Halyna R.

AI_ASSIST.md

@@ -1,16 +1,19 @@
-# AI Usage Log
+# AI assistance log
 
-Record at least one point where you used an AI coding assistant (ChatGPT, Claude, Cursor, GitHub Copilot, Gemini, etc.) during this assignment.
+## Where I used an LLM
 
-## Interaction 1
+I used ChatGPT while debugging the local dbt-to-Databricks setup and while
+organising the assignment repository. The main issues discussed were:
 
-- **Tool used:** (e.g. ChatGPT / Cursor / Claude)
-- **Task / Problem:** (e.g. debugging dbt connection profile / writing PySpark join / configuring Job trigger)
-- **Prompt sent:**
-  > `___`
-- **Output provided by AI:**
-  > `___`
-- **What I kept, changed, or rejected, and why:**
-  > `___`
+- the Databricks adapter and `dbt_utils` package were missing;
+- Git Bash converted `/sql/1.0/warehouses/...` into a Windows path;
+- the source schema was incorrectly set to `public` instead of `nyc_yellow`;
+- dbt initially authenticated with another user's token;
+- the assignment repository contained only empty placeholder files.
 
-*(Ensure no personal passwords, Databricks tokens, or unapproved credentials are included in prompts or logged outputs.)*
+## How I checked the advice
+
+I did not accept the suggestions blindly. I checked them with `dbt --version`,
+`dbt deps`, `dbt debug`, `session_user()`, compiled SQL, and the Databricks error
+messages. I also kept all real tokens outside Git and left real timings,
+Job URLs, and screenshots as manual evidence rather than inventing results.

README.md

@@ -1,74 +1,105 @@
-# HackYourFuture Data Track — Week 13 Assignment
+# Week 13 Assignment — Big Data on Databricks
 
-**Databricks Lab:** PySpark exploration, dbt incremental models on Delta Lake, and Git-backed Job scheduling.
+This repository contains Halyna's Week 13 assignment scaffold for:
 
-Full instructions live in the curriculum: [Week 13 Assignment](https://www.notion.so/hackyourfuture/Assignment-2af50f64ffc98112b371c42a3f469749).
+1. PySpark exploration in a Databricks notebook.
+2. A Week 10 dbt project ported to Databricks and Delta Lake.
+3. A Git-backed Databricks Job with a paused schedule.
 
-## Where to start
+The complete Ukrainian translation of the assignment is in [`ASSIGNMENT_UA.md`](ASSIGNMENT_UA.md).
+A file-by-file explanation is in [`CHANGES_UA.md`](CHANGES_UA.md).
 
-| Folder / File | What to submit | Points (autograder) |
-| --- | --- | --- |
-| `task-1/` | PySpark notebook (`show()` on aggregated results, PySpark-vs-dbt note) | 25 |
-| `task-2/` | Ported dbt project + `WRITEUP.md` (timings + incremental explanation + `DESCRIBE HISTORY`) | 30 |
-| `task-3/` | Git-backed Databricks Job + screenshots + `SCHEDULING.md` (Jobs vs Airflow) | 15 |
-| `AI_ASSIST.md` | Documented LLM usage (prompt, tool, kept/discarded rationale) | 15 |
-| Required files | Presence of all required files across `task-1/`, `task-2/`, `task-3/`, and `AI_ASSIST.md` | 15 |
-| Secrets hygiene | No committed secrets (`profiles.yml`, `.env`, tokens) | Blocker if violated |
-
-**Passing score:** 60/100 on the autograder. Your teacher also reviews quality against the rubric (incremental config, `>` boundary, tool-choice writing, Job configuration).
-
-## Repository layout
+## Repository structure
 
 ```text
-data-assignment-week-13/
+c55-data-week-13/
 ├── task-1/
-│   └── pyspark_exploration.ipynb    # or .py export from Databricks
+│   └── pyspark_exploration.py
 ├── task-2/
-│   ├── dbt_project.yml              # your ported Week 10 project
-│   ├── models/                      # your dbt models
+│   ├── dbt_project.yml
+│   ├── packages.yml
 │   ├── profiles.yml.example
-│   └── WRITEUP.md                   # incremental build write-up + DESCRIBE HISTORY
-├── task-3/                          # Git-backed Job scheduling
-│   ├── SCHEDULING.md                # Jobs vs Airflow write-up + Job Run URL
-│   └── screenshots/                 # Job config, green run, paused trigger
-├── task-4/                          # optional bonuses only (create if needed)
+│   ├── pyproject.toml
+│   ├── models/
+│   ├── tests/
+│   └── WRITEUP.md
+├── task-3/
+│   ├── SCHEDULING.md
+│   └── screenshots/
 ├── .env.example
-├── AI_ASSIST.md                     # LLM interaction log
-└── README.md
+├── AI_ASSIST.md
+└── ASSIGNMENT_UA.md
 ```
 
-## Setup
+## What still requires manual evidence
+
+The code and templates are prepared, but these values cannot be invented and must be added after real Databricks runs:
+
+- the output of Task 1;
+- the full-refresh and incremental wall-clock times;
+- the `DESCRIBE HISTORY` result or screenshot;
+- the Databricks Job Run URL;
+- screenshots of Job configuration, a successful run, and the paused schedule.
+
+## Local dbt setup
+
+Run these commands from `task-2/`:
 
 ```bash
-cp .env.example .env          # fill in Databricks connection values
-cd task-2
 cp profiles.yml.example profiles.yml
-export $(grep -v '^#' ../.env | xargs)   # or source manually
-dbt debug
+cp ../.env.example ../.env
+```
+
+Fill in `../.env` with your personal token and correct warehouse values, then load it in Git Bash:
+
+```bash
+set -a
+source ../.env
+set +a
+```
+
+Git Bash can convert `/sql/...` into a Windows path. Prevent that before running dbt:
+
+```bash
+export MSYS2_ENV_CONV_EXCL='DATABRICKS_HTTP_PATH'
 ```
 
-Use Python 3.11 or 3.12 for dbt if `dbt debug` crashes on import (`uvx --python 3.11 --from dbt-databricks dbt debug`).
+Install and check dbt:
+
+```bash
+uv sync
+uv run dbt deps
+uv run dbt debug
+```
 
-## Check your score locally
+First create the staging views once:
 
 ```bash
-bash .hyf/test.sh
-cat .hyf/score.json
+uv run dbt run --select +fct_trips
 ```
 
-## Scoring ladder (autograder)
+Then run the two commands required for the timing comparison:
 
-| Score | What the grader checks |
-| --- | --- |
-| 15 | Required files present (`task-1` notebook, `task-2/dbt_project.yml`, `WRITEUP.md`, `SCHEDULING.md`, `AI_ASSIST.md`) |
-| 25 | Task 1 notebook mentions `show` and borough/payment_type work |
-| 30 | Task 2 has incremental config (`materialized='incremental'`, `merge`, `unique_key`) and a filled `WRITEUP.md` |
-| 15 | Task 3 has screenshots in `task-3/screenshots/`, Job Run URL, and a filled `SCHEDULING.md` |
-| 15 | `AI_ASSIST.md` contains documented prompt and rationale |
-| Pass | Secrets hygiene (no committed `.env` / `profiles.yml` / `dapi` tokens) |
+```bash
+uv run dbt build --select fct_trips --full-refresh
+uv run dbt build --select fct_trips
+```
+
+## Databricks Job configuration
 
-Governance and streaming bonuses are teacher-reviewed only; they do not affect the autograder score.
+Use these values:
 
-## Instructor / track maintainer
+```text
+Job name: dev_halyna_fct_trips
+Git repository: https://github.com/halyna1995/c55-data-week-13.git
+Branch: main
+Project directory: task-2
+Commands:
+  dbt deps
+  dbt build --select fct_trips
+Warehouse: hyf-dbt-warehouse
+Catalog: hyf
+Schema: dev_halyna
+```
 
-This repo is the Week 13 student scaffold. Teacher rubric: `week_13__assignment_rubric.md` in the [datatrack](https://github.com/HackYourFuture/datatrack) curriculum repo (not shared with students).
+After proving the Job works, add a schedule and leave the trigger **Paused**.

task-1/task-1-week-13-halyna.py

@@ -0,0 +1,48 @@
+# Databricks notebook source
+from pyspark.sql import functions as F
+
+trips = spark.read.table("hyf.nyc_yellow.raw_trips")
+zones = spark.read.table("hyf.nyc_yellow.raw_zones").select(
+    "location_id",
+    "borough",
+)
+
+
+# Question 1: Which pickup borough has the most trips?
+pickup_counts = (
+    trips
+    .join(
+        F.broadcast(zones),
+        trips.pickup_location_id == zones.location_id,
+        "left",
+    )
+    .groupBy("borough")
+    .agg(F.count("*").alias("trip_count"))
+)
+
+top_pickup_borough = (
+    pickup_counts
+    .orderBy(F.desc("trip_count"))
+    .limit(1)
+    .select(
+        F.lit("Pickup borough with most trips").alias("metric"),
+        F.coalesce(F.col("borough"), F.lit("Unknown")).alias("category"),
+        F.col("trip_count").cast("double").alias("value"),
+    )
+)
+
+# Question 2: What is the average total_amount per payment_type?
+average_total_by_payment = (
+    trips
+    .groupBy("payment_type")
+    .agg(F.avg("total_amount").alias("avg_total_amount"))
+    .select(
+        F.lit("Average total_amount by payment_type").alias("metric"),
+        F.col("payment_type").cast("string").alias("category"),
+        F.round("avg_total_amount", 2).cast("double").alias("value"),
+    )
+)
+
+# One final action for both answers.
+results = top_pickup_borough.unionByName(average_total_by_payment)
+results.orderBy("metric", "category").show(truncate=False)

task-2/README.md

@@ -1,12 +1,53 @@
-# Task 2: dbt on Databricks
+# Task 2 — dbt on Databricks
 
-Copy your **Week 10 dbt project** into this folder (or start from the `week-13-ch-4-dbt` branch of [nyc-taxi-dbt-reference](https://github.com/lassebenni/nyc-taxi-dbt-reference)).
+## 1. Create local secret files
 
-**Required changes:**
+```bash
+cp profiles.yml.example profiles.yml
+cp ../.env.example ../.env
+```
 
-1. Install `dbt-databricks` locally (`pip install dbt-databricks` or `uv tool install dbt-databricks`).
-2. Copy `profiles.yml.example` to `profiles.yml` (git-ignored) and export the env vars from `.env.example`.
-3. Configure `fct_trips` as `materialized='incremental'` with `incremental_strategy='merge'` and a real `unique_key`.
-4. Run `dbt build --select fct_trips --full-refresh` for the baseline, then `dbt build --select fct_trips` for the incremental rerun. Document both timings plus your explanation in `WRITEUP.md`.
+Fill `../.env` with your own values. Never commit `.env`, `profiles.yml`, or a token.
 
-**Do not commit:** `profiles.yml`, `.env`, or any Databricks token.
+## 2. Load the environment in Git Bash
+
+```bash
+export MSYS2_ENV_CONV_EXCL='DATABRICKS_HTTP_PATH'
+set -a
+source ../.env
+set +a
+```
+
+## 3. Install and verify
+
+```bash
+uv sync
+uv run dbt deps
+uv run dbt debug
+uv run dbt show --inline "select session_user() as dbt_identity"
+```
+
+The identity must be your own Databricks user, not the teacher's account.
+
+## 4. Create dependencies once
+
+```bash
+uv run dbt run --select +fct_trips
+```
+
+## 5. Required timed runs
+
+```bash
+uv run dbt build --select fct_trips --full-refresh
+uv run dbt build --select fct_trips
+```
+
+Copy the real times to `WRITEUP.md`.
+
+## 6. Verify Delta history
+
+```sql
+DESCRIBE HISTORY hyf.dev_halyna.fct_trips;
+```
+
+Save the result or screenshot and reference it in `WRITEUP.md`.

task-2/WRITEUP.md

@@ -1,24 +1,47 @@
-# Task 2 write-up: incremental build timings & Delta history
+# Task 2 — dbt on Databricks and incremental models
 
-Fill in after running `dbt build --select fct_trips --full-refresh` baseline followed by `dbt build --select fct_trips` incremental rerun against Databricks.
+## Connection check
 
-## First build (full / initial load with --full-refresh)
+- Target catalog: `hyf`
+- Target schema: `dev_halyna`
+- SQL warehouse: `hyf-dbt-warehouse`
+- `dbt debug`: PASS
 
-- **Wall-clock time:**
-- **Notes:** (optional: warehouse size, any errors you fixed)
+## Build timings
 
-## Second build (incremental rerun)
+| Run | Command | Real wall-clock time |
+|---|---|---:|
+| Initial full build | `dbt build --select fct_trips --full-refresh` | 2m8.436s |
+| Incremental rerun | `dbt build --select fct_trips` | 1m54.189s |
 
-- **Wall-clock time:**
+The incremental rerun was modestly faster because `is_incremental()` returned
+true after the target Delta table already existed. The filter using `{{ this }}`
+read the maximum existing `pickup_datetime` from the current `fct_trips` table
+and processed only rows with a strictly newer timestamp instead of rebuilding
+the complete trip history.
 
-## Why was the second run faster?
+The incremental boundary uses `>` rather than `>=`. This prevents rows at the
+existing maximum timestamp from being processed repeatedly and avoids duplicate
+boundary records.
 
-Write two or three sentences in your own words (see the assignment for the concepts you must name):
+The measured difference was relatively small because the total wall-clock time
+also includes dbt startup, SQL warehouse execution, Delta `MERGE`, and data
+tests. However, the Delta table history confirms that the second run used the
+incremental `MERGE` strategy rather than performing another full table rebuild.
 
-`___`
+## Delta history proof
 
-## Delta Table History (DESCRIBE HISTORY)
+ The following command was executed in Databricks SQL Editor:
 
-Paste the output or summary of `DESCRIBE HISTORY hyf.dev_yourname.fct_trips` (showing `CREATE OR REPLACE TABLE` and `MERGE` operations) or reference a screenshot:
+```sql
+DESCRIBE HISTORY hyf.dev_halyna.fct_trips;
+```
 
-`___`
+The Delta history shows:
+
+CREATE OR REPLACE TABLE AS SELECT for the full-refresh builds;
+MERGE for the incremental reruns.
+
+Evidence:
+
+- Suggested screenshot path: `screenshots/delta_history.png`.

task-2/dbt_project.yml

@@ -0,0 +1,22 @@
+name: "nyc_taxi"
+version: "1.0.0"
+config-version: 2
+profile: "nyc_taxi"
+
+model-paths: ["models"]
+test-paths: ["tests"]
+macro-paths: ["macros"]
+seed-paths: ["seeds"]
+snapshot-paths: ["snapshots"]
+analysis-paths: ["analyses"]
+
+clean-targets:
+  - "target"
+  - "dbt_packages"
+
+models:
+  nyc_taxi:
+    staging:
+      +materialized: view
+    marts:
+      +materialized: table