diff --git a/AI_ASSIST.md b/AI_ASSIST.md index b134b96..2b3a68b 100644 --- a/AI_ASSIST.md +++ b/AI_ASSIST.md @@ -1,16 +1,19 @@ -# AI Usage Log +# AI assistance log -Record at least one point where you used an AI coding assistant (ChatGPT, Claude, Cursor, GitHub Copilot, Gemini, etc.) during this assignment. +## Where I used an LLM -## Interaction 1 +I used ChatGPT while debugging the local dbt-to-Databricks setup and while +organising the assignment repository. The main issues discussed were: -- **Tool used:** (e.g. ChatGPT / Cursor / Claude) -- **Task / Problem:** (e.g. debugging dbt connection profile / writing PySpark join / configuring Job trigger) -- **Prompt sent:** - > `___` -- **Output provided by AI:** - > `___` -- **What I kept, changed, or rejected, and why:** - > `___` +- the Databricks adapter and `dbt_utils` package were missing; +- Git Bash converted `/sql/1.0/warehouses/...` into a Windows path; +- the source schema was incorrectly set to `public` instead of `nyc_yellow`; +- dbt initially authenticated with another user's token; +- the assignment repository contained only empty placeholder files. -*(Ensure no personal passwords, Databricks tokens, or unapproved credentials are included in prompts or logged outputs.)* +## How I checked the advice + +I did not accept the suggestions blindly. I checked them with `dbt --version`, +`dbt deps`, `dbt debug`, `session_user()`, compiled SQL, and the Databricks error +messages. I also kept all real tokens outside Git and left real timings, +Job URLs, and screenshots as manual evidence rather than inventing results. diff --git a/README.md b/README.md index fd4900b..c527b82 100644 --- a/README.md +++ b/README.md @@ -1,74 +1,105 @@ -# HackYourFuture Data Track — Week 13 Assignment +# Week 13 Assignment — Big Data on Databricks -**Databricks Lab:** PySpark exploration, dbt incremental models on Delta Lake, and Git-backed Job scheduling. +This repository contains Halyna's Week 13 assignment scaffold for: -Full instructions live in the curriculum: [Week 13 Assignment](https://www.notion.so/hackyourfuture/Assignment-2af50f64ffc98112b371c42a3f469749). +1. PySpark exploration in a Databricks notebook. +2. A Week 10 dbt project ported to Databricks and Delta Lake. +3. A Git-backed Databricks Job with a paused schedule. -## Where to start +The complete Ukrainian translation of the assignment is in [`ASSIGNMENT_UA.md`](ASSIGNMENT_UA.md). +A file-by-file explanation is in [`CHANGES_UA.md`](CHANGES_UA.md). -| Folder / File | What to submit | Points (autograder) | -| --- | --- | --- | -| `task-1/` | PySpark notebook (`show()` on aggregated results, PySpark-vs-dbt note) | 25 | -| `task-2/` | Ported dbt project + `WRITEUP.md` (timings + incremental explanation + `DESCRIBE HISTORY`) | 30 | -| `task-3/` | Git-backed Databricks Job + screenshots + `SCHEDULING.md` (Jobs vs Airflow) | 15 | -| `AI_ASSIST.md` | Documented LLM usage (prompt, tool, kept/discarded rationale) | 15 | -| Required files | Presence of all required files across `task-1/`, `task-2/`, `task-3/`, and `AI_ASSIST.md` | 15 | -| Secrets hygiene | No committed secrets (`profiles.yml`, `.env`, tokens) | Blocker if violated | - -**Passing score:** 60/100 on the autograder. Your teacher also reviews quality against the rubric (incremental config, `>` boundary, tool-choice writing, Job configuration). - -## Repository layout +## Repository structure ```text -data-assignment-week-13/ +c55-data-week-13/ ├── task-1/ -│ └── pyspark_exploration.ipynb # or .py export from Databricks +│ └── pyspark_exploration.py ├── task-2/ -│ ├── dbt_project.yml # your ported Week 10 project -│ ├── models/ # your dbt models +│ ├── dbt_project.yml +│ ├── packages.yml │ ├── profiles.yml.example -│ └── WRITEUP.md # incremental build write-up + DESCRIBE HISTORY -├── task-3/ # Git-backed Job scheduling -│ ├── SCHEDULING.md # Jobs vs Airflow write-up + Job Run URL -│ └── screenshots/ # Job config, green run, paused trigger -├── task-4/ # optional bonuses only (create if needed) +│ ├── pyproject.toml +│ ├── models/ +│ ├── tests/ +│ └── WRITEUP.md +├── task-3/ +│ ├── SCHEDULING.md +│ └── screenshots/ ├── .env.example -├── AI_ASSIST.md # LLM interaction log -└── README.md +├── AI_ASSIST.md +└── ASSIGNMENT_UA.md ``` -## Setup +## What still requires manual evidence + +The code and templates are prepared, but these values cannot be invented and must be added after real Databricks runs: + +- the output of Task 1; +- the full-refresh and incremental wall-clock times; +- the `DESCRIBE HISTORY` result or screenshot; +- the Databricks Job Run URL; +- screenshots of Job configuration, a successful run, and the paused schedule. + +## Local dbt setup + +Run these commands from `task-2/`: ```bash -cp .env.example .env # fill in Databricks connection values -cd task-2 cp profiles.yml.example profiles.yml -export $(grep -v '^#' ../.env | xargs) # or source manually -dbt debug +cp ../.env.example ../.env +``` + +Fill in `../.env` with your personal token and correct warehouse values, then load it in Git Bash: + +```bash +set -a +source ../.env +set +a +``` + +Git Bash can convert `/sql/...` into a Windows path. Prevent that before running dbt: + +```bash +export MSYS2_ENV_CONV_EXCL='DATABRICKS_HTTP_PATH' ``` -Use Python 3.11 or 3.12 for dbt if `dbt debug` crashes on import (`uvx --python 3.11 --from dbt-databricks dbt debug`). +Install and check dbt: + +```bash +uv sync +uv run dbt deps +uv run dbt debug +``` -## Check your score locally +First create the staging views once: ```bash -bash .hyf/test.sh -cat .hyf/score.json +uv run dbt run --select +fct_trips ``` -## Scoring ladder (autograder) +Then run the two commands required for the timing comparison: -| Score | What the grader checks | -| --- | --- | -| 15 | Required files present (`task-1` notebook, `task-2/dbt_project.yml`, `WRITEUP.md`, `SCHEDULING.md`, `AI_ASSIST.md`) | -| 25 | Task 1 notebook mentions `show` and borough/payment_type work | -| 30 | Task 2 has incremental config (`materialized='incremental'`, `merge`, `unique_key`) and a filled `WRITEUP.md` | -| 15 | Task 3 has screenshots in `task-3/screenshots/`, Job Run URL, and a filled `SCHEDULING.md` | -| 15 | `AI_ASSIST.md` contains documented prompt and rationale | -| Pass | Secrets hygiene (no committed `.env` / `profiles.yml` / `dapi` tokens) | +```bash +uv run dbt build --select fct_trips --full-refresh +uv run dbt build --select fct_trips +``` + +## Databricks Job configuration -Governance and streaming bonuses are teacher-reviewed only; they do not affect the autograder score. +Use these values: -## Instructor / track maintainer +```text +Job name: dev_halyna_fct_trips +Git repository: https://github.com/halyna1995/c55-data-week-13.git +Branch: main +Project directory: task-2 +Commands: + dbt deps + dbt build --select fct_trips +Warehouse: hyf-dbt-warehouse +Catalog: hyf +Schema: dev_halyna +``` -This repo is the Week 13 student scaffold. Teacher rubric: `week_13__assignment_rubric.md` in the [datatrack](https://github.com/HackYourFuture/datatrack) curriculum repo (not shared with students). +After proving the Job works, add a schedule and leave the trigger **Paused**. diff --git a/task-1/screenshots/task1_pyspark_code.png b/task-1/screenshots/task1_pyspark_code.png new file mode 100644 index 0000000..1d1b00e Binary files /dev/null and b/task-1/screenshots/task1_pyspark_code.png differ diff --git a/task-1/screenshots/task1_pyspark_results.png b/task-1/screenshots/task1_pyspark_results.png new file mode 100644 index 0000000..b5783b3 Binary files /dev/null and b/task-1/screenshots/task1_pyspark_results.png differ diff --git a/task-1/task-1-week-13-halyna.py b/task-1/task-1-week-13-halyna.py new file mode 100644 index 0000000..bf99a69 --- /dev/null +++ b/task-1/task-1-week-13-halyna.py @@ -0,0 +1,48 @@ +# Databricks notebook source +from pyspark.sql import functions as F + +trips = spark.read.table("hyf.nyc_yellow.raw_trips") +zones = spark.read.table("hyf.nyc_yellow.raw_zones").select( + "location_id", + "borough", +) + + +# Question 1: Which pickup borough has the most trips? +pickup_counts = ( + trips + .join( + F.broadcast(zones), + trips.pickup_location_id == zones.location_id, + "left", + ) + .groupBy("borough") + .agg(F.count("*").alias("trip_count")) +) + +top_pickup_borough = ( + pickup_counts + .orderBy(F.desc("trip_count")) + .limit(1) + .select( + F.lit("Pickup borough with most trips").alias("metric"), + F.coalesce(F.col("borough"), F.lit("Unknown")).alias("category"), + F.col("trip_count").cast("double").alias("value"), + ) +) + +# Question 2: What is the average total_amount per payment_type? +average_total_by_payment = ( + trips + .groupBy("payment_type") + .agg(F.avg("total_amount").alias("avg_total_amount")) + .select( + F.lit("Average total_amount by payment_type").alias("metric"), + F.col("payment_type").cast("string").alias("category"), + F.round("avg_total_amount", 2).cast("double").alias("value"), + ) +) + +# One final action for both answers. +results = top_pickup_borough.unionByName(average_total_by_payment) +results.orderBy("metric", "category").show(truncate=False) diff --git a/task-2/README.md b/task-2/README.md index a68eced..748cb5d 100644 --- a/task-2/README.md +++ b/task-2/README.md @@ -1,12 +1,53 @@ -# Task 2: dbt on Databricks +# Task 2 — dbt on Databricks -Copy your **Week 10 dbt project** into this folder (or start from the `week-13-ch-4-dbt` branch of [nyc-taxi-dbt-reference](https://github.com/lassebenni/nyc-taxi-dbt-reference)). +## 1. Create local secret files -**Required changes:** +```bash +cp profiles.yml.example profiles.yml +cp ../.env.example ../.env +``` -1. Install `dbt-databricks` locally (`pip install dbt-databricks` or `uv tool install dbt-databricks`). -2. Copy `profiles.yml.example` to `profiles.yml` (git-ignored) and export the env vars from `.env.example`. -3. Configure `fct_trips` as `materialized='incremental'` with `incremental_strategy='merge'` and a real `unique_key`. -4. Run `dbt build --select fct_trips --full-refresh` for the baseline, then `dbt build --select fct_trips` for the incremental rerun. Document both timings plus your explanation in `WRITEUP.md`. +Fill `../.env` with your own values. Never commit `.env`, `profiles.yml`, or a token. -**Do not commit:** `profiles.yml`, `.env`, or any Databricks token. +## 2. Load the environment in Git Bash + +```bash +export MSYS2_ENV_CONV_EXCL='DATABRICKS_HTTP_PATH' +set -a +source ../.env +set +a +``` + +## 3. Install and verify + +```bash +uv sync +uv run dbt deps +uv run dbt debug +uv run dbt show --inline "select session_user() as dbt_identity" +``` + +The identity must be your own Databricks user, not the teacher's account. + +## 4. Create dependencies once + +```bash +uv run dbt run --select +fct_trips +``` + +## 5. Required timed runs + +```bash +uv run dbt build --select fct_trips --full-refresh +uv run dbt build --select fct_trips +``` + +Copy the real times to `WRITEUP.md`. + +## 6. Verify Delta history + +```sql +DESCRIBE HISTORY hyf.dev_halyna.fct_trips; +``` + +Save the result or screenshot and reference it in `WRITEUP.md`. diff --git a/task-2/WRITEUP.md b/task-2/WRITEUP.md index f3d1eb3..6b6b7d3 100644 --- a/task-2/WRITEUP.md +++ b/task-2/WRITEUP.md @@ -1,24 +1,47 @@ -# Task 2 write-up: incremental build timings & Delta history +# Task 2 — dbt on Databricks and incremental models -Fill in after running `dbt build --select fct_trips --full-refresh` baseline followed by `dbt build --select fct_trips` incremental rerun against Databricks. +## Connection check -## First build (full / initial load with --full-refresh) +- Target catalog: `hyf` +- Target schema: `dev_halyna` +- SQL warehouse: `hyf-dbt-warehouse` +- `dbt debug`: PASS -- **Wall-clock time:** -- **Notes:** (optional: warehouse size, any errors you fixed) +## Build timings -## Second build (incremental rerun) +| Run | Command | Real wall-clock time | +|---|---|---:| +| Initial full build | `dbt build --select fct_trips --full-refresh` | 2m8.436s | +| Incremental rerun | `dbt build --select fct_trips` | 1m54.189s | -- **Wall-clock time:** +The incremental rerun was modestly faster because `is_incremental()` returned +true after the target Delta table already existed. The filter using `{{ this }}` +read the maximum existing `pickup_datetime` from the current `fct_trips` table +and processed only rows with a strictly newer timestamp instead of rebuilding +the complete trip history. -## Why was the second run faster? +The incremental boundary uses `>` rather than `>=`. This prevents rows at the +existing maximum timestamp from being processed repeatedly and avoids duplicate +boundary records. -Write two or three sentences in your own words (see the assignment for the concepts you must name): +The measured difference was relatively small because the total wall-clock time +also includes dbt startup, SQL warehouse execution, Delta `MERGE`, and data +tests. However, the Delta table history confirms that the second run used the +incremental `MERGE` strategy rather than performing another full table rebuild. -`___` +## Delta history proof -## Delta Table History (DESCRIBE HISTORY) + The following command was executed in Databricks SQL Editor: -Paste the output or summary of `DESCRIBE HISTORY hyf.dev_yourname.fct_trips` (showing `CREATE OR REPLACE TABLE` and `MERGE` operations) or reference a screenshot: +```sql +DESCRIBE HISTORY hyf.dev_halyna.fct_trips; +``` -`___` +The Delta history shows: + +CREATE OR REPLACE TABLE AS SELECT for the full-refresh builds; +MERGE for the incremental reruns. + +Evidence: + +- Suggested screenshot path: `screenshots/delta_history.png`. diff --git a/task-2/dbt_project.yml b/task-2/dbt_project.yml new file mode 100644 index 0000000..198d9c9 --- /dev/null +++ b/task-2/dbt_project.yml @@ -0,0 +1,22 @@ +name: "nyc_taxi" +version: "1.0.0" +config-version: 2 +profile: "nyc_taxi" + +model-paths: ["models"] +test-paths: ["tests"] +macro-paths: ["macros"] +seed-paths: ["seeds"] +snapshot-paths: ["snapshots"] +analysis-paths: ["analyses"] + +clean-targets: + - "target" + - "dbt_packages" + +models: + nyc_taxi: + staging: + +materialized: view + marts: + +materialized: table diff --git a/task-2/models/marts/_fct_trips.yml b/task-2/models/marts/_fct_trips.yml new file mode 100644 index 0000000..41e29b5 --- /dev/null +++ b/task-2/models/marts/_fct_trips.yml @@ -0,0 +1,19 @@ +version: 2 + +models: + - name: fct_trips + description: Incremental NYC yellow taxi mart enriched with pickup and dropoff zones. + columns: + - name: trip_id + description: Surrogate key used by the Delta merge strategy. + data_tests: + - not_null + - unique + - name: pickup_datetime + description: Time when the trip started. + data_tests: + - not_null + - name: pickup_borough + description: Borough of the pickup zone. + - name: dropoff_borough + description: Borough of the dropoff zone. diff --git a/task-2/models/marts/fct_trips.sql b/task-2/models/marts/fct_trips.sql new file mode 100644 index 0000000..5adeb22 --- /dev/null +++ b/task-2/models/marts/fct_trips.sql @@ -0,0 +1,36 @@ +{{ + config( + materialized='incremental', + incremental_strategy='merge', + unique_key='trip_id' + ) +}} + +select + t.trip_id, + t.pickup_datetime, + t.dropoff_datetime, + t.fare_amount, + t.tip_amount, + t.trip_distance, + t.total_amount, + t.trip_duration_minutes, + t.tip_pct, + t.fare_per_mile, + t.payment_type_label, + pz.borough as pickup_borough, + pz.zone as pickup_zone, + dz.borough as dropoff_borough, + dz.zone as dropoff_zone +from {{ ref('stg_trips') }} t +left join {{ ref('stg_zones') }} pz + on t.pickup_location_id = pz.location_id +left join {{ ref('stg_zones') }} dz + on t.dropoff_location_id = dz.location_id + +{% if is_incremental() %} + where t.pickup_datetime > ( + select max(pickup_datetime) + from {{ this }} + ) +{% endif %} diff --git a/task-2/models/staging/_sources.yml b/task-2/models/staging/_sources.yml new file mode 100644 index 0000000..2eb23ab --- /dev/null +++ b/task-2/models/staging/_sources.yml @@ -0,0 +1,12 @@ +version: 2 + +sources: + - name: nyc_taxi + description: Raw NYC yellow taxi data in Databricks Unity Catalog. + database: hyf + schema: nyc_yellow + tables: + - name: raw_trips + description: Yellow taxi trips for 2023-2025, approximately 128 million rows. + - name: raw_zones + description: Lookup table mapping taxi location IDs to boroughs and zones. diff --git a/task-2/models/staging/_stg_trips.yml b/task-2/models/staging/_stg_trips.yml new file mode 100644 index 0000000..42e33b7 --- /dev/null +++ b/task-2/models/staging/_stg_trips.yml @@ -0,0 +1,47 @@ +version: 2 + +models: + - name: stg_trips + description: Cleaned yellow taxi trips with one row per generated trip_id. + data_tests: + - dbt_utils.unique_combination_of_columns: + arguments: + combination_of_columns: + - pickup_datetime + - dropoff_datetime + - pickup_location_id + - fare_amount + config: + severity: warn + columns: + - name: trip_id + description: Surrogate key generated from stable trip fields. + data_tests: + - not_null + - unique + + - name: pickup_datetime + description: Time when the trip started. + data_tests: + - not_null + + - name: pickup_location_id + description: TLC zone ID where the trip started. + data_tests: + - not_null + - relationships: + arguments: + to: ref('stg_zones') + field: location_id + config: + severity: warn + + - name: payment_type + description: TLC payment code. The large dataset also contains code 0. + data_tests: + - not_null: + config: + severity: warn + - accepted_values: + arguments: + values: [0, 1, 2, 3, 4, 5, 6] diff --git a/task-2/models/staging/_stg_zones.yml b/task-2/models/staging/_stg_zones.yml new file mode 100644 index 0000000..47156ad --- /dev/null +++ b/task-2/models/staging/_stg_zones.yml @@ -0,0 +1,13 @@ +version: 2 + +models: + - name: stg_zones + description: One row per TLC taxi zone. + columns: + - name: location_id + description: TLC taxi-zone identifier. + data_tests: + - unique + - not_null + - name: borough + description: NYC borough name. diff --git a/task-2/models/staging/stg_trips.sql b/task-2/models/staging/stg_trips.sql new file mode 100644 index 0000000..908d302 --- /dev/null +++ b/task-2/models/staging/stg_trips.sql @@ -0,0 +1,63 @@ +{{ config(materialized='view') }} + +{% set payment_types = { + 0: 'Flex Fare / unknown', + 1: 'Credit card', + 2: 'Cash', + 3: 'No charge', + 4: 'Dispute', + 5: 'Unknown', + 6: 'Voided trip' +} %} + +with base as ( + select + {{ dbt_utils.generate_surrogate_key([ + 'vendor_id', + 'pickup_datetime', + 'dropoff_datetime', + 'pickup_location_id', + 'dropoff_location_id', + 'fare_amount', + 'trip_distance', + 'total_amount', + 'passenger_count' + ]) }} as trip_id, + pickup_datetime, + dropoff_datetime, + pickup_location_id, + dropoff_location_id, + fare_amount, + tip_amount, + trip_distance, + total_amount, + payment_type, + case + when fare_amount > 0 then round(tip_amount / fare_amount, 4) + else null + end as tip_pct, + case + when trip_distance > 0 then round(fare_amount / trip_distance, 4) + else null + end as fare_per_mile, + case payment_type + {% for code, label in payment_types.items() %} + when {{ code }} then '{{ label }}' + {% endfor %} + else 'Other' + end as payment_type_label, + round( + (unix_timestamp(dropoff_datetime) - unix_timestamp(pickup_datetime)) / 60.0, + 2 + ) as trip_duration_minutes + from {{ source('nyc_taxi', 'raw_trips') }} + where pickup_location_id is not null + and fare_amount >= 0 +) + +select * +from base +qualify row_number() over ( + partition by trip_id + order by pickup_datetime +) = 1 diff --git a/task-2/models/staging/stg_zones.sql b/task-2/models/staging/stg_zones.sql new file mode 100644 index 0000000..1e98897 --- /dev/null +++ b/task-2/models/staging/stg_zones.sql @@ -0,0 +1,6 @@ +select + location_id, + borough, + zone, + service_zone +from {{ source('nyc_taxi', 'raw_zones') }} diff --git a/task-2/package-lock.yml b/task-2/package-lock.yml new file mode 100644 index 0000000..f683b32 --- /dev/null +++ b/task-2/package-lock.yml @@ -0,0 +1,5 @@ +packages: + - name: dbt_utils + package: dbt-labs/dbt_utils + version: 1.4.1 +sha1_hash: e6424ba9e5a22487e47f023803aa4f0411946808 diff --git a/task-2/packages.yml b/task-2/packages.yml new file mode 100644 index 0000000..b26b1ff --- /dev/null +++ b/task-2/packages.yml @@ -0,0 +1,3 @@ +packages: + - package: dbt-labs/dbt_utils + version: [">=1.1.0", "<2.0.0"] diff --git a/task-2/pyproject.toml b/task-2/pyproject.toml new file mode 100644 index 0000000..2892a22 --- /dev/null +++ b/task-2/pyproject.toml @@ -0,0 +1,8 @@ +[project] +name = "week-13-dbt-databricks" +version = "0.1.0" +description = "HYF Week 13 dbt project for Databricks" +requires-python = ">=3.11,<3.14" +dependencies = [ + "dbt-databricks>=1.10,<1.11", +] diff --git a/task-2/screenshots/delta_history.png b/task-2/screenshots/delta_history.png new file mode 100644 index 0000000..279ddee Binary files /dev/null and b/task-2/screenshots/delta_history.png differ diff --git a/task-2/tests/assert_fare_amount_non_negative.sql b/task-2/tests/assert_fare_amount_non_negative.sql new file mode 100644 index 0000000..0b6fbe5 --- /dev/null +++ b/task-2/tests/assert_fare_amount_non_negative.sql @@ -0,0 +1,5 @@ +select + pickup_datetime, + fare_amount +from {{ ref('stg_trips') }} +where fare_amount < 0 diff --git a/task-2/tests/assert_pickup_before_dropoff.sql b/task-2/tests/assert_pickup_before_dropoff.sql new file mode 100644 index 0000000..8ebb206 --- /dev/null +++ b/task-2/tests/assert_pickup_before_dropoff.sql @@ -0,0 +1,8 @@ +{{ config(severity='warn') }} + +select + pickup_datetime, + dropoff_datetime, + pickup_location_id +from {{ ref('stg_trips') }} +where pickup_datetime > dropoff_datetime diff --git a/task-3/SCHEDULING.md b/task-3/SCHEDULING.md index 2780686..26b6f7f 100644 --- a/task-3/SCHEDULING.md +++ b/task-3/SCHEDULING.md @@ -1,23 +1,47 @@ -# Task 3 Write-up: Git-backed Job Scheduling +# Task 3 — Git-backed Databricks Job -## Databricks Job Run URL +## Job configuration -Paste the URL of your successful Job run from the Databricks UI address bar: +- Job name: `dev_halyna_fct_trips` +- Task name: `dbt_fct_trips` +- Source: Git provider +- Repository: `https://github.com/halyna1995/c55-data-week-13.git` +- Branch: `main` +- Project directory: `task-2` +- SQL warehouse: `hyf-dbt-warehouse` +- Catalog: `hyf` +- Schema: `dev_halyna` +- Commands: -`___` +```text +dbt deps +dbt build --select fct_trips +``` -## Screenshots +## Job Run URL -Ensure the following screenshot files exist in `task-3/screenshots/`: +https://adb-7405619530719547.7.azuredatabricks.net/jobs/872113922216601?o=7405619530719547 -1. `job_config.png` — Showing the dbt task configuration with Git repository URL, branch `main`, path `task-2`, and warehouse `hyf-dbt-warehouse`. -2. `job_run_success.png` — Showing a successful run log with a green checkmark and stdout execution output. -3. `job_schedule_paused.png` — Showing the scheduled trigger set to **Paused**. +The Job completed successfully. The incremental fct_trips model was created, +and all three dbt data tests passed. The final output showed: +PASS=4 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=4 -## Orchestration Comparison +## Required screenshots -### When would you choose Databricks Jobs versus Apache Airflow for pipeline orchestration? +- [ ] [`screenshots/job_config.png`](screenshots/job_config.png) +- [ ] [`screenshots/job_success.png`](screenshots/job_success.png) +- [ ] [`screenshots/paused_schedule.png`](screenshots/paused_schedule.png) -Write two to three sentences comparing Databricks Jobs and Apache Airflow in your own words: -`___` +## Databricks Jobs versus Airflow + +I would use a Databricks Job when the workflow is Databricks-native, for example +a dbt task, SQL task, or notebook that reads and writes Unity Catalog tables. +I would use Airflow when one pipeline must coordinate several external systems, +for example an API, Azure Storage, Postgres, Databricks, and notifications. + +## Schedule safety + +A daily schedule was added and immediately paused. This demonstrates that the +Job can run automatically while preventing unnecessary use of the shared class +compute budget. diff --git a/task-3/screenshots/README.md b/task-3/screenshots/README.md new file mode 100644 index 0000000..b8c13a3 --- /dev/null +++ b/task-3/screenshots/README.md @@ -0,0 +1,8 @@ +# Required Job screenshots + +Save the real images with exactly these names: + +1. `job_task_configuration.png` — Git URL, branch, `task-2` project directory, + dbt commands, warehouse, catalog, and schema. +2. `job_successful_run.png` — green run and dbt output. +3. `job_paused_schedule.png` — schedule visible with status Paused. diff --git a/task-3/screenshots/job_config.png b/task-3/screenshots/job_config.png new file mode 100644 index 0000000..75a2c98 Binary files /dev/null and b/task-3/screenshots/job_config.png differ diff --git a/task-3/screenshots/job_success.png b/task-3/screenshots/job_success.png new file mode 100644 index 0000000..d13216e Binary files /dev/null and b/task-3/screenshots/job_success.png differ diff --git a/task-3/screenshots/job_success_overview.png b/task-3/screenshots/job_success_overview.png new file mode 100644 index 0000000..11694b6 Binary files /dev/null and b/task-3/screenshots/job_success_overview.png differ diff --git a/task-3/screenshots/paused_schedule.png b/task-3/screenshots/paused_schedule.png new file mode 100644 index 0000000..964b292 Binary files /dev/null and b/task-3/screenshots/paused_schedule.png differ diff --git a/task-4/BONUSES.md b/task-4/BONUSES.md new file mode 100644 index 0000000..d292c38 --- /dev/null +++ b/task-4/BONUSES.md @@ -0,0 +1,128 @@ +I completed the following optional bonuses: + +- [x] Easy — Workflows alerting +- [x] Medium — Governance +- [x] Harder — Structured Streaming +- [ ] Harder+ — Local PySpark module and pytest + +## Easy — Workflows alerting + +I configured an email notification for failures of the +`dev_halyna_fct_trips` Databricks Job. + +This notification allows pipeline failures to be detected automatically without +manually checking the Databricks Jobs page after every scheduled run. + +![Job failure notification](screenshots/job_notification.png) + +## Medium — Governance + +### Table privileges + +I inspected the access privileges for the table +`hyf.dev_halyna.fct_trips` using the following SQL command: + +```sql +SHOW GRANTS ON TABLE hyf.dev_halyna.fct_trips; + +The result shows the privileges visible to my current Databricks identity. The +displayed privileges are inherited from the hyf.dev_halyna schema. + +## Unity Catalog lineage + +I inspected the Unity Catalog lineage graph for +hyf.dev_halyna.fct_trips. + +The graph shows that the final fct_trips table depends on the upstream dbt +models: + +stg_trips +stg_zones + +This lineage information helps identify where the data came from and which +upstream objects affect the final analytical table. + +## Example PII tag + +The following statement shows how a hypothetical customer e-mail column could +be tagged as sensitive personally identifiable information: + +ALTER TABLE hyf.dev_halyna.fct_trips +ALTER COLUMN hypothetical_customer_email +SET TAGS ( + 'pii' = 'true', + 'classification' = 'sensitive' +); + +This command was documented but not executed because +hypothetical_customer_email is not a real column in the fct_trips table. + +## Harder — Structured Streaming + +I created a Databricks notebook and used the Spark rate source to generate +streaming rows. + +The source generated five rows per second. Each generated record contained: + +timestamp +value + +I also added a derived column: + +value_squared + +The streaming output was written to the following Delta table: + +hyf.dev_halyna.rate_stream_demo + +A Unity Catalog Volume was used for the Structured Streaming checkpoint because +the public DBFS root was not available in the shared workspace. + +## Streaming dashboard + +The streaming dashboard showed the input rate, processing rate, and micro-batch +duration while the streaming query was active. + +## Delta table result + +During the demonstration, the stream generated 2,185 rows. The resulting Delta +table contained the generated timestamp, value, and value_squared +columns. + +## Stopping the stream + +After verifying the result, I explicitly stopped the streaming query: + +query.stop() + +I confirmed that the query was no longer active: + +Streaming query active: False + +Stopping the query was important because an active streaming process would +continue generating data and consuming shared compute resources. + +I also stopped the idle Spark compute after completing the streaming +demonstration. + +## Harder+ — Local PySpark module and pytest + +I created a local PySpark project containing two pure transformation functions: + +calculation of trip counts by pickup borough; +calculation of the average total_amount by payment_type. + +The project also contains pytest tests based on small local fixture DataFrames. +The tests do not read Unity Catalog tables and do not connect to Databricks. + +The local project structure was created successfully, and the required Python +dependencies were installed. However, the tests were not completed successfully. + +PySpark requires a local Java Development Kit because the Apache Spark engine +runs in the Java Virtual Machine. The local Spark session could not start because +Java was not configured, resulting in this error: + +[JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number. + +Because there is no successful green pytest result, I do not claim this bonus +as completed. \ No newline at end of file diff --git a/task-4/pyspark_app/pyproject.toml b/task-4/pyspark_app/pyproject.toml new file mode 100644 index 0000000..27d0fdf --- /dev/null +++ b/task-4/pyspark_app/pyproject.toml @@ -0,0 +1,18 @@ +[project] +name = "week13-pyspark-app" +version = "0.1.0" +requires-python = ">=3.11,<3.13" +dependencies = [ + "pyspark>=3.5,<4.0", +] + +[dependency-groups] +dev = [ + "numpy==1.26.4", + "pandas>=2.0,<3.0", + "pyarrow>=12,<20", + "pytest>=8,<9", +] + +[tool.pytest.ini_options] +pythonpath = ["src"] diff --git a/task-4/pyspark_app/src/pyspark_app/__init__.py b/task-4/pyspark_app/src/pyspark_app/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/task-4/pyspark_app/src/pyspark_app/transform.py b/task-4/pyspark_app/src/pyspark_app/transform.py new file mode 100644 index 0000000..0b56d57 --- /dev/null +++ b/task-4/pyspark_app/src/pyspark_app/transform.py @@ -0,0 +1,39 @@ +from pyspark.sql import DataFrame +from pyspark.sql import functions as F + + +def pickup_borough_counts( + trips: DataFrame, + zones: DataFrame, +) -> DataFrame: + """Count trips for every pickup borough.""" + + zone_lookup = zones.select("location_id", "borough") + + return ( + trips.join( + F.broadcast(zone_lookup), + trips.pickup_location_id == zone_lookup.location_id, + "left", + ) + .groupBy("borough") + .agg(F.count("*").alias("trip_count")) + .orderBy(F.desc("trip_count")) + ) + + +def average_total_by_payment( + trips: DataFrame, +) -> DataFrame: + """Calculate average total amount per payment type.""" + + return ( + trips.groupBy("payment_type") + .agg( + F.round( + F.avg("total_amount"), + 2, + ).alias("avg_total_amount") + ) + .orderBy("payment_type") + ) diff --git a/task-4/pyspark_app/tests/conftest.py b/task-4/pyspark_app/tests/conftest.py new file mode 100644 index 0000000..1771db1 --- /dev/null +++ b/task-4/pyspark_app/tests/conftest.py @@ -0,0 +1,21 @@ +import os + +os.environ.setdefault("PYARROW_IGNORE_TIMEZONE", "1") + +import pytest +from pyspark.sql import SparkSession + + +@pytest.fixture(scope="session") +def spark() -> SparkSession: + session = ( + SparkSession.builder + .master("local[2]") + .appName("week13-pyspark-tests") + .config("spark.ui.enabled", "false") + .getOrCreate() + ) + + yield session + + session.stop() \ No newline at end of file diff --git a/task-4/pyspark_app/tests/test_transform.py b/task-4/pyspark_app/tests/test_transform.py new file mode 100644 index 0000000..0585ee3 --- /dev/null +++ b/task-4/pyspark_app/tests/test_transform.py @@ -0,0 +1,72 @@ +from pyspark.testing import assertDataFrameEqual + +from pyspark_app.transform import ( + average_total_by_payment, + pickup_borough_counts, +) + + +def test_pickup_borough_counts(spark): + """Count trips for every pickup borough.""" + trips = spark.createDataFrame( + [ + (1,), + (1,), + (2,), + (3,), + ], + "pickup_location_id integer", + ) + + zones = spark.createDataFrame( + [ + (1, "Manhattan"), + (2, "Queens"), + (3, "Manhattan"), + ], + "location_id integer, borough string", + ) + + actual = pickup_borough_counts(trips, zones) + + expected = spark.createDataFrame( + [ + ("Manhattan", 3), + ("Queens", 1), + ], + "borough string, trip_count long", + ) + + assertDataFrameEqual( + actual, + expected, + checkRowOrder=True, + ) + + +def test_average_total_by_payment(spark): + """Calculate average total amount per payment type.""" + trips = spark.createDataFrame( + [ + (1, 10.0), + (1, 20.0), + (2, 30.0), + ], + "payment_type integer, total_amount double", + ) + + actual = average_total_by_payment(trips) + + expected = spark.createDataFrame( + [ + (1, 15.0), + (2, 30.0), + ], + "payment_type integer, avg_total_amount double", + ) + + assertDataFrameEqual( + actual, + expected, + checkRowOrder=True, + ) diff --git a/task-4/pyspark_app/uv.lock b/task-4/pyspark_app/uv.lock new file mode 100644 index 0000000..3f2af98 --- /dev/null +++ b/task-4/pyspark_app/uv.lock @@ -0,0 +1,226 @@ +version = 1 +revision = 3 +requires-python = ">=3.11, <3.13" +resolution-markers = [ + "python_full_version >= '3.12'", + "python_full_version < '3.12'", +] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "iniconfig" +version = "2.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, +] + +[[package]] +name = "numpy" +version = "1.26.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/65/6e/09db70a523a96d25e115e71cc56a6f9031e7b8cd166c1ac8438307c14058/numpy-1.26.4.tar.gz", hash = "sha256:2a02aba9ed12e4ac4eb3ea9421c420301a0c6460d9830d74a9df87efa4912010", size = 15786129, upload-time = "2024-02-06T00:26:44.495Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/11/57/baae43d14fe163fa0e4c47f307b6b2511ab8d7d30177c491960504252053/numpy-1.26.4-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:4c66707fabe114439db9068ee468c26bbdf909cac0fb58686a42a24de1760c71", size = 20630554, upload-time = "2024-02-05T23:51:50.149Z" }, + { url = "https://files.pythonhosted.org/packages/1a/2e/151484f49fd03944c4a3ad9c418ed193cfd02724e138ac8a9505d056c582/numpy-1.26.4-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:edd8b5fe47dab091176d21bb6de568acdd906d1887a4584a15a9a96a1dca06ef", size = 13997127, upload-time = "2024-02-05T23:52:15.314Z" }, + { url = "https://files.pythonhosted.org/packages/79/ae/7e5b85136806f9dadf4878bf73cf223fe5c2636818ba3ab1c585d0403164/numpy-1.26.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7ab55401287bfec946ced39700c053796e7cc0e3acbef09993a9ad2adba6ca6e", size = 14222994, upload-time = "2024-02-05T23:52:47.569Z" }, + { url = "https://files.pythonhosted.org/packages/3a/d0/edc009c27b406c4f9cbc79274d6e46d634d139075492ad055e3d68445925/numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:666dbfb6ec68962c033a450943ded891bed2d54e6755e35e5835d63f4f6931d5", size = 18252005, upload-time = "2024-02-05T23:53:15.637Z" }, + { url = "https://files.pythonhosted.org/packages/09/bf/2b1aaf8f525f2923ff6cfcf134ae5e750e279ac65ebf386c75a0cf6da06a/numpy-1.26.4-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:96ff0b2ad353d8f990b63294c8986f1ec3cb19d749234014f4e7eb0112ceba5a", size = 13885297, upload-time = "2024-02-05T23:53:42.16Z" }, + { url = "https://files.pythonhosted.org/packages/df/a0/4e0f14d847cfc2a633a1c8621d00724f3206cfeddeb66d35698c4e2cf3d2/numpy-1.26.4-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:60dedbb91afcbfdc9bc0b1f3f402804070deed7392c23eb7a7f07fa857868e8a", size = 18093567, upload-time = "2024-02-05T23:54:11.696Z" }, + { url = "https://files.pythonhosted.org/packages/d2/b7/a734c733286e10a7f1a8ad1ae8c90f2d33bf604a96548e0a4a3a6739b468/numpy-1.26.4-cp311-cp311-win32.whl", hash = "sha256:1af303d6b2210eb850fcf03064d364652b7120803a0b872f5211f5234b399f20", size = 5968812, upload-time = "2024-02-05T23:54:26.453Z" }, + { url = "https://files.pythonhosted.org/packages/3f/6b/5610004206cf7f8e7ad91c5a85a8c71b2f2f8051a0c0c4d5916b76d6cbb2/numpy-1.26.4-cp311-cp311-win_amd64.whl", hash = "sha256:cd25bcecc4974d09257ffcd1f098ee778f7834c3ad767fe5db785be9a4aa9cb2", size = 15811913, upload-time = "2024-02-05T23:54:53.933Z" }, + { url = "https://files.pythonhosted.org/packages/95/12/8f2020a8e8b8383ac0177dc9570aad031a3beb12e38847f7129bacd96228/numpy-1.26.4-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:b3ce300f3644fb06443ee2222c2201dd3a89ea6040541412b8fa189341847218", size = 20335901, upload-time = "2024-02-05T23:55:32.801Z" }, + { url = "https://files.pythonhosted.org/packages/75/5b/ca6c8bd14007e5ca171c7c03102d17b4f4e0ceb53957e8c44343a9546dcc/numpy-1.26.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:03a8c78d01d9781b28a6989f6fa1bb2c4f2d51201cf99d3dd875df6fbd96b23b", size = 13685868, upload-time = "2024-02-05T23:55:56.28Z" }, + { url = "https://files.pythonhosted.org/packages/79/f8/97f10e6755e2a7d027ca783f63044d5b1bc1ae7acb12afe6a9b4286eac17/numpy-1.26.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9fad7dcb1aac3c7f0584a5a8133e3a43eeb2fe127f47e3632d43d677c66c102b", size = 13925109, upload-time = "2024-02-05T23:56:20.368Z" }, + { url = "https://files.pythonhosted.org/packages/0f/50/de23fde84e45f5c4fda2488c759b69990fd4512387a8632860f3ac9cd225/numpy-1.26.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:675d61ffbfa78604709862923189bad94014bef562cc35cf61d3a07bba02a7ed", size = 17950613, upload-time = "2024-02-05T23:56:56.054Z" }, + { url = "https://files.pythonhosted.org/packages/4c/0c/9c603826b6465e82591e05ca230dfc13376da512b25ccd0894709b054ed0/numpy-1.26.4-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:ab47dbe5cc8210f55aa58e4805fe224dac469cde56b9f731a4c098b91917159a", size = 13572172, upload-time = "2024-02-05T23:57:21.56Z" }, + { url = "https://files.pythonhosted.org/packages/76/8c/2ba3902e1a0fc1c74962ea9bb33a534bb05984ad7ff9515bf8d07527cadd/numpy-1.26.4-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:1dda2e7b4ec9dd512f84935c5f126c8bd8b9f2fc001e9f54af255e8c5f16b0e0", size = 17786643, upload-time = "2024-02-05T23:57:56.585Z" }, + { url = "https://files.pythonhosted.org/packages/28/4a/46d9e65106879492374999e76eb85f87b15328e06bd1550668f79f7b18c6/numpy-1.26.4-cp312-cp312-win32.whl", hash = "sha256:50193e430acfc1346175fcbdaa28ffec49947a06918b7b92130744e81e640110", size = 5677803, upload-time = "2024-02-05T23:58:08.963Z" }, + { url = "https://files.pythonhosted.org/packages/16/2e/86f24451c2d530c88daf997cb8d6ac622c1d40d19f5a031ed68a4b73a374/numpy-1.26.4-cp312-cp312-win_amd64.whl", hash = "sha256:08beddf13648eb95f8d867350f6a018a4be2e5ad54c8d8caed89ebca558b2818", size = 15517754, upload-time = "2024-02-05T23:58:36.364Z" }, +] + +[[package]] +name = "packaging" +version = "26.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d7/f1/e7a6dd94a8d4a5626c03e4e99c87f241ba9e350cd9e6d75123f992427270/packaging-26.2.tar.gz", hash = "sha256:ff452ff5a3e828ce110190feff1178bb1f2ea2281fa2075aadb987c2fb221661", size = 228134, upload-time = "2026-04-24T20:15:23.917Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl", hash = "sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e", size = 100195, upload-time = "2026-04-24T20:15:22.081Z" }, +] + +[[package]] +name = "pandas" +version = "2.3.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, + { name = "python-dateutil" }, + { name = "pytz" }, + { name = "tzdata" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/33/01/d40b85317f86cf08d853a4f495195c73815fdf205eef3993821720274518/pandas-2.3.3.tar.gz", hash = "sha256:e05e1af93b977f7eafa636d043f9f94c7ee3ac81af99c13508215942e64c993b", size = 4495223, upload-time = "2025-09-29T23:34:51.853Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c1/fa/7ac648108144a095b4fb6aa3de1954689f7af60a14cf25583f4960ecb878/pandas-2.3.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:602b8615ebcc4a0c1751e71840428ddebeb142ec02c786e8ad6b1ce3c8dec523", size = 11578790, upload-time = "2025-09-29T23:18:30.065Z" }, + { url = "https://files.pythonhosted.org/packages/9b/35/74442388c6cf008882d4d4bdfc4109be87e9b8b7ccd097ad1e7f006e2e95/pandas-2.3.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:8fe25fc7b623b0ef6b5009149627e34d2a4657e880948ec3c840e9402e5c1b45", size = 10833831, upload-time = "2025-09-29T23:38:56.071Z" }, + { url = "https://files.pythonhosted.org/packages/fe/e4/de154cbfeee13383ad58d23017da99390b91d73f8c11856f2095e813201b/pandas-2.3.3-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b468d3dad6ff947df92dcb32ede5b7bd41a9b3cceef0a30ed925f6d01fb8fa66", size = 12199267, upload-time = "2025-09-29T23:18:41.627Z" }, + { url = "https://files.pythonhosted.org/packages/bf/c9/63f8d545568d9ab91476b1818b4741f521646cbdd151c6efebf40d6de6f7/pandas-2.3.3-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b98560e98cb334799c0b07ca7967ac361a47326e9b4e5a7dfb5ab2b1c9d35a1b", size = 12789281, upload-time = "2025-09-29T23:18:56.834Z" }, + { url = "https://files.pythonhosted.org/packages/f2/00/a5ac8c7a0e67fd1a6059e40aa08fa1c52cc00709077d2300e210c3ce0322/pandas-2.3.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:1d37b5848ba49824e5c30bedb9c830ab9b7751fd049bc7914533e01c65f79791", size = 13240453, upload-time = "2025-09-29T23:19:09.247Z" }, + { url = "https://files.pythonhosted.org/packages/27/4d/5c23a5bc7bd209231618dd9e606ce076272c9bc4f12023a70e03a86b4067/pandas-2.3.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:db4301b2d1f926ae677a751eb2bd0e8c5f5319c9cb3f88b0becbbb0b07b34151", size = 13890361, upload-time = "2025-09-29T23:19:25.342Z" }, + { url = "https://files.pythonhosted.org/packages/8e/59/712db1d7040520de7a4965df15b774348980e6df45c129b8c64d0dbe74ef/pandas-2.3.3-cp311-cp311-win_amd64.whl", hash = "sha256:f086f6fe114e19d92014a1966f43a3e62285109afe874f067f5abbdcbb10e59c", size = 11348702, upload-time = "2025-09-29T23:19:38.296Z" }, + { url = "https://files.pythonhosted.org/packages/9c/fb/231d89e8637c808b997d172b18e9d4a4bc7bf31296196c260526055d1ea0/pandas-2.3.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:6d21f6d74eb1725c2efaa71a2bfc661a0689579b58e9c0ca58a739ff0b002b53", size = 11597846, upload-time = "2025-09-29T23:19:48.856Z" }, + { url = "https://files.pythonhosted.org/packages/5c/bd/bf8064d9cfa214294356c2d6702b716d3cf3bb24be59287a6a21e24cae6b/pandas-2.3.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:3fd2f887589c7aa868e02632612ba39acb0b8948faf5cc58f0850e165bd46f35", size = 10729618, upload-time = "2025-09-29T23:39:08.659Z" }, + { url = "https://files.pythonhosted.org/packages/57/56/cf2dbe1a3f5271370669475ead12ce77c61726ffd19a35546e31aa8edf4e/pandas-2.3.3-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ecaf1e12bdc03c86ad4a7ea848d66c685cb6851d807a26aa245ca3d2017a1908", size = 11737212, upload-time = "2025-09-29T23:19:59.765Z" }, + { url = "https://files.pythonhosted.org/packages/e5/63/cd7d615331b328e287d8233ba9fdf191a9c2d11b6af0c7a59cfcec23de68/pandas-2.3.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b3d11d2fda7eb164ef27ffc14b4fcab16a80e1ce67e9f57e19ec0afaf715ba89", size = 12362693, upload-time = "2025-09-29T23:20:14.098Z" }, + { url = "https://files.pythonhosted.org/packages/a6/de/8b1895b107277d52f2b42d3a6806e69cfef0d5cf1d0ba343470b9d8e0a04/pandas-2.3.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:a68e15f780eddf2b07d242e17a04aa187a7ee12b40b930bfdd78070556550e98", size = 12771002, upload-time = "2025-09-29T23:20:26.76Z" }, + { url = "https://files.pythonhosted.org/packages/87/21/84072af3187a677c5893b170ba2c8fbe450a6ff911234916da889b698220/pandas-2.3.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:371a4ab48e950033bcf52b6527eccb564f52dc826c02afd9a1bc0ab731bba084", size = 13450971, upload-time = "2025-09-29T23:20:41.344Z" }, + { url = "https://files.pythonhosted.org/packages/86/41/585a168330ff063014880a80d744219dbf1dd7a1c706e75ab3425a987384/pandas-2.3.3-cp312-cp312-win_amd64.whl", hash = "sha256:a16dcec078a01eeef8ee61bf64074b4e524a2a3f4b3be9326420cabe59c4778b", size = 10992722, upload-time = "2025-09-29T23:20:54.139Z" }, +] + +[[package]] +name = "pluggy" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, +] + +[[package]] +name = "py4j" +version = "0.10.9.9" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/38/31/0b210511177070c8d5d3059556194352e5753602fa64b85b7ab81ec1a009/py4j-0.10.9.9.tar.gz", hash = "sha256:f694cad19efa5bd1dee4f3e5270eb406613c974394035e5bfc4ec1aba870b879", size = 761089, upload-time = "2025-01-15T03:53:18.624Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/bd/db/ea0203e495be491c85af87b66e37acfd3bf756fd985f87e46fc5e3bf022c/py4j-0.10.9.9-py2.py3-none-any.whl", hash = "sha256:c7c26e4158defb37b0bb124933163641a2ff6e3a3913f7811b0ddbe07ed61533", size = 203008, upload-time = "2025-01-15T03:53:15.648Z" }, +] + +[[package]] +name = "pyarrow" +version = "19.0.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7f/09/a9046344212690f0632b9c709f9bf18506522feb333c894d0de81d62341a/pyarrow-19.0.1.tar.gz", hash = "sha256:3bf266b485df66a400f282ac0b6d1b500b9d2ae73314a153dbe97d6d5cc8a99e", size = 1129437, upload-time = "2025-02-18T18:55:57.027Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/55/f1a8d838ec07fe3ca53edbe76f782df7b9aafd4417080eebf0b42aab0c52/pyarrow-19.0.1-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:cc55d71898ea30dc95900297d191377caba257612f384207fe9f8293b5850f90", size = 30713987, upload-time = "2025-02-18T18:52:20.463Z" }, + { url = "https://files.pythonhosted.org/packages/13/12/428861540bb54c98a140ae858a11f71d041ef9e501e6b7eb965ca7909505/pyarrow-19.0.1-cp311-cp311-macosx_12_0_x86_64.whl", hash = "sha256:7a544ec12de66769612b2d6988c36adc96fb9767ecc8ee0a4d270b10b1c51e00", size = 32135613, upload-time = "2025-02-18T18:52:25.29Z" }, + { url = "https://files.pythonhosted.org/packages/2f/8a/23d7cc5ae2066c6c736bce1db8ea7bc9ac3ef97ac7e1c1667706c764d2d9/pyarrow-19.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0148bb4fc158bfbc3d6dfe5001d93ebeed253793fff4435167f6ce1dc4bddeae", size = 41149147, upload-time = "2025-02-18T18:52:30.975Z" }, + { url = "https://files.pythonhosted.org/packages/a2/7a/845d151bb81a892dfb368bf11db584cf8b216963ccce40a5cf50a2492a18/pyarrow-19.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f24faab6ed18f216a37870d8c5623f9c044566d75ec586ef884e13a02a9d62c5", size = 42178045, upload-time = "2025-02-18T18:52:36.859Z" }, + { url = "https://files.pythonhosted.org/packages/a7/31/e7282d79a70816132cf6cae7e378adfccce9ae10352d21c2fecf9d9756dd/pyarrow-19.0.1-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:4982f8e2b7afd6dae8608d70ba5bd91699077323f812a0448d8b7abdff6cb5d3", size = 40532998, upload-time = "2025-02-18T18:52:42.578Z" }, + { url = "https://files.pythonhosted.org/packages/b8/82/20f3c290d6e705e2ee9c1fa1d5a0869365ee477e1788073d8b548da8b64c/pyarrow-19.0.1-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:49a3aecb62c1be1d822f8bf629226d4a96418228a42f5b40835c1f10d42e4db6", size = 42084055, upload-time = "2025-02-18T18:52:48.749Z" }, + { url = "https://files.pythonhosted.org/packages/ff/77/e62aebd343238863f2c9f080ad2ef6ace25c919c6ab383436b5b81cbeef7/pyarrow-19.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:008a4009efdb4ea3d2e18f05cd31f9d43c388aad29c636112c2966605ba33466", size = 25283133, upload-time = "2025-02-18T18:52:54.549Z" }, + { url = "https://files.pythonhosted.org/packages/78/b4/94e828704b050e723f67d67c3535cf7076c7432cd4cf046e4bb3b96a9c9d/pyarrow-19.0.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:80b2ad2b193e7d19e81008a96e313fbd53157945c7be9ac65f44f8937a55427b", size = 30670749, upload-time = "2025-02-18T18:53:00.062Z" }, + { url = "https://files.pythonhosted.org/packages/7e/3b/4692965e04bb1df55e2c314c4296f1eb12b4f3052d4cf43d29e076aedf66/pyarrow-19.0.1-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:ee8dec072569f43835932a3b10c55973593abc00936c202707a4ad06af7cb294", size = 32128007, upload-time = "2025-02-18T18:53:06.581Z" }, + { url = "https://files.pythonhosted.org/packages/22/f7/2239af706252c6582a5635c35caa17cb4d401cd74a87821ef702e3888957/pyarrow-19.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4d5d1ec7ec5324b98887bdc006f4d2ce534e10e60f7ad995e7875ffa0ff9cb14", size = 41144566, upload-time = "2025-02-18T18:53:11.958Z" }, + { url = "https://files.pythonhosted.org/packages/fb/e3/c9661b2b2849cfefddd9fd65b64e093594b231b472de08ff658f76c732b2/pyarrow-19.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f3ad4c0eb4e2a9aeb990af6c09e6fa0b195c8c0e7b272ecc8d4d2b6574809d34", size = 42202991, upload-time = "2025-02-18T18:53:17.678Z" }, + { url = "https://files.pythonhosted.org/packages/fe/4f/a2c0ed309167ef436674782dfee4a124570ba64299c551e38d3fdaf0a17b/pyarrow-19.0.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:d383591f3dcbe545f6cc62daaef9c7cdfe0dff0fb9e1c8121101cabe9098cfa6", size = 40507986, upload-time = "2025-02-18T18:53:26.263Z" }, + { url = "https://files.pythonhosted.org/packages/27/2e/29bb28a7102a6f71026a9d70d1d61df926887e36ec797f2e6acfd2dd3867/pyarrow-19.0.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:b4c4156a625f1e35d6c0b2132635a237708944eb41df5fbe7d50f20d20c17832", size = 42087026, upload-time = "2025-02-18T18:53:33.063Z" }, + { url = "https://files.pythonhosted.org/packages/16/33/2a67c0f783251106aeeee516f4806161e7b481f7d744d0d643d2f30230a5/pyarrow-19.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:5bd1618ae5e5476b7654c7b55a6364ae87686d4724538c24185bbb2952679960", size = 25250108, upload-time = "2025-02-18T18:53:38.462Z" }, +] + +[[package]] +name = "pygments" +version = "2.20.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" }, +] + +[[package]] +name = "pyspark" +version = "3.5.9" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "py4j" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/95/ce/81e53e729790556e3983e95de1a7d5df91a34adfcd34b5a5ab0e0c6e9b33/pyspark-3.5.9.tar.gz", hash = "sha256:ea27adc39ddac9413b8951e45aa748cbed6c785971b81386efc41938f6243d93", size = 317858593, upload-time = "2026-07-16T08:52:04.494Z" } + +[[package]] +name = "pytest" +version = "8.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "iniconfig" }, + { name = "packaging" }, + { name = "pluggy" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a3/5c/00a0e072241553e1a7496d638deababa67c5058571567b92a7eaa258397c/pytest-8.4.2.tar.gz", hash = "sha256:86c0d0b93306b961d58d62a4db4879f27fe25513d4b969df351abdddb3c30e01", size = 1519618, upload-time = "2025-09-04T14:34:22.711Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a8/a4/20da314d277121d6534b3a980b29035dcd51e6744bd79075a6ce8fa4eb8d/pytest-8.4.2-py3-none-any.whl", hash = "sha256:872f880de3fc3a5bdc88a11b39c9710c3497a547cfa9320bc3c5e62fbf272e79", size = 365750, upload-time = "2025-09-04T14:34:20.226Z" }, +] + +[[package]] +name = "python-dateutil" +version = "2.9.0.post0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "six" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/66/c0/0c8b6ad9f17a802ee498c46e004a0eb49bc148f2fd230864601a86dcf6db/python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 342432, upload-time = "2024-03-01T18:36:20.211Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" }, +] + +[[package]] +name = "pytz" +version = "2026.3.post1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/fb/48/fb042503b6ca6cd271261dc559fd6432f7d8c713153e9ec5c591af4dfc1c/pytz-2026.3.post1.tar.gz", hash = "sha256:2211d3fcf9a797d3405cac96ac7f61d80e6a644f72a3309607282fe8a2010c5d", size = 319745, upload-time = "2026-07-25T15:12:07.385Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0f/7b/39c34ca613b0b198cb866466651b26b045e2009864c5183c979a3b83f383/pytz-2026.3.post1-py2.py3-none-any.whl", hash = "sha256:dd95840dd199baea12d9cc096a1d452caa6596a1c1e4b5f3dbd1541855d5e815", size = 508283, upload-time = "2026-07-25T15:12:05.782Z" }, +] + +[[package]] +name = "six" +version = "1.17.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031, upload-time = "2024-12-04T17:35:28.174Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, +] + +[[package]] +name = "tzdata" +version = "2026.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/92/ff/5a28bdfd8c3ebec42564ac7d0e54ca3db65044a9314a97f9564fa7a1e926/tzdata-2026.3.tar.gz", hash = "sha256:4a1518b8993086a7982523e071643f3c0e5f213e75b21318e78bcabfff9d1415", size = 198674, upload-time = "2026-07-10T08:50:37.887Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e5/6d/b53b99a9f2766d095985947a5782f1702cabb129a34f7a802d7197af832f/tzdata-2026.3-py2.py3-none-any.whl", hash = "sha256:dc096730c87af6cab1b171c9d532be840741ff5d459015e7f6947bd7d7e54931", size = 348168, upload-time = "2026-07-10T08:50:36.46Z" }, +] + +[[package]] +name = "week13-pyspark-app" +version = "0.1.0" +source = { virtual = "." } +dependencies = [ + { name = "pyspark" }, +] + +[package.dev-dependencies] +dev = [ + { name = "numpy" }, + { name = "pandas" }, + { name = "pyarrow" }, + { name = "pytest" }, +] + +[package.metadata] +requires-dist = [{ name = "pyspark", specifier = ">=3.5,<4.0" }] + +[package.metadata.requires-dev] +dev = [ + { name = "numpy", specifier = "==1.26.4" }, + { name = "pandas", specifier = ">=2.0,<3.0" }, + { name = "pyarrow", specifier = ">=12,<20" }, + { name = "pytest", specifier = ">=8,<9" }, +] diff --git a/task-4/screenshots/job_notification.png b/task-4/screenshots/job_notification.png new file mode 100644 index 0000000..a155b01 Binary files /dev/null and b/task-4/screenshots/job_notification.png differ diff --git a/task-4/screenshots/lineage(3).png b/task-4/screenshots/lineage(3).png new file mode 100644 index 0000000..cc3d27c Binary files /dev/null and b/task-4/screenshots/lineage(3).png differ diff --git a/task-4/screenshots/show_grants.png b/task-4/screenshots/show_grants.png new file mode 100644 index 0000000..5759d2e Binary files /dev/null and b/task-4/screenshots/show_grants.png differ diff --git a/task-4/screenshots/streaming_dashboard.png b/task-4/screenshots/streaming_dashboard.png new file mode 100644 index 0000000..6a16283 Binary files /dev/null and b/task-4/screenshots/streaming_dashboard.png differ diff --git a/task-4/screenshots/streaming_result.png b/task-4/screenshots/streaming_result.png new file mode 100644 index 0000000..9ccb8dc Binary files /dev/null and b/task-4/screenshots/streaming_result.png differ diff --git a/task-4/screenshots/streaming_stopped.png b/task-4/screenshots/streaming_stopped.png new file mode 100644 index 0000000..ce84ad4 Binary files /dev/null and b/task-4/screenshots/streaming_stopped.png differ