Baraah A.

[thebaraah]

What I built

• Airflow DAG: dags/taxi_pipeline.py (schedule, ingest_taxi_month → dbt_run → dbt_test, partition from logical date, retries)
• Week 10 dbt project dropped into include/dbt_project/
• Backfill + idempotency evidence (Task 5)
• Operational notes: RUNBOOK.md
• Assignment report: ASSIGNMENT_REPORT.md
• (Target tier) namespaced DAG deployed to the shared class Airflow
• AI usage: AI_ASSIST.md

How to review

• DAG: read dags/taxi_pipeline.py (schedule, task order, catchup=False, retries, partition date).
• Evidence: backfill / idempotency notes in ASSIGNMENT_REPORT.md; operations in RUNBOOK.md.
• AI usage: AI_ASSIST.md.

How to run

From a clean clone, with the Astro CLI installed:

astro dev start
astro dev pytest tests/test_dag_integrity.py --args "-v"

Then open the UI URL, add the azure_pg connection (Admin → Connections), unpause taxi_pipeline, and trigger a run for a real month (e.g. 2024-01-01).

Prerequisite: your Week 10 dbt project in include/dbt_project/ and the azure_pg connection to the shared Postgres.

What reviewers should see (expected results)

Fill in what your run actually produces:

• test_dag_integrity.py result: <e.g. all pass>
• Task order in the Graph view: <e.g. ingest → dbt_run → dbt_test>
• Backfill: <e.g. 7 runs green, re-run is idempotent (same row count)>
• Deployed DAG name (Target tier): <e.g. taxi_pipeline_>

Known limitations / out of scope

• <e.g. Target-tier deploy not attempted; retries set to 2>
• Write "none" if everything in the assignment is done and working.

Self-check

• bash .hyf/test.sh passes
• astro dev pytest tests/test_dag_integrity.py passes
• catchup=False and retries are set
• No credentials committed (connections added in the UI, not in code)
• Backfill / idempotency evidence is in ASSIGNMENT_REPORT.md

[lassebenni]

Reopening to re-trigger grade workflow after checkout fix on main.

[lassebenni]

Autograder note (teacher)

The CI score is a static smoke test only (file presence + code patterns + docs/screenshot files). It does not grade:

• whether screenshots show a real green Graph / Grid / task log / shared-UI run
• before/after backfill row counts
• shared-Airflow deploy proof (merged PR URL + tagged screenshot)

Autograder 100 ≠ assignment pass. Teachers review those from your PR against the Week 12 deliverables. Please make sure ASSIGNMENT_REPORT.md is fully filled (no TODOs), commit at least 3 screenshot images, and include shared-deploy evidence when the class VM is up.

[github-actions]

📝 HackYourFuture auto grade

Assignment Score: 95 / 100 ✅

Status: ✅ Passed
Minimum score to pass: 60
🧪 The auto grade is experimental and still being improved

Test Details

=== Week 12 Autograder — Orchestrated Pipeline ===
  ✓ PASS  found dags/taxi_pipeline.py
  ✓ PASS  found tests/test_dag_integrity.py
  ✓ PASS  found requirements.txt
  ✓ PASS  found RUNBOOK.md
  ✓ PASS  found ASSIGNMENT_REPORT.md
  ✓ PASS  found AI_ASSIST.md
  ✓ PASS  Level 1: required files (20/20 pts)
  ✓ PASS  dags/taxi_pipeline.py: no NotImplementedError stubs left
  ✓ PASS  dags/taxi_pipeline.py: defines a @dag with tasks
  ✓ PASS  Level 2: DAG implemented (15/15 pts)
  ✓ PASS  dags/taxi_pipeline.py: ingest, dbt_run, and dbt_test all present
  ✓ PASS  dags/taxi_pipeline.py: tasks are chained with >>
  ✓ PASS  Level 3: sequential tasks (20/20 pts)
  ✓ PASS  dags/taxi_pipeline.py: dbt runs through uvx (works on the image's Python 3.14)
  ✓ PASS  dags/taxi_pipeline.py: retry behaviour configured (retries=...)
  ✓ PASS  Level 4: uvx dbt + retries (20/20 pts)
  ✓ PASS  dags/taxi_pipeline.py: partition derived from the logical date
  ✗ FAIL  dags/taxi_pipeline.py: max_active_runs not found — set max_active_runs=1 on the @dag decorator (Gotcha #6); CLI --max-active-runs alone is not enough
  ✓ PASS  Level 5: parameterized runs (10/15 pts)
  ✓ PASS  RUNBOOK.md: filled in (1531 chars, no TODO left)
  ✓ PASS  AI_ASSIST.md: filled in (945 chars, no TODO left)
  ✓ PASS  ASSIGNMENT_REPORT.md: filled in (1540 chars, no TODO left)
  ✓ PASS  screenshots: found 4 image file(s) (need ≥3 for Graph + Grid/run + task log)
  ✓ PASS  Level 6: documentation + screenshots (10/10 pts)

Score: 95 / 100  (passing: 60)  pass=true

Reminder: screenshot *content*, shared-Airflow deploy proof, and before/after
row counts are still teacher-reviewed. Autograder green is not a pass — a
high static score is necessary but not sufficient.

[qiraahmad]

The DAG parses successfully and is also able to perform a backfill so that's good, but you need to add max_active_runs=1 on the @dag decorator itself — CLI --max-active-runs alone is not enough (concurrent dbt_run tasks collide on __dbt_backup). Also clean up the leftover starter TODO comments in dags/taxi_pipeline.py.

Please finish the “How to trigger” section in RUNBOOK.md (it currently stops mid-code-block after astro dev start) and use the real DAG id baraah_taxi_pipeline in backfill / ops commands instead of taxi_pipeline.

Please re-run the same 7-month window (with max_active_runs=1 on @dag + CLI) and paste identical per-month counts from airflow_baraah.raw_trips before and after. Also consider filtering the parquet to the logical month before load so adjacent-month spill does not accumulate on re-runs.

Good job! 90/100

[qiraahmad]

Requesting changes for Week 12: add max_active_runs=1 on @dag, and document identical before/after backfill row counts for the same 7-month range.