Mohamad Bader AA

[noneeeed]

Summary

• Databricks Job Run URL: https://adb-7405619530719547.7.azuredatabricks.net/jobs/282713845764221/runs/984569826693564?o=7405619530719547___`

Screenshots`

• Task 1 (PySpark): Notebook in task-1/ with aggregated borough and payment type queries.
• Task 2 (dbt Incremental): Ported dbt project in task-2/ with materialized='incremental', merge strategy, timing comparison, and DESCRIBE HISTORY proof in WRITEUP.md.
• Task 3 (Job Scheduling): Scheduled Databricks Job from GitHub fork on hyf-dbt-warehouse with screenshots and orchestration comparison in task-3/SCHEDULING.md.

How to review

• Open task-2/WRITEUP.md to check initial vs incremental build times and DESCRIBE HISTORY output.
• Open task-3/SCHEDULING.md to verify the Databricks Job Run URL, screenshots, and Airflow comparison.
• Open AI_ASSIST.md for documented LLM interactions.

Secrets hygiene checklist

• [✅ ] No .env or profiles.yml files committed.
• [✅] No Databricks personal access tokens (dapi...) hardcoded in any notebook or SQL file.
• [✅ ] profiles.yml.example and .env.example templates present.

[github-actions]

📝 HackYourFuture auto grade

Assignment Score: 100 / 100 ✅

Status: ✅ Passed
Minimum score to pass: 60
🧪 The auto grade is experimental and still being improved

Test Details

=== Databricks Lab Autograder ===
  ✓ PASS  task-1 notebook: pyspark_exploration.ipynb
  ✓ PASS  task-2/dbt_project.yml
  ✓ PASS  task-2/WRITEUP.md
  ✓ PASS  task-3/SCHEDULING.md
  ✓ PASS  AI_ASSIST.md
  ✓ PASS  Level 1: required files (15/15 pts)
  ✓ PASS  notebook uses show()
  ✓ PASS  notebook addresses borough and/or payment_type
  ✓ PASS  Level 2: PySpark notebook (25/25 pts)
  ✓ PASS  fct_trips incremental materialization
  ✓ PASS  incremental merge config present
  ✓ PASS  WRITEUP explains incremental behavior
  ✓ PASS  WRITEUP references {{ this }} or the boundary filter
  ✓ PASS  Level 3: dbt incremental + WRITEUP (30/30 pts)
  ✓ PASS  task-3/screenshots/ contains 3 screenshot(s)
  ✓ PASS  SCHEDULING.md contains Databricks Job Run URL
  ✓ PASS  SCHEDULING.md contains orchestration comparison
  ✓ PASS  Level 4: Task 3 Job Scheduling (15/15 pts)
  ✓ PASS  AI_ASSIST.md contains documented AI interaction
  ✓ PASS  .gitignore excludes .env and profiles.yml
  ✓ PASS  profiles.yml.example present
  ✓ PASS  Level 5: AI_ASSIST & secrets hygiene (15/15 pts)

Score: 100 / 100  (passing: 60)  pass=true
Total: 100/100 — pass=true (passing threshold: 60)

[jrnhofman]

Total amount per payment type is missing!

[jrnhofman]

is this supposed to be in the notebook? As your missing question is here and not in the notebook :)

[jrnhofman]

this table was not needed but nice that you did anyway!

[jrnhofman]

There is also an objective comparison, it's made here: https://hub.hackyourfuture.nl/Scheduling-dbt-Jobs-40fd6b3bc5a04242af1c72c9ad6b82af look at the section 'Why schedule here (on the platform)' for a detailed overview. Thanks for sharing your opinion though!