MohammedA

[mohammedalfakih-dev]

Summary

• Databricks Job Run URL: (https://adb-7405619530719547.7.azuredatabricks.net/jobs/1042517554723103/runs/latest-success?o=7405619530719547)
• Task 1 (PySpark): Notebook in task-1/ with aggregated borough and payment type queries.
• Task 2 (dbt Incremental): Ported dbt project in task-2/ with materialized='incremental', merge strategy, timing comparison, and DESCRIBE HISTORY proof in WRITEUP.md.
• Task 3 (Job Scheduling): Scheduled Databricks Job from GitHub fork on hyf-dbt-warehouse with screenshots and orchestration comparison in task-3/SCHEDULING.md.

How to review

• Open task-2/WRITEUP.md to check initial vs incremental build times and DESCRIBE HISTORY output.
• Open task-3/SCHEDULING.md to verify the Databricks Job Run URL, screenshots, and Airflow comparison.
• Open AI_ASSIST.md for documented LLM interactions.

Secrets hygiene checklist

• No .env or profiles.yml files committed.
• No Databricks personal access tokens (dapi...) hardcoded in any notebook or SQL file.
• profiles.yml.example and .env.example templates present.

[github-actions]

📝 HackYourFuture auto grade

Assignment Score: 100 / 100 ✅

Status: ✅ Passed
Minimum score to pass: 60
🧪 The auto grade is experimental and still being improved

Test Details

=== Databricks Lab Autograder ===
  ✓ PASS  task-1 notebook: pyspark_exploration.ipynb
  ✓ PASS  task-2/dbt_project.yml
  ✓ PASS  task-2/WRITEUP.md
  ✓ PASS  task-3/SCHEDULING.md
  ✓ PASS  AI_ASSIST.md
  ✓ PASS  Level 1: required files (15/15 pts)
  ✓ PASS  notebook uses show()
  ✓ PASS  notebook addresses borough and/or payment_type
  ✓ PASS  Level 2: PySpark notebook (25/25 pts)
  ✓ PASS  fct_trips incremental materialization
  ✓ PASS  incremental merge config present
  ✓ PASS  WRITEUP explains incremental behavior
  ✓ PASS  WRITEUP references {{ this }} or the boundary filter
  ✓ PASS  Level 3: dbt incremental + WRITEUP (30/30 pts)
  ✓ PASS  task-3/screenshots/ contains 4 screenshot(s)
  ✓ PASS  SCHEDULING.md contains Databricks Job Run URL
  ✓ PASS  SCHEDULING.md contains orchestration comparison
  ✓ PASS  Level 4: Task 3 Job Scheduling (15/15 pts)
  ✓ PASS  AI_ASSIST.md contains documented AI interaction
  ✓ PASS  .gitignore excludes .env and profiles.yml
  ✓ PASS  profiles.yml.example present
  ✓ PASS  Level 5: AI_ASSIST & secrets hygiene (15/15 pts)

Score: 100 / 100  (passing: 60)  pass=true
Total: 100/100 — pass=true (passing threshold: 60)