Pavel T.

[pavel-tisner]

Summary

• Databricks Job Run URL: <paste-your-job-run-url-here>
• Task 1 (PySpark): Notebook in task-1/ with aggregated borough and payment type queries.
• Task 2 (dbt Incremental): Ported dbt project in task-2/ with materialized='incremental', merge strategy, timing comparison, and DESCRIBE HISTORY proof in WRITEUP.md.
• Task 3 (Job Scheduling): Scheduled Databricks Job from GitHub fork on hyf-dbt-warehouse with screenshots and orchestration comparison in task-3/SCHEDULING.md.

How to review

• Open task-2/WRITEUP.md to check initial vs incremental build times and DESCRIBE HISTORY output.
• Open task-3/SCHEDULING.md to verify the Databricks Job Run URL, screenshots, and Airflow comparison.
• Open AI_ASSIST.md for documented LLM interactions.

Secrets hygiene checklist

• No .env or profiles.yml files committed.
• No Databricks personal access tokens (dapi...) hardcoded in any notebook or SQL file.
• profiles.yml.example and .env.example templates present.

[github-actions]

📝 HackYourFuture auto grade

Assignment Score: 80 / 100 ✅

Status: ✅ Passed
Minimum score to pass: 60
🧪 The auto grade is experimental and still being improved

Test Details

=== Databricks Lab Autograder ===
  ✓ PASS  task-1 notebook: pyspark_exploration.ipynb
  ✓ PASS  task-2/dbt_project.yml
  ✓ PASS  task-2/WRITEUP.md
  ✓ PASS  task-3/SCHEDULING.md
  ✓ PASS  AI_ASSIST.md
  ✓ PASS  Level 1: required files (15/15 pts)
  ✓ PASS  notebook uses show()
  ✓ PASS  notebook addresses borough and/or payment_type
  ✓ PASS  Level 2: PySpark notebook (25/25 pts)
  ✓ PASS  fct_trips incremental materialization
  ✓ PASS  incremental merge config present
  ✓ PASS  WRITEUP explains incremental behavior
  ✓ PASS  WRITEUP references {{ this }} or the boundary filter
  ✓ PASS  Level 3: dbt incremental + WRITEUP (30/30 pts)
  ✗ FAIL  task-3/screenshots/ should contain screenshots of Job config, green run, and paused schedule
  ⚠ WARN  SCHEDULING.md missing Databricks Job Run URL (paste URL from address bar)
  ✗ FAIL  SCHEDULING.md should explain when to use Databricks Jobs vs Airflow
  ✓ PASS  Level 4: Task 3 Job Scheduling (0/15 pts)
  ⚠ WARN  AI_ASSIST.md needs your documented prompt and rationale
  ✓ PASS  .gitignore excludes .env and profiles.yml
  ✓ PASS  profiles.yml.example present
  ✓ PASS  Level 5: AI_ASSIST & secrets hygiene (10/15 pts)

Score: 80 / 100  (passing: 60)  pass=true
Total: 80/100 — pass=true (passing threshold: 60)