diff --git a/AI_ASSIST.md b/AI_ASSIST.md index b134b96..81d7bf9 100644 --- a/AI_ASSIST.md +++ b/AI_ASSIST.md @@ -4,13 +4,33 @@ Record at least one point where you used an AI coding assistant (ChatGPT, Claude ## Interaction 1 -- **Tool used:** (e.g. ChatGPT / Cursor / Claude) -- **Task / Problem:** (e.g. debugging dbt connection profile / writing PySpark join / configuring Job trigger) +- **Tool used:** (gemeni flash-lite) +- **Task / Problem:** debugging dbt connection profile - **Prompt sent:** - > `___` + > ERROR:databricks.sql.backend.thrift_backend:ThriftBackend.attempt_request: Exception: + +10:19:34 Databricks adapter: Connection(session-id=Unknown) - Exception while trying to create connection: Error during request to server. + +Error properties: attempt=1/30, bounded-retry-delay=None, elapsed-seconds=0.07423233985900879/900.0, error-message=, http-code=404, method=OpenSession, no-retry-reason=non-retryable error, original-exception=, query-id=None, session-id=None + +10:19:34 Connection test: [ERROR] + + +10:19:34 1 check failed: + +10:19:34 dbt was unable to connect to the specified database. + +The database returned the following error: + + + >Database Error + + Database Error + + Error during request to server. - **Output provided by AI:** - > `___` + > An HTTP 404 (OpenSession not found) error during a dbt debug connection test means that the Databricks SQL driver successfully reached your server hostname, but the specific HTTP Path (http_path) pointing to your SQL Warehouse or cluster is invalid, mistyped, or pointing to a resource that no longer exists.How to Fix ItCheck your profiles.yml file (usually located in ~/.dbt/profiles.yml) and inspect the host and http_path settings. 1. Check for Common Syntax Errors in profiles.ymlNo https:// in the host: The host field should just be the domain name.❌ Incorrect: host: [https://adb-123456789.12.azuredatabricks.net](https://adb-123456789.12.azuredatabricks.net)✔️ Correct: host: adb-123456789.12.azuredatabricks.netMissing Leading Slash (/) in HTTP Path: The http_path must always begin with a forward slash. ❌ Incorrect: http_path: sql/protocolv1/o/...✔️ Correct: http_path: /sql/protocolv1/o/...2. Re-copy the HTTP Path from DatabricksIt is very easy to accidentally miss a character or copy an old cluster path that has been terminated.Log into your Databricks Workspace.Go to SQL Warehouses and click on the warehouse you want to use.Go to the Connection Details tab. Copy the HTTP Path fresh and paste it directly into your profiles.yml.3. Verify the SQL Warehouse is RunningIf the SQL Warehouse is currently Stopped, the connection will fail. Make sure the warehouse status is running/started in the Databricks UI before executing dbt debug. - **What I kept, changed, or rejected, and why:** - > `___` + > after some back and forth it was a naming mistake inside the sql schema not the actual env schema changed it and everything worked *(Ensure no personal passwords, Databricks tokens, or unapproved credentials are included in prompts or logged outputs.)* diff --git a/task-1/README.md b/task-1/README.md index 569d9a9..c978605 100644 --- a/task-1/README.md +++ b/task-1/README.md @@ -12,3 +12,31 @@ Add your Databricks notebook here. - Two or three sentences on when you would choose PySpark versus dbt SQL. See the [Week 13 assignment](https://www.notion.so/hackyourfuture/Assignment-2af50f64ffc98112b371c42a3f469749) for full requirements. + +**reference:** +from pyspark.sql import functions as F + +trips = spark.read.table("hyf.nyc_yellow.raw_trips") +zones = spark.read.table("hyf.nyc_yellow.raw_zones") + +**Question 1:** + +borough_counts = ( + trips + .join(zones, trips.pickup_location_id == zones.location_id) + .groupBy("borough") + .agg(F.count("*").alias("trip_count")) + .orderBy(F.col("trip_count").desc()) +) + +borough_counts.show(1) + +**Question 2:** +payment_avg = ( + trips + .groupBy("payment_type") + .agg(F.round(F.avg("total_amount"), 2).alias("avg_total_amount")) + .orderBy("payment_type") +) + +payment_avg.show() diff --git a/task-1/pyspark_exploration.ipynb b/task-1/pyspark_exploration.ipynb new file mode 100644 index 0000000..d8454e9 --- /dev/null +++ b/task-1/pyspark_exploration.ipynb @@ -0,0 +1,100 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": {}, + "finishTime": 1785326358620, + "inputWidgets": {}, + "nuid": "ff04faac-5cf0-4fe7-bfcd-6abc9f8f2a16", + "showTitle": false, + "startTime": 1785326358084, + "submitTime": 1785326357672, + "tableResultSettingsMap": {}, + "title": "" + } + }, + "outputs": [], + "source": [ + "from pyspark.sql import functions as F\n", + "\n", + "trips = spark.read.table(\"hyf.nyc_yellow.raw_trips\")\n", + "zones = spark.read.table(\"hyf.nyc_yellow.raw_zones\").select(\n", + " \"location_id\",\n", + " \"borough\",\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "application/vnd.databricks.v1+cell": { + "cellMetadata": { + "byteLimit": 2048000, + "rowLimit": 10000 + }, + "finishTime": 1785327406693, + "inputWidgets": {}, + "nuid": "78a135c8-e36c-4292-babd-f6d97f97cd5c", + "showTitle": false, + "startTime": 1785327404237, + "submitTime": 1785327403902, + "tableResultSettingsMap": {}, + "title": "" + } + }, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "output_type": "stream", + "text": [ + "+---------+----------+\n| borough|trip_count|\n+---------+----------+\n|Manhattan| 112028489|\n+---------+----------+\nonly showing top 1 row\n" + ] + } + ], + "source": [ + "borough_counts = (\n", + " trips\n", + " .join(zones, trips.pickup_location_id == zones.location_id)\n", + " .groupBy(\"borough\")\n", + " .agg(F.count(\"*\").alias(\"trip_count\"))\n", + " .orderBy(F.col(\"trip_count\").desc())\n", + ")\n", + "\n", + "borough_counts.show(1)" + ] + } + ], + "metadata": { + "application/vnd.databricks.v1+notebook": { + "computePreferences": null, + "dashboards": [], + "environmentMetadata": { + "base_environment": "", + "environment_version": "5" + }, + "inputWidgetPreferences": null, + "language": "python", + "notebookMetadata": { + "mostRecentlyExecutedCommandWithImplicitDF": { + "commandId": -1, + "dataframes": [ + "_sqldf" + ] + }, + "pythonIndentUnit": 4 + }, + "notebookName": "New Notebook 2026-07-29 13:54:03", + "widgets": {} + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} \ No newline at end of file diff --git a/task-2/.gitignore b/task-2/.gitignore new file mode 100644 index 0000000..5f1af3c --- /dev/null +++ b/task-2/.gitignore @@ -0,0 +1,165 @@ +# dbt +target/ +dbt_packages/ +logs/ +profiles.yml +.user.yml + +# System files +.DS_Store +Thumbs.db +[Dd]esktop.ini + +# hyf +.hyf/score.json + +# Editor and IDE settings +.vscode/ +.idea/ +*.iml +*.code-workspace +*.sublime-project +*.sublime-workspace +.history/ +.ionide/ + +# Logs +logs +*.log +npm-debug.log* +yarn-debug.log* +yarn-error.log* +lerna-debug.log* + +# Diagnostic reports (https://nodejs.org/api/report.html) +report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json + +# Runtime data +pids +*.pid +*.seed +*.pid.lock + +# Directory for instrumented libs generated by jscoverage/JSCover +lib-cov + +# Coverage directory used by tools like istanbul +coverage +*.lcov + +# nyc test coverage +.nyc_output + +# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files) +.grunt + +# Bower dependency directory (https://bower.io/) +bower_components + +# node-waf configuration +.lock-wscript + +# Compiled binary addons (https://nodejs.org/api/addons.html) +build/Release + +# Dependency directories +node_modules/ +jspm_packages/ + +# Snowpack dependency directory (https://snowpack.dev/) +web_modules/ + +# TypeScript cache +*.tsbuildinfo + +# Optional npm cache directory +.npm + +# Optional eslint cache +.eslintcache + +# Optional stylelint cache +.stylelintcache + +# Optional REPL history +.node_repl_history + +# Output of 'npm pack' +*.tgz + +# Yarn Integrity file +.yarn-integrity + +# dotenv environment variable files +.env +.env.* +!.env.example + +# parcel-bundler cache (https://parceljs.org/) +.cache +.parcel-cache + +# Next.js build output +.next +out + +# Nuxt.js build / generate output +.nuxt +dist + +# Gatsby files +.cache/ +# Comment in the public line in if your project uses Gatsby and not Next.js +# https://nextjs.org/blog/next-9-1#public-directory-support +# public + +# vuepress build output +.vuepress/dist + +# vuepress v2.x temp and cache directory +.temp +.cache + +# Sveltekit cache directory +.svelte-kit/ + +# vitepress build output +**/.vitepress/dist + +# vitepress cache directory +**/.vitepress/cache + +# Docusaurus cache and generated files +.docusaurus + +# Serverless directories +.serverless/ + +# FuseBox cache +.fusebox/ + +# DynamoDB Local files +.dynamodb/ + +# Firebase cache directory +.firebase/ + +# TernJS port file +.tern-port + +# Stores VSCode versions used for testing VSCode extensions +.vscode-test + +# yarn v3 +.pnp.* +.yarn/* +!.yarn/patches +!.yarn/plugins +!.yarn/releases +!.yarn/sdks +!.yarn/versions + +# Vite logs files +vite.config.js.timestamp-* +vite.config.ts.timestamp-* + diff --git a/task-2/WRITEUP.md b/task-2/WRITEUP.md index f3d1eb3..051c8d5 100644 --- a/task-2/WRITEUP.md +++ b/task-2/WRITEUP.md @@ -4,21 +4,30 @@ Fill in after running `dbt build --select fct_trips --full-refresh` baseline fol ## First build (full / initial load with --full-refresh) -- **Wall-clock time:** +- **Wall-clock time: 13:51** +94.80s - **Notes:** (optional: warehouse size, any errors you fixed) ## Second build (incremental rerun) -- **Wall-clock time:** - +- **Wall-clock time:13:53** +81.36s + ## Why was the second run faster? -Write two or three sentences in your own words (see the assignment for the concepts you must name): - +Write two or three sentences in your own words (see the assignment for the concepts you must name): is_incremental() speeds up the process because if the table exists it wont rebuild the entire table from scratch and if its true {{this}} filter starts to work and check if there is new inserts to be modified or added `___` ## Delta Table History (DESCRIBE HISTORY) Paste the output or summary of `DESCRIBE HISTORY hyf.dev_yourname.fct_trips` (showing `CREATE OR REPLACE TABLE` and `MERGE` operations) or reference a screenshot: +![Alt Text](describe.png) +version,timestamp,userId,userName,operation,operationParameters,job,notebook,queryHistoryStatementId,clusterId,readVersion,isolationLevel,isBlindAppend,operationMetrics,userMetadata,engineInfo +4,2026-07-30T11:53:06.000Z,142272964130057,baderalmsaddy@gmail.com,MERGE,"{""predicate"":""[\""(trip_id#28336 <=> trip_id#28476)\""]"",""clusterBy"":""[]"",""matchedPredicates"":""[{\""actionType\"":\""update\""}]"",""statsOnLoad"":""false"",""notMatchedBySourcePredicates"":""[]"",""notMatchedPredicates"":""[{\""actionType\"":\""insert\""}]""}",null,null,01f18c0d-0da2-1778-887c-ca46c7e0c62d,null,3,WriteSerializable,false,"{""numTargetRowsCopied"":""0"",""numTargetRowsDeleted"":""0"",""numTargetBytesRemoved"":""0"",""numTargetDeletionVectorsAdded"":""0"",""numTargetRowsMatchedUpdated"":""0"",""numTargetRowsMatchedDeleted"":""0"",""numTargetRowsUpdated"":""0"",""numTargetChangeFilesAdded"":""0"",""numTargetRowsNotMatchedBySourceDeleted"":""0"",""rewriteTimeMs"":""3702"",""numTargetFilesAdded"":""0"",""numTargetBytesAdded"":""0"",""executionTimeMs"":""74496"",""materializeSourceTimeMs"":""70033"",""numTargetRowsInserted"":""0"",""numTargetDeletionVectorsUpdated"":""0"",""scanTimeMs"":""728"",""numOutputRows"":""0"",""numTargetDeletionVectorsRemoved"":""0"",""numTargetRowsNotMatchedBySourceUpdated"":""0"",""numSourceRows"":""0"",""numTargetFilesRemoved"":""0""}",null,Databricks-Runtime/18.x-aarch64-photon-scala2.13 +3,2026-07-30T11:51:16.000Z,142272964130057,baderalmsaddy@gmail.com,CREATE OR REPLACE TABLE AS SELECT,"{""partitionBy"":""[]"",""clusterBy"":""[]"",""description"":null,""isManaged"":""true"",""properties"":""{\""delta.checkpoint.writeStatsAsJson\"":\""false\"",\""delta.checkpoint.writeStatsAsStruct\"":\""true\"",\""delta.parquet.compression.codec\"":\""zstd\"",\""delta.enableDeletionVectors\"":\""true\""}"",""statsOnLoad"":""true""}",null,null,01f18c0c-c1be-1515-b30a-1fb61ac6307f,null,2,WriteSerializable,false,"{""numFiles"":""64"",""numRemovedFiles"":""65"",""numRemovedBytes"":""5159027812"",""numDeletionVectorsRemoved"":""0"",""numOutputRows"":""124241246"",""numOutputBytes"":""5160528189""}",null,Databricks-Runtime/18.x-aarch64-photon-scala2.13 +2,2026-07-30T11:45:23.000Z,142272964130057,baderalmsaddy@gmail.com,MERGE,"{""predicate"":""[\""(trip_id#24719 <=> trip_id#24859)\""]"",""clusterBy"":""[]"",""matchedPredicates"":""[{\""actionType\"":\""update\""}]"",""statsOnLoad"":""false"",""notMatchedBySourcePredicates"":""[]"",""notMatchedPredicates"":""[{\""actionType\"":\""insert\""}]""}",null,null,01f18c0b-f890-1e66-aa1d-5f2c56335061,null,1,WriteSerializable,false,"{""numTargetRowsCopied"":""0"",""numTargetRowsDeleted"":""0"",""numTargetBytesRemoved"":""0"",""numTargetDeletionVectorsAdded"":""0"",""numTargetRowsMatchedUpdated"":""0"",""numTargetRowsMatchedDeleted"":""0"",""numTargetRowsUpdated"":""0"",""numTargetChangeFilesAdded"":""0"",""numTargetRowsNotMatchedBySourceDeleted"":""0"",""rewriteTimeMs"":""1982"",""numTargetFilesAdded"":""0"",""numTargetBytesAdded"":""0"",""executionTimeMs"":""75613"",""materializeSourceTimeMs"":""72818"",""numTargetRowsInserted"":""0"",""numTargetDeletionVectorsUpdated"":""0"",""scanTimeMs"":""790"",""numOutputRows"":""0"",""numTargetDeletionVectorsRemoved"":""0"",""numTargetRowsNotMatchedBySourceUpdated"":""0"",""numSourceRows"":""0"",""numTargetFilesRemoved"":""0""}",null,Databricks-Runtime/18.x-aarch64-photon-scala2.13 +1,2026-07-30T11:41:45.000Z,142272964130057,baderalmsaddy@gmail.com,MERGE,"{""predicate"":""[\""(trip_id#22154 <=> trip_id#22294)\""]"",""clusterBy"":""[]"",""matchedPredicates"":""[{\""actionType\"":\""update\""}]"",""statsOnLoad"":""false"",""notMatchedBySourcePredicates"":""[]"",""notMatchedPredicates"":""[{\""actionType\"":\""insert\""}]""}",null,null,01f18c0b-7800-1e38-bc8d-2bd108a77adb,null,0,WriteSerializable,false,"{""numTargetRowsCopied"":""0"",""numTargetRowsDeleted"":""0"",""numTargetBytesRemoved"":""0"",""numTargetDeletionVectorsAdded"":""0"",""numTargetRowsMatchedUpdated"":""0"",""numTargetRowsMatchedDeleted"":""0"",""numTargetRowsUpdated"":""0"",""numTargetChangeFilesAdded"":""0"",""numTargetRowsNotMatchedBySourceDeleted"":""0"",""rewriteTimeMs"":""2856"",""numTargetFilesAdded"":""0"",""numTargetBytesAdded"":""0"",""executionTimeMs"":""73312"",""materializeSourceTimeMs"":""69355"",""numTargetRowsInserted"":""0"",""numTargetDeletionVectorsUpdated"":""0"",""scanTimeMs"":""1072"",""numOutputRows"":""0"",""numTargetDeletionVectorsRemoved"":""0"",""numTargetRowsNotMatchedBySourceUpdated"":""0"",""numSourceRows"":""0"",""numTargetFilesRemoved"":""0""}",null,Databricks-Runtime/18.x-aarch64-photon-scala2.13 +0,2026-07-30T11:38:34.000Z,142272964130057,baderalmsaddy@gmail.com,CREATE OR REPLACE TABLE AS SELECT,"{""partitionBy"":""[]"",""clusterBy"":""[]"",""description"":null,""isManaged"":""true"",""properties"":""{\""delta.checkpoint.writeStatsAsJson\"":\""false\"",\""delta.checkpoint.writeStatsAsStruct\"":\""true\"",\""delta.parquet.compression.codec\"":\""zstd\"",\""delta.enableDeletionVectors\"":\""true\""}"",""statsOnLoad"":""true""}",null,null,01f18c0a-f72d-1a36-a628-ffc3ee779511,null,null,WriteSerializable,false,"{""numFiles"":""65"",""numRemovedFiles"":""0"",""numRemovedBytes"":""0"",""numDeletionVectorsRemoved"":""0"",""numOutputRows"":""124241246"",""numOutputBytes"":""5159027812""}",null,Databricks-Runtime/18.x-aarch64-photon-scala2.13 + `___` diff --git a/task-2/dbt_project.yml b/task-2/dbt_project.yml new file mode 100644 index 0000000..fca4ee5 --- /dev/null +++ b/task-2/dbt_project.yml @@ -0,0 +1,25 @@ +name: 'nyc_taxi' +version: '1.0.0' +config-version: 2 + +# This project connects to the profile of the same name in profiles.yml. +profile: 'nyc_taxi' + +model-paths: ["models"] +macro-paths: ["macros"] +test-paths: ["tests"] + +target-path: "target" +clean-targets: + - "target" + - "dbt_packages" + +# Folder-level materialization defaults. Staging models stay as views (cheap, +# always fresh); the mart is built as a table (queried repeatedly by the +# dashboard). You can override per model with {{ config(materialized='...') }}. +models: + nyc_taxi: + staging: + +materialized: view + marts: + +materialized: table diff --git a/task-2/describe.png b/task-2/describe.png new file mode 100644 index 0000000..f204ad2 Binary files /dev/null and b/task-2/describe.png differ diff --git a/task-2/macros/safe_divide.sql b/task-2/macros/safe_divide.sql new file mode 100644 index 0000000..8695fe0 --- /dev/null +++ b/task-2/macros/safe_divide.sql @@ -0,0 +1,6 @@ +{% macro safe_divide(numerator, denominator) %} + case + when {{ denominator }} > 0 then round(({{ numerator }} / {{ denominator }})::numeric, 4) + else null + end +{% endmacro %} \ No newline at end of file diff --git a/task-2/models/marts/_fct_daily_borough_stats.yml b/task-2/models/marts/_fct_daily_borough_stats.yml new file mode 100644 index 0000000..716bc4e --- /dev/null +++ b/task-2/models/marts/_fct_daily_borough_stats.yml @@ -0,0 +1,36 @@ +version: 2 + +models: + - name: fct_daily_borough_stats + description: "One row per completed NYC green taxi trip in January 2024, with + pickup/dropoff zone attributes folded in (OBT-style mart). Queried + directly by dashboards and ad-hoc analysis. + + **Grain:** one row per trip. + **Source:** `public.raw_trips` joined to `public.raw_zones` on + `pickup_location_id` and `dropoff_location_id`. + **Not included:** trips where `pickup_location_id` is NULL (dropped + in `stg_trips`); duplicate rows from the TLC source are kept as-is + and surfaced by `dbt_utils.unique_combination_of_columns`." + # TODO: Task 5 -- add the compound uniqueness test on the mart's primary + # key (pickup_borough, pickup_date). You need the dbt_utils package for + # this: declare it in packages.yml and run `dbt deps` first. + tests: + - dbt_utils.unique_combination_of_columns: + combination_of_columns: + - pickup_borough + - pickup_date + severity: warn + columns: + - name: pickup_borough + description: "TLC borough where the trip started" + - name: pickup_date + description: "the day the trip started (date only, no time)" + - name: trip_count + description: "total trips that started in this borough on this day" + - name: total_fare + description: "total revenue from trips that started in this borough on this day in USD" + - name: avg_tip_pct + description: "average tip percentage for trips that started in this borough on this day, expressed as a decimal (e.g. 0.15 = 15%)" + - name: avg_trip_distance + description: "average distance of trips that started in this borough on this day, in miles" diff --git a/task-2/models/marts/fct_daily_borough_stats.sql b/task-2/models/marts/fct_daily_borough_stats.sql new file mode 100644 index 0000000..0e6af8b --- /dev/null +++ b/task-2/models/marts/fct_daily_borough_stats.sql @@ -0,0 +1,41 @@ +-- Mart: daily borough trip statistics. +-- Grain: one row per (pickup_borough, pickup_date). +-- Used to answer: trip volume, revenue, tipping behaviour, and distance profile +-- per borough per day for January 2024. +{{ + config( + materialized='incremental', + incremental_strategy='merge', + unique_key='trip_id' + ) +}} + +WITH trips AS ( + SELECT * + FROM {{ ref('stg_trips') }} +), + +zones AS ( + SELECT * + FROM {{ ref('stg_zones') }} +) + +SELECT + + z.borough AS pickup_borough, + t.pickup_datetime AS pickup_date, + COUNT(*) AS trip_count, + SUM(t.fare_amount) AS total_fare, + AVG(t.tip_pct) AS avg_tip_pct, + AVG(t.trip_distance) AS avg_trip_distance + +FROM trips t +INNER JOIN zones z + ON t.pickup_location_id = z.location_id + +{% if is_incremental() %} +WHERE t.pickup_datetime::date > (select max(pickup_date) from {{ this }}) +{% endif %} + +GROUP BY pickup_borough, pickup_date + diff --git a/task-2/models/marts/fct_trips.sql b/task-2/models/marts/fct_trips.sql new file mode 100644 index 0000000..dcef618 --- /dev/null +++ b/task-2/models/marts/fct_trips.sql @@ -0,0 +1,33 @@ +{{ + config( + materialized='incremental', + incremental_strategy='merge', + unique_key='trip_id' + ) +}} + +select + t.trip_id, + t.pickup_datetime, + t.dropoff_datetime, + t.fare_amount, + t.tip_amount, + t.trip_distance, + t.trip_duration_minutes, + t.tip_pct, + t.fare_per_mile, + t.payment_type_label, + pz.borough as pickup_borough, + pz.zone as pickup_zone, + dz.borough as dropoff_borough, + dz.zone as dropoff_zone +from {{ ref('stg_trips') }} t +left join {{ ref('stg_zones') }} pz + on t.pickup_location_id = pz.location_id +left join {{ ref('stg_zones') }} dz + on t.dropoff_location_id = dz.location_id + +{% if is_incremental() %} + -- On incremental runs, only process trips newer than what we already have. + where t.pickup_datetime > (select max(pickup_datetime) from {{ this }}) +{% endif %} diff --git a/task-2/models/marts/fct_trips.yml b/task-2/models/marts/fct_trips.yml new file mode 100644 index 0000000..b11f8bb --- /dev/null +++ b/task-2/models/marts/fct_trips.yml @@ -0,0 +1,53 @@ +version: 2 + +models: + - name: fct_trips + description: | + One row per completed NYC yellow taxi trip (2023-2025), with + pickup/dropoff zone attributes folded in (OBT-style mart). Queried + directly by dashboards and ad-hoc analysis. + + **Grain:** one row per trip (`trip_id` surrogate key). + **Source:** `hyf.nyc_yellow.raw_trips` joined to `raw_zones` on + `pickup_location_id` and `dropoff_location_id`. + **Not included:** trips where `pickup_location_id` is NULL (dropped + in `stg_trips`); duplicate rows from the TLC source are kept as-is + and surfaced by `dbt_utils.unique_combination_of_columns`. + columns: + - name: trip_id + description: Surrogate key generated in `stg_trips` for incremental merge. + tests: [not_null, unique] + - name: pickup_datetime + description: Wall-clock time the trip began (America/New_York, no timezone attached). + tests: [not_null] + - name: dropoff_datetime + description: Wall-clock time the trip ended. + - name: fare_amount + description: Metered fare in USD, not including tip, tolls, or surcharges. + - name: tip_amount + description: Tip in USD. Non-zero only when payment_type is credit card (1). + - name: trip_distance + description: Distance in miles as reported by the taximeter. + - name: tip_pct + description: | + `tip_amount / fare_amount`, rounded to 4 decimals. NULL when + `fare_amount` is 0 (voided trips, no-charge rides). + - name: fare_per_mile + description: | + `fare_amount / trip_distance`, rounded to 4 decimals. NULL when + `trip_distance` is 0 (data-quality anomalies). + - name: payment_type_label + description: | + Human-readable payment method from the TLC code. See the jinja + dictionary in `stg_trips.sql` for the 1-6 → label mapping. + - name: pickup_borough + description: | + NYC borough of the pickup zone, joined from `stg_zones.borough`. + Values: Manhattan, Brooklyn, Queens, Bronx, Staten Island, EWR, + Unknown, NaN, or NULL when `pickup_location_id` did not resolve. + - name: pickup_zone + description: Human-readable pickup-zone name from `stg_zones.zone`. + - name: dropoff_borough + description: NYC borough of the dropoff zone. + - name: dropoff_zone + description: Human-readable dropoff-zone name. diff --git a/task-2/models/staging/_sources.yml b/task-2/models/staging/_sources.yml new file mode 100644 index 0000000..1eff526 --- /dev/null +++ b/task-2/models/staging/_sources.yml @@ -0,0 +1,10 @@ +version: 2 + +sources: + - name: nyc_taxi + description: Raw NYC yellow taxi trip records and zone lookup on Databricks (Unity Catalog). + database: hyf + schema: nyc_yellow + tables: + - name: raw_trips + - name: raw_zones \ No newline at end of file diff --git a/task-2/models/staging/_stg_trips.yml b/task-2/models/staging/_stg_trips.yml new file mode 100644 index 0000000..5a947ef --- /dev/null +++ b/task-2/models/staging/_stg_trips.yml @@ -0,0 +1,30 @@ +version: 2 + +models: + - name: stg_trips + description: "Cleaned green taxi trips, one row per trip. This reads from the raw_trips source." + columns: + - name: pickup_datetime + description: "when the trip started" + tests: + - not_null + - name: pickup_location_id + description: "TLC zone id where the trip started" + tests: + - not_null + - relationships: + to: ref('stg_zones') + field: location_id + config: + severity: warn + - name: fare_amount + description: "cost of the trip in USD" + - name: tip_amount + description: "tip paid for the trip" + + - name: trip_distance + description: "distance of the trip in miles" + + - name: tip_pct + description: "percentage of tip relative to trip cost" + diff --git a/task-2/models/staging/_stg_zones.yml b/task-2/models/staging/_stg_zones.yml new file mode 100644 index 0000000..13c591d --- /dev/null +++ b/task-2/models/staging/_stg_zones.yml @@ -0,0 +1,15 @@ +version: 2 + +models: + - name: stg_zones + description: "One row per TLC taxi zone (265 zones total)" + columns: + - name: location_id + description: TLC zone ID. + tests: + - unique + - not_null + - name: borough + description: NYC borough (Manhattan, Brooklyn, Queens, Bronx, Staten Island, EWR, Unknown). + tests: + - not_null \ No newline at end of file diff --git a/task-2/models/staging/stg_trips.sql b/task-2/models/staging/stg_trips.sql new file mode 100644 index 0000000..ac1a499 --- /dev/null +++ b/task-2/models/staging/stg_trips.sql @@ -0,0 +1,51 @@ +{{ config(materialized='view') }} + +{% set payment_types = { + 1: 'Credit card', + 2: 'Cash', + 3: 'No charge', + 4: 'Dispute', + 5: 'Unknown', + 6: 'Voided trip' +} %} + +with base as ( + select + {{ dbt_utils.generate_surrogate_key([ + 'vendor_id', 'pickup_datetime', 'dropoff_datetime', 'pickup_location_id', + 'dropoff_location_id', 'fare_amount', 'trip_distance', 'total_amount', 'passenger_count' + ]) }} as trip_id, + pickup_datetime, + dropoff_datetime, + pickup_location_id, + dropoff_location_id, + fare_amount, + tip_amount, + trip_distance, + payment_type, + case + when fare_amount > 0 then round(tip_amount / fare_amount, 4) + else null + end as tip_pct, + case + when trip_distance > 0 then round(fare_amount / trip_distance, 4) + else null + end as fare_per_mile, + case payment_type + {% for code, label in payment_types.items() %} + when {{ code }} then '{{ label }}' + {% endfor %} + else 'Other' + end as payment_type_label, + round((unix_timestamp(dropoff_datetime) - unix_timestamp(pickup_datetime)) / 60.0, 2) as trip_duration_minutes + from {{ source('nyc_taxi', 'raw_trips') }} + where pickup_location_id is not null + and fare_amount >= 0 +) + +select * +from base +qualify row_number() over ( + partition by trip_id + order by pickup_datetime +) = 1 diff --git a/task-2/models/staging/stg_zones.sql b/task-2/models/staging/stg_zones.sql new file mode 100644 index 0000000..1e98897 --- /dev/null +++ b/task-2/models/staging/stg_zones.sql @@ -0,0 +1,6 @@ +select + location_id, + borough, + zone, + service_zone +from {{ source('nyc_taxi', 'raw_zones') }} diff --git a/task-2/package-lock.yml b/task-2/package-lock.yml new file mode 100644 index 0000000..1ce78fc --- /dev/null +++ b/task-2/package-lock.yml @@ -0,0 +1,5 @@ +packages: + - name: dbt_utils + package: dbt-labs/dbt_utils + version: 1.4.1 +sha1_hash: e6424ba9e5a22487e47f023803aa4f0411946808 diff --git a/task-2/packages.yml b/task-2/packages.yml new file mode 100644 index 0000000..71fdb1a --- /dev/null +++ b/task-2/packages.yml @@ -0,0 +1,3 @@ +packages: + - package: dbt-labs/dbt_utils + version: [">=1.1.0", "<2.0.0"] \ No newline at end of file diff --git a/task-2/requirements.txt b/task-2/requirements.txt new file mode 100644 index 0000000..cbfe19a --- /dev/null +++ b/task-2/requirements.txt @@ -0,0 +1,78 @@ +agate==1.9.1 +annotated-types==0.8.0 +attrs==26.1.0 +babel==2.18.0 +certifi==2026.7.22 +cffi==2.1.0 +charset-normalizer==3.4.9 +click==8.4.2 +colorama==0.4.6 +cryptography==49.0.0 +daff==1.4.2 +databricks-sdk==0.117.0 +databricks-sql-connector==4.3.0 +dbt-adapters==1.24.5 +dbt-common==1.37.5 +dbt-core==1.12.0 +dbt-core-experimental-parser==2.0.0a5 +dbt-databricks==1.12.3 +dbt-extractor==0.6.0 +dbt-protos==1.0.541 +dbt-spark==1.10.3 +deepdiff==8.6.2 +et_xmlfile==2.0.0 +google-auth==2.56.2 +idna==3.18 +importlib_metadata==9.0.0 +isodate==0.7.2 +Jinja2==3.1.6 +jsonschema==4.26.0 +jsonschema-specifications==2025.9.1 +leather==0.4.1 +lz4==4.4.5 +MarkupSafe==3.0.3 +mashumaro==3.17 +metricflow==0.211.0 +more-itertools==10.8.0 +msgpack==1.2.1 +networkx==3.6.1 +numpy==2.5.1 +oauthlib==3.3.1 +openpyxl==3.1.5 +orderly-set==5.5.0 +packaging==25.0 +pandas==3.0.5 +parsedatetime==2.6 +pathspec==1.0.4 +protobuf==6.33.6 +pyarrow==25.0.0 +pyasn1==0.6.4 +pyasn1_modules==0.4.2 +pybreaker==1.4.1 +pycparser==3.0 +pydantic==2.12.5 +pydantic_core==2.41.5 +PyJWT==2.13.0 +python-dateutil==2.9.0.post0 +python-dotenv==1.2.2 +python-slugify==8.0.4 +pytimeparse==1.1.8 +pytz==2026.3.post1 +PyYAML==6.0.3 +RapidFuzz==3.14.5 +referencing==0.37.0 +requests==2.34.2 +rpds-py==2026.6.3 +six==1.17.0 +snowplow-tracker==1.1.0 +sqlglot==30.14.0 +sqlparams==6.2.0 +sqlparse==0.5.5 +tabulate==0.10.0 +text-unidecode==1.3 +thrift==0.22.0 +typing-inspection==0.4.2 +typing_extensions==4.16.0 +tzdata==2026.3 +urllib3==2.7.0 +zipp==4.1.0 diff --git a/task-2/tests/assert_avg_tip_pct_within_bounds.sql b/task-2/tests/assert_avg_tip_pct_within_bounds.sql new file mode 100644 index 0000000..054736d --- /dev/null +++ b/task-2/tests/assert_avg_tip_pct_within_bounds.sql @@ -0,0 +1,22 @@ +-- Singular test: flag (borough, date) combinations where avg_tip_pct > 1. +-- A tip_pct > 1 means the average tip exceeded the total fare for that cell, +-- which is unusual and almost always indicates a small-sample bucket (e.g. the +-- Unknown borough) where a few high-tip outliers dominate the average. +-- +-- Set this test to WARN severity by adding an inline config at the top of this +-- file (below these comments): {{ config(severity='warn') }} +-- That keeps a few expected Unknown-borough rows from blocking `dbt build`, while +-- still surfacing them for your reports/answers.md write-up (the rubric requires +-- documenting this finding). Do NOT set a project-level test severity in +-- dbt_project.yml: that would also downgrade your not_null and +-- unique_combination primary-key tests, which you want to stay at ERROR. +-- +-- The test passes (no WARN) when zero rows are returned; any returned rows are flagged. + +-- TODO: write the SELECT here. +-- Query {{ ref('fct_daily_borough_stats') }} and return rows where avg_tip_pct > 1. + +{{ config(severity='warn') }} +select pickup_borough, pickup_date, avg_tip_pct +from {{ ref('fct_daily_borough_stats') }} +where avg_tip_pct > 1 \ No newline at end of file diff --git a/task-3/SCHEDULING.md b/task-3/SCHEDULING.md index 2780686..fd9f2fc 100644 --- a/task-3/SCHEDULING.md +++ b/task-3/SCHEDULING.md @@ -2,8 +2,7 @@ ## Databricks Job Run URL -Paste the URL of your successful Job run from the Databricks UI address bar: - +Paste the URL of your successful Job run from the Databricks UI address bar: https://adb-7405619530719547.7.azuredatabricks.net/jobs/282713845764221/runs/984569826693564?o=7405619530719547 `___` ## Screenshots @@ -18,6 +17,7 @@ Ensure the following screenshot files exist in `task-3/screenshots/`: ### When would you choose Databricks Jobs versus Apache Airflow for pipeline orchestration? -Write two to three sentences comparing Databricks Jobs and Apache Airflow in your own words: +Write two to three sentences comparing Databricks Jobs and Apache Airflow in your own words: in databricks u find everything in its own catagory with a very well made UI except for monitoring, seeing all the jobs clearly in one UI with whats failed and what succeeded in airflow looks simplest on the eyes +databricks on the otherhand even tho more userfriendly and has more advantages (notebooks) unless u need them airflow is simpler to monitor if thats the only goal `___` diff --git a/task-3/screenshots/dbt.png b/task-3/screenshots/dbt.png new file mode 100644 index 0000000..4082447 Binary files /dev/null and b/task-3/screenshots/dbt.png differ diff --git a/task-3/screenshots/git.png b/task-3/screenshots/git.png new file mode 100644 index 0000000..f26c8ac Binary files /dev/null and b/task-3/screenshots/git.png differ diff --git a/task-3/screenshots/schedule.png b/task-3/screenshots/schedule.png new file mode 100644 index 0000000..1662b90 Binary files /dev/null and b/task-3/screenshots/schedule.png differ