Hannah N

task-2/.gitignore

@@ -0,0 +1 @@
+profiles.yml

task-2/WRITEUP.md

@@ -5,15 +5,20 @@ Fill in after running `dbt build --select fct_trips --full-refresh` baseline fol
 ## First build (full / initial load with --full-refresh)
 
 - **Wall-clock time:**
+13:19:59  1 of 4 START sql incremental model dev_hannahwn.fct_trips ...................... [RUN]
+13:20:55  1 of 4 OK created sql incremental model dev_hannahwn.fct_trips ................. [OK in 55.89s]
 - **Notes:** (optional: warehouse size, any errors you fixed)
 
 ## Second build (incremental rerun)
 
 - **Wall-clock time:**
+ 1 of 4 START sql incremental model dev_hannahwn.fct_trips ...................... [RUN]
+13:21:40  1 of 4 OK created sql incremental model dev_hannahwn.fct_trips ................. [OK in 9.84s]
 
 ## Why was the second run faster?
 
-Write two or three sentences in your own words (see the assignment for the concepts you must name):
+The first build builds from scratch thus taking more time. The second run using is_incremental() and so only new rows are read and updated
+
 
 `___`
 
@@ -22,3 +27,5 @@ Write two or three sentences in your own words (see the assignment for the conce
 Paste the output or summary of `DESCRIBE HISTORY hyf.dev_yourname.fct_trips` (showing `CREATE OR REPLACE TABLE` and `MERGE` operations) or reference a screenshot:
 
 `___`
+
+

task-2/dbt_project.yml

@@ -0,0 +1,29 @@
+name: 'nyc_taxi_borough_daily'
+version: '1.0.0'
+config-version: 2
+
+# This project connects to the profile of the same name in profiles.yml.
+profile: 'nyc_taxi_borough_daily'
+
+model-paths: ["models"]
+macro-paths: ["macros"]
+test-paths: ["tests"]
+
+target-path: "target"
+clean-targets:
+  - "target"
+  - "dbt_packages"
+
+# Folder-level materialization defaults. Staging models stay as views (cheap,
+# always fresh); the mart is built as a table (queried repeatedly by the
+# dashboard). You can override per model with {{ config(materialized='...') }}.
+models:
+  nyc_taxi_borough_daily:
+    staging:
+      +materialized: view
+    marts:
+      +materialized: table
+
+# dbt_project.yml
+flags:
+     use_materialization_v2: true

task-2/models/marts/_fct_daily_borough_stats.yml

@@ -0,0 +1,30 @@
+version: 2
+
+models:
+  - name: fct_daily_borough_stats
+
+    description: >
+
+      one row per pickup_borough and pickup_date,the source lineage(built from stg_trips
+      and stg_zones), and atleast one known caveat(we drop rows in staging where pickup location
+      id is null))
+    test:
+      dbt_utils.unique_combination_of_columns:
+        combination_of_columns:
+          - pickup_borough
+          - pickup_date
+    columns:
+      - name: pickup_borough
+        description: groups trips by the pickup_borough, which is derived from the pickup_location_id and the stg_zones table
+
+      - name: pickup_date
+        description: groups trips by the pickup_date, which is derived from the pickup_datetime in stg_trips
+
+      - name: trip_count
+        description: counts the number of trips for each pickup_borough and pickup_date combination
+      - name: total_fare
+        description: sums the fare_amount for each pickup_borough and pickup_date combination
+      - name: avg_tip_pct
+        description: calculates the average tip percentage for each pickup_borough and pickup_date combination
+      - name: avg_trip_distance
+        description: calculates the average trip distance for each pickup_borough and pickup_date combination percentage

task-2/models/marts/_fct_trips.yml

@@ -0,0 +1,57 @@
+version: 2
+
+models:
+  - name: fct_trips
+    description: |
+      One row per completed NYC yellow taxi trip (2023-2025), with
+      pickup/dropoff zone attributes folded in (OBT-style mart). Queried
+      directly by dashboards and ad-hoc analysis.
+
+      **Grain:** one row per trip (`trip_id` surrogate key).
+      **Source:** `hyf.nyc_yellow.raw_trips` joined to `raw_zones` on
+      `pickup_location_id` and `dropoff_location_id`.
+      **Not included:** trips where `pickup_location_id` is NULL (dropped
+      in `stg_trips`); duplicate rows from the TLC source are kept as-is
+      and surfaced by `dbt_utils.unique_combination_of_columns`.
+    columns:
+      - name: trip_id
+        description: Surrogate key generated in `stg_trips` for incremental merge.
+        tests:
+          - unique:
+              config:
+                severity: warn
+          - not_null
+      - name: pickup_datetime
+        description: Wall-clock time the trip began (America/New_York, no timezone attached).
+        tests: [not_null]
+      - name: dropoff_datetime
+        description: Wall-clock time the trip ended.
+      - name: fare_amount
+        description: Metered fare in USD, not including tip, tolls, or surcharges.
+      - name: tip_amount
+        description: Tip in USD. Non-zero only when payment_type is credit card (1).
+      - name: trip_distance
+        description: Distance in miles as reported by the taximeter.
+      - name: tip_pct
+        description: |
+          `tip_amount / fare_amount`, rounded to 4 decimals. NULL when
+          `fare_amount` is 0 (voided trips, no-charge rides).
+      - name: fare_per_mile
+        description: |
+          `fare_amount / trip_distance`, rounded to 4 decimals. NULL when
+          `trip_distance` is 0 (data-quality anomalies).
+      - name: payment_type_label
+        description: |
+          Human-readable payment method from the TLC code. See the jinja
+          dictionary in `stg_trips.sql` for the 1-6 → label mapping.
+      - name: pickup_borough
+        description: |
+          NYC borough of the pickup zone, joined from `stg_zones.borough`.
+          Values: Manhattan, Brooklyn, Queens, Bronx, Staten Island, EWR,
+          Unknown, NaN, or NULL when `pickup_location_id` did not resolve.
+      - name: pickup_zone
+        description: Human-readable pickup-zone name from `stg_zones.zone`.
+      - name: dropoff_borough
+        description: NYC borough of the dropoff zone.
+      - name: dropoff_zone
+        description: Human-readable dropoff-zone name.

task-2/models/marts/fct_daily_borough_stats.sql

@@ -0,0 +1,28 @@
+-- Mart: daily borough trip statistics.
+-- Grain: one row per (pickup_borough, pickup_date).
+-- Used to answer: trip volume, revenue, tipping behaviour, and distance profile
+-- per borough per day for January 2024.
+
+WITH trips AS (
+    SELECT *
+    FROM {{ ref('stg_trips') }}
+),
+
+zones AS (
+    SELECT *
+    FROM {{ ref('stg_zones') }}
+)
+
+SELECT
+    z.borough AS pickup_borough,
+    t.pickup_datetime::date AS pickup_date,
+    COUNT(*) AS trip_count,
+    SUM(t.fare_amount) AS total_fare,
+    AVG({{ dbt_utils.safe_divide('t.tip_amount', 't.fare_amount') }}) AS avg_tip_pct,
+    AVG(t.trip_distance) AS avg_trip_distance
+
+FROM trips t
+INNER JOIN zones z
+    ON t.pickup_location_id = z.location_id
+
+GROUP BY z.borough, t.pickup_datetime::date

task-2/models/marts/fct_trips.sql

@@ -0,0 +1,33 @@
+{{
+  config(
+    materialized='incremental',
+    incremental_strategy='merge',
+    unique_key='trip_id'
+  )
+}}
+
+select
+    t.trip_id,
+    t.pickup_datetime,
+    t.dropoff_datetime,
+    t.fare_amount,
+    t.tip_amount,
+    t.trip_distance,
+    t.trip_duration_minutes,
+    t.tip_pct,
+    t.fare_per_mile,
+    t.payment_type_label,
+    pz.borough as pickup_borough,
+    pz.zone as pickup_zone,
+    dz.borough as dropoff_borough,
+    dz.zone as dropoff_zone
+from {{ ref('stg_trips') }} t
+left join {{ ref('stg_zones') }} pz
+    on t.pickup_location_id = pz.location_id
+left join {{ ref('stg_zones') }} dz
+    on t.dropoff_location_id = dz.location_id
+
+{% if is_incremental() %}
+    -- On incremental runs, only process trips newer than what we already have.
+    where t.pickup_datetime > (select max(pickup_datetime) from {{ this }})
+{% endif %}

task-2/models/marts/fct_trips_docs.md

@@ -0,0 +1,10 @@
+{% docs trip_grain %}
+
+One row per completed taxi trip. "Completed" means the TLC submitted the
+trip record to the public dataset; cancellations and trips in progress
+are not included. Duplicates exist in the source data (roughly 4 rows in
+January 2024 where every column is identical) and are kept as-is; see
+the `dbt_utils.unique_combination_of_columns` test results for the
+current count.
+
+{% enddocs %}

task-2/models/staging/_sources.yml

@@ -0,0 +1,10 @@
+version: 2
+
+sources:
+  - name: nyc_taxi
+    schema: nyc_yellow
+    tables:
+      - name: raw_trips
+        description: One trip per row with detailed information
+      - name: raw_zones
+        description: a table for taxi zones and their boroughs

task-2/models/staging/_stg_trips.yml

@@ -0,0 +1,27 @@
+version: 2
+
+models:
+  - name: stg_trips
+    description: cleaned green taxi trips,one row per trip
+    columns:
+      - name: pickup_datetime
+        description: When trip started. This is used to group trips by day in the mart
+        tests:
+          - not_null
+
+      - name: pickup_location_id
+        description: TLC location ID of the pickup location. This is used to join to the zones table in the mart
+        tests:
+          - not_null
+      - name: fare_amount
+        description: Total fare amount for the trip
+
+      - name: tip_amount
+        description: Tip amount for the trip
+
+      - name: trip_distance
+        description: Distance of the trip
+
+      - name: tip_pct
+        description: Percentage of the fare that was tipped
+

task-2/models/staging/_stg_zones.yml

@@ -0,0 +1,18 @@
+version: 2
+
+models:
+  - name: stg_zones
+    description: one row per taxi zone from the
+    columns:
+      - name: location_id
+        description: TLC location ID of the pickup location. This is used to join to the trips table in the mart
+        tests:
+          - not_null
+          - unique
+
+      - name: borough
+        description: The borough of NYC that the taxi zone is in
+        tests:
+          - not_null
+
+

task-2/models/staging/stg_trips.sql

@@ -0,0 +1,38 @@
+-- Staging model: one row per NYC green taxi trip (January 2024).
+-- Renames source columns, adds derived columns, and filters bad rows.
+-- Downstream: fct_daily_borough_stats joins this to stg_zones.
+SELECT
+    {{ dbt_utils.generate_surrogate_key([
+        'pickup_datetime', 'dropoff_datetime', 'pickup_location_id',
+        'dropoff_location_id', 'fare_amount', 'total_amount', 'passenger_count'
+    ]) }} AS trip_id,
+    pickup_datetime,
+    dropoff_datetime,
+    pickup_location_id,
+    dropoff_location_id,
+    fare_amount,
+    tip_amount,
+    trip_distance,
+    {{ dbt_utils.safe_divide('tip_amount', 'fare_amount') }} AS tip_pct,
+    {{ dbt_utils.safe_divide('fare_amount', 'trip_distance') }} AS fare_per_mile,
+    CASE
+        WHEN payment_type = 1 THEN 'Credit Card'
+        WHEN payment_type = 2 THEN 'Cash'
+        WHEN payment_type = 3 THEN 'No Charge'
+        WHEN payment_type = 4 THEN 'Dispute'
+        WHEN payment_type = 5 THEN 'Unknown'
+        WHEN payment_type = 6 THEN 'Voided Trip'
+        ELSE NULL
+    END AS payment_type_label,
+    CASE
+        WHEN pickup_datetime IS NULL OR dropoff_datetime IS NULL THEN NULL
+        ELSE (unix_timestamp(dropoff_datetime) - unix_timestamp(pickup_datetime)) / 60
+    END AS trip_duration_minutes
+FROM {{ source('nyc_taxi', 'raw_trips') }}
+WHERE
+    pickup_location_id IS NOT NULL
+    AND fare_amount >= 0
+
+
+
+

task-2/models/staging/stg_zones.sql

@@ -0,0 +1,11 @@
+-- Staging model: one row per TLC zone (265 zones).
+-- Exposes location_id and borough for use as a lookup in the mart.
+
+
+
+SELECT
+location_id,
+    borough,
+    zone
+
+FROM {{ source('nyc_taxi', 'raw_zones') }}

task-2/package-lock.yml

@@ -0,0 +1,5 @@
+packages:
+  - name: dbt_utils
+    package: dbt-labs/dbt_utils
+    version: 1.4.1
+sha1_hash: 8b27037b26f3f630c6661194d2470e720c49f6ee

task-2/packages.yml

@@ -0,0 +1,8 @@
+# TODO: Task 5 -- declare the dbt-labs/dbt_utils package here, then run
+# `dbt deps` to install it. You need it for the compound uniqueness test
+# on the mart. See https://hub.getdbt.com/dbt-labs/dbt_utils/latest/
+# for the package block syntax.
+packages:
+  - package: dbt-labs/dbt_utils
+    version: 1.4.1
+

task-3/SCHEDULING.md

@@ -5,6 +5,7 @@
 Paste the URL of your successful Job run from the Databricks UI address bar:
 
 `___`
+https://adb-7405619530719547.7.azuredatabricks.net/jobs/727982853732525/runs/959594307168856?o=7405619530719547
 
 ## Screenshots
 
@@ -21,3 +22,5 @@ Ensure the following screenshot files exist in `task-3/screenshots/`:
 Write two to three sentences comparing Databricks Jobs and Apache Airflow in your own words:
 
 `___`
+
+I would schedule Databricks when I want to see job runs in UI and Apache Airflow when I want to focus on dag runs and task logs