MohammedA

AI_ASSIST.md

@@ -4,13 +4,63 @@ Record at least one point where you used an AI coding assistant (ChatGPT, Claude
 
 ## Interaction 1
 
-- **Tool used:** (e.g. ChatGPT / Cursor / Claude)
-- **Task / Problem:** (e.g. debugging dbt connection profile / writing PySpark join / configuring Job trigger)
+- **Tool used:** ChatGPT
+- **Task / Problem:** Debugging the Databricks HTTP path being incorrectly converted by Git Bash.
 - **Prompt sent:**
-  > `___`
+  > `i got this error
+
+Connection:
+00:21:33 host: **\*\*\*\***\*\***\*\*\*\***\***\*\*\*\***\*\***\*\*\*\***
+00:21:33 http_path: C:/Program Files/Git/sql/1.0**\*\***\*\*\*\***\*\***
+00:21:33 catalog: hyf
+00:21:33 schema: dev_mohammedalfakih
+00:21:33 Registered adapter: databricks=1.12.3
+00:21:34 Databricks adapter: SPOG host (host_type='workspace'): no
+00:21:34 Databricks adapter: workspace_id (from ?o= in http_path): None
+00:21:34 Databricks adapter: databricks-sql-connector version: 4.3.0 (supported)
+00:21:34 Databricks adapter: databricks-sdk version: 0.117.0 (supported)
+00:21:34 databricks-sql-connector adapter: ThriftBackend.attempt_request: Exception: %s
+ERROR:databricks.sql.backend.thrift_backend:ThriftBackend.attempt_request: Exception:
+00:21:34 Databricks adapter: Connection(session-id=Unknown) - Exception while trying to create connection: Error during request to server.
+Error properties: attempt=1/30, bounded-retry-delay=None, elapsed-seconds=0.10899758338928223/900.0, error-message=, http-code=404, method=OpenSession, no-retry-reason=non-retryable error, original-exception=, query-id=None, session-id=None
+00:21:34 Connection test: [ERROR]
+
+00:21:34 1 check failed:
+00:21:34 dbt was unable to connect to the specified database.
+The database returned the following error:
+
+> Database Error
+> Database Error
+
+    Error during request to server.
+
+Check your database credentials and try again. For more information, visit:
+https://docs.getdbt.com/docs/configure-your-profile
+
+---
+
+i notice that http path is loaded in wrong way. i have if in my .env as DATABRICKS_HTTP_PATH=/sql/1.0/warehouses/**\*\*\*\***\*\*\***\*\*\*\***
+but in the error i can see http_path: C:/Program Files/Git/sql/1.0**\*\***\*\*\*\***\*\***
+can u explain why is this happing and how to solve it.
+note: the(\*) is only to hide secrets so I have an real values in my .env`
+
 - **Output provided by AI:**
-  > `___`
+  > `You found the cause correctly. Git Bash uses MSYS2, which automatically converts environment-variable values that look like Unix paths before passing them to native Windows programs. Therefore:
+/sql/1.0/warehouses/...
+was incorrectly converted to:
+C:/Program Files/Git/sql/1.0/warehouses/...
+That produced the HTTP 404. MSYS2 documents this environment-variable path conversion.
+Keep the value in .env unchanged—it is correct.
+Next step only
+In the same Git Bash session, run:
+export MSYS2_ENV_CONV_EXCL=DATABRICKS_HTTP_PATH
+uv run dbt debug --profiles-dir .
+This tells Git Bash not to convert that specific environment variable. The debug output should now show an HTTP path beginning with:
+/sql/1.0/warehouses/
+and ideally finish with:
+All checks passed!
+`
 - **What I kept, changed, or rejected, and why:**
-  > `___`
+  > `I kept the suggested `MSYS2_ENV_CONV_EXCL=DATABRICKS_HTTP_PATH`fix because it prevented Git Bash from converting the Databricks HTTP path, and`dbt debug`then passed. I kept the correct`/sql/1.0/warehouses/...` value unchanged and did not include any real token or credential in the AI prompt.`
 
-*(Ensure no personal passwords, Databricks tokens, or unapproved credentials are included in prompts or logged outputs.)*
+_(Ensure no personal passwords, Databricks tokens, or unapproved credentials are included in prompts or logged outputs.)_

task-1/pyspark_exploration.ipynb

@@ -0,0 +1,227 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {
+      "byteLimit": 2048000,
+      "rowLimit": 10000
+     },
+     "inputWidgets": {},
+     "nuid": "812be89c-94b7-4212-8456-88f19f6fd2f7",
+     "showTitle": false,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "source": [
+    "# Task 1: PySpark Exploration"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 0,
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {
+      "byteLimit": 2048000,
+      "rowLimit": 10000
+     },
+     "finishTime": 1785368532353,
+     "inputWidgets": {},
+     "nuid": "30e0fce1-105c-492a-89cb-3f7aeb888d11",
+     "showTitle": false,
+     "startTime": 1785368531489,
+     "submitTime": 1785368531211,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from pyspark.sql import functions as F\n",
+    "\n",
+    "trips = spark.read.table(\"hyf.nyc_yellow.raw_trips\")\n",
+    "zones = spark.read.table(\"hyf.nyc_yellow.raw_zones\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {
+      "byteLimit": 2048000,
+      "rowLimit": 10000
+     },
+     "inputWidgets": {},
+     "nuid": "7ce90937-7600-4bb1-8a7d-1ea73d9a7376",
+     "showTitle": false,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "source": [
+    "## Question 1: Which pickup borough has the most trips?"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 0,
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {
+      "byteLimit": 2048000,
+      "rowLimit": 10000
+     },
+     "finishTime": 1785368544983,
+     "inputWidgets": {},
+     "nuid": "f6782a41-0e18-46bd-921e-72226d46cc0f",
+     "showTitle": false,
+     "startTime": 1785368532402,
+     "submitTime": 1785368531237,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "outputs": [
+    {
+     "output_type": "stream",
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "+---------+----------+\n|borough  |trip_count|\n+---------+----------+\n|Manhattan|112028489 |\n+---------+----------+\nonly showing top 1 row\n\n"
+     ]
+    }
+   ],
+   "source": [
+    "borough_counts = (\n",
+    "    trips\n",
+    "    .join(\n",
+    "        zones,\n",
+    "        trips.pickup_location_id == zones.location_id,\n",
+    "        \"inner\"\n",
+    "    )\n",
+    "    .groupBy(\"borough\")\n",
+    "    .agg(F.count(\"*\").alias(\"trip_count\"))\n",
+    "    .orderBy(F.col(\"trip_count\").desc())\n",
+    ")\n",
+    "\n",
+    "borough_counts.show(1, truncate=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {
+      "byteLimit": 2048000,
+      "rowLimit": 10000
+     },
+     "inputWidgets": {},
+     "nuid": "4c51b8f0-4857-4ca4-938d-2b29790dd365",
+     "showTitle": false,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "source": [
+    "## Question 2: What is the average total amount per payment type?"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 0,
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {
+      "byteLimit": 2048000,
+      "rowLimit": 10000
+     },
+     "finishTime": 1785368553297,
+     "inputWidgets": {},
+     "nuid": "be5d5e8a-4172-49c2-8508-ba14f345c7e4",
+     "showTitle": false,
+     "startTime": 1785368545006,
+     "submitTime": 1785368531260,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "outputs": [
+    {
+     "output_type": "stream",
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "+------------+----------------+\n|payment_type|avg_total_amount|\n+------------+----------------+\n|0           |23.49           |\n|1           |30.0            |\n|2           |23.75           |\n|3           |9.03            |\n|4           |2.16            |\n|5           |14.89           |\n+------------+----------------+\n\n"
+     ]
+    }
+   ],
+   "source": [
+    "payment_average = (\n",
+    "    trips\n",
+    "    .groupBy(\"payment_type\")\n",
+    "    .agg(\n",
+    "        F.round(\n",
+    "            F.avg(\"total_amount\"),\n",
+    "            2\n",
+    "        ).alias(\"avg_total_amount\")\n",
+    "    )\n",
+    "    .orderBy(\"payment_type\")\n",
+    ")\n",
+    "\n",
+    "payment_average.show(truncate=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {
+      "byteLimit": 2048000,
+      "rowLimit": 10000
+     },
+     "inputWidgets": {},
+     "nuid": "49f61b1d-f921-4a6f-9ba5-0b50aca0a175",
+     "showTitle": false,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "source": [
+    "## Findings\n",
+    "\n",
+    "The pickup borough with the most trips is **Manhattan**, with **112,028,489 trips**.\n",
+    "\n",
+    "The average `total_amount` is **23.49** for payment type 0, **30.00** for type 1, **23.75** for type 2, **9.03** for type 3, **2.16** for type 4, and **14.89** for type 5.\n",
+    "\n",
+    "## PySpark versus dbt SQL\n",
+    "\n",
+    "I would use PySpark for large-scale exploration or transformations that require distributed processing, DataFrame operations, or custom Python logic. I would use dbt SQL for tested, documented, and version-controlled transformations that create reusable analytics models."
+   ]
+  }
+ ],
+ "metadata": {
+  "application/vnd.databricks.v1+notebook": {
+   "computePreferences": null,
+   "dashboards": [],
+   "environmentMetadata": {
+    "base_environment": "",
+    "environment_version": "5"
+   },
+   "inputWidgetPreferences": null,
+   "language": "python",
+   "notebookMetadata": {
+    "pythonIndentUnit": 4
+   },
+   "notebookName": "pyspark_exploration",
+   "widgets": {}
+  },
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}

task-2/WRITEUP.md

@@ -1,24 +1,26 @@
 # Task 2 write-up: incremental build timings & Delta history
 
-Fill in after running `dbt build --select fct_trips --full-refresh` baseline followed by `dbt build --select fct_trips` incremental rerun against Databricks.
+## First build (full refresh)
 
-## First build (full / initial load with --full-refresh)
-
-- **Wall-clock time:**
-- **Notes:** (optional: warehouse size, any errors you fixed)
+- **Wall-clock time:** 20.404 seconds
+- **Result:** Completed successfully against `hyf-dbt-warehouse`.
 
 ## Second build (incremental rerun)
 
-- **Wall-clock time:**
+- **Wall-clock time:** 16.735 seconds
+- **Result:** Completed successfully using a Delta merge.
 
 ## Why was the second run faster?
 
-Write two or three sentences in your own words (see the assignment for the concepts you must name):
+During the full-refresh run, `is_incremental()` was false, so dbt processed the complete source history and recreated the table. During the incremental run, `is_incremental()` was true, and the filter compared incoming pickup dates with the maximum `pickup_date` in `{{ this }}`. This allowed dbt to process only newer data and use a Delta `MERGE`.
 
-`___`
+## Delta Table History
 
-## Delta Table History (DESCRIBE HISTORY)
+Command:
 
-Paste the output or summary of `DESCRIBE HISTORY hyf.dev_yourname.fct_trips` (showing `CREATE OR REPLACE TABLE` and `MERGE` operations) or reference a screenshot:
+`DESCRIBE HISTORY hyf.dev_mohammedalfakih.fct_trips;`
 
-`___`
+| Version | Timestamp               | Operation                         |
+| ------- | ----------------------- | --------------------------------- |
+| 3       | 2026-07-30 00:57:53 UTC | MERGE                             |
+| 1       | 2026-07-30 00:54:53 UTC | CREATE OR REPLACE TABLE AS SELECT |

task-2/dbt_project.yml

@@ -0,0 +1,25 @@
+name: "nyc_taxi_borough_daily"
+version: "1.0.0"
+config-version: 2
+
+# This project connects to the profile of the same name in profiles.yml.
+profile: "nyc_taxi"
+
+model-paths: ["models"]
+macro-paths: ["macros"]
+test-paths: ["tests"]
+
+target-path: "target"
+clean-targets:
+  - "target"
+  - "dbt_packages"
+
+# Folder-level materialization defaults. Staging models stay as views (cheap,
+# always fresh); the mart is built as a table (queried repeatedly by the
+# dashboard). You can override per model with {{ config(materialized='...') }}.
+models:
+  nyc_taxi_borough_daily:
+    staging:
+      +materialized: view
+    marts:
+      +materialized: table

task-2/macros/safe_divide.sql

@@ -0,0 +1,7 @@
+-- safe_divide(numerator, denominator)
+-- Returns numerator / denominator, or NULL when denominator is 0 or NULL.
+
+
+{% macro safe_divide(numerator, denominator) %}
+    ({{ numerator }}::numeric / NULLIF({{ denominator }}::numeric, 0))
+{% endmacro %}