From cae6e2957cd3b8db27f4fd71b94f17f4471cf4f6 Mon Sep 17 00:00:00 2001 From: Carr1005 Date: Fri, 11 Sep 2026 12:06:43 -0700 Subject: [PATCH 1/3] Add the Qdrant Edge course and its AI-Coding-Lab spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New course: Building On-Device AI Memory with Qdrant Edge (5 lessons, Dylan Couzon). Materials are the notebook dump (L3-L5 + helper.py + requirements.txt + ro_shared_data), the five transcripts, the slide descriptions, and environment.md — the AI Coding Lab runtime the build targets. The spec is filed as the variant, spec.coding-agent-lab.md, because it is generated against that environment rather than for an unknown one. Two consequences: - The Decision Ledger carries four rows, not ten. Five of the six learner-context dimensions are fixed by the lab (project and goal are §1, data is the §5 fixture corpus, the runtime is the container, and the course uses no LLM at all, so there is no provider to pick). What remains is course decisions: on-device vs cloud sync, one store vs a store per purpose, similarity vs similarity+freshness, plus the scope-boundary row that presents §1 at the gate. - Environment facts are labelled [environment] throughout and given their own CTX-E provenance row, so the course is never credited with port-4000 rules, the container's provider keys, or the choices-fence question mechanism. The slides earned two changes the other materials could not: R7 now states that the filter runs inside the query rather than trimming its results, with AC7b to discriminate the two (a garden-filtered food query must still return the lone garden note); and R5 records that the course drafted a merged single-inbox display and dropped it — it appears in no video, while the shipped code renders separate lanes and names the blended list as the thing not to do. The decks also carry a third, abandoned lesson numbering, noted in CTX-E so it is never cited. No canonical spec.md yet, so /prepare-build will refuse on this course until one exists. Co-Authored-By: Claude Opus 5 (1M context) --- .../materials/environment.md | 513 +++ .../materials/notebooks/.gitkeep | 0 .../sc-Qdrant-C3-notebook-context.md | 3653 +++++++++++++++++ .../slides/sc-Qdrant-C3-slide-descriptions.md | 440 ++ .../materials/transcripts/.gitkeep | 0 .../transcripts/sc-Qdrant-C3-transcripts.md | 121 + .../spec.coding-agent-lab.md | 772 ++++ 7 files changed, 5499 insertions(+) create mode 100644 courses/building-on-device-ai-memory-with-qdrant-edge/materials/environment.md create mode 100644 courses/building-on-device-ai-memory-with-qdrant-edge/materials/notebooks/.gitkeep create mode 100644 courses/building-on-device-ai-memory-with-qdrant-edge/materials/notebooks/sc-Qdrant-C3-notebook-context.md create mode 100644 courses/building-on-device-ai-memory-with-qdrant-edge/materials/slides/sc-Qdrant-C3-slide-descriptions.md create mode 100644 courses/building-on-device-ai-memory-with-qdrant-edge/materials/transcripts/.gitkeep create mode 100644 courses/building-on-device-ai-memory-with-qdrant-edge/materials/transcripts/sc-Qdrant-C3-transcripts.md create mode 100644 courses/building-on-device-ai-memory-with-qdrant-edge/spec.coding-agent-lab.md diff --git a/courses/building-on-device-ai-memory-with-qdrant-edge/materials/environment.md b/courses/building-on-device-ai-memory-with-qdrant-edge/materials/environment.md new file mode 100644 index 0000000..0cc4f91 --- /dev/null +++ b/courses/building-on-device-ai-memory-with-qdrant-edge/materials/environment.md @@ -0,0 +1,513 @@ +# environment.md — The AI Coding Lab, for lab authors and their coding agents + +This document describes the **AI Coding Lab** (the `agentic-chat-ide` product) from the +point of view of someone building a *new lab* to run inside it. It is written to be +handed to a coding agent (Claude Code, OpenCode, Codex, …) at the start of a lab-authoring +session, so that the agent makes choices that fit this environment and writes those +choices into its own spec. It complements, and does not replace, the [README](README.md) +"Authoring a lab" section, which is the field-level reference for the file formats. + +If you are the agent building a lab: read this whole file before drafting anything. +Section 10 is a checklist of what your lab spec must decide. + +--- + +## 1. What the lab is + +The AI Coding Lab is a browser IDE in which a **learner talks to a coding agent** that +builds software for them inside a sandboxed workspace. There is no terminal and no +notebook. The learner's only instrument is the chat; the agent does all reading, writing, +running and testing. The IDE is deployed one container per learner on DeepLearning.AI's +platform (Pantheon) and opened from a course page, on anything from a desktop browser to +a phone. + +A lab is **content dropped into a generic image**, not a fork of the app: + +| Piece | Where it lands | What it does | +|---|---|---| +| **Instruction artifact** (zip) | `/assignment` | `instructions.md` (the learner brief), optional `rubric.json` (grading), optional `resources.json` + `resources/` (visual aids) | +| **Workspace artifact** (zip) | `/workspace` | Starter code, datasets, step specs — anything the agent should find on disk | + +Both zips are uploaded as course items and downloaded by an init container at sandbox +start. Zip the *contents* (files at the archive root, not nested in a folder). No image +rebuild is needed unless the lab needs heavy system dependencies (see §9). + +The learner's experience, in order: + +1. The IDE opens on an empty chat that says "Welcome to AI coding lab! Click on the button + in the top left to open your instructions." (On phones a hint dot points at the + Instructions tab.) **`instructions.md` is the front door of the lab.** +2. The learner reads the brief, copies a kickoff prompt from it, and sends it to the agent. +3. The agent asks design questions (rendered as clickable answer buttons), builds the app, + serves it on port 4000, and the IDE's preview pane shows it live. +4. Optionally, the learner submits the workspace for grading and gets a rubric-scored + report in the chat. + +--- + +## 2. The three surfaces + +**Sidebar** (left) has three tabs: + +- **History** — the learner's chats. A learner can have many chats; each is an independent + agent session (see §4). +- **Files** — the workspace tree. Clicking a file opens it in the artifact pane. Learners + can create, rename, delete, and edit files, and download the project as a zip. + Dependency and build directories (`node_modules`, `.venv`, `__pycache__`, `dist`, …) are + filtered out; IDE-internal entries (`.chats`, `.git`, `.env`, `.agentic-chat-ide`) are + never shown. +- **Instructions** — `instructions.md`, rendered as markdown. Static; it has no live state. + +**Chat** (center): the thread. Assistant messages are markdown with syntax highlighting. +Tool calls (bash, read, write, edit, search, webfetch, …) render as collapsible cards. +Special constructs — answer buttons, "Learn more" chips, "Dive deeper" cards — are +described in §5. The header shows the model and running cost. + +**Artifact pane** (right): shows one of + +- a workspace file in CodeMirror (read-only by default, Edit toggle, Cmd/Ctrl+S saves; + markdown gets a rendered preview), auto-opened on the first file the agent touches; +- a **learning resource** (an HTML page or image from the assignment) — see §5.3; +- the **live app preview** — an iframe of whatever is listening on port 4000 — see §6. + +**Phones and tablets.** Below 1024px the sidebar becomes a drawer; below 768px the IDE is a +chat-first stack where files, resources and the app preview open as full-screen layers +with a Back button. Everything a lab ships (brief, resources, questions, preview) is used +on a 390px phone by real learners. See [MOBILE-SPEC.md](MOBILE-SPEC.md) for the details. + +--- + +## 3. The assignment directory (`/assignment`) + +Only these files are read. Anything else in the directory is ignored. + +### 3.1 `instructions.md` (required) + +Ordinary markdown, rendered in the Instructions tab and handed verbatim to the grader. +Three things make it more than a static page: + +- **Copy pills on code blocks.** Every fenced code block gets a **Copy** button. On a phone + the tap also closes the drawer and drops the text into the chat composer (creating a + new chat if none is open). **Put the lab's kickoff prompt in the first code fence**, and + one fence per step for multi-step labs, so the learner's action is one tap. +- **Resource markers** (`[phrase](resource:)`, `[phrase](resource-inline:)`) work + here exactly as in chat (§5.3), so the brief can open a visual before the learner ever + talks to the agent. +- **Images and other assets.** Anything under `/assignment` (dotfiles excluded) is served + at `/api/instructions/assets/`, so `![pet](/api/instructions/assets/imgs/pet.png)` + works. Put images in a subfolder and include it in the zip. + +Do **not** put a ` ```choices ` fence in `instructions.md`: the fence is parsed only in +assistant chat messages. In the brief it renders as a code block with a copy pill. + +A good brief tells the learner: what they will build, roughly how long it takes, how the +lab is structured (steps and what each produces), the exact prompt(s) to send, the reminder +that the app must be served on **port 4000** to preview it, and how the work is evaluated. + +### 3.2 `rubric.json` (optional) + +```json +{ + "subject": "Memory service · spec + implementation", + "emphasis": "One-line steer the grader reads before the dimensions (optional).", + "dimensions": [ + { "name": "Spec completeness", "guide": "What the grader checks, in free text." }, + { "name": "Internal consistency", "guide": "…" } + ] +} +``` + +Up to 8 dimensions; each is scored 0–10 by an LLM grader that explores the workspace +itself (§8). Missing or malformed ⇒ a generic four-dimension *spec-quality* rubric +(completeness, clarity, consistency, testability). Write `guide` as instructions to a +grader that will read the learner's files, not as a description for the learner. + +### 3.3 `resources.json` + `resources/` (optional) + +Registry of visual aids the IDE can open beside the chat: + +```json +{ "resources": [ + { "id": "memory-types", "title": "The seven kinds of memory", + "file": "resources/memory-types.html", "kind": "on-mention", + "topics": ["the seven types of memory"] }, + { "id": "step1-toolbox-memory", "title": "Toolbox memory: SQL vs vector store", + "file": "resources/step1-toolbox-memory.html", "kind": "dive-deeper", + "topics": ["toolbox memory"] } ] } +``` + +- `id`: slug (`[a-z0-9_-]`), unique. Markers reference it. +- `file`: relative to `/assignment`; `.html`/`.htm` (fully self-contained) or + `.png/.jpg/.gif/.webp/.svg`. Max 20 entries. Entries with a bad id, duplicate id, + missing file, or unsupported extension are **silently dropped** — check + `GET /api/resources` to see what registered. +- `kind`: `on-mention` (default) or `dive-deeper` — decides the affordance (§5.3). +- `topics`: phrases that tell the **agent** when to surface the resource. Entries with + topics are listed in the agent's standing instructions; entries without topics are + only ever opened from markers you write yourself. + +**Building a resource page.** It is viewed in a same-origin iframe at anything from 320px +to a desktop pane, with no network access beyond the IDE origin. So: one file, inline CSS +and JS, no CDN or font fetches, a viewport meta tag, fluid layout with no horizontal +overflow at 320/480/800px, text ≥11px rendered, touch targets ≥44×44px, accessible names +and states on every control, keyboard reachable. Prefer one shared worked example that the +page reworks under each option over abstract prose — the page opens *next to a question* +and should show the learner how each answer plays out. `e2e/helpers/audit.js` exports +`assertWidgetAudit(page, id, width)` for checking these rules in Playwright. + +--- + +## 4. The agent the learner talks to + +**Engine.** OpenCode CLI (pinned, currently 1.15.x), run once per learner turn as +`opencode run --format json --session --model ""`. +The IDE is the only client; the learner never sees OpenCode's own UI. + +**Models.** On the platform: OpenAI `gpt-5.6-luna` (default), `gpt-5.6-terra`, Anthropic +`claude-sonnet-5`, `claude-sonnet-4-6`, all through DeepLearning.AI's proxy with a +platform-injected token. Google is not available there. The learner can pick a model per +conversation from a picker. **Write specs that work on all four**, or say in the brief +which model the lab assumes. Locking the picker to one model is an image change (§9). + +**Tools and permissions.** The agent has bash, file read/write/edit, glob/grep/list, +webfetch, websearch, todo, and skills. **Every tool is auto-allowed.** `opencode run` is +non-interactive: there are no permission prompts, and nothing pauses for approval. Assume +the agent will run whatever it decides to run. + +**Standing instructions.** OpenCode reads `/workspace/AGENTS.md` at every turn. The IDE +**writes this file itself on every boot** from a fixed template plus a generated appendix +listing your topic-bearing resources. It tells the agent to: + +- build only inside `/workspace` and never touch the IDE's own tree at `/app`; +- serve the app on **port 4000** bound to `0.0.0.0`, frontend and API from **one** server, + long-running servers in the background; +- follow the user's persistence spec literally (in-memory means in-memory); +- reuse the preinstalled stack (§9) and create venvs with `--system-site-packages`; +- use the ` ```choices ` fence for every fixed-choice question, one question per message, + fence last, then stop and wait (§5.1); +- mark the first substantive mention of each registered topic with a resource marker + (§5.3). + +Because the IDE rewrites `AGENTS.md`, **a workspace zip must not ship its own +`AGENTS.md`** — it will be overwritten. Lab-specific agent guidance goes into spec files +seeded into the workspace (§7) and into the kickoff prompt that tells the agent to read +them. + +**What the agent cannot see.** `/assignment` is outside the workspace and is not readable +by the agent. If the agent needs the brief, the rubric, or any lab text, seed a copy into +the workspace. + +**Sessions and memory.** One chat = one OpenCode session; context carries across turns in +that chat and survives the learner navigating away and back, page reloads, and container +restarts. A **new chat starts with no memory of earlier chats**. The only state that +crosses chats is what is on disk in the workspace. Design multi-step labs so that each +step can begin from the files alone (the memory lab's Step 2 spec opens with "read Step +1's configuration before asking anything; do not re-ask it"). + +**Resilience.** A turn that dies on a transient API error, or streams nothing for 180s, is +auto-retried in the same session up to 5 times. Long silent builds (a large `pip install`, +a model download) count as "nothing streamed" only if the agent emits no events at all; +in practice tool calls keep the turn alive. + +**Shell containment.** `kill`/`pkill`/`killall`/`fuser` are shimmed so the agent cannot +kill the IDE's own processes. The agent runs as the non-root `learner` user. + +--- + +## 5. Chat rendering features a lab should design around + +These are the affordances that make a lab feel guided rather than free-form. A lab spec +should say explicitly where each is used. + +### 5.1 The `choices` fence — clickable answers + +An assistant message containing + +````markdown +**Question 1 of 2 — Toolbox memory** + +How should the agent find a tool it can't name? …explanation… + +```choices +SQL table (SQLite) +Vector store (Chroma) +``` +```` + +renders the fence as answer buttons. Exact behaviour, from the parser: + +- The opening line is three backticks immediately followed by `choices`; the block ends at + the next three backticks. One option per line; leading `-`, `*`, or `1.`/`1)` prefixes + are stripped; blank lines are dropped. A fence with no options renders nothing. +- A click sends the option line **verbatim** as an ordinary user message. The learner may + type a free-form answer instead. Specs must accept both ("a typed answer that names an + option counts the same as a click"). +- Buttons are enabled only on the **latest** assistant message while no turn is streaming. + Older questions become inert, so the agent must not rely on the learner revisiting them. +- A line matching `**Question N of M …**` before the fence draws a **question box** around + everything from that label through the buttons (and a following Dive-deeper card). Use + the label to make multi-question sequences trackable; the count is how learner and + agent both know nothing was skipped. +- The base `AGENTS.md` already tells the agent to use the fence for any fixed-choice + question and to ask one question per message. A spec that says "use the interactive + structured-question tool" maps onto this fence; the memory lab's specs spell out the + fence explicitly anyway, which is more robust. + +Design guidance: put explanation in prose *above* the fence, keep option labels short +(suffixes like "(course default)" / "(Recommended)" are fine), one decision per message, +and tell the agent what to do after each answer ("acknowledge and record — no +correction"). Explanations, considerations and trade-offs belong in the spec so the agent +has them to paraphrase. + +### 5.2 Tool cards and file touches + +Every tool call the agent makes is visible as a collapsible card. Writing a file also +opens it in the artifact pane on first touch. Nothing here needs authoring, but it means +the learner *sees* the agent read spec files and run tests — a lab can lean on that +("watch the agent run the Step 1 tests before it asks the next question"). + +### 5.3 Resource markers — "Learn more" and "Dive deeper" + +A marker is a markdown link whose destination is a registered resource id: + +| Marker | Resource `kind` | Renders as | +|---|---|---| +| `[phrase](resource:)` | `on-mention` | The phrase as prose plus a small **Learn more** chip after it | +| `[phrase](resource:)` | `dive-deeper` | The phrase as plain prose; a **Dive deeper** card (title + "See how this choice plays out") **below the answer buttons** of the last `choices` fence, or at the end of the message if there is none | +| `[phrase](resource-inline:)` | any | A **Dive deeper** card **in place of the phrase**, at that position — put it on a line of its own | + +Rules: unregistered ids degrade to plain text (a typo never breaks a message); markers are +ignored inside code blocks; the agent is told to use only listed ids, at most one marker +per resource per message, never inside a `choices` fence, heading, or code block, and +never to mention the syntax to the learner. Dive-deeper cards stay clickable on old +messages; Learn-more chips too. + +Who writes markers: **you**, in `instructions.md` and in seeded spec files that quote the +exact marker for a specific message; **the agent**, for any resource with `topics`, +whenever it substantively discusses that topic. If a resource must appear at one precise +moment (e.g. a "why these are settled" card under a settled-architecture list), don't +rely on topic matching — put the literal marker in the step spec with a table of +"where → marker", as the memory lab does. + +Where it opens: desktop and tablet in the artifact pane (replacing whatever file or +preview was there); phone as a full-screen layer with Back returning to the chat or the +Instructions tab with scroll position preserved. + +### 5.4 Grading report + +When a grading run finishes, a report card (overall score, verdict, per-dimension scores, +"what's working", "push further") renders at the bottom of every chat. See §8. + +--- + +## 6. The app preview (port 4000) + +The artifact pane can show the learner's running app. The rules are strict and the agent +is told them, but a lab's *starter code and spec* must obey them too: + +- **One server, port 4000, bound to 0.0.0.0.** The IDE polls port 4000 every 3s and + auto-reveals the preview when something starts listening. Nothing else is previewable. + The default shape is a backend (FastAPI/uvicorn or Express) serving static files plus + API routes. The agent is told to run a second process only if asked. +- **The preview is reverse-proxied**, not loaded from `localhost:4000`. The iframe (and + "open in new tab") loads `/app-preview/` on the IDE's own origin, and the server forwards + it to `127.0.0.1:4000`. Consequences: + - **All URLs in the learner's app must be relative.** ` +""" + + +def _upload_status(folder): + files = _uploaded(folder) + if not files: + return "Nothing uploaded yet." + return f"{len(files)} ready: {', '.join(f.name for f in files)}" + + +def _reset_uploads_once(): + """Start each fresh kernel with empty upload folders. + + The flag keeps a same-kernel re-run of the first cell from deleting photos + the student just uploaded. Restarting the kernel reloads this module, + resets the flag, and clears the previous session's files. + """ + global _UPLOADS_RESET + if _UPLOADS_RESET: + return + for folder in (TEACH_DIR, TEST_DIR): + path = Path(folder) + if path.is_dir(): + for uploaded in path.iterdir(): + if uploaded.is_file() or uploaded.is_symlink(): + uploaded.unlink() + _UPLOADS_RESET = True + + +def photo_uploader(kind): + """Show the upload button for teaching photos or the held-out test photo.""" + from IPython.display import display, HTML + + _reset_uploads_once() + if kind == "teach": + box = _upload_box( + TEACH_DIR, "Teach with these", + "Two or more photos of one object, from different angles or " + "in different places.", "upload-teach") + elif kind == "test": + box = _upload_box( + TEST_DIR, "Test with this one", + "One more photo of the same object. Keep it out of the teaching " + "photos.", "upload-test") + else: + raise ValueError('kind must be "teach" or "test"') + display(HTML(box)) + + +def object_photos(kind): + """Read teaching photos or the held-out photo, with a bundled fallback.""" + _reset_uploads_once() + teach = [str(f) for f in _uploaded(TEACH_DIR)] + test = [str(f) for f in _uploaded(TEST_DIR)] + if kind == "teach": + if not teach: + print("Bundled example: 2 rubber duck photos") + return [EXAMPLE_OBJECT + f"{i}.jpg" for i in (1, 2)] + if len(teach) < 2: + raise ValueError( + f"Found {len(teach)} teaching photo(s). Upload two or more, " + "or leave this empty for the bundled example." + ) + print(f"{len(teach)} photos to teach with") + return teach + if kind == "test": + if not test: + print("Bundled example: 1 held-out rubber duck photo") + return EXAMPLE_OBJECT + "3.jpg" + if len(test) != 1: + raise ValueError( + f"Found {len(test)} test photos. Upload exactly one, or leave " + "this empty for the bundled example." + ) + print("1 photo held back to test") + return test[0] + raise ValueError('kind must be "teach" or "test"') + + +def show_example_object(): + """Show the bundled rubber duck: two photos to teach with, one held out. + + This runs before the uploaders so the shape of the job is on screen + first: a few views of one object, plus one more photo of it kept aside + to test with. Upload your own object or run straight through with the + duck; `object_photos` falls back to these files either way. + """ + return show_images( + [EXAMPLE_OBJECT + f"{i}.jpg" for i in (1, 2, 3)], + captions=["teach with this", "teach with this", + "held out, to test with"], + title='Bundled example: "rubber duck"', + ) + + +# Shard setup, the offline guard, and benchmark filler ----------------- +# How much of a write to show back. Small: the recording frame is tall, +# not endless, and the point is what landed, not all of it. +PREVIEW_ROWS = 4 +PREVIEW_PHOTOS = 6 + + +def load_memories(path, source_type=None): + """Read a memories JSON file, optionally keeping one source type.""" + memories = json.load(open(path)) + if source_type: + memories = [m for m in memories if m["source_type"] == source_type] + return memories + + +MODEL_NAME = {"text": "Nomic", "image": "CLIP"} + + +def new_shard(directory, text=None, image=None): + """Create an empty shard on a clean directory, one named vector space + per width given: `new_shard("./day_shard", text=768, image=512)`. + + Lesson 3 writes this out in full, an `EdgeConfig` holding one + `EdgeVectorParams` per named vector and then `EdgeShard.create`. After + that first time the repeat lives here. The widths stay arguments, so a + lesson still says on screen which spaces it has and how wide they are. + """ + sizes = {name: size for name, size in + (("text", text), ("image", image)) if size} + config = EdgeConfig(vectors={ + name: EdgeVectorParams(size=size, distance=Distance.Cosine) + for name, size in sizes.items() + }) + shard = EdgeShard.create(fresh_start(directory), config) + print("Memory bank ready:", ", ".join( + f"{name} {size}-d ({MODEL_NAME[name]})" + for name, size in sizes.items())) + return shard + + +def store_notes(shard, notes, preview=True): + """Embed text and voice notes with Nomic and store one point per note. + + The write this wraps is taught in Lesson 3: embed the note, build a + Point with the note as payload, upsert. A voice note embeds its + transcript. + """ + vectors = embed_text([m.get("note") or m["transcript"] for m in notes]) + shard.update(UpdateOperation.upsert_points([ + Point(id=m["id"], vector={"text": v}, payload=m) + for m, v in zip(notes, vectors) + ])) + if preview: + show(memories_table(notes[:PREVIEW_ROWS], + f"Stored {len(notes)} notes")) + else: + print(f"Stored {len(notes)} notes") + + +def store_photos(shard, folder, start_id=1000, preview=True): + """Embed a folder of photos with CLIP and store them in the image vector.""" + photos = sorted(Path(folder).glob("*.jpg")) + vectors = embed_image([str(p) for p in photos]) + shard.update(UpdateOperation.upsert_points([ + Point(id=start_id + i, vector={"image": v}, + payload={"file": p.name, "source_type": "photo"}) + for i, (p, v) in enumerate(zip(photos, vectors)) + ])) + shard.optimize() + if preview: + show(show_images([str(p) for p in photos[:PREVIEW_PHOTOS]], + captions=[p.name for p in photos[:PREVIEW_PHOTOS]], + per_row=6, + title=f"Stored {len(photos)} photos · " + f"{shard.info().points_count} memories")) + else: + print(f"Stored {len(photos)} photos · " + f"{shard.info().points_count} memories") + + +def store_photo_memories(shard, photos, folder, preview=True): + """Embed photo memories with CLIP and store one point per photo.""" + vectors = embed_image([f"{folder}/{m['file']}" for m in photos]) + shard.update(UpdateOperation.upsert_points([ + Point(id=m["id"], vector={"image": v}, payload=m) + for m, v in zip(photos, vectors) + ])) + shard.optimize() + if preview: + show(day_photos(photos[:PREVIEW_PHOTOS], folder, + f"Stored {len(photos)} photos · " + f"{shard.info().points_count} memories")) + else: + print(f"Stored {len(photos)} photos · " + f"{shard.info().points_count} memories") + + +def store_day(shard, notes, photos, folder="./ro_shared_data/images"): + """Load a day's notes and photos into one shard, notes then photos. + + Both writes are taught in Lessons 3 and 4; this is the repeat, so the + lesson that builds the assistant only shows the memory it adds itself. + """ + store_notes(shard, notes, preview=False) + store_photo_memories(shard, photos, folder, preview=False) + + +def save_shard(shard): + """Compact the shard and write it to disk. + + `optimize` builds the index over what was just written, `flush` puts it + on disk, so a taught memory survives the device losing power. + """ + shard.optimize() + shard.flush() + + +def memory_receipt(shard, subject): + """Confirm what the assistant now holds: the new memory, and the total.""" + return receipt_table([ + ("taught", f"{subject}: photo + note"), + ("total", f"{shard.info().points_count} memories"), + ], title="Assistant memory ready") + + +def text_search(shard, query, query_filter=None, limit=4): + """Embed a query with Nomic and return the nearest text memories. + + Lesson 3 teaches both halves in the open: the raw nearest query, then + `query_filter` narrowing it. + """ + return shard.query(QueryRequest( + query=Query.Nearest(embed_query(query), using="text"), + filter=query_filter, + limit=limit, + with_payload=True, + )) + + +def photo_search(shard, description, limit=1): + """Embed a description with CLIP and return the nearest photos. + + The raw cross-modal call is taught in Lesson 3. + """ + return shard.query(QueryRequest( + query=Query.Nearest(embed_query_clip(description), using="image"), + limit=limit, + with_payload=True, + )) + + +def freshness_ranking(latest_time, half_life, weight): + """Build a ranking formula: similarity plus a bonus for recent memories. + + `Expression.Decay` turns a memory's `timestamp` into a freshness score + between 0 and 1: 1 at `latest_time`, half of that `half_life` seconds + earlier, fading from there. `weight` is the most that freshness can add + to the similarity score, so meaning still leads and recency breaks ties. + Memories with no timestamp are treated as current rather than dropped. + """ + freshness = Expression.Decay( + DecayKind.Exp, + Expression.Variable("timestamp"), + target=Expression.Constant(latest_time), + midpoint=0.5, + scale=half_life, + ) + return Formula( + Expression.Sum([ + Expression.Variable("$score"), + Expression.Mult([Expression.Constant(weight), freshness]), + ]), + defaults={"timestamp": latest_time}, + ) + + +def recent_text_search(shard, question, ranking, limit=3): + """Search text memories by meaning, then re-rank with `ranking`. + + Two steps in one request: the prefetch pulls a wider set of candidates + by meaning, and the formula from `freshness_ranking` rescores just those + and keeps the top `limit`. + """ + return shard.query(QueryRequest( + limit=limit, + prefetches=[Prefetch( + limit=20, + query=Query.Nearest(embed_query(question), using="text"), + )], + query=ranking, + with_payload=True, + )) + + +def recall(shard, question): + """One question, two lanes: text memories by Nomic, photos by CLIP. + + Lesson 4 builds this in the open; later lessons import it. Extra text + hits are fetched so one lane cannot crowd out the other. + """ + text_hits = text_search(shard, question, limit=10) + photo_hits = photo_search(shard, question, limit=3) + return { + "Photos": [h for h in photo_hits + if h.payload.get("source_type") == "photo"][:1], + "Voice Notes": [h for h in text_hits + if h.payload.get("source_type") == "voice"][:3], + "Text Notes": [h for h in text_hits + if h.payload.get("source_type") == "text"][:3], + } + + +def recognize(shard, photo): + """Return the closest stored photo. + + Nearest search always returns a match. The notebook applies the decision + threshold separately. + """ + top_match = shard.query(QueryRequest( + query=Query.Nearest(embed_image([photo])[0], using="image"), + limit=1, + with_payload=True, + ))[0] + return top_match + + +def seed_objects(shard, folder="./ro_shared_data/bank"): + """Store three known objects and show them, one photo each at ids 0-2. + + Writes exactly what Lesson 5's `teach` writes: the photo's CLIP vector + with the label as payload, flushed to disk so a taught memory survives + a power cut. + """ + seeds = {"a bicycle": "bicycle.jpg", + "chess pieces": "chess_set.jpg", + "a camera": "camera.jpg"} + paths = [f"{folder}/{f}" for f in seeds.values()] + vectors = embed_image(paths) + shard.update(UpdateOperation.upsert_points([ + Point(id=i, vector={"image": v}, + payload={"label": label, "file": p}) + for i, (label, p, v) in enumerate(zip(seeds, paths, vectors)) + ])) + shard.optimize() + shard.flush() + show(show_images(paths, captions=list(seeds))) + + +def load_day_and_history(folder="./ro_shared_data"): + """The assistant's full memory: today's captures plus the earlier days. + + Returns today's memories, text and voice notes from all days, and today's + photos. + """ + day = load_memories(f"{folder}/memories.json") + history = load_memories(f"{folder}/recent_days.json") + notes = [m for m in day + history + if m["source_type"] in ("text", "voice")] + photos = [m for m in day if m["source_type"] == "photo"] + return day, notes, photos + + +def cloud_client(collection, text=768, image=512): + """Connect to the cluster in QDRANT_URL / QDRANT_API_KEY, collection ready. + + Pasting credentials is how a student opts in, so there is no second + switch to forget: returns a ready qdrant_client.QdrantClient with the + collection created, or None when either variable is empty. A collection + that already exists is left alone and None comes back, because the + course never deletes one. None means every memory stays on the device, + and the calling cell says so. + + The server's vectors_config is the twin of the `EdgeConfig` Lesson 3 + writes out, so it lives here rather than repeating on screen. + """ + import os + if not (os.getenv("QDRANT_URL") and os.getenv("QDRANT_API_KEY")): + return None + from qdrant_client import QdrantClient, models + client = QdrantClient(url=os.environ["QDRANT_URL"], + api_key=os.environ["QDRANT_API_KEY"]) + if client.collection_exists(collection): + print(f"{collection} already exists on the cluster.", + "Delete it there first, or rename the collection here.") + return None + client.create_collection(collection, vectors_config={ + "text": models.VectorParams(size=text, + distance=models.Distance.COSINE), + "image": models.VectorParams(size=image, + distance=models.Distance.COSINE), + }) + return client + + +def cloud_points(shard, limit=1000): + """Every point in a shard, in the shape a Qdrant server takes. + + Same ids, same vectors, same payloads: the format does not change on + the way up. Reading them back is the `ScrollRequest` the appendix + shows in the open one cell earlier. + """ + from qdrant_client import models + records, _ = shard.scroll(ScrollRequest(limit=limit, with_payload=True, + with_vector=True)) + return [models.PointStruct(id=r.id, vector=r.vector, payload=r.payload) + for r in records] + + +def push_note(client, collection, point_id, note): + """Store one text note straight onto the cluster, as another device would. + + Used where the write belongs to some other device in the fleet. A write + the student makes themselves stays in the notebook. + """ + from qdrant_client import models + client.upsert(collection, points=[models.PointStruct( + id=point_id, + vector={"text": embed_text([note])[0]}, + payload={"source_type": "text", "note": note}, + )]) + + +def fetch_snapshot(collection, dest, manifest=None): + """Download a shard snapshot from the cluster in QDRANT_URL to a file. + + With a manifest (from `EdgeShard.snapshot_manifest`), asks the server + for a partial snapshot holding only what this shard is missing. + """ + import os + import urllib.request + base_url = os.environ["QDRANT_URL"] + headers = {"api-key": os.getenv("QDRANT_API_KEY") or ""} + if manifest is None: + url = f"{base_url}/collections/{collection}/shards/0/snapshot" + req = urllib.request.Request(url, headers=headers) + else: + url = (f"{base_url}/collections/{collection}" + "/shards/0/snapshot/partial/create") + headers["Content-Type"] = "application/json" + req = urllib.request.Request( + url, data=json.dumps(manifest).encode(), + headers=headers, method="POST") + with urllib.request.urlopen(req) as response, open(dest, "wb") as f: + f.write(response.read()) + return dest + + +def file_size(path): + """A downloaded snapshot's size, in whichever unit reads better.""" + import os + kb = os.path.getsize(path) / 1024 + return f"{kb / 1024:.1f} MB" if kb >= 1024 else f"{kb:.0f} KB" + + +def fresh_start(directory): + """Delete any previous run's shard directory and recreate it empty. + + A shard the notebook still has bound holds its files open, and Edge flushes + when that object is dropped. Deleting the files first makes the flush fail + inside a destructor, which surfaces as a Rust panic rather than a Python + error. So close any shard the notebook still holds before removing + anything: that makes re-running a setup cell in a live kernel safe, + instead of only working on a clean top-to-bottom run. + + The notebook's own namespace is `__main__`, whatever the call depth, so + this works whether a lesson calls it directly or `new_shard` does. + + Only a directory that already holds a shard can have one open on it, so + the sweep is skipped for a directory that does not exist yet. Without + that guard, opening a second shard closes the first, which is exactly + what a lesson holding two shards at once needs not to happen. + """ + import sys + if any(Path(directory).glob("*")): + notebook = vars(sys.modules.get("__main__", None)) or {} + for value in list(notebook.values()): + if isinstance(value, EdgeShard): + try: + value.close() + except Exception: + pass + gc.collect() + shutil.rmtree(directory, ignore_errors=True) + Path(directory).mkdir(parents=True, exist_ok=True) + return directory + + +# The views the lessons print ------------------------------------------ +QDRANT_RED = "#DC244C" +INK = "#28324D" +MUTED = "#6B7280" +LINE = "#E5E7EB" +FONT = "font-family:system-ui,-apple-system,'Segoe UI',Roboto,sans-serif" +FIG_W = 8.0 # recording frame is 8 wide by 9 high + +MODALITY_COLOR = {"photo": QDRANT_RED, "voice": "#8547FF", "text": INK} +MODALITY_EMOJI = {"photo": "📷", "voice": "🎙️", "text": "📝"} + + +def show(view): + """Put a view on screen from inside a helper, mid-cell.""" + from IPython.display import display + display(view) + + +def _html(markup): + from IPython.display import HTML + return HTML(markup) + + +def _esc(value): + import html + return html.escape(str(value)) + + +def _score_cell(score, peak): + """A score with a proportional bar behind it, still selectable as text. + + Pass `peak=None` where the column mixes score scales: a bar would invite + a comparison between a CLIP score and a Nomic one, which means nothing. + """ + pct = max(0.0, min(1.0, score / peak)) * 100 if peak else 0 + return (f'{score:.3f}') + + +def _table(headers, rows, title=None, caption=None, widths=None, above=""): + """Render a table as HTML. `rows` holds ready-made strings. + + `widths` is one CSS width per column. Without it the browser sizes every + column by its content, which lets a three-character Price column sit on + top of Category and squeezes the memory itself into what is left. The + memory is what the reader came to read, so it gets most of the width. + """ + cols = ("" + + "".join(f'' for w in widths) + + "") if widths else "" + layout = "table-layout:fixed;" if widths else "" + head = "".join( + f'{_esc(h)}' + for h in headers) + body = "".join( + f'{r}' + for i, r in enumerate(rows)) + parts = [f'
'] + if title: + parts.append(f'
{_esc(title)}
') + parts.append(above) + parts.append(f'{cols}' + f'{head}' + f'{body}
') + if caption: + parts.append(f'
{_esc(caption)}
') + parts.append("
") + return "".join(parts) + + +def _cell(value, align="left", color=INK, weight=400, size=13.5, + nowrap=False): + return (f'{_esc(value)}') + + +def _memory_text(payload): + """The words of a memory: its note, its transcript, or its filename.""" + return payload.get("note") or payload.get("transcript") or payload.get("file", "") + + +def _price(payload): + return f"${payload['price']:.0f}" if payload.get("price") is not None else "-" + + +def _has_price(payloads): + """Whether the price column is worth a column at all.""" + return any(p.get("price") is not None for p in payloads) + + +def _result_row(hit, peak, price=True, when=False): + """One table row for a search hit: score, category, price, the memory.""" + p = hit.payload + return (_score_cell(hit.score, peak) + + _cell(p.get("category", "-"), color=MUTED) + + (_cell(_price(p), align="right", color=MUTED) if price else "") + + (_cell(_hhmm(p["timestamp"], "%b %d") if p.get("timestamp") + else "-", color=MUTED, nowrap=True) if when else "") + + _cell(_memory_text(p))) + + +def results_table(hits, title=None, caption=None, query=None, + when=False, price=None): + """Show search hits as a table: score, category, price, and the memory. + + `query` puts the question above the answers, where a reader looks for it. + An empty result renders as the same view with nothing in it, so asking + before and after storing reads as one picture with a row count. + `when` adds the memory's date, for the lessons that rank by it. + `price` defaults to showing the column when a memory carries one. + """ + asked = _query_block(query) if query else "" + if not hits: + return _html( + f'
' + f'
{_esc(title or "No memories found")}
' + + asked + + f'
' + f'Nothing stored yet.
' + + (f'
' + f'{_esc(caption)}
' if caption else '') + '
') + peak = max((h.score for h in hits), default=1.0) + if price is None: + price = _has_price([h.payload for h in hits]) + rows = [_result_row(h, peak, price=price, when=when) for h in hits] + headers = (["Score", "Category"] + (["Price"] if price else []) + + (["When"] if when else []) + ["Memory"]) + widths = {(True, True): ("10%", "12%", "9%", "10%", "59%"), + (True, False): ("10%", "13%", "9%", "68%"), + (False, True): ("10%", "13%", "11%", "66%"), + (False, False): ("10%", "14%", "76%")}[(price, when)] + return _html(_table(headers, rows, title, caption, widths, above=asked)) + + +def memories_table(memories, title=None): + """Show stored memories, which carry no score: category, price, words.""" + price = _has_price(memories) + rows = [_cell(m.get("category", "-"), color=MUTED) + + (_cell(_price(m), align="right", color=MUTED) if price else "") + + _cell(_memory_text(m)) for m in memories] + headers = ["Category"] + (["Price"] if price else []) + ["Memory"] + widths = ("14%", "9%", "77%") if price else ("15%", "85%") + return _html(_table(headers, rows, title, widths=widths)) + + +def receipt_table(rows, title="Restart receipt"): + """Render a list of (label, value) pairs as a two-column table.""" + cells = [_cell(label, color=MUTED) + + _cell(value, weight=700) for label, value in rows] + return _html(_table(["", ""], cells, title)) + + +def _thumb_data_uri(path, size=120): + """Return a base64 data URI for a small thumbnail of an image file.""" + import base64 + import io + from PIL import Image + img = Image.open(path).convert("RGB") + img.thumbnail((size, size)) + buf = io.BytesIO() + img.save(buf, format="JPEG", quality=85) + return "data:image/jpeg;base64," + base64.b64encode(buf.getvalue()).decode() + + +def show_photo_results(hits, image_dir, query): + """Show the photo a description retrieved, large, with its score. + + Only the closest match is shown: the search always returns something, and + a big single answer says that more clearly than a row of runners-up. + """ + hero = hits[0] + uri = _thumb_data_uri(Path(image_dir) / hero.payload["file"], size=420) + return _html( + f'
' + f'
closest photo
' + f'
"{_esc(query)}"
' + f'' + f'
' + f'{_esc(hero.payload["file"])} · similarity ' + f'' + f'{hero.score:.3f}
') + + +def _query_block(text): + """The question, shown above its answers rather than captioned under them.""" + return (f'
{_esc(text)}
') + + +def _cosine(a, b): + """Cosine similarity between two embeddings, the score a search returns.""" + dot = sum(x * y for x, y in zip(a, b)) + return dot / (math.sqrt(sum(x * x for x in a)) + * math.sqrt(sum(y * y for y in b))) + + +def _payload_value(key, value): + """A payload value as stored, glossed where the raw number is unreadable. + + A timestamp is an epoch integer on disk and stays one here, with the time + it stands for beside it: the point of the card is what was really stored. + """ + if key == "timestamp" and isinstance(value, (int, float)): + return f"{value} ({_hhmm(value, '%b %d, %H:%M')})" + return value + + +def vector_preview(text, vector, shown=8): + """One memory beside the start of the vector it became. + + The caption names the vector and counts its dimensions, because those are two + different things and the lesson leans on the difference: one note becomes + one vector, and that vector is a list of coordinates. + """ + numbers = ", ".join(f"{x:+.3f}" for x in vector[:shown]) + return _html( + f'
' + f'
' + f'"{_esc(text)}"
' + f'
' + f'[{numbers}, ...]
' + f'
' + f'{len(vector)} dimensions' + f'
') + + +def point_card(record, vector_name="text", shown=6): + """One point in full: its id, its vector, and its payload. + + Takes anything carrying `.id`, `.vector`, and `.payload`, so it renders a + `Point` the lesson just built as readily as a record read back off disk. + """ + vector = record.vector[vector_name] + # shown=0 where the cell above already printed the vector in full: + # the card is then about the point's shape, not its values twice. + values = ", ".join(f"{x:+.3f}" for x in vector[:shown]) + rows = [_cell(k, color=MUTED, nowrap=True) + _cell(_payload_value(k, v)) + for k, v in record.payload.items()] + return _html( + f'
' + f'
' + f'Point {_esc(record.id)}
' + f'
{_esc(vector_name)}: ' + f'{f"[{values}, ...] " if shown else ""}' + f'{len(vector)} dimensions
' + + _table(["Field", "Value"], rows, + widths=("20%", "80%")) + '
') + + +def _hhmm(ts, fmt="%H:%M"): + from datetime import datetime, timezone + return datetime.fromtimestamp(ts, timezone.utc).strftime(fmt) + + +def day_summary(memories): + """One line: how many captures the day holds, by source type.""" + counts = Counter(m["source_type"] for m in memories) + print(len(memories), "captures:", + ", ".join(f"{v} {k}" for k, v in sorted(counts.items()))) + + +def day_photos(memories, image_dir, title=None): + """A wrapping strip of the day's photos, each stamped with its time.""" + photos = sorted((m for m in memories if m.get("file")), + key=lambda m: m["timestamp"]) + cards = "".join( + f'
' + f'' + f'
' + f'{_hhmm(m["timestamp"])} · {_esc(m.get("store") or m.get("location", ""))}' + f'
' for m in photos) + return _html( + f'
' + f'
📷 {_esc(title) if title else f"{len(photos)} photos, in time order"}
' + f'
{cards}
') + + +def day_notes(memories, limit=10): + """The day's voice and text notes as a table: time, kind, and words. + + Shows the earliest `limit` notes, because a whole day of them runs off + the bottom of the recording frame. Pass a bigger number for more, or + `limit=None` for the lot. The title says how many there are either way. + """ + notes = sorted((m for m in memories if not m.get("file")), + key=lambda m: m["timestamp"]) + total = len(notes) + if limit is not None: + notes = notes[:limit] + # A couple of notes are stamped the evening before, so the date is shown + # whenever the set spans more than one day. + spans_days = len({_hhmm(m["timestamp"], "%j") for m in notes}) > 1 + fmt = "%b %d · %H:%M" if spans_days else "%H:%M" + rows = [] + for m in notes: + kind = m["source_type"] + rows.append(_cell(_hhmm(m["timestamp"], fmt), color=MUTED, + nowrap=True) + + _cell(f'{MODALITY_EMOJI.get(kind, "")} {kind}', + color=MODALITY_COLOR.get(kind, INK), weight=600, + nowrap=True) + + _cell(_memory_text(m))) + title = (f"{total} voice and text notes" if len(notes) == total + else f"First {len(notes)} of {total} voice and text notes") + return _html(_table(["Time", "Kind", "Note"], rows, title, + widths=("15%", "12%", "73%"))) + + +def answers_table(answers, image_dir=None, title="Ask your assistant"): + """Questions answered from both lanes at once: the words and the picture. + + `answers` is a list of (question, text_hit, photo_hit). The two scores sit + in their own columns and carry no bars because they come from different + retrieval lanes and do not compare. When both lanes land on the same + point, the row says so: one memory, reached two ways. + """ + rows = [] + for question, words, photo in answers: + p = words.payload + when = ("you taught this" if p.get("label") + else _hhmm(p["timestamp"], "%b %d") if p.get("timestamp") + else "-") + same = photo is not None and photo.id == words.id + file = photo.payload.get("file", "") if photo is not None else "" + if file and "/" not in file and image_dir: + file = str(Path(image_dir) / file) + thumb = (f'' + if file else "") + tag = ('
✅ same memory
' if same else "") + photo_cell = (f'{thumb}
' + f'{photo.score:.3f}
{tag}' + if photo is not None else _cell("-")) + rows.append(_cell(question, weight=600) + + _cell(_memory_text(p)) + + _score_cell(words.score, None) + + _cell(when, color=MUTED, nowrap=True) + + photo_cell) + return _html(_table(["You asked", "It remembered", "Words", "When", + "Photo"], rows, title)) + + +def memory_inbox(sections, image_dir, min_text_score=None, + min_photo_score=None): + """One question's answers as side-by-side lanes, never one blended list. + + `sections` maps a lane title to a list of ScoredPoint, and each lane is a + column ranked best first. Columns rather than a wrapping row because the + lanes are the point: a reader compares down one lane and across three, + and a wrap would put two lanes' cards on the same line. Every lane is + drawn even when empty. Scores below their model's cutoff are dimmed. + """ + def card(h): + p = h.payload + cutoff = min_photo_score if p.get("file") else min_text_score + weak = cutoff is not None and h.score < cutoff + ctx = " · ".join(x for x in [_hhmm(p["timestamp"]) if p.get("timestamp") else "", + p.get("store"), p.get("location"), + _price(p) if p.get("price") is not None else ""] + if x) + if p.get("file"): + uri = _thumb_data_uri(Path(image_dir) / p["file"], 260) + body = (f'') + else: + body = (f'
{_esc(_memory_text(p))}
') + tag = (' · weaker' + if weak else '') + head = (f'
{_esc(ctx)}
' + if ctx else '') + return (f'
' + f'{head}' + f'{body}
' + f'' + f'{h.score:.3f}{tag}
') + + lanes = [] + for title, hits in sections.items(): + ranked = sorted(hits, key=lambda h: h.score, reverse=True) + inner = ("".join(card(h) for h in ranked) if ranked else + f'
' + f'No matches
') + lanes.append( + f'
' + f'
{_esc(title)}
{inner}
') + return _html( + f'
' + f'
' + f'{"".join(lanes)}
') + + +def threshold_calibration(object_dir, scene_dir, selected, current=None): + """Calibrate image recognition on held-out and unrelated photos. + + Each bundled object keeps its last view out of the teaching set. Positive + scores compare that held-out view with its own taught views. Negative + scores compare held-out and scene photos with taught views of a different + object. `current` may be `(label, score)` for the student's held-out photo. + """ + groups = {} + for path in sorted(Path(object_dir).glob("*.jpg")): + groups.setdefault(path.stem.rsplit("_", 1)[0], []).append(path) + + taught = {label: views[:-1] for label, views in groups.items()} + held_out = {label: views[-1] for label, views in groups.items()} + scenes = sorted(Path(scene_dir).glob("*.jpg")) + paths = [p for views in groups.values() for p in views] + scenes + vectors = dict(zip(paths, embed_image([str(p) for p in paths]))) + + same = [] + different = [] + for label, query in held_out.items(): + same.append(max(_cosine(vectors[query], vectors[p]) + for p in taught[label])) + different.extend( + _cosine(vectors[query], vectors[p]) + for other, views in taught.items() if other != label + for p in views) + different.extend( + _cosine(vectors[scene], vectors[p]) + for scene in scenes for views in taught.values() for p in views) + + same_min = min(same) + different_max = max(different) + fig, ax = plt.subplots(figsize=(FIG_W, 3.2)) + + # Hundreds of negative dots hide the boundary that matters. Show their + # tested range and hardest example instead, then keep each positive test. + different_min = max(0.4, min(different)) + ax.hlines(0, different_min, different_max, color="#C8CEDD", + linewidth=12, alpha=0.65) + ax.scatter([different_max], [0], s=70, color="#8F98B2", + edgecolors="white", linewidths=0.8, zorder=3) + same_y = [0.96 + 0.04 * (i % 3) for i in range(len(same))] + ax.scatter(same, same_y, s=58, color="#009688", + edgecolors="white", linewidths=0.8, zorder=3) + if different_max < same_min: + ax.axvspan(different_max, same_min, color="#009688", alpha=0.09) + ax.axvline(selected, color=QDRANT_RED, lw=2, ls="--") + ax.annotate(f"highest {different_max:.3f}", + xy=(different_max, 0), xytext=(-5, 14), + textcoords="offset points", ha="right", color=MUTED, + fontsize=9) + ax.annotate(f"lowest {same_min:.3f}", + xy=(same_min, 1), xytext=(5, -18), + textcoords="offset points", ha="left", color="#00796B", + fontsize=9) + if current: + ax.scatter([current[1]], [1.25], marker="*", s=170, + color=QDRANT_RED, edgecolors="white", linewidths=0.8, + zorder=4) + ax.annotate(f"your view {current[1]:.3f}", + xy=(current[1], 1.25), xytext=(7, 0), + textcoords="offset points", va="center", + color=QDRANT_RED, fontsize=9, fontweight="bold") + ax.set_xlim(0.4, 1.0) + ax.set_ylim(-0.35, 1.50) + ax.set_yticks([0, 1]) + ax.set_yticklabels([f"{len(different)} non-matches", + f"{len(same)} held-out matches"]) + ax.set_xlabel("similarity to nearest taught view") + ax.set_title("Where should the threshold go?", loc="left") + ax.spines[["top", "right", "left"]].set_visible(False) + ax.tick_params(axis="y", length=0) + fig.tight_layout() + plt.show() + + +# Measured by .build/measure_latency.py on Apple M5 Pro, CPU only, +# Python 3.14.6, 300 queries per size, median reported. The lesson draws +# this curve rather than timing anything live: a 250,000-vector store does +# not fit in the course container, and a number timed on a shared sandbox +# moves on every re-run. Re-measure and paste if the model or Edge changes. +LOOKUP_LATENCY = [ + (1_000, 0.054), + (5_000, 0.179), + (25_000, 0.526), + (100_000, 1.439), + (250_000, 2.646), +] +QUERY_EMBED_MS = 5.50 +MEASURED_ON = "Apple M5 Pro, CPU only, median of 300 queries per size" + + +def latency_curve(points=LOOKUP_LATENCY, embed_ms=QUERY_EMBED_MS): + """Vector lookup time as the store grows, against the cost of embedding. + + Two local costs make up one answer. Embedding the question is a fixed + price the encoder charges whatever the store holds, so it draws as a + flat line. The lookup grows with the number of vectors, so it draws as + a curve. Both stay on the device, which keeps this a breakdown of where + the time goes and never a comparison against a server. + """ + sizes = [n for n, _ in points] + times = [ms for _, ms in points] + fig, ax = plt.subplots(figsize=(FIG_W, 3.5)) + + ax.axhline(embed_ms, color="#8F98B2", lw=2, ls="--") + ax.annotate(f"query embedding · {embed_ms:.1f} ms", + xy=(sizes[0], embed_ms), xytext=(0, 7), + textcoords="offset points", color="#5C6480", fontsize=9.5) + ax.plot(sizes, times, color=QDRANT_RED, lw=2.2, marker="o", + markersize=6, markerfacecolor="white", + markeredgecolor=QDRANT_RED, markeredgewidth=2, zorder=3) + for n, ms in points: + ax.annotate(f"{ms:.2f} ms", xy=(n, ms), xytext=(0, 11), + textcoords="offset points", ha="center", va="bottom", + color=QDRANT_RED, fontsize=9.5, fontweight="bold") + + ax.set_xscale("log") + ax.set_xticks(sizes) + ax.set_xticklabels([f"{n:,}" for n in sizes]) + ax.minorticks_off() + ax.set_xlabel("memories (log scale)") + ax.set_ylabel("milliseconds") + ax.set_ylim(0, max(embed_ms, max(times)) * 1.35) + ax.set_title("Vector lookup as memory grows", loc="left") + ax.spines[["top", "right"]].set_visible(False) + fig.tight_layout() + # One provenance line, kept shorter than the figure: a wider line makes + # matplotlib grow the whole figure past the recording frame. + fig.subplots_adjust(bottom=0.24) + fig.text(0.012, 0.03, MEASURED_ON, fontsize=9.5, color="#4E5366") + plt.show() + + +def show_images(paths, captions=None, per_row=None, title=None, height=170): + """A row of photos with a caption under each, sized to the video frame. + + Every photo gets the same box, whatever its shape, so a row reads as a row + rather than a ragged stack. `contain` keeps the whole subject visible, + which matters when the photo is the evidence. + """ + paths = list(paths) + # Fill the row with what there is, rather than leaving a hole for photos + # that were never uploaded. + per_row = per_row or min(len(paths), 4) or 1 + gap = 14 + cards = [] + for i, path in enumerate(paths): + caption = captions[i] if captions and i < len(captions) else "" + cards.append( + f'
' + f'' + f'
{_esc(caption)}
') + head = (f'
📷 {_esc(title)}
' if title else '') + return _html(f'
{head}' + f'
' + f'{"".join(cards)}
') + + +def show_taught_photos(paths, subject): + """Show the photos used to teach one subject.""" + return show_images( + paths, + title=f'Taught "{subject}" from {len(paths)} photos', + ) + + +def recognition_result(query_photo, top_match, is_known=None, image_dir=None, + threshold=None): + """The photo you showed beside the closest memory, with the verdict. + + `is_known=None` shows the nearest memory without making a decision. + Otherwise it says whether the score cleared the threshold. Both photos + get the same box so the pair reads as a comparison. + """ + stored = top_match.payload["file"] + if image_dir: + stored = str(Path(image_dir) / stored) + label = top_match.payload.get("label", "UNKNOWN") + if is_known is None: + verdict, color, mark = f"Closest memory: {label}", INK, "" + elif is_known: + verdict, color, mark = label, "#009688", "✅" + else: + verdict, color, mark = "UNKNOWN", MUTED, "❓" + + def pane(path, caption): + return (f'
' + f'' + f'
{_esc(caption)}
') + + detail = "" + if is_known is not None and threshold is not None: + if is_known: + detail = (f'{top_match.score:.3f} clears the ' + f'{threshold:.3f} threshold.') + else: + detail = (f'{top_match.score:.3f} is below the ' + f'{threshold:.3f} threshold.') + detail = (f'
{_esc(detail)}
') + + heading = f"{mark} " if mark else "" + score_detail = (f" · closest: {_esc(label)} · similarity " + f"{top_match.score:.3f}" if is_known is False else + f" · similarity {top_match.score:.3f}") + return _html( + f'
' + f'
{heading}{_esc(verdict)}' + f'' + f'{score_detail}
{detail}' + f'
' + + pane(query_photo, "the photo you showed it") + + pane(stored, f"closest memory: {label}") + + '
') + + +# Speech to text for the voice notes ----------------------------------- +WHISPER_MODEL = "whisper-base" + + +@lru_cache(maxsize=1) +def _asr_model(): + import onnx_asr + return onnx_asr.load_model(WHISPER_MODEL, providers=["CPUExecutionProvider"]) + + +def transcribe(audio_path): + """Transcribe one audio file to text with a local Whisper model.""" + return _asr_model().recognize(audio_path).strip() + + +def transcribe_notes(memories, audio_dir): + """Transcribe every voice note in place, then free the speech model. + + Releasing Whisper before the embedding models load keeps the notebook + inside the 4 GB sandbox budget. + """ + voice = [m for m in memories if m["source_type"] == "voice"] + for m in voice: + m["transcript"] = transcribe(f"{audio_dir}/{m['audio_file']}") + _asr_model.cache_clear() + return voice +``` + +--- + +## Shared: data (`ro_shared_data/`) + +The notebooks read this folder as `./ro_shared_data/` relative to the lesson directory. On the +course platform the folder is copied into each lesson; in this repo it sits once at the root. + +### `notes_on_shared_data.txt` (verbatim) + +``` +This directory can contain read only data shared by all the lessons. +This is a good location for shared images or datasets used by multiple lessons. + +When the repo is deployed to the platform, the shared directory will be copied to a subdirectory in each lesson. +So, when referring to the shared data in your code, you should use +./ro_shared_data/shared_file +and not +../shared_data/shared_file. + +Because each lesson is potentially in its own docker container, data written to files in the shared directory is not visible to other lessons. +If you wish, for example to write to a database in lesson1 and have it visible in lesson2, you will need to copy the data from the L1 location to a location under the L2 directory that is not shared. +This is helpful in that students may not run the labs in order, or at all, and will still expect later labs to run in the same way as they see in the video. + +``` + +### `memories.json` + +42 records, one day of captures: 20 `text`, 17 `photo`, 5 `voice`; ids 0 to 41; timestamps +1694624400 to 1694728800 (13 to 14 September 2023, UTC). Fields: `id`, `source_type`, `category`, +`location`, `timestamp`, then `note` and optional `price` for text, `file` for photos, +`transcript` and `audio_file` for voice, and `store` on some records. Read by Lesson 3 (text +records only), Lesson 4 (all), and Lesson 5 (all, through `load_day_and_history`). + +```json +[ + { + "id": 0, + "source_type": "text", + "category": "food", + "location": "5th St", + "timestamp": 1694679480, + "note": "Great little coffee place on 5th with outdoor seating and fast wifi", + "price": 6.0 + }, + { + "id": 1, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1694685600, + "note": "Standup with Sarah moved to Thursday to review the Q3 roadmap" + }, + { + "id": 2, + "source_type": "text", + "category": "errands", + "location": "Home", + "timestamp": 1694682000, + "note": "Pick up dry cleaning before Friday, ticket is on the fridge" + }, + { + "id": 3, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1694691000, + "note": "Idea: batch the weekly report so it drafts itself every Monday" + }, + { + "id": 4, + "source_type": "text", + "category": "social", + "location": "Home", + "timestamp": 1694718000, + "note": "Mum's new address is 14 Elm Court, buzzer 3" + }, + { + "id": 5, + "source_type": "text", + "category": "shopping", + "location": "Mall", + "timestamp": 1694704680, + "note": "Liked the black and white running shoes at the mall, about $45", + "price": 45.0 + }, + { + "id": 6, + "source_type": "text", + "category": "social", + "location": "Home", + "timestamp": 1694725200, + "note": "Book club is reading the new sci-fi novel, we meet next Tuesday" + }, + { + "id": 7, + "source_type": "text", + "category": "health", + "location": "Home", + "timestamp": 1694692800, + "note": "Dentist appointment confirmed for next Wednesday at 2pm" + }, + { + "id": 8, + "source_type": "text", + "category": "food", + "location": "Downtown", + "timestamp": 1694694600, + "note": "Try the new ramen place downtown, everyone raves about the tonkotsu", + "price": 18.0 + }, + { + "id": 9, + "source_type": "text", + "category": "home", + "location": "Home", + "timestamp": 1694712600, + "note": "Water the plants twice a week while the amaryllis is blooming" + }, + { + "id": 10, + "source_type": "text", + "category": "health", + "location": "Gym", + "timestamp": 1694676600, + "note": "Renewed the gym membership, locker code is 4471", + "price": 40.0 + }, + { + "id": 11, + "source_type": "text", + "category": "work", + "location": "Park", + "price": 14, + "timestamp": 1694707200, + "note": "Found a quiet cafe with good wifi to work from near the park" + }, + { + "id": 12, + "source_type": "text", + "category": "errands", + "location": "Home", + "timestamp": 1694721600, + "note": "Remember to call the landlord about the leaking tap" + }, + { + "id": 13, + "source_type": "text", + "category": "food", + "location": "5th St", + "timestamp": 1694678400, + "note": "New bakery on the corner does an amazing morning cronut", + "price": 4.0 + }, + { + "id": 14, + "source_type": "text", + "category": "errands", + "location": "Home", + "timestamp": 1694728800, + "note": "Parking permit renewal is due at the end of the month" + }, + { + "id": 15, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1694700000, + "note": "Meeting notes: ship the edge demo before the conference" + }, + { + "id": 16, + "source_type": "text", + "category": "home", + "location": "Home", + "timestamp": 1694714400, + "note": "Bought a new houseplant for the kitchen windowsill", + "price": 12.0 + }, + { + "id": 17, + "source_type": "text", + "category": "travel", + "location": "Station", + "timestamp": 1694727000, + "note": "Weekend trip: check train times to the coast on Saturday" + }, + { + "id": 18, + "source_type": "text", + "category": "food", + "location": "Office", + "timestamp": 1694624400, + "note": "Coffee run to the espresso bar near the office", + "price": 3.0 + }, + { + "id": 19, + "source_type": "text", + "category": "work", + "location": "Home", + "timestamp": 1694635200, + "note": "Late night fixing the deploy pipeline, finally green" + }, + { + "id": 20, + "source_type": "voice", + "category": "food", + "location": "Downtown", + "timestamp": 1694697840, + "transcript": "Note to self, the ramen downtown was incredible, fourteen dollars and worth it, sat right by the window", + "audio_file": "ramen.wav", + "price": 14.0, + "store": "Ramen-ya" + }, + { + "id": 21, + "source_type": "voice", + "category": "shopping", + "location": "Mall", + "timestamp": 1694705400, + "transcript": "Reminder, buy a birthday present for Alex this week, maybe those headphones he mentioned", + "audio_file": "birthday.wav" + }, + { + "id": 22, + "source_type": "voice", + "category": "work", + "location": "Office", + "timestamp": 1694685960, + "transcript": "Quick memo, the standup is moved to Thursday, tell the rest of the team", + "audio_file": "standup.wav" + }, + { + "id": 23, + "source_type": "voice", + "category": "travel", + "location": "Station", + "timestamp": 1694682720, + "transcript": "Parked the bike near the station, second rack from the entrance", + "audio_file": "bike.wav" + }, + { + "id": 24, + "source_type": "voice", + "category": "errands", + "location": "Home", + "timestamp": 1694719800, + "transcript": "Just remembered, we are low on coffee at home, grab a bag on the way back", + "audio_file": "coffee.wav" + }, + { + "id": 25, + "source_type": "photo", + "category": "food", + "location": "5th St", + "timestamp": 1694679119, + "file": "coffee.jpg", + "store": "Blue Cup" + }, + { + "id": 26, + "source_type": "photo", + "category": "food", + "location": "5th St", + "timestamp": 1694678760, + "file": "bakery.jpg", + "price": 4.0 + }, + { + "id": 27, + "source_type": "photo", + "category": "food", + "location": "Downtown", + "timestamp": 1694696400, + "file": "restaurant.jpg", + "store": "Elizabeth's" + }, + { + "id": 28, + "source_type": "photo", + "category": "food", + "location": "Downtown", + "timestamp": 1694697120, + "file": "ramen.jpg", + "price": 14.0, + "store": "Ramen-ya" + }, + { + "id": 29, + "source_type": "photo", + "category": "food", + "location": "Home", + "timestamp": 1694721600, + "file": "pizza.jpg", + "price": 18.0 + }, + { + "id": 30, + "source_type": "photo", + "category": "shopping", + "location": "Mall", + "timestamp": 1694704680, + "file": "sneakers.jpg", + "price": 45.0, + "store": "SportsWorld" + }, + { + "id": 31, + "source_type": "photo", + "category": "travel", + "location": "5th St", + "timestamp": 1694682360, + "file": "bicycle.jpg" + }, + { + "id": 32, + "source_type": "photo", + "category": "travel", + "location": "Station", + "timestamp": 1694727360, + "file": "train.jpg" + }, + { + "id": 33, + "source_type": "photo", + "category": "travel", + "location": "Downtown", + "timestamp": 1694707920, + "file": "street.jpg" + }, + { + "id": 34, + "source_type": "photo", + "category": "travel", + "location": "Park", + "timestamp": 1694709000, + "file": "park.jpg" + }, + { + "id": 35, + "source_type": "photo", + "category": "home", + "location": "Home", + "timestamp": 1694712960, + "file": "plant.jpg", + "price": 12.0 + }, + { + "id": 36, + "source_type": "photo", + "category": "home", + "location": "Home", + "timestamp": 1694715120, + "file": "kitchen.jpg" + }, + { + "id": 37, + "source_type": "photo", + "category": "social", + "location": "Park", + "timestamp": 1694710800, + "file": "dog.jpg" + }, + { + "id": 38, + "source_type": "photo", + "category": "shopping", + "location": "Mall", + "timestamp": 1694706480, + "file": "book.jpg", + "price": 15.0 + }, + { + "id": 39, + "source_type": "photo", + "category": "work", + "location": "Park", + "timestamp": 1694707560, + "file": "laptop.jpg" + }, + { + "id": 40, + "source_type": "photo", + "category": "work", + "location": "Office", + "timestamp": 1694685600, + "file": "meeting.jpg" + }, + { + "id": 41, + "source_type": "photo", + "category": "health", + "location": "Gym", + "timestamp": 1694676240, + "file": "gym.jpg" + } +] +``` + +### `recent_days.json` + +102 records from the earlier days: 81 `text`, 21 `voice`, no photos; ids 1000 to 1101; timestamps +1692862200 to 1694635200 (24 August to 13 September 2023, UTC). Same fields as `memories.json`. +Read by Lesson 5 only, through `load_day_and_history`, as the assistant's history. Its note +id 1033 (30 August, "locker code at the gym is 2280") and `memories.json` id 10 (14 September, +"locker code is 4471") are the two gym locker memories the freshness section is about. + +```json +[ + { + "id": 1000, + "source_type": "text", + "category": "food", + "location": "5th St", + "timestamp": 1692862200, + "note": "Coffee at Blue Cup, espresso, $5", + "price": 5.0, + "store": "Blue Cup" + }, + { + "id": 1001, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1692867600, + "note": "Standup at 9:30, review dashboard update" + }, + { + "id": 1002, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1692885600, + "note": "Sarah says the kitchen refactor PR is live" + }, + { + "id": 1003, + "source_type": "voice", + "category": "errands", + "location": "Station", + "timestamp": 1692901800, + "transcript": "Note to self, buy milk and bread on the way home" + }, + { + "id": 1004, + "source_type": "text", + "category": "health", + "location": "Gym", + "timestamp": 1692945000, + "note": "Gym early, shoulders and back day" + }, + { + "id": 1005, + "source_type": "text", + "category": "social", + "location": "Office", + "timestamp": 1692957600, + "note": "Meeting with Alex about the coast trip weekend" + }, + { + "id": 1006, + "source_type": "text", + "category": "food", + "location": "Downtown", + "timestamp": 1692964800, + "note": "Lunch at Ramen-ya, tonkotsu broth, $14", + "price": 14.0, + "store": "Ramen-ya" + }, + { + "id": 1007, + "source_type": "text", + "category": "social", + "location": "Home", + "timestamp": 1692975600, + "note": "Book club Friday, new sci-fi novel starts" + }, + { + "id": 1008, + "source_type": "voice", + "category": "social", + "location": null, + "timestamp": 1692990000, + "transcript": "Reminder, Mum's birthday, she wants a scarf" + }, + { + "id": 1009, + "source_type": "text", + "category": "home", + "location": "Home", + "timestamp": 1693000800, + "note": "Fixed the dripping kitchen tap finally" + }, + { + "id": 1010, + "source_type": "text", + "category": "travel", + "location": null, + "timestamp": 1693033200, + "note": "Commute took 35 min, bike path was blocked" + }, + { + "id": 1011, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1693042200, + "note": "Debug production issue in user auth service" + }, + { + "id": 1012, + "source_type": "text", + "category": "food", + "location": "Office", + "timestamp": 1693054800, + "note": "Sandwich from deli, turkey Swiss, $8.50", + "price": 8.5 + }, + { + "id": 1013, + "source_type": "voice", + "category": "errands", + "location": "Home", + "timestamp": 1693065600, + "transcript": "Quick memo, need to buy new lightbulbs for the living room" + }, + { + "id": 1014, + "source_type": "text", + "category": "social", + "location": "Park", + "timestamp": 1693123200, + "note": "Coffee with Alex at Park, meeting was good" + }, + { + "id": 1015, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1693134000, + "note": "Stakeholder meeting about Q3 roadmap" + }, + { + "id": 1016, + "source_type": "text", + "category": "errands", + "location": "Downtown", + "timestamp": 1693146600, + "note": "Groceries, $52, including meat for Sunday roast", + "price": 52.0 + }, + { + "id": 1017, + "source_type": "voice", + "category": "home", + "location": "Home", + "timestamp": 1693159200, + "transcript": "Note to self, landlord says rent stays same next year" + }, + { + "id": 1018, + "source_type": "text", + "category": "home", + "location": "Home", + "timestamp": 1693170000, + "note": "Planted basil in kitchen window, hoping it survives" + }, + { + "id": 1019, + "source_type": "text", + "category": "social", + "location": "Home", + "timestamp": 1693213200, + "note": "Finished sci-fi book, really gripping" + }, + { + "id": 1020, + "source_type": "text", + "category": "food", + "location": "Downtown", + "timestamp": 1693224000, + "note": "Lunch special at Ramen-ya, $11.50 with tea", + "price": 11.5, + "store": "Ramen-ya" + }, + { + "id": 1021, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1693234800, + "note": "Sarah's PR review needs one more iteration" + }, + { + "id": 1022, + "source_type": "voice", + "category": "travel", + "location": "Home", + "timestamp": 1693243800, + "transcript": "Reminder, bike needs new chain soon, check pressure" + }, + { + "id": 1023, + "source_type": "text", + "category": "health", + "location": null, + "timestamp": 1693252800, + "note": "Flu shot appointment next Tuesday at 3pm" + }, + { + "id": 1024, + "source_type": "text", + "category": "health", + "location": "Gym", + "timestamp": 1693290600, + "note": "Gym session, legs, feeling strong today" + }, + { + "id": 1025, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1693303200, + "note": "Code review for new API endpoints done" + }, + { + "id": 1026, + "source_type": "text", + "category": "travel", + "location": "Home", + "timestamp": 1693323000, + "note": "Planning items for weekend coast trip" + }, + { + "id": 1027, + "source_type": "voice", + "category": "errands", + "location": "Home", + "timestamp": 1693335600, + "transcript": "Note to self, return library books tomorrow morning" + }, + { + "id": 1028, + "source_type": "text", + "category": "social", + "location": "Home", + "timestamp": 1693342800, + "note": "Mum called, she likes the scarf idea" + }, + { + "id": 1029, + "source_type": "text", + "category": "food", + "location": "5th St", + "timestamp": 1693382400, + "note": "Morning coffee, oat milk cappuccino, $6", + "price": 6.0, + "store": "Blue Cup" + }, + { + "id": 1030, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1693391400, + "note": "Team standup, release planned for Friday" + }, + { + "id": 1031, + "source_type": "text", + "category": "social", + "location": "Downtown", + "timestamp": 1693400400, + "note": "Lunch meeting with Alex at downtown spot" + }, + { + "id": 1032, + "source_type": "text", + "category": "shopping", + "location": "Mall", + "timestamp": 1693411200, + "note": "Bought scarf for Mum, blue wool, $45", + "price": 45.0 + }, + { + "id": 1033, + "source_type": "voice", + "category": "health", + "location": "Gym", + "timestamp": 1693420200, + "transcript": "Quick memo, locker code at the gym is 2280" + }, + { + "id": 1034, + "source_type": "text", + "category": "travel", + "location": "Home", + "timestamp": 1693432800, + "note": "Packed for coast trip, leaving tomorrow" + }, + { + "id": 1035, + "source_type": "text", + "category": "travel", + "location": "Station", + "timestamp": 1693465200, + "note": "Coast trip day, early morning departure" + }, + { + "id": 1036, + "source_type": "text", + "category": "food", + "location": null, + "timestamp": 1693483200, + "note": "Lunch at beach cafe, fish and chips, $16", + "price": 16.0 + }, + { + "id": 1037, + "source_type": "voice", + "category": "travel", + "location": null, + "timestamp": 1693494000, + "transcript": "Note to self, Alex found great hiking trail near coast" + }, + { + "id": 1038, + "source_type": "text", + "category": "food", + "location": null, + "timestamp": 1693508400, + "note": "Hotel dinner, local seafood place, $38", + "price": 38.0 + }, + { + "id": 1039, + "source_type": "text", + "category": "social", + "location": null, + "timestamp": 1693515600, + "note": "Sunset walk along the beach with Alex" + }, + { + "id": 1040, + "source_type": "text", + "category": "food", + "location": null, + "timestamp": 1693555200, + "note": "Breakfast at cafe, avocado toast, $12", + "price": 12.0 + }, + { + "id": 1041, + "source_type": "text", + "category": "travel", + "location": null, + "timestamp": 1693566000, + "note": "Hiking trail with Alex, saw three deer" + }, + { + "id": 1042, + "source_type": "text", + "category": "shopping", + "location": null, + "timestamp": 1693584000, + "note": "Bought local pottery, $28, hand wash only", + "price": 28.0 + }, + { + "id": 1043, + "source_type": "voice", + "category": "home", + "location": "Home", + "timestamp": 1693596600, + "transcript": "Reminder, the pottery can't go in the dishwasher" + }, + { + "id": 1044, + "source_type": "text", + "category": "social", + "location": null, + "timestamp": 1693602000, + "note": "Evening at hotel, wine and board games" + }, + { + "id": 1045, + "source_type": "text", + "category": "food", + "location": null, + "timestamp": 1693641600, + "note": "Last morning at coast, relaxed breakfast" + }, + { + "id": 1046, + "source_type": "text", + "category": "errands", + "location": null, + "timestamp": 1693659600, + "note": "Drove back, stopped at roadside market" + }, + { + "id": 1047, + "source_type": "text", + "category": "home", + "location": "Home", + "timestamp": 1693677600, + "note": "Back home, unpacking from the trip" + }, + { + "id": 1048, + "source_type": "voice", + "category": "errands", + "location": "Home", + "timestamp": 1693684800, + "transcript": "Note to self, do laundry tomorrow morning" + }, + { + "id": 1049, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1693731600, + "note": "Back at work, caught up on emails" + }, + { + "id": 1050, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1693738800, + "note": "Sarah wants to pair program on auth service" + }, + { + "id": 1051, + "source_type": "text", + "category": "home", + "location": "Home", + "timestamp": 1693753200, + "note": "Finished laundry, put pottery on shelf" + }, + { + "id": 1052, + "source_type": "voice", + "category": "social", + "location": "Home", + "timestamp": 1693765800, + "transcript": "Quick reminder, book club Friday night, bring wine" + }, + { + "id": 1053, + "source_type": "text", + "category": "health", + "location": "Gym", + "timestamp": 1693771200, + "note": "Gym session, arms and core work" + }, + { + "id": 1054, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1693821600, + "note": "Pair programming with Sarah, fixed bug" + }, + { + "id": 1055, + "source_type": "text", + "category": "shopping", + "location": "Mall", + "timestamp": 1693843200, + "note": "Shopping for book club wine, red blend, $18", + "price": 18.0 + }, + { + "id": 1056, + "source_type": "voice", + "category": "home", + "location": "Home", + "timestamp": 1693854000, + "transcript": "Reminder, basil on windowsill needs water today" + }, + { + "id": 1057, + "source_type": "text", + "category": "travel", + "location": null, + "timestamp": 1693900800, + "note": "Commute smooth, bike path working again" + }, + { + "id": 1058, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1693909800, + "note": "Release prep meeting, everything good" + }, + { + "id": 1059, + "source_type": "text", + "category": "food", + "location": "Downtown", + "timestamp": 1693917000, + "note": "Team lunch at sandwich place, $9.50", + "price": 9.5 + }, + { + "id": 1060, + "source_type": "text", + "category": "health", + "location": null, + "timestamp": 1693926000, + "note": "Doctor appointment confirmed, Tuesday 2pm" + }, + { + "id": 1061, + "source_type": "voice", + "category": "social", + "location": "Home", + "timestamp": 1693936800, + "transcript": "Note to self, Mum got the scarf, she loves it" + }, + { + "id": 1062, + "source_type": "text", + "category": "health", + "location": "Park", + "timestamp": 1693944000, + "note": "Evening jog around the park, 3 miles" + }, + { + "id": 1063, + "source_type": "text", + "category": "health", + "location": "Gym", + "timestamp": 1693981800, + "note": "Gym early, chest and triceps day" + }, + { + "id": 1064, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1693990800, + "note": "Final release tests passing, shipping Friday" + }, + { + "id": 1065, + "source_type": "text", + "category": "social", + "location": "Office", + "timestamp": 1694008800, + "note": "Alex texted, wants to hike again next month" + }, + { + "id": 1066, + "source_type": "voice", + "category": "social", + "location": "Home", + "timestamp": 1694019600, + "transcript": "Reminder, need to prep something nice for book club" + }, + { + "id": 1067, + "source_type": "text", + "category": "food", + "location": "5th St", + "timestamp": 1694073600, + "note": "Friday coffee, celebrating the release", + "price": 6.0, + "store": "Blue Cup" + }, + { + "id": 1068, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1694080800, + "note": "Release deployed successfully, no issues" + }, + { + "id": 1069, + "source_type": "text", + "category": "food", + "location": "Downtown", + "timestamp": 1694088000, + "note": "Celebration lunch with Sarah and team", + "price": 15.0 + }, + { + "id": 1070, + "source_type": "text", + "category": "social", + "location": "Home", + "timestamp": 1694098800, + "note": "Left work early, heading to book club" + }, + { + "id": 1071, + "source_type": "voice", + "category": "social", + "location": "Home", + "timestamp": 1694113200, + "transcript": "Book club night, great discussion, wine perfect" + }, + { + "id": 1072, + "source_type": "text", + "category": "social", + "location": "Home", + "timestamp": 1694167200, + "note": "Book club moves to Alex place next time, 22 Birch Lane" + }, + { + "id": 1073, + "source_type": "text", + "category": "food", + "location": "Downtown", + "timestamp": 1694174400, + "note": "Brunch at the cafe, pancakes and coffee, $14", + "price": 14.0 + }, + { + "id": 1074, + "source_type": "text", + "category": "errands", + "location": "Downtown", + "timestamp": 1694185200, + "note": "Grocery shopping, ingredients for new recipe", + "price": 38.0 + }, + { + "id": 1075, + "source_type": "voice", + "category": "food", + "location": "Home", + "timestamp": 1694196000, + "transcript": "Note to self, try the Thai curry recipe this week" + }, + { + "id": 1076, + "source_type": "text", + "category": "health", + "location": "Gym", + "timestamp": 1694203200, + "note": "Gym session, cardio and weights" + }, + { + "id": 1077, + "source_type": "text", + "category": "travel", + "location": "Park", + "timestamp": 1694242800, + "note": "Morning bike ride, clearing my head" + }, + { + "id": 1078, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1694253600, + "note": "Monday standup, planning next sprint" + }, + { + "id": 1079, + "source_type": "text", + "category": "food", + "location": "Office", + "timestamp": 1694264400, + "note": "Lunch, tried Thai curry, really good, $11", + "price": 11.0 + }, + { + "id": 1080, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1694273400, + "note": "Sarah wants to discuss refactoring payment module" + }, + { + "id": 1081, + "source_type": "voice", + "category": "health", + "location": "Home", + "timestamp": 1694286000, + "transcript": "Reminder, doctor appointment tomorrow, bring insurance card" + }, + { + "id": 1082, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1694343600, + "note": "Code review for payment refactoring PR" + }, + { + "id": 1083, + "source_type": "text", + "category": "food", + "location": "Office", + "timestamp": 1694352600, + "note": "Quick lunch, sandwich from deli, $9", + "price": 9.0 + }, + { + "id": 1084, + "source_type": "voice", + "category": "health", + "location": "Office", + "timestamp": 1694354400, + "transcript": "Quick memo, doctor appointment in an hour" + }, + { + "id": 1085, + "source_type": "text", + "category": "health", + "location": null, + "timestamp": 1694361600, + "note": "Doctor visit went well, all clear" + }, + { + "id": 1086, + "source_type": "text", + "category": "food", + "location": "Home", + "timestamp": 1694370600, + "note": "Made Thai curry at home, turned out great" + }, + { + "id": 1087, + "source_type": "text", + "category": "health", + "location": "Gym", + "timestamp": 1694417400, + "note": "Gym session, back to routine" + }, + { + "id": 1088, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1694426400, + "note": "Payment refactoring PR approved, merging today" + }, + { + "id": 1089, + "source_type": "text", + "category": "social", + "location": "Downtown", + "timestamp": 1694433600, + "note": "Lunch with Alex, planning next coast trip" + }, + { + "id": 1090, + "source_type": "text", + "category": "travel", + "location": "Home", + "timestamp": 1694444400, + "note": "Bike chain needs replacing, will do this weekend" + }, + { + "id": 1091, + "source_type": "voice", + "category": "shopping", + "location": "Station", + "timestamp": 1694455200, + "transcript": "Note to self, buy new bike chain at the station shop" + }, + { + "id": 1092, + "source_type": "text", + "category": "social", + "location": "Home", + "timestamp": 1694462400, + "note": "Evening walk with Alex around neighborhood" + }, + { + "id": 1093, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1694516400, + "note": "Sprint planning session for next cycle" + }, + { + "id": 1094, + "source_type": "text", + "category": "shopping", + "location": "Station", + "timestamp": 1694534400, + "note": "Bought new bike chain, $32 at station shop", + "price": 32.0 + }, + { + "id": 1095, + "source_type": "voice", + "category": "travel", + "location": "Home", + "timestamp": 1694547000, + "transcript": "Reminder, replace the bike chain tomorrow afternoon" + }, + { + "id": 1096, + "source_type": "text", + "category": "social", + "location": "Home", + "timestamp": 1694552400, + "note": "Watched sci-fi movie, good one" + }, + { + "id": 1097, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1694588400, + "note": "Office wifi password is on the whiteboard by the kitchen" + }, + { + "id": 1098, + "source_type": "text", + "category": "work", + "location": "Office", + "timestamp": 1694599200, + "note": "Final checks on deployed feature, no issues" + }, + { + "id": 1099, + "source_type": "text", + "category": "travel", + "location": "Home", + "timestamp": 1694617200, + "note": "Replaced bike chain finally, runs smooth" + }, + { + "id": 1100, + "source_type": "voice", + "category": "social", + "location": "Home", + "timestamp": 1694628000, + "transcript": "Note to self, Alex wants to hike again next weekend" + }, + { + "id": 1101, + "source_type": "text", + "category": "health", + "location": "Gym", + "timestamp": 1694635200, + "note": "Evening gym session, finished strong" + } +] +``` + +### `audio/` (6 files) + +`audio/README.md` (verbatim): + +```markdown +# Voice-note audio + +Five short voice memos, one per voice capture in `ro_shared_data/memories.json`, plus `question.wav`, a spoken question Lesson 4 asks the assistant in section 5. Lesson 4 transcribes them on-device with a small Whisper model, so the "voice" modality runs a real speech-to-text step. Each memo maps to its note through the `audio_file` field in `ro_shared_data/memories.json`. `question.wav` stands on its own and belongs to no memory. +``` +Files: `bike.wav`, `birthday.wav`, `coffee.wav`, `question.wav`, `ramen.wav`, `standup.wav` + + +### `images/` (17 files) + +Scene photos for one day, referenced by the `file` field of the photo records in `memories.json` +(Lesson 4 and Lesson 5). Attribution in `images/CREDITS.json` (not embedded). + +Files: `bakery.jpg`, `bicycle.jpg`, `book.jpg`, `coffee.jpg`, `dog.jpg`, `gym.jpg`, `kitchen.jpg`, `laptop.jpg`, `meeting.jpg`, `park.jpg`, `pizza.jpg`, `plant.jpg`, `ramen.jpg`, `restaurant.jpg`, `sneakers.jpg`, `street.jpg`, `train.jpg` + + +### `objects/` (16 files) + +`objects/README.md` (verbatim): + +```markdown +# Object photos + +Six subjects, two or three photos of each: backpack, gaillardia, hardhat, lithops, rubberduck, vase. Each set is several views of the same physical object, which is what Lesson 5 needs to teach recognition from a handful of examples and then test on a view it never saw. + +Lesson 5 falls back to the rubber duck when you upload nothing. To teach one of the other five instead, use the two upload buttons in the lesson's first cell and pick the files from this folder: two or more views on the left to teach with, and one more view of the same object on the right, held back for the test. The vase set, for example, teaches from `vase_1.jpg` and `vase_2.jpg` and tests on `vase_3.jpg`. + +Photo credits, with author and license for each file, are in `CREDITS.json`. +``` +Files: `backpack_1.jpg`, `backpack_2.jpg`, `gaillardia_1.jpg`, `gaillardia_2.jpg`, `gaillardia_3.jpg`, `hardhat_1.jpg`, `hardhat_2.jpg`, `hardhat_3.jpg`, `lithops_1.jpg`, `lithops_2.jpg`, `rubberduck_1.jpg`, `rubberduck_2.jpg`, `rubberduck_3.jpg`, `vase_1.jpg`, `vase_2.jpg`, `vase_3.jpg` + +Attribution in `objects/CREDITS.json` (not embedded). + + +### `bank/` (165 files) + +The image-search bank for Lesson 3 (all 165 stored as photo memories) and the source of +Lesson 5's three seed objects (`bicycle.jpg`, `chess_set.jpg`, `camera.jpg`). One object or +scene per photo, named for its subject. Attribution in `bank/CREDITS.json` (not embedded). + +Files: `airport_terminal.jpg`, `ambulance.jpg`, `autumn_leaves.jpg`, `avocado.jpg`, `backpacking_gear.jpg`, `bagel.jpg`, `banana.jpg`, `baseball_glove.jpg`, `basketball.jpg`, `beach_sunset.jpg`, `bed.jpg`, `beer_glass.jpg`, `bell_pepper.jpg`, `bicycle.jpg`, `bouquet_flowers.jpg`, `bridge_over_river.jpg`, `bus_stop.jpg`, `butterfly.jpg`, `cactus_plant.jpg`, `calculator.jpg`, `camera.jpg`, `cappuccino.jpg`, `carrot.jpg`, `cat.jpg`, `chess_set.jpg`, `chocolate_cake.jpg`, `city_bus.jpg`, `coffee_mug.jpg`, `coins_money.jpg`, `cooking_pot.jpg`, `corn_cob.jpg`, `cow.jpg`, `croissant.jpg`, `cupcake.jpg`, `curry_rice.jpg`, `delivery_van.jpg`, `denim_jacket.jpg`, `desert_dunes.jpg`, `desk_fan.jpg`, `dice.jpg`, `dog_puppy.jpg`, `dolphin.jpg`, `drum_kit.jpg`, `duck_pond.jpg`, `dumbbell_weights.jpg`, `elephant.jpg`, `espresso.jpg`, `fire_extinguisher.jpg`, `fishing_rod.jpg`, `forest_trail.jpg`, `french_fries.jpg`, `fried_chicken.jpg`, `frog.jpg`, `fruit_bowl.jpg`, `full_moon.jpg`, `game_controller.jpg`, `garden_shovel.jpg`, `gas_station.jpg`, `giraffe.jpg`, `goldfish.jpg`, `grocery_store_aisle.jpg`, `gym_interior.jpg`, `hair_dryer.jpg`, `hamburger.jpg`, `hamster.jpg`, `headphones.jpg`, `helicopter.jpg`, `high_heels.jpg`, `hiking_boots.jpg`, `honeybee.jpg`, `horse.jpg`, `ice_cream_cone.jpg`, `jack_o_lantern.jpg`, `keyboard.jpg`, `kitchen_knife.jpg`, `ladder.jpg`, `ladybug.jpg`, `lake.jpg`, `leather_boots.jpg`, `lemon.jpg`, `library_shelves.jpg`, `light_bulb.jpg`, `lightning_storm.jpg`, `lion.jpg`, `map.jpg`, `microphone.jpg`, `microwave_oven.jpg`, `morning_fog.jpg`, `mushroom.jpg`, `necklace.jpg`, `necktie.jpg`, `newspaper.jpg`, `notebook_journal.jpg`, `ocean_waves.jpg`, `office_desk.jpg`, `onion.jpg`, `palm_tree.jpg`, `park_bench.jpg`, `park_fountain.jpg`, `parrot_bird.jpg`, `pencil.jpg`, `penguin.jpg`, `piano.jpg`, `picture_frame.jpg`, `piggy_bank.jpg`, `pizza_slice.jpg`, `rabbit.jpg`, `rainbow_sky.jpg`, `rainy_window.jpg`, `red_wine_glass.jpg`, `refrigerator.jpg`, `river_stream.jpg`, `roller_skates.jpg`, `rose_flower.jpg`, `sailboat.jpg`, `sandwich.jpg`, `scissors.jpg`, `scooter.jpg`, `shopping_cart.jpg`, `skateboard.jpg`, `skyline_at_night.jpg`, `smoothie.jpg`, `snowflakes.jpg`, `snowy_street.jpg`, `soccer_ball.jpg`, `sofa_couch.jpg`, `spider_web.jpg`, `squirrel.jpg`, `stapler.jpg`, `starry_night_sky.jpg`, `statue.jpg`, `steak_dinner.jpg`, `strawberries.jpg`, `subway_platform.jpg`, `sunflower.jpg`, `sunglasses.jpg`, `sushi_platter.jpg`, `swimming_pool.jpg`, `tacos.jpg`, `tea_cup.jpg`, `telescope.jpg`, `tennis_racket.jpg`, `tent_camping.jpg`, `thermometer.jpg`, `tomato.jpg`, `toothbrush.jpg`, `tulip.jpg`, `turtle.jpg`, `vineyard.jpg`, `vinyl_record.jpg`, `waffles.jpg`, `wall_clock.jpg`, `wallet.jpg`, `washing_machine.jpg`, `water_bottle.jpg`, `water_glass.jpg`, `waterfall.jpg`, `watering_can.jpg`, `watermelon.jpg`, `wheat_field.jpg`, `wine_bottle.jpg`, `wool_scarf.jpg`, `wristband.jpg`, `wristwatch.jpg`, `yellow_taxi.jpg` + + +--- + +## Appendix: items the course README lists that are not in this copy of the repo + +- **`Appendix/` (Cloud Sync: One Memory, Many Devices).** The README's lesson table, "Run the + notebooks locally" section and repository layout all name an appendix notebook that syncs one + device's memory to a Qdrant cluster and to a second device. There is no `Appendix/` directory in + `SC-Qdrant-C3-main/`. Its helper functions are present in every copy of helper.py + (`cloud_client`, `cloud_points`, `push_note`, `fetch_snapshot`, `file_size`), and + `requirements.txt` pins `qdrant-client` for it. +- **Saved notebook outputs.** The README and requirements.txt refer to scores "in the saved + outputs"; all three notebooks here have empty outputs. +- **The recognition threshold name.** As noted under Lesson 5, `RECOGNIZE_THRESHOLD` is used in + cell 12 but defined nowhere in the repo. diff --git a/courses/building-on-device-ai-memory-with-qdrant-edge/materials/slides/sc-Qdrant-C3-slide-descriptions.md b/courses/building-on-device-ai-memory-with-qdrant-edge/materials/slides/sc-Qdrant-C3-slide-descriptions.md new file mode 100644 index 0000000..25a255d --- /dev/null +++ b/courses/building-on-device-ai-memory-with-qdrant-edge/materials/slides/sc-Qdrant-C3-slide-descriptions.md @@ -0,0 +1,440 @@ +# Slide descriptions: Building On-Device AI Memory with Qdrant Edge +Source decks: slides/Qdrant_Slides – Wide (16_9).pptx (14 slides), slides/Qdrant_Slides – Tall (8_9).pptx (22 slides) +Instructor: Dylan Couzon + +Rich text descriptions of the course slides, organized by lesson in the same order and with the +same lesson titles as the transcripts in `../transcripts/`. Each entry gives the deck and original +slide number, the time range the slide is on screen in the lesson video, the transcript passage +spoken over it, every word visible on the slide, a description of the visual, and what it teaches. + +## How to read this file + +- **Lesson assignment comes from the videos, not the decks.** The decks use an earlier lesson + plan (the Tall deck's title cards say "Lesson 2: Store and Recall", "Lesson 3: Finding the Right + Memory", and so on, and the Course Map slide lists six lessons). Every slide below was matched to + the lesson video it actually appears in, by sampling frames every 2 seconds and then every 0.5 + seconds around each slide. +- **Only eight deck slides appear in the five videos.** Lesson 1 uses five "Answered on Device" + slides; Lesson 3 uses three Tall-deck slides beside the notebook. Lessons 2, 4, and 5 show no + deck slides at all (only the instructor, the phone app, a screen recording of the repository, or + the notebook). Those lessons are listed with a note so the file still has one section per lesson. +- **Slide numbering** is `Lesson.Order` in on-screen order. The deck and original slide number + follow in parentheses so each entry can be traced back to the .pptx. +- **Timestamps** are `m:ss` in the lesson video and are accurate to about half a second. Several + slides are animated builds; the description notes the order in which elements appear. +- **Text on slide** is transcribed verbatim from the .pptx text runs and the diagram images, + including one typo that is on screen in the video ("Returning closes vectors"). +- **Appendix A** describes the fifteen diagram slides that are in the decks but never appear in a + video. They are grouped by the lesson they were drafted for. This goes slightly beyond the + original request to omit unused slides; it is included because these are substantive teaching + diagrams rather than production slides, and the appendix can be deleted without affecting the + lesson sections. **Appendix B** lists the remaining non-content slides (teaser, placeholders, + acknowledgments, style guide, title cards) in one line each, so all 36 slides are accounted for. +- Course-wide visual conventions: every content slide has the Qdrant logo top left and the + DeepLearning.AI logo top right, a centered title in the deck's coral-red display face, and a + coral-red wave along the bottom edge. Diagrams share a color code: light blue for input or + capture, orange for embedding models, lavender/purple cylinders for the memory store, red for + queries and recall, and teal-green for results and matches. + +--- + +## Lesson 1: Why Devices Need Memory +Video: videos/sc-Qdrant-C3-L1-v2.mp4 (4:36) · Transcript: transcripts/sc-Qdrant-C3-L1-transcript.md + +Five full-screen slides from the Wide deck, all titled "Answered on Device", run as one +continuous animated sequence from 0:45 to 1:50. Before and after this block the video shows the +instructor at a desk with the robot and two cat plushies, phone-app demos, and a screen recording +of the Qdrant Edge web page. Two other on-screen elements are video overlays, not deck slides: the +opening title card "Building AI Assistants with On-Device Memory / Overview of On-Device AI +Memory" (0:00) and the word list "Create / Store / Memory / Recall" (4:24 to 4:33). + +### Slide 1.1 — Answered on Device: from a note to a vector (Wide deck, slide 10) +**On screen:** 0:45–1:06 +**Transcript cue:** "I can take a note, 'I had ramen for lunch.' An embedding model turns the text into a vector. A vector is a list of numbers that represents the meaning of text. A vector search engine then creates a navigable graph to allow search through those vectors to happen extremely fast, sometimes in less than a millisecond." +**Text on slide:** Answered on Device · I had ramen for lunch · Embedding · MODEL · embed on device · 0.12, 0.85 · 0.34, 0.35 · 0.35, 0.72 · 0.81, 0.58 · 0.35, 0.19 · 0.09, 0.45 · 0.32, 0.27 · memory · top matches · stays on device · works offline +**Visual description:** A left-to-right pipeline built up one element at a time. It starts with a +rounded lavender card reading "I had ramen for lunch" with a small writing-hand-and-notepad emoji +above the text (0:46). A dashed arrow leads to a white square labelled "Embedding" over a gear +icon and the word "MODEL", with a small lavender pill underneath reading "embed on device" (0:48). +A second dashed arrow leads to a column of seven number pairs (0.12, 0.85 through 0.32, 0.27) +enclosed in large peach-colored curly braces, representing the vector (0:50). A dashed rounded +rectangle then encloses the note, the model, and the vector. A final dashed arrow points to a +lavender cylinder. The cylinder first appears labelled "memory" (0:51), then swaps to a cylinder +containing a graph of about ten white nodes joined by grey lines, captioned "top matches" in a +hand-lettered face (0:53), and later returns to the plain "memory" label (1:00). A footer line +appears at 0:55 with a green padlock emoji and "stays on device", then a paper-plane emoji and +"works offline". +**What it teaches:** The simplest memory is a text note. The embedding model runs on the device +and turns the note into a list of numbers whose values encode meaning. The vector search engine +stores those vectors and organizes them into a navigable graph so lookup is fast. The footer +states the lesson's two promises: nothing leaves the device, and it works with no network. + +### Slide 1.2 — Answered on Device: asking a question (Wide deck, slide 11) +**On screen:** 1:06–1:21 (continues the animation of Slide 1.1 under the same title) +**Transcript cue:** "A question goes through the same embedding model. For example, 'What did I eat for lunch?' Then that vector search engine returns the approximate nearest neighbor to this query. The closer the vectors, the higher the similarity score. Returning the closest vectors is called retrieval. This is the core pattern you will use in this course." +**Text on slide:** Answered on Device · What did I eat for lunch? · Embedding · MODEL · embed on device · (the same seven number pairs as Slide 1.1) · memory · top matches · high similarity score · high similarity score · Qdrant · Returning closes vectors → **Retrieval** · stays on device · works offline +**Visual description:** Same layout as Slide 1.1, but the note card is replaced by a lavender card +reading "What did I eat for lunch?" (1:05). The embedding model and the vector then disappear so +that the dashed box holds only the question, with a dashed arrow straight to the "top matches" +cylinder (1:06). At 1:13 the cylinder changes to a darker purple cylinder labelled "Qdrant" at its +base, containing a graph of grey nodes and edges; two nodes glow bright green and each is +annotated in hand lettering "high similarity score", with green arrows tracing the search path +through the graph toward them. At 1:15 a caption appears at the top right: "Returning closes +vectors → Retrieval", with "Retrieval" in bold. ("closes" is a typo for "closest" and is on screen +as written.) The padlock and paper-plane footer stays visible throughout. +**What it teaches:** Retrieval is the core pattern of the course. A question is embedded with the +same model as the notes, the engine walks its graph to the nearest stored vectors, and the +closeness of vectors is the similarity score. The green glowing nodes are the approximate nearest +neighbors returned for the query. + +### Slide 1.3 — Answered on Device: CLIP (Wide deck, slide 12) +**On screen:** 1:21–1:39 +**Transcript cue:** "That same idea also works with photos. For this, we use the model called CLIP. CLIP has one encoder for images, another one for text. It creates a similar embedding whether you show it an image of a cat or describe a cat. That means we can search images using words." +**Text on slide:** Answered on Device · cat · **CLIP** → Contrastive Language–Image Pre-training · CLIP Model · image encoder · text encoder · [1,5,6] · [1,8,4] · [1,2,5] · [1,6,3] · [1,3,0] · [0,8,0] · [0,6,4] · [1,5,18] +**Visual description:** On the left a lavender rounded square holds a simple line drawing of a +sitting cat, captioned "cat" beneath it; small pastel stars and dots decorate the corners of the +image, and a faint "stays on device / works offline" footer is part of the image file but is not +legible in the video. The line "CLIP → Contrastive Language–Image Pre-training" appears above the +diagram at 1:24, with "CLIP" in bold. At 1:25 a dashed arrow leads from the cat to a large salmon +box titled "CLIP Model". Inside it, a lighter inner panel holds two darker salmon boxes, "image +encoder" and "text encoder", joined by a red arrow with two small sine waves drawn above and +below it. At 1:29 two columns of three-number vectors appear to the right of the model: a blue +column ([1,5,6], [1,8,4], [1,2,5], [1,6,3]) and a green column ([1,3,0], [0,8,0], [0,6,4], +[1,5,18]). +**What it teaches:** CLIP has two encoders that map images and text into the same vector space, +so the picture of a cat and the word "cat" land near each other. The two columns of vectors +stand in for the image-side and text-side embeddings that can be compared directly. This is what +makes searching photos with words possible. + +### Slide 1.4 — Answered on Device: recognizing what it sees (Wide deck, slide 13) +**On screen:** 1:39–1:48 +**Transcript cue:** "And the same idea lets this robot recognize what it sees by comparing with what it knows." +**Text on slide:** Answered on Device · Similarity Score · cat · CLIP +**Visual description:** A large white thought bubble fills most of the slide, with three small +trailing circles leading to a cartoon robot at the bottom right. The robot is a white rounded +capsule with a two-lens camera "face", speaker slots on its side, and a black baseball cap with +the Qdrant hexagon logo. Inside the bubble, from left to right: the label "Similarity Score" above +a photograph of a real grey-and-white tabby cat sitting on a wooden floor; an arrow into a salmon +box labelled "CLIP" whose inner lavender panel reads "cat" in a typewriter face; an arrow into a +pale green cylinder containing a small graph of green nodes; and a thin loop arrow from the top +of the cylinder back to the CLIP box. A faint dashed frame with pastel confetti (the deck's +underlay image) sits behind the bubble but is barely visible. +**What it teaches:** Recognition on the robot is the same retrieval loop. A camera frame goes +through CLIP, its vector is compared against what is already in memory, and the similarity score +tells the robot whether it is looking at something it knows. The robot is "thinking" the diagram. + +### Slide 1.5 — Answered on Device: the match threshold (Wide deck, slide 14) +**On screen:** 1:48–1:50 +**Transcript cue:** "If the similarity score is above a certain threshold, we have a match. And to do this, we need no LLM, no retraining, or no custom vision pipeline." +**Text on slide:** Answered on Device · similarity score > threshold · 0 · 5 · MATCH! · CLIP +**Visual description:** The same thought bubble and capped robot as Slide 1.4. The cat photo is +replaced by a small panel titled "similarity score > threshold" containing a horizontal slider +from 0 to 5 with a grey handle near 3 and a green check-mark badge at the left end. The CLIP box +now shows a green "MATCH!" badge in its inner panel instead of the word "cat". The green memory +cylinder and the loop arrow back to CLIP are unchanged. +**What it teaches:** A match is a decision rule, not a model: compare the similarity score against +a threshold. Everything the robot does in the demo comes from vectors and a threshold, with no +large language model, no retraining, and no custom vision pipeline. + +--- + +## Lesson 2: Building the Device +Video: videos/sc-Qdrant-C3-L2-v2.mp4 (2:32) · Transcript: transcripts/sc-Qdrant-C3-L2-transcript.md + +No slide from either deck appears in this video. The lesson is shot at the desk with the robot, +the two cat plushies, and the disassembled hardware, and it switches to a screen recording of the +open-source repository and the phone interface. The only graphic elements are video overlays: the +opening title card "Building AI Assistants with On-Device Memory / Inside the AI Assistant +Device" (0:00) and a word list that builds "Camera / Computer / Storage / Interface" at 0:24 to +0:27 while the instructor names the four parts of the robot. + +--- + +## Lesson 3: Store, Find, and Forget Memories +Video: videos/sc-Qdrant-C3-L3-v2.mp4 (6:38) · Transcript: transcripts/sc-Qdrant-C3-L3-transcript.md + +This lesson is a notebook screen recording in the tall half-screen format. Three Tall-deck slides +cut in, each filling the frame in place of the notebook for a few seconds. The opening title card +"Building AI Assistants with On-Device Memory / Building the Memory" (0:00) and the "Find the +latency_curve() method in helper.py for this lesson" callout (5:36 to 5:43) are video overlays, +not deck slides. + +### Slide 3.1 — Anatomy of a point (Tall deck, slide 4) +**On screen:** 2:33–2:44 +**Transcript cue:** "So we use what we call points to store the memory with all the vectors, but also any metadata that is associated with that memory. So we have the ID, we have the source type, which is where the memory came from. We have a category label, the location, the exact timestamp, the text version of the notes and the price when it is applicable." +**Text on slide:** Anatomy of a point · Point · id: 3 · named vector · text · payload · note + fields +**Visual description:** Under the title, a bold "Point" label sits above a tall lavender rounded +rectangle with a purple outline that fills the width of the slide. The rectangle is divided by two +dashed horizontal lines into three rows. Top row: a grey hash icon and "id: 3". Middle row: an +orange waveform icon, "named vector" with a small dot and the qualifier "text", and beneath it a +row of nine small orange-outlined boxes with partial orange fill (a strip of vector cells) ending +in an ellipsis. Bottom row: a grey document icon, "payload" with a dot and the qualifier "note + +fields". The slide appears right after the notebook cell that upserts twenty points and prints +Point 0's fields (id, source_type, category, location, timestamp, note, price). +**What it teaches:** A point is the unit of storage in Qdrant Edge: an id, one or more named +vectors (here the 768-dimensional text vector), and a payload holding the original note plus +metadata fields. The instructor reads the payload fields off the notebook output while the slide +is up. + +### Slide 3.2 — Recognizing with CLIP (Tall deck, slide 9, untitled) +**On screen:** 4:16–4:21 +**Transcript cue:** "Our robot doesn't only process text. It also process images and live video. Now let's add images to our memories." +**Text on slide:** cat · CLIP +**Visual description:** The same thought-bubble diagram as Lesson 1's Slide 1.4, reused in the tall +format without a title: a white thought bubble with a photograph of a tabby cat, an arrow into a +salmon "CLIP" box whose inner panel reads "cat", an arrow into a pale green cylinder with a small +node graph, and a loop arrow from the cylinder back to CLIP. The "Similarity Score" label from the +Lesson 1 version is not present. The capped white robot sits at the bottom right with three small +circles connecting it to the bubble. Above the bubble is empty space with only the Qdrant and +DeepLearning.AI logos. +**What it teaches:** A bridge from text to images. The robot's camera frames go through CLIP into +the same kind of memory store, which is what the next notebook cells build by storing 165 photos. + +### Slide 3.3 — Cross-Model recall (Tall deck, slide 10) +**On screen:** 4:43–4:54 +**Transcript cue:** "Since we're using CLIP, an embedding model that matches text and images, we can ask it a red bicycle, embed that query, and then search. And here we have a red bicycle retrieved." +**Text on slide:** Cross-Model recall · "a red bicycle" · CLIP text tower · text query → image space · text · image · bicycle.jpg +**Visual description:** A vertical flow. At the top, a light-blue rounded box with a speech-bubble +icon holds the query "a red bicycle". A red arrow leads down to an orange rounded box, "CLIP text +tower". From there a red arrow curves down and to the left, annotated in small red text "text +query → image space", and enters a lavender cylinder. Inside the cylinder are two stacked pills: +"text" with a document icon and five vector cells, and "image" with a picture icon and five vector +cells. The "image" pill is outlined in red to show the arrow lands there, skipping the text +vectors. A final red arrow exits the cylinder to the bottom right into a teal-green rounded box +with a picture icon, "bicycle.jpg", and a green check mark. +**What it teaches:** Because CLIP's text encoder and image encoder share one space, a text query +can be embedded with the text tower and searched against the image vectors in the shard. The +shard holds two named vectors per point, and the query chooses which one to search ("using +image" in the notebook). The result is a photo found from words. + +--- + +## Lesson 4: Your On-Device Assistant +Video: videos/sc-Qdrant-C3-L4-v2.mp4 (4:58) · Transcript: transcripts/sc-Qdrant-C3-L4-transcript.md + +No slide from either deck appears in this video. After the opening title card "Building AI +Assistants with On-Device Memory / Building the Voice Assistant" (0:00) and a short on-camera +introduction, the whole lesson is a notebook screen recording (day summary, photo grid, notes +table, voice-note transcription, the recall function, the memory inbox results, and adding a +new memory). The Tall deck's "What comes back" slide was drafted for this lesson but is not used; +see Appendix A. + +--- + +## Lesson 5: Teaching Your Assistant to See +Video: videos/sc-Qdrant-C3-L5-v2.mp4 (5:36) · Transcript: transcripts/sc-Qdrant-C3-L5-transcript.md + +No slide from either deck appears in this video. After the opening title card "Building AI +Assistants with On-Device Memory / Learning Through Memory" (0:00) and an on-camera +introduction, the lesson is a notebook screen recording (object shard, teach and recognize +functions, the rubber-duck example, threshold calibration plot, the assistant's memory, freshness +ranking, and the final question table). The Tall deck's "Teach, Store, and Recognize" and "One +Point, Two Doors" slides were drafted for this lesson but are not used; see Appendix A. + +--- + +## Appendix A: Diagram slides in the decks that do not appear in any video + +Grouped by the lesson each slide was drafted for, using the Tall deck's own title cards and the +topic of the transcript. None of these is on screen in the final videos, so there are no +timestamps. Where a diagram matches part of a transcript, the passage is quoted as context. + +### Drafted for Lesson 1 + +#### A.1 — Frozen and Growing (Wide deck, slides 2 and 3; slide 3 is an identical duplicate) +**Text on slide:** Frozen and Growing · device · model (frozen) · context · your app · write · recall · memory (grows) +**Visual description:** A dashed rounded rectangle labelled "device" in the top-left corner +encloses the whole diagram. On the left, an orange rounded box holds a small neural-network +drawing (three columns of nodes fully connected by orange lines) with a padlock icon in the top +right corner; it is captioned "model (frozen)". In the middle, a light-blue rounded box reads +"your app", with a grey arrow pointing left to the model labelled "context". On the right, a +lavender cylinder captioned "memory (grows)" contains three rows, each with an icon (picture, +waveform, document) followed by seven orange vector cells; three purple plus signs float above +it. Two red curved arrows connect the app and the memory: "write" going in and "recall" coming +out. +**What it teaches:** The course's central framing. The model's weights never change; what grows +is the memory. The app writes new memories and recalls them to give the frozen model context. +Related transcript passage (Lesson 1): "We created, stored, and recalled memories without using +any LLM. What changed was only some vectors." + +#### A.2 — Answered on Device, "where did I put it?" (Wide deck, slide 6) +**Text on slide:** Answered on Device · 🗣 · device · "where did I put it?" · embed on device · top matches · memory · stays on device · works offline +**Visual description:** An earlier, single-image version of the Lesson 1 sequence. A dashed +"device" rectangle encloses everything. Top left, a light-blue rounded box holds the spoken +question "where did I put it?" (the deck also places a speaking-head emoji, 🗣, as a separate +text element). A red arrow leads to a small orange box showing three text lines turning into +three vector cells, captioned "embed on device", then a red arrow into a lavender cylinder +captioned "memory". The cylinder holds three rows of vector cells with picture, waveform, and +document icons; the middle (waveform) row is highlighted with a teal outline and a teal check +mark sits to its right. A teal arrow labelled "top matches" curves from the cylinder back down to +a teal result box at the bottom left containing a picture icon, a document icon, and a check +mark. Bottom right, a padlock with "stays on device" and a paper plane with "works offline". +**What it teaches:** The whole question-to-answer loop happens inside the device boundary: embed +the question locally, find the top matches in local memory, return them. The "device" border and +the footer make the privacy and offline points visually. + +#### A.3 — In-Process and On-Disk (Wide deck, slide 7) +**Text on slide:** In-Process and On-Disk · your app (one process) · upsert() · query() · Qdrant Edge · edge_config.json · segments/ · wal/ · on disk · central Qdrant server · optional sync +**Visual description:** On the left, a dashed rounded rectangle labelled "your app (one process)" +contains a light-blue box with a code-brackets icon and, to its right, a lavender box labelled +"Qdrant Edge". Two horizontal lines join them, labelled "upsert()" above and "query()" below in a +monospace face. A line from Qdrant Edge exits the dashed box to a lavender folder shape on the +right listing three monospace entries: "edge_config.json", "segments/", "wal/"; it is captioned +"on disk" with a small drive icon. Above the folder, a grey rounded box with a database icon +reads "central Qdrant server", connected to the folder by a dashed double-headed arrow labelled +"optional sync". +**What it teaches:** Qdrant Edge is a library inside the application's own process, not a server. +Memory is a directory on disk (config, segments, write-ahead log), and syncing to a central +Qdrant server is optional. Related transcript passage (Lesson 2): "There is no separate vector +database server, and the memory is stored in a folder on the device." + +#### A.4 — Course Map (Wide deck, slide 8) +**Text on slide:** Course Map · capture · embed · store · recall · L2 store + recall · L3 photos · L4 a whole day · L5 teach it to see · L6 on a robot +**Visual description:** Four rounded boxes in a row, joined by grey arrows: light-blue "capture" +(picture, waveform, and document icons), orange "embed" (text lines turning into vector cells), +lavender "store" (a small cylinder), and red "recall" (a magnifying glass). A long red arrow +arcs over the top from recall back to capture, closing the loop. Below, horizontal bars show +which stages each lesson covers: "L2 store + recall" spans store and recall; "L3 photos" sits +under embed; "L4 a whole day" sits under capture; and two full-width grey bars, "L5 teach it to +see" (with a phone icon) and "L6 on a robot" (with a robot icon), span all four stages. +**What it teaches:** The capture, embed, store, recall loop is the spine of the course, and each +lesson adds one stage or one modality. Note that the lesson numbers are from an earlier +six-lesson plan and are off by one from the final videos (the final course has five lessons, with +the robot covered in Lessons 1 and 2). + +### Drafted for Lesson 3 (Tall deck "Lesson 2: Store and Recall" and "Lesson 3: Finding the Right Memory") + +#### A.5 — The memory loop, recall highlighted (Tall deck, slide 3, untitled) +**Text on slide:** capture · embed · store · recall · this lesson +**Visual description:** The four loop stages stacked vertically in the tall format: light-blue +"capture" with a camera icon, orange "embed" with a waveform icon, a lavender "store" cylinder +with a strip of vector cells, and red "recall" with a magnifying glass, joined top to bottom by +grey arrows, with a red arrow curving up the left side from recall back to capture. A +hand-lettered label "this lesson" on the right points with a black arrow at "recall". The three +non-highlighted stages are drawn with lighter text. +**What it teaches:** A section marker: this part of the lesson is about recall (querying). + +#### A.6 — The Vector space (Tall deck, slide 5) +**Text on slide:** The Vector space · stored points · query · nearest +**Visual description:** A two-axis plot drawn by hand, with grey axes and arrowheads. About a +dozen lavender dots are scattered in two loose clusters, labelled "stored points". A red +asterisk at the upper right is labelled "query" in red. A red curved arrow runs from the query to +one dot in the lower-right cluster, which is circled in teal and labelled "nearest" in teal. +**What it teaches:** Similarity search in picture form: every memory is a point in a vector +space, the question is another point, and recall returns the stored point closest to it. + +#### A.7 — The memory loop, embed and recall highlighted (Tall deck, slide 7, untitled) +**Text on slide:** capture · embed · store · recall · this lesson +**Visual description:** Same vertical loop as A.5, but the "this lesson" label has two black +arrows, one to "embed" and one to "recall". +**What it teaches:** A section marker for the part of the lesson that adds a second embedding +model (images) and searches with it. + +#### A.8 — Two Encoders, One Shard (Tall deck, slide 8) +**Text on slide:** Two Encoders, One Shard · Nomic-Embed-Text · 768-d · EdgeShard · text · image · 512-d · CLIP ViT-B/32 +**Visual description:** At the top, an orange rounded box with a document icon reads +"Nomic-Embed-Text"; an orange arrow labelled "768-d" points down into a lavender cylinder labelled +"EdgeShard" to its upper left. Inside the cylinder are two stacked pills: "text" with a document +icon and six vector cells, and "image" with a picture icon and five vector cells. From the +bottom, an orange rounded box with a picture icon reads "CLIP ViT-B/32", and an orange arrow +labelled "512-d" points up into the "image" pill. +**What it teaches:** One shard holds two named vectors per point, each produced by a different +model: the 768-dimensional Nomic text embedding and the 512-dimensional CLIP image embedding. +This matches the shard configuration at the start of the Lesson 3 notebook ("text" size 768, +"image" size 512, cosine distance). + +#### A.9 — Filter inside Query (Tall deck, slide 11) +**Text on slide:** Filter inside Query · query · find: "something to eat" · where: category = food · price < 15 · food · $9 · food · $12 · food · $22 · gear · $45 · taco truck on 5th · 0.92 · pho spot downtown · 0.87 +**Visual description:** At the top, a pale-pink rounded box tagged "query" in small red text holds +two lines separated by a rule: "find:" in red followed by "something to eat" in quotes, and +"where:" in teal followed by "category = food · price < 15". A red arrow leads down into a +lavender cylinder containing four rows. Each row is a strip of five orange vector cells and a +teal tag: "food · $9", "food · $12", "food · $22", "gear · $45". The first two rows are drawn at +full strength with a short red mark on their left; the last two are faded to show they are +excluded. A red arrow leads down to two pink result bars: "taco truck on 5th 0.92" and "pho spot +downtown 0.87". +**What it teaches:** A filter is part of the query, not a post-processing step. The semantic +search ("something to eat") runs only over points whose payload passes the conditions +(category is food, price under 15), and the results still carry similarity scores. This is the +category-and-price filter built in the Lesson 3 notebook. + +### Drafted for Lesson 4 (Tall deck "Lesson 4: Your On-Device Assistant") + +#### A.10 — The memory loop, capture and recall highlighted (Tall deck, slide 13, untitled) +**Text on slide:** capture · embed · store · recall · this lesson +**Visual description:** Same vertical loop as A.5, with "this lesson" pointing at both "capture" +and "recall". +**What it teaches:** A section marker: Lesson 4 captures a whole day (photos, notes, voice) and +recalls from it. + +#### A.11 — What comes back (Tall deck, slide 14) +**Text on slide:** What comes back · "the ramen downtown" · Nomic · CLIP · memory inbox · ramen_shop.jpg · 0.71 · "$14 and worth it" · 0.68 · picked up groceries · 0.42 · weak match +**Visual description:** At the top, a light-blue rounded box with a speech-bubble icon holds the +query "the ramen downtown". Two grey arrows fan out to two orange boxes with waveform icons, +"Nomic" on the left and "CLIP" on the right. Red arrows from both converge into a large +teal-outlined panel titled "memory inbox". Inside are three result rows: a picture icon with +"ramen_shop.jpg" and score 0.71; a microphone icon with the quoted voice note "$14 and worth it" +and score 0.68; and a faded document row "picked up groceries" with score 0.42, annotated +outside the panel with "weak match". +**What it teaches:** One question is embedded twice and searched against both the text and image +vectors, and the results from all modalities (photo, voice note, text note) are merged into one +inbox ranked by score, with low scores flagged as weak. This is the memory inbox display used in +the Lesson 4 notebook. + +### Drafted for Lesson 5 (Tall deck "Lesson 5: Teaching It to See") + +#### A.12 — The memory loop, all of it (Tall deck, slide 16, untitled) +**Text on slide:** capture · embed · store · recall · all of it +**Visual description:** Same vertical loop as A.5, but instead of a pointer, a tall black curly +brace spans all four stages on the right, labelled "all of it". +**What it teaches:** A section marker: the final lesson exercises the whole loop end to end. + +#### A.13 — Teach, Store, and Recognize (Tall deck, slide 17) +**Text on slide:** Teach, Store, and Recognize · photos (teach) · CLIP · object shard · label · nearest match > threshold · new photo (recognize) +**Visual description:** At the top, two small blue picture icons labelled "photos (teach)" send an +orange arrow into an orange "CLIP" box. From CLIP, an orange arrow and a red arrow both point down +into a lavender cylinder labelled "object shard", which contains a strip of six vector cells and a +small lavender pill reading "label". At the bottom left, a single blue picture icon labelled "new +photo (recognize)" sends a long red arrow up the left side into CLIP. From the cylinder, a red +arrow exits to the bottom right into a teal box with a check mark reading "nearest match > +threshold". +**What it teaches:** Teaching and recognizing are the same pipeline through CLIP. Teaching stores +a few photo vectors with a label in a dedicated object shard; recognizing embeds a new photo, +finds the nearest stored view, and accepts it only if the similarity clears a threshold. This is +the teach, recognize, and threshold-calibration sequence in the Lesson 5 notebook. + +#### A.14 — One Point, Two Doors (Tall deck, slide 18) +**Text on slide:** One Point, Two Doors · Point · named vector · image · named vector · text · payload · note · by sight · by words +**Visual description:** The same lavender "Point" card as Slide 3.1, now with three rows: "named +vector · image" (picture icon, six orange vector cells), "named vector · text" (document icon, +six vector cells), and "payload · note" (document icon). Outside the card on the left, a blue +picture icon with a red arrow into the image row is labelled "by sight"; on the right, a blue +speech-bubble icon with a red arrow into the text row is labelled "by words". +**What it teaches:** One memory can be reached two ways. A taught object is stored as a single +point carrying both a CLIP image vector and a text vector for its note, so the assistant can +find it from a new photo or from a typed or spoken question. This is the final "assistant +memory" built in Lesson 5, where the rubber duck is stored with both a photo and a note. + +--- + +## Appendix B: Non-content slides (omitted) + +| Deck | Slide | Title / content | Reason omitted | +| --- | --- | --- | --- | +| Wide | 1 | "Building on-device AI memory / Why Devices Need Memory" with Qdrant and DeepLearning.AI logos | Marketing teaser slide (speaker notes say so); not in any video | +| Wide | 4 | "Demo 1" | Empty placeholder | +| Wide | 5 | "Demo 1" | Empty placeholder | +| Wide | 9 | "Acknowledgments": names and photos of six DeepLearning.AI staff, with a template instruction still on the slide | Credits template; not in any video | +| Tall | 1 | "Building on-device AI memory / Dylan Couzon" | Title card; not in any video | +| Tall | 2 | "Building on-device AI memory / Lesson 2: Store and Recall" | Title card from the earlier lesson plan | +| Tall | 6 | "Building on-device AI memory / Lesson 3: Finding the Right Memory" | Title card from the earlier lesson plan | +| Tall | 12 | "Building on-device AI memory / Lesson 4: Your On-Device Assistant" | Title card; not in any video | +| Tall | 15 | "Building on-device AI memory / Lesson 5: Teaching It to See" | Title card; not in any video | +| Tall | 19 | "Style Guide": typography (Poppins, Verdana for code) and color swatches | Production reference | +| Tall | 20 | "Building AI Assistants with On-Device Memory / Building the Memory" | Older-style title card; the videos use a similar card as a video overlay | +| Tall | 21 | "Building AI Assistants with On-Device Memory / Building the Voice Assistant" | Older-style title card; same note | +| Tall | 22 | "Building AI Assistants with On-Device Memory / Learning Through Memory" | Older-style title card; same note | + +Totals: 8 slides shown in videos + 15 diagram slides in Appendix A + 13 non-content slides in +Appendix B = 36 slides, the full contents of both decks. (Wide slides 2 and 3 are counted +separately but described once, in A.1.) diff --git a/courses/building-on-device-ai-memory-with-qdrant-edge/materials/transcripts/.gitkeep b/courses/building-on-device-ai-memory-with-qdrant-edge/materials/transcripts/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/courses/building-on-device-ai-memory-with-qdrant-edge/materials/transcripts/sc-Qdrant-C3-transcripts.md b/courses/building-on-device-ai-memory-with-qdrant-edge/materials/transcripts/sc-Qdrant-C3-transcripts.md new file mode 100644 index 0000000..8e155ad --- /dev/null +++ b/courses/building-on-device-ai-memory-with-qdrant-edge/materials/transcripts/sc-Qdrant-C3-transcripts.md @@ -0,0 +1,121 @@ +# Transcripts: Building On-Device AI Memory with Qdrant Edge +Source: videos/sc-Qdrant-C3-L1-v2.mp4 to sc-Qdrant-C3-L5-v2.mp4 (24:20 in total) +Instructor: Dylan Couzon + +All five verified lesson transcripts in one file, concatenated in lesson order from +`sc-Qdrant-C3-L1-transcript.md` to `sc-Qdrant-C3-L5-transcript.md`. Each lesson keeps its own +title, source and instructor lines; only the lesson title is demoted from `#` to `##` so this file +has a single top-level heading. The text is otherwise unchanged; see `README.md` for how the +transcripts were produced and `corrections.md` for every edit. + +--- + +## Lesson 1: Why Devices Need Memory +Source: videos/sc-Qdrant-C3-L1-v2.mp4 (4:36) +Instructor: Dylan Couzon + +In this lesson, you'll learn how devices turn text and images into searchable memories. Let's dive in. I interact with these robots through my phone. And here I can ask it, where did you last see my watch? And here you can see that my watch was last seen yesterday in my hotel room. And there's no specific training, there's no LLM running on this device. We basically teach an object once, and then it saves every new sightings of this object. To do this, this robot is using our open-source vector search engine, Qdrant Edge. So how did we do that? You know, starting with the simplest version of a memory, text. + +I can take a note, "I had ramen for lunch." An embedding model turns the text into a vector. A vector is a list of numbers that represents the meaning of text. A vector search engine then creates a navigable graph to allow search through those vectors to happen extremely fast, sometimes in less than a millisecond. A question goes through the same embedding model. For example, "What did I eat for lunch?" Then that vector search engine returns the approximate nearest neighbor to this query. The closer the vectors, the higher the similarity score. + +Returning the closest vectors is called retrieval. This is the core pattern you will use in this course. That same idea also works with photos. For this, we use the model called CLIP. CLIP has one encoder for images, another one for text. It creates a similar embedding whether you show it an image of a cat or describe a cat. That means we can search images using words. And the same idea lets this robot recognize what it sees by comparing with what it knows. + +If the similarity score is above a certain threshold, we have a match. And to do this, we need no LLM, no retraining, or no custom vision pipeline. You might ask, why would we want to keep memory on the device? And first of all, because it can work without an internet connection. This robot is not connected to any network. It only has direct Wi-Fi to my phone. + +So this can be useful in many use cases when, for example, you cannot afford or you cannot wait for a network round trip. Recall can be a lot faster and a lot more reliable when on the device. Also, it allows for your private memories to stay private without any cloud storage or API required. Cloud storage can still be useful when, you know, some memories needs to be shared. And local devices can also have some limitations like less compute and less storage. This is why Qdrant Edge has the capability to sync with a cloud server whenever you need more compute or more storage space. + +But for today, we will only be working locally. You can always use cloud and local retrieval combined together if your use case requires it. Now, let's face this robot towards me. All right. So here you can see that it says Dylan's coffee mug. It doesn't say any coffee mug. It knows that it is mine because it is a view that I taught it. + +And every time it sees that coffee mug, it will create a new sighting saying where and when it was seen. So I can ask it again, where did you last see Dylan's coffee mug? You can see like the latest memories here. And here, you know, I have two cat plushies that are currently being unrecognized. And because I have not taught it yet, I haven't told it what this is. So here I can say, you know, this is my cat Luna. + +This is Luna. And the robots will take a few images, a few representations of that plushie. And every time it sees it, it takes a new image that it turns into vectors. So here I can confirm that memory. And the more it sees Luna, the better it will get at recognizing it. So we can see the score went up to like 97. + +And so this is how we create all the memories. We created, stored, and recalled memories without using any LLM. What changed was only some vectors. Next, we'll open the robots and see how it was built. + +--- + +## Lesson 2: Building the Device +Source: videos/sc-Qdrant-C3-L2-v2.mp4 (2:32) +Instructor: Dylan Couzon + +In this lesson, you'll see how the AI assistant device's hardware, models, and local memory work together. You'll follow the path from a camera image or spoken name to a memory the AI assistant device can find by sight or meaning. Alright, let's get to it. We saw the robot recall my watch. We told it Luna. Now, let's look at how it was built. The camera, the computer, the storage, and the interface. Everything runs inside the application on the device, including the embedding models and vector search. + +There is no separate vector database server, and the memory is stored in a folder on the device. The complete build is open source, with all the code, hardware details, enclosure 3D print files, and setup instructions. Let's take a look at the repo now. This specific robot uses an NVIDIA Jetson Orin with 8 gigabytes of memory. Other parts include an SSD drive, a USB camera, and a 3D printed enclosure. But please note that you do not need any specific hardware for this course. + +You can run the application on a Raspberry Pi 5 or your personal computer. In the very last lesson, we will run this code project on a local computer as well. Let's briefly take a look at how the case was designed. The robot includes no microphone, speaker, or screen. I am connected to it with my phone through direct Wi-Fi, and the phone is the interface. The processing stays on the robot. + +So here, you can see that you have all the multiple representations of all the memories that we have on the robot. And you can see that I can rename, forget them, or drop a specific view. The phone keeps the physical build compact, but you could add the microphone, speaker, or screen to the robot instead. The memory architecture would stay the same, as all the processing is already done on the robot. Then, how do we go from a camera frame to a memory? Our goal today is to create memory, not this exact robot. + +This robot is just one complete example. But you can think about smart glasses, security system, home assistant, or in-the-field device. They all require different cameras, sensors, and interfaces, but it is the same memory loop underneath. Here, you just need to decide what your device should remember, and then design it however you wish. In the next lesson, we will build the core memory loop, which is the brain of the robot. + +--- + +## Lesson 3: Store, Find, and Forget Memories +Source: videos/sc-Qdrant-C3-L3-v2.mp4 (6:38) +Instructor: Dylan Couzon + +What does it take to give an application a memory? In this lesson, you'll build one from scratch and test how it stores, finds, filters, and forgets text and images. Alright, let's go! We start by importing our helper functions and Qdrant Edge. Then we create a shard. So the shard is just the directory where your memory store will live. We'll also call this a collection. So here it is named mem_shard. Then we configure Qdrant Edge. We say which types of vectors do we want. And here we want a text and an image vector. And then we configure the parameters for those vectors. So here the size for the text is 768 embeddings. And this means that the embedding model creates 768 dimensions for the text. And then for the images, we create 512. Then for the distance metrics, we use cosine to compare the vectors to the questions for both text and images. And then we just create the shard. + +Then to verify that our shard is working, we're going to try to ask a question. So the question right now is a good place to eat or drink nearby. But you could change that question to anything else. We're then going to embed that question with the same embedding model that we're using to embed our text memories. And then we're going to query the shard. So you can see that we're requesting for the nearest vectors to our questions and we're doing a limit of 3. And right now nothing returned because we haven't stored any memories yet into our collection. So to create those memories, we're going to import a JSON file that contains text notes as memories, but also a payload on those memories. So metadata like category and price that gives us additional information about those memories. + +Then what we need to do is to embed those memories one by one. And once this is done, this is what a vector looks like. So you can see that it is just an array of numbers that represents the meaning behind the text. Then the only thing that is left to do is to upload those memories into Qdrant. So we use what we call points to store the memory with all the vectors, but also any metadata that is associated with that memory. So we have the ID, we have the source type, which is where the memory came from. We have a category label, the location, the exact timestamp, the text version of the notes and the price when it is applicable. + +Then we can ask our question again, a good place to eat or drink nearby. We emit that query and then we ask our memory store again. And here we can see our top three matches. The first one, a great little coffee place on 5th with outdoor seating and fast Wi-Fi. And our best score is 0.58. But then what happens when I want to query memories from a specific category or a specific price? + +What we're doing here, we're creating field indexes to enable us filtering on specific fields. So here we enable filtering on category and price. Now we want to ask the same question again, a good place to eat or drink nearby. But we have some filters that are applied. We want the category to match food and the price to be lower than $15. Now you can see that all the results match our food category filter and also match our price condition. + +And the best match is a great little coffee place on 5th with outdoor seating and fast Wi-Fi with a score of 0.587. Our robot doesn't only process text. It also process images and live video. Now let's add images to our memories. We're importing a bunch of different images, all representing different objects. We started with 20 text memories. + +And now we stored an additional 165 photos as memory into our memory store for a total of 185 memories. Since we're using CLIP, an embedding model that matches text and images, we can ask it a red bicycle, embed that query, and then search. And here we have a red bicycle retrieved. + +The last thing we may want to do is to forget memories. So here we're just taking the first memory that we've learned. Great little coffee place on 5th with outdoor seating and fast Wi-Fi. And we're forgetting it by calling delete_points. And here you can see the before and after of our question, a good place to eat or drink nearby. Where in the before, we're still retrieving great little coffee place on 5th. + +Now that this memory was deleted, our first result is Found a quiet cafe with good Wi-Fi to work from near the park, which was our second result before deletion. Let's also take a look at the latency. This plot was generated on my own Mac using the code that is in this notebook. Different machines will generate different latency numbers, but the trend should still be similar. You can see here that the x-axis is logarithmic. So the last value is 250 times greater than the first one. + +So you can see that the latency has gone up about 50 times, while the numbers of memories has gone up 250 times. Vector databases are optimized to handle hundreds of thousands or even millions of memories, but forgetting and cleaning up your data is still valuable. In this lesson, you created your first memory store. You stored text notes and images, and you learned to search, add filters, and learned about the latency implications. In the next lesson, you would put all of that together and start building a voice assistant that can search through your day. + +--- + +## Lesson 4: Your On-Device Assistant +Source: videos/sc-Qdrant-C3-L4-v2.mp4 (4:58) +Instructor: Dylan Couzon + +Your day is scattered across photos, voice notes, and text. In this lesson, you'll bring them into one on-device memory. Search it with typed or spoken questions and add a new memory you can recall immediately. Let's have some fun. First, let's import our helper functions and load our memories. Here, you can see that we have 42 different memories, 17 photos, 20 text memories, and 5 voice memories. First, let's look at our image memories. Today, you can see that at 7:20 we went to the gym, then we had some pastries, coffee, looks like we went to work, we had ramen for lunch, we went to do some shopping, we went to the park, seems like we visited downtown, and then we went home around 5:30 p.m. Now, let's take a look at our notes. So you can see that our notes seem to be matching our image memories. We went to the gym this morning, we renewed our membership, we went to a bakery, then we found a coffee place, we parked our bike near the station when we went to work, and then we had some notes about work. Some of these notes were added as text, and some others were added as voice notes. + +So now, let's look into how those voice notes were created. Here, we have a note audio file. "Note to self, the ramen downtown was incredible. $14 and worth it. Sat right by the window." We used a Whisper local model to create that transcript. What we are storing is this transcript, not the audio itself. Now, as we've done before, we will create our memory store, then embed each memory with the source type as metadata, if it was text or voice. And lastly, we will be storing our memory photos. Now, you're ready to recall your day. We're creating a helper function called recall that will allow you to search through your photos, voice notes, and text notes. + +And here, we're going to be asking the question, what was the ramen place downtown? We're calling that recall function. And here, you can see that we're using a minimum text score of 0.6 and a minimum photo score of 0.23. We're using different scores or thresholds because we're using different models. You will learn about those thresholds in more details. As you can see, we have retrieved multiple voice notes and multiple text notes for our search. But every result that falls below the threshold we have set gets ignored. So this is why those results are grayed out here. + +In our top results, you can see that today, at 12:30, I created a text note that says, "Try the new ramen place downtown, everyone raves about the Tonkotsu." Then, at 1:12 p.m., apparently, I went to a place called Ramen-ya, and it looks like I tried the Tonkotsu ramen. Then, after lunch, I created a voice note that said, "Note to self, the ramen downtown was incredible." Then, since we're transcribing the audio into text, the search works exactly the same as a text search. And we ask, hey, where did I park the bike? And we can see a note from this morning, "Park the bike near the station, second rack from the entrance." And it looks like we also took a picture before going into work this morning. + +Looking at the results that did not meet our threshold, you can see here that we're mentioning the park. And so this is why there's a weak match with parking the bike, and this could be the reason why these notes showed up at the top, but still below our threshold. Now, let's test out our assistant. Here, we're adding a note that says, "Left the spare key with the neighbor in apartment 4B." But you could also be adding a voice note or an image here. Now, let's ask the assistant a question related to the memory we just created. + +So here, I'm going to ask it, where did I leave the spare key? But you can ask it any kind of question you want that is related to the memory you just created. And here, our best match with the highest similarity score is the note that we wrote earlier, "Left the spare key with the neighbor in apartment 4B," which is exactly the memory we were looking for. In this lesson, you created your first assistant that can recall different types of memories. In the next lesson, you will teach your assistant to recognize concepts it has never seen before. Alright, see you there! + +--- + +## Lesson 5: Teaching Your Assistant to See +Source: videos/sc-Qdrant-C3-L5-v2.mp4 (5:36) +Instructor: Dylan Couzon + +What if your assistant could learn to recognize something new? In this lesson, you'll teach the assistant device an object from a few photos, test it on a view it hasn't seen, and combine meaning with recency to surface the right memories, all offline. Let's go! As in the previous lessons, we will start by importing our helper functions, Qdrant Edge, and create a new memory store. Then, we will create two more reusable functions. The first one, add_memory, that adds a memory into our memory bank. + +And the second one, teach, which just adds multiple memories at once. Then, let's seed a few objects into our memory store. Alright, and here we have a few sample memories. A bicycle, chess pieces, and a camera. We're gonna be using those two images on the left side to teach, and the third one on the right side to test. If you want to teach your own object, you can use the photo uploader here. + +In the first one, you need to upload two images to teach with, and in the second one, you need to upload one image to test with. The first two images will be replacing these two on the left side, and the last image will be replacing this image on the right side. Uploading your own images is fully optional. If you don't upload any images, the rubber ducks will be used as the default. Now, let's show our test image to the AI assistant to see what happens. And here, you can see that we showed it the image of a duck, but since we have never taught it a duck, the closest match that we have so far is a bicycle, which is not a duck. + +Now, let's teach our AI assistant those two images. We're entering the subject name, "rubber duck", for the metadata, and then we're teaching the photos. And here, you can see that we taught those two views of the rubber duck. Now, let's try to recognize our test image again and see what happens. And here, you can clearly see that the closest memory to the photo that we showed was another rubber duck, with a similarity score of 0.88. Now, retrieval will always return the highest similarity, no matter how low. + +To find the right balance for the threshold, I have run an experiment with a large set of images that includes 220 non-matches and 6 matches, and here is the result. As you can see, the lowest scoring match is at 0.86, and the highest non-match is at 0.74. So 0.8 is the right middle for the threshold in my use case. Depending on the significance of false positives and false negatives, for your use case, you can select a threshold that fits your use case the best. So now, let's build our assistant end-to-end. Let's create a new memory store. + +And now, let's store a few memories from our day. A few notes and photos. Then, we're going to add the memory that we just created. So here are rubber ducks with an additional note: "Rubber duck for the bath from the toy shop on Elm Street." And we can see here that our assistant now has a total of 145 memories. + +To make this AI assistant even better, we might want to prioritize certain types of memories, like the most recent ones. Because here, when we ask the question, "what is my gym locker code?", you can see that the memory from August 30th scored higher than the memory from September 14th. And ideally, we would want the most recent memory with the most recent code to score the highest. To address this problem, you can add a soft forgetting. So here, we're creating a half-life of seven days in seconds. + +And then we're adding a freshness weight. It tells how important it is that the memories should be recent. Now, running that same query with the freshness applied, you can see that the most recent memory is the one that scores the highest by a huge margin. Now that we have all the components for your AI assistant, let's ask it a few questions. What did I get at the toy shop? Show me the bakery. + +Where did I have brunch? And what did the bike chain cost? That first question, "what did I get at the toy shop?", is the rubber duck memory that we created. If you use your own memory and your own photos, you might have a different result here. And then "show me the bakery", "where did I have brunch?", and "what did the bike chain cost?" all seem to fit previous memory well. Here we can see that it remembered a new bakery on the corner. There's an amazing morning cronut. And we have a picture of that cronut. + +Brunch at the cafe with pancakes and coffee for $14. And we see here I had brunch with family and the bike chain. And we can see that we bought it for $32. And we also have found like a picture of our bike attached. And all of this was done through pure retrieval without any LLM involved. In this lesson, you put everything together and taught your assistant a new memory. + +If you apply the same concepts to a video frame by frame, this is how the robot that we've showcased today works. The robot is based on the same assistant code that we've built here. It just runs on a loop frame by frame. In the next lesson, we will run the full AI assistant loop on the personal machine. diff --git a/courses/building-on-device-ai-memory-with-qdrant-edge/spec.coding-agent-lab.md b/courses/building-on-device-ai-memory-with-qdrant-edge/spec.coding-agent-lab.md new file mode 100644 index 0000000..571b6c9 --- /dev/null +++ b/courses/building-on-device-ai-memory-with-qdrant-edge/spec.coding-agent-lab.md @@ -0,0 +1,772 @@ +# Spec: On-Device Memory Assistant — Standalone Takeaway + +> **What you're building.** A page you can open and ask *"where did I leave the bike?"* — and a note +> from this morning, a voice memo, and a photo come back side by side, each with its similarity +> score, the weak ones dimmed rather than dressed up as answers. Show it two photos of an object and +> hold a third back, and it names the third. Delete a memory and watch the next-best answer move up. +> No LLM, no cloud, no API key: every answer comes out of vectors on local disk. +> +> **Fastest path.** Answer the §0 gate's first question with the **recommended baseline build** and +> this gets built end-to-end with no further decisions. The build is **complete when the offline test +> suite passes** — nothing here needs a key, an account, or a network call at query time; the only +> network use is the one-time model download. +> +> **This file is self-contained.** The embedded **Course Context Pack** (`CTX-A`…`CTX-E`, at the end) +> replaces every external course reference — nothing in this spec requires the course platform, its +> notebooks, or its transcripts. `(CTX-X)` anchors mark course-derived knowledge. The **Decision +> Ledger** below holds every point where this build could legitimately diverge, each pinned to one +> course-derived default, so the spec is buildable and evaluatable **as-is**. +> +> **Provenance.** Generated from the *Building On-Device AI Memory with Qdrant Edge* notebook dump +> (`L3`/`L4`/`L5` notebooks, `helper.py`, `requirements.txt`, `ro_shared_data/`), the five lesson +> transcripts, and the slide descriptions, on **2026-09-11**, by spec-generation-guide commit +> **`8e44ecb`**. Lesson numbers follow the **transcripts** (authoritative) — the slide decks carry a +> *third*, earlier numbering that is off by one and must not be used (CTX-E). Slides are cited with +> their status: **shown** (one of the 8 that appear in a lesson video) or **drafted** (one of the 15 +> that exist only in the decks) — a drafted slide never outranks shipped code or narration. Facts +> labelled **[environment]** come from `environment.md` — the AI Coding Lab runtime this build +> targets — **not** from the course; the course is never credited with them. + +--- + +## 0. Before you build — REQUIRED (do this first) + +You are the build agent. Before writing ANY code, you MUST surface the design decisions in the +Decision Ledger (next section) to the person you are building for. Do **not** skip this because the +defaults look complete — the defaults exist so the build is *reproducible*, not because they are the +right choice for this person. + +1. **Use a structured question tool if you have one.** In the AI Coding Lab your structured-question + mechanism is the fenced `choices` block: put the explanation in prose *above* it, put the fence + **last** in the message, one option per line, and stop. Label the question + `**Question N of M — **` on its own line before the fence so it renders as a question + box. A click sends the option line verbatim; a typed answer that names an option counts exactly + the same. Only the latest message's buttons stay live, so never ask anyone to go back and change + an earlier answer — re-ask it as a new question instead. If you are running somewhere else, use + that environment's structured-question tool (e.g. Claude Code's `AskUserQuestion`); only if none + exists, list the rows in your reply and ask for an answer to each. +2. **First question — recommended baseline build, or customize?** Ask exactly one question with two + options: the **recommended baseline build** — every Ledger row resolves to its Default: mostly the + course's own choices, with a lighter stand-in wherever the course's choice needs setup you may not + have (here: a Qdrant cluster URL and API key, on D2); the step-5 checklist marks exactly where the + baseline differs from the course — or **customize** the decisions row by row. *(In this spec every + row's Default happens to be the course's own choice, so a baseline build should print no `≠` + lines in step 5; if you find one, say so rather than assuming it is fine.)* If the baseline is + chosen, skip step 3: go straight to the step-5 checklist and build. If customize is chosen, + continue with step 3. +3. **Present the Ledger ONE ROW AT A TIME — one question per row.** For each row ask a single + question: the **Decision** as the prompt, its **Options** as the choices. Append "(course + default)" to the option the course actually used. You may also mark an option "(Recommended)" — + your judgment for THIS person, made now, at gate time; with no contextual reason to depart, + recommend the row's Default. When your recommended option IS the course's actual choice, merge the + labels into "(Recommended - course default)". A recommendation never removes or moves the "(course + default)" label, and no option is ever labelled with a bare "(default)" — these two labels and + their merged form are the only option labels. Put any realization beyond the offered options under + a free-text answer. Ask about **every** row. A per-call item limit is NEVER a reason to drop, + skip, merge, or silently default a row — make as many separate messages as there are rows. +4. **Presenting any of these questions ENDS YOUR TURN — stop here; write no code, create or edit no + file, take no other build action.** Keep asking, one row at a time, until **every** row has an + answer (a chosen option, an explicit "use the default", or the step-2 baseline answer, which + resolves every row at once). Answers to *some* rows do NOT release the build; "no reply yet" is + not an answer — wait. +5. **Before the first line of code, print a resolved-decision checklist and write it to + `resolved-decisions.md`** in the workspace root — every Ledger row with its final value, each line + carrying a deviation mark: `= course choice`, or `≠ course choice (course used: