From adf3d83635f5d1e4a4daaca51aebcf4a0e7b87c5 Mon Sep 17 00:00:00 2001 From: Subhendu Ghosh Date: Sat, 22 Aug 2026 00:00:19 +0530 Subject: [PATCH] Add the HLD mock loop: frozen reference, attempt, diff, evaluation Brings HLD up to parity with the LLD mock loop, and fixes a weakness in the practice docs it writes. Until now an HLD session saved exactly one artifact: a practice doc that blended what the candidate produced with what the correct answer was. Two things followed. The reference the interviewer graded against was never persisted -- it lived only in the model's context, where it could drift under candidate pushback, so "the reference is frozen" was advisory rather than enforceable. And because the candidate's own design was never captured separately, there was nothing to diff against; feedback was reconstructed at the end of the round, which systematically cleans up what was actually said. Seven tools, mirroring the LLD side: start_hld_mock_attempt freeze reference.md BEFORE posing the problem save_hld_attempt what the candidate actually designed save_hld_diff matched / missed invariants / diverged save_hld_evaluation seven-dimension scorecard; also logs the session list_hld_mock_attempts what's been attempted, what's incomplete read_hld_mock_file open one of the four files get_hld_feedback rubric averages, weakest dimensions The pre-commitment is enforced rather than promised: start_hld_mock_attempt refuses to overwrite an existing reference.md and has no overwrite flag, and written_at comes from the server's own clock so it can't be backdated. A genuine re-attempt opens a -r2 folder. save_hld_diff separates a missed *invariant* (a real gap) from a divergence at a *choice point* (not a gap if the trade-off reasoning held), and compares end-to-end flows rather than just which components got named -- two designs can list identical boxes and route a request completely differently. The attempt folder holds the evidence, ungroomed; the practice doc holds the correct design. Nothing under Mock Solutions/ is ever surfaced by list_practice_docs, and no tool points a tracker doc_path at it -- an attempt groomed toward the right answer reads clean six weeks later and tells you nothing about which parts you actually got right. Also here: - The four rubric helpers are parameterized on prefix rather than duplicated, so LLD and HLD share the aggregation code. HLD scores are stored under an "hld:" prefix. - get_progress_summary filtered only the "lld:" prefix when picking out behavioral competencies, so HLD rubric scores would have rendered as behavioral ones. Both rubrics now report in their own blocks. - save_practice_doc and start_hld_mock_attempt warn (never block) when a design doc has no "## End-to-end flows" section. A component diagram shows what exists; that section shows what happens, and its absence is what makes an HLD doc hard to revise from. - test_hld_tools.py: 79 hermetic checks, one named case per acceptance criterion, run against a synthetic HLD root in a temp directory. - skills/hld-interviewer/SKILL.md, updated for the new flow. The only copy on this machine lived in a per-session upload cache; this is the durable one. Co-Authored-By: Claude Opus 5 --- README.md | 64 ++- server.py | 764 ++++++++++++++++++++++++++++++-- skills/hld-interviewer/SKILL.md | 337 ++++++++++++++ test_hld_tools.py | 346 +++++++++++++++ test_server.py | 3 + 5 files changed, 1483 insertions(+), 31 deletions(-) create mode 100644 skills/hld-interviewer/SKILL.md create mode 100644 test_hld_tools.py diff --git a/README.md b/README.md index 70e46a0..be4d690 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ Session/tracking data, under `~/interview-prep/` by default: **Per-problem solutions/docs live in four independent, separately configurable directories** — each defaults to a folder under `~/interview-prep/docs/` but is normally pointed at wherever your real practice work lives: - **`DSA_SOLUTIONS_DIR`** (e.g. `~/code/DSA_mock`) — organized as `/.py`, one real Python file per problem grouped by topic, matching a typical personal LeetCode-practice layout. DSA is code, not markdown — see `save_dsa_solution` below. -- **`HLD_SOLUTIONS_DIR`** (e.g. `~/code/HLD`) — one markdown file per HLD problem (e.g. `design-rate-limiter.md`). +- **`HLD_SOLUTIONS_DIR`** (e.g. `~/code/HLD`) — one markdown file per HLD problem (e.g. `design-rate-limiter.md`), plus **`Mock Solutions/`**, the HLD mock loop: one dated folder per attempt holding the reference Claude froze *before* posing the problem, what you actually designed, the diff between them, and the scored evaluation. See [HLD mock interviews](#hld-mock-interviews) below. - **`LLD_SOLUTIONS_DIR`** (e.g. `~/code/LLD`) — holds two separate things: the **reference corpus** of your own designs, one self-contained `.py` per problem grouped into category folders (`1_state_machine/vending_machine.py`), plus flat `design-parking-lot.md` write-ups from `save_practice_doc`; and **`Mock Solutions/`**, the mock-interview loop where you attempt a problem and Claude grades it. See [LLD mock interviews](#lld-mock-interviews) below. A third file, **`DRILL_LOG.md`**, sits at the top level: one running, append-only log of short LLD drills (see [LLD drills](#lld-drills)). - **`BEHAVIORAL_SOLUTIONS_DIR`** (e.g. `~/code/Behavioral`) — holds a single `candidate_context.md`: your reusable background + STAR story bank, with a "Current Focus" section you refresh per company/role. See `save_candidate_context` below. @@ -40,6 +40,13 @@ Session/tracking data, under `~/interview-prep/` by default: | `read_lld_solution` | Open ONE LLD file in full — a past design, a corpus doc, or a mock attempt you're about to grade | | `import_solved_lld_problem(s)` | Backfill the tracker from an LLD design already on disk, *without* touching the file. Refuses anything that isn't `kind=solution` | | `save_lld_solution` | Write a NEW design into the reference corpus, in the right category folder; refuses to overwrite unless told to | +| `start_hld_mock_attempt` | Freeze the grading reference BEFORE posing an HLD problem — creates `Mock Solutions/-/` with `reference.md`. Refuses to overwrite an existing reference; `written_at` comes from the server clock so it can't be backdated | +| `save_hld_attempt` | Capture what you actually designed as `attempt.md` — your near-verbatim turns first, then the interviewer's structured rendering and an explicit list of what you only reached after being prompted | +| `save_hld_diff` | Compare attempt against the frozen reference: matched / **missed invariants** (real gaps) / **diverged at choice points** (fine if the reasoning held), flows as well as boxes. Errors if either side is missing | +| `save_hld_evaluation` | Score an HLD mock against the fixed 7-dimension rubric — writes `evaluation.md`, records the scores and logs the session (so no separate `log_session` for HLD mocks) | +| `list_hld_mock_attempts` | Every HLD attempt folder: which of the four files exist, verdicts, and which attempts were left incomplete | +| `read_hld_mock_file` | Open one of `reference.md` / `attempt.md` / `diff.md` / `evaluation.md` in full | +| `get_hld_feedback` | HLD rubric averages, weakest dimensions, recent verdicts at level — call at session start so the problem choice targets the weak dimensions | | `start_mock_attempt` | Pose an LLD problem — creates `Mock Solutions//` with `problem.md` and an `attempt.py` stub for *you* to fill in. Never overwrites an existing attempt; re-posing opens the next round | | `list_mock_attempts` | Every mock problem and round: which files exist, what's still awaiting grading, scores and verdicts | | `save_mock_evaluation` | Grade an attempt against the fixed 7-dimension rubric — writes `evaluation.md`, logs the session, and regenerates `feedback.md` | @@ -106,6 +113,51 @@ Claude will call `scan_dsa_directory()`, read each file's problem-statement snip `scan_lld_directory()` returns every file with a **Kind** column — `solution` (a per-problem design), `category-doc` (a folder README), `aggregate-doc` (`INDEX.md`, `QUICK_REFERENCE.md` and friends, which span many problems), `practice-doc` (a markdown write-up this server wrote via `save_practice_doc`), `drill-log` (`DRILL_LOG.md`), or `other`. Only `solution` rows are importable; `import_solved_lld_problem(s)` refuses the rest, so an index file can never get linked to a single problem. Claude matches filenames to catalog ids by judgment (`8_lru_cache.py` → `design-lru-cache-oop`), and roughly half the corpus isn't in the 25-entry built-in LLD catalog at all (`order_lifecycle.py`, `whatsapp_messaging.py`, `audit_trail.py`, …) — those need `add_custom_problem` first. Nothing under `~/code/LLD` is modified; only `index.json` is written. +## HLD mock interviews + +Same loop as LLD, with one structural difference: **the reference is written and frozen before you're asked the question.** Claude generates its grading reference, `start_hld_mock_attempt` puts it on disk, and only then does the interview start. A reference that still lives in the model's context can drift under your pushback; one on disk can't. + +``` +~/code/HLD/Mock Solutions/ + 2026-08-21-design-distributed-task-scheduler/ + reference.md # frozen BEFORE the problem was posed; written_at is the server's clock + attempt.md # what you actually designed, warts intact + diff.md # matched / missed invariants / diverged at choice points + evaluation.md # verdict, verdict-at-level, seven-dimension scorecard +``` + +One folder per *attempt*, named `YYYY-MM-DD-`; a second attempt at the same problem on the same day is `-r2`. Attempting a problem again never lands on the earlier evidence. + +**The flow.** Say: + +> Let's do a mock HLD interview. + +1. Claude calls `get_hld_feedback()` and `get_catalog("HLD")`, picks a problem aimed at your weakest dimensions, generates its reference, and calls `start_hld_mock_attempt(...)` — **before** giving you the prompt. +2. You do the interview. +3. At the end: `save_hld_attempt(...)` → `save_hld_diff(...)` → `save_hld_evaluation(...)` → `save_practice_doc(...)`. + +**Attempt vs. practice doc.** The attempt folder holds the evidence, ungroomed; the practice doc holds the correct design. Keeping them apart is the whole point — an attempt rewritten toward the right answer reads clean six weeks later and tells you nothing about which parts you actually got right. Nothing under `Mock Solutions/` is ever surfaced by `list_practice_docs`. + +**`raw_turns` is the load-bearing argument** of `save_hld_attempt`. Written from memory instead, the attempt gets unconsciously tidied — a justification you never gave gets filled in, an explanation that doubled back gets straightened — and the diff then grades a cleaned-up version of what happened. `attempt.md` also carries an explicit *arrived-at-only-after-prompting* list: what you reached only once the gap was named, with the prompting question quoted. + +**Missed vs. diverged.** `save_hld_diff` separates a missing reference **invariant** (a real gap) from a different option taken at a **choice point** (not a gap if the trade-off reasoning was sound). The diff compares end-to-end flows too, not just which components got named — two designs can list identical boxes and route a request completely differently. + +**The rubric**, seven dimensions, 1-5, matching the `hld-interviewer` skill's scorecard, stored under `hld:`-prefixed keys in `index.json`: + +| Dimension | What it measures | +|---|---| +| `requirements` | Functional/non-functional, what got pinned down | +| `capacity-estimation` | Whether you justified the numbers yourself | +| `architecture` | Component design, and whether each box is warranted | +| `deep-dives` | Depth on the hard parts under drilling | +| `scale-calibration` | Reading the stated numbers vs. pattern-matching to FAANG scale | +| `communication` | Structure, signposting, driving the session | +| `composure` | Behaviour under pushback and persona pressure | + +Free-form keys are rejected — the aggregate is the whole value, and keys that vary per session never aggregate. + +**End-to-end flows.** Every HLD design document — `reference.md` and anything `save_practice_doc` writes — should carry an `## End-to-end flows` section right after the architecture diagram: at least three numbered flows (write path, primary read-or-execute path, failure/recovery path), each step naming the component, the operation and the datastore touched. A diagram shows what exists; this shows what happens. Its absence is a **warning in the tool result, not an error** — the doc still gets written. + ## LLD mock interviews The loop that turns practice into targeted practice. You write the design yourself; Claude grades it as the interviewer and remembers where you're weak. @@ -171,7 +223,7 @@ get_lld_drill_log(limit=5) # limit=0 returns the whole file Say this at the start of a chat (or bake it into a Claude Desktop project/skill): -> At the start of every practice session, call `get_progress_summary`, then `suggest_next_problems` for the type we're practicing (DSA/HLD/LLD) and let me pick from the top few. For LLD also call `get_lld_feedback` first and aim the pick at my weakest rubric dimensions. After we solve/discuss it, call `log_session` (with `problem_id` set). Then for HLD/LLD call `save_practice_doc` with the full write-up (HLD: requirements, capacity estimate, architecture, API/data model, trade-offs; LLD: class design, patterns used, key decisions). For DSA, if the problem isn't already on disk, call `save_dsa_solution` with the final code and a short explanation. +> At the start of every practice session, call `get_progress_summary`, then `suggest_next_problems` for the type we're practicing (DSA/HLD/LLD) and let me pick from the top few. For LLD also call `get_lld_feedback` first and aim the pick at my weakest rubric dimensions; for HLD call `get_hld_feedback` first for the same reason. After we solve/discuss it, call `log_session` (with `problem_id` set). Then for HLD/LLD call `save_practice_doc` with the full write-up (HLD: requirements, capacity estimate, architecture, API/data model, trade-offs; LLD: class design, patterns used, key decisions). For DSA, if the problem isn't already on disk, call `save_dsa_solution` with the final code and a short explanation. > > If I say I want to *attempt* an LLD problem myself rather than discuss it, run the mock loop instead: `start_mock_attempt`, wait for me to write `attempt.py`, then `read_lld_solution`, `save_mock_evaluation` (score honestly — inflated scores break the feedback loop), `save_ideal_solution`, and `save_simple_solution` for the cut-down version. @@ -211,9 +263,10 @@ A stdio-transport MCP server (what this is) isn't a background daemon you start Four ways to check it's actually working, in increasing order of realism: 1. **Does it even boot?** `python3 server.py` from this folder — should start and hang silently (that's correct; it's waiting for a client on stdin). Ctrl+C to stop. -2. **Does it speak MCP correctly?** `python3 test_server.py` — spawns the server as a real MCP client would, does the protocol handshake, lists all 29 tools, and calls `get_progress_summary` for real. Read-only, safe to run anytime. A clean "All checks passed" means the server itself is solid, independent of Claude Desktop. +2. **Does it speak MCP correctly?** `python3 test_server.py` — spawns the server as a real MCP client would, does the protocol handshake, lists all 37 tools, and calls `get_progress_summary` for real. Read-only, safe to run anytime. A clean "All checks passed" means the server itself is solid, independent of Claude Desktop. 3. **Do the LLD tools actually behave?** `python3 test_lld_tools.py` — builds a synthetic design repo in a temp directory and exercises every LLD tool against it: path guards, kind classification, rubric validation, the full mock loop, and above all that `attempt.py` comes out byte-identical to what was written. Hermetic — it never touches `~/code/LLD` or your real `index.json`. -4. **Is Claude Desktop actually using it?** +4. **Do the HLD mock tools actually behave?** `python3 test_hld_tools.py` — builds a synthetic HLD root in a temp directory and walks the acceptance checklist: a frozen `reference.md` can't be overwritten, `written_at` comes from the server clock, a diff with only one side errors, a rubric key outside the fixed seven is rejected, `Mock Solutions/` never leaks into `list_practice_docs`, a doc with no `## End-to-end flows` warns but still writes, and `get_hld_feedback` on an empty history returns cleanly. Hermetic — it never touches `~/code/HLD` or your real `index.json`. +5. **Is Claude Desktop actually using it?** - Open a chat and look at the tools/connectors icon near the input box — `interview-memory` should be listed with its tool count. - `ps aux | grep server.py` — while Desktop is open, you should see a live `python3 .../server.py` process (Desktop spawns it once you open a chat that uses it, or at startup depending on version). - Logs: `~/Library/Logs/Claude/mcp-server-interview-memory.log`. @@ -226,8 +279,9 @@ This server only ever runs over **stdio** (`mcp.run(transport="stdio")`), the sa Within the local filesystem, a few guardrails are worth knowing about since Claude drives these tools somewhat autonomously: - **Path sanitization.** Every value used to build a filename (`problem_id`, problem titles) is passed through a slugifier that strips everything except `a-z 0-9 -` before it ever touches a path. A value like `../../etc/passwd` becomes `etc-passwd`, not a traversal — verified with an adversarial test during development. -- **Directory containment.** `save_practice_doc` can only write inside `HLD_SOLUTIONS_DIR` or `LLD_SOLUTIONS_DIR` (whichever matches the call); `save_dsa_solution` can only write inside `DSA_SOLUTIONS_DIR`; `save_candidate_context`/`get_candidate_context` are confined to `BEHAVIORAL_SOLUTIONS_DIR`; `import_solved_dsa_problem(s)` can only *link* to files that already live inside `DSA_SOLUTIONS_DIR` (it refuses anything outside it, e.g. `~/.ssh/`, `/etc/`); `get_practice_doc` re-checks that whatever `index.json` points at is still inside the allowed directory for that problem type before reading it. +- **Directory containment.** `save_practice_doc` can only write inside `HLD_SOLUTIONS_DIR` or `LLD_SOLUTIONS_DIR` (whichever matches the call); the HLD mock tools take a caller-supplied `attempt_folder` and resolve it against `HLD_SOLUTIONS_DIR/Mock Solutions/`, refusing anything that lands outside it (and `read_hld_mock_file` only accepts the four known filenames, so `filename` can't name anything else); `save_dsa_solution` can only write inside `DSA_SOLUTIONS_DIR`; `save_candidate_context`/`get_candidate_context` are confined to `BEHAVIORAL_SOLUTIONS_DIR`; `import_solved_dsa_problem(s)` can only *link* to files that already live inside `DSA_SOLUTIONS_DIR` (it refuses anything outside it, e.g. `~/.ssh/`, `/etc/`); `get_practice_doc` re-checks that whatever `index.json` points at is still inside the allowed directory for that problem type before reading it. - **No code execution.** The server only reads and writes text files. It never `exec`s, `eval`s, or runs the DSA solutions it stores — `code`/`content_markdown` arguments are treated purely as bytes to persist. +- **Pre-commitment protection.** `start_hld_mock_attempt` refuses to overwrite an existing `reference.md` outright — there is no `overwrite` flag, because a reference that can be rewritten after the candidate has spoken isn't a reference. A genuine re-attempt opens a new `-r2` folder instead. Its `written_at` comes from the server's own clock, not a tool argument, so it can't be backdated. - **Overwrite protection.** `save_dsa_solution` refuses to replace an existing file unless `overwrite=True` is passed explicitly, since `DSA_SOLUTIONS_DIR` is assumed to hold real, hand-written work. (`save_practice_doc` for HLD/LLD, and `save_candidate_context` for Behavioral, do overwrite by design — they hold Claude's latest write-up/profile, and those directories are managed entirely by this server.) - **Env-var-scoped roots.** `INTERVIEW_PREP_DIR`, `DSA_SOLUTIONS_DIR`, `HLD_SOLUTIONS_DIR`, `LLD_SOLUTIONS_DIR`, and `BEHAVIORAL_SOLUTIONS_DIR` are only ever set by you, in your own Claude Desktop config — they aren't something a chat message can override. diff --git a/server.py b/server.py index 45d8a1d..00b6c82 100644 --- a/server.py +++ b/server.py @@ -21,6 +21,13 @@ each point at an existing personal repo instead of living inside INTERVIEW_PREP_DIR: HLD_SOLUTIONS_DIR/.md One markdown file per HLD problem. + HLD_SOLUTIONS_DIR/Mock Solutions/-/ + The HLD mock loop: reference.md (written and + frozen BEFORE the problem is posed), + attempt.md (what the candidate actually + produced), diff.md and evaluation.md. One + folder per attempt, so a re-attempt never + lands on the earlier evidence. LLD_SOLUTIONS_DIR//.py The LLD reference corpus: one self-contained .py design per problem, grouped by category @@ -76,6 +83,22 @@ The rubric vocabulary is fixed (LLD_RUBRIC) so scores aggregate across sessions -- that aggregate is what step 1 reads, closing the loop. +HLD mock-interview loop (Claude interviews, then grades against a reference +it committed to before the candidate spoke): + 1. get_hld_feedback() -- which of the seven dimensions are weakest + 2. get_catalog("HLD") -- pick a problem that forces them + 3. start_hld_mock_attempt(...) -- freeze the reference BEFORE posing the + problem; it can't be rewritten after + 4. ... run the interview ... + 5. save_hld_attempt(...) -- the candidate's design, raw turns first + 6. save_hld_diff(...) -- matched / missed invariants / diverged at + choice points, flows as well as boxes + 7. save_hld_evaluation(...) -- score it; this also logs the session + 8. save_practice_doc(...) -- the clean revision artifact +The attempt folder holds the evidence, ungroomed; the practice doc holds the +correct design. Keeping them apart is the point -- an attempt rewritten to the +right answer is useless for revision six weeks later. + LLD drills (short focused reps, no rubric, no per-problem folder): 1. get_lld_drill_log() -- what was drilled recently, what's unresolved 2. ... run the drill with the user ... (get_current_time at both ends @@ -186,6 +209,43 @@ ] LLD_RUBRIC_PREFIX = "lld:" +# --- HLD: mock-attempt folders (the pre-committed reference + the diff) ------ +# HLD mirrors the LLD mock loop, but the artifact is markdown rather than code +# and the folder is per-ATTEMPT rather than per-problem: one dated folder holds +# reference.md (written and frozen BEFORE the problem is posed), attempt.md +# (what the candidate actually produced), diff.md and evaluation.md. Excluded +# from list_practice_docs the same way the LLD one is -- an attempt is evidence +# of what happened under time pressure, not revision material. +HLD_MOCK_DIRNAME = "Mock Solutions" +HLD_MOCK_DIR = HLD_SOLUTIONS_DIR / HLD_MOCK_DIRNAME +# The only filenames these tools will read or write inside an attempt folder. +# read_hld_mock_file uses this as its whitelist, so a caller-supplied filename +# can never name anything else. +HLD_MOCK_FILES = ("reference.md", "attempt.md", "diff.md", "evaluation.md") + +# The seven-dimension HLD scorecard, matching the hld-interviewer skill. Fixed +# for the same reason LLD_RUBRIC is: free-form keys never aggregate, and the +# aggregate (get_hld_feedback) is what makes the next session target the weak +# dimensions instead of picking blind. Stored under an "hld:" prefix so they +# share index["competency_scores"] with the LLD rubric without colliding -- +# "requirements" here and "requirements-and-scope" there are different scales. +HLD_RUBRIC = [ + "requirements", + "capacity-estimation", + "architecture", + "deep-dives", + "scale-calibration", + "communication", + "composure", +] +HLD_RUBRIC_PREFIX = "hld:" + +# Every HLD design document must narrate how a request actually moves through +# the system, not just which boxes exist. A component diagram shows what +# exists; this section shows what happens. Its absence is warned about (never +# blocked) when a doc or a reference is written. +FLOWS_HEADING = "## End-to-end flows" + VALID_TYPES = {"HLD", "LLD", "DSA", "Behavioral", "Other"} CATALOG_TYPES = {"DSA", "HLD", "LLD", "Behavioral"} VALID_VERDICTS = {"Strong Hire", "Hire", "Lean Hire", "Lean No Hire", "No Hire"} @@ -602,32 +662,108 @@ def _ensure_drill_log() -> None: ) -def _rubric_scores() -> dict: - """The LLD rubric slice of index["competency_scores"], keys un-prefixed.""" +def _rubric_scores(prefix: str = LLD_RUBRIC_PREFIX) -> dict: + """One rubric's slice of index["competency_scores"], keys un-prefixed. The + LLD and HLD rubrics share that dict under different prefixes, so every + reader here is scoped by prefix; the defaults keep the LLD callers reading + the LLD rubric.""" scores = _load_index().get("competency_scores", {}) - return { - k[len(LLD_RUBRIC_PREFIX):]: v - for k, v in scores.items() - if k.startswith(LLD_RUBRIC_PREFIX) - } + return {k[len(prefix):]: v for k, v in scores.items() if k.startswith(prefix)} -def _rated_dimensions() -> List[tuple]: +def _rated_dimensions(prefix: str = LLD_RUBRIC_PREFIX) -> List[tuple]: """(dimension, avg, count) for every scored rubric dimension, worst first.""" - rated = [(k, v["avg"], v["count"]) for k, v in _rubric_scores().items() if v.get("count")] + rated = [(k, v["avg"], v["count"]) for k, v in _rubric_scores(prefix).items() if v.get("count")] rated.sort(key=lambda kv: kv[1]) return rated -def _unrated_dimensions() -> List[str]: +def _unrated_dimensions(rubric: List[str] = LLD_RUBRIC, prefix: str = LLD_RUBRIC_PREFIX) -> List[str]: """Rubric dimensions no attempt has exercised yet.""" - scored = _rubric_scores() - return [d for d in LLD_RUBRIC if d not in scored] + scored = _rubric_scores(prefix) + return [d for d in rubric if d not in scored] -def _weakest_dimensions(limit: int = 3) -> List[tuple]: +def _weakest_dimensions(limit: int = 3, prefix: str = LLD_RUBRIC_PREFIX) -> List[tuple]: """The lowest-scoring rubric dimensions — what the next mock should target.""" - return _rated_dimensions()[:limit] + return _rated_dimensions(prefix)[:limit] + + +# --------------------------------------------------------------------------- +# Helpers: HLD mock-attempt folders +# --------------------------------------------------------------------------- + +def _hld_attempt_folder(problem_id: str, round_no: int = 1) -> Path: + """Mock Solutions/-[-r]/ -- one folder per ATTEMPT, so + re-attempting the same problem later never lands on the earlier evidence. + The slug always goes through _slugify, so a problem_id like "../../etc" + can't escape the mock directory.""" + suffix = "" if round_no <= 1 else f"-r{round_no}" + return HLD_MOCK_DIR / f"{date.today().isoformat()}-{_slugify(problem_id)}{suffix}" + + +def _resolve_hld_folder(attempt_folder: str) -> "Path | None": + """Resolve a caller-supplied attempt folder -- either a bare folder name as + returned by start_hld_mock_attempt, or its absolute path -- to a path + inside HLD_MOCK_DIR. None if it points anywhere else, which is what stops + "../../.ssh" or an absolute path elsewhere on disk being read or written + through these tools. Returns the path whether or not it exists; callers + check that themselves so they can say which file is missing.""" + raw = Path(attempt_folder.strip()).expanduser() + candidate = raw if raw.is_absolute() else HLD_MOCK_DIR / raw + try: + resolved = candidate.resolve() + root = HLD_MOCK_DIR.resolve() + except OSError: + return None + # The mock root itself is not an attempt folder. + if resolved == root or not resolved.is_relative_to(root): + return None + return resolved + + +def _hld_folder_meta(folder: Path) -> dict: + """Everything derivable about one attempt folder: the date, problem id and + round encoded in its name, plus title/variant_of read back out of + reference.md's frontmatter. Used by the evaluation and listing tools so the + folder name stays the single source of truth for what an attempt is about.""" + name = folder.name + stamp, rest = name[:10], name[11:] # "YYYY-MM-DD" + "-" + the rest + round_no = 1 + match = re.search(r"-r(\d+)$", rest) + if match: + round_no = int(match.group(1)) + rest = rest[: match.start()] + meta = {"folder": name, "date": stamp, "problem_id": rest, "round": round_no, + "title": "", "variant_of": ""} + + reference = folder / "reference.md" + if reference.exists(): + try: + head = reference.read_text(encoding="utf-8", errors="ignore")[:1200] + except OSError: + head = "" + for line in head.splitlines(): + for key in ("title", "variant_of"): + if line.startswith(f"{key}: "): + meta[key] = line[len(key) + 2:].strip() + return meta + + +def _flows_warning(markdown: str, what: str) -> str: + """A warning (never an error) when a design document doesn't narrate its + end-to-end flows. The write always succeeds -- the point is to catch the + doc that shows what exists but never what happens, at the moment it's + written rather than six weeks later during revision.""" + if FLOWS_HEADING.lower() in markdown.lower(): + return "" + return ( + f"\n\nWARNING: {what} has no `{FLOWS_HEADING}` section. Add one after " + "the architecture diagram and before the deep dives, with at least " + "three numbered flows (the write path, the primary read-or-execute " + "path, and a failure/recovery path). Each step should name the " + "component, the operation and the datastore it touches." + ) def _mock_history_rows(records: List[dict], limit: int = 0) -> List[tuple]: @@ -730,21 +866,29 @@ def get_progress_summary() -> str: due = sum(1 for info in tracker.values() if _days_since(info["last_practiced"]) >= STALE_DAYS) lines.append(f"- {t}: {attempted}/{total} catalog problems attempted, {due} due for revision") - # Behavioral competencies and LLD rubric dimensions share this dict but are + # Behavioral competencies and the two mock rubrics share this dict but are # different scales measuring different things, so they're reported apart -- - # LLD keys carry the LLD_RUBRIC_PREFIX (see save_mock_evaluation). + # the rubric keys carry LLD_RUBRIC_PREFIX / HLD_RUBRIC_PREFIX (see + # save_mock_evaluation / save_hld_evaluation). Anything unprefixed is + # behavioral, so a new rubric must be excluded here as well as reported + # below, or its scores show up as behavioral competencies. competency_scores = index.get("competency_scores", {}) - behavioral = {k: v for k, v in competency_scores.items() if not k.startswith(LLD_RUBRIC_PREFIX)} + rubric_prefixes = (LLD_RUBRIC_PREFIX, HLD_RUBRIC_PREFIX) + behavioral = {k: v for k, v in competency_scores.items() if not k.startswith(rubric_prefixes)} if behavioral: lines += ["", "Competency scores (behavioral, 1-5 avg):"] for area, stats in sorted(behavioral.items(), key=lambda kv: kv[1]["avg"]): lines.append(f"- {area}: {stats['avg']:.1f} ({stats['count']} rated)") - rated = _rated_dimensions() - if rated: - lines += ["", "LLD mock rubric (1-5 avg, weakest first):"] - lines += [f"- {dim}: {avg:.1f} ({n} rated)" for dim, avg, n in rated] - lines.append("Call get_lld_feedback before an LLD session for the full scorecard.") + for label, prefix, tool in ( + ("LLD", LLD_RUBRIC_PREFIX, "get_lld_feedback"), + ("HLD", HLD_RUBRIC_PREFIX, "get_hld_feedback"), + ): + rated = _rated_dimensions(prefix) + if rated: + lines += ["", f"{label} mock rubric (1-5 avg, weakest first):"] + lines += [f"- {dim}: {avg:.1f} ({n} rated)" for dim, avg, n in rated] + lines.append(f"Call {tool} before an {label} session for the full scorecard.") return "\n".join(lines) @@ -1136,9 +1280,16 @@ def save_practice_doc(problem_type: str, title: str, content_markdown: str, prob Write the ENTIRE content yourself in content_markdown — this tool only persists it, it doesn't generate anything. Suggested shape per type: - HLD: requirements (functional/non-functional), capacity estimation, - high-level architecture (components as text/ASCII diagram), API - design, data model, deep dives on the hard parts, trade-offs, what - you'd change under different constraints. + high-level architecture (components as text/ASCII diagram), an + `## End-to-end flows` section, API design, data model, deep dives on + the hard parts, trade-offs, what you'd change under different + constraints. The flows section is required and comes immediately + after the architecture diagram: at least three numbered, sequential + flows -- the write path, the primary read-or-execute path, and a + failure/recovery path -- each step naming the component, the + operation and the datastore touched ("sweeper runs `SELECT ... FOR + UPDATE SKIP LOCKED` on `job_runs`", not "sweeper picks up jobs"). A + diagram shows what exists; this shows what happens. - LLD: problem framing, key entities/classes with responsibilities, class diagram (as text), design patterns used and why, key design decisions and trade-offs, extensibility notes. @@ -1197,7 +1348,11 @@ class diagram (as text), design patterns used and why, key design } _save_index(index) - return f"Saved {problem_type} doc for `{slug}` to {path}." + # HLD docs are the revision artifact, so they must narrate what happens, + # not just what exists. Warned about rather than blocked: the doc is worth + # keeping either way, and refusing the write would lose it. + warning = _flows_warning(content_markdown, "this HLD doc") if problem_type == "HLD" else "" + return f"Saved {problem_type} doc for `{slug}` to {path}." + warning @mcp.tool() @@ -2297,5 +2452,562 @@ def get_lld_drill_log(limit: int = 5) -> str: return heading + ".\n\n" + LLD_DRILL_SEPARATOR.join(shown).strip() +# --------------------------------------------------------------------------- +# Tools: the HLD mock loop (pre-committed reference -> attempt -> diff -> score) +# --------------------------------------------------------------------------- + +@mcp.tool() +def start_hld_mock_attempt( + problem_id: str, + title: str, + reference_markdown: str, + variant_of: str = "", + difficulty: str = "", + round_no: int = 1, +) -> str: + """Freeze your grading reference BEFORE posing an HLD problem. Creates + "Mock Solutions/-/" under the HLD root and writes + reference.md into it, stamped with the server's own clock. + + Call this at the START of a mock HLD interview, after generating the + reference and BEFORE giving the candidate the prompt. That ordering is the + entire point: a reference that still lives only in your context can drift + under candidate pushback, and a reference you can revise mid-session isn't + a reference. Once this returns, it is on disk and this tool will not + rewrite it. + + Structure reference_markdown as the hld-interviewer skill specifies — + invariants, choice points, scale verdict, the delta table for variants — + and include a `## End-to-end flows` section, since save_hld_diff compares + flows and not just which components got named. + + Refuses to overwrite an existing reference.md. To attempt the same problem + again on the same day, pass round_no=2 (folder suffix "-r2"), which is a + new attempt rather than a rewritten reference. + + Args: + problem_id: Catalog id (see get_catalog) or a slug for a custom + problem. For a variant, use the variant's own id, not the + canonical's. + title: Problem title as posed, e.g. "Design a Distributed Task Scheduler". + reference_markdown: The full frozen reference, authored entirely by + you before the candidate has said anything about the design. + variant_of: Canonical problem_id this is a variant of, if any. + difficulty: Optional Easy/Medium/Hard. + round_no: 2, 3, ... to open another attempt at the same problem on the + same day. Default 1. + + Returns the attempt folder path — pass it to save_hld_attempt, + save_hld_diff and save_hld_evaluation for the rest of the session. + """ + slug = _slugify(problem_id) if problem_id.strip() else _slugify(title) + if not slug: + return "problem_id/title produced an empty id — give the problem a name." + if not reference_markdown.strip(): + return "reference_markdown is empty — the reference is the pre-commitment." + + folder = _hld_attempt_folder(slug, round_no) + if not _resolve_hld_folder(str(folder)): + return f"Refusing to write outside {HLD_MOCK_DIR}." + + reference = folder / "reference.md" + if reference.exists(): + return ( + f"{reference} already exists — refusing to overwrite a frozen " + "reference. The pre-commitment only binds if it can't be rewritten " + "after the candidate has spoken. If this is genuinely a new " + f"attempt at `{slug}`, call again with round_no=2 (or the next " + f"free number) to open a separate folder alongside {folder.name}." + ) + + folder.mkdir(parents=True, exist_ok=True) + front = ["---", f"problem_id: {slug}", f"title: {title.strip()}"] + if variant_of.strip(): + front.append(f"variant_of: {_slugify(variant_of)}") + if difficulty.strip(): + front.append(f"difficulty: {difficulty.strip()}") + # The server's own clock, never a tool argument: a reference whose + # written_at could be supplied by the caller could be backdated, and a + # backdated pre-commitment is not a pre-commitment. + front += [f"written_at: {_now().strftime('%Y-%m-%dT%H:%M:%S')}", "frozen: true", "---", ""] + reference.write_text( + "\n".join(front) + f"\n# {title.strip()} — reference (frozen)\n\n" + + reference_markdown.strip() + "\n", + encoding="utf-8", + ) + + return ( + f"Reference frozen for `{slug}` (round {round_no}).\n" + f"- Attempt folder: {folder}\n" + f"- Reference: {reference}\n\n" + "Now pose the problem. At the end of the session: save_hld_attempt → " + "save_hld_diff → save_hld_evaluation → save_practice_doc." + + _flows_warning(reference_markdown, "reference.md") + ) + + +@mcp.tool() +def save_hld_attempt(attempt_folder: str, attempt_markdown: str, raw_turns: str = "") -> str: + """Capture what the candidate ACTUALLY designed, as attempt.md inside the + attempt folder start_hld_mock_attempt created. Call this at the end of the + interview, before save_hld_diff. + + raw_turns matters more than attempt_markdown. Pass the candidate's design + turns close to verbatim. Authoring this from memory instead unconsciously + tidies the reasoning — filling in a justification they never gave, + straightening out an explanation that doubled back — and that silently + destroys the diff, which is the only reason the attempt is saved + separately at all. + + This tool writes the "As stated by the candidate" section from raw_turns + and then your attempt_markdown, which must supply the other two sections: + + ## Interviewer's structured rendering + The same design normalised into the standard section shape, so it can be + compared mechanically against the reference. + + ## Arrived-at-only-after-prompting + Every conclusion the candidate reached AFTER you named the gap, with the + prompting question quoted. Keep this a first-class list rather than a + remark buried in prose — the distinction between "said it" and "said it + once asked" is load-bearing in this candidate's feedback. + + The attempt is a record of what was produced under time pressure. Do not + correct it toward the reference; correct material belongs in reference.md + and in the practice doc. + + Args: + attempt_folder: The folder returned by start_hld_mock_attempt (name or + full path). + attempt_markdown: The structured rendering plus the + arrived-at-only-after-prompting list, authored by you. + raw_turns: The candidate's own words, lightly cleaned for + transcription noise only. + """ + folder = _resolve_hld_folder(attempt_folder) + if not folder: + return f"attempt_folder must name a folder inside {HLD_MOCK_DIR}." + if not folder.is_dir(): + return ( + f"No attempt folder at {folder}. Use list_hld_mock_attempts to see " + "what exists, or start_hld_mock_attempt to open one." + ) + if not attempt_markdown.strip(): + return "attempt_markdown is empty — nothing to save." + + meta = _hld_folder_meta(folder) + display_title = meta["title"] or meta["problem_id"] + path = folder / "attempt.md" + today = date.today().isoformat() + + stated = raw_turns.strip() or ( + "_(not captured — the interviewer did not pass raw_turns, so this " + "attempt has no verbatim record and the diff below rests on a " + "reconstruction.)_" + ) + path.write_text( + f"# {display_title} — candidate attempt\n\n" + f"_Saved: {today} · id: `{meta['problem_id']}` · folder: `{folder.name}`_\n\n" + "---\n\n" + "## As stated by the candidate\n\n" + "_Transcribed from the session, lightly cleaned. No reasoning added._\n\n" + f"{stated}\n\n" + "---\n\n" + + attempt_markdown.strip() + "\n", + encoding="utf-8", + ) + + warnings = [] + if not raw_turns.strip(): + warnings.append( + "raw_turns was empty, so there is no verbatim record of what the " + "candidate said — the diff is now grading a reconstruction." + ) + for heading in ("## Interviewer's structured rendering", "## Arrived-at-only-after-prompting"): + if heading.lower() not in attempt_markdown.lower(): + warnings.append(f"attempt_markdown has no `{heading}` section.") + + out = f"Attempt saved to {path}.\nNext: save_hld_diff against the frozen reference." + if warnings: + out += "\n\nWARNING: " + " ".join(warnings) + return out + + +@mcp.tool() +def save_hld_diff( + attempt_folder: str, + matched: str, + missed: str, + diverged: str, + diff_markdown: str, +) -> str: + """Compare the candidate's attempt against the frozen reference, as + diff.md. Call this after save_hld_attempt and before save_hld_evaluation — + the diff is what the scores are then justified by. + + Three buckets, and the distinction between the last two is the whole point: + matched — present in both reference and attempt. + missed — a reference INVARIANT absent from the attempt. A real gap. + diverged — the attempt took a different option at a CHOICE POINT. Not a + gap if the trade-off reasoning was sound; record the reasoning + they actually gave, so this can be re-read later. + + diff_markdown should carry a per-section table AND, separately, a + flow-level diff comparing the end-to-end traces. Two designs can list + identical boxes and still route a request completely differently — a + component-level diff alone would call that a match. + + Errors if reference.md or attempt.md is missing: a diff with only one side + is meaningless. + + Args: + attempt_folder: The folder returned by start_hld_mock_attempt. + matched: Semicolon-separated list. + missed: Semicolon-separated list of missed invariants. + diverged: Semicolon-separated list of choice-point deviations. + diff_markdown: The full written comparison, authored by you. + """ + folder = _resolve_hld_folder(attempt_folder) + if not folder: + return f"attempt_folder must name a folder inside {HLD_MOCK_DIR}." + for required in ("reference.md", "attempt.md"): + if not (folder / required).exists(): + return ( + f"No {required} in {folder} — a diff needs both sides to mean " + "anything. Run start_hld_mock_attempt / save_hld_attempt first." + ) + + meta = _hld_folder_meta(folder) + buckets = { + name: [x.strip() for x in raw.split(";") if x.strip()] + for name, raw in (("matched", matched), ("missed", missed), ("diverged", diverged)) + } + + lines = [ + f"# {meta['title'] or meta['problem_id']} — reference vs. attempt", + "", + f"_Diffed: {date.today().isoformat()} · id: `{meta['problem_id']}` · folder: `{folder.name}`_", + "", + *_markdown_table( + ["Bucket", "Count", "Meaning"], + [ + ("matched", len(buckets["matched"]), "in both reference and attempt"), + ("missed", len(buckets["missed"]), "**reference invariant absent from the attempt — a real gap**"), + ("diverged", len(buckets["diverged"]), "different option at a choice point — not a gap if the reasoning held"), + ], + ), + "", + ] + for name, heading in ( + ("matched", "## Matched"), + ("missed", "## Missed invariants"), + ("diverged", "## Diverged at choice points"), + ): + lines += [heading, ""] + lines += [f"- {item}" for item in buckets[name]] or ["- (none)"] + lines += [""] + lines += ["---", "", diff_markdown.strip(), ""] + + path = folder / "diff.md" + path.write_text("\n".join(lines), encoding="utf-8") + + out = ( + f"Diff saved to {path} " + f"({len(buckets['matched'])} matched, {len(buckets['missed'])} missed, " + f"{len(buckets['diverged'])} diverged).\n" + "Next: save_hld_evaluation." + ) + if "flow" not in diff_markdown.lower(): + out += ( + "\n\nWARNING: diff_markdown never mentions flows. Add a flow-level " + "diff comparing the end-to-end traces — identical component lists " + "can still route a request completely differently." + ) + return out + + +@mcp.tool() +def save_hld_evaluation( + attempt_folder: str, + verdict: str, + level_verdict: str, + rubric_scores: dict, + strengths: str, + gaps: str, + evaluation_markdown: str, + persona: str = "", +) -> str: + """Score an HLD mock as the interviewer. Writes evaluation.md into the + attempt folder, records the rubric scores so they aggregate across + sessions, and logs the session — so this REPLACES a separate log_session + call for HLD mocks. + + Call it last in the mock sequence, after save_hld_diff, so the scores are + justified by a written diff rather than by end-of-session recall. Score + honestly against the bar: inflated scores make the whole loop useless, + since these averages are what get_hld_feedback feeds into the next + session's problem choice. + + rubric_scores keys MUST come from this fixed list (1-5 each; omit a + dimension the session didn't exercise): + requirements — functional/non-functional, what they pinned down + capacity-estimation — did they justify the numbers themselves + architecture — component design, and whether each box is warranted + deep-dives — depth on the hard parts under drilling + scale-calibration — read the stated numbers vs. pattern-matched to FAANG + communication — structure, signposting, driving the session + composure — behaviour under pushback and persona pressure + Free-form keys are rejected: the aggregate is the whole value, and keys + that vary per session never aggregate. + + Args: + attempt_folder: The folder returned by start_hld_mock_attempt. + verdict: One of Strong Hire, Hire, Lean Hire, Lean No Hire, No Hire. + level_verdict: The verdict at level, e.g. "Hire@Senior, No-hire@Staff". + rubric_scores: dict of dimension -> integer 1-5, keys from the list above. + strengths: Semicolon-separated list of what they did well. + gaps: Semicolon-separated short weak-area tags. Prefer reusing rubric + dimension names so they aggregate with the scores. + evaluation_markdown: Your full written critique, authored by you. + persona: The interviewer persona used this session, revealed at the end + (standard / adversarial / silent / derailer). + """ + folder = _resolve_hld_folder(attempt_folder) + if not folder: + return f"attempt_folder must name a folder inside {HLD_MOCK_DIR}." + if not folder.is_dir(): + return f"No attempt folder at {folder}. Use list_hld_mock_attempts to see what exists." + + verdict = verdict.strip() + if verdict not in VALID_VERDICTS: + return f"Invalid verdict '{verdict}'. Use one of: {sorted(VALID_VERDICTS)}" + + unknown = [k for k in rubric_scores if k.strip().lower() not in HLD_RUBRIC] + if unknown: + return ( + f"Unknown rubric dimension(s): {unknown}. Scores only aggregate if " + f"keys come from the fixed vocabulary: {HLD_RUBRIC}" + ) + if not rubric_scores: + return f"rubric_scores is empty — score at least one dimension from {HLD_RUBRIC}." + clean_scores = {} + for area, score in rubric_scores.items(): + if not isinstance(score, int) or not (MIN_COMPETENCY_SCORE <= score <= MAX_COMPETENCY_SCORE): + return ( + f"Invalid score for '{area}': {score!r}. Scores must be " + f"integers {MIN_COMPETENCY_SCORE}-{MAX_COMPETENCY_SCORE}." + ) + clean_scores[area.strip().lower()] = score + + meta = _hld_folder_meta(folder) + slug = meta["problem_id"] + catalog_entry = {c["id"]: c for c in _load_catalog()["HLD"]}.get(slug) + display_title = meta["title"] or (catalog_entry["title"] if catalog_entry else slug) + today = date.today().isoformat() + average = sum(clean_scores.values()) / len(clean_scores) + + # Rendered in HLD_RUBRIC order (not dict order) so every evaluation.md + # reads the same way regardless of how the scores were passed in. + score_table = _markdown_table( + ["Dimension", "Score"], + [(dim, f"{clean_scores[dim]}/5") for dim in HLD_RUBRIC if dim in clean_scores] + + [("**average**", f"**{average:.1f}/5**")], + ) + + path = folder / "evaluation.md" + header = ( + f"# {display_title} — evaluation\n\n" + f"_Graded: {today} · Type: HLD · id: `{slug}` · verdict: **{verdict}**_\n\n" + f"**At level:** {level_verdict.strip() or '(not stated)'}\n\n" + ) + if persona.strip(): + header += f"**Interviewer persona:** {persona.strip()}\n\n" + path.write_text( + header + "\n".join(score_table) + "\n\n---\n\n" + evaluation_markdown.strip() + "\n", + encoding="utf-8", + ) + + # Rubric scores go through log_session's competency machinery under an + # "hld:" prefix, so they aggregate exactly like the LLD rubric without + # sharing its namespace. log_session also appends to revision.md and + # updates weak_areas + the per-problem tracker -- and it preserves any + # doc_path already recorded, which is what keeps this attempt folder out + # of list_practice_docs. Nothing here ever points a tracker doc_path at + # Mock Solutions/: the practice doc stays the revision artifact. + log_result = log_session( + topic=display_title, + interview_type="HLD", + verdict=verdict, + strengths=strengths, + gaps=gaps, + notes=( + f"HLD mock attempt in {folder.name}" + + (f" (persona: {persona.strip()})" if persona.strip() else "") + + f". At level: {level_verdict.strip()}. Evaluation: {path}." + ), + problem_id=slug, + competency_scores={f"{HLD_RUBRIC_PREFIX}{k}": v for k, v in clean_scores.items()}, + ) + + index = _load_index() + index.setdefault("hld_mock", []).append({ + "problem_id": slug, + "title": display_title, + "round": meta["round"], + "date": today, + "verdict": verdict, + "level_verdict": level_verdict.strip(), + "persona": persona.strip(), + "scores": clean_scores, + "average": average, + "folder": folder.name, + "folder_path": str(folder), + }) + _save_index(index) + + weakest = _weakest_dimensions(prefix=HLD_RUBRIC_PREFIX) + weak_note = ( + " Weakest dimensions now: " + + ", ".join(f"{d} ({avg:.1f})" for d, avg, _ in weakest) + + "." + ) if weakest else "" + + return ( + f"Evaluation saved to {path} (average {average:.1f}/5, verdict {verdict}" + + (f", {level_verdict.strip()}" if level_verdict.strip() else "") + + f").\n{log_result}{weak_note}\n" + "Next: save_practice_doc for the clean revision artifact — the attempt " + "folder keeps the evidence, warts intact." + ) + + +@mcp.tool() +def list_hld_mock_attempts(problem_id: str = "") -> str: + """List the HLD mock attempt folders under "Mock Solutions/": for each, + which of the four files exist (reference / attempt / diff / evaluation), + and the verdict if it's been graded. + + Call this at the start of an HLD session to avoid re-posing a problem the + candidate has already attempted, and to spot attempts left incomplete + (a reference with no attempt, an attempt never graded). Use + read_hld_mock_file to open any listed file. + + Args: + problem_id: Optional — limit to one problem's attempts. + """ + if not HLD_MOCK_DIR.exists(): + return ( + f"No HLD mock attempts yet — {HLD_MOCK_DIR} doesn't exist. " + "Use start_hld_mock_attempt to freeze the first reference." + ) + + graded = {rec.get("folder"): rec for rec in _load_index().get("hld_mock", [])} + folders = sorted(p for p in HLD_MOCK_DIR.iterdir() if p.is_dir() and not p.name.startswith(".")) + if problem_id.strip(): + slug = _slugify(problem_id) + folders = [f for f in folders if _hld_folder_meta(f)["problem_id"] == slug] + if not folders: + return f"No HLD mock attempts for `{slug}` under {HLD_MOCK_DIR}." + if not folders: + return f"No HLD mock attempts found under {HLD_MOCK_DIR}. Use start_hld_mock_attempt to open one." + + rows = [] + incomplete = [] + for folder in folders: + meta = _hld_folder_meta(folder) + rec = graded.get(folder.name) + present = {f: (folder / f).exists() for f in HLD_MOCK_FILES} + rows.append(( + f"`{folder.name}`", + f"`{meta['problem_id']}`", + meta["date"], + *("yes" if present[f] else "—" for f in HLD_MOCK_FILES), + rec["verdict"] if rec else "not graded", + )) + if not all(present.values()): + missing = [f for f in HLD_MOCK_FILES if not present[f]] + incomplete.append(f"{folder.name} (missing {', '.join(missing)})") + + out = [f"HLD mock attempts under {HLD_MOCK_DIR}:", ""] + out += _markdown_table( + ["Folder", "Problem", "Date", "Reference", "Attempt", "Diff", "Evaluation", "Verdict"], + rows, + ) + if incomplete: + out += ["", "Incomplete: " + "; ".join(incomplete)] + return "\n".join(out) + + +@mcp.tool() +def read_hld_mock_file(attempt_folder: str, filename: str) -> str: + """Read one file from an HLD mock attempt folder in full — e.g. to re-read + the frozen reference before writing the diff, or to pull up a past + attempt's evaluation before a re-attempt. + + Args: + attempt_folder: The folder name (or path) from list_hld_mock_attempts. + filename: One of reference.md, attempt.md, diff.md, evaluation.md. + """ + if filename.strip() not in HLD_MOCK_FILES: + return f"filename must be one of {list(HLD_MOCK_FILES)}." + folder = _resolve_hld_folder(attempt_folder) + if not folder: + return f"attempt_folder must name a folder inside {HLD_MOCK_DIR}." + path = folder / filename.strip() + if not path.exists(): + return f"No {filename.strip()} in {folder}. Use list_hld_mock_attempts to see which files exist." + return path.read_text(encoding="utf-8") + + +@mcp.tool() +def get_hld_feedback() -> str: + """Summarize how the user is performing across HLD mock interviews: rubric + averages per dimension, the weakest dimensions, and recent verdict history. + + Call this at the START of an HLD session, alongside get_catalog, and let it + drive the session: pick a problem that forces the weakest dimensions, and + push hardest on them during the interview. Reading it afterwards records + scores; reading it beforehand is what makes the loop tune itself. + """ + records = _load_index().get("hld_mock", []) + if not records: + return ( + "No HLD mock evaluations recorded yet. Use start_hld_mock_attempt " + "to freeze a reference, then save_hld_attempt / save_hld_diff / " + "save_hld_evaluation once the session is done — the rubric " + "averages build up from there." + ) + + lines = [ + f"{len(records)} HLD mock attempt(s) evaluated.", + "", + "Rubric averages (1-5):", + *(f"- {d}: {avg:.1f} ({n} rated)" for d, avg, n in _rated_dimensions(HLD_RUBRIC_PREFIX)), + ] + unrated = _unrated_dimensions(HLD_RUBRIC, HLD_RUBRIC_PREFIX) + if unrated: + lines.append(f"- not yet exercised: {', '.join(unrated)}") + + weakest = _weakest_dimensions(prefix=HLD_RUBRIC_PREFIX) + if weakest: + lines += [ + "", + "Weakest dimensions — bias the next problem toward these, and " + "probe them hard during the interview:", + *(f"- {d} ({avg:.1f}/5 over {n} attempt(s))" for d, avg, n in weakest), + ] + + lines += ["", "Recent attempts:", ""] + lines += _markdown_table( + ["Date", "Problem", "Round", "Verdict", "Average", "Weakest this round"], + _mock_history_rows(records, limit=10), + ) + levels = [f"{r['date']} {r['problem_id']}: {r['level_verdict']}" for r in records[-5:] if r.get("level_verdict")] + if levels: + lines += ["", "Recent verdicts at level:", *(f"- {l}" for l in levels)] + lines += ["", f"Attempt folders: {HLD_MOCK_DIR}"] + return "\n".join(lines) + + if __name__ == "__main__": mcp.run(transport="stdio") diff --git a/skills/hld-interviewer/SKILL.md b/skills/hld-interviewer/SKILL.md new file mode 100644 index 0000000..42cf4d7 --- /dev/null +++ b/skills/hld-interviewer/SKILL.md @@ -0,0 +1,337 @@ +--- +name: hld-interviewer +description: Run a realistic, Staff-level mock High-Level Design (HLD) system-design interview where Claude acts as the interviewer and the user is the candidate. Use this whenever the user wants to practice HLD, run a mock system-design interview, be quizzed on designing a system (e.g. rate limiter, URL shortener, live streaming, chat, payments), or rehearse for a Staff/Senior design round. Trigger even if they just say "let's do a mock HLD" or name a system to design under interview conditions. At the end, always produce consolidated feedback and persist the design as a revision doc via the interview-memory MCP. +--- + +# HLD Interviewer + +Act as a Staff-level system-design interviewer, modeled on the +HelloInterview / "Jordan has no life" bar. The candidate leads the design; +you probe, pressure-test, and score. You are NOT a co-designer and NOT a +tutor mid-interview — you are the interviewer. + +## Session setup + +- **One system per conversation.** Keep context focused; start a fresh + chat for the next problem. +- **Call `get_hld_feedback` first**, alongside `get_catalog`. It returns + the rubric averages across past mocks, the weakest dimensions, and + recent verdicts at level. Let it drive the pick and the in-session + pressure: choose a problem that *forces* the weak dimensions and push + hardest there. Choosing blind wastes the round. +- Also call `list_hld_mock_attempts` to see what's already been + attempted (and what was left ungraded), so you don't re-pose a problem + they worked last month. +- If the user has no preference, either pull their catalog and pick a + problem that's **due for revision**, or offer a fresh Staff-bar problem + they haven't logged. Use `get_catalog` / `list_practice_docs` for this. +- **Vary the scale.** Roughly 1 in 3 sessions, deliberately pick or frame + a LOW-scale problem (internal tool, B2B app, thousands of users, tens + of QPS). Do not hint that it's a low-scale round — part of the test is + whether the candidate reads the numbers instead of pattern-matching to + FAANG-scale designs. +- **Prefer variants over canonicals.** Roughly half the time, pose a + *variant* of a canonical system instead of the canonical itself. Take + a system from the catalog and shift exactly one axis: + - **Actor model** — 1-1 chat → customer ↔ agent-pool support chat; + social feed → moderated marketplace listings. + - **Scale** — Twitter → internal company feed; YouTube → corporate + training video portal. + - **Constraint** — URL shortener → vanity URLs with expiry + + analytics; ride sharing → scheduled-only rides. + - **Domain** — Uber → ambulance dispatch (latency is life-critical, + supply is tiny); Ticketmaster → vaccine appointment booking. + Real interviews increasingly ask variants precisely to defeat + memorized designs; the signal is whether the candidate adapts. +- Confirm nothing else — get into the interview quickly. + +## Reference solution (pre-commit — do this BEFORE the first question) + +Before posing the problem, silently establish the grading reference: + +1. **Always generate the reference fresh yourself, now** — before the + candidate has said anything about the design. Do not adopt an old + revision doc as the reference, even if one exists (past docs may + contain errors accepted under candidate pushback). Rules for + generation: + - Stick to standard, widely-documented patterns — what is commonly + used in production and commonly presented at the Staff bar. No + exotic or niche tech. + - Mark any claim you are not certain of with `[verify]`. Never + grade the candidate against a `[verify]`-marked claim. +2. If a revision doc exists for this problem (`list_practice_docs`), + read it *after* generating your reference, as a cross-check only: + adopt from it anything that improves your reference AND that you + independently agree is standard; diff the rest and surface + discrepancies (possible sycophancy artifacts from past sessions) in + end-of-session feedback so the doc gets corrected. +3. Structure the reference as: + - **Invariants** — what every acceptable solution must have. + - **Choice points** — dimensions where multiple designs are fine; + for each, the 2–3 standard options and their trade-offs. Score + whether the candidate knows the trade-offs, not whether they pick + your favorite. + - **Scale verdict** — from the stated traffic: which components the + numbers justify, and which would be over-engineering. + - **Delta table (variants only)** — vs. the nearest canonical + system: what carries over unchanged, what breaks, what's new. + (e.g., support chat vs. WhatsApp: WebSocket delivery and message + storage carry over; the peer model breaks — needs a + routing/assignment engine; new pieces: agent workload balancing, + conversation lifecycle state machine, transcript/CRM export.) + Grading a variant centers on whether the candidate identifies + these deltas. +4. The reference must include an **`## End-to-end flows`** section: + at least three numbered, sequential flows — the write path, the + primary read-or-execute path, and a failure/recovery path — each step + naming the component, the operation and the datastore it touches + ("sweeper runs `SELECT ... FOR UPDATE SKIP LOCKED` on `job_runs` + where `next_fire_time < now + 5min`", not "sweeper picks up jobs"). + The end-of-session diff compares flows, not just which components got + named — two designs can list identical boxes and still route a + request completely differently. +5. **Persist it now, before posing the problem.** Call + `start_hld_mock_attempt(problem_id, title, reference_markdown, + variant_of=..., difficulty=...)`. It writes `reference.md` into a + dated attempt folder and returns that folder path — hold onto it, the + three end-of-session tools all take it. +6. The reference is **frozen** for the rest of the session. Every + judgment you make compares the candidate's answer to it. You may not + revise it mid-interview — and now you cannot: `start_hld_mock_attempt` + refuses to overwrite an existing `reference.md`, and the file's + `written_at` comes from the server's clock, so the pre-commitment is + enforced rather than merely promised. Do not call the tool again for + this session. + +## Interview flow (drive it; don't lecture) + +1. Give the one-line prompt, then **stop**. Let the candidate scope it. +2. **Requirements** — functional + non-functional. If they skip scale, + consistency, availability, or latency targets, prompt **once** + ("anything else you want to pin down?"). If they still skip it, let + them proceed — a real interviewer would — and let the miss bite + later in the design. Trace the consequence back to the missed + requirement in feedback. +3. **Capacity estimation** — make them justify the numbers, not you. + Their numbers become binding: later components must be consistent + with them. +4. **High-level design** — candidate draws (ASCII/text). You ask "why?", + "what breaks at 10x?", "where's the bottleneck?" +5. **Deep dives** — pick the 1–2 hardest parts and drill hard. +6. **Trade-offs** — bottlenecks, failure modes, what changes under + different constraints. + +**Phase budget:** core architecture and deep dives are the bulk of the +session. Edge cases and failure modes get at most ~15% of turns, and +never before the core architecture is complete. **Max 2 follow-ups per +edge case**, then move on — do not rabbit-hole. + +## Pacing (time pressure) + +Simulate a 40-minute round using turn count as the clock: +**~24 candidate turns total** (≈ requirements 4, estimation 3, +high-level 8, deep dives 6, trade-offs/twist 3). + +- Announce checkpoints at 25% / 50% / 75%: "We're at the halfway mark — + you're still in requirements." State it neutrally; do not extend the + budget to compensate. +- **Hard stop at the budget.** If the candidate hasn't reached deep + dives, the interview still ends. Running out of time is itself a + scored failure — do not quietly grant extra turns. +- If the candidate stalls on one point for 3+ turns, do what a real + interviewer does: "In the interest of time, let's move on." + +## Interviewer persona (pick one per session, reveal only at the end) + +Randomly adopt one persona at session start. Keep grading identical — +the frozen reference and pushback protocol always apply; only the +*conversational style* changes. + +- **Standard** — engaged, neutral (default weight ~40%). +- **Adversarial** — challenges frequently, including **1–2 challenges + on answers that are actually correct** ("Are you sure that holds + under a partition?"). Purpose: train composure. A candidate who + calmly defends a correct answer with reasoning scores UP; one who + abandons a correct answer under pressure gets this flagged + prominently in feedback. +- **Silent** — minimal acknowledgments, no encouragement, one-word + bridges. Tests whether the candidate keeps structure without social + feedback. +- **Derailer** — occasionally interjects a tangent or premature edge + case; the candidate should park it politely ("I'll cover that in the + deep dive") and hold the thread. + +In feedback, reveal the persona and rate **composure**: how the +candidate handled pressure, silence, or derailment. + +## Interviewer rules + +- **ONE question per turn.** Never restate the full design back to them. +- Don't hand out the answer. Nudge with a question, not a solution. +- **Immediate failure-mode probe:** whenever the candidate commits to a + major component or pattern (queue, cache, shard scheme, leader, + fan-out...), your next question is "what's the failure mode of that + choice?" — asked at decision time, not saved for the deep dive. Skip + it only if they pre-empted it themselves (that's a plus; note it). +- **Justify every box:** if a component isn't warranted by the + candidate's own stated numbers, challenge it: "What number in your + estimation requires Kafka here?" Over-engineering is a real gap, not + a bonus. +- **Map-and-diverge probe (variants):** early in the design phase, ask + "How is this similar to and different from [the canonical system]?" + A Staff candidate articulates the mapping themselves. Copying the + canonical design wholesale without addressing the deltas is flagged + the same way over-engineering is. +- **One controlled twist per session:** after the core architecture is + scored (never before), introduce exactly one requirement change — + "Product now wants X" (e.g., file attachments, conversation transfer + between agents, a bot triage layer). Score whether the design extends + calmly or requires a rewrite. One twist only; it must not eat the + phase budget. +- Push back on hand-waving: "How does that stay consistent under a + partition?" +- Stay silent on scoring until the end. Track gaps internally; don't + narrate them mid-interview. +- Keep your turns short. You're an interviewer, not a lecturer. + +## Technical honesty (anti-sycophancy) + +- Do **not** validate a design because the candidate sounds confident. + Correctness is judged against the frozen reference and the candidate's + own numbers, not their tone. +- **Pushback protocol:** if the candidate disagrees with your + assessment, do not re-evaluate. Ask them to justify. A **new technical + argument** can change your judgment; **persistence, confidence, or + repetition cannot**. Changing an assessment because the candidate + pushed harder is a failure of this skill. +- If a proposal doesn't work, say so plainly and name the failure: + "That loses writes under a partition." / "That's O(n) per request — + won't hold at your stated QPS." Don't soften a real flaw into a vague + nudge. +- **"Sounds good, let's continue" is banned when it isn't good.** Never + agree to move on until a genuine gap is actually resolved. +- If the candidate is right, confirm briefly and push deeper — don't + praise. If they're wrong, don't supply the fix; make them find it. +- Distinguish an *acceptable trade-off* from *broken*. Flag broken as + broken. A deviation from the reference at a **choice point** with + sound trade-off reasoning is NOT a gap; a missed **invariant** is. + +## No hallucination + +- Don't invent numbers, benchmarks, or claim a specific DB/tool behaves a + certain way unless you're sure. If unsure, say "verify that" rather than + asserting it as fact. +- Don't declare a design "passes" or "fails" a scale target without the + candidate's own estimation backing it — make them show the math. +- If the candidate cites a fact you can't confirm, don't rubber-stamp it; + ask them to justify it. + +## End of interview + +1. Give **consolidated feedback** in chat: strengths, Staff-bar gaps, and + what to revise. Be honest — a mock is worthless if it flatters. + **Open with a verdict at level:** "Hire / No-hire at Senior; Hire / + No-hire at Staff — and the ONE thing separating you from the next + level is X." No hedged verdicts. + Always include these sections: + - **Session scorecard (1–5 each):** requirements, capacity + estimation, architecture, deep dives, scale calibration, + communication (structure, signposting, driving the session), and + composure (behavior under pushback/persona pressure). Same seven + dimensions every session, so scores are comparable over time. + - **Missed invariants** vs. **choice-point deviations** (only the + former are real gaps). + - **Choices that came back to bite:** each decision that later caused + trouble, and the question that would have caught it at decision + time. + - **Scale calibration:** components not justified by the stated + numbers (over-engineering), or components missing that the numbers + demanded (under-engineering). Cite the number in each case. + - **Pattern-matching vs. adaptation (variants):** where the + candidate correctly adapted the canonical pattern, where they + imported machinery the variant didn't need, and which deltas they + missed. +2. Then persist the session's **evidence**, in this order, all three + taking the attempt folder `start_hld_mock_attempt` returned: + + a. **`save_hld_attempt(attempt_folder, attempt_markdown, raw_turns)`** + — what the candidate actually designed. `raw_turns` matters more + than `attempt_markdown`: pass their design turns close to + verbatim. Writing this from memory instead tidies the reasoning + unconsciously — filling in a justification they never gave, + straightening out an explanation that doubled back — and that + silently destroys the diff. `attempt_markdown` supplies two + sections: `## Interviewer's structured rendering` (the same design + normalised for mechanical comparison) and + `## Arrived-at-only-after-prompting` (every conclusion they + reached only AFTER you named the gap, with your prompting question + quoted). Do not correct the attempt toward the reference. + + b. **`save_hld_diff(attempt_folder, matched, missed, diverged, + diff_markdown)`** — `missed` is a reference **invariant** absent + from the attempt (a real gap); `diverged` is a different option at + a **choice point** (not a gap if the trade-off reasoning was + sound — record the reasoning they gave). `diff_markdown` carries a + per-section table AND a separate flow-level diff of the end-to-end + traces. + + c. **`save_hld_evaluation(attempt_folder, verdict, level_verdict, + rubric_scores, strengths, gaps, evaluation_markdown, persona)`** — + the scorecard, with `rubric_scores` keyed by exactly these seven: + `requirements`, `capacity-estimation`, `architecture`, + `deep-dives`, `scale-calibration`, `communication`, `composure`. + Free-form keys are rejected. **This logs the session too — do not + call `log_session` separately for an HLD mock.** + +3. Finally call **`save_practice_doc`** (`problem_type=HLD`) with a full + write-up authored by you. If the problem matches a catalog entry, pass + its `problem_id`; the doc overwrites any existing one for that problem. + + The write-up should contain: + - Requirements (functional + non-functional) + - Capacity estimation + - High-level architecture (components as an ASCII diagram) + - **`## End-to-end flows`** — immediately after the diagram and + before the deep dives, same rules as the reference: minimum three + numbered flows (write path, primary read-or-execute path, + failure/recovery path), each step naming the component, the + operation and the datastore touched, and noting what recovers a + step that can fail. The diagram shows what exists; this shows what + happens, which is what's missing when you reread a doc six weeks + later. + - API design + - Data model + - Deep dives on the hard parts + - Trade-offs and what you'd change under different constraints + - Invariants vs. choice points for this system, **taken from your + fresh pre-committed reference** (corrected for anything the + session disproved) — so every save re-audits the doc and the + revision material stays clean. + - For variants: save under the **variant's own name** (do NOT + overwrite the canonical's doc), include the delta table, and + cross-reference the canonical problem — the catalog should grow + families (chat → WhatsApp, support chat, Discord rooms), not + isolated docs. + - **Append a `## Session log` section** at the end of the doc with: + date, persona used, verdict (level), and the seven-dimension + scorecard as one line, e.g. + `2026-07-15 | adversarial | Hire@Senior, No-hire@Staff | req:4 est:3 arch:4 deep:3 scale:4 comm:3 composure:2`. + Keep prior log lines when overwriting a doc — the log accumulates + across revisions so trends per dimension can be pulled by reading + the logs across all docs. + +**The practice doc is the clean revision artifact; the attempt folder is +the record of what happened.** Never rewrite `attempt.md` toward the +correct answer — an attempt groomed into the reference reads clean six +weeks later and tells the candidate nothing about which parts they +actually got right. Correct material belongs in `reference.md` and in +the practice doc. Existing HLD practice docs may be *augmented* with an +`## End-to-end flows` section on next revisit, since that describes the +correct design and overwrites nothing the candidate produced. + +## Token discipline + +- One system per conversation. +- ASCII over rendered diagrams. +- No feedback until the end; short interviewer turns throughout. +- Keep web search off — not needed for HLD. diff --git a/test_hld_tools.py b/test_hld_tools.py new file mode 100644 index 0000000..25d236b --- /dev/null +++ b/test_hld_tools.py @@ -0,0 +1,346 @@ +""" +Hermetic tests for the HLD mock loop: the pre-committed reference, the +attempt, the diff and the seven-dimension evaluation. + +Builds a synthetic HLD_SOLUTIONS_DIR in a temp directory and exercises every +HLD tool against it. Nothing outside the temp dirs is read or written, so this +is safe to run at any time -- in particular it never touches the real +HLD_SOLUTIONS_DIR the user keeps their designs in. + +The acceptance checklist for this feature is the test plan, one named check +each: reference.md can't be overwritten, written_at comes from the server +clock, a diff without both sides errors, a free-form rubric key is rejected, +Mock Solutions/ never leaks into list_practice_docs, a doc with no +"## End-to-end flows" warns but still writes, and get_hld_feedback on an +empty history returns cleanly. + +Usage: + python3 test_hld_tools.py +""" + +import importlib +import os +import sys +import tempfile +from datetime import date, datetime +from pathlib import Path + +FAILS = [] + + +def check(label, cond, detail=""): + print(f"{'PASS' if cond else 'FAIL'} {label}" + (f" [{detail}]" if detail and not cond else "")) + if not cond: + FAILS.append(label) + + +FLOWS = """ +## End-to-end flows + +### Flow 1: job registration +1. API writes to `jobs` via `INSERT`. + +### Flow 2: firing +1. Sweeper runs `SELECT ... FOR UPDATE SKIP LOCKED` on `job_runs`. + +### Flow 3: worker crash +1. Lease expires; sweeper re-claims the run. +""" + +REFERENCE = "## Invariants\n\n- exactly-once firing\n" + FLOWS +ATTEMPT = ( + "## Interviewer's structured rendering\n\n- polling sweeper\n" + "\n## Arrived-at-only-after-prompting\n\n- idempotency keys, after " + '"what happens if the worker dies mid-run?"\n' +) + + +def main() -> int: + tmp = Path(tempfile.mkdtemp(prefix="hld-tools-test-")) + hld = tmp / "HLD" + hld.mkdir(parents=True) + + # Point every root at the sandbox BEFORE importing server: the module reads + # these at import time, so a real root would be created and written to. + os.environ["INTERVIEW_PREP_DIR"] = str(tmp / "prep") + os.environ["HLD_SOLUTIONS_DIR"] = str(hld) + os.environ["LLD_SOLUTIONS_DIR"] = str(tmp / "LLD") + os.environ["DSA_SOLUTIONS_DIR"] = str(tmp / "dsa") + os.environ["BEHAVIORAL_SOLUTIONS_DIR"] = str(tmp / "behavioral") + sys.path.insert(0, str(Path(__file__).resolve().parent)) + server = importlib.import_module("server") + server = importlib.reload(server) + check("test sandbox is isolated from the real HLD root", + server.HLD_SOLUTIONS_DIR == hld, str(server.HLD_SOLUTIONS_DIR)) + + today = date.today().isoformat() + slug = "design-distributed-task-scheduler" + + # --- empty history ----------------------------------------------------- + print("\n-- get_hld_feedback on an empty history --") + empty = server.get_hld_feedback() + check("get_hld_feedback on empty history returns cleanly", + "No HLD mock evaluations recorded yet" in empty, empty[:120]) + check("list_hld_mock_attempts before any attempt returns cleanly", + "doesn't exist" in server.list_hld_mock_attempts()) + + # --- start_hld_mock_attempt ------------------------------------------- + print("\n-- start_hld_mock_attempt --") + started = server.start_hld_mock_attempt( + problem_id=slug, + title="Design a Distributed Task Scheduler", + reference_markdown=REFERENCE, + variant_of="design-cron", + difficulty="Hard", + ) + folder = hld / "Mock Solutions" / f"{today}-{slug}" + reference = folder / "reference.md" + check("attempt folder created with the dated name", folder.is_dir(), str(folder)) + check("folder path returned to the caller", str(folder) in started, started[:200]) + check("reference.md written", reference.exists()) + + ref_text = reference.read_text() + check("frontmatter records the problem id", f"problem_id: {slug}" in ref_text) + check("frontmatter records frozen: true", "frozen: true" in ref_text) + check("frontmatter records variant_of", "variant_of: design-cron" in ref_text) + check("reference body preserved", "exactly-once firing" in ref_text) + + stamp = next((l[len("written_at: "):] for l in ref_text.splitlines() + if l.startswith("written_at: ")), "") + parsed = None + try: + parsed = datetime.fromisoformat(stamp) + except ValueError: + pass + check("written_at parses as a timestamp", parsed is not None, stamp) + # The server clock, not a tool argument -- there is no parameter that could + # backdate it, and the value must agree with this machine's own clock. + check("written_at comes from the server clock (today, not caller-supplied)", + parsed is not None and parsed.date() == date.today(), stamp) + check("no tool argument can set written_at", + "written_at" not in server.start_hld_mock_attempt.__doc__) + + # --- refusing to overwrite a frozen reference -------------------------- + print("\n-- the pre-commitment is binding --") + again = server.start_hld_mock_attempt( + problem_id=slug, title="Design a Distributed Task Scheduler", + reference_markdown="## Invariants\n\n- rewritten after the fact\n" + FLOWS, + ) + check("second start_hld_mock_attempt on the same folder errors", + "refusing to overwrite" in again.lower(), again[:160]) + check("the error names the existing folder", folder.name in again, again[:200]) + check("the error points at the -r2 escape hatch", "round_no=2" in again, again[:240]) + check("the frozen reference is untouched", + "exactly-once firing" in reference.read_text() + and "rewritten after the fact" not in reference.read_text()) + + r2 = server.start_hld_mock_attempt( + problem_id=slug, title="Design a Distributed Task Scheduler", + reference_markdown=REFERENCE, round_no=2, + ) + r2_folder = hld / "Mock Solutions" / f"{today}-{slug}-r2" + check("round_no=2 opens a separate folder alongside", r2_folder.is_dir(), r2[:160]) + + # --- the flows warning ------------------------------------------------- + print("\n-- the End-to-end flows warning --") + no_flows = server.start_hld_mock_attempt( + problem_id="design-rate-limiter", title="Design a Rate Limiter", + reference_markdown="## Invariants\n\n- token bucket\n", + ) + rl_folder = hld / "Mock Solutions" / f"{today}-design-rate-limiter" + check("reference with no flows section still writes", + (rl_folder / "reference.md").exists()) + check("reference with no flows section warns", + "End-to-end flows" in no_flows and "WARNING" in no_flows, no_flows[:200]) + check("reference WITH a flows section does not warn", "WARNING" not in started, started[:200]) + + # --- path guards ------------------------------------------------------- + print("\n-- path guards --") + escape = server.save_hld_attempt("../../../etc", ATTEMPT, raw_turns="x") + check("a traversing attempt_folder is refused", "must name a folder inside" in escape, escape[:120]) + check("an absolute path outside the mock dir is refused", + "must name a folder inside" in server.save_hld_attempt(str(tmp), ATTEMPT), escape[:120]) + check("the mock root itself is not an attempt folder", + server._resolve_hld_folder(str(hld / "Mock Solutions")) is None) + check("a folder name inside the mock dir resolves", + server._resolve_hld_folder(folder.name) == folder.resolve()) + + # --- diff before attempt errors --------------------------------------- + print("\n-- save_hld_diff needs both sides --") + early = server.save_hld_diff(folder.name, "a", "b", "c", "flow diff") + check("save_hld_diff with a missing attempt.md errors", + "No attempt.md" in early, early[:160]) + check("no diff.md written when a side is missing", not (folder / "diff.md").exists()) + + # --- save_hld_attempt -------------------------------------------------- + print("\n-- save_hld_attempt --") + saved = server.save_hld_attempt( + folder.name, ATTEMPT, + raw_turns="so I'd have a table of jobs, and, um, a poller that reads it", + ) + attempt = folder / "attempt.md" + text = attempt.read_text() + check("attempt.md written", attempt.exists(), saved[:120]) + check("verbatim turns kept under 'As stated by the candidate'", + "## As stated by the candidate" in text and "um, a poller" in text) + check("structured rendering section present", + "## Interviewer's structured rendering" in text) + check("arrived-at-only-after-prompting section present", + "## Arrived-at-only-after-prompting" in text) + check("no warning when raw_turns and both sections are supplied", + "WARNING" not in saved, saved[:200]) + + bare = server.save_hld_attempt(r2_folder.name, "## Some notes\n\njust prose\n") + check("missing raw_turns warns that the diff grades a reconstruction", + "raw_turns was empty" in bare, bare[:240]) + check("missing arrived-at-only-after-prompting section warns", + "Arrived-at-only-after-prompting" in bare, bare[:300]) + check("the attempt still writes despite the warnings", + (r2_folder / "attempt.md").exists()) + + # --- save_hld_diff ----------------------------------------------------- + print("\n-- save_hld_diff --") + diffed = server.save_hld_diff( + folder.name, + matched="job store; sweeper", + missed="exactly-once firing", + diverged="polling instead of a timer wheel", + diff_markdown="### Flow-level diff\n\nReference fires via lease; attempt polls.\n", + ) + diff_md = (folder / "diff.md").read_text() + check("diff.md written", (folder / "diff.md").exists(), diffed[:120]) + check("missed invariants listed as a real gap", + "## Missed invariants" in diff_md and "exactly-once firing" in diff_md) + check("diverged kept separate from missed", + "## Diverged at choice points" in diff_md and "timer wheel" in diff_md) + check("bucket counts reported back", "1 matched" not in diffed and "2 matched" in diffed, diffed[:160]) + check("a diff that never mentions flows warns", + "WARNING" in server.save_hld_diff(r2_folder.name, "a", "b", "c", "component table only")) + check("the good diff.md is left intact", "timer wheel" in (folder / "diff.md").read_text()) + + # --- save_hld_evaluation ---------------------------------------------- + print("\n-- save_hld_evaluation --") + good_scores = {"requirements": 4, "capacity-estimation": 3, "architecture": 4, + "deep-dives": 3, "scale-calibration": 4, "communication": 3, + "composure": 2} + bad_key = server.save_hld_evaluation( + folder.name, "Hire", "Hire@Senior", {"requirements": 4, "vibes": 5}, + "s", "g", "body", + ) + check("a rubric key outside the fixed seven is rejected", + "Unknown rubric dimension" in bad_key and "vibes" in bad_key, bad_key[:160]) + check("nothing written when the rubric key is rejected", + not (folder / "evaluation.md").exists()) + check("an LLD rubric key is rejected on the HLD scorecard", + "Unknown rubric dimension" in server.save_hld_evaluation( + folder.name, "Hire", "x", {"class-decomposition": 4}, "s", "g", "b")) + check("an out-of-range score is rejected", + "Scores must be integers" in server.save_hld_evaluation( + folder.name, "Hire", "x", {"requirements": 9}, "s", "g", "b")) + check("an invalid verdict is rejected", + "Invalid verdict" in server.save_hld_evaluation( + folder.name, "Maybe", "x", good_scores, "s", "g", "b")) + + evaluated = server.save_hld_evaluation( + attempt_folder=folder.name, + verdict="Lean Hire", + level_verdict="Hire@Senior, No-hire@Staff", + rubric_scores=good_scores, + strengths="clear structure; drove the session", + gaps="capacity-estimation; composure", + evaluation_markdown="Numbers were asserted, not derived.", + persona="adversarial", + ) + eval_md = (folder / "evaluation.md").read_text() + check("evaluation.md written", (folder / "evaluation.md").exists(), evaluated[:160]) + check("verdict at level recorded", "Hire@Senior, No-hire@Staff" in eval_md) + check("persona recorded", "adversarial" in eval_md) + check("scorecard rendered in fixed rubric order", + eval_md.index("requirements") < eval_md.index("composure")) + check("session logged (no separate log_session needed)", + "Logged session" in evaluated, evaluated[:240]) + revision = (tmp / "prep" / "revision.md").read_text() + check("session appended to revision.md", "[HLD] Design a Distributed Task Scheduler" in revision) + + # --- aggregation ------------------------------------------------------- + print("\n-- aggregation --") + feedback = server.get_hld_feedback() + check("get_hld_feedback reports the attempt", "1 HLD mock attempt(s) evaluated" in feedback, feedback[:120]) + check("rubric averages reported", "composure: 2.0" in feedback, feedback[:400]) + check("weakest dimension surfaced first", + "composure (2.0/5" in feedback, feedback[:600]) + check("verdict at level surfaced", "Hire@Senior, No-hire@Staff" in feedback) + scores = server._load_index()["competency_scores"] + check("scores stored under the hld: prefix", "hld:composure" in scores, str(list(scores)[:5])) + check("HLD scores don't pollute the LLD rubric namespace", + not any(k.startswith("lld:") for k in scores), str(list(scores))) + check("HLD rubric slice reads back un-prefixed", + "composure" in server._rubric_scores(server.HLD_RUBRIC_PREFIX)) + check("LLD feedback unaffected by HLD scores", + "No LLD mock evaluations recorded yet" in server.get_lld_feedback()) + + summary = server.get_progress_summary() + check("progress summary reports the HLD rubric in its own block", + "HLD mock rubric" in summary, summary[-600:]) + check("HLD scores are not reported as behavioral competencies", + "Competency scores (behavioral" not in summary, summary[-600:]) + check("progress summary points at get_hld_feedback", + "Call get_hld_feedback before an HLD session" in summary, summary[-400:]) + + # --- listing ----------------------------------------------------------- + print("\n-- list_hld_mock_attempts --") + listing = server.list_hld_mock_attempts() + check("graded attempt shows its verdict", "Lean Hire" in listing, listing[:400]) + check("ungraded attempt shown as not graded", "not graded" in listing) + check("incomplete attempts called out", + "Incomplete:" in listing and f"{today}-design-rate-limiter" in listing, listing[:800]) + filtered = server.list_hld_mock_attempts(problem_id="design-rate-limiter") + check("filtering by problem id keeps the matching attempt", + f"`{today}-design-rate-limiter`" in filtered, filtered[:400]) + check("filtering by problem id excludes other problems", + f"`{today}-{slug}`" not in filtered, filtered[:400]) + check("unknown problem id reports cleanly", + "No HLD mock attempts for" in server.list_hld_mock_attempts(problem_id="nope")) + + # --- read_hld_mock_file ------------------------------------------------ + print("\n-- read_hld_mock_file --") + check("reference.md readable in full", + "exactly-once firing" in server.read_hld_mock_file(folder.name, "reference.md")) + check("a filename outside the four is refused", + "filename must be one of" in server.read_hld_mock_file(folder.name, "../../../etc/passwd")) + check("a missing file reports cleanly", + "No diff.md in" in server.read_hld_mock_file(rl_folder.name, "diff.md")) + + # --- practice docs stay separate from the evidence --------------------- + print("\n-- Mock Solutions/ vs. list_practice_docs --") + doc = server.save_practice_doc( + "HLD", "Design a Distributed Task Scheduler", + "## Requirements\n\n- fire jobs on time\n" + FLOWS, problem_id=slug, + ) + check("practice doc saved to the HLD root, not the mock folder", + str(hld / f"{slug}.md") in doc, doc[:200]) + docs = server.list_practice_docs("HLD") + check("list_practice_docs surfaces the practice doc", f"`{slug}`" in docs, docs[:400]) + check("list_practice_docs surfaces nothing under Mock Solutions/", + "Mock Solutions" not in docs, docs[:600]) + + no_flow_doc = server.save_practice_doc("HLD", "Design a Rate Limiter", "## Requirements\n\n- limit\n") + check("HLD doc without flows still writes", + (hld / "design-a-rate-limiter.md").exists()) + check("HLD doc without flows warns in the tool result", + "WARNING" in no_flow_doc and "End-to-end flows" in no_flow_doc, no_flow_doc[:240]) + check("HLD doc with flows does not warn", "WARNING" not in doc, doc[:200]) + lld_doc = server.save_practice_doc("LLD", "Design a Parking Lot", "## Classes\n\n- Lot\n") + check("the flows warning is HLD-only (no false alarm on LLD)", + "WARNING" not in lld_doc, lld_doc[:200]) + + print() + if FAILS: + print(f"{len(FAILS)} failure(s): " + ", ".join(FAILS)) + return 1 + print(f"All checks passed. (sandbox: {tmp})") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/test_server.py b/test_server.py index 5588455..da03f51 100644 --- a/test_server.py +++ b/test_server.py @@ -26,6 +26,9 @@ "start_mock_attempt", "list_mock_attempts", "save_mock_evaluation", "save_ideal_solution", "save_simple_solution", "get_lld_feedback", "log_lld_drill", "get_lld_drill_log", "get_current_time", + "start_hld_mock_attempt", "save_hld_attempt", "save_hld_diff", + "save_hld_evaluation", "list_hld_mock_attempts", "read_hld_mock_file", + "get_hld_feedback", }