feat(experiments): hfir, an assembly-like IR agents write instead of HTML

experiments/hfir/README.md

@@ -0,0 +1,91 @@
+# hfir — assembly for programmatic video
+
+**Status: experiment. Not shipped, not supported, not on any roadmap. It exists to answer one question with evidence instead of opinion.**
+
+## The question
+
+Every code-to-video framework, ours included, is designed to be intuitive to a human. But increasingly no human reads the source. Agents write it, and agents want the opposite of what humans want: not terseness and convention, but explicitness and dials.
+
+So: **what if an agent never wrote HTML at all?**
+
+## What this is
+
+A flat table of `(op, frame, element, property, value)` that compiles to a renderable composition. This is a complete scene:
+
+```
+CANVAS  1280 720 30
+FRAMES  60
+
+DECL    box   rect
+DECL    tag   text
+
+SET   0   box  x 120
+SET   0   box  y 260
+SET   0   box  w 200
+SET   0   box  h 200
+SET   0   box  fill #E0322C
+SET   0   box  opacity 0
+
+RAMP  15  box  opacity 1
+RAMP  45  box  x 960
+RAMP  45  box  fill #0F7A52
+
+SET   30  tag  text FRAME_30_EXACTLY
+```
+
+There is no HTML, no CSS, no cascade, no box model, no layout engine, and no easing vocabulary.
+
+## The one design decision that matters
+
+`SET` and `RAMP` are different opcodes.
+
+- `SET` is a hard step. The value holds until the next op.
+- `RAMP` interpolates linearly into the stated value.
+
+Interpolation is declared **per property, per segment**. There is no implicit tween, no eased default, and no `ease-out` to misinterpret. If a property moves, something said so.
+
+That is the whole "assembly" idea: the agent addresses the machine directly and nothing is inferred on its behalf.
+
+## Try it
+
+```bash
+node experiments/hfir/compile.mjs experiments/hfir/scene.hfir /tmp/hfir/index.html
+npx hyperframes render /tmp/hfir
+```
+
+Compiles 22 ops into 15 animations and renders 60 frames of 720p in **~2.5 seconds**.
+
+## Verified, not asserted
+
+The example scene was rendered and the frames checked against the table by eye:
+
+| Frame   | The table says  | The pixels show                                                                  |
+| ------- | --------------- | -------------------------------------------------------------------------------- |
+| 0       | `box opacity 0` | box genuinely absent, text white at exactly (120,120)                            |
+| 29 → 30 | `SET` is a step | white `NO_HTML_WAS_HARMED` on 29, blue `FRAME_30_EXACTLY` on 30, nothing between |
+| 45      | two `RAMP`s     | box at x=960, colour interpolated red → green                                    |
+
+## Why it targets WAAPI
+
+The compiler emits Web Animations API keyframes rather than CSS keyframes, because seeking a WAAPI animation by `currentTime` is the path our render engine treats as authoritative, and it is a Baseline web standard rather than a Chromium internal.
+
+The useful consequence: the IR gets browser-grade text shaping, compositing and colour **for free**, while the agent never touches any of it. The browser stays the renderer; only the authoring surface changes.
+
+That is also the argument against building a native renderer for this. The explicitness people want from "native" turns out to be a compiler problem, not an engine problem, and solving it this way keeps the installed renderer, the standards floor, and the enormous amount of HTML/CSS in every model's training data.
+
+## What it cannot do yet
+
+Named honestly, because these are the reasons it might not be worth pursuing:
+
+- **No layout.** Everything is absolute pixels. Fine for a machine, miserable for a human, and it means no responsive or flow-based composition.
+- **No text wrapping.** Though `window.__hyperframes.pretext` now measures text without a reflow, so an agent could compute wrap points and emit them as explicit ops. That is the natural next step.
+- **Text is the leak.** WAAPI cannot animate text content, so it is driven by a discrete per-frame table instead. It is the one place the mapping is not 1:1.
+- **No media, no sub-compositions, no audio.** Audio in particular could not be inferred; it would need its own ops.
+
+## The experiment worth running next
+
+This prototype only proves the idea is _possible_. It does not prove it is _better_.
+
+The test that would settle it: give an agent the IR spec and the HTML spec cold, same prompts, and measure first-valid-render rate and repair loops for each. Nobody in the category publishes that number for any format, which is its own opportunity.
+
+If the IR wins, we have found something real. If HTML wins, we have cheaply killed a good-sounding idea, which is worth just as much.

experiments/hfir/compile.mjs

@@ -0,0 +1,228 @@
+#!/usr/bin/env node
+/**
+ * hfir -> HyperFrames composition.
+ *
+ * Compiles a flat table of (op, frame, element, property, value) into a
+ * renderable composition. The agent writing the table never sees HTML, CSS, the
+ * cascade, the box model, or an easing name.
+ *
+ * Two opcodes, and the difference is the whole point:
+ *   SET   frame el prop value   hard step, holds until the next op
+ *   RAMP  frame el prop value   linear interpolation from the previous value
+ *
+ * Interpolation is stated per property per segment, so there is no implicit
+ * tween and no eased default to misread.
+ *
+ * Output targets the Web Animations API rather than CSS keyframes, because
+ * seeking a WAAPI animation by `currentTime` is the path the render engine
+ * treats as authoritative. That also means the IR inherits browser text
+ * shaping, compositing and colour for free.
+ *
+ * Usage: node compile.mjs scene.hfir out/index.html
+ */
+
+import { readFileSync, writeFileSync } from "node:fs";
+
+/** How each IR property maps onto a CSS property and unit. */
+const PROPS = {
+  x: { css: "left", unit: "px" },
+  y: { css: "top", unit: "px" },
+  w: { css: "width", unit: "px" },
+  h: { css: "height", unit: "px" },
+  opacity: { css: "opacity", unit: "" },
+  size: { css: "fontSize", unit: "px" },
+  rotate: { css: "rotate", unit: "deg" },
+  fill: { css: "backgroundColor", unit: "" },
+};
+
+export function parse(src) {
+  const scene = { canvas: null, frames: null, decls: [], ops: [] };
+
+  src.split("\n").forEach((raw, i) => {
+    const line = raw.replace(/;.*$/, "").trim();
+    if (!line) return;
+    const [op, ...rest] = line.split(/\s+/);
+    const at = `line ${i + 1}`;
+
+    switch (op) {
+      case "CANVAS":
+        scene.canvas = { w: +rest[0], h: +rest[1], fps: +rest[2] };
+        break;
+      case "FRAMES":
+        scene.frames = +rest[0];
+        break;
+      case "DECL":
+        scene.decls.push({ id: rest[0], kind: rest[1] });
+        break;
+      case "SET":
+      case "RAMP": {
+        const [frame, el, prop] = rest;
+        if (prop !== "text" && !PROPS[prop]) {
+          throw new Error(`${at}: unknown property "${prop}"`);
+        }
+        scene.ops.push({
+          op,
+          frame: +frame,
+          el,
+          prop,
+          value: rest.slice(3).join(" "),
+        });
+        break;
+      }
+      default:
+        throw new Error(`${at}: unknown op "${op}"`);
+    }
+  });
+
+  if (!scene.canvas) throw new Error("missing CANVAS");
+  if (!scene.frames) throw new Error("missing FRAMES");
+
+  const declared = new Set(scene.decls.map((d) => d.id));
+  for (const o of scene.ops) {
+    if (!declared.has(o.el)) throw new Error(`op targets undeclared element "${o.el}"`);
+    if (o.frame > scene.frames) {
+      throw new Error(`op at frame ${o.frame} is past FRAMES ${scene.frames}`);
+    }
+  }
+  return scene;
+}
+
+/**
+ * One WAAPI animation per (element, property). Offsets land on exact frames, so
+ * seeking to frame N produces the value the table declared for frame N.
+ */
+export function plan(scene) {
+  const groups = new Map();
+  for (const o of scene.ops) {
+    const key = `${o.el}|${o.prop}`;
+    if (!groups.has(key)) groups.set(key, []);
+    groups.get(key).push(o);
+  }
+
+  const animations = [];
+  const textTracks = [];
+
+  for (const [key, list] of groups) {
+    const [el, prop] = key.split("|");
+    list.sort((a, b) => a.frame - b.frame);
+
+    // Text content is not animatable by WAAPI, so it is driven from a discrete
+    // per-frame table on seek. This is the one place the mapping is not 1:1.
+    if (prop === "text") {
+      textTracks.push({ el, steps: list.map((o) => ({ frame: o.frame, value: o.value })) });
+      continue;
+    }
+
+    const isText = scene.decls.find((d) => d.id === el)?.kind === "text";
+    const spec = PROPS[prop];
+    // On a text element, `fill` means the glyph colour, not a background.
+    const css = prop === "fill" && isText ? "color" : spec.css;
+
+    const keyframes = list.map((o) => ({
+      offset: Math.min(1, o.frame / scene.frames),
+      value: spec.unit ? `${o.value}${spec.unit}` : o.value,
+      // A SET holds its value; a RAMP interpolates into it.
+      easing: o.op === "SET" ? "steps(1, end)" : "linear",
+    }));
+
+    animations.push({ el, css, keyframes });
+  }
+
+  return { animations, textTracks };
+}
+
+export function emit(scene, { animations, textTracks }) {
+  const { w, h, fps } = scene.canvas;
+  const durationMs = (scene.frames / fps) * 1000;
+
+  const elements = scene.decls
+    .map((d) => {
+      const textStyle =
+        d.kind === "text" ? "white-space:nowrap;font-family:monospace;font-weight:700;" : "";
+      return (
+        `    <div id="${d.id}" data-start="0" data-duration="${scene.frames / fps}"` +
+        ` style="position:absolute;${textStyle}"></div>`
+      );
+    })
+    .join("\n");
+
+  const animJs = animations
+    .map((a) => {
+      const frames = a.keyframes.map(
+        (k) =>
+          `{ offset: ${k.offset}, ${a.css}: ${JSON.stringify(k.value)}, ` +
+          `easing: ${JSON.stringify(k.easing)} }`,
+      );
+      // A lone keyframe still needs an end frame to have a duration.
+      if (frames.length === 1) frames.push(frames[0].replace(/offset: [\d.]+/, "offset: 1"));
+      return (
+        `  document.getElementById(${JSON.stringify(a.el)}).animate([\n` +
+        `    ${frames.join(",\n    ")}\n  ], { duration: ${durationMs}, fill: "both" });`
+      );
+    })
+    .join("\n");
+
+  const textJs = textTracks.length
+    ? `
+  // Discrete text, applied on every seek from an explicit frame table.
+  var TEXT = ${JSON.stringify(textTracks)};
+  var FPS = ${fps};
+  var probe = document.getElementById(${JSON.stringify(scene.decls[0].id)});
+  function seekSeconds() {
+    var a = probe.getAnimations()[0];
+    return a ? (Number(a.currentTime) || 0) / 1000 : 0;
+  }
+  function applyText() {
+    var f = seekSeconds() * FPS + 1e-4;
+    for (var i = 0; i < TEXT.length; i++) {
+      var el = document.getElementById(TEXT[i].el);
+      var v = "";
+      for (var j = 0; j < TEXT[i].steps.length; j++) {
+        if (f >= TEXT[i].steps[j].frame) v = TEXT[i].steps[j].value;
+      }
+      if (el.textContent !== v) el.textContent = v;
+    }
+    requestAnimationFrame(applyText);
+  }
+  applyText();`
+    : "";
+
+  return `<!doctype html>
+<html>
+<head><meta charset="utf-8"><title>hfir</title><style>body{margin:0}</style></head>
+<body>
+  <!-- Generated from a .hfir table. Do not hand-edit. -->
+  <div id="root" data-composition-id="root" data-duration="${scene.frames / fps}"
+       data-width="${w}" data-height="${h}" data-no-timeline
+       style="position:relative;width:${w}px;height:${h}px;overflow:hidden">
+${elements}
+  </div>
+  <script>
+${animJs}
+${textJs}
+  </script>
+</body>
+</html>
+`;
+}
+
+export function compile(src) {
+  const scene = parse(src);
+  return { scene, html: emit(scene, plan(scene)) };
+}
+
+// CLI
+if (import.meta.url === `file://${process.argv[1]}`) {
+  const [, , input, output] = process.argv;
+  if (!input || !output) {
+    console.error("usage: compile.mjs <scene.hfir> <out/index.html>");
+    process.exit(1);
+  }
+  const { scene, html } = compile(readFileSync(input, "utf8"));
+  writeFileSync(output, html);
+  const { animations, textTracks } = plan(scene);
+  console.log(
+    `${scene.ops.length} ops / ${scene.decls.length} elements -> ` +
+      `${animations.length} animations, ${textTracks.length} text tracks`,
+  );
+}

experiments/hfir/compile.test.mjs

@@ -0,0 +1,49 @@
+import { describe, it, expect } from "vitest";
+
+import { compile, parse, plan } from "./compile.mjs";
+
+const SCENE = `
+CANVAS 100 100 30
+FRAMES 60
+DECL box rect
+SET  0  box opacity 0
+RAMP 30 box opacity 1
+SET  45 box fill #FF0000
+`;
+
+describe("hfir", () => {
+  it("puts a keyframe on the exact frame the table names", () => {
+    const { animations } = plan(parse(SCENE));
+    const opacity = animations.find((a) => a.css === "opacity");
+    // frame 30 of 60 is the midpoint
+    expect(opacity.keyframes.map((k) => k.offset)).toEqual([0, 0.5]);
+  });
+
+  it("makes SET a hard step and RAMP a linear interpolation", () => {
+    const { animations } = plan(parse(SCENE));
+    const opacity = animations.find((a) => a.css === "opacity");
+    expect(opacity.keyframes[0].easing).toBe("steps(1, end)"); // SET
+    expect(opacity.keyframes[1].easing).toBe("linear"); // RAMP
+  });
+
+  it("refuses an op that targets an element nobody declared", () => {
+    expect(() => parse("CANVAS 10 10 30\nFRAMES 1\nSET 0 ghost x 1")).toThrow(/undeclared/);
+  });
+
+  it("refuses a property it cannot map", () => {
+    expect(() => parse("CANVAS 10 10 30\nFRAMES 1\nDECL a rect\nSET 0 a wobble 1")).toThrow(
+      /unknown property/,
+    );
+  });
+
+  it("refuses an op past the end of the scene", () => {
+    expect(() => parse("CANVAS 10 10 30\nFRAMES 10\nDECL a rect\nSET 99 a x 1")).toThrow(/past/);
+  });
+
+  it("emits a composition the renderer can read", () => {
+    const { html } = compile(SCENE);
+    expect(html).toContain('data-composition-id="root"');
+    expect(html).toContain('data-duration="2"'); // 60 frames / 30fps
+    expect(html).toContain(".animate(");
+  });
+});