From 7958e8aac6b4f386e0314200d6fb829d85d30e04 Mon Sep 17 00:00:00 2001 From: Miguel Angel Simon Sierra Date: Mon, 17 Aug 2026 22:23:56 -0400 Subject: [PATCH] feat(experiments): hfir, an assembly-like IR agents write instead of HTML Not shipped and not proposed for shipping. Lives under experiments/ and answers one question with a runnable prototype instead of an argument. The question: every code-to-video framework is designed to read well to a human, but increasingly no human reads the source. Agents want the opposite of what humans want, explicitness over convention. So what if an agent never wrote HTML at all? hfir is a flat table of (op, frame, element, property, value). Two opcodes, and the difference between them is the whole idea: SET is a hard step that holds, RAMP interpolates linearly into the stated value. Interpolation is declared per property per segment, so there is no implicit tween and no easing name to misread. Nothing is inferred. It compiles to WAAPI keyframes rather than CSS, because seeking by currentTime is the path the render engine treats as authoritative and it is a Baseline web standard rather than a Chromium internal. The useful consequence is that the IR inherits browser text shaping, compositing and colour for free while the agent never touches the cascade or the box model. That is also the honest answer to "should we go native": the explicitness people want from native turns out to be a compiler problem rather than an engine one, and doing it this way keeps the installed renderer and the HTML/CSS fluency every model already has. Verified rather than asserted. The example scene renders 60 frames of 720p in about 2.5s, and frames 0, 29, 30 and 45 were checked against the table by eye: opacity 0 is genuinely absent, the SET at frame 30 steps with nothing between 29 and 30, and the two RAMPs land the box at x=960 interpolated red to green. Limits are in the README, including the one real leak: WAAPI cannot animate text content, so text runs off a discrete per-frame table. The prototype only shows this is possible, not that it is better. The test that would settle it is a cold first-valid-render comparison against HTML, which nobody in the category publishes for any format. --- experiments/hfir/README.md | 91 ++++++++++++ experiments/hfir/compile.mjs | 228 ++++++++++++++++++++++++++++++ experiments/hfir/compile.test.mjs | 49 +++++++ experiments/hfir/scene.hfir | 40 ++++++ 4 files changed, 408 insertions(+) create mode 100644 experiments/hfir/README.md create mode 100644 experiments/hfir/compile.mjs create mode 100644 experiments/hfir/compile.test.mjs create mode 100644 experiments/hfir/scene.hfir diff --git a/experiments/hfir/README.md b/experiments/hfir/README.md new file mode 100644 index 0000000000..fb3a6cb8c1 --- /dev/null +++ b/experiments/hfir/README.md @@ -0,0 +1,91 @@ +# hfir — assembly for programmatic video + +**Status: experiment. Not shipped, not supported, not on any roadmap. It exists to answer one question with evidence instead of opinion.** + +## The question + +Every code-to-video framework, ours included, is designed to be intuitive to a human. But increasingly no human reads the source. Agents write it, and agents want the opposite of what humans want: not terseness and convention, but explicitness and dials. + +So: **what if an agent never wrote HTML at all?** + +## What this is + +A flat table of `(op, frame, element, property, value)` that compiles to a renderable composition. This is a complete scene: + +``` +CANVAS 1280 720 30 +FRAMES 60 + +DECL box rect +DECL tag text + +SET 0 box x 120 +SET 0 box y 260 +SET 0 box w 200 +SET 0 box h 200 +SET 0 box fill #E0322C +SET 0 box opacity 0 + +RAMP 15 box opacity 1 +RAMP 45 box x 960 +RAMP 45 box fill #0F7A52 + +SET 30 tag text FRAME_30_EXACTLY +``` + +There is no HTML, no CSS, no cascade, no box model, no layout engine, and no easing vocabulary. + +## The one design decision that matters + +`SET` and `RAMP` are different opcodes. + +- `SET` is a hard step. The value holds until the next op. +- `RAMP` interpolates linearly into the stated value. + +Interpolation is declared **per property, per segment**. There is no implicit tween, no eased default, and no `ease-out` to misinterpret. If a property moves, something said so. + +That is the whole "assembly" idea: the agent addresses the machine directly and nothing is inferred on its behalf. + +## Try it + +```bash +node experiments/hfir/compile.mjs experiments/hfir/scene.hfir /tmp/hfir/index.html +npx hyperframes render /tmp/hfir +``` + +Compiles 22 ops into 15 animations and renders 60 frames of 720p in **~2.5 seconds**. + +## Verified, not asserted + +The example scene was rendered and the frames checked against the table by eye: + +| Frame | The table says | The pixels show | +| ------- | --------------- | -------------------------------------------------------------------------------- | +| 0 | `box opacity 0` | box genuinely absent, text white at exactly (120,120) | +| 29 → 30 | `SET` is a step | white `NO_HTML_WAS_HARMED` on 29, blue `FRAME_30_EXACTLY` on 30, nothing between | +| 45 | two `RAMP`s | box at x=960, colour interpolated red → green | + +## Why it targets WAAPI + +The compiler emits Web Animations API keyframes rather than CSS keyframes, because seeking a WAAPI animation by `currentTime` is the path our render engine treats as authoritative, and it is a Baseline web standard rather than a Chromium internal. + +The useful consequence: the IR gets browser-grade text shaping, compositing and colour **for free**, while the agent never touches any of it. The browser stays the renderer; only the authoring surface changes. + +That is also the argument against building a native renderer for this. The explicitness people want from "native" turns out to be a compiler problem, not an engine problem, and solving it this way keeps the installed renderer, the standards floor, and the enormous amount of HTML/CSS in every model's training data. + +## What it cannot do yet + +Named honestly, because these are the reasons it might not be worth pursuing: + +- **No layout.** Everything is absolute pixels. Fine for a machine, miserable for a human, and it means no responsive or flow-based composition. +- **No text wrapping.** Though `window.__hyperframes.pretext` now measures text without a reflow, so an agent could compute wrap points and emit them as explicit ops. That is the natural next step. +- **Text is the leak.** WAAPI cannot animate text content, so it is driven by a discrete per-frame table instead. It is the one place the mapping is not 1:1. +- **No media, no sub-compositions, no audio.** Audio in particular could not be inferred; it would need its own ops. + +## The experiment worth running next + +This prototype only proves the idea is _possible_. It does not prove it is _better_. + +The test that would settle it: give an agent the IR spec and the HTML spec cold, same prompts, and measure first-valid-render rate and repair loops for each. Nobody in the category publishes that number for any format, which is its own opportunity. + +If the IR wins, we have found something real. If HTML wins, we have cheaply killed a good-sounding idea, which is worth just as much. diff --git a/experiments/hfir/compile.mjs b/experiments/hfir/compile.mjs new file mode 100644 index 0000000000..a8456d59a1 --- /dev/null +++ b/experiments/hfir/compile.mjs @@ -0,0 +1,228 @@ +#!/usr/bin/env node +/** + * hfir -> HyperFrames composition. + * + * Compiles a flat table of (op, frame, element, property, value) into a + * renderable composition. The agent writing the table never sees HTML, CSS, the + * cascade, the box model, or an easing name. + * + * Two opcodes, and the difference is the whole point: + * SET frame el prop value hard step, holds until the next op + * RAMP frame el prop value linear interpolation from the previous value + * + * Interpolation is stated per property per segment, so there is no implicit + * tween and no eased default to misread. + * + * Output targets the Web Animations API rather than CSS keyframes, because + * seeking a WAAPI animation by `currentTime` is the path the render engine + * treats as authoritative. That also means the IR inherits browser text + * shaping, compositing and colour for free. + * + * Usage: node compile.mjs scene.hfir out/index.html + */ + +import { readFileSync, writeFileSync } from "node:fs"; + +/** How each IR property maps onto a CSS property and unit. */ +const PROPS = { + x: { css: "left", unit: "px" }, + y: { css: "top", unit: "px" }, + w: { css: "width", unit: "px" }, + h: { css: "height", unit: "px" }, + opacity: { css: "opacity", unit: "" }, + size: { css: "fontSize", unit: "px" }, + rotate: { css: "rotate", unit: "deg" }, + fill: { css: "backgroundColor", unit: "" }, +}; + +export function parse(src) { + const scene = { canvas: null, frames: null, decls: [], ops: [] }; + + src.split("\n").forEach((raw, i) => { + const line = raw.replace(/;.*$/, "").trim(); + if (!line) return; + const [op, ...rest] = line.split(/\s+/); + const at = `line ${i + 1}`; + + switch (op) { + case "CANVAS": + scene.canvas = { w: +rest[0], h: +rest[1], fps: +rest[2] }; + break; + case "FRAMES": + scene.frames = +rest[0]; + break; + case "DECL": + scene.decls.push({ id: rest[0], kind: rest[1] }); + break; + case "SET": + case "RAMP": { + const [frame, el, prop] = rest; + if (prop !== "text" && !PROPS[prop]) { + throw new Error(`${at}: unknown property "${prop}"`); + } + scene.ops.push({ + op, + frame: +frame, + el, + prop, + value: rest.slice(3).join(" "), + }); + break; + } + default: + throw new Error(`${at}: unknown op "${op}"`); + } + }); + + if (!scene.canvas) throw new Error("missing CANVAS"); + if (!scene.frames) throw new Error("missing FRAMES"); + + const declared = new Set(scene.decls.map((d) => d.id)); + for (const o of scene.ops) { + if (!declared.has(o.el)) throw new Error(`op targets undeclared element "${o.el}"`); + if (o.frame > scene.frames) { + throw new Error(`op at frame ${o.frame} is past FRAMES ${scene.frames}`); + } + } + return scene; +} + +/** + * One WAAPI animation per (element, property). Offsets land on exact frames, so + * seeking to frame N produces the value the table declared for frame N. + */ +export function plan(scene) { + const groups = new Map(); + for (const o of scene.ops) { + const key = `${o.el}|${o.prop}`; + if (!groups.has(key)) groups.set(key, []); + groups.get(key).push(o); + } + + const animations = []; + const textTracks = []; + + for (const [key, list] of groups) { + const [el, prop] = key.split("|"); + list.sort((a, b) => a.frame - b.frame); + + // Text content is not animatable by WAAPI, so it is driven from a discrete + // per-frame table on seek. This is the one place the mapping is not 1:1. + if (prop === "text") { + textTracks.push({ el, steps: list.map((o) => ({ frame: o.frame, value: o.value })) }); + continue; + } + + const isText = scene.decls.find((d) => d.id === el)?.kind === "text"; + const spec = PROPS[prop]; + // On a text element, `fill` means the glyph colour, not a background. + const css = prop === "fill" && isText ? "color" : spec.css; + + const keyframes = list.map((o) => ({ + offset: Math.min(1, o.frame / scene.frames), + value: spec.unit ? `${o.value}${spec.unit}` : o.value, + // A SET holds its value; a RAMP interpolates into it. + easing: o.op === "SET" ? "steps(1, end)" : "linear", + })); + + animations.push({ el, css, keyframes }); + } + + return { animations, textTracks }; +} + +export function emit(scene, { animations, textTracks }) { + const { w, h, fps } = scene.canvas; + const durationMs = (scene.frames / fps) * 1000; + + const elements = scene.decls + .map((d) => { + const textStyle = + d.kind === "text" ? "white-space:nowrap;font-family:monospace;font-weight:700;" : ""; + return ( + `
` + ); + }) + .join("\n"); + + const animJs = animations + .map((a) => { + const frames = a.keyframes.map( + (k) => + `{ offset: ${k.offset}, ${a.css}: ${JSON.stringify(k.value)}, ` + + `easing: ${JSON.stringify(k.easing)} }`, + ); + // A lone keyframe still needs an end frame to have a duration. + if (frames.length === 1) frames.push(frames[0].replace(/offset: [\d.]+/, "offset: 1")); + return ( + ` document.getElementById(${JSON.stringify(a.el)}).animate([\n` + + ` ${frames.join(",\n ")}\n ], { duration: ${durationMs}, fill: "both" });` + ); + }) + .join("\n"); + + const textJs = textTracks.length + ? ` + // Discrete text, applied on every seek from an explicit frame table. + var TEXT = ${JSON.stringify(textTracks)}; + var FPS = ${fps}; + var probe = document.getElementById(${JSON.stringify(scene.decls[0].id)}); + function seekSeconds() { + var a = probe.getAnimations()[0]; + return a ? (Number(a.currentTime) || 0) / 1000 : 0; + } + function applyText() { + var f = seekSeconds() * FPS + 1e-4; + for (var i = 0; i < TEXT.length; i++) { + var el = document.getElementById(TEXT[i].el); + var v = ""; + for (var j = 0; j < TEXT[i].steps.length; j++) { + if (f >= TEXT[i].steps[j].frame) v = TEXT[i].steps[j].value; + } + if (el.textContent !== v) el.textContent = v; + } + requestAnimationFrame(applyText); + } + applyText();` + : ""; + + return ` + +hfir + + +
+${elements} +
+ + + +`; +} + +export function compile(src) { + const scene = parse(src); + return { scene, html: emit(scene, plan(scene)) }; +} + +// CLI +if (import.meta.url === `file://${process.argv[1]}`) { + const [, , input, output] = process.argv; + if (!input || !output) { + console.error("usage: compile.mjs "); + process.exit(1); + } + const { scene, html } = compile(readFileSync(input, "utf8")); + writeFileSync(output, html); + const { animations, textTracks } = plan(scene); + console.log( + `${scene.ops.length} ops / ${scene.decls.length} elements -> ` + + `${animations.length} animations, ${textTracks.length} text tracks`, + ); +} diff --git a/experiments/hfir/compile.test.mjs b/experiments/hfir/compile.test.mjs new file mode 100644 index 0000000000..956682828b --- /dev/null +++ b/experiments/hfir/compile.test.mjs @@ -0,0 +1,49 @@ +import { describe, it, expect } from "vitest"; + +import { compile, parse, plan } from "./compile.mjs"; + +const SCENE = ` +CANVAS 100 100 30 +FRAMES 60 +DECL box rect +SET 0 box opacity 0 +RAMP 30 box opacity 1 +SET 45 box fill #FF0000 +`; + +describe("hfir", () => { + it("puts a keyframe on the exact frame the table names", () => { + const { animations } = plan(parse(SCENE)); + const opacity = animations.find((a) => a.css === "opacity"); + // frame 30 of 60 is the midpoint + expect(opacity.keyframes.map((k) => k.offset)).toEqual([0, 0.5]); + }); + + it("makes SET a hard step and RAMP a linear interpolation", () => { + const { animations } = plan(parse(SCENE)); + const opacity = animations.find((a) => a.css === "opacity"); + expect(opacity.keyframes[0].easing).toBe("steps(1, end)"); // SET + expect(opacity.keyframes[1].easing).toBe("linear"); // RAMP + }); + + it("refuses an op that targets an element nobody declared", () => { + expect(() => parse("CANVAS 10 10 30\nFRAMES 1\nSET 0 ghost x 1")).toThrow(/undeclared/); + }); + + it("refuses a property it cannot map", () => { + expect(() => parse("CANVAS 10 10 30\nFRAMES 1\nDECL a rect\nSET 0 a wobble 1")).toThrow( + /unknown property/, + ); + }); + + it("refuses an op past the end of the scene", () => { + expect(() => parse("CANVAS 10 10 30\nFRAMES 10\nDECL a rect\nSET 99 a x 1")).toThrow(/past/); + }); + + it("emits a composition the renderer can read", () => { + const { html } = compile(SCENE); + expect(html).toContain('data-composition-id="root"'); + expect(html).toContain('data-duration="2"'); // 60 frames / 30fps + expect(html).toContain(".animate("); + }); +}); diff --git a/experiments/hfir/scene.hfir b/experiments/hfir/scene.hfir new file mode 100644 index 0000000000..6382963a5a --- /dev/null +++ b/experiments/hfir/scene.hfir @@ -0,0 +1,40 @@ +; A complete HyperFrames scene with no HTML and no CSS in it. +; Columns: OP FRAME ELEMENT PROPERTY VALUE + +CANVAS 1280 720 30 +FRAMES 60 + +DECL bg rect +DECL box rect +DECL tag text + +; frame 0 - every value stated, nothing inferred +SET 0 bg x 0 +SET 0 bg y 0 +SET 0 bg w 1280 +SET 0 bg h 720 +SET 0 bg fill #101623 + +SET 0 box x 120 +SET 0 box y 260 +SET 0 box w 200 +SET 0 box h 200 +SET 0 box fill #E0322C +SET 0 box opacity 0 + +SET 0 tag x 120 +SET 0 tag y 120 +SET 0 tag size 44 +SET 0 tag fill #FFFFFF +SET 0 tag text NO_HTML_WAS_HARMED + +; RAMP interpolates linearly into the stated value +RAMP 15 box opacity 1 +RAMP 45 box x 960 +RAMP 45 box fill #0F7A52 + +; SET is a hard step - frame 29 and frame 30 differ with nothing in between +SET 30 tag text FRAME_30_EXACTLY +SET 30 tag fill #2B5DE0 + +SET 59 box x 960