diff --git a/experiments/hfir/README.md b/experiments/hfir/README.md new file mode 100644 index 0000000000..fb3a6cb8c1 --- /dev/null +++ b/experiments/hfir/README.md @@ -0,0 +1,91 @@ +# hfir — assembly for programmatic video + +**Status: experiment. Not shipped, not supported, not on any roadmap. It exists to answer one question with evidence instead of opinion.** + +## The question + +Every code-to-video framework, ours included, is designed to be intuitive to a human. But increasingly no human reads the source. Agents write it, and agents want the opposite of what humans want: not terseness and convention, but explicitness and dials. + +So: **what if an agent never wrote HTML at all?** + +## What this is + +A flat table of `(op, frame, element, property, value)` that compiles to a renderable composition. This is a complete scene: + +``` +CANVAS 1280 720 30 +FRAMES 60 + +DECL box rect +DECL tag text + +SET 0 box x 120 +SET 0 box y 260 +SET 0 box w 200 +SET 0 box h 200 +SET 0 box fill #E0322C +SET 0 box opacity 0 + +RAMP 15 box opacity 1 +RAMP 45 box x 960 +RAMP 45 box fill #0F7A52 + +SET 30 tag text FRAME_30_EXACTLY +``` + +There is no HTML, no CSS, no cascade, no box model, no layout engine, and no easing vocabulary. + +## The one design decision that matters + +`SET` and `RAMP` are different opcodes. + +- `SET` is a hard step. The value holds until the next op. +- `RAMP` interpolates linearly into the stated value. + +Interpolation is declared **per property, per segment**. There is no implicit tween, no eased default, and no `ease-out` to misinterpret. If a property moves, something said so. + +That is the whole "assembly" idea: the agent addresses the machine directly and nothing is inferred on its behalf. + +## Try it + +```bash +node experiments/hfir/compile.mjs experiments/hfir/scene.hfir /tmp/hfir/index.html +npx hyperframes render /tmp/hfir +``` + +Compiles 22 ops into 15 animations and renders 60 frames of 720p in **~2.5 seconds**. + +## Verified, not asserted + +The example scene was rendered and the frames checked against the table by eye: + +| Frame | The table says | The pixels show | +| ------- | --------------- | -------------------------------------------------------------------------------- | +| 0 | `box opacity 0` | box genuinely absent, text white at exactly (120,120) | +| 29 → 30 | `SET` is a step | white `NO_HTML_WAS_HARMED` on 29, blue `FRAME_30_EXACTLY` on 30, nothing between | +| 45 | two `RAMP`s | box at x=960, colour interpolated red → green | + +## Why it targets WAAPI + +The compiler emits Web Animations API keyframes rather than CSS keyframes, because seeking a WAAPI animation by `currentTime` is the path our render engine treats as authoritative, and it is a Baseline web standard rather than a Chromium internal. + +The useful consequence: the IR gets browser-grade text shaping, compositing and colour **for free**, while the agent never touches any of it. The browser stays the renderer; only the authoring surface changes. + +That is also the argument against building a native renderer for this. The explicitness people want from "native" turns out to be a compiler problem, not an engine problem, and solving it this way keeps the installed renderer, the standards floor, and the enormous amount of HTML/CSS in every model's training data. + +## What it cannot do yet + +Named honestly, because these are the reasons it might not be worth pursuing: + +- **No layout.** Everything is absolute pixels. Fine for a machine, miserable for a human, and it means no responsive or flow-based composition. +- **No text wrapping.** Though `window.__hyperframes.pretext` now measures text without a reflow, so an agent could compute wrap points and emit them as explicit ops. That is the natural next step. +- **Text is the leak.** WAAPI cannot animate text content, so it is driven by a discrete per-frame table instead. It is the one place the mapping is not 1:1. +- **No media, no sub-compositions, no audio.** Audio in particular could not be inferred; it would need its own ops. + +## The experiment worth running next + +This prototype only proves the idea is _possible_. It does not prove it is _better_. + +The test that would settle it: give an agent the IR spec and the HTML spec cold, same prompts, and measure first-valid-render rate and repair loops for each. Nobody in the category publishes that number for any format, which is its own opportunity. + +If the IR wins, we have found something real. If HTML wins, we have cheaply killed a good-sounding idea, which is worth just as much. diff --git a/experiments/hfir/compile.mjs b/experiments/hfir/compile.mjs new file mode 100644 index 0000000000..a8456d59a1 --- /dev/null +++ b/experiments/hfir/compile.mjs @@ -0,0 +1,228 @@ +#!/usr/bin/env node +/** + * hfir -> HyperFrames composition. + * + * Compiles a flat table of (op, frame, element, property, value) into a + * renderable composition. The agent writing the table never sees HTML, CSS, the + * cascade, the box model, or an easing name. + * + * Two opcodes, and the difference is the whole point: + * SET frame el prop value hard step, holds until the next op + * RAMP frame el prop value linear interpolation from the previous value + * + * Interpolation is stated per property per segment, so there is no implicit + * tween and no eased default to misread. + * + * Output targets the Web Animations API rather than CSS keyframes, because + * seeking a WAAPI animation by `currentTime` is the path the render engine + * treats as authoritative. That also means the IR inherits browser text + * shaping, compositing and colour for free. + * + * Usage: node compile.mjs scene.hfir out/index.html + */ + +import { readFileSync, writeFileSync } from "node:fs"; + +/** How each IR property maps onto a CSS property and unit. */ +const PROPS = { + x: { css: "left", unit: "px" }, + y: { css: "top", unit: "px" }, + w: { css: "width", unit: "px" }, + h: { css: "height", unit: "px" }, + opacity: { css: "opacity", unit: "" }, + size: { css: "fontSize", unit: "px" }, + rotate: { css: "rotate", unit: "deg" }, + fill: { css: "backgroundColor", unit: "" }, +}; + +export function parse(src) { + const scene = { canvas: null, frames: null, decls: [], ops: [] }; + + src.split("\n").forEach((raw, i) => { + const line = raw.replace(/;.*$/, "").trim(); + if (!line) return; + const [op, ...rest] = line.split(/\s+/); + const at = `line ${i + 1}`; + + switch (op) { + case "CANVAS": + scene.canvas = { w: +rest[0], h: +rest[1], fps: +rest[2] }; + break; + case "FRAMES": + scene.frames = +rest[0]; + break; + case "DECL": + scene.decls.push({ id: rest[0], kind: rest[1] }); + break; + case "SET": + case "RAMP": { + const [frame, el, prop] = rest; + if (prop !== "text" && !PROPS[prop]) { + throw new Error(`${at}: unknown property "${prop}"`); + } + scene.ops.push({ + op, + frame: +frame, + el, + prop, + value: rest.slice(3).join(" "), + }); + break; + } + default: + throw new Error(`${at}: unknown op "${op}"`); + } + }); + + if (!scene.canvas) throw new Error("missing CANVAS"); + if (!scene.frames) throw new Error("missing FRAMES"); + + const declared = new Set(scene.decls.map((d) => d.id)); + for (const o of scene.ops) { + if (!declared.has(o.el)) throw new Error(`op targets undeclared element "${o.el}"`); + if (o.frame > scene.frames) { + throw new Error(`op at frame ${o.frame} is past FRAMES ${scene.frames}`); + } + } + return scene; +} + +/** + * One WAAPI animation per (element, property). Offsets land on exact frames, so + * seeking to frame N produces the value the table declared for frame N. + */ +export function plan(scene) { + const groups = new Map(); + for (const o of scene.ops) { + const key = `${o.el}|${o.prop}`; + if (!groups.has(key)) groups.set(key, []); + groups.get(key).push(o); + } + + const animations = []; + const textTracks = []; + + for (const [key, list] of groups) { + const [el, prop] = key.split("|"); + list.sort((a, b) => a.frame - b.frame); + + // Text content is not animatable by WAAPI, so it is driven from a discrete + // per-frame table on seek. This is the one place the mapping is not 1:1. + if (prop === "text") { + textTracks.push({ el, steps: list.map((o) => ({ frame: o.frame, value: o.value })) }); + continue; + } + + const isText = scene.decls.find((d) => d.id === el)?.kind === "text"; + const spec = PROPS[prop]; + // On a text element, `fill` means the glyph colour, not a background. + const css = prop === "fill" && isText ? "color" : spec.css; + + const keyframes = list.map((o) => ({ + offset: Math.min(1, o.frame / scene.frames), + value: spec.unit ? `${o.value}${spec.unit}` : o.value, + // A SET holds its value; a RAMP interpolates into it. + easing: o.op === "SET" ? "steps(1, end)" : "linear", + })); + + animations.push({ el, css, keyframes }); + } + + return { animations, textTracks }; +} + +export function emit(scene, { animations, textTracks }) { + const { w, h, fps } = scene.canvas; + const durationMs = (scene.frames / fps) * 1000; + + const elements = scene.decls + .map((d) => { + const textStyle = + d.kind === "text" ? "white-space:nowrap;font-family:monospace;font-weight:700;" : ""; + return ( + `
` + ); + }) + .join("\n"); + + const animJs = animations + .map((a) => { + const frames = a.keyframes.map( + (k) => + `{ offset: ${k.offset}, ${a.css}: ${JSON.stringify(k.value)}, ` + + `easing: ${JSON.stringify(k.easing)} }`, + ); + // A lone keyframe still needs an end frame to have a duration. + if (frames.length === 1) frames.push(frames[0].replace(/offset: [\d.]+/, "offset: 1")); + return ( + ` document.getElementById(${JSON.stringify(a.el)}).animate([\n` + + ` ${frames.join(",\n ")}\n ], { duration: ${durationMs}, fill: "both" });` + ); + }) + .join("\n"); + + const textJs = textTracks.length + ? ` + // Discrete text, applied on every seek from an explicit frame table. + var TEXT = ${JSON.stringify(textTracks)}; + var FPS = ${fps}; + var probe = document.getElementById(${JSON.stringify(scene.decls[0].id)}); + function seekSeconds() { + var a = probe.getAnimations()[0]; + return a ? (Number(a.currentTime) || 0) / 1000 : 0; + } + function applyText() { + var f = seekSeconds() * FPS + 1e-4; + for (var i = 0; i < TEXT.length; i++) { + var el = document.getElementById(TEXT[i].el); + var v = ""; + for (var j = 0; j < TEXT[i].steps.length; j++) { + if (f >= TEXT[i].steps[j].frame) v = TEXT[i].steps[j].value; + } + if (el.textContent !== v) el.textContent = v; + } + requestAnimationFrame(applyText); + } + applyText();` + : ""; + + return ` + +