The timestamp of various events is under-specified, and not implemented per spec in shipped implementations

[padenot]

(Sorry looks long, but no changes is required to #192 or related work, it's things to make #192 more solid)

There's this bit of normative text in the spec:

For all these events, the timeStamp attribute defined in the DOM Level 2 Event interface must be set to the best possible estimate of when the real-world event which the event object represents occurred.

https://webaudio.github.io/web-speech-api/#:~:text=For%20all%20these,occurred

It turns out that this isn't how it's implemented in Chromium, which dispatches all six events as plain Events, so timeStamp is just the event creation time: https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/modules/speech/speech_recognition.cc;l=435-455. It is what is implemented currently in Firefox, but we haven't shipped and I can make modifications. This proposal aims at keeping what Chromium does today and adding stuff on top in a web-compatible way.

Chromium already carries the kind of clock we actually want, just not for these events and not on the system clock: TimingInformation::audio_start_time is "the amount of audio input into SODA" (which I believe to refer to the underlying recognition system), i.e. an offset in the audio's own timeline. It is plumbed for results only: https://source.chromium.org/chromium/chromium/src/+/main:media/mojo/mojom/speech_recognition_result.h;l=60-65. The client-side endpointer also keeps its speech onset and offset in audio time already (speech_start_time_us_, speech_end_time_us_, exposed via Status()), so the numbers exist at the point where soundstart/soundend are raised. It looks like they're dropped at the IPC boundary (but I'm not fluent in mojo, correct me if I'm wrong, I haven't dug much further).

This gets us in a position to make the spec clearer and more useful for users.

Some context, there are multiple clocks here:

• A DOMHighResTimeStamp, high precision system time, what we typcally set on DOM events and such.
• B the audio device time, clocked to an audio device. This drifts vs the above, by a fraction of a percent, and will eventually not be aligned. This is how an HTMLMediaElement, an AudioContext, a getUserMedia mic MediaStreamTrack, etc., tick. It typically starts at 0, then goes forward. It is latency compensated: it takes into account the audio output latency, if any (for A/V sync in particular).
• C the media time. It can tick at the same rate as the audio device time, but doesn't have to: if we're recognizing a mic, it will, but if we recognize a file that isn't playing out, it can be on the system clock. We will likely add faster-than-realtime recognition later, and that will also be different. It starts at 0 in any case, on those objects, but we can offset it at will (captureStream into another HTMLMediaElement will create a new origin). This isn't exposed to the web.
• D the recognition timeline. It starts at 0 when the "thing" that performs recognition starts accepting audio, and it ticks at the same rate as the media time, drifting or not against the system clock (depending on whether the source/destination of the audio is an audio device).

A, and sometimes C or D, are the system clock. B, and maybe C and D, come from an audio device, which doesn't tick at the same speed. And the origins vary between all of those.

#192's main motivation is having VTT cues be super precise, potentially from a live source (that has its own latency), for replay, and I agree this is an important goal.

But we know that a lot of people watch videos muted and then un-mute them, so we need to account for audio latencies: if we use a latency-compensated clock we get a shift (more so on e.g. a Bluetooth device, or other high latency devices). Note that there is no HTMLMediaElement.outputLatency, so an author can't even compute the correction. This also means that using only the system clock (regular DOM event time-stamping) will not be correct in all cases, especially after long sessions.

Basically, in that setup we do mediaEl.captureStream().getAudioTracks()[0] (or a variant, maybe via an AudioContext, etc.) and pass that track to our SpeechRecognition object's start() method. The init time doesn't matter much, in the sense that audio can immediately start to be buffered and can be sent to the recognition engine wholesale, and we can already get sentences "in the past". So t=0 is precisely when start() was called, timestamped internally in the clock domain of C.

First, implementations shouldn't drop source audio while the recognition engine is warming up (model load, connecting to a service, etc.): if they buffer it insteadaudiostart comes out at 0.0 and all is good, eveyrthing matches.

A slight complication here is start() with no argument the mic doesn't exist until the permission prompt has been answered and the device is open, which can take seconds. There, the origin is the first audio the device produces. Both cases are the same rule — the origin is the position of the source audio at the moment the session starts consuming the source — which is also nicer than tying it to audiostart, since it doesn't depend on when the engine happened to be ready.

Seeking or pausing mid-session needs to be accounted for in this mapping, but we can sidestep the problem by starting another SpeechRecognition session, much like above. It's complicated otherwise: pausing an HTMLMediaElement doesn't "block" the MediaStreamTrack, it keeps delivering silence in real time (mediacapture-main: a muted or disabled track gives the consumer "zero-information-content, which means silence for audio"), so the recognition timeline keeps advancing while the media time doesn't. Seeking adds similar complications.

If we want to align our events with "the audio" (e.g. in the demos by @alan33d, there was a very clear waveform drawn, and subsequently the record/replay one), and we want things to not be internal/magic, I think we can "just" do the following:

[SecureContext, Exposed=Window]
interface SpeechRecognitionAudioEvent : Event {
    constructor(DOMString type, optional SpeechRecognitionAudioEventInit eventInitDict = {});

    // Offset, in seconds, on the session's audio timeline, of the position in
    // the audio that this event reports. 0.0 for "audiostart".
    readonly attribute double time; // bikesheding welcome
};

dictionary SpeechRecognitionAudioEventInit : EventInit {
    double time = 0;
};

and then we inhering our events from that one.

time is a difference in C, with its origin at start(), and is always correct: no conversion to A or B ever happens, so neither drift nor latency compensation can creep in, and the values are the same whether recognition runs in real time or faster. Then we rebase #192 onto it (almost no change), and it's all well defined in terms of audio (media) timeline.

All of this is agreeing what https://github.com/WebAudio/web-speech-api/blob/main/explainers/speech-recognition-result-timestamps.md says and #192 does but goes a little bit deeper and is a bit less hand-wavy, in the interest of maximizing compatibility and making sure scheduling/timings are tack sharps for authors/users.