(expressive mode): register inference replaces presets - #6571
Open
cshape wants to merge 5 commits into
Open
Conversation
Registers Fish Audio (s2 family) across the expr pipeline so inference fishaudio/* voices work in expressive mode: - llm_instructions: sounds-filtered builder β discrete emotion vocabulary (regretful, hopeful, ... sarcastic), sounds (laughing, clear throat), breaks, and an emphasis-only wrapping prosody - convert_markup: expr lowers to Fish's native brackets β [very EMOTION] (intensified so the emotion lands harder), [SOUND], [break]/[long-break] by duration, and [emphasis] word - steering: laughing/reflex_sounds govern the two sounds; disabled labels vanish from the advertised vocabulary and examples - transcript: expr, hallucinated native XML, and Fish's own bracket form are all stripped; brackets=True since Fish's native dialect IS brackets - chunking: no _MAX_INPUT_LEN entry on purpose β Fish's markers are sentence-scoped, so per-sentence emission loses no steering and keeps time-to-first-audio low
gemma keeps labels english on its own, but a weaker llm could write label="ε¬γγ", which lowers to a bracket fish doesn't recognize β make it explicit like the pre-refactor prompts did.
The shared preamble now tells every markup-capable provider's LLM to match delivery to the register of the moment β composed and restrained when the moment is professional or emotionally heavy, loose and bright when it is casual β reassessed every turn. This is the mechanism that lets a single default configuration serve a debt collector and a kids' tutor alike. The fish block adds vocabulary-specific guidance on top, built conditionally so an opted-out option is never referenced, not even prohibitively: laughter guidance only when laugh sounds are advertised, filler guidance only when disfluencies are enabled, and label-polarity guidance (empathetic/sad/regretful/hopeful at heavy moments, never happy/excited against hard news) always.
Register-inference evals showed the presets moved almost nothing the
agent's own persona didn't already move: with the full vocabulary
advertised and the register rule in the shared preamble, models place
laughter and fillers by conversational moment on their own (0.83
laughs/turn at casual moments vs 0.00 at serious, same settings, and
0.00 serious-moment laughs across four llms).
formal/casual also carried an implied delivery contract the data doesn't
support β on fish the two presets resolved to byte-identical
instructions. A locked-down agent is now expressed directly:
{"nonverbal_sounds": {}, "disfluencies": False}.
Two semantic fixes to the steering surface, from review:
- nonverbal_sounds accepts a plain bool: True keeps the provider's full
vocabulary (same as omitting the key), False disables every sound.
- A NonverbalOptions dict is now a genuinely sparse opt-out: omitted
categories default to ENABLED, so {"laughing": False} removes laughter
and nothing else. Previously omitted categories defaulted to off, which
made every nested dict silently disable the whole vocabulary β the same
atomic-mapping footgun previously flagged on the preset merge.
The omitted-off default existed for the presets, which restricted from
zero; with presets gone the default is the full vocabulary and every
control is subtractive.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
stacked on #6528 β the first two commits are that PR; review the last three.
our evals (~2,000 generations, 5 personas Γ serious/neutral/casual moments, 4 llms) suggest the agent persona and conversational context choose the appropriate register on their own, so this replaces the formal/casual presets with:
nonverbal_soundsaccepts a bool, and dicts are sparse opt-outs:{"laughing": False}removes laughter and nothing elsemeasured at the default (everything on): 0.00 laughs/turn at serious moments across all four llms, 0.67β0.83 at casual ones, judge fit 4.59/5 overall.
{"nonverbal_sounds": False, "disfluencies": False}produced 0 sounds and 1 filler across 360 generations.one semantic change to sign off on:
NonverbalOptions()(empty dict) now means "no opt-outs" β everything stays on;nonverbal_sounds=Falseis the all-off form.full findings + reproducible eval harness shared separately.