FIX Scope request converters by prepended message role

[Roman Lutz (romanlutz)]

Description

Request converters could rewrite simulated assistant history when an attack supplied a prepended conversation. This was especially easy to trigger when the AIRT Jailbreak scenario attached its template converter to conversation-based techniques such as context_compliance.

This change makes converter application role-aware and keeps that scoping intact across chat and non-chat targets:

• Prepended request conversion defaults to user-role messages; assistant conversion requires explicit opt-in.
• Non-chat targets convert eligible history before flattening, avoid send-time double conversion, preserve converter piece indexes across retries, and reject non-text converter output that cannot be represented safely in a flattened prompt.
• Jailbreak template conversion is restricted to the scenario-owned prompt_sending delivery. Native system-prompt delivery remains a separate converter-free path.

Tests and Documentation

• 380 focused Python tests covering conversation management, prepended configuration, prompt sending, converter behavior, Crescendo, ShieldGemma, and Jailbreak.
• 1,003 frontend tests, frontend lint, and production build after merging the latest main.
• Ruff, formatting, type checks, diff checks, and repository commit hooks.
• JupyText: N/A (no documentation notebooks changed).

[Richard Lundeen (rlundeen2)]

Overall I think we can make this a lot smaller. The actual bug is narrow: request converters can rewrite simulated assistant history when an attack supplies a prepended conversation. It's easy to hit with context_compliance, where the jailbreak's TextJailbreakConverter ends up rewriting the fake assistant turns. So to me this is about which messages we convert, not new machinery — and I don't think we need to touch ConversationManager or put state on Message.

Here's the direction I'd like instead:

• PrependedConversationConfig stays the sole policy for prepended history. It already picks which roles get converters (apply_converters_to_roles) and which normalizer flattens history. If we default that to ["user"], the assistant-history bug is basically fixed on its own — assistant history is simulated target output, so you have to opt in with ["assistant"] to touch it. If we ever need more control over prepended history, we add it here.
• StrategyConverterConfig gets a way to scope a converter to the first turn. This decides which messages a converter applies to, exactly like apply_converters_to_roles already does. And because it's declarative it rides in the attack identifier, so resume/eval stay deterministic for free.
• Optional implementation detail We can derive "first turn" straight from memory if we want — without threading anything new through the attacks. The normalizer already has the conversation_id and a memory handle, so it can just ask: has the target actually replied yet? Define first-turn as "no real assistant message in this conversation" (prepended turns are stored as simulated_assistant, so they don't count). The nice thing is every case falls out correctly, including the tricky ones — e.g. on a prompt_sending retry we spin up a fresh conversation, so the jailbreak is correctly re-applied, whereas a naive per-send counter would drop it. Multi-turn continuations (Crescendo, TAP nodes copied from a parent) already have a real assistant reply, so they're correctly not re-wrapped.
• Jailbreak scenario: I still like prompt_sending and jailbreak_system_prompt as the defaults. We could add one or two more to show off the scoping (e.g. role play with the jailbreak on the system prompt, or context_compliance with the jailbreak only on the first live turn). But I don't think we should just include all techniques (like how they were before this PR) — they're all a bit different, and the jailbreak locus isn't the same for each. If we do role play, is the jailbreak on the system prompt, the first turn, or every turn? That should be a deliberate choice per technique, which is exactly why the scope belongs on the config.

Net: ConversationManager keeps applying the converter list it's given (role-filtered), no eager conversion, and no _request_converters_applied marker on Message. For non-chat targets we go back to today's behavior — flatten history, prepend it, let the live send convert the block. Unless I'm missing something, that's the whole change.

[Roman Lutz (romanlutz)]

Richard Lundeen (@rlundeen2) I like FIRST_LIVE_TURN as a declarative converter scope, but it does not preserve role scoping for non-chat targets. ConversationManager flattens prepended user/assistant history and the live request into one string before PromptNormalizer runs converters. At that point the converter sees the entire joined block. The turn scope can decide whether the converter runs, but not prevent the assistant portion from being rewritten.

The proposal would work if converter scope were applied while the conversation was still structured, before target normalization/flattening. That requires either the bridge in this PR or a larger pipeline change that introduces structured history adaptation and applies origin/role/turn scope before normalization. So I think FIRST_LIVE_TURN is useful follow-up API design, but it is not by itself a replacement for the current non-chat handling.

[Richard Lundeen (rlundeen2)]

Good catch, I think ConversationManager existed before target capabilities and it shouldn't do message normalization anymore. Agreed with the issue with ConversationManager flattening things. But I don't agree with this: "a larger pipeline change that introduces structured history adaptation" because it's already built and wired. I think the following would be tighter/less error-prone and not significantly more complicated than the current approach of adding converters to the conversation manager.

PromptTarget._get_normalized_conversation_async pulls conversation history from memory, appends the live message, and runs configuration.normalize_async — all after converters have run. The pipeline is capability-gated
and already registers the normalizer that does this exact job

So the shape I'd like to land on:

1. Have initialize_context_async treat non-chat targets like chat targets — write the prepended conversation to memory, structured, with roles.
2. Drop _handle_non_chat_target_async.
3. Gate the squash on missing EDITABLE_HISTORY in the capability→normalizer table (change to ADAPT)
4. Map PrependedConversationConfig.message_normalizer onto normalizer_overrides so callers keep format control.

One wrinkle is multi turn (but not editable) targets like playwright will need to squash but not squash previous turns. But that's the idea that FIRST_TURN could identify.

Also worth noting: _handle_non_chat_target_async only mutates context.next_message and never persists, so prepended history isn't recorded as conversation history for these targets today. Moving the squash into the pipeline fixes that too.

[Roman Lutz (romanlutz)]

But this means normalization happens in the target, and history represents something different, right?

History may show 8 turns of prompt/response but it's perhaps a non-editable history target so everything gets squashed into a single prompt. History should reflect that. Unless I'm misunderstanding.