Improve on-device clarification generation - #2
Conversation
| """ | ||
| static func instructions(hasClarificationHistory: Bool) -> String { | ||
| let task = hasClarificationHistory | ||
| ? """ |
There was a problem hiding this comment.
Can be as simple as:
"""
Given abstract feedback on an iOS app, learn from the user by asking questions about the user's feedback so the developer can more easily implement the user's feedback.
"""
| """ | ||
|
|
||
| return """ | ||
| You help an everyday app user give actionable feedback. |
There was a problem hiding this comment.
Can be as simple as:
"""
Given abstract feedback on an iOS app, learn from the user by asking questions about the user's feedback so the developer can more easily implement the user's feedback.
"""
|
|
||
| @Guide(description: "True for vague negative feedback when one answer would help, including slow, confusing, hard to use, looks wrong, or does not make sense; false for clear positive feedback, false for a concrete product suggestion that names what should change and the desired direction, false after a substantive answer to a visual, content, usability, or accessibility follow-up, and false when a functional report already states the action, observed result, expected result, and frequency") | ||
| var needsClarification: Bool | ||
| @Guide(description: "One exact excerpt from supplied text") |
There was a problem hiding this comment.
Summary can be removed
| @@ -755,37 +583,44 @@ private enum GeneratedFeedbackIssueCategory { | |||
|
|
|||
| enum FeedbackAnalysisPrompt { | |||
There was a problem hiding this comment.
We likely don't need this step, its content can be included in the TestFlight feedback. Doesn't need to be analyzed on the edge device.
|
Implemented the AndreasInk review feedback in 8952d74:
Verification:
The remaining misses are mostly over-questioning already-actionable feedback. The domain-context case also fails now because context is intentionally no longer sent to the edge model. |
|
Updated the branch to enforce at most one generated follow-up (commit 36af9f3). The original feedback is response 1; after the tester answers the single clarification, the conversation finalizes without invoking the model again. Existing response-3/4 notification routing remains readable for upgrade cleanup, but older stored conversations are finalized at the new limit.\n\nValidation:\n- swift build: passed\n- 51 deterministic tests: passed\n- iOS Simulator package build: passed\n- production-path Foundation Models eval: 20/29 = 69.0% (the eval intentionally remains red because its target is 100%)\n\nThe default eval set now includes only cases where production actually calls the model: the initial feedback before the one allowed clarification. |
1 participant
Summary
Verification
Remaining eval note
The local Foundation Model reports reasoning=false. After the schema experiments, later model-backed runs intermittently stalled in the system model service, so the complete 33-case model eval was not rerun successfully on this final commit. The demo logs whether the active device model supports reasoning so the final prompt can be debugged on iPhone.