refactor(sdk): type LLM streaming, message, and tokenizer boundaries

[vedjoshi1]

HUMAN:

---

AGENT:

Why

Keep provider shape variation at typed LLM boundaries and correct the case where
a streaming wrapper's stale completed_response=None erases a yielded completion.

Summary

• Add private stream/event and tokenizer capability adapters; retain plain
  sync/async generators, optional Transformers, counting fallbacks, and output reconstruction.
• Normalize optional LiteLLM metadata and Responses output objects/mappings before
  core conversion. LiteLLM declares these fields but deletes them when absent.
• Use direct authentication config access and remove 14 baseline exceptions
  (98 → 84), without dependency, persisted-schema, routing, or telemetry changes.

Issue Number

Fixes #4976. Completion precedence follows alanhuangyoo's closed, unmerged #4772;
the new regression matrix builds on that report and credits it in the commit.

How to Test

make build
uv run pytest tests/sdk/llm/ tests/cross/test_check_forbidden_dynamic_attributes.py -q
uv run pytest tests/sdk/ -q
uv run pre-commit run --all-files
uv run python scripts/check_forbidden_dynamic_attributes.py --baseline-ref upstream/main
uv run python .pr/local_transport_smoke.py
uv run python .pr/server_binary_smoke.py
uv run python .pr/tokenizer_smoke.py
uv run --isolated --frozen --with transformers==5.17.0 python .pr/tokenizer_smoke.py

Local results: 6,432 SDK tests passed (9 skipped, 19 deselected, 12 xfailed);
1,073 LLM/checker tests passed, including the no-callback reconstruction cases. Whole-repository
pre-commit/Pyright checks passed. The stream regression matrix first reproduced
six failures before the fix, then passed all 15 cases.

Beyond unit tests, all six loopback HTTP/SSE cases passed through the actual SDK
and LiteLLM parsers (sync/async Responses, non-streaming and streaming; sync/async
Chat Completions streaming). Final output, callback text, stable item IDs, and
empty-output reconstruction were asserted. This is local integration evidence,
not a live-provider test.

Token counting passed without Transformers and with real Transformers 5.17.0
loading a local tokenizer (exact count: 2); no model downloads or PyTorch required.

See .pr/verification.md for complete commands and environmental limitations.
The first broad test/build overlap exhausted local disk space; the separate SDK
rerun passed. make build-server succeeded when run separately, producing the
macOS arm64 server executable with all three new private modules included. The
binary's --help worked and its loopback /health endpoint returned HTTP 200.

Video/Screenshots

No UI changes. Reproducible transport and tokenizer smoke scripts, with results,
are under .pr/.

Design Doc

Before/after interfaces and compatibility decisions

Type

• Bug fix
• Feature
• Refactor
• Breaking change
• Docs / chore

Notes

Companion documentation: OpenHands/docs#825.
The human author must complete the reserved HUMAN: section before this draft
is marked ready for review.

[github-actions]

📁 PR Artifacts Notice

This PR contains a .pr/ directory with temporary PR-specific documents. Because this is a fork PR, the workflow will open or update a cleanup PR against main after merge.