Behavior when chunk type check not passed

[johnd0e]

Consider such (real) api response:

data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}],"usage":null}

data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[{"index":0,"delta":{"content":" "},"finish_reason":null}]}

data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[{"index":0,"delta":{"content":" Hello!"},"finish_reason":null}]}

data: {"id":"chatcmpl-7ZZCd6bd5DwGT9UdoK9Ph4","model":"llama2-70b","choices":[],"usage":{"prompt_tokens":11,"total_tokens":15,"completion_tokens":4}}

data: [DONE]

The response objects are missing "object": "chat.completion.chunk", so they aren't passing the check here:

lua-openai/openai/init.moon

Lines 253 to 254 in 08128b9

	if chunk = parse_completion_chunk cjson.decode json_blob
	chunk_callback chunk

This leads to to some consequences:

1. Json is valid and decodes without problems.
2. However parse_completion_chunk doesn't accept it.
3. As a result, it's simply ignored.
4. And chunk_callback is never invoked.

It might be more sensible to throw an error instead of just disregarding it.
Or alternatively, invoke chunk_callback regardless, but with an additional isInvalid argument.

The latter is more flexible.

[johnd0e]

Upon closer inspection, I understand that parse_completion_chunk does more than just verification. It extracts the content and index fields from the entire structure.

I find this problematic because other fields are equally significant! For instance, I require the usage and finish_reason fields.

Moreover, the absence of "object": "chat.completion.chunk" is not the sole reason for parse_completion_chunk failing to capture a chunk; sometimes, the content is missing as well.

Therefore, my proposed solution is not to utilise parse_completion_chunk at all, but to relay the decoded object to the callback as is: main...johnd0e:lua-openai:no-parse

parse_completion_chunk can be provided also, in the API object, to use when needed.

[leafo]

This is partially addressed. The stream handlers in the raw clients now just pass all SSE events with no changes made.

The stream handlers in the ChatSession classes still parse and only pass through the text chunk events. These handlers now pass a second argument of the raw unprocessed event, though, if there's additional information there you need to alter.

In a future update I plan to make the chat session classes have customizable stream event filter function so you can control what gets sent through.