Streaming Agent API (experimental)

Some of the LLM packages, like open-ai, has a streaming version of the Chat Complete function.

To use the streamed API, first call Chat Complete Streamed, and then call Next Event in a loop, to handle all streamed events.

The streamed API supports tool calls and structured output as well. However, unlike Chat Complete, it won't automatically call the LLM again after a set of tool calls is complete. You can do this yourself by appending the tool call results to the context, and then looping back to the Chat Complete Streamed, like so:

In the example above, we explicitly create a message list, instead of using the system message and user message inputs. This is just to make it easier to manage the context when we loop back after tool calls. Another approach would be to use two separate calls to chat complete, one that handles tool calls, and then one that summarizes the results.