Skip to main content
Streaming responses enable your agents to provide real-time feedback, making interactions feel more natural and responsive. Instead of waiting for the entire response, users see content as it’s generated. 💻 Code Examples

Event Streaming

Stream events with tool tracking

OpenAI Streaming

OpenAI-compatible chunks

Thinking Tokens

Stream reasoning content

Why Use Streaming?

Traditional (non-streaming) approach:
  • User waits for entire response
  • No feedback during processing
  • Can feel slow for long responses
Streaming approach:
  • Immediate visual feedback
  • See responses as they’re generated
  • Better user experience
  • Can see tool usage in real-time

Streaming Modes

Tyler supports multiple streaming modes through agent.stream(thread, mode=...): Use await agent.run(thread) for non-streaming execution when you want a completed AgentResult. The default streaming mode with full observability:

OpenAI Streaming (Advanced)

OpenAI mode is for advanced use cases. Tools ARE executed for full agentic behavior, but you only receive raw chunks (no ExecutionEvents).
Stream raw LiteLLM chunks for OpenAI compatibility:
When to use openai mode:
  • Building OpenAI API proxies or gateways
  • Direct integration with OpenAI-compatible clients
  • Minimal latency requirements (no transformation overhead)
How it works:
  • ✅ Tools ARE executed (fully agentic behavior)
  • ✅ Multi-turn iteration supported
  • ✅ Frontend sees finish_reason: "tool_calls" in chunks
  • ⚠️ No ExecutionEvent telemetry (only raw chunks)
  • ⚠️ Silent during tool execution (brief pauses expected)
  • ⚠️ Consumer must handle chunk formatting (SSE serialization)
Matches the pattern from OpenAI’s Agents SDK: Raw chunks → finish_reason=“tool_calls” → [agent executes tools] → more raw chunks → repeat SSE Serialization Example:
See examples/005_openai_streaming.py for a complete working example.

Vercel AI SDK Streaming

Perfect for React/Next.js frontends using @ai-sdk/react’s useChat hook.
Tyler supports the Vercel AI SDK Data Stream Protocol, enabling seamless integration with modern React frontends.
When to use Vercel mode:
  • Building React/Next.js chat interfaces with @ai-sdk/react
  • Direct integration with Vercel’s AI SDK ecosystem
  • Need SSE streams compatible with useChat hook
How it works:
  • ✅ Tools ARE executed (fully agentic behavior)
  • ✅ Thinking/reasoning tokens supported
  • ✅ Pre-formatted SSE strings ready for HTTP response
  • ✅ Compatible with x-vercel-ai-ui-message-stream: v1 protocol
FastAPI Integration Example:
React Frontend with @ai-sdk/react:
See examples/007_vercel_streaming.py for a complete working example.

Understanding Execution Events

ExecutionEvent objects provide detailed information about the agent’s execution:

Thinking Tokens (Reasoning Content)

Requires LiteLLM >= 1.63.0 and a reasoning-capable model like OpenAI o1 or Anthropic Claude with extended thinking.
Models like OpenAI o1 and Anthropic Claude can emit their reasoning process as separate “thinking tokens” alongside the response content. Tyler’s streaming API exposes these as dedicated LLM_THINKING_CHUNK events, allowing you to display reasoning separately from the final answer.

Why Use Thinking Tokens?

  • Transparency: Show users how the AI arrived at its answer
  • Debugging: Trace model reasoning for better agent development
  • UX: Display thinking in a collapsible section or different style
  • Trust: Users can verify the model’s reasoning process

Event Streaming with Thinking

Output:

Thinking in Message Object

After streaming completes, thinking content is stored as a top-level field on the message:

OpenAI Streaming with Thinking

OpenAI mode preserves all thinking fields from LiteLLM:

UI Pattern: Separated Display

A common pattern is showing thinking in a collapsible section:

Supported Models

OpenAI:
  • o1-preview (reasoning_content)
  • o1-mini (reasoning_content)
Anthropic:
  • claude-3-7-sonnet-20250219 with extended thinking
  • Future Claude models with thinking capability
Other Providers via LiteLLM:
  • Deepseek
  • XAI
  • Google AI Studio
  • Perplexity (Magistral models)
  • Groq
See LiteLLM docs for full list.

Backward Compatibility

Models without thinking support work unchanged - no LLM_THINKING_CHUNK events are emitted:
See packages/tyler/examples/006_thinking_tokens.py for complete working examples.

Streaming with Tools

See tool usage in real-time:

Building interactive applications

Terminal chat interface

Web application streaming

For web applications, you can stream to a WebSocket or Server-Sent Events:

Advanced Streaming Patterns

Progress indicators

Show progress for long-running tasks:

Buffered streaming

For smoother output, buffer chunks:

Cancellable streaming

Allow users to stop generation:

Streaming UI Components

Rich terminal UI

Using the rich library for better terminal output:

Token counting

Track tokens during streaming:

Performance tips

1. Chunk Size Optimization

Larger chunks reduce overhead but decrease responsiveness:

2. Async Processing

Process streams asynchronously for better performance:

3. Error Handling in Streams

Real-World Example: Live Research Assistant

Next steps

Slack Integration

Stream responses in Slack

Advanced Patterns

Complex streaming patterns