Event Streaming
Stream events with tool tracking
OpenAI Streaming
OpenAI-compatible chunks
Thinking Tokens
Stream reasoning content
Why Use Streaming?
Traditional (non-streaming) approach:- User waits for entire response
- No feedback during processing
- Can feel slow for long responses
- Immediate visual feedback
- See responses as they’re generated
- Better user experience
- Can see tool usage in real-time
Streaming Modes
Tyler supports multiple streaming modes throughagent.stream(thread, mode=...):
Use
await agent.run(thread) for non-streaming execution when you want a completed AgentResult.
Event Streaming (Recommended)
The default streaming mode with full observability:OpenAI Streaming (Advanced)
Stream raw LiteLLM chunks for OpenAI compatibility:- Building OpenAI API proxies or gateways
- Direct integration with OpenAI-compatible clients
- Minimal latency requirements (no transformation overhead)
- ✅ Tools ARE executed (fully agentic behavior)
- ✅ Multi-turn iteration supported
- ✅ Frontend sees
finish_reason: "tool_calls"in chunks - ⚠️ No ExecutionEvent telemetry (only raw chunks)
- ⚠️ Silent during tool execution (brief pauses expected)
- ⚠️ Consumer must handle chunk formatting (SSE serialization)
examples/005_openai_streaming.py for a complete working example.
Vercel AI SDK Streaming
Perfect for React/Next.js frontends using
@ai-sdk/react’s useChat hook.- Building React/Next.js chat interfaces with
@ai-sdk/react - Direct integration with Vercel’s AI SDK ecosystem
- Need SSE streams compatible with
useChathook
- ✅ Tools ARE executed (fully agentic behavior)
- ✅ Thinking/reasoning tokens supported
- ✅ Pre-formatted SSE strings ready for HTTP response
- ✅ Compatible with
x-vercel-ai-ui-message-stream: v1protocol
examples/007_vercel_streaming.py for a complete working example.
Understanding Execution Events
ExecutionEvent objects provide detailed information about the agent’s execution:Thinking Tokens (Reasoning Content)
Requires LiteLLM >= 1.63.0 and a reasoning-capable model like OpenAI o1 or Anthropic Claude with extended thinking.
LLM_THINKING_CHUNK events, allowing you to display reasoning separately from the final answer.
Why Use Thinking Tokens?
- Transparency: Show users how the AI arrived at its answer
- Debugging: Trace model reasoning for better agent development
- UX: Display thinking in a collapsible section or different style
- Trust: Users can verify the model’s reasoning process
Event Streaming with Thinking
Thinking in Message Object
After streaming completes, thinking content is stored as a top-level field on the message:OpenAI Streaming with Thinking
OpenAI mode preserves all thinking fields from LiteLLM:UI Pattern: Separated Display
A common pattern is showing thinking in a collapsible section:Supported Models
OpenAI:o1-preview(reasoning_content)o1-mini(reasoning_content)
claude-3-7-sonnet-20250219with extended thinking- Future Claude models with thinking capability
- Deepseek
- XAI
- Google AI Studio
- Perplexity (Magistral models)
- Groq
Backward Compatibility
Models without thinking support work unchanged - noLLM_THINKING_CHUNK events are emitted:
packages/tyler/examples/006_thinking_tokens.py for complete working examples.
Streaming with Tools
See tool usage in real-time:Building interactive applications
Terminal chat interface
Web application streaming
For web applications, you can stream to a WebSocket or Server-Sent Events:Advanced Streaming Patterns
Progress indicators
Show progress for long-running tasks:Buffered streaming
For smoother output, buffer chunks:Cancellable streaming
Allow users to stop generation:Streaming UI Components
Rich terminal UI
Using therich library for better terminal output:
Token counting
Track tokens during streaming:Performance tips
1. Chunk Size Optimization
Larger chunks reduce overhead but decrease responsiveness:2. Async Processing
Process streams asynchronously for better performance:3. Error Handling in Streams
Real-World Example: Live Research Assistant
Next steps
Slack Integration
Stream responses in Slack
Advanced Patterns
Complex streaming patterns