How-to Guides
Set Simulation Size
Learn how to choose the right number of personas, conversations, and conversation length for effective testing.
Simulation size directly impacts both the diversity of your testing and your simulation costs. Getting the right balance ensures comprehensive coverage without unnecessary expense.
Run your first simulation, review the results, then adjust based on what you learn:
Questions? Join our developer community or contact support for help choosing the right simulation size.
Three Simulation Size Parameters
Each persona represents a distinct user with unique style, goals, and behavior patterns. Conversations with the same persona will feel like the same type of person is talking.
The maximum number of total conversations to generate. The actual number might be slightly under this, but typically very close.
The maximum number of turns (back-and-forth exchanges) in each conversation. Actual conversations will vary from 3 turns up to your maximum, determined by how the persona and your chatbot interact.
These are maximums - actual conversation lengths and counts may be lower based on natural conversation flow.
Sizing Guidelines
Small Volume (Under 100 conversations)
Maximize diversity with a 5:1 ratio of conversations to personas.- 25 conversations → 5 personas
- 50 conversations → 10 personas
- 75 conversations → 15 personas
Medium Volume (100-500 conversations)
Maintain ratio between 5:1 and 20:1 based on your goals.- Closer to 5:1 for maximum diversity
- Closer to 20:1 for deeper testing of each persona type
- 200 conversations → 10-40 personas
- 300 conversations → 15-60 personas
Large Volume (Over 500 conversations)
Don’t exceed 20:1 ratio (20 conversations per persona).- 500 conversations → at least 25 personas
- 1000 conversations → at least 50 personas
- 2000 conversations → at least 100 personas
Common Testing Scenarios
Quick Feature Test
Quick Feature Test
Goal: Test a specific feature with diverse user typesRecommended Settings:
- 25-50 conversations
- 5-10 personas
- 8-12 max turns (medium length for multi-step processes)
Comprehensive Chatbot Evaluation
Comprehensive Chatbot Evaluation
Goal: Thorough testing across all chatbot capabilitiesRecommended Settings:
- 200-500 conversations
- 20-25 personas
- 12-20 max turns (medium to long for consultative interactions)
Edge Case Discovery
Edge Case Discovery
Goal: Find unusual failure modes and stress-test robustnessRecommended Settings:
- 100-200 conversations
- 10-15 personas
- 15-25 max turns (long conversations reveal complex issues)
Regression Testing
Regression Testing
Goal: Ensure recent changes don’t break existing functionalityRecommended Settings:
- 50-100 conversations
- 10-15 personas
- Match your typical conversation length (3-8 turns for simple Q&A, 15+ for complex workflows)
T-shirt Sizing
Not sure what size to pick? Start with these defaults:| Size | Conversations | Personas | Max Turns |
|---|---|---|---|
| Small | 100 | 10 | 5 |
| Medium | 250 | 25 | 7 |
| Large | 1500 | 75 | 12 |
- Need more diversity? Increase personas.
- Missing edge cases? Increase total conversations.
- Conversations ending too early? Increase max turns.
Resources
Join the Community
Connect with other developers, ask questions, and share feedback.
See Examples
See real simulation examples with different sizing strategies.
Write Chatbot Description
Learn how to write a chatbot description that supports your simulation intent effectively.
Define Simulation Intent
Learn how to define what you want to test with your simulations.
Questions? Join our developer community or contact support for help choosing the right simulation size.