Scenarios & Testing

Scenarios are automated multi-turn conversation tests. You define a sequence of user messages and the expected AI behaviour; Elaras runs the scenario and tells you whether the chatbot responded correctly. Use scenarios to catch regressions when you change your system prompt, knowledge base, or model configuration.

Types of scenarios

Chatbot scenarios

Creating a scenario

In the dashboard, go to Chatbot > Test > New scenario. Define a name and a sequence of turns:

curl -X POST "https://api.elaras.ai/api/developer/v1/chatbots/{chatbot_id}/scenarios" \
  -H "Authorization: Bearer sk_live_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Opening hours enquiry",
    "turns": [
      {
        "user": "What time do you open?",
        "expect": {
          "contains": ["9am", "Monday"]
        }
      }
    ]
  }'
Copy

Turn structure:

Running a scenario

curl -X POST "https://api.elaras.ai/api/developer/v1/chatbots/{chatbot_id}/scenarios/{scenario_id}/run" \
  -H "Authorization: Bearer sk_live_your_key_here"
Copy

Returns a run_id. Poll for the result:

curl "https://api.elaras.ai/api/developer/v1/chatbots/{chatbot_id}/scenario-runs/{run_id}" \
  -H "Authorization: Bearer sk_live_your_key_here"
Copy

The run result includes each turn, the actual AI response, and whether the expectations passed or failed.

Model scenarios

Model scenarios work identically to chatbot scenarios but are attached to a model rather than a chatbot. They are useful for evaluating prompt changes before publishing a new model version.

Drafting scenarios with AI

If you are not sure which scenarios to write, Elaras can draft them based on your system prompt and knowledge base:

curl -X POST "https://api.elaras.ai/api/developer/v1/models/{model_id}/scenarios/draft" \
  -H "Authorization: Bearer sk_live_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"count": 10}'
Copy

Returns a list of suggested scenarios covering common user intents. Review and save the ones that are relevant.

Comparing runs

After making a change to a model version, run all its scenarios and compare the results against a previous run:

curl "https://api.elaras.ai/api/developer/v1/models/{model_id}/scenario-runs/compare?run_a={run_id_a}&run_b={run_id_b}" \
  -H "Authorization: Bearer sk_live_your_key_here"
Copy

The comparison highlights turns where behaviour changed — useful for spotting unintended regressions alongside the intended improvements.

Listing and managing scenarios

# List chatbot scenarios
curl "https://api.elaras.ai/api/developer/v1/chatbots/{chatbot_id}/scenarios" \
  -H "Authorization: Bearer sk_live_your_key_here"

# Update a scenario
curl -X PUT "https://api.elaras.ai/api/developer/v1/chatbots/{chatbot_id}/scenarios/{scenario_id}" \
  -H "Authorization: Bearer sk_live_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"name": "Updated name", "turns": [...]}'

# Delete a scenario
curl -X DELETE "https://api.elaras.ai/api/developer/v1/chatbots/{chatbot_id}/scenarios/{scenario_id}" \
  -H "Authorization: Bearer sk_live_your_key_here"
Copy

CI integration

Run your scenarios as part of your deployment pipeline to catch regressions before publishing a new model version:

#!/bin/bash
set -e

# Trigger a scenario run
RUN=$(curl -s -X POST "https://api.elaras.ai/api/developer/v1/models/$MODEL_ID/scenarios/run-all" \
  -H "Authorization: Bearer $ELARAS_SECRET_KEY" | jq -r '.run_id')

# Poll until complete
while true; do
  STATUS=$(curl -s "https://api.elaras.ai/api/developer/v1/models/$MODEL_ID/scenario-runs/$RUN" \
    -H "Authorization: Bearer $ELARAS_SECRET_KEY" | jq -r '.status')
  [ "$STATUS" = "complete" ] && break
  [ "$STATUS" = "failed" ] && echo "Scenario run failed" && exit 1
  sleep 5
done

# Check results
PASSED=$(curl -s "https://api.elaras.ai/api/developer/v1/models/$MODEL_ID/scenario-runs/$RUN" \
  -H "Authorization: Bearer $ELARAS_SECRET_KEY" | jq '.passed')

[ "$PASSED" = "true" ] || { echo "Scenarios failed — blocking publish"; exit 1; }

echo "All scenarios passed"
Copy

Tips

• Write scenarios for your golden paths first. Cover the top 5–10 questions your chatbot is expected to answer correctly. These are your regression guard.
• Use not_contains for guardrails. Verify the AI is not saying things it should not — hallucinated prices, competitor names, or out-of-scope topics.
• Run scenarios after every knowledge base update. A resync can change how the AI answers questions if the source content changed significantly.
• Draft, then prune. Use AI-drafted scenarios as a starting point, but review them. Auto-generated tests can be vague — sharpen the expectations before relying on them in CI.