# Scenarios & Testing

Scenarios are automated multi-turn conversation tests. You define a sequence of user messages and the expected AI behaviour; Elaras runs the scenario and tells you whether the chatbot responded correctly. Use scenarios to catch regressions when you change your system prompt, knowledge base, or model configuration.

## Types of scenarios

| Type | Attached to | Use for |
|---|---|---|
| **Chatbot scenarios** | A specific chatbot | Testing the full stack — model, knowledge, skills, and flows |
| **Model scenarios** | A model | Testing prompt changes in isolation before assigning the model to a chatbot |

## Chatbot scenarios

### Creating a scenario

In the dashboard, go to **Chatbot > Test > New scenario**. Define a name and a sequence of turns:

```bash
curl -X POST "https://api.elaras.ai/api/developer/v1/chatbots/{chatbot_id}/scenarios" \
  -H "Authorization: Bearer sk_live_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Opening hours enquiry",
    "turns": [
      {
        "user": "What time do you open?",
        "expect": {
          "contains": ["9am", "Monday"]
        }
      }
    ]
  }'
```

**Turn structure:**

| Field | Description |
|---|---|
| `user` | The message the simulated user sends |
| `expect.contains` | Strings that must appear somewhere in the AI response |
| `expect.not_contains` | Strings that must not appear in the AI response |

### Running a scenario

```bash
curl -X POST "https://api.elaras.ai/api/developer/v1/chatbots/{chatbot_id}/scenarios/{scenario_id}/run" \
  -H "Authorization: Bearer sk_live_your_key_here"
```

Returns a `run_id`. Poll for the result:

```bash
curl "https://api.elaras.ai/api/developer/v1/chatbots/{chatbot_id}/scenario-runs/{run_id}" \
  -H "Authorization: Bearer sk_live_your_key_here"
```

The run result includes each turn, the actual AI response, and whether the expectations passed or failed.

## Model scenarios

Model scenarios work identically to chatbot scenarios but are attached to a model rather than a chatbot. They are useful for evaluating prompt changes before publishing a new model version.

### Drafting scenarios with AI

If you are not sure which scenarios to write, Elaras can draft them based on your system prompt and knowledge base:

```bash
curl -X POST "https://api.elaras.ai/api/developer/v1/models/{model_id}/scenarios/draft" \
  -H "Authorization: Bearer sk_live_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"count": 10}'
```

Returns a list of suggested scenarios covering common user intents. Review and save the ones that are relevant.

### Comparing runs

After making a change to a model version, run all its scenarios and compare the results against a previous run:

```bash
curl "https://api.elaras.ai/api/developer/v1/models/{model_id}/scenario-runs/compare?run_a={run_id_a}&run_b={run_id_b}" \
  -H "Authorization: Bearer sk_live_your_key_here"
```

The comparison highlights turns where behaviour changed — useful for spotting unintended regressions alongside the intended improvements.

## Listing and managing scenarios

```bash
# List chatbot scenarios
curl "https://api.elaras.ai/api/developer/v1/chatbots/{chatbot_id}/scenarios" \
  -H "Authorization: Bearer sk_live_your_key_here"

# Update a scenario
curl -X PUT "https://api.elaras.ai/api/developer/v1/chatbots/{chatbot_id}/scenarios/{scenario_id}" \
  -H "Authorization: Bearer sk_live_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"name": "Updated name", "turns": [...]}'

# Delete a scenario
curl -X DELETE "https://api.elaras.ai/api/developer/v1/chatbots/{chatbot_id}/scenarios/{scenario_id}" \
  -H "Authorization: Bearer sk_live_your_key_here"
```

## CI integration

Run your scenarios as part of your deployment pipeline to catch regressions before publishing a new model version:

```bash
#!/bin/bash
set -e

# Trigger a scenario run
RUN=$(curl -s -X POST "https://api.elaras.ai/api/developer/v1/models/$MODEL_ID/scenarios/run-all" \
  -H "Authorization: Bearer $ELARAS_SECRET_KEY" | jq -r '.run_id')

# Poll until complete
while true; do
  STATUS=$(curl -s "https://api.elaras.ai/api/developer/v1/models/$MODEL_ID/scenario-runs/$RUN" \
    -H "Authorization: Bearer $ELARAS_SECRET_KEY" | jq -r '.status')
  [ "$STATUS" = "complete" ] && break
  [ "$STATUS" = "failed" ] && echo "Scenario run failed" && exit 1
  sleep 5
done

# Check results
PASSED=$(curl -s "https://api.elaras.ai/api/developer/v1/models/$MODEL_ID/scenario-runs/$RUN" \
  -H "Authorization: Bearer $ELARAS_SECRET_KEY" | jq '.passed')

[ "$PASSED" = "true" ] || { echo "Scenarios failed — blocking publish"; exit 1; }

echo "All scenarios passed"
```

## Tips

- **Write scenarios for your golden paths first.** Cover the top 5–10 questions your chatbot is expected to answer correctly. These are your regression guard.
- **Use `not_contains` for guardrails.** Verify the AI is not saying things it should not — hallucinated prices, competitor names, or out-of-scope topics.
- **Run scenarios after every knowledge base update.** A resync can change how the AI answers questions if the source content changed significantly.
- **Draft, then prune.** Use AI-drafted scenarios as a starting point, but review them. Auto-generated tests can be vague — sharpen the expectations before relying on them in CI.