Data & Testing

Automated Testing

Scripted regression suites that drive your agents the way real callers do, assert on what matters, and fail a CI build before a bad prompt reaches a phone number.

The model

A suite is a set of scenarios. A scenario is a simulated caller with a persona, a script of intents, fixture data, and a list of assertions. Katexs runs the scenario against a specific agent version in a sandbox — no phone number, no carrier cost, no live tools unless you opt in.

Concept	What it is
Scenario	One simulated conversation with a goal and assertions.
Persona	How the simulated caller behaves: terse, rambling, accented, interrupting.
Fixture	The variable bag and mocked tool responses for that run.
Assertion	A checkable claim about the transcript, variables, tools, or outcome.
Suite	A named group of scenarios run together and reported as one result.

Write a scenario

yaml

scenario: new_patient_books_appointment
agent: agt_7d11c9
persona:
  style: cooperative
  speech_rate: normal
  interruptions: false
fixtures:
  variables:
    business_name: "Northside Dental"
  tools:
    calendar.find_availability:
      returns: { slots: ["2026-08-11T14:00:00-04:00", "2026-08-11T16:30:00-04:00"] }
    calendar.create_event:
      returns: { id: "apt_test_001", status: "confirmed" }
script:
  - caller: "Hi, I'd like to book a cleaning."
  - caller: "I'm a new patient. Dana Whitfield."
  - caller: "Tuesday afternoon works."
  - caller: "Yes, that's perfect."
assertions:
  - outcome_is: booked
  - tool_called: calendar.create_event
  - variable_equals: { caller_name: "Dana Whitfield" }
  - transcript_contains: "confirmation"
  - transcript_excludes: ["I'm not sure", "as an AI"]
  - turns_under: 12
  - duration_under_seconds: 120

Assertion types

Assertion	Checks	Use it for
outcome_is	Terminal outcome tag	The scenario ended the way it should
tool_called / tool_not_called	Tool invocation with optional argument match	Side effects happened exactly once
variable_equals / variable_matches	Collected data	Extraction accuracy
transcript_contains / excludes	Literal or regex on agent turns	Required disclosures, banned phrases
handoff_to	Which Team member or human took over	Routing correctness
rubric	Model-graded judgement against written criteria	Tone, empathy, clarity
turns_under / duration_under_seconds	Efficiency	Preventing prompt bloat

Use rubric assertions sparingly and always alongside deterministic ones. A model grader is useful for tone and useless as your only gate on whether a booking was created.

Personas worth keeping

The interrupter — Talks over the agent mid-sentence. Catches barge-in and turn-taking regressions.
The mumbler — Low-confidence transcription. Verifies your reprompt and confirmation logic.
The off-topic caller — Asks something the agent should refuse or hand off. Guards scope creep.
The adversary — Tries prompt injection and requests for other customers' data. Should always fail to get either.
The silent line — Says nothing. Confirms your no-input fallback and end reason.

Run a suite

bash

curl -X POST https://api.katexs.com/v1/test-runs \
  -H "Authorization: Bearer $KATEXS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "suite": "suite_regression_core",
    "agent_id": "agt_7d11c9",
    "version": 42,
    "parallelism": 8,
    "fail_fast": false
  }'

json

{
  "run_id": "run_5c1a09",
  "status": "completed",
  "passed": 27,
  "failed": 2,
  "duration_seconds": 94,
  "failures": [
    {
      "scenario": "insurance_out_of_network",
      "assertion": "transcript_contains",
      "expected": "out of network",
      "actual": "we accept most plans",
      "transcript_url": "https://app.katexs.com/test-runs/run_5c1a09/insurance_out_of_network"
    }
  ]
}

Wire it into CI

The run endpoint returns a non-zero failure count you can gate on. Combine it with promotion's require_passing_suite so a green build is a precondition for reaching PROD, not a nice-to-have.

yaml

name: agent-regression
on:
  pull_request:
  schedule:
    - cron: "0 6 * * 1-5"
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run Katexs suite
        env:
          KATEXS_API_KEY: ${{ secrets.KATEXS_UAT_KEY }}
        run: |
          npx @katexs/cli test run \
            --suite suite_regression_core \
            --agent agt_7d11c9 \
            --fail-on-any \
            --junit results.xml
      - uses: actions/upload-artifact@v4
        with: { name: katexs-results, path: results.xml }

Practices that keep suites useful

Mock tools by default — Live tools make runs slow, flaky, and occasionally create real appointments. Keep one nightly suite against sandbox credentials for integration coverage.
Turn every incident into a scenario — A production failure that becomes a test never happens twice.
Keep the core suite under two minutes — A slow suite gets skipped. Push exhaustive coverage into the nightly run.
Pin the version — Always test the exact version you intend to promote, not the draft.
Review failures by transcript — The transcript URL in the failure payload shows the whole conversation, not just the failed line.

Enterprise environments Testing Lab