Agent

Custom LLMs

Run your agent's reasoning on your own model endpoint — a fine-tune, a self-hosted open-weights model, or a provider under your own contract — while Katexs handles telephony, tools, and orchestration.

Interface

Your endpoint must accept an OpenAI-compatible chat completions request with streaming enabled and tool calling supported. That is the only contract; anything that speaks it will work.

json

{
  "model": {
    "provider": "custom",
    "url": "https://llm.example.com/v1/chat/completions",
    "auth": { "type": "bearer", "secret_ref": "MY_LLM_KEY" },
    "model": "acme-support-8b",
    "temperature": 0.4,
    "max_tokens": 300,
    "stream": true,
    "supports_tools": true,
    "timeout_ms": 8000,
    "fallback_model": "katexs-fast"
  }
}

Request Katexs sends

json

{
  "model": "acme-support-8b",
  "stream": true,
  "messages": [
    { "role": "system", "content": "<compiled agent prompt + knowledge context>" },
    { "role": "user", "content": "Hi, I need to reschedule Thursday" }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "check_calendar",
        "description": "Find open appointment slots",
        "parameters": { "type": "object", "properties": { "date": { "type": "string" } } }
      }
    }
  ]
}

Response requirements

Requirement	Detail
Streaming	Server-sent events with delta chunks. Non-streaming responses add a full turn of latency.
First token	Under 400ms. Above 1s the conversation feels broken.
Tool calls	Emitted as tool_calls deltas with valid JSON arguments.
Stop	Terminate with finish_reason of stop or tool_calls.
Errors	Non-2xx or timeout triggers the fallback model.

Prompt compilation

Katexs compiles the agent prompt, retrieved knowledge base chunks, conversation behavior rules, and variable substitutions into the system message before sending. Do not re-inject your own system prompt server-side — you will get conflicting instructions and inconsistent tool use.

Choosing a model size

7–8B fine-tunes — Excellent for narrow, scripted flows: booking, qualification, FAQ. Fastest and cheapest.
30B class — Handles multi-step reasoning and messy callers. Watch first-token latency.
Frontier models — Best for open-ended support, but usually only worth it when a smaller model measurably fails in Testing Lab.

Validate before launch

1
Run the suite
Point Testing Lab at the agent and run your full scenario set against the custom endpoint.
2
Compare against baseline
Diff pass rate and latency against the same agent on a stock model.
3
Shadow one number
Route a single low-volume number to it for a day before switching everything.
4
Watch fallbacks
Any pipeline-error-llm above 0.5% means the endpoint is not ready for production traffic.

Keep fallback_model set even when your endpoint is stable. It converts an outage from dropped calls into slightly different-sounding calls.