Nine States for One Phone Call

Call the Agent

The quickest explanation is the working loop itself. Speak Greek into your browser mic; replies come back as PCM audio over one media WebSocket. This is the session code the article describes, with a mocked calendar and bookings. No SMS is sent. It runs on my keys, so sign in for two calls of up to two minutes each. One live call at a time.

Context

Narusec builds a Greek voice agent for call centers. The first job we gave it was to answer a call and book an appointment. We assembled the pipeline from parts: Soniox stt-rt-v5 for transcription, GPT-5.6 Luna on OpenAI's Fast tier for replies, and Gemini 3.1 Flash TTS for speech. That's the configuration I selected below. The browser demo uses a telephony-style media envelope; connecting a phone provider still needs a transport adapter.

The test business is a fictional Athens barbershop. The agent greets you as ο ψηφιακός βοηθός της Ρούλας, checks a fixed set of slots, and returns a mock booking confirmation. Keeping availability fixed makes it easier to compare conversations across runs that each cost real API money.

Choosing Parts With a Stopwatch

Before settling on the model, I built a harness around the real APIs. Each saved run used the same prompt, tools, date, and Greek script for every candidate: transcription repair, ordinary conversation, prompt injection, a fake booking claim, calendar-and-booking flow, and repeated abuse. The script made up to ten LLM requests; failed tool gates skipped later steps. Providers and tiers were pinned, with routing fallback disabled. Each result kept transcripts, tool calls, usage, timings, and hashes of the prompt, tools, and model config.

Latency came first but never got the final vote. GPT-OSS on Groq reached visible text in a median 189 ms and cost $0.00174 across nine LLM requests, yet passed only 7 of 14 checks. Grok took a median 5.6 seconds and had the most expensive run. I put the saved conversations into a blind packet with model names, providers, latency, and cost removed. An agent reviewed the transcripts for naturalness, coherence, repair, tact, suitability for speech, and tool transitions. Groq scored 31/100, Cerebras 39, and Grok 55. Speed alone hadn't bought a usable conversation.

An earlier August 28 round had picked Gemini 3.1 Flash-Lite: 452 ms median first text, 555 ms to complete the LLM response, and 12 of 15 hard checks. The broader August 29 field put both Luna configurations at the top of the transcript review, though both still failed to end the call after repeated harassment. Their one-point dialogue gap was too small to decide on in one review. Reasoning none reached text 162 ms sooner at the median and 184 ms sooner at P90, with lower cost. That was enough to break the tie for this prototype.

The Live Route

This A/B followed the model decision and compared transport and service tier. Fast reported the expected priority tier and reached median first text in 492 ms. It passed 12 of 15 hard checks where the other routes passed 13. One run couldn't establish a quality difference between tiers, so I chose Fast for the lower observed latency. Abuse escalation remained unresolved in all three: the live session still relies on the model to request end_call. A deterministic escalation policy hasn't been implemented.

Architecture

The sentence buffer lets the agent start talking before a slower answer is complete. Once the first meaningful sentence is ready, I allow 150 ms for the rest of the reply to arrive. If it finishes in that window, the whole reply becomes one TTS request. Otherwise the lead goes first and the remaining text becomes one second request when generation finishes. Short acknowledgements wait for more text while the reply is still arriving. The two-request limit applies to each generation; a tool result can start another one.

The State Machine

The first version was an imperative loop with a 400 ms setTimeout debounce, booleans, and a generation counter for stale work. Timing fixes kept adding flags. I moved the control flow into nine XState states, with explicit transitions for accepting caller text, interrupting a reply, and ending the session. The old loop remains as call-session.old.ts in the original Narusec repo. The new machine still needs a generation counter and a playback clock; the states make their transitions easier to follow.

The playback clock is an estimate. Each outgoing PCM chunk adds its duration to playback.until, starting from the later of now or the previous queue end. The browser schedules audio on its own clock and sends no playback acknowledgement. I use the server estimate to wait for queued speech and distinguish a caller continuing before audio starts from someone interrupting it. Transcript guards and confirmation protection still decide whether the interruption is accepted.

Getting Interrupted

Not every sound is an interruption. During greeting or speaking, a Soniox partial with three speech characters is enough, so ναι and όχι can cut playback before a final transcript arrives. During generating, the current partial threshold is four characters, even if audio is already flowing. Finals need at least three speech characters and confidence above 0.7. These checks count letters and digits and reject filler sounds such as μμμ and εεε.

When an interruption lands, I clear the browser's audio queue, invalidate the old generation, and annotate the assistant history with the text prepared for synthesis so far. That list can include words the caller never heard: it has no sentence timings or playback acknowledgements. The note gives the next response interruption context, with an approximate account of the interrupted reply.

The Booking Guard

Before text reaches TTS, a lexical guard checks known Greek booking phrases such as κλείστηκε and επιβεβαιώθηκε, with a check for negation. Without a successful book_appointment result, a detected claim is suppressed and the machine gives the model one retry. The guard covers those phrases; it can miss a different wording and doesn't compare the spoken date, time, or name against the tool result.

After a successful mock booking, the confirmation turn blocks speech-triggered barge-in while audio is estimated to be playing. The booking result already exists by then; this gives its readback room to finish. There's a tradeoff: spoken corrections during that protected audio are ignored. The caller can still hang up at any point.

Ending a Call

An agent-requested ending waits for queued speech. The end_call tool sends the session back to speaking with an end request; once estimated playback finishes, ending waits 200 ms before cleanup. A caller stop or disconnect reaches cleanup directly from whichever state is active.

Putting the Call Together

One caller turn can take several passes through Respond: Luna may request a tool, read its result, and continue the reply. Gemini streams speech as text becomes ready. After playback, the call returns to Listen; an accepted interruption gets there sooner, with the caller's new text retained.

What a Call Costs

At hangup, I store a per-call API cost estimate in SQLite. Soniox supplies processed audio time. For the LLM, I use reported cost where available or price reported tokens at configured rates; missing usage falls back to a marked estimate. Cloud TTS uses a local count of submitted text, including its style prompt, and an estimated character rate. The browser fetches the STT, LLM, and TTS breakdown after disconnect.

The summary is a snapshot taken when cleanup starts, so a cut-off call can miss usage that arrives afterward. Telephony and hosting charges are outside it. I use the breakdown to watch API spend while tuning the conversation, with those limits visible.