The problem
A voice agent on a real phone line has no margin for the two standard LLM failure modes. Latency: a caller hangs up during a five-second pause. Hallucination: an agent inventing an order status on a live call is a business liability, not a demo quirk. Velox AI is built for both constraints — and for the multi-tenant, metered-billing reality of running this as a product.
Architecture
A call arrives at a Twilio number and upgrades to a WebSocket media stream — but only after the server checks the org's remaining minutes, rejecting the handshake with 402 before any audio flows. Deepgram Nova-2 transcribes in real time; an orchestrator assembles system prompt, RAG results, and conversation history; and a Python agent pipeline (Google ADK) routes by difficulty — an optional local Phi-3-mini sidecar for simple turns, Gemini 2.5 Flash for the middle, Gemini Pro for complex queries. Tool calls (order lookup, inventory, calendar booking, CRM, human handoff) execute through a typed registry. The reply streams back through Deepgram Aura or ElevenLabs TTS as 8kHz mulaw. The README's design target for the full turn is under two seconds on Gemini 2.5 Flash.
Grounding is a five-layer RAG system: DSPy prompt optimization (GEPA/MIPROv2), GraphRAG plus hybrid dense-sparse retrieval, agentic patterns (Self-RAG, Corrective RAG, Adaptive RAG), LangGraph multi-agent validation, and guardrails enforcing citations with calibrated abstention — the agent is designed to say "I don't know" below a confidence threshold rather than improvise.
What's measured
Prometheus histograms track LLM, TTS, and end-to-end turn latency per model and provider; counters track call outcomes. Every call writes cost, sentiment, and duration to Postgres. LangFuse traces STT confidence, prompts, and latencies span by span; a DeepEval suite logs to MLflow via an admin endpoint. Billing is enforced, not estimated: Stripe metering gates call minutes per tenant.
Honest limitations
Sub-2-second is the engineering target the latency histograms exist to police — no public p95 benchmark is published, so none is quoted. The five anti-hallucination layers reduce risk; the abstention threshold trades coverage for safety and is tenant-tunable, not proven optimal.