Building AI voice agents that speak Hindi, English and Hinglish

Most small businesses in Bihar miss calls. The clinic receptionist is busy, the coaching centre closes at 7, the property dealer is on site. Every missed call is a missed customer. That's the problem I set out to solve with Cogneora: an AI agent that picks up the phone, in the language the caller actually speaks.

This post walks through how it works.

The pipeline

A phone conversation is a loop with a hard latency budget. The caller speaks, we understand, we decide what to say, and we speak back, ideally in well under a second. Cogneora's voice service is written in Python on Pipecat and chains three stages:

1. Speech-to-text: Sarvam's models, which handle Hindi, English and the mix of both that most callers use.
2. The LLM: a fast model served on Groq that decides what to say and when to call a tool.
3. Text-to-speech: Sarvam's voices, so replies sound natural in Indian languages.

Voice-activity detection (Silero) runs alongside, so callers can interrupt the agent mid-sentence, just as they would with a person.

Calls reach the service two ways: from the browser over WebRTC (for the "try it" demo), or from real phone numbers through Exotel, streamed over a WebSocket as 8 kHz audio.

Grounding answers in the business's own knowledge

An agent that makes things up is worse than voicemail. Each business uploads its documents and website once, and every agent (phone, website chat, follow-ups) answers from that shared knowledge base.

I built the retrieval layer without a framework:

• Embeddings come from a multilingual e5 model running in-process via ONNX. No external embedding API, and no extra cost per query.
• Hybrid search combines pgvector's HNSW index with Postgres full-text search, and merges both result lists with reciprocal rank fusion. Vector search catches meaning; keyword search catches names, prices and phone numbers.
• Website crawling is SSRF-safe, so a malicious URL can't make the server fetch internal addresses.

During a call, the voice service calls back into the API with tools such as search_knowledge, save lead and book appointment.

Every AI call has a price

Small businesses pay in rupees and want predictable bills. Cogneora meters every AI call:

• A model router picks a main and a backup model per feature, and falls back automatically if a provider fails.
• Before a call runs, credits are reserved; afterwards they're settled to the actual cost.
• Invoices are GST-compliant and generated as PDFs.

What I learned

• Latency is the product. Users forgive a slightly simpler answer; they don't forgive silence.
• Hinglish isn't an edge case in India. It's the default.
• Owning the retrieval stack (embeddings, hybrid search, citations) made quality problems debuggable instead of mysterious.

If you run a business that misses calls, or you're building voice AI for Indian languages, I'd love to talk.