Building a WhatsApp voice bot that answers on its own

Text bots are a solved problem. Voice notes are where it gets interesting, because a large share of WhatsApp messages in many markets are voice, and a bot that ignores them ignores most of its inbox.

This is the shape of a bot I built to handle both, and the parts that turned out to matter.

The pipeline

1. 1A webhook receives the message from the WhatsApp Business API. Voice notes arrive as a media id, not as audio.
2. 2Fetch the media, which is a second authenticated request against the media endpoint.
3. 3Transcribe the audio to text with a speech recognition model.
4. 4Load the recent conversation for that sender so the reply has context.
5. 5Send transcript plus context to the language model and get a reply.
6. 6Post the reply back through the API, and store the exchange.

Six steps, and every one of them can fail independently. That is the real design problem, not the model call.

Acknowledge before you process

WhatsApp expects a fast response to the webhook. Transcription plus a model call does not fit inside that window. Answer the webhook immediately and do the work in the background.

@app.post("/webhook")
async def webhook(request: Request, tasks: BackgroundTasks):
    payload = await request.json()
    message = extract_message(payload)
    if message:
        tasks.add_task(handle_message, message)
    # Return now. WhatsApp retries anything slow, and a retry
    # means the same voice note gets answered twice.
    return {"status": "ok"}

Context is what makes it feel like a conversation

Without history the bot answers every message as if it were the first. Someone asks about pricing, then follows up with a question about delivery, and the bot has no idea what is being delivered.

Keep the last handful of turns per sender and pass them with each request. Cap it. An unbounded history grows the prompt until it is slow and expensive, and the oldest turns rarely help.

MAX_TURNS = 8

def build_messages(history, transcript, system_prompt):
    return [
        {"role": "system", "content": system_prompt},
        *history[-MAX_TURNS:],
        {"role": "user", "content": transcript},
    ]

Transcription quality decides everything downstream

The model can only answer the transcript it is given. Voice notes are recorded in noisy places, in mixed languages, at speed. Two things helped more than any prompt engineering.

• Pass a language hint when you know the sender's locale. Guessing the language on a short clip is where most errors come from.
• Give the transcriber a short vocabulary of product names and terms it will not know. Brand names are the words most often mangled, and the ones a reply most needs to get right.

Know when to stop

The most useful thing an automated agent does is recognise what it should not answer. Route anything about refunds, complaints, or anything where the model is not confident straight to a human, and tell the sender that is what is happening.

A bot that answers eight questions out of ten and hands over the other two cleanly is worth more than one that attempts all ten and gets two of them wrong in public.