# AI Voice Agent for Intelligent Call Delegation System

A privacy-first voice agent that conducts real-time phone calls using an entirely local speech and language pipeline, removing per-call inference cost while sustaining conversational latency on CPU-only hardware.

- Role: Team Lead · AI Systems & Backend Engineer (team project)
- Date: Jan 2026 – May 2026
- Categories: project, hackathon
- Tech: Python, FastAPI, Ollama, faster-whisper, Piper TTS, Twilio, RAG, NLTK
- GitHub: https://github.com/AmulyaJain2004/AI-Voice-Calling-Assistant
- Demo: https://drive.google.com/file/d/1vVrGSdgSwzkBOA-jc9plIGaviiQhFFwx/view?usp=sharing

## Overview

Automated call handling usually depends on cloud speech and language APIs, which introduces recurring per-call cost and sends caller audio to third parties. This project investigates whether a conversational voice agent can instead run entirely on local infrastructure while remaining responsive enough for natural telephone dialogue.

## System Design

The agent is organised as a five-stage streaming pipeline: voice activity detection (VAD), speech-to-text (STT), language-model response generation, text-to-speech (TTS), and telephony transport.

- **Speech recognition:** faster-whisper transcribes caller audio locally.
- **Language model:** a small Llama 3.2 model served through Ollama generates responses.
- **Speech synthesis:** Piper TTS renders replies with low synthesis delay.
- **Telephony:** Twilio Media Streams connects the pipeline to live phone calls, served by a FastAPI backend.
- **Turn-taking:** energy-based voice activity detection manages turns and supports barge-in, so the caller can interrupt the agent naturally.

```mermaid
flowchart TD
    Caller(["Caller"]) <-->|"phone call"| Twilio["Twilio Media Streams"]
    Twilio <-->|"audio stream"| API["FastAPI backend"]
    API --> VAD["Voice activity detection<br/>(energy-based)"]
    VAD -->|"caller turn"| STT["Speech-to-text<br/>faster-whisper"]
    STT --> LLM["Response generation<br/>Llama 3.2 via Ollama"]
    Memory[("3-layer memory<br/>ground truth · caller profile · TF-IDF")] <--> LLM
    LLM --> TTS["Text-to-speech<br/>Piper TTS"]
    TTS -->|"reply audio"| API
    VAD -. "barge-in: caller speaks, playback stops" .-> TTS
```

## Memory and Retrieval

Response relevance is improved by a three-layer memory system:

1. **Ground-truth knowledge** that acts as the authoritative source of facts.
2. **Caller profile**, extracted with regular expressions and stored as structured JSON.
3. **Sparse retrieval** over earlier conversation content using TF-IDF.

## Results

The pipeline achieves an end-to-end response time of roughly **1.5 to 2.5 seconds** on CPUs, without any per-call inference cost.

## Hackathon

I continued building the agent at **Build4Bharat: Hackathon 9.0**, organised by the UPES CSI Student Chapter in March 2026, where the team placed in the **top 25** of 40–50 participating teams.

## My Contribution

This was a group project, and I was the team lead. Alongside coordinating the team, I developed the end-to-end pipeline and its integrations, together with the backend and AI components that connect speech, language and telephony stages.

---

Back to the full index: /agent