Voice Agent Framework for Conversational AI

A blueprint notebook showcasing Pipecat AI and NIM in the creation of an AI voice agent. It uses the meta/llama-3.3-70b-instruct LLM model and Riva for STT & TTS. This a launchable on the brev platform.

Pipecat AI is an open-source framework for building voice and multimodal conversational agents. Pipecat simplifies the complex voice-to-voice AI pipeline, and lets developers build AI capabilities easily and with Open Source, commercial, and custom models. The framework was developed by Daily, a company that has provided real-time video and audio communication infrastructure since 2016. It is fully vendor neutral and is not tightly coupled to Daily's infrastructure.

Below is the architecture diagram of the system running on brev:

This notebook can be run on brev or on a local machine.

Run the notebook on brev

Launch from here

Run the notebook locally

Setup JupyterLab

python3.12 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python -m ipykernel install --user --name=venv --display-name "Python3.12"

Setup environment

Add NVIDIA API key to .env.

cp example.env .env
# ... edit .env

Run the Jupyter Notebook

python -m jupyter notebook

Navigate to http://localhost:8888/notebooks/001-hello-pipecat-nim.ipynb

Extras

Run in a command line environment

For convenience, a standalone pipecat can be found here. Edit the system prompt in a separate file in ./prompts/ and then update the prompt.txt symlink.

source .env
python3.12 -m venv venv
source venv/bin/activate
pip install "pipecat-ai[daily,openai,riva,silero]" noaa_sdk python-dotenv
python 001-hello-pipecat-nim.py

Pipecat-AI links and resources

• Pipecat Repo
• Pipecat docs