OpenAI Realtime API + Twilio Voice Integration

A powerful integration that combines OpenAI's Realtime API with Twilio's Voice services to create interactive voice calls with AI responses. This project enables real-time voice conversations between humans and AI, making it perfect for compliance monitoring, customer service automation, and other voice-based AI applications.

• AI sales agent

• AI Support Agent

• AI Cold Calling

• AI Compliance

Features

• Real-time voice streaming between Twilio and OpenAI

• Automatic speech detection and response cancellation

• Configurable voice settings and system prompts

• Environment-based configuration

• WebSocket-based communication

• Support for G711 ULAW audio format

• Interrupt handling for natural conversation flow

• Session management and real-time updates

Prerequisites

Before you begin, ensure you have:

• Python 3.8 or higher

• An OpenAI API key with Realtime API access

• A Twilio account with:

  • Account SID

  • Auth Token

  • Phone Number

• ngrok or a similar tool for exposing a local server to the internet

Installation

1. Clone the repository:

git clone https://github.com/moonabid/ai-call-agent.git
cd ai-call-agent

2. Install required dependencies:

pip install -r requirements.txt

3. Create a .env file in the root directory with your credentials:

OPENAI_API_KEY=your_openai_api_key
TWILIO_ACCOUNT_SID=your_twilio_account_sid
TWILIO_AUTH_TOKEN=your_twilio_auth_token
TWILIO_PHONE_NUMBER=your_twilio_phone_number
NGROK_URL=your_ngrok_url
PORT=5050

Usage

1. Start the server:

uvicorn main:app --port 5050

2. Expose your local server using ngrok:

ngrok http 5050

3. Update your Twilio Voice webhook URL to point to your ngrok URL + /outgoing-call

4. Make a call using the API endpoint:

curl -X POST "http://localhost:5050/make-call" -H "Content-Type: application/json" -d '{"to_phone_number": "+1234567890"}'

Project Structure

.
├── main.py              # Main application file
├── prompts/            
│   └── system_prompt.txt # System instructions for AI
├── requirements.txt     # Python dependencies
├── .env                # Environment variables
├── .gitignore          # Git ignore file
├── LICENSE             # License file
└── README.md           # Project documentation

Configuration

Environment Variables

• OPENAI_API_KEY: Your OpenAI API key

• TWILIO_ACCOUNT_SID: Your Twilio Account SID

• TWILIO_AUTH_TOKEN: Your Twilio Auth Token

• TWILIO_PHONE_NUMBER: Your Twilio phone number

• NGROK_URL: Your ngrok URL

• PORT: Server port (default: 5050)

System Prompt

The AI's behavior can be customized by modifying the system prompt in prompts/system_prompt.txt.

API Endpoints

• GET /: Health check endpoint

• POST /make-call: Initiate a new call

• POST /outgoing-call: Webhook for Twilio voice calls

• WebSocket /media-stream: WebSocket endpoint for media streaming

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

1. Fork the repository

2. Create your feature branch (git checkout -b feature/AmazingFeature)

3. Commit your changes (git commit -m 'Add some AmazingFeature')

4. Push to the branch (git push origin feature/AmazingFeature)

5. Open a Pull Request