# Directory Structure
```
├── .env.example
├── .gitattributes
├── .gitignore
├── assets
│ └── demo.gif
├── eslint.config.mjs
├── LICENSE
├── package-lock.json
├── package.json
├── pnpm-lock.yaml
├── README.md
├── src
│ ├── config
│ │ ├── constants.ts
│ │ └── prompts.ts
│ ├── handlers
│ │ └── openai.handler.ts
│ ├── servers
│ │ ├── mcp.server.ts
│ │ └── voice.server.ts
│ ├── services
│ │ ├── openai
│ │ │ ├── context.service.ts
│ │ │ ├── event.service.ts
│ │ │ └── ws.service.ts
│ │ ├── session-manager.service.ts
│ │ └── twilio
│ │ ├── call.service.ts
│ │ ├── event.service.ts
│ │ └── ws.service.ts
│ ├── start-all.ts
│ ├── types.ts
│ └── utils
│ ├── call-utils.ts
│ └── execution-utils.ts
└── tsconfig.json
```
# Files
--------------------------------------------------------------------------------
/.gitattributes:
--------------------------------------------------------------------------------
```
# Auto detect text files and perform LF normalization
* text=auto
```
--------------------------------------------------------------------------------
/.env.example:
--------------------------------------------------------------------------------
```
# Server configuration
PORT=3004
# Twilio API credentials
TWILIO_ACCOUNT_SID=your_twilio_account_sid
TWILIO_AUTH_TOKEN=your_twilio_auth_token
TWILIO_NUMBER=your_twilio_number
# OpenAI API key
OPENAI_API_KEY=your_openai_api_key
OPENAI_WEBSOCKET_URL=wss://api.openai.com/v1/realtime?model=gpt-4o-mini-realtime-preview
# Ngrok authentication token
NGROK_AUTHTOKEN=your_ngrok_authtoken
```
--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
```
# Dependency directories
node_modules/
jspm_packages/
# Build outputs
dist/
build/
out/
*.tsbuildinfo
# Environment variables
.env
.env.local
.env.development.local
.env.test.local
.env.production.local
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
lerna-debug.log*
# Coverage directory used by tools like istanbul
coverage/
*.lcov
# Cache directories
.npm
.eslintcache
.stylelintcache
.rpt2_cache/
.rts2_cache_cjs/
.rts2_cache_es/
.rts2_cache_umd/
# Runtime data
pids
*.pid
*.seed
*.pid.lock
# IDE and editor folders
.idea/
.vscode/
*.swp
*.swo
.DS_Store
.directory
.project
.settings/
.classpath
.c9/
*.launch
.settings/
*.sublime-workspace
# ngrok
.ngrok/
```
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
```markdown
# Voice Call MCP Server
A Model Context Protocol (MCP) server that enables Claude and other AI assistants to initiate and manage voice calls using Twilio and OpenAI (GPT-4o Realtime model).
Use this as a base to kick-start your AI-powered voice calling explorations, save time and develop additional functionality on top of it.
## Sequence Diagram
```mermaid
sequenceDiagram
participant AI as AI Assistant (e.g., Claude)
participant MCP as MCP Server
participant Twilio as Twilio
participant Phone as Destination Phone
participant OpenAI as OpenAI
AI->>MCP: 1) Initiate outbound call request <br>(POST /calls)
MCP->>Twilio: 2) Place outbound call via Twilio API
Twilio->>Phone: 3) Ring the destination phone
Twilio->>MCP: 4) Call status updates & audio callbacks (webhooks)
MCP->>OpenAI: 5) Forward real-time audio to OpenaAI's realtime model
OpenAI->>MCP: 6) Return voice stream
MCP->>Twilio: 7) Send voice stream
Twilio->>Phone: 8) Forward voice stream
Note over Phone: Two-way conversation continues <br>until the call ends
```
## Features
- Make outbound phone calls via Twilio 📞
- Process call audio in real-time with GPT-4o Realtime model 🎙️
- Real-time language switching during calls 🌐
- Pre-built prompts for common calling scenarios (like restaurant reservations) 🍽️
- Automatic public URL tunneling with ngrok 🔄
- Secure handling of credentials 🔒
## Why MCP?
The Model Context Protocol (MCP) bridges the gap between AI assistants and real-world actions. By implementing MCP, this server allows AI models like Claude to:
1. Initiate actual phone calls on behalf of users
2. Process and respond to real-time audio conversations
3. Execute complex tasks requiring voice communication
This open-source implementation provides transparency and customizability, allowing developers to extend functionality while maintaining control over their data and privacy.
## Requirements
- Node.js >= 22
- If you need to update Node.js, we recommend using `nvm` (Node Version Manager):
```bash
nvm install 22
nvm use 22
```
- Twilio account with API credentials
- OpenAI API key
- Ngrok Authtoken
## Installation
### Manual Installation
1. Clone the repository
```bash
git clone https://github.com/lukaskai/voice-call-mcp-server.git
cd voice-call-mcp-server
```
2. Install dependencies and build
```bash
npm install
npm run build
```
## Configuration
The server requires several environment variables:
- `TWILIO_ACCOUNT_SID`: Your Twilio account SID
- `TWILIO_AUTH_TOKEN`: Your Twilio auth token
- `TWILIO_NUMBER`: Your Twilio number
- `OPENAI_API_KEY`: Your OpenAI API key
- `NGROK_AUTHTOKEN`: Your ngrok authtoken
- `RECORD_CALLS`: Set to "true" to record calls (optional)
### Claude Desktop Configuration
To use this server with Claude Desktop, add the following to your configuration file:
**macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
**Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
```json
{
"mcpServers": {
"voice-call": {
"command": "node",
"args": ["/path/to/your/mcp-new/dist/start-all.cjs"],
"env": {
"TWILIO_ACCOUNT_SID": "your_account_sid",
"TWILIO_AUTH_TOKEN": "your_auth_token",
"TWILIO_NUMBER": "your_e.164_format_number",
"OPENAI_API_KEY": "your_openai_api_key",
"NGROK_AUTHTOKEN": "your_ngrok_authtoken"
}
}
}
}
```
After that, restart Claude Desktop to reload the configuration.
If connected, you should see Voice Call under the 🔨 menu.
## Example Interactions with Claude
Here are some natural ways to interact with the server through Claude:
1. Simple call:
```
Can you call +1-123-456-7890 and let them know I'll be 15 minutes late for our meeting?
```
2. Restaurant reservation:
```
Please call Delicious Restaurant at +1-123-456-7890 and make a reservation for 4 people tonight at 7:30 PM. Please speak in German.
```
3. Appointment scheduling:
```
Please call Expert Dental NYC (+1-123-456-7899) and reschedule my Monday appointment to next Friday between 4–6pm.
```
## Important Notes
1. **Phone Number Format**: All phone numbers must be in E.164 format (e.g., +11234567890)
2. **Rate Limits**: Be aware of your Twilio and OpenAI account's rate limits and pricing
3. **Voice Conversations**: The AI will handle natural conversations in real-time
4. **Call Duration**: Be mindful of call durations as they affect OpenAI API and Twilio costs
5. **Public Exposure**: Be aware that the ngrok tunnel exposes your server publicly for Twilio to reach it (though with a random URL and protected by a random secret)
## Troubleshooting
Common error messages and solutions:
1. "Phone number must be in E.164 format"
- Make sure the phone number starts with "+" and the country code
2. "Invalid credentials"
- Double-check your TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN. You can copy them from the [Twilio Console](https://console.twilio.com)
3. "OpenAI API error"
- Verify your OPENAI_API_KEY is correct and has sufficient credits
4. "Ngrok tunnel failed to start"
- Ensure your NGROK_AUTHTOKEN is valid and not expired
5. "OpenAI Realtime does not detect the end of voice input, or is lagging."
- Sometimes, there might be voice encoding issues between Twilio and the receiver's network operator. Try using a different receiver.
## Contributing
Contributions are welcome! Here are some areas we're looking to improve:
- Implement support for multiple AI models beyond the current implementation
- Add database integration to store conversation history locally and make it accessible for AI context
- Improve latency and response times to enhance call experiences
- Enhance error handling and recovery mechanisms
- Add more pre-built conversation templates for common scenarios
- Implement improved call monitoring and analytics
If you'd like to contribute, please open an issue to discuss your ideas before submitting a pull request.
## License
This project is licensed under the MIT License - see the LICENSE file for details.
## Security
Please do not include any sensitive information (like phone numbers or API credentials) in GitHub issues or pull requests. This server handles sensitive communications; deploy it responsibly and ensure all credentials are kept secure.
## Time For a New Mission?
We’re hiring engineers to build at the frontier of voice AI — and bake it into a next-gen telco.
Curious? Head to [careers.popcorn.space](https://careers.popcorn.space/apply) 🍿 !
```
--------------------------------------------------------------------------------
/src/config/constants.ts:
--------------------------------------------------------------------------------
```typescript
export const LOG_EVENT_TYPES = [
'error',
'session.created',
'response.audio.delta',
'response.audio_transcript.done',
'conversation.item.input_audio_transcription.completed',
];
export const DYNAMIC_API_SECRET = Math.random().toString(36).substring(2, 15) + Math.random().toString(36).substring(2, 15);
export const SHOW_TIMING_MATH = false;
export const VOICE = 'sage';
export const RECORD_CALLS = process.env.RECORD === 'true';
export const GOODBYE_PHRASES = ['bye', 'goodbye', 'have a nice day', 'see you', 'take care'];
```
--------------------------------------------------------------------------------
/src/utils/call-utils.ts:
--------------------------------------------------------------------------------
```typescript
import { WebSocket } from 'ws';
import { GOODBYE_PHRASES } from '../config/constants.js';
export const checkForGoodbye = (text: string): boolean => {
const lowercaseText = text.toLowerCase();
return GOODBYE_PHRASES.some(phrase => lowercaseText.includes(phrase));
};
export const endCall = (ws: WebSocket, openAiWs: WebSocket): void => {
setTimeout(() => {
if (ws.readyState === WebSocket.OPEN) {
ws.close();
}
if (openAiWs.readyState === WebSocket.OPEN) {
openAiWs.close();
}
}, 5000);
};
```
--------------------------------------------------------------------------------
/tsconfig.json:
--------------------------------------------------------------------------------
```json
{
"compilerOptions": {
"target": "es2020",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"declaration": true,
"declarationMap": true,
"sourceMap": true,
"outDir": "./dist",
"strict": true,
"esModuleInterop": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"isolatedModules": true,
"skipLibCheck": true,
"lib": ["es2020", "DOM"],
"allowSyntheticDefaultImports": true
},
"include": ["src/**/*", "openai-realtime-handler.ts"],
"exclude": ["node_modules", "dist"]
}
```
--------------------------------------------------------------------------------
/src/utils/execution-utils.ts:
--------------------------------------------------------------------------------
```typescript
import net from 'net';
export async function isPortInUse(port: number): Promise<boolean> {
return new Promise((resolve) => {
const server = net.createServer()
.once('error', (err: NodeJS.ErrnoException) => {
if (err.code === 'EADDRINUSE') {
resolve(true);
} else {
resolve(false);
}
})
.once('listening', () => {
server.close();
resolve(false);
})
.listen(port);
});
}
```
--------------------------------------------------------------------------------
/eslint.config.mjs:
--------------------------------------------------------------------------------
```
import eslint from '@eslint/js';
import tseslint from 'typescript-eslint';
export default tseslint.config(
eslint.configs.recommended,
...tseslint.configs.recommended,
...tseslint.configs.stylistic,
{
plugins: {
},
rules: {
'@typescript-eslint/no-explicit-any': 'off',
'@typescript-eslint/prefer-for-of': 'off',
'no-trailing-spaces': 'error', // Disallow trailing spaces
'eol-last': ['error', 'always'], // Enforce newline at end of file
'indent': ['error', 4], // Enforce 4 spaces for indentation
'quotes': ['error', 'single'], // Enforce single quotes
'semi': ['error', 'always'], // Enforce semicolons
},
}
);
```
--------------------------------------------------------------------------------
/src/services/openai/context.service.ts:
--------------------------------------------------------------------------------
```typescript
import { generateOutboundCallContext } from '../../config/prompts.js';
import { CallState, ConversationMessage } from '../../types.js';
export class OpenAIContextService {
public initializeCallState(callState: CallState, fromNumber: string, toNumber: string): void {
callState.fromNumber = fromNumber;
callState.toNumber = toNumber;
}
public setupConversationContext(callState: CallState, callContext?: string): void {
callState.initialMessage = 'Hello!';
callState.callContext = generateOutboundCallContext(callState, callContext);
const systemMessage: ConversationMessage = {
role: 'system',
content: callState.callContext
};
callState.conversationHistory = [systemMessage];
const initialMessage: ConversationMessage = {
role: 'user',
content: callState.initialMessage
};
callState.conversationHistory.push(initialMessage);
}
}
```
--------------------------------------------------------------------------------
/src/config/prompts.ts:
--------------------------------------------------------------------------------
```typescript
import { CallState } from '../types.js';
export const generateOutboundCallContext = (callState: CallState, callContext?: string): string => {
return `Please refer to phone call transcripts.
Stay concise and short.
You are assistant (if asked, you phone number with country code is: ${callState.fromNumber}). You are making an outbound call.
Be friendly and speak in human short sentences. Start conversation with how are you. Do not speak in bullet points. Ask one question at a time, tell one sentence at a time.
After successful task completion, say goodbye and end the conversation.
You ARE NOT a receptionist, NOT an administrator, NOT a person making reservation.
You do not provide any other info, which is not related to the goal. You can calling solely to achieve your tasks
You are the customer making a request, not the restaurant staff.
YOU ARE STRICTLY THE ONE MAKING THE REQUEST (and not the one receiving). YOU MUST ACHIEVE YOUR GOAL AS AN ASSITANT AND PERFORM TASK.
Be focused solely on your task:
${callContext ? callContext : ''}`;
};
```
--------------------------------------------------------------------------------
/src/types.ts:
--------------------------------------------------------------------------------
```typescript
// state.ts - Shared state variables
export enum CallType {
OUTBOUND = 'OUTBOUND',
}
export interface ConversationMessage {
role: 'system' | 'user' | 'assistant';
content: string;
name?: string;
}
export class CallState {
// Call identification
streamSid = '';
callSid = '';
// Call type and direction
callType: CallType = CallType.OUTBOUND;
// Phone numbers
fromNumber = '';
toNumber = '';
// Call context and conversation
callContext = '';
initialMessage = '';
conversationHistory: ConversationMessage[] = [];
// Speech state
speaking = false;
// Timing and processing state
llmStart = 0;
firstByte = true;
sendFirstSentenceInputTime: number | null = null;
// Media processing state
latestMediaTimestamp = 0;
responseStartTimestampTwilio: number | null = null;
lastAssistantItemId: string | null = null;
markQueue: string[] = [];
hasSeenMedia = false;
constructor(callType: CallType = CallType.OUTBOUND) {
this.callType = callType;
}
}
/**
* Configuration for the OpenAI WebSocket connection
*/
export interface OpenAIConfig {
apiKey: string;
websocketUrl: string;
voice: string;
temperature: number;
}
/**
* Configuration for Twilio client
*/
export interface TwilioConfig {
accountSid: string;
authToken: string;
recordCalls: boolean;
}
```
--------------------------------------------------------------------------------
/package.json:
--------------------------------------------------------------------------------
```json
{
"name": "voice-call-mcp-server",
"main": "dist/start-all.js",
"type": "module",
"scripts": {
"start-all": "tsx src/start-all.ts",
"start": "node dist/start-all.cjs",
"build": "npm-run-all clean build:app",
"build:app": "tsup src/start-all.ts",
"clean": "rimraf dist"
},
"dependencies": {
"@modelcontextprotocol/sdk": "1.8.0",
"@ngrok/ngrok": "^1.4.1",
"axios": "^1.6.8",
"body-parser": "^1.20.2",
"colors": "^1.4.0",
"cors": "^2.8.5",
"dotenv": "^16.4.5",
"eslint-plugin-simple-import-sort": "^12.1.1",
"express": "^4.18.3",
"express-ws": "^5.0.2",
"form-data": "^4.0.0",
"google-protobuf": "^3.21.4",
"httpdispatcher": "^2.2.0",
"ngrok": "5.0.0-beta.2",
"node-fetch": "^2.7.0",
"node-vad": "^1.1.4",
"openai": "^4.85.1",
"peerjs": "^1.5.4",
"perf_hooks": "^0.0.1",
"protobufjs": "^7.4.0",
"twilio": "^5.0.1",
"uuid": "^9.0.1",
"websocket": "^1.0.28",
"zod": "^3.22.4"
},
"devDependencies": {
"@eslint/js": "^9.21.0",
"@types/cors": "^2.8.17",
"@types/express": "^4.17.21",
"@types/express-ws": "^3.0.4",
"@types/node": "^20.11.30",
"@types/uuid": "^9.0.8",
"@types/websocket": "^1.0.10",
"@types/ws": "^8.5.10",
"dotenv": "^16.4.5",
"eslint": "^9.21.0",
"globals": "^16.0.0",
"npm-run-all": "^4.1.5",
"rimraf": "^5.0.5",
"tsup": "^8.0.2",
"tsx": "^4.7.1",
"typescript": "^5.4.2",
"typescript-eslint": "^8.24.1"
},
"author": "Popcorn",
"license": "MIT",
"packageManager": "[email protected]+sha512.6b865ad4b62a1d9842b61d674a393903b871d9244954f652b8842c2b553c72176b278f64c463e52d40fff8aba385c235c8c9ecf5cc7de4fd78b8bb6d49633ab6"
}
```
--------------------------------------------------------------------------------
/src/services/twilio/call.service.ts:
--------------------------------------------------------------------------------
```typescript
import twilio from 'twilio';
import { DYNAMIC_API_SECRET, RECORD_CALLS } from '../../config/constants.js';
/**
* Service for handling Twilio call operations
*/
export class TwilioCallService {
private readonly twilioClient: twilio.Twilio;
/**
* Create a new Twilio call service
* @param twilioClient The Twilio client
*/
constructor(twilioClient: twilio.Twilio) {
this.twilioClient = twilioClient;
}
/**
* Start recording a call
* @param callSid The SID of the call to record
*/
public async startRecording(callSid: string): Promise<void> {
if (!RECORD_CALLS || !callSid) {
return;
}
try {
await this.twilioClient.calls(callSid)
.recordings
.create();
} catch (error) {
console.error(`Failed to start recording for call ${callSid}:`, error);
}
}
/**
* End a call
* @param callSid The SID of the call to end
*/
public async endCall(callSid: string): Promise<void> {
if (!callSid) {
return;
}
try {
await this.twilioClient.calls(callSid)
.update({ status: 'completed' });
} catch (error) {
console.error(`Failed to end call ${callSid}:`, error);
}
}
public async makeCall(twilioCallbackUrl: string, toNumber: string, callContext = ''): Promise<string> {
try {
const twilioClient = twilio(process.env.TWILIO_ACCOUNT_SID, process.env.TWILIO_AUTH_TOKEN);
const callContextEncoded = encodeURIComponent(callContext);
const call = await twilioClient.calls.create({
to: toNumber,
from: process.env.TWILIO_NUMBER || '',
url: `${twilioCallbackUrl}/call/outgoing?apiSecret=${DYNAMIC_API_SECRET}&callType=outgoing&callContext=${callContextEncoded}`,
});
return call.sid;
} catch (error) {
console.error(`Error making call: ${error}`);
throw error;
}
}
}
```
--------------------------------------------------------------------------------
/src/servers/voice.server.ts:
--------------------------------------------------------------------------------
```typescript
import dotenv from 'dotenv';
import express, { Response } from 'express';
import VoiceResponse from 'twilio/lib/twiml/VoiceResponse.js';
import ExpressWs from 'express-ws';
import { WebSocket } from 'ws';
import { CallType } from '../types.js';
import { DYNAMIC_API_SECRET } from '../config/constants.js';
import { CallSessionManager } from '../handlers/openai.handler.js';
dotenv.config();
export class VoiceServer {
private app: express.Application & { ws: any };
private port: number;
private sessionManager: CallSessionManager;
private callbackUrl: string;
constructor(callbackUrl: string, sessionManager: CallSessionManager) {
this.callbackUrl = callbackUrl;
this.port = parseInt(process.env.PORT || '3004');
this.app = ExpressWs(express()).app;
this.sessionManager = sessionManager;
this.configureMiddleware();
this.setupRoutes();
}
private configureMiddleware(): void {
this.app.use(express.json());
this.app.use(express.urlencoded({ extended: false }));
}
private setupRoutes(): void {
this.app.post('/call/outgoing', this.handleOutgoingCall.bind(this));
this.app.ws('/call/connection-outgoing/:secret', this.handleOutgoingConnection.bind(this));
}
private async handleOutgoingCall(req: express.Request, res: Response): Promise<void> {
const apiSecret = req.query.apiSecret?.toString();
if (req.query.apiSecret?.toString() !== DYNAMIC_API_SECRET) {
res.status(401).json({ error: 'Unauthorized: Invalid or missing API secret' });
return;
}
const fromNumber = req.body.From;
const toNumber = req.body.To;
const callContext = req.query.callContext?.toString();
const twiml = new VoiceResponse();
const connect = twiml.connect();
const stream = connect.stream({
url: `${this.callbackUrl.replace('https://', 'wss://')}/call/connection-outgoing/${apiSecret}`,
});
stream.parameter({ name: 'fromNumber', value: fromNumber });
stream.parameter({ name: 'toNumber', value: toNumber });
stream.parameter({ name: 'callContext', value: callContext });
res.writeHead(200, { 'Content-Type': 'text/xml' });
res.end(twiml.toString());
}
private handleOutgoingConnection(ws: WebSocket, req: express.Request): void {
if (req.params.secret !== DYNAMIC_API_SECRET) {
ws.close(1008, 'Unauthorized: Invalid or missing API secret');
return;
}
this.sessionManager.createSession(ws, CallType.OUTBOUND);
}
public start(): void {
this.app.listen(this.port);
}
}
```
--------------------------------------------------------------------------------
/src/services/session-manager.service.ts:
--------------------------------------------------------------------------------
```typescript
import { WebSocket } from 'ws';
import twilio from 'twilio';
import { CallType } from '../types.js';
import { OpenAIContextService } from './openai/context.service.js';
import { OpenAICallHandler } from '../handlers/openai.handler.js';
/**
* Manages multiple concurrent call sessions
*/
export class SessionManagerService {
private readonly activeSessions: Map<string, OpenAICallHandler>;
private readonly twilioClient: twilio.Twilio;
private readonly contextService: OpenAIContextService;
/**
* Create a new session manager
* @param twilioConfig Configuration for the Twilio client
*/
constructor(twilioClient: twilio.Twilio) {
this.activeSessions = new Map();
this.twilioClient = twilioClient;
this.contextService = new OpenAIContextService();
}
/**
* Creates a new call session and adds it to the active sessions
* @param ws The WebSocket connection
* @param callType The type of call
*/
public createSession(ws: WebSocket, callType: CallType): void {
const handler = new OpenAICallHandler(ws, callType, this.twilioClient, this.contextService);
this.registerSessionCleanup(ws);
this.addSession(ws, handler);
}
/**
* Register cleanup for a session
* @param ws The WebSocket connection
*/
private registerSessionCleanup(ws: WebSocket): void {
ws.on('close', () => {
this.removeSession(ws);
});
}
/**
* Add a session to active sessions
* @param ws The WebSocket connection
* @param handler The OpenAI call handler
*/
private addSession(ws: WebSocket, handler: OpenAICallHandler): void {
this.activeSessions.set(this.getSessionKey(ws), handler);
}
/**
* Removes a session from active sessions
* @param ws The WebSocket connection
*/
private removeSession(ws: WebSocket): void {
const sessionKey = this.getSessionKey(ws);
if (this.activeSessions.has(sessionKey)) {
this.activeSessions.delete(sessionKey);
}
}
/**
* Generates a unique key for a session based on the WebSocket object
* @param ws The WebSocket connection
* @returns A unique key for the session
*/
private getSessionKey(ws: WebSocket): string {
return ws.url || ws.toString();
}
/**
* Get the Twilio client
* @returns The Twilio client
*/
public getTwilioClient(): twilio.Twilio {
return this.twilioClient;
}
/**
* Get the context service
* @returns The context service
*/
public getContextService(): OpenAIContextService {
return this.contextService;
}
}
```
--------------------------------------------------------------------------------
/src/services/twilio/ws.service.ts:
--------------------------------------------------------------------------------
```typescript
import { WebSocket } from 'ws';
import { CallState } from '../../types.js';
import { SHOW_TIMING_MATH } from '../../config/constants.js';
/**
* Service for handling Twilio WebSocket streams
*/
export class TwilioWsService {
private readonly webSocket: WebSocket;
private readonly callState: CallState;
/**
* Create a new Twilio stream service
* @param webSocket The Twilio WebSocket connection
* @param callState The state of the call
*/
constructor(webSocket: WebSocket, callState: CallState) {
this.webSocket = webSocket;
this.callState = callState;
}
/**
* Close the WebSocket connection
*/
public close(): void {
if (this.webSocket.readyState === WebSocket.OPEN) {
this.webSocket.close();
}
}
/**
* Send a mark event to Twilio
*/
public sendMark(): void {
if (!this.callState.streamSid) {
return;
}
const markEvent = {
event: 'mark',
streamSid: this.callState.streamSid,
mark: { name: 'responsePart' }
};
this.webSocket.send(JSON.stringify(markEvent));
this.callState.markQueue.push('responsePart');
}
/**
* Send audio data to Twilio
* @param payload The audio payload to send
*/
public sendAudio(payload: string): void {
if (!this.callState.streamSid) {
return;
}
const audioDelta = {
event: 'media',
streamSid: this.callState.streamSid,
media: { payload }
};
this.webSocket.send(JSON.stringify(audioDelta));
}
/**
* Clear the Twilio stream
*/
public clearStream(): void {
if (!this.callState.streamSid) {
return;
}
this.webSocket.send(JSON.stringify({
event: 'clear',
streamSid: this.callState.streamSid
}));
}
/**
* Set up event handlers for the Twilio WebSocket
* @param onMessage Callback for handling messages from Twilio
* @param onClose Callback for when the connection is closed
*/
public setupEventHandlers(
onMessage: (message: Buffer | string) => void,
onClose: () => void
): void {
this.webSocket.on('message', onMessage);
this.webSocket.on('close', onClose);
}
/**
* Process a Twilio start event
* @param data The start event data
*/
public processStartEvent(data: any): void {
this.callState.streamSid = data.start.streamSid;
this.callState.responseStartTimestampTwilio = null;
this.callState.latestMediaTimestamp = 0;
this.callState.callSid = data.start.callSid;
}
/**
* Process a Twilio mark event
*/
public processMarkEvent(): void {
if (this.callState.markQueue.length > 0) {
this.callState.markQueue.shift();
}
}
/**
* Process a Twilio media event
* @param data The media event data
*/
public processMediaEvent(data: any): void {
this.callState.latestMediaTimestamp = data.media.timestamp;
if (SHOW_TIMING_MATH) {
// console.log(`Received media message with timestamp: ${this.callState.latestMediaTimestamp}ms`);
}
}
}
```
--------------------------------------------------------------------------------
/src/services/openai/ws.service.ts:
--------------------------------------------------------------------------------
```typescript
import { WebSocket } from 'ws';
import { OpenAIConfig } from '../../types.js';
import { SHOW_TIMING_MATH } from '../../config/constants.js';
/**
* Service for handling OpenAI API interactions
*/
export class OpenAIWsService {
private webSocket: WebSocket | null = null;
private readonly config: OpenAIConfig;
/**
* Create a new OpenAI service
* @param config Configuration for the OpenAI API
*/
constructor(config: OpenAIConfig) {
this.config = config;
}
/**
* Initialize the WebSocket connection to OpenAI
* @param onMessage Callback for handling messages from OpenAI
* @param onOpen Callback for when the connection is opened
* @param onError Callback for handling errors
*/
public initialize(
onMessage: (data: WebSocket.Data) => void,
onOpen: () => void,
onError: (error: Error) => void
): void {
this.webSocket = new WebSocket(this.config.websocketUrl, {
headers: {
Authorization: `Bearer ${this.config.apiKey}`,
'OpenAI-Beta': 'realtime=v1'
}
});
this.webSocket.on('open', onOpen);
this.webSocket.on('message', onMessage);
this.webSocket.on('error', onError);
}
/**
* Initialize the session with OpenAI
* @param callContext The context for the call
*/
public initializeSession(callContext: string): void {
if (!this.webSocket || this.webSocket.readyState !== WebSocket.OPEN) {
return;
}
const sessionUpdate = {
type: 'session.update',
session: {
turn_detection: { type: 'server_vad' },
input_audio_format: 'g711_ulaw',
output_audio_format: 'g711_ulaw',
voice: this.config.voice,
instructions: callContext,
modalities: ['text', 'audio'],
temperature: this.config.temperature,
'input_audio_transcription': {
'model': 'whisper-1'
},
}
};
this.webSocket.send(JSON.stringify(sessionUpdate));
}
/**
* Close the WebSocket connection
*/
public close(): void {
if (this.webSocket && this.webSocket.readyState === WebSocket.OPEN) {
this.webSocket.close();
}
}
/**
* Forward audio data to OpenAI
* @param audioPayload The audio payload to forward
*/
public sendAudio(audioPayload: string): void {
if (!this.webSocket || this.webSocket.readyState !== WebSocket.OPEN) {
return;
}
const audioAppend = {
type: 'input_audio_buffer.append',
audio: audioPayload
};
this.webSocket.send(JSON.stringify(audioAppend));
}
/**
* Truncate the assistant's response
* @param itemId The ID of the assistant's response
* @param elapsedTime The time elapsed since the response started
*/
public truncateAssistantResponse(itemId: string, elapsedTime: number): void {
if (!this.webSocket || this.webSocket.readyState !== WebSocket.OPEN) {
return;
}
const truncateEvent = {
type: 'conversation.item.truncate',
item_id: itemId,
content_index: 0,
audio_end_ms: elapsedTime
};
if (SHOW_TIMING_MATH) {
console.error('Sending truncation event:', JSON.stringify(truncateEvent));
}
this.webSocket.send(JSON.stringify(truncateEvent));
}
/**
* Check if the WebSocket is connected
*/
public isConnected(): boolean {
return this.webSocket !== null && this.webSocket.readyState === WebSocket.OPEN;
}
}
```
--------------------------------------------------------------------------------
/src/services/openai/event.service.ts:
--------------------------------------------------------------------------------
```typescript
import { WebSocket } from 'ws';
import { CallState } from '../../types.js';
import { LOG_EVENT_TYPES, SHOW_TIMING_MATH } from '../../config/constants.js';
import { checkForGoodbye } from '../../utils/call-utils.js';
/**
* Service for processing OpenAI events
*/
export class OpenAIEventService {
private readonly callState: CallState;
private readonly onEndCall: () => void;
private readonly onSendAudioToTwilio: (payload: string) => void;
private readonly onTruncateResponse: () => void;
/**
* Create a new OpenAI event processor
* @param callState The state of the call
* @param onEndCall Callback for ending the call
* @param onSendAudioToTwilio Callback for sending audio to Twilio
* @param onTruncateResponse Callback for truncating the response
*/
constructor(
callState: CallState,
onEndCall: () => void,
onSendAudioToTwilio: (payload: string) => void,
onTruncateResponse: () => void
) {
this.callState = callState;
this.onEndCall = onEndCall;
this.onSendAudioToTwilio = onSendAudioToTwilio;
this.onTruncateResponse = onTruncateResponse;
}
/**
* Process an OpenAI message
* @param data The message data
*/
public processMessage(data: WebSocket.Data): void {
try {
const response = JSON.parse(data.toString());
if (LOG_EVENT_TYPES.includes(response.type)) {
// console.log(`Received event: ${response.type}`, response);
}
this.processEvent(response);
} catch (error) {
console.error('Error processing OpenAI message:', error, 'Raw message:', data);
}
}
/**
* Process an OpenAI event
* @param response The event data
*/
private processEvent(response: any): void {
switch (response.type) {
case 'conversation.item.input_audio_transcription.completed':
this.handleTranscriptionCompleted(response.transcript);
break;
case 'response.audio_transcript.done':
this.handleAudioTranscriptDone(response.transcript);
break;
case 'response.audio.delta':
if (response.delta) {
this.handleAudioDelta(response);
}
break;
case 'input_audio_buffer.speech_started':
this.onTruncateResponse();
break;
}
}
/**
* Handle a transcription completed event
* @param transcription The transcription text
*/
private handleTranscriptionCompleted(transcription: string): void {
if (!transcription) {
return;
}
this.callState.conversationHistory.push({
role: 'user',
content: transcription
});
if (checkForGoodbye(transcription)) {
this.onEndCall();
}
}
/**
* Handle an audio transcript done event
* @param transcript The transcript text
*/
private handleAudioTranscriptDone(transcript: string): void {
if (!transcript) {
return;
}
this.callState.conversationHistory.push({
role: 'assistant',
content: transcript
});
}
/**
* Handle an audio delta event
* @param response The event data
*/
private handleAudioDelta(response: any): void {
this.onSendAudioToTwilio(response.delta);
if (!this.callState.responseStartTimestampTwilio) {
this.callState.responseStartTimestampTwilio = this.callState.latestMediaTimestamp;
if (SHOW_TIMING_MATH) {
// console.log(`Setting start timestamp for new response: ${this.callState.responseStartTimestampTwilio}ms`);
}
}
if (response.item_id) {
this.callState.lastAssistantItemId = response.item_id;
}
}
}
```
--------------------------------------------------------------------------------
/src/start-all.ts:
--------------------------------------------------------------------------------
```typescript
import dotenv from 'dotenv';
import ngrok from '@ngrok/ngrok';
import { isPortInUse } from './utils/execution-utils.js';
import { VoiceCallMcpServer } from './servers/mcp.server.js';
import { TwilioCallService } from './services/twilio/call.service.js';
import { VoiceServer } from './servers/voice.server.js';
import twilio from 'twilio';
import { CallSessionManager } from './handlers/openai.handler.js';
// Load environment variables
dotenv.config();
// Define required environment variables
const REQUIRED_ENV_VARS = [
'TWILIO_ACCOUNT_SID',
'TWILIO_AUTH_TOKEN',
'OPENAI_API_KEY',
'NGROK_AUTHTOKEN',
'TWILIO_NUMBER'
] as const;
/**
* Validates that all required environment variables are present
* @returns true if all variables are present, exits process otherwise
*/
function validateEnvironmentVariables(): boolean {
for (const envVar of REQUIRED_ENV_VARS) {
if (!process.env[envVar]) {
console.error(`Error: ${envVar} environment variable is required`);
process.exit(1);
}
}
return true;
}
/**
* Sets up the port for the application
*/
function setupPort(): number {
const PORT = process.env.PORT || '3004';
process.env.PORT = PORT;
return parseInt(PORT);
}
/**
* Establishes ngrok tunnel for external access
* @param portNumber - The port number to forward
* @returns The public URL provided by ngrok
*/
async function setupNgrokTunnel(portNumber: number): Promise<string> {
const listener = await ngrok.forward({
addr: portNumber,
authtoken_from_env: true
});
const twilioCallbackUrl = listener.url();
if (!twilioCallbackUrl) {
throw new Error('Failed to obtain ngrok URL');
}
return twilioCallbackUrl;
}
/**
* Sets up graceful shutdown handlers
*/
function setupShutdownHandlers(): void {
process.on('SIGINT', async () => {
try {
await ngrok.disconnect();
} catch (err) {
console.error('Error killing ngrok:', err);
}
process.exit(0);
});
}
/**
* Retries starting the server when the port is in use
* @param portNumber - The port number to check
*/
function scheduleServerRetry(portNumber: number): void {
console.error(`Port ${portNumber} is already in use. Server may already be running.`);
console.error('Will retry in 15 seconds...');
const RETRY_INTERVAL_MS = 15000;
const retryInterval = setInterval(async () => {
const stillInUse = await isPortInUse(portNumber);
if (!stillInUse) {
clearInterval(retryInterval);
main();
} else {
console.error(`Port ${portNumber} is still in use. Will retry in 15 seconds...`);
}
}, RETRY_INTERVAL_MS);
}
async function main(): Promise<void> {
try {
validateEnvironmentVariables();
const portNumber = setupPort();
const twilioClient = twilio(process.env.TWILIO_ACCOUNT_SID, process.env.TWILIO_AUTH_TOKEN);
const sessionManager = new CallSessionManager(twilioClient);
const twilioCallService = new TwilioCallService(twilioClient);
// Check if port is already in use
const portInUse = await isPortInUse(portNumber);
if (portInUse) {
scheduleServerRetry(portNumber);
return;
}
// Establish ngrok connectivity
const twilioCallbackUrl = await setupNgrokTunnel(portNumber);
// Start the main HTTP server
const server = new VoiceServer(twilioCallbackUrl, sessionManager);
server.start();
const mcpServer = new VoiceCallMcpServer(twilioCallService, twilioCallbackUrl);
await mcpServer.start();
// Set up graceful shutdown
setupShutdownHandlers();
} catch (error) {
console.error('Error starting services:', error);
process.exit(1);
}
}
// Start the main function
main();
```
--------------------------------------------------------------------------------
/src/services/twilio/event.service.ts:
--------------------------------------------------------------------------------
```typescript
import { CallState } from '../../types.js';
import { OpenAIContextService } from '../openai/context.service.js';
import { RECORD_CALLS, SHOW_TIMING_MATH } from '../../config/constants.js';
import { TwilioCallService } from './call.service.js';
/**
* Service for processing Twilio events
*/
export class TwilioEventService {
private readonly callState: CallState;
private readonly twilioCallService: TwilioCallService;
private readonly contextService: OpenAIContextService;
private readonly onForwardAudioToOpenAI: (payload: string) => void;
/**
* Create a new Twilio event processor
* @param callState The state of the call
* @param twilioCallService The Twilio call service
* @param contextService The context service
* @param onForwardAudioToOpenAI Callback for forwarding audio to OpenAI
*/
constructor(
callState: CallState,
twilioCallService: TwilioCallService,
contextService: OpenAIContextService,
onForwardAudioToOpenAI: (payload: string) => void,
) {
this.callState = callState;
this.twilioCallService = twilioCallService;
this.contextService = contextService;
this.onForwardAudioToOpenAI = onForwardAudioToOpenAI;
}
/**
* Process a Twilio message
* @param message The message data
*/
public async processMessage(message: Buffer | string): Promise<void> {
try {
const data = JSON.parse(message.toString());
await this.processEvent(data);
} catch (error) {
console.error('Error parsing message:', error, 'Message:', message);
}
}
/**
* Process a Twilio event
* @param data The event data
*/
private async processEvent(data: any): Promise<void> {
switch (data.event) {
case 'media':
await this.handleMediaEvent(data);
break;
case 'start':
await this.handleStartEvent(data);
break;
case 'mark':
this.handleMarkEvent();
break;
default:
console.error('Received non-media event:', data.event);
break;
}
}
/**
* Handle a Twilio media event
* @param data The event data
*/
private async handleMediaEvent(data: any): Promise<void> {
this.callState.latestMediaTimestamp = data.media.timestamp;
if (SHOW_TIMING_MATH) {
// console.log(`Received media message with timestamp: ${this.callState.latestMediaTimestamp}ms`);
}
await this.handleFirstMediaEventIfNeeded();
this.onForwardAudioToOpenAI(data.media.payload);
}
/**
* Handle the first media event if it hasn't been handled yet
*/
private async handleFirstMediaEventIfNeeded(): Promise<void> {
if (this.callState.hasSeenMedia) {
return;
}
this.callState.hasSeenMedia = true;
if (RECORD_CALLS && this.callState.callSid) {
await this.startCallRecording();
}
}
/**
* Start recording the call
*/
private async startCallRecording(): Promise<void> {
await this.twilioCallService.startRecording(this.callState.callSid);
}
/**
* Handle a Twilio start event
* @param data The event data
*/
private async handleStartEvent(data: any): Promise<void> {
this.callState.streamSid = data.start.streamSid;
this.callState.responseStartTimestampTwilio = null;
this.callState.latestMediaTimestamp = 0;
this.contextService.initializeCallState(this.callState, data.start.customParameters.fromNumber, data.start.customParameters.toNumber);
this.contextService.setupConversationContext(this.callState, data.start.customParameters.callContext);
this.callState.callSid = data.start.callSid;
}
/**
* Handle a Twilio mark event
*/
private handleMarkEvent(): void {
if (this.callState.markQueue.length > 0) {
this.callState.markQueue.shift();
}
}
}
```
--------------------------------------------------------------------------------
/src/servers/mcp.server.ts:
--------------------------------------------------------------------------------
```typescript
import { McpServer, ResourceTemplate } from '@modelcontextprotocol/sdk/server/mcp.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import { z } from 'zod';
import { TwilioCallService } from '../services/twilio/call.service.js';
export class VoiceCallMcpServer {
private server: McpServer;
private twilioCallService: TwilioCallService;
private twilioCallbackUrl: string;
constructor(twilioCallService: TwilioCallService, twilioCallbackUrl: string) {
this.twilioCallbackUrl = twilioCallbackUrl;
this.twilioCallService = twilioCallService;
this.server = new McpServer({
name: 'Voice Call MCP Server',
version: '1.0.0',
description: 'MCP server that provides tools for initiating phone calls via Twilio'
});
this.registerTools();
this.registerResources();
this.registerPrompts();
}
private registerTools(): void {
this.server.tool(
'trigger-call',
'Trigger an outbound phone call via Twilio',
{
toNumber: z.string().describe('The phone number to call'),
callContext: z.string().describe('Context for the call')
},
async ({ toNumber, callContext }) => {
try {
const callSid = await this.twilioCallService.makeCall(this.twilioCallbackUrl, toNumber, callContext);
return {
content: [{
type: 'text',
text: JSON.stringify({
status: 'success',
message: 'Call triggered successfully',
callSid: callSid
})
}]
};
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
return {
content: [{
type: 'text',
text: JSON.stringify({
status: 'error',
message: `Failed to trigger call: ${errorMessage}`
})
}],
isError: true
};
}
}
);
}
private registerResources(): void {
this.server.resource(
'get-latest-call',
new ResourceTemplate('call://transcriptions', { list: undefined }),
async () => {
// TODO: get call transcription
return {
contents: [{
text: JSON.stringify({
transcription: '{}',
status: 'completed',
}),
uri: 'call://transcriptions/latest',
mimeType: 'application/json'
}]
};
}
);
}
private registerPrompts(): void {
this.server.prompt(
'make-restaurant-reservation',
'Create a prompt for making a restaurant reservation by phone',
{
restaurantNumber: z.string().describe('The phone number of the restaurant'),
peopleNumber: z.string().describe('The number of people in the party'),
date: z.string().describe('Date of the reservation'),
time: z.string().describe('Preferred time for the reservation')
},
({ restaurantNumber, peopleNumber, date, time }) => {
return {
messages: [{
role: 'user',
content: {
type: 'text',
text: `You are calling a restaurant to book a table for ${peopleNumber} people on ${date} at ${time}. Call the restaurant at ${restaurantNumber} from ${process.env.TWILIO_NUMBER}.`
}
}]
};
}
);
}
public async start(): Promise<void> {
const transport = new StdioServerTransport();
await this.server.connect(transport);
}
}
```
--------------------------------------------------------------------------------
/src/handlers/openai.handler.ts:
--------------------------------------------------------------------------------
```typescript
import { WebSocket } from 'ws';
import twilio from 'twilio';
import dotenv from 'dotenv';
import { CallState, CallType, OpenAIConfig } from '../types.js';
import { VOICE } from '../config/constants.js';
import { OpenAIContextService } from '../services/openai/context.service.js';
import { OpenAIWsService } from '../services/openai/ws.service.js';
import { TwilioWsService } from '../services/twilio/ws.service.js';
import { OpenAIEventService } from '../services/openai/event.service.js';
import { TwilioEventService } from '../services/twilio/event.service.js';
import { SessionManagerService } from '../services/session-manager.service.js';
import { TwilioCallService } from '../services/twilio/call.service.js';
dotenv.config();
/**
* Handles the communication between Twilio and OpenAI for voice calls
*/
export class OpenAICallHandler {
private readonly twilioStream: TwilioWsService;
private readonly openAIService: OpenAIWsService;
private readonly openAIEventProcessor: OpenAIEventService;
private readonly twilioEventProcessor: TwilioEventService;
private readonly twilioCallService: TwilioCallService;
private readonly callState: CallState;
constructor(ws: WebSocket, callType: CallType, twilioClient: twilio.Twilio, contextService: OpenAIContextService) {
this.callState = new CallState(callType);
// Initialize Twilio services
this.twilioStream = new TwilioWsService(ws, this.callState);
this.twilioCallService = new TwilioCallService(twilioClient);
// Initialize OpenAI service
const openAIConfig: OpenAIConfig = {
apiKey: process.env.OPENAI_API_KEY || '',
websocketUrl: process.env.OPENAI_WEBSOCKET_URL || 'wss://api.openai.com/v1/realtime?model=gpt-4o-mini-realtime-preview',
voice: VOICE,
temperature: 0.6
};
this.openAIService = new OpenAIWsService(openAIConfig);
// Initialize event processors
this.openAIEventProcessor = new OpenAIEventService(
this.callState,
() => this.endCall(),
(payload) => this.twilioStream.sendAudio(payload),
() => this.handleSpeechStartedEvent()
);
this.twilioEventProcessor = new TwilioEventService(
this.callState,
this.twilioCallService,
contextService,
(payload) => this.openAIService.sendAudio(payload),// Log the first media event
);
this.setupEventHandlers();
this.initializeOpenAI();
}
private endCall(): void {
if (this.callState.callSid) {
this.twilioCallService.endCall(this.callState.callSid);
}
setTimeout(() => {
this.closeWebSockets();
}, 5000);
}
private closeWebSockets(): void {
this.twilioStream.close();
this.openAIService.close();
}
private initializeOpenAI(): void {
this.openAIService.initialize(
(data) => this.openAIEventProcessor.processMessage(data),
() => {
setTimeout(() => this.openAIService.initializeSession(this.callState.callContext), 100);
},
(error) => console.error('Error in the OpenAI WebSocket:', error)
);
}
private handleSpeechStartedEvent(): void {
if (this.callState.markQueue.length === 0 || this.callState.responseStartTimestampTwilio === null || !this.callState.lastAssistantItemId) {
return;
}
const elapsedTime = this.callState.latestMediaTimestamp - this.callState.responseStartTimestampTwilio;
this.openAIService.truncateAssistantResponse(this.callState.lastAssistantItemId, elapsedTime);
this.twilioStream.clearStream();
this.resetResponseState();
}
private resetResponseState(): void {
this.callState.markQueue = [];
this.callState.lastAssistantItemId = null;
this.callState.responseStartTimestampTwilio = null;
}
private setupEventHandlers(): void {
this.twilioStream.setupEventHandlers(
async (message) => await this.twilioEventProcessor.processMessage(message),
async () => {
this.openAIService.close();
}
);
}
}
/**
* Manages multiple concurrent call sessions
*/
export class CallSessionManager {
private readonly sessionManager: SessionManagerService;
constructor(twilioClient: twilio.Twilio) {
this.sessionManager = new SessionManagerService(twilioClient);
}
/**
* Creates a new call session
* @param ws The WebSocket connection
* @param callType The type of call
*/
public createSession(ws: WebSocket, callType: CallType): void {
this.sessionManager.createSession(ws, callType);
}
}
```