Real-TimeVoice Assistant | ElevenLabs & React Guide

Build a real-time voice assistant using ElevenLabs Voice Agents and React. This guide simplifies voice AI development, creating interactive conversational apps with a single API.

Real-TimeVoice Assistant | ElevenLabs & React Guide

Voice AI has evolved rapidly in the last few years. Earlier, building a voice assistant meant manually combining multiple systems like Speech-to-Text, large language models, and Text-to-Speech engines. This made development complex, slow, and difficult to scale.

Voice AI has evolved rapidly in the last few years. Earlier, building a voice assistant meant manually combining multiple systems like Speech-to-Text, large language models, and Text-to-Speech engines. This made development complex, slow, and difficult to scale. Today, platforms like ElevenLabs Voice Agents simplify this entire process. Instead of stitching together multiple services, you can now build a complete conversational voice assistant using a single API. In this guide, we’ll walk through how to build a real-time voice assistant using ElevenLabs and React. The goal is simple: a user speaks into the browser, and the AI responds back with a natural voice in real time. We’ll use: ElevenLabs Agent API React frontend Browser microphone access By the end, you’ll have a working voice assistant you can actually interact with in your browser. Why ElevenLabs Voice Agents Make Things Easier Traditionally, a voice assistant is built using three separate components: Speech-to-Text (to understand user input) LLM (to generate a response) Text-to-Speech (to speak the response back) While this works, it introduces complexity. You need to manage multiple APIs, handle data flow between them, and deal with added latency and failure points. ElevenLabs simplifies this entire architecture by combining everything into a single Voice Agent. So instead of managing multiple services, your application only needs to communicate with: 👉 A single Agent API This significantly reduces complexity and speeds up development. How the Flow Works (High-Level Overview) Before jumping into code, it helps to understand the flow of interaction: The user speaks into the microphone → The React app captures the audio → The audio is sent to ElevenLabs Agent API → The agent processes the request and generates a response → The AI-generated voice is returned and played in the browser In simple terms: Microphone → React App → ElevenLabs Agent → Voice Response Setting Up Your Voice Agent in ElevenLabs Before writing any code, we first need to create and configure the voice agent. Start by logging into the ElevenLabs dashboard and navigating to the Agents section inside Voice Lab. From there, create a new agent. When setting it up, you define a few basic things: A name for the agent (for example, “Customer Support Assistant”) The personality and behavior style (friendly, professional, concise, etc.) A system instruction that guides how the agent responds For example: You are a helpful assistant that answers clearly and concisely. You also select a voice and language depending on your use case. Once the agent is created, ElevenLabs provides two important values: Agent ID API Key These are what your React application will use to communicate with the agent. Building the React Voice Interface Now that the backend setup is ready, let’s build a simple frontend interface. The idea here is not to create a complex UI, but to focus on interaction: Start recording voice Stop recording Send audio to the AI Play the response This keeps the experience minimal and focused. React Implementation Below is a simple implementation that handles microphone input and communicates with the ElevenLabs agent: import React, { useState, useRef } from "react"; import axios from "axios"; function App() { const [recording, setRecording] = useState(false); const mediaRecorderRef = useRef(null); const audioChunks = useRef([]); const startRecording = async () => { const stream = await navigator.mediaDevices.getUserMedia({ audio: true }); const mediaRecorder = new MediaRecorder(stream); mediaRecorderRef.current = mediaRecorder; audioChunks.current = []; mediaRecorder.ondataavailable = (e) => { audioChunks.current.push(e.data); }; mediaRecorder.onstop = sendToAgent; mediaRecorder.start(); setRecording(true); }; const stopRecording = () => { mediaRecorderRef.current.stop(); setRecording(false); }; const sendToAgent = async () => { const audioBlob = new Blob(audioChunks.current, { type: "audio/webm" }); const formData = new FormData(); formData.append("audio", audioBlob); const response = await axios.post( "https://api.elevenlabs.io/v1/agents/YOUR_AGENT_ID/interact", formData, { headers: { "xi-api-key": "YOUR_API_KEY" }, responseType: "arraybuffer" } ); const audioBlobResponse = new Blob([response.data], { type: "audio/mpeg" }); const audioUrl = URL.createObjectURL(audioBlobResponse); const audio = new Audio(audioUrl); audio.play(); }; return (

🎙️ Voice Assistant

{recording ? "Stop" : "Start Talking"}

{recording ? "Listening..." : "Idle"}

); } export default App; What’s Happening Behind the Scenes When the user interacts with the app, a single API call handles the entire voice pipeline. The request is sent to: POST /v1/agents/{agent_id}/interact The API expects: The user’s audio input An API key in headers And it returns: A generated audio response from the agent This response is then played directly in the browser. Why This Approach Works So Well Compared to a traditional voice AI setup, the difference is significant. With a traditional system, you would manage: Speech-to-text separately A language model separately Text-to-speech separately But with ElevenLabs: Everything is handled through one API The integration is much simpler Response time is optimized Code complexity is significantly reduced This makes it ideal for fast development and production-ready prototypes. What You Can Build Next Once the basic version is working, there are several meaningful upgrades you can explore: One improvement is real-time streaming, where responses are played as they are generated instead of waiting for the full output. Another enhancement is combining text and voice, where you display the conversation alongside audio playback for better UX. You can also introduce voice switching, allowing users to choose different AI voices dynamically. Finally, adding interruption support can make the assistant feel more natural by letting users interrupt while the AI is speaking. Important Considerations Before Production Before deploying something like this in a real application, a few things should be kept in mind. Always ensure microphone permissions are handled properly. Audio input is sensitive and requires user consent. It’s also important to limit or manage audio size to avoid performance issues. Never expose your API key directly in a frontend application in production. It should always be secured through a backend layer. Finally, keep an eye on API usage, as real-time voice processing can become costly at scale.