The Gemini Live API allows for real-time, bidirectional interaction with Gemini models, supporting audio, video, and text inputs and native audio outputs. This guide explains how to integrate directly with the API using raw WebSockets.
Overview
The Gemini Live API uses WebSockets for real-time communication. Unlike using an SDK, this approach involves directly managing the WebSocket connection and sending/receiving messages in a specific JSON format defined by the API.
Key concepts:
- WebSocket Endpoint: The specific URL to connect to.
- Message Format: All communication is done via JSON messages conforming to
LiveSessionRequestandLiveSessionResponsestructures. - Session Management: You are responsible for maintaining the WebSocket connection.
Authentication
Authentication is handled by including your API key as a query parameter in the WebSocket URL.
The endpoint format is:
wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=YOUR_API_KEY
Replace YOUR_API_KEY with your actual API key.
Authentication with Ephemeral Tokens
If you are using ephemeral tokens, you need to connect to the v1alpha endpoint.
The ephemeral token needs to be passed as an access_token query parameter.
The endpoint format for ephemeral keys is:
wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContentConstrained?access_token={short-lived-token}
Replace {short-lived-token} with the actual ephemeral token.
Connecting to the Live API
To start a live session, establish a WebSocket connection to the authenticated endpoint.
The first message sent over the WebSocket must be a LiveSessionRequest containing the config.
For the full configuration options, see the Live API - WebSockets API reference.
Python
import asyncio
import websockets
import json
API_KEY = "YOUR_API_KEY"
MODEL_NAME = "gemini-2.5-flash-native-audio-preview-12-2025"
WS_URL = f"wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key={API_KEY}"
async def connect_and_configure():
async with websockets.connect(WS_URL) as websocket:
print("WebSocket Connected")
# 1. Send the initial configuration
config_message = {
"config": {
"model": f"models/{MODEL_NAME}",
"responseModalities": ["AUDIO"],
"systemInstruction": {
"parts": [{"text": "You are a helpful assistant."}]
}
}
}
await websocket.send(json.dumps(config_message))
print("Configuration sent")
# Keep the session alive for further interactions
await asyncio.sleep(3600) # Example: keep open for an hour
async def main():
await connect_and_configure()
if __name__ == "__main__":
asyncio.run(main())
JavaScript
const API_KEY = "YOUR_API_KEY";
const MODEL_NAME = "gemini-2.5-flash-native-audio-preview-12-2025";
const WS_URL = `wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=${API_KEY}`;
const websocket = new WebSocket(WS_URL);
websocket.onopen = () => {
console.log('WebSocket Connected');
// 1. Send the initial configuration
const configMessage = {
config: {
model: `models/${MODEL_NAME}`,
responseModalities: ['AUDIO'],
systemInstruction: {
parts: [{ text: 'You are a helpful assistant.' }]
}
}
};
websocket.send(JSON.stringify(configMessage));
console.log('Configuration sent');
};
websocket.onmessage = (event) => {
const response = JSON.parse(event.data);
console.log('Received:', response);
// Handle different types of responses here
};
websocket.onerror = (error) => {
console.error('WebSocket Error:', error);
};
websocket.onclose = () => {
console.log('WebSocket Closed');
};
Sending text
To send text input, construct a LiveSessionRequest with the realtimeInput field populated with text.
Python
# Inside the websocket context
async def send_text(websocket, text):
text_message = {
"realtimeInput": {
"text": text
}
}
await websocket.send(json.dumps(text_message))
print(f"Sent text: {text}")
# Example usage: await send_text(websocket, "Hello, how are you?")
JavaScript
function sendTextMessage(text) {
if (websocket.readyState === WebSocket.OPEN) {
const textMessage = {
realtimeInput: {
text: text
}
};
websocket.send(JSON.stringify(textMessage));
console.log('Text message sent:', text);
} else {
console.warn('WebSocket not open.');
}
}
// Example usage:
sendTextMessage("Hello, how are you?");
Sending audio
Audio needs to be sent as raw PCM data (raw 16-bit PCM audio, 16kHz, little-endian). Construct a LiveSessionRequest with the realtimeInput field, containing a Blob with the audio data. The mimeType is crucial.
Python
# Inside the websocket context
async def send_audio_chunk(websocket, chunk_bytes):
import base64
encoded_data = base64.b64encode(chunk_bytes).decode('utf-8')
audio_message = {
"realtimeInput": {
"audio": {
"data": encoded_data,
"mimeType": "audio/pcm;rate=16000"
}
}
}
await websocket.send(json.dumps(audio_message))
# print("Sent audio chunk") # Avoid excessive logging
# Assuming 'chunk' is your raw PCM audio bytes
# await send_audio_chunk(websocket, chunk)
JavaScript
// Assuming 'chunk' is a Buffer of raw PCM audio
function sendAudioChunk(chunk) {
if (websocket.readyState === WebSocket.OPEN) {
const audioMessage = {
realtimeInput: {
audio: {
data: chunk.toString('base64'),
mimeType: 'audio/pcm;rate=16000'
}
}
};
websocket.send(JSON.stringify(audioMessage));
// console.log('Sent audio chunk');
}
}
// Example usage: sendAudioChunk(audioBuffer);
For an example of how to get the audio from the client device (e.g. the browser) see the end-to-end example on GitHub.
Sending video
Video frames are sent as individual images (e.g., JPEG or PNG). Similar to audio, use realtimeInput with a Blob, specifying the correct mimeType.
Python
# Inside the websocket context
async def send_video_frame(websocket, frame_bytes, mime_type="image/jpeg"):
import base64
encoded_data = base64.b64encode(frame_bytes).decode('utf-8')
video_message = {
"realtimeInput": {
"video": {
"data": encoded_data,
"mimeType": mime_type
}
}
}
await websocket.send(json.dumps(video_message))
# print("Sent video frame")
# Assuming 'frame' is your JPEG-encoded image bytes
# await send_video_frame(websocket, frame)
JavaScript
// Assuming 'frame' is a Buffer of JPEG-encoded image data
function sendVideoFrame(frame, mimeType = 'image/jpeg') {
if (websocket.readyState === WebSocket.OPEN) {
const videoMessage = {
realtimeInput: {
video: {
data: frame.toString('base64'),
mimeType: mimeType
}
}
};
websocket.send(JSON.stringify(videoMessage));
// console.log('Sent video frame');
}
}
// Example usage: sendVideoFrame(jpegBuffer);
For an example of how to get the video from the client device (e.g. the browser) see the end-to-end example on GitHub.
Receiving responses
The WebSocket will send back LiveSessionResponse messages. You need to parse these JSON messages and handle different types of content.
Python
# Inside the websocket context, in a receive loop
async def receive_loop(websocket):
async for message in websocket:
response = json.loads(message)
print("Received:", response)
if "serverContent" in response:
server_content = response["serverContent"]
# Receiving Audio
if "modelTurn" in server_content and "parts" in server_content["modelTurn"]:
for part in server_content["modelTurn"]["parts"]:
if "inlineData" in part:
audio_data_b64 = part["inlineData"]["data"]
# Process or play the base64 encoded audio data
# audio_data = base64.b64decode(audio_data_b64)
print(f"Received audio data (base64 len: {len(audio_data_b64)})")
# Receiving Text Transcriptions
if "inputTranscription" in server_content:
print(f"User: {server_content['inputTranscription']['text']}")
if "outputTranscription" in server_content:
print(f"Gemini: {server_content['outputTranscription']['text']}")
# Handling Tool Calls
if "toolCall" in response:
await handle_tool_call(websocket, response["toolCall"])
# Example usage: await receive_loop(websocket)
For an example of how to handle the response, see the end-to-end example on GitHub.
JavaScript
websocket.onmessage = (event) => {
const response = JSON.parse(event.data);
console.log('Received:', response);
if (response.serverContent) {
const serverContent = response.serverContent;
// Receiving Audio
if (serverContent.modelTurn?.parts) {
for (const part of serverContent.modelTurn.parts) {
if (part.inlineData) {
const audioData = part.inlineData.data; // Base64 encoded string
// Process or play audioData
console.log(`Received audio data (base64 len: ${audioData.length})`);
}
}
}
// Receiving Text Transcriptions
if (serverContent.inputTranscription) {
console.log('User:', serverContent.inputTranscription.text);
}
if (serverContent.outputTranscription) {
console.log('Gemini:', serverContent.outputTranscription.text);
}
}
// Handling Tool Calls
if (response.toolCall) {
handleToolCall(response.toolCall);
}
};
Handling tool calls
When the model requests a tool call, the LiveSessionResponse will contain a toolCall field. You must execute the function locally and send the result back to the WebSocket using a LiveSessionRequest with the toolResponse field.
Python
# Placeholder for your tool function
def my_tool_function(args):
print(f"Executing tool with args: {args}")
# Implement your tool logic here
return {"status": "success", "data": "some result"}
async def handle_tool_call(websocket, tool_call):
function_responses = []
for fc in tool_call["functionCalls"]:
# 1. Execute the function locally
try:
result = my_tool_function(fc.get("args", {}))
response_data = {"result": result}
except Exception as e:
print(f"Error executing tool {fc['name']}: {e}")
response_data = {"error": str(e)}
# 2. Prepare the response
function_responses.append({
"name": fc["name"],
"id": fc["id"],
"response": response_data
})
# 3. Send the tool response back to the session
tool_response_message = {
"toolResponse": {
"functionResponses": function_responses
}
}
await websocket.send(json.dumps(tool_response_message))
print("Sent tool response")
# This function is called within the receive_loop when a toolCall is detected.
JavaScript
// Placeholder for your tool function
function myToolFunction(args) {
console.log(`Executing tool with args:`, args);
// Implement your tool logic here
return { status: 'success', data: 'some result' };
}
function handleToolCall(toolCall) {
const functionResponses = [];
for (const fc of toolCall.functionCalls) {
// 1. Execute the function locally
let result;
try {
result = myToolFunction(fc.args || {});
} catch (e) {
console.error(`Error executing tool ${fc.name}:`, e);
result = { error: e.message };
}
// 2. Prepare the response
functionResponses.push({
name: fc.name,
id: fc.id,
response: { result }
});
}
// 3. Send the tool response back to the session
if (websocket.readyState === WebSocket.OPEN) {
const toolResponseMessage = {
toolResponse: {
functionResponses: functionResponses
}
};
websocket.send(JSON.stringify(toolResponseMessage));
console.log('Sent tool response');
} else {
console.warn('WebSocket not open to send tool response.');
}
}
// This function is called within websocket.onmessage when a toolCall is detected.
What's next
- Read the full Live API Capabilities guide for key capabilities and configurations; including Voice Activity Detection and native audio features.
- Read the Tool use guide to learn how to integrate Live API with tools and function calling.
- Read the Session management guide for managing long running conversations.
- Read the Ephemeral tokens guide for secure authentication in client-to-server applications.
- For more information about the underlying WebSockets API, see the WebSockets API reference.