# Gemini Webhooks API Note: This API is in Beta. Endpoints are under `/v1beta/`. Webhooks allow the Gemini API to push real-time notifications to your server when asynchronous or Long-Running Operations (LROs) complete. This replaces the need to poll the API for status updates, reducing latency and overhead. ## Webhooks ### CreateWebhook `POST https://generativelanguage.googleapis.com/v1beta/webhooks` Creates a new Webhook. #### Request Body - **name** (`string`) Optional. The user-provided name of the webhook. - **uri** (`string`) *(Required)* Required. The URI to which webhook events will be sent. - **subscribed_events** (`array (enum (string))`) *(Required)* Required. The events that the webhook is subscribed to. Available events: - batch.succeeded - batch.expired - batch.failed - interaction.requires_action - interaction.completed - interaction.failed - video.generated Possible values: - `batch.succeeded`: Batch processing finished successfully. - `batch.expired`: Batch has not been processed within the 48h timeframe. - `batch.failed`: Batch job failed. - `interaction.requires_action`: Interaction requires action (e.g., function calling). - `interaction.completed`: Interaction completed successfully. - `interaction.failed`: Interaction failed. - `video.generated`: Video generation completed. #### Response - **name** (`string`) Optional. The user-provided name of the webhook. - **uri** (`string`) Required. The URI to which webhook events will be sent. - **subscribed_events** (`array (enum (string))`) Required. The events that the webhook is subscribed to. Available events: - batch.succeeded - batch.expired - batch.failed - interaction.requires_action - interaction.completed - interaction.failed - video.generated Possible values: - `batch.succeeded`: Batch processing finished successfully. - `batch.expired`: Batch has not been processed within the 48h timeframe. - `batch.failed`: Batch job failed. - `interaction.requires_action`: Interaction requires action (e.g., function calling). - `interaction.completed`: Interaction completed successfully. - `interaction.failed`: Interaction failed. - `video.generated`: Video generation completed. - **create_time** (`string`) Output only. The timestamp when the webhook was created. - **update_time** (`string`) Output only. The timestamp when the webhook was last updated. - **signing_secrets** (`array (SigningSecret)`) Output only. The signing secrets associated with this webhook. - **truncated_secret** (`string`) Output only. The truncated version of the signing secret. - **expire_time** (`string`) Output only. The expiration date of the signing secret. - **state** (`enum (string)`) Output only. The state of the webhook. Possible values: - `enabled`: The webhook is enabled. - `disabled`: The webhook is disabled by the user. - `disabled_due_to_failed_deliveries`: The webhook is disabled due to failed deliveries. - **new_signing_secret** (`string`) Output only. The new signing secret for the webhook. Only populated on create. - **id** (`string`) Output only. The ID of the webhook. #### Examples **Example** **REST** ```sh curl -X POST 'https://generativelanguage.googleapis.com/v1beta/webhooks' \ -H 'Content-Type: application/json' \ -d '{ "name": "string", "uri": "string", "subscribed_events": [ "batch.succeeded" ], "create_time": "string", "update_time": "string", "signing_secrets": [ { "truncated_secret": "string", "expire_time": "string" } ], "state": "enabled", "new_signing_secret": "string", "id": "string" }' ``` Response: ```json { "name": "string", "uri": "string", "subscribed_events": [ "batch.succeeded" ], "create_time": "string", "update_time": "string", "signing_secrets": [ { "truncated_secret": "string", "expire_time": "string" } ], "state": "enabled", "new_signing_secret": "string", "id": "string" } ``` --- ### PingWebhook `POST https://generativelanguage.googleapis.com/v1beta/webhooks/{id}:ping` Sends a ping event to a Webhook. #### Parameters - **id** (`string`) *(Required)* Required. The ID of the webhook to ping. Format: `{webhook_id}` #### Request Body #### Response #### Examples **Example** **REST** ```sh curl -X POST 'https://generativelanguage.googleapis.com/v1beta/webhooks/{id}:ping' \ -H 'Content-Type: application/json' \ ``` --- ### RotateSigningSecret `POST https://generativelanguage.googleapis.com/v1beta/webhooks/{id}:rotateSigningSecret` Generates a new signing secret for a Webhook. #### Parameters - **id** (`string`) *(Required)* Required. The ID of the webhook for which to generate a signing secret. Format: `{webhook_id}` #### Request Body - **revocation_behavior** (`enum (string)`) Optional. The revocation behavior for previous signing secrets. Possible values: - `revoke_previous_secrets_after_h24`: Generate a new signing secret and revoke all previous secrets after 24 hours. Default and safest option for migrations. - `revoke_previous_secrets_immediately`: Revoke all previous secrets immediately. Use with caution as this can interrupt ongoing notifications. #### Response - **secret** (`string`) Output only. The newly generated signing secret. #### Examples **Example** **REST** ```sh curl -X POST 'https://generativelanguage.googleapis.com/v1beta/webhooks/{id}:rotateSigningSecret' \ -H 'Content-Type: application/json' \ -d '{ "revocation_behavior": "revoke_previous_secrets_after_h24" }' ``` Response: ```json { "secret": "string" } ``` --- ### ListWebhooks `GET https://generativelanguage.googleapis.com/v1beta/webhooks` Lists all Webhooks. #### Parameters - **page_size** (`integer`) Optional. The maximum number of webhooks to return. The service may return fewer than this value. If unspecified, at most 50 webhooks will be returned. The maximum value is 1000. - **page_token** (`string`) Optional. A page token, received from a previous `ListWebhooks` call. Provide this to retrieve the subsequent page. #### Response - **webhooks** (`array (Webhook)`) The webhooks. - **next_page_token** (`string`) A token, which can be sent as `page_token` to retrieve the next page. If this field is omitted, there are no subsequent pages. #### Examples **Example** **REST** ```sh curl -X GET 'https://generativelanguage.googleapis.com/v1beta/webhooks' \ -H 'Content-Type: application/json' \ ``` Response: ```json { "webhooks": [ { "name": "string", "uri": "string", "subscribed_events": [ "batch.succeeded" ], "create_time": "string", "update_time": "string", "signing_secrets": [ { "truncated_secret": "string", "expire_time": "string" } ], "state": "enabled", "new_signing_secret": "string", "id": "string" } ], "next_page_token": "string" } ``` --- ### GetWebhook `GET https://generativelanguage.googleapis.com/v1beta/webhooks/{id}` Gets a specific Webhook. #### Parameters - **id** (`string`) *(Required)* Required. The ID of the webhook to retrieve. #### Response - **name** (`string`) Optional. The user-provided name of the webhook. - **uri** (`string`) Required. The URI to which webhook events will be sent. - **subscribed_events** (`array (enum (string))`) Required. The events that the webhook is subscribed to. Available events: - batch.succeeded - batch.expired - batch.failed - interaction.requires_action - interaction.completed - interaction.failed - video.generated Possible values: - `batch.succeeded`: Batch processing finished successfully. - `batch.expired`: Batch has not been processed within the 48h timeframe. - `batch.failed`: Batch job failed. - `interaction.requires_action`: Interaction requires action (e.g., function calling). - `interaction.completed`: Interaction completed successfully. - `interaction.failed`: Interaction failed. - `video.generated`: Video generation completed. - **create_time** (`string`) Output only. The timestamp when the webhook was created. - **update_time** (`string`) Output only. The timestamp when the webhook was last updated. - **signing_secrets** (`array (SigningSecret)`) Output only. The signing secrets associated with this webhook. - **truncated_secret** (`string`) Output only. The truncated version of the signing secret. - **expire_time** (`string`) Output only. The expiration date of the signing secret. - **state** (`enum (string)`) Output only. The state of the webhook. Possible values: - `enabled`: The webhook is enabled. - `disabled`: The webhook is disabled by the user. - `disabled_due_to_failed_deliveries`: The webhook is disabled due to failed deliveries. - **new_signing_secret** (`string`) Output only. The new signing secret for the webhook. Only populated on create. - **id** (`string`) Output only. The ID of the webhook. #### Examples **Example** **REST** ```sh curl -X GET 'https://generativelanguage.googleapis.com/v1beta/webhooks/{id}' \ -H 'Content-Type: application/json' \ ``` Response: ```json { "name": "string", "uri": "string", "subscribed_events": [ "batch.succeeded" ], "create_time": "string", "update_time": "string", "signing_secrets": [ { "truncated_secret": "string", "expire_time": "string" } ], "state": "enabled", "new_signing_secret": "string", "id": "string" } ``` --- ### UpdateWebhook `PATCH https://generativelanguage.googleapis.com/v1beta/webhooks/{id}` Updates an existing Webhook. #### Parameters - **id** (`string`) *(Required)* Required. The ID of the webhook to update. - **update_mask** (`string`) Optional. The list of fields to update. #### Request Body - **name** (`string`) Optional. The user-provided name of the webhook. - **uri** (`string`) Optional. The URI to which webhook events will be sent. - **subscribed_events** (`array (enum (string))`) Optional. The events that the webhook is subscribed to. Available events: - batch.succeeded - batch.expired - batch.failed - interaction.requires_action - interaction.completed - interaction.failed - video.generated Possible values: - `batch.succeeded`: Batch processing finished successfully. - `batch.expired`: Batch has not been processed within the 48h timeframe. - `batch.failed`: Batch job failed. - `interaction.requires_action`: Interaction requires action (e.g., function calling). - `interaction.completed`: Interaction completed successfully. - `interaction.failed`: Interaction failed. - `video.generated`: Video generation completed. - **state** (`enum (string)`) Optional. The state of the webhook. Possible values: - `enabled`: The webhook is enabled. - `disabled`: The webhook is disabled by the user. - `disabled_due_to_failed_deliveries`: The webhook is disabled due to failed deliveries. #### Response - **name** (`string`) Optional. The user-provided name of the webhook. - **uri** (`string`) Required. The URI to which webhook events will be sent. - **subscribed_events** (`array (enum (string))`) Required. The events that the webhook is subscribed to. Available events: - batch.succeeded - batch.expired - batch.failed - interaction.requires_action - interaction.completed - interaction.failed - video.generated Possible values: - `batch.succeeded`: Batch processing finished successfully. - `batch.expired`: Batch has not been processed within the 48h timeframe. - `batch.failed`: Batch job failed. - `interaction.requires_action`: Interaction requires action (e.g., function calling). - `interaction.completed`: Interaction completed successfully. - `interaction.failed`: Interaction failed. - `video.generated`: Video generation completed. - **create_time** (`string`) Output only. The timestamp when the webhook was created. - **update_time** (`string`) Output only. The timestamp when the webhook was last updated. - **signing_secrets** (`array (SigningSecret)`) Output only. The signing secrets associated with this webhook. - **truncated_secret** (`string`) Output only. The truncated version of the signing secret. - **expire_time** (`string`) Output only. The expiration date of the signing secret. - **state** (`enum (string)`) Output only. The state of the webhook. Possible values: - `enabled`: The webhook is enabled. - `disabled`: The webhook is disabled by the user. - `disabled_due_to_failed_deliveries`: The webhook is disabled due to failed deliveries. - **new_signing_secret** (`string`) Output only. The new signing secret for the webhook. Only populated on create. - **id** (`string`) Output only. The ID of the webhook. #### Examples **Example** **REST** ```sh curl -X PATCH 'https://generativelanguage.googleapis.com/v1beta/webhooks/{id}' \ -H 'Content-Type: application/json' \ -d '{ "name": "string", "uri": "string", "subscribed_events": [ "batch.succeeded" ], "state": "enabled" }' ``` Response: ```json { "name": "string", "uri": "string", "subscribed_events": [ "batch.succeeded" ], "create_time": "string", "update_time": "string", "signing_secrets": [ { "truncated_secret": "string", "expire_time": "string" } ], "state": "enabled", "new_signing_secret": "string", "id": "string" } ``` --- ### DeleteWebhook `DELETE https://generativelanguage.googleapis.com/v1beta/webhooks/{id}` Deletes a Webhook. #### Parameters - **id** (`string`) *(Required)* Required. The ID of the webhook to delete. Format: `{webhook_id}` #### Response #### Examples **Example** **REST** ```sh curl -X DELETE 'https://generativelanguage.googleapis.com/v1beta/webhooks/{id}' \ -H 'Content-Type: application/json' \ ``` --- ## Resources ### Webhook { #Resource:Webhook } A Webhook resource. **Properties:** - **name** (`string`) Optional. The user-provided name of the webhook. - **uri** (`string`) *(Required)* Required. The URI to which webhook events will be sent. - **subscribed_events** (`array (enum (string))`) *(Required)* Required. The events that the webhook is subscribed to. Available events: - batch.succeeded - batch.expired - batch.failed - interaction.requires_action - interaction.completed - interaction.failed - video.generated Possible values: - `batch.succeeded`: Batch processing finished successfully. - `batch.expired`: Batch has not been processed within the 48h timeframe. - `batch.failed`: Batch job failed. - `interaction.requires_action`: Interaction requires action (e.g., function calling). - `interaction.completed`: Interaction completed successfully. - `interaction.failed`: Interaction failed. - `video.generated`: Video generation completed. - **create_time** (`string`) Output only. The timestamp when the webhook was created. - **update_time** (`string`) Output only. The timestamp when the webhook was last updated. - **signing_secrets** (`array (SigningSecret)`) Output only. The signing secrets associated with this webhook. - **truncated_secret** (`string`) Output only. The truncated version of the signing secret. - **expire_time** (`string`) Output only. The expiration date of the signing secret. - **state** (`enum (string)`) Output only. The state of the webhook. Possible values: - `enabled`: The webhook is enabled. - `disabled`: The webhook is disabled by the user. - `disabled_due_to_failed_deliveries`: The webhook is disabled due to failed deliveries. - **new_signing_secret** (`string`) Output only. The new signing secret for the webhook. Only populated on create. - **id** (`string`) Output only. The ID of the webhook. **JSON Representation:** ```json { "name": "string", "uri": "string", "subscribed_events": [ "batch.succeeded" ], "create_time": "string", "update_time": "string", "signing_secrets": [ { "truncated_secret": "string", "expire_time": "string" } ], "state": "enabled", "new_signing_secret": "string", "id": "string" } ``` ## Data Models ### InteractionSseEvent { #Resource:InteractionSseEvent } **Polymorphic Types:** (Discriminator: `event_type`)- **InteractionCreatedEvent** - - **event_type** (`object`) *(Required)* Value: `interaction.created` - **event_id** (`string`) The event_id token to be used to resume the interaction stream, from this event. - **metadata** (`StreamMetadata`) Optional metadata accompanying ANY streamed event. - **total_usage** (`Usage`) - **total_input_tokens** (`integer`) Number of tokens in the prompt (context). - **input_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of input token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_cached_tokens** (`integer`) Number of tokens in the cached part of the prompt (the cached content). - **cached_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of cached token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_output_tokens** (`integer`) Total number of tokens across all the generated responses. - **output_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of output token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_tool_use_tokens** (`integer`) Number of tokens present in tool-use prompt(s). - **tool_use_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of tool-use token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_thought_tokens** (`integer`) Number of tokens of thoughts for thinking models. - **total_tokens** (`integer`) Total token count for the interaction request (prompt + responses + other internal tokens). - **grounding_tool_count** (`array (GroundingToolCount)`) Grounding tool count. - **type** (`enum (string)`) The grounding tool type associated with the count. Possible values: - `google_search`: Grounding with Google Web Search and Image Search, & Web Grounding for Enterprise. - `google_maps`: Grounding with Google Maps. - `retrieval`: Grounding with customer's data, for example, VertexAISearch. - **count** (`integer`) The number of grounding tool counts. - **interaction** (`InteractionSseEventInteraction`) *(Required)* Partial interaction resource emitted when the stream is created. - **id** (`string`) *(Required)* Required. Output only. A unique identifier for the interaction completion. - **object** (`string`) Output only. The resource type. - **model** (`string`) The model that will complete your prompt. - **agent** (`string`) The agent to interact with. - **status** (`enum (string)`) *(Required)* Required. Output only. The status of the interaction. Possible values: - `in_progress`: The interaction is in progress. - `requires_action`: The interaction requires action/input from the user. - `completed`: The interaction is completed. - `failed`: The interaction failed. - `cancelled`: The interaction was cancelled. - `incomplete`: The interaction is completed, but contains incomplete results (e.g. hitting max_tokens). - **created** (`string`) Output only. The time at which the response was created in ISO 8601 format. - **updated** (`string`) Output only. The time at which the response was last updated in ISO 8601 format. - **service_tier** (`ServiceTier`) The service tier for the interaction. Possible values: - `flex`: Flex service tier. - `standard`: Standard service tier. - `priority`: Priority service tier. - **usage** (`Usage`) Output only. Statistics on the interaction request's token usage. - **total_input_tokens** (`integer`) Number of tokens in the prompt (context). - **input_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of input token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_cached_tokens** (`integer`) Number of tokens in the cached part of the prompt (the cached content). - **cached_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of cached token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_output_tokens** (`integer`) Total number of tokens across all the generated responses. - **output_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of output token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_tool_use_tokens** (`integer`) Number of tokens present in tool-use prompt(s). - **tool_use_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of tool-use token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_thought_tokens** (`integer`) Number of tokens of thoughts for thinking models. - **total_tokens** (`integer`) Total token count for the interaction request (prompt + responses + other internal tokens). - **grounding_tool_count** (`array (GroundingToolCount)`) Grounding tool count. - **type** (`enum (string)`) The grounding tool type associated with the count. Possible values: - `google_search`: Grounding with Google Web Search and Image Search, & Web Grounding for Enterprise. - `google_maps`: Grounding with Google Maps. - `retrieval`: Grounding with customer's data, for example, VertexAISearch. - **count** (`integer`) The number of grounding tool counts. - **steps** (`array (Step)`) Output only. The steps that make up the interaction, if included in this event. **Possible Types:** (Discriminator: `type`) - **UserInputStep**: Input provided by the user. - **content** (`array (Content)`) **Possible Types:** (Discriminator: `type`) - **TextContent**: A text content block. - **type** (`object`) *(Required)* Value: `text` - **text** (`string`) *(Required)* Required. The text content. - **annotations** (`array (Annotation)`) Citation information for model-generated content. **Possible Types:** (Discriminator: `type`) - **UrlCitation**: A URL citation annotation. - **type** (`object`) *(Required)* Value: `url_citation` - **url** (`string`) The URL. - **title** (`string`) The title of the URL. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **FileCitation**: A file citation annotation. - **type** (`object`) *(Required)* Value: `file_citation` - **document_uri** (`string`) The URI of the file. - **file_name** (`string`) The name of the file. - **source** (`string`) Source attributed for a portion of the text. - **custom_metadata** (`object`) User provided metadata about the retrieved context. - **page_number** (`integer`) Page number of the cited document, if applicable. - **media_id** (`string`) Media ID in-case of image citations, if applicable. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **PlaceCitation**: A place citation annotation. - **type** (`object`) *(Required)* Value: `place_citation` - **place_id** (`string`) The ID of the place, in `places/{place_id}` format. - **name** (`string`) Title of the place. - **url** (`string`) URI reference of the place. - **review_snippets** (`array (ReviewSnippet)`) Snippets of reviews that are used to generate answers about the features of a given place in Google Maps. - **title** (`string`) Title of the review. - **url** (`string`) A link that corresponds to the user review on Google Maps. - **review_id** (`string`) The ID of the review snippet. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **ImageContent**: An image content block. - **type** (`object`) *(Required)* Value: `image` - **data** (`string`) The image content. - **uri** (`string`) The URI of the image. - **mime_type** (`enum (string)`) The mime type of the image. Possible values: - `image/png`: PNG image format - `image/jpeg`: JPEG image format - `image/webp`: WebP image format - `image/heic`: HEIC image format - `image/heif`: HEIF image format - `image/gif`: GIF image format - `image/bmp`: BMP image format - `image/tiff`: TIFF image format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **AudioContent**: An audio content block. - **type** (`object`) *(Required)* Value: `audio` - **data** (`string`) The audio content. - **uri** (`string`) The URI of the audio. - **mime_type** (`enum (string)`) The mime type of the audio. Possible values: - `audio/wav`: WAV audio format - `audio/mp3`: MP3 audio format - `audio/aiff`: AIFF audio format - `audio/aac`: AAC audio format - `audio/ogg`: OGG audio format - `audio/flac`: FLAC audio format - `audio/mpeg`: MPEG audio format - `audio/m4a`: M4A audio format - `audio/l16`: L16 audio format - `audio/opus`: OPUS audio format - `audio/alaw`: ALAW audio format - `audio/mulaw`: MULAW audio format - **channels** (`integer`) The number of audio channels. - **sample_rate** (`integer`) The sample rate of the audio. - **DocumentContent**: A document content block. - **type** (`object`) *(Required)* Value: `document` - **data** (`string`) The document content. - **uri** (`string`) The URI of the document. - **mime_type** (`enum (string)`) The mime type of the document. Possible values: - `application/pdf`: PDF document format - `text/csv`: CSV document format - **VideoContent**: A video content block. - **type** (`object`) *(Required)* Value: `video` - **data** (`string`) The video content. - **uri** (`string`) The URI of the video. - **mime_type** (`enum (string)`) The mime type of the video. Possible values: - `video/mp4`: MP4 video format - `video/mpeg`: MPEG video format - `video/mpg`: MPG video format - `video/mov`: MOV video format - `video/avi`: AVI video format - `video/x-flv`: FLV video format - `video/webm`: WebM video format - `video/wmv`: WMV video format - `video/3gpp`: 3GPP video format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **type** (`object`) *(Required)* Value: `user_input` - **ModelOutputStep**: Output generated by the model. - **type** (`object`) *(Required)* Value: `model_output` - **content** (`array (Content)`) **Possible Types:** (Discriminator: `type`) - **TextContent**: A text content block. - **type** (`object`) *(Required)* Value: `text` - **text** (`string`) *(Required)* Required. The text content. - **annotations** (`array (Annotation)`) Citation information for model-generated content. **Possible Types:** (Discriminator: `type`) - **UrlCitation**: A URL citation annotation. - **type** (`object`) *(Required)* Value: `url_citation` - **url** (`string`) The URL. - **title** (`string`) The title of the URL. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **FileCitation**: A file citation annotation. - **type** (`object`) *(Required)* Value: `file_citation` - **document_uri** (`string`) The URI of the file. - **file_name** (`string`) The name of the file. - **source** (`string`) Source attributed for a portion of the text. - **custom_metadata** (`object`) User provided metadata about the retrieved context. - **page_number** (`integer`) Page number of the cited document, if applicable. - **media_id** (`string`) Media ID in-case of image citations, if applicable. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **PlaceCitation**: A place citation annotation. - **type** (`object`) *(Required)* Value: `place_citation` - **place_id** (`string`) The ID of the place, in `places/{place_id}` format. - **name** (`string`) Title of the place. - **url** (`string`) URI reference of the place. - **review_snippets** (`array (ReviewSnippet)`) Snippets of reviews that are used to generate answers about the features of a given place in Google Maps. - **title** (`string`) Title of the review. - **url** (`string`) A link that corresponds to the user review on Google Maps. - **review_id** (`string`) The ID of the review snippet. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **ImageContent**: An image content block. - **type** (`object`) *(Required)* Value: `image` - **data** (`string`) The image content. - **uri** (`string`) The URI of the image. - **mime_type** (`enum (string)`) The mime type of the image. Possible values: - `image/png`: PNG image format - `image/jpeg`: JPEG image format - `image/webp`: WebP image format - `image/heic`: HEIC image format - `image/heif`: HEIF image format - `image/gif`: GIF image format - `image/bmp`: BMP image format - `image/tiff`: TIFF image format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **AudioContent**: An audio content block. - **type** (`object`) *(Required)* Value: `audio` - **data** (`string`) The audio content. - **uri** (`string`) The URI of the audio. - **mime_type** (`enum (string)`) The mime type of the audio. Possible values: - `audio/wav`: WAV audio format - `audio/mp3`: MP3 audio format - `audio/aiff`: AIFF audio format - `audio/aac`: AAC audio format - `audio/ogg`: OGG audio format - `audio/flac`: FLAC audio format - `audio/mpeg`: MPEG audio format - `audio/m4a`: M4A audio format - `audio/l16`: L16 audio format - `audio/opus`: OPUS audio format - `audio/alaw`: ALAW audio format - `audio/mulaw`: MULAW audio format - **channels** (`integer`) The number of audio channels. - **sample_rate** (`integer`) The sample rate of the audio. - **DocumentContent**: A document content block. - **type** (`object`) *(Required)* Value: `document` - **data** (`string`) The document content. - **uri** (`string`) The URI of the document. - **mime_type** (`enum (string)`) The mime type of the document. Possible values: - `application/pdf`: PDF document format - `text/csv`: CSV document format - **VideoContent**: A video content block. - **type** (`object`) *(Required)* Value: `video` - **data** (`string`) The video content. - **uri** (`string`) The URI of the video. - **mime_type** (`enum (string)`) The mime type of the video. Possible values: - `video/mp4`: MP4 video format - `video/mpeg`: MPEG video format - `video/mpg`: MPG video format - `video/mov`: MOV video format - `video/avi`: AVI video format - `video/x-flv`: FLV video format - `video/webm`: WebM video format - `video/wmv`: WMV video format - `video/3gpp`: 3GPP video format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **error** (`Status`) The error result of the operation in case of failure or cancellation. - **code** (`integer`) The status code, which should be an enum value of google.rpc.Code. - **message** (`string`) A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client. - **details** (`array (object)`) A list of messages that carry the error details. There is a common set of message types for APIs to use. - **ThoughtStep**: A thought step. - **type** (`object`) *(Required)* Value: `thought` - **signature** (`string`) A signature hash for backend validation. - **summary** (`array (ThoughtSummaryContent)`) A summary of the thought. **Possible Types:** (Discriminator: `type`) - **TextContent**: A text content block. - **type** (`object`) *(Required)* Value: `text` - **text** (`string`) *(Required)* Required. The text content. - **annotations** (`array (Annotation)`) Citation information for model-generated content. **Possible Types:** (Discriminator: `type`) - **UrlCitation**: A URL citation annotation. - **type** (`object`) *(Required)* Value: `url_citation` - **url** (`string`) The URL. - **title** (`string`) The title of the URL. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **FileCitation**: A file citation annotation. - **type** (`object`) *(Required)* Value: `file_citation` - **document_uri** (`string`) The URI of the file. - **file_name** (`string`) The name of the file. - **source** (`string`) Source attributed for a portion of the text. - **custom_metadata** (`object`) User provided metadata about the retrieved context. - **page_number** (`integer`) Page number of the cited document, if applicable. - **media_id** (`string`) Media ID in-case of image citations, if applicable. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **PlaceCitation**: A place citation annotation. - **type** (`object`) *(Required)* Value: `place_citation` - **place_id** (`string`) The ID of the place, in `places/{place_id}` format. - **name** (`string`) Title of the place. - **url** (`string`) URI reference of the place. - **review_snippets** (`array (ReviewSnippet)`) Snippets of reviews that are used to generate answers about the features of a given place in Google Maps. - **title** (`string`) Title of the review. - **url** (`string`) A link that corresponds to the user review on Google Maps. - **review_id** (`string`) The ID of the review snippet. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **ImageContent**: An image content block. - **type** (`object`) *(Required)* Value: `image` - **data** (`string`) The image content. - **uri** (`string`) The URI of the image. - **mime_type** (`enum (string)`) The mime type of the image. Possible values: - `image/png`: PNG image format - `image/jpeg`: JPEG image format - `image/webp`: WebP image format - `image/heic`: HEIC image format - `image/heif`: HEIF image format - `image/gif`: GIF image format - `image/bmp`: BMP image format - `image/tiff`: TIFF image format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **FunctionCallStep**: A function tool call step. - **type** (`object`) *(Required)* Value: `function_call` - **name** (`string`) *(Required)* Required. The name of the tool to call. - **arguments** (`object`) *(Required)* Required. The arguments to pass to the function. - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **CodeExecutionCallStep**: Code execution call step. - **type** (`object`) *(Required)* Value: `code_execution_call` - **arguments** (`CodeExecutionCallStepArguments`) *(Required)* Required. The arguments to pass to the code execution. - **language** (`enum (string)`) Programming language of the `code`. Possible values: - `python`: Python >= 3.10, with numpy and simpy available. - **code** (`string`) The code to be executed. - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **signature** (`string`) A signature hash for backend validation. - **UrlContextCallStep**: URL context call step. - **type** (`object`) *(Required)* Value: `url_context_call` - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **signature** (`string`) A signature hash for backend validation. - **arguments** (`UrlContextCallArguments`) *(Required)* The arguments to pass to the URL context. - **urls** (`array (string)`) The URLs to fetch. - **McpServerToolCallStep**: MCPServer tool call step. - **type** (`object`) *(Required)* Value: `mcp_server_tool_call` - **name** (`string`) *(Required)* Required. The name of the tool which was called. - **server_name** (`string`) *(Required)* Required. The name of the used MCP server. - **arguments** (`object`) *(Required)* Required. The JSON object of arguments for the function. - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **GoogleSearchCallStep**: Google Search call step. - **type** (`object`) *(Required)* Value: `google_search_call` - **arguments** (`GoogleSearchCallStepArguments`) *(Required)* Required. The arguments to pass to Google Search. - **queries** (`array (string)`) Web search queries for the following-up web search. - **search_type** (`enum (string)`) The type of search grounding enabled. Possible values: - `web_search`: Setting this field enables web search. Only text results are returned. - `image_search`: Setting this field enables image search. Image bytes are returned. - `enterprise_web_search`: Setting this field enables enterprise web search. - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **signature** (`string`) A signature hash for backend validation. - **FileSearchCallStep**: File Search call step. - **type** (`object`) *(Required)* Value: `file_search_call` - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **signature** (`string`) A signature hash for backend validation. - **GoogleMapsCallStep**: Google Maps call step. - **type** (`object`) *(Required)* Value: `google_maps_call` - **arguments** (`GoogleMapsCallStepArguments`) The arguments to pass to the Google Maps tool. - **queries** (`array (string)`) The queries to be executed. - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **signature** (`string`) A signature hash for backend validation. - **FunctionResultStep**: Result of a function tool call. - **type** (`object`) *(Required)* Value: `function_result` - **name** (`string`) The name of the tool that was called. - **is_error** (`boolean`) Whether the tool call resulted in an error. - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **result** (`array (ImageContent or TextContent) or object or string`) *(Required)* The result of the tool call. - **CodeExecutionResultStep**: Code execution result step. - **type** (`object`) *(Required)* Value: `code_execution_result` - **result** (`string`) *(Required)* Required. The output of the code execution. - **is_error** (`boolean`) Whether the code execution resulted in an error. - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **signature** (`string`) A signature hash for backend validation. - **UrlContextResultStep**: URL context result step. - **type** (`object`) *(Required)* Value: `url_context_result` - **result** (`array (UrlContextResult)`) *(Required)* Required. The results of the URL context. - **url** (`string`) The URL that was fetched. - **status** (`enum (string)`) The status of the URL retrieval. Possible values: - `success`: Url retrieval is successful. - `error`: Url retrieval is failed due to error. - `paywall`: Url retrieval is failed because the content is behind paywall. - `unsafe`: Url retrieval is failed because the content is unsafe. - **is_error** (`boolean`) Whether the URL context resulted in an error. - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **signature** (`string`) A signature hash for backend validation. - **GoogleSearchResultStep**: Google Search result step. - **type** (`object`) *(Required)* Value: `google_search_result` - **result** (`array (GoogleSearchResultItem)`) *(Required)* Required. The results of the Google Search. - **search_suggestions** (`string`) Web content snippet that can be embedded in a web page or an app webview. - **is_error** (`boolean`) Whether the Google Search resulted in an error. - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **signature** (`string`) A signature hash for backend validation. - **McpServerToolResultStep**: MCPServer tool result step. - **type** (`object`) *(Required)* Value: `mcp_server_tool_result` - **name** (`string`) Name of the tool which is called for this specific tool call. - **server_name** (`string`) The name of the used MCP server. - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **result** (`array (ImageContent or TextContent) or object or string`) *(Required)* The output from the MCP server call. Can be simple text or rich content. - **FileSearchResultStep**: File Search result step. - **type** (`object`) *(Required)* Value: `file_search_result` - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **signature** (`string`) A signature hash for backend validation. - **GoogleMapsResultStep**: Google Maps result step. - **type** (`object`) *(Required)* Value: `google_maps_result` - **result** (`array (GoogleMapsResultItem)`) *(Required)* - **places** (`array (GoogleMapsResultPlaces)`) - **place_id** (`string`) - **name** (`string`) - **url** (`string`) - **review_snippets** (`array (ReviewSnippet)`) - **title** (`string`) Title of the review. - **url** (`string`) A link that corresponds to the user review on Google Maps. - **review_id** (`string`) The ID of the review snippet. - **widget_context_token** (`string`) - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **signature** (`string`) A signature hash for backend validation. **Examples** **Interaction Created** ```json { "event_type": "interaction.created", "interaction": { "id": "v1_ChdXS0l4YWZXTk9xbk0xZThQczhEcmlROBIXV0tJeGFmV05PcW5NMWU4UHM4RHJpUTg", "model": "gemini-3.5-flash", "status": "in_progress", "created": "2025-12-04T15:01:45Z", "updated": "2025-12-04T15:01:45Z" }, "event_id": "evt_123" } ``` **Interaction Created** ```json { "event_type": "interaction.created", "interaction": { "id": "v1_ChdXS0l4YWZXTk9xbk0xZThQczhEcmlROBIXV0tJeGFmV05PcW5NMWU4UHM4RHJpUTg", "model": "gemini-3-flash-preview", "object": "interaction", "status": "in_progress" }, "event_id": "evt_123" } ``` - **InteractionCompletedEvent** - - **event_type** (`object`) *(Required)* Value: `interaction.completed` - **event_id** (`string`) The event_id token to be used to resume the interaction stream, from this event. - **metadata** (`StreamMetadata`) Optional metadata accompanying ANY streamed event. - **total_usage** (`Usage`) - **total_input_tokens** (`integer`) Number of tokens in the prompt (context). - **input_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of input token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_cached_tokens** (`integer`) Number of tokens in the cached part of the prompt (the cached content). - **cached_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of cached token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_output_tokens** (`integer`) Total number of tokens across all the generated responses. - **output_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of output token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_tool_use_tokens** (`integer`) Number of tokens present in tool-use prompt(s). - **tool_use_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of tool-use token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_thought_tokens** (`integer`) Number of tokens of thoughts for thinking models. - **total_tokens** (`integer`) Total token count for the interaction request (prompt + responses + other internal tokens). - **grounding_tool_count** (`array (GroundingToolCount)`) Grounding tool count. - **type** (`enum (string)`) The grounding tool type associated with the count. Possible values: - `google_search`: Grounding with Google Web Search and Image Search, & Web Grounding for Enterprise. - `google_maps`: Grounding with Google Maps. - `retrieval`: Grounding with customer's data, for example, VertexAISearch. - **count** (`integer`) The number of grounding tool counts. - **interaction** (`InteractionSseEventInteraction`) *(Required)* Partial completed interaction resource emitted at the end of the stream. - **id** (`string`) *(Required)* Required. Output only. A unique identifier for the interaction completion. - **object** (`string`) Output only. The resource type. - **model** (`string`) The model that will complete your prompt. - **agent** (`string`) The agent to interact with. - **status** (`enum (string)`) *(Required)* Required. Output only. The status of the interaction. Possible values: - `in_progress`: The interaction is in progress. - `requires_action`: The interaction requires action/input from the user. - `completed`: The interaction is completed. - `failed`: The interaction failed. - `cancelled`: The interaction was cancelled. - `incomplete`: The interaction is completed, but contains incomplete results (e.g. hitting max_tokens). - **created** (`string`) Output only. The time at which the response was created in ISO 8601 format. - **updated** (`string`) Output only. The time at which the response was last updated in ISO 8601 format. - **service_tier** (`ServiceTier`) The service tier for the interaction. Possible values: - `flex`: Flex service tier. - `standard`: Standard service tier. - `priority`: Priority service tier. - **usage** (`Usage`) Output only. Statistics on the interaction request's token usage. - **total_input_tokens** (`integer`) Number of tokens in the prompt (context). - **input_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of input token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_cached_tokens** (`integer`) Number of tokens in the cached part of the prompt (the cached content). - **cached_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of cached token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_output_tokens** (`integer`) Total number of tokens across all the generated responses. - **output_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of output token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_tool_use_tokens** (`integer`) Number of tokens present in tool-use prompt(s). - **tool_use_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of tool-use token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_thought_tokens** (`integer`) Number of tokens of thoughts for thinking models. - **total_tokens** (`integer`) Total token count for the interaction request (prompt + responses + other internal tokens). - **grounding_tool_count** (`array (GroundingToolCount)`) Grounding tool count. - **type** (`enum (string)`) The grounding tool type associated with the count. Possible values: - `google_search`: Grounding with Google Web Search and Image Search, & Web Grounding for Enterprise. - `google_maps`: Grounding with Google Maps. - `retrieval`: Grounding with customer's data, for example, VertexAISearch. - **count** (`integer`) The number of grounding tool counts. - **steps** (`array (Step)`) Output only. The steps that make up the interaction, if included in this event. **Possible Types:** (Discriminator: `type`) - **UserInputStep**: Input provided by the user. - **content** (`array (Content)`) **Possible Types:** (Discriminator: `type`) - **TextContent**: A text content block. - **type** (`object`) *(Required)* Value: `text` - **text** (`string`) *(Required)* Required. The text content. - **annotations** (`array (Annotation)`) Citation information for model-generated content. **Possible Types:** (Discriminator: `type`) - **UrlCitation**: A URL citation annotation. - **type** (`object`) *(Required)* Value: `url_citation` - **url** (`string`) The URL. - **title** (`string`) The title of the URL. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **FileCitation**: A file citation annotation. - **type** (`object`) *(Required)* Value: `file_citation` - **document_uri** (`string`) The URI of the file. - **file_name** (`string`) The name of the file. - **source** (`string`) Source attributed for a portion of the text. - **custom_metadata** (`object`) User provided metadata about the retrieved context. - **page_number** (`integer`) Page number of the cited document, if applicable. - **media_id** (`string`) Media ID in-case of image citations, if applicable. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **PlaceCitation**: A place citation annotation. - **type** (`object`) *(Required)* Value: `place_citation` - **place_id** (`string`) The ID of the place, in `places/{place_id}` format. - **name** (`string`) Title of the place. - **url** (`string`) URI reference of the place. - **review_snippets** (`array (ReviewSnippet)`) Snippets of reviews that are used to generate answers about the features of a given place in Google Maps. - **title** (`string`) Title of the review. - **url** (`string`) A link that corresponds to the user review on Google Maps. - **review_id** (`string`) The ID of the review snippet. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **ImageContent**: An image content block. - **type** (`object`) *(Required)* Value: `image` - **data** (`string`) The image content. - **uri** (`string`) The URI of the image. - **mime_type** (`enum (string)`) The mime type of the image. Possible values: - `image/png`: PNG image format - `image/jpeg`: JPEG image format - `image/webp`: WebP image format - `image/heic`: HEIC image format - `image/heif`: HEIF image format - `image/gif`: GIF image format - `image/bmp`: BMP image format - `image/tiff`: TIFF image format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **AudioContent**: An audio content block. - **type** (`object`) *(Required)* Value: `audio` - **data** (`string`) The audio content. - **uri** (`string`) The URI of the audio. - **mime_type** (`enum (string)`) The mime type of the audio. Possible values: - `audio/wav`: WAV audio format - `audio/mp3`: MP3 audio format - `audio/aiff`: AIFF audio format - `audio/aac`: AAC audio format - `audio/ogg`: OGG audio format - `audio/flac`: FLAC audio format - `audio/mpeg`: MPEG audio format - `audio/m4a`: M4A audio format - `audio/l16`: L16 audio format - `audio/opus`: OPUS audio format - `audio/alaw`: ALAW audio format - `audio/mulaw`: MULAW audio format - **channels** (`integer`) The number of audio channels. - **sample_rate** (`integer`) The sample rate of the audio. - **DocumentContent**: A document content block. - **type** (`object`) *(Required)* Value: `document` - **data** (`string`) The document content. - **uri** (`string`) The URI of the document. - **mime_type** (`enum (string)`) The mime type of the document. Possible values: - `application/pdf`: PDF document format - `text/csv`: CSV document format - **VideoContent**: A video content block. - **type** (`object`) *(Required)* Value: `video` - **data** (`string`) The video content. - **uri** (`string`) The URI of the video. - **mime_type** (`enum (string)`) The mime type of the video. Possible values: - `video/mp4`: MP4 video format - `video/mpeg`: MPEG video format - `video/mpg`: MPG video format - `video/mov`: MOV video format - `video/avi`: AVI video format - `video/x-flv`: FLV video format - `video/webm`: WebM video format - `video/wmv`: WMV video format - `video/3gpp`: 3GPP video format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **type** (`object`) *(Required)* Value: `user_input` - **ModelOutputStep**: Output generated by the model. - **type** (`object`) *(Required)* Value: `model_output` - **content** (`array (Content)`) **Possible Types:** (Discriminator: `type`) - **TextContent**: A text content block. - **type** (`object`) *(Required)* Value: `text` - **text** (`string`) *(Required)* Required. The text content. - **annotations** (`array (Annotation)`) Citation information for model-generated content. **Possible Types:** (Discriminator: `type`) - **UrlCitation**: A URL citation annotation. - **type** (`object`) *(Required)* Value: `url_citation` - **url** (`string`) The URL. - **title** (`string`) The title of the URL. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **FileCitation**: A file citation annotation. - **type** (`object`) *(Required)* Value: `file_citation` - **document_uri** (`string`) The URI of the file. - **file_name** (`string`) The name of the file. - **source** (`string`) Source attributed for a portion of the text. - **custom_metadata** (`object`) User provided metadata about the retrieved context. - **page_number** (`integer`) Page number of the cited document, if applicable. - **media_id** (`string`) Media ID in-case of image citations, if applicable. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **PlaceCitation**: A place citation annotation. - **type** (`object`) *(Required)* Value: `place_citation` - **place_id** (`string`) The ID of the place, in `places/{place_id}` format. - **name** (`string`) Title of the place. - **url** (`string`) URI reference of the place. - **review_snippets** (`array (ReviewSnippet)`) Snippets of reviews that are used to generate answers about the features of a given place in Google Maps. - **title** (`string`) Title of the review. - **url** (`string`) A link that corresponds to the user review on Google Maps. - **review_id** (`string`) The ID of the review snippet. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **ImageContent**: An image content block. - **type** (`object`) *(Required)* Value: `image` - **data** (`string`) The image content. - **uri** (`string`) The URI of the image. - **mime_type** (`enum (string)`) The mime type of the image. Possible values: - `image/png`: PNG image format - `image/jpeg`: JPEG image format - `image/webp`: WebP image format - `image/heic`: HEIC image format - `image/heif`: HEIF image format - `image/gif`: GIF image format - `image/bmp`: BMP image format - `image/tiff`: TIFF image format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **AudioContent**: An audio content block. - **type** (`object`) *(Required)* Value: `audio` - **data** (`string`) The audio content. - **uri** (`string`) The URI of the audio. - **mime_type** (`enum (string)`) The mime type of the audio. Possible values: - `audio/wav`: WAV audio format - `audio/mp3`: MP3 audio format - `audio/aiff`: AIFF audio format - `audio/aac`: AAC audio format - `audio/ogg`: OGG audio format - `audio/flac`: FLAC audio format - `audio/mpeg`: MPEG audio format - `audio/m4a`: M4A audio format - `audio/l16`: L16 audio format - `audio/opus`: OPUS audio format - `audio/alaw`: ALAW audio format - `audio/mulaw`: MULAW audio format - **channels** (`integer`) The number of audio channels. - **sample_rate** (`integer`) The sample rate of the audio. - **DocumentContent**: A document content block. - **type** (`object`) *(Required)* Value: `document` - **data** (`string`) The document content. - **uri** (`string`) The URI of the document. - **mime_type** (`enum (string)`) The mime type of the document. Possible values: - `application/pdf`: PDF document format - `text/csv`: CSV document format - **VideoContent**: A video content block. - **type** (`object`) *(Required)* Value: `video` - **data** (`string`) The video content. - **uri** (`string`) The URI of the video. - **mime_type** (`enum (string)`) The mime type of the video. Possible values: - `video/mp4`: MP4 video format - `video/mpeg`: MPEG video format - `video/mpg`: MPG video format - `video/mov`: MOV video format - `video/avi`: AVI video format - `video/x-flv`: FLV video format - `video/webm`: WebM video format - `video/wmv`: WMV video format - `video/3gpp`: 3GPP video format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **error** (`Status`) The error result of the operation in case of failure or cancellation. - **code** (`integer`) The status code, which should be an enum value of google.rpc.Code. - **message** (`string`) A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client. - **details** (`array (object)`) A list of messages that carry the error details. There is a common set of message types for APIs to use. - **ThoughtStep**: A thought step. - **type** (`object`) *(Required)* Value: `thought` - **signature** (`string`) A signature hash for backend validation. - **summary** (`array (ThoughtSummaryContent)`) A summary of the thought. **Possible Types:** (Discriminator: `type`) - **TextContent**: A text content block. - **type** (`object`) *(Required)* Value: `text` - **text** (`string`) *(Required)* Required. The text content. - **annotations** (`array (Annotation)`) Citation information for model-generated content. **Possible Types:** (Discriminator: `type`) - **UrlCitation**: A URL citation annotation. - **type** (`object`) *(Required)* Value: `url_citation` - **url** (`string`) The URL. - **title** (`string`) The title of the URL. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **FileCitation**: A file citation annotation. - **type** (`object`) *(Required)* Value: `file_citation` - **document_uri** (`string`) The URI of the file. - **file_name** (`string`) The name of the file. - **source** (`string`) Source attributed for a portion of the text. - **custom_metadata** (`object`) User provided metadata about the retrieved context. - **page_number** (`integer`) Page number of the cited document, if applicable. - **media_id** (`string`) Media ID in-case of image citations, if applicable. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **PlaceCitation**: A place citation annotation. - **type** (`object`) *(Required)* Value: `place_citation` - **place_id** (`string`) The ID of the place, in `places/{place_id}` format. - **name** (`string`) Title of the place. - **url** (`string`) URI reference of the place. - **review_snippets** (`array (ReviewSnippet)`) Snippets of reviews that are used to generate answers about the features of a given place in Google Maps. - **title** (`string`) Title of the review. - **url** (`string`) A link that corresponds to the user review on Google Maps. - **review_id** (`string`) The ID of the review snippet. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **ImageContent**: An image content block. - **type** (`object`) *(Required)* Value: `image` - **data** (`string`) The image content. - **uri** (`string`) The URI of the image. - **mime_type** (`enum (string)`) The mime type of the image. Possible values: - `image/png`: PNG image format - `image/jpeg`: JPEG image format - `image/webp`: WebP image format - `image/heic`: HEIC image format - `image/heif`: HEIF image format - `image/gif`: GIF image format - `image/bmp`: BMP image format - `image/tiff`: TIFF image format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **FunctionCallStep**: A function tool call step. - **type** (`object`) *(Required)* Value: `function_call` - **name** (`string`) *(Required)* Required. The name of the tool to call. - **arguments** (`object`) *(Required)* Required. The arguments to pass to the function. - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **CodeExecutionCallStep**: Code execution call step. - **type** (`object`) *(Required)* Value: `code_execution_call` - **arguments** (`CodeExecutionCallStepArguments`) *(Required)* Required. The arguments to pass to the code execution. - **language** (`enum (string)`) Programming language of the `code`. Possible values: - `python`: Python >= 3.10, with numpy and simpy available. - **code** (`string`) The code to be executed. - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **signature** (`string`) A signature hash for backend validation. - **UrlContextCallStep**: URL context call step. - **type** (`object`) *(Required)* Value: `url_context_call` - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **signature** (`string`) A signature hash for backend validation. - **arguments** (`UrlContextCallArguments`) *(Required)* The arguments to pass to the URL context. - **urls** (`array (string)`) The URLs to fetch. - **McpServerToolCallStep**: MCPServer tool call step. - **type** (`object`) *(Required)* Value: `mcp_server_tool_call` - **name** (`string`) *(Required)* Required. The name of the tool which was called. - **server_name** (`string`) *(Required)* Required. The name of the used MCP server. - **arguments** (`object`) *(Required)* Required. The JSON object of arguments for the function. - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **GoogleSearchCallStep**: Google Search call step. - **type** (`object`) *(Required)* Value: `google_search_call` - **arguments** (`GoogleSearchCallStepArguments`) *(Required)* Required. The arguments to pass to Google Search. - **queries** (`array (string)`) Web search queries for the following-up web search. - **search_type** (`enum (string)`) The type of search grounding enabled. Possible values: - `web_search`: Setting this field enables web search. Only text results are returned. - `image_search`: Setting this field enables image search. Image bytes are returned. - `enterprise_web_search`: Setting this field enables enterprise web search. - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **signature** (`string`) A signature hash for backend validation. - **FileSearchCallStep**: File Search call step. - **type** (`object`) *(Required)* Value: `file_search_call` - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **signature** (`string`) A signature hash for backend validation. - **GoogleMapsCallStep**: Google Maps call step. - **type** (`object`) *(Required)* Value: `google_maps_call` - **arguments** (`GoogleMapsCallStepArguments`) The arguments to pass to the Google Maps tool. - **queries** (`array (string)`) The queries to be executed. - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **signature** (`string`) A signature hash for backend validation. - **FunctionResultStep**: Result of a function tool call. - **type** (`object`) *(Required)* Value: `function_result` - **name** (`string`) The name of the tool that was called. - **is_error** (`boolean`) Whether the tool call resulted in an error. - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **result** (`array (ImageContent or TextContent) or object or string`) *(Required)* The result of the tool call. - **CodeExecutionResultStep**: Code execution result step. - **type** (`object`) *(Required)* Value: `code_execution_result` - **result** (`string`) *(Required)* Required. The output of the code execution. - **is_error** (`boolean`) Whether the code execution resulted in an error. - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **signature** (`string`) A signature hash for backend validation. - **UrlContextResultStep**: URL context result step. - **type** (`object`) *(Required)* Value: `url_context_result` - **result** (`array (UrlContextResult)`) *(Required)* Required. The results of the URL context. - **url** (`string`) The URL that was fetched. - **status** (`enum (string)`) The status of the URL retrieval. Possible values: - `success`: Url retrieval is successful. - `error`: Url retrieval is failed due to error. - `paywall`: Url retrieval is failed because the content is behind paywall. - `unsafe`: Url retrieval is failed because the content is unsafe. - **is_error** (`boolean`) Whether the URL context resulted in an error. - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **signature** (`string`) A signature hash for backend validation. - **GoogleSearchResultStep**: Google Search result step. - **type** (`object`) *(Required)* Value: `google_search_result` - **result** (`array (GoogleSearchResultItem)`) *(Required)* Required. The results of the Google Search. - **search_suggestions** (`string`) Web content snippet that can be embedded in a web page or an app webview. - **is_error** (`boolean`) Whether the Google Search resulted in an error. - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **signature** (`string`) A signature hash for backend validation. - **McpServerToolResultStep**: MCPServer tool result step. - **type** (`object`) *(Required)* Value: `mcp_server_tool_result` - **name** (`string`) Name of the tool which is called for this specific tool call. - **server_name** (`string`) The name of the used MCP server. - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **result** (`array (ImageContent or TextContent) or object or string`) *(Required)* The output from the MCP server call. Can be simple text or rich content. - **FileSearchResultStep**: File Search result step. - **type** (`object`) *(Required)* Value: `file_search_result` - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **signature** (`string`) A signature hash for backend validation. - **GoogleMapsResultStep**: Google Maps result step. - **type** (`object`) *(Required)* Value: `google_maps_result` - **result** (`array (GoogleMapsResultItem)`) *(Required)* - **places** (`array (GoogleMapsResultPlaces)`) - **place_id** (`string`) - **name** (`string`) - **url** (`string`) - **review_snippets** (`array (ReviewSnippet)`) - **title** (`string`) Title of the review. - **url** (`string`) A link that corresponds to the user review on Google Maps. - **review_id** (`string`) The ID of the review snippet. - **widget_context_token** (`string`) - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **signature** (`string`) A signature hash for backend validation. **Examples** **Interaction Completed** ```json { "event_type": "interaction.completed", "interaction": { "id": "v1_ChdXS0l4YWZXTk9xbk0xZThQczhEcmlROBIXV0tJeGFmV05PcW5NMWU4UHM4RHJpUTg", "model": "gemini-3.5-flash", "status": "completed", "created": "2025-12-04T15:01:45Z", "updated": "2025-12-04T15:01:45Z" }, "event_id": "evt_123" } ``` **Interaction Completed** ```json { "event_type": "interaction.completed", "interaction": { "id": "v1_ChdXS0l4YWZXTk9xbk0xZThQczhEcmlROBIXV0tJeGFmV05PcW5NMWU4UHM4RHJpUTg", "model": "gemini-3-flash-preview", "object": "interaction", "status": "completed", "created": "2025-12-04T15:01:45Z", "updated": "2025-12-04T15:01:45Z" }, "event_id": "evt_123" } ``` - **InteractionStatusUpdate** - - **event_type** (`object`) *(Required)* Value: `interaction.status_update` - **interaction_id** (`string`) *(Required)* - **status** (`enum (string)`) *(Required)* Possible values: - `in_progress`: The interaction is in progress. - `requires_action`: The interaction requires action/input from the user. - `completed`: The interaction is completed. - `failed`: The interaction failed. - `cancelled`: The interaction was cancelled. - `incomplete`: The interaction is completed, but contains incomplete results (e.g. hitting max_tokens). - `budget_exceeded`: The interaction was halted because the token budget was exceeded. - **event_id** (`string`) The event_id token to be used to resume the interaction stream, from this event. - **metadata** (`StreamMetadata`) Optional metadata accompanying ANY streamed event. - **total_usage** (`Usage`) - **total_input_tokens** (`integer`) Number of tokens in the prompt (context). - **input_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of input token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_cached_tokens** (`integer`) Number of tokens in the cached part of the prompt (the cached content). - **cached_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of cached token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_output_tokens** (`integer`) Total number of tokens across all the generated responses. - **output_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of output token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_tool_use_tokens** (`integer`) Number of tokens present in tool-use prompt(s). - **tool_use_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of tool-use token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_thought_tokens** (`integer`) Number of tokens of thoughts for thinking models. - **total_tokens** (`integer`) Total token count for the interaction request (prompt + responses + other internal tokens). - **grounding_tool_count** (`array (GroundingToolCount)`) Grounding tool count. - **type** (`enum (string)`) The grounding tool type associated with the count. Possible values: - `google_search`: Grounding with Google Web Search and Image Search, & Web Grounding for Enterprise. - `google_maps`: Grounding with Google Maps. - `retrieval`: Grounding with customer's data, for example, VertexAISearch. - **count** (`integer`) The number of grounding tool counts. **Examples** **Interaction Status Update** ```json { "event_type": "interaction.status_update", "interaction_id": "v1_ChdTMjQ0YWJ5TUF1TzcxZThQdjRpcnFRcxIXUzI0NGFieU1BdU83MWU4UHY0aXJxUXM", "status": "in_progress" } ``` - **ErrorEvent** - - **event_type** (`object`) *(Required)* Value: `error` - **error** (`Error`) - **code** (`string`) A URI that identifies the error type. - **message** (`string`) A human-readable error message. - **event_id** (`string`) The event_id token to be used to resume the interaction stream, from this event. - **metadata** (`StreamMetadata`) Optional metadata accompanying ANY streamed event. - **total_usage** (`Usage`) - **total_input_tokens** (`integer`) Number of tokens in the prompt (context). - **input_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of input token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_cached_tokens** (`integer`) Number of tokens in the cached part of the prompt (the cached content). - **cached_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of cached token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_output_tokens** (`integer`) Total number of tokens across all the generated responses. - **output_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of output token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_tool_use_tokens** (`integer`) Number of tokens present in tool-use prompt(s). - **tool_use_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of tool-use token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_thought_tokens** (`integer`) Number of tokens of thoughts for thinking models. - **total_tokens** (`integer`) Total token count for the interaction request (prompt + responses + other internal tokens). - **grounding_tool_count** (`array (GroundingToolCount)`) Grounding tool count. - **type** (`enum (string)`) The grounding tool type associated with the count. Possible values: - `google_search`: Grounding with Google Web Search and Image Search, & Web Grounding for Enterprise. - `google_maps`: Grounding with Google Maps. - `retrieval`: Grounding with customer's data, for example, VertexAISearch. - **count** (`integer`) The number of grounding tool counts. **Examples** **Error Event** ```json { "event_type": "error", "error": { "message": "Failed to get completed interaction: Result not found.", "code": "not_found" } } ``` - **StepStart** - - **event_type** (`object`) *(Required)* Value: `step.start` - **index** (`integer`) *(Required)* - **step** (`Step`) *(Required)* **Possible Types:** (Discriminator: `type`) - **UserInputStep**: Input provided by the user. - **content** (`array (Content)`) **Possible Types:** (Discriminator: `type`) - **TextContent**: A text content block. - **type** (`object`) *(Required)* Value: `text` - **text** (`string`) *(Required)* Required. The text content. - **annotations** (`array (Annotation)`) Citation information for model-generated content. **Possible Types:** (Discriminator: `type`) - **UrlCitation**: A URL citation annotation. - **type** (`object`) *(Required)* Value: `url_citation` - **url** (`string`) The URL. - **title** (`string`) The title of the URL. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **FileCitation**: A file citation annotation. - **type** (`object`) *(Required)* Value: `file_citation` - **document_uri** (`string`) The URI of the file. - **file_name** (`string`) The name of the file. - **source** (`string`) Source attributed for a portion of the text. - **custom_metadata** (`object`) User provided metadata about the retrieved context. - **page_number** (`integer`) Page number of the cited document, if applicable. - **media_id** (`string`) Media ID in-case of image citations, if applicable. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **PlaceCitation**: A place citation annotation. - **type** (`object`) *(Required)* Value: `place_citation` - **place_id** (`string`) The ID of the place, in `places/{place_id}` format. - **name** (`string`) Title of the place. - **url** (`string`) URI reference of the place. - **review_snippets** (`array (ReviewSnippet)`) Snippets of reviews that are used to generate answers about the features of a given place in Google Maps. - **title** (`string`) Title of the review. - **url** (`string`) A link that corresponds to the user review on Google Maps. - **review_id** (`string`) The ID of the review snippet. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **ImageContent**: An image content block. - **type** (`object`) *(Required)* Value: `image` - **data** (`string`) The image content. - **uri** (`string`) The URI of the image. - **mime_type** (`enum (string)`) The mime type of the image. Possible values: - `image/png`: PNG image format - `image/jpeg`: JPEG image format - `image/webp`: WebP image format - `image/heic`: HEIC image format - `image/heif`: HEIF image format - `image/gif`: GIF image format - `image/bmp`: BMP image format - `image/tiff`: TIFF image format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **AudioContent**: An audio content block. - **type** (`object`) *(Required)* Value: `audio` - **data** (`string`) The audio content. - **uri** (`string`) The URI of the audio. - **mime_type** (`enum (string)`) The mime type of the audio. Possible values: - `audio/wav`: WAV audio format - `audio/mp3`: MP3 audio format - `audio/aiff`: AIFF audio format - `audio/aac`: AAC audio format - `audio/ogg`: OGG audio format - `audio/flac`: FLAC audio format - `audio/mpeg`: MPEG audio format - `audio/m4a`: M4A audio format - `audio/l16`: L16 audio format - `audio/opus`: OPUS audio format - `audio/alaw`: ALAW audio format - `audio/mulaw`: MULAW audio format - **channels** (`integer`) The number of audio channels. - **sample_rate** (`integer`) The sample rate of the audio. - **DocumentContent**: A document content block. - **type** (`object`) *(Required)* Value: `document` - **data** (`string`) The document content. - **uri** (`string`) The URI of the document. - **mime_type** (`enum (string)`) The mime type of the document. Possible values: - `application/pdf`: PDF document format - `text/csv`: CSV document format - **VideoContent**: A video content block. - **type** (`object`) *(Required)* Value: `video` - **data** (`string`) The video content. - **uri** (`string`) The URI of the video. - **mime_type** (`enum (string)`) The mime type of the video. Possible values: - `video/mp4`: MP4 video format - `video/mpeg`: MPEG video format - `video/mpg`: MPG video format - `video/mov`: MOV video format - `video/avi`: AVI video format - `video/x-flv`: FLV video format - `video/webm`: WebM video format - `video/wmv`: WMV video format - `video/3gpp`: 3GPP video format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **type** (`object`) *(Required)* Value: `user_input` - **ModelOutputStep**: Output generated by the model. - **type** (`object`) *(Required)* Value: `model_output` - **content** (`array (Content)`) **Possible Types:** (Discriminator: `type`) - **TextContent**: A text content block. - **type** (`object`) *(Required)* Value: `text` - **text** (`string`) *(Required)* Required. The text content. - **annotations** (`array (Annotation)`) Citation information for model-generated content. **Possible Types:** (Discriminator: `type`) - **UrlCitation**: A URL citation annotation. - **type** (`object`) *(Required)* Value: `url_citation` - **url** (`string`) The URL. - **title** (`string`) The title of the URL. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **FileCitation**: A file citation annotation. - **type** (`object`) *(Required)* Value: `file_citation` - **document_uri** (`string`) The URI of the file. - **file_name** (`string`) The name of the file. - **source** (`string`) Source attributed for a portion of the text. - **custom_metadata** (`object`) User provided metadata about the retrieved context. - **page_number** (`integer`) Page number of the cited document, if applicable. - **media_id** (`string`) Media ID in-case of image citations, if applicable. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **PlaceCitation**: A place citation annotation. - **type** (`object`) *(Required)* Value: `place_citation` - **place_id** (`string`) The ID of the place, in `places/{place_id}` format. - **name** (`string`) Title of the place. - **url** (`string`) URI reference of the place. - **review_snippets** (`array (ReviewSnippet)`) Snippets of reviews that are used to generate answers about the features of a given place in Google Maps. - **title** (`string`) Title of the review. - **url** (`string`) A link that corresponds to the user review on Google Maps. - **review_id** (`string`) The ID of the review snippet. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **ImageContent**: An image content block. - **type** (`object`) *(Required)* Value: `image` - **data** (`string`) The image content. - **uri** (`string`) The URI of the image. - **mime_type** (`enum (string)`) The mime type of the image. Possible values: - `image/png`: PNG image format - `image/jpeg`: JPEG image format - `image/webp`: WebP image format - `image/heic`: HEIC image format - `image/heif`: HEIF image format - `image/gif`: GIF image format - `image/bmp`: BMP image format - `image/tiff`: TIFF image format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **AudioContent**: An audio content block. - **type** (`object`) *(Required)* Value: `audio` - **data** (`string`) The audio content. - **uri** (`string`) The URI of the audio. - **mime_type** (`enum (string)`) The mime type of the audio. Possible values: - `audio/wav`: WAV audio format - `audio/mp3`: MP3 audio format - `audio/aiff`: AIFF audio format - `audio/aac`: AAC audio format - `audio/ogg`: OGG audio format - `audio/flac`: FLAC audio format - `audio/mpeg`: MPEG audio format - `audio/m4a`: M4A audio format - `audio/l16`: L16 audio format - `audio/opus`: OPUS audio format - `audio/alaw`: ALAW audio format - `audio/mulaw`: MULAW audio format - **channels** (`integer`) The number of audio channels. - **sample_rate** (`integer`) The sample rate of the audio. - **DocumentContent**: A document content block. - **type** (`object`) *(Required)* Value: `document` - **data** (`string`) The document content. - **uri** (`string`) The URI of the document. - **mime_type** (`enum (string)`) The mime type of the document. Possible values: - `application/pdf`: PDF document format - `text/csv`: CSV document format - **VideoContent**: A video content block. - **type** (`object`) *(Required)* Value: `video` - **data** (`string`) The video content. - **uri** (`string`) The URI of the video. - **mime_type** (`enum (string)`) The mime type of the video. Possible values: - `video/mp4`: MP4 video format - `video/mpeg`: MPEG video format - `video/mpg`: MPG video format - `video/mov`: MOV video format - `video/avi`: AVI video format - `video/x-flv`: FLV video format - `video/webm`: WebM video format - `video/wmv`: WMV video format - `video/3gpp`: 3GPP video format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **error** (`Status`) The error result of the operation in case of failure or cancellation. - **code** (`integer`) The status code, which should be an enum value of google.rpc.Code. - **message** (`string`) A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client. - **details** (`array (object)`) A list of messages that carry the error details. There is a common set of message types for APIs to use. - **ThoughtStep**: A thought step. - **type** (`object`) *(Required)* Value: `thought` - **signature** (`string`) A signature hash for backend validation. - **summary** (`array (ThoughtSummaryContent)`) A summary of the thought. **Possible Types:** (Discriminator: `type`) - **TextContent**: A text content block. - **type** (`object`) *(Required)* Value: `text` - **text** (`string`) *(Required)* Required. The text content. - **annotations** (`array (Annotation)`) Citation information for model-generated content. **Possible Types:** (Discriminator: `type`) - **UrlCitation**: A URL citation annotation. - **type** (`object`) *(Required)* Value: `url_citation` - **url** (`string`) The URL. - **title** (`string`) The title of the URL. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **FileCitation**: A file citation annotation. - **type** (`object`) *(Required)* Value: `file_citation` - **document_uri** (`string`) The URI of the file. - **file_name** (`string`) The name of the file. - **source** (`string`) Source attributed for a portion of the text. - **custom_metadata** (`object`) User provided metadata about the retrieved context. - **page_number** (`integer`) Page number of the cited document, if applicable. - **media_id** (`string`) Media ID in-case of image citations, if applicable. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **PlaceCitation**: A place citation annotation. - **type** (`object`) *(Required)* Value: `place_citation` - **place_id** (`string`) The ID of the place, in `places/{place_id}` format. - **name** (`string`) Title of the place. - **url** (`string`) URI reference of the place. - **review_snippets** (`array (ReviewSnippet)`) Snippets of reviews that are used to generate answers about the features of a given place in Google Maps. - **title** (`string`) Title of the review. - **url** (`string`) A link that corresponds to the user review on Google Maps. - **review_id** (`string`) The ID of the review snippet. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **ImageContent**: An image content block. - **type** (`object`) *(Required)* Value: `image` - **data** (`string`) The image content. - **uri** (`string`) The URI of the image. - **mime_type** (`enum (string)`) The mime type of the image. Possible values: - `image/png`: PNG image format - `image/jpeg`: JPEG image format - `image/webp`: WebP image format - `image/heic`: HEIC image format - `image/heif`: HEIF image format - `image/gif`: GIF image format - `image/bmp`: BMP image format - `image/tiff`: TIFF image format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **FunctionCallStep**: A function tool call step. - **type** (`object`) *(Required)* Value: `function_call` - **name** (`string`) *(Required)* Required. The name of the tool to call. - **arguments** (`object`) *(Required)* Required. The arguments to pass to the function. - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **CodeExecutionCallStep**: Code execution call step. - **type** (`object`) *(Required)* Value: `code_execution_call` - **arguments** (`CodeExecutionCallStepArguments`) *(Required)* Required. The arguments to pass to the code execution. - **language** (`enum (string)`) Programming language of the `code`. Possible values: - `python`: Python >= 3.10, with numpy and simpy available. - **code** (`string`) The code to be executed. - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **signature** (`string`) A signature hash for backend validation. - **UrlContextCallStep**: URL context call step. - **type** (`object`) *(Required)* Value: `url_context_call` - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **signature** (`string`) A signature hash for backend validation. - **arguments** (`UrlContextCallArguments`) *(Required)* The arguments to pass to the URL context. - **urls** (`array (string)`) The URLs to fetch. - **McpServerToolCallStep**: MCPServer tool call step. - **type** (`object`) *(Required)* Value: `mcp_server_tool_call` - **name** (`string`) *(Required)* Required. The name of the tool which was called. - **server_name** (`string`) *(Required)* Required. The name of the used MCP server. - **arguments** (`object`) *(Required)* Required. The JSON object of arguments for the function. - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **GoogleSearchCallStep**: Google Search call step. - **type** (`object`) *(Required)* Value: `google_search_call` - **arguments** (`GoogleSearchCallStepArguments`) *(Required)* Required. The arguments to pass to Google Search. - **queries** (`array (string)`) Web search queries for the following-up web search. - **search_type** (`enum (string)`) The type of search grounding enabled. Possible values: - `web_search`: Setting this field enables web search. Only text results are returned. - `image_search`: Setting this field enables image search. Image bytes are returned. - `enterprise_web_search`: Setting this field enables enterprise web search. - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **signature** (`string`) A signature hash for backend validation. - **FileSearchCallStep**: File Search call step. - **type** (`object`) *(Required)* Value: `file_search_call` - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **signature** (`string`) A signature hash for backend validation. - **GoogleMapsCallStep**: Google Maps call step. - **type** (`object`) *(Required)* Value: `google_maps_call` - **arguments** (`GoogleMapsCallStepArguments`) The arguments to pass to the Google Maps tool. - **queries** (`array (string)`) The queries to be executed. - **id** (`string`) *(Required)* Required. A unique ID for this specific tool call. - **signature** (`string`) A signature hash for backend validation. - **FunctionResultStep**: Result of a function tool call. - **type** (`object`) *(Required)* Value: `function_result` - **name** (`string`) The name of the tool that was called. - **is_error** (`boolean`) Whether the tool call resulted in an error. - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **result** (`array (ImageContent or TextContent) or object or string`) *(Required)* The result of the tool call. - **CodeExecutionResultStep**: Code execution result step. - **type** (`object`) *(Required)* Value: `code_execution_result` - **result** (`string`) *(Required)* Required. The output of the code execution. - **is_error** (`boolean`) Whether the code execution resulted in an error. - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **signature** (`string`) A signature hash for backend validation. - **UrlContextResultStep**: URL context result step. - **type** (`object`) *(Required)* Value: `url_context_result` - **result** (`array (UrlContextResult)`) *(Required)* Required. The results of the URL context. - **url** (`string`) The URL that was fetched. - **status** (`enum (string)`) The status of the URL retrieval. Possible values: - `success`: Url retrieval is successful. - `error`: Url retrieval is failed due to error. - `paywall`: Url retrieval is failed because the content is behind paywall. - `unsafe`: Url retrieval is failed because the content is unsafe. - **is_error** (`boolean`) Whether the URL context resulted in an error. - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **signature** (`string`) A signature hash for backend validation. - **GoogleSearchResultStep**: Google Search result step. - **type** (`object`) *(Required)* Value: `google_search_result` - **result** (`array (GoogleSearchResultItem)`) *(Required)* Required. The results of the Google Search. - **search_suggestions** (`string`) Web content snippet that can be embedded in a web page or an app webview. - **is_error** (`boolean`) Whether the Google Search resulted in an error. - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **signature** (`string`) A signature hash for backend validation. - **McpServerToolResultStep**: MCPServer tool result step. - **type** (`object`) *(Required)* Value: `mcp_server_tool_result` - **name** (`string`) Name of the tool which is called for this specific tool call. - **server_name** (`string`) The name of the used MCP server. - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **result** (`array (ImageContent or TextContent) or object or string`) *(Required)* The output from the MCP server call. Can be simple text or rich content. - **FileSearchResultStep**: File Search result step. - **type** (`object`) *(Required)* Value: `file_search_result` - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **signature** (`string`) A signature hash for backend validation. - **GoogleMapsResultStep**: Google Maps result step. - **type** (`object`) *(Required)* Value: `google_maps_result` - **result** (`array (GoogleMapsResultItem)`) *(Required)* - **places** (`array (GoogleMapsResultPlaces)`) - **place_id** (`string`) - **name** (`string`) - **url** (`string`) - **review_snippets** (`array (ReviewSnippet)`) - **title** (`string`) Title of the review. - **url** (`string`) A link that corresponds to the user review on Google Maps. - **review_id** (`string`) The ID of the review snippet. - **widget_context_token** (`string`) - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **signature** (`string`) A signature hash for backend validation. - **event_id** (`string`) The event_id token to be used to resume the interaction stream, from this event. - **metadata** (`StreamMetadata`) Optional metadata accompanying ANY streamed event. - **total_usage** (`Usage`) - **total_input_tokens** (`integer`) Number of tokens in the prompt (context). - **input_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of input token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_cached_tokens** (`integer`) Number of tokens in the cached part of the prompt (the cached content). - **cached_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of cached token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_output_tokens** (`integer`) Total number of tokens across all the generated responses. - **output_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of output token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_tool_use_tokens** (`integer`) Number of tokens present in tool-use prompt(s). - **tool_use_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of tool-use token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_thought_tokens** (`integer`) Number of tokens of thoughts for thinking models. - **total_tokens** (`integer`) Total token count for the interaction request (prompt + responses + other internal tokens). - **grounding_tool_count** (`array (GroundingToolCount)`) Grounding tool count. - **type** (`enum (string)`) The grounding tool type associated with the count. Possible values: - `google_search`: Grounding with Google Web Search and Image Search, & Web Grounding for Enterprise. - `google_maps`: Grounding with Google Maps. - `retrieval`: Grounding with customer's data, for example, VertexAISearch. - **count** (`integer`) The number of grounding tool counts. **Examples** **Step Start** ```json { "event_type": "step.start", "index": 0, "step": { "type": "model_output" } } ``` - **StepDelta** - - **event_type** (`object`) *(Required)* Value: `step.delta` - **index** (`integer`) *(Required)* - **delta** (`StepDeltaData`) *(Required)* **Possible Types:** (Discriminator: `type`) - **TextDelta**: - **type** (`object`) *(Required)* Value: `text` - **text** (`string`) *(Required)* - **ImageDelta**: - **type** (`object`) *(Required)* Value: `image` - **data** (`string`) - **uri** (`string`) - **mime_type** (`enum (string)`) Possible values: - `image/png`: PNG image format - `image/jpeg`: JPEG image format - `image/webp`: WebP image format - `image/heic`: HEIC image format - `image/heif`: HEIF image format - `image/gif`: GIF image format - `image/bmp`: BMP image format - `image/tiff`: TIFF image format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **AudioDelta**: - **type** (`object`) *(Required)* Value: `audio` - **data** (`string`) - **uri** (`string`) - **mime_type** (`enum (string)`) Possible values: - `audio/wav`: WAV audio format - `audio/mp3`: MP3 audio format - `audio/aiff`: AIFF audio format - `audio/aac`: AAC audio format - `audio/ogg`: OGG audio format - `audio/flac`: FLAC audio format - `audio/mpeg`: MPEG audio format - `audio/m4a`: M4A audio format - `audio/l16`: L16 audio format - `audio/opus`: OPUS audio format - `audio/alaw`: ALAW audio format - `audio/mulaw`: MULAW audio format - **sample_rate** (`integer`) The sample rate of the audio. - **channels** (`integer`) The number of audio channels. - **DocumentDelta**: - **type** (`object`) *(Required)* Value: `document` - **data** (`string`) - **uri** (`string`) - **mime_type** (`enum (string)`) Possible values: - `application/pdf`: PDF document format - `text/csv`: CSV document format - **VideoDelta**: - **type** (`object`) *(Required)* Value: `video` - **data** (`string`) - **uri** (`string`) - **mime_type** (`enum (string)`) Possible values: - `video/mp4`: MP4 video format - `video/mpeg`: MPEG video format - `video/mpg`: MPG video format - `video/mov`: MOV video format - `video/avi`: AVI video format - `video/x-flv`: FLV video format - `video/webm`: WebM video format - `video/wmv`: WMV video format - `video/3gpp`: 3GPP video format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **ThoughtSummaryDelta**: - **type** (`object`) *(Required)* Value: `thought_summary` - **content** (`Content`) A new summary item to be added to the thought. **Possible Types:** (Discriminator: `type`) - **TextContent**: A text content block. - **type** (`object`) *(Required)* Value: `text` - **text** (`string`) *(Required)* Required. The text content. - **annotations** (`array (Annotation)`) Citation information for model-generated content. **Possible Types:** (Discriminator: `type`) - **UrlCitation**: A URL citation annotation. - **type** (`object`) *(Required)* Value: `url_citation` - **url** (`string`) The URL. - **title** (`string`) The title of the URL. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **FileCitation**: A file citation annotation. - **type** (`object`) *(Required)* Value: `file_citation` - **document_uri** (`string`) The URI of the file. - **file_name** (`string`) The name of the file. - **source** (`string`) Source attributed for a portion of the text. - **custom_metadata** (`object`) User provided metadata about the retrieved context. - **page_number** (`integer`) Page number of the cited document, if applicable. - **media_id** (`string`) Media ID in-case of image citations, if applicable. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **PlaceCitation**: A place citation annotation. - **type** (`object`) *(Required)* Value: `place_citation` - **place_id** (`string`) The ID of the place, in `places/{place_id}` format. - **name** (`string`) Title of the place. - **url** (`string`) URI reference of the place. - **review_snippets** (`array (ReviewSnippet)`) Snippets of reviews that are used to generate answers about the features of a given place in Google Maps. - **title** (`string`) Title of the review. - **url** (`string`) A link that corresponds to the user review on Google Maps. - **review_id** (`string`) The ID of the review snippet. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **ImageContent**: An image content block. - **type** (`object`) *(Required)* Value: `image` - **data** (`string`) The image content. - **uri** (`string`) The URI of the image. - **mime_type** (`enum (string)`) The mime type of the image. Possible values: - `image/png`: PNG image format - `image/jpeg`: JPEG image format - `image/webp`: WebP image format - `image/heic`: HEIC image format - `image/heif`: HEIF image format - `image/gif`: GIF image format - `image/bmp`: BMP image format - `image/tiff`: TIFF image format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **AudioContent**: An audio content block. - **type** (`object`) *(Required)* Value: `audio` - **data** (`string`) The audio content. - **uri** (`string`) The URI of the audio. - **mime_type** (`enum (string)`) The mime type of the audio. Possible values: - `audio/wav`: WAV audio format - `audio/mp3`: MP3 audio format - `audio/aiff`: AIFF audio format - `audio/aac`: AAC audio format - `audio/ogg`: OGG audio format - `audio/flac`: FLAC audio format - `audio/mpeg`: MPEG audio format - `audio/m4a`: M4A audio format - `audio/l16`: L16 audio format - `audio/opus`: OPUS audio format - `audio/alaw`: ALAW audio format - `audio/mulaw`: MULAW audio format - **channels** (`integer`) The number of audio channels. - **sample_rate** (`integer`) The sample rate of the audio. - **DocumentContent**: A document content block. - **type** (`object`) *(Required)* Value: `document` - **data** (`string`) The document content. - **uri** (`string`) The URI of the document. - **mime_type** (`enum (string)`) The mime type of the document. Possible values: - `application/pdf`: PDF document format - `text/csv`: CSV document format - **VideoContent**: A video content block. - **type** (`object`) *(Required)* Value: `video` - **data** (`string`) The video content. - **uri** (`string`) The URI of the video. - **mime_type** (`enum (string)`) The mime type of the video. Possible values: - `video/mp4`: MP4 video format - `video/mpeg`: MPEG video format - `video/mpg`: MPG video format - `video/mov`: MOV video format - `video/avi`: AVI video format - `video/x-flv`: FLV video format - `video/webm`: WebM video format - `video/wmv`: WMV video format - `video/3gpp`: 3GPP video format - **resolution** (`MediaResolution`) The resolution of the media. Possible values: - `low`: Low resolution. - `medium`: Medium resolution. - `high`: High resolution. - `ultra_high`: Ultra high resolution. - **ThoughtSignatureDelta**: - **type** (`object`) *(Required)* Value: `thought_signature` - **signature** (`string`) Signature to match the backend source to be part of the generation. - **TextAnnotationDelta**: - **type** (`object`) *(Required)* Value: `text_annotation_delta` - **annotations** (`array (Annotation)`) Citation information for model-generated content. **Possible Types:** (Discriminator: `type`) - **UrlCitation**: A URL citation annotation. - **type** (`object`) *(Required)* Value: `url_citation` - **url** (`string`) The URL. - **title** (`string`) The title of the URL. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **FileCitation**: A file citation annotation. - **type** (`object`) *(Required)* Value: `file_citation` - **document_uri** (`string`) The URI of the file. - **file_name** (`string`) The name of the file. - **source** (`string`) Source attributed for a portion of the text. - **custom_metadata** (`object`) User provided metadata about the retrieved context. - **page_number** (`integer`) Page number of the cited document, if applicable. - **media_id** (`string`) Media ID in-case of image citations, if applicable. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **PlaceCitation**: A place citation annotation. - **type** (`object`) *(Required)* Value: `place_citation` - **place_id** (`string`) The ID of the place, in `places/{place_id}` format. - **name** (`string`) Title of the place. - **url** (`string`) URI reference of the place. - **review_snippets** (`array (ReviewSnippet)`) Snippets of reviews that are used to generate answers about the features of a given place in Google Maps. - **title** (`string`) Title of the review. - **url** (`string`) A link that corresponds to the user review on Google Maps. - **review_id** (`string`) The ID of the review snippet. - **start_index** (`integer`) Start of segment of the response that is attributed to this source. Index indicates the start of the segment, measured in bytes. - **end_index** (`integer`) End of the attributed segment, exclusive. - **ArgumentsDelta**: - **type** (`object`) *(Required)* Value: `arguments_delta` - **arguments** (`string`) - **CodeExecutionCallDelta**: - **type** (`object`) *(Required)* Value: `code_execution_call` - **arguments** (`CodeExecutionCallArguments`) *(Required)* - **language** (`enum (string)`) Programming language of the `code`. Possible values: - `python`: Python >= 3.10, with numpy and simpy available. - **code** (`string`) The code to be executed. - **signature** (`string`) A signature hash for backend validation. - **UrlContextCallDelta**: - **type** (`object`) *(Required)* Value: `url_context_call` - **arguments** (`UrlContextCallArguments`) *(Required)* - **urls** (`array (string)`) The URLs to fetch. - **signature** (`string`) A signature hash for backend validation. - **GoogleSearchCallDelta**: - **type** (`object`) *(Required)* Value: `google_search_call` - **arguments** (`GoogleSearchCallArguments`) *(Required)* - **queries** (`array (string)`) Web search queries for the following-up web search. - **signature** (`string`) A signature hash for backend validation. - **McpServerToolCallDelta**: - **type** (`object`) *(Required)* Value: `mcp_server_tool_call` - **name** (`string`) *(Required)* - **server_name** (`string`) *(Required)* - **arguments** (`object`) *(Required)* - **FileSearchCallDelta**: - **type** (`object`) *(Required)* Value: `file_search_call` - **signature** (`string`) A signature hash for backend validation. - **GoogleMapsCallDelta**: - **type** (`object`) *(Required)* Value: `google_maps_call` - **arguments** (`GoogleMapsCallArguments`) The arguments to pass to the Google Maps tool. - **queries** (`array (string)`) The queries to be executed. - **signature** (`string`) A signature hash for backend validation. - **CodeExecutionResultDelta**: - **type** (`object`) *(Required)* Value: `code_execution_result` - **result** (`string`) *(Required)* - **is_error** (`boolean`) - **signature** (`string`) A signature hash for backend validation. - **UrlContextResultDelta**: - **type** (`object`) *(Required)* Value: `url_context_result` - **result** (`array (UrlContextResult)`) *(Required)* - **url** (`string`) The URL that was fetched. - **status** (`enum (string)`) The status of the URL retrieval. Possible values: - `success`: Url retrieval is successful. - `error`: Url retrieval is failed due to error. - `paywall`: Url retrieval is failed because the content is behind paywall. - `unsafe`: Url retrieval is failed because the content is unsafe. - **is_error** (`boolean`) - **signature** (`string`) A signature hash for backend validation. - **GoogleSearchResultDelta**: - **type** (`object`) *(Required)* Value: `google_search_result` - **result** (`array (GoogleSearchResult)`) *(Required)* - **search_suggestions** (`string`) Web content snippet that can be embedded in a web page or an app webview. - **is_error** (`boolean`) - **signature** (`string`) A signature hash for backend validation. - **McpServerToolResultDelta**: - **type** (`object`) *(Required)* Value: `mcp_server_tool_result` - **name** (`string`) - **server_name** (`string`) - **result** (`array (ImageContent or TextContent) or object or string`) *(Required)* - **FileSearchResultDelta**: - **type** (`object`) *(Required)* Value: `file_search_result` - **result** (`array (FileSearchResult)`) *(Required)* - **signature** (`string`) A signature hash for backend validation. - **GoogleMapsResultDelta**: - **type** (`object`) *(Required)* Value: `google_maps_result` - **result** (`array (GoogleMapsResult)`) The results of the Google Maps. - **places** (`array (Places)`) The places that were found. - **place_id** (`string`) The ID of the place, in `places/{place_id}` format. - **name** (`string`) Title of the place. - **url** (`string`) URI reference of the place. - **review_snippets** (`array (ReviewSnippet)`) Snippets of reviews that are used to generate answers about the features of a given place in Google Maps. - **title** (`string`) Title of the review. - **url** (`string`) A link that corresponds to the user review on Google Maps. - **review_id** (`string`) The ID of the review snippet. - **widget_context_token** (`string`) Resource name of the Google Maps widget context token. - **signature** (`string`) A signature hash for backend validation. - **FunctionResultDelta**: - **type** (`object`) *(Required)* Value: `function_result` - **name** (`string`) - **is_error** (`boolean`) - **call_id** (`string`) *(Required)* Required. ID to match the ID from the function call block. - **result** (`array (ImageContent or TextContent) or object or string`) *(Required)* - **event_id** (`string`) The event_id token to be used to resume the interaction stream, from this event. - **metadata** (`StepDeltaMetadata`) Optional metadata accompanying ANY streamed event. - **total_usage** (`Usage`) Statistics on the interaction request's token usage. - **total_input_tokens** (`integer`) Number of tokens in the prompt (context). - **input_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of input token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_cached_tokens** (`integer`) Number of tokens in the cached part of the prompt (the cached content). - **cached_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of cached token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_output_tokens** (`integer`) Total number of tokens across all the generated responses. - **output_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of output token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_tool_use_tokens** (`integer`) Number of tokens present in tool-use prompt(s). - **tool_use_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of tool-use token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_thought_tokens** (`integer`) Number of tokens of thoughts for thinking models. - **total_tokens** (`integer`) Total token count for the interaction request (prompt + responses + other internal tokens). - **grounding_tool_count** (`array (GroundingToolCount)`) Grounding tool count. - **type** (`enum (string)`) The grounding tool type associated with the count. Possible values: - `google_search`: Grounding with Google Web Search and Image Search, & Web Grounding for Enterprise. - `google_maps`: Grounding with Google Maps. - `retrieval`: Grounding with customer's data, for example, VertexAISearch. - **count** (`integer`) The number of grounding tool counts. **Examples** **Step Delta** ```json { "event_type": "step.delta", "index": 0, "delta": { "type": "text", "text": "Hello" } } ``` - **StepStop** - - **event_type** (`object`) *(Required)* Value: `step.stop` - **index** (`integer`) *(Required)* - **usage** (`Usage`) Cumulative model usage stats from the start of the session. - **total_input_tokens** (`integer`) Number of tokens in the prompt (context). - **input_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of input token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_cached_tokens** (`integer`) Number of tokens in the cached part of the prompt (the cached content). - **cached_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of cached token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_output_tokens** (`integer`) Total number of tokens across all the generated responses. - **output_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of output token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_tool_use_tokens** (`integer`) Number of tokens present in tool-use prompt(s). - **tool_use_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of tool-use token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_thought_tokens** (`integer`) Number of tokens of thoughts for thinking models. - **total_tokens** (`integer`) Total token count for the interaction request (prompt + responses + other internal tokens). - **grounding_tool_count** (`array (GroundingToolCount)`) Grounding tool count. - **type** (`enum (string)`) The grounding tool type associated with the count. Possible values: - `google_search`: Grounding with Google Web Search and Image Search, & Web Grounding for Enterprise. - `google_maps`: Grounding with Google Maps. - `retrieval`: Grounding with customer's data, for example, VertexAISearch. - **count** (`integer`) The number of grounding tool counts. - **step_usage** (`Usage`) Model usage stats for this specific step. - **total_input_tokens** (`integer`) Number of tokens in the prompt (context). - **input_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of input token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_cached_tokens** (`integer`) Number of tokens in the cached part of the prompt (the cached content). - **cached_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of cached token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_output_tokens** (`integer`) Total number of tokens across all the generated responses. - **output_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of output token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_tool_use_tokens** (`integer`) Number of tokens present in tool-use prompt(s). - **tool_use_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of tool-use token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_thought_tokens** (`integer`) Number of tokens of thoughts for thinking models. - **total_tokens** (`integer`) Total token count for the interaction request (prompt + responses + other internal tokens). - **grounding_tool_count** (`array (GroundingToolCount)`) Grounding tool count. - **type** (`enum (string)`) The grounding tool type associated with the count. Possible values: - `google_search`: Grounding with Google Web Search and Image Search, & Web Grounding for Enterprise. - `google_maps`: Grounding with Google Maps. - `retrieval`: Grounding with customer's data, for example, VertexAISearch. - **count** (`integer`) The number of grounding tool counts. - **event_id** (`string`) The event_id token to be used to resume the interaction stream, from this event. - **metadata** (`StreamMetadata`) Optional metadata accompanying ANY streamed event. - **total_usage** (`Usage`) - **total_input_tokens** (`integer`) Number of tokens in the prompt (context). - **input_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of input token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_cached_tokens** (`integer`) Number of tokens in the cached part of the prompt (the cached content). - **cached_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of cached token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_output_tokens** (`integer`) Total number of tokens across all the generated responses. - **output_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of output token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_tool_use_tokens** (`integer`) Number of tokens present in tool-use prompt(s). - **tool_use_tokens_by_modality** (`array (ModalityTokens)`) A breakdown of tool-use token usage by modality. - **modality** (`ResponseModality`) The modality associated with the token count. Possible values: - `text`: Indicates the model should return text. - `image`: Indicates the model should return images. - `audio`: Indicates the model should return audio. - `video`: Indicates the model should return video. - `document`: Indicates the model should return documents. - **tokens** (`integer`) Number of tokens for the modality. - **total_thought_tokens** (`integer`) Number of tokens of thoughts for thinking models. - **total_tokens** (`integer`) Total token count for the interaction request (prompt + responses + other internal tokens). - **grounding_tool_count** (`array (GroundingToolCount)`) Grounding tool count. - **type** (`enum (string)`) The grounding tool type associated with the count. Possible values: - `google_search`: Grounding with Google Web Search and Image Search, & Web Grounding for Enterprise. - `google_maps`: Grounding with Google Maps. - `retrieval`: Grounding with customer's data, for example, VertexAISearch. - **count** (`integer`) The number of grounding tool counts. **Examples** **Step Stop** ```json { "event_type": "step.stop", "index": 0 } ``` **JSON Representation:** ```json { "event_type": {}, "event_id": "string", "metadata": { "total_usage": { "total_input_tokens": 0, "input_tokens_by_modality": [ { "modality": "text", "tokens": 0 } ], "total_cached_tokens": 0, "cached_tokens_by_modality": [ { "modality": "text", "tokens": 0 } ], "total_output_tokens": 0, "output_tokens_by_modality": [ { "modality": "text", "tokens": 0 } ], "total_tool_use_tokens": 0, "tool_use_tokens_by_modality": [ { "modality": "text", "tokens": 0 } ], "total_thought_tokens": 0, "total_tokens": 0, "grounding_tool_count": [ { "type": "google_search", "count": 0 } ] } }, "interaction": { "id": "string", "object": "string", "model": "string", "agent": "string", "status": "in_progress", "created": "string", "updated": "string", "service_tier": "flex", "usage": { "total_input_tokens": 0, "input_tokens_by_modality": [ { "modality": "text", "tokens": 0 } ], "total_cached_tokens": 0, "cached_tokens_by_modality": [ { "modality": "text", "tokens": 0 } ], "total_output_tokens": 0, "output_tokens_by_modality": [ { "modality": "text", "tokens": 0 } ], "total_tool_use_tokens": 0, "tool_use_tokens_by_modality": [ { "modality": "text", "tokens": 0 } ], "total_thought_tokens": 0, "total_tokens": 0, "grounding_tool_count": [ { "type": "google_search", "count": 0 } ] }, "steps": [ { "content": [ "..." ], "type": {} } ] } } ```