Video analysis with Gemma

View on ai.google.dev Run in Google Colab Run in Kaggle Open in Vertex AI View source on GitHub

Video data is a rich source of information that can help you accomplish tasks and understand the world around you. Using Gemma with video data can help you understand spacial relationships, interpret human interactions, and assist with situational awareness. This tutorial shows you how to get started processing video data with Gemma using Hugging Face Transformers. The Transformers Python library provides a API for accessing pre-trained generative AI models, including Gemma. For more information, see the Transformers documentation.

Setup

Before starting this tutorial, complete the following steps:

  • Get access to Gemma by logging into Hugging Face and selecting Acknowledge license for a Gemma model.
  • Select a Colab runtime with sufficient resources to run the Gemma model size you want to run. Learn more.
  • Generate a Hugging Face Access Token and add it to your Colab environment.

Configure Access Token

Add your access token to Colab to enable downloading of Gemma models from the Hugging Face web site. Use the Colab Secrets feature to securely save your token without adding it to your working code.

To add your Hugging Face Access Token as a Secret:

  1. Open the secrets tab by selecting the key icon on left side of the interface, or select Tools > Command pallete, type secrets, and press Enter.
  2. Select Add new secret to add a new secret entry.
  3. In the Name field, enter HF_TOKEN.
  4. In the Value field, enter the text of your Hugging Face Access Token.
  5. In the Notebook access field, select the switch to enable access.

Once you have entered your Access Token as HF_TOKEN and value, you can access and set it within your Colab notebook environment using the following code:

from google.colab import userdata
from huggingface_hub import login

# Login into Hugging Face Hub
hf_token = userdata.get('HF_TOKEN') # If you are running inside a Google Colab
login(hf_token)

Install Python packages

Install the Hugging Face libraries required for running the Gemma model and making requests.

# Install Pytorch & other libraries
%pip install "torch>=2.4.0"

# Install a transformers version that supports Gemma 3n (>= 4.53.0)
%pip install "transformers>=4.53.0"

Prepare video data prompt

Gemma models interpret video data by breaking it down into it's component parts, specifically as images of video frames and audio clips extracted from the video file. The following code example shows how to extract frames and audio from a video file.

wget http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4 -O /content/video.mp4
mkdir /content/frames
# extract frames from video at 1 frame per second (video is 15 seconds)
ffmpeg -i /content/video.mp4 -vf fps=1 /content/frames/%04d.jpg
# extract audio from video as a single, 15-second clip
ffmpeg -i /content/video.mp4 -vn -c:a copy /content/audio.aac

Once you have extracted frames and audio from your video file into separate image files and audio files, you can assemble those components into a prompt. The following code example shows how to add the extracted video data to a prompt structure, in preparation for prompting the model.

import os

content = []

for file in os.listdir("/content/frames"):
  content.append({"type": "image", "url": f"/content/frames/{file}"})

content.append({"type": "audio", "audio": "/content/audio.aac"})
content.append({"type": "text", "text": "What is shown in this video?"})

messages = [
    {
        "role": "user",
        "content": content
    },
]

For longer videos, you may need to reduce the number of frames per second captured to fit within Gemma model's context window. For audio clips, you should limit the total length of any audio clip to 30 seconds or less for best results.

Configure model

When loading a Gemma model for use with audio data, configure the Transformer instance specifically for use with image and audio data. In particular, you must define a processor and model object using the AutoProcessor and AutoModelForImageTextToText classes, as shown in the following code example:

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

GEMMA_MODEL_ID = "google/gemma-3n-E4B-it"

processor = AutoProcessor.from_pretrained(GEMMA_MODEL_ID, device_map="auto")
model = AutoModelForImageTextToText.from_pretrained(
            GEMMA_MODEL_ID, torch_dtype="auto", device_map="auto")

Run video data request

Once you have created a prompt with video data and configured the Gemma model processor and model objects, you can run the prompt to generate output. The following example code shows a request using a chat template, output generation, decoding of the response:

input_ids = processor.apply_chat_template(
        messages,
        add_generation_prompt=True,
        tokenize=True, return_dict=True,
        return_tensors="pt",
)
input_ids = input_ids.to(model.device, dtype=model.dtype)

# Generate output from the model
outputs = model.generate(**input_ids, max_new_tokens=128)

# decode and print the output as text
text = processor.batch_decode(
    outputs,
    skip_special_tokens=False,
    clean_up_tokenization_spaces=False
)
print(text[0])

Next steps

Build and explore more with Gemma models: