Skip to content

Integration · Models

Gemini API integration for images, documents, audio and video

We use Google’s Gemini models when the input is not just text: photos of a damaged part, scanned PDFs, call recordings or short videos. The build runs on paid API terms or on Vertex AI in your Google Cloud project, so the data terms match your contract.

Book a Strategy CallGet an AI Plan

Google Gemini

Models

In our build stackA tool we build with. Not a partnership, certification or reseller status.

Access

Authentication and least-privilege scopes

A paid-tier Gemini API key held server-side, or a Vertex AI service account in your Google Cloud project

  • vertex: aiplatform.user role
  • service account: one per service
  • storage: one input bucket
  • api key: Gemini API only

For regulated data we prefer Vertex AI: your IAM roles, your project’s region and its data-governance controls apply.

Objects

What we read and what we write

  • generateContent request

    Read: NoWrite: Yes

    Text plus inline image, audio, video or PDF parts.

  • Files API file

    Read: YesWrite: Yes

    Large inputs uploaded once, referenced by URI and deleted after the storage period.

  • Response schema

    Read: NoWrite: Yes

    JSON schema the output must follow; values are still validated in code.

  • Response + usage metadata

    Read: YesWrite: No

    Text or JSON plus token counts per call.

  • Cloud Storage object (Vertex AI)

    Read: YesWrite: No

    Inputs referenced from your bucket instead of uploaded.

Data flow

System of record and write-back

Originals stay in your storage. The service sends the parts a task needs, receives structured fields and writes them to your system with the source file linked.

  1. An image, PDF, recording or video arrives from your app, inbox or bucket.
  2. Small files go inline; large ones go through the Files API or a Cloud Storage reference.
  3. Gemini returns fields that follow a JSON schema, and code validates the values.
  4. Results are written to your system with a link back to the original.
Gemini multimodal path (illustration)
  • Photo upload
  • Scanned PDF
  • Call recording
Your service on Vertex AI or the Gemini API
  • Claims or ticket system
  • Review queue

Worked example

Google Gemini in a build

Tabs for image, PDF, audio and video inputs, each describing what the build reads from that input and what is checked before saving.

A photo of a damaged parcel becomes a short report: damage type, visible labels and whether a person should check it.

  • Label text read from the photo
  • Damage category from your list
  • Low confidence goes to review

Sample workflows

What teams build with Google Gemini

  • Pattern 01

    Photo triage for claims and warranties

    Photos on a claim are described against your damage categories. Anything unclear or high-value goes to an adjuster with the photo and the model’s notes.

  • Pattern 02

    Scanned document intake

    Invoices, delivery notes and forms are read as page images, so tables and stamps survive. Totals are re-added in code before anything posts.

  • Pattern 03

    Call and meeting summaries

    Recordings become summaries with action items in your CRM, each linked to the moment in the recording it came from.

Failure handling

What breaks, and what the build does about it

  • The input is too large to send inline

    Big files go through the Files API or a Cloud Storage reference, and the job records which route it used.

  • The model misreads a field

    Totals, dates and IDs are checked in code against your records; failures go to a review queue with the source page or frame.

  • An uploaded file expired before a rerun

    Files API uploads are temporary, so originals stay in your storage and a rerun uploads again.

Fit · Google Gemini

When it is the wrong choice

Saying no early is cheaper than rebuilding later.
  • Inputs are plain text and the task is short. A text-only model is simpler and often cheaper.
  • You planned to use the free tier with customer data. Free-tier content may be used to improve Google products and seen by reviewers.
  • Images or recordings show people who have not agreed to AI processing. Sort out consent before the pipeline.

Alternatives worth pricing

  • OpenAI

    Text-first features that need strict schemas and batch jobs.

  • Anthropic Claude

    Long text documents with page-level citations.

  • Open-weight models

    Vision or text models that must run inside your network.

Where it is used

Related use cases, services and guides

Limits

Limits and common errors

Vendor limits change; each line cites the vendor page and the date we checked it.

  • Files API uploads are stored for 48 hours, up to 2 GB per file and 20 GB per project. Use it when a request would exceed 100 MB, or 50 MB for PDFs.

    Source: Files API (opens in a new tab) · Google AI for Developers · accessed

  • PDFs of up to 50 MB or 1,000 pages are accepted; each page costs about 258 tokens, and native vision reads images, charts and tables, not only extracted text.

    Source: Document understanding (opens in a new tab) · Google AI for Developers · accessed

  • A prompt can carry up to 9.5 hours of audio, tokenized at 32 tokens per second.

    Source: Audio understanding (opens in a new tab) · Google AI for Developers · accessed

  • Unpaid services may use content to improve Google products, with human review; paid services do not. Apps offered to users in the EEA, Switzerland or the UK must use paid services.

    Source: Gemini API Additional Terms of Service (opens in a new tab) · Google AI for Developers · accessed

  • On Vertex AI, Google does not use your data to train models without permission. Gemini models cache customer data by default, and caching can be turned off per project.

    Source: Generative AI and data governance (opens in a new tab) · Google Cloud Docs · accessed

Security

Security notes

  • Inputs are read from your storage through a service account, never from public links.
  • Images and recordings with personal data are processed only for the stated purpose and deleted on your schedule.
  • Service accounts hold only the Vertex AI and storage roles the pipeline needs.

FAQ

Frequently asked questions

Does Google use our Gemini API data to improve its products?

On the paid Gemini API and on Vertex AI, Google states it does not use your prompts or responses to improve its products. The free tier is different: content may be used for improvement and seen by reviewers, so customer data never goes through it.

Should we use the Gemini API or Vertex AI?

The Gemini API is the quickest start. Vertex AI fits when you need Google Cloud IAM, a chosen region, logging in your project and your existing cloud agreement; many builds move there before regulated data arrives.

Can Gemini read scanned PDFs with tables and handwriting?

It reads each page as an image as well as text, so tables, stamps and handwriting stay in context. We still check totals and IDs in code, because a confident misreading looks just like a correct one.

How do you handle long recordings or videos?

They go through the Files API or a Cloud Storage reference rather than the request body, and the original stays in your storage. Summaries carry timestamps so a reviewer can jump to the moment a point came from.

Can Gemini return data in our exact JSON format?

Yes, with a JSON schema in the request. Only part of JSON Schema is supported and very deep schemas can be rejected, so we keep schemas flat and validate every value in code before saving.

Next step

Have a workflow in mind? Let’s scope it together.

Book a free 30-minute strategy call. Bring one process you want to automate and we will talk through scope, integrations and a sensible first step.

Book a Strategy CallGet an AI Plan

Free 30-minute call · no commitment

Send an RFP or security questionnaire