Skip to content

Integration · Models

OpenAI API integration with data controls, spend limits and a fallback

We build OpenAI features behind a small gateway you own: one place that picks the model, sets data controls, enforces spend limits, validates output against a schema and falls back when a call fails. Your app never talks to the API directly.

Book a Strategy CallGet an AI Plan

OpenAI

Models

In our build stackA tool we build with. Not a partnership, certification or reseller status.

Access

Authentication and least-privilege scopes

Project-scoped API keys in your OpenAI organisation, held by the gateway and never by a browser

  • project: one per environment
  • key: restricted permissions
  • limits: spend limit + alerts
  • members: least privilege

Each environment gets its own project, key and spend limit, so a test loop cannot spend the production budget. Keys rotate when someone leaves.

Objects

What we read and what we write

  • Responses API request

    Read: NoWrite: Yes

    Prompt, tools and JSON schema; storage set per feature.

  • Response object

    Read: YesWrite: No

    Output, refusal field, status and token usage, logged per call.

  • Embeddings

    Read: YesWrite: Yes

    Vectors for retrieval; this endpoint stores no application state.

  • Files + vector stores

    Read: YesWrite: Yes

    Kept until deleted; used only when a feature needs hosted retrieval.

  • Batch job

    Read: YesWrite: Yes

    Large offline jobs that do not need an instant answer.

  • Rate-limit headers

    Read: YesWrite: No

    x-ratelimit-* and Retry-After drive the gateway’s backoff.

Data flow

System of record and write-back

Your database and documents stay the source of truth. The gateway sends only the context a request needs and logs model, tokens and cost for every call.

  1. Your app calls the gateway, never OpenAI directly.
  2. The gateway checks the budget, removes data the model does not need and picks the model for the task.
  3. The response is validated against the JSON schema; a refusal or incomplete status is handled, not passed on.
  4. After repeated failures the gateway routes to an approved fallback model and records the switch.
OpenAI request path (illustration)
  • Your app
  • Batch jobs
Your model gateway
  • OpenAI API
  • Fallback model
  • Usage + cost log

Worked example

OpenAI in a build

A request trace: budget check, strict-schema call, a rate-limit error, a backoff retry, schema validation, and the fallback rule if retries run out.
One request through the gateway (Sample data)Sample data
  1. PLANRouterTask: structured answer, low risk. Model tier: small.
  2. TOOLBudgetProject spend check: within the monthly limit.
  3. TOOLGatewayCall primary model with strict JSON schema, store off.
  4. ERRORGateway429 rate_limit_error · Retry-After: 2
  5. RETRYGatewayWait Retry-After plus jitter, attempt 2 of 3.
  6. OKValidatorStatus complete, no refusal, schema valid.
  7. NOTEGatewayHad retries run out: fallback model, switch logged.
  8. DONERouterAnswer returned; model, tokens and cost logged.

Sample workflows

What teams build with OpenAI

  • Pattern 01

    Structured extraction from documents

    Emails, forms and PDFs become typed fields against a strict schema. Low-confidence fields go to a person before anything is saved.

  • Pattern 02

    Grounded answers over your content

    Retrieval finds the passages, the model answers from them with citations, and the assistant declines when nothing relevant is found.

  • Pattern 03

    Overnight enrichment in batches

    Classification and tagging of large record sets run through the Batch API, away from live traffic and its rate limits.

Failure handling

What breaks, and what the build does about it

  • Rate limits or overload

    Wait at least Retry-After, add jitter, cap attempts and total wait, and never retry quota or billing errors.

  • The model refuses or stops early

    The refusal field and incomplete status are checked before parsing; the user gets a clear message or a person.

  • A model is retired or changes behaviour

    Prompts and eval sets live in version control, and a new model must pass your eval set before the gateway switches.

Fit · OpenAI

When it is the wrong choice

Saying no early is cheaper than rebuilding later.
  • Data must not leave your network or region and the residency options do not cover you. Use an open-weight model in your environment.
  • The task is arithmetic, lookups or fixed rules. Plain code is cheaper, faster and exact.
  • You want cited answers but nobody keeps the source documents current. Fix the content first.

Alternatives worth pricing

Where it is used

Related use cases, services and guides

Limits

Limits and common errors

Vendor limits change; each line cites the vendor page and the date we checked it.

  • API data is not used to train OpenAI models unless you opt in. Abuse-monitoring logs are kept up to 30 days by default; zero data retention needs OpenAI’s approval.

    Source: Data controls in the OpenAI platform (opens in a new tab) · OpenAI API Docs · accessed

  • The Responses API stores response data for at least 30 days by default or when store is true; Chat Completions stores state only when store is enabled.

    Source: Data controls in the OpenAI platform (opens in a new tab) · OpenAI API Docs · accessed

  • Regional storage and processing covers the US, Europe and the UAE; India, the UK and several others get regional storage only. Non-US regions need modified abuse monitoring or zero data retention.

    Source: Data controls in the OpenAI platform (opens in a new tab) · OpenAI API Docs · accessed

  • Limits apply per organisation and project in RPM, RPD, TPM, TPD and IPM. A 429 may carry Retry-After; failed requests still count, and spend alerts or hard spend limits can be set.

    Source: Rate limits (opens in a new tab) · OpenAI API Docs · accessed

  • Structured Outputs with strict mode keeps responses to your JSON Schema, but a refusal or an incomplete response may not match it, so both are checked.

    Source: Structured Outputs (opens in a new tab) · OpenAI API Docs · accessed

Security

Security notes

  • Keys sit in the gateway’s secrets store, scoped per project, and never reach a browser or mobile app.
  • Personal data the model does not need is removed or masked before the call.
  • Retrieved text and user input are treated as untrusted; write actions need a rule or a person to approve them.

FAQ

Frequently asked questions

Does OpenAI train on the data we send through the API?

Not by default. OpenAI states that API data is not used to train its models unless you opt in. Abuse-monitoring logs are kept for a limited time, zero data retention is open to approved organisations, and we turn storage off where a feature does not need it.

Which OpenAI model should we use?

The smallest one that passes your eval set for the task. We test a few candidates on your real examples, compare quality, latency and cost per request, and record the choice so it can be revisited when new models ship.

How do you keep an AI feature from running up a large OpenAI bill?

Each environment has its own project with a spend limit and alerts, every request caps its output length, and the gateway logs cost per call. A runaway loop hits the limit instead of your card.

Can our OpenAI data stay in India or the EU?

There are data residency options, with storage and processing in some regions and storage only in others, and non-US regions need OpenAI’s approval for reduced retention. We check your region against the current list before the design is final.

What happens to the feature when OpenAI has an outage?

The gateway retries with backoff, then routes to a fallback model you approved, which can be another provider or a model you host. Users see a slower answer or a clear message, never a raw error.

Next step

Have a workflow in mind? Let’s scope it together.

Book a free 30-minute strategy call. Bring one process you want to automate and we will talk through scope, integrations and a sensible first step.

Book a Strategy CallGet an AI Plan

Free 30-minute call · no commitment

Send an RFP or security questionnaire