Integration · Models
Self-hosted open-weight models, inside your own perimeter
When data cannot leave your network or cloud account, we run open-weight models there: weights pulled from Hugging Face, licence checked, served behind an internal endpoint and tested against your eval set before anyone relies on them.
Open-weight models
Models
Access
Authentication and least-privilege scopes
A read-scoped Hugging Face token used to pull weights, then internal service credentials for the inference endpoint
- hub token: read only
- endpoint: internal network only
- gpu hosts: no inbound internet
- logs: your monitoring stack
Once the weights are pulled and verified, the serving hosts need no internet access. Updates are a reviewed pull, never an automatic one.
Objects
What we read and what we write
Model repository
Read: YesWrite: No
Weights and config pulled in safetensors format; licence and model card recorded.
Gated access request
Read: YesWrite: Yes
The model owner’s terms accepted under your organisation’s account before download.
Inference endpoint
Read: YesWrite: Yes
An internal HTTP API with the request shape of hosted APIs, so app code can switch providers.
Embedding model
Read: YesWrite: Yes
Runs alongside for retrieval, so documents are never sent outside.
Prompt + response logs
Read: YesWrite: Yes
Kept in your logging stack under your retention rules.
Eval set
Read: YesWrite: Yes
Your real examples with expected answers, rerun on every model change.
Data flow
System of record and write-back
Documents, prompts, outputs and logs stay inside your perimeter. The only outside contact is the reviewed download of model weights.
- Pick a model and record its licence from the model card.
- Pull the weights once, verify them and keep them in your artefact store.
- Serve them on your GPUs or a private cloud instance behind an internal endpoint.
- Apps and retrieval call that endpoint; logs go to your own monitoring.
| Concern | Hosted API | Self-hosted weights |
|---|---|---|
| Where data goes | To the provider, under its terms | Stays in your network |
| Who operates it | The provider | You, or us with ongoing support |
| Cost shape | Per token | Hardware plus operations |
| Upgrades | The provider’s schedule | Your reviewed schedule |
Worked example
Open-weight models in a build
- Your documents
- Your apps
- Staff questions
- Your database
- Audit log
- Answers with sources
Sample workflows
What teams build with Open-weight models
Pattern 01
Private knowledge assistant
Staff ask questions over internal documents. Retrieval and generation both run inside your network, and every answer cites its source file.
Pattern 02
Sensitive document classification
Contracts, patient letters or HR files are tagged and routed without leaving your environment; uncertain items go to a person.
Pattern 03
Drafting next to the data
Report and reply drafts are generated beside the records they draw on, then reviewed and sent by a person.
Failure handling
What breaks, and what the build does about it
Quality drops after a model change
Every change reruns your eval set, and the previous version stays deployable until the new one passes.
GPU memory runs out under load
Requests queue with a timeout and a clear message; capacity is sized from a peak load test, not an average.
A new release changes its licence
Upgrades start by reading the new model card and licence, and nothing is pulled automatically.
Fit · Open-weight models
When it is the wrong choice
- Your data may go to a hosted provider under its data terms. Hosted APIs are cheaper to start and need no GPUs.
- You need the strongest available model for hard, open-ended reasoning. Self-hosted models can trail the newest hosted ones.
- Nobody can run GPU servers and ongoing support is not an option. An unmaintained model server ages quickly.
Alternatives worth pricing
Hosted models with storage off, spend limits and regional options.
Claude through your cloud provider
Hosted models processed under your existing cloud agreement.
Multimodal models inside your Google Cloud project.
Where it is used
Related use cases, services and guides
Limits
Limits and common errors
Gated models require you to share your username and email and accept the author’s conditions; downloads then need a user access token, and authors can revoke access at any time.
Source: Gated models (opens in a new tab) · Hugging Face Hub Docs · accessed
Safetensors stores tensors safely, unlike pickle-based files that can run code when loaded, and stays fast through zero-copy loading.
Source: Safetensors (opens in a new tab) · Hugging Face Docs · accessed
Most of Mistral’s open models use Apache 2.0; some use a modified MIT licence that requires a commercial licence above a revenue threshold. Each model card states its terms.
Source: Under which license are Mistral’s open models available? (opens in a new tab) · Mistral AI Help Center · accessed
Security
Security notes
- Serving hosts accept no traffic from the internet and make no outbound calls once weights are installed.
- Models load only from verified safetensors copies in your artefact store.
- Prompts and outputs are logged inside your network under your retention rules, with role-based access to the logs.
FAQ
Frequently asked questions
Which open-weight model should we self-host?
The smallest one that passes your eval set on your hardware, under a licence that fits your use. We shortlist a few from different families, test them on your real questions and choose on quality, speed and memory.
What hardware do we need to run a model in house?
It depends on the model size, the precision it runs at and how many people use it at once. We size it from a load test on your own questions; many internal assistants start on one GPU server or a private cloud instance.
Are open-weight models free to use commercially?
Not always. Many Mistral models use Apache 2.0, while other licences add attribution, acceptable-use rules or thresholds for large companies. We record the licence from the model card before building and check it again on every upgrade.
Can a self-hosted model match a hosted API?
For focused tasks such as classification, extraction and answering from your documents, often yes, once it is tested on your data. For the hardest open-ended reasoning the newest hosted models tend to lead, so we test before promising either way.
Does self-hosting mean nothing ever leaves our network?
That is the design: documents, prompts, outputs and logs stay inside. The one exception is the reviewed download of weights and updates, which happens on your schedule and can pass through a proxy you control.
Next step
Have a workflow in mind? Let’s scope it together.
Book a free 30-minute strategy call. Bring one process you want to automate and we will talk through scope, integrations and a sensible first step.
Free 30-minute call · no commitment