Introducing the YellowDog Inference Portal: private, multi-cloud, sovereign inference at scale for any open model

  • YellowDog Blog
YellowDog Inference Portal: inference nodes connected across clouds and regions worldwide

Give your team its own private endpoint for open models. The YellowDog Inference Portal serves those models from GPUs in your own cloud accounts and on-premises compute, placing workloads wherever capacity is available.

Executive summary

Open-weight models are now good enough to run real products. Running them yourself is still hard: you have to find GPUs, size each model to them, serve it, scale it and put a secure API in front of it.

The YellowDog Inference Portal is our answer. It provisions GPU instances through the YellowDog platform across AWS, Google Cloud, Azure and Oracle Cloud, serves a wide range of standard and customised models, and puts one OpenAI- and Anthropic-compatible endpoint in front of the lot to optimise and secure access. The management portal handles deployment, autoscaling, API keys and guardrails, so a model goes from its repo to a live, scaled endpoint without anyone writing infrastructure code.

Why organisations want their own inference

Hosted model APIs are the quickest way to start, but they ask you to send prompts and data to someone else’s infrastructure, accept their choice of models and pay per token at their price. For regulated industries, for proprietary or fine-tuned models, and for steady high-volume workloads, that trade stops making sense.

The alternative, self-hosting, has its own problems. GPU capacity is scarce and unevenly spread: the instance type you want is often unavailable in the region you want it, at the moment you want it. Each model needs sizing against the memory on each card, and a mis-sized model fails quietly, after the instance is already billing. And once it is running, someone still has to scale it as demands change, secure it and give callers keys.

Our inference portal is built to remove that operational burden while keeping the models, the data and the spend inside your own accounts.

What it does

The YellowDog Inference Portal is a self-contained inference deployment: a gateway, a management portal and a set of model pools, running in its own Kubernetes namespace, with its own database, its own sign-in and its own YellowDog key. Each customer or team gets its own, at its own address.

Inside it, every model runs as a pool of GPU nodes. Callers never see the pools. They see a single endpoint that speaks the OpenAI API and the Anthropic Messages API, meaning that existing SDKs, agent frameworks and tools such as Claude Code work by changing a base URL and a key.

From the portal, an operator can:

  • Add any authorised compute resource available
  • Deploy a model from the catalogue, or add a new one by pasting its Hugging Face ID
  • Scale each pool automatically on queue depth, KV-cache pressure and time to first token, with operating hours that shut pools down out of hours
  • Issue API keys per caller, scoped to chosen models, with rate limits, budgets and expiry
  • Watch cost per hour, node health and latency for every pool, live

How it works

When a model is deployed, the YellowDog Inference Portal first identifies which instance types have enough GPU memory to serve it. It checks the model’s weights, KV cache and headroom against each GPU’s memory, excluding instances that could boot but never serve the model. It then produces an ordered list of suitable instance types for each cloud, cheapest first.

That list becomes a YellowDog waterfall: a compute requirement whose sources run through every zone of the first region, then the next region, then the next cloud. YellowDog’s provisioning works down the waterfall until it finds capacity, so a pool lands wherever the GPUs are rather than failing on the first sold-out zone.

Each node boots with a small script that installs vLLM, loads the model’s weights and serves them over HTTPS, using a certificate issued by the YellowDog Inference Portal’s own certificate authority. The gateway, LiteLLM in front of vllm-router, discovers nodes as they become ready and routes each request to a pool serving that model, with prefix-aware routing so repeated prompt openings reuse a node’s cache.

The control plane is deliberately simple. The only thing it ever changes on a running pool is the node count. Everything else – placement, booting, joining the gateway – follows from that by polling, so there is no fragile push path to break.

Diagram of the YellowDog Inference Portal: your applications call one gateway endpoint, which routes requests to model pools on AWS, Google Cloud and your own GPUs, while the manager sets each pool's node count and the YellowDog platform finds capacity

How the YellowDog Inference Portal fits together

Callers see one endpoint; the manager sets each pool’s node count and YellowDog finds the capacity.

Key features

Multi-cloud and hybrid by design

A single pool can span AWS, Google Cloud, Azure and Oracle Cloud, and many regions within each. Operators choose how the waterfall is ordered: by cost (the default), by performance, by value, or by latency to the gateway. Clouds and regions are configured on the portal’s Compute Resources page, where the inference portal checks each region, sets up its network and firewall rule, and tests a template before it is offered for deployment.

It is hybrid as well as multi-cloud. Machines you already own, an on-premises GPU server or a workstation, can join a pool with a single script and are probed, certified and routed to like any cloud node, at no hourly cost. The gateway can also front pools running as pods on Kubernetes alongside cloud instances, so a model can move between the two without callers noticing.

Hugging Face integration

To add a model from the Hugging Face Hub, enter its ID on the Models page. The portal reads its configuration, determines its memory footprint and KV-cache cost, and proposes a waterfall for every cloud. It explains its sizing and flags anything it could not establish. Quantisation formats are matched to GPU generations that support them, while multimodal models receive extra headroom for their vision or audio towers. Serving a model depends on compatibility with the deployment’s serving software and available hardware.

The catalogue itself is configuration rather than code, so a deployment ships exactly the models it chooses, imported from a file or added one ID at a time.

Private models and private data

Gated and private Hugging Face repositories work with a token that is set once on the portal, sealed at rest and passed only to the nodes that need it. Weights are downloaded straight into your own cloud accounts and never pass through a third party.

The same applies to prompts and responses. Traffic runs over TLS from your gateway to your nodes, each inference portal is a dedicated customer specific instance with its own isolated state and certificate authority and an optional PII guardrail that can monitor, mask or block personal data per model before it reaches any model at all.

Built for operators

  • Autoscaling on queue depth, KV-cache use and time to first token, with named policies and operating hours
  • Cost visibility per node and per pool, and a cost note before deploying any high-end GPU
  • API keys with per-model scopes, rate limits, budgets and rotation
  • Single sign-on with Microsoft Entra or Google, plus an administrative basic login
  • One routing mode per pool: prefix-aware, round robin or session affinity

What our partners say

“We’re pleased to be partnering with YellowDog. Their Inference Portal integrated easily into our existing infrastructure and adds another cost-effective option for hosting open-source models.”

Arun Steward, Senior Software Engineer, Boltzbit

Looking ahead

The hard part of running open models is no longer the models. It is finding the GPUs, sizing to them and operating what runs on them. The YellowDog Inference Portal turns that into a portal and an endpoint, on capacity YellowDog can source across every major cloud and on hardware you already own.

Key takeaways

  • The YellowDog Inference Portal serves any Hugging Face model, public or private, from GPUs in your own on-premises installation or cloud accounts, behind one OpenAI- and Anthropic-compatible endpoint
  • Waterfalls across regions and clouds place each pool wherever capacity exists, ordered by cost, performance, value or latency
  • Models, weights, prompts and responses stay inside your accounts, with per-model PII guardrails and keys scoped to the models each caller may use

To see the YellowDog Inference Portal running your own models, book a demo or contact our team at contact@yellowdog.ai.