Hybrid AI on the IBM AI Appliance: Open Models On Premises, Claude When You Need It

The IBM AI Appliance delivers exactly that. Built on IBM Fusion HCI with NVIDIA H200 NVL GPUs, the IBM Storage Scale System 6000 and Red Hat AI Enterprise, it runs open models on your own floor and can route selected requests to Anthropic Claude through a single governed gateway.

The appliance runs fully on its own. The frontier connection is an option you switch on, only for the applications, users and data you approve.

Most work runs on the appliance. Claude handles the tasks you designate, with only the context you approve.

Red Hat OpenShift AI Models-as-a-Service presents every model, on premises or in the cloud, behind one interface.

• Same access rules for every model. Teams sign in with corporate credentials, and each application reaches only the models it is cleared for.
• Restricted data stays on your floor. Applications that handle restricted data are permitted the on-premises model only.
• One controlled path out. Only the gateway can reach the Anthropic service.
• Every token metered. Usage is reported by user, team and model, for on-premises and Claude requests alike.

Routing to external models such as Claude is available now as a Technology Preview in Red Hat OpenShift AI 3.5: ready to evaluate today, with production use following general availability.

Most open models in use today are small enough for one or two GPUs. In Hugging Face's Summer 2026 report, only 3% of this year's downloads of models that declare their size went to models above 70 billion parameters.

The smallest NVIDIA H200 configuration in Red Hat AI's validated model matrix, at the precision shown, or at the model's published precision. Qwen3.8-27B is not yet validated by Red Hat; it runs on one H200 in a published SGLang benchmark. The appliance's H200 NVL GPUs carry the same 141 GB of memory. Li9 sizes each deployment to its context length and number of users.

Each Entry Level GPU server holds four H200 NVL GPUs, so even the largest models in the table run within one server, and the appliance can serve several models side by side.

Cloud AI is priced per token: the text sent to a model and the text it returns. On the appliance, tokens are produced at a fixed cost. We estimated what an Entry Level IBM AI Appliance, with 8 NVIDIA H200 NVL GPUs, produces running non-stop for 3 years, then priced exactly those tokens in the cloud.

We used two leading open models of Q4 2026, chosen for their capabilities, their use in pilots and production, and US enterprise requirements: Gemma 4 from Google, validated by Red Hat on NVIDIA H200, and Qwen3.8 from Alibaba, the most downloaded open model family on Hugging Face. Both are Apache 2.0 licensed.

Swipe the table sideways to see the Claude prices.

Estimated with the same method for both models: each model's published SGLang benchmark on one NVIDIA H200 at full load, scaled to 8 GPUs and reduced to 84% for the H200 NVL's lower compute rating. Each row prices exactly the tokens in that row; the benchmarks used different request sizes, so token totals are not compared between rows. Same model in the cloud: Gemma 4 on Google Vertex AI, Qwen3.8 on Alibaba Cloud. Published rates as of October 27, 2026. Your own figures are confirmed by measurement during implementation.

The appliance delivers this capacity at a fixed cost, with data and models kept on your floor and no dependence on an outside service. Per-token rates in the cloud vary widely: Claude Haiku is priced for high-volume tasks, while Sonnet and Opus, the models suited to the hardest work, carry the highest rates. Running routine work on the appliance and routing Claude only the tasks that need it keeps frontier spending to the work that requires it. Open and frontier models differ in capability, so the table compares token cost, not answer quality.

Every published IBM AI Appliance configuration, from Entry Level to Maximum, includes the IBM Storage Scale System 6000, and it changes what long conversations cost the GPU. As a model reads a request, it builds working state, known as the KV cache. When GPU memory fills, that state is normally discarded and rebuilt from scratch. On the appliance, it can be saved to the array and read straight back.

In IBM's published test on IBM Storage Scale, a 130,000-token request began answering in under 0.6 seconds instead of 32 seconds. Li9 confirms the supported serving configuration with IBM and Red Hat for each deployment. On the Claude side, Anthropic's prompt caching serves the same purpose for repeated context.

Hybrid AI works when one platform controls the data, the models and the spending. The IBM AI Appliance brings that control to a single rack:

• Built and tested before it ships. IBM integrates and tests every appliance in the factory, then repeats the tests on site.
• One call for the whole stack. IBM supports every hardware and software component, whether IBM, Red Hat or NVIDIA.
• Grows in place. Start at 8 GPUs and grow to 32 in the same rack, with no redesign.
• Delivered by Li9. Our engineers design, implement and support the appliance alongside IBM.

• Red Hat OpenShift AI 3.5: Technology Preview features
• Red Hat AI: Validated model support matrix
• SGLang: Gemma 4 speed benchmark
• SGLang: Qwen3.8-27B FP8 on H200
• Gemma 4 26B A4B cloud pricing
• Qwen3.8-27B cloud pricing
• Anthropic: Claude pricing
• Most downloaded open models, September 2026
• IBM Redbooks: Context Without Limits
• Hugging Face: State of Open Models, Summer 2026