Running DeepSeek's DeepInfer on dedicated B200/B300 GPUs: what it does, what it doesn't, and what it costs

Most AI teams discover the same thing about a year into production: the model isn’t the hard part. The bill is. Inference runs every hour of every day, on-demand rates move under you, and the engine deciding how many tokens you get per GPU-hour is usually a black box you rent rather than own.

That is why DeepSeek open-sourcing DeepInfer — the inference framework behind its own large-scale service — is worth a careful look. Paired with GPUs you reserve rather than rent by the hour, it gives teams a credible way to own both the software and the cost line of their inference stack. It also comes with real limits, and it’s better to know them upfront.

01 / WHAT DEEPINFER IS

DeepSeek describes DeepInfer as the framework that powers its own production service at scale: a single cluster running it serves 155 trillion tokens a day, according to DeepSeek’s deployment Q&A. The repository went public in October 2026 and passed 12,000 GitHub stars within three weeks.*

Today it supports one model, DeepSeek-V4.1-Flash, on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, with attention and context parallelism, MoE expert parallelism with load balancing, speculative decoding, asynchronous CPU/GPU scheduling, batch-invariant inference with deterministic sampling on NVIDIA hardware, and structured output constrained to a JSON schema, regex or grammar. Underneath, it runs on DeepSeek’s own open kernel libraries — FlashMLA, DeepGEMM, DeepEP — with FP4 for the KV cache and MoE expert weights.

It’s just as important to know what the release leaves out. By DeepSeek’s own description, it “provides offline inference” — there is no HTTP serving API, no KV cache system, no prefill/decode disaggregation and no load balancing across replicas. The README lists those three as required for a complete inference service and explicitly not included. Treat DeepInfer today as a strong engine for batch work and the foundation of a service, not a drop-in replacement for a hosted endpoint.

On performance, read the README’s own prefill and decode charts rather than anyone’s summary of them — they were measured on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain. One figure DeepSeek does state directly: confidence-scheduled verification delivers 4–7% more throughput at the production operating point, compared with always verifying five draft tokens.

02 / WHY DEDICATED, RESERVED BLACKWELL CAPACITY FITS

• 01The hardware matches. SM100 is NVIDIA’s compute capability 10.0 — the B200, per NVIDIA’s own table. The B300 is a separate compute capability (10.3) and isn’t named in DeepInfer’s README; if you’re planning around B300s, validate the build on your target nodes before you commit. Hopper (H100/H200) isn’t supported, and community pull requests to add it were still open as of this writing.
• 02The engine rewards steady load. Per-GPU throughput climbs as batches fill up in DeepInfer’s own benchmarks. An engine that performs best when kept busy pairs naturally with capacity priced for being kept busy — reserved capacity only pays off at high utilization, because you pay for every reserved hour whether you use it or not.
• 03The cost becomes predictable. A fixed rate for a fixed term turns inference from a variable bill into a line item finance can plan around. Cloud providers have long traded discounts for commitment — AWS advertises up to 72% off on-demand EC2 rates for one- to three-year Savings Plans.*
• 04You keep control of your data. Open weights, an open-source engine and dedicated GPUs mean prompts and outputs stay on infrastructure you control. Our five sites in Texas, Ohio, Virginia, Ontario and Alberta each run 100–200 GPUs under HIPAA and SOC 2.
• 05The capacity is actually yours. Dedicated GPUs run 24/7 for production workloads — you aren’t competing for them by the hour. Confirm tenancy, SLA and network scope in writing; our FAQ covers the same ground.

One caution on scale: DeepSeek’s Q&A puts the smallest efficient deployment at 64 GPUs of prefill plus 64 of decode, and its decode benchmark ran inside a single scale-up domain. A standard 8-GPU HGX B200 or B300 board is one NVLink domain of eight GPUs. On 8-GPU nodes, start with eight-way expert parallelism — the example’s own default — and treat the Q&A’s EP8 figures as your reference, not the EP64 curve.

03 / GETTING STARTED, WITHOUT THE RUNBOOK

Standing up DeepInfer today is a build-from-source job, not a package install: Linux, GCC 13 or later, Python 3.12 and CUDA 13, with DeepSeek’s kernel libraries pulled in as git submodules. The default checkpoint, DeepSeek-V4.1-Flash-1017, downloads from Hugging Face on first run, or points at a local copy for locked-down networks. The examples support several parallel layouts out of the box — expert, data and context parallelism — and two settings matter most for production: one enables confidence-scheduled speculative decoding, the other enables batch-invariant, reproducible output. DeepSeek also ships an evaluation harness against AIME 2026, GPQA Diamond and HLE, so you can check a build against DeepSeek’s reference scores before trusting it with real traffic.

None of this is hard. It’s just not a managed service yet — which is exactly where dedicated infrastructure, and a team that runs GPUs for a living, earns its keep.

04 / WHERE THIS OPENS UP NEW GROUND

These are illustrative patterns, not customer stories. A capable release this early creates room to move before the rest of the market catches up. Three stand out.

• 01Nightly batch processing. An insurer extracting fields from claims, contracts and correspondence overnight doesn’t need a serving API — the workload is batch by nature, which is exactly what an offline engine suits. JSON-schema structured output keeps results machine-readable, and the GPUs stay busy on a predictable schedule.
• 02Reproducible evaluation and regression testing. A product team re-running thousands of prompts every time it changes a prompt, retriever or fine-tune gets a real advantage from batch-invariant inference and deterministic sampling: a change in output reflects a change in the system, not a change in batch composition, which makes regressions far easier to trust.
• 03A platform team building its own endpoint. A company that wants V4.1-Flash behind its own API, for data-residency reasons, can start with DeepInfer on one 8-GPU node for batch jobs while building the pieces the release leaves out — serving, KV cache storage, load balancing — then grow toward the prefill/decode layout DeepSeek describes as traffic justifies it.

05 / WHAT IT COSTS

Our own published figures use a target rate of about $5 per GPU-hour for qualifying commitments of at least 36 months, against about $10 on-demand.* The best fit is 24/7 workloads with $50K or more in monthly GPU spend. At that rate, 32 GPUs running around the clock cost about $1.4M a year less than the same GPUs rented on-demand.

All of these are illustrative figures, not quotes. Pricing depends on GPU type, node configuration, quantity, region and term, and the ~$5 target applies only to qualifying deployments. Commit only to the baseline you’re sure of, and keep cloud capacity for spikes.

DeepInfer gives you DeepSeek’s production engine for DeepSeek-V4.1-Flash: open, well documented, and tuned for Blackwell-class SM100 GPUs. It’s offline-first, so treat it as a strong engine for steady batch work and the core of a service you’ll build out — not a finished product. If that steady work is already costing you $50K or more a month, reserved B200 or B300 capacity is worth modelling against it. Request a quote for your GPU count, region and timeline, or reach the team at contactus@icecastle.si.