Baking a 27B Coder Model Down to INT4 on One Arc Pro B65

Latest release: v1.1. Same weights as v1.0, with a second way to serve them: our INT4 DFlash2 drafter at k=7 decodes at 89.3 tok/s on the B65, against 75.8 with the built-in MTP head, at 128k context instead of 262k. All seven tasks are within the bar. Download it from Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked and pin --revision v1.1 for this exact build. Every version gets a log entry at the bottom of this page.

How this started

Wrapzii, whose baked Swift-1.5 INT4 build we have benchmarked before, asked us to run Jackrong/Qwopus3.8-27B-Coder-Flash through our bake. The goal was to make it a fast local model on a single Intel Arc Pro B65 32 GB, served from the machine that runs our coding harness.

The model only exists in BF16: 18 safetensors shards, 55.6 GB, a Qwen3.8 hybrid with 64 layers plus one MTP (multi-token prediction) layer and a vision tower. That doesn’t fit in 32 GB, and nobody had published a GPTQ or AWQ build of the Coder variant. Every model we had baked before started from someone else’s INT4 body. This time we had to quantize the body ourselves, on a host with 30 GB of RAM, before the bake could even start.

What “the bake” is

A typical GPTQ INT4 checkpoint quantizes the transformer body and leaves two large pieces in 16-bit: the output head (lm_head, 248k × 5120) and the MTP draft head that speculative decoding uses. On a card limited by memory bandwidth, those two are a large share of the bytes read for every token. Our bake quantizes both to INT4 with GPTQ, using Hessians captured from the model’s own activations inside the serving engine. It then only keeps a layer if it beats plain round-to-nearest (RTN) quantization on held-out rows. The result is one checkpoint that stock vLLM serves with no runtime patches to the weights.

Building it on one card

The body was quantized with GPTQModel 7.3.2 on the B65 itself, inside an XPU container built on the same vLLM image we serve with. Settings: 4-bit, symmetric, group size 128, act-order off, damping 0.05, 512 calibration samples of 2048 tokens (C4, code and chat).

The first problem was host memory, not the GPU. Short 2-, 4- and 8-layer smoke runs showed resident memory growing by about 1 GB per layer. That would have run out of RAM halfway through a 64-layer model. It wasn’t a leak in the quantizer: glibc’s per-thread malloc arenas were fragmenting. Setting MALLOC_ARENA_MAX=2, calling malloc_trim(0) between layers and finalizing tensors synchronously cut the 8-layer peak from 23.7 GB to 17.5 GB and made the per-layer floor flat. The full body then quantized in 3 h 12 min (11,542 s), peaking at about 27.8 GB on the first layer and staying near 17 GB after that.

The bake itself followed. Hessians for the head and MTP layers took 46 minutes. The gate passed 9 of 9: every baked linear beat RTN on the same held-out rows. The INT4 lm_head’s output error was 4.66%, against 8.16% for RTN, and the MTP linears were about 40% better than RTN. The assembled checkpoint is 16 GB.

First speed check: a quick BetterBench run, MTP speculative decoding with k=6 at 262k context, gave 78.9 tok/s weighted decode and about 1,250 tok/s prefill (page). That was fast enough to be worth checking whether the model was still any good.

Setting the bar before measuring

We wrote the quality thresholds down and hashed the file before any quantized model was evaluated. We recorded each later change to the protocol as a dated addendum with its own hash. There were four, all made before any INT4 results existed: the LiveCodeBench window, the GSM8K answer extraction for chat models, the BFCL data version, and an exploratory BF16 thinking-on arm that is not gated. The bars:

The seven tasks are HumanEval+, MBPP+, LiveCodeBench v6, GSM8K, an MMLU-Pro subset, IFEval and a 400-item BFCL tool-calling subset. Every task compares the same questions item by item, so we report each one as a paired delta with a 95% bootstrap interval, not as two separate scores.

Renting the yardstick

A 55 GB BF16 model won’t run on the B65, so we rented the reference. Over three Vast.ai rentals of an RTX PRO 6000 96 GB, running the same vLLM 0.30, we captured the BF16 model’s top-64 next-token log-probabilities on a fixed set of 512 × 2048 tokens, ran all seven tasks with thinking off, ran the needle-in-a-haystack tests at 32k and 128k, and then ran the seven tasks again with thinking on. The three rentals cost roughly $35 in total, by our own estimate from the hourly rates.

Two of those rentals kept billing for hours after their work was done. The rental tool rightly won’t destroy a box without a human answering its prompt, and nobody was there to answer. Rented boxes now get a detached teardown watcher that ends them the moment their results are pulled back.

The thinking-on numbers aren’t gated, but they show how much of this model’s ability sits behind its reasoning: LiveCodeBench rises from 41.1% to 57.1% and MMLU-Pro from 71.8% to 79.2% (study page with both, BF16 baselines).

Three arms, one change at a time

To show which step costs what, the INT4 build was measured as a chain. Each arm adds one change:

• A: the INT4 body with the head and MTP left in full precision.
• B: A plus the baked INT4 head and MTP. This is the arm the thresholds gate.
• C: B served with MTP speculative decoding at k=6, for the T7 parity check.

Does it predict the same tokens?

Perplexity and KL divergence pass with room to spare. Top-1 agreement misses by a quarter of a point on B but not on A, so the shortfall comes from quantizing the output head, not the body.

Does it still get answers right?

Six of the seven tasks are within the bar or within chance. HumanEval+ and LiveCodeBench drop more than 2 points, but their intervals include zero, so by the rule we registered they don’t count. BFCL doesn’t pass: both INT4 arms lose 8–10 points on tool calling, far outside chance.

The tool-calling bug

Almost all of the BFCL loss was in two categories: parallel and parallel_multiple, where the right answer is several tool calls in one turn. On 24 of the 25 parallel items arm B lost, it made exactly one call where BF16 made two to four. Then it stopped. Its replies were less than half as long (median 54 tokens vs 134). The parser wasn’t dropping anything: the model was ending its turn.

The token right after the first </tool_call> showed why. BF16 strongly prefers a newline there, the start of the next call, with <|im_end|> (end of turn) about 2–3 nats behind. The INT4 model has them the other way round. That is a 2.5-nat shift in a single decision, not a near-tie flipped by rounding.

We ruled things out one at a time:

• The tool and reasoning parsers are the same on both sides.
• The tokenizer files the quantizer had re-saved gave the same probabilities as the original files. They will still be restored before upload.
• The FP8 KV cache: the same margins with the default cache.
• FP16 vs BF16 serving: the same margins.
• The baked head: arm A, with the full-precision head, fails worse.

That left the body, and its calibration set: 40% C4 web text, 40% code, 20% chat, and not one tool call or tools system prompt. GPTQ shapes its rounding with statistics of the activations it sees during calibration. The format of a multi-call reply was never among them.

The fix was to quantize the body again with 20% of the calibration replaced by Hermes function-calling conversations, rendered through the model’s own chat template with its tools, half of them parallel calls. Then we rebaked the head and MTP on the new body.

B2 recovers almost all of it and passes. Its next-token numbers barely moved: KL divergence 0.041, perplexity +1.7% on both corpora, top-1 92.84%. It still misses the 93% top-1 bar by a fraction of a point, as B did.

This recalibration is a post-hoc change: we made it after seeing the BFCL result, not as part of the plan we registered. Every page that shows B2 says so. B2 is the checkpoint we published, so we reran all seven tasks on it as shipped (next section).

All seven tasks on the shipped build

The last open question was whether B2 holds up on the other six tasks, served the way people will run it: MTP speculative decoding at k=6, on for every task and for the next-token logprobs. It does. Every task is inside the −2 point bar or has a confidence interval that includes zero.

Over the whole suite the MTP head kept 46.8% of what it drafted: 80% at the first position, falling to 62%, 48%, 37%, 30% and 23%. That is about 3.8 tokens per verify step under greedy decoding.

Where it stands on the bars

T7 needs explaining. Six of the 32 prompts produce a different greedy output with speculative decoding on. At every one of those divergence points, the top two candidates were tied: three exactly, three within a single FP16 rounding step (0.0156). Batched verification breaks a tie one way and plain decoding breaks it the other. Neither is wrong, but we wrote “identical”, so it’s reported as a fail with the evidence. The needle tests were 100% at 32k and 128k for both BF16 and v1.0.

T6 fails, clearly. Jackrong’s own Q4_K_M GGUF stays closer to BF16 than our build: mean KL divergence 0.025 against our 0.041, top-1 agreement 94.9% against 92.8%, perplexity +0.7% against +1.7%. The two files are about the same size (16.8 and 17.1 GB). Q4_K_M spends its bits on the text layers, with 67 of its tensors at 6 bits; ours spends about 1 GB on a full-precision vision tower and embeddings, and quantizes the body to a uniform 4 bits. To compare them we ran the GGUF in llama.cpp on the same card, dumped the same top-64 logprobs on the same tokens, and scored both with the same code. That isn’t the method we registered (it needed a rental), so the change is dated in the thresholds file. The Q8_0 GGUF, measured the same way, lands at 0.002, so the engine difference is small and the gap is the quantization. What our build buys is the serving stack: vLLM on the B65 with speculative decoding. A better body is the way to close T6, and it’s the next quality study.

The full study, with every chart and caveat, is on its QualityBench page.

Why it’s slower than Swift

With B2 in hand, a quick BetterBench run measured 75.8 tok/s weighted decode (page). On the same card and image, the Swift-1.5 bake with the same MTP drafter reaches 80.5, and with a DFlash2 drafter 96.7.

Speculative decoding speed comes down to two numbers: how long one verify step takes, and how many drafted tokens each step keeps. The step time is the same for both models, 42 ms, so the INT4 body and its kernels aren’t the difference. The difference is acceptance: Qwopus keeps about 3.0 tokens per step under BetterBench’s sampling, Swift 3.35. Under greedy decoding the MTP head’s guesses land 81% of the time at the first position, then 61%, 45%, 33%, 23% and 17% down the draft.

We checked the MTP head’s weights against the base Qwen3.8-27B. All 15 tensors are byte-for-byte identical. The fine-tune changed the model but never trained its draft head, so the drafter is still guessing what the base model would say, not what Qwopus says. Swift’s MTP head turned out to be stock as well.

Chasing 115 tok/s

Our target is 115 tok/s weighted decode on a quick BetterBench run. Decode speed is tokens kept per step divided by step time. The step is already close to the bandwidth floor: about 27 ms for the target model against roughly 24 ms to read its bytes. So the gain has to come from the drafter. At the step time of the best drafter we have, reaching 115 means keeping about 5 tokens per step, which needs a per-position acceptance of about 0.88 at a draft length of 7.

None of this work changes the target model’s weights. With greedy decoding, speculative decoding doesn’t change the output, so the quality study above stays valid. Each new drafter only needs the parity check.

A different drafter, and how long its drafts should be

First we swapped the MTP head for the stock DFlash2 drafter that gave Swift its 96.7. It drafts a block of tokens in one pass instead of one at a time. It costs context: with the drafter loaded, the card only fits 128k, not 262k. Then we swept the draft length for both drafters with a short probe that records acceptance at every position.

k=7 is the best setting, and the reason it peaks there is the step time, not the drafter. From k=8 on, the verify step carries nine or more query tokens and gets about 5 ms slower. That cancels the extra tokens a longer draft keeps. A quick BetterBench run of DFlash2 at k=7 measured 89.3 tok/s (page), up from 75.8. By category it keeps 5.2 tokens per step on JSON and 4.7 on summaries, but only 2.8 on reasoning and 2.5 on prose. The losses are in thinking text and free prose.

A new drafter needs the parity check. With greedy decoding, 28 of the 32 parity prompts came out identical to plain decoding, and all four differences are at a tie, the same pattern as the MTP head. We released this configuration as v1.1.

The same sweep answered a question about the bake. With a full-precision head and MTP, the drafter keeps about the same number of tokens per step under sampling as with the baked INT4 head (3.21 vs 3.25). The baked build is still 36% faster there, because it reads far fewer bytes per step. Quantizing the head costs no acceptance.

Training a drafter on Qwopus’s own answers

The obvious next step was self-distillation: teach the drafter what Qwopus says, not what the base model says. We wrote 10,000 prompts in BetterBench’s category mix, drawn from public datasets, and dropped any prompt that shared an 8-word run with BetterBench’s own prompts. The BF16 model answered each one with thinking on, on a rented H100: 14M tokens in 104 minutes. Then we fine-tuned z-lab’s DFlash2 drafter on those answers with SpecForge, on two rented H200s: two epochs, 58 minutes of training. Data and training together cost about $24 in rentals.

It didn’t work. Every checkpoint is within the noise of the stock drafter and far from the 5.0 gate. Training accuracy on answers the drafter hadn’t seen yet stayed flat for the whole first epoch, and rose only in the second, on answers it had already seen. So the stock drafter wasn’t badly matched to Qwopus in the first place. The tokens it misses are in text that is hard to guess for any drafter of its size. Much more data could still move it (z-lab trained on about a million conversations, not ten thousand), but this run gives no reason to expect a large gain. The full numbers are on the drafter study page.

The verify step

That left the engine. A profile of the k=7 decode step showed about 7 ms per step where the GPU sat waiting for the host. With speculation off it was 2 ms. The cause was in vLLM’s metadata builder for the linear-attention (GDN) layers: on every speculative step it indexed two GPU tensors with a boolean mask that lives on the CPU. The size of that result depends on the data, so the host has to wait for the GPU before it can go on, every step. That cancels the overlap that async scheduling is supposed to give.

On that path the padded sequences always sit at the back, so the mask is just “the first n rows” and a slice gives the same tensors without the wait. The fix is a 50-line boot patch, now in the B65 repo. On a single prompt the k=7 step went from 47.9 to 44.4 ms. Two quick BetterBench runs of the patched profile measured 92.5 and 96.0 tok/s, up from 89.3 (page for the second run). Quick runs vary by a few tok/s, so we show both. Speculative parity improved too: 30 of 32 greedy outputs identical, both differences at exact ties.

The other half of the verify-step problem is still there. The INT4 matrix multiply gets slower when the verify step carries nine or more query tokens: in an isolated kernel timing, the step’s GEMMs cost 22.7 ms at up to eight rows and 25.9 ms from nine to sixteen. We wrote a Triton INT4 kernel for those shapes. It was correct, and 3.5 times slower than the oneDNN kernel it was meant to replace, so we dropped it. Longer drafts still don’t pay off, and k=7 stays the setting.

Where we stopped, and why

115 tok/s needs about five kept tokens per step at a 44 ms step. Every drafter we have keeps four to 4.2 under sampling, and even a perfect fix for the nine-token slowdown projects to about 95–100. We have no lever left that gets to 115 on this card without a much larger drafter-training run. So we’re retiring the target and shipping what we measured: v1.1.1, at 92.5 to 96 tok/s, now registered in our local coding agent as the default for this model, with the 262k-context MTP profile next to it.

We also have a W4A8 build planned, with INT8 activations on selected layers. We expect it to speed up prefill, not decode, because decode on this card is limited by memory bandwidth and W4A8 reads the same 4-bit weights. It also has little quality headroom: B2’s KL divergence is 0.041 against a 0.05 bar. So it will be its own study, later.

Things that went wrong

• A benchmark that cancelled itself. evalplus 0.3.1 cancels every request after 100 seconds and retries forever. With thinking on, the answers took longer than that, so each problem was sent again and again while the abandoned generations kept running upstream: 56 requests in flight and a KV cache 96% full. A wrapper with a one-hour timeout and three tries fixed it.
• vLLM’s default batch size on a hybrid model. vLLM 0.30’s default of 1,024 sequences exceeded the model’s 670 linear-attention cache blocks and crashed the BF16 server. Capping it at 256 fixed it.
• Calibration that depends on the library version. transformers 5.22 on the host builds the C4 calibration windows in a different order from 5.16 in the quantization image. Calibration data is now always built inside the image.
• Tokenizer files rewritten by the quantizer. They’re harmless for serving, but they don’t match upstream. The originals go back before upload.
• A rented host that disappeared. The first H100 generating the training data went offline mid-run and took 3–4 thousand answers with it. A replacement finished the job, and long rentals now copy their partial results home every 10 minutes.
• Four false starts on the training box. A download flag that only took its first pattern, an export mode that doesn’t support DFlash, a config path it wanted as a file, and a port the smoke test’s server held for 40 seconds after it exited. Each one stopped before training began, and the box was never torn down by mistake. Together they cost about 15 minutes of rental.

Releases

Every page from this run

• The model on Hugging Face, with the tags v1.0, v1.0.1, v1.1 and v1.1.1.
• BF16 vs INT4 v1.0 quality study: the shipped build against BF16, all seven tasks (current).
• The ablation study: arms A, B and B2 against BF16.
• Drafter study: the draft-length sweep and the self-distilled drafter.
• Quick BetterBench, DFlash2 k=7 with the verify-step patch: 96.0 tok/s (second of two runs; the first measured 92.5), 128k context.
• Quick BetterBench, v1.0 with stock DFlash2 k=7: 89.3 tok/s, 128k context.
• Quick BetterBench, B2 on the B65: 75.8 tok/s, MTP k=6.
• Quick BetterBench, first bake (B) on the B65: 78.9 tok/s.
• Study with complete thinking-on results, before B2 existed.
• BF16 quality reference on its own.
• BF16 task baselines, thinking off and on.

Pages on launch80 are self-reported until they are verified, and each one carries a badge saying which it is.

Log

• 1 NovRequest from Wrapzii. Thresholds registered and hashed. BF16 download (55.6 GB). Body quantization began after the malloc-arena fix.
• 2 NovBake passed its gate 9/9 and the first quick BetterBench ran at 78.9 tok/s. Three BF16 reference rentals. Arms A, B and C measured. BFCL regression found and traced to calibration. Body recalibrated with tool calls, giving A′ and B2; B2 passes BFCL. Study and B2 BetterBench published. MTP head found to be stock. The full seven-task rerun on B2 as shipped started.
• 2 Nov · v1.0B2 released on Hugging Face as v1.0, with the source tokenizer files restored. The model card carries the pre-registered results so far, the post-hoc disclosure, and a version table.
• 2 Nov · v1.0.1All seven tasks rerun on the v1.0 checkpoint as shipped, MTP k=6: all seven pass. New single-arm study page and an updated model card (tag v1.0.1; weights unchanged). The drafter measurements toward 115 tok/s started.
• 3 NovDraft-length sweep done: stock DFlash2 at k=7 is best, and a quick BetterBench run measured 89.3 tok/s (page). Training data generated on a rented H100 (one host outage, re-rented). DFlash2 drafter fine-tuned on Qwopus’s answers on two rented H200s; no checkpoint beat stock (drafter study). Next: the verify-step slowdown at nine query tokens.
• 3 Nov · v1.1The 89.3 tok/s configuration released as v1.1: same weights, a model card with both serving profiles, and the drafter’s card updated with its Qwopus numbers. Spec-decoding parity for the new profile: 28 of 32 identical, all four differences at a tie.
• 3 Nov · v1.1.1Verify-step profile found a per-step host wait in vLLM’s GDN metadata; a boot patch removes it: 92.5 and 96.0 tok/s in two quick runs (page), parity 30 of 32. A Triton INT4 kernel for the nine-token shapes was 3.5× slower than oneDNN and was dropped. The 115 tok/s target is retired. T6 measured: our build fails against Jackrong’s Q4_K_M GGUF (KLD 0.041 vs 0.025). Released as v1.1.1 (card only) and registered in our coding agent.
• 4 Nov · v2.0 not releasedA follow-up build (AutoRound asymmetric body, g32 head) got closer to BF16 on every measure, KLD 0.041 → 0.033, but decoded about 4% slower and could not run the DFlash2 drafter. v1.1.1 stays the release. Write-up: A Better INT4 Qwopus We Decided Not to Ship.