Parallel Inference & API Fallback

Real parallel benchmarks, context scaling tables, and API as the true "speed tier"

Real Benchmarks (measured on this hardware)

Single-stream generation (measured, @34K context)

Parallel scaling: 0.8B Q4_K_M (real concurrent requests, 128 tokens each)

Context Scaling (real measured data)

All models use Q8 KV cache + 262K max context + activation rotation. Clean restart per test.

9B IQ4_NL (Interactive tier)

35B-A3B IQ4_NL regex (Workhorse fast mode)

35B-A3B IQ4_NL -cmoe (Workhorse full context mode)

Prompt Processing Time (real)

VRAM Budget: Qwen3.5-4B Q4_K_M

Qwen3.5-4B's hybrid architecture means KV cache per token is only ~32KB (vs ~144KB for a standard 4B transformer):

You could theoretically run 16 parallel slots at 16K context each on your 3060 with Qwen3.5-4B. The Gated DeltaNet layers use O(1) state, so only the 8 attention layers contribute to KV cache.

Gemma 4 vs Qwen 3.5: Head-to-Head for Parallel Agentic Use

Small Models (sub-4B)

MoE Tier (26B class)

Candidates to Manually Benchmark Per Tier

Gemma 4 Known Bugs in llama.cpp

Parallel Inference: Gotchas

1. Quadratic attention overhead: Unified KV cache means attention is computed over the entire cache then masked per-slot — wasteful with many slots
2. CPU sampling bottleneck: Sampling is per-slot on CPU. Your 2014 Xeon will struggle at high N — watch for GPU utilization dropping below 40%
3. KV cache full = sequence killed: When cache fills, llama.cpp terminates the largest sequence. No graceful offloading.
4. Nondeterministic outputs: cache_prompt can cause non-identical logits across different batch sizes
5. One server, not four: Multiple llama-server instances each load model weights separately. On 12GB, only one server with --parallel N is viable.

Benchmark Commands

Run this to get your actual numbers

# Qwen3.5-4B: parallel scaling benchmark
llama-batched-bench -m Qwen3.5-4B-Q4_K_M.gguf -ngl 99 -fa 1 \
  -c 32768 -b 2048 -ub 512 \
  -npp 512,2048 -ntg 128,512 -npl 1,2,4,8

# Gemma 4 E2B: parallel scaling benchmark
llama-batched-bench -m gemma-4-E2B-it-Q8_0.gguf -ngl 99 -fa 1 \
  -c 16384 -b 2048 -ub 512 \
  -npp 512,2048 -ntg 128,512 -npl 1,2,4,8

# Single-stream baseline for each candidate
llama-bench -m Qwen3.5-4B-Q4_K_M.gguf -ngl 99 -fa 1
llama-bench -m gemma-4-E2B-it-Q8_0.gguf -ngl 99 -fa 1
llama-bench -m gemma-4-E4B-it-Q8_0.gguf -ngl 99 -fa 1

Parallel server for agentic use

# Qwen3.5-4B with 4 parallel agent slots
llama-server -hf unsloth/Qwen3.5-4B-GGUF:Q4_K_M \
  -ngl 99 -fa --jinja \
  --parallel 4 -c 32768 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --port 8080

# Each slot gets 8K context (32768 / 4)
# Total KV cache: ~1GB (Q8_0) — leaves 6.75GB free

API Fallback: Models Too Big for Local

For frontier models that can't run on 12GB, cheap API options ranked by price:

Speed-Optimized APIs (for latency-sensitive agent loops)

Monthly Cost Comparison (1000 queries/day, ~1K input + 500 output tokens each)

Aggregators

• OpenRouter — single API key for all models, automatic failover, often cheapest. OpenAI-compatible format.
• Groq — unbeatable for latency (500-1000 t/s server-side). Limited model selection (Llama/Qwen/GPT-OSS only).
• Together.ai — batch API at 50% off. Free tier available.
• Fireworks.ai — prompt caching at ~50% discount. Good for repeated system prompts in agentic loops.
• NanoGPT — pay-as-you-go aggregator starting at $0.01 top-up. Not a dedicated plan for any model.

Hybrid Strategy: Local + API (data-driven)