Mini Whale 1 12B

Abstract

We present Mini Whale 1 12B, a Mixture-of-Experts language model that fuses a Qwen3-4B host with 260 SwiGLU coding experts extracted from DeepSeek-V4-Flash. The architecture uses low-rank bridge layers (rank 7) to translate between the host's 2560-dimensional representation space and the experts' 4096-dimensional space, a sqrt-softplus router for top-2 expert selection, and residual-safe gating to preserve the host's reasoning ability. The model is fine-tuned with QLoRA (4-bit NF4) and optimized for local inference on a single RTX 3060 12GB. With DSpark speculative decoding, we achieve 10 tokens/second — a 2× speedup over baseline 4-bit generation — while maintaining Qwen3-quality reasoning and DeepSeek-quality code generation.

1. Architecture

1.1 Overview

Mini Whale 1 12B is built on a "fusion" principle: rather than training a new model from scratch, we take an existing language model (Qwen3-4B) and augment each of its 36 transformer layers with coding experts borrowed from a larger, specialized model (DeepSeek-V4-Flash). The host model handles language understanding, reasoning, and general text generation. The experts handle code generation. A learned router decides, per token, which experts to activate.

Input Tokens
    │
    ▼
┌─────────────────────────────────────────────┐
│  Qwen3-4B Host (36 layers, hidden=2560)     │
│                                             │
│  For each layer:                            │
│  1. Self-Attention (SDPA, 32 heads)         │
│  2. Host MLP (SwiGLU, 9728 intermediate)    │
│  3. Coding Expert Augmentation:             │
│     a. Bridge In  (2560 → 4096)             │
│     b. Router (sqrt-softplus, top-2)        │
│     c. Expert SwiGLU (2 of N experts)       │
│     d. Bridge Out (4096 → 2560)             │
│     e. RMSNorm + Sigmoid Gate               │
│     f. Residual Repair (rank-7)             │
│     g. Residual Addition                    │
│  4. RMSNorm                                 │
└─────────────────────────────────────────────┘
    │
    ▼
LM Head (tied embeddings, 151,936 vocab)
    │
    ▼
Output Logits → Token

1.2 Expert Distribution

All 36 layers are augmented, but the number of experts per layer varies based on the original DeepSeek-V4-Flash architecture. Early layers use fewer experts (1–8) for basic coding patterns. Deep layers (29–35) use up to 25 experts for complex code synthesis. In total, 303 expert instances are spread across the model, drawn from a pool of 260 unique experts (some shared across layers).

1.3 Bridge Layers

The bridge is the key architectural innovation. Qwen3 operates in a 2560-dimensional representation space; DeepSeek's experts expect 4096-dimensional inputs. A direct projection would either lose information (2560 → 4096 is an up-projection) or overwhelm the experts with out-of-distribution inputs. We use a low-rank (rank 7) linear bridge in both directions:

• bridge_in: Linear(2560, 4096) — maps host hidden states to expert space
• bridge_out: Linear(4096, 2560) — maps expert outputs back to host space

The low rank keeps the parameter count small (~18K per bridge per layer) and acts as a bottleneck that prevents the coding path from overwhelming the host's residual stream.

1.4 Router

Expert selection uses a sqrt-softplus scoring function rather than standard softmax. This is smoother than softmax and prevents expert collapse (one expert dominating all tokens):

logits = router.gate(hidden_states)              # (tokens, num_experts)
scores = sqrt(clamp(softplus(logits), min=1e-6)) # sqrt-softplus
topk_weights, topk_indices = scores.topk(2)      # top-2 selection
topk_weights = topk_weights / topk_weights.sum() # normalize

The clamp(min=1e-6) before sqrt is critical — without it, very negative logits produce softplus → 0 → sqrt(0) = 0, but the gradient sqrt'(0) = ∞, causing NaN during training.

1.5 Residual Safety

The coding delta is gated and clamped to prevent it from disrupting the host's representation. Three mechanisms work together:

1. Sigmoid gate (init -2.0 → ~12% pass-through) — controls how much coding signal enters the residual stream
2. Residual clamp — coding delta is scaled to at most 10% of the host signal's norm
3. Rank-7 repair — a low-rank residual correction that compensates for any disruption

This ensures the coding experts contribute to the output without dominating it. The host model's reasoning ability is preserved.

1.6 SwiGLU Clamping

DeepSeek-V4-Flash uses swiglu_limit=10.0 to clamp expert intermediate activations. Without this clamp, outlier values grow exponentially across 36 layers — layer 0 produces max ~8.6, layer 3 reaches NaN. The clamp is baked into the SwiGLUExpert.forward method:

gate_up = F.silu(gate_proj(x)) * up_proj(x)
gate_up = gate_up.clamp(-10.0, 10.0)   # ← critical
return down_proj(gate_up)

2. Training

2.1 QLoRA Fine-Tuning

The model is fine-tuned using QLoRA: the host model is frozen in 4-bit NF4 quantization, while LoRA adapters are trained on attention projections (q/k/v/o) and the bridge/router/repair layers. Only 19.5M parameters are trainable — 0.16% of the total 12B.

2.2 Training Configuration

• Dataset: Coding instructions (Python, JavaScript, TypeScript)
• Steps: 500
• Loss: 0.62 → 0.19
• Optimizer: AdamW, lr=2e-4, cosine schedule
• Hardware: RTX 3060 12GB (local)
• VRAM: 6.6 GB (model + optimizer states)

2.3 Merge

After training, LoRA adapters are merged into the base model using a streaming tensor-by-tensor merge: 216 LoRA modules merged (B = A @ B for each adapter), 144 checkpoint overrides applied, 1285 tensors copied unchanged. The output is a 23.4 GB BF16 model in 5 safetensors shards.

3. Quantization & Local Inference

3.1 4-bit NF4 Quantization

All linear layers (attention projections + MLP) are quantized to 4-bit NF4 (NormalFloat 4-bit) using bitsandbytes. This reduces the model from 23.4 GB (BF16) to 8.9 GB (4-bit), fitting comfortably within the 12 GB VRAM of an RTX 3060.

3.2 SDPA Attention

The model uses PyTorch's Scaled Dot-Product Attention (SDPA) fused CUDA kernel instead of the default "eager" Python-loop attention. This requires all RMSNorm/LayerNorm weights to be in bfloat16 (not float32), which is handled automatically by the from_pretrained override in the model code.

3.3 VRAM Breakdown

On RTX 3060 12GB, Windows reserves ~0.5 GB for display. If reserved VRAM exceeds ~11.5 GB, CUDA silently spills to system RAM via PCIe, causing a 50× slowdown (from 10 tok/s to 0.2 tok/s). The shared-embedding configuration (drafter reuses the target's embedding layer) saves 0.8 GB, keeping reserved VRAM at 11.4 GB — just under the limit.

4. DSpark Speculative Decoding

4.1 Method

DSpark (Draft Speculative) decoding uses a small drafter model (5-layer Qwen3, 2560 hidden) to predict 7 tokens in a single forward pass. The target model then verifies all 7 tokens in one forward pass. The longest matching prefix (greedy argmax) is accepted, plus one bonus token from the target.

┌─────────────┐     ┌──────────────┐
│   Drafter   │────▶│    Target    │
│  (5 layers) │     │  (36 layers) │
│  Predicts   │     │  Verifies    │
│  7 tokens   │     │  in 1 pass   │
└─────────────┘     └──────────────┘
     ~32ms              ~330ms

4.2 Optimizations

• Forward hooks on 5 target layers (not output_hidden_states=True on all 36) — captures hidden states for the drafter without overhead
• Single target forward per block — no sequential fallback, even on rejection
• 4-bit drafter — quantized to NF4 to save 0.6 GB VRAM
• Shared embedding — drafter reuses target's embedding layer (saves 0.8 GB)
• Sliding window KV cache (512 tokens) — prevents speed degradation on long sequences

4.3 Acceptance Rate

At 2.4/7 average acceptance, each block produces 3.4 tokens (2.4 accepted + 1 bonus) in ~400ms (330ms target + 70ms drafter), yielding ~8.5 tok/s. With BF16 attention (enabled via shared embedding), the target forward drops to ~300ms, yielding ~10 tok/s.

5. Performance

5.1 Speed

The sliding window KV cache keeps speed constant at 8.5 tok/s for sequences of any length. Without it, speed degrades from 10 tok/s to 1.1 tok/s at 1265 tokens due to KV cache growth.

5.2 Quality

The model produces Qwen3-quality chain-of-thought reasoning (the host handles this) and DeepSeek-quality code generation (the experts handle this). The residual-safe gating ensures seamless switching between reasoning and code within a single generation.

5.3 Example Output

Prompt: "Write a Python function to check if a number is prime."

Output (excerpt):
  Okay, I need to write a Python function to check if a
  number is prime. Let me think about how to approach this.

  First, a prime number is a number greater than 1 that has
  no divisors other than 1 and itself...

  1. Check if the number is less than 2 → return False.
  2. Check if the number is 2 → return True
  3. Check if the number is even → if yes, return False
  4. Iterate from 3 to sqrt(n), stepping by 2
  5. For each i, check if n is divisible by i
  6. If none divide n, return True

6. Key Config

7. Availability

The model is publicly available on HuggingFace:

huggingface.co/Akahsizrr/Mini-Whale-1-12B ↗

All runtime fixes (SwiGLU clamping, router stability, meta-param auto-init, norm dtype casting) are baked into the model code. Loading with AutoModelForCausalLM.from_pretrained() + a BitsAndBytesConfig produces a working model with zero post-load patching.

Citation

@misc{miniwhale1,
  title  = {Mini Whale 1 12B: Fusing Qwen3-4B with DeepSeek Coding Experts},
  author = {Vasko Djack},
  year   = {2026},
  url    = {https://huggingface.co/Akahsizrr/Mini-Whale-1-12B}
}

Acknowledgments

• Qwen3-4B by Alibaba/Qwen Team — host model
• DeepSeek-V4-Flash by DeepSeek AI — coding experts
• DSpark speculative decoding framework
• bitsandbytes for 4-bit NF4 quantization