---
topic: ai-engineering
author: Crashtech Editorial
published: Aug 5, 2026 · read: 5 min
---

Soft Labels and Teacher Ceilings: What Knowledge Distillation Actually Transfers, and Where It Stops

How dark knowledge, soft probability distributions, and synthetic reasoning traces allow 7B student models to outperform 32B base models.

– –

In modern machine learning benchmarks, an apparent paradox frequently appears: a small 7-billion-parameter language model fine-tuned on the outputs of a frontier 671-billion-parameter reasoning model can decisively outperform a traditional 32-billion or 70-billion-parameter base model on specialized math and coding benchmarks.

How can a model with a fraction of the parameter capacity outscore systems four times its size?

The answer lies in Knowledge Distillation.

Distillation is often conflated with model compression (like quantization or pruning), but it is a fundamentally distinct discipline. Compression squashes an existing model’s weights into fewer bits. Distillation trains a brand new, separate neural network from scratch, using the probabilistic reasoning outputs of a massive “teacher” model as its supervision signal.

Understanding how distillation transfers “dark knowledge,” where it hits hard capacity ceilings, and how it is revolutionizing edge inference is essential for modern AI systems architects.

+-------------------------------------------------------------+
|             THE DISTILLATION PARADOX                        |
|                                                             |
| Large Teacher Model (671B Params): Generates Reasoning Data |
|                            |                                |
|                            v                                |
| Small Student Model (7B Params): Learns Reasoning Patterns  |
|                            |                                |
| Result on Math Benchmarks: Student (7B) > Base Model (32B)  |
+-------------------------------------------------------------+

Hard Labels vs. Soft Labels: Unlocking “Dark Knowledge”

To understand why distillation works, consider how neural networks are traditionally trained.

In standard supervised learning, models train on hard labels. Given an image of a cat, the ground truth vector is a one-hot vector:

Target = [cat: 1.0, dog: 0.0, car: 0.0, fox: 0.0]

This binary signal discards an enormous amount of structural reality. A cat is biologically related to a dog and looks somewhat like a fox; it shares zero characteristics with a car.

A massive teacher model does not output a binary choice. Its final softmax layer emits a continuous probability distribution across its entire vocabulary, known as soft labels:

Teacher Output = [cat: 0.72, dog: 0.18, fox: 0.099, car: 0.001]

Geoffrey Hinton and colleagues famously coined the term “Dark Knowledge” to describe the rich structural information hidden inside these non-target probabilities. The small probability assigned to “dog” and “fox” teaches the student model the geometric relationships between concepts.

Hard Target:   [Cat: 1.0,  Dog: 0.0,  Car: 0.0]   -> Zero relationship data
Soft Teacher:  [Cat: 0.75, Dog: 0.24, Car: 0.001] -> Encodes semantic similarity!

Temperature Scaling

To expose this dark knowledge during distillation, researchers scale the softmax function using a Temperature parameter ($T$):

q_i = exp(z_i / T) / sum_j(exp(z_j / T))

When $T = 1$, the softmax distribution is sharp, dominated by the winning token. When $T$ is raised (e.g., $T = 4$ or $T = 8$), the probability distribution softens, amplifying the dark knowledge in the tail probabilities so the student can absorb subtle semantic nuances.

The Three Flavors of Modern Distillation

Depending on what level of access you have to the teacher model, distillation takes three architectural forms:

[1. Output Distillation]    : Student matches Teacher's final token logits (Kullback-Leibler Divergence)
[2. Feature Distillation]   : Student matches intermediate hidden states & attention maps
[3. Synthetic Trajectory]   : Student trains on Chain-of-Thought (CoT) reasoning traces (DeepSeek style)

1. Logit Distillation (Matching Probabilities)

The original form of distillation. The student’s training loss is a weighted combination of standard cross-entropy on ground truth and Kullback-Leibler (KL) Divergence against the teacher’s soft probability vector:

Loss = (1 - alpha) * CrossEntropy(y, sigma(z_s)) + alpha * (T^2) * KL_Divergence(sigma(z_t / T), sigma(z_s / T))

2. Feature and Representation Distillation

Instead of only matching final outputs, feature distillation forces the student’s intermediate hidden layers to mimic the teacher’s internal vector representations. This is widely used when training high-efficiency embedding models (such as EmbeddingGemma).

3. Synthetic Reasoning Distillation (The 2025-2026 Paradigm)

Frontier labs rarely expose full logit distributions over their commercial APIs; they return plain text.

Modern distillation leverages large reasoning models (such as DeepSeek-R1 or o1) to generate hundreds of thousands of long-form Chain-of-Thought (CoT) reasoning trajectories. The student model is fine-tuned directly on these verified thinking processes:

  • The teacher generates the step-by-step mathematical proof or debugging sequence.
  • Flawed traces are filtered out using automated compilers or unit tests.
  • The student model is trained on the curated traces, effectively inheriting the teacher’s structured reasoning habits without requiring 600B parameters.

The Limits: Where Distillation Hits the Wall

Despite its dramatic success on narrow benchmarks, distillation is not a magical silver bullet. It is bounded by four fundamental constraints:

1. The Teacher Ceiling

A student cannot reliably exceed the broad cognitive frontier of its teacher. If the teacher possesses systematic blind spots, factual hallucinations, or historical biases, those errors are baked directly into the student’s weights with high confidence.

2. The Capacity Gap

If the size difference between teacher and student is too vast (e.g., trying to distill a 400B model directly into a 500M edge model), the student lacks the representational capacity to model the teacher’s complex decision manifold. In such cases, researchers use a Teacher Assistant: a medium-sized intermediate model (e.g., 70B) is trained first, and then distills down to the smallest model.

3. General Knowledge vs. Task Narrowing

Distillation excels on formal, rule-bound domains: competition mathematics, code generation, SQL synthesis, and structured JSON parsing.

However, on broad open-domain knowledge, historical trivia, and nuanced multi-lingual discourse, parameter count is king. A 7B student simply does not have enough physical memory in its feed-forward layers to store the facts of the world, and will hallucinate broad knowledge far more frequently than a 70B base model.

4. Subliminal Trait Transmission

Recent research revealed an intriguing phenomenon: student models inherit unprompted stylistic quirks, political leanings, and subtle reasoning traits from their teacher even when the training data is scrubbed of explicit keywords. The structure of the reasoning path itself acts as a carrier wave for the teacher’s latent behavioral traits.

Strategic Decision Matrix

DimensionKnowledge DistillationPost-Training Quantization (PTQ)
Model IdentityTrains a separate, brand-new modelCompresses the exact same model weights
Silicon RequirementsStudent runs natively on 1 consumer GPUModel still needs enough VRAM for parameters
Accuracy ProfileOutperforms base models on narrow domainsMinor perplexity degradation across all tasks
Training CostRequires high compute to train studentNear-zero compute (algorithmic conversion)

For engineering teams building production AI applications, knowledge distillation is the premier lever for unit economics: deploy frontier teachers to synthesize verified reasoning paths, and distill that knowledge into ultra-fast, single-GPU student models that execute at millisecond latencies.

Advertisement

/* Comments */