Model Training
Distillation (Model Distillation)
Teaching a small, fast AI model to mimic a large, expensive one, so you get similar results at a fraction of the cost.
Definition
A technique where a smaller "student" model is trained to replicate the behavior of a larger "teacher" model. The student learns from the teacher's soft probability outputs, not just hard labels.
Why it matters
Enables deploying AI on edge devices and reducing inference costs while maintaining quality.
Related terms in Model Training
Adversarial Training
Teaching an AI to defend itself by constantly attacking it with tricky or malicious inputs during training.
Autoencoders
A neural network that learns to compress data into a small code and then unzip it back to the original.
Dropout
Randomly turning off some neurons during training so the AI doesn't over-memorize and can generalize better.
Epoch
One complete pass through the entire training dataset, the AI sees every example once per epoch.
From vocabulary to outcomes
Ready to put Distillation (Model Distillation) to work?
Knowing the term is step one. Deploying it inside a revenue architecture that compounds is what Sophizo builds.
Book a Discovery Call