#model-distillation

#model-distillation

DeepSeek trained its R1 reasoning model for $294,000 using 512 Nvidia H800 chips in 80 hours, employing distillation and some A100s.

Student models trained on teacher model outputs can acquire unrelated traits and misaligned behaviors through distillation, transferring subtle biases even when explicit cues are filtered.

Fine-tuned models can inherit traits from base models despite efforts to filter data, requiring stricter safety evaluations.