distiller

Distiller Trainer API

This module contains the implementation of a distiller trainer that can be used to train a student model

Classes

Name	Description
Distiller	A trainer for distillation

Machine-readable model

• DistillerclassTrain a student using target labels and a teacher's softened predictions.

class

Distiller

Python

Train a student using target labels and a teacher's softened predictions.

helia_edge/trainers/distiller.py:16

Distiller(student: keras.models.Model, teacher: keras.models.Model)

Train a student using target labels and a teacher’s softened predictions.

call() returns the student’s output. The distillation objective evaluates the teacher with training=False and applies temperature-scaled softmax along axis 1, so predictions must have their class dimension on that axis. Configure the optimizer, two loss functions and mixing weight with compile(). The objective does not apply sample_weight.

Parameters

Parameters of Distiller

Name	Type	Default	Description
student	keras.models.Model	Required	Keras model whose predictions are returned and scored.
teacher	keras.models.Model	Required	Keras model providing the reference predictions.

attribute

teacher

Python

helia_edge/trainers/distiller.py:36

teacher: keras.models.Model = teacher

attribute

student

Python

helia_edge/trainers/distiller.py:37

student: keras.models.Model = student

method

compile

Python

Configure the distiller.

helia_edge/trainers/distiller.py:39

compile(
    optimizer: keras.optimizers.Optimizer,
    metrics: list[keras.metrics.Metric],
    student_loss_fn: keras.losses.Loss,
    distillation_loss_fn: keras.losses.Loss,
    alpha: float = 0.1,
    temperature: float = 3,
)

Configure the distiller.

Parameters

Parameters of compile

Name	Type	Default	Description
optimizer	keras.optimizers.Optimizer	Required	Keras optimizer for the student weights
metrics	list[keras.metrics.Metric]	Required	Keras metrics for evaluation
student_loss_fn	keras.losses.Loss	Required	Loss function of difference between student predictions and ground-truth
distillation_loss_fn	keras.losses.Loss	Required	Loss function of difference between soft student predictions and soft teacher predictions
alpha	float	0.1	weight to student_loss_fn and 1-alpha to distillation_loss_fn. Defaults to 0.1.
temperature	float	3	Temperature for softening probability distributions. Defaults to 3.

method

compute_loss

Python

helia_edge/trainers/distiller.py:66

compute_loss(x=None, y=None, y_pred=None, sample_weight=None, allow_empty=False)

method

call

Python

helia_edge/trainers/distiller.py:78

call(x)