losses

AdditiveAngularMarginLoss

class AdditiveAngularMarginLoss(embedding_features, n_classes, scale=64.0, margin=0.5, **kwargs)

AKA ArcFaceLoss.

Reference

Deng et al. : ArcFace: Additive Angular Margin Loss for Deep Face Recognition

Note

  • this loss has weights and requires an optimizer.

  • margin is already expressed in radians.

Parameters

  • embedding_features (int) – number of embedding features inputs are expected to be (batch, embedding_features).

  • n_classes (int) – number of classes in projection.

  • scale (float) – Defaults to 64.0.

  • margin (float) – margin in radians. Defaults to 0.5 (28.6 degrees).

  • eps – epsilon for clamping . Defaults to 1e-7.

  • reduction – batch reduction for this loss should be one of ‘mean’, ‘sum’, ‘none’. Defaults to ‘mean’.

Example

>>> import torch
>>> from hearth.losses import AdditiveAngularMarginLoss
>>> _ = torch.manual_seed(666)
>>>
>>>
>>> # this would be embeddings coming out of your model...
>>> emb = torch.normal(0, 1, size=(5, 128))
>>> targets = torch.randint(0, 10, size=(5,))
>>> loss = AdditiveAngularMarginLoss(128, 10)
>>> loss(emb, targets)
tensor(41.8191, grad_fn=<MeanBackward0>)

BinaryFocalLoss

class BinaryFocalLoss(alpha=0.25, gamma=2.0, mask_target_value=- 1, reduction='mean')

Binary focal loss.

Focal loss is a proposed solution for handling class imbalambce in segmentation.

Reference

Li et al. : Focal Loss for Dense Object Detection

Parameters

  • alpha (float) – class weighting factor. Defaults to 0.25 (defaults taken from paper).

  • gamma (float) – focusing factor. Defaults to 2.0 (default taken from paper).

  • mask_target_value (int) – this index will be masked when seen in the targets upon computing the loss. Defaults to -1.

  • reduction (str) – string name of reduction. Defaults to ‘mean’.

Shape

  • inputs: \((N, *)\) where \(*\) means, any number of additional dimensions if inputs have an extra single dimension thats ok… they will be reshaped as targets.

  • targets: \((N, *)\).

  • output: scalar unless reduction is 'none', then \((N, *)\), same shape as targets.

Example

>>> import torch
>>> from hearth.losses import BinaryFocalLoss
>>>
>>> targets = torch.tensor([0, 1, 0, 0, 1, 0], dtype=torch.float32)
>>> inputs = torch.tensor([-1.1645,  -0.2928, -0.5685, -0.8038, -0.0211,  2.0062])
>>> loss = BinaryFocalLoss()
>>> loss
BinaryFocalLoss(alpha=0.25, gamma=2.0, mask_target_value=-1, reduction='mean')

>>> loss(inputs, targets)
tensor(0.2399)

if your inputs have an extra dim thats ok:

>>> loss(inputs.unsqueeze(-1), targets)
tensor(0.2399)

supports masking by value for instance if we want to predict a binary value over time and the last timestep for the last example is missing, we can mask by setting mask_target_value to -1 and padding targets with -1:

>>> masked_targets =  torch.tensor([[0, 1, 0], [0, 1, -1]], dtype=torch.float32)
>>> inputs = inputs.reshape((2, 3, -1)) # (batch, time, 1)
>>> loss(inputs, masked_targets)
tensor(0.0393)

when reduction is 'none' we will return one loss value per step as you can see the masked targets are 0.0:

>>> loss = BinaryFocalLoss(reduction='none')
>>> loss(inputs, masked_targets)
tensor([[0.0115, 0.0697, 0.0440],
        [0.0265, 0.0449, 0.0000]])
forward(inputs, targets)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Return type

Tensor

extra_repr()

Set the extra representation of the module

To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable.

Return type

str

LargeMarginCosineLoss

class LargeMarginCosineLoss(embedding_features, n_classes, scale=30.0, margin=0.4, **kwargs)

AKA CosFaceLoss.

Reference

Wang et al. : CosFace: Large Margin Cosine Loss for Deep Face Recognition

Note

  • this loss has weights and requires an optimizer.

Parameters

  • embedding_features (int) – number of embedding features inputs are expected to be (batch, embedding_features).

  • n_classes (int) – number of classes in projection.

  • scale (float) – Defaults to 30.0.

  • margin (float) – Defaults to 0.4.

  • eps – epsilon for clamping . Defaults to 1e-7.

  • reduction – batch reduction for this loss should be one of ‘mean’, ‘sum’, ‘none’. Defaults to ‘mean’.

Example

>>> import torch
>>> from hearth.losses import LargeMarginCosineLoss
>>> _ = torch.manual_seed(666)
>>>
>>>
>>> # this would be embeddings coming out of your model...
>>> emb = torch.normal(0, 1, size=(5, 128))
>>> targets = torch.randint(0, 10, size=(5,))
>>> loss = LargeMarginCosineLoss(128, 10)
>>> loss(emb, targets)
tensor(17.5245, grad_fn=<MeanBackward0>)

MaskedMSELoss

class MaskedMSELoss(mask_target_value=- inf, reduction='mean')

MSELoss with support for masked targets.

Parameters

  • mask_target_value – ignore targets with this value. Defaults to -inf.

  • reduction (str) – Defaults to ‘mean’.

Example

>>> import torch
>>> _ = torch.manual_seed(0)
>>> from hearth.losses import MaskedMSELoss
>>>
>>> ninf = -float('inf')
>>> loss = MaskedMSELoss()
>>> inputs = torch.rand(3, 5) # (batch, timesteps)
>>> targets = torch.tensor([[ 1.1721,  0.3909, -5.2731,    ninf,   ninf],
...                         [ 2.4388,  2.5159, -1.0815, -1.9472, -0.5450],
...                         [ 4.0665, -2.5141,    ninf,    ninf,   ninf]])
>>> loss(inputs, targets)
tensor(7.0101)
forward(input, target)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Return type

Tensor

reduction: str

MultiHeadLoss

class MultiHeadLoss(*, weights=None, aggregate_key='weighted_sum', **kwargs)

wrapper for losses for models with multiple output heads.

Parameters

  • weights (Union[NumberDict, Dict[str, int], None]) – a mapping of key to scalar weight, will be multiplied with losses before summing to create the aggregate value at aggregate_key, need not sum to 1. Defaults to None (all losses weighted evenly).

  • aggregate_key (str) – the key to use for the aggregate loss. Defaults to ‘weighted_sum’.

Example

>>> import torch
>>> from torch import nn
>>> from hearth.losses import MultiHeadLoss
>>>
>>> _ =torch.manual_seed(0)
>>>
>>> loss = MultiHeadLoss(a=nn.BCEWithLogitsLoss(),
...                      b=nn.CrossEntropyLoss())
>>> loss
MultiHeadLoss(a=BCEWithLogitsLoss(),
              b=CrossEntropyLoss(),
              weights=NumberDict({'a': 1.0, 'b': 1.0}),
              aggregate_key=weighted_sum)

multihead loss expects inputs and targets to be dicts with containing the it’s keys (in this case ‘a’ and ‘b’):

>>> batch_size = 10
>>> inputs = {'a': torch.rand(batch_size, 1),
...           'b': torch.normal(batch_size, 1, size=(batch_size, 4))}
>>> targets = {'a': torch.rand(batch_size, 1).round(),
...            'b': torch.randint(4, size=(batch_size,))}
>>>
>>> loss(inputs, targets)
TensorDict({'a': tensor(0.5791), 'b': tensor(1.2425), 'weighted_sum': tensor(1.8216)})

weighted_sum is the default aggregate key. this is the bit you should call backward on. you can change the default aggregate key if you like by specifying it at init.

>>> loss = MultiHeadLoss(a=nn.BCEWithLogitsLoss(),
...                      b=nn.CrossEntropyLoss(),
...                      aggregate_key='sally')
>>> loss(inputs, targets)
TensorDict({'a': tensor(0.5791), 'b': tensor(1.2425), 'sally': tensor(1.8216)})

you can aslo specify weights at init to weight contribution losses differently:

>>> loss = MultiHeadLoss(a=nn.BCEWithLogitsLoss(),
...                      b=nn.CrossEntropyLoss(),
...                      weights={'a': .2, 'b':.8})
>>> loss(inputs, targets)
TensorDict({'a': tensor(0.5791), 'b': tensor(1.2425), 'weighted_sum': tensor(1.1098)})
property weights
forward(inputs, targets, **kwargs)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Return type

TensorDict

training: bool

MulticlassFocalLoss

class MulticlassFocalLoss(alpha=None, gamma=2.0, mask_target_value=- 1, reduction='mean')

multiclass focal loss.

Focal loss is a proposed solution for handling class imbalambce in segmentation.

Reference

Li et al. : Focal Loss for Dense Object Detection

Parameters

  • alpha (Union[Tensor, float, None]) – class weighting factor, may be a scalar, or a tensor with one weight per class if None alpha=1.0. Defaults to None.

  • gamma (float) – focusing factor. Defaults to 2.0.

  • mask_target_value (int) – this index will be masked when seen in the targets upon computing the loss. Defaults to -1.

  • reduction (str) – string . Defaults to ‘mean’.

Shape

  • inputs: \((N, *)\) where \(*\) means, any number of additional dimensions

  • targets: \((N, *, C)\), where C = number of classes.

  • output: scalar unless reduction is 'none', then \((N, *)\), same shape as targets.

Example

>>> import torch
>>> from hearth.losses import MulticlassFocalLoss
>>>
>>> inp = torch.tensor([[ 1.3053, -0.6421, -1.2027,  1.0494, -1.9540],
...                     [ 0.6801,  0.2266,  0.8120,  0.9490, -0.8120],
...                     [-0.8212,  0.8024,  0.8370, -0.3272,  0.6125],
...                     [-0.1975, -1.0706,  2.6819, -0.4297,  0.1980],
...                     [ 0.8256,  1.7839, -1.5876,  1.7705, -1.7051],
...                     [ 0.1288,  1.0981,  0.0570, -1.1684,  0.4567],
...                     [ 0.5658,  1.3948, -1.1457, -0.5921, -0.8026],
...                     [-0.3989,  0.6574,  0.3411, -1.9814,  0.2935]])
>>>
>>> targets = torch.tensor([0, 1, 4, 2, 3, 0, 2, 2])
>>> loss = MulticlassFocalLoss()
>>> loss
MulticlassFocalLoss(alpha=1.0, gamma=2.0, mask_target_value=-1, reduction='mean')

>>> loss(inp, targets)
tensor(0.9478)

pass class weights for the alpha value:

>>> weights = torch.tensor([1.0, 2.0, 0.3, 2.1, .5])
>>> loss = MulticlassFocalLoss(alpha=weights)
>>> loss(inp, targets)
tensor(0.8010)

supports masking by value for instance if we want to predict classes over time and the last two timesteps for the last batch are missing we can mask them by setting to mask_target_value to -1 and padding targets with -1:

>>> masked_targets =  torch.tensor([[0, 1, 4, 2], [3, 0, -1, -1]]) # (batch, time)
>>> inp = inp.reshape((2, 4, -1)) # (batch, time, classes)
>>> loss(inp, masked_targets)
tensor(0.8885)

when reduction is 'none' we will return one loss value per step as you can see the masked targets are 0.0:

>>> loss = MulticlassFocalLoss(alpha=weights, reduction='none')
>>> loss(inp, masked_targets)
tensor([[1.8430e-01, 2.7830e+00, 4.0199e-01, 1.6719e-03],
        [6.7119e-01, 1.2889e+00, 0.0, 0.0]])
property alpha
forward(inputs, targets)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Return type

Tensor

extra_repr()

Set the extra representation of the module

To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable.

Return type

str

SphereEmbeddingLoss

class SphereEmbeddingLoss(embedding_features, n_classes, scale=64.0, margin=1.35, **kwargs)

AKA SphereFaceLoss.

Reference

Liu et al. : SphereFace: Deep Hypersphere Embedding for Face Recognition

Note

  • this loss has weights and requires an optimizer.

Parameters

  • embedding_features (int) – number of embedding features inputs are expected to be (batch, embedding_features).

  • n_classes (int) – number of classes in projection.

  • scale (float) – Defaults to 64.0.

  • margin (float) – Defaults to 1.35.

  • eps – epsilon for clamping . Defaults to 1e-7.

  • reduction – batch reduction for this loss should be one of ‘mean’, ‘sum’, ‘none’. Defaults to ‘mean’.

Example

>>> import torch
>>> from hearth.losses import SphereEmbeddingLoss
>>> _ = torch.manual_seed(666)
>>>
>>>
>>> # this would be embeddings coming out of your model...
>>> emb = torch.normal(0, 1, size=(5, 128))
>>> targets = torch.randint(0, 10, size=(5,))
>>> loss = SphereEmbeddingLoss(128, 10)
>>> loss(emb, targets)
tensor(44.7385, grad_fn=<MeanBackward0>)