Implements positional encoding as commonly used in transformer architectures.
Positional encoding introduces a way to inject information about the order of the input data (e.g., sequence positions) into a neural network that otherwise lacks a sense of position due to its permutation-invariant nature. This class computes sinusoidal encodings based on the position of each element in the input and concatenates them with the original input features.
Attributes
Name
Type
Description
frequencies
torch.Tensor
A tensor containing the frequencies used to calculate the sinusoidal components. The frequencies are powers of 2, scaled by the base frequency.
encode_all
bool
Determines whether the positional encoding is applied to all features of the input or only the first feature (e.g., time component).
base_freq
float
The base frequency used to scale the sinusoidal components, defaulting to 2 * pi.
Parameters
Name
Type
Description
Default
nr_frequencies
int
The number of sinusoidal frequencies to compute. This determines the dimensionality of the positional encoding for each input feature.
required
encode_all
(bool, optional(default=True))
If True, the positional encoding is computed for all features in the input. Otherwise, it is computed only for the first feature (e.g., the time dimension).
Computes and concatenates positional encodings with the input tensor.
Parameters
Name
Type
Description
Default
t_theta
torch.Tensor
Input tensor of shape (batch_size, input_dim), where input_dim is the dimensionality of the input features.
required
Returns
Name
Type
Description
torch.Tensor
A tensor containing the input features concatenated with the positional encodings. The output shape will be: - (batch_size, input_dim + 2 * nr_frequencies) if encode_all is True. - (batch_size, input_dim + 2 * nr_frequencies) if encode_all is False, but positional encodings are computed only for the first input feature.