fusionlab.nn.components.PositionalEncoding

class fusionlab.nn.components.PositionalEncoding

Bases: Layer, NNLearner

Sinusoidal positional encoding (Transformer-style).

This layer adds a deterministic (non-trainable) sinusoidal table to the input sequence so the model can distinguish positions.

Key design goals (why this implementation looks “special”)

1. Graph-scope safety We build the table with NumPy inside build() and store it as a non-trainable Keras weight via add_weight(…).

   Why: tensors created with TF ops inside build() can end up attached to a temporary FuncGraph during tracing. Later, when the model is re-traced or used in a different graph context (fit, SavedModel, etc.), those tensors can become “out of scope” and crash. A Keras weight is a TF Variable and is safe across graphs.

2. Serialization safety across Keras 2 and Keras 3 Older saved checkpoints may have no positional_encoding weight (legacy versions stored a plain tensor attribute). This class tolerates that during load by overriding:

   • set_weights (Keras 2 / H5-like paths)

   • load_own_variables (Keras 3 object-based save paths)

   If the weight is missing, we keep the freshly initialized constant.

param max_length:

Maximum sequence length supported by the precomputed table.

type max_length:

int, default 2048

Notes

The output shape matches the input: (B, T, D). The training argument is accepted for API compatibility only.

References

Vaswani et al., 2017, “Attention is All You Need”.

__init__(max_length=2048, **kwargs)

Parameters:

max_length (int)

Methods

__init__([max_length])
add_loss(loss)	Can be called inside of the call() method to add a scalar loss.
add_metric(*args, **kwargs)
add_variable(shape, initializer[, dtype, ...])	Add a weight variable to the layer.
add_weight([shape, initializer, dtype, ...])	Add a weight variable to the layer.
build(input_shape)	Create the fixed sinusoidal table once.
build_from_config(config)	Builds the layer's states with the supplied config dict.
call(inputs[, training])	Add positional encoding to the input.
compute_mask(inputs, previous_mask)
compute_output_shape(*args, **kwargs)
compute_output_spec(*args, **kwargs)
count_params()	Count the total number of scalars composing the weights.
from_config(config)	Creates an operation from its config.
get_build_config()	Returns a dictionary with the layer's input shape.
get_config()	Keras serialization config.
get_params([deep])	Get the parameters for this learner.
get_weights()	Return the values of layer.weights as a list of NumPy arrays.
help(**kwargs)
load(file_path[, format])	Load the learner's state from a specified file in the desired format.
load_own_variables(store)	Keras 3 object-based loading hook.
quantize(mode[, type_check, config])
quantized_build(input_shape, mode)
quantized_call(*args, **kwargs)
rematerialized_call(layer_call, *args, **kwargs)	Enable rematerialization dynamically for layer's call method.
save([file_path, format, overwrite, ...])	Save the learner's state to a specified file in the desired format.
save_own_variables(store)	Saves the state of the layer.
set_params(**params)	Set the parameters of this learner.
set_weights(weights)	Keras 2 / H5-style loading hook.
stateless_call(trainable_variables, ...[, ...])	Call the layer without any side effects.
summary()	Provide a summary of the learner's parameters.
symbolic_call(*args, **kwargs)

Attributes

compute_dtype	The dtype of the computations performed by the layer.
dtype	Alias of layer.variable_dtype.
dtype_policy
input	Retrieves the input tensor(s) of a symbolic operation.
input_dtype	The dtype layer inputs should be converted to.
input_spec
losses	List of scalar losses from add_loss, regularizers and sublayers.
metrics	List of all metrics.
metrics_variables	List of all metric variables.
my_params
non_trainable_variables	List of all non-trainable layer state.
non_trainable_weights	List of all non-trainable weight variables of the layer.
output	Retrieves the output tensor(s) of a layer.
path	The path of the layer.
quantization_mode	The quantization mode of this layer, None if not quantized.
supports_masking	Whether this layer supports computing a mask using compute_mask.
trainable	Settable boolean, whether this layer should be trainable or not.
trainable_variables	List of all trainable layer state.
trainable_weights	List of all trainable weight variables of the layer.
variable_dtype	The dtype of the state (weights) of the layer.
variables	List of all layer state, including random seeds.
weights	List of all weight variables of the layer.

__init__(max_length=2048, **kwargs)

Parameters:

max_length (int)

build(input_shape)

Create the fixed sinusoidal table once.

input_shape is expected to be (B, T, D). We only require D (feature dimension) to build the table.

Parameters:

input_shape (TensorShape)

set_weights(weights)

Keras 2 / H5-style loading hook.

Legacy checkpoints may provide an EMPTY list for this layer (because old versions had no variables). If so, accept it and keep the newly-initialized constant weight.

load_own_variables(store)

Keras 3 object-based loading hook.

In Keras 3, variable loading may use an internal “store” dict. Legacy saves might not include ‘positional_encoding’. If missing, do nothing and keep the initialized constant.

call(inputs, training=False)

Add positional encoding to the input.

inputs: (B, T, D) returns: (B, T, D)

Parameters:

inputs (Tensor)

Return type:

Tensor

get_config()

Keras serialization config.

Keep config minimal and JSON-serializable.

Return type:

dict

help(**kwargs)

my_params = PositionalEncoding(max_length=2048)