fusionlab.nn.components.CrossAttention

class fusionlab.nn.components.CrossAttention

Bases: Layer, NNLearner

CrossAttention layer that attends one source sequence to another [1].

This layer transforms two input sources, source1 and source2, into a shared dimensionality via separate dense layers, then applies multi-head attention using source1 as the query and source2 as both key and value. The output shape depends on the specified units.

\[\mathbf{H}_{\text{out}} = \text{MHA}( \mathbf{W}_{1}\,\mathbf{S}_1,\, \mathbf{W}_{2}\,\mathbf{S}_2,\, \mathbf{W}_{2}\,\mathbf{S}_2 )\]

where \(\mathbf{S}_1\) and \(\mathbf{S}_2\) are the two source sequences.

Parameters:

• units (int) – Dimensionality for the internal projections of the query/key/value in multi-head attention.

• num_heads (int) – Number of attention heads.

Notes

Cross attention is particularly useful when focusing on how one sequence (the query) relates to another (the key/value). For example, in multi-modal time series settings, one might attend dynamic covariates to static ones or vice versa.

call(`inputs`, training=False)

Forward pass of the cross-attention layer.

get_config()

Returns the configuration dictionary for serialization.

from_config(`config`)

Creates a new layer from the given config.

Examples

>>> from fusionlab.nn.components import CrossAttention
>>> import tensorflow as tf
>>> # Two sequences of shape (batch_size, time_steps, features)
>>> source1 = tf.random.normal((32, 10, 64))
>>> source2 = tf.random.normal((32, 10, 64))
>>> # Instantiate the CrossAttention layer
>>> cross_attn = CrossAttention(units=64, num_heads=4)
>>> # Forward pass
>>> outputs = cross_attn([source1, source2])

See also

HierarchicalAttention

Another attention-based layer focusing on short/long-term sequences.

MemoryAugmentedAttention

Uses a learned memory matrix to enhance representations.

References

[1]

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). “Attention is all you need.” In Advances in Neural Information Processing Systems (pp. 5998-6008).

__init__(units, num_heads)

Initialize the CrossAttention layer.

Parameters:

• units (int) – Number of output units for the internal Dense projections and multi-head attention dimension.

• num_heads (int) – Number of attention heads to use in the multi-head attention module.

Methods

__init__(units, num_heads)	Initialize the CrossAttention layer.
add_loss(losses, **kwargs)	Add loss tensor(s), potentially dependent on layer inputs.
add_metric(value[, name])	Adds metric tensor to the layer.
add_update(updates)	Add update op(s), potentially dependent on layer inputs.
add_variable(*args, **kwargs)	Deprecated, do NOT use! Alias for add_weight.
add_weight([name, shape, dtype, ...])	Adds a new variable to the layer.
build(input_shape)	Creates the variables of the layer (for subclass implementers).
build_from_config(config)	Builds the layer's states with the supplied config dict.
call(inputs[, training])	Forward pass of CrossAttention.
compute_mask(inputs[, mask])	Computes an output mask tensor.
compute_output_shape(input_shape)	Computes the output shape of the layer.
compute_output_signature(input_signature)	Compute the output tensor signature of the layer based on the inputs.
count_params()	Count the total number of scalars composing the weights.
finalize_state()	Finalizes the layers state after updating layer weights.
from_config(config)	Create a new CrossAttention layer from the given config dictionary.
get_build_config()	Returns a dictionary with the layer's input shape.
get_config()	Returns configuration dictionary for this layer.
get_input_at(node_index)	Retrieves the input tensor(s) of a layer at a given node.
get_input_mask_at(node_index)	Retrieves the input mask tensor(s) of a layer at a given node.
get_input_shape_at(node_index)	Retrieves the input shape(s) of a layer at a given node.
get_output_at(node_index)	Retrieves the output tensor(s) of a layer at a given node.
get_output_mask_at(node_index)	Retrieves the output mask tensor(s) of a layer at a given node.
get_output_shape_at(node_index)	Retrieves the output shape(s) of a layer at a given node.
get_params([deep])	Get the parameters for this learner.
get_weights()	Returns the current weights of the layer, as NumPy arrays.
help(**kwargs)
load(file_path[, format])	Load the learner's state from a specified file in the desired format.
load_own_variables(store)	Loads the state of the layer.
save([file_path, format, overwrite, ...])	Save the learner's state to a specified file in the desired format.
save_own_variables(store)	Saves the state of the layer.
set_params(**params)	Set the parameters of this learner.
set_weights(weights)	Sets the weights of the layer, from NumPy arrays.
summary()	Provide a summary of the learner's parameters.
with_name_scope(method)	Decorator to automatically enter the module name scope.

Attributes

activity_regularizer	Optional regularizer function for the output of this layer.
compute_dtype	The dtype of the layer's computations.
dtype	The dtype of the layer weights.
dtype_policy	The dtype policy associated with this layer.
dynamic	Whether the layer is dynamic (eager-only); set in the constructor.
inbound_nodes	Return Functional API nodes upstream of this layer.
input	Retrieves the input tensor(s) of a layer.
input_mask	Retrieves the input mask tensor(s) of a layer.
input_shape	Retrieves the input shape(s) of a layer.
input_spec	InputSpec instance(s) describing the input format for this layer.
losses	List of losses added using the add_loss() API.
metrics	List of metrics attached to the layer.
my_params
name	Name of the layer (string), set in the constructor.
name_scope	Returns a tf.name_scope instance for this class.
non_trainable_variables	Sequence of non-trainable variables owned by this module and its submodules.
non_trainable_weights	List of all non-trainable weights tracked by this layer.
outbound_nodes	Return Functional API nodes downstream of this layer.
output	Retrieves the output tensor(s) of a layer.
output_mask	Retrieves the output mask tensor(s) of a layer.
output_shape	Retrieves the output shape(s) of a layer.
stateful
submodules	Sequence of all sub-modules.
supports_masking	Whether this layer supports computing a mask using compute_mask.
trainable
trainable_variables	Sequence of trainable variables owned by this module and its submodules.
trainable_weights	List of all trainable weights tracked by this layer.
updates
variable_dtype	Alias of Layer.dtype, the dtype of the weights.
variables	Returns the list of all layer variables/weights.
weights	Returns the list of all layer variables/weights.

__init__(units, num_heads)

Initialize the CrossAttention layer.

Parameters:

• units (int) – Number of output units for the internal Dense projections and multi-head attention dimension.

• num_heads (int) – Number of attention heads to use in the multi-head attention module.

call(inputs, training=False)

Forward pass of CrossAttention.

Parameters:

• inputs (list of tf.Tensor) – A list [source1, source2], each of shape (batch_size, time_steps, features).

• training (bool, optional) – Indicates if the layer is in training mode (for dropout, if any). Defaults to False.

Returns:

A tensor of shape (batch_size, time_steps, units) representing cross-attended features.

Return type:

tf.Tensor

get_config()

Returns configuration dictionary for this layer.

Returns:

Configuration dictionary, including ‘units’.

Return type:

dict

classmethod from_config(config)

Create a new CrossAttention layer from the given config dictionary.

Parameters:

config (dict) – Configuration as returned by get_config.

Returns:

A new instance of CrossAttention.

Return type:

CrossAttention

help(**kwargs)

my_params = CrossAttention(units, num_heads)