fusionlab.nn.components.MultiModalEmbedding

class fusionlab.nn.components.MultiModalEmbedding

Bases: Layer, NNLearner

MultiModalEmbedding layer for embedding multiple input modalities into a common feature space and concatenating them along the last dimension.

This layer takes a list of tensors, each representing a different modality with the same batch and time dimensions. It applies a dense projection (with activation) to each modality, converting them to the same dimensionality before concatenation.

\[\mathbf{H}_{out} = \text{Concat}\big( \text{Dense}(\mathbf{M_1}),\, \text{Dense}(\mathbf{M_2}),\,\dots\big)\]

where each \(\mathbf{M_i}\) is a tensor for a specific modality.

Parameters:

embed_dim (int) – Dimensionality of the output embedding for each modality.

Notes

This layer expects each input modality tensor to have the same batch and time dimensions, but potentially different feature dimensions.

call(`inputs`, training=False)

Forward pass that projects each modality separately, then concatenates.

get_config()

Returns a configuration dictionary for serialization.

from_config(`config`)

Recreates the layer from a config dict.

Examples

>>> from fusionlab.nn.components import MultiModalEmbedding
>>> import tensorflow as tf
>>> # Suppose we have two modalities:
... #   dynamic_modality  : (batch, time, dyn_dim)
... #   future_modality   : (batch, time, fut_dim)
>>> dyn_input = tf.random.normal((32, 10, 16))
>>> fut_input = tf.random.normal((32, 10, 8))
>>> # Instantiate the layer
>>> mm_embed = MultiModalEmbedding(embed_dim=32)
>>> # Forward pass with both modalities
>>> outputs = mm_embed([dyn_input, fut_input])

See also

LearnedNormalization

Normalizes input features before embedding.

HierarchicalAttention

Another specialized layer that can be used after embeddings are computed.

__init__(embed_dim)

Parameters:

embed_dim (int)

Methods

__init__(embed_dim)
add_loss(loss)	Can be called inside of the call() method to add a scalar loss.
add_metric(*args, **kwargs)
add_variable(shape, initializer[, dtype, ...])	Add a weight variable to the layer.
add_weight([shape, initializer, dtype, ...])	Add a weight variable to the layer.
build(input_shape)	Build method that creates a Dense layer for each modality based on input_shape.
build_from_config(config)	Builds the layer's states with the supplied config dict.
call(inputs[, training])	Forward pass: project each modality into embed_dim and concatenate.
compute_mask(inputs, previous_mask)
compute_output_shape(*args, **kwargs)
compute_output_spec(*args, **kwargs)
count_params()	Count the total number of scalars composing the weights.
from_config(config)	Recreates a MultiModalEmbedding layer from a config dictionary.
get_build_config()	Returns a dictionary with the layer's input shape.
get_config()	Returns the configuration dictionary of this layer.
get_params([deep])	Get the parameters for this learner.
get_weights()	Return the values of layer.weights as a list of NumPy arrays.
help(**kwargs)
load(file_path[, format])	Load the learner's state from a specified file in the desired format.
load_own_variables(store)	Loads the state of the layer.
quantize(mode[, type_check, config])
quantized_build(input_shape, mode)
quantized_call(*args, **kwargs)
rematerialized_call(layer_call, *args, **kwargs)	Enable rematerialization dynamically for layer's call method.
save([file_path, format, overwrite, ...])	Save the learner's state to a specified file in the desired format.
save_own_variables(store)	Saves the state of the layer.
set_params(**params)	Set the parameters of this learner.
set_weights(weights)	Sets the values of layer.weights from a list of NumPy arrays.
stateless_call(trainable_variables, ...[, ...])	Call the layer without any side effects.
summary()	Provide a summary of the learner's parameters.
symbolic_call(*args, **kwargs)

Attributes

compute_dtype	The dtype of the computations performed by the layer.
dtype	Alias of layer.variable_dtype.
dtype_policy
input	Retrieves the input tensor(s) of a symbolic operation.
input_dtype	The dtype layer inputs should be converted to.
input_spec
losses	List of scalar losses from add_loss, regularizers and sublayers.
metrics	List of all metrics.
metrics_variables	List of all metric variables.
my_params
non_trainable_variables	List of all non-trainable layer state.
non_trainable_weights	List of all non-trainable weight variables of the layer.
output	Retrieves the output tensor(s) of a layer.
path	The path of the layer.
quantization_mode	The quantization mode of this layer, None if not quantized.
supports_masking	Whether this layer supports computing a mask using compute_mask.
trainable	Settable boolean, whether this layer should be trainable or not.
trainable_variables	List of all trainable layer state.
trainable_weights	List of all trainable weight variables of the layer.
variable_dtype	The dtype of the state (weights) of the layer.
variables	List of all layer state, including random seeds.
weights	List of all weight variables of the layer.

__init__(embed_dim)

Parameters:

embed_dim (int)

build(input_shape)

Build method that creates a Dense layer for each modality based on input_shape.

Parameters:

input_shape (list of tuples) – Each tuple corresponds to a modality’s shape, typically (batch_size, time_steps, feature_dim).

help(**kwargs)

my_params = MultiModalEmbedding(embed_dim)

call(inputs, training=False)

Forward pass: project each modality into embed_dim and concatenate.

Parameters:

• inputs (list of tf.Tensor) – Each tensor has shape \((B, T, D_i)\) where D_i can vary by modality.

• training (bool, optional) – Indicates if the layer is in training mode. Defaults to False.

Returns:

A concatenated embedding of shape \((B, T, \sum_{i}(\text{embed_dim}))\).

Return type:

tf.Tensor

get_config()

Returns the configuration dictionary of this layer.

Returns:

Configuration including embed_dim.

Return type:

dict

classmethod from_config(config)

Recreates a MultiModalEmbedding layer from a config dictionary.

Parameters:

config (dict) – Configuration as produced by get_config.

Returns:

A new instance of this layer.

Return type:

MultiModalEmbedding