fusionlab.nn.components.PositionwiseFeedForward

class fusionlab.nn.components.PositionwiseFeedForward

Bases: Layer, NNLearner

Implements the Position-wise Feed-Forward Network (FFN) layer.

This layer is a core component of a standard Transformer block, typically applied after the multi-head attention sub-layer. Its purpose is to process the context-rich output from the attention mechanism at each position independently, adding non-linearity and transformative capacity to the model.

The network consists of two fully-connected (Dense) layers with a non-linear activation function in between. The first layer expands the input dimensionality, and the second layer projects it back down.

Parameters:

• embed_dim (int) – The input and output dimensionality of the layer. This must match the embedding dimension of the Transformer, often denoted as \(d_{model}\).

• ffn_dim (int) – The dimensionality of the inner, expanded hidden layer. It is common practice in Transformer architectures to set this to four times the embed_dim.

• activation (str, optional) – The activation function to use in the inner layer. Any valid Keras activation string is accepted. Defaults to "relu".

• dropout_rate (float, optional) – The dropout rate applied for regularization, typically after the first activation function. Defaults to 0.1.

• **kwargs – Standard keyword arguments for a Keras Layer.

Notes

The “position-wise” nature of this layer is its defining characteristic. The same instance of this layer, with the exact same set of learned weights (\(W_1, b_1, W_2, b_2\)), is applied to the feature vector at every single position (e.g., time step) in the input sequence. It does not mix information between positions; that task is handled by the preceding self-attention layer.

The mathematical operation for a single position vector \(x\) is:

\[ext{FFN}(x) = ext{Linear}_2( ext{activation}( ext{Linear}_1(x)))\]

The residual connection (\(x + ext{Dropout}( ext{FFN}(x))\)) is typically applied outside this layer, within the main Transformer block.

See also

fusionlab.nn.components.TransformerEncoderLayer

A typical consumer of this layer.

tf.keras.layers.Dense

The core building block of the FFN.

References

[1]

Vaswani, A., et al. “Attention Is All You Need.” NeurIPS 2017.

Examples

>>> import tensorflow as tf
>>> # Create a dummy input tensor (batch, sequence_length, embed_dim)
>>> input_tensor = tf.random.normal((32, 50, 128))
...
>>> # Instantiate the FFN layer
>>> ffn_layer = PositionwiseFeedForward(embed_dim=128, ffn_dim=512)
...
>>> # Pass the input through the layer
>>> output_tensor = ffn_layer(input_tensor, training=True)
...
>>> # The output shape remains the same as the input shape
>>> print(f"Input Shape: {input_tensor.shape}")
>>> print(f"Output Shape: {output_tensor.shape}")
Input Shape: (32, 50, 128)
Output Shape: (32, 50, 128)

__init__(embed_dim, ffn_dim, activation='relu', dropout_rate=0.1, **kwargs)

Parameters:

• embed_dim (int)

• ffn_dim (int)

• activation (str)

• dropout_rate (float)

Methods

__init__(embed_dim, ffn_dim[, activation, ...])
add_loss(losses, **kwargs)	Add loss tensor(s), potentially dependent on layer inputs.
add_metric(value[, name])	Adds metric tensor to the layer.
add_update(updates)	Add update op(s), potentially dependent on layer inputs.
add_variable(*args, **kwargs)	Deprecated, do NOT use! Alias for add_weight.
add_weight([name, shape, dtype, ...])	Adds a new variable to the layer.
build(input_shape)	Creates the variables of the layer (for subclass implementers).
build_from_config(config)	Builds the layer's states with the supplied config dict.
call(x[, training])	Defines the forward pass for the FFN layer.
compute_mask(inputs[, mask])	Computes an output mask tensor.
compute_output_shape(input_shape)	Computes the output shape of the layer.
compute_output_signature(input_signature)	Compute the output tensor signature of the layer based on the inputs.
count_params()	Count the total number of scalars composing the weights.
finalize_state()	Finalizes the layers state after updating layer weights.
from_config(config)	Creates a layer from its config.
get_build_config()	Returns a dictionary with the layer's input shape.
get_config()	Returns the configuration of the layer for serialization.
get_input_at(node_index)	Retrieves the input tensor(s) of a layer at a given node.
get_input_mask_at(node_index)	Retrieves the input mask tensor(s) of a layer at a given node.
get_input_shape_at(node_index)	Retrieves the input shape(s) of a layer at a given node.
get_output_at(node_index)	Retrieves the output tensor(s) of a layer at a given node.
get_output_mask_at(node_index)	Retrieves the output mask tensor(s) of a layer at a given node.
get_output_shape_at(node_index)	Retrieves the output shape(s) of a layer at a given node.
get_params([deep])	Get the parameters for this learner.
get_weights()	Returns the current weights of the layer, as NumPy arrays.
help(**kwargs)
load(file_path[, format])	Load the learner's state from a specified file in the desired format.
load_own_variables(store)	Loads the state of the layer.
save([file_path, format, overwrite, ...])	Save the learner's state to a specified file in the desired format.
save_own_variables(store)	Saves the state of the layer.
set_params(**params)	Set the parameters of this learner.
set_weights(weights)	Sets the weights of the layer, from NumPy arrays.
summary()	Provide a summary of the learner's parameters.
with_name_scope(method)	Decorator to automatically enter the module name scope.

Attributes

activity_regularizer	Optional regularizer function for the output of this layer.
compute_dtype	The dtype of the layer's computations.
dtype	The dtype of the layer weights.
dtype_policy	The dtype policy associated with this layer.
dynamic	Whether the layer is dynamic (eager-only); set in the constructor.
inbound_nodes	Return Functional API nodes upstream of this layer.
input	Retrieves the input tensor(s) of a layer.
input_mask	Retrieves the input mask tensor(s) of a layer.
input_shape	Retrieves the input shape(s) of a layer.
input_spec	InputSpec instance(s) describing the input format for this layer.
losses	List of losses added using the add_loss() API.
metrics	List of metrics attached to the layer.
my_params
name	Name of the layer (string), set in the constructor.
name_scope	Returns a tf.name_scope instance for this class.
non_trainable_variables	Sequence of non-trainable variables owned by this module and its submodules.
non_trainable_weights	List of all non-trainable weights tracked by this layer.
outbound_nodes	Return Functional API nodes downstream of this layer.
output	Retrieves the output tensor(s) of a layer.
output_mask	Retrieves the output mask tensor(s) of a layer.
output_shape	Retrieves the output shape(s) of a layer.
stateful
submodules	Sequence of all sub-modules.
supports_masking	Whether this layer supports computing a mask using compute_mask.
trainable
trainable_variables	Sequence of trainable variables owned by this module and its submodules.
trainable_weights	List of all trainable weights tracked by this layer.
updates
variable_dtype	Alias of Layer.dtype, the dtype of the weights.
variables	Returns the list of all layer variables/weights.
weights	Returns the list of all layer variables/weights.

__init__(embed_dim, ffn_dim, activation='relu', dropout_rate=0.1, **kwargs)

Parameters:

• embed_dim (int)

• ffn_dim (int)

• activation (str)

• dropout_rate (float)

call(x, training=False)

Defines the forward pass for the FFN layer.

Parameters:

• x (Tensor)

• training (bool)

Return type:

Tensor

get_config()

Returns the configuration of the layer for serialization.

help(**kwargs)

my_params = PositionwiseFeedForward(embed_dim, ffn_dim, activation='relu', dropout_rate=0.1)