Custom Activation Functions

This module defines custom activation functions used in neural networks.

Description

CReLU (Concatenated ReLU) [1] applies ReLU activation to both the input and its negation, then concatenates the results along the last dimension:

\[\text{CReLU}(x) = \text{ReLU}([x, -x])\]

This doubles the number of features and enhances the representational capacity of the network.

Classes

class objectrl.utils.custom_act.CReLU

Bases: Module

Concatenated ReLU (CReLU) activation module.

This module implements the CReLU activation function, which concatenates the ReLU activations of both the input and its negation along the last dimension:

CReLU(x) = ReLU([x, -x])

This increases the representational capacity of the model by doubling the number of features while preserving non-linearity.

__init__() → None

Initialize the CReLU module. :param None:

Returns:

None

forward(x: Tensor) → Tensor

Forward pass through the CReLU activation. :param x: Input tensor of shape (…, features) :type x: torch.Tensor

Returns:

Output tensor with ReLU applied to both x and -x,

concatenated along the last dimension (shape: (…, 2 * features))

Return type:

torch.Tensor

References

[1]

Abbas Z. et al., (2023), Loss of Plasticity in Continual Deep Reinforcement Learning