Critic Networks

Critic Networks#

This module defines various critic (value function) network architectures used in reinforcement learning, including deterministic, probabilistic, and Bayesian variants.

Descriptions#

  • CriticNet: A basic deterministic Q-network estimating expected returns.

  • ValueNet: A simplified Q-network for value estimation, ignoring action input.

  • CriticNetProbabilistic: Outputs both mean and uncertainty in Q-values.

  • BNNCriticNet: A Bayesian Q-network using the local reparameterization trick.

  • EMstyle: An encoder network for expectation-maximization-style latent modeling.

  • DQNNet: A Q-network architecture designed for discrete action spaces (e.g., DQN).

  • QuantileCriticNet: A quantile regression-based critic network for distributional RL.

Classes#

class objectrl.nets.critic_nets.CriticNet(dim_state: int, dim_act: int, depth: int = 3, width: int = 256, act: Literal['relu', 'crelu'] = 'relu', has_norm: bool = False)[source]#

Bases: Module

Deterministic Critic Network (Q-network). Estimates the expected return (Q-value) for a given state-action pair.

Parameters:

  • dim_state (int) – Dimension of observation space.

  • dim_act (int) – Dimension of action space.

  • depth (int) – Number of hidden layers.

  • width (int) – Width of each hidden layer.

  • act (str) – Activation function to use.

  • has_norm (bool) – Whether to include normalization layers.

__init__(dim_state: int, dim_act: int, depth: int = 3, width: int = 256, act: Literal['relu', 'crelu'] = 'relu', has_norm: bool = False) → None[source]#

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(x: Tensor) → Tensor[source]#

Forward pass of the critic network.

Parameters:

x (Tensor) – Concatenated observation and action tensor.

class objectrl.nets.critic_nets.ValueNet(dim_state: int, dim_act: int, depth: int = 3, width: int = 256, act: str = 'relu', has_norm: bool = False)[source]#

Bases: CriticNet

Value network for estimating V(s) without action input.

Inherits from CriticNet but ignores action dimensions by setting dim_act to 0. Suitable for use in value-based methods like PPO or baseline estimation.

Parameters:

  • dim_state (int) – Dimension of the input state.

  • dim_act (int) – Unused, kept for compatibility (should be 0).

  • depth (int) – Number of hidden layers in the network.

  • width (int) – Width (number of units) in each hidden layer.

  • act (str) – Activation function to use (“relu” or “crelu”).

  • has_norm (bool) – Whether to apply normalization (e.g., LayerNorm).

__init__(dim_state: int, dim_act: int, depth: int = 3, width: int = 256, act: str = 'relu', has_norm: bool = False) → None[source]#

Initialize internal Module state, shared by both nn.Module and ScriptModule.

class objectrl.nets.critic_nets.CriticNetProbabilistic(dim_state: int, dim_act: int, depth: int = 3, width: int = 256, act: Literal['relu', 'crelu'] = 'relu', has_norm: bool = False)[source]#

Bases: Module

Probabilistic Critic Network.

Parameters:

  • dim_state (int) – Observation space dimension.

  • dim_act (int) – Action space dimension.

  • depth (int) – Number of hidden layers.

  • width (int) – Width of each hidden layer.

  • act (str) – Activation function to use.

  • has_norm (bool) – Whether to use normalization layers.

__init__(dim_state: int, dim_act: int, depth: int = 3, width: int = 256, act: Literal['relu', 'crelu'] = 'relu', has_norm: bool = False) → None[source]#

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(x: Tensor) → Tensor[source]#

Forward pass of the probabilistic critic.

Parameters:

x (Tensor) – Concatenated observation and action tensor.

class objectrl.nets.critic_nets.BNNCriticNet(dim_state: int, dim_act: int, depth: int = 3, width: int = 256, act: Literal['relu', 'crelu'] = 'relu', has_norm: bool = False)[source]#

Bases: Module

A Bayesian Critic Network (Q-network).

Parameters:

  • dim_state (int) – Observation space dimension.

  • dim_act (int) – Action space dimension.

  • depth (int) – Number of hidden layers.

  • width (int) – Width of each hidden layer.

  • act (Literal["relu", "crelu"]) – Activation function to use.

  • has_norm (bool) – Whether to include normalization layers.

__init__(dim_state: int, dim_act: int, depth: int = 3, width: int = 256, act: Literal['relu', 'crelu'] = 'relu', has_norm: bool = False) → None[source]#

Initialize internal Module state, shared by both nn.Module and ScriptModule.

map(on: bool = True) → None[source]#

Switch maximum a posteriori mode on/off

forward(x: Tensor) → Tensor | tuple[Tensor, Tensor | None][source]#

Forward pass of the BNNCriticNet

class objectrl.nets.critic_nets.EMstyle(dim_state: int, dim_act: int, depth: int = 3, width: int = 256, act: Literal['relu', 'crelu'] = 'relu', has_norm: bool = False)[source]#

Bases: Module

Encoder network for EM-style models.

Parameters:

  • dim_state (int) – Observation space dimension.

  • dim_act (int) – Action space dimension.

  • depth (int) – Number of hidden layers.

  • width (int) – Hidden layer width and output dimensionality.

  • act (str) – Activation function to use.

  • has_norm (bool) – Whether to use normalization layers.

__init__(dim_state: int, dim_act: int, depth: int = 3, width: int = 256, act: Literal['relu', 'crelu'] = 'relu', has_norm: bool = False) → None[source]#

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(x: Tensor) → Tensor[source]#

Forward pass to produce latent feature encoding.

Parameters:

x (Tensor) – Concatenated input tensor.

class objectrl.nets.critic_nets.DQNNet(dim_state: int, dim_act: int, depth: int = 3, width: int = 256, act: str = 'relu', has_norm: bool = False)[source]#

Bases: Module

Deterministic Critic Network (Q-network).

Parameters:

  • dim_state (int) – Dimension of observation space.

  • dim_act (int) – Dimension of action space.

  • depth (int) – Number of hidden layers.

  • width (int) – Width of each hidden layer.

  • act (str) – Activation function to use.

  • has_norm (bool) – Whether to include normalization layers.

__init__(dim_state: int, dim_act: int, depth: int = 3, width: int = 256, act: str = 'relu', has_norm: bool = False) → None[source]#

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(x: Tensor) → Tensor[source]#

Forward pass of the critic network.

Parameters:

x (Tensor) – Concatenated observation and action tensor.

class objectrl.nets.critic_nets.QuantileCriticNet(dim_state: int, dim_act: int, depth: int, width: int = 256, act: Literal['relu', 'crelu'] = 'relu', has_norm: bool = True)[source]#

Bases: Module

Quantile Critic Network for Distributional RL. Estimates the quantile values for given state-action pairs.

Parameters:

  • dim_state (int) – Dimension of observation space.

  • dim_act (int) – Dimension of action space.

  • depth (int) – Number of hidden layers.

  • width (int) – Width of each hidden layer.

  • act (str) – Activation function to use.

  • has_norm (bool) – Whether to include normalization layers.

__init__(dim_state: int, dim_act: int, depth: int, width: int = 256, act: Literal['relu', 'crelu'] = 'relu', has_norm: bool = True) → None[source]#

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(xtau: tuple[Tensor, Tensor]) → Tensor[source]#

Forward pass of the critic network. :param xtau: Tuple containing concatenated observation-action tensor and quantile fractions. :type xtau: tuple[Tensor, Tensor]

Returns:

Estimated quantile values.

Return type:

Tensor

Contents