rlightning.policy.utils.losses

rlightning.policy.utils.losses.compute_ppo_actor_critic_loss(**kwargs) → Tuple[MockModule('torch.Tensor'), Dict][source]

Compute PPO actor-critic loss.

Parameters:

**kwargs – Keyword arguments forwarded to actor and critic loss functions.

Returns:

Total loss and metrics dictionary.

Return type:

Tuple[torch.Tensor, Dict]

rlightning.policy.utils.losses.compute_ppo_actor_loss(logprobs: MockModule('torch.Tensor'), old_logprobs: MockModule('torch.Tensor'), clip_ratio_low: float, clip_ratio_high: float, advantages: MockModule('torch.Tensor'), loss_mask: MockModule('torch.Tensor') | None = None, c_clip: float | None = None, loss_agg_func: ~typing.Callable[[...], MockModule('torch.Tensor')] | None = <function masked_mean>, max_episode_steps: int | None = None, loss_mask_sum: MockModule('torch.Tensor') | None = None, critic_warmup: bool | None = False, **kwargs) → Tuple[MockModule('torch.Tensor'), Dict][source]

Compute PPO actor loss.

Parameters:

  • logprobs (torch.FloatTensor) – Log probabilities of actions.

  • old_logprobs (torch.FloatTensor) – Old log probabilities of actions.

  • clip_ratio_low (float) – Lower bound of clipping ratio.

  • clip_ratio_high (float) – Upper bound of clipping ratio.

  • advantages (torch.FloatTensor) – GAE (normalized) advantages.

  • loss_mask (Optional[torch.BoolTensor]) – Mask for valid entries.

  • c_clip (Optional[float]) – Optional clipping coefficient.

  • loss_agg_func (Optional[Callable[..., torch.Tensor]]) – Aggregation function.

  • max_episode_steps (Optional[int]) – Max episode length for normalization.

  • loss_mask_sum (Optional[torch.Tensor]) – Sum of mask values for normalization.

  • critic_warmup (Optional[bool]) – If True, zero out actor loss.

  • **kwargs – Unused extra keyword arguments.

Returns:

Actor loss and metrics dictionary.

Return type:

Tuple[torch.Tensor, Dict]

rlightning.policy.utils.losses.compute_ppo_critic_loss(values: MockModule('torch.Tensor'), returns: MockModule('torch.Tensor'), prev_values: MockModule('torch.Tensor'), value_clip: float, huber_delta: float, loss_mask: MockModule('torch.Tensor') | None = None, max_episode_steps: int | None = None, loss_mask_sum: MockModule('torch.Tensor') | None = None, **kwargs) → Tuple[MockModule('torch.Tensor'), Dict][source]

Compute PPO critic loss.

Parameters:

  • values (torch.Tensor) – Current value predictions.

  • returns (torch.Tensor) – Return values.

  • prev_values (torch.Tensor) – Previous value predictions.

  • value_clip (float) – Value clipping threshold.

  • huber_delta (float) – Huber loss delta parameter.

  • loss_mask (Optional[torch.Tensor]) – Mask for valid entries.

  • max_episode_steps (Optional[int]) – Max episode length for normalization.

  • loss_mask_sum (Optional[torch.Tensor]) – Sum of mask values for normalization.

  • **kwargs – Unused extra keyword arguments.

Returns:

Critic loss and metrics dictionary.

Return type:

Tuple[torch.Tensor, Dict]