Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ Jacobians (generally of the losses with respect to the parameters). Its interfac
to that of [`torch.autograd`](https://docs.pytorch.org/docs/stable/autograd):
[`autojac.jac`](https://torchjd.org/stable/docs/autojac/jac) is analog to
[`autograd.grad`](https://docs.pytorch.org/docs/stable/generated/torch.autograd.grad) but
returns Jacobians insteads of gradients, and
returns Jacobians instead of gradients, and
[`autojac.backward`](https://torchjd.org/stable/docs/autojac/backward) is analog to
[`autograd.backward`](https://docs.pytorch.org/docs/stable/generated/torch.autograd.backward)
but accumulates Jacobians in the `.jac` fields of parameters instead of gradients in the `.grad`
Expand Down
2 changes: 1 addition & 1 deletion skills/research-method/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: Researches about a new method (scalarizer or aggregator) from the s

# Research new method

This skill researches about a new method by reading the paper and the existing implementations, proposing a plan for integrating an implmentation in TorchJD.
This skill researches about a new method by reading the paper and the existing implementations, proposing a plan for integrating an implementation in TorchJD.

**For agents:** invoke as `/implement-method method-name (paper-name)` (e.g. `/implement-method upgrad (Jacobian Descent for Multi-objective Optimization)`).
If no method name is provided, ask the user for the name of the method and the title of the paper.
Expand Down
4 changes: 2 additions & 2 deletions src/torchjd/_linalg/_generalized_gramian.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ def reshape(gramian: PSDTensor, half_shape: list[int]) -> PSDTensor:

:param gramian: Gramian to reshape. Can be a generalized Gramian.
:param half_shape: First half of the target shape, the shape of the output is therefore
`shape + shape[::-1]`.
`half_shape + half_shape[::-1]`.
"""

# Example 1: `gramian` of shape [4, 3, 2, 2, 3, 4] and `half_shape` of [8, 3]:
Expand Down Expand Up @@ -59,7 +59,7 @@ def movedim(gramian: PSDTensor, half_source: list[int], half_destination: list[i
must be done simultaneously on the first half of the dimensions and on the second half of the
dimensions reversed.

:param gramian: Gramian to reshape. Can be a generalized Gramian.
:param gramian: Gramian whose dimensions should be moved. Can be a generalized Gramian.
:param half_source: Source dimensions, that should be in the range [-gramian.ndim//2,
gramian.ndim//2[. Its elements should be unique.
:param half_destination: Destination dimensions, that should be in the range
Expand Down
2 changes: 1 addition & 1 deletion src/torchjd/aggregation/_aligned_mtl.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ class AlignedMTL(GramianWeightedAggregator):
`Independent Component Alignment for Multi-Task Learning
<https://openaccess.thecvf.com/content/CVPR2023/papers/Senushkin_Independent_Component_Alignment_for_Multi-Task_Learning_CVPR_2023_paper.pdf>`_.

:param pref_vector: The preference vector to use. If not provided, defaults to
:param pref_vector: The preference vector to use. If not provided, defaults to
:math:`\begin{bmatrix} \frac{1}{m} & \dots & \frac{1}{m} \end{bmatrix}^T \in \mathbb{R}^m`.
:param scale_mode: The scaling mode used to build the balance transformation. ``"min"`` uses
the smallest eigenvalue (default), ``"median"`` uses the median eigenvalue, and ``"rmse"``
Expand Down
2 changes: 1 addition & 1 deletion src/torchjd/aggregation/_cr_mogm.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ class CRMOGMWeighting(Weighting[_T], Stateful):
Creating the corresponding :class:`~torchjd.aggregation.Aggregator` from a wrapped weighting can
be done by composing it with the appropriate aggregator subclass
(:class:`~torchjd.aggregation.WeightedAggregator` or
:class:`~torchjd.aggregation.GramianWeightedAggregator`)
:class:`~torchjd.aggregation.GramianWeightedAggregator`).

The following example shows how to instantiate a Gramian-based weighted aggregator whose
Gramian weighting is wrapped by CR-MOGM.
Expand Down
2 changes: 1 addition & 1 deletion src/torchjd/aggregation/_fairgrad.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ class FairGradWeighting(_WithOptionalDeps, _GramianWeighting, _NonDifferentiable
"""

_REQUIRED_DEPS = ["numpy", "scipy"]
_INSTALL_HINT = 'Install it with: pip install "torchjd[fairgrad]"'
_INSTALL_HINT = 'Install them with: pip install "torchjd[fairgrad]"'

def __init__(self, alpha: float, max_iters: int | None = None) -> None:
super().__init__()
Expand Down
3 changes: 2 additions & 1 deletion src/torchjd/aggregation/_sdmgrad.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@ class SDMGradWeighting(_MatrixWeighting, Stateful, _NonDifferentiable):
:param lambda_: Non-negative coefficient controlling how strongly the descent direction is pulled
toward the preference direction. Must be non-negative.
:param pref_vector: The preference vector :math:`\tilde w` defining the target direction. If not
provided, defaults to the uniform vector :math:`[1/m, \ldots, 1/m]` (i.e. the target diection is the average gradient).
provided, defaults to the uniform vector :math:`[1/m, \ldots, 1/m]` (i.e. the target
direction is the average gradient).

.. note::
The inner simplex-projected solver is adapted from the `official implementation
Expand Down
2 changes: 1 addition & 1 deletion src/torchjd/aggregation/_sum.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ def forward(self, matrix: Tensor, /) -> Tensor:

class Sum(WeightedAggregator):
"""
:class:`~torchjd.aggregation.WeightedAggregator` that sums of the rows of the input
:class:`~torchjd.aggregation.WeightedAggregator` that sums the rows of the input
matrices.
"""

Expand Down
2 changes: 1 addition & 1 deletion src/torchjd/aggregation/_upgrad.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ class UPGrad(GramianWeightedAggregator, _NonDifferentiable):
matrix onto the dual cone of all rows of this matrix, and that combines the result, as proposed
in `Jacobian Descent For Multi-Objective Optimization <https://arxiv.org/pdf/2406.16232>`_.

:param pref_vector: The preference vector used to combine the projected rows. If not provided,
:param pref_vector: The preference vector used to combine the projected rows. If not provided,
defaults to :math:`\begin{bmatrix} \frac{1}{m} & \dots & \frac{1}{m} \end{bmatrix}^T \in
\mathbb{R}^m`.
:param projector: The :class:`~torchjd.linalg.DualConeProjector` used to compute the projection.
Expand Down
4 changes: 2 additions & 2 deletions src/torchjd/autogram/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@
tensor of outputs (generally losses) with respect to some modules' parameters. This Gramian contains
all the inner products between pairs of gradients, and is thus a sufficient statistic for most
weighting methods. The algorithm is formally defined in Section 6 of `Jacobian Descent For
Multi-Objective Optimization <https://arxiv.org/pdf/2406.16232>`_).
Multi-Objective Optimization <https://arxiv.org/pdf/2406.16232>`_.

Due to computing the Gramian iteratively over the layers, without ever having to store the full
Jacobian in memory, this method is much more memory-efficient than
:doc:`autojac <../autojac/index>`, which makes it often much faster. Note that we're still working
on making autogram faster and more memory-efficient, and it's interface may change in future
on making autogram faster and more memory-efficient, and its interface may change in future
releases.

The list of Weightings compatible with ``autogram`` is:
Expand Down
4 changes: 2 additions & 2 deletions src/torchjd/autogram/_engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,8 @@ class Engine:
<https://docs.pytorch.org/docs/stable/generated/torch.nn.Transformer.html>`_, which use a
dropout function (rather than a `Dropout
<https://docs.pytorch.org/docs/stable/generated/torch.nn.Dropout.html>`_ layer) in a
module with some trainable parameters, has to be used with
``dropout=0.0``. Note that a `Dropout
module with some trainable parameters, have to be used with
``dropout=0.0``. Note that `Dropout
<https://docs.pytorch.org/docs/stable/generated/torch.nn.Dropout.html>`_ layers are
entirely supported and should be preferred. It is also perfectly fine for random modules
to have child modules that have trainable parameters, so if you have a random module with
Expand Down
4 changes: 2 additions & 2 deletions src/torchjd/autogram/_jacobian_computer.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@

class JacobianComputer(ABC):
"""
Abstract class to computes Jacobians for a module's forward pass with respect to its parameters.
Abstract class to compute Jacobians for a module's forward pass with respect to its parameters.

:params module: The module to differentiate.
:param module: The module to differentiate.
"""

def __init__(self, module: nn.Module) -> None:
Expand Down
2 changes: 1 addition & 1 deletion src/torchjd/autogram/_module_hook_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ def hook_module(self, module: nn.Module, gramian_computer: GramianComputer) -> N
"""
Add a module hook used to insert Jacobian accumulation nodes into the backward graph.

The hook injects a AutogramNode function into the computation graph after the module,
The hook injects an AutogramNode function into the computation graph after the module,
enabling Gramian computation.
"""

Expand Down
2 changes: 1 addition & 1 deletion src/torchjd/autojac/_accumulation.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ def is_tensor_with_jac(t: Tensor) -> TypeGuard[TensorWithJac]:
def accumulate_jacs(params: Iterable[Tensor], jacobians: Iterable[Tensor]) -> None:
for param, jac in zip(params, jacobians, strict=True):
_check_expects_grad(param, field_name=".jac")
# We that the shape is correct to be consistent with torch, that checks that the grad
# We check that the shape is correct to be consistent with torch, that checks that the grad
# shape is correct before assigning it.
if jac.shape[1:] != param.shape:
raise RuntimeError(
Expand Down
2 changes: 1 addition & 1 deletion src/torchjd/autojac/_jac_to_grad.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ def jac_to_grad(
.. admonition::
Example

This example shows how to use ``jac_to_grad`` after a call to ``backward``
This example shows how to use ``jac_to_grad`` after a call to ``backward``.

>>> import torch
>>>
Expand Down
2 changes: 1 addition & 1 deletion src/torchjd/autojac/_transform/_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@


class RequirementError(ValueError):
"""Inappropriate set of inputs keys."""
"""Inappropriate set of input keys."""


class Transform(ABC):
Expand Down
2 changes: 1 addition & 1 deletion src/torchjd/linalg/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
"""
This module provides utilitary linear algebra methods as well as types to represent specific
This module provides utility linear algebra methods as well as types to represent specific
structural properties.
"""

Expand Down
3 changes: 2 additions & 1 deletion src/torchjd/scalarization/_uw.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,8 @@ class UW(Scalarizer, Stateful):
:param shape: The shape of the values to scalarize, used to create one log-variance per value.
An ``int`` ``n`` is interpreted as the shape ``(n,)``.

The following example shows train a model with Uncertainty Weighting, as described in the paper.
The following example shows how to train a model with Uncertainty Weighting, as described in
the paper.

>>> import torch
>>> from torch.nn import Linear
Expand Down
Loading