Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 23 additions & 11 deletions docs/guide/execution/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,32 @@
Execution
=========

With a :class:`.Transformation` defined, the next step is to execute this.
The easiest way to run it is to use the :ref:`quickrun CLI tool <userguide_quickrun>`.
More advanced options are available through first considering the
:ref:`theory of the execution model<userguide_execution_theory>`
then :ref:`reading on the available Python functions<reference_execution>`.
Once you have planned your calculations, each :class:`.Transformation` needs to be executed.
Its simulations are run and the results analysed to give a free energy estimate.
**openfe** supports two approaches, which differ in who organises the work.
Both can be used from the command line or from Python.

:ref:`Running Transformations individually <userguide_quickrun>`
You run one Transformation at a time and decide where and when each one runs,
for example by submitting one cluster job per Transformation.
On the command line this is ``openfe quickrun``; in Python, :func:`.execute_DAG`.

:ref:`Task-based execution <userguide_task_based_execution>` (experimental feature)
**openfe** splits a whole :class:`.AlchemicalNetwork` into tasks and keeps track of which are ready to run.
You start one or more workers, and each one picks up the next available task:
``openfe run-task`` on the command line, or :class:`.Worker` in Python.

.. toctree::
execution_theory
quickrun_execution
task_based_execution
execution_theory

.. admonition:: How-to guides

* :doc:`/cookbook/dumping_transformation`
.. admonition:: Related How-To Guides

.. TODO: add the Running calculations how-tos (quickrun, repeats in parallel, SLURM,
resuming, NVIDIA MPS, task-based execution) to this box.
* :doc:`/how_to_guides/dumping_transformation`
* :doc:`/how_to_guides/run_quickrun`
* :doc:`/how_to_guides/resume_quickrun`
* :doc:`/how_to_guides/run_on_slurm`
* :doc:`/how_to_guides/run_repeats_in_parallel`
* :doc:`/how_to_guides/run_with_mps`
* :doc:`/how_to_guides/run_task_based`
217 changes: 32 additions & 185 deletions docs/guide/execution/quickrun_execution.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
.. _userguide_quickrun:

Execution with Quickrun
Execution with ``quickrun``
=======================

The planning and preparation of a campaign of alchemical simulations using ``openfe`` is intended to be achievable on a local workstation in a matter of minutes.
Expand All @@ -10,201 +10,48 @@ These serialized JSON files are the currency of executing a campaign of simulati

To read the ``Transformation`` information and execute the simulation, the command line interface provides the ``openfe quickrun`` command, the full details of which are given in :ref:`the CLI reference section<cli_quickrun>`.

Overview
--------

Basic quickrun usage
--------------------

The ``quickrun`` command takes in the ``Transformation`` information represented as JSON, then executes a simulation according to those specifications.
For example, the following command executes a simulation defined by ``transformation.json`` and produces a results file named ``results.json``.

::

> openfe quickrun transformation.json -d workdir/ -o workdir/results.json
``openfe quickrun`` executes one :class:`.Transformation`.
A Transformation contains everything needed to run a free energy calculation:
the two :class:`.ChemicalSystem`\s, the :class:`.Protocol` and, if the Protocol needs one,
an :class:`.AtomMapping` (see :ref:`alchemical_network_model`).

Comment thread
atravitz marked this conversation as resolved.
Resolved
The ``-d`` / ``--work-dir`` flag controls where working files (checkpoints, trajectory data, etc...) are written.
If it is omitted, the current directory will be used.
When quickrun starts, it creates the ``ProtocolDAG`` of the Transformation:
the setup, simulation and analysis tasks (``ProtocolUnit``\s) of every repeat (see :ref:`userguide_execution_theory`).
quickrun then executes these tasks one at a time, and each task writes its files to its own subdirectory of the work directory.
If a task fails, quickrun retries it up to two more times; if it still fails, quickrun stops.

The ``-o`` flag controls where the results file will be written.
If it is omitted, results are written to a file named ``<transformation_key>_results.json`` in the working directory, where ``<transformation_key>`` is a unique identifier.
When all tasks have finished, or one has failed, quickrun combines the task results into a :class:`.ProtocolResult`
and writes the free energy estimate and its uncertainty to the results file.

.. _userguide_quickrun_resume:
quickrun keeps track of which tasks have finished, so an interrupted run can be resumed with ``--resume``.
Finished tasks are skipped, and an interrupted simulation continues from its last checkpoint
(see :doc:`/how_to_guides/resume_quickrun`).

Resuming a halted job
Organizing a campaign
---------------------

When ``openfe quickrun`` starts, it saves a plan of the simulation to a cache file before execution begins:

.. code:: bash

<work-dir>/quickrun_cache/dag-cache-<key>.json

Where ``<key>`` is a unique identifier based on the ``-o`` file path and Transformation.
This cache is automatically removed once the job completes.

If a job is interrupted (e.g. due to a wall-time limit, node failure, or manual cancellation), you can resume the interrupted job by passing the ``--resume`` flag:

.. code:: bash

> openfe quickrun transformation.json -d workdir/ -o workdir/results.json --resume

The planned simulation cache will be used to identify where in the simulation process it left off and, if supported by the Transformation Protocol, how to resume.

.. note::

The same ``-d`` / ``--work-dir`` and ``-o`` flag arguments used in the
original run must be specified so that ``quickrun`` can locate the cache file.

If you pass ``--resume`` but no cache file is found (e.g. the job never started), the following warning is printed and a fresh execution begins.

.. code:: bash

openfe quickrun was run with --resume, but no cached results found at
<path-to-cache-file>. Starting new execution.

If the cache file is corrupted (e.g. due to an incomplete write at the moment of interruption), ``quickrun --resume`` will raise an error with instructions to rerun the simulation:

.. code:: bash

Recovery failed, please remove <work-dir>/quickrun_cache/dag-cache-<key>.json
before executing a new transformation simulation.

If you do not pass the ``--resume`` flag, the code will detect the partially complete transformation and prevent you from accidentally starting a duplicate run.
The following error will be raised:

.. code:: bash

Transformation has been started but is incomplete. Please remove
<work-dir>/quickrun_cache/dag-cache-<key>.json and rerun, or resume
execution using the ``--resume`` flag.


Executing within a job submission script
----------------------------------------

You may need to submit computational jobs to a queueing engine, such as Slurm.
The ``openfe quickrun`` command can be used within a submission script as follows:

.. code-block:: bash

#!/bin/bash

#SBATCH --job-name="openfe job"
#SBATCH --mem-per-cpu=2G

# activate an appropriate conda environment, or any "module load" commands required to
conda activate openfe_env

openfe quickrun transformation.json -d workdir/ -o workdir/results.json


Parallel execution of repeats with Quickrun
===========================================

Serial execution of multiple repeats of a transformation can be inefficient when simulation times are long.
Higher throughput can be achieved with parallel execution by running one repeat per HPC job.
Most protocols are set up to run three repeats in serial by default, but this can be changed by either:

1. Defining the protocol setting ``protocol_repeats`` - see the :ref:`protocol configuration guide <cookbook/choose_protocol.nblink>` for more details.
2. Using the ``openfe plan-rhfe-network`` (or ``plan-rbfe-network``) command line flag ``--n-protocol-repeats``.

Each transformation can then be executed multiple times via the ``openfe quickrun`` command to produce a set of repeats.
However, **you must use unique results files for each repeat to ensure they don't overwrite each other**.
We recommend using folders named ``results_x`` where x is 0-2 to store the repeated calculations as our :ref:`openfe gather <cli_gather>` command also supports this file structure.

Below is an example of a simple script that will create and submit a separate job script (``\*.job`` named file) for every alchemical transformation (for the simplest SLURM use case) in a network running each repeat in parallel and writing the results to a unique folder:
You decide where and when each Transformation runs, for example by
:doc:`submitting one job per Transformation to a cluster </how_to_guides/run_on_slurm>`.
You can run multiple simulations in parallel by running several quickrun calls at the same time,
for example one per Transformation, :doc:`one per repeat </how_to_guides/run_repeats_in_parallel>`,
or :doc:`several on the same GPU </how_to_guides/run_with_mps>`.
The results files are then collected with :ref:`openfe gather <cli_gather>`.

.. code-block:: bash
.. TODO: link the "Gather free energy results" how-to instead of the CLI reference once it exists.

for file in network_setup/transformations/*.json; do
relpath="${file:30}" # strip off "network_setup/"
dirpath=${relpath%.*} # strip off final ".json"
jobpath="network_setup/transformations/${dirpath}.job"
if [ -f "${jobpath}" ]; then
echo "${jobpath} already exists"
exit 1
fi
for repeat in {0..2}; do
cmd="openfe quickrun ${file} -o results_${repeat}/${relpath} -d results_${repeat}/${dirpath}"
echo -e "#!/usr/bin/env bash\n${cmd}" > "${jobpath}"
sbatch "${jobpath}"
done
done
If you would rather have **openfe** schedule the tasks of a whole network for you,
see :ref:`userguide_task_based_execution`.

This should result in the following file structure after execution:
.. admonition:: How-to guides

::

results_parallel
├── results_0
│   ├── rbfe_lig_ejm_31_complex_lig_ejm_42_complex.json
│   ├── shared_HybridTopologyMultiStateAnalysisUnit-5e0825de1dd045818cdc3428205c1cf7_attempt_0
│   │   ├── forward_reverse_convergence.png
│   │   ├── ligand_RMSD.png
│   │   ├── mbar_overlap_matrix.png
│   │   ├── replica_exchange_matrix.png
│   │   ├── replica_state_timeseries.png
│   │   └── structural_analysis.npz
│   ├── shared_HybridTopologyMultiStateSimulationUnit-144be594cf024cb19152cfe5e0b3fb7d_attempt_0
│   │   ├── checkpoint.chk
│   │   ├── simulation.nc
│  │  └── simulation_real_time_analysis.yaml
│   └── shared_HybridTopologySetupUnit-01b5afe1972c4e2f9d0943da43b4b19c_attempt_0
│   ├── A_db.json
│   ├── B_db.json
│   ├── hybrid_positions.npy
│   ├── hybrid_system.pdb
│   └── hybrid_system.xml.bz2
├── results_1
│   ├── rbfe_lig_ejm_31_complex_lig_ejm_42_complex.json
│   ├── shared_HybridTopologyMultiStateAnalysisUnit-7986bec616a74929aee85e900535f4a2_attempt_0
│   │   ├── forward_reverse_convergence.png
│   │   ├── ligand_RMSD.png
│   │   ├── mbar_overlap_matrix.png
│   │   ├── replica_exchange_matrix.png
│   │   ├── replica_state_timeseries.png
│   │   └── structural_analysis.npz
│   ├── shared_HybridTopologyMultiStateSimulationUnit-18eb295b7123444f9ac66ff3caffcab8_attempt_0
│   │   ├── checkpoint.chk
│   │   ├── simulation.nc
│  │  └── simulation_real_time_analysis.yaml
│   └── shared_HybridTopologySetupUnit-3d8ccb1ef5124bd4ba20e0047aad0b5f_attempt_0
│   ├── A_db.json
│   ├── B_db.json
│   ├── hybrid_positions.npy
│   ├── hybrid_system.pdb
│   └── hybrid_system.xml.bz2
└── results_2
├── rbfe_lig_ejm_31_complex_lig_ejm_42_complex.json
├── shared_HybridTopologyMultiStateAnalysisUnit-ac5fad8ad1fb49598f80018713dce070_attempt_0
│   ├── forward_reverse_convergence.png
│   ├── ligand_RMSD.png
│   ├── mbar_overlap_matrix.png
│   ├── replica_exchange_matrix.png
│   ├── replica_state_timeseries.png
│   └── structural_analysis.npz
├── shared_HybridTopologyMultiStateSimulationUnit-73abea21b423444881bd8f21415c937f_attempt_0
│   ├── checkpoint.chk
   │   ├── simulation.nc
  │  └── simulation_real_time_analysis.yaml
└── shared_HybridTopologySetupUnit-79bc9b63321945338a3b69d9f94ee15b_attempt_0
├── A_db.json
├── B_db.json
├── hybrid_positions.npy
├── hybrid_system.pdb
└── hybrid_system.xml.bz2

The results of which can be gathered from the CLI using the ``openfe gather`` command, in this case you should direct
it to the root directory which includes the repeat results and it will automatically collate the information

::

> openfe gather results_parallel

Optimizing GPU performance with NVIDIA MPS
==========================================

You can further optimize execution of ``openfe quickrun`` using NVIDIA's Multi-Process Service (MPS).
See NVIDIA's documentation on `MPS for OpenFE free energy calculations <https://developer.nvidia.com/blog/maximizing-openmm-molecular-dynamics-throughput-with-nvidia-multi-process-service/?ref=blog.omsf.io#mps_for_openfe_free_energy_calculations>`_ for details.
* :doc:`/how_to_guides/run_quickrun`
* :doc:`/how_to_guides/resume_quickrun`
* :doc:`/how_to_guides/run_on_slurm`
* :doc:`/how_to_guides/run_repeats_in_parallel`
* :doc:`/how_to_guides/run_with_mps`

See Also
--------
Expand Down
4 changes: 2 additions & 2 deletions docs/guide/protocols/absolutebinding.rst
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ the `Boresch restraint analysis tutorial
The multistate trajectory (``simulation.nc``) and its topology
(``alchemical_system.pdb``) can be analysed further using `openfe_analysis`_.

.. TODO: Link to the trajectory visualisation cookbook when published.
.. TODO: Link to the trajectory visualisation how_to_guide when published.

See Also
--------
Expand All @@ -154,7 +154,7 @@ See Also

**How-to guides**

:ref:`How-to guides <cookbooks>`
:ref:`How-to guides <how_to_guides>`

**API Documentation**

Expand Down
4 changes: 2 additions & 2 deletions docs/guide/protocols/absolutesolvation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ the structural analysis is skipped. The multistate trajectories
(``solvent.nc`` and ``vacuum.nc``) can be analysed further using
`openfe_analysis`_.

.. TODO: Link to the trajectory visualisation cookbook when published.
.. TODO: Link to the trajectory visualisation how_to_guide when published.


See Also
Expand All @@ -135,7 +135,7 @@ See Also

**How-to guides**

:ref:`How-to guides <cookbooks>`
:ref:`How-to guides <how_to_guides>`

**API Documentation**

Expand Down
2 changes: 1 addition & 1 deletion docs/guide/protocols/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -243,7 +243,7 @@ These limitations apply to all binding free energy protocols.
See also
--------

* :doc:`Choose and configure a protocol </cookbook/choose_protocol>`
* :doc:`Choose and configure a protocol </how_to_guides/choose_protocol>`
* :ref:`Assessing simulation health <userguide_simulation_health>`
* :doc:`Troubleshooting simulations </guide/troubleshooting>`

Expand Down
4 changes: 2 additions & 2 deletions docs/guide/protocols/relativehybridtopology.rst
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ evidence of ligand displacement.
The multistate trajectory (``simulation.nc``) and the hybrid topology
(``hybrid_system.pdb``) can be analysed further using `openfe_analysis`_.

.. TODO: Link to the trajectory visualisation cookbook when published.
.. TODO: Link to the trajectory visualisation how_to_guide when published.


See Also
Expand All @@ -219,7 +219,7 @@ See Also

**How-to guides**

:ref:`How-to guides <cookbooks>`
:ref:`How-to guides <how_to_guides>`

**API Documentation**

Expand Down
4 changes: 2 additions & 2 deletions docs/guide/protocols/septop.rst
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ the `Boresch restraint analysis tutorial
The multistate trajectory (``simulation.nc``) and its topology
(``alchemical_system.pdb``) can be analysed further using `openfe_analysis`_.

.. TODO: Link to the trajectory visualisation cookbook when published.
.. TODO: Link to the trajectory visualisation how_to_guide when published.

See Also
--------
Expand All @@ -165,7 +165,7 @@ See Also

**How-to guides**

:ref:`How-to guides <cookbooks>`
:ref:`How-to guides <how_to_guides>`

**API Documentation**

Expand Down
4 changes: 2 additions & 2 deletions docs/guide/results/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ and entire networks of results analysed.

.. admonition:: How-to guides

* :doc:`/cookbook/dumping_transformation`
* :doc:`/cookbook/jq_inspection`
* :doc:`/how_to_guides/dumping_transformation`
* :doc:`/how_to_guides/jq_inspection`

.. TODO: add the gather, load results, cinnabar and openfe_analysis how-tos.
2 changes: 1 addition & 1 deletion docs/guide/results/simulation_health.rst
Original file line number Diff line number Diff line change
Expand Up @@ -454,7 +454,7 @@ Some problems only show up across a network. Relative free energies around a
cycle should sum to zero within their uncertainties, and each edge should
agree with the value implied by the rest of the network. Large deviations
point to unreliable edges worth checking with the diagnostics above. These checks need a network with redundant edges
(see :doc:`Generate ligand networks </cookbook/generate_ligand_network>`).
(see :doc:`Generate ligand networks </how_to_guides/generate_ligand_network>`).
To calculate cycle closure errors, see the `cinnabar cycle closure tutorial
<https://cinnabar.openfree.energy/en/latest/tutorials/cycle-closure.html>`_.

Expand Down
Loading
Loading