feat(cli): added run benchmarks command to execute benchmark instances on PR

docs/getting-started/cli-reference.md

@@ -139,6 +139,189 @@ nexus get benchmark-requirements <package_name>
 nexus get benchmark-requirements terratorch
 ```
 
+## nexus run
+
+Execute benchmark instances.
+
+### Subcommands
+
+#### nexus run benchmarks
+
+Execute benchmarks from a GitHub Pull Request. This command identifies new or
+changed benchmark instances in a PR and optionally executes them using the `ado`
+CLI.
+
+**Usage:**
+
+```bash
+nexus run benchmarks --pr <pr_url> [OPTIONS]
+```
+
+!!! warning
+
+    This command requires the [GitHub CLI (`gh`)](https://cli.github.com/) to be installed and
+    configured in the command execution environment.
+
+**Required Options:**
+
+- `--pr <pr_url>`: GitHub Pull Request URL (e.g.,
+  `https://github.com/IBM/algorithm-nexus/pull/123`)
+
+**Optional Flags:**
+
+- `--remote <path>`: Execute operations on a remote Ray cluster using the
+  specified configuration file. When provided, the command automatically
+  installs the required benchmark packages in the Ray environment. Read
+  [`Running ado on remote Ray clusters`](https://ibm.github.io/ado/getting-started/remote_run/)
+  to discover the remote context configuration format.
+- `--context <path>`: Path to ADO context YAML file (samplestore context). Read
+  [`Working with Contexts`](https://ibm.github.io/ado/resources/metastore/#working-with-contexts)
+  to discover how to manage contexts.
+- `--dry-run`: List benchmark instances without executing them (dry run)
+- `--output-file <path>`: Output file path for execution results. If not
+  specified, results are printed to screen.
+- `-o, --output-format <format>`: Output format: 'json' or 'yaml'. Can be used
+  with or without `--output-file`. When used without `--output-file`, prints
+  formatted output to console. When used with `--output-file`, overrides format
+  inference from file extension (defaults to json).
+
+**Behavior:**
+
+The command automatically:
+
+1. Checks if the local repository is on the same commit as the PR
+2. If not, checks out the PR code to a temporary directory
+3. Analyzes the PR to find new or changed benchmark instances
+4. Executes the benchmarks (unless `--dry-run` is specified)
+5. Writes results to the output file
+
+**Benchmark Instance Detection:**
+
+A benchmark instance is detected as changed if any file within its directory is
+modified in the PR. This includes:
+
+- Changes to `space.yaml` files
+- Changes to any other files in the `benchmark_instances/<instance-name>/`
+  directory
+- New benchmark instance directories (any new folder under
+  `benchmark_instances/`)
+
+The detection works for both:
+
+- **Model-level instances**:
+  `packages/<package>/models/<model>/benchmark_instances/<instance>/`
+- **Package-level instances**:
+  `packages/<package>/benchmark_instances/<instance>/`
+
+!!! note
+
+    When not running in remote mode (`--remote` not set), the benchmark instances will be executed with
+    `ado` in the local environment. It is the user responsibility to ensure that the required
+    benchmark packages are installed in the local python environment. Benchmark packages are listed for
+    each nexus package in the `nexus.yaml` configuration.
+
+**Examples:**
+
+List benchmarks in a PR without executing (dry run):
+
+```bash
+nexus run benchmarks --pr https://github.com/IBM/algorithm-nexus/pull/123 --dry-run
+```
+
+Execute benchmarks locally:
+
+```bash
+nexus run benchmarks --pr https://github.com/IBM/algorithm-nexus/pull/123
+```
+
+Execute benchmarks on a remote Ray cluster:
+
+```bash
+nexus run benchmarks \
+  --pr https://github.com/IBM/algorithm-nexus/pull/123 \
+  --remote path/to/remote-context.yaml \
+  --context path/to/ado-context.yaml
+```
+
+Execute benchmarks and save results to a file:
+
+```bash
+nexus run benchmarks \
+  --pr https://github.com/IBM/algorithm-nexus/pull/123 \
+  --output-file benchmark_results.json
+```
+
+Execute benchmarks and save results in YAML format:
+
+```bash
+nexus run benchmarks \
+  --pr https://github.com/IBM/algorithm-nexus/pull/123 \
+  --output-file results.yaml \
+  --output-format yaml
+```
+
+**Output Format:**
+
+By default, the command outputs results in a human-readable format showing the
+status, message, and IDs for each benchmark instance.
+
+To get structured output (JSON or YAML), use the `--output-format` option:
+
+```bash
+# Print JSON to console
+algorithm-nexus run packages/terratorch/models/prithvi/benchmark_instances/flood-test \
+  --output-format json
+
+# Print YAML to console
+algorithm-nexus run packages/terratorch/models/prithvi/benchmark_instances/flood-test \
+  --output-format yaml
+```
+
+The structured output (JSON) has the following format:
+
+```json
+{
+    "instances": [
+        {
+            "instance_path": "packages/<package>/models/<model>/benchmark_instances/test_benchmark",
+            "status": "started",
+            "message": "Successfully started on Ray cluster with job ID: raysubmit_snRVd4ZqTTKcaR3W | Space ID: space-a009d7-default",
+            "space_id": "space-a009d7-default",
+            "operation_id": "randomwalk-123456-default",
+            "ray_job_id": "raysubmit_snRVd4ZqTTKcaR3W"
+        }
+    ]
+}
+```
+
+**Status Values:**
+
+- `success`: Benchmark completed successfully (local execution)
+- `started`: Benchmark started on Ray cluster (remote execution)
+- `failed`: Benchmark execution failed
+- `unknown`: Status could not be determined
+
+**Execution Mode Comparison:**
+
+| Field          | Local Mode (`--remote` not set)                                              | Remote Mode (`--remote` set)                                                 |
+| -------------- | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
+| `status`       | `success` (on success)<br>`failed` (on error)<br>`unknown` (if undetermined) | `started` (on success)<br>`failed` (on error)<br>`unknown` (if undetermined) |
+| `operation_id` | ADO operation ID                                                             | `null`                                                                       |
+| `ray_job_id`   | `null`                                                                       | Ray job ID                                                                   |
+| Notes          | Operation completes locally                                                  | Job submitted to Ray cluster                                                 |
+
+When `operation_id` is `null` (in remote mode), users will need to inspect the
+Ray job logs to extract the ADO operation ID once execution completes.
+
+In case of failure (either mode), the `status` will be `failed` and the
+`message` field will contain the reason for the failure.
+
+**Exit Codes:**
+
+- `0`: All benchmarks executed successfully
+- `1`: One or more benchmarks failed or an error occurred
+- `130`: Interrupted by user (Ctrl+C)
+
 ## Common Workflows
 
 ### Validating a New Package
@@ -177,6 +360,35 @@ Retrieve specific information about a package:
 nexus get benchmark-requirements my-package
 ```
 
+### Running Benchmarks from a Pull Request
+
+Execute benchmarks from a GitHub PR to validate changes:
+
+```bash
+# First, check what benchmarks would be executed (dry run)
+nexus run benchmarks \
+  --pr https://github.com/IBM/algorithm-nexus/pull/123 \
+  --dry-run
+
+# Execute benchmarks locally
+nexus run benchmarks \
+  --pr https://github.com/IBM/algorithm-nexus/pull/123
+
+# Execute benchmarks on a remote Ray cluster
+nexus run benchmarks \
+  --pr https://github.com/IBM/algorithm-nexus/pull/123 \
+  --remote config/remote-context.yaml \
+  --context config/ado-context.yaml \
+  --output-file pr123_results.json
+```
+
+The command will:
+
+1. Automatically detect if your local repo is on the PR commit
+2. Checkout the PR code if needed (to a temporary directory)
+3. Find all new or changed benchmark instances
+4. Execute them and report results
+
 ## Exit Codes
 
 All commands follow standard Unix exit code conventions:

pyproject.toml

@@ -22,6 +22,7 @@ candidate = [
   "vllm==0.19.1",
 ]
 cli = [
+  "ado-core>=1.8.0",
   "pydantic>=2.13.3",
   "pyyaml>=6.0.3",
   "rich>=15.0.0",

src/algorithm_nexus/cli.py

@@ -24,6 +24,7 @@
     list_benchmark_packages,
     list_packages,
 )
+from algorithm_nexus.commands.run import run_benchmarks
 from algorithm_nexus.commands.validate import validate
 
 console = Console()
@@ -48,6 +49,13 @@
 )
 app.add_typer(get_app, name="get")
 
+# Create subcommand group for 'run'
+run_app = typer.Typer(
+    help="Execute benchmarks and operations.",
+    no_args_is_help=True,
+)
+app.add_typer(run_app, name="run")
+
 
 @app.callback(invoke_without_command=True)
 def main_callback(ctx: typer.Context) -> None:
@@ -65,6 +73,9 @@ def main_callback(ctx: typer.Context) -> None:
 # Register get commands
 get_app.command(name="benchmark-requirements")(get_benchmark_requirements)
 
+# Register run commands
+run_app.command(name="benchmarks")(run_benchmarks)
+
 
 def main() -> None:
     """Entry point for the CLI application."""

src/algorithm_nexus/commands/list.py

@@ -340,11 +340,9 @@ def list_benchmark_experiments(
 
         # Add instructions for getting more details
         console.print("[bold]For further details on each experiment:[/bold]")
-        console.print("1. Install ado")
-        console.print("   [cyan]uv pip install ado-core[/cyan]")
-        console.print("2. Install the Benchmark package the experiment belongs to")
+        console.print("1. Install the Benchmark package the experiment belongs to")
         console.print("   [cyan]uv pip install <benchmark_package>[/cyan]")
-        console.print("3. Describe the experiment")
+        console.print("2. Describe the experiment")
         console.print("   [cyan]ado describe experiment <experiment_id>[/cyan]\n")