Skip to content

LCORE-2341-unified mode feature files #2020

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
radofuchs wants to merge 3 commits into
base: main
Choose a base branch
from
Open

LCORE-2341-unified mode feature files #2020

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8161786
LCORE-2341-unified mode feature files
Jun 29, 2026
25c96be
mark cases as skipped
Jun 29, 2026
da9416d
mark cases as skipped
Jun 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 116 additions & 0 deletions tests/e2e/features/unified-mode-boot.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
@e2e_group_2 @skip

@coderabbitai coderabbitai Bot Jun 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Feature files carrying a top-level skip tag:"
rg -n '^@.*\bskip\b' tests/e2e/features

echo
echo "Behave hook handling for skip-related tags:"
fd -HI 'environment.py' tests/e2e . -x sh -c '
  echo "--- $1 ---"
  rg -n -C3 "skip|skip-in-server-mode|skip-in-library-mode" "$1" || true
' sh {}

Repository: lightspeed-core/lightspeed-stack

Length of output: 8141

Remove the feature-level @skip unless this suite is meant to stay disabled. tests/e2e/features/environment.py skips any scenario tagged @skip, so this feature will never execute as written.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e/features/unified-mode-boot.feature` at line 1, The feature is
currently tagged with `@skip`, which prevents every scenario in unified-mode-boot
from running because environment.py skips any scenario carrying that tag. Remove
the feature-level `@skip` from unified-mode-boot.feature unless this suite is
intentionally disabled, and keep the existing e2e_group_2 tag so the scenarios
can execute normally.

Feature: Unified mode configuration boot

Background:
Given The service is started locally
And The system is in default state
And REST API service prefix is /v1
And the Lightspeed stack configuration directory is "tests/e2e/configuration/unified-mode"


# --- library mode (@skip-in-server-mode) ---

@skip-in-server-mode
Scenario: Unified config with inference.providers boots and serves requests in library mode
Given The service uses the lightspeed-stack-unified-providers.yaml configuration
And The service is restarted
When I access endpoint "readiness" using HTTP GET method
Then The status code of the response is 200
And The body of the response has the following schema
"""
{
"ready": "bool",
"reason": "str",
"providers": "list[str]"
}
"""
When I use "query" to ask question
"""
{"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"}
"""
Then The status code of the response is 200


@skip-in-server-mode
Scenario: Unified config with llama_stack.config only boots and serves requests in library mode
Given The service uses the lightspeed-stack-unified-config-only.yaml configuration
And The service is restarted
When I access endpoint "readiness" using HTTP GET method
Then The status code of the response is 200
When I use "query" to ask question
"""
{"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"}
"""
Then The status code of the response is 200


@skip-in-server-mode
Scenario: Unified config with relative profile path boots in library mode
Given The service uses the lightspeed-stack-unified-relative-profile.yaml configuration
And The service is restarted
When I access endpoint "readiness" using HTTP GET method
Then The status code of the response is 200


@skip-in-server-mode
Scenario: Unified config with absolute profile path boots in library mode
Given The service uses the lightspeed-stack-unified-absolute-profile.yaml configuration
And The service is restarted
When I access endpoint "readiness" using HTTP GET method
Then The status code of the response is 200
Comment on lines +47 to +60

@coderabbitai coderabbitai Bot Jun 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Exercise query serving for the profile-path variants.

These four scenarios stop at /readiness, so they never verify that the relative/absolute profile configs can actually serve inference requests. That leaves the boot contract untested for those variants and would miss regressions where startup succeeds but profile resolution breaks query handling.

Also applies to: 101-116

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e/features/unified-mode-boot.feature` around lines 47 - 60, The
profile-path boot scenarios only verify readiness and never exercise query
serving, so the relative and absolute variants can still break inference without
failing the test. Update the unified-mode boot scenarios in
unified-mode-boot.feature to include an actual inference/query request after the
service restarts and readiness passes, using the existing scenario patterns for
the relative and absolute profile-path cases so both boot and serve behavior are
covered.



# --- server mode (@skip-in-library-mode) ---

@skip-in-library-mode
Scenario: Unified config with inference.providers boots and serves requests in server mode
Given The service uses the lightspeed-stack-unified-providers.yaml configuration
And Llama Stack is restarted
And Lightspeed Stack is restarted
When I access endpoint "readiness" using HTTP GET method
Then The status code of the response is 200
And The body of the response has the following schema
"""
{
"ready": "bool",
"reason": "str",
"providers": "list[str]"
}
"""
When I use "query" to ask question
"""
{"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"}
"""
Then The status code of the response is 200


@skip-in-library-mode
Scenario: Unified config with llama_stack.config only boots and serves requests in server mode
Given The service uses the lightspeed-stack-unified-config-only.yaml configuration
And Llama Stack is restarted
And Lightspeed Stack is restarted
When I access endpoint "readiness" using HTTP GET method
Then The status code of the response is 200
When I use "query" to ask question
"""
{"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"}
"""
Then The status code of the response is 200


@skip-in-library-mode
Scenario: Unified config with relative profile path boots in server mode
Given The service uses the lightspeed-stack-unified-relative-profile.yaml configuration
And Llama Stack is restarted
And Lightspeed Stack is restarted
When I access endpoint "readiness" using HTTP GET method
Then The status code of the response is 200


@skip-in-library-mode
Scenario: Unified config with absolute profile path boots in server mode
Given The service uses the lightspeed-stack-unified-absolute-profile.yaml configuration
And Llama Stack is restarted
And Lightspeed Stack is restarted
When I access endpoint "readiness" using HTTP GET method
Then The status code of the response is 200
39 changes: 39 additions & 0 deletions tests/e2e/features/unified-mode-legacy.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
@e2e_group_2 @skip

@coderabbitai coderabbitai Bot Jun 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Check how `@skip` is handled in the E2E harness and where it is used.
fd -i 'environment.py' tests/e2e .
rg -n -C2 --glob 'tests/e2e/**' '`@skip`\b|`@skip-in-server-mode`|`@skip-in-library-mode`'
rg -n -C2 --glob '*' 'behave|--tags|`@skip`|skip-in-server-mode|skip-in-library-mode'

Repository: lightspeed-core/lightspeed-stack

Length of output: 326

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== feature file ==\n'
cat -n tests/e2e/features/unified-mode-legacy.feature

printf '\n== environment.py ==\n'
cat -n tests/e2e/features/environment.py

printf '\n== tag references in e2e ==\n'
rg -n -C2 '`@skip`\b|`@skip-in-server-mode`|`@skip-in-library-mode`|before_tag|after_tag|active_tags|tag_expression|behave' tests/e2e

Repository: lightspeed-core/lightspeed-stack

Length of output: 50388

🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from pathlib import Path
p = Path('tests/e2e/features/unified-mode-legacy.feature')
text = p.read_text()
print('top tags:', text.splitlines()[:5])
print('has scenario tags:', '`@skip-in-server-mode`' in text, '`@skip-in-library-mode`' in text)
PY

Repository: lightspeed-core/lightspeed-stack

Length of output: 365

Remove the feature-level @skip tag. tests/e2e/features/unified-mode-legacy.feature:1 skips both scenarios, so this legacy coverage never runs; the scenario-level mode tags already gate the two cases.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e/features/unified-mode-legacy.feature` at line 1, Remove the
feature-level `@skip` tag from the unified-mode-legacy feature so the scenarios
are no longer skipped entirely; keep the existing scenario-level mode tags to
control which legacy cases run. Update the feature header in
tests/e2e/features/unified-mode-legacy.feature so only the intended scenario
gating remains.

Feature: Legacy two-file configuration during deprecation window

Background:
Given The service is started locally
And The system is in default state
And REST API service prefix is /v1
And the Lightspeed stack configuration directory is "tests/e2e/configuration"


# --- library mode (@skip-in-server-mode) ---

@skip-in-server-mode
Scenario: Legacy two-file configuration still boots and serves requests in library mode
Given The service uses the lightspeed-stack.yaml configuration
And The service is restarted
When I access endpoint "readiness" using HTTP GET method
Then The status code of the response is 200
When I use "query" to ask question
"""
{"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"}
"""
Then The status code of the response is 200


# --- server mode (@skip-in-library-mode) ---

@skip-in-library-mode
Scenario: Legacy two-file configuration still boots and serves requests in server mode
Given The service uses the lightspeed-stack.yaml configuration
And Llama Stack is restarted
And Lightspeed Stack is restarted
When I access endpoint "readiness" using HTTP GET method
Then The status code of the response is 200
When I use "query" to ask question
"""
{"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"}
"""
Then The status code of the response is 200
53 changes: 53 additions & 0 deletions tests/e2e/features/unified-mode-migration.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
@e2e_group_2 @skip

@coderabbitai coderabbitai Bot Jun 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Remove the file-level @skip.

Line 1 skips the entire feature, so none of these newly added migration scenarios will execute even though this PR registers them as E2E coverage. That leaves the migration path untested in CI. As per coding guidelines, tests/e2e/**/*.{py,feature} should be used for behave-based end-to-end testing, and a file-level skip prevents that coverage from running.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e/features/unified-mode-migration.feature` at line 1, The feature is
currently marked with a file-level `@skip`, which prevents all unified mode
migration scenarios from running in E2E. Remove the `@skip` from the feature
header in unified-mode-migration.feature so behave executes the registered
migration coverage, while keeping any narrower scenario-level tags only if
needed.

Source: Coding guidelines

Feature: Legacy to unified configuration migration

Background:
Given The service is started locally
And The system is in default state
And REST API service prefix is /v1
And the Lightspeed stack configuration directory is "tests/e2e/configuration/unified-mode"


Scenario: migrate-config produces a unified configuration from a legacy pair
When lightspeed-stack --migrate-config is run for the legacy migration fixture pair
Then the file lightspeed-stack-unified-migrated.yaml contains native_override
And the file lightspeed-stack-unified-migrated.yaml does not contain library_client_config_path


Scenario: migrate then synthesize round-trips to the original run.yaml
When lightspeed-stack --migrate-config is run for the legacy migration fixture pair
And the active unified configuration is synthesized to run.yaml
Then the synthesized run.yaml is byte-identical to the legacy migration fixture run.yaml


# --- library mode (@skip-in-server-mode) ---

@skip-in-server-mode
Scenario: Migrated unified configuration drives byte-identical Llama Stack behavior in library mode
When lightspeed-stack --migrate-config is run for the legacy migration fixture pair
Given The service uses the lightspeed-stack-unified-migrated.yaml configuration
And The service is restarted
When I access endpoint "readiness" using HTTP GET method
Then The status code of the response is 200
When I use "query" to ask question
"""
{"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"}
"""
Then The status code of the response is 200
Comment on lines +25 to +36

@coderabbitai coderabbitai Bot Jun 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🔵 Trivial

Reorder steps to follow standard Given-When-Then structure.

In both library and server mode scenarios, the When step running --migrate-config precedes the Given step that configures the service to use the migrated file. While behave executes steps sequentially regardless of keyword, standard BDD style places all Given preconditions before When actions. Consider restructuring so that Given The service uses... appears before When lightspeed-stack --migrate-config is run..., or use And to clarify that the migration is part of the action phase.

📝 Suggested reordering for library mode scenario 
   `@skip-in-server-mode`
   Scenario: Migrated unified configuration drives byte-identical Llama Stack behavior in library mode
-     When lightspeed-stack --migrate-config is run for the legacy migration fixture pair
     Given The service uses the lightspeed-stack-unified-migrated.yaml configuration
+     When lightspeed-stack --migrate-config is run for the legacy migration fixture pair
       And The service is restarted

Or if the migrate-config must happen first:

   `@skip-in-server-mode`
   Scenario: Migrated unified configuration drives byte-identical Llama Stack behavior in library mode
-     When lightspeed-stack --migrate-config is run for the legacy migration fixture pair
-     Given The service uses the lightspeed-stack-unified-migrated.yaml configuration
+     Given The legacy migration fixture pair has been migrated to lightspeed-stack-unified-migrated.yaml
+       And The service uses the lightspeed-stack-unified-migrated.yaml configuration
       And The service is restarted

Also applies to: 41-53

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e/features/unified-mode-migration.feature` around lines 25 - 36, The
migrated unified-mode scenarios have the action step out of standard
Given-When-Then order; in the affected feature cases, move the service
configuration precondition into the Given phase before the migration action, or
rewrite the migration step as part of the When/And action phase. Update the
affected scenarios in unified-mode-migration.feature so the step sequence around
lightspeed-stack --migrate-config, The service uses the
lightspeed-stack-unified-migrated.yaml configuration, and The service is
restarted reads clearly and follows BDD structure.



# --- server mode (@skip-in-library-mode) ---

@skip-in-library-mode
Scenario: Migrated unified configuration drives byte-identical Llama Stack behavior in server mode
When lightspeed-stack --migrate-config is run for the legacy migration fixture pair
Given The service uses the lightspeed-stack-unified-migrated.yaml configuration
And Llama Stack is restarted
And Lightspeed Stack is restarted
When I access endpoint "readiness" using HTTP GET method
Then The status code of the response is 200
When I use "query" to ask question
"""
{"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"}
"""
Then The status code of the response is 200
59 changes: 59 additions & 0 deletions tests/e2e/features/unified-mode-synthesis.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
@e2e_group_2 @skip

@coderabbitai coderabbitai Bot Jun 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify how `@skip` is handled in this repo's behave setup and where else it is used.
fd -i 'environment.py' tests . -x sh -c 'echo "=== $1 ==="; sed -n "1,220p" "$1"' sh {}
echo
rg -n -C2 '`@skip`\b|skip-in-server-mode|skip-in-library-mode' tests/e2e

Repository: lightspeed-core/lightspeed-stack

Length of output: 32630

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== tests/e2e/features/unified-mode-synthesis.feature ==="
sed -n '1,120p' tests/e2e/features/unified-mode-synthesis.feature

echo
echo "=== Behave availability ==="
python3 - <<'PY'
try:
    import behave
    print("behave:", behave.__version__)
except Exception as e:
    print("behave_import_error:", type(e).__name__, e)
PY

Repository: lightspeed-core/lightspeed-stack

Length of output: 3113

Remove the feature-level @skip tag. This disables all six scenarios in tests/e2e/features/unified-mode-synthesis.feature, so the new synthesis coverage never runs.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e/features/unified-mode-synthesis.feature` at line 1, Remove the
feature-level `@skip` tag from unified-mode-synthesis.feature so the six synthesis
scenarios are no longer disabled. Update the feature header in
tests/e2e/features/unified-mode-synthesis.feature to keep the existing
`@e2e_group_2` tag but drop `@skip`, ensuring the scenarios run normally.

Feature: Unified mode configuration synthesis

Background:
Given The service is started locally
And The system is in default state
And the Lightspeed stack configuration directory is "tests/e2e/configuration/unified-mode"


Scenario: native_override replaces an overlapping scalar key
Given The service uses the lightspeed-stack-unified-native-override-scalar.yaml configuration
When the active unified configuration is synthesized to run.yaml
Then the synthesized run.yaml contains the native_override scalar value for safety.excluded_categories


Scenario: native_override replaces an overlapping list key wholesale
Given The service uses the lightspeed-stack-unified-native-override-list.yaml configuration
When the active unified configuration is synthesized to run.yaml
Then the synthesized run.yaml contains exactly the native_override list for apis


Scenario: LCORE-emitted secrets remain as environment references on disk
Given The service uses the lightspeed-stack-unified-providers.yaml configuration
When the active unified configuration is synthesized to run.yaml
Then the synthesized run.yaml contains ${env.OPENAI_API_KEY}
And the synthesized run.yaml does not contain the resolved OPENAI_API_KEY value


Scenario: Synthesized run.yaml is written with owner-only permissions
Given The service uses the lightspeed-stack-unified-providers.yaml configuration
When the active unified configuration is synthesized to run.yaml
Then the synthesized run.yaml file permissions are 0600


Scenario: synthesized-config-output overrides the default synthesis location
Given The service uses the lightspeed-stack-unified-providers.yaml configuration
And lightspeed-stack is started with --synthesized-config-output set to a custom path
When the active unified configuration is synthesized to run.yaml
Then the synthesized run.yaml is written to the custom output path
And the default synthesized run.yaml path does not exist


# --- library mode (@skip-in-server-mode) ---

@skip-in-server-mode
Scenario: Synthesized run.yaml path is logged at startup in library mode
Given The service uses the lightspeed-stack-unified-providers.yaml configuration
And The service is restarted
Then the lightspeed-stack container logs contain synthesized run.yaml


# --- server mode (@skip-in-library-mode) ---

@skip-in-library-mode
Scenario: Synthesized run.yaml path is logged at startup in server mode
Given The service uses the lightspeed-stack-unified-providers.yaml configuration
And Llama Stack is restarted
And Lightspeed Stack is restarted
Then the lightspeed-stack container logs contain synthesized run.yaml
19 changes: 19 additions & 0 deletions tests/e2e/features/unified-mode-validation.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
@e2e_group_2 @skip

@coderabbitai coderabbitai Bot Jun 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Check how `@skip` is handled in the E2E harness and where it is used.
fd -i 'environment.py' tests/e2e .
rg -n -C2 --glob 'tests/e2e/**' '`@skip`\b|`@skip-in-server-mode`|`@skip-in-library-mode`'
rg -n -C2 --glob '*' 'behave|--tags|`@skip`|skip-in-server-mode|skip-in-library-mode'

Repository: lightspeed-core/lightspeed-stack

Length of output: 326

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n== feature file ==\n'
cat -n tests/e2e/features/unified-mode-validation.feature

printf '\n== environment.py outline ==\n'
ast-grep outline tests/e2e/features/environment.py --view expanded || true

printf '\n== environment.py ==\n'
cat -n tests/e2e/features/environment.py

printf '\n== skip tag usage ==\n'
rg -n -C2 '`@skip`\b|skip-in-server-mode|skip-in-library-mode|--tags|behave' tests/e2e . --glob '!**/.git/**'

Repository: lightspeed-core/lightspeed-stack

Length of output: 50390

Remove the feature-level @skip tag.
That tag skips both validation scenarios, so the new --migrate-config error path never runs.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/e2e/features/unified-mode-validation.feature` at line 1, The
feature-level `@skip` tag on unified-mode-validation.feature is preventing both
scenarios from running, including the new --migrate-config error path. Remove
the top-level `@skip` from the feature declaration so the validation scenarios
execute normally, while keeping the existing scenario-level tags or any
selective skips only where explicitly needed.

Feature: Unified mode configuration validation

Background:
Given The service is started locally
And The system is in default state
And the Lightspeed stack configuration directory is "tests/e2e/configuration/unified-mode"


Scenario: inference.providers together with library_client_config_path fails at load
Given The service uses the lightspeed-stack-invalid-providers-and-legacy.yaml configuration
When configuration validation is attempted for the active configuration
Then the validation error contains --migrate-config


Scenario: llama_stack.config together with library_client_config_path fails at load
Given The service uses the lightspeed-stack-invalid-config-and-legacy.yaml configuration
When configuration validation is attempted for the active configuration
Then the validation error contains --migrate-config
5 changes: 5 additions & 0 deletions tests/e2e/test_list.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,8 @@ features/tls-ca.feature
features/tls-mtls.feature
features/tls-tlsv13.feature
features/opentelemetry.feature
features/unified-mode-boot.feature
features/unified-mode-legacy.feature
features/unified-mode-validation.feature
features/unified-mode-migration.feature
features/unified-mode-synthesis.feature
Loading