diff --git a/tests/e2e/features/unified-mode-boot.feature b/tests/e2e/features/unified-mode-boot.feature new file mode 100644 index 000000000..ef7b34dca --- /dev/null +++ b/tests/e2e/features/unified-mode-boot.feature @@ -0,0 +1,116 @@ +@e2e_group_2 @skip +Feature: Unified mode configuration boot + + Background: + Given The service is started locally + And The system is in default state + And REST API service prefix is /v1 + And the Lightspeed stack configuration directory is "tests/e2e/configuration/unified-mode" + + + # --- library mode (@skip-in-server-mode) --- + + @skip-in-server-mode + Scenario: Unified config with inference.providers boots and serves requests in library mode + Given The service uses the lightspeed-stack-unified-providers.yaml configuration + And The service is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is 200 + And The body of the response has the following schema + """ + { + "ready": "bool", + "reason": "str", + "providers": "list[str]" + } + """ + When I use "query" to ask question + """ + {"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + + + @skip-in-server-mode + Scenario: Unified config with llama_stack.config only boots and serves requests in library mode + Given The service uses the lightspeed-stack-unified-config-only.yaml configuration + And The service is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is 200 + When I use "query" to ask question + """ + {"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + + + @skip-in-server-mode + Scenario: Unified config with relative profile path boots in library mode + Given The service uses the lightspeed-stack-unified-relative-profile.yaml configuration + And The service is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is 200 + + + @skip-in-server-mode + Scenario: Unified config with absolute profile path boots in library mode + Given The service uses the lightspeed-stack-unified-absolute-profile.yaml configuration + And The service is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is 200 + + + # --- server mode (@skip-in-library-mode) --- + + @skip-in-library-mode + Scenario: Unified config with inference.providers boots and serves requests in server mode + Given The service uses the lightspeed-stack-unified-providers.yaml configuration + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is 200 + And The body of the response has the following schema + """ + { + "ready": "bool", + "reason": "str", + "providers": "list[str]" + } + """ + When I use "query" to ask question + """ + {"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + + + @skip-in-library-mode + Scenario: Unified config with llama_stack.config only boots and serves requests in server mode + Given The service uses the lightspeed-stack-unified-config-only.yaml configuration + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is 200 + When I use "query" to ask question + """ + {"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + + + @skip-in-library-mode + Scenario: Unified config with relative profile path boots in server mode + Given The service uses the lightspeed-stack-unified-relative-profile.yaml configuration + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is 200 + + + @skip-in-library-mode + Scenario: Unified config with absolute profile path boots in server mode + Given The service uses the lightspeed-stack-unified-absolute-profile.yaml configuration + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is 200 diff --git a/tests/e2e/features/unified-mode-legacy.feature b/tests/e2e/features/unified-mode-legacy.feature new file mode 100644 index 000000000..0bee463ec --- /dev/null +++ b/tests/e2e/features/unified-mode-legacy.feature @@ -0,0 +1,39 @@ +@e2e_group_2 @skip +Feature: Legacy two-file configuration during deprecation window + + Background: + Given The service is started locally + And The system is in default state + And REST API service prefix is /v1 + And the Lightspeed stack configuration directory is "tests/e2e/configuration" + + + # --- library mode (@skip-in-server-mode) --- + + @skip-in-server-mode + Scenario: Legacy two-file configuration still boots and serves requests in library mode + Given The service uses the lightspeed-stack.yaml configuration + And The service is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is 200 + When I use "query" to ask question + """ + {"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + + + # --- server mode (@skip-in-library-mode) --- + + @skip-in-library-mode + Scenario: Legacy two-file configuration still boots and serves requests in server mode + Given The service uses the lightspeed-stack.yaml configuration + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is 200 + When I use "query" to ask question + """ + {"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 diff --git a/tests/e2e/features/unified-mode-migration.feature b/tests/e2e/features/unified-mode-migration.feature new file mode 100644 index 000000000..31db10022 --- /dev/null +++ b/tests/e2e/features/unified-mode-migration.feature @@ -0,0 +1,53 @@ +@e2e_group_2 @skip +Feature: Legacy to unified configuration migration + + Background: + Given The service is started locally + And The system is in default state + And REST API service prefix is /v1 + And the Lightspeed stack configuration directory is "tests/e2e/configuration/unified-mode" + + + Scenario: migrate-config produces a unified configuration from a legacy pair + When lightspeed-stack --migrate-config is run for the legacy migration fixture pair + Then the file lightspeed-stack-unified-migrated.yaml contains native_override + And the file lightspeed-stack-unified-migrated.yaml does not contain library_client_config_path + + + Scenario: migrate then synthesize round-trips to the original run.yaml + When lightspeed-stack --migrate-config is run for the legacy migration fixture pair + And the active unified configuration is synthesized to run.yaml + Then the synthesized run.yaml is byte-identical to the legacy migration fixture run.yaml + + + # --- library mode (@skip-in-server-mode) --- + + @skip-in-server-mode + Scenario: Migrated unified configuration drives byte-identical Llama Stack behavior in library mode + When lightspeed-stack --migrate-config is run for the legacy migration fixture pair + Given The service uses the lightspeed-stack-unified-migrated.yaml configuration + And The service is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is 200 + When I use "query" to ask question + """ + {"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 + + + # --- server mode (@skip-in-library-mode) --- + + @skip-in-library-mode + Scenario: Migrated unified configuration drives byte-identical Llama Stack behavior in server mode + When lightspeed-stack --migrate-config is run for the legacy migration fixture pair + Given The service uses the lightspeed-stack-unified-migrated.yaml configuration + And Llama Stack is restarted + And Lightspeed Stack is restarted + When I access endpoint "readiness" using HTTP GET method + Then The status code of the response is 200 + When I use "query" to ask question + """ + {"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"} + """ + Then The status code of the response is 200 diff --git a/tests/e2e/features/unified-mode-synthesis.feature b/tests/e2e/features/unified-mode-synthesis.feature new file mode 100644 index 000000000..57fd20aeb --- /dev/null +++ b/tests/e2e/features/unified-mode-synthesis.feature @@ -0,0 +1,59 @@ +@e2e_group_2 @skip +Feature: Unified mode configuration synthesis + + Background: + Given The service is started locally + And The system is in default state + And the Lightspeed stack configuration directory is "tests/e2e/configuration/unified-mode" + + + Scenario: native_override replaces an overlapping scalar key + Given The service uses the lightspeed-stack-unified-native-override-scalar.yaml configuration + When the active unified configuration is synthesized to run.yaml + Then the synthesized run.yaml contains the native_override scalar value for safety.excluded_categories + + + Scenario: native_override replaces an overlapping list key wholesale + Given The service uses the lightspeed-stack-unified-native-override-list.yaml configuration + When the active unified configuration is synthesized to run.yaml + Then the synthesized run.yaml contains exactly the native_override list for apis + + + Scenario: LCORE-emitted secrets remain as environment references on disk + Given The service uses the lightspeed-stack-unified-providers.yaml configuration + When the active unified configuration is synthesized to run.yaml + Then the synthesized run.yaml contains ${env.OPENAI_API_KEY} + And the synthesized run.yaml does not contain the resolved OPENAI_API_KEY value + + + Scenario: Synthesized run.yaml is written with owner-only permissions + Given The service uses the lightspeed-stack-unified-providers.yaml configuration + When the active unified configuration is synthesized to run.yaml + Then the synthesized run.yaml file permissions are 0600 + + + Scenario: synthesized-config-output overrides the default synthesis location + Given The service uses the lightspeed-stack-unified-providers.yaml configuration + And lightspeed-stack is started with --synthesized-config-output set to a custom path + When the active unified configuration is synthesized to run.yaml + Then the synthesized run.yaml is written to the custom output path + And the default synthesized run.yaml path does not exist + + + # --- library mode (@skip-in-server-mode) --- + + @skip-in-server-mode + Scenario: Synthesized run.yaml path is logged at startup in library mode + Given The service uses the lightspeed-stack-unified-providers.yaml configuration + And The service is restarted + Then the lightspeed-stack container logs contain synthesized run.yaml + + + # --- server mode (@skip-in-library-mode) --- + + @skip-in-library-mode + Scenario: Synthesized run.yaml path is logged at startup in server mode + Given The service uses the lightspeed-stack-unified-providers.yaml configuration + And Llama Stack is restarted + And Lightspeed Stack is restarted + Then the lightspeed-stack container logs contain synthesized run.yaml diff --git a/tests/e2e/features/unified-mode-validation.feature b/tests/e2e/features/unified-mode-validation.feature new file mode 100644 index 000000000..fe86c5ce5 --- /dev/null +++ b/tests/e2e/features/unified-mode-validation.feature @@ -0,0 +1,19 @@ +@e2e_group_2 @skip +Feature: Unified mode configuration validation + + Background: + Given The service is started locally + And The system is in default state + And the Lightspeed stack configuration directory is "tests/e2e/configuration/unified-mode" + + + Scenario: inference.providers together with library_client_config_path fails at load + Given The service uses the lightspeed-stack-invalid-providers-and-legacy.yaml configuration + When configuration validation is attempted for the active configuration + Then the validation error contains --migrate-config + + + Scenario: llama_stack.config together with library_client_config_path fails at load + Given The service uses the lightspeed-stack-invalid-config-and-legacy.yaml configuration + When configuration validation is attempted for the active configuration + Then the validation error contains --migrate-config diff --git a/tests/e2e/test_list.txt b/tests/e2e/test_list.txt index 8d6dd7fa8..8747bac96 100644 --- a/tests/e2e/test_list.txt +++ b/tests/e2e/test_list.txt @@ -32,3 +32,8 @@ features/tls-ca.feature features/tls-mtls.feature features/tls-tlsv13.feature features/opentelemetry.feature +features/unified-mode-boot.feature +features/unified-mode-legacy.feature +features/unified-mode-validation.feature +features/unified-mode-migration.feature +features/unified-mode-synthesis.feature \ No newline at end of file