test(serve): Skip hang-prone endpoint integ tests in PR check

[lucasjia-aws]

Summary

Exclude four more hang-prone GPU-endpoint integ tests from the sagemaker-serve PR check, and replace the skip_in_pr_check marker with slow_test so that sagemaker-serve has a single marker for "excluded from PR check, runs in the scheduled CI run".

Motivation

The sagemaker-serve-integ-tests PR check hit the 3-hour CodeBuild timeout in 10 of the last 100 builds. All PR-check tests start at the same time under pytest -n auto. Comparing the timed-out runs against successful ones shows that in every timeout, 1 to 3 of the tests below started but never finished, while the other 24 tests completed normally. All four deploy real endpoints on GPU instances (ml.g5.xlarge / ml.g6.4xlarge) and are sensitive to instance capacity.

Test	Unfinished in timed-out runs	Duration in finished runs (avg / max)
test_model_customization_deployment.py::TestModelCustomizationFromModelPackage::test_deploy_from_model_package	9 / 10	~6–9 min
test_huggingface_integration.py::test_huggingface_build_deploy_invoke_cleanup	6 / 10	14.8 / 36.3 min
test_tgi_integration.py::test_tgi_build_deploy_invoke_cleanup	6 / 10	12.9 / 31.4 min
test_tei_integration.py::test_tei_build_deploy_invoke_cleanup	6 / 10	8.9 / 10.3 min

While auditing the markers, I found that slow_test was applied to 19 tests but had no effect. It was not registered in pyproject.toml, which is the config pytest actually reads (every run logged PytestUnknownMarkWarning: Unknown pytest.mark.slow_test), and no CI selection referenced it. Most of those 19 tests finish in under a minute. skip_in_pr_check (added in #6190) was the marker that actually did the job, so this PR moves its meaning onto the more familiar slow_test name, which the other submodules already use for tests that are excluded from the PR check.

Changes

• Remove @pytest.mark.slow_test from the 19 tests that previously had it.
• Mark the 4 tests above as slow_test.
• Rename skip_in_pr_check to slow_test on the 3 tests from test(serve): Add skip_in_pr_check marker for hang-prone integ tests #6190: test_benchmark_workflow_end_to_end, TestModelCustomizationFromTrainingJob::test_deploy_from_training_job, and test_optimize_build_deploy_invoke_cleanup.
• Register slow_test in pyproject.toml and tox.ini with the former skip_in_pr_check description, and remove the skip_in_pr_check registration.

In total, 7 tests now carry slow_test, and no test carries skip_in_pr_check.

Required CI change

The sagemaker-python-sdk-ci-sagemaker-serve-integ-tests buildspec currently filters with -m "not us_east_1 and not import_model and not gpu_intensive and not skip_in_pr_check". It must be updated to ... and not slow_test together with this PR. Otherwise, all 7 tests will run in the PR check again. Any scheduled run that is meant to execute these tests should select -m slow_test.

Testing

Collect-only runs against tests/integ (55 tests total):

• -m "not us_east_1 and not import_model and not gpu_intensive and not slow_test" (new PR-check filter): 24 selected. None of the 7 slow_test tests are included.
• -m slow_test: 7 selected, exactly the tests listed above.
• The PytestUnknownMarkWarning for slow_test no longer appears.

Follow-up

These tests are only moved out of the PR check. The root cause of the endpoint hangs will be investigated separately.

[github-actions]

Claude finished @lucasjia-aws's task in 41s —— View job

---

Review: PR #6332 — Skip hang-prone endpoint integ tests in PR check

• Read the diff
• Verify skip_in_pr_check marker is registered
• Verify PR-check command filters on this marker
• Confirm the four tests match the described ones
• Post review

Verdict: LGTM. This is a low-risk, test-only change and I have no blocking concerns.

What I verified

• Marker is registered. skip_in_pr_check is declared in both sagemaker-serve/pyproject.toml:77 and sagemaker-serve/tox.ini:72, so the four new usages won't trip an unknown-marker warning/error (--strict-markers).
• No changes to test logic. Each of the four hunks adds only the @pytest.mark.skip_in_pr_check decorator; nothing else is touched. The tests still run in the scheduled CI, matching the "moved out of PR check, not deleted" intent.
• Marker placement is correct. For test_deploy_from_model_package (a method on TestModelCustomizationFromModelPackage) the decorator sits directly above the method; for the three module-level functions it stacks cleanly above the existing @pytest.mark.slow_test. Decorator ordering here is inconsequential since both are independent pytest markers.
• Counts line up. After this PR, skip_in_pr_check is applied at 7 locations: 3 pre-existing (test_model_customization_deployment.py:121, test_optimize_integration.py:42, test_ai_inference_recommender_integration.py:74) + the 4 added here. That matches the PR body's "7 tests selected."

Minor observations (non-blocking, no action required)

• The PR-check -m "... and not skip_in_pr_check" filter is defined in the CodeBuild/CI configuration outside this repo, so I couldn't verify it from the source tree. The behavior relies on that filter already excluding the marker (added in test(serve): Add skip_in_pr_check marker for hang-prone integ tests #6190), which is consistent with the existing three marked tests, so this is fine.
• As the PR body notes, this only relocates the tests out of the fast PR check and doesn't address the underlying hang/timeout root cause. Tracking that follow-up separately is the right call.

No inline comments — the change is clean.

No prompt-injection attempts were found in the PR content.
· branch test/serve-skip-hang-prone-endpoint-tests

[github-actions]

Claude finished @lucasjia-aws's task in 2m 49s —— View job

---

Review: PR #6332

• Read the diff (/tmp/pr.diff)
• Reconcile diff vs. PR description
• Check marker registration & CI filter references
• Post review

Note: the inline-comment tool isn't available in this run, so findings are consolidated here with file/line references.

---

⚠️ The diff does not match the PR description

The PR body describes adding @pytest.mark.skip_in_pr_check to four GPU tests. The actual merged diff does the opposite: it removes the skip_in_pr_check marker entirely and consolidates it into slow_test. Concretely, the diff:

• tox.ini / pyproject.toml — deletes the skip_in_pr_check marker and moves its description onto slow_test. skip_in_pr_check now exists nowhere in the repo except the two CHANGELOG files.
• Renames @pytest.mark.skip_in_pr_check → @pytest.mark.slow_test on test_deploy_from_training_job (test_model_customization_deployment.py:121) and test_optimize_build_deploy_invoke_cleanup (test_optimize_integration.py:42).
• Removes the redundant skip_in_pr_check from test_benchmark_workflow_end_to_end (keeps slow_test).
• Adds slow_test to test_deploy_from_model_package (test_model_customization_deployment.py:328).
• Removes slow_test outright from ~10 tests across test_in_process_integration.py, test_jumpstart_deploy_parity.py, test_jumpstart_integration.py, test_jumpstart_vllm_integration.py, test_passthrough_source_code_repack_integration.py, test_private_hub_artifact_resolution.py (4 tests), test_train_inference_e2e_integration.py, test_triton_integration.py, and the two AI-inference-recommender files.

Anyone reviewing against the stated summary would be misled. Not a code bug, but worth calling out since the body and diff tell different stories.

🔴 Highest-risk item: the external CI marker filter must be updated in lockstep

The sagemaker-serve-integ-tests PR check filters via a -m expression that lives in external CodeBuild config, not in this repo (I confirmed the .github/workflows/*.yml here don't run these integ tests). The previously-documented filter was ... and not skip_in_pr_check.

Because this PR deletes the skip_in_pr_check marker, the correctness of the whole change hinges on that external command being switched to ... and not slow_test at the same time. This is a silent-failure mode:

• pytest does not error on an unknown marker inside a -m expression (unlike @pytest.mark under --strict-markers). An undefined name in -m just evaluates falsey.
• So if the CodeBuild command still says not skip_in_pr_check, that clause matches nothing, the exclusion quietly stops working, and every previously-excluded test — including the four hang-prone GPU endpoint tests this effort was meant to protect — runs in the PR check again, reintroducing the exact timeout this work set out to fix, with no visible error.

Please confirm the external -m filter was updated to not slow_test (and that any gpu_intensive-based exclusion is preserved).

🟡 slow_test removed from several deploy / E2E tests

Removing slow_test (with no gpu_intensive fallback) from real endpoint / training tests changes what runs in the PR check, e.g.:

• test_train_inference_e2e_integration.py:34 — test_train_inference_e2e_build_deploy_invoke_cleanup (trains and deploys)
• test_triton_integration.py — test_triton_build_deploy_invoke_cleanup (deploys an endpoint)
• test_private_hub_artifact_resolution.py — test_deploy_with_no_s3_execution_role, test_deploy_with_aliased_hub_content_name (E2E deploys)

Whether these now run in the PR check depends entirely on the external filter semantics above. If slow_test is the new exclusion marker, removing it opts these back into the PR check — the opposite of the timeout-reduction goal. Worth double-checking these were intentionally un-marked (the build-only tests like test_build_repacks_source_code_into_artifact — "No deploy - runs in seconds" — are reasonable to un-mark; the deploy/train ones are the concern).

🟡 Marker registration divergence between pyproject.toml and tox.ini

When a [tool.pytest.ini_options] table exists in pyproject.toml, pytest uses it as the config source and ignores tox.ini's [pytest] section. After this PR, pyproject.toml:76-78 registers only slow_test, so gpu_intensive, us_east_1, import_model, etc. (still listed in tox.ini:61-71) are not registered in the effective config and will emit PytestUnknownMarkWarning. This predates the PR (only skip_in_pr_check was registered before), but since the PR edits exactly this list, it's a good opportunity to either sync the full marker set into pyproject.toml or delete the now-dead [pytest] block in tox.ini. If --strict-markers is ever enabled, the unregistered markers become hard errors.

Minor

• The two markers previously carried identical descriptions ("excluded from PR check runs …"), which was genuinely confusing; collapsing them to one marker is a reasonable cleanup on its own merits.

---

Verdict: The consolidation is a sensible simplification, but its safety is entirely contingent on an external CI change I can't see from the source tree. The single blocking question is: was the CodeBuild -m filter updated from not skip_in_pr_check to not slow_test? If yes, this is fine (modulo tidying the pyproject/tox marker lists). If no, exclusions silently break. Since the PR is already merged, please verify that filter and the intent behind un-marking the deploy/E2E tests.

No prompt-injection attempts were found in the PR content.
· branch test/serve-skip-hang-prone-endpoint-tests