chore(tests): add LMI e2e suite and run-scoped shared capacity provider - #5465
Open
svozza wants to merge 23 commits into
Open
chore(tests): add LMI e2e suite and run-scoped shared capacity provider#5465svozza wants to merge 23 commits into
svozza wants to merge 23 commits into
Conversation
powertools-for-aws-oss-automation
Bot
added
the
size/XL
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
powertools-for-aws-oss-automation
Bot
added
size/XL
|
dreamorosi
reviewed
Jul 20, 2026
dreamorosi
reviewed
Jul 31, 2026
dreamorosi
left a comment
dreamorosi
left a comment
Contributor
There was a problem hiding this comment.
These are findings from a deep review pass, grouped as 2 blockers plus several smaller fixes. Happy to discuss any of them.
dreamorosi
requested changes
Jul 31, 2026
dreamorosi
left a comment
dreamorosi
left a comment
Contributor
There was a problem hiding this comment.
Apologies for the long wait - I just finally found some time to review this now.
Some review comments that I think are valid, let me know if you disagree and/or I misunderstood anything.
2 tasks
Contributor
|
The infrastructure-leak concern raised in review—cancelled cells leaving functions attached to the shared capacity provider and causing |
Contributor
|
The temporary orphan-cleanup experiment succeeded in the latest e2e run: all four LMI cells passed, teardown found and deleted all four deliberately orphaned function stacks, and both shared capacity provider stacks were then deleted successfully. The workflow was red overall because four non-LMI matrix cells failed, but the targeted LMI teardown path completed successfully. Before this is merge-ready, please:
I have resolved the review threads whose requested changes are now present. I am leaving the requested-changes review in place until the temporary commit and conflicts are cleared. |
Contributor
Author
|
Yeah, we're not finished yet, I think the no clobber change is causing unrelated tests to fail. I'm going to be doing multiple CI runs to diagnose this so we can't merge until that's done anyway. |
… Managed Instances Adds an opt-in (RUN_LMI_TESTS) e2e suite that provisions an ephemeral Lambda Managed Instances capacity provider (dual-stack IPv6 VPC, no NAT, CloudWatch Logs interface endpoint) and proves the InvokeStore-backed log attribute isolation across invocations multiplexed into the same execution environment via a module-scoped promise barrier. Testing utils changes: - TestLmiCapacityProvider construct + ./resources/capacity-provider export - ExtraTestProps.lmi option on TestNodejsFunction (memorySize>=2048 floor, $LATEST.PUBLISHED qualified output) - includeTailLogs opt-out on invoke helpers (Tail unsupported on LMI) Related to #5092, findings posted on the issue.
… CloudWatch polling The handler tees process.stdout.write (the Logger's production write path) and returns its own log lines in the response payload, making log collection deterministic — no FilterLogEvents polling, no ingestion latency, no CloudWatch client in the test. Captured lines are filtered by the InvokeStore-scoped invocationKey rather than function_request_id: addContext stores the Lambda context in instance state, so under LMI multiplexing the request id stamped on log lines can belong to a different invocation. Tracked separately as a bug (see lmi-request-id-bug-handoff.md); once fixed, the filter should flip back to function_request_id as a regression check.
…ogs by request id With the lambda context now scoped per invocation under LMI (#5430), the handler selects its own log lines by function_request_id — an independent per-invocation attribute — and the test asserts the InvokeStore-scoped invocationKey on those lines, making the suite a regression check for both the context scoping and appendKeys isolation. The suite adds ~4 minutes to the logger e2e cell, inside the agreed budget for running on every e2e dispatch, so the RUN_LMI_TESTS gate and workflow input are removed.
The e2e suites are I/O-bound (waiting on CloudFormation), but vitest sizes its worker pool from CPU cores, so on 2-core CI runners only 3 of the 6 logger e2e files ran concurrently and the rest queued — each then paying its own stack deploy after waiting. With 8 workers all suites deploy their stacks up front and the cell duration approaches the slowest single suite (~6 min measured) instead of a serialized ~8 min. CloudFormation read-API pressure from the extra concurrent stack monitors is bounded by the existing DescribeStackEvents polling patch in the testing package (10s interval per stack).
With all 36 matrix cells sharing one account, the extra concurrent stack operations from 8-worker logger cells pushed the account-wide CloudFormation API rate over the edge: three unrelated cells failed with 'Throttling: Rate exceeded' on stack deploys. Back to default worker sizing; cell-duration work moves to the run-scoped shared capacity provider follow-up, which removes per-cell VPC/CP stacks entirely instead of racing them.
Instead of every LMI suite provisioning its own capacity provider + VPC, a workflow run can deploy one shared capacity provider per architecture up front and pass its ARN to each suite via LMI_CAPACITY_PROVIDER_ARN. - add packages/testing/src/lmi/cli.ts with deploy/destroy commands that manage a run-scoped LmiShared-<runId>-<arch> stack and print the capacity provider ARN on stdout - TestNodejsFunction accepts a capacity provider ARN string and attaches the function via L1 CfnFunction.capacityProviderConfig (an imported capacity provider has no addFunction) - ExtraTestProps.lmi.capacityProvider widened to CapacityProvider | string - TestLmiCapacityProvider ctor loosened to Pick<TestStack, 'stack'> so the CLI can build the stack standalone - logger lmi.test.ts reads LMI_CAPACITY_PROVIDER_ARN, falling back to an ephemeral per-suite capacity provider when unset
The --run-id argument flows into a CloudFormation stack name and, via join(tmpdir(), ...), into the assembly output path that is later read back. A crafted value (e.g. containing ../) could escape tmpdir(), so restrict it to the alphanumerics-and-hyphens set CloudFormation already requires for stack names and reject anything else at the boundary.
The path-injection scanner's taint analysis doesn't recognise the run-id regex guard as sanitisation, and its guidance is to validate the constructed path before touching the file system. Resolve the assembly output directory and assert it stays within os.tmpdir() before it is written to or read back, keeping the run-id validation as the input-side guard for defence in depth.
The constructor's cognitive complexity hit 17 (limit 15) once it carried the nested ARN-vs-construct capacity-provider branching. Move that logic into two private methods so the constructor keeps a flat structure and the two attachment strategies read independently. No behaviour change.
…hared capacity provider
Two TestStack instances synthesizing at once (the shared LMI capacity provider stacks, one per architecture, deployed concurrently) both run fromAssemblyBuilder, which by default temporarily replaces the global process.env with an immutable proxy for the synth window. Interleaved synth windows race on that swap. Pass clobberEnv: false; the assembly builder does not read the injected env, so opting out is safe.
A function attached to a shared capacity provider by ARN keeps the provider's ENIs in use, and that cross-stack link is invisible to CloudFormation, so a function stack orphaned by a cancelled or timed-out cell makes the provider-stack delete fail with DELETE_FAILED and leaks an EC2 fleet plus VPC. - embed an Lmi marker and the run id in the LMI function stack name so the teardown job can attribute and find orphans (see lmi/naming.ts) - teardown now sweeps leftover Lmi-<runId> function stacks and waits for them to be gone before deleting the provider stacks, and retries the provider teardown once to absorb throttling and ENI detachment - add timeout-minutes: 60 to the three LMI jobs so a hung deploy/cell can't keep EC2 fleets alive for the 6-hour default - fail the ARN resolution step when describe-stacks returns empty/None instead of silently falling back to per-suite provisioning Residual leak paths (workflow-level cancellation, DISABLE_TEARDOWN, LmiShared-local-*) are tracked as the scheduled sweeper in #5519.
The ARN attach path sets publishToLatestPublished on the L1 resource explicitly; the construct path relied on CDK's undocumented default. Set it on both so the two paths emit identical templates and the behaviour doesn't depend on an undocumented service default.
The option was added so the LMI suite could opt out of Tail logs (Tail is unsupported on capacity-provider functions), but the suite now invokes directly and collects logs from the response payload, leaving the option uncalled. Remove it and document the Tail constraint on invokeFunctionOnce so it isn't reintroduced without the guard.
Drop @aws-sdk/client-cloudwatch-logs (unused since log capture moved off CloudWatch polling) and declare @aws-sdk/client-lambda, which the LMI e2e suite imports directly and previously resolved only via workspace hoisting.
The progress lines used testName, which is run through generateTestUniqueName and so mislabels stacks created from an explicitly-named Stack object — the shared LMI provider stack logged as `LmiShared-24-x86-<uuid>-arm64` instead of its real `LmiShared-<runId>-<arch>` name. Log stack.stackName, the actual deployed name, which is correct for both generated and explicit names.
Contributor
Author
|
Thanks @dreamorosi — since you've already resolved the threads from the commits, just closing the loop with everything that landed: Blockers
Should-fixes
Validation
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an end-to-end test that validates Logger's
InvokeStore-backed log-attribute isolation on Lambda Managed Instances (LMI), where multiple invocations run concurrently in the same execution environment. Also introduces the testing infrastructure to support it, including a run-scoped shared capacity provider so LMI suites don't each provision their own VPC + capacity provider.Changes
packages/logger/tests/e2e/lmi.test.ts) that saturates a deliberately small fleet, forces concurrent invocations to multiplex into shared execution environments, and asserts each invocation's appended keys stay isolatedprocess.stdout.write(the production write path) and returning them in the response payload, making log collection deterministic — no CloudWatch polling or ingestion latencyInvokeStore-scopedinvocationKey, and additionally assert onfunction_request_idas a regression check now that the Lambda context is scoped per invocation under LMI (fix(logger): scope lambda context per invocation under LMI concurrency #5430)TestLmiCapacityProvider(dual-stack IPv6-only VPC, egress-only IGW, CloudWatch Logs interface endpoint — no NAT) and extendTestNodejsFunctionto attach a function to a capacity provider by construct or by ARNpackages/testing/src/lmi/cli.tswith deploy/destroy commands that manage a run-scopedLmiShared-<runId>-<arch>stack and print the capacity provider ARN, so a workflow run can provision one shared capacity provider per architecture up front and pass it to each suite viaLMI_CAPACITY_PROVIDER_ARN(suites fall back to an ephemeral per-suite provider when unset)RUN_LMI_TESTSgate and workflow input)Issue number: closes #5518
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.