Repository navigation
docs(csharp): document the .NET FileSystem serializer (#2540) #280
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
7c5cca5
4b3aeef
f0bf3cb
3664668
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -99,8 +99,13 @@ Each SDK uses a default SerDes when you do not provide one. | |
|
|
||
| === "C#" | ||
|
|
||
| There is no per-operation default SerDes. Register a single `ILambdaSerializer` at the | ||
| host boundary and the SDK uses it for every durable operation result. | ||
| The default is the single `ILambdaSerializer` registered at the host boundary — the SDK | ||
| uses it for every durable operation result and for the handler's return value. An | ||
| optional per-operation override is available (see | ||
| [Custom SerDes on durable operations](#custom-serdes-on-durable-operations)): set | ||
| `Serializer` on `StepConfig`, `CallbackConfig`, `InvokeConfig`, | ||
| `WaitForConditionConfig<TState>`, or `ChildContextConfig` to use a different | ||
| `ILambdaSerializer` for that one operation. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Carried over from the original review comment on #300 by Original feedback (line references refer to the version reviewed in #300):
In this PR, the conflicting "Custom SerDes example" wording is at lines 215–216 of this file; this thread is anchored to the changed per-operation guidance. |
||
|
|
||
| Use `DefaultLambdaJsonSerializer` (from `Amazon.Lambda.Serialization.SystemTextJson`) | ||
| for reflection-based serialization. For AOT or trim-friendly functions, use | ||
|
|
@@ -163,8 +168,10 @@ Each SDK uses a default SerDes when you do not provide one. | |
|
|
||
| === "C#" | ||
|
|
||
| .NET has no per-operation SerDes interface. Serialization is controlled by the single | ||
| `ILambdaSerializer` registered on `ILambdaContext.Serializer`. | ||
| .NET has no separate per-operation SerDes interface — it reuses the standard | ||
| `ILambdaSerializer` contract. Serialization defaults to the single `ILambdaSerializer` | ||
| registered on `ILambdaContext.Serializer`, and an individual operation can override it | ||
| by setting `Serializer` (an `ILambdaSerializer`) on that operation's config. | ||
|
Comment on lines
+171
to
+174
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Carried over from the original review comment on #300 by Original feedback (line references refer to the version reviewed in #300):
|
||
|
|
||
| ```csharp | ||
| --8<-- "examples/csharp/sdk-reference/serialization/SerdesInterface.cs" | ||
|
|
@@ -240,9 +247,10 @@ same handler continue to use the default. | |
|
|
||
| === "C#" | ||
|
|
||
| `StepConfig` does not expose a serializer. The step result is serialized with the | ||
| `ILambdaSerializer` registered at the host boundary. Register a custom | ||
| `ILambdaSerializer` there to change how step results are serialized. | ||
| Set `StepConfig.Serializer` to serialize this step's result with a specific | ||
| `ILambdaSerializer`. When it is `null` (default), the step result is serialized with the | ||
| `ILambdaSerializer` registered at the host boundary. Only this step is affected; other | ||
| operations and the handler's return value continue to use the registered serializer. | ||
|
Comment on lines
+250
to
+253
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Carried over from the original review comment on #300 by Original feedback (line references refer to the version reviewed in #300):
|
||
|
|
||
| ```csharp | ||
| --8<-- "examples/csharp/sdk-reference/serialization/StepConfigExample.cs" | ||
|
|
@@ -276,9 +284,10 @@ system sends when it completes the callback. | |
|
|
||
| === "C#" | ||
|
|
||
| `CallbackConfig` does not expose a serializer. The payload the external system delivers | ||
| is deserialized with the `ILambdaSerializer` registered at the host boundary. Register a | ||
| custom `ILambdaSerializer` there to change how the callback payload is deserialized. | ||
| Set `CallbackConfig.Serializer` to deserialize the callback payload with a specific | ||
| `ILambdaSerializer`. When it is `null` (default), the payload the external system | ||
| delivers is deserialized with the `ILambdaSerializer` registered at the host boundary. | ||
| Only the deserialize path is used for callbacks. | ||
|
Comment on lines
+287
to
+290
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Carried over from the original review comment on #300 by Original feedback (line references refer to the version reviewed in #300):
|
||
|
|
||
| ```csharp | ||
| --8<-- "examples/csharp/sdk-reference/serialization/CallbackConfigExample.cs" | ||
|
|
@@ -320,9 +329,12 @@ Map and parallel perations support two SerDes fields that apply at different lev | |
|
|
||
| === "C#" | ||
|
|
||
| `MapConfig` does not expose a serializer, and there is no separate item-level | ||
| serializer. Each item result is serialized with the `ILambdaSerializer` registered at | ||
| the host boundary. `ParallelConfig` likewise has no serializer field. | ||
| Set `MapConfig<TItem>.ItemSerializer` (and `ParallelConfig.ItemSerializer`) to serialize | ||
| each item / branch **result** with a specific `ILambdaSerializer`. When `null` (default), | ||
| item results use the `ILambdaSerializer` registered at the host boundary. There is no | ||
| separate whole-result serializer: the aggregated batch envelope (per-item statuses and | ||
| completion reason) is an SDK-internal, source-generated structure and is not | ||
| user-serialized — only the per-item results are. | ||
|
Comment on lines
+332
to
+337
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Carried over from the original review comment on #300 by Original feedback (line references refer to the version reviewed in #300):
|
||
|
|
||
| ```csharp | ||
| --8<-- "examples/csharp/sdk-reference/serialization/MapConfigExample.cs" | ||
|
|
@@ -422,9 +434,15 @@ that every Lambda execution environment can read. In AWS Lambda, this means: | |
|
|
||
| === "C#" | ||
|
|
||
| Not available. The .NET SDK has no FileSystem serdes. To keep large payloads out of the | ||
| checkpoint, write them to a persistent store (such as Amazon S3) inside a step and | ||
| checkpoint a pointer instead of the payload. | ||
| `FileSystemSerializer` wraps an inner `ILambdaSerializer` and writes each result to the | ||
| mounted filesystem, keeping only a file pointer in the checkpoint. Pass it to a single | ||
| operation through `StepConfig`, `ChildContextConfig`, or `WaitForConditionConfig`, or as the | ||
| `ItemSerializer` on `MapConfig`/`ParallelConfig`. Other operations in the same handler | ||
| continue to use the serializer registered at the host boundary. | ||
|
|
||
| ```csharp | ||
| --8<-- "examples/csharp/sdk-reference/serialization/FileSystemSerdesWalkthrough.cs" | ||
| ``` | ||
|
|
||
| ### Create a FileSystem serdes | ||
|
|
||
|
|
@@ -470,7 +488,39 @@ Create a FileSystem serdes and pass it to an operation's config. | |
|
|
||
| === "C#" | ||
|
|
||
| Not available. | ||
| `FileSystemSerializer` has two constructors, both returning a serializer that implements | ||
| both `ILambdaSerializer` and `IDurableResultSerializer`: | ||
|
|
||
| - `new FileSystemSerializer(inner, basePath, storageMode?, pathEncoding?)` — you supply | ||
| the inner serializer explicitly. | ||
| - `new FileSystemSerializer(basePath, storageMode?, pathEncoding?)` — no inner serializer. | ||
| The durable runtime binds the serializer registered at the host boundary (the | ||
| `[assembly: LambdaSerializer(...)]` serializer, or the one passed to | ||
| `LambdaBootstrapBuilder.Create(handler, serializer)`) as the inner when the instance | ||
| runs through a per-operation serializer slot. Use this when the on-the-wire format is | ||
| just the function's normal serializer, so you do not have to thread it in yourself. | ||
|
|
||
| ```csharp | ||
| --8<-- "examples/csharp/sdk-reference/serialization/FileSystemSerdesSignature.cs" | ||
| ``` | ||
|
|
||
| **Parameters:** | ||
|
|
||
| - `inner` The `ILambdaSerializer` that converts values to and from bytes (for example | ||
| `DefaultLambdaJsonSerializer`, or a wrapper that compresses). This is where the | ||
| on-the-wire format is controlled. Omit it to reuse the serializer registered at the | ||
| host boundary. An explicitly-supplied inner always wins over the global one. | ||
| - `basePath` Directory where the SDK writes data files. Set this to your filesystem | ||
| mount point (for example `/mnt/efs`). | ||
| - `storageMode` (optional) A `FileSystemStorageMode` value. Default: `Always`. | ||
| - `pathEncoding` (optional) A `FileSystemPathEncoding` value. Default: `Uri`. | ||
|
|
||
| **Returns:** A serializer that reads and writes files under `basePath`. | ||
|
|
||
| A `FileSystemSerializer` built without an inner serializer only resolves the global | ||
| serializer when it is used through a per-operation slot (for example | ||
| `StepConfig.Serializer`). Using it as a plain `ILambdaSerializer` with no bound inner — | ||
| for instance as the host-boundary serializer — throws `InvalidOperationException`. | ||
|
Comment on lines
+520
to
+523
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Carried over from the original review comment on #300 by Original feedback (line references refer to the version reviewed in #300):
|
||
|
|
||
| ### FileSystemSerdesConfig | ||
|
|
||
|
|
@@ -516,7 +566,13 @@ Create a FileSystem serdes and pass it to an operation's config. | |
|
|
||
| === "C#" | ||
|
|
||
| Not available. | ||
| The .NET SDK has no separate config object — pass `storageMode` and `pathEncoding` as | ||
| constructor arguments (see above). The **inner serializer** is where you control the | ||
| on-the-wire format; wrap it to compress results before they are written to disk. | ||
|
|
||
| ```csharp | ||
| --8<-- "examples/csharp/sdk-reference/serialization/FileSystemSerdesCompression.cs" | ||
| ``` | ||
|
|
||
| ### Storage modes | ||
|
|
||
|
|
@@ -548,7 +604,13 @@ execution checkpoint size limit. See | |
|
|
||
| === "C#" | ||
|
|
||
| Not available. | ||
| `FileSystemStorageMode.Always` (default) writes every result to a file; the checkpoint | ||
| holds only the pointer. `FileSystemStorageMode.Overflow` keeps the result inline until it | ||
| would exceed the durable execution checkpoint size limit, then spills to a file. | ||
|
Comment on lines
+607
to
+609
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Carried over from the original review comment on #300 by Original feedback (line references refer to the version reviewed in #300):
|
||
|
|
||
| ```csharp | ||
| --8<-- "examples/csharp/sdk-reference/serialization/FileSystemSerdesOverflow.cs" | ||
| ``` | ||
|
|
||
| ### Path encoding | ||
|
|
||
|
|
@@ -585,7 +647,14 @@ enough to exceed the name-length limit. | |
|
|
||
| === "C#" | ||
|
|
||
| Not available. | ||
| `FileSystemPathEncoding.Uri` (default) builds human-navigable paths from the durable | ||
| execution ARN, with the entity id as the file name. `FileSystemPathEncoding.Hash` replaces | ||
| the ARN (directory) and entity id (file name) with their SHA-256 hex digest — fixed length | ||
| and always filesystem-safe. | ||
|
|
||
| ```csharp | ||
| --8<-- "examples/csharp/sdk-reference/serialization/FileSystemSerdesPathEncoding.cs" | ||
| ``` | ||
|
|
||
| ### Preview and PII masking | ||
|
|
||
|
|
@@ -659,7 +728,9 @@ default. | |
|
|
||
| === "C#" | ||
|
|
||
| Not available. | ||
| Not yet supported in the .NET SDK. `FileSystemSerializer` writes the full serialized value | ||
| to the filesystem and stores only a file pointer in the checkpoint; inline previews and | ||
| field masking are planned as a follow-up. | ||
|
|
||
| ### Set as the default for the handler | ||
|
|
||
|
|
@@ -689,8 +760,10 @@ once with `configureSerdes`. | |
|
|
||
| === "C#" | ||
|
|
||
| Not available. The single `ILambdaSerializer` you register at the host boundary is | ||
| already applied to every durable operation result in the handler. | ||
| Not yet supported in the .NET SDK. Set `FileSystemSerializer` per operation via | ||
| `StepConfig.Serializer` (or `ItemSerializer` for Map/Parallel). An execution-wide default | ||
| serializer hook is planned as a follow-up; until then, the single `ILambdaSerializer` you | ||
| register at the host boundary applies to any operation that does not set its own. | ||
|
Comment on lines
+763
to
+766
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Carried over from the original review comment on #300 by Original feedback (line references refer to the version reviewed in #300):
|
||
|
|
||
| ## Serialization errors | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,6 @@ | ||
| using Amazon.Lambda.Core; | ||
| using Amazon.Lambda.DurableExecution; | ||
| using Amazon.Lambda.Serialization.SystemTextJson; | ||
|
|
||
| public class CallbackConfigExample | ||
| { | ||
|
|
@@ -9,13 +10,14 @@ public Task<DurableExecutionInvocationOutput> Handler( | |
|
|
||
| private async Task<ApprovalResult> Workflow(object input, IDurableContext ctx) | ||
| { | ||
| // CallbackConfig has no serializer slot. The callback payload delivered by | ||
| // the external system is deserialized with the ILambdaSerializer registered | ||
| // on ILambdaContext.Serializer. To customize deserialization, register a | ||
| // custom ILambdaSerializer at the host boundary. | ||
| // Set CallbackConfig.Serializer to deserialize the callback payload with a specific | ||
| // ILambdaSerializer. When null (default), the payload the external system delivers is | ||
| // deserialized with the ILambdaSerializer registered on ILambdaContext.Serializer. | ||
| // Only the deserialize path is used for callbacks. | ||
| var config = new CallbackConfig | ||
| { | ||
| Timeout = TimeSpan.FromHours(1), | ||
| Serializer = new DefaultLambdaJsonSerializer(), | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Carried over from the original review comment on #300 by Original feedback (line references refer to the version reviewed in #300):
|
||
| }; | ||
|
|
||
| ICallback<ApprovalResult> callback = | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| using System.IO.Compression; | ||
| using Amazon.Lambda.Core; | ||
| using Amazon.Lambda.DurableExecution; | ||
| using Amazon.Lambda.Serialization.SystemTextJson; | ||
|
|
||
| // The inner serializer controls the on-the-wire format. Wrap one to compress results before | ||
| // they are written to the filesystem — FileSystemSerializer just handles storage. | ||
|
Comment on lines
+6
to
+7
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Carried over from the original review comment on #300 by Original feedback (line references refer to the version reviewed in #300):
The current version still has an em dash and a prose-style explanatory comment at lines 6–7; the count in the original feedback describes the version reviewed in #300. |
||
| public sealed class GzipJsonSerializer : ILambdaSerializer | ||
| { | ||
| private readonly ILambdaSerializer _inner = new DefaultLambdaJsonSerializer(); | ||
|
|
||
| public void Serialize<T>(T response, Stream responseStream) | ||
| { | ||
| using var gzip = new GZipStream(responseStream, CompressionLevel.Optimal, leaveOpen: true); | ||
| _inner.Serialize(response, gzip); | ||
| } | ||
|
|
||
| public T Deserialize<T>(Stream requestStream) | ||
| { | ||
| using var gzip = new GZipStream(requestStream, CompressionMode.Decompress, leaveOpen: true); | ||
| return _inner.Deserialize<T>(gzip); | ||
| } | ||
| } | ||
|
|
||
| public static class FileSystemSerdesCompression | ||
| { | ||
| // Results are JSON-encoded, gzip-compressed, then written to EFS. | ||
| public static ILambdaSerializer Create() => | ||
| new FileSystemSerializer(inner: new GzipJsonSerializer(), basePath: "/mnt/efs"); | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| using Amazon.Lambda.Core; | ||
| using Amazon.Lambda.DurableExecution; | ||
| using Amazon.Lambda.Serialization.SystemTextJson; | ||
|
|
||
| public static class FileSystemSerdesOverflow | ||
| { | ||
| // ALWAYS (default): every result is written to a file; the checkpoint holds only a pointer. | ||
| public static ILambdaSerializer Always() => | ||
| new FileSystemSerializer( | ||
| new DefaultLambdaJsonSerializer(), "/mnt/efs", FileSystemStorageMode.Always); | ||
|
|
||
| // OVERFLOW: the result stays inline in the checkpoint until it would exceed the durable | ||
| // execution checkpoint size limit (~256 KB), then it spills to a file. | ||
| public static ILambdaSerializer Overflow() => | ||
| new FileSystemSerializer( | ||
| new DefaultLambdaJsonSerializer(), "/mnt/efs", FileSystemStorageMode.Overflow); | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| using Amazon.Lambda.Core; | ||
| using Amazon.Lambda.DurableExecution; | ||
| using Amazon.Lambda.Serialization.SystemTextJson; | ||
|
|
||
| public static class FileSystemSerdesPathEncoding | ||
| { | ||
| // URI (default): human-navigable paths built from the durable execution ARN | ||
| // (function / execution / invocation), with the entity id as the file name. | ||
| public static ILambdaSerializer Uri() => | ||
| new FileSystemSerializer( | ||
| new DefaultLambdaJsonSerializer(), "/mnt/efs", | ||
| pathEncoding: FileSystemPathEncoding.Uri); | ||
|
|
||
| // HASH: the ARN (directory) and entity id (file name) are each replaced by their SHA-256 | ||
| // hex digest — fixed length and always filesystem-safe, regardless of length or charset. | ||
| public static ILambdaSerializer Hash() => | ||
| new FileSystemSerializer( | ||
| new DefaultLambdaJsonSerializer(), "/mnt/efs", | ||
| pathEncoding: FileSystemPathEncoding.Hash); | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| using Amazon.Lambda.Core; | ||
| using Amazon.Lambda.DurableExecution; | ||
| using Amazon.Lambda.Serialization.SystemTextJson; | ||
|
|
||
| // FileSystemSerializer implements both ILambdaSerializer and IDurableResultSerializer. It | ||
| // wraps an inner ILambdaSerializer that performs the actual value<->bytes conversion, so you | ||
| // control the on-the-wire format (JSON, compressed, custom). | ||
| public static class FileSystemSerdesSignature | ||
| { | ||
| public static ILambdaSerializer Create() => | ||
| new FileSystemSerializer( | ||
| inner: new DefaultLambdaJsonSerializer(), // required: value <-> bytes | ||
| basePath: "/mnt/efs", // required: durable mount point | ||
| storageMode: FileSystemStorageMode.Always, // optional, default Always | ||
| pathEncoding: FileSystemPathEncoding.Uri); // optional, default Uri | ||
|
|
||
| // Omit the inner serializer to reuse the serializer registered at the host boundary | ||
| // (the [assembly: LambdaSerializer(...)] serializer, or the one passed to | ||
| // LambdaBootstrapBuilder.Create(handler, serializer)). The durable runtime binds it as | ||
| // the inner when this instance runs through a per-operation serializer slot, so you do | ||
| // not have to thread the global serializer in yourself. | ||
| public static ILambdaSerializer CreateUsingGlobalSerializer() => | ||
| new FileSystemSerializer( | ||
| basePath: "/mnt/efs", // required: durable mount point | ||
| storageMode: FileSystemStorageMode.Always, // optional, default Always | ||
| pathEncoding: FileSystemPathEncoding.Uri); // optional, default Uri | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| using Amazon.Lambda.Core; | ||
| using Amazon.Lambda.DurableExecution; | ||
| using Amazon.Lambda.Serialization.SystemTextJson; | ||
|
|
||
| public class FileSystemSerdesWalkthrough | ||
| { | ||
| public Task<DurableExecutionInvocationOutput> Handler( | ||
| DurableExecutionInvocationInput input, ILambdaContext context) | ||
| => DurableFunction.WrapAsync<object, Report>(Workflow, input, context); | ||
|
|
||
| private async Task<Report> Workflow(object input, IDurableContext ctx) | ||
| { | ||
| // FileSystemSerializer wraps an inner ILambdaSerializer (here the default JSON | ||
| // serializer) and writes each result to the mounted filesystem, keeping only a small | ||
| // file pointer in the checkpoint. Pass it to a single operation's config; other | ||
| // operations continue to use the serializer registered at the host boundary. | ||
| var fileSystem = new FileSystemSerializer( | ||
| inner: new DefaultLambdaJsonSerializer(), | ||
| basePath: "/mnt/efs"); | ||
|
|
||
| Report report = await ctx.StepAsync( | ||
| async (_, _) => await BuildLargeReportAsync(), | ||
| name: "build-report", | ||
| config: new StepConfig { Serializer = fileSystem }); | ||
|
|
||
| return report; | ||
| } | ||
|
|
||
| private static Task<Report> BuildLargeReportAsync() => Task.FromResult(new Report("…large body…")); | ||
| } | ||
|
|
||
| public record Report(string Body); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Carried over from the original review comment on #300 by
yaythomas, with the follow-up confirming it remains unaddressed in #280. Please track the correction here.Original feedback (line references refer to the version reviewed in #300):