Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 96 additions & 23 deletions docs/sdk-reference/state/serialization.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,13 @@ Each SDK uses a default SerDes when you do not provide one.

=== "C#"

There is no per-operation default SerDes. Register a single `ILambdaSerializer` at the
host boundary and the SDK uses it for every durable operation result.
The default is the single `ILambdaSerializer` registered at the host boundary — the SDK
uses it for every durable operation result and for the handler's return value. An
optional per-operation override is available (see
[Custom SerDes on durable operations](#custom-serdes-on-durable-operations)): set
`Serializer` on `StepConfig`, `CallbackConfig`, `InvokeConfig`,
`WaitForConditionConfig<TState>`, or `ChildContextConfig` to use a different
`ILambdaSerializer` for that one operation.
Comment on lines +102 to +108

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carried over from the original review comment on #300 by yaythomas, with the follow-up confirming it remains unaddressed in #280. Please track the correction here.

Original feedback (line references refer to the version reviewed in #300):

The C# tabs on this page state one fact five times: the registered serializer is the default, and a per-operation serializer overrides it. The statements are at lines 91-94, 132-138, 231-234, 247-248, and 331-334. Could the page state it once, here, and keep the later tabs to what each section adds? This list also leaves out ItemSerializer on MapConfig<TItem> and ParallelConfig, and the per-branch serializer on IDurableParallel.Branch.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carried over from the original review comment on #300 by yaythomas, with the follow-up confirming it remains unaddressed in #280. Please track the correction here.

Original feedback (line references refer to the version reviewed in #300):

This tab now says a per-operation override affects only the selected operation. The C# tab of "Custom SerDes example" (lines 284-285) still says "The custom serializer applies to every durable operation result, not a single operation." With StepConfig.Serializer and the other per-operation properties, that sentence is no longer true. Could that tab change in this PR too, so the page gives one answer?

In this PR, the conflicting "Custom SerDes example" wording is at lines 215–216 of this file; this thread is anchored to the changed per-operation guidance.


Use `DefaultLambdaJsonSerializer` (from `Amazon.Lambda.Serialization.SystemTextJson`)
for reflection-based serialization. For AOT or trim-friendly functions, use
Expand Down Expand Up @@ -163,8 +168,10 @@ Each SDK uses a default SerDes when you do not provide one.

=== "C#"

.NET has no per-operation SerDes interface. Serialization is controlled by the single
`ILambdaSerializer` registered on `ILambdaContext.Serializer`.
.NET has no separate per-operation SerDes interface — it reuses the standard
`ILambdaSerializer` contract. Serialization defaults to the single `ILambdaSerializer`
registered on `ILambdaContext.Serializer`, and an individual operation can override it
by setting `Serializer` (an `ILambdaSerializer`) on that operation's config.
Comment on lines +171 to +174

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carried over from the original review comment on #300 by yaythomas, with the follow-up confirming it remains unaddressed in #280. Please track the correction here.

Original feedback (line references refer to the version reviewed in #300):

Core 2.0.0 added IDurableResultSerializer, an optional interface for per-operation serializers. Its Serialize<T> and Deserialize<T> overloads receive a DurableSerializationContext, which has EntityId and DurableExecutionArn properties. The runtime calls these overloads when the configured serializer implements the interface. FileSystemSerializer implements it. So "no separate per-operation SerDes interface" is not accurate for 2.x. Could this tab document IDurableResultSerializer, the way the Java tab now documents SerDesContext?


```csharp
--8<-- "examples/csharp/sdk-reference/serialization/SerdesInterface.cs"
Expand Down Expand Up @@ -240,9 +247,10 @@ same handler continue to use the default.

=== "C#"

`StepConfig` does not expose a serializer. The step result is serialized with the
`ILambdaSerializer` registered at the host boundary. Register a custom
`ILambdaSerializer` there to change how step results are serialized.
Set `StepConfig.Serializer` to serialize this step's result with a specific
`ILambdaSerializer`. When it is `null` (default), the step result is serialized with the
`ILambdaSerializer` registered at the host boundary. Only this step is affected; other
operations and the handler's return value continue to use the registered serializer.
Comment on lines +250 to +253

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carried over from the original review comment on #300 by yaythomas, with the follow-up confirming it remains unaddressed in #280. Please track the correction here.

Original feedback (line references refer to the version reviewed in #300):

This uses passive voice (Voice and Tone). How about this?

Set StepConfig.Serializer to serialize this step's result with a specific ILambdaSerializer. When it is null (the default), the SDK serializes the step result with the ILambdaSerializer registered at the host boundary. Other operations and the handler's return value keep that serializer.


```csharp
--8<-- "examples/csharp/sdk-reference/serialization/StepConfigExample.cs"
Expand Down Expand Up @@ -276,9 +284,10 @@ system sends when it completes the callback.

=== "C#"

`CallbackConfig` does not expose a serializer. The payload the external system delivers
is deserialized with the `ILambdaSerializer` registered at the host boundary. Register a
custom `ILambdaSerializer` there to change how the callback payload is deserialized.
Set `CallbackConfig.Serializer` to deserialize the callback payload with a specific
`ILambdaSerializer`. When it is `null` (default), the payload the external system
delivers is deserialized with the `ILambdaSerializer` registered at the host boundary.
Only the deserialize path is used for callbacks.
Comment on lines +287 to +290

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carried over from the original review comment on #300 by yaythomas, with the follow-up confirming it remains unaddressed in #280. Please track the correction here.

Original feedback (line references refer to the version reviewed in #300):

This also uses passive voice. How about this?

Set CallbackConfig.Serializer to deserialize the callback payload with a specific ILambdaSerializer. When it is null, the SDK deserializes the payload with the ILambdaSerializer registered at the host boundary. The SDK never serializes a callback result, so it calls only Deserialize.


```csharp
--8<-- "examples/csharp/sdk-reference/serialization/CallbackConfigExample.cs"
Expand Down Expand Up @@ -320,9 +329,12 @@ Map and parallel perations support two SerDes fields that apply at different lev

=== "C#"

`MapConfig` does not expose a serializer, and there is no separate item-level
serializer. Each item result is serialized with the `ILambdaSerializer` registered at
the host boundary. `ParallelConfig` likewise has no serializer field.
Set `MapConfig<TItem>.ItemSerializer` (and `ParallelConfig.ItemSerializer`) to serialize
each item / branch **result** with a specific `ILambdaSerializer`. When `null` (default),
item results use the `ILambdaSerializer` registered at the host boundary. There is no
separate whole-result serializer: the aggregated batch envelope (per-item statuses and
completion reason) is an SDK-internal, source-generated structure and is not
user-serialized — only the per-item results are.
Comment on lines +332 to +337

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carried over from the original review comment on #300 by yaythomas, with the follow-up confirming it remains unaddressed in #280. Please track the correction here.

Original feedback (line references refer to the version reviewed in #300):

The last sentence here chains two colons, uses passive voice, and names an implementation detail ("source-generated"). How about this?

Set MapConfig<TItem>.ItemSerializer or ParallelConfig.ItemSerializer to serialize each item or branch result with a specific ILambdaSerializer. When it is null, item results use the serializer registered at the host boundary. The SDK writes the batch envelope (item statuses and the completion reason) itself, so there is no whole-result serializer.


```csharp
--8<-- "examples/csharp/sdk-reference/serialization/MapConfigExample.cs"
Expand Down Expand Up @@ -422,9 +434,15 @@ that every Lambda execution environment can read. In AWS Lambda, this means:

=== "C#"

Not available. The .NET SDK has no FileSystem serdes. To keep large payloads out of the
checkpoint, write them to a persistent store (such as Amazon S3) inside a step and
checkpoint a pointer instead of the payload.
`FileSystemSerializer` wraps an inner `ILambdaSerializer` and writes each result to the
mounted filesystem, keeping only a file pointer in the checkpoint. Pass it to a single
operation through `StepConfig`, `ChildContextConfig`, or `WaitForConditionConfig`, or as the
`ItemSerializer` on `MapConfig`/`ParallelConfig`. Other operations in the same handler
continue to use the serializer registered at the host boundary.

```csharp
--8<-- "examples/csharp/sdk-reference/serialization/FileSystemSerdesWalkthrough.cs"
```

### Create a FileSystem serdes

Expand Down Expand Up @@ -470,7 +488,39 @@ Create a FileSystem serdes and pass it to an operation's config.

=== "C#"

Not available.
`FileSystemSerializer` has two constructors, both returning a serializer that implements
both `ILambdaSerializer` and `IDurableResultSerializer`:

- `new FileSystemSerializer(inner, basePath, storageMode?, pathEncoding?)` — you supply
the inner serializer explicitly.
- `new FileSystemSerializer(basePath, storageMode?, pathEncoding?)` — no inner serializer.
The durable runtime binds the serializer registered at the host boundary (the
`[assembly: LambdaSerializer(...)]` serializer, or the one passed to
`LambdaBootstrapBuilder.Create(handler, serializer)`) as the inner when the instance
runs through a per-operation serializer slot. Use this when the on-the-wire format is
just the function's normal serializer, so you do not have to thread it in yourself.

```csharp
--8<-- "examples/csharp/sdk-reference/serialization/FileSystemSerdesSignature.cs"
```

**Parameters:**

- `inner` The `ILambdaSerializer` that converts values to and from bytes (for example
`DefaultLambdaJsonSerializer`, or a wrapper that compresses). This is where the
on-the-wire format is controlled. Omit it to reuse the serializer registered at the
host boundary. An explicitly-supplied inner always wins over the global one.
- `basePath` Directory where the SDK writes data files. Set this to your filesystem
mount point (for example `/mnt/efs`).
- `storageMode` (optional) A `FileSystemStorageMode` value. Default: `Always`.
- `pathEncoding` (optional) A `FileSystemPathEncoding` value. Default: `Uri`.

**Returns:** A serializer that reads and writes files under `basePath`.

A `FileSystemSerializer` built without an inner serializer only resolves the global
serializer when it is used through a per-operation slot (for example
`StepConfig.Serializer`). Using it as a plain `ILambdaSerializer` with no bound inner —
for instance as the host-boundary serializer — throws `InvalidOperationException`.
Comment on lines +520 to +523

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carried over from the original review comment on #300 by SilanHe, with the follow-up confirming it remains unaddressed in #280. Please track the correction here.

Original feedback (line references refer to the version reviewed in #300):

C# failure mode is half right. NotSupportedException is thrown when used as a plain ILambdaSerializer. InvalidOperationException is thrown only when the inner serializer is missing.


### FileSystemSerdesConfig

Expand Down Expand Up @@ -516,7 +566,13 @@ Create a FileSystem serdes and pass it to an operation's config.

=== "C#"

Not available.
The .NET SDK has no separate config object — pass `storageMode` and `pathEncoding` as
constructor arguments (see above). The **inner serializer** is where you control the
on-the-wire format; wrap it to compress results before they are written to disk.

```csharp
--8<-- "examples/csharp/sdk-reference/serialization/FileSystemSerdesCompression.cs"
```

### Storage modes

Expand Down Expand Up @@ -548,7 +604,13 @@ execution checkpoint size limit. See

=== "C#"

Not available.
`FileSystemStorageMode.Always` (default) writes every result to a file; the checkpoint
holds only the pointer. `FileSystemStorageMode.Overflow` keeps the result inline until it
would exceed the durable execution checkpoint size limit, then spills to a file.
Comment on lines +607 to +609

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carried over from the original review comment on #300 by yaythomas, with the follow-up confirming it remains unaddressed in #280. Please track the correction here.

Original feedback (line references refer to the version reviewed in #300):

In Overflow mode, the .NET serializer base64-encodes the serialized bytes into a {"data": ...} envelope. It then compares the envelope's size with 255 KiB. Base64 adds a third to the size. I tested 2.2.0: a 190 KiB string stayed inline, and a 192 KiB string went to a file. So inline storage ends near 191 KiB of serialized bytes, not at the checkpoint limit. The comment in FileSystemSerdesOverflow.cs (lines 12-13) also says "~256 KB". Could both state the 255 KiB envelope rule, the way the Java tab states its own?


```csharp
--8<-- "examples/csharp/sdk-reference/serialization/FileSystemSerdesOverflow.cs"
```

### Path encoding

Expand Down Expand Up @@ -585,7 +647,14 @@ enough to exceed the name-length limit.

=== "C#"

Not available.
`FileSystemPathEncoding.Uri` (default) builds human-navigable paths from the durable
execution ARN, with the entity id as the file name. `FileSystemPathEncoding.Hash` replaces
the ARN (directory) and entity id (file name) with their SHA-256 hex digest — fixed length
and always filesystem-safe.

```csharp
--8<-- "examples/csharp/sdk-reference/serialization/FileSystemSerdesPathEncoding.cs"
```

### Preview and PII masking

Expand Down Expand Up @@ -659,7 +728,9 @@ default.

=== "C#"

Not available.
Not yet supported in the .NET SDK. `FileSystemSerializer` writes the full serialized value
to the filesystem and stores only a file pointer in the checkpoint; inline previews and
field masking are planned as a follow-up.

### Set as the default for the handler

Expand Down Expand Up @@ -689,8 +760,10 @@ once with `configureSerdes`.

=== "C#"

Not available. The single `ILambdaSerializer` you register at the host boundary is
already applied to every durable operation result in the handler.
Not yet supported in the .NET SDK. Set `FileSystemSerializer` per operation via
`StepConfig.Serializer` (or `ItemSerializer` for Map/Parallel). An execution-wide default
serializer hook is planned as a follow-up; until then, the single `ILambdaSerializer` you
register at the host boundary applies to any operation that does not set its own.
Comment on lines +763 to +766

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carried over from the original review comment on #300 by yaythomas, with the follow-up confirming it remains unaddressed in #280. Please track the correction here.

Original feedback (line references refer to the version reviewed in #300):

"Not yet supported" and "is not supported" say the same thing, and "yet" promises future work. The Java tab gives the reason, and the same reason applies here: FileSystemSerializer throws NotSupportedException at the host boundary. How about this?

The FileSystem serializer cannot be the handler-wide default, because the host-boundary serializer also handles Lambda input and output, which have no operation identity. Set it per operation through StepConfig.Serializer or ItemSerializer.


## Serialization errors

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
using Amazon.Lambda.Core;
using Amazon.Lambda.DurableExecution;
using Amazon.Lambda.Serialization.SystemTextJson;

public class CallbackConfigExample
{
Expand All @@ -9,13 +10,14 @@ public Task<DurableExecutionInvocationOutput> Handler(

private async Task<ApprovalResult> Workflow(object input, IDurableContext ctx)
{
// CallbackConfig has no serializer slot. The callback payload delivered by
// the external system is deserialized with the ILambdaSerializer registered
// on ILambdaContext.Serializer. To customize deserialization, register a
// custom ILambdaSerializer at the host boundary.
// Set CallbackConfig.Serializer to deserialize the callback payload with a specific
// ILambdaSerializer. When null (default), the payload the external system delivers is
// deserialized with the ILambdaSerializer registered on ILambdaContext.Serializer.
// Only the deserialize path is used for callbacks.
var config = new CallbackConfig
{
Timeout = TimeSpan.FromHours(1),
Serializer = new DefaultLambdaJsonSerializer(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carried over from the original review comment on #300 by yaythomas, with the follow-up confirming it remains unaddressed in #280. Please track the correction here.

Original feedback (line references refer to the version reviewed in #300):

DefaultLambdaJsonSerializer is also the usual host-boundary serializer, so this override looks the same as the default. The Step and Map examples use CamelCaseLambdaJsonSerializer. Could this one use it too?

};

ICallback<ApprovalResult> callback =
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
using System.IO.Compression;
using Amazon.Lambda.Core;
using Amazon.Lambda.DurableExecution;
using Amazon.Lambda.Serialization.SystemTextJson;

// The inner serializer controls the on-the-wire format. Wrap one to compress results before
// they are written to the filesystem — FileSystemSerializer just handles storage.
Comment on lines +6 to +7

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Carried over from the original review comment on #300 by SilanHe, with the follow-up confirming it remains unaddressed in #280. Please track the correction here.

Original feedback (line references refer to the version reviewed in #300):

Four em-dashes, plus long comments that repeat the page prose. Both break the authoring guide.

The current version still has an em dash and a prose-style explanatory comment at lines 6–7; the count in the original feedback describes the version reviewed in #300.

public sealed class GzipJsonSerializer : ILambdaSerializer
{
private readonly ILambdaSerializer _inner = new DefaultLambdaJsonSerializer();

public void Serialize<T>(T response, Stream responseStream)
{
using var gzip = new GZipStream(responseStream, CompressionLevel.Optimal, leaveOpen: true);
_inner.Serialize(response, gzip);
}

public T Deserialize<T>(Stream requestStream)
{
using var gzip = new GZipStream(requestStream, CompressionMode.Decompress, leaveOpen: true);
return _inner.Deserialize<T>(gzip);
}
}

public static class FileSystemSerdesCompression
{
// Results are JSON-encoded, gzip-compressed, then written to EFS.
public static ILambdaSerializer Create() =>
new FileSystemSerializer(inner: new GzipJsonSerializer(), basePath: "/mnt/efs");
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
using Amazon.Lambda.Core;
using Amazon.Lambda.DurableExecution;
using Amazon.Lambda.Serialization.SystemTextJson;

public static class FileSystemSerdesOverflow
{
// ALWAYS (default): every result is written to a file; the checkpoint holds only a pointer.
public static ILambdaSerializer Always() =>
new FileSystemSerializer(
new DefaultLambdaJsonSerializer(), "/mnt/efs", FileSystemStorageMode.Always);

// OVERFLOW: the result stays inline in the checkpoint until it would exceed the durable
// execution checkpoint size limit (~256 KB), then it spills to a file.
public static ILambdaSerializer Overflow() =>
new FileSystemSerializer(
new DefaultLambdaJsonSerializer(), "/mnt/efs", FileSystemStorageMode.Overflow);
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
using Amazon.Lambda.Core;
using Amazon.Lambda.DurableExecution;
using Amazon.Lambda.Serialization.SystemTextJson;

public static class FileSystemSerdesPathEncoding
{
// URI (default): human-navigable paths built from the durable execution ARN
// (function / execution / invocation), with the entity id as the file name.
public static ILambdaSerializer Uri() =>
new FileSystemSerializer(
new DefaultLambdaJsonSerializer(), "/mnt/efs",
pathEncoding: FileSystemPathEncoding.Uri);

// HASH: the ARN (directory) and entity id (file name) are each replaced by their SHA-256
// hex digest — fixed length and always filesystem-safe, regardless of length or charset.
public static ILambdaSerializer Hash() =>
new FileSystemSerializer(
new DefaultLambdaJsonSerializer(), "/mnt/efs",
pathEncoding: FileSystemPathEncoding.Hash);
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
using Amazon.Lambda.Core;
using Amazon.Lambda.DurableExecution;
using Amazon.Lambda.Serialization.SystemTextJson;

// FileSystemSerializer implements both ILambdaSerializer and IDurableResultSerializer. It
// wraps an inner ILambdaSerializer that performs the actual value<->bytes conversion, so you
// control the on-the-wire format (JSON, compressed, custom).
public static class FileSystemSerdesSignature
{
public static ILambdaSerializer Create() =>
new FileSystemSerializer(
inner: new DefaultLambdaJsonSerializer(), // required: value <-> bytes
basePath: "/mnt/efs", // required: durable mount point
storageMode: FileSystemStorageMode.Always, // optional, default Always
pathEncoding: FileSystemPathEncoding.Uri); // optional, default Uri

// Omit the inner serializer to reuse the serializer registered at the host boundary
// (the [assembly: LambdaSerializer(...)] serializer, or the one passed to
// LambdaBootstrapBuilder.Create(handler, serializer)). The durable runtime binds it as
// the inner when this instance runs through a per-operation serializer slot, so you do
// not have to thread the global serializer in yourself.
public static ILambdaSerializer CreateUsingGlobalSerializer() =>
new FileSystemSerializer(
basePath: "/mnt/efs", // required: durable mount point
storageMode: FileSystemStorageMode.Always, // optional, default Always
pathEncoding: FileSystemPathEncoding.Uri); // optional, default Uri
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
using Amazon.Lambda.Core;
using Amazon.Lambda.DurableExecution;
using Amazon.Lambda.Serialization.SystemTextJson;

public class FileSystemSerdesWalkthrough
{
public Task<DurableExecutionInvocationOutput> Handler(
DurableExecutionInvocationInput input, ILambdaContext context)
=> DurableFunction.WrapAsync<object, Report>(Workflow, input, context);

private async Task<Report> Workflow(object input, IDurableContext ctx)
{
// FileSystemSerializer wraps an inner ILambdaSerializer (here the default JSON
// serializer) and writes each result to the mounted filesystem, keeping only a small
// file pointer in the checkpoint. Pass it to a single operation's config; other
// operations continue to use the serializer registered at the host boundary.
var fileSystem = new FileSystemSerializer(
inner: new DefaultLambdaJsonSerializer(),
basePath: "/mnt/efs");

Report report = await ctx.StepAsync(
async (_, _) => await BuildLargeReportAsync(),
name: "build-report",
config: new StepConfig { Serializer = fileSystem });

return report;
}

private static Task<Report> BuildLargeReportAsync() => Task.FromResult(new Report("…large body…"));
}

public record Report(string Body);
10 changes: 7 additions & 3 deletions examples/csharp/sdk-reference/serialization/MapConfigExample.cs
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
using Amazon.Lambda.Core;
using Amazon.Lambda.DurableExecution;
using Amazon.Lambda.Serialization.SystemTextJson;

public class MapConfigExample
{
Expand All @@ -9,12 +10,15 @@ public Task<DurableExecutionInvocationOutput> Handler(

private async Task<IReadOnlyList<ProcessedItem>> Workflow(object input, IDurableContext ctx)
{
// MapConfig has no serializer slot. Each item result is serialized with the
// ILambdaSerializer registered on ILambdaContext.Serializer. To customize
// serialization, register a custom ILambdaSerializer at the host boundary.
// Set MapConfig.ItemSerializer to serialize each item's RESULT with a specific
// ILambdaSerializer. When null (default), item results are serialized with the
// ILambdaSerializer registered on ILambdaContext.Serializer. This controls only the
// per-item result — not the aggregated batch envelope (statuses / completion
// reason), which is SDK-internal. ParallelConfig has the same ItemSerializer field.
var config = new MapConfig<string>
{
MaxConcurrency = 3,
ItemSerializer = new CamelCaseLambdaJsonSerializer(),
};

var items = new[] { "a", "b", "c" };
Expand Down
Loading
Loading