EF persistence for failed messages and streamline known endpoints#5638
EF persistence for failed messages and streamline known endpoints#5638johnsimons wants to merge 5 commits into
Conversation
8f7ce88 to
9346fb1
Compare
| const string CatalogName = "ServiceControlFullTextCatalog"; | ||
|
|
||
| // Both statements are guarded: an instance without Full-Text Search installed still migrates, | ||
| // it just has no full text index. They also cannot run inside a transaction, so the migration |
There was a problem hiding this comment.
do we actually need to support instances without FTS?
If so we probably need to surface that as an error and have a way to recreate this index so that there is an upgrade path.
There was a problem hiding this comment.
I am not sure, I remember as part of the spikes we allowed customers to configure it.
@warwickschroeder any thoughts?
There was a problem hiding this comment.
We can make a decision, later on this?
Without FTS, they would get no search at all.
I don't think we want to support LIKE %blah% !
From our spikes FTS was not an issue at all, so i think we can make it a required thing.
There was a problem hiding this comment.
Not having it as an option gives the user the option to not having to store the body in the DB and only use cheap external storage. That may be worth it?
This change introduces the core EF Core schema and entities for storing failed messages, including their associated groups and retry information. It also sets up full-text search capabilities for failed message headers, bodies, and message types in both PostgreSQL and SQL Server. The separate 'insert-only' table and background reconciler for KnownEndpoints have been removed, simplifying how known endpoints are managed. A new `MaxBodySizeToStore` setting is added to control message body storage.
This change completes the error ingestion process for PostgreSQL and SQL Server persisters, enabling robust and concurrent storage of failed messages. It introduces a detailed design document (`error-ingestion-design.md`) outlining the complex write path.
Adds a `RetentionSweeper` background service to automatically delete resolved and archived failed messages that have exceeded the configured retention period. The sweeper runs periodically in batches, handling external message bodies and tolerating body deletion failures.
9346fb1 to
ed0b1a6
Compare
warwickschroeder
left a comment
warwickschroeder
left a comment
There was a problem hiding this comment.
Review so far. Not finished yet. Will review the rest tonight/tomorrow.
| When `BodyStoredExternally` is true the external copy is authoritative and `BodyText` is a | ||
| search aid only; it must never be served as the body. `BodySize` is always the true original | ||
| size. External writes happen before the row that points at them is committed. |
There was a problem hiding this comment.
For storage cost reasons, could we give the option to only store in cheap external storage and not in the DB at all? This would mean giving up Full Text Search, but maybe thats less important for error messages and more important for auditing, as auditing would typically have alot more messages to search through.
There was a problem hiding this comment.
But that would be weird, given that then we could search audits but not errors, and specially now that we have an agregate list of messages (All messages).
It is possible that we are trying to optimise things too early.
There was a problem hiding this comment.
Yeh. Maybe that option is a bit premature.
3 participants
This change introduces the core EF Core schema and entities for storing failed messages, including their associated groups and retry information. It also sets up full-text search capabilities for failed message headers, bodies, and message types in both PostgreSQL and SQL Server.
The separate 'insert-only' table and background reconciler for KnownEndpoints have been removed, simplifying how known endpoints are managed. A new
MaxBodySizeToStoresetting is added to control message body storage.