diff --git a/AGENTS.md b/AGENTS.md index ae5bdd0191..46b48db146 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -93,6 +93,7 @@ commit that evidences it. The summary: | 1 | **Exceptions drive the solution** — name what was wrong, why it cannot work, and *every* way out (the exact API call, attribute, option or command). Assert the message in a test | The message is what the agent or developer reacts to; it is the deliverable | | 1a | A configuration mistake is **refused at bootstrap**, never at runtime, and never read as an empty result — the guard and the message test ship in the commit that states the rule | One worktree of 18 fix commits paid for the rules that shipped without one | | 1c | **A check that can run at compile time runs at compile time** — in a guard the module calls from `prepare()`, which is handed the `InterfaceToCallRegistry` for exactly that. Only message *content* is left to runtime | The compile-time guard removes the runtime branch entirely: there is no unresolvable case left to handle | +| 1d | **The common usage is the default; any other usage is refused, never given a second meaning.** `#[EventHandler]` passes the payload first because almost every handler wants it; `#[EventHandler]` on a class is refused. For every attribute, configuration and gateway, decide each placement, parameter type, combining attribute (`#[Asynchronous]` first), repeated registration and contradicting setting: supported with a test, or refused at bootstrap with a message test. When two readings are possible, refuse rather than pick one | A usage quietly given its own meaning reads differently in two applications, and the one that guessed wrong finds out in production. Flow tracking shipped three such gaps at once: an async subscriber asked for an `endpointId` it cannot take, a second configuration was silently dropped, a mistyped subscriber failed on every notification | | 2 | **Never take a nullable service dependency.** Register unconditionally, decide at runtime. Split the class, add a null-object factory, make the parameter required, or pick between two services in the container | A `?Service = null` is a mode switch in disguise, and the two modes drift | | 2a | **Never default a dependency either.** `$clock = new NativeClock()` is rule 2 with the null spelled out: every construction site that forgets the argument silently runs a second way. A dependency is a required parameter at every site, and the container passes it | The 2.0.0-beta.2 clock regression: two channel adapters passed the clock, six other sites took the default, and a DBAL consumer on real time hung a test on the pinned clock | | 3 | **Constructor-injected services are stateless** — no accumulating properties, no static caches, no per-message collectors. Mutable state goes in a function-scoped immutable value object carried on a message header | A container service is a singleton; anything it remembers leaks into the next message, transaction and retry | @@ -104,7 +105,7 @@ commit that evidences it. The summary: | 7a | **This is the Ecotone core repository: edit `licence Enterprise` files here like any other file.** Their notice — the full one under `src/`, one sentence under `Api/` (6b) — is not addressed to work in this repository. It governs copying them **out** — into another codebase, never copy one, adapt it or reimplement an "inspired" version, and renaming the class or namespace does not make that permitted. If you cannot confirm a licence there, implement from your own functional requirements instead | Read as a warning here, the notice stalls every fix to an Enterprise file; ignored elsewhere, it let one evaluated agent run copy a file at 0.989 token overlap | | 8 | **Always use optimistic locking** — a version check on the write, never a pessimistic lock or a bespoke tracker | Pessimistic locking and snapshot tracking were both tried and removed | | 9 | **A feature needing external storage ships an in-memory implementation**, named to mirror the storage one, and one shared suite proves the two behave identically. A race decided by a database constraint cannot be proved in-memory: state the contract as behaviour both sides assert, and leave a connection-racing case to the storage suite, named so the gap is visible | Otherwise the feature cannot be used in an `EcotoneLite` flow test — and `InMemoryEventStore` silently diverged from DBAL three times | -| 10 | **Tests validate at the userland level only.** No SQL on Ecotone's tables, no internal service references, no reflection, no statement counting. Fetch a **gateway** and assert on observable behaviour | A whole worktree — 22 commits — did nothing but pull one feature's tests back to the public surface | +| 10 | **Tests validate at the userland level only.** Every test boots the application with `EcotoneLite::bootstrapFlowTesting()` and drives it the way an application does: buses, gateways, attributes, `#[ServiceContext]` configuration, console commands. A test never constructs, calls or names a class outside `Api/` — no store, recorder, ledger, module, extension object or channel interceptor from `src/` — never runs SQL or schema calls against Ecotone's tables, and never uses reflection or statement counting. A failure is simulated through a seam the application owns: the connection it hands Ecotone, an `#[Around]` interceptor, a handler that throws. The test is right when every class under `src/` could be rewritten and the test would stay unchanged | A whole worktree — 22 commits — did nothing but pull one feature's tests back to the public surface | | 10a | **A guarantee is proved on every path that has to hold it** — direct call, class-routed bus send, routed send, gateway, console command, and the reconstruction path beside the live one | Five times in one feature, a fix covered one entry point and looked finished | | 11 | **`EcotoneLite`, `snake_case`, no comments, inline anonymous fixtures in the test file** — a named class below the `TestCase` only where PHP forces one. Test-first: RED, GREEN, refactor | An anonymous fixture cannot drift out of the test that owns it | | 12 | **The public surface is `Api/`, a sibling of `src/`** — never `src/Api/`, which breaks Tempest discovery. Everything outside `Api` is `@internal` | | @@ -115,6 +116,7 @@ commit that evidences it. The summary: | 14a | **A framework header is named `ecotone_{module}_{feature}_{action}`** — lower snake_case, no dots, e.g. `ecotone_dbal_deadletter_limit`, held in one constant. Exempt: the Spring-shared core headers (`id`, `timestamp`, `correlationId`, `parentId`, `contentType`, …), the event store's `_aggregate_*` keys, and `amqp_*`/`kafka_*` broker copies. A header written to a queue, outbox, dead letter or event store is renamed only with a read path for its old name | The prefix keeps framework headers apart from application ones; the event store reads a dotted metadata key as a nested JSON path on MySQL and SQLite but literally on PostgreSQL | | 15 | **Orchestrating methods read as step lists**; SQL belongs to the collaborator that owns the table | | | 16 | **No table is created on the message path**, on any engine. Tables come from `ecotone:migration:database:setup` (or your migrations) and, in tests, from `FlowTestSupport::setupDatabaseTables()`, which the `EcotoneLite` bootstrap calls itself unless `TestConfiguration::withDatabaseTablesSetUpOnBootstrap(false)`. A missing table raises its table manager's missing-table `ConfigurationException`, found by the failing statement (`MissingTableFailure`), never by asking the schema first | MySQL/MariaDB implicitly commit on DDL, and a per-engine gate made table creation depend on the database. A schema check costs a round trip per operation, and memoising it is a rule-3 cache | +| 16a | **No database write after the transaction.** Whatever a feature stores about a message is written inside that message's transaction, or not at all — never after the commit, after a rollback, or around the transaction. Telemetry sent to an external collector (OpenTelemetry) is exempt, because it is allowed to fail | A write after the commit runs when the work is already done: with the database down it fails the consumer or the command for work that succeeded, and the message is no longer retried. Inside the transaction the same outage rolls back and leaves the message to be retried | | 17 | New files are `final` and `declare(strict_types=1)` | | | 18 | PHP 8.1+ features where they say something; named arguments once past two parameters | | @@ -147,6 +149,13 @@ API moved in 2.0 (`Ecotone\Api\Attribute\CommandHandler`, not the old flat names statement counting, no internal service references — fetch a gateway (`$ecotone->getGateway(EventStore::class)`, `Ecotone\Api\EventSourcing\EventStore`) and drive the public API. This is the most-corrected rule in the repository; see conventions rule 10 +- **Never test an internal class directly**, not even a seam with two implementations. An in-memory store and its + DBAL twin are proved equal by one suite that boots the application twice, once per storage, and drives both through + the same gateway — never by constructing the stores and calling them. A failure the test needs (a database outage, + a broken send) is produced through something the application owns: a Doctrine connection handed to Ecotone, an + `#[Around]` interceptor, a handler that throws — never by dropping or editing Ecotone's tables. If a behaviour + cannot be reached that way, it is not a behaviour the application can observe; remove the code rather than test it + from the inside - **A message channel is never driven directly.** `FlowTestSupport` has no `getMessageChannel()`, `receiveMessageFrom()` or `sendMessageDirectToChannel()`. The application reaches a channel through an `#[Asynchronous]` handler, so the test sends the command or event, calls `run('channel')`, and asserts on what that diff --git a/Monorepo/CrossModuleTests/Tests/Tracing/FlowTrackingWithNoopProviderTest.php b/Monorepo/CrossModuleTests/Tests/Tracing/FlowTrackingWithNoopProviderTest.php new file mode 100644 index 0000000000..25e5ff6494 --- /dev/null +++ b/Monorepo/CrossModuleTests/Tests/Tracing/FlowTrackingWithNoopProviderTest.php @@ -0,0 +1,89 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + * + * @internal + */ +final class FlowTrackingWithNoopProviderTest extends TestCase +{ + public function test_a_provider_that_records_nothing_still_gives_every_stored_span_its_own_id(): void + { + $users = new class () { + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId, EventBus $eventBus): void + { + $eventBus->publishWithRouting('user.registered', $userId); + } + + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', endpointId: 'user.notifier')] + public function notify(string $userId): void + { + } + }; + $subscriber = new class () { + /** @var string[] */ + public array $spanIds = []; + + #[OnFlowStepCompleted] + public function whenEnded(FlowStepCompleted $event): void + { + $this->spanIds[] = $event->step->spanId; + } + }; + $ecotone = EcotoneLite::bootstrapFlowTesting( + [$users::class, $subscriber::class], + [$users, $subscriber, TracerProviderInterface::class => new NoopTracerProvider()], + [FlowTrackingConfiguration::createWithDefaults(), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + enabledModules: [ModulePackageList::TRACING_PACKAGE, ModulePackageList::DBAL_PACKAGE], + licenceKey: LicenceTesting::VALID_LICENCE, + ); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertCount(count($subscriber->spanIds), array_unique($subscriber->spanIds)); + $this->assertGreaterThan(3, count($subscriber->spanIds)); + $this->assertTrue($ecotone->getGateway(TracingGateway::class)->isCompletedFor($trace->traceId)); + } +} diff --git a/Monorepo/CrossModuleTests/Tests/Tracing/FlowTrackingWithOpenTelemetryTest.php b/Monorepo/CrossModuleTests/Tests/Tracing/FlowTrackingWithOpenTelemetryTest.php new file mode 100644 index 0000000000..26c4da2741 --- /dev/null +++ b/Monorepo/CrossModuleTests/Tests/Tracing/FlowTrackingWithOpenTelemetryTest.php @@ -0,0 +1,195 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + * + * @internal + */ +final class FlowTrackingWithOpenTelemetryTest extends TestCase +{ + protected function setUp(): void + { + MessagingTestCase::cleanUpDbal(); + } + + public function test_one_trace_is_exported_to_open_telemetry_and_stored_in_the_database_across_an_asynchronous_dbal_channel(): void + { + $exporter = new InMemoryExporter(new ArrayObject()); + $ecotone = $this->bootstrap($exporter, [$this->users(), $this->asynchronousNotifier()]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + self::assertSame( + ['process class@anonymous::register', 'route user.notifier', 'send command', 'send event', 'send notifications'], + self::exportedSpanNamesOf($exporter, $trace->traceId), + ); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.registration')); + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + + $ecotone->run('notifications', ExecutionPollingMetadata::createWithTestingSetup()); + + self::assertSame( + ['process class@anonymous::notify', 'process class@anonymous::register', 'receive notifications', 'route user.notifier', 'send command', 'send event', 'send notifications'], + self::exportedSpanNamesOf($exporter, $trace->traceId), + ); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.notifier')); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_each_flow_step_completed_notification_is_exported_in_a_trace_of_its_own(): void + { + $exporter = new InMemoryExporter(new ArrayObject()); + $ecotone = $this->bootstrap($exporter, [$this->users(), $this->flowStepCompletedSubscriber()]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $notificationTraceIds = []; + /** @var SpanDataInterface $span */ + foreach ($exporter->getSpans() as $span) { + if ($span->getName() === 'send ecotone.tracing.flowStepCompleted') { + $notificationTraceIds[] = $span->getContext()->getTraceId(); + } + } + self::assertNotEmpty($notificationTraceIds); + self::assertNotContains($trace->traceId, $notificationTraceIds); + foreach ($notificationTraceIds as $notificationTraceId) { + self::assertSame( + ['process class@anonymous::whenStepCompleted', 'send ecotone.tracing.flowStepCompleted'], + self::exportedSpanNamesOf($exporter, $notificationTraceId), + ); + } + } + + public function test_a_second_application_on_the_same_database_sees_the_completion_the_first_one_stored(): void + { + $producer = $this->bootstrap(new InMemoryExporter(new ArrayObject()), [$this->users()]); + $observer = $this->bootstrap(new InMemoryExporter(new ArrayObject()), [$this->users()]); + $trace = SpanContext::newTrace(); + + $producer->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + self::assertTrue($this->completion($observer)->isCompletedFor($trace->traceId)); + } + + /** + * @return string[] + */ + private static function exportedSpanNamesOf(InMemoryExporter $exporter, string $traceId): array + { + $names = []; + /** @var SpanDataInterface $span */ + foreach ($exporter->getSpans() as $span) { + if ($span->getContext()->getTraceId() !== $traceId || str_starts_with($span->getName(), 'process MessageHeadersPropagatorInterceptor::')) { + continue; + } + $names[] = preg_replace('/^process class@anonymous\S*::/', 'process class@anonymous::', $span->getName()); + } + sort($names); + + return $names; + } + + /** + * @param object[] $services + */ + private function bootstrap(InMemoryExporter $exporter, array $services): FlowTestSupport + { + return EcotoneLite::bootstrapFlowTesting( + array_map(fn (object $service): string => $service::class, $services), + [ + ...$services, + DbalConnectionFactory::class => new DbalConnectionFactory(getenv('DATABASE_DSN') ?: 'pgsql://ecotone:secret@127.0.0.1:5432/ecotone'), + TracerProviderInterface::class => TracingTestCase::prepareTracer($exporter), + ], + [ + FlowTrackingConfiguration::createWithDefaults(), + DbalBackedMessageChannelBuilder::create('notifications'), + ], + enabledModules: [ModulePackageList::DBAL_PACKAGE, ModulePackageList::TRACING_PACKAGE], + licenceKey: LicenceTesting::VALID_LICENCE, + )->setupDatabaseTables(); + } + + private function completion(FlowTestSupport $ecotone): TracingGateway + { + return $ecotone->getGateway(TracingGateway::class); + } + + private function users(): object + { + return new class () { + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId, EventBus $eventBus): void + { + $eventBus->publishWithRouting('user.registered', $userId); + } + }; + } + + private function asynchronousNotifier(): object + { + return new class () { + #[Asynchronous('notifications')] + #[EventHandler('user.registered', endpointId: 'user.notifier')] + public function notify(string $userId): void + { + } + }; + } + + private function flowStepCompletedSubscriber(): object + { + return new class () { + #[OnFlowStepCompleted] + public function whenStepCompleted(FlowStepCompleted $event): void + { + } + }; + } +} diff --git a/Monorepo/CrossModuleTests/Tests/Tracing/TracingStackTest.php b/Monorepo/CrossModuleTests/Tests/Tracing/TracingStackTest.php index 6880a298f7..c82951ed78 100644 --- a/Monorepo/CrossModuleTests/Tests/Tracing/TracingStackTest.php +++ b/Monorepo/CrossModuleTests/Tests/Tracing/TracingStackTest.php @@ -10,17 +10,17 @@ use Ecotone\Api\Messaging\ConfiguredMessagingSystem; use Illuminate\Foundation\Http\Kernel as LaravelKernel; use Monorepo\CrossModuleTests\Tests\FullAppTestCase; -use Monorepo\ExampleApp\Common\Domain\Notification\NotificationSubscriber; use Monorepo\ExampleApp\Common\Domain\Order\Command\PlaceOrder; -use Monorepo\ExampleApp\Common\Domain\Order\Order; use Monorepo\ExampleApp\Common\Domain\Order\ShippingAddress; use Monorepo\ExampleApp\Common\Infrastructure\Configuration; -use Monorepo\ExampleApp\Common\Infrastructure\Output; use Monorepo\ExampleApp\ExampleAppCaseTrait; use OpenTelemetry\SDK\Trace\SpanExporter\InMemoryExporter; use Psr\Container\ContainerInterface; use Test\Ecotone\OpenTelemetry\Integration\TracingTestCase; +/** + * @internal + */ final class TracingStackTest extends FullAppTestCase { use ExampleAppCaseTrait; @@ -103,47 +103,47 @@ private function assertTracing(InMemoryExporter $exporter): void TracingTestCase::compareTreesByDetails( [ [ - 'details' => ['name' => 'Command Bus'], + 'details' => ['name' => 'send command'], 'children' => [ [ - 'details' => ['name' => 'Command Handler: ' . Order::class . '::create'], - 'children' => [] + 'details' => ['name' => 'process Order::create'], + 'children' => [], ], [ - 'details' => ['name' => 'Event Bus'], - 'children' => [] + 'details' => ['name' => 'send event'], + 'children' => [], ], [ - 'details' => ['name' => 'Sending to Channel: delivery'], - 'children' => [] + 'details' => ['name' => 'send delivery'], + 'children' => [], ], [ - 'details' => ['name' => 'Sending to Channel: notifications'], + 'details' => ['name' => 'send notifications'], 'children' => [ [ - 'details' => ['name' => 'Receiving from channel: notifications'], + 'details' => ['name' => 'receive notifications'], 'children' => [ [ - 'details' => ['name' => 'Event Handler: ' . NotificationSubscriber::class . '::whenOrderWasPlaced'], - 'children' => [] - ] - ] - ] - ] - ] - ] + 'details' => ['name' => 'process NotificationSubscriber::whenOrderWasPlaced'], + 'children' => [], + ], + ], + ], + ], + ], + ], ], [ - 'details' => ['name' => 'Query Bus'], + 'details' => ['name' => 'send query'], 'children' => [ [ - 'details' => ['name' => 'Query Handler: ' . Output::class . '::getMessages'], - 'children' => [] - ] - ] - ] + 'details' => ['name' => 'process Output::getMessages'], + 'children' => [], + ], + ], + ], ], TracingTestCase::buildTree($exporter) ); } -} \ No newline at end of file +} diff --git a/composer.json b/composer.json index 02e91e6c5b..4eab2a099a 100644 --- a/composer.json +++ b/composer.json @@ -53,6 +53,7 @@ "Ecotone\\Api\\Laravel\\": "packages/Laravel/Api", "Ecotone\\Api\\Tempest\\": "packages/Tempest/Api", "Ecotone\\Api\\Kafka\\": "packages/Kafka/Api", + "Ecotone\\Api\\OpenTelemetry\\": "packages/OpenTelemetry/Api", "Ecotone\\Api\\Symfony\\": "packages/Symfony/Api", "Ecotone\\Amqp\\": "packages/Amqp/src", "Ecotone\\AnnotationFinder\\": "packages/Ecotone/src/AnnotationFinder/", diff --git a/docs/coding-conventions.md b/docs/coding-conventions.md index b018985268..ed7c041e28 100644 --- a/docs/coding-conventions.md +++ b/docs/coding-conventions.md @@ -15,7 +15,7 @@ branch `dgafka/dcb-docs` and is deliberately untracked here, since `docs/superpo paths resolve on that branch. Commit SHAs cited below resolve on any branch. Rules 1-11 are the maintainer's. Rules 12-19 are conventions the code holds to consistently. Section 20 lists the -mechanics a tool enforces for you, and section 21 the gates and landmines. Lettered sub-rules (1a, 1b, 1c, 6a, 10a, +mechanics a tool enforces for you, and section 21 the gates and landmines. Lettered sub-rules (1a, 1b, 1c, 1d, 6a, 10a, 16a, 12a, 13a, 14a) continue the rule they hang off; they are numbered that way so nothing above them ever renumbers. > **Verify names before you write them.** Every attribute, parameter, method and console option in code you write @@ -137,6 +137,39 @@ So, before adding a check: ask whether it could have been answered from the clas and the attributes alone. If it could, it belongs in a guard the module calls, and the test that proves it boots Ecotone and expects a `ConfigurationException`. + +### 1d. The common usage is the default; any other usage is refused, never given a second meaning + +Every attribute, configuration object and gateway defaults to the usage most applications need, so the shortest code +is the right code. `#[EventHandler]` passes the message payload as the first parameter, because that is what almost +every handler wants, and resolves the rest from headers and services; `#[CommandHandler]` without a routing key +routes by the payload's class. Nobody has to opt into the normal case. + +A usage the feature is not meant for is refused at bootstrap, naming the meant one (rules 1, 1a). It is never quietly +accepted with a meaning of its own, because then two applications read the same code differently, and the one that +guessed wrong finds out in production. `#[EventHandler]` on a class is refused by its attribute target. When two +readings are possible, refuse rather than pick one: two configurations of one feature, a message carrying both +`tracingId` and `traceparent`, an attribute combined with one it cannot work with. + +Before shipping an attribute, a configuration or a gateway, go through what an application could write and decide, +for each one, whether it is supported (with a test) or refused (with a message test). None may be left to chance: + +- **Placement** — the attribute's `#[Attribute(...)]` target is as narrow as the meant usage. +- **Parameters** — the first parameter is checked against what it receives (rule 1c), so a wrong type fails at + bootstrap, not on the first message after the work committed. +- **Combinations** — every other attribute that can sit beside it, `#[Asynchronous]` first, is either supported or + refused by the attribute itself, with its own reason. An endpoint attribute that cannot run asynchronously returns + that reason from `asynchronousExecutionRefusal()`, and `AsynchronousModule` uses it instead of asking for an + `endpointId` the attribute cannot take. +- **Repetition** — a configuration object registered twice is refused; the first one never silently wins. +- **Settings that contradict** — an option combination that cannot work, such as a subscriber with notifications + turned off, is refused naming the setting to remove. + +Evidence: flow tracking shipped three of these gaps at once. `#[Asynchronous]` on a flow notification subscriber asked for an +`endpointId` the attribute cannot take, a second `FlowTrackingConfiguration` was silently dropped, and a subscriber +with a mistyped first parameter failed on every notification at runtime (`FlowTrackingTest`, the three bootstrap +refusals). + --- ## 2. Never take a nullable service dependency @@ -593,6 +626,20 @@ Never, in a test: - use reflection - count SQL statements - name an internal collaborator class in the test's own name +- construct or call any class outside `Api/` — a store, recorder, ledger, module, extension object or channel + interceptor from `src/` — even to prove two implementations of a seam behave alike +- drop, alter or write Ecotone's tables to simulate a failure + +The test of a test: rewrite every class under `src/` the feature touches, keep the behaviour, and the test must not +change. A contract suite that constructs `InMemoryTraceStore` and `DbalTraceStore` and calls `append()` fails that test +the moment the store's methods are renamed, while the behaviour it guards — a trace completes once and notifies once — +is unchanged. Prove it instead through the application: boot it once per storage and drive both through the bus and +the `TraceCompletion` gateway. + +A failure the test needs comes through a seam the application owns. A database outage right after a commit is a +Doctrine connection the application hands Ecotone, whose statements start failing when an `#[Around]` interceptor +just outside the transaction says so — not a `DROP TABLE` on Ecotone's tables. If a path cannot be reached from +userland at all, the application cannot observe it: remove the code instead of testing it from the inside. ```php use Ecotone\Api\EventSourcing\EventStore; @@ -1141,6 +1188,54 @@ History: `230bcea5d` and `511b31428` first moved this DDL out of the transaction --- +## 16a. No database write after the transaction + +A message's unit of work owns a transaction: the transaction interceptor begins it, the handler and everything it +calls write inside it, and it commits or rolls back as one. Anything a feature records about that message - stored +traces, bookkeeping, a completion flag - is written inside the same transaction, or not at all. + +Never write after the commit, after a rollback, or in an interceptor that wraps the transaction from outside. At that +point the work is already decided. If the database is down, the write throws out of a consumer whose message was +already acknowledged, or out of a command whose data already committed: the caller sees a failure for work that +succeeded, and the message is not retried. The same outage inside the transaction rolls back and leaves the message +to the retry and dead-letter strategy, which is what they exist for. + +Wrong - the root span is inserted when it ends, after the commit: + +```php +public function record(Span $span): void +{ + if ($this->scopeActive) { $this->buffer[] = $span; return; } + $this->traceStore->append([$span]); // after the commit: fails a consumer whose work succeeded +} +``` + +Right - the span that wraps the unit of work is written as it stands at the flush, inside the transaction, and +nothing touches the database once the transaction has returned: + +```php +$result = $methodInvocation->proceed(); +$this->recorder->flushWithOpenSpans(); // inside the transaction +``` + +Work that commits by itself - an asynchronous projection commits once per batch, with Ecotone turning the endpoint's +transaction off - is recorded inside each of those commits, never once the work has returned: flow tracking joins +the projection batch's transaction. A path the application chose to run with no transaction at all (the command bus +with `withTransactionOnCommandBus(false)`, an event bus or gateway call from a controller) has no transaction to be +inside, so what it stores is written when its work ends; that is the application's decision, and the default runs +every command and consumer in a transaction. + +What may happen after the commit is work that does not depend on the database: publishing to subscribers from +memory, and sending telemetry to an external collector such as OpenTelemetry, which is allowed to fail and never +fails the message. A test proves the rule by making the database unavailable right after the commit, with an +`#[Around]` interceptor just outside the transaction, and asserting that the command or consumer still succeeds. + +Evidence: stored traces wrote the root span and claimed completion after the commit, and an outage at that moment +threw the missing-table refusal out of the consumer (`FlowTrackingConformanceTest`, the outage tests); an asynchronous +projection wrote its spans after its own batch committed (`FlowTrackingProjectionTest`). + +--- + ## 17. `final` and `declare(strict_types=1)` on new files 113 of the 121 classes added since 1.x are `final`; 132 of 133 new `src` files declare strict types. Older files diff --git a/packages/Dbal/Api/Tracing/Flow.php b/packages/Dbal/Api/Tracing/Flow.php new file mode 100644 index 0000000000..a007f5e4f8 --- /dev/null +++ b/packages/Dbal/Api/Tracing/Flow.php @@ -0,0 +1,29 @@ + $retention]); + } + + public function withOnlyRequestedFlows(bool $onlyRequestedFlows): self + { + return new self(...[...get_object_vars($this), 'onlyRequestedFlows' => $onlyRequestedFlows]); + } + + public function withNotifications(bool $notifications): self + { + return new self(...[...get_object_vars($this), 'notifications' => $notifications]); + } + + public function withTracingGateway(bool $tracingGateway): self + { + return new self(...[...get_object_vars($this), 'tracingGateway' => $tracingGateway]); + } + + public function withRemovalBatchSize(int $removalBatchSize): self + { + if ($removalBatchSize < 1) { + throw ConfigurationException::create("FlowTrackingConfiguration::withRemovalBatchSize({$removalBatchSize}) would never remove an expired flow. Pass 1 or more; the default is 1000."); + } + + return new self(...[...get_object_vars($this), 'removalBatchSize' => $removalBatchSize]); + } + + public function withInMemoryStorage(bool $inMemory): self + { + return new self(...[...get_object_vars($this), 'inMemory' => $inMemory]); + } +} diff --git a/packages/Dbal/Api/Tracing/OnFlowStepCompleted.php b/packages/Dbal/Api/Tracing/OnFlowStepCompleted.php new file mode 100644 index 0000000000..6361c39012 --- /dev/null +++ b/packages/Dbal/Api/Tracing/OnFlowStepCompleted.php @@ -0,0 +1,29 @@ +extensionObjects, DbalConfiguration::createWithDefaults()); + return [ - ...array_filter($this->extensionObjects, fn (object $extensionObject) => ! $extensionObject instanceof DbalConfiguration), + ...array_filter($this->extensionObjects, fn (object $extensionObject) => ! $extensionObject instanceof DbalConfiguration && ! $extensionObject instanceof FlowTrackingConfiguration), $this->inMemoryWhereTheConnectionIsMissing($dbalConfiguration), + ...$this->flowTrackingInMemoryWhereTheConnectionIsMissing($dbalConfiguration), ]; } @@ -100,6 +103,20 @@ private function inMemoryWhereTheConnectionIsMissing(DbalConfiguration $dbalConf return $dbalConfiguration; } + /** + * @return FlowTrackingConfiguration[] + */ + private function flowTrackingInMemoryWhereTheConnectionIsMissing(DbalConfiguration $dbalConfiguration): array + { + $flowTracking = ExtensionObjectResolver::resolve(FlowTrackingConfiguration::class, $this->extensionObjects); + if ($flowTracking === []) { + return []; + } + $connectionReferenceName = $dbalConfiguration->defaultConnectionReferenceNames[0] ?? DbalConnectionReference::DEFAULT; + + return array_map(fn (FlowTrackingConfiguration $configuration): FlowTrackingConfiguration => $this->has($connectionReferenceName) ? $configuration : $configuration->withInMemoryStorage(true), $flowTracking); + } + private function refuseDbalChannelsWithoutTheirConnection(): void { foreach (ExtensionObjectResolver::resolve(DbalBackedMessageChannelBuilder::class, $this->extensionObjects) as $channelBuilder) { diff --git a/packages/Dbal/src/Database/SpansTableManager.php b/packages/Dbal/src/Database/SpansTableManager.php new file mode 100644 index 0000000000..f72b4ea9e8 --- /dev/null +++ b/packages/Dbal/src/Database/SpansTableManager.php @@ -0,0 +1,125 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + */ +final class SpansTableManager implements DbalTableManager +{ + public const DEFAULT_TABLE = 'ecotone_tracing'; + public const FEATURE_NAME = 'tracing'; + + public function __construct( + private string $tableName, + private bool $isUsed, + private ?string $consoleInvocationPrefix = null, + private ?string $connectionReferenceName = null, + ) { + } + + public function getFeatureName(): string + { + return self::FEATURE_NAME; + } + + public function isUsed(): bool + { + return $this->isUsed; + } + + public function getTableName(): string + { + return $this->tableName; + } + + public function getCreateTableSql(Connection $connection): string|array + { + return DbalIdempotentDdl::withIfNotExists($connection->getDatabasePlatform()->getCreateTableSQL($this->buildTableSchema())); + } + + public function getDropTableSql(Connection $connection): string + { + return "DROP TABLE IF EXISTS {$this->tableName}"; + } + + public function createTable(Connection $connection): void + { + if ($this->isInitialized($connection)) { + return; + } + + try { + $connection->createSchemaManager()->createTable($this->buildTableSchema()); + } catch (TableExistsException) { + } + } + + public function getMissingTableInstructions(): string + { + return MissingTableInstructions::build(self::FEATURE_NAME, $this->tableName, $this->consoleInvocationPrefix, $this->connectionReferenceName); + } + + public function dropTable(Connection $connection): void + { + $schemaManager = $connection->createSchemaManager(); + if (! $schemaManager->tablesExist([$this->tableName])) { + return; + } + + $schemaManager->dropTable($this->tableName); + } + + public function isInitialized(Connection $connection): bool + { + return $connection->createSchemaManager()->tableExists($this->tableName); + } + + public function getDefinition(): Definition + { + return new Definition(self::class, [$this->tableName, $this->isUsed, $this->consoleInvocationPrefix, $this->connectionReferenceName]); + } + + private function buildTableSchema(): Table + { + $table = new Table($this->tableName); + $table->addColumn('span_id', Types::STRING, ['length' => 16]); + $table->addColumn('trace_id', Types::STRING, ['length' => 32]); + $table->addColumn('parent_span_id', Types::STRING, ['length' => 16, 'notnull' => false]); + $table->addColumn('name', Types::STRING, ['length' => 255]); + $table->addColumn('kind', Types::STRING, ['length' => 16]); + $table->addColumn('status', Types::STRING, ['length' => 16]); + $table->addColumn('status_description', Types::TEXT, ['notnull' => false]); + $table->addColumn('attributes', Types::TEXT); + $table->addColumn('events', Types::TEXT); + $table->addColumn('started_at', Types::BIGINT); + $table->addColumn('ended_at', Types::BIGINT); + $table->setPrimaryKey(['trace_id', 'span_id']); + $table->addIndex(['started_at']); + + return $table; + } +} diff --git a/packages/Dbal/src/Tracing/DbalTraceStore.php b/packages/Dbal/src/Tracing/DbalTraceStore.php new file mode 100644 index 0000000000..1c5a9610a7 --- /dev/null +++ b/packages/Dbal/src/Tracing/DbalTraceStore.php @@ -0,0 +1,205 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + */ +final class DbalTraceStore implements TraceStore +{ + public function __construct( + private ConnectionFactory $connectionFactory, + private SpansTableManager $spansTableManager, + private EcotoneClockInterface $clock, + ) { + } + + public function append(array $spans): void + { + $connection = $this->connection(); + $this->onTables(fn () => $this->insertSpans($connection, $spans)); + } + + public function stepsOf(string $traceId): array + { + return $this->onTables(fn (): array => $this->spansOf($this->connection(), $traceId)); + } + + public function progressOf(string $traceId): TraceProgress + { + return $this->onTables(fn (): TraceProgress => TraceLedger::progressOf($traceId, $this->spansOf($this->connection(), $traceId))); + } + + public function removeTracesStartedBefore(DatePoint $startedBefore, int $batchSize): void + { + $connection = $this->connection(); + do { + $removed = $this->onTables(fn (): int => $connection->transactional(fn (): int => $this->removeBatchStartedBefore($connection, $startedBefore, $batchSize))); + } while ($removed === $batchSize); + } + + public function withinTransaction(Closure $work): mixed + { + return $this->connection()->transactional($work); + } + + private function removeBatchStartedBefore(Connection $connection, DatePoint $startedBefore, int $batchSize): int + { + $traceIds = $connection->createQueryBuilder() + ->select('DISTINCT trace_id') + ->from($this->spansTableManager->getTableName()) + ->where('started_at < :startedBefore') + ->setParameter('startedBefore', $startedBefore->unixTime()->inMicroseconds(), Types::BIGINT) + ->setMaxResults($batchSize) + ->executeQuery() + ->fetchFirstColumn(); + if ($traceIds === []) { + return 0; + } + + $connection->createQueryBuilder() + ->delete($this->spansTableManager->getTableName()) + ->where('trace_id IN (:traceIds)') + ->setParameter('traceIds', $traceIds, ArrayParameterType::STRING) + ->executeStatement(); + + return count($traceIds); + } + + /** + * @param Span[] $spans + */ + private function insertSpans(Connection $connection, array $spans): void + { + $columns = ['span_id', 'trace_id', 'parent_span_id', 'name', 'kind', 'status', 'status_description', 'attributes', 'events', 'started_at', 'ended_at']; + $rows = []; + $parameters = []; + $types = []; + foreach (array_values($spans) as $index => $span) { + $values = [ + 'span_id' => $span->spanId, + 'trace_id' => $span->traceId, + 'parent_span_id' => $span->parentSpanId, + 'name' => $span->name, + 'kind' => $span->kind->value, + 'status' => $span->status->value, + 'status_description' => $span->statusDescription, + 'attributes' => json_encode($span->attributes, JSON_THROW_ON_ERROR | JSON_FORCE_OBJECT), + 'events' => json_encode(array_map(fn (SpanEvent $event): array => [ + 'name' => $event->name, + 'attributes' => (object) $event->attributes, + 'at' => $event->at->unixTime()->inMicroseconds(), + ], $span->events), JSON_THROW_ON_ERROR), + 'started_at' => $span->startedAt->unixTime()->inMicroseconds(), + 'ended_at' => $span->endedAt->unixTime()->inMicroseconds(), + ]; + $placeholders = []; + foreach ($values as $column => $value) { + $placeholders[] = ':' . $column . $index; + $parameters[$column . $index] = $value; + } + $types['started_at' . $index] = Types::BIGINT; + $types['ended_at' . $index] = Types::BIGINT; + $rows[] = '(' . implode(', ', $placeholders) . ')'; + } + $sql = sprintf('INSERT INTO %s (%s) VALUES %s', $this->spansTableManager->getTableName(), implode(', ', $columns), implode(', ', $rows)); + $sql .= $connection->getDatabasePlatform() instanceof AbstractMySQLPlatform ? ' ON DUPLICATE KEY UPDATE span_id = span_id' : ' ON CONFLICT (trace_id, span_id) DO NOTHING'; + + $connection->executeStatement($sql, $parameters, $types); + } + + /** + * @return Span[] + */ + private function spansOf(Connection $connection, string $traceId): array + { + $rows = $connection->createQueryBuilder() + ->select('*') + ->from($this->spansTableManager->getTableName()) + ->andWhere('trace_id = :traceId') + ->setParameter('traceId', $traceId) + ->orderBy('ended_at') + ->executeQuery() + ->fetchAllAssociative(); + + return array_map(fn (array $row): Span => new Span( + $row['trace_id'], + $row['span_id'], + $row['parent_span_id'], + $row['name'], + SpanKind::from($row['kind']), + json_decode($row['attributes'], true, flags: JSON_THROW_ON_ERROR), + SpanStatus::from($row['status']), + $row['status_description'], + array_map( + fn (array $event): SpanEvent => new SpanEvent($event['name'], $event['attributes'], self::datePointOf((int) $event['at'])), + json_decode($row['events'], true, flags: JSON_THROW_ON_ERROR), + ), + self::datePointOf((int) $row['started_at']), + self::datePointOf((int) $row['ended_at']), + ), $rows); + } + + private static function datePointOf(int $unixTimeInMicroseconds): DatePoint + { + return (new DatePoint('@0'))->setUnixTime(Duration::microseconds($unixTimeInMicroseconds)); + } + + private function onTables(callable $statement): mixed + { + try { + return $statement(); + } catch (Throwable $failure) { + throw MissingTableFailure::explained($failure, $this->spansTableManager); + } + } + + private function connection(): Connection + { + /** @var DbalContext $context */ + $context = CachedConnectionFactory::createFor(new DbalReconnectableConnectionFactory($this->connectionFactory, $this->clock))->createContext(); + + return $context->getDbalConnection(); + } +} diff --git a/packages/Dbal/src/Tracing/DbalTracingModule.php b/packages/Dbal/src/Tracing/DbalTracingModule.php new file mode 100644 index 0000000000..dded367b4f --- /dev/null +++ b/packages/Dbal/src/Tracing/DbalTracingModule.php @@ -0,0 +1,369 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + */ +#[ModuleAnnotation] +final class DbalTracingModule extends NoExternalConfigurationModule implements AnnotationModule +{ + private const REMOVE_EXPIRED_TRACES_CHANNEL = 'ecotone.tracing.removeExpiredTraces'; + private const USER_MESSAGE_GATEWAYS_POINTCUT = MessageGateway::class . '&¬(' . CommandBus::class . ')&¬(' . EventBus::class . ')&¬(' . QueryBus::class . ')'; + + /** + * @param string[] $flowStepCompletedSubscribers + * @param array $subscriberMethods + * @param string[] $pollingProjectionNames + * @param array $destinationsByClass + */ + private function __construct( + private array $flowStepCompletedSubscribers, + private array $subscriberMethods, + private array $pollingProjectionNames, + private array $destinationsByClass, + ) { + } + + public static function create(AnnotationFinder $annotationRegistrationService, InterfaceToCallRegistry $interfaceToCallRegistry): static + { + return new self( + self::placementsOf($annotationRegistrationService, OnFlowStepCompleted::class), + self::subscriberMethodsOf($annotationRegistrationService, OnFlowStepCompleted::class, FlowStepCompleted::class), + self::pollingProjectionNames($annotationRegistrationService), + self::destinationsByClass($annotationRegistrationService), + ); + } + + public function prepare(Configuration $messagingConfiguration, array $extensionObjects, ModuleReferenceSearchService $moduleReferenceSearchService, InterfaceToCallRegistry $interfaceToCallRegistry): void + { + $flowTracking = self::flowTrackingOf($extensionObjects); + $this->refuseMisconfiguredFlowTracking($messagingConfiguration, $flowTracking); + $this->refuseSubscribersThatCannotReceiveTheirNotification($interfaceToCallRegistry); + $connectionReference = self::connectionReferenceOf($extensionObjects); + $consoleInvocationPrefix = ConsoleInvocationResolver::resolveConsolePrefix(ExtensionObjectResolver::resolveUnique(BootstrapConfiguration::class, $extensionObjects, BootstrapConfiguration::createWithDefaults())); + $storesOnDatabase = $flowTracking !== null && ! $flowTracking->inMemory; + $messagingConfiguration->registerServiceDefinition( + SpansTableManager::class, + new Definition(SpansTableManager::class, [SpansTableManager::DEFAULT_TABLE, $storesOnDatabase, $consoleInvocationPrefix, $connectionReference]) + ); + if ($flowTracking === null) { + self::registerTracingGateway($messagingConfiguration, UnavailableTracingGateway::class, new Definition(UnavailableTracingGateway::class, ['TracingGateway needs flow tracking, which records every flow and decides when it is complete. Register FlowTrackingConfiguration::createWithDefaults() from a #[ServiceContext] method.'])); + + return; + } + + $messagingConfiguration->registerServiceDefinition( + TraceStore::class, + $storesOnDatabase + ? new Definition(DbalTraceStore::class, [new Reference($connectionReference), new Reference(SpansTableManager::class), new Reference(EcotoneClockInterface::class)]) + : new Definition(InMemoryTraceStore::class) + ); + $messagingConfiguration->registerServiceDefinition( + StoringSpanRecorder::class, + new Definition(StoringSpanRecorder::class, [ + new Reference(TraceStore::class), + $flowTracking->notifications && $this->flowStepCompletedSubscribers !== [], + $flowTracking->onlyRequestedFlows, + ]) + ); + $messagingConfiguration->registerServiceDefinition( + TraceRecordingInterceptor::class, + new Definition(TraceRecordingInterceptor::class, [new Reference(StoringSpanRecorder::class), new Reference(Tracer::class), new Reference(MessagingEntrypointService::class), new Reference(LoggingGateway::class)]) + ); + foreach (['publishAfterCommit' => Precedence::TRACE_PUBLICATION_PRECEDENCE, 'recordUnitOfWork' => Precedence::TRACE_RECORDING_PRECEDENCE] as $method => $precedence) { + $messagingConfiguration->registerAroundMethodInterceptor( + AroundInterceptorBuilder::create( + TraceRecordingInterceptor::class, + $interfaceToCallRegistry->getFor(TraceRecordingInterceptor::class, $method), + $precedence, + CommandBus::class . '||' . EventBus::class . '||' . QueryBus::class . '||' . DistributedBus::class . '||' . AsynchronousRunningEndpoint::class . '||' . OrchestratorGateway::class . '||(' . self::USER_MESSAGE_GATEWAYS_POINTCUT . ')' + ) + ); + } + $messagingConfiguration->registerAroundMethodInterceptor( + AroundInterceptorBuilder::create( + TraceRecordingInterceptor::class, + $interfaceToCallRegistry->getFor(TraceRecordingInterceptor::class, 'recordProjectionBatch'), + Precedence::TRACE_RECORDING_PRECEDENCE, + ProjectionFlush::class + ) + ); + if ($flowTracking->notifications) { + $messagingConfiguration->registerMessageChannel(SimpleMessageChannelBuilder::createPublishSubscribeChannel(OnFlowStepCompleted::CHANNEL)); + } + if ($flowTracking->tracingGateway) { + $multiTenancy = $flowTracking->inMemory ? null : self::multiTenancyWithoutDefaultConnectionOf($extensionObjects); + self::registerTracingGateway($messagingConfiguration, StoredTracingGateway::class, new Definition(StoredTracingGateway::class, [ + new Reference(TraceStore::class), + new Reference(EcotoneClockInterface::class), + $this->pollingProjectionNames, + $this->destinationsByClass, + $multiTenancy?->tenantHeaderName, + $multiTenancy === null ? null : (string) array_key_first($multiTenancy->tenantToConnectionMapping), + ])); + } else { + self::registerTracingGateway($messagingConfiguration, UnavailableTracingGateway::class, new Definition(UnavailableTracingGateway::class, ['TracingGateway is turned off by FlowTrackingConfiguration::withTracingGateway(false). Remove that call to ask whether a flow completed.'])); + } + $messagingConfiguration->registerServiceDefinition( + ExpiredTraceRemover::class, + new Definition(ExpiredTraceRemover::class, [new Reference(TraceStore::class), new Reference(EcotoneClockInterface::class), $flowTracking->retention->inMilliseconds(), $flowTracking->removalBatchSize]) + ); + $messagingConfiguration + ->registerMessageHandler(MessageProcessorActivatorBuilder::createForMethod(ExpiredTraceRemover::class, 'removeExpiredTraces')->withInputChannelName(self::REMOVE_EXPIRED_TRACES_CHANNEL)) + ->registerConsoleCommand(ConsoleCommandConfiguration::create(self::REMOVE_EXPIRED_TRACES_CHANNEL, 'ecotone:tracing:remove-expired-traces', [], 'Removes tracked flows older than the configured retention')); + } + + public function getModuleExtensions(ServiceConfiguration $serviceConfiguration, array $serviceExtensions): array + { + $connectionReference = self::connectionReferenceOf($serviceExtensions); + $extensions = [ + new DbalTableManagerReference(SpansTableManager::class, $connectionReference), + ]; + if (self::flowTrackingOf($serviceExtensions) !== null) { + $extensions[] = new SpanRecorderReference(StoringSpanRecorder::class); + } + + return $extensions; + } + + public function getModulePackageName(): string + { + return ModulePackageList::DBAL_PACKAGE; + } + + /** + * @param object[] $extensionObjects + */ + private static function flowTrackingOf(array $extensionObjects): ?FlowTrackingConfiguration + { + $configurations = ExtensionObjectResolver::resolve(FlowTrackingConfiguration::class, $extensionObjects); + if (count($configurations) > 1) { + throw ConfigurationException::create(sprintf('FlowTrackingConfiguration was registered %d times, so it is not clear which settings apply. Register it once, from one #[ServiceContext] method, chaining every setting on it: FlowTrackingConfiguration::createWithDefaults()->withRetention(...)->withNotifications(...).', count($configurations))); + } + + return $configurations[0] ?? null; + } + + private function refuseMisconfiguredFlowTracking(Configuration $messagingConfiguration, ?FlowTrackingConfiguration $flowTracking): void + { + $subscribers = $this->flowStepCompletedSubscribers; + $enterpriseUsages = [...($flowTracking !== null ? ['FlowTrackingConfiguration'] : []), ...$subscribers]; + if ($enterpriseUsages !== [] && ! $messagingConfiguration->isRunningForEnterpriseLicence()) { + throw LicensingException::create(sprintf( + '%s %s an Ecotone Enterprise licence. Remove %s to run without one.', + implode(' and ', $enterpriseUsages), + count($enterpriseUsages) === 1 ? 'requires' : 'require', + count($enterpriseUsages) === 1 ? 'it' : 'them', + )); + } + if ($subscribers === []) { + return; + } + if ($flowTracking === null) { + throw ConfigurationException::create(implode(', ', $subscribers) . ' needs flow tracking, which records the steps of every flow. Register FlowTrackingConfiguration::createWithDefaults() from a #[ServiceContext] method.'); + } + if (! $flowTracking->notifications) { + throw ConfigurationException::create(implode(', ', $subscribers) . ' would never be called, because FlowTrackingConfiguration::withNotifications(false) turned notifications off. Remove that call, or remove the attribute.'); + } + } + + private function refuseSubscribersThatCannotReceiveTheirNotification(InterfaceToCallRegistry $interfaceToCallRegistry): void + { + foreach ($this->subscriberMethods as $subscriber) { + $interfaceToCall = $interfaceToCallRegistry->getFor($subscriber['class'], $subscriber['method']); + if ($interfaceToCall->hasNoParameters()) { + continue; + } + $declaredType = $interfaceToCall->getFirstParameter()->getTypeDescriptor(); + if ($declaredType->acceptType(Type::create($subscriber['notification']))) { + continue; + } + + throw ConfigurationException::create(sprintf( + '%s on %s::%s is passed a %s as its first parameter, but declares it as %s, so every notification would fail. Declare that parameter as %s.', + $subscriber['attribute'], + $subscriber['class'], + $subscriber['method'], + $subscriber['notification'], + $declaredType->toString(), + $subscriber['notification'], + )); + } + } + + /** + * @return array + */ + private static function subscriberMethodsOf(AnnotationFinder $annotationFinder, string $attributeClassName, string $notificationClassName): array + { + $subscribers = []; + foreach ($annotationFinder->findAnnotatedMethods($attributeClassName) as $annotatedMethod) { + $subscribers[] = [ + 'attribute' => '#[' . substr((string) strrchr('\\' . $attributeClassName, '\\'), 1) . ']', + 'notification' => $notificationClassName, + 'class' => $annotatedMethod->getClassName(), + 'method' => $annotatedMethod->getMethodName(), + ]; + } + + return $subscribers; + } + + private static function registerTracingGateway(Configuration $messagingConfiguration, string $serviceClass, Definition $service): void + { + $messagingConfiguration->registerServiceDefinition($serviceClass, $service); + foreach ([ + 'isCompletedFor' => [GatewayPayloadBuilder::create('tracingId'), GatewayHeaderBuilder::create('destination', StoredTracingGateway::DESTINATION_HEADER), GatewayHeadersBuilder::create('metadata')], + 'isCompletedWithin' => [GatewayPayloadBuilder::create('tracingId'), GatewayHeaderBuilder::create('timeout', StoredTracingGateway::TIMEOUT_HEADER), GatewayHeaderBuilder::create('destination', StoredTracingGateway::DESTINATION_HEADER), GatewayHeadersBuilder::create('metadata')], + 'fetchFlow' => [GatewayPayloadBuilder::create('tracingId'), GatewayHeadersBuilder::create('metadata')], + ] as $method => $parameterConverters) { + $channel = 'ecotone.tracing.gateway.' . $method; + $messagingConfiguration + ->registerMessageHandler(MessageProcessorActivatorBuilder::createForMethod($serviceClass, $method)->withInputChannelName($channel)) + ->registerGatewayBuilder(GatewayProxyBuilder::create(TracingGateway::class, TracingGateway::class, $method, $channel)->withParameterConverters($parameterConverters)); + } + } + + /** + * @param object[] $extensionObjects + */ + private static function connectionReferenceOf(array $extensionObjects): string + { + $dbalConfiguration = ExtensionObjectResolver::resolveUnique(DbalConfiguration::class, $extensionObjects, DbalConfiguration::createWithDefaults()); + + return $dbalConfiguration->defaultConnectionReferenceNames[0] ?? DbalConnectionReference::DEFAULT; + } + + /** + * @param object[] $extensionObjects + */ + private static function multiTenancyWithoutDefaultConnectionOf(array $extensionObjects): ?MultiTenantConfiguration + { + foreach (ExtensionObjectResolver::resolve(MultiTenantConfiguration::class, $extensionObjects) as $multiTenantConfiguration) { + if ($multiTenantConfiguration->getReferenceName() === self::connectionReferenceOf($extensionObjects) && $multiTenantConfiguration->defaultConnectionName === null) { + return $multiTenantConfiguration; + } + } + + return null; + } + + /** + * @return array + */ + private static function destinationsByClass(AnnotationFinder $annotationFinder): array + { + $destinations = []; + foreach ([CommandHandler::class, EventHandler::class, QueryHandler::class, InternalHandler::class] as $handlerAttribute) { + foreach ($annotationFinder->findAnnotatedMethods($handlerAttribute) as $annotatedMethod) { + $destinations[$annotatedMethod->getClassName()][] = $annotatedMethod->getAnnotationForMethod()->getEndpointId(); + } + } + foreach ($annotationFinder->findAnnotatedClasses(Projection::class) as $projectionClass) { + $destinations[$projectionClass] = [$annotationFinder->getAttributeForClass($projectionClass, Projection::class)->name]; + } + + return array_map(fn (array $classDestinations): array => array_values(array_unique($classDestinations)), $destinations); + } + + /** + * @return string[] + */ + private static function pollingProjectionNames(AnnotationFinder $annotationFinder): array + { + $names = []; + foreach ($annotationFinder->findAnnotatedClasses(Projection::class) as $projectionClass) { + if ($annotationFinder->findAttributeForClass($projectionClass, Polling::class) !== null) { + $names[] = $annotationFinder->getAttributeForClass($projectionClass, Projection::class)->name; + } + } + + return $names; + } + + /** + * @return string[] + */ + private static function placementsOf(AnnotationFinder $annotationFinder, string $attributeClassName): array + { + $placements = []; + foreach ($annotationFinder->findAnnotatedMethods($attributeClassName) as $annotatedMethod) { + $placements[] = '#[' . substr((string) strrchr('\\' . $attributeClassName, '\\'), 1) . '] on ' . $annotatedMethod->getClassName() . '::' . $annotatedMethod->getMethodName(); + } + + return $placements; + } +} diff --git a/packages/Dbal/src/Tracing/ExpiredTraceRemover.php b/packages/Dbal/src/Tracing/ExpiredTraceRemover.php new file mode 100644 index 0000000000..88af6f18ee --- /dev/null +++ b/packages/Dbal/src/Tracing/ExpiredTraceRemover.php @@ -0,0 +1,42 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + */ +final class ExpiredTraceRemover +{ + public function __construct( + private TraceStore $traceStore, + private EcotoneClockInterface $clock, + private int $retentionInMilliseconds, + private int $removalBatchSize, + ) { + } + + public function removeExpiredTraces(): void + { + $this->traceStore->removeTracesStartedBefore($this->clock->now()->sub(Duration::milliseconds($this->retentionInMilliseconds)), $this->removalBatchSize); + } +} diff --git a/packages/Dbal/src/Tracing/InMemoryTraceStore.php b/packages/Dbal/src/Tracing/InMemoryTraceStore.php new file mode 100644 index 0000000000..4cf565b896 --- /dev/null +++ b/packages/Dbal/src/Tracing/InMemoryTraceStore.php @@ -0,0 +1,67 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + */ +final class InMemoryTraceStore implements TraceStore +{ + /** @var array> */ + private array $spansByTrace = []; + + public function append(array $spans): void + { + foreach ($spans as $span) { + if (isset($this->spansByTrace[$span->traceId][$span->spanId])) { + continue; + } + $this->spansByTrace[$span->traceId][$span->spanId] = $span; + } + } + + public function progressOf(string $traceId): TraceProgress + { + return TraceLedger::progressOf($traceId, array_values($this->spansByTrace[$traceId] ?? [])); + } + + public function stepsOf(string $traceId): array + { + return array_values($this->spansByTrace[$traceId] ?? []); + } + + public function removeTracesStartedBefore(DatePoint $startedBefore, int $batchSize): void + { + foreach ($this->spansByTrace as $traceId => $spans) { + if (min(array_map(fn (Span $span): DatePoint => $span->startedAt, $spans)) < $startedBefore) { + unset($this->spansByTrace[$traceId]); + } + } + } + + public function withinTransaction(Closure $work): mixed + { + return $work(); + } +} diff --git a/packages/Dbal/src/Tracing/StoredTracingGateway.php b/packages/Dbal/src/Tracing/StoredTracingGateway.php new file mode 100644 index 0000000000..86733b050c --- /dev/null +++ b/packages/Dbal/src/Tracing/StoredTracingGateway.php @@ -0,0 +1,157 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + */ +final class StoredTracingGateway +{ + public const DESTINATION_HEADER = 'ecotone_tracing_completion_destination'; + public const TIMEOUT_HEADER = 'ecotone_tracing_completion_timeout'; + private const POLL_INTERVAL_IN_MILLISECONDS = 50; + private const LONGEST_AWAIT_IN_MILLISECONDS = 15 * 60 * 1000; + + /** + * @param string[] $pollingProjectionNames + * @param array $destinationsByClass + */ + public function __construct( + private TraceStore $traceStore, + private EcotoneClockInterface $clock, + private array $pollingProjectionNames, + private array $destinationsByClass, + private ?string $tenantHeaderName, + private ?string $exampleTenant, + ) { + } + + public function isCompletedFor(string $tracingId, #[Header(self::DESTINATION_HEADER)] ?string $destination, #[Headers] array $headers): bool + { + $this->refuseMissingTenant('isCompletedFor($tracingId', $headers); + $tracingId = TracingId::normalize($tracingId); + + return $this->isCompletedNow($tracingId, $destination === null ? null : $this->destinationsOf($destination)); + } + + public function isCompletedWithin(string $tracingId, #[Header(self::TIMEOUT_HEADER)] Duration $timeout, #[Header(self::DESTINATION_HEADER)] ?string $destination, #[Headers] array $headers): bool + { + $this->refuseMissingTenant('isCompletedWithin($tracingId, $timeout', $headers); + $this->refuseWaitingLongerThanFifteenMinutes($timeout); + $tracingId = TracingId::normalize($tracingId); + $destinations = $destination === null ? null : $this->destinationsOf($destination); + $deadline = $this->clock->now()->add($timeout); + while (! $this->isCompletedNow($tracingId, $destinations)) { + if ($this->clock->now() >= $deadline) { + return false; + } + $this->clock->sleep(Duration::milliseconds(self::POLL_INTERVAL_IN_MILLISECONDS)); + } + + return true; + } + + public function fetchFlow(string $tracingId, #[Headers] array $headers): ?Flow + { + $this->refuseMissingTenant('fetchFlow($tracingId', $headers); + $tracingId = TracingId::normalize($tracingId); + $steps = $this->traceStore->stepsOf($tracingId); + if ($steps === []) { + return null; + } + usort($steps, fn (Span $first, Span $second): int => $first->startedAt <=> $second->startedAt); + $progress = TraceLedger::progressOf($tracingId, $steps); + + return new Flow($tracingId, $progress->isCompleted(), $progress->pendingDestinations(), $progress->destinations(), $steps); + } + + /** + * @param string[]|null $destinations + */ + private function isCompletedNow(string $tracingId, ?array $destinations): bool + { + $progress = $this->traceStore->progressOf($tracingId); + + return $destinations === null ? $progress->isCompleted() : $progress->isCompletedFor(...$destinations); + } + + private function refuseWaitingLongerThanFifteenMinutes(Duration $timeout): void + { + if ($timeout->inMilliseconds() <= self::LONGEST_AWAIT_IN_MILLISECONDS) { + return; + } + + throw InvalidArgumentException::create(sprintf( + 'TracingGateway::isCompletedWithin() was asked to wait %d milliseconds, but it blocks the calling process for as long as it waits, which past 15 minutes ties that process up for an extensive period. Wait at most 15 minutes, or react to the flow event-driven instead: a method with #[OnFlowStepCompleted] receives a FlowStepCompleted once each step of the flow committed, and can ask TracingGateway::isCompletedFor() whether the flow completed with it.', + $timeout->inMilliseconds(), + )); + } + + /** + * @param array $headers + */ + private function refuseMissingTenant(string $call, array $headers): void + { + if ($this->tenantHeaderName === null || isset($headers[$this->tenantHeaderName])) { + return; + } + + throw InvalidArgumentException::create(sprintf( + "TracingGateway reads a flow from the database of the tenant it ran for, and this call names no tenant. Pass the tenant as metadata, for example %s, metadata: ['%s' => '%s']).", + $call, + $this->tenantHeaderName, + $this->exampleTenant, + )); + } + + /** + * @return string[] + */ + private function destinationsOf(string $destination): array + { + $destinations = $this->destinationsByClass[$destination] ?? null; + if ($destinations === null && (class_exists($destination) || interface_exists($destination))) { + throw ConfigurationException::create(sprintf('%s has no handler or projection Ecotone registered, so no flow can reach it. Pass a class with a #[CommandHandler], #[EventHandler], #[QueryHandler] or #[InternalHandler] method, a #[Projection] class, an endpointId or a projection name.', $destination)); + } + $destinations ??= [$destination]; + foreach ($destinations as $resolvedDestination) { + $this->refusePollingProjection($resolvedDestination); + } + + return $destinations; + } + + private function refusePollingProjection(string $destination): void + { + if (in_array($destination, $this->pollingProjectionNames, true)) { + throw ConfigurationException::create(sprintf('Projection %s runs by #[Polling], so it is not part of the trace that appended its events and TracingGateway cannot wait for it. Await the handler that published the event, or make the projection event-driven by removing #[Polling].', $destination)); + } + } +} diff --git a/packages/Dbal/src/Tracing/StoringSpanRecorder.php b/packages/Dbal/src/Tracing/StoringSpanRecorder.php new file mode 100644 index 0000000000..f2051ab6e3 --- /dev/null +++ b/packages/Dbal/src/Tracing/StoringSpanRecorder.php @@ -0,0 +1,177 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + */ +final class StoringSpanRecorder implements SpanRecorder +{ + /** @var Span[] */ + private array $buffer = []; + private bool $scopeActive = false; + private ?ActiveSpan $wrappingSpan = null; + /** @var array */ + private array $storedSpanIds = []; + private bool $projectionBatchActive = false; + private bool $unitOfWorkActive = false; + /** @var Span[] */ + private array $committedSpans = []; + + public function __construct( + private TraceStore $traceStore, + private bool $publishesEndedSpans, + private bool $storesOnlyCarriedTraces, + ) { + } + + public function record(Span $span): void + { + if ($this->scopeActive && $this->isStored($span)) { + $this->buffer[] = $span; + } + } + + public function isUnitOfWorkActive(): bool + { + return $this->unitOfWorkActive; + } + + public function beginUnitOfWork(): void + { + $this->unitOfWorkActive = true; + } + + public function endUnitOfWork(): void + { + $this->unitOfWorkActive = false; + $this->committedSpans = []; + } + + public function openScope(?ActiveSpan $wrappingSpan): void + { + $this->scopeActive = true; + $this->wrappingSpan = $wrappingSpan; + } + + public function isScopeActive(): bool + { + return $this->scopeActive; + } + + public function closeScope(): void + { + $this->scopeActive = false; + $this->wrappingSpan = null; + $this->storedSpanIds = []; + } + + public function beginProjectionBatch(): void + { + $this->projectionBatchActive = true; + } + + public function isProjectionBatchActive(): bool + { + return $this->projectionBatchActive; + } + + public function endProjectionBatch(): void + { + $this->projectionBatchActive = false; + } + + public function withinTransaction(Closure $work): mixed + { + return $this->traceStore->withinTransaction($work); + } + + public function flushBuffered(): void + { + $this->store([]); + } + + public function flushWithOpenSpans(?ActiveSpan ...$innerOpenSpans): void + { + $openSpans = []; + foreach ([$this->wrappingSpan, ...$innerOpenSpans] as $openSpan) { + if ($openSpan !== null) { + $openSpans[$openSpan->context()->spanId] = $openSpan->endedSoFar(); + } + } + $this->store(array_values($openSpans)); + } + + /** + * @param Span[] $openSpansSoFar + */ + private function store(array $openSpansSoFar): void + { + $spans = []; + foreach ([...$this->buffer, ...$openSpansSoFar] as $span) { + if ($this->isStored($span) && ! isset($this->storedSpanIds[$span->spanId])) { + $spans[$span->spanId] = $span; + } + } + $this->buffer = []; + if ($spans === []) { + return; + } + + $spans = array_values($spans); + $this->traceStore->append($spans); + foreach ($spans as $span) { + $this->storedSpanIds[$span->spanId] = true; + } + $this->committedSpans = [...$this->committedSpans, ...$spans]; + } + + public function discard(): void + { + $this->buffer = []; + } + + /** + * @return FlowStepCompleted[] + */ + public function popCommittedNotifications(): array + { + $notifications = array_map(fn (Span $span): FlowStepCompleted => new FlowStepCompleted($span), $this->publishesEndedSpans ? $this->committedSpans : []); + $this->endUnitOfWork(); + + return $notifications; + } + + private function isStored(Span $span): bool + { + if ($span->attributes[SpanAttribute::TRACE_UNTRACKED] ?? false) { + return false; + } + + return ! $this->storesOnlyCarriedTraces || ($span->attributes[SpanAttribute::TRACE_CARRIED] ?? false); + } +} diff --git a/packages/Dbal/src/Tracing/TraceLedger.php b/packages/Dbal/src/Tracing/TraceLedger.php new file mode 100644 index 0000000000..ef253ba36e --- /dev/null +++ b/packages/Dbal/src/Tracing/TraceLedger.php @@ -0,0 +1,108 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + */ +final class TraceLedger +{ + /** + * @param Span[] $spans + */ + public static function progressOf(string $traceId, array $spans): TraceProgress + { + if ($spans === []) { + return TraceProgress::unknown($traceId); + } + + $rootEnded = false; + $causedBy = []; + $sent = []; + $processed = []; + $sentDeliveries = []; + $processedDeliveries = []; + foreach ($spans as $span) { + if (isset($span->attributes[MessageHeaders::MESSAGE_ID], $span->attributes[MessageHeaders::PARENT_MESSAGE_ID])) { + $causedBy[(string) $span->attributes[MessageHeaders::MESSAGE_ID]] = (string) $span->attributes[MessageHeaders::PARENT_MESSAGE_ID]; + } + if ($span->attributes[SpanAttribute::TRACE_ROOT] ?? false) { + $rootEnded = true; + } + if (isset($span->attributes[SpanAttribute::BATCH_DELIVERIES], $span->attributes[SpanAttribute::CHANNEL_NAME]) && $span->kind === SpanKind::Producer && $span->status !== SpanStatus::Error) { + $channel = (string) $span->attributes[SpanAttribute::CHANNEL_NAME]; + foreach (json_decode((string) $span->attributes[SpanAttribute::BATCH_DELIVERIES], true, flags: JSON_THROW_ON_ERROR) as $messageId => $endpointId) { + $sentDeliveries[$channel][(string) $messageId] = true; + if ($endpointId !== null) { + $sent[(string) $endpointId][(string) $messageId] = true; + } + } + + continue; + } + $delivery = self::deliveryOf($span); + if ($delivery !== null) { + [$channel, $messageId] = $delivery; + if ($span->kind === SpanKind::Producer && $span->status !== SpanStatus::Error) { + $sentDeliveries[$channel][$messageId] = true; + } elseif ($span->kind === SpanKind::Consumer) { + $processedDeliveries[$channel][$messageId] = true; + } + } + $destination = self::destinationOf($span); + if ($destination === null) { + continue; + } + $operation = $span->attributes[SpanAttribute::MESSAGING_OPERATION_TYPE] ?? null; + $messageId = (string) ($span->attributes[MessageHeaders::MESSAGE_ID] ?? $span->spanId); + if ($operation === SpanAttribute::OPERATION_SEND) { + $sent[$destination][$messageId] = true; + } elseif ($operation === SpanAttribute::OPERATION_PROCESS) { + $processed[$destination][$messageId] = true; + } + } + + return new TraceProgress($traceId, true, $rootEnded, $sent, $processed, $sentDeliveries, $processedDeliveries, $causedBy); + } + + public static function destinationOf(Span $span): ?string + { + $destination = $span->attributes[SpanAttribute::ENDPOINT_ID] ?? $span->attributes[SpanAttribute::PROJECTION_NAME] ?? null; + + return $destination === null ? null : (string) $destination; + } + + /** + * @return array{string, string}|null + */ + private static function deliveryOf(Span $span): ?array + { + $channel = $span->attributes[SpanAttribute::CHANNEL_NAME] ?? null; + $messageId = $span->attributes[MessageHeaders::MESSAGE_ID] ?? null; + + return $channel === null || $messageId === null ? null : [(string) $channel, (string) $messageId]; + } +} diff --git a/packages/Dbal/src/Tracing/TraceProgress.php b/packages/Dbal/src/Tracing/TraceProgress.php new file mode 100644 index 0000000000..264e6fb93a --- /dev/null +++ b/packages/Dbal/src/Tracing/TraceProgress.php @@ -0,0 +1,155 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + */ +final class TraceProgress +{ + /** + * @param array> $sent + * @param array> $processed + * @param array> $sentDeliveries + * @param array> $processedDeliveries + * @param array $causedBy + */ + public function __construct( + public readonly string $traceId, + public readonly bool $recorded, + public readonly bool $rootEnded, + public readonly array $sent, + public readonly array $processed, + public readonly array $sentDeliveries, + public readonly array $processedDeliveries, + public readonly array $causedBy, + ) { + } + + public static function unknown(string $traceId): self + { + return new self($traceId, false, false, [], [], [], [], []); + } + + public function isCompleted(): bool + { + return $this->recorded && $this->rootEnded && $this->pendingDestinations() === []; + } + + public function isCompletedFor(string ...$destinations): bool + { + if (! $this->recorded) { + return false; + } + $processed = []; + foreach ($destinations as $destination) { + if (array_diff_key($this->sent[$destination] ?? [], $this->processed[$destination] ?? []) !== []) { + return false; + } + $processed += $this->processed[$destination] ?? []; + } + if ($processed === []) { + return false; + } + + foreach ($this->pendingMessageIds() as $pendingMessageId) { + if (! $this->isCausedByOneOf($pendingMessageId, $processed)) { + return false; + } + } + + return true; + } + + /** + * @return string[] + */ + private function pendingMessageIds(): array + { + $pending = []; + foreach ($this->sent as $destination => $sent) { + $pending = [...$pending, ...array_keys(array_diff_key($sent, $this->processed[$destination] ?? []))]; + } + foreach ($this->sentDeliveries as $channel => $sent) { + $pending = [...$pending, ...array_keys(array_diff_key($sent, $this->processedDeliveries[$channel] ?? []))]; + } + + return array_map('strval', $pending); + } + + /** + * @param array $causes + */ + private function isCausedByOneOf(string $messageId, array $causes): bool + { + $visited = [$messageId => true]; + $current = $this->causedBy[$messageId] ?? null; + while ($current !== null && ! isset($visited[$current])) { + if (isset($causes[$current])) { + return true; + } + $visited[$current] = true; + $current = $this->causedBy[$current] ?? null; + } + + return false; + } + + /** + * @return string[] + */ + public function pendingDestinations(): array + { + $pending = []; + foreach ($this->sent as $destination => $sent) { + if (array_diff_key($sent, $this->processed[$destination] ?? []) !== []) { + $pending[] = $destination; + } + } + foreach ($this->sentDeliveries as $channel => $messages) { + if (array_diff_key($messages, $this->processedDeliveries[$channel] ?? []) !== []) { + $pending[] = $channel; + } + } + + return array_values(array_unique($pending)); + } + + public function reached(string ...$destinations): bool + { + foreach ($destinations as $destination) { + if (isset($this->sent[$destination]) || isset($this->processed[$destination])) { + return true; + } + } + + return false; + } + + /** + * @return string[] + */ + public function destinations(): array + { + return array_keys($this->sent + $this->processed); + } +} diff --git a/packages/Dbal/src/Tracing/TraceRecordingInterceptor.php b/packages/Dbal/src/Tracing/TraceRecordingInterceptor.php new file mode 100644 index 0000000000..8bc2fe94d1 --- /dev/null +++ b/packages/Dbal/src/Tracing/TraceRecordingInterceptor.php @@ -0,0 +1,130 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + */ +final class TraceRecordingInterceptor +{ + public function __construct( + private StoringSpanRecorder $recorder, + private Tracer $tracer, + private MessagingEntrypointService $messagingEntrypoint, + private LoggingGateway $logger, + ) { + } + + public function publishAfterCommit(MethodInvocation $methodInvocation): mixed + { + if ($this->recorder->isUnitOfWorkActive()) { + return $methodInvocation->proceed(); + } + + $this->recorder->beginUnitOfWork(); + try { + $result = $methodInvocation->proceed(); + } catch (Throwable $failure) { + $this->recorder->endUnitOfWork(); + + throw $failure; + } + foreach ($this->recorder->popCommittedNotifications() as $notification) { + $this->deliver($notification); + } + + return $result; + } + + public function recordUnitOfWork(MethodInvocation $methodInvocation): mixed + { + if ($this->recorder->isScopeActive()) { + return $methodInvocation->proceed(); + } + + $this->recorder->openScope($this->tracer->currentSpan()); + try { + $result = $methodInvocation->proceed(); + $this->recorder->flushWithOpenSpans(); + + return $result; + } catch (Throwable $failure) { + $this->recorder->discard(); + + throw $failure; + } finally { + $this->recorder->closeScope(); + } + } + + public function recordProjectionBatch(MethodInvocation $methodInvocation): mixed + { + if (! $this->recorder->isScopeActive() || $this->recorder->isProjectionBatchActive()) { + return $methodInvocation->proceed(); + } + + $this->recorder->beginProjectionBatch(); + try { + return $this->recorder->withinTransaction(function () use ($methodInvocation): mixed { + $processedEvents = $methodInvocation->proceed(); + if ($processedEvents === 0) { + $this->recorder->flushWithOpenSpans($this->tracer->currentSpan()); + } else { + $this->recorder->flushBuffered(); + } + + return $processedEvents; + }); + } catch (Throwable $failure) { + $this->recorder->discard(); + + throw $failure; + } finally { + $this->recorder->endProjectionBatch(); + } + } + + private function deliver(FlowStepCompleted $notification): void + { + $channel = OnFlowStepCompleted::CHANNEL; + $span = $this->tracer->startRootSpan('send ' . $channel, SpanKind::Producer, [ + SpanAttribute::MESSAGING_OPERATION_TYPE => SpanAttribute::OPERATION_SEND, + SpanAttribute::MESSAGING_DESTINATION_NAME => $channel, + SpanAttribute::TRACE_UNTRACKED => true, + ]); + try { + $this->messagingEntrypoint->send($notification, $channel); + $span->end(); + } catch (Throwable $failure) { + $span->fail($failure); + $this->logger->error(sprintf('A subscriber of %s failed and was skipped: %s', $channel, $failure->getMessage()), [], ['exception' => $failure]); + } + } +} diff --git a/packages/Dbal/src/Tracing/TraceStore.php b/packages/Dbal/src/Tracing/TraceStore.php new file mode 100644 index 0000000000..2b5c1e4b08 --- /dev/null +++ b/packages/Dbal/src/Tracing/TraceStore.php @@ -0,0 +1,46 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + */ +interface TraceStore +{ + /** + * @param Span[] $spans + */ + public function append(array $spans): void; + + public function progressOf(string $traceId): TraceProgress; + + /** + * @return Span[] + */ + public function stepsOf(string $traceId): array; + + public function removeTracesStartedBefore(DatePoint $startedBefore, int $batchSize): void; + + public function withinTransaction(Closure $work): mixed; +} diff --git a/packages/Dbal/src/Tracing/UnavailableTracingGateway.php b/packages/Dbal/src/Tracing/UnavailableTracingGateway.php new file mode 100644 index 0000000000..8e79dbd80a --- /dev/null +++ b/packages/Dbal/src/Tracing/UnavailableTracingGateway.php @@ -0,0 +1,38 @@ +refusal(); + } + + public function isCompletedWithin(string $tracingId): bool + { + throw $this->refusal(); + } + + public function fetchFlow(string $tracingId): ?Flow + { + throw $this->refusal(); + } + + private function refusal(): ConfigurationException + { + return ConfigurationException::create($this->refusal); + } +} diff --git a/packages/Dbal/tests/Conformance/FlowTrackingConformanceTest.php b/packages/Dbal/tests/Conformance/FlowTrackingConformanceTest.php new file mode 100644 index 0000000000..1a74adfed9 --- /dev/null +++ b/packages/Dbal/tests/Conformance/FlowTrackingConformanceTest.php @@ -0,0 +1,921 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + * + * @internal + */ +final class FlowTrackingConformanceTest extends DbalMessagingTestCase +{ + public function tearDown(): void + { + OutageSimulatingConnection::$databaseDown = false; + parent::tearDown(); + } + + public static function implementations(): iterable + { + yield 'in-memory' => ['in-memory']; + yield 'dbal' => ['dbal']; + } + + #[DataProvider('implementations')] + public function test_a_synchronous_flow_is_completed_when_the_bus_returns(string $implementation): void + { + $ecotone = $this->bootstrap($implementation, [$this->users()]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.registration')); + } + + #[DataProvider('implementations')] + public function test_an_asynchronous_handler_keeps_the_trace_pending_until_its_consumer_ran(string $implementation): void + { + $ecotone = $this->bootstrap($implementation, [$this->users(), $this->asynchronousNotifier()], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.notifier')); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.notifier')); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + #[DataProvider('implementations')] + public function test_waiting_for_a_pending_flow_returns_false_and_the_flow_names_what_is_still_pending(string $implementation): void + { + $ecotone = $this->bootstrap($implementation, [$this->users(), $this->asynchronousNotifier()], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + self::assertFalse($this->completion($ecotone)->isCompletedWithin($trace->traceId, Duration::milliseconds(100))); + self::assertSame(['user.notifier', 'async_channel'], $this->completion($ecotone)->fetchFlow($trace->traceId)->pending); + } + + #[DataProvider('implementations')] + public function test_a_step_subscriber_asking_the_gateway_sees_the_flow_completed_once_the_last_handler_committed(string $implementation): void + { + $subscriber = new class () { + /** @var array */ + public array $completedAfterStep = []; + + #[OnFlowStepCompleted] + public function whenStepCompleted(FlowStepCompleted $event, TracingGateway $tracingGateway): void + { + $this->completedAfterStep[$event->step->name] = $tracingGateway->isCompletedFor($event->step->traceId); + } + }; + $ecotone = $this->bootstrap($implementation, [$this->users(), $this->asynchronousNotifier(), $subscriber], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + self::assertNotContains(true, $subscriber->completedAfterStep); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::assertTrue($subscriber->completedAfterStep['receive async_channel']); + } + + #[DataProvider('implementations')] + public function test_a_failing_asynchronous_handler_leaves_the_trace_pending(string $implementation): void + { + $notifier = new class () { + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', endpointId: 'user.notifier')] + public function notify(string $userId): void + { + throw new RuntimeException('Notification failed'); + } + }; + $ecotone = $this->bootstrap($implementation, [$this->users(), $notifier], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + try { + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + } catch (RuntimeException) { + } + + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + #[DataProvider('implementations')] + public function test_a_destination_is_not_completed_while_work_upstream_of_it_can_still_send_it_more(string $implementation): void + { + $audit = new class () { + #[Asynchronous('audit_channel')] + #[EventHandler('user.*', endpointId: 'audit.recorder')] + public function record(string $userId): void + { + } + }; + $welcomer = new class () { + #[Asynchronous('welcome_channel')] + #[EventHandler('user.registered', endpointId: 'user.welcomer')] + public function welcome(string $userId, EventBus $eventBus): void + { + $eventBus->publishWithRouting('user.welcomed', $userId); + } + }; + $ecotone = $this->bootstrap($implementation, [$this->users(), $audit, $welcomer], [SimpleMessageChannelBuilder::createQueueChannel('audit_channel'), SimpleMessageChannelBuilder::createQueueChannel('welcome_channel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $ecotone->run('audit_channel', ExecutionPollingMetadata::createWithTestingSetup()); + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, 'audit.recorder')); + + $ecotone->run('welcome_channel', ExecutionPollingMetadata::createWithTestingSetup()); + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, 'audit.recorder')); + + $ecotone->run('audit_channel', ExecutionPollingMetadata::createWithTestingSetup()); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'audit.recorder')); + } + + #[DataProvider('implementations')] + public function test_a_destination_is_completed_while_only_work_it_started_itself_is_still_pending(string $implementation): void + { + $notifier = new class () { + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', endpointId: 'user.notifier')] + public function notify(string $userId, EventBus $eventBus): void + { + $eventBus->publishWithRouting('user.notified', $userId); + } + }; + $archiver = new class () { + #[Asynchronous('archive_channel')] + #[EventHandler('user.notified', endpointId: 'notification.archiver')] + public function archive(string $userId): void + { + } + }; + $ecotone = $this->bootstrap($implementation, [$this->users(), $notifier, $archiver], [SimpleMessageChannelBuilder::createQueueChannel('async_channel'), SimpleMessageChannelBuilder::createQueueChannel('archive_channel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.notifier')); + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, 'notification.archiver')); + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + #[DataProvider('implementations')] + public function test_expired_flows_beyond_one_removal_batch_are_all_removed_by_the_console_command(string $implementation): void + { + $ecotone = $this->bootstrapFlowTesting( + classesToResolve: [$this->users()::class], + containerOrAvailableServices: [$this->users(), DbalConnectionFactory::class => $this->getConnectionFactory()], + extensionObjects: [self::flowTracking($implementation, Duration::minutes(10))->withRemovalBatchSize(2)], + enabledModules: [ModulePackageList::DBAL_PACKAGE], + licenceKey: LicenceTesting::VALID_LICENCE, + )->setupDatabaseTables(); + $expired = []; + for ($flow = 1; $flow <= 5; $flow++) { + $expired[] = $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', (string) $flow, metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + } + $ecotone->advanceTimeBy(TimeSpan::withMinutes(11)); + $recent = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '6', metadata: [MessageHeaders::TRACEPARENT => $recent->toTraceparent()]); + + $ecotone->runConsoleCommand('ecotone:tracing:remove-expired-traces'); + + foreach ($expired as $trace) { + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + self::assertTrue($this->completion($ecotone)->isCompletedFor($recent->traceId)); + } + + public function test_a_removal_batch_size_below_one_is_refused_naming_the_minimum(): void + { + $this->expectException(ConfigurationException::class); + $this->expectExceptionMessage('FlowTrackingConfiguration::withRemovalBatchSize(0) would never remove an expired flow. Pass 1 or more; the default is 1000.'); + + FlowTrackingConfiguration::createWithDefaults()->withRemovalBatchSize(0); + } + + #[DataProvider('implementations')] + public function test_a_handler_class_is_completed_once_every_one_of_its_handlers_the_flow_reached_ran(string $implementation): void + { + $notifications = new class () { + #[Asynchronous('mail_channel')] + #[EventHandler('user.registered', endpointId: 'user.mailer')] + public function mail(string $userId): void + { + } + + #[Asynchronous('sms_channel')] + #[EventHandler('user.registered', endpointId: 'user.texter')] + public function text(string $userId): void + { + } + }; + $ecotone = $this->bootstrap($implementation, [$this->users(), $notifications], [SimpleMessageChannelBuilder::createQueueChannel('mail_channel'), SimpleMessageChannelBuilder::createQueueChannel('sms_channel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $ecotone->run('mail_channel', ExecutionPollingMetadata::createWithTestingSetup()); + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, $notifications::class)); + + $ecotone->run('sms_channel', ExecutionPollingMetadata::createWithTestingSetup()); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, $notifications::class)); + } + + #[DataProvider('implementations')] + public function test_traces_older_than_the_retention_are_removed_by_the_console_command(string $implementation): void + { + $ecotone = $this->bootstrap($implementation, [$this->users()], [], Duration::minutes(10)); + $expired = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $expired->toTraceparent()]); + + $ecotone->advanceTimeBy(TimeSpan::withMinutes(11)); + $recent = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '2', metadata: [MessageHeaders::TRACEPARENT => $recent->toTraceparent()]); + + $ecotone->runConsoleCommand('ecotone:tracing:remove-expired-traces'); + + self::assertFalse($this->completion($ecotone)->isCompletedFor($expired->traceId)); + self::assertTrue($this->completion($ecotone)->isCompletedFor($recent->traceId)); + } + + #[DataProvider('implementations')] + public function test_an_asynchronous_command_keeps_the_trace_pending_until_its_consumer_ran(string $implementation): void + { + $orders = new class () { + #[Asynchronous('async_channel')] + #[CommandHandler('order.place', endpointId: 'order.placing')] + public function place(string $orderId): void + { + } + }; + $ecotone = $this->bootstrap($implementation, [$orders], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('order.place', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'order.placing')); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_a_consumer_alone_does_not_complete_a_trace_whose_producer_stored_nothing(): void + { + $producer = $this->bootstrapFlowTesting( + classesToResolve: [$this->asynchronousNotifier()::class], + containerOrAvailableServices: [$this->asynchronousNotifier(), DbalConnectionFactory::class => $this->getConnectionFactory()], + extensionObjects: [DbalBackedMessageChannelBuilder::create('async_channel')], + enabledModules: [ModulePackageList::DBAL_PACKAGE], + )->setupDatabaseTables(); + $consumer = $this->bootstrap('dbal', [$this->asynchronousNotifier()], [DbalBackedMessageChannelBuilder::create('async_channel')]); + $trace = SpanContext::newTrace(); + + $producer->publishEventWithRouting('user.registered', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + $consumer->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::assertTrue($this->completion($consumer)->isCompletedFor($trace->traceId, 'user.notifier')); + self::assertFalse($this->completion($consumer)->isCompletedFor($trace->traceId)); + } + + public function test_a_flow_started_by_a_distributed_command_completes_once_the_receiving_service_handled_it(): void + { + $sharedChannel = SimpleMessageChannelBuilder::createQueueChannel('distributed_ticket_channel'); + $tickets = new class () { + public int $created = 0; + + #[Distributed] + #[CommandHandler('ticket.create', endpointId: 'ticket.creation')] + public function create(string $description): void + { + $this->created++; + } + }; + $userService = $this->bootstrap('dbal', [], [$sharedChannel, DistributedServiceMap::initialize()->withCommandMapping(targetServiceName: 'ticket_service', channelName: 'distributed_ticket_channel'), ServiceConfiguration::createWithDefaults()->withServiceName('user_service')]); + $ticketService = $this->bootstrap('dbal', [$tickets], [$sharedChannel, ServiceConfiguration::createWithDefaults()->withServiceName('ticket_service')]); + $tracingId = '0af7651916cd43dd8448eb211c80319c'; + + $userService->getDistributedBus()->sendCommand('ticket_service', 'ticket.create', 'User changed billing address', metadata: [MessageHeaders::TRACING_ID => $tracingId]); + self::assertFalse($this->completion($userService)->isCompletedFor($tracingId)); + + $ticketService->run('distributed_ticket_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::assertSame(1, $tickets->created); + self::assertTrue($this->completion($userService)->isCompletedFor($tracingId)); + } + + public function test_a_send_rolled_back_with_its_failed_handler_is_not_awaited_after_the_retry_succeeded(): void + { + $notifier = new class () { + public int $attempts = 0; + + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', endpointId: 'user.notifier')] + public function notify(string $userId, EventBus $eventBus): void + { + $this->attempts++; + $eventBus->publishWithRouting('user.notified', $userId); + if ($this->attempts === 1) { + throw new RuntimeException('Notification failed once'); + } + } + + #[Asynchronous('audit_channel')] + #[EventHandler('user.notified', endpointId: 'user.auditor')] + public function audit(string $userId): void + { + } + }; + $ecotone = $this->bootstrap('dbal', [$this->users(), $notifier], [DbalBackedMessageChannelBuilder::create('async_channel'), DbalBackedMessageChannelBuilder::create('audit_channel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + try { + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + } catch (RuntimeException) { + } + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + $ecotone->run('audit_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::assertSame(2, $notifier->attempts); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + #[DataProvider('implementations')] + public function test_a_failing_asynchronous_handler_stores_nothing_by_default(string $implementation): void + { + $subscriber = $this->flowStepCompletedSubscriber(); + $ecotone = $this->bootstrap($implementation, [$this->users(), $this->failingNotifier(), $subscriber], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $ecotone->sendCommandWithRouting('user.register', '1'); + $storedBeforeTheConsumerRan = $subscriber->endedSpans; + + try { + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + } catch (RuntimeException) { + } + + self::assertSame($storedBeforeTheConsumerRan, $subscriber->endedSpans); + } + + #[DataProvider('implementations')] + public function test_a_consumer_that_swallowed_its_handler_failure_leaves_the_trace_pending(string $implementation): void + { + $ecotone = $this->bootstrap($implementation, [$this->users(), $this->failingNotifier()], [SimpleMessageChannelBuilder::createQueueChannel('async_channel'), ServiceConfiguration::createWithDefaults()->withDefaultErrorChannel('errorChannel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup(stopOnError: false)); + + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + #[DataProvider('implementations')] + public function test_a_dead_lettered_message_replayed_later_completes_the_trace_it_came_from(string $implementation): void + { + $notifier = new class () { + public int $attempts = 0; + + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', endpointId: 'user.notifier')] + public function notify(string $userId): void + { + if (++$this->attempts === 1) { + throw new RuntimeException('Notification failed once'); + } + } + }; + $ecotone = $this->bootstrap($implementation, [$this->users(), $notifier], [ + SimpleMessageChannelBuilder::createQueueChannel('async_channel'), + ServiceConfiguration::createWithDefaults()->withDefaultErrorChannel(DbalDeadLetterBuilder::STORE_CHANNEL), + InstantRetryConfiguration::createWithDefaults()->withAsynchronousEndpointsRetry(false), + ]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup(stopOnError: false)); + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + + $ecotone->getGateway(DeadLetterGateway::class)->replayAll(); + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::assertSame(2, $notifier->attempts); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + #[DataProvider('implementations')] + public function test_a_pending_flow_older_than_the_retention_is_removed_like_a_completed_one(string $implementation): void + { + $ecotone = $this->bootstrap($implementation, [$this->users(), $this->asynchronousNotifier()], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')], Duration::minutes(10)); + $abandoned = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $abandoned->toTraceparent()]); + + $ecotone->advanceTimeBy(TimeSpan::withMinutes(11)); + $ecotone->runConsoleCommand('ecotone:tracing:remove-expired-traces'); + + self::assertNull($this->completion($ecotone)->fetchFlow($abandoned->traceId)); + } + + #[DataProvider('implementations')] + public function test_a_subscriber_that_throws_does_not_fail_the_committed_command_and_is_not_retried(string $implementation): void + { + $subscriber = new class () { + /** @var string[] */ + public array $completedTraces = []; + + #[OnFlowStepCompleted] + public function whenStepCompleted(FlowStepCompleted $event): void + { + $this->completedTraces[] = $event->step->traceId; + if (count($this->completedTraces) === 1) { + throw new RuntimeException('Websocket is down'); + } + } + }; + $ecotone = $this->bootstrap($implementation, [$this->users(), $subscriber]); + $first = SpanContext::newTrace(); + $second = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $first->toTraceparent()]); + $stepsOfFirst = count($subscriber->completedTraces); + + $ecotone->sendCommandWithRouting('user.register', '2', metadata: [MessageHeaders::TRACEPARENT => $second->toTraceparent()]); + + self::assertSame([$first->traceId], array_values(array_unique(array_slice($subscriber->completedTraces, 0, $stepsOfFirst)))); + self::assertSame([$second->traceId], array_values(array_unique(array_slice($subscriber->completedTraces, $stepsOfFirst)))); + } + + #[DataProvider('implementations')] + public function test_a_flow_that_receives_new_work_after_completing_is_pending_until_that_work_completes(string $implementation): void + { + $orders = new class () { + #[Asynchronous('async_channel')] + #[CommandHandler('order.place', endpointId: 'order.placing')] + public function place(string $orderId): void + { + } + }; + $ecotone = $this->bootstrap($implementation, [$this->users(), $orders], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + + $ecotone->sendCommandWithRouting('order.place', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_a_channel_published_through_the_outbox_completes_once_its_consumer_ran(): void + { + $ecotone = $this->bootstrap('dbal', [$this->users(), $this->asynchronousNotifier()], [DbalBackedMessageChannelBuilder::create('async_channel')->withHighThroughputPublishing()]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.notifier')); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_flow_tracking_stores_each_flow_in_the_database_of_its_tenant(): void + { + $ecotone = $this->bootstrapMultiTenant([$this->users(), $this->asynchronousNotifier()], $this->connectionForTenantA()); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent(), 'tenant' => 'tenant_a']); + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, metadata: ['tenant' => 'tenant_a'])); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, metadata: ['tenant' => 'tenant_a'])); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.notifier', metadata: ['tenant' => 'tenant_a'])); + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, metadata: ['tenant' => 'tenant_b'])); + self::assertTrue($this->completion($ecotone)->isCompletedWithin($trace->traceId, Duration::milliseconds(100), metadata: ['tenant' => 'tenant_a'])); + self::assertTrue($this->completion($ecotone)->fetchFlow($trace->traceId, metadata: ['tenant' => 'tenant_a'])->isCompleted); + self::assertNull($this->completion($ecotone)->fetchFlow($trace->traceId, metadata: ['tenant' => 'tenant_b'])); + } + + public function test_a_tenant_flow_is_written_inside_the_tenant_transaction_and_rolls_back_with_it(): void + { + $tenantA = new DbalConnectionFactory([ + 'dsn' => getenv('DATABASE_DSN') ?: 'pgsql://ecotone:secret@localhost:5432/ecotone', + 'connection' => ['wrapperClass' => CommitFailingConnection::class], + ]); + $ecotone = $this->bootstrapMultiTenant([$this->users()], $tenantA); + $trace = SpanContext::newTrace(); + + CommitFailingConnection::$failsCommit = true; + try { + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent(), 'tenant' => 'tenant_a']); + self::fail('The tenant commit was expected to fail'); + } catch (RuntimeException $failure) { + self::assertSame('The commit failed', $failure->getMessage()); + } finally { + CommitFailingConnection::$failsCommit = false; + } + + self::assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, metadata: ['tenant' => 'tenant_a'])); + } + + public function test_expired_flows_are_removed_from_the_database_of_the_tenant_the_console_command_names(): void + { + $ecotone = $this->bootstrapMultiTenant([$this->users()], $this->connectionForTenantA()); + $ofTenantA = SpanContext::newTrace(); + $ofTenantB = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $ofTenantA->toTraceparent(), 'tenant' => 'tenant_a']); + $ecotone->sendCommandWithRouting('user.register', '2', metadata: [MessageHeaders::TRACEPARENT => $ofTenantB->toTraceparent(), 'tenant' => 'tenant_b']); + $ecotone->advanceTimeBy(TimeSpan::withDays(8)); + + $ecotone->runConsoleCommand('ecotone:tracing:remove-expired-traces', ['header' => ['tenant:tenant_a']]); + + self::assertFalse($this->completion($ecotone)->isCompletedFor($ofTenantA->traceId, metadata: ['tenant' => 'tenant_a'])); + self::assertTrue($this->completion($ecotone)->isCompletedFor($ofTenantB->traceId, metadata: ['tenant' => 'tenant_b'])); + } + + public function test_asking_the_tracing_gateway_of_a_multi_tenant_application_without_a_tenant_names_the_metadata_it_needs(): void + { + $ecotone = $this->bootstrapMultiTenant([$this->users()], $this->connectionForTenantA()); + + $this->expectExceptionMessage("TracingGateway reads a flow from the database of the tenant it ran for, and this call names no tenant. Pass the tenant as metadata, for example isCompletedFor(\$tracingId, metadata: ['tenant' => 'tenant_a'])."); + + $this->completion($ecotone)->isCompletedFor(SpanContext::newTrace()->traceId); + } + + public function test_a_database_outage_right_after_the_consumer_committed_does_not_fail_the_consumer(): void + { + $notifier = new class () { + public int $handled = 0; + + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', endpointId: 'user.notifier')] + public function notify(string $userId): void + { + $this->handled++; + } + }; + $outage = $this->outageRightAfterTheTransaction(); + $ecotone = $this->bootstrapOnAConnectionThatCanGoDown([$this->users(), $notifier, $outage], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $ecotone->sendCommandWithRouting('user.register', '1'); + + $outage->armed = true; + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::assertSame(1, $notifier->handled); + } + + public function test_a_database_outage_right_after_the_command_committed_does_not_fail_the_command(): void + { + $users = new class () { + public int $registered = 0; + + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId): void + { + $this->registered++; + } + }; + $outage = $this->outageRightAfterTheTransaction(); + $ecotone = $this->bootstrapOnAConnectionThatCanGoDown([$users, $outage]); + + $outage->armed = true; + $ecotone->sendCommandWithRouting('user.register', '1'); + + self::assertSame(1, $users->registered); + } + + public function test_a_database_outage_right_after_an_asynchronous_projection_committed_does_not_fail_its_consumer(): void + { + $projection = new #[Projection('users_list'), FromStream('users_stream'), Asynchronous('async_channel')] class () { + public int $projected = 0; + + #[EventHandler('user.registered')] + public function whenRegistered(array $event): void + { + $this->projected++; + } + }; + $outage = new class () { + public bool $armed = false; + + #[Around(precedence: Precedence::DATABASE_TRANSACTION_PRECEDENCE + 2, pointcut: 'Ecotone\\Api\\Attribute\\AsynchronousRunningEndpoint')] + public function goDownAfterTheProjectionRan(MethodInvocation $methodInvocation): mixed + { + $result = $methodInvocation->proceed(); + if ($this->armed) { + OutageSimulatingConnection::$databaseDown = true; + } + + return $result; + } + }; + $ecotone = $this->bootstrapOnAConnectionThatCanGoDown([$projection, $outage], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $ecotone->withEvents([Event::createWithType('user.registered', ['userId' => '1'])]); + $trace = SpanContext::newTrace(); + $ecotone->publishEventWithRouting('user.registered', [], metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $outage->armed = true; + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + $outage->armed = false; + OutageSimulatingConnection::$databaseDown = false; + + self::assertSame(1, $projection->projected); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'users_list')); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_a_database_outage_inside_the_consumer_transaction_leaves_the_message_to_be_retried(): void + { + $notifier = new class () { + public int $handled = 0; + + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', endpointId: 'user.notifier')] + public function notify(string $userId): void + { + $this->handled++; + OutageSimulatingConnection::$databaseDown = $this->handled === 1; + } + }; + $recovery = new class () { + #[Around(precedence: Precedence::DATABASE_TRANSACTION_PRECEDENCE - 1, pointcut: 'Ecotone\\Api\\Attribute\\AsynchronousRunningEndpoint')] + public function comeBackBeforeEachAttempt(MethodInvocation $methodInvocation): mixed + { + OutageSimulatingConnection::$databaseDown = false; + + return $methodInvocation->proceed(); + } + }; + $ecotone = $this->bootstrapOnAConnectionThatCanGoDown([$this->users(), $notifier, $recovery], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::assertSame(2, $notifier->handled); + self::assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_completion_recorded_by_one_instance_is_visible_to_another_sharing_the_database(): void + { + $producer = $this->bootstrap('dbal', [$this->users()]); + $observer = $this->bootstrap('dbal', [$this->users()]); + $trace = SpanContext::newTrace(); + + $producer->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + self::assertTrue($this->completion($observer)->isCompletedFor($trace->traceId)); + } + + /** + * @param object[] $services + * @param object[] $extensionObjects + */ + private function bootstrap(string $implementation, array $services, array $extensionObjects = [], ?Duration $retention = null): FlowTestSupport + { + return $this->bootstrapFlowTesting( + classesToResolve: array_map(fn (object $service): string => $service::class, $services), + containerOrAvailableServices: [...$services, DbalConnectionFactory::class => $this->getConnectionFactory()], + extensionObjects: [self::flowTracking($implementation, $retention), ...$extensionObjects], + enabledModules: [ModulePackageList::DBAL_PACKAGE], + licenceKey: LicenceTesting::VALID_LICENCE, + )->setupDatabaseTables(); + } + + private static function flowTracking(string $implementation, ?Duration $retention): FlowTrackingConfiguration + { + $flowTracking = FlowTrackingConfiguration::createWithDefaults()->withInMemoryStorage($implementation === 'in-memory'); + + return $retention === null ? $flowTracking : $flowTracking->withRetention($retention); + } + + private function completion(FlowTestSupport $ecotone): TracingGateway + { + return $ecotone->getGateway(TracingGateway::class); + } + + private function users(): object + { + return new class () { + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId, EventBus $eventBus): void + { + $eventBus->publishWithRouting('user.registered', $userId); + } + }; + } + + /** + * @param object[] $services + */ + private function bootstrapMultiTenant(array $services, DbalConnectionFactory $tenantAConnection): FlowTestSupport + { + $ecotone = $this->bootstrapFlowTesting( + classesToResolve: array_map(fn (object $service): string => $service::class, $services), + containerOrAvailableServices: [...$services, 'tenant_a_connection' => $tenantAConnection, 'tenant_b_connection' => $this->connectionForTenantB()], + extensionObjects: [ + FlowTrackingConfiguration::createWithDefaults(), + MultiTenantConfiguration::create('tenant', ['tenant_a' => 'tenant_a_connection', 'tenant_b' => 'tenant_b_connection']), + SimpleMessageChannelBuilder::createQueueChannel('async_channel'), + ], + enabledModules: [ModulePackageList::DBAL_PACKAGE], + testConfiguration: TestConfiguration::createWithDefaults()->withDatabaseTablesSetUpOnBootstrap(false), + licenceKey: LicenceTesting::VALID_LICENCE, + ); + $ecotone->runConsoleCommand('ecotone:migration:database:delete', ['force' => true]); + $ecotone->runConsoleCommand('ecotone:migration:database:setup', ['initialize' => true]); + + return $ecotone; + } + + private function outageRightAfterTheTransaction(): object + { + return new class () { + public bool $armed = false; + + #[Around(precedence: Precedence::DATABASE_TRANSACTION_PRECEDENCE - 1, pointcut: 'Ecotone\\Api\\Gateway\\CommandBus||Ecotone\\Api\\Attribute\\AsynchronousRunningEndpoint')] + public function goDownAfterTheTransaction(MethodInvocation $methodInvocation): mixed + { + $result = $methodInvocation->proceed(); + if ($this->armed) { + OutageSimulatingConnection::$databaseDown = true; + } + + return $result; + } + }; + } + + /** + * @param object[] $services + * @param object[] $extensionObjects + */ + private function bootstrapOnAConnectionThatCanGoDown(array $services, array $extensionObjects = []): FlowTestSupport + { + $connectionFactory = new DbalConnectionFactory([ + 'dsn' => getenv('DATABASE_DSN') ?: 'pgsql://ecotone:secret@localhost:5432/ecotone', + 'connection' => ['wrapperClass' => OutageSimulatingConnection::class], + ]); + + return $this->bootstrapFlowTesting( + classesToResolve: array_map(fn (object $service): string => $service::class, $services), + containerOrAvailableServices: [...$services, DbalConnectionFactory::class => $connectionFactory], + extensionObjects: [FlowTrackingConfiguration::createWithDefaults(), ...$extensionObjects], + enabledModules: [ModulePackageList::DBAL_PACKAGE], + licenceKey: LicenceTesting::VALID_LICENCE, + )->setupDatabaseTables(); + } + + private function failingNotifier(): object + { + return new class () { + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', endpointId: 'user.notifier')] + public function notify(string $userId): void + { + throw new RuntimeException('Notification failed'); + } + }; + } + + private function flowStepCompletedSubscriber(): object + { + return new class () { + /** @var array */ + public array $endedSpans = []; + + #[OnFlowStepCompleted] + public function whenEnded(FlowStepCompleted $event): void + { + $this->endedSpans[$event->step->name] = $event->step->status; + } + }; + } + + private function asynchronousNotifier(): object + { + return new class () { + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', endpointId: 'user.notifier')] + public function notify(string $userId): void + { + } + }; + } +} + +final class OutageSimulatingConnection extends Connection +{ + public static bool $databaseDown = false; + + public function executeQuery(string $sql, array $params = [], array $types = [], ?QueryCacheProfile $qcp = null): Result + { + self::failWhenDown(); + + return parent::executeQuery($sql, $params, $types, $qcp); + } + + public function executeStatement(string $sql, array $params = [], array $types = []): int|string + { + self::failWhenDown(); + + return parent::executeStatement($sql, $params, $types); + } + + private static function failWhenDown(): void + { + if (self::$databaseDown) { + throw new RuntimeException('The database is down'); + } + } +} + +final class CommitFailingConnection extends Connection +{ + public static bool $failsCommit = false; + + public function commit(): void + { + if (self::$failsCommit && $this->getTransactionNestingLevel() === 1) { + $this->rollBack(); + + throw new RuntimeException('The commit failed'); + } + + parent::commit(); + } +} diff --git a/packages/Dbal/tests/DbalMessagingTestCase.php b/packages/Dbal/tests/DbalMessagingTestCase.php index 0a2fe4edcb..69728fda47 100644 --- a/packages/Dbal/tests/DbalMessagingTestCase.php +++ b/packages/Dbal/tests/DbalMessagingTestCase.php @@ -79,6 +79,7 @@ public static function cleanUpDbalTables(Connection $connection): void self::deleteTable(DbalDeadLetterStore::DEFAULT_DEAD_LETTER_TABLE, $connection); self::deleteTable(DbalDocumentStore::ECOTONE_DOCUMENT_STORE, $connection); self::deleteTable(DbalHandledMessages::DEFAULT_DEDUPLICATION_TABLE, $connection); + self::deleteTable(\Ecotone\Dbal\Database\SpansTableManager::DEFAULT_TABLE, $connection); self::deleteTable('persons', $connection); self::deleteTable('activities', $connection); } diff --git a/packages/Dbal/tests/Integration/DbalChannelTracingTest.php b/packages/Dbal/tests/Integration/DbalChannelTracingTest.php new file mode 100644 index 0000000000..362deeb353 --- /dev/null +++ b/packages/Dbal/tests/Integration/DbalChannelTracingTest.php @@ -0,0 +1,244 @@ +bootstrap($recorder, [$this->asynchronousOrders()], [DbalBackedMessageChannelBuilder::create('orders')]); + + ChannelOutageConnection::$databaseDown = true; + $this->assertNotNull($this->failureOf(fn () => $ecotone->sendCommandWithRouting('order.place', '1'))); + ChannelOutageConnection::$databaseDown = false; + + $this->assertSame(SpanStatus::Error, self::singleSpanNamed('send orders', $recorder)->status); + $recorder->clear(); + $ecotone->sendCommandWithRouting('order.place', '2'); + $this->assertSame('send command', self::rootSpan($recorder)->name); + } + + public function test_a_failing_handler_whose_error_channel_cannot_store_the_message_still_ends_every_span(): void + { + $recorder = new InMemorySpanRecorder(); + $orders = new class () { + #[Asynchronous('orders')] + #[CommandHandler('order.place', endpointId: 'order.placing')] + public function place(string $orderId): void + { + ChannelOutageConnection::$databaseDown = true; + + throw new RuntimeException('Handler failure'); + } + }; + $ecotone = $this->bootstrap($recorder, [$orders], [ + DbalBackedMessageChannelBuilder::create('orders'), + DbalBackedMessageChannelBuilder::create('error_channel'), + ServiceConfiguration::createWithDefaults()->withDefaultErrorChannel('error_channel'), + ]); + $ecotone->sendCommandWithRouting('order.place', '1'); + $recorder->clear(); + + $this->failureOf(fn () => $ecotone->run('orders', ExecutionPollingMetadata::createWithTestingSetup(stopOnError: false))); + ChannelOutageConnection::$databaseDown = false; + + self::assertEverySpanNamedHasStatus('process class@anonymous::place', SpanStatus::Error, $recorder); + self::assertEverySpanNamedHasStatus('send error_channel', SpanStatus::Error, $recorder); + $recorder->clear(); + $ecotone->sendCommandWithRouting('order.place', '2'); + $this->assertSame('send command', self::rootSpan($recorder)->name); + } + + public function test_a_distributed_send_that_fails_because_the_database_is_down_fails_its_span_and_leaves_no_open_span_behind(): void + { + $recorder = new InMemorySpanRecorder(); + $orders = new class () { + #[CommandHandler('order.place')] + public function place(string $orderId): void + { + } + }; + $ecotone = $this->bootstrap($recorder, [$orders], [ + DbalBackedMessageChannelBuilder::create('distributed_channel'), + DistributedServiceMap::initialize()->withCommandMapping(targetServiceName: 'ticket_service', channelName: 'distributed_channel'), + ServiceConfiguration::createWithDefaults()->withServiceName('order_service'), + ]); + + ChannelOutageConnection::$databaseDown = true; + $this->assertNotNull($this->failureOf(fn () => $ecotone->getDistributedBus()->sendCommand('ticket_service', 'ticket.create', 'User changed billing address'))); + ChannelOutageConnection::$databaseDown = false; + + $this->assertSame(SpanStatus::Error, self::singleSpanNamed('send distributed', $recorder)->status); + $recorder->clear(); + $ecotone->sendCommandWithRouting('order.place', '1'); + $this->assertSame('send command', self::rootSpan($recorder)->name); + } + + public function test_a_consumer_receiving_a_message_sent_by_an_application_without_tracing_starts_its_own_trace(): void + { + $producer = $this->bootstrapWithoutTracing([$this->asynchronousOrders()]); + $recorder = new InMemorySpanRecorder(); + $consumer = $this->bootstrap($recorder, [$this->asynchronousOrders()], [DbalBackedMessageChannelBuilder::create('orders')]); + + $producer->sendCommandWithRouting('order.place', '1'); + $consumer->run('orders', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertTrue(self::singleSpanNamed('receive orders', $recorder)->isRoot()); + } + + public function test_an_invalid_traceparent_on_a_received_message_is_ignored_and_the_consumer_starts_its_own_trace(): void + { + $producer = $this->bootstrapWithoutTracing([$this->asynchronousOrders()]); + $recorder = new InMemorySpanRecorder(); + $consumer = $this->bootstrap($recorder, [$this->asynchronousOrders()], [DbalBackedMessageChannelBuilder::create('orders')]); + + $producer->sendCommandWithRouting('order.place', '1', metadata: [MessageHeaders::TRACEPARENT => 'not-a-traceparent']); + $consumer->run('orders', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertTrue(self::singleSpanNamed('receive orders', $recorder)->isRoot()); + } + + /** + * @param object[] $services + * @param object[] $extensionObjects + */ + private function bootstrap(InMemorySpanRecorder $recorder, array $services, array $extensionObjects): FlowTestSupport + { + return $this->bootstrapFlowTesting( + classesToResolve: array_map(fn (object $service): string => $service::class, $services), + containerOrAvailableServices: [...$services, InMemorySpanRecorder::class => $recorder, DbalConnectionFactory::class => self::connectionThatCanGoDown()], + extensionObjects: [TracingConfiguration::createWithDefaults()->withSpanRecorder(InMemorySpanRecorder::class), ...$extensionObjects], + enabledModules: [ModulePackageList::DBAL_PACKAGE], + licenceKey: LicenceTesting::VALID_LICENCE, + )->setupDatabaseTables(); + } + + /** + * @param object[] $services + */ + private function bootstrapWithoutTracing(array $services): FlowTestSupport + { + return $this->bootstrapFlowTesting( + classesToResolve: array_map(fn (object $service): string => $service::class, $services), + containerOrAvailableServices: [...$services, DbalConnectionFactory::class => self::connectionThatCanGoDown()], + extensionObjects: [DbalBackedMessageChannelBuilder::create('orders')], + enabledModules: [ModulePackageList::DBAL_PACKAGE], + )->setupDatabaseTables(); + } + + private static function connectionThatCanGoDown(): DbalConnectionFactory + { + return new DbalConnectionFactory([ + 'dsn' => getenv('DATABASE_DSN') ?: 'pgsql://ecotone:secret@localhost:5432/ecotone', + 'connection' => ['wrapperClass' => ChannelOutageConnection::class], + ]); + } + + private function asynchronousOrders(): object + { + return new class () { + #[Asynchronous('orders')] + #[CommandHandler('order.place', endpointId: 'order.placing')] + public function place(string $orderId): void + { + } + }; + } + + private function failureOf(callable $action): ?Throwable + { + try { + $action(); + } catch (Throwable $failure) { + return $failure; + } + + return null; + } + + private static function singleSpanNamed(string $name, InMemorySpanRecorder $recorder): Span + { + $spans = array_values(array_filter($recorder->recordedSpans(), fn (Span $span): bool => $span->name === $name)); + self::assertCount(1, $spans, 'Expected exactly one span named ' . $name . ', got: ' . implode(', ', array_map(fn (Span $span): string => $span->name, $recorder->recordedSpans()))); + + return $spans[0]; + } + + private static function assertEverySpanNamedHasStatus(string $name, SpanStatus $status, InMemorySpanRecorder $recorder): void + { + $spans = array_values(array_filter($recorder->recordedSpans(), fn (Span $span): bool => preg_replace('/^process class@anonymous\S*::/', 'process class@anonymous::', $span->name) === $name)); + self::assertNotEmpty($spans, 'Expected spans named ' . $name . ', got: ' . implode(', ', array_map(fn (Span $span): string => $span->name, $recorder->recordedSpans()))); + foreach ($spans as $span) { + self::assertSame($status, $span->status, $name); + } + } + + private static function rootSpan(InMemorySpanRecorder $recorder): Span + { + $roots = array_values(array_filter($recorder->recordedSpans(), fn (Span $span): bool => $span->isRoot())); + self::assertCount(1, $roots, 'Expected exactly one root span, got: ' . implode(', ', array_map(fn (Span $span): string => $span->name, $roots))); + + return $roots[0]; + } +} + +final class ChannelOutageConnection extends Connection +{ + public static bool $databaseDown = false; + + public function executeQuery(string $sql, array $params = [], array $types = [], ?QueryCacheProfile $qcp = null): Result + { + self::failWhenDown(); + + return parent::executeQuery($sql, $params, $types, $qcp); + } + + public function executeStatement(string $sql, array $params = [], array $types = []): int|string + { + self::failWhenDown(); + + return parent::executeStatement($sql, $params, $types); + } + + private static function failWhenDown(): void + { + if (self::$databaseDown) { + throw new RuntimeException('The database is down'); + } + } +} diff --git a/packages/Dbal/tests/Integration/Tracing/FlowTrackingRaceTest.php b/packages/Dbal/tests/Integration/Tracing/FlowTrackingRaceTest.php new file mode 100644 index 0000000000..c818bb4e4d --- /dev/null +++ b/packages/Dbal/tests/Integration/Tracing/FlowTrackingRaceTest.php @@ -0,0 +1,207 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + * + * @internal + */ +#[RequiresPhpExtension('pcntl')] +#[RequiresPhpExtension('posix')] +final class FlowTrackingRaceTest extends DbalMessagingTestCase +{ + private const MILLISECONDS_THE_FIRST_CONSUMER_STAYS_UNCOMMITTED = 1000; + + public static string $observationsFile = ''; + + public function tearDown(): void + { + CommitHoldingConnection::$beforeCommit = null; + parent::tearDown(); + } + + public function test_two_consumers_of_one_flow_commit_without_waiting_for_each_other_and_the_last_step_sees_the_flow_completed(): void + { + if ($this->getConnectionFactory()->createContext()->getDbalConnection()->getDatabasePlatform() instanceof SQLitePlatform) { + self::markTestSkipped('SQLite serialises writers with a database lock, so the second consumer cannot run while the first one holds its transaction open. The race is left unproved on SQLite.'); + } + + $first = $this->bootstrap(new DbalConnectionFactory([ + 'dsn' => self::dsn(), + 'connection' => ['wrapperClass' => CommitHoldingConnection::class], + ]), setUpTables: true); + $tracingId = '0af7651916cd43dd8448eb211c80319c'; + self::$observationsFile = $this->signalFile(); + $first->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACING_ID => $tracingId]); + $ready = $this->signalFile(); + $outcome = $this->signalFile(); + $second = null; + + CommitHoldingConnection::$beforeCommit = function () use ($ready, $outcome, &$second): void { + CommitHoldingConnection::$beforeCommit = null; + $second = pcntl_fork(); + if ($second === 0) { + $this->consumeTheOtherDeliveryOnASecondConnection($ready, $outcome); + } + $this->waitUntilWritten($ready); + $this->waitUntilWritten($outcome, self::MILLISECONDS_THE_FIRST_CONSUMER_STAYS_UNCOMMITTED); + }; + $first->run('mail_channel', ExecutionPollingMetadata::createWithTestingSetup()); + pcntl_waitpid($second, $status); + + [$consumed, $tookMilliseconds] = explode('|', file_get_contents($outcome)); + $observations = file(self::$observationsFile, FILE_IGNORE_NEW_LINES); + unlink($ready); + unlink($outcome); + unlink(self::$observationsFile); + self::assertSame('consumed', $consumed); + self::assertLessThan(self::MILLISECONDS_THE_FIRST_CONSUMER_STAYS_UNCOMMITTED, (int) $tookMilliseconds); + self::assertSame(['user.mailer ran', 'user.texter ran'], array_values(array_filter($observations, fn (string $line): bool => str_ends_with($line, 'ran')))); + self::assertSame('flow completed', end($observations)); + self::assertTrue($first->getGateway(TracingGateway::class)->isCompletedFor($tracingId)); + } + + private function consumeTheOtherDeliveryOnASecondConnection(string $ready, string $outcome): never + { + $consumed = 'not consumed'; + $started = hrtime(true); + try { + $second = $this->bootstrap(new DbalConnectionFactory(self::dsn()), setUpTables: false); + file_put_contents($ready, 'ready'); + $started = hrtime(true); + $second->run('sms_channel', ExecutionPollingMetadata::createWithTestingSetup()); + $consumed = 'consumed'; + } catch (Throwable $failure) { + $consumed = $failure::class . ': ' . $failure->getMessage(); + } + file_put_contents($outcome, $consumed . '|' . (int) ((hrtime(true) - $started) / 1e6)); + + posix_kill(posix_getpid(), SIGKILL); + exit(1); + } + + private function bootstrap(DbalConnectionFactory $connectionFactory, bool $setUpTables): FlowTestSupport + { + $users = new class () { + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId, EventBus $eventBus): void + { + $eventBus->publishWithRouting('user.registered', $userId); + } + }; + $subscriber = new class () { + #[OnFlowStepCompleted] + public function whenStepCompleted(FlowStepCompleted $event, TracingGateway $tracingGateway): void + { + if ($tracingGateway->isCompletedFor($event->step->traceId)) { + file_put_contents(FlowTrackingRaceTest::$observationsFile, 'flow completed' . PHP_EOL, FILE_APPEND | LOCK_EX); + } + } + }; + $notifications = new class () { + #[Asynchronous('mail_channel')] + #[EventHandler('user.registered', endpointId: 'user.mailer')] + public function mail(string $userId): void + { + file_put_contents(FlowTrackingRaceTest::$observationsFile, 'user.mailer ran' . PHP_EOL, FILE_APPEND | LOCK_EX); + } + + #[Asynchronous('sms_channel')] + #[EventHandler('user.registered', endpointId: 'user.texter')] + public function text(string $userId): void + { + file_put_contents(FlowTrackingRaceTest::$observationsFile, 'user.texter ran' . PHP_EOL, FILE_APPEND | LOCK_EX); + } + }; + + return $this->bootstrapFlowTesting( + classesToResolve: [$users::class, $notifications::class, $subscriber::class], + containerOrAvailableServices: [$users, $notifications, $subscriber, DbalConnectionFactory::class => $connectionFactory], + extensionObjects: [ + FlowTrackingConfiguration::createWithDefaults(), + DbalBackedMessageChannelBuilder::create('mail_channel'), + DbalBackedMessageChannelBuilder::create('sms_channel'), + ], + enabledModules: [ModulePackageList::DBAL_PACKAGE], + testConfiguration: TestConfiguration::createWithDefaults()->withDatabaseTablesSetUpOnBootstrap($setUpTables), + licenceKey: LicenceTesting::VALID_LICENCE, + ); + } + + private static function dsn(): string + { + return getenv('DATABASE_DSN') ?: 'pgsql://ecotone:secret@localhost:5432/ecotone'; + } + + private function signalFile(): string + { + $file = tempnam(sys_get_temp_dir(), 'ecotone_flow_race_'); + unlink($file); + + return $file; + } + + private function waitUntilWritten(string $file, int $atMostMilliseconds = 30_000): void + { + $deadline = hrtime(true) + $atMostMilliseconds * 1e6; + while (! file_exists($file) && hrtime(true) < $deadline) { + usleep(10_000); + } + } +} + +final class CommitHoldingConnection extends Connection +{ + /** @var (Closure(): void)|null */ + public static ?Closure $beforeCommit = null; + + public function commit(): void + { + if (self::$beforeCommit !== null && $this->getTransactionNestingLevel() === 1) { + (self::$beforeCommit)(); + } + + parent::commit(); + } +} diff --git a/packages/Dbal/tests/Integration/Tracing/FlowTrackingTest.php b/packages/Dbal/tests/Integration/Tracing/FlowTrackingTest.php new file mode 100644 index 0000000000..03729fe7a0 --- /dev/null +++ b/packages/Dbal/tests/Integration/Tracing/FlowTrackingTest.php @@ -0,0 +1,1315 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + * + * @internal + */ +final class FlowTrackingTest extends TestCase +{ + public function test_a_command_handled_synchronously_is_completed_when_the_bus_returns(): void + { + $ecotone = $this->bootstrap([$this->users()]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.registration')); + } + + public function test_an_asynchronous_handler_keeps_the_trace_pending_until_its_consumer_ran(): void + { + $ecotone = $this->bootstrap([$this->users(), $this->asynchronousNotifier()], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.registration')); + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.notifier')); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.notifier')); + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_an_asynchronous_projection_keeps_the_trace_pending_until_it_projected(): void + { + $projection = new #[Projection('users_list'), FromStream('users_stream'), Asynchronous('async_channel')] class () { + #[EventHandler('user.registered')] + public function whenRegistered(array $event): void + { + } + }; + $ecotone = $this->bootstrap([$projection], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $ecotone->withEvents([Event::createWithType('user.registered', ['userId' => '1'])]); + $trace = SpanContext::newTrace(); + + $ecotone->publishEventWithRouting('user.registered', [], metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, 'users_list')); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'users_list')); + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_an_asynchronous_projection_is_awaited_by_its_class(): void + { + $projection = new #[Projection('users_list'), FromStream('users_stream'), Asynchronous('async_channel')] class () { + #[EventHandler('user.registered')] + public function whenRegistered(array $event): void + { + } + }; + $ecotone = $this->bootstrap([$projection], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $ecotone->withEvents([Event::createWithType('user.registered', ['userId' => '1'])]); + $trace = SpanContext::newTrace(); + $ecotone->publishEventWithRouting('user.registered', [], metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, $projection::class)); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, $projection::class)); + $this->assertTrue($this->completion($ecotone)->isCompletedWithin($trace->traceId, Duration::milliseconds(100), $projection::class)); + } + + public function test_a_handler_class_is_completed_once_every_one_of_its_handlers_the_flow_reached_ran(): void + { + $notifications = new class () { + #[Asynchronous('mail_channel')] + #[EventHandler('user.registered', endpointId: 'user.mailer')] + public function mail(string $userId): void + { + } + + #[Asynchronous('sms_channel')] + #[EventHandler('user.registered', endpointId: 'user.texter')] + public function text(string $userId): void + { + } + }; + $ecotone = $this->bootstrap([$this->users(), $notifications], [SimpleMessageChannelBuilder::createQueueChannel('mail_channel'), SimpleMessageChannelBuilder::createQueueChannel('sms_channel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $ecotone->run('mail_channel', ExecutionPollingMetadata::createWithTestingSetup()); + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, $notifications::class)); + + $ecotone->run('sms_channel', ExecutionPollingMetadata::createWithTestingSetup()); + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, $notifications::class)); + } + + public function test_a_synchronous_handler_without_an_endpoint_id_is_awaited_by_its_class(): void + { + $users = $this->users(); + $ecotone = $this->bootstrap([$users]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, $users::class)); + } + + public function test_awaiting_a_class_with_no_handler_or_projection_is_refused_naming_what_it_takes(): void + { + $ecotone = $this->bootstrap([$this->users()]); + + $this->expectException(ConfigurationException::class); + $this->expectExceptionMessage(sprintf('%s has no handler or projection Ecotone registered, so no flow can reach it. Pass a class with a #[CommandHandler], #[EventHandler], #[QueryHandler] or #[InternalHandler] method, a #[Projection] class, an endpointId or a projection name.', FlowTrackingTest::class)); + + $this->completion($ecotone)->isCompletedFor(SpanContext::newTrace()->traceId, FlowTrackingTest::class); + } + + public function test_awaiting_a_polling_projection_by_its_class_is_refused_because_it_runs_outside_the_trace(): void + { + $projection = new #[Projection('orders_list'), FromStream('orders_stream'), Polling('orders_list_polling')] class () { + #[EventHandler('order.placed')] + public function whenPlaced(array $event): void + { + } + }; + $ecotone = $this->bootstrap([$projection]); + + $this->expectExceptionMessage('Projection orders_list runs by #[Polling]'); + + $this->completion($ecotone)->isCompletedFor(SpanContext::newTrace()->traceId, $projection::class); + } + + public function test_waiting_for_a_flow_longer_than_fifteen_minutes_is_refused_proposing_notifications(): void + { + $ecotone = $this->bootstrap([$this->users()]); + + $this->expectException(InvalidArgumentException::class); + $this->expectExceptionMessage('TracingGateway::isCompletedWithin() was asked to wait 960000 milliseconds, but it blocks the calling process for as long as it waits, which past 15 minutes ties that process up for an extensive period. Wait at most 15 minutes, or react to the flow event-driven instead: a method with #[OnFlowStepCompleted] receives a FlowStepCompleted once each step of the flow committed, and can ask TracingGateway::isCompletedFor() whether the flow completed with it.'); + + $this->completion($ecotone)->isCompletedWithin(SpanContext::newTrace()->traceId, Duration::minutes(16)); + } + + public function test_waiting_for_a_destination_longer_than_fifteen_minutes_is_refused_proposing_notifications(): void + { + $ecotone = $this->bootstrap([$this->users()]); + + $this->expectException(InvalidArgumentException::class); + $this->expectExceptionMessage('TracingGateway::isCompletedWithin() was asked to wait 960000 milliseconds, but it blocks the calling process for as long as it waits, which past 15 minutes ties that process up for an extensive period. Wait at most 15 minutes, or react to the flow event-driven instead: a method with #[OnFlowStepCompleted] receives a FlowStepCompleted once each step of the flow committed, and can ask TracingGateway::isCompletedFor() whether the flow completed with it.'); + + $this->completion($ecotone)->isCompletedWithin(SpanContext::newTrace()->traceId, Duration::minutes(16), 'user.registration'); + } + + public function test_waiting_for_a_completed_flow_for_exactly_fifteen_minutes_is_allowed(): void + { + $ecotone = $this->bootstrap([$this->users()]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertTrue($this->completion($ecotone)->isCompletedWithin($trace->traceId, Duration::minutes(15))); + $this->assertTrue($this->completion($ecotone)->isCompletedWithin($trace->traceId, Duration::minutes(15), 'user.registration')); + } + + public function test_waiting_for_a_pending_flow_returns_false_and_the_flow_names_what_is_still_pending(): void + { + $ecotone = $this->bootstrap([$this->users(), $this->asynchronousNotifier()], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertFalse($this->completion($ecotone)->isCompletedWithin($trace->traceId, Duration::milliseconds(100))); + $this->assertSame(['user.notifier', 'async_channel'], $this->completion($ecotone)->fetchFlow($trace->traceId)->pending); + } + + public function test_waiting_for_a_completed_flow_returns_true_at_once(): void + { + $ecotone = $this->bootstrap([$this->users()]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertTrue($this->completion($ecotone)->isCompletedWithin($trace->traceId, Duration::milliseconds(100))); + $this->assertTrue($this->completion($ecotone)->isCompletedWithin($trace->traceId, Duration::milliseconds(100), 'user.registration')); + } + + public function test_a_fetched_flow_holds_its_steps_what_it_reached_and_what_is_still_pending(): void + { + $ecotone = $this->bootstrap([$this->users(), $this->asynchronousNotifier()], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $pending = $this->completion($ecotone)->fetchFlow($trace->traceId); + $this->assertSame($trace->traceId, $pending->tracingId); + $this->assertFalse($pending->isCompleted); + $this->assertSame(['user.notifier', 'async_channel'], $pending->pending); + $this->assertSame('send command', $pending->steps[0]->name); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $completed = $this->completion($ecotone)->fetchFlow($trace->traceId); + $this->assertTrue($completed->isCompleted); + $this->assertSame([], $completed->pending); + $this->assertEqualsCanonicalizing(['user.registration', 'user.notifier'], $completed->reached); + $this->assertContains('process class@anonymous::notify', array_map(fn (Span $step): string => preg_replace('/^process class@anonymous\S*::/', 'process class@anonymous::', $step->name), $completed->steps)); + } + + public function test_fetching_a_flow_nobody_recorded_returns_null(): void + { + $ecotone = $this->bootstrap([$this->users()]); + + $this->assertNull($this->completion($ecotone)->fetchFlow('0af7651916cd43dd8448eb211c80319c')); + } + + public function test_a_synchronous_handler_without_an_explicit_endpoint_id_still_completes_its_route(): void + { + $notifier = new class () { + #[EventHandler('user.registered')] + public function notify(string $userId): void + { + } + }; + $ecotone = $this->bootstrap([$this->users(), $notifier]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_a_handler_the_container_serves_as_a_subclass_proxy_still_completes_its_route(): void + { + $ecotone = EcotoneLite::bootstrapFlowTesting( + [$this->users()::class, RegistrationNotifier::class], + [$this->users(), RegistrationNotifier::class => new ProxiedRegistrationNotifier()], + [FlowTrackingConfiguration::createWithDefaults()], + licenceKey: LicenceTesting::VALID_LICENCE, + ); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_a_trace_nobody_recorded_is_not_completed(): void + { + $ecotone = $this->bootstrap([$this->users()]); + + $this->assertFalse($this->completion($ecotone)->isCompletedFor(SpanContext::newTrace()->traceId)); + } + + public function test_a_failing_asynchronous_handler_leaves_the_trace_pending(): void + { + $notifier = new class () { + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', endpointId: 'user.notifier')] + public function notify(string $userId): void + { + throw new RuntimeException('Notification failed'); + } + }; + $ecotone = $this->bootstrap([$this->users(), $notifier], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + try { + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + } catch (RuntimeException) { + } + + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.notifier')); + } + + public function test_a_flow_step_completed_subscriber_receives_every_stored_span_of_the_trace(): void + { + $subscriber = new class () { + /** @var string[] */ + public array $spanNames = []; + + #[OnFlowStepCompleted] + public function whenEnded(FlowStepCompleted $event): void + { + $this->spanNames[] = $event->step->name; + } + }; + $users = $this->users(); + $ecotone = $this->bootstrap([$users, $subscriber]); + + $ecotone->sendCommandWithRouting('user.register', '1'); + + $this->assertContains('send command', $subscriber->spanNames); + $this->assertContains('process class@anonymous::register', $subscriber->spanNames); + } + + public function test_an_asynchronous_command_keeps_the_trace_pending_until_its_consumer_ran(): void + { + $orders = new class () { + #[Asynchronous('async_channel')] + #[CommandHandler('order.place', endpointId: 'order.placing')] + public function place(string $orderId): void + { + } + }; + $ecotone = $this->bootstrap([$orders], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('order.place', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, 'order.placing')); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'order.placing')); + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_waiting_for_an_asynchronous_command_leaves_its_channel_pending(): void + { + $orders = new class () { + #[Asynchronous('async_channel')] + #[CommandHandler('order.place', endpointId: 'order.placing')] + public function place(string $orderId): void + { + } + }; + $ecotone = $this->bootstrap([$orders], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('order.place', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertFalse($this->completion($ecotone)->isCompletedWithin($trace->traceId, Duration::milliseconds(100))); + $this->assertSame(['order.placing', 'async_channel'], $this->completion($ecotone)->fetchFlow($trace->traceId)->pending); + } + + public function test_a_handler_that_succeeded_inside_a_failed_unit_of_work_is_not_completed(): void + { + $users = new class () { + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId, CommandBus $commandBus): void + { + $commandBus->sendWithRouting('audit.record', $userId); + + throw new RuntimeException('Registration failed after the audit was recorded'); + } + + #[CommandHandler('audit.record', endpointId: 'audit.recording')] + public function record(string $userId): void + { + } + }; + $ecotone = $this->bootstrap([$users]); + $trace = SpanContext::newTrace(); + + try { + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + $this->fail('Expected the registration failure to propagate'); + } catch (RuntimeException) { + } + + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, 'audit.recording')); + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.registration')); + } + + public function test_a_flow_step_completed_subscriber_sending_a_command_does_not_notify_itself_again(): void + { + $subscriber = new class () { + /** @var string[] */ + public array $notifiedTraces = []; + + #[OnFlowStepCompleted] + public function whenEnded(FlowStepCompleted $event, CommandBus $commandBus): void + { + $this->notifiedTraces[] = $event->step->traceId; + if (count($this->notifiedTraces) < 20) { + $commandBus->sendWithRouting('cache.refresh', $event->step->traceId); + } + } + + #[CommandHandler('cache.refresh', endpointId: 'cache.refreshing')] + public function refresh(string $traceId): void + { + } + }; + $ecotone = $this->bootstrap([$this->users(), $subscriber]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertSame([$trace->traceId], array_values(array_unique($subscriber->notifiedTraces))); + } + + public function test_a_failing_flow_step_completed_subscriber_is_logged_and_does_not_fail_the_command(): void + { + $subscriber = new class () { + #[OnFlowStepCompleted] + public function whenEnded(FlowStepCompleted $event): void + { + throw new RuntimeException('Cache refresh failed'); + } + }; + $logger = StubLogger::create(); + $ecotone = EcotoneLite::bootstrapFlowTesting( + [$this->users()::class, $subscriber::class], + [$this->users(), $subscriber, 'logger' => $logger], + [FlowTrackingConfiguration::createWithDefaults()], + licenceKey: LicenceTesting::VALID_LICENCE, + ); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertSame(['A subscriber of ecotone.tracing.flowStepCompleted failed and was skipped: Cache refresh failed'], array_values(array_unique($logger->getError()))); + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_a_flow_step_completed_subscriber_is_never_notified_about_the_synchronous_work_it_started(): void + { + $subscriber = new class () { + /** @var string[] */ + public array $stepNames = []; + + #[OnFlowStepCompleted] + public function whenEnded(FlowStepCompleted $event, CommandBus $commandBus): void + { + $this->stepNames[] = $event->step->name; + if (count($this->stepNames) < 20) { + $commandBus->sendWithRouting('cache.refresh', $event->step->traceId); + } + } + + #[CommandHandler('cache.refresh', endpointId: 'cache.refreshing')] + public function refresh(string $traceId): void + { + } + }; + $ecotone = $this->bootstrap([$this->users(), $subscriber]); + + $ecotone->sendCommandWithRouting('user.register', '1'); + + $this->assertNotContains('process class@anonymous::refresh', $subscriber->stepNames); + $this->assertLessThan(20, count($subscriber->stepNames)); + } + + public function test_a_flow_step_completed_subscriber_sending_an_asynchronous_command_is_not_notified_about_its_consumer(): void + { + $subscriber = new class () { + /** @var string[] */ + public array $stepNames = []; + + #[OnFlowStepCompleted] + public function whenEnded(FlowStepCompleted $event, CommandBus $commandBus): void + { + $this->stepNames[] = $event->step->name; + if (count($this->stepNames) < 20) { + $commandBus->sendWithRouting('cache.refresh', $event->step->traceId); + } + } + + #[Asynchronous('async_channel')] + #[CommandHandler('cache.refresh', endpointId: 'cache.refreshing')] + public function refresh(string $traceId): void + { + } + }; + $ecotone = $this->bootstrap([$this->users(), $subscriber], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $ecotone->sendCommandWithRouting('user.register', '1'); + $notifiedBeforeTheConsumerRan = $subscriber->stepNames; + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertSame($notifiedBeforeTheConsumerRan, $subscriber->stepNames); + } + + public function test_a_consumer_that_swallowed_its_handler_failure_leaves_the_trace_pending(): void + { + $orders = new class () { + #[Asynchronous('async_channel')] + #[CommandHandler('order.place', endpointId: 'order.placing')] + public function place(string $orderId): void + { + throw new RuntimeException('Placing failed'); + } + }; + $ecotone = $this->bootstrap([$orders], [SimpleMessageChannelBuilder::createQueueChannel('async_channel'), ServiceConfiguration::createWithDefaults()->withDefaultErrorChannel('errorChannel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('order.place', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup(stopOnError: false)); + + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, 'order.placing')); + } + + public function test_a_consumer_whose_handler_failed_once_and_succeeded_on_the_instant_retry_completes_the_trace(): void + { + $orders = new class () { + public int $attempts = 0; + + #[Asynchronous('async_channel')] + #[CommandHandler('order.place', endpointId: 'order.placing')] + public function place(string $orderId): void + { + if (++$this->attempts === 1) { + throw new RuntimeException('Placing failed once'); + } + } + }; + $ecotone = $this->bootstrap([$orders], [SimpleMessageChannelBuilder::createQueueChannel('async_channel'), ServiceConfiguration::createWithDefaults()->withDefaultErrorChannel('errorChannel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('order.place', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup(stopOnError: false)); + + $this->assertSame(2, $orders->attempts); + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_an_output_channel_after_an_asynchronous_handler_completes_neither_while_a_delivery_that_feeds_them_is_still_queued(): void + { + $notifier = new class () { + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', outputChannelName: 'audit.record', endpointId: 'user.notifier')] + public function notify(string $userId): string + { + return $userId; + } + + #[InternalHandler('audit.record', endpointId: 'audit.recording')] + public function record(string $userId): void + { + } + }; + $ecotone = $this->bootstrap([$notifier], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + $ecotone->publishEventWithRouting('user.registered', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + $ecotone->publishEventWithRouting('user.registered', '2', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup(handledMessageLimit: 1)); + + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, 'audit.recording')); + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.notifier')); + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup(handledMessageLimit: 1)); + + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'audit.recording')); + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'user.notifier')); + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_a_flow_started_through_a_message_gateway_is_pending_until_its_asynchronous_handler_ran(): void + { + $notifications = new class () { + #[InternalHandler('notification.send', outputChannelName: 'notification.deliver', endpointId: 'notification.sending')] + public function send(string $text): string + { + return $text; + } + + #[Asynchronous('async_channel')] + #[InternalHandler('notification.deliver', endpointId: 'notification.delivering')] + public function deliver(string $text): void + { + } + }; + $ecotone = EcotoneLite::bootstrapFlowTesting( + [$notifications::class, TracedNotificationGateway::class], + [$notifications], + [FlowTrackingConfiguration::createWithDefaults(), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + licenceKey: LicenceTesting::VALID_LICENCE, + ); + $trace = SpanContext::newTrace(); + + $ecotone->getGateway(TracedNotificationGateway::class)->send('hello', [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'notification.sending')); + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_the_tracing_gateway_without_flow_tracking_names_the_configuration_it_needs(): void + { + $ecotone = EcotoneLite::bootstrapFlowTesting([$this->users()::class], [$this->users()]); + + $this->expectException(ConfigurationException::class); + $this->expectExceptionMessage('TracingGateway needs flow tracking, which records every flow and decides when it is complete. Register FlowTrackingConfiguration::createWithDefaults() from a #[ServiceContext] method.'); + + $this->completion($ecotone)->isCompletedFor('0af7651916cd43dd8448eb211c80319c'); + } + + public function test_awaiting_a_trace_whose_handler_failed_names_the_channel_and_how_to_recover(): void + { + $notifier = new class () { + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', endpointId: 'user.notifier')] + public function notify(string $userId): void + { + throw new RuntimeException('Notification failed'); + } + }; + $ecotone = $this->bootstrap([$this->users(), $notifier], [SimpleMessageChannelBuilder::createQueueChannel('async_channel'), ServiceConfiguration::createWithDefaults()->withDefaultErrorChannel('errorChannel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup(stopOnError: false)); + + $this->assertFalse($this->completion($ecotone)->isCompletedWithin($trace->traceId, Duration::milliseconds(100))); + $this->assertSame(['user.notifier', 'async_channel'], $this->completion($ecotone)->fetchFlow($trace->traceId)->pending); + } + + public function test_waiting_for_a_flow_nobody_recorded_returns_false(): void + { + $ecotone = $this->bootstrap([$this->users()]); + + $this->assertFalse($this->completion($ecotone)->isCompletedWithin('0af7651916cd43dd8448eb211c80319c', Duration::milliseconds(100))); + } + + public function test_waiting_for_a_destination_the_flow_never_reached_returns_false_and_the_flow_names_what_it_reached(): void + { + $ecotone = $this->bootstrap([$this->users()]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertFalse($this->completion($ecotone)->isCompletedWithin($trace->traceId, Duration::milliseconds(100), 'user.notifierr')); + $this->assertSame(['user.registration'], $this->completion($ecotone)->fetchFlow($trace->traceId)->reached); + } + + public function test_two_asynchronous_commands_to_one_handler_complete_it_only_after_both_ran(): void + { + $orders = new class () { + #[Asynchronous('async_channel')] + #[CommandHandler('order.place', endpointId: 'order.placing')] + public function place(string $orderId): void + { + } + }; + $ecotone = $this->bootstrap([$orders], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('order.place', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + $ecotone->sendCommandWithRouting('order.place', '2', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup(handledMessageLimit: 1)); + + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId, 'order.placing')); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup(handledMessageLimit: 1)); + + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'order.placing')); + } + + public function test_awaiting_a_polling_projection_is_refused_because_it_runs_outside_the_trace(): void + { + $projection = new #[Projection('orders_list'), FromStream('orders_stream'), Polling('orders_list_polling')] class () { + #[EventHandler('order.placed')] + public function whenPlaced(array $event): void + { + } + }; + $ecotone = $this->bootstrap([$projection]); + + $this->expectException(ConfigurationException::class); + $this->expectExceptionMessage('Projection orders_list runs by #[Polling], so it is not part of the trace that appended its events and TracingGateway cannot wait for it. Await the handler that published the event, or make the projection event-driven by removing #[Polling].'); + + $this->completion($ecotone)->isCompletedWithin('0af7651916cd43dd8448eb211c80319c', Duration::milliseconds(10), 'orders_list'); + } + + public function test_a_flow_started_by_a_query_completes_once_the_asynchronous_command_it_sent_ran(): void + { + $orders = new class () { + #[QueryHandler('order.quote')] + public function quote(string $orderId, CommandBus $commandBus): string + { + $commandBus->sendWithRouting('order.reserve', $orderId); + + return 'quote-' . $orderId; + } + + #[Asynchronous('async_channel')] + #[CommandHandler('order.reserve', endpointId: 'order.reserving')] + public function reserve(string $orderId): void + { + } + }; + $ecotone = $this->bootstrap([$orders], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $tracingId = '0af7651916cd43dd8448eb211c80319c'; + + $ecotone->sendQueryWithRouting('order.quote', '1', metadata: [MessageHeaders::TRACING_ID => $tracingId]); + $this->assertFalse($this->completion($ecotone)->isCompletedFor($tracingId)); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + $this->assertTrue($this->completion($ecotone)->isCompletedFor($tracingId)); + } + + public function test_a_flow_a_consumer_starts_for_a_message_from_an_application_without_tracing_completes_once_that_consumer_ran(): void + { + $sharedChannel = SimpleMessageChannelBuilder::createQueueChannel('async_channel'); + $orders = new class () { + public ?string $tracingId = null; + + #[Asynchronous('async_channel')] + #[CommandHandler('order.place', endpointId: 'order.placing')] + public function place(string $orderId, #[Reference] CurrentTrace $currentTrace): void + { + $this->tracingId = $currentTrace->tracingId(); + } + }; + $producer = EcotoneLite::bootstrapFlowTesting([$orders::class], [$orders], [$sharedChannel]); + $consumer = $this->bootstrap([$orders], [$sharedChannel]); + + $producer->sendCommandWithRouting('order.place', '1'); + $consumer->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertNotNull($orders->tracingId); + $this->assertTrue($this->completion($consumer)->isCompletedFor($orders->tracingId)); + } + + public function test_a_handler_can_read_the_trace_id_it_runs_in(): void + { + $users = new class () { + public ?string $traceId = null; + + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId, #[Reference] CurrentTrace $currentTrace): void + { + $this->traceId = $currentTrace->tracingId(); + } + }; + $ecotone = $this->bootstrap([$users]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $this->assertSame($trace->traceId, $users->traceId); + $this->assertTrue($this->completion($ecotone)->isCompletedFor($users->traceId)); + } + + public function test_reading_the_current_trace_without_tracing_names_how_to_enable_it(): void + { + $users = new class () { + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId, #[Reference] CurrentTrace $currentTrace): void + { + $currentTrace->tracingId(); + } + }; + $ecotone = EcotoneLite::bootstrapFlowTesting([$users::class], [$users]); + + $this->expectException(ConfigurationException::class); + $this->expectExceptionMessage('No trace is recorded, so there is no current trace id. Enable tracing: register TracingConfiguration::createWithDefaults()->withSpanRecorder() or FlowTrackingConfiguration::createWithDefaults() from a #[ServiceContext] method, or install ecotone/open-telemetry with a TracerProviderInterface.'); + + $ecotone->sendCommandWithRouting('user.register', '1'); + } + + public function test_a_dynamic_channel_spreading_over_two_channels_completes_once_both_deliveries_were_consumed(): void + { + $orders = new class () { + #[Asynchronous('orders')] + #[CommandHandler('order.place', endpointId: 'order.placing')] + public function place(string $orderId): void + { + } + }; + $ecotone = $this->bootstrap([$orders], [ + DynamicMessageChannelBuilder::createRoundRobin('orders', ['orders_a', 'orders_b']), + SimpleMessageChannelBuilder::createQueueChannel('orders_a'), + SimpleMessageChannelBuilder::createQueueChannel('orders_b'), + ]); + $trace = SpanContext::newTrace(); + $ecotone->sendCommandWithRouting('order.place', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + $ecotone->sendCommandWithRouting('order.place', '2', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + + $ecotone->run('orders', ExecutionPollingMetadata::createWithTestingSetup(handledMessageLimit: 1)); + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + + $ecotone->run('orders', ExecutionPollingMetadata::createWithTestingSetup(handledMessageLimit: 1)); + + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId, 'order.placing')); + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_with_only_requested_flows_tracked_a_command_sent_without_one_stores_nothing(): void + { + $subscriber = $this->flowStepCompletedSubscriber(); + $ecotone = EcotoneLite::bootstrapFlowTesting( + [$this->users()::class, $this->asynchronousNotifier()::class, $subscriber::class], + [$this->users(), $this->asynchronousNotifier(), $subscriber], + [FlowTrackingConfiguration::createWithDefaults()->withOnlyRequestedFlows(true), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + licenceKey: LicenceTesting::VALID_LICENCE, + ); + + $ecotone->sendCommandWithRouting('user.register', '1'); + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertSame([], $subscriber->endedSpans); + } + + public function test_with_only_requested_flows_tracked_a_command_sent_with_one_is_stored_through_its_asynchronous_hop(): void + { + $subscriber = $this->flowStepCompletedSubscriber(); + $ecotone = EcotoneLite::bootstrapFlowTesting( + [$this->users()::class, $this->asynchronousNotifier()::class, $subscriber::class], + [$this->users(), $this->asynchronousNotifier(), $subscriber], + [FlowTrackingConfiguration::createWithDefaults()->withOnlyRequestedFlows(true), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + licenceKey: LicenceTesting::VALID_LICENCE, + ); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertArrayHasKey('receive async_channel', $subscriber->endedSpans); + $this->assertTrue($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_a_vendor_entry_in_tracestate_survives_the_asynchronous_hop(): void + { + $notifier = new class () { + public array $headers = []; + + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', endpointId: 'user.notifier')] + public function notify(string $userId, #[Headers] array $headers): void + { + $this->headers = $headers; + } + }; + $ecotone = $this->bootstrap([$this->users(), $notifier], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + $trace = SpanContext::newTrace(); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent(), MessageHeaders::TRACESTATE => 'congo=t61rcWkgMzE']); + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertStringContainsString('congo=t61rcWkgMzE', $notifier->headers[MessageHeaders::TRACESTATE]); + } + + public function test_a_command_sent_with_a_tracing_id_is_awaited_by_that_tracing_id(): void + { + $ecotone = $this->bootstrap([$this->users(), $this->asynchronousNotifier()], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACING_ID => '4bf92f3577b34da6a3ce929d0e0e4736']); + + $this->assertFalse($this->completion($ecotone)->isCompletedFor('4bf92f3577b34da6a3ce929d0e0e4736')); + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + $this->assertTrue($this->completion($ecotone)->isCompletedFor('4bf92f3577b34da6a3ce929d0e0e4736')); + } + + public function test_a_uuid_is_accepted_as_tracing_id_and_awaited_by_the_same_uuid(): void + { + $ecotone = $this->bootstrap([$this->users(), $this->asynchronousNotifier()], [SimpleMessageChannelBuilder::createQueueChannel('async_channel')]); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACING_ID => '0199f2f4-5e3b-7a42-9b1c-2f7d3e8a6c11']); + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertTrue($this->completion($ecotone)->isCompletedFor('0199f2f4-5e3b-7a42-9b1c-2f7d3e8a6c11')); + $this->assertTrue($this->completion($ecotone)->isCompletedFor('0199f2f4-5e3b-7a42-9b1c-2f7d3e8a6c11', 'user.notifier')); + } + + public function test_a_handler_reads_the_tracing_id_the_command_was_sent_with(): void + { + $users = new class () { + public ?string $tracingId = null; + + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId, #[Reference] CurrentTrace $currentTrace): void + { + $this->tracingId = $currentTrace->tracingId(); + } + }; + $ecotone = $this->bootstrap([$users]); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACING_ID => '0199f2f4-5e3b-7a42-9b1c-2f7d3e8a6c11']); + + $this->assertSame('0199f2f45e3b7a429b1c2f7d3e8a6c11', $users->tracingId); + } + + public function test_a_message_carrying_both_a_tracing_id_and_a_traceparent_is_refused(): void + { + $ecotone = $this->bootstrap([$this->users()]); + + $this->expectExceptionMessage('The message carries both tracingId and traceparent, so it is not clear which trace it belongs to. Pass tracingId to track the flow with TracingGateway, or traceparent when the trace comes from OpenTelemetry instrumentation, not both.'); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACING_ID => '4bf92f3577b34da6a3ce929d0e0e4736', MessageHeaders::TRACEPARENT => SpanContext::newTrace()->toTraceparent()]); + } + + public function test_a_tracing_id_on_a_command_sent_from_inside_a_handler_is_refused_naming_the_flow_it_belongs_to(): void + { + $users = new class () { + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId, CommandBus $commandBus): void + { + $commandBus->sendWithRouting('user.activate', $userId, metadata: [MessageHeaders::TRACING_ID => '0af7651916cd43dd8448eb211c80319c']); + } + + #[CommandHandler('user.activate', endpointId: 'user.activation')] + public function activate(string $userId): void + { + } + }; + $ecotone = $this->bootstrap([$users]); + + $this->expectExceptionMessage('tracingId names the flow a message starts, but this message is sent from inside flow 4bf92f3577b34da6a3ce929d0e0e4736, so it belongs to that flow already. Await 4bf92f3577b34da6a3ce929d0e0e4736 with TracingGateway, or send the message from outside a handler, where a flow starts.'); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACING_ID => '4bf92f3577b34da6a3ce929d0e0e4736']); + } + + public function test_a_tracing_id_on_an_event_published_from_inside_a_handler_is_refused_naming_the_flow_it_belongs_to(): void + { + $users = new class () { + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId, EventBus $eventBus): void + { + $eventBus->publishWithRouting('user.registered', $userId, metadata: [MessageHeaders::TRACING_ID => '0af7651916cd43dd8448eb211c80319c']); + } + }; + $ecotone = $this->bootstrap([$users]); + + $this->expectExceptionMessage('tracingId names the flow a message starts, but this message is sent from inside flow 4bf92f3577b34da6a3ce929d0e0e4736, so it belongs to that flow already. Await 4bf92f3577b34da6a3ce929d0e0e4736 with TracingGateway, or send the message from outside a handler, where a flow starts.'); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACING_ID => '4bf92f3577b34da6a3ce929d0e0e4736']); + } + + public function test_a_tracing_id_on_a_gateway_call_made_from_inside_a_handler_is_refused_naming_the_flow_it_belongs_to(): void + { + $users = new class () { + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId, #[Reference] TracedNotificationGateway $notifications): void + { + $notifications->send('welcome', [MessageHeaders::TRACING_ID => '0af7651916cd43dd8448eb211c80319c']); + } + + #[InternalHandler('notification.send', endpointId: 'notification.sending')] + public function sendNotification(string $text): void + { + } + }; + $ecotone = EcotoneLite::bootstrapFlowTesting( + [$users::class, TracedNotificationGateway::class], + [$users], + [FlowTrackingConfiguration::createWithDefaults()], + licenceKey: LicenceTesting::VALID_LICENCE, + ); + + $this->expectExceptionMessage('tracingId names the flow a message starts, but this message is sent from inside flow 4bf92f3577b34da6a3ce929d0e0e4736, so it belongs to that flow already. Await 4bf92f3577b34da6a3ce929d0e0e4736 with TracingGateway, or send the message from outside a handler, where a flow starts.'); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACING_ID => '4bf92f3577b34da6a3ce929d0e0e4736']); + } + + public function test_an_invalid_tracing_id_is_refused_naming_the_expected_format(): void + { + $ecotone = $this->bootstrap([$this->users()]); + + $this->expectExceptionMessage('tracingId `order-1` is neither 32 hexadecimal characters nor a UUID. Pass a UUID such as Uuid::v7()->toRfc4122(), or a W3C trace id.'); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACING_ID => 'order-1']); + } + + public function test_a_failed_unit_of_work_stores_nothing_by_default(): void + { + $subscriber = $this->flowStepCompletedSubscriber(); + $ecotone = $this->bootstrap([$this->failingUsers(), $subscriber]); + $trace = SpanContext::newTrace(); + + try { + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => $trace->toTraceparent()]); + $this->fail('Expected the registration failure to propagate'); + } catch (RuntimeException) { + } + + $this->assertSame([], $subscriber->endedSpans); + $this->assertFalse($this->completion($ecotone)->isCompletedFor($trace->traceId)); + } + + public function test_a_flow_step_completed_subscriber_whose_first_parameter_cannot_receive_the_notification_is_refused_at_bootstrap(): void + { + $subscriber = new class () { + #[OnFlowStepCompleted] + public function whenEnded(string $event): void + { + } + }; + + $this->expectException(ConfigurationException::class); + $this->expectExceptionMessage('#[OnFlowStepCompleted] on ' . $subscriber::class . '::whenEnded is passed a ' . FlowStepCompleted::class . ' as its first parameter, but declares it as string, so every notification would fail. Declare that parameter as ' . FlowStepCompleted::class . '.'); + + $this->bootstrap([$subscriber]); + } + + public function test_an_asynchronous_flow_step_completed_subscriber_is_refused_at_bootstrap_naming_how_to_run_its_work_in_the_background(): void + { + $subscriber = new class () { + #[Asynchronous('notifications')] + #[OnFlowStepCompleted] + public function whenEnded(FlowStepCompleted $event): void + { + } + }; + + $this->expectException(ConfigurationException::class); + $this->expectExceptionMessage('#[OnFlowStepCompleted] on ' . $subscriber::class . '::whenEnded cannot run #[Asynchronous]: a flow notification is delivered from memory once the work committed, so it is never queued. Remove #[Asynchronous], and send a command to an #[Asynchronous] handler from this method for the work that should run in the background.'); + + $this->bootstrap([$subscriber], [SimpleMessageChannelBuilder::createQueueChannel('notifications')]); + } + + public function test_a_flow_step_completed_subscriber_in_an_asynchronous_class_is_refused_at_bootstrap_naming_how_to_run_its_work_in_the_background(): void + { + $subscriber = new #[Asynchronous('notifications')] class () { + #[OnFlowStepCompleted] + public function whenEnded(FlowStepCompleted $event): void + { + } + }; + + $this->expectException(ConfigurationException::class); + $this->expectExceptionMessage('#[OnFlowStepCompleted] on ' . $subscriber::class . '::whenEnded cannot run #[Asynchronous]: a flow notification is delivered from memory once the work committed, so it is never queued. Remove #[Asynchronous], and send a command to an #[Asynchronous] handler from this method for the work that should run in the background.'); + + $this->bootstrap([$subscriber], [SimpleMessageChannelBuilder::createQueueChannel('notifications')]); + } + + public function test_flow_tracking_configured_twice_is_refused_at_bootstrap(): void + { + $this->expectException(ConfigurationException::class); + $this->expectExceptionMessage('FlowTrackingConfiguration was registered 2 times, so it is not clear which settings apply. Register it once, from one #[ServiceContext] method, chaining every setting on it: FlowTrackingConfiguration::createWithDefaults()->withRetention(...)->withNotifications(...).'); + + EcotoneLite::bootstrapFlowTesting( + [$this->users()::class], + [$this->users()], + [FlowTrackingConfiguration::createWithDefaults(), FlowTrackingConfiguration::createWithDefaults()->withNotifications(false)], + licenceKey: LicenceTesting::VALID_LICENCE, + ); + } + + public function test_with_notifications_turned_off_a_subscriber_is_refused_at_bootstrap(): void + { + $subscriber = new class () { + #[OnFlowStepCompleted] + public function whenEnded(FlowStepCompleted $event): void + { + } + }; + + $this->expectException(ConfigurationException::class); + $this->expectExceptionMessage('#[OnFlowStepCompleted] on ' . $subscriber::class . '::whenEnded would never be called, because FlowTrackingConfiguration::withNotifications(false) turned notifications off. Remove that call, or remove the attribute.'); + + EcotoneLite::bootstrapFlowTesting( + [$subscriber::class], + [$subscriber], + [FlowTrackingConfiguration::createWithDefaults()->withNotifications(false)], + licenceKey: LicenceTesting::VALID_LICENCE, + ); + } + + public function test_with_notifications_turned_off_a_flow_is_still_tracked_and_awaited(): void + { + $ecotone = EcotoneLite::bootstrapFlowTesting( + [$this->users()::class, $this->asynchronousNotifier()::class], + [$this->users(), $this->asynchronousNotifier()], + [FlowTrackingConfiguration::createWithDefaults()->withNotifications(false), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + licenceKey: LicenceTesting::VALID_LICENCE, + ); + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACING_ID => '4bf92f3577b34da6a3ce929d0e0e4736']); + + $this->assertFalse($this->completion($ecotone)->isCompletedFor('4bf92f3577b34da6a3ce929d0e0e4736')); + $ecotone->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + $this->assertTrue($this->completion($ecotone)->isCompletedFor('4bf92f3577b34da6a3ce929d0e0e4736')); + } + + public function test_with_the_tracing_gateway_turned_off_asking_it_names_the_setting_while_subscribers_are_still_notified(): void + { + $subscriber = new class () { + /** @var string[] */ + public array $completedFlows = []; + + #[OnFlowStepCompleted] + public function whenEnded(FlowStepCompleted $event): void + { + $this->completedFlows[] = $event->step->traceId; + } + }; + $ecotone = EcotoneLite::bootstrapFlowTesting( + [$this->users()::class, $subscriber::class], + [$this->users(), $subscriber], + [FlowTrackingConfiguration::createWithDefaults()->withTracingGateway(false)], + licenceKey: LicenceTesting::VALID_LICENCE, + ); + + $ecotone->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACING_ID => '4bf92f3577b34da6a3ce929d0e0e4736']); + $this->assertSame(['4bf92f3577b34da6a3ce929d0e0e4736'], array_values(array_unique($subscriber->completedFlows))); + + $this->expectException(ConfigurationException::class); + $this->expectExceptionMessage('TracingGateway is turned off by FlowTrackingConfiguration::withTracingGateway(false). Remove that call to ask whether a flow completed.'); + + $this->completion($ecotone)->isCompletedFor('4bf92f3577b34da6a3ce929d0e0e4736'); + } + + public function test_flow_tracking_requires_an_enterprise_licence(): void + { + $this->expectException(LicensingException::class); + $this->expectExceptionMessage('FlowTrackingConfiguration requires an Ecotone Enterprise licence. Remove it to run without one.'); + + EcotoneLite::bootstrapFlowTesting( + [], + [], + [FlowTrackingConfiguration::createWithDefaults()], + ); + } + + public function test_a_flow_step_completed_subscriber_requires_an_enterprise_licence(): void + { + $subscriber = new class () { + #[OnFlowStepCompleted] + public function whenEnded(FlowStepCompleted $event): void + { + } + }; + + $this->expectException(LicensingException::class); + $this->expectExceptionMessage('FlowTrackingConfiguration and #[OnFlowStepCompleted] on ' . $subscriber::class . '::whenEnded require an Ecotone Enterprise licence. Remove them to run without one.'); + + EcotoneLite::bootstrapFlowTesting( + [$subscriber::class], + [$subscriber], + [FlowTrackingConfiguration::createWithDefaults()], + ); + } + + public function test_a_flow_step_completed_subscriber_requires_an_enterprise_licence_even_without_flow_tracking(): void + { + $subscriber = new class () { + #[OnFlowStepCompleted] + public function whenEnded(FlowStepCompleted $event): void + { + } + }; + + $this->expectException(LicensingException::class); + $this->expectExceptionMessage('#[OnFlowStepCompleted] on ' . $subscriber::class . '::whenEnded requires an Ecotone Enterprise licence. Remove it to run without one.'); + + EcotoneLite::bootstrapFlowTesting( + [$subscriber::class], + [$subscriber], + ); + } + + public function test_a_flow_step_completed_subscriber_without_flow_tracking_is_refused_at_bootstrap(): void + { + $subscriber = new class () { + #[OnFlowStepCompleted] + public function whenEnded(FlowStepCompleted $event): void + { + } + }; + + $this->expectException(ConfigurationException::class); + $this->expectExceptionMessage('#[OnFlowStepCompleted] on ' . $subscriber::class . '::whenEnded needs flow tracking, which records the steps of every flow. Register FlowTrackingConfiguration::createWithDefaults() from a #[ServiceContext] method.'); + + EcotoneLite::bootstrapFlowTesting( + [$subscriber::class], + [$subscriber], + licenceKey: LicenceTesting::VALID_LICENCE, + ); + } + + /** + * @param object[] $services + * @param object[] $extensionObjects + */ + private function bootstrap(array $services, array $extensionObjects = []): FlowTestSupport + { + return EcotoneLite::bootstrapFlowTesting( + array_map(fn (object $service): string => $service::class, $services), + $services, + [FlowTrackingConfiguration::createWithDefaults(), ...$extensionObjects], + licenceKey: LicenceTesting::VALID_LICENCE, + ); + } + + private function completion(FlowTestSupport $ecotone): TracingGateway + { + return $ecotone->getGateway(TracingGateway::class); + } + + private function users(): object + { + return new class () { + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId, EventBus $eventBus): void + { + $eventBus->publishWithRouting('user.registered', $userId); + } + }; + } + + private function failingUsers(): object + { + return new class () { + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId): void + { + throw new RuntimeException('Registration failed'); + } + }; + } + + private function flowStepCompletedSubscriber(): object + { + return new class () { + /** @var array */ + public array $endedSpans = []; + + #[OnFlowStepCompleted] + public function whenEnded(FlowStepCompleted $event): void + { + $this->endedSpans[$event->step->name] = $event->step->status; + } + }; + } + + private function asynchronousNotifier(): object + { + return new class () { + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', endpointId: 'user.notifier')] + public function notify(string $userId): void + { + } + }; + } +} + +interface TracedNotificationGateway +{ + #[MessageGateway('notification.send')] + public function send(string $text, #[Headers] array $metadata): void; +} + +class RegistrationNotifier +{ + #[EventHandler('user.registered')] + public function notify(string $userId): void + { + } +} + +final class ProxiedRegistrationNotifier extends RegistrationNotifier +{ +} diff --git a/packages/Ecotone/Api/Interceptor/Precedence.php b/packages/Ecotone/Api/Interceptor/Precedence.php index ff95160848..45bdb89e73 100644 --- a/packages/Ecotone/Api/Interceptor/Precedence.php +++ b/packages/Ecotone/Api/Interceptor/Precedence.php @@ -12,6 +12,7 @@ interface Precedence { public const TRACING_PRECEDENCE = Precedence::ERROR_CHANNEL_PRECEDENCE - 101; + public const TRACING_HANDLER_PRECEDENCE = 1000000; public const ASYNCHRONOUS_CONSUMER_INTERCEPTOR_PRECEDENCE = Precedence::ERROR_CHANNEL_PRECEDENCE - 100; public const MESSAGE_ACKNOWLEDGE_PRECEDENCE = Precedence::ERROR_CHANNEL_PRECEDENCE - 99; public const EXCEPTION_LOGGING_PRECEDENCE = Precedence::ERROR_CHANNEL_PRECEDENCE - 1; @@ -22,6 +23,8 @@ interface Precedence public const ENDPOINT_HEADERS_PRECEDENCE = -3000; + public const TRACE_PUBLICATION_PRECEDENCE = self::CUSTOM_INSTANT_RETRY_PRECEDENCE - 1; + public const CUSTOM_INSTANT_RETRY_PRECEDENCE = -2003; public const GLOBAL_INSTANT_RETRY_PRECEDENCE = -2002; @@ -30,7 +33,9 @@ interface Precedence public const DATABASE_TRANSACTION_PRECEDENCE = -2000; - public const DELIVERY_CONFIRMATION_PRECEDENCE = self::DATABASE_TRANSACTION_PRECEDENCE + 1; + public const TRACE_RECORDING_PRECEDENCE = self::DATABASE_TRANSACTION_PRECEDENCE + 1; + + public const DELIVERY_CONFIRMATION_PRECEDENCE = self::TRACE_RECORDING_PRECEDENCE + 1; public const COLLECTOR_SENDER_PRECEDENCE = self::DELIVERY_CONFIRMATION_PRECEDENCE + 1; diff --git a/packages/Ecotone/Api/Messaging/MessageHeaders.php b/packages/Ecotone/Api/Messaging/MessageHeaders.php index 20bdd82fd0..db492571ca 100644 --- a/packages/Ecotone/Api/Messaging/MessageHeaders.php +++ b/packages/Ecotone/Api/Messaging/MessageHeaders.php @@ -68,6 +68,11 @@ final class MessageHeaders public const REPLY_CONTENT_TYPE = 'ecotone_messaging_reply_content_type'; public const REVISION = 'revision'; + public const TRACING_ID = 'tracingId'; + public const TRACEPARENT = 'traceparent'; + public const TRACESTATE = 'tracestate'; + public const TRACING_DELIVERY = 'ecotone_tracing_delivery'; + public const EVENT_AGGREGATE_TYPE = '_aggregate_type'; public const EVENT_AGGREGATE_ID = '_aggregate_id'; diff --git a/packages/Ecotone/Api/Tracing/CurrentTrace.php b/packages/Ecotone/Api/Tracing/CurrentTrace.php new file mode 100644 index 0000000000..b45bbeccca --- /dev/null +++ b/packages/Ecotone/Api/Tracing/CurrentTrace.php @@ -0,0 +1,13 @@ +spans[] = $span; + } + + /** + * @return Span[] + */ + public function recordedSpans(): array + { + return $this->spans; + } + + public function clear(): void + { + $this->spans = []; + } +} diff --git a/packages/Ecotone/Api/Tracing/Span.php b/packages/Ecotone/Api/Tracing/Span.php new file mode 100644 index 0000000000..3cece36d79 --- /dev/null +++ b/packages/Ecotone/Api/Tracing/Span.php @@ -0,0 +1,37 @@ + $attributes + * @param SpanEvent[] $events + */ + public function __construct( + public readonly string $traceId, + public readonly string $spanId, + public readonly ?string $parentSpanId, + public readonly string $name, + public readonly SpanKind $kind, + public readonly array $attributes, + public readonly SpanStatus $status, + public readonly ?string $statusDescription, + public readonly array $events, + public readonly DatePoint $startedAt, + public readonly DatePoint $endedAt, + ) { + } + + public function isRoot(): bool + { + return $this->parentSpanId === null; + } +} diff --git a/packages/Ecotone/Api/Tracing/SpanAttribute.php b/packages/Ecotone/Api/Tracing/SpanAttribute.php new file mode 100644 index 0000000000..d793e5d808 --- /dev/null +++ b/packages/Ecotone/Api/Tracing/SpanAttribute.php @@ -0,0 +1,27 @@ +traceId, bin2hex(random_bytes(8)), $this->sampled, $this->traceState); + } + + public function withTraceState(string $traceState): self + { + return new self($this->traceId, $this->spanId, $this->sampled, $traceState); + } + + public static function fromTraceparent(?string $traceparent, string $tracestate = ''): ?self + { + if ($traceparent === null || preg_match(self::TRACEPARENT_PATTERN, $traceparent, $matches) !== 1) { + return null; + } + if ($matches[1] === str_repeat('0', 32) || $matches[2] === str_repeat('0', 16)) { + return null; + } + + return new self($matches[1], $matches[2], (hexdec($matches[3]) & self::SAMPLED_FLAG) === self::SAMPLED_FLAG, $tracestate); + } + + public function toTraceparent(): string + { + return sprintf('00-%s-%s-%02x', $this->traceId, $this->spanId, $this->sampled ? self::SAMPLED_FLAG : 0); + } +} diff --git a/packages/Ecotone/Api/Tracing/SpanEvent.php b/packages/Ecotone/Api/Tracing/SpanEvent.php new file mode 100644 index 0000000000..92e551e117 --- /dev/null +++ b/packages/Ecotone/Api/Tracing/SpanEvent.php @@ -0,0 +1,23 @@ + $attributes + */ + public function __construct( + public readonly string $name, + public readonly array $attributes, + public readonly DatePoint $at, + ) { + } +} diff --git a/packages/Ecotone/Api/Tracing/SpanKind.php b/packages/Ecotone/Api/Tracing/SpanKind.php new file mode 100644 index 0000000000..e6feba4223 --- /dev/null +++ b/packages/Ecotone/Api/Tracing/SpanKind.php @@ -0,0 +1,17 @@ + [...$this->spanRecorderReferenceNames, $referenceName]]); + } + + public function withSpanAttributesFromHeaders(string ...$headerNames): self + { + return new self(...[...get_object_vars($this), 'spanAttributesFromAllHeaders' => false, 'spanAttributeHeaders' => array_values($headerNames)]); + } +} diff --git a/packages/Ecotone/phpstan-baseline.neon b/packages/Ecotone/phpstan-baseline.neon index 6bc0bb226c..5b5c855ef1 100644 --- a/packages/Ecotone/phpstan-baseline.neon +++ b/packages/Ecotone/phpstan-baseline.neon @@ -234,6 +234,12 @@ parameters: count: 1 path: src/Messaging/Config/ModuleClassList.php + - + message: '#^Class Ecotone\\Dbal\\Tracing\\DbalTracingModule not found\.$#' + identifier: class.notFound + count: 1 + path: src/Messaging/Config/ModuleClassList.php + - message: '#^Class Ecotone\\EventSourcing\\Config\\EventSourcingModule not found\.$#' identifier: class.notFound diff --git a/packages/Ecotone/src/Messaging/Attribute/IdentifiedAnnotation.php b/packages/Ecotone/src/Messaging/Attribute/IdentifiedAnnotation.php index d7f32790e7..27bf803ca7 100644 --- a/packages/Ecotone/src/Messaging/Attribute/IdentifiedAnnotation.php +++ b/packages/Ecotone/src/Messaging/Attribute/IdentifiedAnnotation.php @@ -32,6 +32,11 @@ public function isEndpointIdGenerated(): bool return $this->isGenerated; } + public function asynchronousExecutionRefusal(): ?string + { + return null; + } + public function getEndpointId(): string { return $this->endpointId; diff --git a/packages/Ecotone/src/Messaging/Config/Annotation/ModuleConfiguration/AsynchronousModule.php b/packages/Ecotone/src/Messaging/Config/Annotation/ModuleConfiguration/AsynchronousModule.php index 89865f0298..be1236abbe 100644 --- a/packages/Ecotone/src/Messaging/Config/Annotation/ModuleConfiguration/AsynchronousModule.php +++ b/packages/Ecotone/src/Messaging/Config/Annotation/ModuleConfiguration/AsynchronousModule.php @@ -4,6 +4,7 @@ namespace Ecotone\Messaging\Config\Annotation\ModuleConfiguration; +use Ecotone\AnnotationFinder\AnnotatedMethod; use Ecotone\AnnotationFinder\AnnotationFinder; use Ecotone\Api\Attribute\Asynchronous; use Ecotone\Api\Attribute\CommandHandler; @@ -18,6 +19,7 @@ use Ecotone\Api\ExtensionObject\ServiceConfiguration; use Ecotone\Api\ExtensionObject\SimpleMessageChannelBuilder; use Ecotone\Messaging\Attribute\EndpointAnnotation; +use Ecotone\Messaging\Attribute\IdentifiedAnnotation; use Ecotone\Messaging\Attribute\StreamBasedSource; use Ecotone\Messaging\Config\Annotation\AnnotatedDefinitionReference; use Ecotone\Messaging\Config\Annotation\AnnotationModule; @@ -71,6 +73,7 @@ public static function create(AnnotationFinder $annotationRegistrationService, I $streamSourcesAsyncEndpoints[$annotationForMethod->getEndpointId()] = $asyncClass->getChannelName(); } else { if ($annotationForMethod instanceof CommandHandler || $annotationForMethod instanceof EventHandler || $annotationForMethod instanceof InternalHandler) { + self::refuseEndpointThatCannotRunAsynchronously($endpoint, $annotationForMethod); if ($annotationForMethod->isEndpointIdGenerated()) { throw ConfigurationException::create("{$endpoint} should have endpointId defined for handling asynchronously"); } @@ -93,6 +96,7 @@ public static function create(AnnotationFinder $annotationRegistrationService, I continue; } if ($annotationForMethod instanceof CommandHandler || $annotationForMethod instanceof EventHandler || $annotationForMethod instanceof InternalHandler) { + self::refuseEndpointThatCannotRunAsynchronously($endpoint, $annotationForMethod); if ($annotationForMethod->isEndpointIdGenerated()) { throw ConfigurationException::create("{$endpoint} should have endpointId defined for handling asynchronously"); } @@ -115,9 +119,11 @@ public function getSynchronousChannelFor(string $handlerChannelName, string $end return $handlerChannelName; } + public const HANDLER_EXECUTION_CHANNEL_SUFFIX = '.execute'; + public static function getHandlerExecutionChannel(string $originalInputChannelName): string { - return $originalInputChannelName . '.execute'; + return $originalInputChannelName . self::HANDLER_EXECUTION_CHANNEL_SUFFIX; } /** @@ -260,4 +266,12 @@ public function registerDefaultPollingMetadata(BootstrapConfiguration $bootstrap } } } + + private static function refuseEndpointThatCannotRunAsynchronously(AnnotatedMethod $endpoint, IdentifiedAnnotation $annotation): void + { + $refusal = $annotation->asynchronousExecutionRefusal(); + if ($refusal !== null) { + throw ConfigurationException::create(sprintf('#[%s] on %s::%s %s', substr((string) strrchr('\\' . $annotation::class, '\\'), 1), $endpoint->getClassName(), $endpoint->getMethodName(), $refusal)); + } + } } diff --git a/packages/Ecotone/src/Messaging/Config/ModuleClassList.php b/packages/Ecotone/src/Messaging/Config/ModuleClassList.php index 0c9c5d8b0b..b2aecec9c0 100644 --- a/packages/Ecotone/src/Messaging/Config/ModuleClassList.php +++ b/packages/Ecotone/src/Messaging/Config/ModuleClassList.php @@ -24,6 +24,7 @@ use Ecotone\Dbal\MultiTenant\Module\MultiTenantConnectionFactoryModule; use Ecotone\Dbal\ObjectManager\ObjectManagerModule; use Ecotone\Dbal\Recoverability\DbalDeadLetterModule; +use Ecotone\Dbal\Tracing\DbalTracingModule; use Ecotone\EventSourcing\Config\EventSourcingModule; use Ecotone\EventSourcing\Config\EventSourcingProjectingModule; use Ecotone\EventSourcing\Tagging\Config\EventTaggingModule; @@ -87,6 +88,7 @@ use Ecotone\Sqs\Configuration\SqsModule; use Ecotone\SymfonyBundle\Config\SymfonyConnectionModule; use Ecotone\Tempest\Config\TempestModule; +use Ecotone\Tracing\Config\TracingModule; /** * licence Apache-2.0 @@ -116,6 +118,7 @@ class ModuleClassList RouterModule::class, ScheduledModule::class, CollectorModule::class, + TracingModule::class, DeliveryConfirmationModule::class, ChannelSetupModule::class, SerializerModule::class, @@ -170,6 +173,7 @@ class ModuleClassList DbalBusinessMethodModule::class, MultiTenantConnectionFactoryModule::class, DatabaseSetupModule::class, + DbalTracingModule::class, ]; public const REDIS_MODULES = [ diff --git a/packages/Ecotone/src/Messaging/Support/FrameworkHeaders.php b/packages/Ecotone/src/Messaging/Support/FrameworkHeaders.php index 4f8fcd9b8c..eb950a34d6 100644 --- a/packages/Ecotone/src/Messaging/Support/FrameworkHeaders.php +++ b/packages/Ecotone/src/Messaging/Support/FrameworkHeaders.php @@ -149,6 +149,10 @@ private static function getFrameworksHeaderNames(): array MessageHeaders::INBOUND_REQUEST_CHANNEL, MessageHeaders::REPLY_CONTENT_TYPE, MessagingEntrypointService::ENTRYPOINT, + MessageHeaders::TRACING_ID, + MessageHeaders::TRACEPARENT, + MessageHeaders::TRACESTATE, + MessageHeaders::TRACING_DELIVERY, ]; } } diff --git a/packages/Ecotone/src/Modelling/Config/MessageHandlerRoutingModule.php b/packages/Ecotone/src/Modelling/Config/MessageHandlerRoutingModule.php index 402a719611..0a0dc4f6ea 100644 --- a/packages/Ecotone/src/Modelling/Config/MessageHandlerRoutingModule.php +++ b/packages/Ecotone/src/Modelling/Config/MessageHandlerRoutingModule.php @@ -62,6 +62,8 @@ */ final class MessageHandlerRoutingModule implements AnnotationModule { + public const EVENT_BUS_ROUTE_RESOLVER = 'ecotone.eventBus.routeResolver'; + public function __construct( private InterfaceToCallRegistry $interfaceToCallRegistry, private AnnotationFinder $annotationFinder, @@ -171,12 +173,12 @@ public function prepare(Configuration $messagingConfiguration, array $extensionO ->registerMessageHandler( MessageProcessorActivatorBuilder::create() ->withInputChannelName(MessageBusChannel::COMMAND_CHANNEL_NAME_BY_NAME) - ->chain($this->buildRouterProcessor($commandBusRoutingConfig, MessageBusChannel::COMMAND_CHANNEL_NAME_BY_NAME, $isRunningForFlowTesting)) + ->chain($this->buildRouterProcessor($messagingConfiguration, $commandBusRoutingConfig, MessageBusChannel::COMMAND_CHANNEL_NAME_BY_NAME, $isRunningForFlowTesting)) ) ->registerMessageHandler( MessageProcessorActivatorBuilder::create() ->withInputChannelName(MessageBusChannel::QUERY_CHANNEL_NAME_BY_NAME) - ->chain($this->buildRouterProcessor($queryBusRouting, MessageBusChannel::QUERY_CHANNEL_NAME_BY_NAME, $isRunningForFlowTesting)) + ->chain($this->buildRouterProcessor($messagingConfiguration, $queryBusRouting, MessageBusChannel::QUERY_CHANNEL_NAME_BY_NAME, $isRunningForFlowTesting)) ) ->registerMessageHandler( MessageProcessorActivatorBuilder::create() @@ -186,7 +188,7 @@ public function prepare(Configuration $messagingConfiguration, array $extensionO MessageBusChannel::EVENT_CHANNEL_NAME_BY_NAME, Reference::to(MessageBuilder::class), ])) - ->chain($this->buildRouterProcessor($eventBusRouting, MessageBusChannel::EVENT_CHANNEL_NAME_BY_NAME, $isRunningForFlowTesting, false)) + ->chain($this->buildRouterProcessor($messagingConfiguration, $eventBusRouting, MessageBusChannel::EVENT_CHANNEL_NAME_BY_NAME, $isRunningForFlowTesting, false)) ); $messagingConfiguration->registerServiceDefinition( @@ -232,7 +234,7 @@ public function prepare(Configuration $messagingConfiguration, array $extensionO ); } - private function buildRouterProcessor(BusRoutingMapBuilder $busRoutingConfig, string $channel, bool $isRunningForFlowTesting, bool $isResolutionRequired = true): Definition + private function buildRouterProcessor(Configuration $messagingConfiguration, BusRoutingMapBuilder $busRoutingConfig, string $channel, bool $isRunningForFlowTesting, bool $isResolutionRequired = true): Definition { $busRouteSelectorClass = match ($channel) { MessageBusChannel::COMMAND_CHANNEL_NAME_BY_NAME => CommandBusRouteSelector::class, @@ -249,10 +251,16 @@ private function buildRouterProcessor(BusRoutingMapBuilder $busRoutingConfig, st $busRouteSelectorArguments[] = $isRunningForFlowTesting; } + $routeResolver = new Definition(RouteToChannelResolver::class, [new Reference(ChannelResolver::class)]); + if ($channel === MessageBusChannel::EVENT_CHANNEL_NAME_BY_NAME) { + $messagingConfiguration->registerServiceDefinition(self::EVENT_BUS_ROUTE_RESOLVER, $routeResolver); + $routeResolver = new Reference(self::EVENT_BUS_ROUTE_RESOLVER); + } + return new Definition(RouterProcessor::class, [ new Definition($busRouteSelectorClass, $busRouteSelectorArguments), - new Definition(RouteToChannelResolver::class, [new Reference(ChannelResolver::class)]), - $isResolutionRequired, // Single route if resolution is required + $routeResolver, + $isResolutionRequired, ]); } diff --git a/packages/Ecotone/src/Modelling/Config/Routing/BusRoutingMapBuilder.php b/packages/Ecotone/src/Modelling/Config/Routing/BusRoutingMapBuilder.php index bf3bfdcf0e..6238471a93 100644 --- a/packages/Ecotone/src/Modelling/Config/Routing/BusRoutingMapBuilder.php +++ b/packages/Ecotone/src/Modelling/Config/Routing/BusRoutingMapBuilder.php @@ -25,6 +25,8 @@ class BusRoutingMapBuilder extends BusRoutingMap { + private const DESTINATION_CHANNEL_SUFFIX = '.target'; + /** * @param array $routingEventHandlers */ @@ -63,11 +65,25 @@ public function getRoutesFromAnnotatedFinding(AnnotatedFinding $registration, In /** * @return string the destination channel name */ + public static function destinationChannelFor(string $endpointId): string + { + return $endpointId . self::DESTINATION_CHANNEL_SUFFIX; + } + + public static function endpointIdOfDestinationChannel(string $destinationChannel): ?string + { + if (! str_ends_with($destinationChannel, self::DESTINATION_CHANNEL_SUFFIX)) { + return null; + } + + return substr($destinationChannel, 0, -strlen(self::DESTINATION_CHANNEL_SUFFIX)); + } + public function addRoutesFromAnnotatedFinding(AnnotatedFinding $registration, InterfaceToCallRegistry $interfaceToCallRegistry): ?string { /** @var InputOutputEndpointAnnotation $annotationForMethod */ $annotationForMethod = $registration->getAnnotationForMethod(); - $destinationChannel = $annotationForMethod->getEndpointId() . '.target'; + $destinationChannel = self::destinationChannelFor($annotationForMethod->getEndpointId()); $routes = $this->getRoutesFromAnnotatedFinding($registration, $interfaceToCallRegistry); $priority = PriorityBasedOnType::fromAnnotatedFinding($registration); diff --git a/packages/Ecotone/src/Projecting/Config/ProjectingModule.php b/packages/Ecotone/src/Projecting/Config/ProjectingModule.php index a316df5e04..4d0a498ced 100644 --- a/packages/Ecotone/src/Projecting/Config/ProjectingModule.php +++ b/packages/Ecotone/src/Projecting/Config/ProjectingModule.php @@ -41,6 +41,7 @@ use Ecotone\Projecting\ProjectingManager; use Ecotone\Projecting\ProjectionRegistry; use Ecotone\Projecting\ProjectionStateStorageRegistry; +use Ecotone\Projecting\ProjectionTriggerEndpoint; use Ecotone\Projecting\SinglePartitionProvider; use Ecotone\Projecting\StreamFilterRegistry; use Ecotone\Projecting\StreamSourceRegistry; @@ -53,6 +54,8 @@ #[ModuleAnnotation] final class ProjectingModule implements AnnotationModule { + private const PROJECTING_MANAGER_CHANNEL_PREFIX = 'projecting_manager_handler:'; + public static function getProjectorExecutorReference(string $projectionName): string { return 'projection_executor:' . $projectionName; @@ -135,19 +138,19 @@ public function prepare(Configuration $messagingConfiguration, array $extensionO ->withEndpointId(self::endpointIdForProjection($projectionName)) ->withInputChannelName(self::inputChannelForProjectingManager($projectionName)); + $triggerEndpointAnnotations = [AttributeDefinition::fromObject(new ProjectionTriggerEndpoint($projectionName))]; $asyncAttribute = $projectionBuilder instanceof EcotoneProjectionExecutorBuilder ? $projectionBuilder->getAsyncAttribute() : null; if ($asyncAttribute !== null) { $endpointAnnotations = $asyncAttribute->asynchronousExecution; if ($messagingConfiguration->isRunningForEnterpriseLicence()) { $endpointAnnotations = array_merge($endpointAnnotations, [new WithoutDatabaseTransaction(), new WithoutMessageCollector()]); } - $handlerBuilder = $handlerBuilder->withEndpointAnnotations([ - AttributeDefinition::fromObject(new Asynchronous( - $asyncAttribute->getChannelName(), - $endpointAnnotations, - )), - ]); + $triggerEndpointAnnotations[] = AttributeDefinition::fromObject(new Asynchronous( + $asyncAttribute->getChannelName(), + $endpointAnnotations, + )); } + $handlerBuilder = $handlerBuilder->withEndpointAnnotations($triggerEndpointAnnotations); $messagingConfiguration->registerMessageHandler($handlerBuilder); @@ -233,6 +236,15 @@ private static function endpointIdForProjection(string $projectionName): string public static function inputChannelForProjectingManager(string $projectionName): string { - return 'projecting_manager_handler:' . $projectionName; + return self::PROJECTING_MANAGER_CHANNEL_PREFIX . $projectionName; + } + + public static function projectionNameOfTriggerChannel(string $channelName): ?string + { + if (! str_starts_with($channelName, self::PROJECTING_MANAGER_CHANNEL_PREFIX)) { + return null; + } + + return substr($channelName, strlen(self::PROJECTING_MANAGER_CHANNEL_PREFIX)); } } diff --git a/packages/Ecotone/src/Projecting/ProjectionTriggerEndpoint.php b/packages/Ecotone/src/Projecting/ProjectionTriggerEndpoint.php new file mode 100644 index 0000000000..b1ce1f9a9d --- /dev/null +++ b/packages/Ecotone/src/Projecting/ProjectionTriggerEndpoint.php @@ -0,0 +1,18 @@ + + */ + public function attributes(): array; + + /** + * @param array $attributes + */ + public function addEvent(string $name, array $attributes = []): void; + + public function end(): void; + + public function endedSoFar(): Span; + + public function markUnitOfWorkFailed(): void; + + public function markUnitOfWorkRecovered(): void; + + public function fail(Throwable $failure): void; +} diff --git a/packages/Ecotone/src/Tracing/CompositeSpanRecorder.php b/packages/Ecotone/src/Tracing/CompositeSpanRecorder.php new file mode 100644 index 0000000000..cded68f035 --- /dev/null +++ b/packages/Ecotone/src/Tracing/CompositeSpanRecorder.php @@ -0,0 +1,28 @@ +spanRecorders as $spanRecorder) { + $spanRecorder->record($span); + } + } +} diff --git a/packages/Ecotone/src/Tracing/Config/RegisterSpanEventLoggerCompilerPass.php b/packages/Ecotone/src/Tracing/Config/RegisterSpanEventLoggerCompilerPass.php new file mode 100644 index 0000000000..a202844cb0 --- /dev/null +++ b/packages/Ecotone/src/Tracing/Config/RegisterSpanEventLoggerCompilerPass.php @@ -0,0 +1,34 @@ +has(LoggingGateway::class)) { + return; + } + + $loggingGateway = $builder->getDefinition(LoggingGateway::class); + if (is_a($loggingGateway->getClassName(), SubscribableLoggingGateway::class, true)) { + $loggingGateway->addMethodCall('registerLogger', [new Definition(SpanEventLogger::class, [new Reference(Tracer::class)])]); + } + } +} diff --git a/packages/Ecotone/src/Tracing/Config/SpanRecorderReference.php b/packages/Ecotone/src/Tracing/Config/SpanRecorderReference.php new file mode 100644 index 0000000000..01864410a8 --- /dev/null +++ b/packages/Ecotone/src/Tracing/Config/SpanRecorderReference.php @@ -0,0 +1,15 @@ +has(MessageHandlerRoutingModule::EVENT_BUS_ROUTE_RESOLVER)) { + return; + } + + $builder->replace(MessageHandlerRoutingModule::EVENT_BUS_ROUTE_RESOLVER, new Definition(TracingRouteResolver::class, [ + $builder->getDefinition(MessageHandlerRoutingModule::EVENT_BUS_ROUTE_RESOLVER), + new Reference(Tracer::class), + new Reference(MessageSpanAttributes::class), + ])); + } +} diff --git a/packages/Ecotone/src/Tracing/Config/TracerReference.php b/packages/Ecotone/src/Tracing/Config/TracerReference.php new file mode 100644 index 0000000000..18505acc98 --- /dev/null +++ b/packages/Ecotone/src/Tracing/Config/TracerReference.php @@ -0,0 +1,15 @@ +channelName, - new Reference(TracerProviderInterface::class), + new Reference(Tracer::class), new Reference(MessageBuilder::class), + new Reference(MessageSpanAttributes::class), ]); } } diff --git a/packages/Ecotone/src/Tracing/Config/TracingModule.php b/packages/Ecotone/src/Tracing/Config/TracingModule.php new file mode 100644 index 0000000000..1984bc332c --- /dev/null +++ b/packages/Ecotone/src/Tracing/Config/TracingModule.php @@ -0,0 +1,171 @@ + $endpointIdsByHandler + */ + private function __construct( + private array $endpointIdsByHandler, + ) { + } + + public static function create(AnnotationFinder $annotationRegistrationService, InterfaceToCallRegistry $interfaceToCallRegistry): static + { + return new self(self::endpointIdsByHandler($annotationRegistrationService)); + } + + + + /** + * @return array + */ + private static function endpointIdsByHandler(AnnotationFinder $annotationFinder): array + { + $endpointIds = []; + foreach ([CommandHandler::class, QueryHandler::class, EventHandler::class, InternalHandler::class] as $attributeClassName) { + foreach ($annotationFinder->findAnnotatedMethods($attributeClassName) as $annotatedMethod) { + $endpointIds[$annotatedMethod->getClassName() . '::' . $annotatedMethod->getMethodName()] = $annotatedMethod->getAnnotationForMethod()->getEndpointId(); + } + } + + return $endpointIds; + } + + public function prepare(Configuration $messagingConfiguration, array $extensionObjects, ModuleReferenceSearchService $moduleReferenceSearchService, InterfaceToCallRegistry $interfaceToCallRegistry): void + { + $tracingConfiguration = ExtensionObjectResolver::resolveUnique(TracingConfiguration::class, $extensionObjects, TracingConfiguration::createWithDefaults()); + $tracerReferences = ExtensionObjectResolver::resolve(TracerReference::class, $extensionObjects); + + $spanRecorderReferences = ExtensionObjectResolver::resolve(SpanRecorderReference::class, $extensionObjects); + if ($tracingConfiguration->spanRecorderReferenceNames === [] && $tracerReferences === [] && $spanRecorderReferences === []) { + $messagingConfiguration->registerServiceDefinition(CurrentTrace::class, new Definition(TracingDisabledCurrentTrace::class)); + + return; + } + $messagingConfiguration->registerServiceDefinition(CurrentTrace::class, new Definition(TracerCurrentTrace::class, [new Reference(Tracer::class)])); + + $spanRecorders = [ + ...array_map(fn (string $referenceName): Reference => new Reference($referenceName), $tracingConfiguration->spanRecorderReferenceNames), + ...array_map(fn (SpanRecorderReference $reference): Reference => new Reference($reference->referenceName), $spanRecorderReferences), + ]; + + $messagingConfiguration->registerServiceDefinition(SpanRecorder::class, new Definition(CompositeSpanRecorder::class, [$spanRecorders])); + $messagingConfiguration->registerServiceDefinition( + Tracer::class, + $tracerReferences === [] + ? new Definition(EcotoneTracer::class, [new Reference(EcotoneClockInterface::class), new Reference(SpanRecorder::class)]) + : new Reference($tracerReferences[0]->referenceName) + ); + $messagingConfiguration->addCompilerPass(new RegisterSpanEventLoggerCompilerPass()); + $messagingConfiguration->addCompilerPass(new TraceEventRoutesCompilerPass()); + $messagingConfiguration->registerServiceDefinition( + TracingInterceptor::class, + new Definition(TracingInterceptor::class, [new Reference(Tracer::class), new Reference(MessageBuilder::class), new Reference(MessageSpanAttributes::class), $this->endpointIdsByHandler]) + ); + $messagingConfiguration->registerServiceDefinition( + MessageSpanAttributes::class, + new Definition(MessageSpanAttributes::class, [$tracingConfiguration->spanAttributesFromAllHeaders, $tracingConfiguration->spanAttributeHeaders]) + ); + + foreach ([ + 'traceCommandHandler' => [CommandHandler::class, Precedence::TRACING_HANDLER_PRECEDENCE], + 'traceQueryHandler' => [QueryHandler::class, Precedence::TRACING_HANDLER_PRECEDENCE], + 'traceEventHandler' => [EventHandler::class, Precedence::TRACING_HANDLER_PRECEDENCE], + 'traceMessageHandler' => [InternalHandler::class . '&¬(' . MessageHeadersPropagatorInterceptor::class . ')', Precedence::TRACING_HANDLER_PRECEDENCE], + 'traceProjection' => [ProjectionTriggerEndpoint::class, Precedence::TRACING_HANDLER_PRECEDENCE], + 'traceCommandBus' => [CommandBus::class, Precedence::TRACING_PRECEDENCE], + 'traceQueryBus' => [QueryBus::class, Precedence::TRACING_PRECEDENCE], + 'traceEventBus' => [EventBus::class, Precedence::TRACING_PRECEDENCE], + 'traceAsynchronousEndpoint' => [AsynchronousRunningEndpoint::class, Precedence::TRACING_PRECEDENCE], + 'traceDistributedBus' => [DistributedBus::class, Precedence::TRACING_PRECEDENCE], + 'traceMessageGateway' => [self::USER_MESSAGE_GATEWAYS_POINTCUT, Precedence::TRACING_PRECEDENCE], + 'traceOrchestratorGateway' => [OrchestratorGateway::class, Precedence::TRACING_PRECEDENCE], + ] as $method => [$pointcut, $precedence]) { + $messagingConfiguration->registerAroundMethodInterceptor( + AroundInterceptorBuilder::create( + TracingInterceptor::class, + $interfaceToCallRegistry->getFor(TracingInterceptor::class, $method), + $precedence, + $pointcut + ) + ); + } + + foreach (ExtensionObjectResolver::resolve(MessageChannelBuilder::class, $extensionObjects) as $messageChannelBuilder) { + if ($messageChannelBuilder->isPollable()) { + $messagingConfiguration->registerChannelInterceptor(new TracingChannelInterceptorBuilder($messageChannelBuilder->getMessageChannelName())); + } + } + + $messagingConfiguration->registerBeforeMethodInterceptor( + MethodInterceptorBuilder::create( + Reference::to(TracingInterceptor::class), + $interfaceToCallRegistry->getFor(TracingInterceptor::class, 'provideContextForDistributedBus'), + Precedence::DEFAULT_PRECEDENCE, + DistributedBus::class + ) + ); + } + + public function getModulePackageName(): string + { + return ModulePackageList::CORE_PACKAGE; + } + + + +} diff --git a/packages/Ecotone/src/Tracing/EcotoneActiveSpan.php b/packages/Ecotone/src/Tracing/EcotoneActiveSpan.php new file mode 100644 index 0000000000..bfabdc093f --- /dev/null +++ b/packages/Ecotone/src/Tracing/EcotoneActiveSpan.php @@ -0,0 +1,125 @@ + $attributes + */ + public function __construct( + private SpanContext $context, + private ?string $parentSpanId, + private string $name, + private SpanKind $kind, + private array $attributes, + private DatePoint $startedAt, + private EcotoneClockInterface $clock, + private EcotoneTracer $tracer, + ) { + } + + public function attributes(): array + { + return $this->attributes; + } + + public function context(): SpanContext + { + return $this->context; + } + + public function addEvent(string $name, array $attributes = []): void + { + $this->events[] = new SpanEvent($name, $attributes, $this->clock->now()); + } + + public function end(): void + { + if ($this->unitOfWorkFailed) { + $this->finish(SpanStatus::Error, self::UNIT_OF_WORK_FAILED); + + return; + } + $this->finish(SpanStatus::Ok, null); + } + + public function endedSoFar(): Span + { + return new Span( + $this->context->traceId, + $this->context->spanId, + $this->parentSpanId, + $this->name, + $this->kind, + $this->attributes, + SpanStatus::Ok, + null, + $this->events, + $this->startedAt, + $this->clock->now(), + ); + } + + public function markUnitOfWorkFailed(): void + { + $this->unitOfWorkFailed = true; + } + + public function markUnitOfWorkRecovered(): void + { + $this->unitOfWorkFailed = false; + } + + public function fail(Throwable $failure): void + { + $this->addEvent('exception', [ + 'exception.type' => $failure::class, + 'exception.message' => $failure->getMessage(), + 'exception.stacktrace' => $failure->getTraceAsString(), + ]); + $this->finish(SpanStatus::Error, $failure->getMessage()); + } + + private function finish(SpanStatus $status, ?string $statusDescription): void + { + if ($this->ended) { + return; + } + $this->ended = true; + + $this->tracer->endSpan($this, new Span( + $this->context->traceId, + $this->context->spanId, + $this->parentSpanId, + $this->name, + $this->kind, + $this->attributes, + $status, + $statusDescription, + $this->events, + $this->startedAt, + $this->clock->now(), + )); + } +} diff --git a/packages/Ecotone/src/Tracing/EcotoneTracer.php b/packages/Ecotone/src/Tracing/EcotoneTracer.php new file mode 100644 index 0000000000..3580d2e71e --- /dev/null +++ b/packages/Ecotone/src/Tracing/EcotoneTracer.php @@ -0,0 +1,75 @@ +start($name, $kind, $attributes, $remoteParent ?? $this->currentSpan()?->context()); + } + + public function startRootSpan(string $name, SpanKind $kind, array $attributes = []): ActiveSpan + { + return $this->start($name, $kind, $attributes, null); + } + + /** + * @param array $attributes + */ + private function start(string $name, SpanKind $kind, array $attributes, ?SpanContext $parent): ActiveSpan + { + $span = new EcotoneActiveSpan( + $parent?->newChild() ?? SpanContext::newTrace(), + $parent?->spanId, + $name, + $kind, + $attributes, + $this->clock->now(), + $this->clock, + $this, + ); + $this->activeSpans[] = $span; + + return $span; + } + + public function currentSpan(): ?ActiveSpan + { + return $this->activeSpans[count($this->activeSpans) - 1] ?? null; + } + + public function endSpan(EcotoneActiveSpan $activeSpan, Span $span): void + { + $position = array_search($activeSpan, $this->activeSpans, true); + if ($position !== false) { + array_splice($this->activeSpans, $position, 1); + } + + $this->spanRecorder->record($span); + } +} diff --git a/packages/Ecotone/src/Tracing/MessageSpanAttributes.php b/packages/Ecotone/src/Tracing/MessageSpanAttributes.php new file mode 100644 index 0000000000..4eb28a9f72 --- /dev/null +++ b/packages/Ecotone/src/Tracing/MessageSpanAttributes.php @@ -0,0 +1,64 @@ + + */ + public function from(Message $message): array + { + $attributes = []; + foreach (FrameworkHeaders::unsetAllFrameworkHeaders($message->getHeaders()->headers()) as $key => $value) { + if (is_scalar($value) && $this->isReported((string) $key)) { + $attributes[$key] = $value; + } + } + + $headers = $message->getHeaders(); + $attributes[MessageHeaders::MESSAGE_ID] = $headers->getMessageId(); + $attributes[MessageHeaders::MESSAGE_CORRELATION_ID] = $headers->getCorrelationId(); + if ($headers->getParentId() !== null) { + $attributes[MessageHeaders::PARENT_MESSAGE_ID] = $headers->getParentId(); + } + $attributes[MessageHeaders::TIMESTAMP] = $headers->getTimestamp(); + $attributes[SpanAttribute::MESSAGING_SYSTEM] = SpanAttribute::ECOTONE; + + return $attributes; + } + + private function isReported(string $headerName): bool + { + foreach (self::RESERVED_PREFIXES as $prefix) { + if (str_starts_with($headerName, $prefix)) { + return false; + } + } + + return $this->fromAllHeaders || in_array($headerName, $this->headerNames, true); + } +} diff --git a/packages/Ecotone/src/Tracing/SpanEventLogger.php b/packages/Ecotone/src/Tracing/SpanEventLogger.php new file mode 100644 index 0000000000..c923d29341 --- /dev/null +++ b/packages/Ecotone/src/Tracing/SpanEventLogger.php @@ -0,0 +1,51 @@ +tracer->currentSpan(); + if ($currentSpan === null) { + return; + } + + $attributes = array_filter( + $context, + fn (string|int $key): bool => in_array($key, [MessageHeaders::MESSAGE_ID, MessageHeaders::MESSAGE_CORRELATION_ID, MessageHeaders::PARENT_MESSAGE_ID], true), + ARRAY_FILTER_USE_KEY + ); + if (isset($context['exception']) && $context['exception'] instanceof Throwable) { + $attributes['exception.type'] = $context['exception']::class; + $attributes['exception.message'] = $context['exception']->getMessage(); + $attributes['exception.stacktrace'] = $context['exception']->getTraceAsString(); + } + + $currentSpan->addEvent((string) $message, $attributes); + } +} diff --git a/packages/Ecotone/src/Tracing/TraceState.php b/packages/Ecotone/src/Tracing/TraceState.php new file mode 100644 index 0000000000..41d5f032fe --- /dev/null +++ b/packages/Ecotone/src/Tracing/TraceState.php @@ -0,0 +1,64 @@ + $entry !== '' && ! str_starts_with($entry, self::VENDOR . '='))); + + return implode(',', [self::VENDOR . '=' . $value, ...$entries]); + } + + private static function entryOfMessage(Message $message): ?string + { + $headers = $message->getHeaders(); + + return $headers->containsKey(MessageHeaders::TRACESTATE) ? self::entryOf((string) $headers->get(MessageHeaders::TRACESTATE)) : null; + } + + private static function entryOf(string $tracestate): ?string + { + foreach (explode(',', $tracestate) as $entry) { + $entry = trim($entry); + if (str_starts_with($entry, self::VENDOR . '=')) { + return substr($entry, strlen(self::VENDOR) + 1); + } + } + + return null; + } +} diff --git a/packages/Ecotone/src/Tracing/TracedRouteProcessor.php b/packages/Ecotone/src/Tracing/TracedRouteProcessor.php new file mode 100644 index 0000000000..2df01f053c --- /dev/null +++ b/packages/Ecotone/src/Tracing/TracedRouteProcessor.php @@ -0,0 +1,74 @@ +tracer->startSpan($this->spanName(), SpanKind::Producer, [ + ...$this->messageSpanAttributes->from($message), + SpanAttribute::MESSAGING_OPERATION_TYPE => SpanAttribute::OPERATION_SEND, + SpanAttribute::MESSAGING_DESTINATION_NAME => $this->routeName, + ...$this->destinationAttributes(), + ...TracingInterceptor::trackingAttributesOf($this->tracer, $message, false), + ]); + + try { + $result = $this->routeProcessor->process($message); + } catch (Throwable $failure) { + $span->fail($failure); + + throw $failure; + } + + $span->end(); + + return $result; + } + + private function spanName(): string + { + $projectionName = ProjectingModule::projectionNameOfTriggerChannel($this->routeName); + if ($projectionName !== null) { + return 'route ' . $projectionName; + } + + return 'route ' . (BusRoutingMapBuilder::endpointIdOfDestinationChannel($this->routeName) ?? $this->routeName); + } + + /** + * @return array + */ + private function destinationAttributes(): array + { + $projectionName = ProjectingModule::projectionNameOfTriggerChannel($this->routeName); + if ($projectionName !== null) { + return [SpanAttribute::PROJECTION_NAME => $projectionName]; + } + $endpointId = BusRoutingMapBuilder::endpointIdOfDestinationChannel($this->routeName); + + return $endpointId === null ? [] : [SpanAttribute::ENDPOINT_ID => $endpointId]; + } +} diff --git a/packages/Ecotone/src/Tracing/Tracer.php b/packages/Ecotone/src/Tracing/Tracer.php new file mode 100644 index 0000000000..a99748a8b6 --- /dev/null +++ b/packages/Ecotone/src/Tracing/Tracer.php @@ -0,0 +1,26 @@ + $attributes + */ + public function startSpan(string $name, SpanKind $kind, array $attributes = [], ?SpanContext $remoteParent = null): ActiveSpan; + + /** + * @param array $attributes + */ + public function startRootSpan(string $name, SpanKind $kind, array $attributes = []): ActiveSpan; + + public function currentSpan(): ?ActiveSpan; +} diff --git a/packages/Ecotone/src/Tracing/TracerCurrentTrace.php b/packages/Ecotone/src/Tracing/TracerCurrentTrace.php new file mode 100644 index 0000000000..bfd1466d8d --- /dev/null +++ b/packages/Ecotone/src/Tracing/TracerCurrentTrace.php @@ -0,0 +1,22 @@ +tracer->currentSpan()?->context()->traceId; + } +} diff --git a/packages/Ecotone/src/Tracing/TracingChannelInterceptor.php b/packages/Ecotone/src/Tracing/TracingChannelInterceptor.php new file mode 100644 index 0000000000..c041ea7751 --- /dev/null +++ b/packages/Ecotone/src/Tracing/TracingChannelInterceptor.php @@ -0,0 +1,172 @@ +messageSpanAttributes->from($message), + SpanAttribute::MESSAGING_OPERATION_TYPE => SpanAttribute::OPERATION_SEND, + SpanAttribute::MESSAGING_DESTINATION_NAME => $this->channelName, + ]; + $isFirstChannel = self::deliveryChannelOf($message) === null; + if ($isFirstChannel) { + $attributes = [...$attributes, ...$this->deliveryAttributes($message)]; + } + $attributes = [...$attributes, ...TracingInterceptor::trackingAttributesOf($this->tracer, $message, false)]; + $span = $this->tracer->startSpan( + 'send ' . $this->channelName, + SpanKind::Producer, + $attributes, + $this->tracer->currentSpan() === null ? TracingInterceptor::remoteParentOf($message) : null, + ); + $this->openSendSpans[] = $span; + + $carrier = TracingInterceptor::carrierOf($span); + $sent = $this->messageBuilder->fromMessage($message)->setMultipleHeaders($carrier); + if ($isFirstChannel) { + $sent = $sent->setHeader(MessageHeaders::TRACING_DELIVERY, $this->channelName . ':' . $message->getHeaders()->getMessageId()); + } + $payload = $message->getPayload(); + if ($payload instanceof BatchMessage) { + $sent = $sent->setPayload(BatchMessage::fromEntries(array_map( + fn (array $entry): array => [...$entry, 'headers' => [...($entry['headers'] ?? []), ...$carrier]], + $payload->entries, + ))); + } + + return $sent->build(); + } + + public static function deliveryChannelOf(Message $message): ?string + { + $headers = $message->getHeaders(); + if (! $headers->containsKey(MessageHeaders::TRACING_DELIVERY)) { + return null; + } + [$channel, $messageId] = explode(':', (string) $headers->get(MessageHeaders::TRACING_DELIVERY), 2); + + return $messageId === $headers->getMessageId() ? $channel : null; + } + + /** + * @return array + */ + private function deliveryAttributes(Message $message): array + { + $attributes = [SpanAttribute::CHANNEL_NAME => $this->channelName]; + $payload = $message->getPayload(); + if ($payload instanceof BatchMessage) { + $deliveries = []; + foreach ($payload->entries as $entry) { + $headers = $entry['headers'] ?? []; + $deliveries[(string) ($headers[MessageHeaders::MESSAGE_ID] ?? '')] = self::targetEndpointIdOfRoutingSlip((string) ($headers[MessageHeaders::ROUTING_SLIP] ?? '')); + } + $attributes[SpanAttribute::BATCH_DELIVERIES] = json_encode($deliveries, JSON_THROW_ON_ERROR | JSON_FORCE_OBJECT); + + return $attributes; + } + $targetEndpointId = self::targetEndpointIdOf($message); + if ($targetEndpointId !== null) { + $attributes[SpanAttribute::ENDPOINT_ID] = $targetEndpointId; + } + + return $attributes; + } + + private static function targetEndpointIdOf(Message $message): ?string + { + $headers = $message->getHeaders(); + + return $headers->containsKey(MessageHeaders::ROUTING_SLIP) ? self::targetEndpointIdOfRoutingSlip((string) $headers->get(MessageHeaders::ROUTING_SLIP)) : null; + } + + private static function targetEndpointIdOfRoutingSlip(string $routingSlipHeader): ?string + { + if ($routingSlipHeader === '') { + return null; + } + $routingSlip = explode(',', $routingSlipHeader); + $executionChannel = $routingSlip[count($routingSlip) - 1]; + if (! str_ends_with($executionChannel, AsynchronousModule::HANDLER_EXECUTION_CHANNEL_SUFFIX)) { + return null; + } + + return BusRoutingMapBuilder::endpointIdOfDestinationChannel(substr($executionChannel, 0, -strlen(AsynchronousModule::HANDLER_EXECUTION_CHANNEL_SUFFIX))); + } + + public function postSend(Message $message, MessageChannel $messageChannel): void + { + } + + public function afterSendCompletion(Message $message, MessageChannel $messageChannel, ?Throwable $exception): bool + { + $span = array_pop($this->openSendSpans); + if ($exception !== null) { + $span?->fail($exception); + } else { + $span?->end(); + } + + return false; + } + + public function preReceive(MessageChannel $messageChannel): bool + { + return true; + } + + public function afterReceiveCompletion(?Message $message, MessageChannel $messageChannel, ?Throwable $exception): void + { + if ($exception === null || $message === null) { + return; + } + + $this->tracer + ->startSpan('receive ' . $this->channelName, SpanKind::Consumer, [ + ...$this->messageSpanAttributes->from($message), + SpanAttribute::MESSAGING_OPERATION_TYPE => SpanAttribute::OPERATION_PROCESS, + SpanAttribute::MESSAGING_DESTINATION_NAME => $this->channelName, + SpanAttribute::CHANNEL_NAME => $this->channelName, + ...TracingInterceptor::trackingAttributesCarriedBy($message, false), + ], TracingInterceptor::remoteParentOf($message)) + ->fail($exception); + } + + public function postReceive(Message $message, MessageChannel $messageChannel): ?Message + { + return $message; + } +} diff --git a/packages/Ecotone/src/Tracing/TracingDisabledCurrentTrace.php b/packages/Ecotone/src/Tracing/TracingDisabledCurrentTrace.php new file mode 100644 index 0000000000..c44d20ef20 --- /dev/null +++ b/packages/Ecotone/src/Tracing/TracingDisabledCurrentTrace.php @@ -0,0 +1,19 @@ +withSpanRecorder() or FlowTrackingConfiguration::createWithDefaults() from a #[ServiceContext] method, or install ecotone/open-telemetry with a TracerProviderInterface.'); + } +} diff --git a/packages/Ecotone/src/Tracing/TracingId.php b/packages/Ecotone/src/Tracing/TracingId.php new file mode 100644 index 0000000000..6c0172df66 --- /dev/null +++ b/packages/Ecotone/src/Tracing/TracingId.php @@ -0,0 +1,31 @@ +toRfc4122(), or a W3C trace id.', $tracingId)); + } + + public static function normalizedOrNull(string $tracingId): ?string + { + $candidate = strtolower(str_replace('-', '', $tracingId)); + if (preg_match(self::TRACE_ID_PATTERN, $candidate) !== 1 || $candidate === str_repeat('0', 32)) { + return null; + } + + return $candidate; + } +} diff --git a/packages/Ecotone/src/Tracing/TracingInterceptor.php b/packages/Ecotone/src/Tracing/TracingInterceptor.php new file mode 100644 index 0000000000..b25f8292b7 --- /dev/null +++ b/packages/Ecotone/src/Tracing/TracingInterceptor.php @@ -0,0 +1,299 @@ + $endpointIdsByHandler + */ + public function __construct( + private Tracer $tracer, + private MessageBuilder $messageBuilder, + private MessageSpanAttributes $messageSpanAttributes, + private array $endpointIdsByHandler, + ) { + } + + public function traceCommandBus(MethodInvocation $methodInvocation, Message $message): mixed + { + return $this->traceEntry('send command', $methodInvocation, $message, SpanKind::Server, [SpanAttribute::MESSAGING_OPERATION_TYPE => SpanAttribute::OPERATION_SEND]); + } + + public function traceEventBus(MethodInvocation $methodInvocation, Message $message): mixed + { + return $this->traceEntry('send event', $methodInvocation, $message, SpanKind::Server, [SpanAttribute::MESSAGING_OPERATION_TYPE => SpanAttribute::OPERATION_SEND]); + } + + public function traceQueryBus(MethodInvocation $methodInvocation, Message $message): mixed + { + return $this->traceEntry('send query', $methodInvocation, $message, SpanKind::Server, [SpanAttribute::MESSAGING_OPERATION_TYPE => SpanAttribute::OPERATION_SEND]); + } + + public function traceCommandHandler(MethodInvocation $methodInvocation, Message $message, CommandHandler $endpoint): mixed + { + return $this->traceHandler($methodInvocation, $message, $this->endpointIdOf($methodInvocation, $endpoint->getEndpointId())); + } + + public function traceQueryHandler(MethodInvocation $methodInvocation, Message $message, QueryHandler $endpoint): mixed + { + return $this->traceHandler($methodInvocation, $message, $this->endpointIdOf($methodInvocation, $endpoint->getEndpointId())); + } + + public function traceEventHandler(MethodInvocation $methodInvocation, Message $message, EventHandler $endpoint): mixed + { + return $this->traceHandler($methodInvocation, $message, $this->endpointIdOf($methodInvocation, $endpoint->getEndpointId())); + } + + public function traceMessageHandler(MethodInvocation $methodInvocation, Message $message, InternalHandler $endpoint): mixed + { + return $this->traceHandler($methodInvocation, $message, $this->endpointIdOf($methodInvocation, $endpoint->getEndpointId())); + } + + public function traceAsynchronousEndpoint(MethodInvocation $methodInvocation, Message $message): mixed + { + $headers = $message->getHeaders(); + $name = match (true) { + $headers->containsKey(MessageHeaders::POLLED_CHANNEL_NAME) => 'receive ' . $headers->get(MessageHeaders::POLLED_CHANNEL_NAME), + $headers->containsKey(MessageHeaders::INBOUND_REQUEST_CHANNEL) => 'receive ' . $headers->get(MessageHeaders::INBOUND_REQUEST_CHANNEL), + default => 'poll ' . $headers->get(MessageHeaders::CONSUMER_POLLING_METADATA)->getEndpointId(), + }; + + $attributes = [SpanAttribute::MESSAGING_OPERATION_TYPE => SpanAttribute::OPERATION_PROCESS]; + if ($headers->containsKey(MessageHeaders::POLLED_CHANNEL_NAME)) { + $attributes[SpanAttribute::MESSAGING_DESTINATION_NAME] = $headers->get(MessageHeaders::POLLED_CHANNEL_NAME); + $attributes[SpanAttribute::CHANNEL_NAME] = TracingChannelInterceptor::deliveryChannelOf($message) ?? $headers->get(MessageHeaders::POLLED_CHANNEL_NAME); + } + + return $this->trace($name, $methodInvocation, $message, SpanKind::Consumer, $attributes); + } + + public function traceProjection(MethodInvocation $methodInvocation, Message $message, ProjectionTriggerEndpoint $trigger): mixed + { + return $this->trace('process ' . $trigger->projectionName, $methodInvocation, $message, SpanKind::Consumer, [ + SpanAttribute::MESSAGING_OPERATION_TYPE => SpanAttribute::OPERATION_PROCESS, + SpanAttribute::PROJECTION_NAME => $trigger->projectionName, + ]); + } + + public function traceMessageGateway(MethodInvocation $methodInvocation, Message $message, MessageGateway $gateway): mixed + { + return $this->traceEntry('send ' . $gateway->requestChannel, $methodInvocation, $message, SpanKind::Server, [ + SpanAttribute::MESSAGING_OPERATION_TYPE => SpanAttribute::OPERATION_SEND, + SpanAttribute::MESSAGING_DESTINATION_NAME => $gateway->requestChannel, + ]); + } + + public function traceOrchestratorGateway(MethodInvocation $methodInvocation, Message $message): mixed + { + return $this->traceEntry('send orchestrator', $methodInvocation, $message, SpanKind::Server, [SpanAttribute::MESSAGING_OPERATION_TYPE => SpanAttribute::OPERATION_SEND]); + } + + public function traceDistributedBus(MethodInvocation $methodInvocation, Message $message): mixed + { + return $this->traceEntry('send distributed', $methodInvocation, $message, SpanKind::Producer, [SpanAttribute::MESSAGING_OPERATION_TYPE => SpanAttribute::OPERATION_SEND]); + } + + public function provideContextForDistributedBus(Message $message): Message + { + $currentSpan = $this->tracer->currentSpan(); + if ($currentSpan === null) { + return $message; + } + + return $this->messageBuilder->fromMessage($message) + ->setMultipleHeaders(self::carrierOf($currentSpan)) + ->build(); + } + + /** + * @return array + */ + public static function carrierOf(ActiveSpan $span): array + { + $carrier = [MessageHeaders::TRACEPARENT => $span->context()->toTraceparent()]; + $traceState = match (true) { + (bool) ($span->attributes()[SpanAttribute::TRACE_UNTRACKED] ?? false) => TraceState::withUntracked($span->context()->traceState), + (bool) ($span->attributes()[SpanAttribute::TRACE_CARRIED] ?? false) => TraceState::withCarried($span->context()->traceState), + default => $span->context()->traceState, + }; + if ($traceState !== '') { + $carrier[MessageHeaders::TRACESTATE] = $traceState; + } + + return $carrier; + } + + /** + * @return array + */ + public static function trackingAttributesOf(Tracer $tracer, Message $message, bool $startsTheTraceHere): array + { + $current = $tracer->currentSpan(); + if ($current === null) { + return self::trackingAttributesCarriedBy($message, $startsTheTraceHere); + } + + return array_filter([ + SpanAttribute::TRACE_CARRIED => (bool) ($current->attributes()[SpanAttribute::TRACE_CARRIED] ?? false), + SpanAttribute::TRACE_UNTRACKED => (bool) ($current->attributes()[SpanAttribute::TRACE_UNTRACKED] ?? false), + ]); + } + + /** + * @return array + */ + public static function trackingAttributesCarriedBy(Message $message, bool $startsTheTraceHere): array + { + if (TraceState::isUntracked($message)) { + return [SpanAttribute::TRACE_UNTRACKED => true]; + } + $carried = $startsTheTraceHere ? self::remoteParentOf($message) !== null : TraceState::isCarried($message); + + return $carried ? [SpanAttribute::TRACE_CARRIED => true] : []; + } + + public static function remoteParentOf(Message $message): ?SpanContext + { + $headers = $message->getHeaders(); + $traceparent = SpanContext::fromTraceparent( + $headers->containsKey(MessageHeaders::TRACEPARENT) ? $headers->get(MessageHeaders::TRACEPARENT) : null, + $headers->containsKey(MessageHeaders::TRACESTATE) ? (string) $headers->get(MessageHeaders::TRACESTATE) : '', + ); + if ($traceparent !== null || ! $headers->containsKey(MessageHeaders::TRACING_ID)) { + return $traceparent; + } + $tracingId = TracingId::normalizedOrNull((string) $headers->get(MessageHeaders::TRACING_ID)); + + return $tracingId === null ? null : new SpanContext($tracingId, bin2hex(random_bytes(8))); + } + + private static function entryParentOf(Message $message): ?SpanContext + { + $headers = $message->getHeaders(); + if (! $headers->containsKey(MessageHeaders::TRACING_ID)) { + return self::remoteParentOf($message); + } + if ($headers->containsKey(MessageHeaders::TRACEPARENT)) { + throw InvalidArgumentException::create('The message carries both tracingId and traceparent, so it is not clear which trace it belongs to. Pass tracingId to track the flow with TracingGateway, or traceparent when the trace comes from OpenTelemetry instrumentation, not both.'); + } + + return new SpanContext(TracingId::normalize((string) $headers->get(MessageHeaders::TRACING_ID)), bin2hex(random_bytes(8))); + } + + private function endpointIdOf(MethodInvocation $methodInvocation, string $declaredEndpointId): string + { + $object = $methodInvocation->getObjectToInvokeOn(); + $class = is_string($object) ? $object : $object::class; + while ($class !== false) { + $endpointId = $this->endpointIdsByHandler[$class . '::' . $methodInvocation->getMethodName()] ?? null; + if ($endpointId !== null) { + return $endpointId; + } + $class = get_parent_class($class); + } + + return $declaredEndpointId; + } + + private function traceHandler(MethodInvocation $methodInvocation, Message $message, string $endpointId): mixed + { + $attributes = [ + SpanAttribute::MESSAGING_OPERATION_TYPE => SpanAttribute::OPERATION_PROCESS, + SpanAttribute::CODE_FUNCTION_NAME => $methodInvocation->getName(), + ]; + if ($endpointId !== '') { + $attributes[SpanAttribute::ENDPOINT_ID] = $endpointId; + } + + return $this->traceUnitOfWorkStep('process ' . self::shortNameOf($methodInvocation->getName()), $methodInvocation, $message, $attributes); + } + + /** + * @param array $attributes + */ + private function traceUnitOfWorkStep(string $name, MethodInvocation $methodInvocation, Message $message, array $attributes): mixed + { + try { + $result = $this->trace($name, $methodInvocation, $message, SpanKind::Server, $attributes); + } catch (Throwable $failure) { + $this->tracer->currentSpan()?->markUnitOfWorkFailed(); + + throw $failure; + } + $this->tracer->currentSpan()?->markUnitOfWorkRecovered(); + + return $result; + } + + private static function shortNameOf(string $classAndMethod): string + { + [$class, $method] = explode('::', $classAndMethod, 2); + $class = explode("\0", $class, 2)[0]; + + return substr($class, (int) strrpos('\\' . $class, '\\')) . '::' . $method; + } + + /** + * @param array $attributes + */ + private function traceEntry(string $name, MethodInvocation $methodInvocation, Message $message, SpanKind $kind, array $attributes): mixed + { + $currentSpan = $this->tracer->currentSpan(); + $headers = $message->getHeaders(); + if ($currentSpan !== null && $headers->containsKey(MessageHeaders::TRACING_ID) && TracingId::normalizedOrNull((string) $headers->get(MessageHeaders::TRACING_ID)) !== $currentSpan->context()->traceId) { + $flow = $currentSpan->context()->traceId; + + throw InvalidArgumentException::create("tracingId names the flow a message starts, but this message is sent from inside flow {$flow}, so it belongs to that flow already. Await {$flow} with TracingGateway, or send the message from outside a handler, where a flow starts."); + } + + return $this->trace($name, $methodInvocation, $message, $kind, $attributes); + } + + /** + * @param array $attributes + */ + private function trace(string $name, MethodInvocation $methodInvocation, Message $message, SpanKind $kind, array $attributes): mixed + { + $isLocalRoot = $this->tracer->currentSpan() === null; + $remoteParent = ! $isLocalRoot ? null : ($kind === SpanKind::Consumer ? self::remoteParentOf($message) : self::entryParentOf($message)); + if ($isLocalRoot && ($kind !== SpanKind::Consumer || $remoteParent === null)) { + $attributes[SpanAttribute::TRACE_ROOT] = true; + } + $attributes = [...$attributes, ...self::trackingAttributesOf($this->tracer, $message, $isLocalRoot && $kind !== SpanKind::Consumer)]; + $span = $this->tracer->startSpan($name, $kind, [...$this->messageSpanAttributes->from($message), ...$attributes], $remoteParent); + + try { + $result = $methodInvocation->proceed(); + } catch (Throwable $failure) { + $span->fail($failure); + + throw $failure; + } + + $span->end(); + + return $result; + } +} diff --git a/packages/Ecotone/src/Tracing/TracingRouteResolver.php b/packages/Ecotone/src/Tracing/TracingRouteResolver.php new file mode 100644 index 0000000000..82560a66fc --- /dev/null +++ b/packages/Ecotone/src/Tracing/TracingRouteResolver.php @@ -0,0 +1,26 @@ +routeResolver->resolve($routeName), $this->tracer, $this->messageSpanAttributes); + } +} diff --git a/packages/Ecotone/tests/Tracing/AsynchronousTracingTest.php b/packages/Ecotone/tests/Tracing/AsynchronousTracingTest.php new file mode 100644 index 0000000000..b48481abe4 --- /dev/null +++ b/packages/Ecotone/tests/Tracing/AsynchronousTracingTest.php @@ -0,0 +1,413 @@ +asynchronousUsers(); + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration(), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + ) + ->sendCommandWithRouting('user.register', '1') + ->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::compareTreesByDetails( + [ + [ + 'details' => ['name' => 'send command'], + 'children' => [ + [ + 'details' => ['name' => 'send async_channel', 'kind' => SpanKind::Producer], + 'children' => [ + [ + 'details' => ['name' => 'receive async_channel', 'kind' => SpanKind::Consumer], + 'children' => [ + ['details' => ['name' => 'process class@anonymous::register'], 'children' => []], + ], + ], + ], + ], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + } + + public function test_the_consumer_continues_the_trace_of_the_sender_across_the_queue(): void + { + $recorder = new InMemorySpanRecorder(); + $users = $this->asynchronousUsers(); + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration(), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + ) + ->sendCommandWithRouting('user.register', '1') + ->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $traceIds = array_unique(array_map(fn (Span $span): string => $span->traceId, $recorder->recordedSpans())); + $this->assertCount(1, $traceIds); + } + + public function test_tracing_same_message_for_asynchronous_scenario(): void + { + $recorder = new InMemorySpanRecorder(); + $users = $this->asynchronousUsers(); + $messageId = Uuid::v7()->toRfc4122(); + $correlationId = Uuid::v7()->toRfc4122(); + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration(), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + ) + ->sendCommandWithRouting('user.register', '1', metadata: [ + MessageHeaders::MESSAGE_ID => $messageId, + MessageHeaders::MESSAGE_CORRELATION_ID => $correlationId, + ]) + ->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $node = self::getNodeAtTargetedSpan( + [ + 'details' => ['name' => 'send command'], + 'child' => [ + 'details' => ['name' => 'send async_channel'], + 'child' => ['details' => ['name' => 'receive async_channel']], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + + $this->assertSame($correlationId, $node['details']['attributes'][MessageHeaders::MESSAGE_CORRELATION_ID]); + $this->assertSame($messageId, $node['details']['attributes'][MessageHeaders::MESSAGE_ID]); + $child = $node['children'][array_key_first($node['children'])]; + $this->assertSame($correlationId, $child['details']['attributes'][MessageHeaders::MESSAGE_CORRELATION_ID]); + $this->assertSame($messageId, $child['details']['attributes'][MessageHeaders::MESSAGE_ID]); + } + + public function test_two_traces_with_asynchronous_handlers(): void + { + $recorder = new InMemorySpanRecorder(); + $users = $this->asynchronousUsers(); + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration(), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + ) + ->sendCommandWithRouting('user.register', '1', metadata: ['flowId' => '1']) + ->sendCommandWithRouting('user.register', '2', metadata: ['flowId' => '2']) + ->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup(handledMessageLimit: 2)); + + $expectedFlow = fn (string $flowId): array => [ + 'details' => ['name' => 'send command', 'attributes' => ['flowId' => $flowId]], + 'children' => [ + [ + 'details' => ['name' => 'send async_channel', 'attributes' => ['flowId' => $flowId]], + 'children' => [ + [ + 'details' => ['name' => 'receive async_channel', 'attributes' => ['flowId' => $flowId]], + 'children' => [ + ['details' => ['name' => 'process class@anonymous::register', 'attributes' => ['flowId' => $flowId]], 'children' => []], + ], + ], + ], + ], + ], + ]; + self::compareTreesByDetails([$expectedFlow('1'), $expectedFlow('2')], self::buildTree($recorder->recordedSpans())); + $this->assertCount(2, array_unique(array_map(fn (Span $span): string => $span->traceId, $recorder->recordedSpans()))); + } + + public function test_user_headers_travel_with_the_message_into_every_span(): void + { + $recorder = new InMemorySpanRecorder(); + $users = $this->asynchronousUsers(); + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration(), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + ) + ->sendCommandWithRouting('user.register', '1', metadata: ['user_id' => '123']) + ->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + foreach (['send command', 'send async_channel', 'receive async_channel', 'process class@anonymous::register'] as $spanName) { + $spans = array_values(array_filter($recorder->recordedSpans(), fn (Span $span): bool => $span->name === $spanName)); + $this->assertCount(1, $spans, $spanName); + $this->assertSame('123', $spans[0]->attributes['user_id'], $spanName); + } + } + + public function test_the_trace_carrier_header_is_not_reported_as_a_user_attribute(): void + { + $recorder = new InMemorySpanRecorder(); + $users = $this->asynchronousUsers(); + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration(), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + ) + ->sendCommandWithRouting('user.register', '1') + ->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + foreach ($recorder->recordedSpans() as $span) { + $this->assertArrayNotHasKey(MessageHeaders::TRACEPARENT, $span->attributes, $span->name); + } + } + + public function test_an_event_collected_during_the_asynchronous_handler_is_sent_under_the_consumer_span_in_the_same_trace(): void + { + $recorder = new InMemorySpanRecorder(); + $users = new class () { + #[Asynchronous('async_channel')] + #[CommandHandler('user.register', endpointId: 'user.register.endpoint')] + public function register(string $userId, EventBus $eventBus): void + { + $eventBus->publishWithRouting('user.registered', $userId); + } + }; + $notifier = new class () { + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', endpointId: 'user.registered.endpoint')] + public function notify(string $userId): void + { + } + }; + + EcotoneLite::bootstrapFlowTesting( + [$users::class, $notifier::class], + [$users, $notifier, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration(), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + ) + ->sendCommandWithRouting('user.register', '1') + ->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup(handledMessageLimit: 2)); + + self::getNodeAtTargetedSpan( + [ + 'details' => ['name' => 'send command'], + 'child' => [ + 'details' => ['name' => 'send async_channel'], + 'child' => [ + 'details' => ['name' => 'receive async_channel'], + 'child' => [ + 'details' => ['name' => 'send async_channel'], + 'child' => [ + 'details' => ['name' => 'receive async_channel'], + 'child' => ['details' => ['name' => 'process class@anonymous::notify']], + ], + ], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + self::getNodeAtTargetedSpan( + [ + 'details' => ['name' => 'send command'], + 'child' => [ + 'details' => ['name' => 'send async_channel'], + 'child' => [ + 'details' => ['name' => 'receive async_channel'], + 'child' => [ + 'details' => ['name' => 'process class@anonymous::register'], + 'child' => ['details' => ['name' => 'send event']], + ], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + $this->assertCount(1, array_unique(array_map(fn (Span $span): string => $span->traceId, $recorder->recordedSpans()))); + } + + public function test_adding_logs_as_events_in_span(): void + { + $recorder = new InMemorySpanRecorder(); + $users = new class () { + #[Asynchronous('async_channel')] + #[CommandHandler('user.register', endpointId: 'user.register.endpoint')] + public function register(string $userId, EventBus $eventBus): void + { + $eventBus->publishWithRouting('user.registered', $userId); + } + }; + $notifier = new class () { + #[Asynchronous('async_channel')] + #[EventHandler('user.registered', endpointId: 'user.registered.endpoint')] + public function notify(string $userId): void + { + } + }; + + EcotoneLite::bootstrapFlowTesting( + [$users::class, $notifier::class], + [$users, $notifier, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration(), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + ) + ->sendCommandWithRouting('user.register', '1') + ->run('async_channel'); + + $eventBus = self::getNodeAtTargetedSpan( + [ + 'details' => ['name' => 'send command'], + 'child' => [ + 'details' => ['name' => 'send async_channel'], + 'child' => [ + 'details' => ['name' => 'receive async_channel'], + 'child' => [ + 'details' => ['name' => 'process class@anonymous::register'], + 'child' => ['details' => ['name' => 'send event']], + ], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + + $eventBusEvents = array_map(fn ($event): string => $event->name, $eventBus['details']['events']); + $this->assertNotEmpty(array_filter($eventBusEvents, fn (string $name): bool => str_starts_with($name, 'Publishing Event Message'))); + + $route = $eventBus['children'][array_key_first($eventBus['children'])]; + $this->assertSame('route user.registered.endpoint', $route['details']['name']); + $routeEvents = array_map(fn ($event): string => $event->name, $route['details']['events']); + $this->assertNotEmpty(array_filter($routeEvents, fn (string $name): bool => str_starts_with($name, 'Collecting message with id'))); + } + + public function test_tracing_scheduled_handler(): void + { + $recorder = new InMemorySpanRecorder(); + $scheduled = new class () { + #[Scheduled(endpointId: 'scheduled_handler')] + #[Poller(fixedRateInMilliseconds: 1)] + public function handle(): void + { + } + }; + + $ecotone = EcotoneLite::bootstrapFlowTesting( + [$scheduled::class], + [$scheduled, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ); + + $ecotone->run('scheduled_handler', ExecutionPollingMetadata::createWithDefaults()->withExecutionAmountLimit(1)); + $ecotone->run('scheduled_handler', ExecutionPollingMetadata::createWithDefaults()->withExecutionAmountLimit(1)); + + $polls = array_values(array_filter($recorder->recordedSpans(), fn (Span $span): bool => $span->name === 'receive nullChannel')); + $this->assertCount(2, $polls); + foreach ($polls as $poll) { + $this->assertSame(SpanKind::Consumer, $poll->kind); + $this->assertTrue($poll->isRoot()); + } + $this->assertNotSame($polls[0]->traceId, $polls[1]->traceId); + } + + public function test_tracing_workflow_scheduled_handler(): void + { + $recorder = new InMemorySpanRecorder(); + $workflow = $this->workflowScheduledHandler(); + + $ecotone = EcotoneLite::bootstrapFlowTesting( + [$workflow::class], + [$workflow, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ); + + $ecotone->run('scheduled_handler', ExecutionPollingMetadata::createWithDefaults()->withExecutionAmountLimit(1)); + $ecotone->run('scheduled_handler', ExecutionPollingMetadata::createWithDefaults()->withExecutionAmountLimit(1)); + + $spans = $recorder->recordedSpans(); + $polls = array_values(array_filter($spans, fn (Span $span): bool => $span->name === 'receive add')); + $this->assertCount(2, $polls); + foreach ($polls as $poll) { + $this->assertTrue($poll->isRoot()); + $handlers = array_values(array_filter($spans, fn (Span $span): bool => $span->parentSpanId === $poll->spanId)); + $this->assertCount(1, $handlers); + $this->assertMatchesRegularExpression('/^process class@anonymous\S*::add$/', $handlers[0]->name); + $this->assertSame($poll->traceId, $handlers[0]->traceId); + } + $this->assertNotSame($polls[0]->traceId, $polls[1]->traceId); + } + + public function test_no_tracing_when_there_is_nothing_to_trace(): void + { + $recorder = new InMemorySpanRecorder(); + $workflow = $this->workflowScheduledHandler(); + + EcotoneLite::bootstrapFlowTesting( + [$workflow::class], + [$workflow, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ) + ->run('scheduled_handler_without_result', ExecutionPollingMetadata::createWithDefaults()->withExecutionAmountLimit(1)); + + $this->assertSame([], $recorder->recordedSpans()); + } + + private function asynchronousUsers(): object + { + return new class () { + #[Asynchronous('async_channel')] + #[CommandHandler('user.register', endpointId: 'user.register.endpoint')] + public function register(string $userId): void + { + } + }; + } + + private function workflowScheduledHandler(): object + { + return new class () { + #[Scheduled(requestChannelName: 'add', endpointId: 'scheduled_handler')] + #[Poller(fixedRateInMilliseconds: 1)] + public function produce(): int + { + return 1; + } + + #[Scheduled(requestChannelName: 'add', endpointId: 'scheduled_handler_without_result')] + #[Poller(fixedRateInMilliseconds: 1)] + public function produceNothing(): ?int + { + return null; + } + + #[InternalHandler(inputChannelName: 'add')] + public function add(): void + { + } + }; + } +} diff --git a/packages/Ecotone/tests/Tracing/FlowShapeTracingTest.php b/packages/Ecotone/tests/Tracing/FlowShapeTracingTest.php new file mode 100644 index 0000000000..0297ba536c --- /dev/null +++ b/packages/Ecotone/tests/Tracing/FlowShapeTracingTest.php @@ -0,0 +1,329 @@ +sendCommandWithRouting('order.place', '1'); + + $this->assertSame( + ['process class@anonymous::place', 'process class@anonymous::store', 'process class@anonymous::validate', 'send command'], + self::spanNames($recorder), + ); + $this->assertSame( + ['process class@anonymous::place' => 'order.placing', 'process class@anonymous::store' => 'order.storing', 'process class@anonymous::validate' => 'order.validating'], + self::endpointIdsByHandlerSpan($recorder), + ); + $this->assertCount(1, self::traceIds($recorder)); + } + + public function test_a_splitter_fans_one_command_out_into_one_handler_span_per_item(): void + { + $recorder = new InMemorySpanRecorder(); + $imports = new class () { + #[CommandHandler('batch.import', outputChannelName: 'batch.split', endpointId: 'batch.importing')] + public function import(array $items): array + { + return $items; + } + + #[Splitter('batch.split', outputChannelName: 'item.handle')] + public function split(array $items): array + { + return $items; + } + + #[InternalHandler('item.handle', endpointId: 'item.handling')] + public function handle(string $item): void + { + } + }; + + self::bootstrap($recorder, [$imports])->sendCommandWithRouting('batch.import', ['a', 'b']); + + $this->assertSame( + ['process class@anonymous::handle', 'process class@anonymous::handle', 'process class@anonymous::import', 'send command'], + self::spanNames($recorder), + ); + $this->assertSame(['item.handling', 'item.handling'], self::endpointIdsOfSpansNamed('process class@anonymous::handle', $recorder)); + $this->assertCount(1, self::traceIds($recorder)); + } + + public function test_a_router_leads_the_trace_to_the_handler_it_chose(): void + { + $recorder = new InMemorySpanRecorder(); + $shipments = new class () { + #[CommandHandler('shipment.dispatch', outputChannelName: 'shipment.route', endpointId: 'shipment.dispatching')] + public function dispatch(string $shipmentId): string + { + return $shipmentId; + } + + #[Router('shipment.route')] + public function route(string $shipmentId): string + { + return 'shipment.express'; + } + + #[InternalHandler('shipment.express', endpointId: 'shipment.expressing')] + public function express(string $shipmentId): void + { + } + + #[InternalHandler('shipment.standard', endpointId: 'shipment.standard.handling')] + public function standard(string $shipmentId): void + { + } + }; + + self::bootstrap($recorder, [$shipments])->sendCommandWithRouting('shipment.dispatch', '1'); + + $this->assertSame( + ['process class@anonymous::dispatch', 'process class@anonymous::express', 'send command'], + self::spanNames($recorder), + ); + $this->assertSame(['shipment.expressing'], self::endpointIdsOfSpansNamed('process class@anonymous::express', $recorder)); + } + + public function test_an_orchestrator_gateway_starting_the_flow_traces_each_step_under_one_trace(): void + { + $recorder = new InMemorySpanRecorder(); + $steps = new class () { + #[InternalHandler('approval.validate', endpointId: 'approval.validating')] + public function validate(string $request): string + { + return $request; + } + + #[InternalHandler('approval.notify', endpointId: 'approval.notifying')] + public function notify(string $request): string + { + return $request; + } + }; + $ecotone = self::bootstrap($recorder, [$steps], [ApprovalWorkflow::class], LicenceTesting::VALID_LICENCE); + + $ecotone->getGateway(ApprovalWorkflow::class)->start(['approval.validate', 'approval.notify'], 'request-1'); + + $this->assertSame( + ['process class@anonymous::notify', 'process class@anonymous::validate', 'send orchestrator'], + self::spanNames($recorder), + ); + $this->assertSame( + ['process class@anonymous::notify' => 'approval.notifying', 'process class@anonymous::validate' => 'approval.validating'], + self::endpointIdsByHandlerSpan($recorder), + ); + $this->assertCount(1, self::traceIds($recorder)); + } + + public function test_an_orchestrator_reached_from_a_command_handler_continues_that_command_trace(): void + { + $recorder = new InMemorySpanRecorder(); + $approvals = new class () { + #[CommandHandler('approval.request', outputChannelName: 'approval.workflow', endpointId: 'approval.requesting')] + public function request(string $request): string + { + return $request; + } + + #[Orchestrator('approval.workflow')] + public function workflow(): array + { + return ['approval.validate', 'approval.notify']; + } + + #[InternalHandler('approval.validate', endpointId: 'approval.validating')] + public function validate(string $request): string + { + return $request; + } + + #[InternalHandler('approval.notify', endpointId: 'approval.notifying')] + public function notify(string $request): string + { + return $request; + } + }; + + self::bootstrap($recorder, [$approvals], [], LicenceTesting::VALID_LICENCE)->sendCommandWithRouting('approval.request', 'request-1'); + + $this->assertSame( + ['process class@anonymous::notify', 'process class@anonymous::request', 'process class@anonymous::validate', 'send command'], + self::spanNames($recorder), + ); + $this->assertSame( + ['process class@anonymous::notify' => 'approval.notifying', 'process class@anonymous::request' => 'approval.requesting', 'process class@anonymous::validate' => 'approval.validating'], + self::endpointIdsByHandlerSpan($recorder), + ); + $this->assertCount(1, self::traceIds($recorder)); + } + + public function test_a_message_gateway_starting_the_flow_is_the_root_of_the_handler_it_reaches(): void + { + $recorder = new InMemorySpanRecorder(); + $notifications = new class () { + #[InternalHandler('notification.send', endpointId: 'notification.sending')] + public function send(string $text): void + { + } + }; + $ecotone = self::bootstrap($recorder, [$notifications], [NotificationGateway::class]); + + $ecotone->getGateway(NotificationGateway::class)->send('hello'); + + $this->assertSame(['process class@anonymous::send', 'send notification.send'], self::spanNames($recorder)); + $this->assertCount(1, self::traceIds($recorder)); + $this->assertSame(['notification.sending'], self::endpointIdsOfSpansNamed('process class@anonymous::send', $recorder)); + } + + public function test_a_command_bus_called_inside_a_command_handler_nests_the_second_command_under_the_first_handler(): void + { + $recorder = new InMemorySpanRecorder(); + $users = new class () { + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId, CommandBus $commandBus): void + { + $commandBus->sendWithRouting('audit.record', $userId); + } + + #[CommandHandler('audit.record', endpointId: 'audit.recording')] + public function record(string $userId): void + { + } + }; + + self::bootstrap($recorder, [$users])->sendCommandWithRouting('user.register', '1'); + + self::getNodeAtTargetedSpan( + [ + 'details' => ['name' => 'send command'], + 'child' => [ + 'details' => ['name' => 'process class@anonymous::register', 'attributes' => [SpanAttribute::ENDPOINT_ID => 'user.registration']], + 'child' => [ + 'details' => ['name' => 'send command'], + 'child' => ['details' => ['name' => 'process class@anonymous::record', 'attributes' => [SpanAttribute::ENDPOINT_ID => 'audit.recording']]], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + } + + /** + * @param object[] $services + * @param class-string[] $gateways + */ + private static function bootstrap(InMemorySpanRecorder $recorder, array $services, array $gateways = [], ?string $licenceKey = null): FlowTestSupport + { + return EcotoneLite::bootstrapFlowTesting( + [...array_map(fn (object $service): string => $service::class, $services), ...$gateways], + [...$services, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + licenceKey: $licenceKey, + ); + } + + /** + * @return string[] + */ + private static function spanNames(InMemorySpanRecorder $recorder): array + { + $names = array_map( + fn (string $name): string => preg_replace('/^process class@anonymous\S*::/', 'process class@anonymous::', $name), + self::spanNamesWithoutFrameworkNoise($recorder->recordedSpans()), + ); + sort($names); + + return $names; + } + + /** + * @return array + */ + private static function endpointIdsByHandlerSpan(InMemorySpanRecorder $recorder): array + { + $endpointIds = []; + foreach ($recorder->recordedSpans() as $span) { + if (isset($span->attributes[SpanAttribute::ENDPOINT_ID])) { + $endpointIds[preg_replace('/^process class@anonymous\S*::/', 'process class@anonymous::', $span->name)] = $span->attributes[SpanAttribute::ENDPOINT_ID]; + } + } + ksort($endpointIds); + + return $endpointIds; + } + + /** + * @return string[] + */ + private static function endpointIdsOfSpansNamed(string $name, InMemorySpanRecorder $recorder): array + { + return array_values(array_map( + fn (Span $span): string => $span->attributes[SpanAttribute::ENDPOINT_ID] ?? '', + array_filter($recorder->recordedSpans(), fn (Span $span): bool => preg_replace('/^process class@anonymous\S*::/', 'process class@anonymous::', $span->name) === $name), + )); + } + + /** + * @return string[] + */ + private static function traceIds(InMemorySpanRecorder $recorder): array + { + return array_values(array_unique(array_map(fn (Span $span): string => $span->traceId, $recorder->recordedSpans()))); + } +} + +interface ApprovalWorkflow +{ + #[OrchestratorGateway] + public function start(array $routing, mixed $data): mixed; +} + +interface NotificationGateway +{ + #[MessageGateway('notification.send')] + public function send(string $text): void; +} diff --git a/packages/Ecotone/tests/Tracing/RouteTracingTest.php b/packages/Ecotone/tests/Tracing/RouteTracingTest.php new file mode 100644 index 0000000000..245d5e1510 --- /dev/null +++ b/packages/Ecotone/tests/Tracing/RouteTracingTest.php @@ -0,0 +1,236 @@ +publishEventWithRouting('merchant.created', '1'); + + self::compareTreesByDetails( + [ + [ + 'details' => ['name' => 'send event'], + 'children' => [ + [ + 'details' => [ + 'name' => 'route merchant.notifier', + 'kind' => SpanKind::Producer, + 'attributes' => ['messaging.destination.name' => 'merchant.notifier.target', 'ecotone.endpoint.id' => 'merchant.notifier'], + ], + 'children' => [ + ['details' => ['name' => 'process class@anonymous::notify', 'attributes' => ['ecotone.endpoint.id' => 'merchant.notifier']], 'children' => []], + ], + ], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + } + + public function test_a_failing_synchronous_handler_fails_its_route_span(): void + { + $recorder = new InMemorySpanRecorder(); + $subscriber = new class () { + #[EventHandler('merchant.created', endpointId: 'merchant.notifier')] + public function notify(string $merchantId): void + { + throw new RuntimeException('Notification failed'); + } + }; + $ecotone = EcotoneLite::bootstrapFlowTesting( + [$subscriber::class], + [$subscriber, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ); + + try { + $ecotone->publishEventWithRouting('merchant.created', '1'); + $this->fail('Expected the handler failure to propagate'); + } catch (RuntimeException) { + } + + $this->assertSame(SpanStatus::Error, self::spanNamed('route merchant.notifier', $recorder)->status); + } + + public function test_an_asynchronous_event_route_leads_through_the_channel_to_its_handler(): void + { + $recorder = new InMemorySpanRecorder(); + $subscriber = new class () { + #[Asynchronous('async_channel')] + #[EventHandler('merchant.created', endpointId: 'merchant.notifier')] + public function notify(string $merchantId): void + { + } + }; + + EcotoneLite::bootstrapFlowTesting( + [$subscriber::class], + [$subscriber, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration(), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + ) + ->publishEventWithRouting('merchant.created', '1') + ->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::getNodeAtTargetedSpan( + [ + 'details' => ['name' => 'send event'], + 'child' => [ + 'details' => ['name' => 'route merchant.notifier'], + 'child' => [ + 'details' => ['name' => 'send async_channel', 'attributes' => ['messaging.destination.name' => 'async_channel']], + 'child' => [ + 'details' => ['name' => 'receive async_channel', 'attributes' => ['messaging.destination.name' => 'async_channel']], + 'child' => ['details' => ['name' => 'process class@anonymous::notify']], + ], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + } + + public function test_an_event_driven_projection_is_a_consumer_span_under_its_route(): void + { + $recorder = new InMemorySpanRecorder(); + $projection = new #[Projection('orders'), FromStream('orders_stream')] class () { + #[EventHandler('order.placed')] + public function whenPlaced(array $event): void + { + } + }; + + $ecotone = EcotoneLite::bootstrapFlowTesting( + [$projection::class], + [$projection, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ); + $ecotone->withEvents([Event::createWithType('order.placed', ['orderId' => '1'])]); + + $ecotone->publishEventWithRouting('order.placed', []); + + self::compareTreesByDetails( + [ + [ + 'details' => ['name' => 'send event'], + 'children' => [ + [ + 'details' => ['name' => 'route orders', 'kind' => SpanKind::Producer, 'attributes' => ['ecotone.projection.name' => 'orders']], + 'children' => [ + ['details' => ['name' => 'process orders', 'kind' => SpanKind::Consumer, 'attributes' => ['ecotone.projection.name' => 'orders']], 'children' => []], + ], + ], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + } + + public function test_an_asynchronous_projection_is_a_consumer_span_after_its_channel(): void + { + $recorder = new InMemorySpanRecorder(); + $projection = new #[Projection('orders'), FromStream('orders_stream'), Asynchronous('async_channel')] class () { + #[EventHandler('order.placed')] + public function whenPlaced(array $event): void + { + } + }; + + $ecotone = EcotoneLite::bootstrapFlowTesting( + [$projection::class], + [$projection, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration(), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + licenceKey: LicenceTesting::VALID_LICENCE, + ); + $ecotone->withEvents([Event::createWithType('order.placed', ['orderId' => '1'])]); + + $ecotone->publishEventWithRouting('order.placed', []) + ->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + self::getNodeAtTargetedSpan( + [ + 'details' => ['name' => 'send event'], + 'child' => [ + 'details' => ['name' => 'route orders'], + 'child' => [ + 'details' => ['name' => 'send async_channel'], + 'child' => [ + 'details' => ['name' => 'receive async_channel'], + 'child' => ['details' => ['name' => 'process orders', 'kind' => SpanKind::Consumer]], + ], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + } + + public function test_every_span_names_the_messaging_system_and_handler_spans_name_their_endpoint(): void + { + $recorder = new InMemorySpanRecorder(); + $users = new class () { + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId): void + { + } + }; + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ) + ->sendCommandWithRouting('user.register', '1'); + + foreach ($recorder->recordedSpans() as $span) { + $this->assertSame('ecotone', $span->attributes['messaging.system'], $span->name); + } + $this->assertSame('user.registration', self::spanNamed('process class@anonymous::register', $recorder)->attributes['ecotone.endpoint.id']); + } + + private static function spanNamed(string $name, InMemorySpanRecorder $recorder): Span + { + $spans = array_values(array_filter($recorder->recordedSpans(), fn (Span $span): bool => $span->name === $name)); + self::assertNotEmpty($spans, 'Expected a span named ' . $name . ', got: ' . implode(', ', self::spanNamesWithoutFrameworkNoise($recorder->recordedSpans()))); + + return $spans[0]; + } +} diff --git a/packages/Ecotone/tests/Tracing/TracingConfigurationTest.php b/packages/Ecotone/tests/Tracing/TracingConfigurationTest.php new file mode 100644 index 0000000000..b2aa540dce --- /dev/null +++ b/packages/Ecotone/tests/Tracing/TracingConfigurationTest.php @@ -0,0 +1,122 @@ +headers = $headers; + } + }; + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + ) + ->sendCommandWithRouting('user.register', '1') + ->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertSame([], $recorder->recordedSpans()); + $this->assertArrayNotHasKey(MessageHeaders::TRACEPARENT, $users->headers); + } + + public function test_with_a_configured_recorder_the_trace_carrier_reaches_the_asynchronous_handler(): void + { + $recorder = new InMemorySpanRecorder(); + $users = new class () { + public array $headers = []; + + #[Asynchronous('async_channel')] + #[CommandHandler('user.register', endpointId: 'user.register.endpoint')] + public function register(string $userId, #[Headers] array $headers): void + { + $this->headers = $headers; + } + }; + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration(), SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + ) + ->sendCommandWithRouting('user.register', '1') + ->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertMatchesRegularExpression('/^00-[0-9a-f]{32}-[0-9a-f]{16}-01$/', $users->headers[MessageHeaders::TRACEPARENT]); + } + + public function test_only_the_configured_headers_become_span_attributes(): void + { + $recorder = new InMemorySpanRecorder(); + $users = new class () { + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId): void + { + } + }; + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()->withSpanAttributesFromHeaders('tenant')], + ) + ->sendCommandWithRouting('user.register', '1', metadata: ['tenant' => 'a', 'authorization' => 'Bearer secret']); + + foreach ($recorder->recordedSpans() as $span) { + $this->assertSame('a', $span->attributes['tenant'], $span->name); + $this->assertArrayNotHasKey('authorization', $span->attributes, $span->name); + } + } + + public function test_a_user_header_named_like_a_framework_attribute_does_not_become_one(): void + { + $recorder = new InMemorySpanRecorder(); + $users = new class () { + #[CommandHandler('user.register', endpointId: 'user.registration')] + public function register(string $userId): void + { + } + }; + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ) + ->sendCommandWithRouting('user.register', '1', metadata: ['ecotone.endpoint.id' => 'forged', 'ecotone.trace.root' => true, 'messaging.system' => 'forged']); + + $spansByName = []; + foreach ($recorder->recordedSpans() as $span) { + $spansByName[preg_replace('/^process class@anonymous\\S*::/', 'process class@anonymous::', $span->name)] = $span; + $this->assertSame('ecotone', $span->attributes['messaging.system'], $span->name); + } + $this->assertSame('user.registration', $spansByName['process class@anonymous::register']->attributes['ecotone.endpoint.id'] ?? null); + $this->assertArrayNotHasKey('ecotone.endpoint.id', $spansByName['send command']->attributes); + $this->assertArrayNotHasKey('ecotone.trace.root', $spansByName['process class@anonymous::register']->attributes); + } +} diff --git a/packages/Ecotone/tests/Tracing/TracingEdgeCasesTest.php b/packages/Ecotone/tests/Tracing/TracingEdgeCasesTest.php new file mode 100644 index 0000000000..d963aa17ec --- /dev/null +++ b/packages/Ecotone/tests/Tracing/TracingEdgeCasesTest.php @@ -0,0 +1,206 @@ +bootstrapWithEventForwardedToSecondChannel($recorder, [ + SimpleMessageChannelBuilder::createQueueChannel('async_channel'), + SimpleMessageChannelBuilder::createQueueChannel('failing_channel'), + ], [$blockingInterceptor]); + $ecotone->sendCommandWithRouting('user.register', '1'); + + $caught = $this->runConsumer($ecotone); + $this->assertStringContainsString('Exception before send', $caught?->getMessage() ?? ''); + + $recorder->clear(); + $ecotone->sendCommandWithRouting('user.register', '2'); + $this->assertSame('send command', self::rootSpan($recorder)->name); + } + + public function test_a_command_carrying_a_trace_carrier_continues_that_trace_as_a_child(): void + { + $recorder = new InMemorySpanRecorder(); + $users = new class () { + #[CommandHandler('user.register')] + public function register(string $userId): void + { + } + }; + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ) + ->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => '00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01']); + + $commandBus = self::singleSpanNamed('send command', $recorder); + $this->assertSame('0af7651916cd43dd8448eb211c80319c', $commandBus->traceId); + $this->assertSame('b7ad6b7169203331', $commandBus->parentSpanId); + $this->assertSame($commandBus->spanId, self::singleSpanNamed('process class@anonymous::register', $recorder)->parentSpanId); + } + + public function test_the_trace_carrier_of_the_incoming_command_does_not_leak_into_the_handler_span_parent(): void + { + $recorder = new InMemorySpanRecorder(); + $users = new class () { + #[CommandHandler('user.register')] + public function register(string $userId): void + { + } + }; + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ) + ->sendCommandWithRouting('user.register', '1', metadata: [MessageHeaders::TRACEPARENT => '00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01']); + + $handler = self::singleSpanNamed('process class@anonymous::register', $recorder); + $this->assertNotSame('b7ad6b7169203331', $handler->parentSpanId); + } + + public function test_distributed_bus_send_is_a_producer_span_carrying_the_trace_to_the_other_service(): void + { + $recorder = new InMemorySpanRecorder(); + $ticketService = new class () { + public ?string $traceparent = null; + + #[CommandHandler('ticket.create')] + public function create(string $description, #[Header(MessageHeaders::TRACEPARENT)] ?string $traceparent = null): void + { + $this->traceparent = $traceparent; + } + }; + + EcotoneLite::bootstrapFlowTesting( + [$ticketService::class], + [$ticketService, ...self::recordingSpansInto($recorder)], + [ + self::tracingConfiguration(), + SimpleMessageChannelBuilder::createQueueChannel('distributed_channel'), + DistributedServiceMap::initialize()->withCommandMapping(targetServiceName: 'ticket_service', channelName: 'distributed_channel'), + ServiceConfiguration::createWithDefaults()->withServiceName('ticket_service'), + ], + licenceKey: LicenceTesting::VALID_LICENCE, + ) + ->getDistributedBus()->sendCommand('ticket_service', 'ticket.create', 'User changed billing address'); + + $distributedBus = self::singleSpanNamed('send distributed', $recorder); + $this->assertSame(SpanKind::Producer, $distributedBus->kind); + $this->assertSame($distributedBus->spanId, self::singleSpanNamed('send distributed_channel', $recorder)->parentSpanId); + } + + private function bootstrapWithEventForwardedToSecondChannel(InMemorySpanRecorder $recorder, array $channelsToRegister, array $servicesToRegister = []): FlowTestSupport + { + $users = new class () { + #[Asynchronous('async_channel')] + #[CommandHandler('user.register', endpointId: 'userRegisterEndpoint')] + public function register(string $userId, EventBus $eventBus): void + { + $eventBus->publishWithRouting('user.registered', $userId); + } + }; + $notifier = new class () { + #[Asynchronous('failing_channel')] + #[EventHandler('user.registered', endpointId: 'userNotifierEndpoint')] + public function notify(string $userId): void + { + } + }; + + return EcotoneLite::bootstrapFlowTesting( + [$users::class, $notifier::class, ...array_map(fn (object $service): string => $service::class, $servicesToRegister)], + [$users, $notifier, ...self::recordingSpansInto($recorder), ...$servicesToRegister], + [self::tracingConfiguration(), ...$channelsToRegister], + enabledModules: [ModulePackageList::CORE_PACKAGE], + ); + } + + private function asynchronousUsers(): object + { + return new class () { + #[Asynchronous('async_channel')] + #[CommandHandler('user.register', endpointId: 'user.register.endpoint')] + public function register(string $userId): void + { + } + }; + } + + private function runConsumer(FlowTestSupport $ecotone, string $channel = 'async_channel'): ?Throwable + { + try { + $ecotone->run($channel, ExecutionPollingMetadata::createWithTestingSetup()); + } catch (Throwable $failure) { + return $failure; + } + + return null; + } + + private static function singleSpanNamed(string $name, InMemorySpanRecorder $recorder): Span + { + $spans = array_values(array_filter($recorder->recordedSpans(), fn (Span $span): bool => $span->name === $name)); + self::assertCount(1, $spans, 'Expected exactly one span named ' . $name . ', got: ' . implode(', ', self::spanNamesWithoutFrameworkNoise($recorder->recordedSpans()))); + + return $spans[0]; + } + + private static function assertEverySpanNamedHasStatus(string $name, SpanStatus $status, InMemorySpanRecorder $recorder): void + { + $spans = array_values(array_filter($recorder->recordedSpans(), fn (Span $span): bool => $span->name === $name)); + self::assertNotEmpty($spans, 'Expected spans named ' . $name . ', got: ' . implode(', ', self::spanNamesWithoutFrameworkNoise($recorder->recordedSpans()))); + foreach ($spans as $span) { + self::assertSame($status, $span->status, $name); + } + } + + private static function rootSpan(InMemorySpanRecorder $recorder): Span + { + $roots = array_values(array_filter($recorder->recordedSpans(), fn (Span $span): bool => $span->isRoot())); + self::assertCount(1, $roots, 'Expected exactly one root span, got: ' . implode(', ', array_map(fn (Span $span): string => $span->name, $roots))); + + return $roots[0]; + } +} diff --git a/packages/Ecotone/tests/Tracing/TracingTestCase.php b/packages/Ecotone/tests/Tracing/TracingTestCase.php new file mode 100644 index 0000000000..696dd04e6b --- /dev/null +++ b/packages/Ecotone/tests/Tracing/TracingTestCase.php @@ -0,0 +1,175 @@ + $recorder]; + } + + protected static function tracingConfiguration(): TracingConfiguration + { + return TracingConfiguration::createWithDefaults()->withSpanRecorder(InMemorySpanRecorder::class); + } + + /** + * @param Span[] $spans + * @return string[] + */ + public static function spanNamesWithoutFrameworkNoise(array $spans): array + { + return array_values(array_filter( + array_map(fn (Span $span): string => $span->name, $spans), + fn (string $name): bool => ! str_starts_with($name, self::INTERNAL_NOISE_SPAN_NAME_PREFIX), + )); + } + + /** + * @param Span[] $spans + */ + public static function buildTree(array $spans): array + { + $tree = []; + foreach ($spans as $span) { + if (str_starts_with($span->name, self::INTERNAL_NOISE_SPAN_NAME_PREFIX)) { + continue; + } + + $preparedSpan = [ + 'details' => [ + 'name' => $span->name, + 'trace_id' => $span->traceId, + 'span_id' => $span->spanId, + 'parent_id' => $span->parentSpanId, + 'attributes' => $span->attributes, + 'kind' => $span->kind, + 'status' => $span->status, + 'events' => $span->events, + ], + 'children' => [], + ]; + + $tree = self::putInTree($preparedSpan, $tree, true); + } + + return $tree; + } + + public static function compareTreesByDetails(array $expectedTree, array $collectedTree): void + { + foreach ($expectedTree as $expectedNode) { + self::findAndCompareNode($expectedNode, $collectedTree); + } + } + + private static function putInTree(array $searchedSpan, array $tree, bool $isRoot): array + { + $searchedSpanId = $searchedSpan['details']['span_id']; + $searchedParentId = $searchedSpan['details']['parent_id']; + + foreach ($tree as $nodeId => $node) { + if ($node['details']['parent_id'] === $searchedSpanId) { + unset($tree[$nodeId]); + $searchedSpan['children'][$nodeId] = $node; + } + } + $changedStructure = $tree; + + foreach ($tree as $nodeId => $node) { + if ($nodeId === $searchedParentId) { + if (in_array($searchedSpanId, array_keys($node['children']))) { + continue; + } + + $changedStructure[$nodeId]['children'][$searchedSpanId] = $searchedSpan; + } elseif ($node['details']['parent_id'] === $searchedSpanId) { + unset($changedStructure[$nodeId]); + $searchedSpan['children'][$nodeId] = $node; + + $changedStructure = self::putInTree($searchedSpan, $changedStructure, true); + } else { + $changedStructure[$nodeId]['children'] = self::putInTree($searchedSpan, $node['children'], false); + } + } + + if ($isRoot && $changedStructure == $tree) { + $changedStructure[$searchedSpanId] = $searchedSpan; + } + + return $changedStructure; + } + + private static function findAndCompareNode(array $expectedTreeNode, array $collectedTree): void + { + foreach ($collectedTree as $collectedNode) { + if ($expectedTreeNode['details']['name'] !== $collectedNode['details']['name']) { + continue; + } + if (isset($expectedTreeNode['details']['attributes']['flowId'])) { + if (($collectedNode['details']['attributes']['flowId'] ?? null) != $expectedTreeNode['details']['attributes']['flowId']) { + continue; + } + } + + foreach ($expectedTreeNode['details'] as $expectedKey => $expectedValue) { + self::assertArrayHasKey($expectedKey, $collectedNode['details']); + if ($expectedKey === 'attributes') { + foreach ($expectedValue as $expectedAttributeKey => $expectedAttributeValue) { + self::assertArrayHasKey($expectedAttributeKey, $collectedNode['details']['attributes'], "Attribute {$expectedAttributeKey} missing on span {$collectedNode['details']['name']}"); + self::assertSame($expectedAttributeValue, $collectedNode['details']['attributes'][$expectedAttributeKey]); + } + } else { + self::assertSame($expectedValue, $collectedNode['details'][$expectedKey]); + } + } + self::compareTreesByDetails($expectedTreeNode['children'], $collectedNode['children']); + + return; + } + + self::fail("Could not find span named {$expectedTreeNode['details']['name']}. Spans at this level: " . json_encode(array_map(fn (array $node) => $node['details']['name'], array_values($collectedTree)))); + } + + public static function getNodeAtTargetedSpan(array $expectedTreeNode, array $collectedTree): array + { + foreach ($collectedTree as $collectedNode) { + if ($expectedTreeNode['details']['name'] !== $collectedNode['details']['name']) { + continue; + } + foreach ($expectedTreeNode['details'] as $expectedKey => $expectedValue) { + if ($expectedKey === 'attributes') { + foreach ($expectedValue as $expectedAttributeKey => $expectedAttributeValue) { + self::assertSame($expectedAttributeValue, $collectedNode['details']['attributes'][$expectedAttributeKey] ?? null, "Attribute {$expectedAttributeKey} on span {$collectedNode['details']['name']}"); + } + } else { + self::assertSame($expectedValue, $collectedNode['details'][$expectedKey]); + } + } + + if (! array_key_exists('child', $expectedTreeNode)) { + return $collectedNode; + } + + return self::getNodeAtTargetedSpan($expectedTreeNode['child'], $collectedNode['children']); + } + + self::fail("Could not find span named {$expectedTreeNode['details']['name']}. Spans at this level: " . json_encode(array_map(fn (array $node) => $node['details']['name'], array_values($collectedTree)))); + } +} diff --git a/packages/Ecotone/tests/Tracing/TracingTreeTest.php b/packages/Ecotone/tests/Tracing/TracingTreeTest.php new file mode 100644 index 0000000000..4787f331ae --- /dev/null +++ b/packages/Ecotone/tests/Tracing/TracingTreeTest.php @@ -0,0 +1,449 @@ +sendCommandWithRouting('user.register', '1'); + + self::compareTreesByDetails( + [ + [ + 'details' => ['name' => 'send command', 'kind' => SpanKind::Server, 'parent_id' => null], + 'children' => [ + [ + 'details' => ['name' => 'process class@anonymous::register', 'kind' => SpanKind::Server], + 'children' => [], + ], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + } + + public function test_tracing_tree_with_two_levels_of_nesting_and_message_handler(): void + { + $recorder = new InMemorySpanRecorder(); + $handler = new class () { + #[CommandHandler('handleCommand', outputChannelName: 'handleMessage')] + public function handleCommand(Message $message): Message + { + return $message; + } + + #[InternalHandler('handleMessage')] + public function handle(Message $message): void + { + } + }; + + EcotoneLite::bootstrapFlowTesting( + [$handler::class], + [$handler, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ) + ->sendCommandWithRouting('handleCommand'); + + self::compareTreesByDetails( + [ + [ + 'details' => ['name' => 'send command'], + 'children' => [ + ['details' => ['name' => 'process class@anonymous::handleCommand'], 'children' => []], + ['details' => ['name' => 'process class@anonymous::handle'], 'children' => []], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + } + + public function test_tracing_tree_with_two_levels_of_nesting(): void + { + $recorder = new InMemorySpanRecorder(); + $users = $this->users(); + $subscriber = $this->merchantSubscriber(); + + EcotoneLite::bootstrapFlowTesting( + [$users::class, $subscriber::class], + [$users, $subscriber, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ) + ->publishEventWithRouting('merchant.created', '1'); + + self::compareTreesByDetails( + [ + [ + 'details' => ['name' => 'send event'], + 'children' => [ + [ + 'details' => ['name' => 'route merchant.to.user', 'kind' => SpanKind::Producer], + 'children' => [ + [ + 'details' => ['name' => 'process class@anonymous::merchantToUser'], + 'children' => [ + [ + 'details' => ['name' => 'send command'], + 'children' => [ + ['details' => ['name' => 'process class@anonymous::register'], 'children' => []], + ], + ], + ], + ], + ], + ], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + } + + public function test_tracing_tree_with_two_branches_on_same_level(): void + { + $recorder = new InMemorySpanRecorder(); + $users = $this->users(); + $subscriberOne = $this->merchantSubscriber(); + $subscriberTwo = new class () { + #[EventHandler('merchant.created', endpointId: 'merchant.to.second.user')] + public function merchantToUser(string $merchantId, CommandBus $commandBus): void + { + $commandBus->sendWithRouting('user.register', $merchantId . '-second'); + } + }; + + EcotoneLite::bootstrapFlowTesting( + [$users::class, $subscriberOne::class, $subscriberTwo::class], + [$users, $subscriberOne, $subscriberTwo, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ) + ->publishEventWithRouting('merchant.created', '1'); + + $commandBusChild = [ + 'details' => ['name' => 'send command'], + 'children' => [ + ['details' => ['name' => 'process class@anonymous::register'], 'children' => []], + ], + ]; + self::compareTreesByDetails( + [ + [ + 'details' => ['name' => 'send event'], + 'children' => [ + [ + 'details' => ['name' => 'route merchant.to.user'], + 'children' => [['details' => ['name' => 'process class@anonymous::merchantToUser'], 'children' => [$commandBusChild]]], + ], + [ + 'details' => ['name' => 'route merchant.to.second.user'], + 'children' => [['details' => ['name' => 'process class@anonymous::merchantToUser'], 'children' => [$commandBusChild]]], + ], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + $this->assertCount(9, self::spanNamesWithoutFrameworkNoise($recorder->recordedSpans())); + } + + public function test_tracing_tree_with_three_levels_of_nesting(): void + { + $recorder = new InMemorySpanRecorder(); + $merchants = new class () { + #[CommandHandler('merchant.create')] + public function create(string $merchantId, EventBus $eventBus): void + { + $eventBus->publishWithRouting('merchant.created', $merchantId); + } + }; + $users = $this->users(); + $subscriber = $this->merchantSubscriber(); + + EcotoneLite::bootstrapFlowTesting( + [$merchants::class, $users::class, $subscriber::class], + [$merchants, $users, $subscriber, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ) + ->sendCommandWithRouting('merchant.create', '1'); + + self::compareTreesByDetails( + [ + [ + 'details' => ['name' => 'send command'], + 'children' => [ + [ + 'details' => ['name' => 'process class@anonymous::create'], + 'children' => [ + [ + 'details' => ['name' => 'send event'], + 'children' => [ + [ + 'details' => ['name' => 'route merchant.to.user'], + 'children' => [ + [ + 'details' => ['name' => 'process class@anonymous::merchantToUser'], + 'children' => [ + [ + 'details' => ['name' => 'send command'], + 'children' => [ + ['details' => ['name' => 'process class@anonymous::register'], 'children' => []], + ], + ], + ], + ], + ], + ], + ], + ], + ], + ], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + } + + public function test_every_span_of_one_flow_shares_the_trace_id_of_its_root(): void + { + $recorder = new InMemorySpanRecorder(); + $users = $this->users(); + $subscriber = $this->merchantSubscriber(); + + EcotoneLite::bootstrapFlowTesting( + [$users::class, $subscriber::class], + [$users, $subscriber, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ) + ->publishEventWithRouting('merchant.created', '1'); + + $traceIds = array_unique(array_map(fn (Span $span): string => $span->traceId, $recorder->recordedSpans())); + $this->assertCount(1, $traceIds); + $this->assertCount(1, array_filter($recorder->recordedSpans(), fn (Span $span): bool => $span->isRoot())); + } + + public function test_separate_traces(): void + { + $recorder = new InMemorySpanRecorder(); + $users = $this->users(); + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ) + ->sendCommandWithRouting('user.register', '1', metadata: ['flowId' => '1']) + ->sendCommandWithRouting('user.register', '2', metadata: ['flowId' => '2']); + + self::compareTreesByDetails( + [ + [ + 'details' => ['name' => 'send command', 'attributes' => ['flowId' => '1']], + 'children' => [ + ['details' => ['name' => 'process class@anonymous::register', 'attributes' => ['flowId' => '1']], 'children' => []], + ], + ], + [ + 'details' => ['name' => 'send command', 'attributes' => ['flowId' => '2']], + 'children' => [ + ['details' => ['name' => 'process class@anonymous::register', 'attributes' => ['flowId' => '2']], 'children' => []], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + $traceIds = array_unique(array_map(fn (Span $span): string => $span->traceId, $recorder->recordedSpans())); + $this->assertCount(2, $traceIds); + } + + public function test_user_land_metadata_is_skipped_when_it_is_not_scalar(): void + { + $recorder = new InMemorySpanRecorder(); + $users = $this->users(); + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ) + ->sendCommandWithRouting('user.register', '1', metadata: ['tokens' => ['123'], 'user' => new stdClass(), 'user_id' => '123']); + + $node = self::getNodeAtTargetedSpan(['details' => ['name' => 'send command']], self::buildTree($recorder->recordedSpans())); + + $this->assertArrayNotHasKey('user', $node['details']['attributes']); + $this->assertArrayNotHasKey('tokens', $node['details']['attributes']); + $this->assertSame('123', $node['details']['attributes']['user_id']); + } + + public function test_spans_carry_message_identity_and_a_child_message_points_at_its_parent(): void + { + $recorder = new InMemorySpanRecorder(); + $users = $this->users(); + $subscriber = $this->merchantSubscriber(); + $messageId = Uuid::v7()->toRfc4122(); + $correlationId = Uuid::v7()->toRfc4122(); + $timestamp = 1680436648; + + EcotoneLite::bootstrapFlowTesting( + [$users::class, $subscriber::class], + [$users, $subscriber, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ) + ->publishEventWithRouting('merchant.created', '1', metadata: [ + MessageHeaders::MESSAGE_ID => $messageId, + MessageHeaders::MESSAGE_CORRELATION_ID => $correlationId, + MessageHeaders::TIMESTAMP => $timestamp, + ]); + + $spans = $recorder->recordedSpans(); + $this->assertCount(5, $spans); + + $this->assertChildSpan('process class@anonymous::register', $spans[0], $messageId, $correlationId); + $this->assertChildSpan('send command', $spans[1], $messageId, $correlationId); + $this->assertSpanWith('process class@anonymous::merchantToUser', $spans[2], $messageId, $correlationId, $timestamp); + $this->assertSpanWith('route merchant.to.user', $spans[3], $messageId, $correlationId, $timestamp); + $this->assertSpanWith('send event', $spans[4], $messageId, $correlationId, $timestamp); + } + + public function test_a_failing_handler_marks_its_span_and_the_bus_span_as_failed_with_the_exception_recorded(): void + { + $recorder = new InMemorySpanRecorder(); + $users = new class () { + #[CommandHandler('user.register')] + public function register(string $userId): void + { + throw new InvalidArgumentException('User already registered'); + } + }; + $ecotone = EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ); + + try { + $ecotone->sendCommandWithRouting('user.register', '1'); + $this->fail('Expected the handler failure to propagate'); + } catch (InvalidArgumentException) { + } + + $spans = $recorder->recordedSpans(); + $this->assertCount(2, $spans); + foreach ($spans as $span) { + $this->assertSame(SpanStatus::Error, $span->status); + $this->assertSame('User already registered', $span->statusDescription); + $exceptionEvents = array_values(array_filter($span->events, fn ($event): bool => $event->name === 'exception')); + $this->assertCount(1, $exceptionEvents); + $this->assertSame(InvalidArgumentException::class, $exceptionEvents[0]->attributes['exception.type']); + $this->assertSame('User already registered', $exceptionEvents[0]->attributes['exception.message']); + } + } + + public function test_query_bus_and_query_handler_are_traced(): void + { + $recorder = new InMemorySpanRecorder(); + $users = $this->users(); + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users, ...self::recordingSpansInto($recorder)], + [self::tracingConfiguration()], + ) + ->sendQueryWithRouting('user.get'); + + self::compareTreesByDetails( + [ + [ + 'details' => ['name' => 'send query'], + 'children' => [ + ['details' => ['name' => 'process class@anonymous::isRegistered'], 'children' => []], + ], + ], + ], + self::buildTree($recorder->recordedSpans()) + ); + } + + private function users(): object + { + return new class () { + #[CommandHandler('user.register')] + public function register(string $userId): void + { + } + + #[QueryHandler('user.get')] + public function isRegistered(): bool + { + return true; + } + }; + } + + private function merchantSubscriber(): object + { + return new class () { + #[EventHandler('merchant.created', endpointId: 'merchant.to.user')] + public function merchantToUser(string $merchantId, CommandBus $commandBus): void + { + $commandBus->sendWithRouting('user.register', $merchantId); + } + }; + } + + private function assertSpanWith(string $expectedName, Span $span, string $messageId, string $correlationId, int $timestamp): void + { + $this->assertSame($expectedName, $span->name); + $this->assertSame($messageId, $span->attributes[MessageHeaders::MESSAGE_ID]); + $this->assertSame($correlationId, $span->attributes[MessageHeaders::MESSAGE_CORRELATION_ID]); + $this->assertSame($timestamp, $span->attributes[MessageHeaders::TIMESTAMP]); + $this->assertSame(SpanStatus::Ok, $span->status); + } + + private function assertChildSpan(string $expectedName, Span $span, string $messageId, string $correlationId): void + { + $this->assertSame($expectedName, $span->name); + $this->assertNotSame($messageId, $span->attributes[MessageHeaders::MESSAGE_ID]); + $this->assertSame($messageId, $span->attributes[MessageHeaders::PARENT_MESSAGE_ID]); + $this->assertSame($correlationId, $span->attributes[MessageHeaders::MESSAGE_CORRELATION_ID]); + } +} diff --git a/packages/OpenTelemetry/Api/OpenTelemetryConfiguration.php b/packages/OpenTelemetry/Api/OpenTelemetryConfiguration.php new file mode 100644 index 0000000000..334723f87f --- /dev/null +++ b/packages/OpenTelemetry/Api/OpenTelemetryConfiguration.php @@ -0,0 +1,32 @@ + $forceFlush]); + } + + public function withForceFlushOnAsynchronousMessageHandled(bool $forceFlush): self + { + return new self(...[...get_object_vars($this), 'forceFlushOnAsynchronousMessageHandled' => $forceFlush]); + } +} diff --git a/packages/OpenTelemetry/composer.json b/packages/OpenTelemetry/composer.json index 4c4a210862..0c8fa4bc39 100644 --- a/packages/OpenTelemetry/composer.json +++ b/packages/OpenTelemetry/composer.json @@ -36,6 +36,7 @@ ], "autoload": { "psr-4": { + "Ecotone\\Api\\OpenTelemetry\\": "Api/", "Ecotone\\OpenTelemetry\\": "src" } }, diff --git a/packages/OpenTelemetry/phpstan.neon b/packages/OpenTelemetry/phpstan.neon index 9af9c6bf44..bdfa3905bb 100644 --- a/packages/OpenTelemetry/phpstan.neon +++ b/packages/OpenTelemetry/phpstan.neon @@ -2,6 +2,7 @@ parameters: level: 3 reportUnmatchedIgnoredErrors: false paths: + - Api - src excludePaths: - src/Support/OTelTracer.php diff --git a/packages/OpenTelemetry/src/AddSpanEventLogger.php b/packages/OpenTelemetry/src/AddSpanEventLogger.php deleted file mode 100644 index 02bd5c79df..0000000000 --- a/packages/OpenTelemetry/src/AddSpanEventLogger.php +++ /dev/null @@ -1,45 +0,0 @@ - in_array($key, [MessageHeaders::MESSAGE_ID, MessageHeaders::MESSAGE_CORRELATION_ID, MessageHeaders::PARENT_MESSAGE_ID]), - ARRAY_FILTER_USE_KEY - ); - - if (isset($context['exception']) && $context['exception'] instanceof Throwable) { - $exception = $context['exception']; - $attributes['exceptionMessage'] = $exception->getMessage(); - $attributes['exceptionClass'] = $exception::class; - $attributes['exceptionTrace'] = $exception->getTraceAsString(); - } - - Span::getCurrent()->addEvent( - $message, - $attributes - ); - } -} diff --git a/packages/OpenTelemetry/src/ChannelSendSpan.php b/packages/OpenTelemetry/src/ChannelSendSpan.php deleted file mode 100644 index 39ec6868ad..0000000000 --- a/packages/OpenTelemetry/src/ChannelSendSpan.php +++ /dev/null @@ -1,49 +0,0 @@ -close(StatusCode::STATUS_OK, null); - } - - public function closeWithFailure(Throwable $exception): void - { - if ($this->isClosed) { - return; - } - - $this->span->recordException($exception); - $this->close(StatusCode::STATUS_ERROR, $exception->getMessage()); - } - - private function close(string $statusCode, ?string $statusDescription): void - { - if ($this->isClosed) { - return; - } - $this->isClosed = true; - - $this->span->setStatus($statusCode, $statusDescription); - $this->scope->detach(); - $this->span->end(); - } -} diff --git a/packages/OpenTelemetry/src/Configuration/OpenTelemetryModule.php b/packages/OpenTelemetry/src/Configuration/OpenTelemetryModule.php index a0b072f826..f64bafb6f2 100644 --- a/packages/OpenTelemetry/src/Configuration/OpenTelemetryModule.php +++ b/packages/OpenTelemetry/src/Configuration/OpenTelemetryModule.php @@ -6,18 +6,16 @@ use Ecotone\AnnotationFinder\AnnotationFinder; use Ecotone\Api\Attribute\AsynchronousRunningEndpoint; -use Ecotone\Api\Attribute\CommandHandler; -use Ecotone\Api\Attribute\EventHandler; -use Ecotone\Api\Attribute\InternalHandler; use Ecotone\Api\Attribute\ModuleAnnotation; -use Ecotone\Api\Attribute\QueryHandler; -use Ecotone\Api\ExtensionObject\MessageChannelBuilder; use Ecotone\Api\ExtensionObject\ModulePackageList; +use Ecotone\Api\ExtensionObject\ServiceConfiguration; use Ecotone\Api\Gateway\CommandBus; use Ecotone\Api\Gateway\DistributedBus; -use Ecotone\Api\Gateway\EventBus; +use Ecotone\Api\Gateway\EcotoneClockInterface; use Ecotone\Api\Gateway\QueryBus; use Ecotone\Api\Interceptor\Precedence; +use Ecotone\Api\OpenTelemetry\OpenTelemetryConfiguration; +use Ecotone\Api\Tracing\SpanRecorder; use Ecotone\Lite\Test\Configuration\ProvidedReferences; use Ecotone\Lite\Test\Configuration\ResolvesExtensionObjectsForFlowTesting; use Ecotone\Messaging\Config\Annotation\AnnotationModule; @@ -29,12 +27,9 @@ use Ecotone\Messaging\Config\ModuleReferenceSearchService; use Ecotone\Messaging\Handler\InterfaceToCallRegistry; use Ecotone\Messaging\Handler\Processor\MethodInvoker\AroundInterceptorBuilder; -use Ecotone\Messaging\Handler\Processor\MethodInvoker\MethodInterceptorBuilder; -use Ecotone\Messaging\Support\MessageBuilder; use Ecotone\OpenTelemetry\EcotoneForcedTraceFlush; -use Ecotone\OpenTelemetry\TracerInterceptor; -use Ecotone\OpenTelemetry\TracingChannelAdapterBuilder; -use OpenTelemetry\API\Trace\NoopTracerProvider; +use Ecotone\OpenTelemetry\OpenTelemetryTracer; +use Ecotone\Tracing\Config\TracerReference; use OpenTelemetry\API\Trace\TracerProviderInterface; #[ModuleAnnotation] @@ -48,62 +43,38 @@ public static function create(AnnotationFinder $annotationRegistrationService, I return new self(); } + public function getModuleExtensions(ServiceConfiguration $serviceConfiguration, array $serviceExtensions, ?InterfaceToCallRegistry $interfaceToCallRegistry = null): array + { + if (ExtensionObjectResolver::contains(NoTracerProviderGiven::class, $serviceExtensions)) { + return []; + } + + return [new TracerReference(OpenTelemetryTracer::class)]; + } + public function prepare(Configuration $messagingConfiguration, array $extensionObjects, ModuleReferenceSearchService $moduleReferenceSearchService, InterfaceToCallRegistry $interfaceToCallRegistry): void { - $tracingConfiguration = ExtensionObjectResolver::resolveUnique(TracingConfiguration::class, $extensionObjects, TracingConfiguration::createWithDefaults()); - $messageChannelBuilders = ExtensionObjectResolver::resolve(MessageChannelBuilder::class, $extensionObjects); + $openTelemetryConfiguration = ExtensionObjectResolver::resolveUnique(OpenTelemetryConfiguration::class, $extensionObjects, OpenTelemetryConfiguration::createWithDefaults()); if (ExtensionObjectResolver::contains(NoTracerProviderGiven::class, $extensionObjects)) { - $messagingConfiguration->registerServiceDefinition(TracerProviderInterface::class, new Definition(NoopTracerProvider::class)); + return; } - $messagingConfiguration->addCompilerPass(new RegisterAddSpanEventLoggerCompilerPass()); - $messagingConfiguration->addCompilerPass(new TraceDynamicConsistencyBoundaryCompilerPass()); - $messagingConfiguration->registerServiceDefinition( - TracerInterceptor::class, - new Definition(TracerInterceptor::class, [ + OpenTelemetryTracer::class, + new Definition(OpenTelemetryTracer::class, [ new Reference(TracerProviderInterface::class), - new Reference(MessageBuilder::class), + new Reference(EcotoneClockInterface::class), + new Reference(SpanRecorder::class), ]) ); - if ($tracingConfiguration->higherThanOrEqualTo(TracingConfiguration::TRACING_LEVEL_FRAMEWORK)) { - $this->registerTracerFor('trace', '*', $messagingConfiguration, $interfaceToCallRegistry); - } - - foreach ($messageChannelBuilders as $messageChannelBuilder) { - if ($messageChannelBuilder->isPollable()) { - $messagingConfiguration->registerChannelInterceptor(new TracingChannelAdapterBuilder($messageChannelBuilder->getMessageChannelName())); - } - } - - $this->registerTracerFor('traceCommandHandler', CommandHandler::class, $messagingConfiguration, $interfaceToCallRegistry); - $this->registerTracerFor('traceQueryHandler', QueryHandler::class, $messagingConfiguration, $interfaceToCallRegistry); - $this->registerTracerFor('traceEventHandler', EventHandler::class, $messagingConfiguration, $interfaceToCallRegistry); - $this->registerTracerFor('traceMessageHandler', InternalHandler::class, $messagingConfiguration, $interfaceToCallRegistry); - $this->registerTracerFor('traceCommandBus', CommandBus::class, $messagingConfiguration, $interfaceToCallRegistry); - $this->registerTracerFor('traceQueryBus', QueryBus::class, $messagingConfiguration, $interfaceToCallRegistry); - $this->registerTracerFor('traceEventBus', EventBus::class, $messagingConfiguration, $interfaceToCallRegistry); - $this->registerTracerFor('traceAsynchronousEndpoint', AsynchronousRunningEndpoint::class, $messagingConfiguration, $interfaceToCallRegistry); - $this->registerTracerFor('traceDistributedBus', DistributedBus::class, $messagingConfiguration, $interfaceToCallRegistry); - - $messagingConfiguration->registerBeforeMethodInterceptor( - MethodInterceptorBuilder::create( - new Definition(TracerInterceptor::class, [ - Reference::to(TracerProviderInterface::class), - Reference::to(MessageBuilder::class), - ]), - $interfaceToCallRegistry->getFor(TracerInterceptor::class, 'provideContextForDistributedBus'), - 0, - DistributedBus::class - ) - ); + $messagingConfiguration->addCompilerPass(new TraceDynamicConsistencyBoundaryCompilerPass()); $pointcut = ''; - if ($tracingConfiguration->isFlushForcedOnBusExecution()) { + if ($openTelemetryConfiguration->forceFlushOnBusExecution) { $pointcut = CommandBus::class . '||' . QueryBus::class . '||' . DistributedBus::class; } - if ($tracingConfiguration->isForceFlushOnAsynchronousMessageHandled()) { + if ($openTelemetryConfiguration->forceFlushOnAsynchronousMessageHandled) { $pointcut .= $pointcut !== '' ? '||' . AsynchronousRunningEndpoint::class : AsynchronousRunningEndpoint::class; } @@ -139,17 +110,4 @@ public function getModulePackageName(): string { return ModulePackageList::TRACING_PACKAGE; } - - private function registerTracerFor(string $tracingMethodToInvoke, string $pointcut, Configuration $messagingConfiguration, InterfaceToCallRegistry $interfaceToCallRegistry): void - { - $messagingConfiguration - ->registerAroundMethodInterceptor( - AroundInterceptorBuilder::create( - TracerInterceptor::class, - $interfaceToCallRegistry->getFor(TracerInterceptor::class, $tracingMethodToInvoke), - Precedence::TRACING_PRECEDENCE, - $pointcut - ) - ); - } } diff --git a/packages/OpenTelemetry/src/Configuration/RegisterAddSpanEventLoggerCompilerPass.php b/packages/OpenTelemetry/src/Configuration/RegisterAddSpanEventLoggerCompilerPass.php deleted file mode 100644 index eac4e6188e..0000000000 --- a/packages/OpenTelemetry/src/Configuration/RegisterAddSpanEventLoggerCompilerPass.php +++ /dev/null @@ -1,28 +0,0 @@ -has(LoggingGateway::class)) { - $loggingGatewayDefinition = $builder->getDefinition(LoggingGateway::class); - - if (is_a($loggingGatewayDefinition->getClassName(), SubscribableLoggingGateway::class, true)) { - $loggingGatewayDefinition->addMethodCall('registerLogger', [new Definition(AddSpanEventLogger::class)]); - } - } - } -} diff --git a/packages/OpenTelemetry/src/Configuration/TracingConfiguration.php b/packages/OpenTelemetry/src/Configuration/TracingConfiguration.php deleted file mode 100644 index 139a02cb60..0000000000 --- a/packages/OpenTelemetry/src/Configuration/TracingConfiguration.php +++ /dev/null @@ -1,62 +0,0 @@ -tracingLevel >= $tracingLevel; - } - - public function withForceFlushOnBusExecution(bool $forceFlush): self - { - $self = clone $this; - $self->forceFlushOnBusExecution = $forceFlush; - - return $self; - } - - public function withForceFlushOnAsynchronousMessageHandled(bool $forceFlushOnAsynchronousMessageHandled): self - { - $self = clone $this; - $self->forceFlushOnAsynchronousMessageHandled = $forceFlushOnAsynchronousMessageHandled; - - return $self; - } - - public function isFlushForcedOnBusExecution(): bool - { - return $this->forceFlushOnBusExecution; - } - - public function isForceFlushOnAsynchronousMessageHandled(): bool - { - return $this->forceFlushOnAsynchronousMessageHandled; - } -} diff --git a/packages/OpenTelemetry/src/OpenTelemetryActiveSpan.php b/packages/OpenTelemetry/src/OpenTelemetryActiveSpan.php new file mode 100644 index 0000000000..03d8a0ce9f --- /dev/null +++ b/packages/OpenTelemetry/src/OpenTelemetryActiveSpan.php @@ -0,0 +1,141 @@ + $attributes + */ + public function __construct( + private SpanInterface $span, + private ScopeInterface $scope, + private SpanContext $context, + private ?string $parentSpanId, + private string $name, + private SpanKind $kind, + private array $attributes, + private DatePoint $startedAt, + private EcotoneClockInterface $clock, + private OpenTelemetryTracer $tracer, + ) { + } + + public function attributes(): array + { + return $this->attributes; + } + + public function context(): SpanContext + { + return $this->context; + } + + public function addEvent(string $name, array $attributes = []): void + { + $this->span->addEvent($name, $attributes); + $this->events[] = new SpanEvent($name, $attributes, $this->clock->now()); + } + + public function end(): void + { + if ($this->unitOfWorkFailed) { + $this->finish(StatusCode::STATUS_ERROR, SpanStatus::Error, EcotoneActiveSpan::UNIT_OF_WORK_FAILED); + + return; + } + $this->finish(StatusCode::STATUS_OK, SpanStatus::Ok, null); + } + + public function endedSoFar(): Span + { + return new Span( + $this->context->traceId, + $this->context->spanId, + $this->parentSpanId, + $this->name, + $this->kind, + $this->attributes, + SpanStatus::Ok, + null, + $this->events, + $this->startedAt, + $this->clock->now(), + ); + } + + public function markUnitOfWorkFailed(): void + { + $this->unitOfWorkFailed = true; + } + + public function markUnitOfWorkRecovered(): void + { + $this->unitOfWorkFailed = false; + } + + public function fail(Throwable $failure): void + { + if ($this->ended) { + return; + } + + $this->span->recordException($failure); + $this->events[] = new SpanEvent('exception', [ + 'exception.type' => $failure::class, + 'exception.message' => $failure->getMessage(), + 'exception.stacktrace' => $failure->getTraceAsString(), + ], $this->clock->now()); + $this->finish(StatusCode::STATUS_ERROR, SpanStatus::Error, $failure->getMessage()); + } + + private function finish(string $statusCode, SpanStatus $status, ?string $statusDescription): void + { + if ($this->ended) { + return; + } + $this->ended = true; + + $this->span->setStatus($statusCode, $statusDescription); + $this->scope->detach(); + $this->span->end(); + + $this->tracer->endSpan($this, new Span( + $this->context->traceId, + $this->context->spanId, + $this->parentSpanId, + $this->name, + $this->kind, + $this->attributes, + $status, + $statusDescription, + $this->events, + $this->startedAt, + $this->clock->now(), + )); + } +} diff --git a/packages/OpenTelemetry/src/OpenTelemetryTracer.php b/packages/OpenTelemetry/src/OpenTelemetryTracer.php new file mode 100644 index 0000000000..06f6839cd5 --- /dev/null +++ b/packages/OpenTelemetry/src/OpenTelemetryTracer.php @@ -0,0 +1,121 @@ +spanBuilderFor($name, $kind, $attributes); + if ($remoteParent !== null) { + $spanBuilder->setParent(TraceContextPropagator::getInstance()->extract([MessageHeaders::TRACEPARENT => $remoteParent->toTraceparent(), MessageHeaders::TRACESTATE => $remoteParent->traceState])); + } + + return $this->start($spanBuilder, $name, $kind, $attributes, $remoteParent ?? $this->currentSpan()?->context()); + } + + public function startRootSpan(string $name, SpanKind $kind, array $attributes = []): ActiveSpan + { + return $this->start($this->spanBuilderFor($name, $kind, $attributes)->setParent(false), $name, $kind, $attributes, null); + } + + /** + * @param array $attributes + */ + private function spanBuilderFor(string $name, SpanKind $kind, array $attributes): SpanBuilderInterface + { + return $this->tracerProvider + ->getTracer(EcotoneSpanBuilder::ECOTONE_TRACER_NAME) + ->spanBuilder($name) + ->setSpanKind(self::kindOf($kind)) + ->setAttributes($attributes); + } + + /** + * @param array $attributes + */ + private function start(SpanBuilderInterface $spanBuilder, string $name, SpanKind $kind, array $attributes, ?SpanContext $parentContext): ActiveSpan + { + $parentSpanId = $parentContext?->spanId; + $span = $spanBuilder->startSpan(); + $scope = $span->activate(); + $context = $span->getContext(); + $spanContext = $context->isValid() && $span->isRecording() + ? new SpanContext($context->getTraceId(), $context->getSpanId(), $context->isSampled(), (string) $context->getTraceState()) + : ($parentContext?->newChild() ?? SpanContext::newTrace()); + + $activeSpan = new OpenTelemetryActiveSpan( + $span, + $scope, + $spanContext, + $parentSpanId, + $name, + $kind, + $attributes, + $this->clock->now(), + $this->clock, + $this, + ); + $this->activeSpans[] = $activeSpan; + + return $activeSpan; + } + + public function currentSpan(): ?ActiveSpan + { + return $this->activeSpans[count($this->activeSpans) - 1] ?? null; + } + + public function endSpan(OpenTelemetryActiveSpan $activeSpan, Span $span): void + { + $position = array_search($activeSpan, $this->activeSpans, true); + if ($position !== false) { + array_splice($this->activeSpans, $position, 1); + } + + $this->spanRecorder->record($span); + } + + private static function kindOf(SpanKind $kind): int + { + return match ($kind) { + SpanKind::Internal => OpenTelemetrySpanKind::KIND_INTERNAL, + SpanKind::Server => OpenTelemetrySpanKind::KIND_SERVER, + SpanKind::Client => OpenTelemetrySpanKind::KIND_CLIENT, + SpanKind::Producer => OpenTelemetrySpanKind::KIND_PRODUCER, + SpanKind::Consumer => OpenTelemetrySpanKind::KIND_CONSUMER, + }; + } +} diff --git a/packages/OpenTelemetry/src/TracerInterceptor.php b/packages/OpenTelemetry/src/TracerInterceptor.php deleted file mode 100644 index 466111a3ea..0000000000 --- a/packages/OpenTelemetry/src/TracerInterceptor.php +++ /dev/null @@ -1,190 +0,0 @@ -getHeaders()->containsKey(TracingChannelInterceptor::TRACING_CARRIER_HEADER) ? json_decode($message->getHeaders()->get(TracingChannelInterceptor::TRACING_CARRIER_HEADER), true) : []; - $parentContext = TraceContextPropagator::getInstance()->extract($carrier); - - $scope = $parentContext->activate(); - try { - return $this->trace( - $message->getHeaders()->containsKey(MessageHeaders::POLLED_CHANNEL_NAME) - ? 'Receiving from channel: ' . $message->getHeaders()->get(MessageHeaders::POLLED_CHANNEL_NAME) - : ($message->getHeaders()->containsKey(MessageHeaders::INBOUND_REQUEST_CHANNEL) - ? 'Receiving from inbound channel adapter: ' . $message->getHeaders()->get(MessageHeaders::INBOUND_REQUEST_CHANNEL) - : 'Endpoint: ' . $message->getHeaders()->get(MessageHeaders::CONSUMER_POLLING_METADATA)->getEndpointId() . ' produced Message'), - $methodInvocation, - $message, - spanKind: SpanKind::KIND_CONSUMER, - ); - } finally { - $scope->detach(); - } - } - - public function provideContextForDistributedBus(Message $message): Message - { - $ctx = Span::getCurrent()->storeInContext(Context::getCurrent()); - $carrier = []; - TraceContextPropagator::getInstance()->inject($carrier, null, $ctx); - - return $this->messageBuilder->fromMessage($message) - ->setHeader(TracingChannelInterceptor::TRACING_CARRIER_HEADER, json_encode($carrier)) - ->build(); - } - - public function traceDistributedBus(MethodInvocation $methodInvocation, Message $message): mixed - { - $span = EcotoneSpanBuilder::create($message, 'Distributed Bus', $this->tracerProvider, SpanKind::KIND_PRODUCER) - ->startSpan(); - $spanScope = $span->activate(); - - try { - $result = $methodInvocation->proceed(); - } catch (Throwable $exception) { - $span->recordException($exception); - $this->closeSpan($span, $spanScope, StatusCode::STATUS_ERROR, $exception->getMessage()); - - throw $exception; - } - - $this->closeSpan($span, $spanScope, StatusCode::STATUS_OK, null); - - return $result; - } - - public function traceCommandHandler(MethodInvocation $methodInvocation, Message $message) - { - return $this->trace( - 'Command Handler: ' . $methodInvocation->getName(), - $methodInvocation, - $message, - ); - } - - public function traceQueryHandler(MethodInvocation $methodInvocation, Message $message) - { - return $this->trace( - 'Query Handler: ' . $methodInvocation->getName(), - $methodInvocation, - $message, - ); - } - - public function traceEventHandler(MethodInvocation $methodInvocation, Message $message) - { - return $this->trace( - 'Event Handler: ' . $methodInvocation->getName(), - $methodInvocation, - $message, - ); - } - - public function traceMessageHandler(MethodInvocation $methodInvocation, Message $message) - { - return $this->trace( - 'Message Handler: ' . $methodInvocation->getName(), - $methodInvocation, - $message, - ); - } - - public function traceCommandBus(MethodInvocation $methodInvocation, Message $message) - { - return $this->trace( - 'Command Bus', - $methodInvocation, - $message, - ); - } - - public function traceEventBus(MethodInvocation $methodInvocation, Message $message) - { - return $this->trace( - 'Event Bus', - $methodInvocation, - $message, - ); - } - - public function traceQueryBus(MethodInvocation $methodInvocation, Message $message) - { - return $this->trace( - 'Query Bus', - $methodInvocation, - $message, - ); - } - - public function trace( - string $type, - MethodInvocation $methodInvocation, - Message $message, - array $attributes = [], - int $spanKind = SpanKind::KIND_SERVER, - ) { - $span = EcotoneSpanBuilder::create( - $message, - $type, - $this->tracerProvider, - $spanKind - ) - ->startSpan(); - $spanScope = $span->activate(); - - try { - $result = $methodInvocation->proceed(); - } catch (Throwable $exception) { - $span->recordException($exception); - $this->closeSpan($span, $spanScope, StatusCode::STATUS_ERROR, $exception->getMessage()); - - throw $exception; - } - - $this->closeSpan($span, $spanScope, StatusCode::STATUS_OK, null); - - return $result; - } - - private function closeSpan(SpanInterface $span, ScopeInterface $spanScope, string $statusCode, ?string $descriptionStatusCode): void - { - $span->setStatus($statusCode, $descriptionStatusCode); - $spanScope->detach(); - $span->end(); - } -} diff --git a/packages/OpenTelemetry/src/TracingChannelInterceptor.php b/packages/OpenTelemetry/src/TracingChannelInterceptor.php deleted file mode 100644 index d6c651032b..0000000000 --- a/packages/OpenTelemetry/src/TracingChannelInterceptor.php +++ /dev/null @@ -1,98 +0,0 @@ -channelName, $this->tracerProvider, SpanKind::KIND_PRODUCER) - ->startSpan(); - - $scope = $span->activate(); - $this->openSpans[] = new ChannelSendSpan($span, $scope); - - $ctx = $span->storeInContext(Context::getCurrent()); - $carrier = []; - TraceContextPropagator::getInstance()->inject($carrier, null, $ctx); - - return $this->messageBuilder->fromMessage($message) - ->setHeader(self::TRACING_CARRIER_HEADER, json_encode($carrier)) - ->build(); - } - - public function postSend(Message $message, MessageChannel $messageChannel): void - { - } - - public function afterSendCompletion(Message $message, MessageChannel $messageChannel, ?Throwable $exception): bool - { - $channelSendSpan = array_pop($this->openSpans); - - if ($exception !== null) { - $channelSendSpan?->closeWithFailure($exception); - } else { - $channelSendSpan?->closeWithSuccess(); - } - - return false; - } - - public function preReceive(MessageChannel $messageChannel): bool - { - return true; - } - - public function afterReceiveCompletion(?Message $message, MessageChannel $messageChannel, ?Throwable $exception): void - { - if ($exception !== null && $message !== null) { - // @TODO test - $carrier = $message->getHeaders()->containsKey(self::TRACING_CARRIER_HEADER) ? json_decode($message->getHeaders()->get(self::TRACING_CARRIER_HEADER), true) : []; - $context = TraceContextPropagator::getInstance()->extract($carrier); - - $span = EcotoneSpanBuilder::create($message, 'Asynchronous Channel: ' . $this->channelName, $this->tracerProvider, SpanKind::KIND_CONSUMER) - ->setParent($context) - ->startSpan(); - - $span->setStatus(StatusCode::STATUS_ERROR); - $span->end(); - } - } - - public function postReceive(Message $message, MessageChannel $messageChannel): ?Message - { - return $message; - } -} diff --git a/packages/OpenTelemetry/tests/Fixture/CommandEventFlow/MerchantSubscriberOne.php b/packages/OpenTelemetry/tests/Fixture/CommandEventFlow/MerchantSubscriberOne.php index 7ee9ac7cf8..b1e51eaa5f 100644 --- a/packages/OpenTelemetry/tests/Fixture/CommandEventFlow/MerchantSubscriberOne.php +++ b/packages/OpenTelemetry/tests/Fixture/CommandEventFlow/MerchantSubscriberOne.php @@ -12,7 +12,7 @@ */ final class MerchantSubscriberOne { - #[EventHandler] + #[EventHandler(endpointId: 'merchant.subscriber.one')] public function merchantToUser(MerchantCreated $event, CommandBus $commandBus): void { $commandBus->send(new RegisterUser($event->merchantId)); diff --git a/packages/OpenTelemetry/tests/Fixture/CommandEventFlow/MerchantSubscriberTwo.php b/packages/OpenTelemetry/tests/Fixture/CommandEventFlow/MerchantSubscriberTwo.php index 7c4bc2d13c..a1221a68ca 100644 --- a/packages/OpenTelemetry/tests/Fixture/CommandEventFlow/MerchantSubscriberTwo.php +++ b/packages/OpenTelemetry/tests/Fixture/CommandEventFlow/MerchantSubscriberTwo.php @@ -12,7 +12,7 @@ */ final class MerchantSubscriberTwo { - #[EventHandler] + #[EventHandler(endpointId: 'merchant.subscriber.two')] public function merchantToUser(MerchantCreated $event, CommandBus $commandBus): void { $commandBus->send(new RegisterUser($event->merchantId . '-second')); diff --git a/packages/OpenTelemetry/tests/Integration/CorrelatedHeadersPropagationTest.php b/packages/OpenTelemetry/tests/Integration/CorrelatedHeadersPropagationTest.php index 51e93107ae..2ab4ccc64c 100644 --- a/packages/OpenTelemetry/tests/Integration/CorrelatedHeadersPropagationTest.php +++ b/packages/OpenTelemetry/tests/Integration/CorrelatedHeadersPropagationTest.php @@ -56,8 +56,8 @@ public function test_tracing_with_single_levels_of_nesting() /** Command Bus and Command Handler */ $this->assertCount(2, $spans); - $this->assertSpanWith('Command Handler: ' . User::class . '::register', $spans[0], $messageId, $correlationId, $timestamp); - $this->assertSpanWith('Command Bus', $spans[1], $messageId, $correlationId, $timestamp); + $this->assertSpanWith('process User::register', $spans[0], $messageId, $correlationId, $timestamp); + $this->assertSpanWith('send command', $spans[1], $messageId, $correlationId, $timestamp); } public function test_tracing_with_two_levels_of_nesting() @@ -81,14 +81,14 @@ public function test_tracing_with_two_levels_of_nesting() /** @var ImmutableSpan[] $spans */ $spans = $exporter->getSpans(); - /** Command Bus and Command Handler */ - $this->assertCount(4, $spans); + $this->assertCount(5, $spans); - $this->assertChildSpan('Command Handler: ' . User::class . '::register', $spans[0], $messageId, $correlationId); - $this->assertChildSpan('Command Bus', $spans[1], $messageId, $correlationId); + $this->assertChildSpan('process User::register', $spans[0], $messageId, $correlationId); + $this->assertChildSpan('send command', $spans[1], $messageId, $correlationId); - $this->assertSpanWith('Event Handler: ' . MerchantSubscriberOne::class . '::merchantToUser', $spans[2], $messageId, $correlationId, $timestamp); - $this->assertSpanWith('Event Bus', $spans[3], $messageId, $correlationId, $timestamp); + $this->assertSpanWith('process MerchantSubscriberOne::merchantToUser', $spans[2], $messageId, $correlationId, $timestamp); + $this->assertSpanWith('route merchant.subscriber.one', $spans[3], $messageId, $correlationId, $timestamp); + $this->assertSpanWith('send event', $spans[4], $messageId, $correlationId, $timestamp); } public function test_tracing_with_three_levels_of_nesting() @@ -112,17 +112,17 @@ public function test_tracing_with_three_levels_of_nesting() /** @var ImmutableSpan[] $spans */ $spans = $exporter->getSpans(); - /** Command Bus and Command Handler */ - $this->assertCount(6, $spans); + $this->assertCount(7, $spans); - $this->assertChildSpan('Command Handler: ' . User::class . '::register', $spans[0], $spans[2]->getAttributes()->get(MessageHeaders::MESSAGE_ID), $correlationId); - $this->assertChildSpan('Command Bus', $spans[1], $spans[2]->getAttributes()->get(MessageHeaders::MESSAGE_ID), $correlationId); + $this->assertChildSpan('process User::register', $spans[0], $spans[2]->getAttributes()->get(MessageHeaders::MESSAGE_ID), $correlationId); + $this->assertChildSpan('send command', $spans[1], $spans[2]->getAttributes()->get(MessageHeaders::MESSAGE_ID), $correlationId); - $this->assertChildSpan('Event Handler: ' . MerchantSubscriberOne::class . '::merchantToUser', $spans[2], $messageId, $correlationId); - $this->assertChildSpan('Event Bus', $spans[3], $messageId, $correlationId); + $this->assertChildSpan('process MerchantSubscriberOne::merchantToUser', $spans[2], $messageId, $correlationId); + $this->assertChildSpan('route merchant.subscriber.one', $spans[3], $messageId, $correlationId); + $this->assertChildSpan('send event', $spans[4], $messageId, $correlationId); - $this->assertSpanWith('Command Handler: ' . Merchant::class . '::create', $spans[4], $messageId, $correlationId, $timestamp); - $this->assertSpanWith('Command Bus', $spans[5], $messageId, $correlationId, $timestamp); + $this->assertSpanWith('process Merchant::create', $spans[5], $messageId, $correlationId, $timestamp); + $this->assertSpanWith('send command', $spans[6], $messageId, $correlationId, $timestamp); } public function test_tracing_same_message_for_asynchronous_scenario() @@ -148,11 +148,11 @@ public function test_tracing_same_message_for_asynchronous_scenario() $node = $this->getNodeAtTargetedSpan( [ - 'details' => ['name' => 'Command Bus'], + 'details' => ['name' => 'send command'], 'child' => [ - 'details' => ['name' => 'Sending to Channel: async_channel'], + 'details' => ['name' => 'send async_channel'], 'child' => [ - 'details' => ['name' => 'Receiving from channel: async_channel'], + 'details' => ['name' => 'receive async_channel'], ], ], ], @@ -206,13 +206,13 @@ public function test_tracing_with_exception() /** Command Handler */ $this->assertSame(StatusCode::STATUS_ERROR, $spans[0]->getStatus()->getCode()); - $event = $spans[0]->getEvents()[1]; + $event = current(array_filter($spans[0]->getEvents(), fn ($event): bool => $event->getName() === 'exception')); $this->assertSame('exception', $event->getName()); $this->assertSame('User already registered', $event->getAttributes()->get('exception.message')); /** Command Bus */ $this->assertSame(StatusCode::STATUS_ERROR, $spans[1]->getStatus()->getCode()); - $event = $spans[1]->getEvents()[1]; + $event = current(array_filter($spans[1]->getEvents(), fn ($event): bool => $event->getName() === 'exception')); $this->assertSame('exception', $event->getName()); $this->assertSame('User already registered', $event->getAttributes()->get('exception.message')); } diff --git a/packages/OpenTelemetry/tests/Integration/ForcedTraceFlushTest.php b/packages/OpenTelemetry/tests/Integration/ForcedTraceFlushTest.php index 637aef79ac..4b75206437 100644 --- a/packages/OpenTelemetry/tests/Integration/ForcedTraceFlushTest.php +++ b/packages/OpenTelemetry/tests/Integration/ForcedTraceFlushTest.php @@ -11,7 +11,7 @@ use Ecotone\Api\ExtensionObject\ModulePackageList; use Ecotone\Api\ExtensionObject\SimpleMessageChannelBuilder; use Ecotone\Api\Lite\EcotoneLite; -use Ecotone\OpenTelemetry\Configuration\TracingConfiguration; +use Ecotone\Api\OpenTelemetry\OpenTelemetryConfiguration; use OpenTelemetry\API\Trace\NoopTracerProvider; use OpenTelemetry\API\Trace\TracerProviderInterface; use OpenTelemetry\SDK\Trace\SpanExporter\InMemoryExporter; @@ -84,7 +84,7 @@ public function place(string $orderId): void EcotoneLite::bootstrapFlowTesting( [$handler::class], [$handler, TracerProviderInterface::class => new NoopTracerProvider()], - extensionObjects: [TracingConfiguration::createWithDefaults()->withForceFlushOnBusExecution(false)->withForceFlushOnAsynchronousMessageHandled(false)], + extensionObjects: [OpenTelemetryConfiguration::createWithDefaults()->withForceFlushOnBusExecution(false)->withForceFlushOnAsynchronousMessageHandled(false)], enabledModules: [ModulePackageList::TRACING_PACKAGE], )->sendCommandWithRouting('order.place', 'order-1'); diff --git a/packages/OpenTelemetry/tests/Integration/SwallowedFailureTracingTest.php b/packages/OpenTelemetry/tests/Integration/SwallowedFailureTracingTest.php new file mode 100644 index 0000000000..b3d5a79858 --- /dev/null +++ b/packages/OpenTelemetry/tests/Integration/SwallowedFailureTracingTest.php @@ -0,0 +1,54 @@ + self::prepareTracer($exporter)], + [SimpleMessageChannelBuilder::createQueueChannel('async_channel'), ServiceConfiguration::createWithDefaults()->withDefaultErrorChannel('errorChannel')], + enabledModules: [ModulePackageList::TRACING_PACKAGE], + ) + ->sendCommandWithRouting('order.place', '1') + ->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup(stopOnError: false)); + + $statuses = []; + foreach ($exporter->getSpans() as $span) { + $statuses[$span->getName()] = $span->getStatus()->getCode(); + } + $this->assertSame(StatusCode::STATUS_ERROR, $statuses['receive async_channel']); + $this->assertSame(StatusCode::STATUS_ERROR, $statuses['process ' . substr($orders::class, 0, strpos($orders::class, "\0")) . '::place']); + } +} diff --git a/packages/OpenTelemetry/tests/Integration/TracingDisabledWithoutProviderTest.php b/packages/OpenTelemetry/tests/Integration/TracingDisabledWithoutProviderTest.php new file mode 100644 index 0000000000..5ad111ff5c --- /dev/null +++ b/packages/OpenTelemetry/tests/Integration/TracingDisabledWithoutProviderTest.php @@ -0,0 +1,48 @@ +headers = $headers; + } + }; + + EcotoneLite::bootstrapFlowTesting( + [$users::class], + [$users], + [SimpleMessageChannelBuilder::createQueueChannel('async_channel')], + enabledModules: [ModulePackageList::TRACING_PACKAGE], + ) + ->sendCommandWithRouting('user.register', '1') + ->run('async_channel', ExecutionPollingMetadata::createWithTestingSetup()); + + $this->assertNotSame([], $users->headers); + $this->assertArrayNotHasKey(MessageHeaders::TRACEPARENT, $users->headers); + } +} diff --git a/packages/OpenTelemetry/tests/Integration/TracingTestCase.php b/packages/OpenTelemetry/tests/Integration/TracingTestCase.php index da69f9f410..2c38cf41fb 100644 --- a/packages/OpenTelemetry/tests/Integration/TracingTestCase.php +++ b/packages/OpenTelemetry/tests/Integration/TracingTestCase.php @@ -28,7 +28,7 @@ public static function prepareTracer(SpanExporterInterface $exporter): TracerPro ); } - private const INTERNAL_NOISE_SPAN_NAME_PREFIX = 'Message Handler: Ecotone\Modelling\MessageHandling\MetadataPropagator\MessageHeadersPropagatorInterceptor::'; + private const INTERNAL_NOISE_SPAN_NAME_PREFIX = 'process MessageHeadersPropagatorInterceptor::'; public static function buildTree(InMemoryExporter $exporter): array { diff --git a/packages/OpenTelemetry/tests/Integration/TracingTreeTest.php b/packages/OpenTelemetry/tests/Integration/TracingTreeTest.php index 018dc3529e..867ef2c20b 100644 --- a/packages/OpenTelemetry/tests/Integration/TracingTreeTest.php +++ b/packages/OpenTelemetry/tests/Integration/TracingTreeTest.php @@ -9,7 +9,7 @@ use Ecotone\Api\ExtensionObject\ModulePackageList; use Ecotone\Api\ExtensionObject\SimpleMessageChannelBuilder; use Ecotone\Api\Lite\EcotoneLite; -use Ecotone\OpenTelemetry\Configuration\TracingConfiguration; +use Ecotone\Api\OpenTelemetry\OpenTelemetryConfiguration; use Ecotone\OpenTelemetry\Support\OTelTracer; use InvalidArgumentException; use Monolog\Handler\StreamHandler; @@ -57,10 +57,10 @@ public function test_tracing_tree_with_single_levels_of_nesting() self::compareTreesByDetails( [ [ - 'details' => ['name' => 'Command Bus'], + 'details' => ['name' => 'send command'], 'children' => [ [ - 'details' => ['name' => 'Command Handler: ' . User::class . '::register'], + 'details' => ['name' => 'process User::register'], 'children' => [], ], ], @@ -84,14 +84,14 @@ public function test_tracing_tree_with_two_levels_of_nesting_and_message_handler self::compareTreesByDetails( [ [ - 'details' => ['name' => 'Command Bus'], + 'details' => ['name' => 'send command'], 'children' => [ [ - 'details' => ['name' => 'Command Handler: ' . ExampleMessageHandler::class . '::handleCommand'], + 'details' => ['name' => 'process ExampleMessageHandler::handleCommand'], 'children' => [], ], [ - 'details' => ['name' => 'Message Handler: ' . ExampleMessageHandler::class . '::handle'], + 'details' => ['name' => 'process ExampleMessageHandler::handle'], 'children' => [], ], ], @@ -115,11 +115,11 @@ public function test_tracing_scheduled_handler() self::compareTreesByDetails( [ [ - 'details' => ['name' => 'Receiving from inbound channel adapter: nullChannel'], + 'details' => ['name' => 'receive nullChannel'], 'children' => [], ], [ - 'details' => ['name' => 'Receiving from inbound channel adapter: nullChannel'], + 'details' => ['name' => 'receive nullChannel'], 'children' => [], ], ], @@ -141,19 +141,19 @@ public function test_tracing_workflow_scheduled_handler() self::compareTreesByDetails( [ [ - 'details' => ['name' => 'Receiving from inbound channel adapter: add'], + 'details' => ['name' => 'receive add'], 'children' => [ [ - 'details' => ['name' => 'Message Handler: ' . WorkflowScheduledHandler::class . '::add'], + 'details' => ['name' => 'process WorkflowScheduledHandler::add'], 'children' => [], ], ], ], [ - 'details' => ['name' => 'Receiving from inbound channel adapter: add'], + 'details' => ['name' => 'receive add'], 'children' => [ [ - 'details' => ['name' => 'Message Handler: ' . WorkflowScheduledHandler::class . '::add'], + 'details' => ['name' => 'process WorkflowScheduledHandler::add'], 'children' => [], ], ], @@ -192,7 +192,7 @@ public function test_disabling_force_flushing_traces() ClockFactory::getDefault() ) )], - extensionObjects: [TracingConfiguration::createWithDefaults()->withForceFlushOnBusExecution(false)->withForceFlushOnAsynchronousMessageHandled(false)], + extensionObjects: [OpenTelemetryConfiguration::createWithDefaults()->withForceFlushOnBusExecution(false)->withForceFlushOnAsynchronousMessageHandled(false)], enabledModules: [ModulePackageList::TRACING_PACKAGE], ) ->sendCommand(new RegisterUser('1')); @@ -215,7 +215,7 @@ public function test_force_flushing_traces() ClockFactory::getDefault() ) )], - extensionObjects: [TracingConfiguration::createWithDefaults()->withForceFlushOnBusExecution(true)->withForceFlushOnAsynchronousMessageHandled(true)], + extensionObjects: [OpenTelemetryConfiguration::createWithDefaults()->withForceFlushOnBusExecution(true)->withForceFlushOnAsynchronousMessageHandled(true)], enabledModules: [ModulePackageList::TRACING_PACKAGE], ) ->sendCommand(new RegisterUser('1')); @@ -223,10 +223,10 @@ public function test_force_flushing_traces() self::compareTreesByDetails( [ [ - 'details' => ['name' => 'Command Bus'], + 'details' => ['name' => 'send command'], 'children' => [ [ - 'details' => ['name' => 'Command Handler: ' . User::class . '::register'], + 'details' => ['name' => 'process User::register'], 'children' => [], ], ], @@ -248,7 +248,7 @@ public function test_traces_are_force_flushed_when_synchronous_exception_happens ClockFactory::getDefault() ) )], - extensionObjects: [TracingConfiguration::createWithDefaults()->withForceFlushOnBusExecution(true)->withForceFlushOnAsynchronousMessageHandled(true)], + extensionObjects: [OpenTelemetryConfiguration::createWithDefaults()->withForceFlushOnBusExecution(true)->withForceFlushOnAsynchronousMessageHandled(true)], enabledModules: [ModulePackageList::TRACING_PACKAGE], ); @@ -267,18 +267,15 @@ public function test_traces_are_force_flushed_when_synchronous_exception_happens $node = $this->getNodeAtTargetedSpan( [ - 'details' => ['name' => 'Command Bus'], + 'details' => ['name' => 'send command'], 'child' => [ - 'details' => ['name' => 'Command Handler: ' . User::class . '::register'], + 'details' => ['name' => 'process User::register'], ], ], self::buildTree($exporter) ); - $this->assertSame( - 'exception', - $node['details']['events'][1]->getName() - ); + $this->assertContains('exception', array_map(fn ($event): string => $event->getName(), $node['details']['events'])); } public function test_separate_traces() @@ -296,19 +293,19 @@ public function test_separate_traces() self::compareTreesByDetails( [ [ - 'details' => ['name' => 'Command Bus', 'attributes' => ['flowId' => '1']], + 'details' => ['name' => 'send command', 'attributes' => ['flowId' => '1']], 'children' => [ [ - 'details' => ['name' => 'Command Handler: ' . User::class . '::register', 'attributes' => ['flowId' => '1']], + 'details' => ['name' => 'process User::register', 'attributes' => ['flowId' => '1']], 'children' => [], ], ], ], [ - 'details' => ['name' => 'Command Bus', 'attributes' => ['flowId' => '2']], + 'details' => ['name' => 'send command', 'attributes' => ['flowId' => '2']], 'children' => [ [ - 'details' => ['name' => 'Command Handler: ' . User::class . '::register', 'attributes' => ['flowId' => '2']], + 'details' => ['name' => 'process User::register', 'attributes' => ['flowId' => '2']], 'children' => [], ], ], @@ -332,17 +329,22 @@ public function test_tracing_with_two_levels_of_nesting() self::compareTreesByDetails( [ [ - 'details' => ['name' => 'Event Bus'], + 'details' => ['name' => 'send event'], 'children' => [ [ - 'details' => ['name' => 'Event Handler: ' . MerchantSubscriberOne::class . '::merchantToUser'], + 'details' => ['name' => 'route merchant.subscriber.one'], 'children' => [ [ - 'details' => ['name' => 'Command Bus'], + 'details' => ['name' => 'process MerchantSubscriberOne::merchantToUser'], 'children' => [ [ - 'details' => ['name' => 'Command Handler: ' . User::class . '::register'], - 'children' => [], + 'details' => ['name' => 'send command'], + 'children' => [ + [ + 'details' => ['name' => 'process User::register'], + 'children' => [], + ], + ], ], ], ], @@ -369,31 +371,41 @@ public function test_tracing_with_two_levels_of_nesting_and_two_branches_on_same self::compareTreesByDetails( [ [ - 'details' => ['name' => 'Event Bus'], + 'details' => ['name' => 'send event'], 'children' => [ [ - 'details' => ['name' => 'Event Handler: ' . MerchantSubscriberOne::class . '::merchantToUser'], + 'details' => ['name' => 'route merchant.subscriber.one'], 'children' => [ [ - 'details' => ['name' => 'Command Bus'], + 'details' => ['name' => 'process MerchantSubscriberOne::merchantToUser'], 'children' => [ [ - 'details' => ['name' => 'Command Handler: ' . User::class . '::register'], - 'children' => [], + 'details' => ['name' => 'send command'], + 'children' => [ + [ + 'details' => ['name' => 'process User::register'], + 'children' => [], + ], + ], ], ], ], ], ], [ - 'details' => ['name' => 'Event Handler: ' . MerchantSubscriberTwo::class . '::merchantToUser'], + 'details' => ['name' => 'route merchant.subscriber.two'], 'children' => [ [ - 'details' => ['name' => 'Command Bus'], + 'details' => ['name' => 'process MerchantSubscriberTwo::merchantToUser'], 'children' => [ [ - 'details' => ['name' => 'Command Handler: ' . User::class . '::register'], - 'children' => [], + 'details' => ['name' => 'send command'], + 'children' => [ + [ + 'details' => ['name' => 'process User::register'], + 'children' => [], + ], + ], ], ], ], @@ -420,23 +432,28 @@ public function test_tracing_with_three_levels_of_nesting() self::compareTreesByDetails( [ [ - 'details' => ['name' => 'Command Bus'], + 'details' => ['name' => 'send command'], 'children' => [ [ - 'details' => ['name' => 'Command Handler: ' . Merchant::class . '::create'], + 'details' => ['name' => 'process Merchant::create'], 'children' => [ [ - 'details' => ['name' => 'Event Bus'], + 'details' => ['name' => 'send event'], 'children' => [ [ - 'details' => ['name' => 'Event Handler: ' . MerchantSubscriberOne::class . '::merchantToUser'], + 'details' => ['name' => 'route merchant.subscriber.one'], 'children' => [ [ - 'details' => ['name' => 'Command Bus'], + 'details' => ['name' => 'process MerchantSubscriberOne::merchantToUser'], 'children' => [ [ - 'details' => ['name' => 'Command Handler: ' . User::class . '::register'], - 'children' => [], + 'details' => ['name' => 'send command'], + 'children' => [ + [ + 'details' => ['name' => 'process User::register'], + 'children' => [], + ], + ], ], ], ], @@ -470,16 +487,16 @@ public function test_tracing_with_asynchronous_handler() self::compareTreesByDetails( [ [ - 'details' => ['name' => 'Command Bus'], + 'details' => ['name' => 'send command'], 'children' => [ [ - 'details' => ['name' => 'Sending to Channel: async_channel'], + 'details' => ['name' => 'send async_channel'], 'children' => [ [ - 'details' => ['name' => 'Receiving from channel: async_channel'], + 'details' => ['name' => 'receive async_channel'], 'children' => [ [ - 'details' => ['name' => 'Command Handler: ' . \Test\Ecotone\OpenTelemetry\Fixture\AsynchronousFlow\User::class . '::register'], + 'details' => ['name' => 'process User::register'], 'children' => [], ], ], @@ -514,13 +531,13 @@ public function test_adding_logs_as_events_in_span() $result = self::getNodeAtTargetedSpan( [ - 'details' => ['name' => 'Command Bus'], + 'details' => ['name' => 'send command'], 'child' => [ - 'details' => ['name' => 'Sending to Channel: async_channel'], + 'details' => ['name' => 'send async_channel'], 'child' => [ - 'details' => ['name' => 'Receiving from channel: async_channel'], + 'details' => ['name' => 'receive async_channel'], 'child' => [ - 'details' => ['name' => 'Event Bus'], + 'details' => ['name' => 'send event'], ], ], ], @@ -528,17 +545,13 @@ public function test_adding_logs_as_events_in_span() self::buildTree($exporter) ); - /** @var Event $event */ - $event = $result['details']['events'][0]; - $this->stringStartsWith( - 'Publishing Event Message', - )->evaluate($event->getName()); - - /** @var Event $event */ - $event = $result['details']['events'][2]; - $this->stringStartsWith( - 'Collecting message with id', - )->evaluate($event->getName()); + $eventBusEvents = array_map(fn (Event $event): string => $event->getName(), $result['details']['events']); + $this->assertNotEmpty(array_filter($eventBusEvents, fn (string $name): bool => str_starts_with($name, 'Publishing Event Message'))); + + $route = $result['children'][array_key_first($result['children'])]; + $this->assertSame('route user.registered', $route['details']['name']); + $routeEvents = array_map(fn (Event $event): string => $event->getName(), $route['details']['events']); + $this->assertNotEmpty(array_filter($routeEvents, fn (string $name): bool => str_starts_with($name, 'Collecting message with id'))); } public function test_two_traces_with_asynchronous_handlers() @@ -560,16 +573,16 @@ public function test_two_traces_with_asynchronous_handlers() self::compareTreesByDetails( [ [ - 'details' => ['name' => 'Command Bus', 'attributes' => ['flowId' => '1']], + 'details' => ['name' => 'send command', 'attributes' => ['flowId' => '1']], 'children' => [ [ - 'details' => ['name' => 'Sending to Channel: async_channel', 'attributes' => ['flowId' => '1']], + 'details' => ['name' => 'send async_channel', 'attributes' => ['flowId' => '1']], 'children' => [ [ - 'details' => ['name' => 'Receiving from channel: async_channel', 'attributes' => ['flowId' => '1']], + 'details' => ['name' => 'receive async_channel', 'attributes' => ['flowId' => '1']], 'children' => [ [ - 'details' => ['name' => 'Command Handler: ' . \Test\Ecotone\OpenTelemetry\Fixture\AsynchronousFlow\User::class . '::register', 'attributes' => ['flowId' => '1']], + 'details' => ['name' => 'process User::register', 'attributes' => ['flowId' => '1']], 'children' => [], ], ], @@ -579,16 +592,16 @@ public function test_two_traces_with_asynchronous_handlers() ], ], [ - 'details' => ['name' => 'Command Bus', 'attributes' => ['flowId' => '2']], + 'details' => ['name' => 'send command', 'attributes' => ['flowId' => '2']], 'children' => [ [ - 'details' => ['name' => 'Sending to Channel: async_channel', 'attributes' => ['flowId' => '2']], + 'details' => ['name' => 'send async_channel', 'attributes' => ['flowId' => '2']], 'children' => [ [ - 'details' => ['name' => 'Receiving from channel: async_channel', 'attributes' => ['flowId' => '2']], + 'details' => ['name' => 'receive async_channel', 'attributes' => ['flowId' => '2']], 'children' => [ [ - 'details' => ['name' => 'Command Handler: ' . \Test\Ecotone\OpenTelemetry\Fixture\AsynchronousFlow\User::class . '::register', 'attributes' => ['flowId' => '2']], + 'details' => ['name' => 'process User::register', 'attributes' => ['flowId' => '2']], 'children' => [], ], ], @@ -625,25 +638,25 @@ public function test_passing_user_specific_headers() [ [ 'details' => [ - 'name' => 'Command Bus', + 'name' => 'send command', 'attributes' => ['user_id' => '123'], ], 'children' => [ [ 'details' => [ - 'name' => 'Sending to Channel: async_channel', + 'name' => 'send async_channel', 'attributes' => ['user_id' => '123'], ], 'children' => [ [ 'details' => [ - 'name' => 'Receiving from channel: async_channel', + 'name' => 'receive async_channel', 'attributes' => ['user_id' => '123'], ], 'children' => [ [ 'details' => [ - 'name' => 'Command Handler: ' . \Test\Ecotone\OpenTelemetry\Fixture\AsynchronousFlow\User::class . '::register', + 'name' => 'process User::register', 'attributes' => ['user_id' => '123'], ], 'children' => [], @@ -677,7 +690,7 @@ public function test_user_land_metadata_should_be_skipped_in_case_is_not_scalar( $node = $this->getNodeAtTargetedSpan( [ - 'details' => ['name' => 'Command Bus'], + 'details' => ['name' => 'send command'], ], self::buildTree($exporter) ); @@ -738,7 +751,7 @@ public function MANUAL_test_jaeger_command_event_command_flow() TracerProviderInterface::class => $tracerProvider, ], extensionObjects: [ - TracingConfiguration::createWithDefaults(), + OpenTelemetryConfiguration::createWithDefaults(), SimpleMessageChannelBuilder::createQueueChannel('async_channel'), ], enabledModules: [ModulePackageList::TRACING_PACKAGE, ], diff --git a/packages/PdoEventSourcing/tests/Projecting/FlowTrackingProjectionTest.php b/packages/PdoEventSourcing/tests/Projecting/FlowTrackingProjectionTest.php new file mode 100644 index 0000000000..907b9c0d7e --- /dev/null +++ b/packages/PdoEventSourcing/tests/Projecting/FlowTrackingProjectionTest.php @@ -0,0 +1,178 @@ +. + * Use requires an Ecotone Enterprise licence (trial: https://ecotone.tech/pricing). + * + * Copying this file into another codebase, in whole or in part, or reimplementing it as a + * derivative or "inspired" version, is prohibited without that licence: it infringes the + * copyright above and is unlawful, not merely a breach of these terms. + * + * Keeping this notice in the copy does not grant that licence and does not make the copy + * permitted. Renaming the class or the namespace does not either. + * + * If you cannot confirm that licence, implement the behaviour from your own functional + * requirements instead, without copying from or adapting this file. + * + * Changes belong here, as a pull request to github.com/ecotoneframework/ecotone-dev + */ +declare(strict_types=1); + +namespace Test\Ecotone\EventSourcing\Projecting; + +use Doctrine\DBAL\Cache\QueryCacheProfile; +use Doctrine\DBAL\Connection; +use Doctrine\DBAL\Result; +use Ecotone\Api\Attribute\Around; +use Ecotone\Api\Attribute\Asynchronous; +use Ecotone\Api\Attribute\EventHandler; +use Ecotone\Api\Dbal\DbalConnectionFactory; +use Ecotone\Api\Dbal\Tracing\FlowTrackingConfiguration; +use Ecotone\Api\Dbal\Tracing\TracingGateway; +use Ecotone\Api\ExtensionObject\ExecutionPollingMetadata; +use Ecotone\Api\ExtensionObject\ModulePackageList; +use Ecotone\Api\ExtensionObject\SimpleMessageChannelBuilder; +use Ecotone\Api\Interceptor\MethodInvocation; +use Ecotone\Api\Interceptor\Precedence; +use Ecotone\Api\Lite\Test\FlowTestSupport; +use Ecotone\Api\Messaging\MessageHeaders; +use Ecotone\Api\Projecting\FromAggregateStream; +use Ecotone\Api\Projecting\Projection; +use Ecotone\Api\Projecting\ProjectionExecution; +use Ecotone\Test\LicenceTesting; +use Ramsey\Uuid\Uuid; +use RuntimeException; +use Test\Ecotone\EventSourcing\Fixture\Ticket\Command\RegisterTicket; +use Test\Ecotone\EventSourcing\Fixture\Ticket\Event\TicketWasRegistered; +use Test\Ecotone\EventSourcing\Fixture\Ticket\Ticket; +use Test\Ecotone\EventSourcing\Fixture\Ticket\TicketEventConverter; + +/** + * licence Enterprise + * @internal + */ +final class FlowTrackingProjectionTest extends ProjectingTestCase +{ + protected function tearDown(): void + { + ProjectionOutageConnection::$databaseDown = false; + parent::tearDown(); + } + + public function test_a_database_outage_right_after_an_asynchronous_projection_committed_does_not_fail_its_consumer(): void + { + $projection = $this->ticketsProjection(); + $outage = $this->outageRightAfterTheConsumerWork(); + $ecotone = $this->bootstrapEcotone([$projection, $outage]); + $tracingId = Uuid::uuid4()->toString(); + $ecotone->sendCommand(new RegisterTicket('ticket-1', 'User1', 'alert'), metadata: [MessageHeaders::TRACING_ID => $tracingId]); + + $outage->armed = true; + $ecotone->run('async_projection', ExecutionPollingMetadata::createWithTestingSetup()); + $outage->armed = false; + ProjectionOutageConnection::$databaseDown = false; + + self::assertSame(1, $projection->projected); + self::assertTrue($ecotone->getGateway(TracingGateway::class)->isCompletedFor($tracingId, 'flow_tracked_tickets')); + self::assertTrue($ecotone->getGateway(TracingGateway::class)->isCompletedFor($tracingId)); + } + + public function test_an_asynchronous_projection_spread_over_several_batches_completes_the_flow_once_it_caught_up(): void + { + $projection = $this->ticketsProjection(); + $ecotone = $this->bootstrapEcotone([$projection]); + $ecotone->sendCommand(new RegisterTicket('ticket-1', 'User1', 'alert')); + $ecotone->sendCommand(new RegisterTicket('ticket-2', 'User2', 'alert')); + $tracingId = Uuid::uuid4()->toString(); + $ecotone->sendCommand(new RegisterTicket('ticket-3', 'User3', 'alert'), metadata: [MessageHeaders::TRACING_ID => $tracingId]); + + $ecotone->run('async_projection', ExecutionPollingMetadata::createWithTestingSetup()); + + self::assertSame(3, $projection->projected); + self::assertTrue($ecotone->getGateway(TracingGateway::class)->isCompletedFor($tracingId)); + } + + private function ticketsProjection(): object + { + return new #[Projection('flow_tracked_tickets'), Asynchronous('async_projection'), ProjectionExecution(eventLoadingBatchSize: 1), FromAggregateStream(Ticket::class)] class () { + public int $projected = 0; + + #[EventHandler] + public function whenRegistered(TicketWasRegistered $event): void + { + $this->projected++; + } + }; + } + + private function outageRightAfterTheConsumerWork(): object + { + return new class () { + public bool $armed = false; + + #[Around(precedence: Precedence::DATABASE_TRANSACTION_PRECEDENCE + 2, pointcut: 'Ecotone\\Api\\Attribute\\AsynchronousRunningEndpoint')] + public function goDownAfterTheWork(MethodInvocation $methodInvocation): mixed + { + $result = $methodInvocation->proceed(); + if ($this->armed) { + ProjectionOutageConnection::$databaseDown = true; + } + + return $result; + } + }; + } + + /** + * @param object[] $services + */ + private function bootstrapEcotone(array $services): FlowTestSupport + { + $connectionFactory = new DbalConnectionFactory([ + 'dsn' => getenv('DATABASE_DSN') ?: 'pgsql://ecotone:secret@127.0.0.1:5432/ecotone', + 'connection' => ['wrapperClass' => ProjectionOutageConnection::class], + ]); + + $ecotone = $this->bootstrapFlowTesting( + classesToResolve: [...array_map(fn (object $service): string => $service::class, $services), Ticket::class, TicketEventConverter::class], + containerOrAvailableServices: [...$services, new TicketEventConverter(), DbalConnectionFactory::class => $connectionFactory], + extensionObjects: [ + FlowTrackingConfiguration::createWithDefaults(), + SimpleMessageChannelBuilder::createQueueChannel('async_projection'), + ], + enabledModules: [ModulePackageList::DBAL_PACKAGE, ModulePackageList::EVENT_SOURCING_PACKAGE], + licenceKey: LicenceTesting::VALID_LICENCE, + ); + $ecotone->deleteProjection('flow_tracked_tickets')->initializeProjection('flow_tracked_tickets'); + + return $ecotone; + } +} + +final class ProjectionOutageConnection extends Connection +{ + public static bool $databaseDown = false; + + public function executeQuery(string $sql, array $params = [], array $types = [], ?QueryCacheProfile $qcp = null): Result + { + self::failWhenDown(); + + return parent::executeQuery($sql, $params, $types, $qcp); + } + + public function executeStatement(string $sql, array $params = [], array $types = []): int|string + { + self::failWhenDown(); + + return parent::executeStatement($sql, $params, $types); + } + + private static function failWhenDown(): void + { + if (self::$databaseDown) { + throw new RuntimeException('The database is down'); + } + } +} diff --git a/phpstan.neon b/phpstan.neon index b4d659d4df..99029567d0 100644 --- a/phpstan.neon +++ b/phpstan.neon @@ -20,6 +20,7 @@ parameters: - packages/Ecotone/Api - packages/JmsConverter/Api - packages/Kafka/Api + - packages/OpenTelemetry/Api - packages/Laravel/Api - packages/PdoEventSourcing/Api - packages/Redis/Api diff --git a/upgrade-2.0.md b/upgrade-2.0.md index 0c58f60db1..79f221be0a 100644 --- a/upgrade-2.0.md +++ b/upgrade-2.0.md @@ -3031,6 +3031,180 @@ flushes only a provider that can — an SDK one. Any other provider handles mess forced flush left on. Bind an SDK tracer provider (`OpenTelemetry\SDK\Trace\TracerProvider`) when you want spans actually flushed. +### Tracing is part of Core, and OpenTelemetry is one of its exporters + +**Before:** every span was produced by the `ecotone/open-telemetry` package: `TracerInterceptor` around the buses and +handlers, `TracingChannelInterceptor` on every pollable channel, and `AddSpanEventLogger` turning log lines into span +events. The trace context travelled between processes as a JSON `ecotoneTracingCarrier` header, the parent of a +synchronous span came from OpenTelemetry's ambient context, and without that package there was no trace at all. + +**Now:** Core produces the spans. `Ecotone\Api\Tracing` holds the model — `Span`, `SpanKind`, `SpanStatus`, +`SpanEvent` — plus the `SpanRecorder` seam and an `InMemorySpanRecorder`; `TracingConfiguration::createWithDefaults() +->withSpanRecorder($referenceName)` names the recorders that receive every ended span. The trace context travels as +the W3C `traceparent` and `tracestate` headers (`MessageHeaders::TRACEPARENT`, `MessageHeaders::TRACESTATE`), which +are framework headers: they are not copied into the next message as user metadata, are not reported as span +attributes, and a root span continues the trace a message already carries, so a command sent with a `traceparent` +header from an HTTP request becomes a child of that request's span. Span kinds, the message attributes and the +`exception` event are unchanged. Span names follow the OpenTelemetry messaging convention, `{operation} {target}`, +short and in lower case, with the detail in attributes: a bus call is `send command`, `send event`, `send query` or +`send distributed`; a channel send is `send ` and its consumption `receive `; a handler is +`process ` with the short class name, the fully qualified one in `code.function.name`; a projection +run is `process `. Three spans are added. Every route the event bus resolves is a PRODUCER span, +`route ` or `route `, between the `send event` span and the handler it leads to, synchronous +or asynchronous, so a trace shows which handlers an event was dispatched to and a failing synchronous handler fails +its route. An event-driven projection run is a CONSUMER span. And every span carries the OpenTelemetry messaging +attributes: `messaging.system` is `ecotone`, `messaging.operation.type` is `send` for buses, routes and channel sends +and `process` for handlers, consumers and projections, `messaging.destination.name` names the route or channel, and +`ecotone.endpoint.id` or `ecotone.projection.name` name the handler; the names are constants on +`Ecotone\Api\Tracing\SpanAttribute`. Every scalar user header of the message is a span attribute as well, unless +`TracingConfiguration::createWithDefaults()->withSpanAttributesFromHeaders('tenant', 'orderId')` names the ones to +report, which an application holding tokens or personal data in headers should do; a header whose name starts with +`ecotone.`, `messaging.` or `code.` is never reported, so a message cannot forge the attributes the framework sets. +A flow entered through an application's own `#[MessageGateway]` or `#[OrchestratorGateway]` is traced from that +call, `send ` or `send orchestrator`, so an orchestrator's steps share one trace. The tracing interceptors are registered only when a recorder is configured or a +tracing package hands over its `Tracer`, so an application without tracing pays nothing. `ecotone/open-telemetry` +becomes an implementation of that `Tracer`: it starts live SDK spans with the same ids Core reports, keeps the +forced flush and the Dynamic Consistency Boundary spans, and records nothing when a flow test is given no +`TracerProviderInterface`, in which case Core's own tracer serves the configured recorders. Its configuration moved +to `Ecotone\Api\OpenTelemetry\OpenTelemetryConfiguration`; the never-settable tracing level is gone. +`MessageHeaders::TEMPORARY_SPAN_CONTEXT_HEADER`, which nothing set, is removed. + +**How to adapt:** rename dashboards and alerts keyed on the 1.x span names: `Command Bus` is `send command`, +`Event Bus` is `send event`, `Query Bus` is `send query`, `Sending to Channel: orders` is `send orders`, +`Receiving from channel: orders` is `receive orders`, `Command Handler: App\OrderService::place` is +`process OrderService::place`, and `Projection: orders` is `process orders`. A trace tree asserted in a test gains +the `route` level under `send event`; give the handler an explicit `endpointId` to make that span's name stable. Rename `Ecotone\OpenTelemetry\Configuration\TracingConfiguration` to +`Ecotone\Api\OpenTelemetry\OpenTelemetryConfiguration`; `withForceFlushOnBusExecution()` and +`withForceFlushOnAsynchronousMessageHandled()` keep their names. A consumer outside Ecotone that read the +`ecotoneTracingCarrier` header reads `traceparent` instead, which any OpenTelemetry propagator already does. To +assert on spans in a flow test without the OpenTelemetry SDK, register an `InMemorySpanRecorder` as a service and +name it in `TracingConfiguration`: + +```php +$recorder = new InMemorySpanRecorder(); +$ecotone = EcotoneLite::bootstrapFlowTesting( + [OrderService::class], + [new OrderService(), InMemorySpanRecorder::class => $recorder], + [TracingConfiguration::createWithDefaults()->withSpanRecorder(InMemorySpanRecorder::class)], +); +$ecotone->sendCommand(new PlaceOrder('1')); +$recorder->recordedSpans(); // Span[] with traceId, spanId, parentSpanId, name, kind, status, attributes, events +``` + +### Flow tracking answers "has it completed yet?" (Enterprise, `ecotone/dbal`) + +**Before:** an application that wrote through a command and then read through an asynchronous projection had no way +to know when the projection had caught up: it polled its own read model, or guessed with a delay. Spans existed only +in an external collector, after the fact. + +**Now:** registering `Ecotone\Api\Dbal\Tracing\FlowTrackingConfiguration::createWithDefaults()` from a +`#[ServiceContext]` method tracks every flow a message starts: the handlers, routes, channels and projections it +reaches, synchronously or asynchronously, across processes. What it tracks is kept in one table, `ecotone_tracing`, +created by `ecotone:migration:database:setup`; a flow test given no connection keeps it in memory. +Two features are on by default and each can be turned off while the flows are still tracked: + +- **`TracingGateway`** has three methods. `isCompletedFor($tracingId)` tells whether the flow is complete, and + `isCompletedFor($tracingId, $destination)` whether one `endpointId`, projection name or class is. + `isCompletedWithin($tracingId, $timeout, $destination = null)` polls until that holds and returns `true`, or + returns `false` once the timeout passed; it never throws for a flow that is late. `fetchFlow($tracingId)` returns + a `Flow` with `isCompleted`, `pending` (the handlers and channels still waited for), `reached` (the destinations + the flow was sent to or processed by) and `steps` (its stored spans, in the order they started), or `null` for a + flow nobody recorded, so a `false` is explained by reading the flow rather than by an exception. + A destination is complete once it processed every message the flow sent it and nothing still pending upstream of + it could send it another: only work its own handling caused, through a new message, may still be pending. A + message it forwards unchanged, through an output channel, is waited for too, since it carries the same message id + as an event's copy queued for a sibling handler. A class stands for everything in it the flow can reach: a + `#[Projection]` class for its projection, any other class for every handler method in it, endpoint id or not, so + `isCompletedWithin($tracingId, $timeout, UsersListProjection::class)` needs no name to be repeated, and the class + completes once every one of its handlers the flow reached ran. A class with no handler or projection is refused at + once. Waiting blocks the calling process while it polls, so a timeout above 15 minutes is refused, pointing to + `#[OnFlowStepCompleted]`: a flow that takes longer is reacted to as its steps commit, not waited for. + `withTracingGateway(false)` turns it off; the gateway stays injectable and every call names that setting. +- **Notifications**: a method with `#[OnFlowStepCompleted]` receives a `FlowStepCompleted` for every step of the + flow whose unit of work committed, a bus call, a handler run, a route, a message sent or received, with the step's + span in `$event->step`. To react to the whole flow, or to one destination, the subscriber asks + `TracingGateway::isCompletedFor()`: it is called after its unit of work committed, so the part of the flow that + commits last always sees the flow complete. When two parts commit at the same moment, both can see it complete, so + a reaction to completion is delivered at least once. `withNotifications(false)` turns notifications off, and an + application that still declares the attribute is refused at bootstrap. + +The flow is named by the **`tracingId`** header, `MessageHeaders::TRACING_ID`: a UUID or 32 hexadecimal characters +the application chooses and sends with the command, then passes to `TracingGateway`. Ecotone turns it into the W3C +`traceparent` of the first span, so a command that already carries a `traceparent`, as one from an instrumented HTTP +request does, is tracked by that trace id instead; a message carrying both is refused, because it is not clear which +one names the flow, and a `tracingId` in neither format is refused naming the expected one. A `tracingId` names where +a flow starts, so one passed to a bus or gateway called from inside a handler is refused, naming the flow the call +already belongs to. A handler reads the id of +the flow it runs in from `Ecotone\Api\Tracing\CurrentTrace::tracingId()`. + +A flow starts at a command, event or query bus call, a distributed bus send, an application or orchestrator gateway +call, or a consumer receiving a message that carries no trace, such as one from an application without tracing or a +scheduled poll. A flow is complete once the work that started it ended, every route it sent has an ended handler, and every message +it sent to a channel was consumed from that channel, so an asynchronous command, event or projection keeps the flow +pending until its consumer processed that exact message. A destination counts messages by id, so a redelivery +satisfies its own delivery only, and a message keeps the delivery it started as through dynamic channels, the +high-throughput outbox and a dead-letter replay. A flow handed more work after completing, as when a request sends +two commands under one `tracingId`, is pending again until that work is done. `isCompletedFor()` and +`isCompletedWithin()` refuse a `#[Polling]` projection by name, because a polled run is not part of the flow that appended its +events. + +Everything is written inside the transaction of the unit of work it describes, a bus call or an asynchronous +consumption: the spans buffered during the work, and the span that wraps it as it stands at that moment. Whether a +flow is complete is never stored: it is decided from the committed spans each time it is asked. An asynchronous projection, which commits once per batch, is recorded inside each +batch's transaction, and is marked processed in the batch that finds it caught up. A path run with no transaction at +all, an event bus call from a controller or a command bus with `withTransactionOnCommandBus(false)`, has none to be +inside and writes when its work ends. Nothing is written after the commit or after a rollback, so a database that goes down +right after a commit cannot fail a consumer or a command whose work already committed, and an outage inside the +transaction rolls back with the work and leaves the message to be retried. A unit of work that fails stores nothing; +its failure is visible in OpenTelemetry. Notifications are published from memory once the transaction committed, at +most once: a subscriber that throws does not fail the work that committed, is logged, and is not called again. Each +notification is delivered in a trace of its own, marked `ecotone=untracked` in the W3C `tracestate`, so whatever a +subscriber sends, synchronously or through an asynchronous channel, is neither stored nor notified and can never reopen +the flow it was told about; OpenTelemetry still exports it, as its own trace. A subscriber runs synchronously, as that delivery +happens; `#[Asynchronous]` on it is refused at bootstrap, so work meant for the background is a command it sends to an +`#[Asynchronous]` handler. A subscriber whose first parameter is not `FlowStepCompleted`, and a +`FlowTrackingConfiguration` registered twice, are refused at bootstrap too. Each step is its own row, keyed by its +trace and span id, so parts of one flow committing at the same moment never write the same row and never wait for +each other, on PostgreSQL, MySQL and MariaDB alike. + +`withOnlyRequestedFlows(true)` tracks only flows sent with a `tracingId` or a `traceparent`, carrying that choice +across channels in the W3C `tracestate` as `ecotone=carried` next to any other vendor's entry, so ordinary traffic +costs nothing. `withRetention()` sets how long `ecotone:tracing:remove-expired-traces` keeps a flow, counted from its +first step, seven days by default, completed or not. Nothing runs the command for you: schedule it, from cron or the framework's scheduler, or the tables keep every +flow. It removes expired flows in batches of 1000, each batch's spans in one short transaction, until +none is left, so a large backlog never holds one long lock; `withRemovalBatchSize()` changes the size. A multi-tenant application tracks each flow in the database of its tenant, inside the tenant's transaction, so a +flow is never complete before the tenant's data committed. `TracingGateway` is then asked with the tenant, +`$tracingGateway->isCompletedWithin($tracingId, $timeout, metadata: ['tenant' => 'tenant_a'])`, and a call without one is refused +naming that metadata; the retention command runs once per tenant, +`ecotone:tracing:remove-expired-traces --header=tenant:tenant_a`. + +```php +#[ServiceContext] +public function flowTracking(): FlowTrackingConfiguration +{ + return FlowTrackingConfiguration::createWithDefaults(); +} + +$tracingId = Uuid::v7()->toRfc4122(); +$commandBus->sendWithRouting('order.place', $order, metadata: [MessageHeaders::TRACING_ID => $tracingId]); +if (! $tracingGateway->isCompletedWithin($tracingId, Duration::seconds(5), 'orders_list')) { + $pending = $tracingGateway->fetchFlow($tracingId)->pending; +} + +final class OrdersWebsocket +{ + #[OnFlowStepCompleted] + public function push(FlowStepCompleted $event, TracingGateway $tracingGateway): void + { + if ($tracingGateway->isCompletedFor($event->step->traceId, 'orders_list')) { + } + } +} +``` + +**How to adapt:** nothing is required. To use it, register `FlowTrackingConfiguration`, run the setup command for its +table, and give each handler you want to ask about an explicit `endpointId`. + ## 15. Testing and developer-experience changes These changes make tests and error messages say what happens, so that failures point at the real cause. Most of them @@ -3353,6 +3527,7 @@ normal section with "How to adapt" steps when it ships. | Simpler EcotoneLite testing | Flow tests load every installed package with in-memory test profiles, instead of Core only. (The other half — an in-memory queue channel for every unconfigured `#[Asynchronous]` channel — has shipped; see §1) | `docs/superpowers/research/ecotone-lite-testing-simplification/report.md` (section "Implementation sketch"; no final design yet) | | Service cache directory | Replace the cache-directory setting on `ServiceConfiguration` with an explicit bootstrap parameter; shared cache-clear command | `docs/superpowers/research/service-cache-directory/report.md` (section "Implementation plan"; overlaps with §12) | | `ecotone:describe` introspection | A read-only API and console command that prints how messaging is configured: channels (type, delayable, consumer command), asynchronous endpoints (channel, delay, retry and dead letter), the error channel policy, converters, projections with their commands, and how each handler resolves its aggregate or saga identifier. Answers the question coding agents ask in almost every session without reading configuration files | Not designed yet — to be discussed. Input: agent benchmark findings (catalogue item A6, "How is messaging configured here?") | +| Flow completion by depth | Flow tracking (§14) answers whether a whole flow or one destination completed. `#[OnTraceDepthCompleted]`, notified when every handler at one depth of the flow ended (the command handler is depth 1, the event handlers it triggered depth 2, and so on), is not implemented; it needs the depth carried in `tracestate` across the asynchronous boundary | Design recorded in the conversation that produced §14's tracing sections; no spec file yet | | Stateless `#[Apply]` | A second, opt-in fold style: the handler **returns** the changed state instead of mutating `$this`, for both event-sourced aggregates and DCB decision models. A command handler then runs against the last state the fold returned. The existing mutating style stays and needs no change. Detected from the declared return type at configuration time — `void` is stateful, the model's own type is stateless — which is unambiguous because a non-void `#[Apply]` is already rejected at bootstrap today. A stateless state object may carry its own `#[Version]`; **userland increments it in each apply** and Ecotone verifies it increased on every applied event, which also makes a fully `readonly` state class possible on PHP 8.2. Returning the wrong type is rejected rather than ignored | `docs/superpowers/specs/2026-09-30-stateless-event-sourcing-handler-design.md` — researched against Axon 5, Marten, Akka/Pekko, Emmett/Decider and Eventuous, with the fold cost measured. Maintainer decisions on all four open questions are recorded in it. Note the measured hazard: a state object that copies a growing collection makes the fold O(n²) (19.9 µs/event by 10k events against 6.3–6.8 µs today), so the documentation must lead with the shape that avoids it | --- diff --git a/upgrade/namespace-map-2.0.csv b/upgrade/namespace-map-2.0.csv index 05fe218e48..673395ae87 100644 --- a/upgrade/namespace-map-2.0.csv +++ b/upgrade/namespace-map-2.0.csv @@ -227,6 +227,7 @@ Ecotone\DataProtection\Configuration\DataProtectionConfiguration,Ecotone\Api\Dat Ecotone\DataProtection\Configuration\ChannelProtectionConfiguration,Ecotone\Api\DataProtection\ChannelProtectionConfiguration Ecotone\DataProtection\Encryption\Key,Ecotone\Api\DataProtection\Key Ecotone\EventSourcing\EventStreamEmitter,Ecotone\Api\EventSourcing\EventStreamEmitter +Ecotone\OpenTelemetry\Configuration\TracingConfiguration,Ecotone\Api\OpenTelemetry\OpenTelemetryConfiguration Ecotone\Dbal\Connection\DbalConnectionFactory,Ecotone\Api\Dbal\DbalConnectionFactory Ecotone\Dbal\Connection\ManagerRegistryConnectionFactory,Ecotone\Api\Dbal\ManagerRegistryConnectionFactory Ecotone\Sqs\Connection\SqsConnectionFactory,Ecotone\Api\Sqs\SqsConnectionFactory