Skip to content

DS-3508/[Data Streams] gRPC transport for EA - #23233

Merged
denis-chernov-smartcontract merged 18 commits into
developfrom
DS-3508-bridge-connection-manager
Aug 19, 2026
Merged

denis-chernov-smartcontract merged 18 commits into
developfrom
DS-3508-bridge-connection-manager

Conversation

@denis-chernov-smartcontract

@denis-chernov-smartcontract denis-chernov-smartcontract commented Jul 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Introduces an opt-in connection manager for external adapters that support the streams-adapter gRPC protocol. When a bridge is created with UseConnectionManager=true, the BridgeTask no longer performs a synchronous HTTP request per pipeline run. Instead, it reads from a process-level observation cache populated by a persistent, per-bridge gRPC subscription to the adapter.

Changes

New package core/services/pipeline/bridgeconn

  • BridgeConnManager: package-level singleton that owns one observation cache and one EAConn per bridge.
  • EAConn: persistent gRPC client that:
    • dials the adapter host derived from the bridge URL,
    • sends a full subscription snapshot every 10 s,
    • prunes assets idle for ≥ 30 s (indirect unsubscribe),
    • reconnects with exponential backoff on failures,
    • accepts SubscribeResponse messages, validates payload_hash against registered assets, and writes the observation into the cache.
  • grpcStreamClient: gRPC dialer supporting both plaintext (http://) and TLS (https://) transports with no application tokens or client certificates.
  • Protobuf contract defined in core/services/pipeline/bridgeconn/streamspb/streams.proto (StreamService/Subscribe).

Pipeline bridge task integration

  • BridgeTask.Run checks bridge.UseConnectionManager.
  • If enabled, it uses BridgeConnManager.GetObservation(...) to return a cached observation.
  • The cache key is SHA256(bridgeName + subscriptionData), matching the adapter-side payload_hash.
  • Cache entries are valid for 60 s; missing or expired entries return ErrBridgeObservationNotFound / ErrBridgeObservationExpired.
  • Telemetry (latencies, request/response data) is still emitted for this path.

Bridge model & API

  • Added use_connection_manager boolean to bridge_types (migration 0304_add_use_connection_manager_to_bridge_types.sql).
  • Added UseConnectionManager to BridgeType, BridgeTypeRequest, BridgeTypeAuthentication, and presenters.
  • Exposed useConnectionManager in GraphQL schema, resolvers, and create/update inputs.

Build tooling

  • Added google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.6.2 to the protoc make target.

Tests

  • Unit tests in bridgeconn/bridge_conn_manager_test.go for:
    • concurrent one-EAConn-per-bridge behavior,
    • asset registration/refresh/pruning,
    • observation handling, cache hit/miss, TTL expiry, missing data field,
    • reconnect backoff.
  • Unit test in pipeline/task.bridge_test.go for the connection-manager path.
  • End-to-end integration test TestIntegration_LLO_bridgeConnManagerHappyPath in ocr2/plugins/llo/integration_test.go, plus a fake gRPC streams-adapter server in helpers_test.go.

Opt-in behavior

UseConnectionManager defaults to false. Existing bridges and jobs continue to use the standard HTTP request path.

Known limitations / follow-up

@denis-chernov-smartcontract denis-chernov-smartcontract added the build-publish Build and Publish image to SDLC label Jul 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

✅ No conflicts with other open PRs targeting develop

@denis-chernov-smartcontract denis-chernov-smartcontract removed the build-publish Build and Publish image to SDLC label Jul 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

@trunk-io

trunk-io Bot commented Jul 26, 2026 •

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge

View Full Report ↗︎ ⋅ Docs

@denis-chernov-smartcontract
denis-chernov-smartcontract force-pushed the DS-3508-bridge-connection-manager branch from 1381ead to 1b55e94 Compare July 26, 2026 05:43
@denis-chernov-smartcontract denis-chernov-smartcontract added the build-publish Build and Publish image to SDLC label Jul 26, 2026
@denis-chernov-smartcontract
denis-chernov-smartcontract force-pushed the DS-3508-bridge-connection-manager branch 2 times, most recently from ba8c45a to b9beb5d Compare July 26, 2026 06:14
@denis-chernov-smartcontract
denis-chernov-smartcontract force-pushed the DS-3508-bridge-connection-manager branch from b9beb5d to 4b565bd Compare July 26, 2026 06:24
@denis-chernov-smartcontract
denis-chernov-smartcontract force-pushed the DS-3508-bridge-connection-manager branch 3 times, most recently from 4691835 to 93c16c9 Compare August 18, 2026 03:56
@denis-chernov-smartcontract
denis-chernov-smartcontract force-pushed the DS-3508-bridge-connection-manager branch from 93c16c9 to 688b42f Compare August 18, 2026 04:34
@denis-chernov-smartcontract
denis-chernov-smartcontract marked this pull request as ready for review August 18, 2026 05:16
Comment thread core/services/pipeline/bridgeconn/bridge_conn_manager.go Outdated
Comment thread core/services/pipeline/bridgeconn/eaconn.go Outdated
Comment thread core/services/pipeline/task.bridge.go
@cl-sonarqube-production

Copy link
Copy Markdown

@denis-chernov-smartcontract
denis-chernov-smartcontract added this pull request to the merge queue Aug 19, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 19, 2026
@denis-chernov-smartcontract
denis-chernov-smartcontract added this pull request to the merge queue Aug 19, 2026
Merged via the queue into develop with commit d9b280f Aug 19, 2026
259 of 269 checks passed
@denis-chernov-smartcontract
denis-chernov-smartcontract deleted the DS-3508-bridge-connection-manager branch August 19, 2026 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build-publish Build and Publish image to SDLC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants