Skip to content

Composio memory sync: add Shopify (shopify) sync pipeline #84

Description

@M3gA-Mind

Important

The pipeline body goes in tinyhumansai/tinycortex; the wiring goes here. This is a cross-repo deliverable.

How a Composio memory-source sync is actually built (Gmail is the worked example):

Piece Lives in Job
Enginerun_incremental_sync(source: &dyn IncrementalSource, …) (vendor/tinycortex/src/memory/sync/orchestrator.rs:113) tinycortex paging, incremental filtering, ingestion — generic, toolkit-agnostic
PipelineGmailSyncPipeline (vendor/tinycortex/src/memory/sync/composio/gmail.rs:18, ~229 lines) tinycortex implements SyncPipeline + IncrementalSource; answers toolkit-specific questions (toolkit()"gmail", action()GMAIL_FETCH_EMAILS, arguments(), extract_page()). Its tick() is one line — it hands itself to the engine
ProviderComposioProvider (src/openhuman/memory_sync/composio/providers/gmail/) openhuman account/profile, daily request budget, HTML→text post-processing, cursor/state; delegates ingestion to tinycortex
Selector — the match at src/openhuman/tinycortex/sync.rs:344 openhuman maps a toolkit slug → pipeline. A missing arm is what raises tinycortex sync does not support toolkit '<slug>'

Placement decision: the pipeline body lands in tinycortex, alongside the existing six (gmail, github, notion, linear, clickup, slack). To be precise about why: this is a deliberate consistency decision, not a technical constraintIncrementalSource is an unsealed pub trait over public types (orchestrator.rs:53-104) and the engine takes a trait object, so openhuman could legally implement a pipeline locally under the orphan rule. We are keeping all pipeline bodies in one repo instead.

Order of work:

  1. tinyhumansai/tinycortex — implement the pipeline (SyncPipeline + IncrementalSource), register it, cut a release.
  2. tinyhumansai/openhuman — bump the vendor/tinycortex submodule pointer (pinned at 9a0603af), add the sync.rs:344 arm, and register the ComposioProvider so memory_sources.supported_toolkits advertises the slug.

This issue stays on openhuman because step 2 is here and this is where the user-visible failure surfaces. Do not close it on the tinycortex release alone — it is done when the slug actually syncs end to end.

Dedupe (read before writing): set metadata.path_scope to a stable collection scope; per-item IDs are dedupe keys only. PR tinyhumansai/openhuman#4953 fixed exactly this bug class (stable document_id as the upsert key) — do not reintroduce it.

Coordinate with tinyhumansai/openhuman#4993, which adds a drift guard asserting the advertised set == the syncable set. As each pipeline registers, its slug moves from "advertised but not syncable" to both.


Summary

Composio memory-source sync has no pipeline for shopify (Shopify). Connecting it succeeds and reports ACTIVE, then fails at sync with tinycortex sync does not support toolkit 'shopify' — so the source silently never ingests store data.

Problem

src/openhuman/tinycortex/sync.rs:344-353 matches on the toolkit slug and only implements six: gmail, github, notion, linear, clickup, slack. shopify falls to the _ arm:

tinycortex sync does not support toolkit 'shopify'

But shopify is advertised in CAPABILITY_TOOLKITS (src/openhuman/memory_sync/composio/providers/mod.rs:63), so the app offers it as connectable. The connect flow gives no indication sync is unimplemented — the user gets a source that looks healthy and produces nothing.

Solution

Implement a Shopify sync pipeline and register it so memory_sources.supported_toolkits reports it:

  • Fetch orders/products — value as a memory source is questionable; consider closing as wontfix.
  • Map to memory documents with a stable dedupe key — set metadata.path_scope to a stable collection scope (per-item IDs are dedupe keys only; see AGENTS.md).
  • Register the provider so it appears in all_composio_sync_providers() and is selectable in tinycortex/sync.rs.
  • Follow the existing pipelines as the pattern (GmailSyncPipeline for message-shaped sources, NotionSyncPipeline for document-shaped ones).

Related: PR tinyhumansai/openhuman#4953 fixed a sync upsert-key defect (use a stable document_id) — apply that lesson here rather than reintroducing it.

Acceptance criteria

  • Pipeline implemented and registeredmemory_sources.supported_toolkits includes shopify.
  • Sync works end to end — connecting Shopify ingests real content; no does not support toolkit error.
  • Stable dedupe — re-syncing does not duplicate; path_scope set to a stable scope.
  • Regression coverage — pipeline unit test; diff coverage >= 80%.
  • No PII/secrets logged — debug logging present on the new flow, content-free.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions