Skip to content

Latest commit

 

History

243 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Record Store

CI Documentation OpenSSF Scorecard Latest release Apache-2.0 license

Record Store is a self-hosted, S3-compatible records store: one authoritative copy of an object, integrity you can prove to somebody else, history you can hold a deployment to, and share and embed links that make a stored object usable without copying it somewhere else.

That is a different job from a general object store. A records store is what you reach for when the question is not "where did we put the file" but "can we show this is the file, unchanged, and show who touched it". Record Store is built for the second question:

  • One authoritative copy. Payloads are immutable and addressed by generated identifiers, versioning keeps history rather than overwriting it, and an object written over S3 and one written through the console are the same object under the same rules. There is no second copy to drift.
  • Provable integrity. Every payload is checksummed on write and verified on read. Object Lock enforces GOVERNANCE and COMPLIANCE retention and legal holds, so a retained version refuses deletion by anyone — including the root credential. Proof bundles are signed documents a third party can check offline, against the file and nothing else: no server, no network, no credential.
  • Provable history. A durable audit trail records who did what, separately from the storage-event feed. Making that trail tamper-evident — a hash chain, checkpoints, and external anchoring, so a past state is provable against someone with disk access — is in progress and not yet shipped. Proof bundles already carry the section and report it as unavailable rather than implying it is covered.
  • Usable links. A share link gives a person read access to one object; an embed link gives a site or an application read-only bytes. Both are capabilities, not credentials, and both resolve through the same authoritative object.

A deployment is one process on one machine, with one copy of your data. No external database, message broker, or coordination service runs alongside it. Durability is whatever the storage underneath it gives you, so use redundant disks and take backups; if the machine is gone, the service is down until you restore it. Replication and erasure coding are not implemented, and the honest reason is that they are substantial work we intend to fund with adoption rather than ship ahead of it. Single-node is the supported shape today, and we would rather say so than imply a cluster story we cannot stand behind.

Public S3 traffic uses port 7600, the native management API uses 7601, and the web console uses 7602. Every listener is configurable. Record Store is provided and maintained by Open Elements®.

Documentation

Full documentation — installation, configuration, deployment, security, and reference — is published at https://openelementslabs.github.io/record-store/ and lives in docs/.

To build it locally:

pip install --require-hashes -r requirements-docs.txt
mkdocs serve

Install

Record Store publishes production container images for linux/amd64 and linux/arm64 to the GitHub Container Registry:

docker pull ghcr.io/openelementslabs/record-store:latest
docker pull ghcr.io/openelementslabs/record-store-console:latest

latest tracks the newest stable release. Name a version instead — 0.1.3, 0.1, or a digest — for anything you intend to keep running.

Both packages are public, so no docker login is needed. To run both from the published images, with RECORD_STORE_VERSION selecting the tag:

RECORD_STORE_VERSION=latest \
  docker compose --env-file .env -f deploy/docker/compose.ghcr.yml up -d

Each release carries SPDX SBOMs and a SHA256SUMS file covering every asset. Images built since attestation was enabled also carry signed provenance, verifiable with gh attestation verify; 0.1.1 and earlier do not — see Verifying a Release for what can be checked and what that limitation means.

See Installation, Container Images, and Verifying a Release. Released versions are recorded in CHANGELOG.md.

Supported S3 surface

  • AWS Signature Version 4 header authentication and presigned GET/PUT URLs
  • ListBuckets, CreateBucket, HeadBucket, and empty DeleteBucket
  • streaming PutObject, GetObject, HeadObject, and idempotent DeleteObject
  • ListObjectsV2 with bounded pagination, prefix, delimiter, and continuation tokens
  • multipart create, streamed part upload, persisted part listing, completion, abort, and upload listing
  • bucket versioning (Disabled, Enabled, and Suspended), immutable version reads/deletes, delete markers, and ListObjectVersions
  • Object Lock with AWS semantics: GOVERNANCE/COMPLIANCE retention, legal holds, per-bucket defaults, and a governance bypass gated on its own policy permission
  • per-bucket CORS configuration, unsigned browser preflights, and CORS headers on matching S3 responses
  • streaming same-bucket and cross-bucket CopyObject with COPY and REPLACE metadata directives
  • bounded, open-ended, and suffix byte ranges
  • If-Match, If-None-Match, If-Modified-Since, and If-Unmodified-Since
  • content type, x-amz-meta-*, SHA-256 checksum validation, single-part ETags, and multipart ETags

Presigned multipart part uploads use the same canonical SigV4 verifier. ACLs, UploadPartCopy, server-side encryption headers, and AWS's aws-chunked trailing-checksum encoding are not implemented. Unsupported operations or semantic headers return S3 XML NotImplemented; they are never silently accepted.

Object Lock is enforced inside the metadata transaction that would remove a version, so a retention placed concurrently cannot be raced. A COMPLIANCE retention binds every caller including the root credential; a GOVERNANCE retention yields only to x-amz-bypass-governance-retention: true presented by a credential holding s3:BypassGovernanceRetention, and every bypass is audited. Object Lock is chosen when a bucket is created and cannot be enabled later, because doing so would claim protection over versions written without it. It is enforced by Record Store rather than by the filesystem: it stops deletions through the API, not someone with access to the data directory — see Object Lock and Trust.

Architecture and durability

Protocol crates call shared application services; they do not access filesystem internals. Both protocol surfaces go through the same service layer, so an object written over S3 and an object written through the console are the same object under the same rules:

record-store-s3 ─────┐                         ┌── filesystem store
            ├──> record-store-service ────────>├── checksum verification
record-store-api ────┘          │              └── objects/
                       ▼
              metadata catalog
              (buckets, objects, versions)

A deployment is one process with one copy of your data. A successful write means the payload was streamed to a temporary file, checksummed, fsynced, and atomically renamed into place, with metadata published afterwards — so it survives process crash and power loss to the extent the filesystem honours fsync, and does not survive losing the disk. Redundancy under the data directory is the redundancy you have: use RAID, a mirrored pool, or a replicated volume, and take backups. Erasure coding is not implemented; the unused record-store-erasure crate is not wired into any code path.

Payloads are immutable and addressed by generated UUIDs. Logical bucket names and object keys never become filesystem paths. Uploads stream through bounded chunks into create-only temporary files while SHA-256 and MD5 are calculated, then use fsync and atomic rename before metadata publication.

Optional encryption at rest uses a random per-object or per-part data key, chunked AES-256-GCM authenticated encryption, and a master-key-wrapped data key. The payload header persists the algorithm/format version, nonces, logical size, object binding, and a non-secret key reference. Reads and byte ranges remain streaming and authenticate every accessed chunk. Enable it with RECORD_STORE_STORAGE_ENCRYPTION_ENABLED=true; the stable RECORD_STORE_CREDENTIAL_MASTER_KEY is then mandatory. Existing plaintext objects remain readable when encryption is first enabled, while all new object and multipart payloads are encrypted. Once an encrypted-store marker exists, startup refuses a missing, mismatched, or disabled key configuration rather than making data unreadable silently.

A durable publication journal resolves the payload/metadata crash window on startup. Replaced and deleted payloads use a durable cleanup queue. Multipart completion has durable completing state and startup reconciliation. Metadata schema version 5 uses ordered, non-destructive migrations.

Local state uses this layout:

<data-directory>/
├── metadata/catalog.redb
├── metadata/credentials.redb
├── metadata/audit.redb
├── metadata/events.redb
├── metadata/lifecycle.redb
├── objects/<2 hex>/<2 hex>/<object UUID>
├── system/
└── tmp/

Keep the temporary directory on the same filesystem as the data directory so publication by rename remains atomic.

Build and test

Rust 1.97.1 is selected by rust-toolchain.toml. A system protoc is not required.

cargo fmt --all --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-features --locked
cargo build --workspace --release --locked

Dependency security is checked with tests/rust-audit.sh, which runs cargo audit --deny warnings with no exceptions. The 2026-08-22 review upgraded quick-xml to 0.41.0 for RUSTSEC-2026-0194 and RUSTSEC-2026-0195. RUSTSEC-2026-0235 was carried for a while as a narrow exception — rkyv 0.7.46 reached Cargo.lock only as an inactive optional serialization backend of rust_decimal through openraft -> byte-unit, and was never compiled — and is now simply gone: rust_decimal 1.43.0 dropped that optional backend. --deny warnings additionally makes a yanked crate a failure rather than a note.

One dependency decision is settled ahead of the code that needs it, with the condition that ends it. On 2026-09-19, RFC 3161 anchoring was decided on der 0.7 and cms 0.2, because cms 0.3 exists only as 0.3.0-pre.2, and a pre-release — which promises no compatibility and can be yanked or re-cut under the same version — is not acceptable in a project built with --deny warnings. Revisit when cms 0.3 reaches a stable release. Neither crate is in Cargo.lock yet; they arrive with the anchoring work, and they will bring a duplicate const-oid with them — 0.9.6 through der 0.7 alongside the 0.10.2 already present through digest 0.11. That was weighed and accepted: there is no advisory against either, and rsa, the crate that would make an ASN.1 stack an audit problem, stays out of the tree. Because the two versions give unrelated ObjectIdentifier types, SHA-256's identifier is defined locally in crates/record-store-proof/src/anchor.rs and checked against the X.690 encoding rules, so the duplication cannot surface as a confusing comparison failure while parsing a TSTInfo. The decision and its removal condition are recorded in Audit Chain and Checkpoints.

The parsers that run before a request is authenticated are fuzzed. Targets live in fuzz/ and cover the S3 XML request bodies, the Authorization header, the presigned-URL query, the Range header, the ListObjectsV2 query, and bucket-name and object-key validation. Each asserts an invariant rather than only the absence of a panic — that an accepted range lies inside the object, that an accepted object key holds no .. or empty segment. CI builds and briefly runs every target; a real campaign is FUZZ_SECONDS=3600 tests/fuzz-smoke.sh. See Testing.

Storage microbenchmarks are reproducible with cargo bench -p record-store-storage --bench storage.

Real-client compatibility checks exercise boto3, AWS SDK for JavaScript v3, AWS SDK for Go, and AWS SDK for Java v2 against an ephemeral encrypted Record Store data directory on the fixed listeners. They cover bucket/object I/O, listing, multipart completion, presigned requests, browser CORS, ranges, versioning, historical reads, and copy behavior:

bash tests/compatibility/run.sh

The runner installs pinned client dependencies into a temporary directory and removes all test state when it exits.

Run

Record Store intentionally has no built-in credentials. Use distinct, stable secrets:

export RECORD_STORE_ROOT_ACCESS_KEY='local-admin'
export RECORD_STORE_ROOT_SECRET_KEY='replace-with-a-long-random-secret'
export RECORD_STORE_CREDENTIAL_MASTER_KEY='replace-with-a-stable-32-byte-or-longer-master-key'
export RECORD_STORE_MANAGEMENT_SYSTEM_TOKEN='replace-with-a-distinct-32-byte-or-longer-token'
export RECORD_STORE_STORAGE_ENCRYPTION_ENABLED=true
cargo run --bin record-store -- server

The equivalent daemon entry point is cargo run --bin record-store-server. Defaults remain:

S3 API          http://localhost:7600 (also serves /e/<token> embeds)
Management API  http://localhost:7601
Web console     http://localhost:7602 (also serves /s/<token> share pages)

Load the example file with cargo run --bin record-store -- server --config record-store.example.toml; secrets should still come from the environment.

AWS CLI

Configure path-style access and a root or policy-authorized service-account credential:

export AWS_ACCESS_KEY_ID="$RECORD_STORE_ROOT_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="$RECORD_STORE_ROOT_SECRET_KEY"
export AWS_DEFAULT_REGION=us-east-1
export AWS_EC2_METADATA_DISABLED=true
export AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED
export AWS_RESPONSE_CHECKSUM_VALIDATION=WHEN_REQUIRED
aws configure set s3.addressing_style path

aws --endpoint-url http://localhost:7600 s3api list-buckets
aws --endpoint-url http://localhost:7600 s3api create-bucket --bucket demo
aws --endpoint-url http://localhost:7600 s3api put-bucket-versioning \
  --bucket demo --versioning-configuration Status=Enabled
aws --endpoint-url http://localhost:7600 s3api put-bucket-cors --bucket demo \
  --cors-configuration '{"CORSRules":[{"AllowedOrigins":["https://app.example.com"],"AllowedMethods":["PUT","GET","HEAD"],"AllowedHeaders":["content-type","x-amz-*"],"ExposeHeaders":["ETag","x-amz-version-id"],"MaxAgeSeconds":3600}]}'
aws --endpoint-url http://localhost:7600 s3 cp ./example.pdf s3://demo/example.pdf
aws --endpoint-url http://localhost:7600 s3 cp s3://demo/example.pdf ./downloaded.pdf
aws --endpoint-url http://localhost:7600 s3api list-object-versions --bucket demo

When using a named profile, apply path-style addressing to that profile as well: aws configure set s3.addressing_style path --profile PROFILE. Keep the endpoint as a plain URL; shell commands must not contain Markdown link syntax. The checksum environment settings avoid the aws-chunked trailer encoding that Record Store intentionally reports as unsupported.

Set RECORD_STORE_ROOT_S3_ENABLED=false after service-account policies are established to keep root credentials off the application data plane.

Browser access is denied by default. Configure CORS on each bucket that a web origin may reach; Record Store does not apply a deployment-wide wildcard. A successful preflight is unauthenticated but grants only the origins, methods, and request headers stored on that bucket. The following signed request still needs its ordinary S3 permission or valid presigned URL. Record Store never emits Access-Control-Allow-Credentials because S3 browser authorization belongs in the signature rather than ambient cookies.

Management API and CLI

Only GET /health and GET /ready are public. System information is part of the authenticated management plane, and GET /metrics accepts only the dedicated RECORD_STORE_METRICS_SCRAPE_TOKEN. Set RECORD_STORE_MANAGEMENT_TOKEN in the CLI environment to the configured system, storage, or auditor token.

If no system token is configured, legacy root Basic authentication remains available for development compatibility and Record Store emits a warning. Management roles are separate from S3 policies: system administrators have full access, storage administrators manage storage/buckets/integrity/lifecycle, and auditors have read-only access to audit and operational metadata.

export RECORD_STORE_MANAGEMENT_TOKEN="$RECORD_STORE_MANAGEMENT_SYSTEM_TOKEN"
cargo run --bin record-store -- status
cargo run --bin record-store -- bucket list
cargo run --bin record-store -- bucket create demo
cargo run --bin record-store -- bucket versioning enable demo
cargo run --bin record-store -- bucket object-lock show demo
cargo run --bin record-store -- service-account create my-app
cargo run --bin record-store -- credential rotate <account-id>
cargo run --bin record-store -- policy create ./policy.json
cargo run --bin record-store -- policy attach <policy-id> <account-id>
cargo run --bin record-store -- webhook list
cargo run --bin record-store -- audit --limit 100
cargo run --bin record-store -- verify object demo path/to/object
cargo run --bin record-store -- storage inspect
cargo run --bin record-store -- storage repair              # dry run
cargo run --bin record-store -- storage repair --apply      # explicit orphan deletion

Service-account and webhook signing secrets are returned only when created or rotated. Stored signing material is encrypted with AES-256-GCM under the injected RECORD_STORE_CREDENTIAL_MASTER_KEY. The same injected master material derives a domain-separated object key-encryption key when payload encryption is enabled. Record Store refuses to create encrypted credentials without it and refuses startup if encrypted records or payload state exist but the key is unavailable. The master key is never stored by Record Store.

S3 service accounts use attached allow/deny policies. Explicit deny overrides allow; no matching allow is an implicit deny. Policy resources use canonical decoded logical keys and support only a trailing wildcard, avoiding filesystem or ambiguous wildcard semantics.

Webhooks and lifecycle

Storage events are persisted separately from audit events. Matching webhook deliveries run outside the object upload response path, use HMAC-SHA256 signatures, persist state across restart, and stop after bounded exponential retries. HTTPS and public network targets are the safe defaults; HTTP and private targets require explicit configuration. Redirects are disabled and attempts have a fixed timeout.

Lifecycle rules support prefix-scoped current-object expiration and non-current-version expiration. The supervised worker scans indexed metadata in bounded pages, persists a cursor per rule, and writes an audit event for each successful deletion.

Offline metadata backup

Stop Record Store before backup or restore. The command obtains an exclusive data-directory lock, so it refuses to race a running server. Backups contain versioned, SHA-256-verified metadata database files, not object payloads or configuration secrets.

cargo run --bin record-store -- server backup-metadata ./backup-2026-08-21
cargo run --bin record-store -- server restore-metadata ./backup-2026-08-21

Restore refuses an incompatible manifest or a non-empty target metadata directory.

Configuration

Configuration file values overlay defaults, then environment variables take precedence. Unknown fields and invalid values fail startup.

Environment variable Configuration field
RECORD_STORE_S3_BIND server.s3_bind
RECORD_STORE_API_BIND server.api_bind
RECORD_STORE_SHUTDOWN_TIMEOUT_SECONDS server.shutdown_grace_period_seconds
RECORD_STORE_STORAGE_DATA_DIRECTORY storage.data_directory
RECORD_STORE_STORAGE_TEMPORARY_DIRECTORY storage.temporary_directory
RECORD_STORE_STORAGE_ENCRYPTION_ENABLED storage.encryption_enabled
RECORD_STORE_ROOT_ACCESS_KEY auth.root_access_key
RECORD_STORE_ROOT_SECRET_KEY auth.root_secret_key
RECORD_STORE_CREDENTIAL_MASTER_KEY auth.credential_master_key
RECORD_STORE_ROOT_S3_ENABLED auth.root_s3_enabled
RECORD_STORE_MANAGEMENT_SYSTEM_TOKEN auth.management_system_token
RECORD_STORE_MANAGEMENT_STORAGE_TOKEN auth.management_storage_token
RECORD_STORE_MANAGEMENT_AUDITOR_TOKEN auth.management_auditor_token
RECORD_STORE_METRICS_SCRAPE_TOKEN auth.metrics_scrape_token
RECORD_STORE_MAX_CONCURRENT_OPERATIONS limits.maximum_concurrent_operations
RECORD_STORE_MAX_HEADER_BYTES limits.maximum_header_bytes
RECORD_STORE_WEBHOOK_ALLOW_HTTP webhooks.allow_http
RECORD_STORE_WEBHOOK_ALLOW_PRIVATE_NETWORKS webhooks.allow_private_networks
RECORD_STORE_WEBHOOK_TIMEOUT_SECONDS webhooks.request_timeout_seconds
RECORD_STORE_WEBHOOK_MAXIMUM_ATTEMPTS webhooks.maximum_attempts
RECORD_STORE_WEBHOOK_POLL_INTERVAL_SECONDS webhooks.poll_interval_seconds
RECORD_STORE_LIFECYCLE_INTERVAL_SECONDS lifecycle.interval_seconds
RECORD_STORE_LIFECYCLE_BATCH_SIZE lifecycle.batch_size
RECORD_STORE_OBJECT_LOCK_CLOCK_WATERMARK_INTERVAL_SECONDS object_lock.clock_watermark_interval_seconds
RECORD_STORE_OBJECT_LOCK_CLOCK_BACKWARDS_TOLERANCE_SECONDS object_lock.clock_backwards_tolerance_seconds
RECORD_STORE_SHARING_SHARES_ENABLED sharing.shares_enabled
RECORD_STORE_SHARING_EMBEDS_ENABLED sharing.embeds_enabled
RECORD_STORE_SHARING_MAXIMUM_LIFETIME_DAYS sharing.maximum_lifetime_days
RECORD_STORE_SHARING_REQUIRE_EXPIRATION sharing.require_expiration
RECORD_STORE_SHARING_REQUIRE_PASSWORD sharing.require_share_password
RECORD_STORE_SHARING_MAXIMUM_ACCESS_COUNT sharing.maximum_access_count
RECORD_STORE_SHARING_PASSWORD_ATTEMPTS_PER_MINUTE sharing.password_attempts_per_minute
RECORD_STORE_SHARING_TOKEN_PROBES_PER_MINUTE sharing.token_probes_per_minute
RECORD_STORE_SHARING_UNLOCK_LIFETIME_HOURS sharing.unlock_lifetime_hours
RECORD_STORE_SHARING_PREVIEW_TEXT_LIMIT_BYTES sharing.preview_text_limit_bytes
RECORD_STORE_SHARING_SHARE_BASE_URL sharing.share_base_url
RECORD_STORE_SHARING_EMBED_BASE_URL sharing.embed_base_url
RECORD_STORE_LOG observability.log_filter
RECORD_STORE_LOG_JSON observability.json
RECORD_STORE_CONFIG_FILE server/CLI configuration selection

Preview, share links, and embeds

Stored objects are usable directly rather than only administrable. The console previews an object; a share link gives a person read access to one object through a Record Store page; an embed link gives a website or application a read-only URL for the bytes. All three resolve through the same authoritative object service, so there is no second copy of anything.

                          Record Store object
                               │
            ┌──────────────────┼──────────────────┐
            ▼                  ▼                  ▼
         Preview            Share              Embed
      authenticated       a person        a site or an app
      console :7602      /s/<token>          /e/<token>
                        console :7602       S3 API :7600

A share link and an embed link are different capabilities, and they are published in different places. A share is a page Record Store renders, so it lives on the console alongside the viewer that shows it. An embed serves object bytes into somebody else's page, so it lives on the S3-compatible endpoint that already publishes object bytes — which is what lets a deployment expose storage to the internet while the management plane and the console stay closed. Set sharing.embed_base_url when storage is published under its own hostname.

Both are capabilities rather than credentials. The opaque token in the path is the entire authorization; it names one object and one version policy and can express nothing else. Neither can list, write, delete, or reach any other object, and neither is ever an S3 credential. Every request re-resolves the token against durable state, so a revocation takes effect on the next one.

Share link Embed link
Intended for A person A website or application
Delivered by Console :7602 S3 API :7600
Version Current, or a pinned VersionId Current, or a pinned VersionId
Access View, download, or both Read-only bytes
Optional controls Password, expiry, strict access budget Origin allowlist, expiry
Caching no-store, so revocation is immediate Short, bounded revalidation

Only media types Record Store is prepared to be responsible for are served inline: JPEG, PNG, WebP, GIF, MP4, WebM, MP3, Ogg, WAV, PDF, plain text, Markdown, CSV, and JSON. A declared type is corroborated against the object's leading bytes before anything is rendered, so an upload labelled image/png that begins with <html> is refused. HTML, SVG, XML, and script are never rendered inline and never embeddable inline; they remain downloadable as attachments. Downloads are unchanged: always Content-Disposition: attachment, always nosniff, whatever the object turns out to be.

Capability tokens carry 256 bits of entropy from the operating system's cryptographic generator. They are stored as a lookup digest plus an AES-256-GCM-sealed copy under the deployment's master key, so an administrator can copy a link again without Record Store holding it in the clear. Share passwords are stored as salted Argon2 hashes, never a digest, and repeated attempts are throttled per link and per client so a public link cannot be locked for everyone. Capability tokens are redacted from request logs and audit records; audit entries name a share or embed by its stable non-secret identifier instead.

Web console

The console is an administrative interface for Record Store. It is a client of the management API on 7601 and is never required: Record Store stays fully operable through the CLI and the API alone.

Applications ──────► S3 API        :7600
Embedding sites ───► S3 API        :7600  /e/<token>
Share recipients ──► Web console   :7602  /s/<token>
Administrators ────► Web console   :7602 ──► Management API :7601

The browser talks only to the console's own origin. The console server attaches the management credential and forwards the request to 7601, so the credential lives in an HTTP-only cookie the page cannot read, no CORS configuration is needed, and the browser never reaches the management API, the stored objects, or the metadata catalog.

Public share pages are served by the same application but authorize differently: that boundary attaches no credential at all, because the token in the path is the authorization. Embed bytes do not pass through the console.

After sign-in, the console reads GET /api/v1/system/info for the deployment's capability set and renders only the screens that capability set supports.

Develop

Requires Node 24 and a running Record Store server.

cd console
npm install
RECORD_STORE_API_URL=http://127.0.0.1:7601 npm run dev   # http://localhost:7602

Sign in with a management role token, for example the value of RECORD_STORE_MANAGEMENT_SYSTEM_TOKEN. An auditor token signs in to a read-only console.

Validate

cd console
npm run lint
npm run typecheck
npm run test
npm run build

End-to-end tests drive a real Record Store server rather than a mock, so console and API drift is caught rather than papered over:

cd console
npm run test:e2e:install   # once, downloads Chromium
npm run test:e2e

Configuration

Variable Purpose
RECORD_STORE_API_URL management API base URL, default http://127.0.0.1:7601
RECORD_STORE_CONSOLE_SECURE_COOKIES force the session cookie's Secure flag; defaults to on in production
PORT console listener, default 7602

RECORD_STORE_API_URL is read on the server at runtime, so one image works in any deployment and no localhost assumption is compiled into the bundle.

Object uploads

The browser sends an object as one streaming PUT. The File handle itself is the request body, so bytes travel from disk to the network without passing through the page's heap; object size is not bounded by browser memory.

There is no resume. An interrupted upload fails and has to be sent again from the first byte, and the console states that rather than implying otherwise. Resumable browser uploads need multipart operations the management API does not expose yet: presigned part URLs, so control requests go to 7601 while part bodies go straight to the S3 API on 7600 and no long-lived secret reaches the page. The transport is one injected function in console/features/objects/upload-transport.ts, so such a strategy can replace it without touching the queue, progress, retry, or cancellation UI above it.

Docker

The Compose files below build from source, which is what you want while developing. For a real deployment, use the published images through deploy/docker/compose.ghcr.yml — see Install.

Compose variables may be kept in a repo-root .env file (which Git ignores) and loaded explicitly with --env-file .env. Use deploy/docker/compose.console.yml for Record Store plus the console, or deploy/docker/compose.yml for the server on its own.

docker build -f deploy/docker/Dockerfile -t record-store .
docker run --read-only \
  -e RECORD_STORE_ROOT_ACCESS_KEY \
  -e RECORD_STORE_ROOT_SECRET_KEY \
  -e RECORD_STORE_CREDENTIAL_MASTER_KEY \
  -e RECORD_STORE_MANAGEMENT_SYSTEM_TOKEN \
  -e RECORD_STORE_STORAGE_ENCRYPTION_ENABLED=true \
  -p 7600:7600 -p 7601:7601 \
  -v record-store-data:/var/lib/record-store record-store

The default Compose file (deploy/docker/compose.yml) runs the server on its own. It publishes only S3 on localhost:7600 and management on localhost:7601. Development secrets have explicit local defaults and must not be copied into production:

docker compose -f deploy/docker/compose.yml up --build -d
docker compose -f deploy/docker/compose.yml ps

A second Compose file (deploy/docker/compose.console.yml) runs the server together with the web console. It publishes S3 on 7600, management on 7601, and the console on 7602:

docker compose --env-file .env -f deploy/docker/compose.console.yml up --build -d
# open http://localhost:7602 and sign in with RECORD_STORE_MANAGEMENT_SYSTEM_TOKEN

The Compose network carries plaintext traffic and is intended for local development. Terminate TLS in a reverse proxy in front of 7600 and 7602 for any real deployment, and keep 7601 private.

The runtime image is non-root, supports a read-only root filesystem, publishes only ports selected by the operator, uses the management health endpoint, and performs SIGTERM-aware graceful shutdown across the HTTP listeners and background workers.

Repository structure

apps/record-store-server       startup, listeners, backup, and worker supervision
apps/record-store-cli          server and management command-line interface
crates/record-store-core       validated domain model
crates/record-store-service    shared bucket/object application services
crates/record-store-s3         S3 protocol, SigV4, XML, multipart, and versioning
crates/record-store-api        native management HTTP API and management roles
crates/record-store-storage    streaming filesystem backend and recovery journal
crates/record-store-metadata   durable indexed catalog and ordered migrations
crates/record-store-auth       encrypted credentials and authorization policies
crates/record-store-audit      durable bounded security audit trail
crates/record-store-sharing    share and embed capabilities
crates/record-store-events     durable events and signed webhook delivery
crates/record-store-lifecycle  incremental lifecycle expiration worker
crates/record-store-config     configuration loading and validation
crates/record-store-observability structured tracing initialization
console/              web console: Next.js, React, Tailwind, TanStack
deploy/docker/        container and Compose definitions
docs/                 MkDocs documentation site
.github/workflows/    CI, documentation, and the release pipeline

License

Apache License 2.0. See LICENSE.

About

Record Store is a self-hosted object storage service s3-compatible

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages