Skip to content
openclawPublic

About

Shared Go infrastructure for local-first crawler archives.

Topics

Resources

Contributing

Security policy

Stars

69 stars

Watchers

1 watching

Forks

Latest commit

 

History

498 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

crawlkit 🧰 — one kit, many crawlers

crawlkit banner

CI Go module Go License

crawlkit is the shared Go library for local-first crawler archives. It gives crawler authors provider-neutral building blocks for config paths, SQLite stores, snapshots, backups, synchronization, search, terminal interfaces, and automation.

Provider APIs, authentication, schemas, privacy filters, and user-facing command contracts stay in the downstream crawl apps.

Install

crawlkit requires Go 1.27.0 or newer.

Development builds prefer Go 1.27.4 through the toolchain directive in go.mod.

Add the package you need to a Go module. For the quick start below:

go get github.com/openclaw/crawlkit/store@latest

Install the optional archive controller:

go install github.com/openclaw/crawlkit/cmd/crawlctl@latest

The latest module version is available through the Go module proxy. Signed CLI archives and the release process are documented in Publishing Crawlkit.

Quick start

This example opens an in-memory SQLite store with crawlkit's connection defaults, applies a schema, writes a row, and reads it back:

package main

import (
	"context"
	"fmt"

	"github.com/openclaw/crawlkit/store"
)

func main() {
	ctx := context.Background()
	db, err := store.Open(ctx, store.Options{
		Path:   ":memory:",
		Schema: `create table items (title text not null)`,
	})
	if err != nil {
		panic(err)
	}
	defer db.Close()

	if _, err := db.DB().ExecContext(ctx, `insert into items values (?)`, "first crawl"); err != nil {
		panic(err)
	}
	rows, err := db.Query(ctx, `select title from items`)
	if err != nil {
		panic(err)
	}
	fmt.Println(rows.Values[0]["title"])
}

Run it from a module that depends on crawlkit:

go run .
first crawl

Package map

Area Packages What they own
Local data config, store, state, cache Runtime paths, SQLite access, sync cursors, and safe cache snapshots
Portable archives snapshot, backup, mirror JSONL/Gzip packs, encrypted backups, sidecars, and Git-backed history
Search embed, vector Embedding providers, vector encoding, exact search, and result fusion
App contracts control, output, progress Machine-readable metadata, output formats, and CI-safe progress logs
Remote archives remote Provider-neutral HTTP client and versioned archive protocol
Background processing worker Bounded continuous workers over application-owned durable queues
User surfaces scheduler, tui, releasecheck Refresh jobs, terminal browsing, and release notices

See the package guide for the complete inventory and Go package reference for exported APIs.

crawlctl

crawlctl discovers installed crawl apps through their machine-readable metadata, runs configured refresh jobs under a held OS lock, and records JSONL run history.

Discovery and refresh commands allow two seconds for inherited output pipes to close after the command exits or its context is canceled. A descendant that keeps a pipe open then produces an error instead of blocking the controller and its run lock indefinitely. This does not impose a runtime limit on active jobs or terminate descendant processes.

The lock file is persistent: do not remove or replace it to unlock the runner. Closing the owning handle or exiting releases the lock. Keep its directory private; on Windows, file access follows the directory's ACLs. Unsupported OS locking fails closed.

Stop all old runners before upgrading from PID-file locking. Legacy or ambiguous lock contents require an explicit stopped-runner migration: verify all old runners are stopped, archive that legacy file once, then start the new runner. There is no mixed old/new runner safety or automatic stale-PID reclamation.

Default discovery checks gitcrawl, discrawl, notcrawl, wacrawl, telecrawl, slacrawl, graincrawl, imsgcrawl, photoscrawl, and weicrawl. Use --app to select binaries explicitly. Discovery does not automatically schedule source-specific commands such as Photos' import_apple. Discovery reports malformed metadata and missing manifest IDs in the app's error field. Supported legacy binaries retain their fallback manifests. Legacy fallback also recognizes supported binaries selected by absolute or relative executable path. Discovery records an absolute executable path so saved jobs keep working when the runner starts from another directory.

Jobs using {repo} require at least one nonblank repository; blank-only lists report a failed job instead of silently doing no work.

Scheduler due checks use the exact job and repository identity, so colon-containing names do not share scheduling history.

If a write is interrupted, history reads ignore a truncated final JSON value and the next run repairs that tail before appending. Valid final records without a trailing newline are retained; complete corrupt records still report an error.

History reads accept the full records written by the runner, including long command arguments. Explicit configuration paths work without a home directory; default and ~/ paths still require one. Saving a controller configuration applies private file permissions before replacing existing content.

Command Purpose
init Discover crawl apps and write a controller config
discover Print discovered crawl apps
run Run enabled refresh jobs
status Show the latest recorded job status
logs Print recent job logs (--tail requires a positive integer)
install Install or render a periodic schedule
uninstall Remove an installed periodic schedule

Use crawlctl run -- JOB or crawlctl logs -- JOB for job names that look like flags. Options before -- retain their normal behavior; names after it stay literal.

The install command resolves relative --config paths before creating a schedule, so the scheduled run uses the same config and log directory from another working directory.

Scheduling uses launchd on macOS, systemd user units on Linux, Task Scheduler on Windows, and cron rendering as the portable fallback. Cron requires whole-minute intervals below one hour that divide 60 evenly, such as 5m, 12m, or 30m; other intervals are rejected because minute steps restart each hour.

Launchd and systemd plans require whole-second intervals of at least one minute. Windows Task Scheduler plans require whole-minute intervals from 1m through 1439m.

Cron plans escape percent signs in executable, config, and log paths before cron processes the command, including when adjacent to literal backslashes. Paths containing line feeds are rejected because each cron entry must occupy one physical line.

Systemd plans preserve literal argument text, including quotes, percent signs, and dollar signs. Executable paths must follow systemd rules: quotes, backslashes, and control characters are rejected before installation. Dry-run previews can still render plans for another operating system.

See Background workers for content-triggered processing alongside ingestion.

Private native backup transport

crawlctl preservation transfers existing native backup files to the opt-in admin-only preservation API in crawl-remote. It does not query a provider, prune a store, decompress a backup, or activate a publication. Configure the service's dedicated private preservation bucket before using it. Enabling these routes advertises the admin auth role; keep them on an isolated endpoint until existing contract consumers support that role. The default base contract and existing archive publication paths remain unchanged.

Provide the admin credential through an environment variable managed by your normal credential helper. Do not put a credential on the command line. For example, with PRESERVATION_TOKEN already injected:

crawlctl preservation upload \
  --endpoint https://archive.example.com \
  --app discrawl --archive discrawl/example \
  --token-env PRESERVATION_TOKEN \
  --file native.sqlite.zst=/backups/native.sqlite.zst \
  --file media/attachment.bin=/backups/attachment.bin > upload-receipt.json

crawlctl preservation download \
  --endpoint https://archive.example.com \
  --app discrawl --archive discrawl/example \
  --token-env PRESERVATION_TOKEN \
  --manifest <manifest_sha256-from-receipt> \
  --output /restore/new-empty-destination

crawlctl preservation inspect \
  --endpoint https://archive.example.com \
  --app discrawl --archive discrawl/example \
  --token-env PRESERVATION_TOKEN \
  --manifest <manifest_sha256-from-receipt>

inspect fetches and verifies only the pinned manifest, without downloading chunks or writing files. Its JSON output includes manifest_verified: true; this does not verify current chunk availability or qualify a database restore.

Use crawlctl preservation verify with the same flags as inspect to stream every chunk and verify its length and hash plus each complete-file hash without writing files. It returns files_verified: true only after all content passes; restore_qualified remains false. Use download when a local copy is needed.

Keep the upload receipt in the recovery inventory. Its manifest_sha256 pins all file names, complete-file hashes, and ordered chunk hashes. Repeating an unchanged upload reuses immutable object identities; it does not fetch upstream data again. The default chunk size is 64 MiB; --chunk-size can reduce it. A manifest supports up to 64 nonempty files and 256 chunks, at most 16 GiB. For a larger inventory, retain every separate manifest digest explicitly. The default per-request timeout is five minutes and can be set with --timeout.

For a source file larger than one manifest, --file-range NAME=OFFSET:LENGTH:PATH reads an exact section without creating a temporary copy. Offset and length are decimal byte counts; length must be positive and the section must fit the source file. For example, an 8 GiB section can use --file-range pack.part-000=0:8589934592:/backups/archive.pack. Use a separate archive identifier for each volume. The library equivalent is PreservationSource.Offset and PreservationSource.Length; leaving both zero keeps the whole-file behavior.

A section receipt verifies only those selected bytes. Retain the original file size and SHA-256, each section's offset and length, and all manifest pins in the recovery inventory. Restore each pinned volume, concatenate in recorded offset order, and verify the original complete-file checksum before declaring the original file recoverable. Range upload does not provide a snapshot of a file that is still changing.

Upload reads sources twice and holds at most one chunk plus bounded copy buffers at a time. Sources must already be closed, coherent backups: use the native SQLite backup owner first, not an actively written database/WAL pair. Source size, each chunk, and the complete-file hash are checked before manifest publication. Failed uploads can leave private unreferenced chunks; they never publish a successful receipt or change the current archive.

Download creates a new destination (0700 directories and 0600 files on Unix) and never overwrites an existing directory. On Windows, use a parent directory with an appropriately restricted ACL. It verifies the exact manifest bytes, every downloaded chunk, and each complete file. Failure leaves a reported partial destination; retry into another new directory. Success reports files_verified: true, while restore_qualified remains false. Decompress separately, run SQLite integrity and native fidelity checks, and account for external files before declaring a backup recoverable or changing retention. This transport does not certify upstream freshness, exclusion policy, query coverage, or restored schemas.

The same operations are available through remote.PlanPreservation, Client.UploadPreservation, Client.FetchPreservationManifest, and Client.DownloadPreservationFile. A failed download may have written partial bytes to the supplied writer; callers must discard those bytes.

Boundaries

crawlkit accepts shared mechanics only when they are provider-neutral, reusable by at least two apps, and preserve each app's database and CLI contracts. The ownership map tracks what belongs here and what remains in GitHub-, Discord-, Slack-, Notion-, and other provider-specific applications.

The remote package owns the Go client and v1 wire contract for hosted archives. Worker deployment, D1 schema, authentication policy, and secrets live outside this module; see the remote contract and Cloudflare archive design.

Safety

Tests and examples use temporary or in-memory data. They do not access app runtime stores such as ~/.config/gitcrawl, ~/.slacrawl, ~/.discrawl, or ~/.notcrawl.

Snapshot imports reject absolute and parent-traversing shard paths while preserving literal root and shard names, including current-directory roots. This is a lexical check: snapshots must still come from a trusted filesystem tree because shard reads follow symlinks.

Pass a plain filesystem path to store.Open unless SQLite driver parameters are intentional. A caller-supplied file: URI keeps its query parameters, which can override crawlkit's default pragmas or fail connection validation.

Development

make check

This runs module tidiness, formatting, vet, dead-code and vulnerability checks, unit tests, and race tests with GOWORK=off. See CONTRIBUTING.md for the compatibility rules.

Keep modernc.org/libc at the exact version required by the selected modernc.org/sqlite module. SQLite v1.62.0 requires libc v1.80.1; its runtime dependency must be updated together with SQLite, not independently.

SQLite operations waiting for a write lock honor their context cancellation or deadline, including during the default five-second busy timeout.

When nesting JSON produced by a scalar subquery inside another JSON constructor, mark the subquery result explicitly with json(...), for example json_array(json((select json_group_array(value) from items))). SQLite 3.54 does not preserve the JSON subtype through that subquery boundary. Explicit conversion also works on SQLite 3.53 and keeps arrays from becoming quoted strings.

License

MIT

About

Shared Go infrastructure for local-first crawler archives.

Topics

Resources

Contributing

Security policy

Stars

69 stars

Watchers

1 watching

Forks

Releases

Used by

Contributors

Languages