Entity Identifier Resolver

Entity Identifier Resolver (EIR) is a local entity database and search engine written in Rust.

EIR stores structured entities and resolves queries against them using aliases, tokens, tags, attributes, sources, and relationships.

Status

EIR is under active development.

The core database, storage, indexing, and search systems are implemented and tested. The CLI provides tools for working with EIR databases.

⚠️ EIR is not encrypted. Do not use it to store sensitive data without appropriate external protection.

---

Architecture

flowchart TB
    CLI["eir-cli"]

    ENGINE["Engine"]

    DB["Database"]
    RESOLVER["Resolver"]
    STORAGE["Storage"]

    QUERY["Query"]
    SEARCH["Search"]
    INDEXES["Indexes"]

    CLI --> ENGINE

    ENGINE --> DB
    ENGINE --> RESOLVER
    ENGINE --> STORAGE

    RESOLVER --> QUERY
    QUERY --> SEARCH
    SEARCH --> INDEXES

    DB --> INDEXES

Loading

At the center is Engine, which coordinates the database, resolver, and storage backend.

The main flow is:

Entity Documents
       │
       ▼
    Database
       │
       ├── Registries
       └── Indexes
              │
              ▼
           Resolver
              │
              ▼
          Search Results

---

Project Structure

EntityIdentifierResolver/
├── crates/
│   ├── eir-core/
│   ├── eir-cli/
│   └── eir-version/
├── apps/
├── docs/
└── fixtures/

eir-core

The core EIR library containing:

• database and engine
• entity model
• storage
• indexes
• query parsing
• search and ranking

eir-cli

Command-line interface for creating, searching, inspecting, and maintaining databases.

eir-version

Database/version-related functionality.

---

Entity Model

An entity can contain aliases, tags, attributes, relationships, and sources.

{
  "id": 1001,
  "aliases": [
    "FizzBerry Spark",
    "FizzBerry"
  ],
  "tags": [
    "drink",
    "berry"
  ],
  "attributes": [],
  "relationships": [],
  "sources": [
    {
      "provider": "Open Food Facts",
      "verified": true
    }
  ]
}

---

Search

EIR combines multiple search signals.

flowchart LR
    Q["Query"] --> P["Parser"]
    P --> PLAN["Planner"]

    PLAN --> EXACT["Exact"]
    PLAN --> PREFIX["Prefix"]
    PLAN --> FUZZY["Fuzzy"]
    PLAN --> TOKEN["Token"]
    PLAN --> TAG["Tag"]
    PLAN --> PROPERTY["Property"]
    PLAN --> RELATIONSHIP["Relationship"]

    EXACT --> C["Candidates"]
    PREFIX --> C
    FUZZY --> C
    TOKEN --> C
    TAG --> C
    PROPERTY --> C
    RELATIONSHIP --> C

    C --> R["Ranker"]
    R --> RESULTS["Results"]

Loading

Search results include the signals that contributed to a match, making results easier to understand and debug.

---

Storage

An EIR database uses a logical .eir database identity together with its supporting storage:

database/
├── database.eir
├── eir.toml
├── segments/
└── wal/

The storage layer uses persistent segments and a write-ahead log.

flowchart LR
    MUTATION["Insert / Update / Remove"]
    WAL["WAL"]
    DB["Database"]
    SEGMENTS["Segments"]

    MUTATION --> WAL
    MUTATION --> DB
    DB --> SEGMENTS

Loading

Compaction rewrites persistent storage to remove obsolete data.

---

CLI

The CLI provides the main database management interface:

init
build
stats
inspect
search
insert
remove
update
compact
merge
server
completions

Example:

cargo eir init data nutrition

cargo eir build \
  --input fixtures/entities.json \
  --database data/nutrition

cargo eir search \
  data/nutrition/nutrition.eir \
  "FizzBerry"

See CLI Documentation for the complete command reference.

---

Development

Clone the repository:

git clone https://github.com/HandsOnDigits/EntityIdentifierResolver.git
cd EntityIdentifierResolver

Check the workspace:

cargo check --workspace

Run the tests:

cargo test --workspace

Format the code:

cargo fmt --all

---

Documentation

• CLI — command reference and database operations
• Repository — source code and development

---

Design Goals

EIR is designed around a few simple principles:

• Local — no remote service required
• Structured — entities contain more than just names
• Fast — specialized indexes for different search operations
• Explainable — search results expose matching signals
• Embeddable — the core engine is independent of the CLI
• Recoverable — persistent storage uses snapshots and WAL

---

Entity Identifier Resolver — TODO

🏗️ Core Architecture

Completed

• Rust workspace
• eir-core crate
• eir-cli crate
• eir-version crate
• Entity model
• EntityID
• EntityType
• Entity aliases
• Tags
• Sources
• Attributes
• Relationships
• Registries / interners
• Database abstraction
• Engine abstraction
• Resolver abstraction

Planned

• Stabilize public core API
• Document core architecture
• Define database compatibility/versioning policy
• Improve error model

---

🔎 Search & Entity Resolution

Completed

• Exact alias search
• Prefix alias search
• Fuzzy alias search
• Token search
• Tag search
• Attribute/property search
• Relationship search
• Alias index
• Prefix trie
• Fuzzy/BK-tree index
• Token inverted index
• Posting lists
• Query parser
• Query intent
• Query filters
• Search planner
• Search executor
• Search stages
• Candidate collection
• Search signals
• Ranking
• Search explanations
• Search tests

Planned

• Improve ranking quality
• Tune fuzzy matching
• Add more query operators
• Improve relationship queries
• Improve search explanations
• Benchmark search performance
• Test against larger real-world datasets
• Add configurable ranking strategies

---

💾 Database & Storage

Completed

• Database lifecycle
• Database creation
• Database opening
• .eir database identity
• eir.toml configuration
• Storage configuration
• DEIR storage format
• Storage segments
• Segment manager
• Backend abstraction
• Write-ahead log (WAL)
• WAL replay
• Database recovery
• Snapshot persistence
• Flush
• Index rebuilding
• Database statistics

---

✏️ Entity Mutations

Completed

• Insert entities
• Remove entities
• Update entities
• Duplicate entity detection
• Entity validation
• Index rebuild after mutation
• WAL support for insert
• WAL support for remove
• WAL support for update
• Mutation recovery tests

---

🧹 Compaction

Completed

• Compact command
• Segment rewrite
• Remove obsolete storage data
• Report storage size before/after
• Report reclaimed space
• Compaction tests

Planned

• Add automatic compaction policy
• Add configurable compaction thresholds

---

🔀 Database Merge

Completed

• Merge command
• Merge two databases
• Duplicate entity ID detection
• Reject output/input collisions
• Combine entity collections
• Rebuild merged indexes
• Merge tests

Planned

• Support merging more than two databases
• Improve merge performance
• Add merge conflict strategies
• Document registry remapping
• Add large-database merge benchmarks

---

🖥️ CLI

Completed

• CLI with Clap
• init
• build
• stats
• inspect
• search
• insert
• remove
• update
• compact
• merge
• server command structure
• Shell completions
• CLI integration tests
• Database lifecycle tests

Planned

• Finish server implementation
• Improve CLI output formatting
• Add machine-readable output
• Improve error messages
• Improve command help
• Add CLI benchmarks
• Add support for CSV file import

---

🌐 Server / API

Completed

• Server command scaffold
• Server lifecycle command structure

Planned

• HTTP API
• Search endpoint
• Entity lookup endpoint
• Entity insertion endpoint
• Entity update endpoint
• Entity removal endpoint
• Database statistics endpoint
• Health endpoint
• API documentation
• Authentication strategy
• Request validation
• API integration tests

---

📦 Data & Fixtures

Completed

• JSON entity fixtures
• Test entities
• Test sources
• Test tags
• Test attributes
• Test relationships
• Larger test database
• CLI lifecycle fixture tests

Planned

• CSV entity fixtures

---

🧪 Testing

Completed

• Unit tests
• Database lifecycle tests
• Persistence tests
• WAL recovery tests
• Insert tests
• Remove tests
• Update tests
• Compaction tests
• Merge tests
• Search tests
• Query tests
• CLI tests
• Duplicate ID tests
• Output/input collision tests

Planned

• Large dataset tests
• Performance benchmarks
• Search relevance benchmarks
• Storage benchmarks
• Fuzz testing
• Crash/recovery testing
• Concurrency testing

---

📚 Documentation

Completed

• CLI documentation
• Architecture documentation
• Search architecture documentation
• Storage documentation
• Mermaid architecture diagrams
• README architecture overview

Planned

• Keep README aligned with source
• Keep CLI docs aligned with commands
• Database format documentation
• Entity schema documentation
• Search/query documentation
• Storage format specification
• API documentation
• Contributor guide
• Architecture decision records

---

🚀 Performance

Planned

• Benchmark database creation
• Benchmark inserts
• Benchmark updates
• Benchmark deletes
• Benchmark search
• Benchmark fuzzy search
• Benchmark index building
• Benchmark database opening
• Benchmark WAL replay
• Benchmark compaction
• Benchmark merge
• Memory usage profiling
• Large-dataset testing
• Investigate GPU acceleration

---

🔐 Security & Privacy

Current

• Local-first architecture
• No search history by default
• No external service required for core search
• No telemetry in the core engine

Planned

• Document security model
• Document filesystem permissions
• Optional database encryption strategy
• API authentication
• API authorization
• Security audit

---

🎯 Project Milestones

Phase 1 — Core Engine

• Entity model
• Database
• Storage
• Indexing
• Resolver
• Search

Phase 2 — Database Lifecycle

• Persistence
• WAL
• Recovery
• Insert
• Update
• Remove
• Compaction
• Merge

Phase 3 — CLI

• Database creation
• Build
• Search
• Inspect
• Stats
• Mutations
• Maintenance commands

Phase 4 — Production Readiness

• Performance benchmarks
• Large dataset testing
• Complete documentation
• Stable public API
• Error handling review
• Recovery testing

Phase 5 — API & Applications

• HTTP server
• API
• TypeScript client

License

See the repository for the current license.

Contains AI Assisted Code