diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 2a8a408a98..62681543c0 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -60,8 +60,11 @@ For example: ## Documentation Status - +Select one option and provide the documentation location when applicable. + +- [ ] `Doc - TODO`: required documentation is pending; complete it before merging. +- [ ] `Doc - Done`: documentation is included here or linked below. +- [ ] `Doc - No Need`: no user-visible documentation is affected. -- [ ] `Doc - TODO` -- [ ] `Doc - Done` -- [ ] `Doc - No Need` +Documentation files in this PR or paired hugegraph-doc PR: + diff --git a/.serena/memories/architecture_and_modules.md b/.serena/memories/architecture_and_modules.md index a74d658209..687ae2251f 100644 --- a/.serena/memories/architecture_and_modules.md +++ b/.serena/memories/architecture_and_modules.md @@ -32,4 +32,5 @@ Distributed storage + Raft: `hg-store-core`, `hg-store-node`, `hg-store-client`, - **hugegraph-cluster-test**: Cluster integration tests ## Distributed Deployment (BETA) -PD + Store + Server (3+ nodes each), all gRPC. Docker compose configs in `docker/` directory, using bridge networking (migrated from host mode). +PD manages placement and metadata; Store owns partition data; Server serves graph queries. +Development and HA topologies differ; see `docker/README.md` for node counts and configuration. diff --git a/.serena/memories/code_style_and_conventions.md b/.serena/memories/code_style_and_conventions.md index 7a4c310e0b..701757a1b6 100644 --- a/.serena/memories/code_style_and_conventions.md +++ b/.serena/memories/code_style_and_conventions.md @@ -24,5 +24,5 @@ - Validate: `mvn apache-rat:check -ntp` + `mvn editorconfig:check` ## Build -- Java 11 target, `-Xlint:unchecked`, Lombok 1.18.30 (provided/optional) +- Java 11 target except Commons (Java 8 target); compiler/dependency versions come from module POMs. - Swagger: `io.swagger.core.v3:swagger-jaxrs2-jakarta` for REST API docs diff --git a/.serena/memories/ecosystem_and_related_projects.md b/.serena/memories/ecosystem_and_related_projects.md index 6bb1344d66..5f405287d9 100644 --- a/.serena/memories/ecosystem_and_related_projects.md +++ b/.serena/memories/ecosystem_and_related_projects.md @@ -7,7 +7,7 @@ |------|---------| | hugegraph-toolchain | Loader, Hubble (visualization), Tools CLI, Java Client | | hugegraph-computer | OLAP: PageRank, Connected Components, Shortest Path | -| incubator-hugegraph-ai | Graph RAG, KG construction, NL→Gremlin/Cypher | +| apache/hugegraph-ai | Graph RAG, KG construction, NL→Gremlin/Cypher | | hugegraph-doc | Docs & website (hugegraph.apache.org) | ## Data Flow @@ -20,4 +20,5 @@ Sources → hugegraph-loader → hugegraph-server → Hubble / Computer / AI - Queries: Gremlin (TinkerPop 3.5.1), OpenCypher, REST API + Swagger UI - Storage: RocksDB (default), HStore (distributed) -## Version: Server 1.7.0, TinkerPop 3.5.1, Java 11+ +## Versions +Read Server revision from root `pom.xml` and TinkerPop from `hugegraph-server/pom.xml`. diff --git a/.serena/memories/implementation_patterns_and_guidelines.md b/.serena/memories/implementation_patterns_and_guidelines.md index d04e33ce56..26d19ac26f 100644 --- a/.serena/memories/implementation_patterns_and_guidelines.md +++ b/.serena/memories/implementation_patterns_and_guidelines.md @@ -20,7 +20,7 @@ ## gRPC Protocol - PD protos: `hugegraph-pd/hg-pd-grpc/src/main/proto/` - Store protos: `hugegraph-store/hg-store-grpc/src/main/proto/` -- After `.proto` changes: `mvn clean compile` → `target/generated-sources/protobuf/` +- After `.proto` changes: `mvn clean compile`; PD/Store gRPC POMs output to each module's `src/main/java/`. ## Query Languages - **Gremlin**: Native TinkerPop 3.5.1 @@ -36,10 +36,10 @@ - **Backends in CI**: memory, rocksdb, hbase (matrix) - **Single test class**: `mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,memory -Dtest=ClassName` - TinkerPop tests: only on `release-*`/`test-*` branches -- Raft tests: only on `test*`/`raft*` branches +- Server Raft API tests are branch-gated in `server-ci.yml`; Store raft-core tests run in normal `pd-store-ci.yml` CI. ## Docker -- Single-node: `docker/docker-compose.yml` (bridge network, pd+store+server) +- Standalone: `docker/docker-compose.yml` (bridge network, Server + Hubble) - Cluster: `docker/docker-compose-3pd-3store-3server.yml` - Container logs: stdout-based diff --git a/.serena/memories/key_file_locations.md b/.serena/memories/key_file_locations.md index 5f2ade9424..ac10c668e8 100644 --- a/.serena/memories/key_file_locations.md +++ b/.serena/memories/key_file_locations.md @@ -16,7 +16,7 @@ - Tests: `hugegraph-test/src/main/java/.../` → `unit/`, `core/`, `api/`, `tinkerpop/` ## Docker -- `docker/docker-compose.yml` — Single-node (bridge network, pd+store+server) +- `docker/docker-compose.yml` — Standalone Server + Hubble (bridge network) - `docker/docker-compose-3pd-3store-3server.yml` — 3-node cluster - `docker/docker-compose.dev.yml` — Dev mode diff --git a/.serena/memories/project_overview.md b/.serena/memories/project_overview.md index 6375a3104e..a08aea9ec6 100644 --- a/.serena/memories/project_overview.md +++ b/.serena/memories/project_overview.md @@ -22,5 +22,5 @@ Apache HugeGraph is a fast-speed, highly-scalable graph database supporting 10+ - **Storage**: RocksDB (default/embedded), HStore (distributed/production), HBase (deprecated; planned for removal in 2.0) ## Version -- Current: 1.7.0 (`${revision}` property, Maven flatten plugin) +- Read `${revision}` from the root `pom.xml`; Maven flatten resolves it for published artifacts. - License: Apache License 2.0 diff --git a/.serena/memories/suggested_commands.md b/.serena/memories/suggested_commands.md index 346304432f..c1aa4d8e5b 100644 --- a/.serena/memories/suggested_commands.md +++ b/.serena/memories/suggested_commands.md @@ -5,7 +5,7 @@ mvn clean install -DskipTests # Full build mvn clean install -pl hugegraph-server -am -DskipTests # Server only mvn clean compile -U -Dmaven.javadoc.skip=true -ntp # Compile only -mvn clean package -DskipTests # Distribution → install-dist/target/ +mvn clean package -DskipTests # Aggregate distribution → root target/ ``` ## Test @@ -19,7 +19,7 @@ mvn test -pl hugegraph-server/hugegraph-test -am -P api-test,rocksdb # Single test class mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,memory -Dtest=YourTestClass -# TinkerPop compliance (release/test branches only) +# TinkerPop compliance (can be run locally on any branch) mvn test -pl hugegraph-server/hugegraph-test -am -P tinkerpop-structure-test,memory # PD/Store (build struct first) @@ -45,8 +45,9 @@ bin/enable-auth.sh # Enable auth ``` ## Docker +Create the required authentication environment first; follow `docker/README.md`. ```bash -cd docker && docker compose up -d # Single-node (bridge network) +cd docker && docker compose up -d # Standalone Server + Hubble (bridge network) cd docker && docker compose -f docker-compose-3pd-3store-3server.yml up -d # Cluster ``` diff --git a/.serena/memories/task_completion_checklist.md b/.serena/memories/task_completion_checklist.md index bcee316d5a..22a12b4db7 100644 --- a/.serena/memories/task_completion_checklist.md +++ b/.serena/memories/task_completion_checklist.md @@ -1,6 +1,8 @@ # Task Completion Checklist -## 1. Code Quality (MANDATORY) +## 1. Validation scope +Documentation-only changes: check links, paths and `git diff --check`. +For code changes, select relevant checks below and follow root `AGENTS.md` before pushing. ```bash mvn apache-rat:check -ntp # License headers mvn editorconfig:check # Style (.editorconfig) @@ -10,7 +12,7 @@ mvn clean compile -Dmaven.javadoc.skip=true # Compile warnings ## 2. Testing - Choose backend: `memory` (fast), `rocksdb` (realistic), `hbase` (deprecated compatibility) -- Single test: `-Dtest=ClassName` works with all profiles +- Single-test selection: check the module test POM and confirm the requested test actually ran. - Bug fix → existing tests; New feature → write tests; Refactor → affected module tests ## 3. Dependencies (if adding new) @@ -23,10 +25,13 @@ mvn clean compile -Dmaven.javadoc.skip=true # Compile warnings - `server-ci.yml`: memory/rocksdb/hbase × Java 11 - `rerun-ci.yml`: auto-retries flaky failures - `licence-checker.yml`: header validation -- Raft tests: only `test*`/`raft*` branches +- Server Raft API tests are branch-gated; Store raft-core and core tests run in normal PD/Store CI. - TinkerPop tests: only `release-*`/`test-*` branches -## 5. Commit +## 5. Documentation +- Follow root `AGENTS.md`: user-visible feature/configuration/deployment changes ship with matching docs. + +## 6. Commit - NEVER commit unless explicitly asked - Format: `feat|fix|refactor(module): msg` - Include issue ID if available diff --git a/AGENTS.md b/AGENTS.md index 07daf17662..a38801710f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,108 +1,62 @@ # AGENTS.md -Single source of truth for AI coding agents. -README.md covers human-facing deployment/ecosystem context; only consult it on demand. +Repository-specific guidance; module files add local details. -## Stack & Modules +## Key relationships -Apache HugeGraph — Apache TinkerPop 3 compliant graph database. -Java 11+, Maven 3.5+. Version managed via `${revision}` (currently `1.8.0`). +```text +Server (hugegraph-server): graph engine + REST/Gremlin API + ├─ RocksDB: embedded backend + └─ HStore adapter → Store client → Store: partition data + Raft +PD: placement + metadata; consulted by Server and Store +Struct: shared types/codecs; separate implementations also exist in Server +Commons: shared utilities; independent compiler and test settings ``` -Client (Gremlin / Cypher / REST) - │ -Server = hugegraph-server - ├─ hugegraph-api REST, Gremlin/Cypher, auth - ├─ hugegraph-core engine, schema, traversal, BackendStore interface - └─ Backend impls rocksdb (default, embedded) │ hstore (distributed) - ▼ - hugegraph-pd (placement) + hugegraph-store (Raft) -``` - -Top-level modules: `hugegraph-server` · `hugegraph-pd` · `hugegraph-store` · -`hugegraph-commons` (shared utils & RPC) · `hugegraph-struct` (data types; dep of PD/Store). -Server submodules worth knowing: `hugegraph-core`, `hugegraph-api`, -`hugegraph-rocksdb`, `hugegraph-hstore`, `hugegraph-test`, `hugegraph-dist`. +## Load on demand -## Code Search Anchors - -| Area | Path | +| Work area | Read when working there | |---|---| -| Graph engine | `hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/` | -| REST APIs | `hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/` | -| Backend interface | `hugegraph-server/hugegraph-core/.../backend/store/BackendStore.java` | -| Auth | `hugegraph-server/hugegraph-api/.../api/auth/` | -| gRPC protos | `hugegraph-{pd,store}/hg-{pd,store}-grpc/src/main/proto/` | - -Config roots (under each dist module's `src/assembly/static/conf/`): -- Server — `hugegraph.properties`, `rest-server.properties`, `gremlin-server.yaml` -- PD / Store — `application.yml` - -## Build +| Graph engine, API, backends | [Server](hugegraph-server/AGENTS.md) | +| Placement, metadata, PD client | [PD](hugegraph-pd/AGENTS.md) | +| Distributed storage and client | [Store](hugegraph-store/AGENTS.md) | +| Shared data types and serialization | [Struct](hugegraph-struct/AGENTS.md) | +| Utilities and RPC | [Commons](hugegraph-commons/AGENTS.md) | +| Container deployment | [Docker guide](docker/README.md) | +| PR requirements | [Contribution guide](docs/CONTRIBUTING.md) | + +## Repository constraints + +- Use existing module boundaries; keep unrelated refactors out of a fix. +- User-visible feature, configuration or deployment behavior changes must ship with matching docs. + Update in-repository docs in the same PR; link a paired `apache/hugegraph-doc` PR when website + docs are affected and coordinate both merges. A follow-up issue alone does not satisfy this rule. + Internal-only changes can use `Doc - No Need`. +- Keep README as an entry point; link detailed deployment instructions instead of duplicating them. +- Keep AGENTS.md under 100 lines where practical. Shared rules belong here, local exceptions + in module files. Keep key relationships, common commands and non-obvious pitfalls inline; + reference existing sources for versions, configuration details and lengthy procedures. + +## Build and validation + +Commands run from the repository root. Java 11+ and Maven 3.5+; +versions come from [pom.xml](pom.xml), including `${revision}`. ```bash -# All modules mvn clean install -DskipTests - -# Single module mvn clean install -pl hugegraph-server -am -DskipTests ``` -Distributed build order (for HStore-enabled dev): - -```bash -mvn install -pl hugegraph-struct -am -DskipTests # 1. shared data types -mvn clean package -pl hugegraph-pd -am -DskipTests # 2. placement driver -mvn clean package -pl hugegraph-store -am -DskipTests # 3. distributed storage -mvn clean package -pl hugegraph-server -am -DskipTests # 4. server -``` - -Runtime scripts (human-run) live in `hugegraph-server/hugegraph-dist/src/assembly/static/bin/`: -`init-store.sh`, `start-hugegraph.sh`, `stop-hugegraph.sh`. - -## Testing - -Server tests implicitly prefix `mvn test -pl hugegraph-server/hugegraph-test -am`: - -| Profile | Suffix | -|---|---| -| Unit | `-P unit-test` | -| Core | `-P core-test,rocksdb` (swap `rocksdb` for `memory`) | -| API | `-P api-test,rocksdb` | -| TinkerPop structure / process | `-P tinkerpop-{structure,process}-test,memory` | -| Single class | `-P core-test,rocksdb -Dtest=YourTestClass` | - -PD / Store tests (need `hugegraph-struct` installed first): - -```bash -mvn install -pl hugegraph-struct -am -DskipTests -mvn test -pl hugegraph-pd/hg-pd-test -am -mvn test -pl hugegraph-store/hg-store-test -am -``` - -Before writing new tests, check existing suites under `hugegraph-server/hugegraph-test/`. - -## Style & Pre-commit - -- Line 120, 4-space indent, LF, UTF-8, **no star imports** -- Commit format: `feat|fix|refactor(module): msg` -- Run before pushing: - ```bash - mvn editorconfig:format # enforce code style - mvn clean compile -Dmaven.javadoc.skip=true # surface warnings - ``` - -## Cross-module notes - -- `.proto` edits: `mvn clean compile` regenerates gRPC stubs under - `target/generated-sources/protobuf/` (output packages `*/grpc/` are excluded from Apache RAT). -- Adding a third-party dep: update `install-dist/release-docs/{LICENSE,NOTICE,licenses/}` - and `install-dist/scripts/dependency/known-dependencies.txt`. -- `hugegraph-commons` is shared by every module; `hugegraph-struct` must precede PD/Store; - server backends depend on `hugegraph-core`. - -## Additional context files - -`.serena/memories/` — notably `suggested_commands.md` and `task_completion_checklist.md` -when a task needs depth beyond this file. +For separate distributed-module builds, install `hugegraph-struct` first, +then build PD, Store and Server. Module guidance covers tests and CI prerequisites. +Commons tests require `-DskipCommonsTests=false`; a successful build does not imply its tests ran. + +- Java style: 120 columns, 4 spaces, no star imports; see [.editorconfig](.editorconfig). +- Before pushing code: `mvn editorconfig:format`, + `mvn clean compile -Dmaven.javadoc.skip=true`, and relevant module tests. + Documentation-only changes need link/path checks and `git diff --check`. +- PD/Store `.proto` builds generate Java into each `hg-*-grpc/src/main/java/` directory; + see the corresponding gRPC module POM before editing or cleaning generated sources. +- New dependencies require updating [release LICENSE/NOTICE/licenses](install-dist/release-docs/) + and [dependency inventory](install-dist/scripts/dependency/known-dependencies.txt). diff --git a/README.md b/README.md index e84260f5cc..0e48ced340 100644 --- a/README.md +++ b/README.md @@ -315,7 +315,7 @@ For comprehensive documentation, visit the [HugeGraph Documentation](https://hug | Understand configuration | `hugegraph-dist` | `src/assembly/static/conf/` | | Run tests | `hugegraph-test` | Test suites with multiple profiles | -For detailed architecture and development guidance, see [AGENTS.md](AGENTS.md). +For repository constraints and module guidance, see [AGENTS.md](AGENTS.md).
For Contributors @@ -323,7 +323,7 @@ For detailed architecture and development guidance, see [AGENTS.md](AGENTS.md). **New to HugeGraph?** Follow this path to get started: 1. **Understand the Architecture** - - Read [AGENTS.md](AGENTS.md) for detailed module structure and development patterns + - Read [AGENTS.md](AGENTS.md) for key component relationships and module guidance - Review the [Architecture Diagram](#architecture) above 2. **Set Up Your Environment** diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 7056afd978..bfa9d7e66f 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -125,6 +125,13 @@ Note that since GitHub requires submitting code through `username + token` (inst Go to the web page of GitHub fork repo, there would be a chance to create a Pull Request after pushing to a new branch, click the button "Compare & pull request" to do it. Then edit the description for proposed changes, which can just be copied from the commit message. +User-visible feature, configuration or deployment behavior changes must ship with matching +documentation. Include repository documentation in the same PR. When website documentation +is affected, link a paired [hugegraph-doc](https://github.com/apache/hugegraph-doc) PR and +coordinate both merges. A follow-up issue alone does not replace required documentation. +In the PR template, mark pending documentation as `Doc - TODO` until it is ready; +use `Doc - Done` with the documentation files or PR link, or `Doc - No Need` for internal-only changes. + Note: please make sure the email address you used to submit the code is bound to the GitHub account. For how to bind the email address, please refer to https://github.com/settings/emails: image diff --git a/hugegraph-commons/AGENTS.md b/hugegraph-commons/AGENTS.md index c21ab4a5dc..179e01300c 100644 --- a/hugegraph-commons/AGENTS.md +++ b/hugegraph-commons/AGENTS.md @@ -1,285 +1,32 @@ -# AGENTS.md +# Commons guidance -This file provides guidance to an AI coding tool when working with code in this repository. +Shared rules: [root AGENTS.md](../AGENTS.md). Paths here are relative to this module. -## Project Overview +## Entry points -hugegraph-commons is a shared utility module for Apache HugeGraph and its peripheral components. It provides core infrastructure components (locks, config, events, iterators, REST client, RPC framework) to simplify development across the HugeGraph ecosystem. +- Config, locks, events, iterators and REST: `hugegraph-common/src/main/java/org/apache/hugegraph/`. +- RPC framework: `hugegraph-rpc/`, which depends on `hugegraph-common`. +- Commons targets Java 8 bytecode in [pom.xml](pom.xml), although CI uses JDK 11. + Do not infer its language/API baseline from Server's JDK requirement. -**Technology Stack**: -- Java 8+ (compiler source/target: 1.8) -- Apache Maven 3.5+ -- Apache Commons Configuration2 for config management -- OkHttp 4.10.0 for REST client (hugegraph-common) -- Sofa-RPC 5.7.6 for RPC framework (hugegraph-rpc) -- JUnit 4.13.1 and Mockito 4.1.0 for testing +## Tests -## Architecture - -### Two-Module Structure - -This is a Maven multi-module project with 2 main modules: - -1. **hugegraph-common**: Core utilities library - - Lock implementations (atomic, key, row, lock groups) - - Configuration system with type-safe options - - Event hub for async notifications - - Iterator utilities (map, filter, flat-map, batch) - - RESTful client (OkHttp-based) - - Utilities (perf analysis, version checking, collections, logging) - - License management - -2. **hugegraph-rpc**: RPC communication framework - - Sofa-RPC based client/server implementation - - Consumer and provider configuration - - Service registration and discovery - - **Depends on hugegraph-common** - -### Key Design Patterns - -1. **Type-Safe Configuration System**: `HugeConfig` + `OptionSpace` pattern - - Config options defined as typed `ConfigOption` objects - - Supports both `.properties` and `.yaml` files - - Options organized in `OptionSpace` groups for validation - - Security checks on load - -2. **Lock Hierarchy**: Multiple lock implementations for different use cases - - `AtomicLock`: Basic atomic locking - - `KeyLock`: Lock by specific key - - `RowLock`: Row-level locking for table-like structures - - `LockGroup`: Manage multiple related locks - - `LockManager`: Central lock coordination - -3. **Event System**: Async event notification - - `EventHub`: Central event dispatcher - - `EventListener`: Typed event handlers - - Thread-safe event publishing - -4. **Iterator Composition**: Chainable iterator wrappers - - `MapperIterator`, `FilterIterator`, `LimitIterator` - - `FlatMapperIterator` for nested iteration - - `BatchMapperIterator` for batch processing - - All extend `ExtendableIterator` base - -5. **RPC Architecture**: Sofa-RPC abstraction layer - - `RpcServer`: Service provider side - - `RpcClientProvider`: Service consumer side - - `RpcProviderConfig`/`RpcConsumerConfig`: Configuration wrappers - - Supports multiple protocols (bolt, rest, grpc) - -## Build & Development Commands - -### Prerequisites -```bash -# Verify Java version (8+ required) -java -version - -# Verify Maven version (3.5+ required) -mvn -version -``` - -### Build Commands - -```bash -# Clean build without tests (fastest) -mvn clean install -DskipTests - -# Build with tests enabled -mvn clean install - -# Build specific module only -mvn clean install -pl hugegraph-common -DskipTests -mvn clean install -pl hugegraph-rpc -am -DskipTests # -am includes dependencies - -# Compile with warnings visible -mvn clean compile -Dmaven.javadoc.skip=true -``` - -**Note**: Tests are skipped by default via `true` in pom.xml. To run tests, override with `-DskipCommonsTests=false`. - -### Testing +Tests are skipped by default through `skipCommonsTests`. After installing dependencies, +run from the repository root: ```bash -# Run all tests (override default skip) -mvn test -DskipCommonsTests=false - -# Run tests for specific module -mvn test -pl hugegraph-common -DskipCommonsTests=false -mvn test -pl hugegraph-rpc -am -DskipCommonsTests=false - -# Run single test class -mvn test -pl hugegraph-common -Dtest=HugeConfigTest -DskipCommonsTests=false - -# Run test suite (includes all unit tests) -mvn test -pl hugegraph-common -Dtest=UnitTestSuite -DskipCommonsTests=false +mvn test -pl hugegraph-commons/hugegraph-common -Dtest=UnitTestSuite -DskipCommonsTests=false +mvn test -pl hugegraph-commons/hugegraph-rpc -Dtest=UnitTestSuite -DskipCommonsTests=false ``` -### Code Quality - -```bash -# License header check (Apache RAT) -mvn apache-rat:check - -# Checkstyle validation -mvn checkstyle:check - -# Both checks run automatically during validate phase -mvn validate -``` - -### Code Coverage - -```bash -# Generate JaCoCo coverage report -mvn clean test -DskipCommonsTests=false -# Report: target/jacoco/index.html -``` - -## Important File Locations - -### Source Code Structure -- hugegraph-common sources: `hugegraph-common/src/main/java/org/apache/hugegraph/` - - `concurrent/`: Lock implementations - - `config/`: Configuration system (HugeConfig, OptionSpace, ConfigOption) - - `event/`: Event hub and listeners - - `iterator/`: Iterator utilities - - `rest/`: REST client implementation - - `util/`: Various utilities (collections, logging, version, etc.) - - `perf/`: Performance measurement (PerfUtil, Stopwatch) - - `license/`: License management - -- hugegraph-rpc sources: `hugegraph-rpc/src/main/java/org/apache/hugegraph/` - - `rpc/`: RPC server and client implementations - - `config/`: RPC-specific config options - -### Test Structure -- Unit tests: `hugegraph-{module}/src/test/java/org/apache/hugegraph/unit/` -- Test suites: `UnitTestSuite.java` lists all test classes -- Test utilities: `hugegraph-common/src/main/java/org/apache/hugegraph/testutil/` - - `Whitebox`: Reflection utilities for testing private members - - `Assert`: Enhanced assertion utilities - -### Configuration Files -- Parent POM: `pom.xml` (defines all dependencies and versions) -- Module POMs: `hugegraph-{module}/pom.xml` -- Test resources: `hugegraph-rpc/src/test/resources/*.properties` -- Checkstyle config: `style/checkstyle.xml` (referenced in parent POM) - -### Version Management -- Version property: `${revision}` in pom.xml (currently 1.7.0) -- Version classes: - - `hugegraph-common/src/main/java/org/apache/hugegraph/version/CommonVersion.java` - - `hugegraph-rpc/src/main/java/org/apache/hugegraph/version/RpcVersion.java` -- **IMPORTANT**: When changing version in pom.xml, also update version in these Java files - -## Development Workflow - -### Module Dependencies - -**Dependency order**: -``` -hugegraph-common (no internal dependencies) - ↓ -hugegraph-rpc (depends on hugegraph-common) -``` - -When making changes: -- Changes to `hugegraph-common` require rebuilding `hugegraph-rpc` -- Always build common first: `mvn install -pl hugegraph-common -DskipTests` -- Then build rpc: `mvn install -pl hugegraph-rpc -am -DskipTests` - -### Working with Configuration System - -When adding new configuration options: -1. Define `ConfigOption` in appropriate config class (e.g., `RpcOptions.java`) -2. Register option in an `OptionSpace` for validation -3. Load via `HugeConfig.get(option)` or `config.get(option)` -4. Example: -```java -public static final ConfigOption RPC_SERVER_HOST = - new ConfigOption<>("rpc.server_host", "...", "127.0.0.1"); -``` - -### Working with RPC Framework - -RPC configuration pattern: -- Server side: Create `RpcServer` with `HugeConfig` -- Client side: Use `RpcClientProvider` for service proxies -- Config files: Properties format with `rpc.*` keys -- Protocol: Default is Sofa-Bolt (binary protocol) - -### Testing Patterns - -1. **Unit tests** extend `BaseUnitTest` (for common module) or use standard JUnit -2. **Test organization**: Tests mirror source package structure -3. **Naming**: `{ClassName}Test.java` for unit tests -4. **Mocking**: Use Mockito for external dependencies -5. **Reflection testing**: Use `Whitebox.setInternalState()` for private field access - -### Adding Dependencies - -When adding third-party dependencies: -1. Add to `dependencyManagement` section in parent pom if used by multiple modules -2. Declare version in `` section -3. Add license info to `hugegraph-dist/release-docs/licenses/` -4. Update `hugegraph-dist/release-docs/LICENSE` -5. Update `hugegraph-dist/release-docs/NOTICE` if upstream has NOTICE - -## Common Workflows - -### Running a Specific Test - -```bash -# Single test class -mvn test -pl hugegraph-common -Dtest=HugeConfigTest -DskipCommonsTests=false - -# Single test method -mvn test -pl hugegraph-common -Dtest=HugeConfigTest#testGetOption -DskipCommonsTests=false - -# Pattern matching -mvn test -pl hugegraph-common -Dtest=*ConfigTest -DskipCommonsTests=false -``` - -### Debugging Tips - -1. **Enable debug logging**: Modify `log4j2.xml` in test resources -2. **Maven debug output**: Add `-X` flag to any Maven command -3. **Skip checkstyle temporarily**: Add `-Dcheckstyle.skip=true` -4. **Force dependency updates**: `mvn clean install -U` - -### Working with Parent POM - -This module has a parent POM (`../pom.xml` - hugegraph main project). If working standalone: -- The `` property comes from parent (1.7.0) -- Flatten Maven plugin resolves `${revision}` to actual version -- `.flattened-pom.xml` is auto-generated (excluded from RAT checks) - -## Special Notes - -### Version Synchronization - -Three places to update when changing version: -1. `pom.xml`: `` property -2. `hugegraph-common/.../CommonVersion.java`: Update version constant -3. `hugegraph-rpc/.../RpcVersion.java`: Update version constant - -### REST Client Implementation - -The REST client in `hugegraph-common/rest/` uses OkHttp (not Jersey as older docs suggest): -- Switched from Jersey to OkHttp in recent versions -- Supports connection pooling, timeouts, interceptors -- See `AbstractRestClient.java` for base implementation - -### RPC Version Note - -hugegraph-rpc uses Sofa-RPC 5.7.6 which has known security issues. There's a TODO to upgrade to 5.12+: -- See comment in `hugegraph-rpc/pom.xml:65-66` -- This is a known technical debt item +- Tests live in each module's `src/test/java/`. +- CI selects `UnitTestSuite`; add new tests to the relevant suite. +- Full setup: [Commons CI](../.github/workflows/commons-ci.yml). -### Checkstyle Configuration +## Compatibility -Checkstyle runs during `validate` phase by default: -- Config: `style/checkstyle.xml` -- Failures block the build -- Skip with `-Dcheckstyle.skip=true` for quick iteration -- Always fix before committing +- Utility behavior is shared across components, including iterator cleanup, + configuration parsing and lock semantics. +- Version changes: check `CommonVersion.java` and `RpcVersion.java` alongside the POM. +- Details on demand: [Common README](hugegraph-common/README.md) and + [RPC README](hugegraph-rpc/README.md). diff --git a/hugegraph-pd/AGENTS.md b/hugegraph-pd/AGENTS.md index aaaa861f39..57feb79304 100644 --- a/hugegraph-pd/AGENTS.md +++ b/hugegraph-pd/AGENTS.md @@ -1,376 +1,32 @@ -# AGENTS.md +# PD guidance -This file provides guidance to an AI coding tool when working with code in this repository. +Shared rules: [root AGENTS.md](../AGENTS.md). Paths here are relative to this module. -## Project Overview +## Entry points -HugeGraph PD (Placement Driver) is a meta server for distributed HugeGraph deployments, responsible for: -- Service discovery and registration -- Partition information storage and management -- Store node monitoring and scheduling -- Metadata coordination using Raft consensus +- Partition allocation and Store registration: `hg-pd-core/src/main/java/org/apache/hugegraph/pd/`. +- Metadata persistence and Raft: its `meta/` and `raft/` packages. +- RPC/REST handlers: `hg-pd-service/`; discovery, locks and watches: `hg-pd-client/`. +- Wire definitions: [protos](hg-pd-grpc/src/main/proto/). +- Shipped config: [distribution conf](hg-pd-dist/src/assembly/static/conf/). + Container endpoints/peers: [Docker guide](../docker/README.md). -**Technology Stack**: -- Java 11+ (required) -- Apache Maven 3.5+ -- gRPC + Protocol Buffers for RPC communication -- JRaft (Ant Design's Raft implementation) for consensus -- RocksDB for metadata persistence -- Spring Boot for REST APIs and dependency injection +## Build and tests -## Module Architecture - -HugeGraph PD is a Maven multi-module project with 8 modules: - -``` -hugegraph-pd/ -├── hg-pd-grpc # Protocol Buffers definitions and generated stubs -├── hg-pd-common # Shared utilities and common interfaces -├── hg-pd-core # Core PD logic (Raft, metadata, services) -├── hg-pd-service # gRPC service implementations and REST APIs -├── hg-pd-client # Java client library for PD -├── hg-pd-cli # Command-line interface tools -├── hg-pd-test # Integration and unit tests -└── hg-pd-dist # Distribution assembly (bin scripts, configs) -``` - -### Module Dependency Flow - -``` -hg-pd-grpc (protos) - ↓ -hg-pd-common (utilities) - ↓ -hg-pd-core (Raft + metadata stores) - ↓ -hg-pd-service (gRPC + REST endpoints) - ↓ -hg-pd-dist (assembly) - -hg-pd-client (depends on hg-pd-grpc, hg-pd-common) -hg-pd-cli (depends on hg-pd-client) -hg-pd-test (depends on hg-pd-core, hg-pd-service) -``` - -### Core Components (hg-pd-core) - -**Metadata Stores** (`meta/` package): -- `MetadataRocksDBStore`: RocksDB-backed persistence layer -- `PartitionMeta`: Partition assignment and shard group management -- `StoreInfoMeta`: Store node information and health status -- `TaskInfoMeta`: Distributed task coordination -- `IdMetaStore`, `ConfigMetaStore`, `DiscoveryMetaStore`: Domain-specific metadata - -**Services** (root package): -- `PartitionService`: Partition allocation, balancing, and splitting -- `StoreNodeService`: Store registration, heartbeat processing, status monitoring -- `StoreMonitorDataService`: Metrics collection and time-series data storage -- `TaskScheduleService`: Automated partition patrol and rebalancing -- `KvService`, `IdService`, `ConfigService`, `LogService`: Utility services - -**Raft Layer** (`raft/` package): -- `RaftEngine`: Raft group management and leadership -- `RaftStateMachine`: State machine applying metadata operations -- `RaftTaskHandler`: Async task execution via Raft proposals -- `KVOperation`, `KVStoreClosure`: Raft operation abstractions - -**Service Layer** (hg-pd-service): -- `ServiceGrpc`: Main gRPC service endpoint (partition, store, discovery RPCs) -- `PartitionAPI`: REST API for partition management -- `PDPulseService`: Heartbeat and monitoring -- `DiscoveryService`: Service discovery and registration - -### gRPC Protocol Definitions - -Located in `hg-pd-grpc/src/main/proto/`: -- `pdpb.proto`: Main PD service (GetMembers, RegisterStore, GetPartition) -- `metapb.proto`: Metadata objects (Partition, Shard, Store, Graph) -- `meta.proto`: Store and partition metadata -- `discovery.proto`: Service discovery protocol -- `kv.proto`: Key-value operations -- `pd_pulse.proto`: Heartbeat and monitoring -- `pd_watch.proto`: Watch notifications -- `metaTask.proto`: Task coordination - -**Important**: Generated Java code from `.proto` files is excluded from source control (`.gitignore`) and Apache RAT checks (Jacoco config). Regenerate after proto changes. - -## Build & Development Commands - -### Building PD Module - -```bash -# From hugegraph root directory, build PD and dependencies -mvn clean package -pl hugegraph-pd -am -DskipTests - -# From hugegraph-pd directory, build all modules -mvn clean install -DskipTests - -# Build with tests -mvn clean install - -# Build distribution package only -mvn clean package -pl hg-pd-dist -am -DskipTests -# Output: hugegraph-pd/apache-hugegraph-pd-.tar.gz -``` - -### Running Tests - -PD tests use Maven profiles defined in `pom.xml`: - -```bash -# All tests (default profiles active) -mvn test - -# Specific test profile -mvn test -P pd-core-test -mvn test -P pd-common-test -mvn test -P pd-client-test -mvn test -P pd-rest-test - -# Single test class (from hugegraph-pd directory) -mvn test -pl hg-pd-test -am -Dtest=StoreNodeServiceTest -mvn test -pl hg-pd-test -am -Dtest=PartitionServiceTest - -# From hugegraph root directory -mvn test -pl hugegraph-pd/hg-pd-test -am -``` - -Test files are located in `hg-pd-test/src/main/java/` (note: not `src/test/java`). - -### Regenerating gRPC Stubs - -```bash -# After modifying .proto files -mvn clean compile - -# Generated files location: -# target/generated-sources/protobuf/java/ -# target/generated-sources/protobuf/grpc-java/ -``` - -### Code Quality - -```bash -# License header check (Apache RAT) -mvn apache-rat:check - -# Clean build artifacts -mvn clean -# This also removes: *.tar, *.tar.gz, .flattened-pom.xml -``` - -## Running HugeGraph PD - -### Distribution Structure - -After building, extract the tarball: -``` -apache-hugegraph-pd-/ -├── bin/ -│ ├── start-hugegraph-pd.sh # Start PD server -│ ├── stop-hugegraph-pd.sh # Stop PD server -│ └── util.sh # Utility functions -├── conf/ -│ ├── application.yml # Main configuration -│ ├── application.yml.template # Configuration template -│ ├── log4j2.xml # Logging configuration -│ └── verify-license.json # License verification (optional) -├── lib/ # JAR dependencies -├── logs/ # Runtime logs -└── pd_data/ # RocksDB metadata storage (created at runtime) -``` - -### Starting PD - -```bash -cd apache-hugegraph-pd-/ -bin/start-hugegraph-pd.sh - -# With custom GC options -bin/start-hugegraph-pd.sh -g g1 - -# With custom JVM options -bin/start-hugegraph-pd.sh -j "-Xmx8g -Xms4g" - -# With OpenTelemetry enabled -bin/start-hugegraph-pd.sh -y true -``` - -Default ports: -- gRPC: 8686 (configure in `application.yml`: `grpc.port`) -- REST API: 8620 (configure in `application.yml`: `server.port`) -- Raft: 8610 (configure in `application.yml`: `raft.address`) - -JVM memory defaults (in `start-hugegraph-pd.sh`): -- Max heap: 32 GB -- Min heap: 512 MB - -### Stopping PD +Run from the repository root: ```bash -bin/stop-hugegraph-pd.sh -``` - -This sends SIGTERM to the PD process (tracked in `bin/pid`). - -## Key Configuration (application.yml) - -### Critical Settings for Distributed Deployment - -```yaml -grpc: - host: 127.0.0.1 # MUST change to actual IPv4 address in production - port: 8686 - -raft: - address: 127.0.0.1:8610 # This node's Raft address - peers-list: 127.0.0.1:8610 # Comma-separated list of all PD peers - # Example: 192.168.1.1:8610,192.168.1.2:8610,192.168.1.3:8610 - -pd: - data-path: ./pd_data # RocksDB metadata storage path - initial-store-count: 1 # Min stores required for cluster availability - initial-store-list: 127.0.0.1:8500 # Auto-activated store nodes (grpc_ip:grpc_port) - patrol-interval: 1800 # Partition rebalancing interval (seconds) - -partition: - default-shard-count: 1 # Replicas per partition (typically 3 in production) - store-max-shard-count: 12 # Max partitions per store - -store: - max-down-time: 172800 # Seconds before store is permanently offline (48h) - monitor_data_enabled: true # Enable metrics collection - monitor_data_interval: 1 minute # Metrics collection interval - monitor_data_retention: 1 day # Metrics retention period -``` - -### Common Configuration Errors - -1. **Raft peer discovery failure**: `raft.peers-list` must include all PD nodes' `raft.address` values -2. **Store connection issues**: `grpc.host` must be a reachable IP (not `127.0.0.1`) for distributed deployments. In Docker bridge networking, use the container hostname (e.g., `pd0`) set via `HG_PD_GRPC_HOST` env var. -3. **Split-brain scenarios**: Always run 3 or 5 PD nodes in production for Raft quorum -4. **Partition imbalance**: Adjust `patrol-interval` for faster/slower rebalancing - -## Development Workflows - -### Adding a New gRPC Service - -1. Define `.proto` messages and service in `hg-pd-grpc/src/main/proto/` -2. Run `mvn compile` to generate Java stubs -3. Implement service in `hg-pd-service/src/main/java/.../service/` -4. Register service in gRPC server initialization (check existing `ServiceGrpc.java` pattern) -5. Add client methods in `hg-pd-client/` if needed - -### Adding a New Metadata Store - -1. Create meta class in `hg-pd-core/src/main/java/.../meta/` -2. Use `MetadataRocksDBStore` as the underlying persistence layer -3. Implement metadata operations as Raft proposals via `RaftTaskHandler` -4. Add corresponding service methods in `hg-pd-core/.../Service.java` -5. Expose via gRPC in `hg-pd-service/` - -### Modifying Partition Logic - -- Core partition logic: `hg-pd-core/.../PartitionService.java` (69KB file, 2000+ lines) -- Key methods: - - `splitPartition()`: Partition splitting logic - - `updatePartitionLeader()`: Leader election handling - - `balancePartitions()`: Auto-balancing algorithm - - `getPartitionByCode()`: Partition routing -- All partition changes must go through Raft consensus -- Test with `hg-pd-test/.../core/PartitionServiceTest.java` - -### Debugging Raft Issues - -- Enable Raft logging in `conf/log4j2.xml`: Set `com.alipay.sofa.jraft` to DEBUG -- Check Raft state: Leader election happens in `RaftEngine.java` -- Raft snapshots stored in `pd_data/raft/snapshot/` -- Raft logs stored in `pd_data/raft/log/` - -## Testing Strategy - -### Test Organization - -Tests are in `hg-pd-test/src/main/java/` (non-standard location): -- `BaseTest.java`: Base class with common setup -- `core/`: Core service tests (PartitionService, StoreNodeService, etc.) -- Suite tests: `PDCoreSuiteTest.java` runs all core tests - -### Running Integration Tests - -```bash -# From hugegraph root, run PD integration tests +mvn install -pl hugegraph-struct -am -DskipTests mvn test -pl hugegraph-pd/hg-pd-test -am - -# These tests start embedded PD instances and verify: -# - Raft consensus and leader election -# - Partition allocation and balancing -# - Store heartbeat and monitoring -# - Metadata persistence and recovery -``` - -## Docker Deployment - -### Building Docker Image - -```bash -# From hugegraph root directory -docker build -f hugegraph-pd/Dockerfile -t hugegraph-pd:latest . ``` -The Dockerfile uses multi-stage build: -1. Stage 1: Build with Maven -2. Stage 2: Runtime with OpenJDK 11 - -### Running in Docker - -```bash -# Single PD node (development) -docker run -d -p 8620:8620 -p 8686:8686 -p 8610:8610 \ - -v /path/to/pd_data:/hugegraph-pd/pd_data \ - hugegraph-pd:latest - -# For production clusters, use Docker Compose or Kubernetes -# See: ../docker/docker-compose-3pd-3store-3server.yml and ../docker/README.md -``` - -Exposed ports: 8620 (REST), 8686 (gRPC), 8610 (Raft) - -## Cross-Module Dependencies - -When working on PD: -- **hugegraph-struct**: Protocol definitions shared with hugegraph-store - - Build struct first: `mvn install -pl hugegraph-struct -am -DskipTests` - - Required before building PD if struct changed -- **hugegraph-commons**: RPC framework, locks, and configuration utilities - - Changes to commons may affect PD's `hg-pd-common` module - -## CI/CD Integration - -PD tests run in `pd-store-ci.yml` GitHub Actions workflow: -- Triggered on pushes to PD module files -- Runs `mvn test -pl hugegraph-pd/hg-pd-test -am` -- JaCoCo coverage excludes generated gRPC code and config classes - -## Important Notes - -### Generated Code Exclusions -- `hg-pd-grpc/src/main/java/` is excluded from git (see `.gitignore`) -- Apache RAT skips `**/grpc/**.*` (see `pom.xml` Jacoco config) -- Always run `mvn clean compile` after pulling proto changes - -### Raft Consensus Requirements -- PD uses JRaft for distributed consensus -- All metadata writes are Raft proposals (see `KVOperation`, `KVStoreClosure`) -- Raft group requires 3 or 5 nodes for fault tolerance in production -- Single-node mode (peers-list with one address) is for development only +- Tests live in `hg-pd-test/src/main/java/`, not `src/test/java/`. +- Test profiles in the [parent POM](pom.xml) activate matching Surefire executions; + suite selection is configured in the [test POM](hg-pd-test/pom.xml). +- Service setup: [PD/Store CI](../.github/workflows/pd-store-ci.yml). -### Store Interaction -- PD does not store graph data; it only stores metadata about store nodes and partitions -- Actual graph data resides in hugegraph-store nodes -- PD coordinates store nodes but doesn't handle data plane traffic +## Cross-module traps -### Version Compatibility -- PD version must match hugegraph-server and hugegraph-store versions -- Version managed via `${revision}` property (inherited from parent POM) -- Current version: 1.7.0 +- Trace metadata mutations through `RaftTaskHandler` / `KVOperation` and the state machine. +- Client changes affect Server and Store; inspect watch/lock callers before changing + transport lifecycle or timeout behavior. diff --git a/hugegraph-server/AGENTS.md b/hugegraph-server/AGENTS.md index 0b6da40820..d093c2c138 100644 --- a/hugegraph-server/AGENTS.md +++ b/hugegraph-server/AGENTS.md @@ -1,258 +1,35 @@ -# AGENTS.md +# Server guidance -This file provides guidance to an AI coding tool when working with code in this repository. +Shared rules: [root AGENTS.md](../AGENTS.md). Paths here are relative to this module. -## Project Overview +## Entry points -HugeGraph Server is the graph engine layer of Apache HugeGraph, consisting of: -- **REST API Layer** (hugegraph-api): RESTful APIs for graph operations, Gremlin/Cypher queries, schema management, and authentication -- **Graph Engine Layer** (hugegraph-core): TinkerPop 3 implementation, schema management, traversal optimization, task scheduling -- **Backend Interface**: Abstraction layer for pluggable storage backends -- **Storage Backend Implementations**: RocksDB (default), HStore (distributed), HBase (deprecated; planned for removal in 2.0), and Memory (test-only) +- Engine/schema/traversal: `hugegraph-core/src/main/java/org/apache/hugegraph/`. +- REST, auth and graph lifecycle: `hugegraph-api/src/main/java/org/apache/hugegraph/`. +- Backend contract: `hugegraph-core/src/main/java/org/apache/hugegraph/backend/store/BackendStore.java`. +- HStore adapter: `hugegraph-hstore/`; distributed client lives in [Store](../hugegraph-store/AGENTS.md). +- Shipped configs/scripts: [distribution sources](hugegraph-dist/src/assembly/static/). + Graph configs are in `conf/graphs/`; REST and Gremlin configs are in `conf/`. -Technology: Java 11+, Maven 3.5+, Apache TinkerPop 3.5.1, Jersey 3.0 (REST), gRPC (distributed communication) +## Tests -## Build Commands +Run from the repository root: -### Full Build ```bash -# Build all modules (from hugegraph-server directory) -mvn clean install -DskipTests - -# Build with tests -mvn clean install - -# Build specific module -mvn clean install -pl hugegraph-core -am -DskipTests -``` - -### Code Quality -```bash -# Run checkstyle validation -mvn checkstyle:check - -# Checkstyle runs automatically during 'validate' phase -# Configuration: ../style/checkstyle.xml -``` - -## Testing - -### Running Tests - -**Test profiles** (`-P` flag): -- `core-test` (default): Core graph engine tests -- `unit-test`: Unit tests only (memory backend) -- `api-test`: REST API tests -- `tinkerpop-structure-test`: TinkerPop structure compliance -- `tinkerpop-process-test`: TinkerPop process compliance - -**Backend profiles** (combine with test profiles): -- `memory`: In-memory backend (default for tests) -- `rocksdb`: RocksDB backend -- `hbase`: HBase backend (deprecated; planned for removal in 2.0) -- `hstore`: HStore backend - -```bash -# Unit tests (from hugegraph-server/) -mvn test -pl hugegraph-test -am -P unit-test - -# Core tests with RocksDB backend -mvn test -pl hugegraph-test -am -P core-test,rocksdb - -# API tests with memory backend -mvn test -pl hugegraph-test -am -P api-test,memory - -# Run specific test class -mvn test -pl hugegraph-test -am -P core-test,memory -Dtest=YourTestClassName - -# TinkerPop compliance tests (for release validation) -mvn test -pl hugegraph-test -am -P tinkerpop-structure-test,memory -mvn test -pl hugegraph-test -am -P tinkerpop-process-test,memory -``` - -### Test Module Structure -All tests are in `hugegraph-test/` which depends on all other modules. Tests are organized by: -- Unit tests: `src/test/java/.../unit/` -- Core tests: `src/test/java/.../core/` -- API tests: `src/test/java/.../api/` - -## Module Structure - -Multi-module Maven project with 8 submodules: - -``` -hugegraph-server/ -├── hugegraph-core # Graph engine, TinkerPop impl, schema, backend interface -├── hugegraph-api # REST API, Gremlin/Cypher endpoints, authentication -├── hugegraph-rocksdb # RocksDB backend (default, embedded) -├── hugegraph-hstore # HStore backend (distributed, production) -├── hugegraph-hbase # HBase backend (deprecated; planned for removal in 2.0) -├── hugegraph-dist # Distribution packaging, scripts, configs -├── hugegraph-test # All test suites -└── hugegraph-example # Example code -``` - -### Key Package Structure (hugegraph-core) - -``` -org/apache/hugegraph/ -├── HugeGraph.java # Main graph interface -├── StandardHugeGraph.java # Core implementation -├── HugeFactory.java # Factory for graph instances -├── backend/ # Backend abstraction and implementations -│ ├── store/BackendStore.java # Storage backend interface -│ ├── serializer/ # Data serialization -│ └── tx/ # Transaction management -├── schema/ # Schema management (VertexLabel, EdgeLabel, etc.) -├── structure/ # Graph elements (Vertex, Edge, Property) -├── traversal/ # Gremlin traversal optimization -├── task/ # Async task scheduling and execution -├── auth/ # Authentication and authorization -├── job/ # Job management (rebuild, compact, etc.) -└── meta/ # Metadata management -``` - -### Key Package Structure (hugegraph-api) - -``` -org/apache/hugegraph/ -├── api/ # REST API endpoints -│ ├── graph/GraphAPI.java # Graph CRUD operations -│ ├── schema/ # Schema APIs -│ ├── gremlin/GremlinAPI.java # Gremlin query endpoint -│ ├── cypher/CypherAPI.java # Cypher query endpoint (via OpenCypher) -│ ├── auth/ # Authentication APIs -│ └── job/TaskAPI.java # Job/Task management APIs -├── server/RestServer.java # Jersey/Grizzly REST server -├── auth/ # Auth filters and handlers -└── metrics/ # Metrics collection +mvn test -pl hugegraph-server/hugegraph-test -am -P unit-test +mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,rocksdb +mvn test -pl hugegraph-server/hugegraph-test -am -P api-test,rocksdb ``` -## Architecture Patterns - -### Pluggable Backend Architecture -All storage backends implement the `BackendStore` interface from hugegraph-core. New backends can be added as separate modules without modifying core code. Backend selection is via `backend` property in `conf/hugegraph.properties`. - -### TinkerPop Compliance -Full Apache TinkerPop 3 implementation with: -- Custom traversal strategies in `hugegraph-core/src/main/java/org/apache/hugegraph/traversal/` -- Structure and process API implementations -- Gremlin groovy script support via `gremlin-groovy` integration - -### Multi-Language Query Support -- **Gremlin**: Native support via TinkerPop -- **Cypher**: OpenCypher implementation in `hugegraph-api/src/main/java/org/apache/hugegraph/opencypher/` - - Translates Cypher to Gremlin using `opencypher-gremlin` library - -### gRPC for Distributed Components -When working with HStore backend or distributed features: -- Protocol Buffer definitions: `hugegraph-core/src/main/resources/proto/` -- Generated code: `target/generated-sources/protobuf/java/` -- Regenerate after .proto changes: `mvn clean compile` - -### Authentication System -Authentication is **optional and disabled by default**: -- Enable: `bin/enable-auth.sh` or via configuration -- Implementation: `hugegraph-api/src/main/java/org/apache/hugegraph/auth/` -- Multi-level access control: Users, Groups, Projects, Targets, Permissions -- Core auth logic: `hugegraph-core/src/main/java/org/apache/hugegraph/auth/` - -## Running the Server - -After building, server distribution is in `hugegraph-dist/target/`. Scripts are in `hugegraph-dist/src/assembly/static/bin/`: - -```bash -# Initialize backend storage (first time only) -bin/init-store.sh - -# Start server (REST API on 8080, Gremlin on 8182) -bin/start-hugegraph.sh - -# Stop server -bin/stop-hugegraph.sh - -# Gremlin console (interactive) -bin/gremlin-console.sh - -# Monitor server status -bin/monitor-hugegraph.sh - -# Enable authentication -bin/enable-auth.sh -``` - -## Configuration Files - -Located in `hugegraph-dist/src/assembly/static/conf/`: -- **`hugegraph.properties`**: Main server configuration (backend, storage paths, cache) -- **`rest-server.properties`**: REST API settings (host, port, thread pool, SSL) -- **`gremlin-server.yaml`**: Gremlin server configuration (WebSocket, serializers) -- **`log4j2.xml`**: Logging configuration - -## Common Development Tasks - -### Adding a New REST API Endpoint -1. Create API class in `hugegraph-api/src/main/java/org/apache/hugegraph/api/` -2. Extend `ApiBase` or relevant base class -3. Use JAX-RS annotations (`@Path`, `@GET`, `@POST`, etc.) -4. Add Swagger/OpenAPI annotations for documentation -5. Add tests in `hugegraph-test/src/test/java/.../api/` - -### Adding Backend Support -1. Create new module: `hugegraph-{backend-name}/` -2. Add dependency on `hugegraph-core` -3. Implement `BackendStore` interface -4. Implement `BackendStoreProvider` for factory -5. Add backend module to parent `pom.xml` `` section -6. Add tests combining with test profiles - -### Modifying Schema or Graph Elements -- Schema definitions: `hugegraph-core/src/main/java/org/apache/hugegraph/schema/` -- Graph structure: `hugegraph-core/src/main/java/org/apache/hugegraph/structure/` -- Always consider backward compatibility for stored data -- Update serializers if changing storage format - -### Working with Transactions -Transaction management is in `hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/`: -- `GraphTransaction`: Vertex/Edge operations -- `SchemaTransaction`: Schema operations -- `IndexTransaction`: Index operations -- Follow the transaction lifecycle pattern in existing code - -## Important Notes - -### Code Style -- Checkstyle configuration: `../style/checkstyle.xml` -- Enforced during Maven `validate` phase -- Import code style into IDE (IntelliJ IDEA recommended) - -### License Headers -All Java files must have Apache License header. Verified via `maven-checkstyle-plugin`. - -### Version Management -- Version managed via `${revision}` property (currently 1.7.0) -- Uses `flatten-maven-plugin` for CI-friendly versioning -- Don't hardcode versions in module POMs - -### Cross-Module Dependencies -``` -hugegraph-api → hugegraph-core → hugegraph-commons (external) -hugegraph-{backend} → hugegraph-core -hugegraph-test → all modules -``` - -### Generated Code -- Protobuf Java classes: Generated, not manually edited -- Located in `target/generated-sources/` -- Excluded from checkstyle/license checks - -### Backend Selection at Runtime -Backends are loaded via ServiceLoader pattern. The `backend` property in `hugegraph.properties` determines which implementation is used. All backend JARs must be on classpath. +- Suite sources are in `hugegraph-test/src/main/java/`, not `src/test/java/`. +- Combine test and backend profiles. TinkerPop profiles: + `tinkerpop-structure-test`, `tinkerpop-process-test`. +- Selection and suite registration: [test POM](hugegraph-test/pom.xml). +- Service setup: [Server CI](../.github/workflows/server-ci.yml); + HStore setup: [PD/Store CI](../.github/workflows/pd-store-ci.yml). -## Debugging Tips +## Cross-module traps -- Detailed logging: Edit `hugegraph-dist/src/assembly/static/conf/log4j2.xml` -- View effective config: `bin/dump-conf.sh` -- Arthas diagnostics: Built-in (version 3.7.1) -- Backend state inspection: `bin/dump-store.sh` -- Raft cluster tools (for distributed): `bin/raft-tools.sh` +- Struct has separate ID/query/serializer implementations. For format changes, inspect + [Struct](../hugegraph-struct/AGENTS.md) and both read/write paths. +- Authentication spans REST and Gremlin; inspect both shipped configs when changing bootstrap. diff --git a/hugegraph-store/AGENTS.md b/hugegraph-store/AGENTS.md index 8b5ef46bab..c3bf63e5bf 100644 --- a/hugegraph-store/AGENTS.md +++ b/hugegraph-store/AGENTS.md @@ -1,312 +1,35 @@ -# AGENTS.md +# Store guidance -This file provides guidance to an AI coding tool when working with code in this repository. +Shared rules: [root AGENTS.md](../AGENTS.md). Paths here are relative to this module. -## Project Overview +## Entry points -HugeGraph Store is a distributed storage backend for Apache HugeGraph, using RocksDB as the underlying storage engine with Raft consensus protocol for distributed coordination. It is designed for production-scale deployments requiring high availability and horizontal scalability. +- Routing, retry and streaming: `hg-store-client/`. +- Partition lifecycle, Raft and snapshots: `hg-store-core/src/main/java/org/apache/hugegraph/store/`. +- RocksDB access: `hg-store-rocksdb/`; RPC/REST serving: `hg-store-node/`. +- Wire definitions: [protos](hg-store-grpc/src/main/proto/). +- Shipped configs/scripts: [distribution sources](hg-store-dist/src/assembly/static/). +- Server adapter: [HStore backend](../hugegraph-server/hugegraph-hstore/). -**Technology Stack**: -- Java 11+ -- RocksDB: Embedded key-value storage engine -- Raft (JRaft): Distributed consensus protocol -- gRPC: Inter-node communication -- Protocol Buffers: Data serialization +## Build and tests -## Architecture +Run from the repository root: -### Module Structure - -HugeGraph Store consists of 9 submodules: - -``` -hugegraph-store/ -├── hg-store-common # Shared utilities, constants, query abstractions -├── hg-store-grpc # gRPC protocol definitions (proto files) and generated stubs -├── hg-store-client # Client library for connecting to Store cluster -├── hg-store-rocksdb # RocksDB abstraction and optimizations -├── hg-store-core # Core storage logic, partition management -├── hg-store-node # Store node server implementation with Raft -├── hg-store-dist # Distribution packaging, scripts, configs -├── hg-store-cli # Command-line tools for cluster management -└── hg-store-test # Integration and unit tests -``` - -### Key Package Structure - -``` -org/apache/hugegraph/store/ -├── grpc/ # Generated gRPC stubs (do not edit manually) -├── client/ # Client API for Store operations -├── node/ # Store node server and Raft integration -├── core/ # Core storage abstractions -│ ├── store/ # Store interface and implementations -│ ├── partition/ # Partition management -│ └── raft/ # Raft consensus integration -├── rocksdb/ # RocksDB wrapper and optimizations -├── query/ # Query processing and aggregation -└── util/ # Common utilities -``` - -### Distributed Architecture - -Store operates as a cluster of nodes: -- **Store Nodes**: 3+ nodes (typically 3 or 5 for Raft quorum) -- **Raft Groups**: Data partitioned into Raft groups for replication -- **PD Coordination**: Requires hugegraph-pd for cluster metadata and partition assignment -- **Client Access**: hugegraph-server connects via hg-store-client - -## Build Commands - -### Prerequisites ```bash -# HugeGraph Store depends on hugegraph-struct -# Build struct module first from repository root -cd /path/to/hugegraph-org mvn install -pl hugegraph-struct -am -DskipTests +mvn test -pl hugegraph-store/hg-store-test -am ``` -### Full Build -```bash -# From hugegraph-store directory -mvn clean install -DskipTests - -# Build with tests -mvn clean install - -# Build specific module (e.g., client only) -mvn clean install -pl hg-store-client -am -DskipTests -``` - -### Testing - -**Test profiles** (defined in pom.xml): -- `store-client-test` (default): Client library tests -- `store-core-test` (default): Core storage tests -- `store-common-test` (default): Common utilities tests -- `store-rocksdb-test` (default): RocksDB abstraction tests -- `store-server-test` (default): Store node server tests -- `store-raftcore-test` (default): Raft consensus tests - -```bash -# Run all tests (from hugegraph-store/) -mvn test -pl hg-store-test -am - -# Run specific test class -mvn test -pl hg-store-test -am -Dtest=YourTestClassName - -# Run tests for specific module -mvn test -pl hg-store-core -am -mvn test -pl hg-store-client -am -``` - -### Code Quality -```bash -# License header check (Apache RAT) - from repository root -mvn apache-rat:check - -# EditorConfig validation - from repository root -mvn editorconfig:check -``` - -## Running Store Cluster - -Scripts are located in `hg-store-dist/src/assembly/static/bin/`: - -```bash -# Start Store node -bin/start-hugegraph-store.sh - -# Stop Store node -bin/stop-hugegraph-store.sh - -# Restart Store node -bin/restart-hugegraph-store.sh -``` - -**Important**: For a functional distributed cluster, you need: -1. HugeGraph PD cluster running (3+ nodes) -2. HugeGraph Store cluster (3+ nodes) -3. Proper configuration pointing Store nodes to PD cluster - -See Docker Compose examples in the repository root `../docker/` directory. Single-node quickstart (pre-built images): `../docker/docker-compose.yml`. Single-node dev build (from source): `../docker/docker-compose.dev.yml`. 3-node cluster: `../docker/docker-compose-3pd-3store-3server.yml`. See `../docker/README.md` for the full setup guide. - -## Configuration Files - -Located in `hg-store-dist/src/assembly/static/conf/`: - -- **`application.yml`**: Main Store node configuration - - RocksDB settings (data paths, cache sizes, compaction) - - Raft configuration (election timeout, snapshot interval) - - Network settings (gRPC ports) - - Store capacity and partition management - -- **`application-pd.yml`**: PD client configuration - - PD cluster endpoints - - Heartbeat intervals - - Partition query settings - -- **`log4j2.xml`**: Logging configuration - -## Important Development Notes - -### gRPC Protocol Definitions - -Protocol Buffer files are in `hg-store-grpc/src/main/proto/`: -- `store_common.proto` - Common data structures -- `store_session.proto` - Client-server session management -- `store_state.proto` - Cluster state and metadata -- `store_stream_meta.proto` - Streaming operations -- `graphpb.proto` - Graph data structures -- `query.proto` - Query operations -- `healthy.proto` - Health check endpoints - -**When modifying `.proto` files**: -1. Edit the `.proto` file in `hg-store-grpc/src/main/proto/` -2. Run `mvn clean compile` to regenerate Java stubs -3. Generated code appears in `target/generated-sources/protobuf/` -4. Generated files are excluded from license checks - -### Module Dependencies - -Build order matters due to dependencies: -``` -hugegraph-struct (external) - ↓ -hg-store-common - ↓ -hg-store-grpc → hg-store-rocksdb - ↓ -hg-store-core - ↓ -hg-store-client, hg-store-node - ↓ -hg-store-cli, hg-store-dist, hg-store-test -``` - -Always build `hugegraph-struct` first, then Store modules follow Maven reactor order. - -### Working with RocksDB - -Store uses RocksDB for persistent storage: -- Abstraction layer: `hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/` -- Column families for different data types -- Custom compaction and compression settings -- Optimized for graph workloads (vertices, edges, indexes) - -Configuration in `application.yml`: -- `rocksdb.data-path` - Data directory location -- `rocksdb.block-cache-size` - In-memory cache size -- `rocksdb.write-buffer-size` - Write buffer configuration - -### Raft Consensus Integration - -Store uses JRaft (Ant Financial's Raft implementation): -- Each partition is a Raft group with 3 replicas (typically) -- Leader election, log replication, snapshot management -- Configuration: `raft.*` settings in `application.yml` - -Key Raft operations: -- Snapshot creation and loading -- Log compaction -- Leadership transfer -- Membership changes - -### Client Development - -When working with `hg-store-client`: -- Client connects to PD to discover Store nodes -- Automatic failover and retry logic -- Connection pooling and load balancing -- Batch operations support - -Example usage in hugegraph-server: -- Backend: `hugegraph-server/hugegraph-hstore/` -- Client integration: Uses `hg-store-client` library - -### Partition Management - -Data is partitioned for distributed storage: -- Partition assignment managed by PD -- Partition splitting and merging (future feature) -- Partition rebalancing on node addition/removal -- Hash-based partition key distribution - -## Common Development Tasks - -### Adding New gRPC Service - -1. Define service in appropriate `.proto` file in `hg-store-grpc/src/main/proto/` -2. Add message definitions for request/response -3. Run `mvn clean compile` to generate stubs -4. Implement service in `hg-store-node/` server -5. Add client methods in `hg-store-client/` -6. Add tests in `hg-store-test/` - -### Modifying Storage Engine - -1. Core storage interfaces: `hg-store-core/src/main/java/org/apache/hugegraph/store/core/store/` -2. RocksDB implementation: `hg-store-rocksdb/` -3. Update Raft state machine if needed: `hg-store-node/src/main/java/org/apache/hugegraph/store/node/raft/` -4. Consider backward compatibility for stored data format - -### Adding Query Operations - -1. Query abstractions: `hg-store-common/src/main/java/org/apache/hugegraph/store/query/` -2. Aggregation functions: `hg-store-common/.../query/func/` -3. Update proto definitions if new query types needed -4. Implement in `hg-store-core/` and expose via gRPC - -### Cluster Testing - -For distributed cluster tests: -- Module: `hugegraph-cluster-test/` (repository root) -- Requires: PD cluster + Store cluster + Server instances -- Docker Compose recommended for local testing -- CI/CD: See `.github/workflows/cluster-test-ci.yml` - -## Debugging Tips - -- **Logging**: Edit `hg-store-dist/src/assembly/static/conf/log4j2.xml` for detailed logs -- **Raft State**: Check Raft logs and snapshots in data directory -- **RocksDB Stats**: Enable RocksDB statistics in `application.yml` -- **gRPC Tracing**: Enable gRPC logging for request/response debugging -- **PD Connection**: Verify Store can connect to PD endpoints -- **Health Checks**: Use gRPC health check service for node status - -## Cross-Repository Integration - -Store integrates with other HugeGraph components: - -1. **hugegraph-pd**: Cluster metadata and partition management - - Store registers with PD on startup - - PD assigns partitions to Store nodes - - Heartbeat mechanism for health monitoring - -2. **hugegraph-server**: Graph engine uses Store as backend - - Backend implementation: `hugegraph-server/hugegraph-hstore/` - - Uses `hg-store-client` for Store cluster access - - Configuration: `backend=hstore` in `hugegraph.properties` - -3. **hugegraph-commons**: Shared utilities - - RPC framework: `hugegraph-commons/hugegraph-rpc/` - - Common utilities: `hugegraph-commons/hugegraph-common/` - -## Version Management - -- Version managed via `${revision}` property (currently 1.7.0) -- Flatten Maven plugin for CI-friendly versioning -- Must match version of other HugeGraph components (server, PD) - -## Special Notes - -### Performance Tuning +- Tests live in `hg-store-test/src/main/java/`, not `src/test/java/`. +- Test profiles in the [parent POM](pom.xml) activate matching Surefire executions. + Check [test POM](hg-store-test/pom.xml) suite includes when adding tests. +- CoreSuiteTest and BatchGraphIsolationTest need separate JVM forks: Store shutdown leaves + singleton state closed. Preserve `reuseForks=false` in the core test execution. +- Test prerequisites: [PD/Store CI](../.github/workflows/pd-store-ci.yml). + Cluster setup: [Docker guide](../docker/README.md). -Key performance factors: -- RocksDB block cache size (memory) -- Raft batch size and flush interval -- gRPC connection pool size -- Partition count and distribution -- Network latency between nodes +## Cross-module traps -Refer to `application.yml` for tuning parameters. +- PD manages placement; Store owns graph data and partition Raft groups. + Routing or membership changes cross both components. +- Codec changes must agree with [Struct](../hugegraph-struct/AGENTS.md) and the Server adapter. diff --git a/hugegraph-struct/AGENTS.md b/hugegraph-struct/AGENTS.md index dea4f06288..49f27a31f7 100644 --- a/hugegraph-struct/AGENTS.md +++ b/hugegraph-struct/AGENTS.md @@ -1,273 +1,31 @@ -# AGENTS.md +# Struct guidance -This file provides guidance to an AI coding tool when working with code in this repository. +Shared rules: [root AGENTS.md](../AGENTS.md). Paths here are relative to this module. -## Module Overview +## Entry points -**hugegraph-struct** is a foundational data structures module that defines the core abstractions shared across HugeGraph distributed components. This module **must be built before hugegraph-pd and hugegraph-store** as they depend on its structure definitions. +Sources: `src/main/java/org/apache/hugegraph/`. -**Key Responsibilities**: -- Schema element definitions (VertexLabel, EdgeLabel, PropertyKey, IndexLabel) -- Graph element structures (BaseVertex, BaseEdge, BaseProperty) -- Binary serialization/deserialization for efficient storage and RPC -- Type system definitions (HugeType enum, data types, ID strategies) -- Query abstractions (Query, ConditionQuery, IdQuery, Aggregate) -- Chinese text analyzers (multiple implementations: Jieba, IK, HanLP, etc.) -- Authentication utilities (JWT token generation, constants) +- Schema metadata: `struct/schema/`; graph instances: `structure/`. +- IDs, binary serialization and conditions: `id/`, `serializer/`, `query/`. +- Type codes: `type/`; metadata client lifecycle: `meta/`. +- Dependency versions and compiler level: [pom.xml](pom.xml). -## Build Commands +## Build and tests -### Building This Module +Run from the repository root: ```bash -# From hugegraph-struct directory -mvn clean install -DskipTests - -# Build with tests (if any exist in future) -mvn clean install - -# From parent directory (hugegraph root) -mvn install -pl hugegraph-struct -am -DskipTests -``` - -### Dependency Chain - -This module is a **critical dependency** for distributed components: - -```bash -# Correct build order for distributed components: -# 1. Build hugegraph-struct first mvn install -pl hugegraph-struct -am -DskipTests - -# 2. Then build PD -mvn clean package -pl hugegraph-pd -am -DskipTests - -# 3. Then build Store -mvn clean package -pl hugegraph-store -am -DskipTests -``` - -## Code Architecture - -### Package Structure - -``` -org.apache.hugegraph/ -├── struct/schema/ # Schema element definitions -│ ├── SchemaElement # Base class for all schema types -│ ├── VertexLabel # Vertex label definitions -│ ├── EdgeLabel # Edge label definitions -│ ├── PropertyKey # Property key definitions -│ ├── IndexLabel # Index label definitions -│ └── builder/ # Builder pattern implementations -├── structure/ # Graph element structures -│ ├── BaseElement # Base class for vertices/edges -│ ├── BaseVertex # Vertex implementation -│ ├── BaseEdge # Edge implementation -│ ├── BaseProperty # Property implementation -│ └── builder/ # Element builders -├── type/ # Type system -│ ├── HugeType # Enum for all graph types (VERTEX, EDGE, etc.) -│ ├── GraphType # Type interface -│ ├── Namifiable # Name-based types -│ ├── Idfiable # ID-based types -│ └── define/ # Type definitions (DataType, IdStrategy, etc.) -├── id/ # ID generation and management -│ ├── Id # ID interface -│ ├── IdGenerator # ID generation utilities -│ ├── EdgeId # Edge-specific ID handling -│ └── IdUtil # ID utility methods -├── serializer/ # Binary serialization -│ ├── BytesBuffer # Buffer for binary I/O -│ ├── BinaryElementSerializer # Element serialization -│ └── DirectBinarySerializer # Direct binary access -├── query/ # Query abstractions -│ ├── Query # Base query interface -│ ├── ConditionQuery # Conditional queries -│ ├── IdQuery # ID-based queries -│ ├── Condition # Query conditions -│ └── Aggregate # Aggregation queries -├── analyzer/ # Text analyzers (Chinese NLP) -│ ├── Analyzer # Base analyzer interface -│ ├── AnalyzerFactory # Factory for creating analyzers -│ ├── IKAnalyzer # IK Chinese word segmentation -│ ├── JiebaAnalyzer # Jieba segmentation -│ ├── HanLPAnalyzer # HanLP NLP -│ ├── AnsjAnalyzer # Ansj segmentation -│ ├── WordAnalyzer # Word-based analysis -│ ├── JcsegAnalyzer # Jcseg segmentation -│ ├── MMSeg4JAnalyzer # MMSeg4J segmentation -│ └── SmartCNAnalyzer # Lucene SmartCN -├── auth/ # Authentication utilities -│ ├── TokenGenerator # JWT token generation -│ └── AuthConstant # Auth constants -├── backend/ # Backend abstractions -│ ├── BinaryId # Binary ID representation -│ ├── BackendColumn # Column abstraction -│ └── Shard # Shard information -├── options/ # Configuration options -│ ├── CoreOptions # Core configuration -│ └── AuthOptions # Auth configuration -├── util/ # Utilities -│ ├── StringEncoding # String encoding utilities -│ ├── GraphUtils # Graph utility methods -│ ├── LZ4Util # LZ4 compression -│ ├── Blob # Binary blob handling -│ └── collection/ # Collection utilities (IdSet, CollectionFactory) -└── exception/ # Exception hierarchy - ├── HugeException # Base exception - ├── BackendException # Backend errors - ├── NotSupportException # Unsupported operations - ├── NotFoundException # Not found errors - └── NotAllowException # Permission errors -``` - -### Key Architectural Concepts - -#### 1. Two-Layer Schema System - -The module defines a dual schema hierarchy: - -- **`struct.schema.*`**: Schema element definitions (VertexLabel, EdgeLabel, etc.) - these are *metadata* about the graph structure -- **`structure.*`**: Actual graph elements (BaseVertex, BaseEdge, etc.) - these are *data* instances - -The schema layer defines the "blueprint" while the structure layer implements the "instances". - -#### 2. Type System - -The `HugeType` enum (type/HugeType.java) defines all possible types: -- Schema types: `VERTEX_LABEL`, `EDGE_LABEL`, `PROPERTY_KEY`, `INDEX_LABEL` -- Data types: `VERTEX`, `EDGE`, `PROPERTY`, `AGGR_PROPERTY_V`, `AGGR_PROPERTY_E` -- Special types: `META`, `COUNTER`, `TASK`, `OLAP`, `INDEX` - -#### 3. ID Management - -IDs are critical for distributed systems: -- `Id` interface provides abstraction over different ID types -- `IdGenerator` creates IDs based on strategy (AUTO_INCREMENT, PRIMARY_KEY, CUSTOMIZE) -- `EdgeId` uses special encoding: source vertex ID + edge label ID + sort values + target vertex ID -- Binary serialization optimizes ID storage - -#### 4. Binary Serialization - -`BytesBuffer` and serializers enable: -- Efficient storage in RocksDB and other backends -- Fast gRPC message passing between PD/Store/Server -- Compact on-disk and in-memory representation - -#### 5. Query Abstraction - -Query classes provide backend-agnostic query building: -- `Query`: Base interface with limit, offset, ordering -- `ConditionQuery`: Supports conditions (EQ, GT, LT, IN, CONTAINS, etc.) -- `IdQuery`: Direct ID-based lookups -- `Aggregate`: Aggregation operations (SUM, MAX, MIN, AVG) - -## Dependencies - -### Critical Dependencies - -- **hg-pd-client** (${project.version}): PD client for metadata coordination -- **hugegraph-common** (${project.version}): Shared utilities -- **Apache TinkerPop 3.5.1**: Graph computing framework -- **Guava 25.1-jre**: Google utilities -- **Eclipse Collections 10.4.0**: High-performance collections -- **fastutil 8.1.0**: Fast primitive collections - -### Text Analysis Dependencies - -Multiple Chinese NLP libraries for different use cases: -- **jieba-analysis 1.0.2**: Popular Chinese word segmentation -- **IKAnalyzer 2012_u6**: IK word segmentation -- **HanLP portable-1.5.0**: Natural language processing -- **Ansj 5.1.6**: Ansj segmentation -- **Word 1.3**: APDPlat word segmentation -- **Jcseg 2.2.0**: Jcseg segmentation -- **mmseg4j-core 1.10.0**: MMSeg4J segmentation -- **lucene-analyzers-smartcn 7.4.0**: Lucene SmartCN - -### Security Dependencies - -- **jjwt-api/impl/jackson 0.11.2**: JWT token handling -- **jbcrypt 0.4**: Password hashing - -## Development Notes - -### When Modifying This Module - -1. **Understand the impact**: Changes here affect hugegraph-pd, hugegraph-store, and hugegraph-server -2. **Rebuild dependent modules**: After modifying, rebuild PD and Store modules -3. **Binary compatibility**: Serialization changes require careful version migration -4. **ID changes**: Modifying ID generation can break existing data - -### Working with Schema Elements - -When adding or modifying schema elements in `struct/schema/`: -- Extend `SchemaElement` base class -- Implement required interfaces (`Namifiable`, `Typifiable`) -- Add corresponding `HugeType` enum value if needed -- Update serialization logic in `BinaryElementSerializer` -- Verify schema builder patterns in `struct/schema/builder/` - -### Working with Binary Serialization - -When modifying serialization: -- Changes to `BytesBuffer` format require version migration -- Test with all backends (RocksDB, HStore) -- Ensure backward compatibility or provide migration path -- Update both write and read paths consistently - -### Adding Text Analyzers - -To add a new text analyzer: -1. Implement the `Analyzer` interface in `analyzer/` -2. Register in `AnalyzerFactory` -3. Add dependency to pom.xml -4. Test with Chinese text queries - -## Common Patterns - -### Creating Schema Elements - -```java -// Schema elements use builders -PropertyKey propertyKey = schema.propertyKey("name") - .asText() - .valueSingle() - .create(); -``` - -### ID Generation - -```java -// Generate IDs based on strategy -Id id = IdGenerator.of(value, IdType.LONG); -Id edgeId = EdgeId.parse(sourceId, direction, label, sortValues, targetId); +mvn test -pl hugegraph-struct -am ``` -### Binary Serialization - -```java -// Write to buffer -BytesBuffer buffer = BytesBuffer.allocate(size); -buffer.writeId(id); -buffer.writeString(name); - -// Read from buffer -Id id = buffer.readId(); -String name = buffer.readString(); -``` - -## Cross-Module References - -This module is referenced by: -- **hugegraph-pd**: Uses schema definitions for metadata management -- **hugegraph-store**: Uses serialization for storage and RPC -- **hugegraph-server/hugegraph-core**: Uses all abstractions for graph operations -- **hugegraph-server/hugegraph-api**: Uses structures for REST API serialization +Tests exist under `src/test/java/`. See [PD/Store CI](../.github/workflows/pd-store-ci.yml) +for the Struct job and downstream build setup. -## License and Compliance +## Compatibility -This module follows Apache Software Foundation guidelines: -- All files must have Apache 2.0 license headers -- Third-party dependencies require license documentation in `install-dist/release-docs/licenses/` -- Excluded from Apache RAT: None (all source files checked) +- Server core has separate query, ID and serializer implementations; + matching class names do not guarantee matching behavior. +- Type codes, ID encoding and binary layout affect persisted data and Store decoding. + Check corresponding Server and Store readers/writers before changing the format.