From 96a48ad677d2af09e9a01eacf8251af102dd5efc Mon Sep 17 00:00:00 2001 From: Himanshu Verma Date: Wed, 9 Sep 2026 13:32:12 +0530 Subject: [PATCH 1/2] doc: state the design philosophy and the doc-sync rule for contributors AGENTS.md covered structure, build and test workflow, but said nothing about how code is expected to be written here, so agents fell back on generic best practice and reached for new abstractions. The PR template already asks for a documentation status, but "Doc - TODO" means "you will update later" with nothing attached to it, so the follow-up often does not happen and the docs drift away from the code. CONTRIBUTING did not mention documentation at all in the section on opening a PR. Adds a short design philosophy section to AGENTS.md, one visible field under the existing Documentation Status options for the hugegraph-doc PR link, and the paired doc PR expectation in CONTRIBUTING where a contributor reads it before opening the PR. The three options keep their current wording so the block stays identical to the one in hugegraph-toolchain and hugegraph-computer. --- .github/PULL_REQUEST_TEMPLATE.md | 2 ++ AGENTS.md | 16 ++++++++++++++++ docs/CONTRIBUTING.md | 6 ++++++ 3 files changed, 24 insertions(+) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 2a8a408a98..7d63f9c3ca 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -65,3 +65,5 @@ For example: - [ ] `Doc - TODO` - [ ] `Doc - Done` - [ ] `Doc - No Need` + +Related hugegraph-doc PR or issue: diff --git a/AGENTS.md b/AGENTS.md index 07daf17662..9edb9ae778 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -83,6 +83,18 @@ mvn test -pl hugegraph-store/hg-store-test -am Before writing new tests, check existing suites under `hugegraph-server/hugegraph-test/`. +## Design philosophy + +Match the codebase, do not improve on it. + +- Extend existing code before introducing a new abstraction. A new interface, factory + or base class needs a reason beyond symmetry. +- Keep the change minimal: the smallest edit that fully solves the issue. Adjacent + cleanups belong in their own PR. +- One concern per PR. Do not bundle an unrelated fix because you were already in the file. +- Comments stay short. When something needs real context to explain, link the GitHub + issue instead of writing a paragraph in the source. + ## Style & Pre-commit - Line 120, 4-space indent, LF, UTF-8, **no star imports** @@ -101,6 +113,10 @@ Before writing new tests, check existing suites under `hugegraph-server/hugegrap and `install-dist/scripts/dependency/known-dependencies.txt`. - `hugegraph-commons` is shared by every module; `hugegraph-struct` must precede PD/Store; server backends depend on `hugegraph-core`. +- Feature, config and `Dockerfile` changes ship a paired PR in `apache/hugegraph-doc`. + Land both together; docs that trail the code go stale without anyone noticing. +- README: wrap at 120, keep it thin, push detail to hugegraph-doc, and fold the + non-urgent parts into `
`. ## Additional context files diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 7056afd978..8d63e0608c 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -125,6 +125,12 @@ Note that since GitHub requires submitting code through `username + token` (inst Go to the web page of GitHub fork repo, there would be a chance to create a Pull Request after pushing to a new branch, click the button "Compare & pull request" to do it. Then edit the description for proposed changes, which can just be copied from the commit message. +If the change touches a feature, a configuration item or a `Dockerfile`, open the matching +documentation PR in [hugegraph-doc](https://github.com/apache/hugegraph-doc) at the same time +and link the two together. Both should merge together. Documentation that trails the code +goes stale quietly, and the gap is only found when a user or a search tool reads the wrong +thing. + Note: please make sure the email address you used to submit the code is bound to the GitHub account. For how to bind the email address, please refer to https://github.com/settings/emails: image From 899f0722c80e66a02e87120b51d64d62999359b4 Mon Sep 17 00:00:00 2001 From: imbajin Date: Thu, 17 Sep 2026 21:43:15 +0800 Subject: [PATCH 2/2] docs: streamline agent guidance and doc sync - keep module guidance concise with on-demand references - align documentation requirements across contributor entry points - correct stale Serena notes against current source - clarify README navigation and generated source paths --- .github/PULL_REQUEST_TEMPLATE.md | 11 +- .serena/memories/architecture_and_modules.md | 3 +- .../memories/code_style_and_conventions.md | 2 +- .../ecosystem_and_related_projects.md | 5 +- .../implementation_patterns_and_guidelines.md | 6 +- .serena/memories/key_file_locations.md | 2 +- .serena/memories/project_overview.md | 2 +- .serena/memories/suggested_commands.md | 7 +- .serena/memories/task_completion_checklist.md | 13 +- AGENTS.md | 156 +++---- README.md | 4 +- docs/CONTRIBUTING.md | 11 +- hugegraph-commons/AGENTS.md | 295 +------------- hugegraph-pd/AGENTS.md | 384 +----------------- hugegraph-server/AGENTS.md | 271 ++---------- hugegraph-store/AGENTS.md | 323 ++------------- hugegraph-struct/AGENTS.md | 278 +------------ 17 files changed, 191 insertions(+), 1582 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 7d63f9c3ca..62681543c0 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -60,10 +60,11 @@ For example: ## Documentation Status - +Select one option and provide the documentation location when applicable. -- [ ] `Doc - TODO` -- [ ] `Doc - Done` -- [ ] `Doc - No Need` +- [ ] `Doc - TODO`: required documentation is pending; complete it before merging. +- [ ] `Doc - Done`: documentation is included here or linked below. +- [ ] `Doc - No Need`: no user-visible documentation is affected. -Related hugegraph-doc PR or issue: +Documentation files in this PR or paired hugegraph-doc PR: + diff --git a/.serena/memories/architecture_and_modules.md b/.serena/memories/architecture_and_modules.md index a74d658209..687ae2251f 100644 --- a/.serena/memories/architecture_and_modules.md +++ b/.serena/memories/architecture_and_modules.md @@ -32,4 +32,5 @@ Distributed storage + Raft: `hg-store-core`, `hg-store-node`, `hg-store-client`, - **hugegraph-cluster-test**: Cluster integration tests ## Distributed Deployment (BETA) -PD + Store + Server (3+ nodes each), all gRPC. Docker compose configs in `docker/` directory, using bridge networking (migrated from host mode). +PD manages placement and metadata; Store owns partition data; Server serves graph queries. +Development and HA topologies differ; see `docker/README.md` for node counts and configuration. diff --git a/.serena/memories/code_style_and_conventions.md b/.serena/memories/code_style_and_conventions.md index 7a4c310e0b..701757a1b6 100644 --- a/.serena/memories/code_style_and_conventions.md +++ b/.serena/memories/code_style_and_conventions.md @@ -24,5 +24,5 @@ - Validate: `mvn apache-rat:check -ntp` + `mvn editorconfig:check` ## Build -- Java 11 target, `-Xlint:unchecked`, Lombok 1.18.30 (provided/optional) +- Java 11 target except Commons (Java 8 target); compiler/dependency versions come from module POMs. - Swagger: `io.swagger.core.v3:swagger-jaxrs2-jakarta` for REST API docs diff --git a/.serena/memories/ecosystem_and_related_projects.md b/.serena/memories/ecosystem_and_related_projects.md index 6bb1344d66..5f405287d9 100644 --- a/.serena/memories/ecosystem_and_related_projects.md +++ b/.serena/memories/ecosystem_and_related_projects.md @@ -7,7 +7,7 @@ |------|---------| | hugegraph-toolchain | Loader, Hubble (visualization), Tools CLI, Java Client | | hugegraph-computer | OLAP: PageRank, Connected Components, Shortest Path | -| incubator-hugegraph-ai | Graph RAG, KG construction, NL→Gremlin/Cypher | +| apache/hugegraph-ai | Graph RAG, KG construction, NL→Gremlin/Cypher | | hugegraph-doc | Docs & website (hugegraph.apache.org) | ## Data Flow @@ -20,4 +20,5 @@ Sources → hugegraph-loader → hugegraph-server → Hubble / Computer / AI - Queries: Gremlin (TinkerPop 3.5.1), OpenCypher, REST API + Swagger UI - Storage: RocksDB (default), HStore (distributed) -## Version: Server 1.7.0, TinkerPop 3.5.1, Java 11+ +## Versions +Read Server revision from root `pom.xml` and TinkerPop from `hugegraph-server/pom.xml`. diff --git a/.serena/memories/implementation_patterns_and_guidelines.md b/.serena/memories/implementation_patterns_and_guidelines.md index d04e33ce56..26d19ac26f 100644 --- a/.serena/memories/implementation_patterns_and_guidelines.md +++ b/.serena/memories/implementation_patterns_and_guidelines.md @@ -20,7 +20,7 @@ ## gRPC Protocol - PD protos: `hugegraph-pd/hg-pd-grpc/src/main/proto/` - Store protos: `hugegraph-store/hg-store-grpc/src/main/proto/` -- After `.proto` changes: `mvn clean compile` → `target/generated-sources/protobuf/` +- After `.proto` changes: `mvn clean compile`; PD/Store gRPC POMs output to each module's `src/main/java/`. ## Query Languages - **Gremlin**: Native TinkerPop 3.5.1 @@ -36,10 +36,10 @@ - **Backends in CI**: memory, rocksdb, hbase (matrix) - **Single test class**: `mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,memory -Dtest=ClassName` - TinkerPop tests: only on `release-*`/`test-*` branches -- Raft tests: only on `test*`/`raft*` branches +- Server Raft API tests are branch-gated in `server-ci.yml`; Store raft-core tests run in normal `pd-store-ci.yml` CI. ## Docker -- Single-node: `docker/docker-compose.yml` (bridge network, pd+store+server) +- Standalone: `docker/docker-compose.yml` (bridge network, Server + Hubble) - Cluster: `docker/docker-compose-3pd-3store-3server.yml` - Container logs: stdout-based diff --git a/.serena/memories/key_file_locations.md b/.serena/memories/key_file_locations.md index 5f2ade9424..ac10c668e8 100644 --- a/.serena/memories/key_file_locations.md +++ b/.serena/memories/key_file_locations.md @@ -16,7 +16,7 @@ - Tests: `hugegraph-test/src/main/java/.../` → `unit/`, `core/`, `api/`, `tinkerpop/` ## Docker -- `docker/docker-compose.yml` — Single-node (bridge network, pd+store+server) +- `docker/docker-compose.yml` — Standalone Server + Hubble (bridge network) - `docker/docker-compose-3pd-3store-3server.yml` — 3-node cluster - `docker/docker-compose.dev.yml` — Dev mode diff --git a/.serena/memories/project_overview.md b/.serena/memories/project_overview.md index 6375a3104e..a08aea9ec6 100644 --- a/.serena/memories/project_overview.md +++ b/.serena/memories/project_overview.md @@ -22,5 +22,5 @@ Apache HugeGraph is a fast-speed, highly-scalable graph database supporting 10+ - **Storage**: RocksDB (default/embedded), HStore (distributed/production), HBase (deprecated; planned for removal in 2.0) ## Version -- Current: 1.7.0 (`${revision}` property, Maven flatten plugin) +- Read `${revision}` from the root `pom.xml`; Maven flatten resolves it for published artifacts. - License: Apache License 2.0 diff --git a/.serena/memories/suggested_commands.md b/.serena/memories/suggested_commands.md index 346304432f..c1aa4d8e5b 100644 --- a/.serena/memories/suggested_commands.md +++ b/.serena/memories/suggested_commands.md @@ -5,7 +5,7 @@ mvn clean install -DskipTests # Full build mvn clean install -pl hugegraph-server -am -DskipTests # Server only mvn clean compile -U -Dmaven.javadoc.skip=true -ntp # Compile only -mvn clean package -DskipTests # Distribution → install-dist/target/ +mvn clean package -DskipTests # Aggregate distribution → root target/ ``` ## Test @@ -19,7 +19,7 @@ mvn test -pl hugegraph-server/hugegraph-test -am -P api-test,rocksdb # Single test class mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,memory -Dtest=YourTestClass -# TinkerPop compliance (release/test branches only) +# TinkerPop compliance (can be run locally on any branch) mvn test -pl hugegraph-server/hugegraph-test -am -P tinkerpop-structure-test,memory # PD/Store (build struct first) @@ -45,8 +45,9 @@ bin/enable-auth.sh # Enable auth ``` ## Docker +Create the required authentication environment first; follow `docker/README.md`. ```bash -cd docker && docker compose up -d # Single-node (bridge network) +cd docker && docker compose up -d # Standalone Server + Hubble (bridge network) cd docker && docker compose -f docker-compose-3pd-3store-3server.yml up -d # Cluster ``` diff --git a/.serena/memories/task_completion_checklist.md b/.serena/memories/task_completion_checklist.md index bcee316d5a..22a12b4db7 100644 --- a/.serena/memories/task_completion_checklist.md +++ b/.serena/memories/task_completion_checklist.md @@ -1,6 +1,8 @@ # Task Completion Checklist -## 1. Code Quality (MANDATORY) +## 1. Validation scope +Documentation-only changes: check links, paths and `git diff --check`. +For code changes, select relevant checks below and follow root `AGENTS.md` before pushing. ```bash mvn apache-rat:check -ntp # License headers mvn editorconfig:check # Style (.editorconfig) @@ -10,7 +12,7 @@ mvn clean compile -Dmaven.javadoc.skip=true # Compile warnings ## 2. Testing - Choose backend: `memory` (fast), `rocksdb` (realistic), `hbase` (deprecated compatibility) -- Single test: `-Dtest=ClassName` works with all profiles +- Single-test selection: check the module test POM and confirm the requested test actually ran. - Bug fix → existing tests; New feature → write tests; Refactor → affected module tests ## 3. Dependencies (if adding new) @@ -23,10 +25,13 @@ mvn clean compile -Dmaven.javadoc.skip=true # Compile warnings - `server-ci.yml`: memory/rocksdb/hbase × Java 11 - `rerun-ci.yml`: auto-retries flaky failures - `licence-checker.yml`: header validation -- Raft tests: only `test*`/`raft*` branches +- Server Raft API tests are branch-gated; Store raft-core and core tests run in normal PD/Store CI. - TinkerPop tests: only `release-*`/`test-*` branches -## 5. Commit +## 5. Documentation +- Follow root `AGENTS.md`: user-visible feature/configuration/deployment changes ship with matching docs. + +## 6. Commit - NEVER commit unless explicitly asked - Format: `feat|fix|refactor(module): msg` - Include issue ID if available diff --git a/AGENTS.md b/AGENTS.md index 9edb9ae778..a38801710f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,124 +1,62 @@ # AGENTS.md -Single source of truth for AI coding agents. -README.md covers human-facing deployment/ecosystem context; only consult it on demand. +Repository-specific guidance; module files add local details. -## Stack & Modules +## Key relationships -Apache HugeGraph — Apache TinkerPop 3 compliant graph database. -Java 11+, Maven 3.5+. Version managed via `${revision}` (currently `1.8.0`). +```text +Server (hugegraph-server): graph engine + REST/Gremlin API + ├─ RocksDB: embedded backend + └─ HStore adapter → Store client → Store: partition data + Raft +PD: placement + metadata; consulted by Server and Store +Struct: shared types/codecs; separate implementations also exist in Server +Commons: shared utilities; independent compiler and test settings ``` -Client (Gremlin / Cypher / REST) - │ -Server = hugegraph-server - ├─ hugegraph-api REST, Gremlin/Cypher, auth - ├─ hugegraph-core engine, schema, traversal, BackendStore interface - └─ Backend impls rocksdb (default, embedded) │ hstore (distributed) - ▼ - hugegraph-pd (placement) + hugegraph-store (Raft) -``` - -Top-level modules: `hugegraph-server` · `hugegraph-pd` · `hugegraph-store` · -`hugegraph-commons` (shared utils & RPC) · `hugegraph-struct` (data types; dep of PD/Store). -Server submodules worth knowing: `hugegraph-core`, `hugegraph-api`, -`hugegraph-rocksdb`, `hugegraph-hstore`, `hugegraph-test`, `hugegraph-dist`. +## Load on demand -## Code Search Anchors - -| Area | Path | +| Work area | Read when working there | |---|---| -| Graph engine | `hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/` | -| REST APIs | `hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/` | -| Backend interface | `hugegraph-server/hugegraph-core/.../backend/store/BackendStore.java` | -| Auth | `hugegraph-server/hugegraph-api/.../api/auth/` | -| gRPC protos | `hugegraph-{pd,store}/hg-{pd,store}-grpc/src/main/proto/` | - -Config roots (under each dist module's `src/assembly/static/conf/`): -- Server — `hugegraph.properties`, `rest-server.properties`, `gremlin-server.yaml` -- PD / Store — `application.yml` - -## Build +| Graph engine, API, backends | [Server](hugegraph-server/AGENTS.md) | +| Placement, metadata, PD client | [PD](hugegraph-pd/AGENTS.md) | +| Distributed storage and client | [Store](hugegraph-store/AGENTS.md) | +| Shared data types and serialization | [Struct](hugegraph-struct/AGENTS.md) | +| Utilities and RPC | [Commons](hugegraph-commons/AGENTS.md) | +| Container deployment | [Docker guide](docker/README.md) | +| PR requirements | [Contribution guide](docs/CONTRIBUTING.md) | + +## Repository constraints + +- Use existing module boundaries; keep unrelated refactors out of a fix. +- User-visible feature, configuration or deployment behavior changes must ship with matching docs. + Update in-repository docs in the same PR; link a paired `apache/hugegraph-doc` PR when website + docs are affected and coordinate both merges. A follow-up issue alone does not satisfy this rule. + Internal-only changes can use `Doc - No Need`. +- Keep README as an entry point; link detailed deployment instructions instead of duplicating them. +- Keep AGENTS.md under 100 lines where practical. Shared rules belong here, local exceptions + in module files. Keep key relationships, common commands and non-obvious pitfalls inline; + reference existing sources for versions, configuration details and lengthy procedures. + +## Build and validation + +Commands run from the repository root. Java 11+ and Maven 3.5+; +versions come from [pom.xml](pom.xml), including `${revision}`. ```bash -# All modules mvn clean install -DskipTests - -# Single module mvn clean install -pl hugegraph-server -am -DskipTests ``` -Distributed build order (for HStore-enabled dev): - -```bash -mvn install -pl hugegraph-struct -am -DskipTests # 1. shared data types -mvn clean package -pl hugegraph-pd -am -DskipTests # 2. placement driver -mvn clean package -pl hugegraph-store -am -DskipTests # 3. distributed storage -mvn clean package -pl hugegraph-server -am -DskipTests # 4. server -``` - -Runtime scripts (human-run) live in `hugegraph-server/hugegraph-dist/src/assembly/static/bin/`: -`init-store.sh`, `start-hugegraph.sh`, `stop-hugegraph.sh`. - -## Testing - -Server tests implicitly prefix `mvn test -pl hugegraph-server/hugegraph-test -am`: - -| Profile | Suffix | -|---|---| -| Unit | `-P unit-test` | -| Core | `-P core-test,rocksdb` (swap `rocksdb` for `memory`) | -| API | `-P api-test,rocksdb` | -| TinkerPop structure / process | `-P tinkerpop-{structure,process}-test,memory` | -| Single class | `-P core-test,rocksdb -Dtest=YourTestClass` | - -PD / Store tests (need `hugegraph-struct` installed first): - -```bash -mvn install -pl hugegraph-struct -am -DskipTests -mvn test -pl hugegraph-pd/hg-pd-test -am -mvn test -pl hugegraph-store/hg-store-test -am -``` - -Before writing new tests, check existing suites under `hugegraph-server/hugegraph-test/`. - -## Design philosophy - -Match the codebase, do not improve on it. - -- Extend existing code before introducing a new abstraction. A new interface, factory - or base class needs a reason beyond symmetry. -- Keep the change minimal: the smallest edit that fully solves the issue. Adjacent - cleanups belong in their own PR. -- One concern per PR. Do not bundle an unrelated fix because you were already in the file. -- Comments stay short. When something needs real context to explain, link the GitHub - issue instead of writing a paragraph in the source. - -## Style & Pre-commit - -- Line 120, 4-space indent, LF, UTF-8, **no star imports** -- Commit format: `feat|fix|refactor(module): msg` -- Run before pushing: - ```bash - mvn editorconfig:format # enforce code style - mvn clean compile -Dmaven.javadoc.skip=true # surface warnings - ``` - -## Cross-module notes - -- `.proto` edits: `mvn clean compile` regenerates gRPC stubs under - `target/generated-sources/protobuf/` (output packages `*/grpc/` are excluded from Apache RAT). -- Adding a third-party dep: update `install-dist/release-docs/{LICENSE,NOTICE,licenses/}` - and `install-dist/scripts/dependency/known-dependencies.txt`. -- `hugegraph-commons` is shared by every module; `hugegraph-struct` must precede PD/Store; - server backends depend on `hugegraph-core`. -- Feature, config and `Dockerfile` changes ship a paired PR in `apache/hugegraph-doc`. - Land both together; docs that trail the code go stale without anyone noticing. -- README: wrap at 120, keep it thin, push detail to hugegraph-doc, and fold the - non-urgent parts into `
`. - -## Additional context files - -`.serena/memories/` — notably `suggested_commands.md` and `task_completion_checklist.md` -when a task needs depth beyond this file. +For separate distributed-module builds, install `hugegraph-struct` first, +then build PD, Store and Server. Module guidance covers tests and CI prerequisites. +Commons tests require `-DskipCommonsTests=false`; a successful build does not imply its tests ran. + +- Java style: 120 columns, 4 spaces, no star imports; see [.editorconfig](.editorconfig). +- Before pushing code: `mvn editorconfig:format`, + `mvn clean compile -Dmaven.javadoc.skip=true`, and relevant module tests. + Documentation-only changes need link/path checks and `git diff --check`. +- PD/Store `.proto` builds generate Java into each `hg-*-grpc/src/main/java/` directory; + see the corresponding gRPC module POM before editing or cleaning generated sources. +- New dependencies require updating [release LICENSE/NOTICE/licenses](install-dist/release-docs/) + and [dependency inventory](install-dist/scripts/dependency/known-dependencies.txt). diff --git a/README.md b/README.md index e84260f5cc..0e48ced340 100644 --- a/README.md +++ b/README.md @@ -315,7 +315,7 @@ For comprehensive documentation, visit the [HugeGraph Documentation](https://hug | Understand configuration | `hugegraph-dist` | `src/assembly/static/conf/` | | Run tests | `hugegraph-test` | Test suites with multiple profiles | -For detailed architecture and development guidance, see [AGENTS.md](AGENTS.md). +For repository constraints and module guidance, see [AGENTS.md](AGENTS.md).
For Contributors @@ -323,7 +323,7 @@ For detailed architecture and development guidance, see [AGENTS.md](AGENTS.md). **New to HugeGraph?** Follow this path to get started: 1. **Understand the Architecture** - - Read [AGENTS.md](AGENTS.md) for detailed module structure and development patterns + - Read [AGENTS.md](AGENTS.md) for key component relationships and module guidance - Review the [Architecture Diagram](#architecture) above 2. **Set Up Your Environment** diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 8d63e0608c..bfa9d7e66f 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -125,11 +125,12 @@ Note that since GitHub requires submitting code through `username + token` (inst Go to the web page of GitHub fork repo, there would be a chance to create a Pull Request after pushing to a new branch, click the button "Compare & pull request" to do it. Then edit the description for proposed changes, which can just be copied from the commit message. -If the change touches a feature, a configuration item or a `Dockerfile`, open the matching -documentation PR in [hugegraph-doc](https://github.com/apache/hugegraph-doc) at the same time -and link the two together. Both should merge together. Documentation that trails the code -goes stale quietly, and the gap is only found when a user or a search tool reads the wrong -thing. +User-visible feature, configuration or deployment behavior changes must ship with matching +documentation. Include repository documentation in the same PR. When website documentation +is affected, link a paired [hugegraph-doc](https://github.com/apache/hugegraph-doc) PR and +coordinate both merges. A follow-up issue alone does not replace required documentation. +In the PR template, mark pending documentation as `Doc - TODO` until it is ready; +use `Doc - Done` with the documentation files or PR link, or `Doc - No Need` for internal-only changes. Note: please make sure the email address you used to submit the code is bound to the GitHub account. For how to bind the email address, please refer to https://github.com/settings/emails: image diff --git a/hugegraph-commons/AGENTS.md b/hugegraph-commons/AGENTS.md index c21ab4a5dc..179e01300c 100644 --- a/hugegraph-commons/AGENTS.md +++ b/hugegraph-commons/AGENTS.md @@ -1,285 +1,32 @@ -# AGENTS.md +# Commons guidance -This file provides guidance to an AI coding tool when working with code in this repository. +Shared rules: [root AGENTS.md](../AGENTS.md). Paths here are relative to this module. -## Project Overview +## Entry points -hugegraph-commons is a shared utility module for Apache HugeGraph and its peripheral components. It provides core infrastructure components (locks, config, events, iterators, REST client, RPC framework) to simplify development across the HugeGraph ecosystem. +- Config, locks, events, iterators and REST: `hugegraph-common/src/main/java/org/apache/hugegraph/`. +- RPC framework: `hugegraph-rpc/`, which depends on `hugegraph-common`. +- Commons targets Java 8 bytecode in [pom.xml](pom.xml), although CI uses JDK 11. + Do not infer its language/API baseline from Server's JDK requirement. -**Technology Stack**: -- Java 8+ (compiler source/target: 1.8) -- Apache Maven 3.5+ -- Apache Commons Configuration2 for config management -- OkHttp 4.10.0 for REST client (hugegraph-common) -- Sofa-RPC 5.7.6 for RPC framework (hugegraph-rpc) -- JUnit 4.13.1 and Mockito 4.1.0 for testing +## Tests -## Architecture - -### Two-Module Structure - -This is a Maven multi-module project with 2 main modules: - -1. **hugegraph-common**: Core utilities library - - Lock implementations (atomic, key, row, lock groups) - - Configuration system with type-safe options - - Event hub for async notifications - - Iterator utilities (map, filter, flat-map, batch) - - RESTful client (OkHttp-based) - - Utilities (perf analysis, version checking, collections, logging) - - License management - -2. **hugegraph-rpc**: RPC communication framework - - Sofa-RPC based client/server implementation - - Consumer and provider configuration - - Service registration and discovery - - **Depends on hugegraph-common** - -### Key Design Patterns - -1. **Type-Safe Configuration System**: `HugeConfig` + `OptionSpace` pattern - - Config options defined as typed `ConfigOption` objects - - Supports both `.properties` and `.yaml` files - - Options organized in `OptionSpace` groups for validation - - Security checks on load - -2. **Lock Hierarchy**: Multiple lock implementations for different use cases - - `AtomicLock`: Basic atomic locking - - `KeyLock`: Lock by specific key - - `RowLock`: Row-level locking for table-like structures - - `LockGroup`: Manage multiple related locks - - `LockManager`: Central lock coordination - -3. **Event System**: Async event notification - - `EventHub`: Central event dispatcher - - `EventListener`: Typed event handlers - - Thread-safe event publishing - -4. **Iterator Composition**: Chainable iterator wrappers - - `MapperIterator`, `FilterIterator`, `LimitIterator` - - `FlatMapperIterator` for nested iteration - - `BatchMapperIterator` for batch processing - - All extend `ExtendableIterator` base - -5. **RPC Architecture**: Sofa-RPC abstraction layer - - `RpcServer`: Service provider side - - `RpcClientProvider`: Service consumer side - - `RpcProviderConfig`/`RpcConsumerConfig`: Configuration wrappers - - Supports multiple protocols (bolt, rest, grpc) - -## Build & Development Commands - -### Prerequisites -```bash -# Verify Java version (8+ required) -java -version - -# Verify Maven version (3.5+ required) -mvn -version -``` - -### Build Commands - -```bash -# Clean build without tests (fastest) -mvn clean install -DskipTests - -# Build with tests enabled -mvn clean install - -# Build specific module only -mvn clean install -pl hugegraph-common -DskipTests -mvn clean install -pl hugegraph-rpc -am -DskipTests # -am includes dependencies - -# Compile with warnings visible -mvn clean compile -Dmaven.javadoc.skip=true -``` - -**Note**: Tests are skipped by default via `true` in pom.xml. To run tests, override with `-DskipCommonsTests=false`. - -### Testing +Tests are skipped by default through `skipCommonsTests`. After installing dependencies, +run from the repository root: ```bash -# Run all tests (override default skip) -mvn test -DskipCommonsTests=false - -# Run tests for specific module -mvn test -pl hugegraph-common -DskipCommonsTests=false -mvn test -pl hugegraph-rpc -am -DskipCommonsTests=false - -# Run single test class -mvn test -pl hugegraph-common -Dtest=HugeConfigTest -DskipCommonsTests=false - -# Run test suite (includes all unit tests) -mvn test -pl hugegraph-common -Dtest=UnitTestSuite -DskipCommonsTests=false +mvn test -pl hugegraph-commons/hugegraph-common -Dtest=UnitTestSuite -DskipCommonsTests=false +mvn test -pl hugegraph-commons/hugegraph-rpc -Dtest=UnitTestSuite -DskipCommonsTests=false ``` -### Code Quality - -```bash -# License header check (Apache RAT) -mvn apache-rat:check - -# Checkstyle validation -mvn checkstyle:check - -# Both checks run automatically during validate phase -mvn validate -``` - -### Code Coverage - -```bash -# Generate JaCoCo coverage report -mvn clean test -DskipCommonsTests=false -# Report: target/jacoco/index.html -``` - -## Important File Locations - -### Source Code Structure -- hugegraph-common sources: `hugegraph-common/src/main/java/org/apache/hugegraph/` - - `concurrent/`: Lock implementations - - `config/`: Configuration system (HugeConfig, OptionSpace, ConfigOption) - - `event/`: Event hub and listeners - - `iterator/`: Iterator utilities - - `rest/`: REST client implementation - - `util/`: Various utilities (collections, logging, version, etc.) - - `perf/`: Performance measurement (PerfUtil, Stopwatch) - - `license/`: License management - -- hugegraph-rpc sources: `hugegraph-rpc/src/main/java/org/apache/hugegraph/` - - `rpc/`: RPC server and client implementations - - `config/`: RPC-specific config options - -### Test Structure -- Unit tests: `hugegraph-{module}/src/test/java/org/apache/hugegraph/unit/` -- Test suites: `UnitTestSuite.java` lists all test classes -- Test utilities: `hugegraph-common/src/main/java/org/apache/hugegraph/testutil/` - - `Whitebox`: Reflection utilities for testing private members - - `Assert`: Enhanced assertion utilities - -### Configuration Files -- Parent POM: `pom.xml` (defines all dependencies and versions) -- Module POMs: `hugegraph-{module}/pom.xml` -- Test resources: `hugegraph-rpc/src/test/resources/*.properties` -- Checkstyle config: `style/checkstyle.xml` (referenced in parent POM) - -### Version Management -- Version property: `${revision}` in pom.xml (currently 1.7.0) -- Version classes: - - `hugegraph-common/src/main/java/org/apache/hugegraph/version/CommonVersion.java` - - `hugegraph-rpc/src/main/java/org/apache/hugegraph/version/RpcVersion.java` -- **IMPORTANT**: When changing version in pom.xml, also update version in these Java files - -## Development Workflow - -### Module Dependencies - -**Dependency order**: -``` -hugegraph-common (no internal dependencies) - ↓ -hugegraph-rpc (depends on hugegraph-common) -``` - -When making changes: -- Changes to `hugegraph-common` require rebuilding `hugegraph-rpc` -- Always build common first: `mvn install -pl hugegraph-common -DskipTests` -- Then build rpc: `mvn install -pl hugegraph-rpc -am -DskipTests` - -### Working with Configuration System - -When adding new configuration options: -1. Define `ConfigOption` in appropriate config class (e.g., `RpcOptions.java`) -2. Register option in an `OptionSpace` for validation -3. Load via `HugeConfig.get(option)` or `config.get(option)` -4. Example: -```java -public static final ConfigOption RPC_SERVER_HOST = - new ConfigOption<>("rpc.server_host", "...", "127.0.0.1"); -``` - -### Working with RPC Framework - -RPC configuration pattern: -- Server side: Create `RpcServer` with `HugeConfig` -- Client side: Use `RpcClientProvider` for service proxies -- Config files: Properties format with `rpc.*` keys -- Protocol: Default is Sofa-Bolt (binary protocol) - -### Testing Patterns - -1. **Unit tests** extend `BaseUnitTest` (for common module) or use standard JUnit -2. **Test organization**: Tests mirror source package structure -3. **Naming**: `{ClassName}Test.java` for unit tests -4. **Mocking**: Use Mockito for external dependencies -5. **Reflection testing**: Use `Whitebox.setInternalState()` for private field access - -### Adding Dependencies - -When adding third-party dependencies: -1. Add to `dependencyManagement` section in parent pom if used by multiple modules -2. Declare version in `` section -3. Add license info to `hugegraph-dist/release-docs/licenses/` -4. Update `hugegraph-dist/release-docs/LICENSE` -5. Update `hugegraph-dist/release-docs/NOTICE` if upstream has NOTICE - -## Common Workflows - -### Running a Specific Test - -```bash -# Single test class -mvn test -pl hugegraph-common -Dtest=HugeConfigTest -DskipCommonsTests=false - -# Single test method -mvn test -pl hugegraph-common -Dtest=HugeConfigTest#testGetOption -DskipCommonsTests=false - -# Pattern matching -mvn test -pl hugegraph-common -Dtest=*ConfigTest -DskipCommonsTests=false -``` - -### Debugging Tips - -1. **Enable debug logging**: Modify `log4j2.xml` in test resources -2. **Maven debug output**: Add `-X` flag to any Maven command -3. **Skip checkstyle temporarily**: Add `-Dcheckstyle.skip=true` -4. **Force dependency updates**: `mvn clean install -U` - -### Working with Parent POM - -This module has a parent POM (`../pom.xml` - hugegraph main project). If working standalone: -- The `` property comes from parent (1.7.0) -- Flatten Maven plugin resolves `${revision}` to actual version -- `.flattened-pom.xml` is auto-generated (excluded from RAT checks) - -## Special Notes - -### Version Synchronization - -Three places to update when changing version: -1. `pom.xml`: `` property -2. `hugegraph-common/.../CommonVersion.java`: Update version constant -3. `hugegraph-rpc/.../RpcVersion.java`: Update version constant - -### REST Client Implementation - -The REST client in `hugegraph-common/rest/` uses OkHttp (not Jersey as older docs suggest): -- Switched from Jersey to OkHttp in recent versions -- Supports connection pooling, timeouts, interceptors -- See `AbstractRestClient.java` for base implementation - -### RPC Version Note - -hugegraph-rpc uses Sofa-RPC 5.7.6 which has known security issues. There's a TODO to upgrade to 5.12+: -- See comment in `hugegraph-rpc/pom.xml:65-66` -- This is a known technical debt item +- Tests live in each module's `src/test/java/`. +- CI selects `UnitTestSuite`; add new tests to the relevant suite. +- Full setup: [Commons CI](../.github/workflows/commons-ci.yml). -### Checkstyle Configuration +## Compatibility -Checkstyle runs during `validate` phase by default: -- Config: `style/checkstyle.xml` -- Failures block the build -- Skip with `-Dcheckstyle.skip=true` for quick iteration -- Always fix before committing +- Utility behavior is shared across components, including iterator cleanup, + configuration parsing and lock semantics. +- Version changes: check `CommonVersion.java` and `RpcVersion.java` alongside the POM. +- Details on demand: [Common README](hugegraph-common/README.md) and + [RPC README](hugegraph-rpc/README.md). diff --git a/hugegraph-pd/AGENTS.md b/hugegraph-pd/AGENTS.md index aaaa861f39..57feb79304 100644 --- a/hugegraph-pd/AGENTS.md +++ b/hugegraph-pd/AGENTS.md @@ -1,376 +1,32 @@ -# AGENTS.md +# PD guidance -This file provides guidance to an AI coding tool when working with code in this repository. +Shared rules: [root AGENTS.md](../AGENTS.md). Paths here are relative to this module. -## Project Overview +## Entry points -HugeGraph PD (Placement Driver) is a meta server for distributed HugeGraph deployments, responsible for: -- Service discovery and registration -- Partition information storage and management -- Store node monitoring and scheduling -- Metadata coordination using Raft consensus +- Partition allocation and Store registration: `hg-pd-core/src/main/java/org/apache/hugegraph/pd/`. +- Metadata persistence and Raft: its `meta/` and `raft/` packages. +- RPC/REST handlers: `hg-pd-service/`; discovery, locks and watches: `hg-pd-client/`. +- Wire definitions: [protos](hg-pd-grpc/src/main/proto/). +- Shipped config: [distribution conf](hg-pd-dist/src/assembly/static/conf/). + Container endpoints/peers: [Docker guide](../docker/README.md). -**Technology Stack**: -- Java 11+ (required) -- Apache Maven 3.5+ -- gRPC + Protocol Buffers for RPC communication -- JRaft (Ant Design's Raft implementation) for consensus -- RocksDB for metadata persistence -- Spring Boot for REST APIs and dependency injection +## Build and tests -## Module Architecture - -HugeGraph PD is a Maven multi-module project with 8 modules: - -``` -hugegraph-pd/ -├── hg-pd-grpc # Protocol Buffers definitions and generated stubs -├── hg-pd-common # Shared utilities and common interfaces -├── hg-pd-core # Core PD logic (Raft, metadata, services) -├── hg-pd-service # gRPC service implementations and REST APIs -├── hg-pd-client # Java client library for PD -├── hg-pd-cli # Command-line interface tools -├── hg-pd-test # Integration and unit tests -└── hg-pd-dist # Distribution assembly (bin scripts, configs) -``` - -### Module Dependency Flow - -``` -hg-pd-grpc (protos) - ↓ -hg-pd-common (utilities) - ↓ -hg-pd-core (Raft + metadata stores) - ↓ -hg-pd-service (gRPC + REST endpoints) - ↓ -hg-pd-dist (assembly) - -hg-pd-client (depends on hg-pd-grpc, hg-pd-common) -hg-pd-cli (depends on hg-pd-client) -hg-pd-test (depends on hg-pd-core, hg-pd-service) -``` - -### Core Components (hg-pd-core) - -**Metadata Stores** (`meta/` package): -- `MetadataRocksDBStore`: RocksDB-backed persistence layer -- `PartitionMeta`: Partition assignment and shard group management -- `StoreInfoMeta`: Store node information and health status -- `TaskInfoMeta`: Distributed task coordination -- `IdMetaStore`, `ConfigMetaStore`, `DiscoveryMetaStore`: Domain-specific metadata - -**Services** (root package): -- `PartitionService`: Partition allocation, balancing, and splitting -- `StoreNodeService`: Store registration, heartbeat processing, status monitoring -- `StoreMonitorDataService`: Metrics collection and time-series data storage -- `TaskScheduleService`: Automated partition patrol and rebalancing -- `KvService`, `IdService`, `ConfigService`, `LogService`: Utility services - -**Raft Layer** (`raft/` package): -- `RaftEngine`: Raft group management and leadership -- `RaftStateMachine`: State machine applying metadata operations -- `RaftTaskHandler`: Async task execution via Raft proposals -- `KVOperation`, `KVStoreClosure`: Raft operation abstractions - -**Service Layer** (hg-pd-service): -- `ServiceGrpc`: Main gRPC service endpoint (partition, store, discovery RPCs) -- `PartitionAPI`: REST API for partition management -- `PDPulseService`: Heartbeat and monitoring -- `DiscoveryService`: Service discovery and registration - -### gRPC Protocol Definitions - -Located in `hg-pd-grpc/src/main/proto/`: -- `pdpb.proto`: Main PD service (GetMembers, RegisterStore, GetPartition) -- `metapb.proto`: Metadata objects (Partition, Shard, Store, Graph) -- `meta.proto`: Store and partition metadata -- `discovery.proto`: Service discovery protocol -- `kv.proto`: Key-value operations -- `pd_pulse.proto`: Heartbeat and monitoring -- `pd_watch.proto`: Watch notifications -- `metaTask.proto`: Task coordination - -**Important**: Generated Java code from `.proto` files is excluded from source control (`.gitignore`) and Apache RAT checks (Jacoco config). Regenerate after proto changes. - -## Build & Development Commands - -### Building PD Module - -```bash -# From hugegraph root directory, build PD and dependencies -mvn clean package -pl hugegraph-pd -am -DskipTests - -# From hugegraph-pd directory, build all modules -mvn clean install -DskipTests - -# Build with tests -mvn clean install - -# Build distribution package only -mvn clean package -pl hg-pd-dist -am -DskipTests -# Output: hugegraph-pd/apache-hugegraph-pd-.tar.gz -``` - -### Running Tests - -PD tests use Maven profiles defined in `pom.xml`: - -```bash -# All tests (default profiles active) -mvn test - -# Specific test profile -mvn test -P pd-core-test -mvn test -P pd-common-test -mvn test -P pd-client-test -mvn test -P pd-rest-test - -# Single test class (from hugegraph-pd directory) -mvn test -pl hg-pd-test -am -Dtest=StoreNodeServiceTest -mvn test -pl hg-pd-test -am -Dtest=PartitionServiceTest - -# From hugegraph root directory -mvn test -pl hugegraph-pd/hg-pd-test -am -``` - -Test files are located in `hg-pd-test/src/main/java/` (note: not `src/test/java`). - -### Regenerating gRPC Stubs - -```bash -# After modifying .proto files -mvn clean compile - -# Generated files location: -# target/generated-sources/protobuf/java/ -# target/generated-sources/protobuf/grpc-java/ -``` - -### Code Quality - -```bash -# License header check (Apache RAT) -mvn apache-rat:check - -# Clean build artifacts -mvn clean -# This also removes: *.tar, *.tar.gz, .flattened-pom.xml -``` - -## Running HugeGraph PD - -### Distribution Structure - -After building, extract the tarball: -``` -apache-hugegraph-pd-/ -├── bin/ -│ ├── start-hugegraph-pd.sh # Start PD server -│ ├── stop-hugegraph-pd.sh # Stop PD server -│ └── util.sh # Utility functions -├── conf/ -│ ├── application.yml # Main configuration -│ ├── application.yml.template # Configuration template -│ ├── log4j2.xml # Logging configuration -│ └── verify-license.json # License verification (optional) -├── lib/ # JAR dependencies -├── logs/ # Runtime logs -└── pd_data/ # RocksDB metadata storage (created at runtime) -``` - -### Starting PD - -```bash -cd apache-hugegraph-pd-/ -bin/start-hugegraph-pd.sh - -# With custom GC options -bin/start-hugegraph-pd.sh -g g1 - -# With custom JVM options -bin/start-hugegraph-pd.sh -j "-Xmx8g -Xms4g" - -# With OpenTelemetry enabled -bin/start-hugegraph-pd.sh -y true -``` - -Default ports: -- gRPC: 8686 (configure in `application.yml`: `grpc.port`) -- REST API: 8620 (configure in `application.yml`: `server.port`) -- Raft: 8610 (configure in `application.yml`: `raft.address`) - -JVM memory defaults (in `start-hugegraph-pd.sh`): -- Max heap: 32 GB -- Min heap: 512 MB - -### Stopping PD +Run from the repository root: ```bash -bin/stop-hugegraph-pd.sh -``` - -This sends SIGTERM to the PD process (tracked in `bin/pid`). - -## Key Configuration (application.yml) - -### Critical Settings for Distributed Deployment - -```yaml -grpc: - host: 127.0.0.1 # MUST change to actual IPv4 address in production - port: 8686 - -raft: - address: 127.0.0.1:8610 # This node's Raft address - peers-list: 127.0.0.1:8610 # Comma-separated list of all PD peers - # Example: 192.168.1.1:8610,192.168.1.2:8610,192.168.1.3:8610 - -pd: - data-path: ./pd_data # RocksDB metadata storage path - initial-store-count: 1 # Min stores required for cluster availability - initial-store-list: 127.0.0.1:8500 # Auto-activated store nodes (grpc_ip:grpc_port) - patrol-interval: 1800 # Partition rebalancing interval (seconds) - -partition: - default-shard-count: 1 # Replicas per partition (typically 3 in production) - store-max-shard-count: 12 # Max partitions per store - -store: - max-down-time: 172800 # Seconds before store is permanently offline (48h) - monitor_data_enabled: true # Enable metrics collection - monitor_data_interval: 1 minute # Metrics collection interval - monitor_data_retention: 1 day # Metrics retention period -``` - -### Common Configuration Errors - -1. **Raft peer discovery failure**: `raft.peers-list` must include all PD nodes' `raft.address` values -2. **Store connection issues**: `grpc.host` must be a reachable IP (not `127.0.0.1`) for distributed deployments. In Docker bridge networking, use the container hostname (e.g., `pd0`) set via `HG_PD_GRPC_HOST` env var. -3. **Split-brain scenarios**: Always run 3 or 5 PD nodes in production for Raft quorum -4. **Partition imbalance**: Adjust `patrol-interval` for faster/slower rebalancing - -## Development Workflows - -### Adding a New gRPC Service - -1. Define `.proto` messages and service in `hg-pd-grpc/src/main/proto/` -2. Run `mvn compile` to generate Java stubs -3. Implement service in `hg-pd-service/src/main/java/.../service/` -4. Register service in gRPC server initialization (check existing `ServiceGrpc.java` pattern) -5. Add client methods in `hg-pd-client/` if needed - -### Adding a New Metadata Store - -1. Create meta class in `hg-pd-core/src/main/java/.../meta/` -2. Use `MetadataRocksDBStore` as the underlying persistence layer -3. Implement metadata operations as Raft proposals via `RaftTaskHandler` -4. Add corresponding service methods in `hg-pd-core/.../Service.java` -5. Expose via gRPC in `hg-pd-service/` - -### Modifying Partition Logic - -- Core partition logic: `hg-pd-core/.../PartitionService.java` (69KB file, 2000+ lines) -- Key methods: - - `splitPartition()`: Partition splitting logic - - `updatePartitionLeader()`: Leader election handling - - `balancePartitions()`: Auto-balancing algorithm - - `getPartitionByCode()`: Partition routing -- All partition changes must go through Raft consensus -- Test with `hg-pd-test/.../core/PartitionServiceTest.java` - -### Debugging Raft Issues - -- Enable Raft logging in `conf/log4j2.xml`: Set `com.alipay.sofa.jraft` to DEBUG -- Check Raft state: Leader election happens in `RaftEngine.java` -- Raft snapshots stored in `pd_data/raft/snapshot/` -- Raft logs stored in `pd_data/raft/log/` - -## Testing Strategy - -### Test Organization - -Tests are in `hg-pd-test/src/main/java/` (non-standard location): -- `BaseTest.java`: Base class with common setup -- `core/`: Core service tests (PartitionService, StoreNodeService, etc.) -- Suite tests: `PDCoreSuiteTest.java` runs all core tests - -### Running Integration Tests - -```bash -# From hugegraph root, run PD integration tests +mvn install -pl hugegraph-struct -am -DskipTests mvn test -pl hugegraph-pd/hg-pd-test -am - -# These tests start embedded PD instances and verify: -# - Raft consensus and leader election -# - Partition allocation and balancing -# - Store heartbeat and monitoring -# - Metadata persistence and recovery -``` - -## Docker Deployment - -### Building Docker Image - -```bash -# From hugegraph root directory -docker build -f hugegraph-pd/Dockerfile -t hugegraph-pd:latest . ``` -The Dockerfile uses multi-stage build: -1. Stage 1: Build with Maven -2. Stage 2: Runtime with OpenJDK 11 - -### Running in Docker - -```bash -# Single PD node (development) -docker run -d -p 8620:8620 -p 8686:8686 -p 8610:8610 \ - -v /path/to/pd_data:/hugegraph-pd/pd_data \ - hugegraph-pd:latest - -# For production clusters, use Docker Compose or Kubernetes -# See: ../docker/docker-compose-3pd-3store-3server.yml and ../docker/README.md -``` - -Exposed ports: 8620 (REST), 8686 (gRPC), 8610 (Raft) - -## Cross-Module Dependencies - -When working on PD: -- **hugegraph-struct**: Protocol definitions shared with hugegraph-store - - Build struct first: `mvn install -pl hugegraph-struct -am -DskipTests` - - Required before building PD if struct changed -- **hugegraph-commons**: RPC framework, locks, and configuration utilities - - Changes to commons may affect PD's `hg-pd-common` module - -## CI/CD Integration - -PD tests run in `pd-store-ci.yml` GitHub Actions workflow: -- Triggered on pushes to PD module files -- Runs `mvn test -pl hugegraph-pd/hg-pd-test -am` -- JaCoCo coverage excludes generated gRPC code and config classes - -## Important Notes - -### Generated Code Exclusions -- `hg-pd-grpc/src/main/java/` is excluded from git (see `.gitignore`) -- Apache RAT skips `**/grpc/**.*` (see `pom.xml` Jacoco config) -- Always run `mvn clean compile` after pulling proto changes - -### Raft Consensus Requirements -- PD uses JRaft for distributed consensus -- All metadata writes are Raft proposals (see `KVOperation`, `KVStoreClosure`) -- Raft group requires 3 or 5 nodes for fault tolerance in production -- Single-node mode (peers-list with one address) is for development only +- Tests live in `hg-pd-test/src/main/java/`, not `src/test/java/`. +- Test profiles in the [parent POM](pom.xml) activate matching Surefire executions; + suite selection is configured in the [test POM](hg-pd-test/pom.xml). +- Service setup: [PD/Store CI](../.github/workflows/pd-store-ci.yml). -### Store Interaction -- PD does not store graph data; it only stores metadata about store nodes and partitions -- Actual graph data resides in hugegraph-store nodes -- PD coordinates store nodes but doesn't handle data plane traffic +## Cross-module traps -### Version Compatibility -- PD version must match hugegraph-server and hugegraph-store versions -- Version managed via `${revision}` property (inherited from parent POM) -- Current version: 1.7.0 +- Trace metadata mutations through `RaftTaskHandler` / `KVOperation` and the state machine. +- Client changes affect Server and Store; inspect watch/lock callers before changing + transport lifecycle or timeout behavior. diff --git a/hugegraph-server/AGENTS.md b/hugegraph-server/AGENTS.md index 0b6da40820..d093c2c138 100644 --- a/hugegraph-server/AGENTS.md +++ b/hugegraph-server/AGENTS.md @@ -1,258 +1,35 @@ -# AGENTS.md +# Server guidance -This file provides guidance to an AI coding tool when working with code in this repository. +Shared rules: [root AGENTS.md](../AGENTS.md). Paths here are relative to this module. -## Project Overview +## Entry points -HugeGraph Server is the graph engine layer of Apache HugeGraph, consisting of: -- **REST API Layer** (hugegraph-api): RESTful APIs for graph operations, Gremlin/Cypher queries, schema management, and authentication -- **Graph Engine Layer** (hugegraph-core): TinkerPop 3 implementation, schema management, traversal optimization, task scheduling -- **Backend Interface**: Abstraction layer for pluggable storage backends -- **Storage Backend Implementations**: RocksDB (default), HStore (distributed), HBase (deprecated; planned for removal in 2.0), and Memory (test-only) +- Engine/schema/traversal: `hugegraph-core/src/main/java/org/apache/hugegraph/`. +- REST, auth and graph lifecycle: `hugegraph-api/src/main/java/org/apache/hugegraph/`. +- Backend contract: `hugegraph-core/src/main/java/org/apache/hugegraph/backend/store/BackendStore.java`. +- HStore adapter: `hugegraph-hstore/`; distributed client lives in [Store](../hugegraph-store/AGENTS.md). +- Shipped configs/scripts: [distribution sources](hugegraph-dist/src/assembly/static/). + Graph configs are in `conf/graphs/`; REST and Gremlin configs are in `conf/`. -Technology: Java 11+, Maven 3.5+, Apache TinkerPop 3.5.1, Jersey 3.0 (REST), gRPC (distributed communication) +## Tests -## Build Commands +Run from the repository root: -### Full Build ```bash -# Build all modules (from hugegraph-server directory) -mvn clean install -DskipTests - -# Build with tests -mvn clean install - -# Build specific module -mvn clean install -pl hugegraph-core -am -DskipTests -``` - -### Code Quality -```bash -# Run checkstyle validation -mvn checkstyle:check - -# Checkstyle runs automatically during 'validate' phase -# Configuration: ../style/checkstyle.xml -``` - -## Testing - -### Running Tests - -**Test profiles** (`-P` flag): -- `core-test` (default): Core graph engine tests -- `unit-test`: Unit tests only (memory backend) -- `api-test`: REST API tests -- `tinkerpop-structure-test`: TinkerPop structure compliance -- `tinkerpop-process-test`: TinkerPop process compliance - -**Backend profiles** (combine with test profiles): -- `memory`: In-memory backend (default for tests) -- `rocksdb`: RocksDB backend -- `hbase`: HBase backend (deprecated; planned for removal in 2.0) -- `hstore`: HStore backend - -```bash -# Unit tests (from hugegraph-server/) -mvn test -pl hugegraph-test -am -P unit-test - -# Core tests with RocksDB backend -mvn test -pl hugegraph-test -am -P core-test,rocksdb - -# API tests with memory backend -mvn test -pl hugegraph-test -am -P api-test,memory - -# Run specific test class -mvn test -pl hugegraph-test -am -P core-test,memory -Dtest=YourTestClassName - -# TinkerPop compliance tests (for release validation) -mvn test -pl hugegraph-test -am -P tinkerpop-structure-test,memory -mvn test -pl hugegraph-test -am -P tinkerpop-process-test,memory -``` - -### Test Module Structure -All tests are in `hugegraph-test/` which depends on all other modules. Tests are organized by: -- Unit tests: `src/test/java/.../unit/` -- Core tests: `src/test/java/.../core/` -- API tests: `src/test/java/.../api/` - -## Module Structure - -Multi-module Maven project with 8 submodules: - -``` -hugegraph-server/ -├── hugegraph-core # Graph engine, TinkerPop impl, schema, backend interface -├── hugegraph-api # REST API, Gremlin/Cypher endpoints, authentication -├── hugegraph-rocksdb # RocksDB backend (default, embedded) -├── hugegraph-hstore # HStore backend (distributed, production) -├── hugegraph-hbase # HBase backend (deprecated; planned for removal in 2.0) -├── hugegraph-dist # Distribution packaging, scripts, configs -├── hugegraph-test # All test suites -└── hugegraph-example # Example code -``` - -### Key Package Structure (hugegraph-core) - -``` -org/apache/hugegraph/ -├── HugeGraph.java # Main graph interface -├── StandardHugeGraph.java # Core implementation -├── HugeFactory.java # Factory for graph instances -├── backend/ # Backend abstraction and implementations -│ ├── store/BackendStore.java # Storage backend interface -│ ├── serializer/ # Data serialization -│ └── tx/ # Transaction management -├── schema/ # Schema management (VertexLabel, EdgeLabel, etc.) -├── structure/ # Graph elements (Vertex, Edge, Property) -├── traversal/ # Gremlin traversal optimization -├── task/ # Async task scheduling and execution -├── auth/ # Authentication and authorization -├── job/ # Job management (rebuild, compact, etc.) -└── meta/ # Metadata management -``` - -### Key Package Structure (hugegraph-api) - -``` -org/apache/hugegraph/ -├── api/ # REST API endpoints -│ ├── graph/GraphAPI.java # Graph CRUD operations -│ ├── schema/ # Schema APIs -│ ├── gremlin/GremlinAPI.java # Gremlin query endpoint -│ ├── cypher/CypherAPI.java # Cypher query endpoint (via OpenCypher) -│ ├── auth/ # Authentication APIs -│ └── job/TaskAPI.java # Job/Task management APIs -├── server/RestServer.java # Jersey/Grizzly REST server -├── auth/ # Auth filters and handlers -└── metrics/ # Metrics collection +mvn test -pl hugegraph-server/hugegraph-test -am -P unit-test +mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,rocksdb +mvn test -pl hugegraph-server/hugegraph-test -am -P api-test,rocksdb ``` -## Architecture Patterns - -### Pluggable Backend Architecture -All storage backends implement the `BackendStore` interface from hugegraph-core. New backends can be added as separate modules without modifying core code. Backend selection is via `backend` property in `conf/hugegraph.properties`. - -### TinkerPop Compliance -Full Apache TinkerPop 3 implementation with: -- Custom traversal strategies in `hugegraph-core/src/main/java/org/apache/hugegraph/traversal/` -- Structure and process API implementations -- Gremlin groovy script support via `gremlin-groovy` integration - -### Multi-Language Query Support -- **Gremlin**: Native support via TinkerPop -- **Cypher**: OpenCypher implementation in `hugegraph-api/src/main/java/org/apache/hugegraph/opencypher/` - - Translates Cypher to Gremlin using `opencypher-gremlin` library - -### gRPC for Distributed Components -When working with HStore backend or distributed features: -- Protocol Buffer definitions: `hugegraph-core/src/main/resources/proto/` -- Generated code: `target/generated-sources/protobuf/java/` -- Regenerate after .proto changes: `mvn clean compile` - -### Authentication System -Authentication is **optional and disabled by default**: -- Enable: `bin/enable-auth.sh` or via configuration -- Implementation: `hugegraph-api/src/main/java/org/apache/hugegraph/auth/` -- Multi-level access control: Users, Groups, Projects, Targets, Permissions -- Core auth logic: `hugegraph-core/src/main/java/org/apache/hugegraph/auth/` - -## Running the Server - -After building, server distribution is in `hugegraph-dist/target/`. Scripts are in `hugegraph-dist/src/assembly/static/bin/`: - -```bash -# Initialize backend storage (first time only) -bin/init-store.sh - -# Start server (REST API on 8080, Gremlin on 8182) -bin/start-hugegraph.sh - -# Stop server -bin/stop-hugegraph.sh - -# Gremlin console (interactive) -bin/gremlin-console.sh - -# Monitor server status -bin/monitor-hugegraph.sh - -# Enable authentication -bin/enable-auth.sh -``` - -## Configuration Files - -Located in `hugegraph-dist/src/assembly/static/conf/`: -- **`hugegraph.properties`**: Main server configuration (backend, storage paths, cache) -- **`rest-server.properties`**: REST API settings (host, port, thread pool, SSL) -- **`gremlin-server.yaml`**: Gremlin server configuration (WebSocket, serializers) -- **`log4j2.xml`**: Logging configuration - -## Common Development Tasks - -### Adding a New REST API Endpoint -1. Create API class in `hugegraph-api/src/main/java/org/apache/hugegraph/api/` -2. Extend `ApiBase` or relevant base class -3. Use JAX-RS annotations (`@Path`, `@GET`, `@POST`, etc.) -4. Add Swagger/OpenAPI annotations for documentation -5. Add tests in `hugegraph-test/src/test/java/.../api/` - -### Adding Backend Support -1. Create new module: `hugegraph-{backend-name}/` -2. Add dependency on `hugegraph-core` -3. Implement `BackendStore` interface -4. Implement `BackendStoreProvider` for factory -5. Add backend module to parent `pom.xml` `` section -6. Add tests combining with test profiles - -### Modifying Schema or Graph Elements -- Schema definitions: `hugegraph-core/src/main/java/org/apache/hugegraph/schema/` -- Graph structure: `hugegraph-core/src/main/java/org/apache/hugegraph/structure/` -- Always consider backward compatibility for stored data -- Update serializers if changing storage format - -### Working with Transactions -Transaction management is in `hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/`: -- `GraphTransaction`: Vertex/Edge operations -- `SchemaTransaction`: Schema operations -- `IndexTransaction`: Index operations -- Follow the transaction lifecycle pattern in existing code - -## Important Notes - -### Code Style -- Checkstyle configuration: `../style/checkstyle.xml` -- Enforced during Maven `validate` phase -- Import code style into IDE (IntelliJ IDEA recommended) - -### License Headers -All Java files must have Apache License header. Verified via `maven-checkstyle-plugin`. - -### Version Management -- Version managed via `${revision}` property (currently 1.7.0) -- Uses `flatten-maven-plugin` for CI-friendly versioning -- Don't hardcode versions in module POMs - -### Cross-Module Dependencies -``` -hugegraph-api → hugegraph-core → hugegraph-commons (external) -hugegraph-{backend} → hugegraph-core -hugegraph-test → all modules -``` - -### Generated Code -- Protobuf Java classes: Generated, not manually edited -- Located in `target/generated-sources/` -- Excluded from checkstyle/license checks - -### Backend Selection at Runtime -Backends are loaded via ServiceLoader pattern. The `backend` property in `hugegraph.properties` determines which implementation is used. All backend JARs must be on classpath. +- Suite sources are in `hugegraph-test/src/main/java/`, not `src/test/java/`. +- Combine test and backend profiles. TinkerPop profiles: + `tinkerpop-structure-test`, `tinkerpop-process-test`. +- Selection and suite registration: [test POM](hugegraph-test/pom.xml). +- Service setup: [Server CI](../.github/workflows/server-ci.yml); + HStore setup: [PD/Store CI](../.github/workflows/pd-store-ci.yml). -## Debugging Tips +## Cross-module traps -- Detailed logging: Edit `hugegraph-dist/src/assembly/static/conf/log4j2.xml` -- View effective config: `bin/dump-conf.sh` -- Arthas diagnostics: Built-in (version 3.7.1) -- Backend state inspection: `bin/dump-store.sh` -- Raft cluster tools (for distributed): `bin/raft-tools.sh` +- Struct has separate ID/query/serializer implementations. For format changes, inspect + [Struct](../hugegraph-struct/AGENTS.md) and both read/write paths. +- Authentication spans REST and Gremlin; inspect both shipped configs when changing bootstrap. diff --git a/hugegraph-store/AGENTS.md b/hugegraph-store/AGENTS.md index 8b5ef46bab..c3bf63e5bf 100644 --- a/hugegraph-store/AGENTS.md +++ b/hugegraph-store/AGENTS.md @@ -1,312 +1,35 @@ -# AGENTS.md +# Store guidance -This file provides guidance to an AI coding tool when working with code in this repository. +Shared rules: [root AGENTS.md](../AGENTS.md). Paths here are relative to this module. -## Project Overview +## Entry points -HugeGraph Store is a distributed storage backend for Apache HugeGraph, using RocksDB as the underlying storage engine with Raft consensus protocol for distributed coordination. It is designed for production-scale deployments requiring high availability and horizontal scalability. +- Routing, retry and streaming: `hg-store-client/`. +- Partition lifecycle, Raft and snapshots: `hg-store-core/src/main/java/org/apache/hugegraph/store/`. +- RocksDB access: `hg-store-rocksdb/`; RPC/REST serving: `hg-store-node/`. +- Wire definitions: [protos](hg-store-grpc/src/main/proto/). +- Shipped configs/scripts: [distribution sources](hg-store-dist/src/assembly/static/). +- Server adapter: [HStore backend](../hugegraph-server/hugegraph-hstore/). -**Technology Stack**: -- Java 11+ -- RocksDB: Embedded key-value storage engine -- Raft (JRaft): Distributed consensus protocol -- gRPC: Inter-node communication -- Protocol Buffers: Data serialization +## Build and tests -## Architecture +Run from the repository root: -### Module Structure - -HugeGraph Store consists of 9 submodules: - -``` -hugegraph-store/ -├── hg-store-common # Shared utilities, constants, query abstractions -├── hg-store-grpc # gRPC protocol definitions (proto files) and generated stubs -├── hg-store-client # Client library for connecting to Store cluster -├── hg-store-rocksdb # RocksDB abstraction and optimizations -├── hg-store-core # Core storage logic, partition management -├── hg-store-node # Store node server implementation with Raft -├── hg-store-dist # Distribution packaging, scripts, configs -├── hg-store-cli # Command-line tools for cluster management -└── hg-store-test # Integration and unit tests -``` - -### Key Package Structure - -``` -org/apache/hugegraph/store/ -├── grpc/ # Generated gRPC stubs (do not edit manually) -├── client/ # Client API for Store operations -├── node/ # Store node server and Raft integration -├── core/ # Core storage abstractions -│ ├── store/ # Store interface and implementations -│ ├── partition/ # Partition management -│ └── raft/ # Raft consensus integration -├── rocksdb/ # RocksDB wrapper and optimizations -├── query/ # Query processing and aggregation -└── util/ # Common utilities -``` - -### Distributed Architecture - -Store operates as a cluster of nodes: -- **Store Nodes**: 3+ nodes (typically 3 or 5 for Raft quorum) -- **Raft Groups**: Data partitioned into Raft groups for replication -- **PD Coordination**: Requires hugegraph-pd for cluster metadata and partition assignment -- **Client Access**: hugegraph-server connects via hg-store-client - -## Build Commands - -### Prerequisites ```bash -# HugeGraph Store depends on hugegraph-struct -# Build struct module first from repository root -cd /path/to/hugegraph-org mvn install -pl hugegraph-struct -am -DskipTests +mvn test -pl hugegraph-store/hg-store-test -am ``` -### Full Build -```bash -# From hugegraph-store directory -mvn clean install -DskipTests - -# Build with tests -mvn clean install - -# Build specific module (e.g., client only) -mvn clean install -pl hg-store-client -am -DskipTests -``` - -### Testing - -**Test profiles** (defined in pom.xml): -- `store-client-test` (default): Client library tests -- `store-core-test` (default): Core storage tests -- `store-common-test` (default): Common utilities tests -- `store-rocksdb-test` (default): RocksDB abstraction tests -- `store-server-test` (default): Store node server tests -- `store-raftcore-test` (default): Raft consensus tests - -```bash -# Run all tests (from hugegraph-store/) -mvn test -pl hg-store-test -am - -# Run specific test class -mvn test -pl hg-store-test -am -Dtest=YourTestClassName - -# Run tests for specific module -mvn test -pl hg-store-core -am -mvn test -pl hg-store-client -am -``` - -### Code Quality -```bash -# License header check (Apache RAT) - from repository root -mvn apache-rat:check - -# EditorConfig validation - from repository root -mvn editorconfig:check -``` - -## Running Store Cluster - -Scripts are located in `hg-store-dist/src/assembly/static/bin/`: - -```bash -# Start Store node -bin/start-hugegraph-store.sh - -# Stop Store node -bin/stop-hugegraph-store.sh - -# Restart Store node -bin/restart-hugegraph-store.sh -``` - -**Important**: For a functional distributed cluster, you need: -1. HugeGraph PD cluster running (3+ nodes) -2. HugeGraph Store cluster (3+ nodes) -3. Proper configuration pointing Store nodes to PD cluster - -See Docker Compose examples in the repository root `../docker/` directory. Single-node quickstart (pre-built images): `../docker/docker-compose.yml`. Single-node dev build (from source): `../docker/docker-compose.dev.yml`. 3-node cluster: `../docker/docker-compose-3pd-3store-3server.yml`. See `../docker/README.md` for the full setup guide. - -## Configuration Files - -Located in `hg-store-dist/src/assembly/static/conf/`: - -- **`application.yml`**: Main Store node configuration - - RocksDB settings (data paths, cache sizes, compaction) - - Raft configuration (election timeout, snapshot interval) - - Network settings (gRPC ports) - - Store capacity and partition management - -- **`application-pd.yml`**: PD client configuration - - PD cluster endpoints - - Heartbeat intervals - - Partition query settings - -- **`log4j2.xml`**: Logging configuration - -## Important Development Notes - -### gRPC Protocol Definitions - -Protocol Buffer files are in `hg-store-grpc/src/main/proto/`: -- `store_common.proto` - Common data structures -- `store_session.proto` - Client-server session management -- `store_state.proto` - Cluster state and metadata -- `store_stream_meta.proto` - Streaming operations -- `graphpb.proto` - Graph data structures -- `query.proto` - Query operations -- `healthy.proto` - Health check endpoints - -**When modifying `.proto` files**: -1. Edit the `.proto` file in `hg-store-grpc/src/main/proto/` -2. Run `mvn clean compile` to regenerate Java stubs -3. Generated code appears in `target/generated-sources/protobuf/` -4. Generated files are excluded from license checks - -### Module Dependencies - -Build order matters due to dependencies: -``` -hugegraph-struct (external) - ↓ -hg-store-common - ↓ -hg-store-grpc → hg-store-rocksdb - ↓ -hg-store-core - ↓ -hg-store-client, hg-store-node - ↓ -hg-store-cli, hg-store-dist, hg-store-test -``` - -Always build `hugegraph-struct` first, then Store modules follow Maven reactor order. - -### Working with RocksDB - -Store uses RocksDB for persistent storage: -- Abstraction layer: `hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/` -- Column families for different data types -- Custom compaction and compression settings -- Optimized for graph workloads (vertices, edges, indexes) - -Configuration in `application.yml`: -- `rocksdb.data-path` - Data directory location -- `rocksdb.block-cache-size` - In-memory cache size -- `rocksdb.write-buffer-size` - Write buffer configuration - -### Raft Consensus Integration - -Store uses JRaft (Ant Financial's Raft implementation): -- Each partition is a Raft group with 3 replicas (typically) -- Leader election, log replication, snapshot management -- Configuration: `raft.*` settings in `application.yml` - -Key Raft operations: -- Snapshot creation and loading -- Log compaction -- Leadership transfer -- Membership changes - -### Client Development - -When working with `hg-store-client`: -- Client connects to PD to discover Store nodes -- Automatic failover and retry logic -- Connection pooling and load balancing -- Batch operations support - -Example usage in hugegraph-server: -- Backend: `hugegraph-server/hugegraph-hstore/` -- Client integration: Uses `hg-store-client` library - -### Partition Management - -Data is partitioned for distributed storage: -- Partition assignment managed by PD -- Partition splitting and merging (future feature) -- Partition rebalancing on node addition/removal -- Hash-based partition key distribution - -## Common Development Tasks - -### Adding New gRPC Service - -1. Define service in appropriate `.proto` file in `hg-store-grpc/src/main/proto/` -2. Add message definitions for request/response -3. Run `mvn clean compile` to generate stubs -4. Implement service in `hg-store-node/` server -5. Add client methods in `hg-store-client/` -6. Add tests in `hg-store-test/` - -### Modifying Storage Engine - -1. Core storage interfaces: `hg-store-core/src/main/java/org/apache/hugegraph/store/core/store/` -2. RocksDB implementation: `hg-store-rocksdb/` -3. Update Raft state machine if needed: `hg-store-node/src/main/java/org/apache/hugegraph/store/node/raft/` -4. Consider backward compatibility for stored data format - -### Adding Query Operations - -1. Query abstractions: `hg-store-common/src/main/java/org/apache/hugegraph/store/query/` -2. Aggregation functions: `hg-store-common/.../query/func/` -3. Update proto definitions if new query types needed -4. Implement in `hg-store-core/` and expose via gRPC - -### Cluster Testing - -For distributed cluster tests: -- Module: `hugegraph-cluster-test/` (repository root) -- Requires: PD cluster + Store cluster + Server instances -- Docker Compose recommended for local testing -- CI/CD: See `.github/workflows/cluster-test-ci.yml` - -## Debugging Tips - -- **Logging**: Edit `hg-store-dist/src/assembly/static/conf/log4j2.xml` for detailed logs -- **Raft State**: Check Raft logs and snapshots in data directory -- **RocksDB Stats**: Enable RocksDB statistics in `application.yml` -- **gRPC Tracing**: Enable gRPC logging for request/response debugging -- **PD Connection**: Verify Store can connect to PD endpoints -- **Health Checks**: Use gRPC health check service for node status - -## Cross-Repository Integration - -Store integrates with other HugeGraph components: - -1. **hugegraph-pd**: Cluster metadata and partition management - - Store registers with PD on startup - - PD assigns partitions to Store nodes - - Heartbeat mechanism for health monitoring - -2. **hugegraph-server**: Graph engine uses Store as backend - - Backend implementation: `hugegraph-server/hugegraph-hstore/` - - Uses `hg-store-client` for Store cluster access - - Configuration: `backend=hstore` in `hugegraph.properties` - -3. **hugegraph-commons**: Shared utilities - - RPC framework: `hugegraph-commons/hugegraph-rpc/` - - Common utilities: `hugegraph-commons/hugegraph-common/` - -## Version Management - -- Version managed via `${revision}` property (currently 1.7.0) -- Flatten Maven plugin for CI-friendly versioning -- Must match version of other HugeGraph components (server, PD) - -## Special Notes - -### Performance Tuning +- Tests live in `hg-store-test/src/main/java/`, not `src/test/java/`. +- Test profiles in the [parent POM](pom.xml) activate matching Surefire executions. + Check [test POM](hg-store-test/pom.xml) suite includes when adding tests. +- CoreSuiteTest and BatchGraphIsolationTest need separate JVM forks: Store shutdown leaves + singleton state closed. Preserve `reuseForks=false` in the core test execution. +- Test prerequisites: [PD/Store CI](../.github/workflows/pd-store-ci.yml). + Cluster setup: [Docker guide](../docker/README.md). -Key performance factors: -- RocksDB block cache size (memory) -- Raft batch size and flush interval -- gRPC connection pool size -- Partition count and distribution -- Network latency between nodes +## Cross-module traps -Refer to `application.yml` for tuning parameters. +- PD manages placement; Store owns graph data and partition Raft groups. + Routing or membership changes cross both components. +- Codec changes must agree with [Struct](../hugegraph-struct/AGENTS.md) and the Server adapter. diff --git a/hugegraph-struct/AGENTS.md b/hugegraph-struct/AGENTS.md index dea4f06288..49f27a31f7 100644 --- a/hugegraph-struct/AGENTS.md +++ b/hugegraph-struct/AGENTS.md @@ -1,273 +1,31 @@ -# AGENTS.md +# Struct guidance -This file provides guidance to an AI coding tool when working with code in this repository. +Shared rules: [root AGENTS.md](../AGENTS.md). Paths here are relative to this module. -## Module Overview +## Entry points -**hugegraph-struct** is a foundational data structures module that defines the core abstractions shared across HugeGraph distributed components. This module **must be built before hugegraph-pd and hugegraph-store** as they depend on its structure definitions. +Sources: `src/main/java/org/apache/hugegraph/`. -**Key Responsibilities**: -- Schema element definitions (VertexLabel, EdgeLabel, PropertyKey, IndexLabel) -- Graph element structures (BaseVertex, BaseEdge, BaseProperty) -- Binary serialization/deserialization for efficient storage and RPC -- Type system definitions (HugeType enum, data types, ID strategies) -- Query abstractions (Query, ConditionQuery, IdQuery, Aggregate) -- Chinese text analyzers (multiple implementations: Jieba, IK, HanLP, etc.) -- Authentication utilities (JWT token generation, constants) +- Schema metadata: `struct/schema/`; graph instances: `structure/`. +- IDs, binary serialization and conditions: `id/`, `serializer/`, `query/`. +- Type codes: `type/`; metadata client lifecycle: `meta/`. +- Dependency versions and compiler level: [pom.xml](pom.xml). -## Build Commands +## Build and tests -### Building This Module +Run from the repository root: ```bash -# From hugegraph-struct directory -mvn clean install -DskipTests - -# Build with tests (if any exist in future) -mvn clean install - -# From parent directory (hugegraph root) -mvn install -pl hugegraph-struct -am -DskipTests -``` - -### Dependency Chain - -This module is a **critical dependency** for distributed components: - -```bash -# Correct build order for distributed components: -# 1. Build hugegraph-struct first mvn install -pl hugegraph-struct -am -DskipTests - -# 2. Then build PD -mvn clean package -pl hugegraph-pd -am -DskipTests - -# 3. Then build Store -mvn clean package -pl hugegraph-store -am -DskipTests -``` - -## Code Architecture - -### Package Structure - -``` -org.apache.hugegraph/ -├── struct/schema/ # Schema element definitions -│ ├── SchemaElement # Base class for all schema types -│ ├── VertexLabel # Vertex label definitions -│ ├── EdgeLabel # Edge label definitions -│ ├── PropertyKey # Property key definitions -│ ├── IndexLabel # Index label definitions -│ └── builder/ # Builder pattern implementations -├── structure/ # Graph element structures -│ ├── BaseElement # Base class for vertices/edges -│ ├── BaseVertex # Vertex implementation -│ ├── BaseEdge # Edge implementation -│ ├── BaseProperty # Property implementation -│ └── builder/ # Element builders -├── type/ # Type system -│ ├── HugeType # Enum for all graph types (VERTEX, EDGE, etc.) -│ ├── GraphType # Type interface -│ ├── Namifiable # Name-based types -│ ├── Idfiable # ID-based types -│ └── define/ # Type definitions (DataType, IdStrategy, etc.) -├── id/ # ID generation and management -│ ├── Id # ID interface -│ ├── IdGenerator # ID generation utilities -│ ├── EdgeId # Edge-specific ID handling -│ └── IdUtil # ID utility methods -├── serializer/ # Binary serialization -│ ├── BytesBuffer # Buffer for binary I/O -│ ├── BinaryElementSerializer # Element serialization -│ └── DirectBinarySerializer # Direct binary access -├── query/ # Query abstractions -│ ├── Query # Base query interface -│ ├── ConditionQuery # Conditional queries -│ ├── IdQuery # ID-based queries -│ ├── Condition # Query conditions -│ └── Aggregate # Aggregation queries -├── analyzer/ # Text analyzers (Chinese NLP) -│ ├── Analyzer # Base analyzer interface -│ ├── AnalyzerFactory # Factory for creating analyzers -│ ├── IKAnalyzer # IK Chinese word segmentation -│ ├── JiebaAnalyzer # Jieba segmentation -│ ├── HanLPAnalyzer # HanLP NLP -│ ├── AnsjAnalyzer # Ansj segmentation -│ ├── WordAnalyzer # Word-based analysis -│ ├── JcsegAnalyzer # Jcseg segmentation -│ ├── MMSeg4JAnalyzer # MMSeg4J segmentation -│ └── SmartCNAnalyzer # Lucene SmartCN -├── auth/ # Authentication utilities -│ ├── TokenGenerator # JWT token generation -│ └── AuthConstant # Auth constants -├── backend/ # Backend abstractions -│ ├── BinaryId # Binary ID representation -│ ├── BackendColumn # Column abstraction -│ └── Shard # Shard information -├── options/ # Configuration options -│ ├── CoreOptions # Core configuration -│ └── AuthOptions # Auth configuration -├── util/ # Utilities -│ ├── StringEncoding # String encoding utilities -│ ├── GraphUtils # Graph utility methods -│ ├── LZ4Util # LZ4 compression -│ ├── Blob # Binary blob handling -│ └── collection/ # Collection utilities (IdSet, CollectionFactory) -└── exception/ # Exception hierarchy - ├── HugeException # Base exception - ├── BackendException # Backend errors - ├── NotSupportException # Unsupported operations - ├── NotFoundException # Not found errors - └── NotAllowException # Permission errors -``` - -### Key Architectural Concepts - -#### 1. Two-Layer Schema System - -The module defines a dual schema hierarchy: - -- **`struct.schema.*`**: Schema element definitions (VertexLabel, EdgeLabel, etc.) - these are *metadata* about the graph structure -- **`structure.*`**: Actual graph elements (BaseVertex, BaseEdge, etc.) - these are *data* instances - -The schema layer defines the "blueprint" while the structure layer implements the "instances". - -#### 2. Type System - -The `HugeType` enum (type/HugeType.java) defines all possible types: -- Schema types: `VERTEX_LABEL`, `EDGE_LABEL`, `PROPERTY_KEY`, `INDEX_LABEL` -- Data types: `VERTEX`, `EDGE`, `PROPERTY`, `AGGR_PROPERTY_V`, `AGGR_PROPERTY_E` -- Special types: `META`, `COUNTER`, `TASK`, `OLAP`, `INDEX` - -#### 3. ID Management - -IDs are critical for distributed systems: -- `Id` interface provides abstraction over different ID types -- `IdGenerator` creates IDs based on strategy (AUTO_INCREMENT, PRIMARY_KEY, CUSTOMIZE) -- `EdgeId` uses special encoding: source vertex ID + edge label ID + sort values + target vertex ID -- Binary serialization optimizes ID storage - -#### 4. Binary Serialization - -`BytesBuffer` and serializers enable: -- Efficient storage in RocksDB and other backends -- Fast gRPC message passing between PD/Store/Server -- Compact on-disk and in-memory representation - -#### 5. Query Abstraction - -Query classes provide backend-agnostic query building: -- `Query`: Base interface with limit, offset, ordering -- `ConditionQuery`: Supports conditions (EQ, GT, LT, IN, CONTAINS, etc.) -- `IdQuery`: Direct ID-based lookups -- `Aggregate`: Aggregation operations (SUM, MAX, MIN, AVG) - -## Dependencies - -### Critical Dependencies - -- **hg-pd-client** (${project.version}): PD client for metadata coordination -- **hugegraph-common** (${project.version}): Shared utilities -- **Apache TinkerPop 3.5.1**: Graph computing framework -- **Guava 25.1-jre**: Google utilities -- **Eclipse Collections 10.4.0**: High-performance collections -- **fastutil 8.1.0**: Fast primitive collections - -### Text Analysis Dependencies - -Multiple Chinese NLP libraries for different use cases: -- **jieba-analysis 1.0.2**: Popular Chinese word segmentation -- **IKAnalyzer 2012_u6**: IK word segmentation -- **HanLP portable-1.5.0**: Natural language processing -- **Ansj 5.1.6**: Ansj segmentation -- **Word 1.3**: APDPlat word segmentation -- **Jcseg 2.2.0**: Jcseg segmentation -- **mmseg4j-core 1.10.0**: MMSeg4J segmentation -- **lucene-analyzers-smartcn 7.4.0**: Lucene SmartCN - -### Security Dependencies - -- **jjwt-api/impl/jackson 0.11.2**: JWT token handling -- **jbcrypt 0.4**: Password hashing - -## Development Notes - -### When Modifying This Module - -1. **Understand the impact**: Changes here affect hugegraph-pd, hugegraph-store, and hugegraph-server -2. **Rebuild dependent modules**: After modifying, rebuild PD and Store modules -3. **Binary compatibility**: Serialization changes require careful version migration -4. **ID changes**: Modifying ID generation can break existing data - -### Working with Schema Elements - -When adding or modifying schema elements in `struct/schema/`: -- Extend `SchemaElement` base class -- Implement required interfaces (`Namifiable`, `Typifiable`) -- Add corresponding `HugeType` enum value if needed -- Update serialization logic in `BinaryElementSerializer` -- Verify schema builder patterns in `struct/schema/builder/` - -### Working with Binary Serialization - -When modifying serialization: -- Changes to `BytesBuffer` format require version migration -- Test with all backends (RocksDB, HStore) -- Ensure backward compatibility or provide migration path -- Update both write and read paths consistently - -### Adding Text Analyzers - -To add a new text analyzer: -1. Implement the `Analyzer` interface in `analyzer/` -2. Register in `AnalyzerFactory` -3. Add dependency to pom.xml -4. Test with Chinese text queries - -## Common Patterns - -### Creating Schema Elements - -```java -// Schema elements use builders -PropertyKey propertyKey = schema.propertyKey("name") - .asText() - .valueSingle() - .create(); -``` - -### ID Generation - -```java -// Generate IDs based on strategy -Id id = IdGenerator.of(value, IdType.LONG); -Id edgeId = EdgeId.parse(sourceId, direction, label, sortValues, targetId); +mvn test -pl hugegraph-struct -am ``` -### Binary Serialization - -```java -// Write to buffer -BytesBuffer buffer = BytesBuffer.allocate(size); -buffer.writeId(id); -buffer.writeString(name); - -// Read from buffer -Id id = buffer.readId(); -String name = buffer.readString(); -``` - -## Cross-Module References - -This module is referenced by: -- **hugegraph-pd**: Uses schema definitions for metadata management -- **hugegraph-store**: Uses serialization for storage and RPC -- **hugegraph-server/hugegraph-core**: Uses all abstractions for graph operations -- **hugegraph-server/hugegraph-api**: Uses structures for REST API serialization +Tests exist under `src/test/java/`. See [PD/Store CI](../.github/workflows/pd-store-ci.yml) +for the Struct job and downstream build setup. -## License and Compliance +## Compatibility -This module follows Apache Software Foundation guidelines: -- All files must have Apache 2.0 license headers -- Third-party dependencies require license documentation in `install-dist/release-docs/licenses/` -- Excluded from Apache RAT: None (all source files checked) +- Server core has separate query, ID and serializer implementations; + matching class names do not guarantee matching behavior. +- Type codes, ID encoding and binary layout affect persisted data and Store decoding. + Check corresponding Server and Store readers/writers before changing the format.