diff --git a/docs/src/concepts/index.md b/docs/src/concepts/index.md index a8eaf6c2..cc6ee3b1 100644 --- a/docs/src/concepts/index.md +++ b/docs/src/concepts/index.md @@ -1,9 +1,9 @@ --- title: Core Concepts -description: Understand the fundamental concepts of IDP-Core - Entity Templates, Entities, Properties, Relations, and Audit tracking +description: Understand the fundamental concepts of IDP-Core - Entity Templates, Entities, Properties, and Relations --- -IDP-Core sits at the center of a flexible, runtime-configurable data model. This section explains the fundamental concepts you need to understand, including entity management and comprehensive audit tracking. +IDP-Core sits at the center of a flexible, runtime-configurable data model. This section explains the fundamental concepts you need to understand, including entity management and relationships. ## Overview @@ -49,24 +49,12 @@ graph TB Connections between entities forming a knowledge graph. -- 🌐 **[Webhooks](webhooks.md)** - - --- - - Runtime-configurable connectors to push your data from external systems within your IDP. You can map any source in your data model in minutes. - - πŸ” **[Filtering Entities](entity-filtering.md)** --- Query entities by attributes, property values, and relations using the filter DSL. -- πŸ“œ **[Audit](audit.md)** - - --- - - Track all changes over time with comprehensive revision history and user attribution. - --- @@ -142,4 +130,5 @@ Dive deeper into each concept: - **[Entity Templates](entity-templates.md)** - Learn how to design your data model - **[Properties](properties.md)** - Understand property types and validation - **[Relations](relations.md)** - Connect your entities into a graph -- **[Webhooks](webhooks.md)** - Configure inbound integrations and security strategies +- **[Data Integration](../features/data-integration.md)** - Ingest and map data from external systems +- **[Audit](../features/audit.md)** - Track changes to entities over time diff --git a/docs/src/contributing/code/audit-implemantation.md b/docs/src/contributing/code/audit-implemantation.md deleted file mode 100644 index b486ce70..00000000 --- a/docs/src/contributing/code/audit-implemantation.md +++ /dev/null @@ -1,200 +0,0 @@ -# Adding Audit Logging to Domain Objects - -This guide explains how to integrate our Hibernate Envers audit logging architecture for a new or existing domain object, and how to expose an endpoint to retrieve its history. This is part of our Hexagonal Architecture and ensures we keep a strict separation between database tracking and domain logic. - -## Architecture Overview - -We use **Hibernate Envers** with native entity tracking enabled (`track_entities_changed_in_revision: true`). - -- **Global Transaction Log:** `envers_transaction_log` tracks the revision number, timestamp, and the user's `auth_id`. -- **Modified Entities Log:** `envers_modified_entities` natively tracks which JPA entity classes were modified in a given transaction. -- **Entity-Specific Audit Tables:** (for example `entity_aud`, `property_aud`). These store the actual state snapshots of the modified rows. - ---- - -## Step 1: Add `@Audited` to the JPA Entity - -Navigate to your JPA Entity class (for example `PropertyJpaEntity`) in the `infrastructure.adapters.persistence.model` package. - -```java -import org.hibernate.envers.Audited; -import org.hibernate.envers.NotAudited; - -@Entity -@Table(name = "property") -@Audited(withModifiedFlag=true) // <--- Add this annotation to track changes and flag column modified -public class PropertyJpaEntity { - - @Id - private UUID id; - - private String name; - - // Use @NotAudited on fields or lazy relationships you DO NOT want to track - @NotAudited - @OneToMany(mappedBy = "property") - private Set children; -} -``` - -### Collection Type Considerations for Auditing - -When your entity contains collections (`@ElementCollection` or `@OneToMany`), choose the appropriate collection type: - -**Use `Set` (Recommended):** - -```java -@ElementCollection(fetch = FetchType.EAGER) -@CollectionTable(name = "related_items", joinColumns = @JoinColumn(name = "parent_id")) -@Audited(withModifiedFlag = true) -private Set relatedItems; // Order doesn't matter -``` - -**Use `List` (Only When Order Matters):** - -```java -@ElementCollection(fetch = FetchType.EAGER) -@CollectionTable(name = "ordered_items", joinColumns = @JoinColumn(name = "parent_id")) -@OrderColumn(name = "item_order") // <--- Explicitly track order -@Audited(withModifiedFlag = true) -private List orderedItems; // Order is significant -``` - -**Key Difference in Audit Schema:** - -- **Set**: Composite PK is `(parent_id, rev, item_id)` - simpler -- **List**: Composite PK is `(parent_id, rev, item_order)` - adds order tracking column - -See Infrastructure Layer instructions for complete Set vs List guidance and when to use each type. - -## Step 2: Create the Flyway Migration - -Create a new Flyway migration script (for example `V4_2__audit_property.sql`) in `src/main/resources/db/migration/` to define the specific `_aud` table for your entity. - -**Rules for Audit Tables:** - -1. Suffix the table name with `_aud` (for example `property_aud`). -2. Add a `rev` column (`BIGINT NOT NULL`). -3. Add a `revtype` column (`SMALLINT`) to track the operation (0=ADD, 1=MOD, 2=DEL). -4. Replicate the fields from the base table that you are auditing. -5. Make the Primary Key a composite of the entity's ID and `rev`. -6. Add the Foreign Key linking to `envers_transaction_log`. - -```sql -CREATE TABLE property_aud -( - id UUID NOT NULL, - rev BIGINT NOT NULL, - revtype SMALLINT, - name VARCHAR(255), - value VARCHAR(255), - PRIMARY KEY (id, rev), - CONSTRAINT fk_property_aud_revinfo FOREIGN KEY (rev) REFERENCES envers_transaction_log (rev) ON DELETE CASCADE -); - -CREATE INDEX idx_property_aud_rev ON property_aud (rev); -``` - -## Step 3: Implement the Domain Port and Persistence Adapter - -Create your port in the domain (for example `PropertyAuditPort`), then implement the adapter using Hibernate Envers' `AuditReader`. - -```java -@Component -@RequiredArgsConstructor -public class PostgresPropertyAuditAdapter implements PropertyAuditPort { - - private final EntityManager entityManager; - - @Override - public List getPropertyAuditHistory(UUID propertyId) { - AuditReader auditReader = AuditReaderFactory.get(entityManager); - - // Query Envers for the specific entity class - @SuppressWarnings("unchecked") - List revisions = auditReader.createQuery() - .forRevisionsOfEntity(PropertyJpaEntity.class, false, true) - .add(AuditEntity.id().eq(propertyId)) - .addOrder(AuditEntity.revisionNumber().desc()) - .getResultList(); - - return revisions.stream().map(this::mapToDomainAuditInfo).toList(); - } - - private AuditInfo mapToDomainAuditInfo(Object[] revision) { - PropertyJpaEntity snapshot = (PropertyJpaEntity) revision[0]; - CustomRevisionEntity revEntity = (CustomRevisionEntity) revision[1]; - RevisionType revType = (RevisionType) revision[2]; - - // Map to your domain record... - } -} -``` - -## Step 4: Add/Update Domain Service and REST Endpoint - -Depending on the domain design, you can either reuse existing files to minimize boilerplate, or create dedicated audit files for clean separation of concerns. - -1. **Service:** Create `PropertyAuditService` to handle business logic (like ensuring the parent object exists before querying history). -2. **Controller:** Expose the endpoint, using standard DTOs formatted with Jackson's `SnakeCaseStrategy`. - -### Approach 1: Reusing Existing Structures (Recommended for simple resources) - -If you already have a PropertyService and a PropertyController, simply append the new functionality directly to them to keep things concise. - -1. **Service:** In PropertyService: Inject the PropertyAuditPort and add a getPropertyHistory(UUID id) method. -2. **Controller:** In PropertyController: Expose a nested route following clean RESTful guidelines. - -```java -// Within your existing PropertyController.java -@GetMapping("/{propertyId}/history") -public List getPropertyHistory(@PathVariable UUID propertyId) { -return auditMapper.fromDomainList(propertyService.getPropertyHistory(propertyId)); -} -``` - -### Approach 2: Creating Dedicated Audit Handlers (Recommended for complex auditing rules) - -If retrieving audit details requires dedicated permissions, complex filtering, or distinct business rules, decouple them into dedicated files. - -1. **Service:** Establish a PropertyAuditService to guarantee domain invariant verification (for example verifying object access permissions before compiling historical records). -2. **Controller:** Build a focused controller parsing outputs cleanly with Jackson's SnakeCaseStrategy. - -```java -package com.company.project.infrastructure.adapters.api; - -import lombok.RequiredArgsConstructor; -import org.springframework.web.bind.annotation.*; -import java.util.List; -import java.util.UUID; - -@RestController -@RequestMapping("/api/v1/audit/") -@RequiredArgsConstructor -public class PropertyAuditController { - - private final PropertyAuditService auditService; - private final PropertyAuditDtoOutMapper mapper; - - @GetMapping("properties/{propertyId}") - public List getPropertyAuditHistory(@PathVariable UUID propertyId) { - return mapper.fromDomainList(auditService.getHistory(propertyId)); - } -} -``` - -```java -@RestController -@RequestMapping("/api/v1/audit/") -@RequiredArgsConstructor -public class PropertyAuditController { - - private final PropertyAuditService auditService; - private final PropertyAuditDtoOutMapper mapper; - - @GetMapping("properties/{propertyId}") - public List getPropertyAuditHistory(@PathVariable UUID propertyId) { - return mapper.fromDomainList(auditService.getHistory(propertyId)); - } -} -``` diff --git a/docs/src/deployment/configuration.md b/docs/src/deployment/configuration.md index 981fc68e..276b89b6 100644 --- a/docs/src/deployment/configuration.md +++ b/docs/src/deployment/configuration.md @@ -1,5 +1,5 @@ --- -title: Configuration Reference +title: General Configuration description: Complete configuration reference for IDP-Core --- diff --git a/docs/src/concepts/audit.md b/docs/src/features/audit.md similarity index 98% rename from docs/src/concepts/audit.md rename to docs/src/features/audit.md index 8649f1d5..b7c35e77 100644 --- a/docs/src/concepts/audit.md +++ b/docs/src/features/audit.md @@ -655,6 +655,6 @@ curl -s http://localhost:8084/api/v1/audit/entities/web-service/my-service | \ ## Next Steps -- **[Entities](entities.md)** - Entity structure and lifecycle -- **[Properties](properties.md)** - Property types and validation -- **[Relations](relations.md)** - Entity relationships +- **[Entities](../concepts/entities.md)** - Entity structure and lifecycle +- **[Properties](../concepts/properties.md)** - Property types and validation +- **[Relations](../concepts/relations.md)** - Entity relationships diff --git a/docs/src/features/data-integration.md b/docs/src/features/data-integration.md index 193d6e29..13cc888b 100644 --- a/docs/src/features/data-integration.md +++ b/docs/src/features/data-integration.md @@ -1,11 +1,11 @@ --- -title: Data Integration +title: Overview description: Connect to any data source through Webhooks, Kafka, or Pub/Sub with runtime-configurable mappings status: πŸ• Doing --- > [!IMPORTANT] -> This document describes a feature that is not yet developed. The content is subject to change and may not reflect the final implementation. +> This document describes a feature that is not fully developed yet. The content is subject to change and may not reflect the final implementation. The Internal Developer Platform provides flexible data integration to connect to any source and map incoming data to your entities at runtime without code changes. That's powerful for rapid adaptation. diff --git a/docs/src/concepts/entity-dynamic-mapping.md b/docs/src/features/entity-dynamic-mapping.md similarity index 95% rename from docs/src/concepts/entity-dynamic-mapping.md rename to docs/src/features/entity-dynamic-mapping.md index b2bd8487..225cf044 100644 --- a/docs/src/concepts/entity-dynamic-mapping.md +++ b/docs/src/features/entity-dynamic-mapping.md @@ -5,7 +5,7 @@ description: Understand Dynamic mappings and JSLT expressions. ## Overview -A mapping targets one Entity Template and describes how to derive entity fields from the incoming JSON payload with a JSLT filter and entity projections. +A mapping targets one Entity Template and describes how to derive entity fields from the incoming JSON payload with a JSLT filter and entity projections. It is linked to a data integration configuration (Webhook, Kafka, etc) as it describes how to transform incoming payloads. ## Entity Dynamic Mapping Fields @@ -35,6 +35,20 @@ A mapping targets one Entity Template and describes how to derive entity fields | `name` | βœ… | Relation name from the Entity Template | | `target_entity_identifiers` | βœ… | Array of JSLT expressions to extract target identifiers | +> [!TIP] +> To remove all target relations while preserving the entity, set `target_entity_identifiers` to `["null"]`. +> Setting `[]` or `[""]` will not work. +> +> ```json +> "relations": [ +> { +> "name": "your-relation-name", +> "target_entity_identifiers": ["null"] +> } +> ] +> ``` +> + ### Mapping Actions The `action` field determines how the resolved entity payload modifies the entity in IDP-Core: diff --git a/docs/src/features/graph.md b/docs/src/features/graph.md index b2042bd1..c67daf94 100644 --- a/docs/src/features/graph.md +++ b/docs/src/features/graph.md @@ -496,4 +496,4 @@ Overlay health metrics onto graph nodes to quickly identify troubled components. - **[Data Integration](data-integration.md)** - Connect external systems and populate the graph with data - **[Entity Templates](../concepts/entity-templates.md)** - Define the types of entities in your graph -- **[Audit Trail](../concepts/audit.md)** - Track changes to entities and relationships +- **[Audit Trail](audit.md)** - Track changes to entities and relationships diff --git a/docs/src/features/index.md b/docs/src/features/index.md index 3d887cb3..a6f13ee7 100644 --- a/docs/src/features/index.md +++ b/docs/src/features/index.md @@ -1,9 +1,9 @@ --- title: Features -description: Explore the Internal Developer Platform features - Data Integration, Scorecards, Dashboards, Self-Service Actions, and AI Integration +description: Explore IDP-Core features, including data integration, audit history, graphs, scorecards, and self-service actions --- -The Internal Developer Platform provides a comprehensive set of features to build your Internal Developer Platform. This section covers current capabilities and the roadmap for future development. +IDP-Core provides features for integrating data, tracking changes, and building your Internal Developer Platform. This section covers available capabilities and planned features. ## Feature Overview @@ -13,17 +13,27 @@ The Internal Developer Platform provides a comprehensive set of features to buil --- - Connect to any data source through Webhooks, Kafka, or Pub/Sub. Map incoming data to entities using JSLT expressions. + Connect external systems and map incoming data to entities. **Status:** πŸ• Doing + **Details:** [Webhooks](webhooks.md) Β· [Entity Dynamic Mappings](entity-dynamic-mapping.md) + - πŸ•ΈοΈ **[Graph](graph.md)** --- Visualize entity relationships and dependencies as interactive graphs. - **Status:** πŸ• Done + **Status:** βœ… Available + +- πŸ“œ **[Audit](audit.md)** + + --- + + Track entity changes with revision history and user attribution. + + **Status:** βœ… Available - πŸ“ˆ **[Scorecards](scorecards.md)** diff --git a/docs/src/concepts/webhooks.md b/docs/src/features/webhooks.md similarity index 74% rename from docs/src/concepts/webhooks.md rename to docs/src/features/webhooks.md index d1030c8c..ff4da621 100644 --- a/docs/src/concepts/webhooks.md +++ b/docs/src/features/webhooks.md @@ -17,7 +17,7 @@ A webhook connector combines three concerns: ```mermaid flowchart LR - S[External system] --> E[POST /webhooks/{configurationId}] + S[External system] --> E["POST /webhooks/{configurationId}"] E --> H[InboundWebhookHandler] H --> D[Security dispatcher] D --> C[WebhookConnector] @@ -30,7 +30,7 @@ M --> T[Entity Template] A webhook connector is the runtime configuration stored by IDP-Core for one inbound integration. | Field | Type | Description | -|-----------------------|---------|--------------------------------------------------------| +| --------------------- | ------- | ------------------------------------------------------ | | `identifier` | String | Stable key used in the webhook URL and management APIs | | `name` | String | Human-readable name | | `description` | String | Optional explanation of the connector purpose | @@ -46,7 +46,10 @@ A webhook connector is the runtime configuration stored by IDP-Core for one inbo "name": "GitHub repositories", "description": "Receives repository events from GitHub", "enabled": true, - "mapping_identifiers": ["github-repo-update-mapping", "github-repo-delete-mapping"], + "mapping_identifiers": [ + "github-repo-update-mapping", + "github-repo-delete-mapping" + ], "security": { "type": "HMAC_SHA256", "config": { @@ -77,19 +80,20 @@ This validation keeps the connector configuration aligned with the current data Each connector declares one security type. IDP-Core validates the configuration at creation time and validates requests again at runtime. -| Type | Required configuration keys | Runtime behavior | -|----------------|---------------------------------------------------|--------------------------------------------------------------------------------------------------| -| `HMAC_SHA256` | `header_name`, `secret_alias`, `prefix` | Computes the SHA-256 HMAC of the raw body and compares it with the request header | -| `STATIC_TOKEN` | `header_name`, `secret_alias` | Compares a header value with a secret loaded from the environment | -| `BASIC_AUTH` | `username`, `secret_alias` | Compares the `Authorization: Basic ...` header with the configured username and secret | -| `JWT_BEARER` | `jwks_uri`, `client_id_field`, `client_id_values` | Validates the bearer token against a JWKS endpoint, then checks caller identity claim allow-list | -| `NONE` | none | Skips authentication | +| Type | Required configuration keys | Optional configuration keys | +| -------------- | ------------------------------------------------- | ------------------------------- | +| `HMAC_SHA256` | `header_name`, `secret_alias` | `prefix` (default is `sha256=`) | +| `STATIC_TOKEN` | `header_name`, `secret_alias` | None | +| `BASIC_AUTH` | `username`, `secret_alias` | None | +| `JWT_BEARER` | `jwks_uri`, `client_id_field`, `client_id_values` | `expected_audience` | +| `NONE` | None | None | > [!IMPORTANT] -> Security configuration keys accept `snake_case` and `camelCase` variants for the supported fields. +> Configuration keys accept `snake_case` and `camelCase` variants where supported. HMAC prefix uses the `prefix` key and has a default set to `sha256=`. > [!WARNING] -> `secret_alias` must reference an environment variable alias in `UPPER_SNAKE_CASE`. It does not store the raw secret -value in the connector configuration. +> Secret aliases reference environment variables; they do not store the raw secret value in the connector configuration. +> Use `MY_SECRET`, `env:MY_SECRET`, or `${MY_SECRET}` as the value. You can also use `secret_alias_env` or +> `secretAliasEnv` as the key and set its value to the environment variable name. ### Example Security Configurations @@ -149,16 +153,20 @@ value in the connector configuration. - `jwks_uri` (required): literal HTTPS URL used to validate JWT signatures. - `client_id_field` (required): claim name used to identify the caller. Allowed values are only `azp` or `email`. - `client_id_values` (required): comma-separated allow-list of accepted claim values. -- `expected_audience` (optional): comma-separated allow-list for the `aud` claim. If present, at least one JWT audience must match. -- `allowed-jwks-hosts` (optional, environment-backed): comma-separated allow-list of trusted JWKS hosts under `idp.security.webhook`. Set `ALLOWED_JWKS_HOSTS` when you need to permit a specific issuer host explicitly. +- `expected_audience` (optional): comma-separated allow-list for the `aud` claim. If present, at least one JWT audience must match. If omitted, IDP-Core does not check the audience. > [!WARNING] > `jwks_uri` must use a resolvable public HTTPS host. Local hosts, private networks, link-local addresses, and group-address destinations are rejected at connector creation. -> If you configure `allowed-jwks-hosts`, the listed hosts bypass this rejection path intentionally. -> This allow-list is configured under `idp.security.webhook.allowed-jwks-hosts`. `jwks_uri` is stored as a literal URL in the connector configuration. Environment references are not supported for this field. +### JWKS Host Allow-List + +The optional server-level property `idp.security.webhook.allowed-jwks-hosts` defaults to an empty list. Set the +`ALLOWED_JWKS_HOSTS` environment variable to a comma-separated list of hosts to allow. When the list is non-empty, +`jwks_uri` must use one of those hosts; allow-listed hosts bypass the private and local address rejection checks. +The URI must still use HTTPS. + ## Runtime Flow The webhook runtime uses a single generic endpoint: @@ -184,7 +192,7 @@ The request flow is: You manage webhook connectors through the inbound webhook management API, which exposes standard CRUD methods. | HTTP Method | Endpoint | Purpose | -|-------------|-----------------------------------------|------------------| +| ----------- | --------------------------------------- | ---------------- | | `POST` | `/api/v1/inbound_webhooks` | Create connector | | `GET` | `/api/v1/inbound_webhooks` | List connectors | | `GET` | `/api/v1/inbound_webhooks/{identifier}` | Get connector | @@ -194,7 +202,7 @@ You manage webhook connectors through the inbound webhook management API, which This separation keeps configuration management under versioned API routes while the event ingestion endpoint stays simple for external systems. -## When to Use Webhooks +## When to use Webhooks Use webhooks when an external system can push JSON events over HTTP and you want to: @@ -207,7 +215,7 @@ Use webhooks when an external system can push JSON events over HTTP and you want ## Next Steps -- **[Entity Templates](entity-templates.md)** - Define the target structures that mappings reference -- **[Entities](entities.md)** - Understand the records produced by successful ingestion -- **[Relations](relations.md)** - Model links that webhook mappings can populate +- **[Entity Templates](../concepts/entity-templates.md)** - Define the target structures that mappings reference +- **[Entities](../concepts/entities.md)** - Understand the records produced by successful ingestion +- **[Relations](../concepts/relations.md)** - Model links that webhook mappings can populate - **[Data Integration](../features/data-integration.md)** - Explore the broader ingestion roadmap diff --git a/docs/src/getting-started/index.md b/docs/src/getting-started/index.md index c3207c16..9ff1394f 100644 --- a/docs/src/getting-started/index.md +++ b/docs/src/getting-started/index.md @@ -62,7 +62,7 @@ Start the PostgreSQL service using Docker Compose: ```bash git clone https://github.com/decathlon/internal-developer-platform.git cd internal-developer-platform -docker-compose up -d +docker compose up -d ``` Build and run the Internal Developer Platform app: diff --git a/docs/src/getting-started/installation.md b/docs/src/getting-started/installation.md index 6031df0a..37b0c814 100644 --- a/docs/src/getting-started/installation.md +++ b/docs/src/getting-started/installation.md @@ -21,7 +21,7 @@ The recommended way to get started with the Internal Developer Platform is using 2. **Start the PG database with Docker Compose** ```bash - docker-compose up -d + docker compose up -d ``` 3. **Run the Internal Developer Platform Application** diff --git a/docs/zensical.toml b/docs/zensical.toml index f9434c84..343946a8 100644 --- a/docs/zensical.toml +++ b/docs/zensical.toml @@ -37,12 +37,17 @@ nav = [ "concepts/entity-filtering.md" ]}, "concepts/properties.md", - "concepts/relations.md", - "concepts/webhooks.md" + "concepts/relations.md" ]}, { "Features" = [ "features/index.md", - "features/data-integration.md", + "features/audit.md", + { "Data Integration" = [ + "features/data-integration.md", + "features/entity-dynamic-mapping.md", + "features/webhooks.md" + ]}, + "features/graph.md", "features/scorecards.md", "features/self-service-actions.md" ]}, @@ -54,7 +59,10 @@ nav = [ "deployment/docker.md", "deployment/kubernetes.md", "deployment/observability.md", - "deployment/configuration.md" + { "Configuration Reference" = [ + "deployment/configuration.md", + "concepts/authentication.md" + ]}, ]}, { "Contributing" = [ "contributing/index.md", @@ -69,11 +77,16 @@ nav = [ ]}, { "Architecture Decision Records" = [ "contributing/adrs/index.md", - "contributing/adrs/0001-doc-site-engine.md" + "contributing/adrs/0001-doc-site-engine.md", + "contributing/adrs/0002-code-architecture-pattern.md", + "contributing/adrs/0003-data-ingestion-framework.md", + "contributing/adrs/0004-dynamic-mapping-dsl.md" ]}, "contributing/development-setup.md", + "contributing/documentation.md", "contributing/pull-requests.md", "contributing/testing.md", + "contributing/owasp-zap-security.md", "contributing/ci-workflow.md", ]} ]