diff --git a/docs/configuration.md b/docs/configuration.md index 522794e..4bc0814 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -4,34 +4,35 @@ Documentation for configuring Shuffle. Most information is related to onprem and ## Table of contents -* [Introduction](#introduction) -* [Updating Shuffle](#updating-shuffle) -* [Production readiness](#production-readiness) -* [Shuffle Scaling](#scaling-shuffle) -* [Distributed Caching](#distributed-caching) -* [Kubernetes](#kubernetes) -* [Shuffle Apps on Kubernetes](#shuffle-apps-on-kubernetes) -* [Swarm HAProxy for worker load balancing](#swarm-haproxy-for-worker-load-balancing) -* [Swarm overlay network encryption](#swarm-overlay-network-encryption) -* [No Internet Install](#no-internet-install) -* [Proxy Configuration](#proxy-configuration) -* [App Certificates](#app-certificates) -* [HTTPS](#https) -* [OpenSearch TLS certificate setup](#opensearch-tls-certificate-setup) -* [IPv6](#ipv6) -* [Database](#database) -* [Database Change](#change-the-database-from-opensearch-to-elasticsearch) -* [Network Configuration](#network-configuration) -* [Docker Version error](#docker-version-error) -* [Database indexes](#database-indexes-opensearch) -* [Re-indexing & Index Management](#re-indexing--index-management) -* [Uptime monitoring](#uptime-monitoring) -* [Debugging](#debugging) -* [Execution Debugging](#execution-debugging) -* [Known Bugs](#known-bugs) -* [Shuffle Server Healthcheck](#shuffle-server-healthcheck) -* [Using podman](#using-podman) -* [Marketplace setup](#marketplace-setup) +- [Introduction](#introduction) +- [Updating Shuffle](#updating-shuffle) +- [Production readiness](#production-readiness) +- [Shuffle Scaling](#scaling-shuffle) +- [Distributed Caching](#distributed-caching) +- [Kubernetes](#kubernetes) +- [Shuffle Apps on Kubernetes](#shuffle-apps-on-kubernetes) +- [Swarm HAProxy for worker load balancing](#swarm-haproxy-for-worker-load-balancing) +- [Swarm overlay network encryption](#swarm-overlay-network-encryption) +- [No Internet Install](#no-internet-install) +- [Proxy Configuration](#proxy-configuration) +- [App Certificates](#app-certificates) +- [HTTPS](#https) +- [OpenSearch TLS certificate setup](#opensearch-tls-certificate-setup) +- [IPv6](#ipv6) +- [Database](#database) +- [Database Change](#change-the-database-from-opensearch-to-elasticsearch) +- [Network Configuration](#network-configuration) +- [Docker Version error](#docker-version-error) +- [Database indexes](#database-indexes-opensearch) +- [OpenSearch Permissions (Security Plugin / RBAC)](#opensearch-permissions-security-plugin--rbac) +- [Re-indexing & Index Management](#re-indexing--index-management) +- [Uptime monitoring](#uptime-monitoring) +- [Debugging](#debugging) +- [Execution Debugging](#execution-debugging) +- [Known Bugs](#known-bugs) +- [Shuffle Server Healthcheck](#shuffle-server-healthcheck) +- [Using podman](#using-podman) +- [Marketplace setup](#marketplace-setup) ## Introduction @@ -67,6 +68,7 @@ git pull docker-compose pull docker-compose up -d ``` + **Please note:** We will no longer provide support for older versions after **January 31, 2026**. To ensure uninterrupted access and the best possible experience, please upgrade to **Shuffle v2.1.1**. @@ -89,10 +91,12 @@ Shuffle is by default configured to be easy to start using. This means we have h ![image](https://github.com/user-attachments/assets/1bf288e0-fbd7-47c1-aba2-5269acaa4f8d) **Here are the things we'll dive into** + - [Environment Variables](#environment_variables) - [High Availability](#high-availability) ### Environment Variables + With Shuffle being a very technical system, it is important to understand that you have a lot of control mechanisms available to you in your local installation. [Check the .env file on your server](https://github.com/Shuffle/Shuffle/blob/main/.env) to see what the default values are. They are broadly split into the following: @@ -106,6 +110,7 @@ With Shuffle being a very technical system, it is important to understand that y **PS:** Most of our environment variables start with `SHUFFLE_` ### Servers + When setting up Shuffle for production, we always recommend two or more servers (VMs), but it works fine with one to start. These are MINIMUM requirements, and we recommend adding more to avoid congestion. The webserver is where your users and Shuffle's API is. Opensearch is a RAM heavy database, and we are doing A LOT of caching with to ensure scalable stability. @@ -119,11 +124,13 @@ The [default docker-compose file](https://github.com/shuffle/Shuffle/blob/main/d ### Hybrid Cloud Configuration -* Onprem: If you want to try using Hybrid Shuffle, see [Cloud sync documentation](/docs/organizations#cloud_synchronization) -* Cloud: If you want access to on-premises resources and API's, [set up extra Environments](/docs/organizations#environments) +- Onprem: If you want to try using Hybrid Shuffle, see [Cloud sync documentation](/docs/organizations#cloud_synchronization) +- Cloud: If you want access to on-premises resources and API's, [set up extra Environments](/docs/organizations#environments) ### High Availability + When running Shuffle on multiple servers, you need to take multiple things into account. Among them are: + - Can the servers talk to each other? - Do you want all Shuffle services on all servers? - Do you want distributed storage? @@ -133,20 +140,23 @@ When running Shuffle on multiple servers, you need to take multiple things into ![image](https://github.com/user-attachments/assets/5248417c-c47b-4397-95c5-b01e1f5b4082) Here is a breakdown of the previous High Availability image of Shuffle, and how it works: + 1. All the **Green** colored services are our providers, meaning they are built by someone else than Shuffle, but used in the Shuffle stack. Here is our recommendation on scaling these services: - * [Opensearch (Database)](https://opensearch.org/docs/latest/tuning-your-cluster/). Elasticsearch also works. If you are using more than one entrypoint to Opensearch/Elasticsearch, [add the URL's comma separated in the .env file](https://github.com/Shuffle/Shuffle/blob/c5ef50f523c041efaf53a1e285c1b19a30201e67/.env#L104). - * [Memcached (Shared Memory)](#distributed_caching): We recommend starting with Memcached on a single server, and only scaling up as needed. Shuffle can run without it, but when scaling runtime locations it gives the Backend, Orborus and Workers a shared cache instead of separate process-local caches. Add multiple [comma separated URL's here](https://github.com/Shuffle/Shuffle/blob/c5ef50f523c041efaf53a1e285c1b19a30201e67/.env#L84) to configure multiple instances. + - [Opensearch (Database)](https://opensearch.org/docs/latest/tuning-your-cluster/). Elasticsearch also works. If you are using more than one entrypoint to Opensearch/Elasticsearch, [add the URL's comma separated in the .env file](https://github.com/Shuffle/Shuffle/blob/c5ef50f523c041efaf53a1e285c1b19a30201e67/.env#L104). + - [Memcached (Shared Memory)](#distributed_caching): We recommend starting with Memcached on a single server, and only scaling up as needed. Shuffle can run without it, but when scaling runtime locations it gives the Backend, Orborus and Workers a shared cache instead of separate process-local caches. Add multiple [comma separated URL's here](https://github.com/Shuffle/Shuffle/blob/c5ef50f523c041efaf53a1e285c1b19a30201e67/.env#L84) to configure multiple instances. 2. **NFS** is Network File Storage. This is for you to be able to store files across multiple servers. This is required if you are running multiple instances of the Shuffle backend, and for them to have consistent access to the Files that you store. Only configure this if you are storing files in Shuffle. When NFS is set up, [mount your NFS storage to ./shuffle-files](https://github.com/Shuffle/Shuffle/blob/c5ef50f523c041efaf53a1e285c1b19a30201e67/docker-compose.yml#L28). 3. The **Blue** services are YOUR services. These can be in your Cloud, Onprem etc. The service in Shuffle that needs access to this are the `Apps`, which have their network configuration copied from the `Orborus` container. If you have on-premises services that Shuffle needs access to, set up an Orborus instance in the same network, which has access to your Shuffle instance + the service in question. 4. The **orange** services are Shuffle's containers. Below is a breakdown of what and how to use them. -* **Backend:** Handles all of Shuffle's API requests, and are typically routed through the Frontend service. This means that it usually does NOT expose a port. **Scale across ALL available servers.** Available on shuffle-http://backend:5001 in the container network. -* **Frontend:** Handles frontend & backend routing, as well as the default certificates. **Scale across ALL available servers.** Available on http://shuffle-frontend:3001 or https://shuffle-frontend:3443 in the container network. -* **Orborus / Worker / Apps:** Orborus is in charge of this stack, and is how you control all three services. Orborus receives jobs from the Backend, and does NOT expose any port. Read more about configuring, scaling and managing them in your instance on the [/admin?tab=locations](/admin?tab=locations) page, or in the next section. + +- **Backend:** Handles all of Shuffle's API requests, and are typically routed through the Frontend service. This means that it usually does NOT expose a port. **Scale across ALL available servers.** Available on shuffle-http://backend:5001 in the container network. +- **Frontend:** Handles frontend & backend routing, as well as the default certificates. **Scale across ALL available servers.** Available on http://shuffle-frontend:3001 or https://shuffle-frontend:3443 in the container network. +- **Orborus / Worker / Apps:** Orborus is in charge of this stack, and is how you control all three services. Orborus receives jobs from the Backend, and does NOT expose any port. Read more about configuring, scaling and managing them in your instance on the [/admin?tab=locations](/admin?tab=locations) page, or in the next section. ### Scaling Runtime Locations + Orborus can run in Docker-swarm mode, and in early 2023, with Kubernetes. This makes the workflow executions **A LOT** faster, use less resources, making it more scalable both on a single, as well as across multiple servers. Since September 2024, scale has been partially open source, and can be achieved with changing environment variables in the "Orborus" container for Shuffle. [Click here for Kubernetes details](https://github.com/Shuffle/Shuffle/tree/2.0.0/functions/kubernetes#instructions). If you have received a licensed version, don't forget step 3 to load in the correct worker. Let's begin with setting up Docker, Docker Compose, and creating a Docker Swarm network with two manager nodes involves several steps. Below is a step-by-step guide to achieve this: @@ -164,6 +174,7 @@ Docker Compose Installation Guide: https://docs.docker.com/compose/install/ Step 3: Load the license **(skip if not a customer)** You should have received a license from the Shuffle team, which comes in form of a URL. This URL can be used to download the licensed version of the Worker as many times as you want. After downloading it, you need to docker load the file. + ``` wget docker load -i shuffle-worker.zip @@ -172,7 +183,9 @@ docker load -i shuffle-worker.zip After these have been ran, it should be clear what the docker image is. This docker image needs to be used in the `SHUFFLE_WORKER_IMAGE` environment variable in step 4. **Step 4: Configure Orborus Environment Variables:** + 1. Add and change the following environment variables for Orborus in the docker-compose.yml file. `BASE_URL` is the external URL of the server you're running Shuffle on (the one you visit Shuffle with in your browser): + ``` # Required: - SHUFFLE_SWARM_CONFIG=run # Enables SWARM scaling @@ -202,6 +215,7 @@ To make swarm work, Please make sure that [these ports are open](https://docs.do **It is recommended to make sure that these ports are ONLY open internally to be sure that everything is secure.** 2. When step 1 is configured, take down the stack and pull it back up AFTER initializing swarm: + ``` docker swarm init docker-compose down @@ -242,14 +256,17 @@ If you get EOFs or timeouts for workers in machine B, look [here](https://shuffl ![](Aspose.Words.81096d25-bbff-47b2-a5ee-1ac38ad8ca4e.001.jpeg) ### Swarm: Manual worker service (Orborus scheduler-only) + If you want Orborus to only schedule jobs while you manage workers as a Swarm service yourself, you can pre-create the `shuffle-workers` service and point Orborus at it. Orborus will still poll the backend queue and send execution requests to `/api/v1/execute`. 1. Create the overlay network (if not already created): + ``` docker network create --driver=overlay --attachable shuffle_swarm_executions ``` 2. Create the worker service manually: + ``` docker service create \ --name shuffle-workers \ @@ -264,6 +281,7 @@ docker service create \ 3. Deploy `shuffle-orborus` after the worker service exists so the first execution can dispatch immediately. 4. Configure Orborus to use the manual worker service: + ``` - SHUFFLE_SWARM_CONFIG=run - SHUFFLE_WORKER_SERVER_URL=http://shuffle-workers:33333 @@ -275,6 +293,7 @@ docker service create \ 5. Ensure Orborus is attached to the same overlay network as `shuffle-workers`. **Limitations and gotchas** + - Orborus still calls the Docker Swarm API (service list/create, network attach). It must run with manager-level Docker API access (direct socket or a socket proxy that exposes swarm/service/network endpoints). - Workers also need manager-level Docker API access to deploy app services during executions; without it, app execution fails. - If `shuffle-workers` is deleted or scaled to 0, Orborus will try to recreate it and may fail if it lacks permissions. @@ -337,6 +356,7 @@ SHUFFLE_MEMCACHED=:PORT ``` ### Distributed Caching + Once you have a scalable version of Shuffle, using Docker Swarm or Kubernetes, it becomes important for short-lived execution data to be visible across services. Shuffle supports distributed caching [in the form of Memcached](https://hub.docker.com/_/memcached). Memcached helps reduce database load and gives runtime-location components a shared cache while workers are deployed and executions are updated. - Backend @@ -346,6 +366,7 @@ Once you have a scalable version of Shuffle, using Docker Swarm or Kubernetes, i The database remains the source of truth. Memcached is used for cache keys such as execution data, action results, validation data, health/stat counters and worker coordination. If `SHUFFLE_MEMCACHED` is empty, each service falls back to its own local in-memory cache. To make use of Memcached, start a Memcached service on a host or service network Shuffle can access, then configure each service to use it with `SHUFFLE_MEMCACHED`. The default port is 11211. Here is a quickstart that reserves 1024 Mb of memory: + ``` docker run --name shuffle-cache -p 11211:11211 -d memcached -m 1024 ``` @@ -353,6 +374,7 @@ docker run --name shuffle-cache -p 11211:11211 -d memcached -m 1024 **PS: This requires swap limit capabilities on the Docker host. [More about running it in Docker here](https://hub.docker.com/_/memcached)** Once this is up, it will be listening on port 11211. From here, set `SHUFFLE_MEMCACHED` on the Backend and Orborus. Orborus forwards the same value into workers it starts, including single Docker workers, the `shuffle-workers` Swarm service and Kubernetes worker deployments. Here's an example that fits into your docker-compose file: + ``` services: shuffle-backend: @@ -364,6 +386,7 @@ services: ``` Set the same value on Orborus: + ``` services: orborus: @@ -374,6 +397,7 @@ services: ``` You can additionally add this to your docker compose with the following setting: + ``` memcached: image: memcached:latest @@ -389,17 +413,19 @@ You can additionally add this to your docker compose with the following setting: ``` ### Memcached on a Docker Swarm network + When running with `SHUFFLE_SWARM_CONFIG=run`, Memcached must be reachable from the worker network. Orborus creates or uses the overlay network from `SHUFFLE_SWARM_NETWORK_NAME`, and workers use that network when they execute workflows. If Memcached only exists on the main Shuffle network, workers may fail to resolve or connect to it. You do not need to expose Memcached on the host machine for this setup. Do not add `-p 11211:11211`, `--publish`, or a Compose `ports:` entry unless something outside Docker must connect to Memcached. For Shuffle Swarm, keep Memcached internal and use `tasks.shuffle-cache:11211` from services attached to the overlay network. The common pattern is: -* Backend reaches Memcached on the main Shuffle network. -* Orborus reaches Memcached and forwards `SHUFFLE_MEMCACHED` to workers. -* Workers reach Memcached on the execution overlay network. +- Backend reaches Memcached on the main Shuffle network. +- Orborus reaches Memcached and forwards `SHUFFLE_MEMCACHED` to workers. +- Workers reach Memcached on the execution overlay network. Attach Memcached to both networks if backend and workers are not on the same overlay: + ``` docker network create --driver=overlay --attachable shuffle docker network create --driver=overlay --attachable shuffle_swarm_executions @@ -415,11 +441,13 @@ docker service create \ ``` Then set the same endpoint on backend and Orborus: + ``` - SHUFFLE_MEMCACHED=tasks.shuffle-cache:11211 ``` If you manage the worker service manually, attach it to the same execution network: + ``` docker service create \ --name shuffle-workers \ @@ -430,6 +458,7 @@ docker service create \ ``` For a Swarm stack file, the important parts look like this: + ``` services: shuffle-cache: @@ -467,14 +496,17 @@ networks: ``` Verify connectivity from a temporary container on the execution network: + ``` docker run --rm --network shuffle_swarm_executions alpine sh -c "apk add --no-cache busybox-extras >/dev/null && nc -vz tasks.shuffle-cache 11211" ``` ### Multi-server memcached + You can run Memcached on multiple servers as well, but may run into key inconsistency if clients do not route the same key to the same server. This should however not affect how things run in Shuffle, as we verify and fix request data. To do this, add multiple Memcached instances to the environment variable, comma separated. Example: + ``` - SHUFFLE_MEMCACHED=10.0.0.1:11211,10.0.0.2:11211,10.0.0.3:11211 ``` @@ -502,6 +534,7 @@ To implement this, follow these steps: - Define in all your nodes the list of hostnames in /etc/hosts so all the machines can do the necessary IP resolutions when they have a hostname. - Configure an OpenSearch database cluster. For this step, it is recommended to follow the official documentation [https://opensearch.org/docs/latest/tuning-your-cluster/index/](https://opensearch.org/docs/latest/tuning-your-cluster/index/). Anyways to achieve this in a easy way you just need to change this settings in each opensearch node and reload the service: + ``` cluster.name -> Set its new value to "shuffle-cluster". node.name -> Set its new value to the hostname of the current node. @@ -552,6 +585,7 @@ networks: shuffle: driver: bridge ``` + Also for this step you need to change the value of a variable inside the .env so it looks like this:`SHUFFLE_OPENSEARCH_URL=SHUFFLE_OPENSEARCH_URL=https://192.168.0.30:9200,https://192.168.0.31:9200,https://192.168.0.32:9200`. Each one of that IPs is the corresponding one to each opensearch node. - Deploy as many Orborus nodes as desired. At a minimum, it is recommended to deploy 2 nodes. If scaling is required regarding the maximum number of concurrently executing Workflows, the number of these nodes can be increased. The dockerfile of each one of the orborus nodes should look like this: @@ -594,6 +628,7 @@ Shuffle use with Kubernetes is now possible due to help from our contributors. Y Due to Kubernetes not being capable of building Shuffle Apps directly, an additional container for building them is available. ### Shuffle Apps on Kubernetes + By default, Shuffle Worker creates a Kubernetes Deployment and Service for each app. Each app and version has its own Deployment and Service. Shuffle automatically deploys a set of apps, and other apps are deployed on demand when they are first used. You can use `app.*` Helm values to control parts of the app deployment (resources, security context, etc.). These values are converted to environment variables on Orborus, and Orborus passes them to Worker when creating app Deployments. When `worker.enableHelmDeployment` is set, the app configuration is set on the worker directly. This configuration applies to all apps (you cannot scale or resource-tune individual apps in this mode). @@ -601,6 +636,7 @@ You can use `app.*` Helm values to control parts of the app deployment (resource If you want to set CPU and memory for apps that are created dynamically by Worker, use `app.resources` (or `app.resourcesPreset`) in the chart. These values are applied to all dynamically created app Deployments. If you want full control, you can deploy apps using Helm instead. This gives you: + - full control over app deployments via Helm values - granular control per app and version (replicas, resources, etc.) - fewer issues with on-demand started apps (see https://github.com/Shuffle/Shuffle/issues/1739) @@ -633,6 +669,7 @@ apps: ``` Notes: + - The key under `apps` is only used to identify the app in values. It can be any unique name. - You can override any `app.*` value for a specific app via `apps..*` (for example, `apps.shuffleTools.replicaCount`). - Hybrid mode is supported: deploy some apps with Helm while still letting Worker create others on demand. @@ -640,6 +677,7 @@ Notes: If you do not want Worker to manage app deployments, set `worker.manageAppDeployments=true`. This removes the required permissions from the Shuffle Worker Kubernetes Service Account and requires you to deploy all apps manually using Helm. #### Shuffle App Service Accounts + By default, apps use a shared `shuffle-app` service account. If you deploy apps with Helm, you can use a dedicated service account per app. ```yaml @@ -664,10 +702,13 @@ apps: All app service accounts use the `shuffle-app` role by default. ### Orborus with Kubernetes + To configure Kubernetes, you need to specify a single environment variable for Orborus: RUNNING_MODE. By setting the environment variable RUNNING_MODE=kubernetes, execution should work as expected! ### Scaling Kubernetes + To scale Shuffle in Kubernetes, use the following environment variables in the Orborus container: + ```bash SHUFFLE_SCALE_REPLICAS=3 # HPA coming soon. This is for static scaling. SHUFFLE_WORKER_IMAGE=ghcr.io/shuffle/shuffle-worker-scale:nightly @@ -677,9 +718,11 @@ SHUFFLE_MEMCACHED=shuffle-memcached:11211 # this depends on your setup. ``` ### Private apps on Kubernetes + To run private apps on Kubernetes, you need a private container registry to pull images from. Once you configure `REGISTRY_URL` to point to your private registry, Shuffle will automatically pull app images from there. To install the Shuffle Helm chart pointing to your custom registry, run: + ```bash helm install shuffle oci://ghcr.io/shuffle/charts/shuffle \ --namespace shuffle \ @@ -690,6 +733,7 @@ helm install shuffle oci://ghcr.io/shuffle/charts/shuffle \ If your registry allows unauthenticated pushes (or you're not using Docker Hub), you can skip this step. However, if your private registry requires authentication, create and share a Docker registry secret using: + ```bash kubectl create secret docker-registry \ --docker-server= \ @@ -704,6 +748,7 @@ kubectl set env deployment/backend SHUFFLE_REGISTRY_SECRET="" After switching to your private registry, the images for already-installed apps will not be available in the new registry. To avoid image-pull failures, you can mirror all existing Shuffle app images from Docker Hub into your private registry. You can use `skopeo` to mirror the entire repository (docker.io/frikky/shuffle) or run the following script to pull and push every tag manually: + ```bash registry="your-registry.domain:5000" @@ -724,12 +769,15 @@ done ``` ### Swarm HAProxy for worker load balancing + In Docker Swarm, Orborus sends requests to `shuffle-workers:33333` through the Swarm VIP. The routing mesh is not load-aware, so requests can hit saturated workers and time out under load. You can improve distribution by placing HAProxy in front of the worker tasks and pointing Orborus to HAProxy. This can be done after the Swarm stack is up, without changing how Orborus deploys workers or apps. #### 1) Create the HAProxy config + Create a file named `haproxy.cfg` on a Swarm manager: + ```cfg global log stdout format raw local0 @@ -761,12 +809,15 @@ backend worker_back ``` #### 2) Deploy HAProxy as a Swarm service + Ensure the worker overlay network exists (Orborus creates `shuffle_swarm_executions` by default): + ```bash docker network create --driver=overlay --attachable shuffle_swarm_executions ``` Create the HAProxy config and deploy it. Do not publish port 33333 on HAProxy if the worker service already uses it (Swarm only allows one published service per port). Orborus can reach HAProxy over the overlay network: + ```bash docker config create haproxy-cfg ./haproxy.cfg @@ -778,17 +829,21 @@ docker service create \ ``` #### 3) Point Orborus to HAProxy + Set the worker server URL so Orborus sends executions through HAProxy: + ```bash SHUFFLE_WORKER_SERVER_URL=http://shuffle-haproxy:33333 ``` Restart Orborus after updating the environment: + ```bash docker restart shuffle-orborus ``` #### 4) Verify + ```bash docker service logs shuffle-haproxy --tail 200 docker service ps shuffle-haproxy @@ -1093,14 +1148,14 @@ If tasks stay `Pending`, lower replicas or add more eligible nodes. If you want encrypted node-to-node traffic on Swarm overlays, pre-create encrypted networks and point your stack file to those external networks. -1) Pre-create encrypted overlays: +1. Pre-create encrypted overlays: ```bash docker network create --driver overlay --attachable --opt encrypted shuffle docker network create --driver overlay --attachable --opt encrypted swarm_executions ``` -2) In your stack/compose config file, use external networks: +2. In your stack/compose config file, use external networks: ```yaml networks: @@ -1112,13 +1167,13 @@ networks: name: swarm_executions ``` -3) Deploy as normal: +3. Deploy as normal: ```bash docker stack deploy -c docker-compose.yml shuffle ``` -4) Verify encryption and service attachment: +4. Verify encryption and service attachment: ```bash docker network inspect shuffle | grep -i encrypted @@ -1369,6 +1424,7 @@ For additional OpenSearch sizing, shard allocation, and cluster tuning guidance, - https://opensearch.org/docs/latest/tuning-your-cluster/ ## Networking + Networking with Shuffle is pretty straight forward. What we check for are the following: - Can Shuffle reach your services? @@ -1380,29 +1436,32 @@ There are however many things that can go wrong with these simple mechanisms, le Proxies are another requirement to many enterprises, hence it's an important feature to support. There are two places where proxies can be implemented: -* Shuffle Backend: Connects to Github and Dockerhub. -* Shuffle Orborus: Connects to Dockerhub and Shuffle Backend. +- Shuffle Backend: Connects to Github and Dockerhub. +- Shuffle Orborus: Connects to Dockerhub and Shuffle Backend. **PS: Orborus settings are also set for the Worker** To configure these, there are two options: -* Internal vs External proxy (shuffle vs apps) -* Individual containers -* Globally for Docker +- Internal vs External proxy (shuffle vs apps) +- Individual containers +- Globally for Docker To **DISABLE** proxy for **internal** Shuffle traffic (between internal containers), add the following environment variables to Orborus: + ``` SHUFFLE_INTERNAL_HTTP_PROXY=noproxy SHUFFLE_INTERNAL_HTTPS_PROXY=noproxy ``` To disable proxy use for specific domains, use NO_PROXY with comma separation between domains: + ``` NO_PROXY=myinternal-domain.com,random.org ``` If you need a separate no-proxy list for internal Shuffle traffic, set one of these on Orborus: + ``` SHUFFLE_INTERNAL_NO_PROXY=shuffle-backend,shuffle-workers # Legacy alias also supported by backend/shared code: @@ -1456,6 +1515,7 @@ HTTPS_PROXY= ``` ### Opensearch / Elasticsearch proxies + Connections from Shuffle's backend to the Opensearch database **does NOT** follow normal HTTP_PROXY and NOPROXY environment variables. Opensearch and Elasticsearch proxy configuration can be set using the `SHUFFLE_OPENSEARCH_PROXY` environment variable. @@ -1469,10 +1529,13 @@ To set up proxies in individual containers, open docker-compose.yml and add the ![Proxy containers](https://github.com/shuffle/shuffle-docs/blob/master/assets/proxy-containers.png?raw=true) ### Orborus running on a different network + All you'll need to do is allow orborus to have access to the backend OR frontend of Shuffle. ### Internal vs External proxy + As of November 2023, we added another way to configure a difference between these two: + - Internal tools like Backend -> Orborus -> Worker <-> Apps - Apps -> External tools @@ -1515,10 +1578,10 @@ The New Routing steps are automatic as long as you update the BASE_URL to point Necessary info for the truststore to create TLS/SSL certificates: -* Certificates are located in ./frontend/certs. -* ./frontend/README.md contains information on generating a self-signed cert -* (default): Privatekey is named privkey.pem -* (default): Fullchain is named fullchain.pem +- Certificates are located in ./frontend/certs. +- ./frontend/README.md contains information on generating a self-signed cert +- (default): Privatekey is named privkey.pem +- (default): Fullchain is named fullchain.pem If you want to change this, edit ./frontend/Dockerfile and ./frontend/nginx.conf. @@ -1531,21 +1594,25 @@ After changing certificates, you can rebuild the entire frontend by running (./f Make sure that the output image is the same in your docker-compose.yml file. This should work seemlessly for you next. ### App Certificates + As of November 2023, it is possible to mount folders into apps. This is in order for you to have better control of what Shuffle Apps can do, with the main reason being to manage certificates or dynamic, large files. To mount in certificates to ALL App containers, add the following environment variable to the "Orborus" container, but change the source and destination folder. The item BEFORE the colon (:) is the source folder on your machine, with the one AFTER the colon (:) being for the destination folder in the app itself. If you want more multiple folders mounted, add them with a comma. Folders MUST exist, otherwise apps may not run, and the Worker will throw an error in the logs with Workflow Runs not finishing. If you are in production, we recommend trying this feature in a separate Runtime Location. + ``` SHUFFLE_VOLUME_BINDS="/etc/ssl/certs:/usr/local/share/ca-certificates,/srcfolder:/dstfolder" ``` ### Using the Nginx Reverse Proxy for TLS/SSL + If you intend to use Nginx as a Reverse Proxy, the main steps are below. [Here is a basic single-server architecture for it](https://jamboard.google.com/d/1zJU8yMzbsu-XWeZnch_5MoDwmMNkkN8ZmoGNLCaHPlU/edit?usp=sharing). The Docker version is further down. 1. [Install Nginx](https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-20-04) on your server (find the correct distro), or in a [Docker container by itself](https://www.theserverside.com/blog/Coffee-Talk-Java-News-Stories-and-Opinions/Docker-Nginx-reverse-proxy-setup-example). 2. Make sure you have a VALID certificate that matches your domain/hostname and [add this to your Nginx server](https://phoenixnap.com/kb/install-ssl-certificate-nginx) 3. In the nginx.conf file (/etc/nginx/conf.d/default.conf or similar), under "server", add the information below. Make sure to change the "proxy_pass" part. This is how it will redirect all /api requests. + ``` location / { proxy_pass SHUFFLE FRONTENDIP; @@ -1559,10 +1626,13 @@ location / { proxy_ssl_verify off; } ``` + 4. Restart Nginx! `systemctl restart nginx` ### Nginx in Docker + 1. Add the following service to your docker-compose.yml + ``` nginx-proxy: image: nginx:latest @@ -1578,6 +1648,7 @@ location / { ``` 2. Add a new nginx configuration file called `nginx-conf` with the following (you may add additional Nginx configuration to this): + ``` server { listen 443 ssl; @@ -1605,10 +1676,11 @@ server { 4. Restart everything: `docker-compose down; docker-compose up -d` ### Internal Certificate Authority -By default, certificates are not being verified when outbound traffic goes from Shuffle. This is due to the massive use of self-signed certificates when using internal services. You may ignore certificate warnings by adding `SHUFFLE_SKIPSSL_VERIFY=true` to the environment of each relevant service - most notably used for Orborus. If you want to accept your Certificate Authority for all requests, there are a few ways to do this: + +By default, certificates are not being verified when outbound traffic goes from Shuffle. This is due to the massive use of self-signed certificates when using internal services. You may ignore certificate warnings by adding `SHUFFLE_SKIPSSL_VERIFY=true` to the environment of each relevant service - most notably used for Orborus. If you want to accept your Certificate Authority for all requests, there are a few ways to do this: 1. Mount your CA certificates (recommended): Add the `./certs:/certs` mount to the Orborus service in your docker-compose.yml. Ensure that the shuffle directory contains a certs subdirectory with all the necessary certificate files. This will automatically append all certificates in `./certs` to the system's root CA. -2. Docker Daemon level - point to your cert: `$ dockerd --tlscacert=/path/to/custom-ca-cert.pem` +2. Docker Daemon level - point to your cert: `$ dockerd --tlscacert=/path/to/custom-ca-cert.pem` 3. Add it to every app (per-image configuration). You can do this by modifying the Dockerfile for an app and manually building it with the certificate in the Dockerfile of each Docker image. Restart Shuffle after this is done. As this may require advanced Docker understanding, reach out to ask us about it: [support@shuffler.io](mailto:support@shuffler.io) @@ -1619,7 +1691,6 @@ Shuffle supports IPv6 in Docker by default, but your docker engine may not. IPv6 [https://docs.docker.com/config/daemon/ipv6/](https://docs.docker.com/config/daemon/ipv6/) - ### Enterprise Environments In most enterprise environments, Shuffle will be behind firewalls, proxies and other networking equipment. If this is the case, below are the requirements to make Shuffle work anywhere. The most common issue has to do with downloads from Alpine linux's Docker images while Shuffle is running. @@ -1628,20 +1699,17 @@ In most enterprise environments, Shuffle will be behind firewalls, proxies and o ### Change the Database from OpenSearch to Elasticsearch -- Open the Docker-compose.yml file in the Shuffle directory. Find the OpenSearch container section and either comment out or remove the details. Save your modifications to the file. +- Open the Docker-compose.yml file in the Shuffle directory. Find the OpenSearch container section and either comment out or remove the details. Save your modifications to the file. - ![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/5a9ff541-14f4-4ddc-8c04-08a874ffc3ff) + ![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/5a9ff541-14f4-4ddc-8c04-08a874ffc3ff) -- Now open the .env file and change the below value in the .env from false to true for Elasticsearch database enable. +- Now open the .env file and change the below value in the .env from false to true for Elasticsearch database enable. ![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/f1116c97-daa2-48ba-80f4-f14803c1629d) -- Find the part in the .env file that defines database configurations. Update the Elasticsearch host configuration using your Elasticsearch IP address. - - ![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/13277ad5-269d-44a7-ab3b-13916bb9ce0e) - - +- Find the part in the .env file that defines database configurations. Update the Elasticsearch host configuration using your Elasticsearch IP address. +![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/13277ad5-269d-44a7-ab3b-13916bb9ce0e) ### Domain Whitelisting @@ -1667,7 +1735,7 @@ production.cloudflare.docker.com # Protects of DockerHub ### Incoming IP Whitelisting -When using Shuffle in the cloud (*.shuffler.io), the incoming IP to your services by default will be be from our cloud functions, if you are not using [Runtime Locations](/admin?tab=locations). The range is **not static**, and may wary based on region. Here's a list (mostly IPv6 as of 2025): +When using Shuffle in the cloud (\*.shuffler.io), the incoming IP to your services by default will be be from our cloud functions, if you are not using [Runtime Locations](/admin?tab=locations). The range is **not static**, and may wary based on region. Here's a list (mostly IPv6 as of 2025): ``` Default (London): 2600:1900:2000:2a:400::0 -> 2600:1900:2000:2a:400::ffff @@ -1688,10 +1756,12 @@ Architecture connecting from cloud to onprem (hybrid): ![image](https://github.com/user-attachments/assets/7f0b6146-ebae-4133-bbc7-8b158d48c3a9) ### Static inbound IPs for cloud -As a customer of Shuffle we can provide you with a static **IP range** with secondary domains that can be used for cloud routing. May incur costs depending on needs. Contact support@shuffler.io for more info. + +As a customer of Shuffle we can provide you with a static **IP range** with secondary domains that can be used for cloud routing. May incur costs depending on needs. Contact support@shuffler.io for more info. Areas of relevance: -- Your Environment -> Shuffle Cloud (*.shuffler.io) + +- Your Environment -> Shuffle Cloud (\*.shuffler.io) - Shuffle Cloud -> Your Enviroment ### Manual Docker image transfers @@ -1722,9 +1792,11 @@ docker load shuffle_tools.tar ``` ### No Internet Install + This procedure will help you export what you need to run Shuffle on a no internet host. **The following features will not work without internet:** + - Cloud Sync - Automatic Health System (/health page) - UI app activations @@ -1732,9 +1804,11 @@ This procedure will help you export what you need to run Shuffle on a no interne - Depending on your network: Search engine. This is a frontend feature, if you are in a no-internet zone, it will stop working. 1. Prerequise -* Both machines has Docker and Docker Compose installed already -* Your host machine already needs the images on it to make them exportable +- Both machines has Docker and Docker Compose installed already + +- Your host machine already needs the images on it to make them exportable + 2. Pull images on original machine Shuffle need a few base images to work: @@ -1810,141 +1884,292 @@ To modify the database location, change "DB_LOCATION" in .env (root dir) to your Shuffle uses OpenSearch (or Elasticsearch) as its primary database. Below is a complete list of all indexes used by Shuffle, organized by category. #### Default Credentials + The default OpenSearch credentials for Shuffle are: + - **Username:** `admin` - **Password:** `StrongShufflePassword321!` These can be changed via environment variables in your `.env` file: + - `SHUFFLE_OPENSEARCH_USERNAME` - `SHUFFLE_OPENSEARCH_PASSWORD` #### Index Prefix + You can set a custom prefix for all indexes using the `SHUFFLE_OPENSEARCH_INDEX_PREFIX` environment variable. This is useful when running multiple Shuffle instances on a single OpenSearch cluster. Example: If `SHUFFLE_OPENSEARCH_INDEX_PREFIX=prod`, the `workflow` index becomes `prod_workflow`. #### Core Indexes -| Index | Description | -|-------|-------------| -| `workflow` | Workflow definitions and configurations | -| `workflowexecution` | Workflow execution records and results | -| `workflowapp` | Workflow application definitions | -| `workflowappauth` | App authentication configurations | -| `workflowappauthgroup` | Authentication group configurations | -| `workflowqueue-{env}` | Execution queue per environment (dynamic, based on Orborus environment) | -| `workflow_revisions` | Workflow version history | -| `app_revisions` | App version history | +| Index | Description | +| ---------------------- | ----------------------------------------------------------------------- | +| `workflow` | Workflow definitions and configurations | +| `workflowexecution` | Workflow execution records and results | +| `workflowapp` | Workflow application definitions | +| `workflowappauth` | App authentication configurations | +| `workflowappauthgroup` | Authentication group configurations | +| `workflowqueue-{env}` | Execution queue per environment (dynamic, based on Orborus environment) | +| `workflow_revisions` | Workflow version history | +| `app_revisions` | App version history | #### Organization & User Indexes -| Index | Description | -|-------|-------------| -| `organizations` | Organization data and settings | -| `users` | User accounts and profiles | -| `sessions` | User session tokens | -| `apikey` | API key storage | -| `partners` | Partner organization data | -| `org_statistics` | Organization usage statistics | -| `org_cache` | Organization cache data (KV store) | -| `org_cache_revisions` | Cache version history | +| Index | Description | +| --------------------- | ---------------------------------- | +| `organizations` | Organization data and settings | +| `users` | User accounts and profiles | +| `sessions` | User session tokens | +| `apikey` | API key storage | +| `partners` | Partner organization data | +| `org_statistics` | Organization usage statistics | +| `org_cache` | Organization cache data (KV store) | +| `org_cache_revisions` | Cache version history | #### Files & Storage Indexes -| Index | Description | -|-------|-------------| -| `files` | File metadata storage | -| `datastore_category` | Datastore category definitions | -| `datastore_ngram` | N-gram data for search functionality | -| `oauth2_storage` | OAuth2 token storage | +| Index | Description | +| -------------------- | ------------------------------------ | +| `files` | File metadata storage | +| `datastore_category` | Datastore category definitions | +| `datastore_ngram` | N-gram data for search functionality | +| `oauth2_storage` | OAuth2 token storage | #### Triggers & Automation Indexes -| Index | Description | -|-------|-------------| -| `hooks` | Webhook configurations | -| `schedules` | Scheduled task definitions | -| `pipelines` | Pipeline definitions | -| `trigger_auth` | Trigger authentication data | -| `environments` | Environment configurations | +| Index | Description | +| -------------------- | --------------------------- | +| `hooks` | Webhook configurations | +| `schedules` | Scheduled task definitions | +| `pipelines` | Pipeline definitions | +| `trigger_auth` | Trigger authentication data | +| `environments` | Environment configurations | | `gmail_subscription` | Gmail trigger subscriptions | #### Notifications & Communication Indexes -| Index | Description | -|-------|-------------| -| `notifications` | Notification records | -| `conversations` | AI conversation data | +| Index | Description | +| ----------------------- | --------------------- | +| `notifications` | Notification records | +| `conversations` | AI conversation data | | `conversation_metadata` | Conversation metadata | -| `suggestions` | System suggestions | +| `suggestions` | System suggestions | #### Statistics & Monitoring Indexes -| Index | Description | -|-------|-------------| -| `platform_health` | Platform health check data | -| `app_execution_values` | App execution value cache | -| `app_stats` | App usage statistics | -| `creator_stats` | Creator statistics | -| `environment_stats` | Environment statistics | -| `singul_stats` | Singul integration stats | +| Index | Description | +| ----------------------- | -------------------------- | +| `platform_health` | Platform health check data | +| `app_execution_values` | App execution value cache | +| `app_stats` | App usage statistics | +| `creator_stats` | Creator statistics | +| `environment_stats` | Environment statistics | +| `singul_stats` | Singul integration stats | | `live_execution_status` | Real-time execution status | #### Detection & Rules Indexes -| Index | Description | -|-------|-------------| -| `disabled_rules` | Disabled detection rules | +| Index | Description | +| ---------------- | -------------------------------- | +| `disabled_rules` | Disabled detection rules | | `selected_rules` | Selected/enabled detection rules | #### Other Indexes -| Index | Description | -|-------|-------------| -| `openapi3` | OpenAPI spec storage | -| `usecases` | Use case definitions | -| `reseller_deal` | Reseller deal data | -| `training` | Training/ML data | -| `synckey` | Cloud sync key data | -| `shuffle_logs` | Shuffle system logs | - -#### Indexes with Aliasing & Rollover - -The following 11 indexes are configured with **aliasing and automatic rollover** for better scaling. These indexes are created with the pattern `{index}-000001` and have an alias pointing to the write index. This is handled by `InitOpensearchIndexes()` on startup. - -``` -workflowexecution, datastore_ngram, org_cache, org_cache_revisions, -notifications, shuffle_logs, environments, org_statistics, -workflowapp, workflow, workflow_revisions -``` +| Index | Description | +| --------------- | -------------------- | +| `openapi3` | OpenAPI spec storage | +| `usecases` | Use case definitions | +| `reseller_deal` | Reseller deal data | +| `training` | Training/ML data | +| `synckey` | Cloud sync key data | +| `shuffle_logs` | Shuffle system logs | + +#### Indexes with Rollover (Append-Only Stores) + +Only genuinely **append-only** data gets aliasing and automatic rollover for scaling. These are created with the pattern `{index}-000001`, use an alias pointing to the write index, and may roll into new generations (with optional ISM retention). This is handled by `InitOpensearchIndices()` on startup. + +``` +shuffle_logs, workflow_revisions, org_cache_revisions, workflowexecution (archive - see below) +``` + +**Stateful keyed stores stay on a single backing index** (no rollover). Each document has a stable `_id` that is updated in place — e.g. executions move `RUNNING -> FINISHED/ABORTED`, notifications toggle `read`/`ignored` — so rolling them would split an `_id` across generations and silently produce duplicate documents (OpenSearch routes alias writes to the current write-index generation with no cross-generation `_id` awareness). On startup, Shuffle collapses any legacy multi-generation state back into one backing index, keeping the newest copy of each `_id`: + +``` +datastore_ngram, org_cache, environments, org_statistics, +workflowapp, workflow, datastore_category, notifications, workflowexecution_live +``` + +#### workflowexecution: Hot/Cold Lifecycle + +`workflowexecution` is neither a pure append-only store nor a simple keyed +store: the same `execution_id` is rewritten in place while a run is +in-flight (`EXECUTING` → `FINISHED`/`ABORTED`), which makes plain rollover +unsafe, but its overall volume needs rollover-style scaling once history +accumulates. Shuffle handles this with a two-index split: + +- **`workflowexecution_live`** - a small, single, keyed index holding every + execution that is not yet in a terminal state, plus terminal executions + still inside a short grace period. All execution writes land here first. +- **`workflowexecution`** - the existing index/alias, which now serves as an + append-only, rollover-managed **archive**. A background sweep moves + confirmed-terminal executions here once they're past the grace period. + +Reads check the live index first and fall back to the archive; list/history +queries search both in a single request. Existing deployments upgrade with +zero downtime: the current `workflowexecution` index becomes the archive +as-is (no reindex of history), and only currently in-flight executions are +copied into the new live index on first startup after upgrading. + +If a rerun/continue is attempted on an execution that has already been +archived, the API returns a clean `{"success": false, "reason": "..."}` +response instead of an error, since archived executions are immutable. + +Configuration: + +- `OPENSEARCH_EXECUTION_GRACE_PERIOD` (default `1h`, Go duration string, e.g. `90m`) - how long a terminal + execution stays in the live index before becoming archivable. +- `OPENSEARCH_EXECUTION_ARCHIVE_SWEEP_INTERVAL` (default `30m`, Go duration string, e.g. `15m`) - how often + the archival sweep runs. +- Archive retention defaults to `365` days and is configured the same way + as every other rollover index's retention - via the existing + `OPENSEARCH_INDEX_RETENTION_DAYS` JSON env var (e.g. + `{"workflowexecution": 180}`). No dedicated env var was added for this. +- `SHUFFLE_SKIP_EXECUTION_LIVE_MIGRATION` (default unset/runs) - skips only + the one-time startup migration of pre-upgrade in-flight executions into + `workflowexecution_live`. Safe to disable: any execution left behind + self-heals the first time it's written to with a non-terminal status. + Useful for very large deployments that want to run this migration + manually during a maintenance window instead of automatically on every + restart. +- `SHUFFLE_SKIP_EXECUTION_ARCHIVAL_SWEEP` (default unset/runs) - skips only + the recurring sweep that moves terminal executions from live to archive. + Disabling this defeats the point of the hot/cold split (the live index + will grow unbounded), so a warning is logged on every startup while set. + +**Note:** these two flags are independent of the existing +`SHUFFLE_SKIP_OPENSEARCH_INDEX_INIT`, which only controls whether Shuffle +creates/manages OpenSearch indexes, mapping templates, and ISM policies - +for operators who provision their OpenSearch schema externally. Setting +that flag does not disable the execution lifecycle jobs above; they are +started independently so self-hosted operators who manage their own index +infrastructure still get the scaling benefit of this feature. **Default rollover conditions:** + - Max age: 90 days - Max size: 40GB - Max documents: 1,000,000 **Default index settings:** + - Shards: 3 - Replicas: 1 - Refresh interval: 30s You can customize these with environment variables: -- `OPENSEARCH_INDEX_CONFIG` - Custom JSON for index settings/mappings -- `OPENSEARCH_INDEX_ROLLOVER` - Custom JSON for rollover conditions -- `SHUFFLE_SKIP_OPENSEARCH_INDEX_INIT=true` - Skip automatic index initialization + +- `OPENSEARCH_INDEX_CONFIG` - Custom JSON for index settings/mappings. When set, Shuffle does not manage mappings or rollover for you (you own the schema). No default (unset = Shuffle-managed schema). +- `OPENSEARCH_INDEX_ROLLOVER` - Custom JSON for rollover conditions. Default (equivalent to leaving unset): + ```json + {"max_age": "90d", "max_size": "40gb", "max_docs": 1000000} + ``` +- `OPENSEARCH_USE_ISM_ROLLOVER` - Set to `false` to use direct shard rollover instead of the ISM (Index State Management) plugin. Default: `true` (ISM-managed). +- `OPENSEARCH_ISM_POLICY_NAME` - Name for the ISM rollover/retention policy. Default: `shuffle-rollover`. +- `OPENSEARCH_INDEX_RETENTION_DAYS` - Optional JSON map of index to retention period (e.g. `{"shuffle_logs":"90d"}`), used to delete old rolled generations via ISM. No default (unset = indexes never expire on their own). +- `OPENSEARCH_NOTIFICATION_RETENTION_DAYS` - `notifications` has no permanent terminal state (read/ignored can toggle back and forth), so instead of rollover it uses a daily cleanup sweep that deletes notifications which are read-or-ignored and older than this many days. Default: `0` (disabled/opt-in). Example: `30`. +- `SHUFFLE_SKIP_OPENSEARCH_INDEX_INIT=true` - Skip automatic index initialization. Default: unset (index management runs). + +#### Automatic Mapping & Field Changes + +Shuffle applies curated field mappings (typed `keyword` ids, `date`/`epoch_second` timestamps, `long` counters, `boolean` flags, and `index:false` for binary blobs like images) when an index is created fresh. Because the behavior of these fields is set at index creation, Shuffle keeps them current automatically: + +- **Single/keyed indexes** are re-indexed into a fresh generation on startup if their live mapping no longer matches (idempotent, no-op when up to date). +- **Append/rollover indexes** have an index mapping template registered so every **future** rolled generation is created with the current mappings. Existing backing indexes are left as-is (they are not re-indexed automatically). + +These are skipped when `OPENSEARCH_INDEX_CONFIG` is set. + +> Note: Mappings are immutable in OpenSearch. Re-indexing is the only way to change a field type on existing data. The automatic migration above handles this for single/keyed stores; for very large append stores, you may still need a manual re-index (see below). + +> Exception: `created`/`edited`/`started_at` on `workflowexecution` are mapped `long` rather than `date`, since these fields are sorted across `workflowexecution_live` and the archive (and across archive generations) - `date` fields store doc values internally as epoch milliseconds regardless of the `epoch_second` format annotation, which would silently corrupt sort order against older generations that still have these fields inferred as `long`. + +### OpenSearch Permissions (Security Plugin / RBAC) + +If your OpenSearch cluster has the security plugin enabled - especially if it's a **shared cluster** also holding other tenants' data (SIEM logs, SRE data, other applications) - the credential Shuffle uses should be scoped down to the minimum required, not given the `all_access` role. + +#### Recommended index permissions + +Grant `indices_all` (or a narrower CRUD set, if you prefer) only on Shuffle's own index patterns: + +``` +_* +``` + +(or `*` if you don't set `SHUFFLE_OPENSEARCH_INDEX_PREFIX`). This is enough for all of Shuffle's normal document read/write/search operations, and for listing/inspecting its own indexes and aliases (Shuffle uses the per-index `_settings` and `_alias` endpoints - not the cluster-level `_cat/indices`/`_cat/aliases` - specifically so index enumeration works with index-scoped permissions only). + +#### Cluster permissions + +Shuffle does need the following cluster-level admin permissions to manage its own rollover/retention and index mapping lifecycle: + +| Permission | Used for | +| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | +| `cluster:monitor/main` | Basic startup connectivity check (`GET /`) | +| `cluster:admin/opendistro/ism/policy/*` | Creating/updating the ISM rollover + retention policy (`shuffle-rollover-*`) | +| `cluster:admin/opendistro/ism/managedindex/*` | Attaching/managing the ISM policy on Shuffle's rollover-managed indexes | +| `indices:admin/index_template/put`, `indices:admin/index_template/get` | Registering the index mapping template so new rollover generations get correct field types | +| `cluster:monitor/task/get` | Polling the reindex task started for one-time legacy index/alias collision migrations to completion, and detecting per-document failures instead of silently losing data | +| `cluster:monitor/tasks/lists` | Checking whether a matching reindex is already running (e.g. started by another backend replica) before starting a duplicate one - best-effort only, gracefully skipped if not granted | + +**Known residual risk:** OpenSearch's security plugin has no per-resource-pattern scoping for ISM policies or index templates - these are all-or-nothing cluster-wide grants. A credential with these permissions can, in principle, create/overwrite an ISM policy or index template belonging to a _different_ tenant on the same cluster, even if its index permissions are correctly scoped to `_*`. This is a structural limitation of OpenSearch's permission model, not something Shuffle can restrict further from the application side. If you run a genuinely shared, multi-tenant cluster, mitigate this by: + +- Alerting on ISM policy or index template writes whose ID/pattern doesn't match Shuffle's own naming, sourced from audit logs of the Shuffle credential. +- Using a dedicated OpenSearch cluster (or dedicated node role/tenant) for Shuffle if this residual risk isn't acceptable for your compliance requirements. + +#### Example role (matches what Shuffle actually needs) + +```json +{ + "cluster_permissions": [ + "cluster:monitor/main", + "cluster:admin/opendistro/ism/policy/*", + "cluster:admin/opendistro/ism/managedindex/*", + "indices:admin/index_template/put", + "indices:admin/index_template/get", + "cluster:monitor/task/get", + "cluster:monitor/tasks/lists" + ], + "index_permissions": [ + { + "index_patterns": ["_*"], + "allowed_actions": ["indices_all"] + } + ] +} +``` + +**Note:** without `cluster:monitor/task/get`, Shuffle can still start a legacy index/alias collision migration, but cannot poll it to completion or detect document-level failures - it will log a permission-denied error and safely retry on the next restart rather than silently assuming success. `cluster:monitor/tasks/lists` is used only to avoid starting a duplicate reindex when another replica already started one; if missing, Shuffle simply skips that check and may occasionally start redundant (harmless, idempotent) duplicate work instead. Neither permission affects normal indexing/search/rollover. + +Bind this role to the OpenSearch user configured via `SHUFFLE_OPENSEARCH_USERNAME` / `SHUFFLE_OPENSEARCH_PASSWORD`. ### Re-indexing & Index Management #### When to Re-index + Re-indexing is rarely needed since Shuffle uses automatic rollover for high-volume indexes. However, you may need to re-index when: -- **`workflowexecution` is slow** - The most common issue. If this index grew large before rollover was configured, queries become slow. Re-index into a fresh index with proper aliasing. +- **`workflowexecution` is slow** - Handled automatically by the hot/cold + lifecycle split above (see workflowexecution_live/archive). Manual + re-indexing of `workflowexecution` should no longer be necessary; if it + still occurs, check that `OPENSEARCH_EXECUTION_ARCHIVE_SWEEP_INTERVAL` is + running (look for "Archived N terminal executions" log lines). - **Changing shard count** - OpenSearch locks primary shard count at creation. If you need more shards for a large dataset, you must re-index. - **Field mapping changes** - OpenSearch doesn't allow changing field types (e.g., `text` to `keyword`). Schema changes require re-indexing. - **Index corruption** - Rare, but if an index gets corrupted, re-index from backups. - **Cluster migration** - Moving from single-node to clustered OpenSearch, or migrating between clusters. #### Re-indexing a Single Index + ```bash # Using OpenSearch API directly curl -X POST "https://localhost:9200/_reindex" -H 'Content-Type: application/json' -d' @@ -1957,6 +2182,7 @@ curl -X POST "https://localhost:9200/_reindex" -H 'Content-Type: application/jso ``` #### Checking Index Health + ```bash # List all Shuffle indexes curl -X GET "https://localhost:9200/_cat/indices/*workflow*?v" @@ -1969,7 +2195,9 @@ curl -X GET "https://localhost:9200/workflowexecution/_alias" ``` #### Manual Rollover + If an index grows too large before automatic rollover triggers: + ```bash curl -X POST "https://localhost:9200/workflowexecution/_rollover" -H 'Content-Type: application/json' -d' { @@ -1980,7 +2208,9 @@ curl -X POST "https://localhost:9200/workflowexecution/_rollover" -H 'Content-Ty ``` #### Deleting Old Index Data + For indexes with rollover, you can delete old backing indexes: + ```bash # List backing indexes curl -X GET "https://localhost:9200/_cat/indices/workflowexecution-*?v" @@ -2006,8 +2236,8 @@ As Shuffle has a lot of individual parts, debugging can be quite tricky. To get | worker-8a666e4f-e544-440e-bf0f-4220e7cc9e25 | | | | ### Disabling Image downloads -There are cases where automatic image downloads may occur, such as when you restart Orborus. You can disable this with `SHUFFLE_AUTO_IMAGE_DOWNLOAD=false` +There are cases where automatic image downloads may occur, such as when you restart Orborus. You can disable this with `SHUFFLE_AUTO_IMAGE_DOWNLOAD=false` ### Execution debugging @@ -2023,9 +2253,9 @@ Execution debugging might be the most notable issue you might explain. This is b As previously stated, a lot can go wrong. Here's the most common issues: -* Networking (firewalls / proxies) -* Badly formed apps. -* Bad environment +- Networking (firewalls / proxies) +- Badly formed apps. +- Bad environment ### General debugging @@ -2079,8 +2309,6 @@ docker logs -f CONTAINER_ID As can be seen in the image above, is shows the exact execution order it takes. It starts by finding the parents, before executing the child process after it's finished. Take note of the specific apps being executed as well. It says "Time to execute \ with app \. This indicates the app THAT WILL be executed. The following lines saying "Container \ is the container created with this app. - - 5. App debugging in itself might be the trickiest. There are a lot of factors like branches, bad workflow building etc that might come into play. This builds on the same concept as the worker, where you pass the container ID it specified. Get the app logs @@ -2114,20 +2342,22 @@ For now, the docker socket is required to run Shuffle. Whether you run with Kube **Usage of the socket**: -* Backend (Not required, but used for app management) -* Orborus (Required, deploying Workers) -* Worker (Required, deploying Apps. Apps DONT have access to the socket.) +- Backend (Not required, but used for app management) +- Orborus (Required, deploying Workers) +- Worker (Required, deploying Apps. Apps DONT have access to the socket.) **API's in use** -* Backend: Create, Make, Export docker images. No direct container management. -* Orborus: Download and Remove images. Make, List and Remove containers. Make, List and Remove services. -* Worker: Download and Remove images. Make, List and Remove containers. Make, List and Remove services. +- Backend: Create, Make, Export docker images. No direct container management. +- Orborus: Download and Remove images. Make, List and Remove containers. Make, List and Remove services. +- Worker: Download and Remove images. Make, List and Remove containers. Make, List and Remove services. ### Docker Socket Proxy + In certain scenarios or environments, you may find the docker socket to not have the right permissions, or running the socket directly on your software to be against internal policies. To solve this problem, we've built support for the [docker socket proxy](https://github.com/Tecnativa/docker-socket-proxy), which will give the containers the same permissions, but without the socket being directly mounted in the same container. Another good reason to use the docker socket proxy is to control the docker permissions required. To use the docker socket proxy, add the following to your docker-compose.yml as a service. This will lauch it together with the rest: + ``` docker-socket-proxy: image: tecnativa/docker-socket-proxy @@ -2157,6 +2387,7 @@ To use the docker socket proxy, add the following to your docker-compose.yml as ``` When done, remove the "/var/run/docker.sock" volume from the backend and orborus services in the docker-compose. To enable the docker rerouting, add this environment variable to both of them + ``` - DOCKER_HOST=tcp://docker-socket-proxy:2375 ``` @@ -2189,7 +2420,6 @@ For this, the scripts have been prepared with the alerting mechanism which will This script will determine whether or not the disc space is more than 75% full. If so, an alert will be sent to your Webhook URL. Replace the script's with your Webhook URL. - ``` #!/bin/sh df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | grep -v overlay | while read output; @@ -2207,7 +2437,6 @@ done This script will determine whether or not the memory utilization is more than 70%. If so, an alert will be sent to your Webhook URL. Replace the script's with your Webhook URL. - ``` #check server health STATUS="$(curl http://172.17.14.102:3001/api/v1/_ah/health)" @@ -2253,50 +2482,40 @@ Our main goal is to provide stable support for docker. But a lot of members from You can do it like this: - - ``` sed -E "s/(.*)=['\"]?([^'\"]*)['\"]?/\1=\2/" .env -i ``` - 2. Edit the existing docker-compose.yml to support podman by: In shuffle-backend, comment back in the volume: /var/run/docker.sock:/var/run/docker.sock -And in environments, Comment back out: # DOCKER_HOST=tcp://docker-socket-proxy:2375 - +And in environments, Comment back out: # DOCKER_HOST=tcp://docker-socket-proxy:2375 So that the shuffle-backend service block ends up looking like this: - - ```yaml - backend: - image: ghcr.io/shuffle/shuffle-backend:latest - container_name: shuffle-backend - hostname: ${BACKEND_HOSTNAME} - # Here for debugging: - ports: - - "${BACKEND_PORT}:5001" - networks: - - shuffle - volumes: - - /var/run/docker.sock:/var/run/docker.sock - - ${SHUFFLE_APP_HOTLOAD_LOCATION}:/shuffle-apps:z - - ${SHUFFLE_FILE_LOCATION}:/shuffle-files:z - env_file: .env - environment: - #- DOCKER_HOST=tcp://docker-socket-proxy:2375 # we commented this out - - SHUFFLE_APP_HOTLOAD_FOLDER=/shuffle-apps - - SHUFFLE_FILE_LOCATION=/shuffle-files +backend: + image: ghcr.io/shuffle/shuffle-backend:latest + container_name: shuffle-backend + hostname: ${BACKEND_HOSTNAME} + # Here for debugging: + ports: + - "${BACKEND_PORT}:5001" + networks: + - shuffle + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - ${SHUFFLE_APP_HOTLOAD_LOCATION}:/shuffle-apps:z + - ${SHUFFLE_FILE_LOCATION}:/shuffle-files:z + env_file: .env + environment: + #- DOCKER_HOST=tcp://docker-socket-proxy:2375 # we commented this out + - SHUFFLE_APP_HOTLOAD_FOLDER=/shuffle-apps + - SHUFFLE_FILE_LOCATION=/shuffle-files ``` - - 3. Run Shuffle with podman by: - - ```bash sudo podman-compose -f docker-compose.yml pull sudo podman-compose -f docker-compose.yml up diff --git a/docs/extensions.md b/docs/extensions.md index 040c4d8..a5ca1de 100644 --- a/docs/extensions.md +++ b/docs/extensions.md @@ -1,59 +1,61 @@ # Shuffle extensions + This is documentation for integrating and sending data from third-party services to Shuffle, along with being a brief overview of our Research & Development (R&D) activities. Not to be confused with [apps](/apps) and [workflows](/workflows) ## Table of contents -* [Introduction](#introduction) -* [Single Signon (OpenID/SAML)](#single-signon-sso) - * [Okta](#okta) - * [Google SSO - SAML](#google-saml-sso) - * [Auth0](#auth0) - * [PingIdentity](#ping-id) - * [Keycloak - OpenID](#keycloak) - * [Azure AD - OpenID](#azure-ad) - * [Other SSO providers](#other) - * [Testing SSO](#sso-testing) - * [SSO Required for Org](#sso-required-for-org) - * [Auto Provisioning](#auto-provisioning) - * [Role Base Restriction](#role-base-restriction) - * [Skip SSO for admin in suborgs](#skip-sso-for-admin-in-suborgs) -* [Singul](#singul) -* [AI Agents](#ai-agents) -* [CACAO](#cacao) -* [Detection Manager](#detection-manager) -* [KMS](#KMS) -* [Webhooks](#webhooks) - * [Wazuh Webhook](#wazuh) - * [TheHive Webhook](#thehive) - * [Logz.io Webhook](#logzio) - * [MISP forwarder](#misp) - * [AWS S3 forwarder](#aws-s3-forwarder) - * [QRadar Webhook](#qradar) - * [FortiSIEM Webhook](#fortisiem) - * [ELK Webhook](#elk) - * [Cortex Webhook](#cortex) - * [Splunk Webhook](#splunk-siem) - * [Eventlog Analyzer - SIEM](#eventlog-analyzer) - * [ServicePilot - SIEM](#servicepilot-siem) -## Introduction -From the start, Shuffle has been a platform about integrations. We've focused on making them as open and usable as possible, but were missing one part; inbound data. The general way Shuffle handles this has been through third-party API's, where we poll for data on a schedule. There are however some cases where this doesn't do the trick. That's what extensions are. +- [Introduction](#introduction) +- [Single Signon (OpenID/SAML)](#single-signon-sso) + - [OpenID Connect requirements](#openid-connect-requirements) + - [Okta](#okta) + - [Google SSO - SAML](#google-saml-sso) + - [Auth0](#auth0) + - [PingIdentity](#ping-id) + - [Keycloak - OpenID](#keycloak) + - [Azure AD - OpenID](#azure-ad) + - [Other SSO providers](#other) + - [Testing SSO](#sso-testing) + - [SSO Required for Org](#sso-required-for-org) + - [Auto Provisioning](#auto-provisioning) + - [Role Base Restriction](#role-base-restriction) + - [Skip SSO for admin in suborgs](#skip-sso-for-admin-in-suborgs) +- [Singul](#singul) +- [AI Agents](#ai-agents) +- [CACAO](#cacao) +- [Detection Manager](#detection-manager) +- [KMS](#KMS) +- [Webhooks](#webhooks) + - [Wazuh Webhook](#wazuh) + - [TheHive Webhook](#thehive) + - [Logz.io Webhook](#logzio) + - [MISP forwarder](#misp) + - [AWS S3 forwarder](#aws-s3-forwarder) + - [QRadar Webhook](#qradar) + - [FortiSIEM Webhook](#fortisiem) + - [ELK Webhook](#elk) + - [Cortex Webhook](#cortex) + - [Splunk Webhook](#splunk-siem) + - [Eventlog Analyzer - SIEM](#eventlog-analyzer) + - [ServicePilot - SIEM](#servicepilot-siem) -These integrations will typically entail third party services connecting to Shuffle with inbound Webhooks as triggers in a workflow. +## Introduction -## Single Signon -Shuffle added Single Signon (SAML v1.0) from Shuffle version 0.9.16 & OpenID in Shuffle version 1.0.0. This allows you to log into Shuffle from other identity platforms, entirely controlled by you. SSO is available for **onprem AND cloud**. It works by setting an Entrypoint (IdP) and X509 Certificate, both used to validate who the requester is. This can be added in [your admin panel](/admin). +From the start, Shuffle has been a platform about integrations. We've focused on making them as open and usable as possible, but were missing one part; inbound data. The general way Shuffle handles this has been through third-party API's, where we poll for data on a schedule. There are however some cases where this doesn't do the trick. That's what extensions are. -Note: This feature will be depreciated on cloud by 10th December 2025. Please migrate to OAuth2/OpenID Connect as soon as possible. +These integrations will typically entail third party services connecting to Shuffle with inbound Webhooks as triggers in a workflow. ### Org Swapping behavior + If an Organization requires SSO, it will FORCE you through the SSO login unless your session already has been through SSO for that organization. This may feel counter-intuitive at first, but is a required system as each organization is controlled for SSO individually, and there is no limit to how many organizations a user can have. -### Using ANY SSO platform -**ONPREM ONLY:** You will have to change the SSO_REDIRECT_URL variable in the .env file to match your front end server link i.e `SSO_REDIRECT_URL=http://:` +### Using ANY SSO platform + +**ONPREM ONLY:** You will have to change the SSO_REDIRECT_URL variable in the .env file to match your front end server link i.e `SSO_REDIRECT_URL=http://:` ![Single Signon button](https://github.com/frikky/shuffle-docs/blob/master/assets/sso-3.png?raw=true) **How it works:** + - Every user gets the role "user" by default. - If the user DOESN'T exist, it will be created, based on an identifier provided from the SSO provider. - If the user DOES exist, it will simply log them in. @@ -62,32 +64,67 @@ If an Organization requires SSO, it will FORCE you through the SSO login unless **PS**: In some cases, the persons' username may be appear an ID in Shuffle. If so, an admin should login to Shuffle and change their username. SSO should still work. **PPS**: The callback URL/redirect URL/recipient URL is `https://:/api/v1/login_sso` +### OpenID Connect requirements + +Shuffle supports OpenID Connect (OIDC) login with the **Authorization Code Flow with PKCE**. The OIDC server must support: + +- `response_type=code` +- `code_challenge_method=S256` +- The `openid` and `email` scopes +- A signed `id_token` returned from the token endpoint +- Provider discovery and JWKS for token signature verification +- A stable issuer URL derived from the configured authorization or token endpoint + +Shuffle generates a high-entropy PKCE verifier, sends the SHA-256 based `S256` code challenge in the authorization URL, stores the verifier server-side, and sends the original verifier to the token endpoint during callback handling. Shuffle does **not** support implicit flow, hybrid flow, `response_type=id_token`, or `response_mode=form_post` for OIDC login. + +Configure the OIDC client with this redirect URI: + +- Cloud: `https://shuffler.io/api/v1/login_openid` +- On-prem: `https:///api/v1/login_openid` + +The client may be public or confidential. If no client secret is configured, Shuffle performs a public-client authorization code exchange protected by PKCE and does not send a `client_secret` parameter. If the OIDC server requires confidential-client authentication, configure the client secret in Shuffle; Shuffle uses it only server-to-server at the token endpoint and does not send it in the browser authorization URL. The ID token must contain a `sub` claim and should contain an `email` claim. Shuffle can read roles from `roles`, `groups`, or Keycloak-style `realm_access.roles`; supported Shuffle role names are `shuffle-admin`, `shuffle-user`, and `shuffle-org-reader`. + +Supported OIDC servers include any standards-compliant provider with Authorization Code Flow and PKCE S256 support. Common examples: + +| Type | Examples | +| ----------- | --------------------------------------------------------------------------------------------------------------- | +| Self-hosted | Keycloak, Authentik, Dex, Zitadel, Ory Hydra, Authelia | +| Cloud | Okta, Microsoft Entra ID / Azure AD, Auth0, Google Workspace / Cloud Identity, PingOne, OneLogin, Zitadel Cloud | + +For multi-replica Shuffle backend deployments, configure `SHUFFLE_MEMCACHED` for every backend replica. OIDC login state is created before the browser redirect and consumed when the IdP calls back; without shared Memcached, the callback may reach a different backend replica and fail with invalid OpenID state. Single-replica deployments do not require Memcached for OIDC, and sticky sessions can avoid the symptom, but shared Memcached is the recommended configuration for multi-replica backends. + ![Single signon configuration](https://github.com/frikky/shuffle-docs/blob/master/assets/sso-1.png?raw=true) ### Okta -To use Okta SSO with Shuffle, first make an app in Okta. Their guide for making an app [can be found here](https://developer.okta.com/docs/guides/build-sso-integration/saml2/create-your-app/). + +To use Okta SSO with Shuffle, first make an app in Okta. Their guide for making an app [can be found here](https://developer.okta.com/docs/guides/build-sso-integration/saml2/create-your-app/). Once an application is made, it's time to find the required information. Go to the app > Sign On > Click "View Setup Instructions" under SAML 2.0. This will open a new page with the credentials. ![Okta SSO setup](https://github.com/frikky/shuffle-docs/blob/master/assets/sso-2.png?raw=true) **Move these fields over to Shuffle:** -* Identity Provider Single Sign-On URL -> SSO Entrypoint (IdP) -* X.509 -> SSO Certificate (X509) + +- Identity Provider Single Sign-On URL -> SSO Entrypoint (IdP) +- X.509 -> SSO Certificate (X509) After adding them, click "Save", saving the configuration. After saving, log out of your user to verify the SSO configuration. If you don't see a button for "Use SSO", you most likely configured the wrong organization. ### Okta OpenID + To use Okta OpenID with Shuffle, first make an app in Okta. **Configure the following in Okta:** -* Sign-in redirect URIs -> `https://shuffler.io/api/v1/login_openid` -* Initiate login URI -> `https://shuffler.io/api/v1/login_openid` -* Under Client Credentials, enable "Require PKCE as additional verification" -* Under Grant type, enable "Implicit (hybrid)" and "Allow ID Token with implicit grant type" + +- Sign-in redirect URIs -> `https://shuffler.io/api/v1/login_openid` +- Initiate login URI -> `https://shuffler.io/api/v1/login_openid` +- Under Client Credentials, enable "Require PKCE as additional verification" with `S256` +- Under Grant type, enable "Authorization Code" +- Do not enable implicit or hybrid ID Token grant types for Shuffle OIDC login After adding them, click "Save", saving the configuration. After saving, log out of your user to verify the SSO configuration. ### Google SAML SSO + • Log in to your Google Workspace portal as admin -> Apps -> Web and mobile apps. • Click Add app -> Add custom SAML app. @@ -98,76 +135,82 @@ After adding them, click "Save", saving the configuration. After saving, log out • Next page will be the SP information, this is where you should provide the Single Sign On URL, and SP Entity ID. - - ACS URL : - - For Cloud : https://shuffler.io/api/v1/login_sso - - For On-prem: https://:3443/api/v1/login_sso - - Entity ID : shuffle-saml - - Name ID: Basic Information > Primary Email - ![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/7958a55b-c932-4b5c-8910-dfb42703b695) + - ACS URL : + +- For Cloud : https://shuffler.io/api/v1/login_sso +- For On-prem: https://:3443/api/v1/login_sso +- Entity ID : shuffle-saml +- Name ID: Basic Information > Primary Email + ![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/7958a55b-c932-4b5c-8910-dfb42703b695) - • The next page is where you add the below attributes and click on Finish and the app is added to the Google Workspace. - ![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/2bd0c217-de1c-41a4-82b9-7672ed2a4984) +• The next page is where you add the below attributes and click on Finish and the app is added to the Google Workspace. +![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/2bd0c217-de1c-41a4-82b9-7672ed2a4984) - • After completing the app creation on Google Workspace. You can select the shuffle app under the "Web and mobile apps". Click on the shuffle app, and note - the SPID in the URL, that is the SPID that will be needed. +• After completing the app creation on Google Workspace. You can select the shuffle app under the "Web and mobile apps". Click on the shuffle app, and note +the SPID in the URL, that is the SPID that will be needed. - ![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/cfd1fc98-8efe-4aa2-a4af-4ea60d10a3d6) +![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/cfd1fc98-8efe-4aa2-a4af-4ea60d10a3d6) - • Now we configure the Google SAML SSO in shuffle portal. Log in to your Shuffle web portal. - - • Click on profile -> Admin --> Edit Details --> Now add the Entrypoint (IdP) and SSO Certificate under the SAML SSO and click on save. +• Now we configure the Google SAML SSO in shuffle portal. Log in to your Shuffle web portal. + +• Click on profile -> Admin --> Edit Details --> Now add the Entrypoint (IdP) and SSO Certificate under the SAML SSO and click on save. + +**Note : The IdP metadata contains the Idp ID and certificate. Use this link as your SSO entry point, substituting your SP ID for "YourSPID" and your IDP ID for "YourIDPId."** - **Note : The IdP metadata contains the Idp ID and certificate. Use this link as your SSO entry point, substituting your SP ID for "YourSPID" and your IDP ID for "YourIDPId."** - - SSO Entrypoint (IdP) : https://accounts.google.com/o/saml2/initsso?idpid=C0ivw&spid=634181550209&forceauthn=false -### Auth0 +### Auth0 + To use Auth0 SSO with Shuffle, first make an app on [https://manage.auth0.com](https://manage.auth0.com). Documentation can be found [here](https://auth0.com/docs/configure/saml-configuration/configure-auth0-saml-identity-provider). -After the app is made, click "Addons" > "SAML2 Web App". +After the app is made, click "Addons" > "SAML2 Web App". ![Auth0 SSO setup](https://github.com/frikky/shuffle-docs/blob/master/assets/sso-4.png?raw=true) **In the popup, move the data of these fields to Shuffle:** -* Identity Provider Login URL -> SSO Entrypoint (IdP) -* Identity Provider Certificate -> SSO Certificate (X509) + +- Identity Provider Login URL -> SSO Entrypoint (IdP) +- Identity Provider Certificate -> SSO Certificate (X509) Open the Certificate file in a text editor, and copy it's contents. After adding them, click "Save", saving the configuration. After saving, log out of your user to verify the SSO configuration. If you don't see a button for "Use SSO", you most likely configured the wrong organization. -### PingIdentity +### PingIdentity + To use PingID SSO with Shuffle, first make an app on [https://console.pingone.eu/](https://docs.pingidentity.com/bundle/pingoneforenterprise/page/xsh1564020480660-1.html). Documentation can be found [here](https://docs.pingidentity.com/bundle/pingoneforenterprise/page/xsh1564020480660-1.html). After the app is made, click the dropdown for it on the right side > Configuration > find these fields. ![PingID SSO setup Shuffle](https://github.com/frikky/shuffle-docs/blob/master/assets/sso-5.png?raw=true) **In the view above, move the data of these fields to Shuffle:** -* INITIATE SINGLE SIGN-ON URL -> SSO Entrypoint (IdP) -* DOWNLOAD METADATA -> SSO Certificate (X509) + +- INITIATE SINGLE SIGN-ON URL -> SSO Entrypoint (IdP) +- DOWNLOAD METADATA -> SSO Certificate (X509) Open the Certificate file in a text editor, and copy it's contents in the field. After adding them, click "Save", saving the configuration. After saving, log out of your user to verify the SSO configuration. If you don't see a button for "Use SSO", you most likely configured the wrong organization. ### Keycloak -Open ID SSO setup with Keycloak: -- Click on clients -- Create new client -- Set client id -- Ensure client protocol is openid-connect -- Hit save - -You now have a client. Click on settings and configure them as follows -- Your client ID -- Set up a name -- Client Protocol set to openid-connect -- Access Type set to public -- Standard Flow Enabled toggled to ON -- Direct access grants toggled to ON + +OpenID Connect SSO setup with Keycloak: + +- Click **Clients** +- Create a new client +- Set **Client type** / **Client protocol** to `openid-connect` +- Set the client ID you want Shuffle to use +- Enable **Standard flow** +- Disable **Implicit flow**. Shuffle does not support implicit or `form_post` ID token login. +- Disable **Direct access grants** unless you need them for another application. +- Enable **Proof Key for Code Exchange (PKCE)** with `S256` when your Keycloak version exposes this option. +- If you use a confidential Keycloak client, enable client authentication and copy the client secret into Shuffle. Shuffle only sends the secret server-to-server during the token exchange. ![Valid redirect URI](https://user-images.githubusercontent.com/31187099/162692070-ccc54692-6793-4331-adc7-2356d0fd5397.jpg) -For the Valid Redirect URI https://shuffler.io/api/v1/login_openid +For the valid redirect URI, use your Shuffle callback URL: + +- Cloud: `https://shuffler.io/api/v1/login_openid` +- On-prem: `https:///api/v1/login_openid` - Backchannel Logout URL https://shuffler.io/* - Backchannel Logout Session toggled to ON @@ -179,16 +222,21 @@ Valid Request URIs https://shuffler.io/login?autologin=true ![Auto login = true](https://user-images.githubusercontent.com/31187099/162692186-6ac71b93-01be-4cb1-83f4-b7cb17758378.jpg?raw=true) Once this is done head over to your shuffle instance. + 1. Click on Admin button 2. Scroll down and click on the downward facing arrow beneath Organization overview 3. Scroll down again to OpenID connect 4. Fill in the client ID (It should be the same as what you entered in Keycloak) -5. Authorization URL http://:/auth/realms/openid/protocol/openid-connect/auth -6. Token URL http://:/auth/realms/openid/protocol/openid-connect/token +5. If the Keycloak client is confidential, fill in the client secret. +6. Authorization URL `https:///realms//protocol/openid-connect/auth` +7. Token URL `https:///realms//protocol/openid-connect/token` + +Older Keycloak deployments may include `/auth` before `/realms/...`; use the URLs shown in your realm's OpenID endpoint configuration. ![OIDC shuffle-side](https://user-images.githubusercontent.com/31187099/162689392-51fcb2e9-3d89-4066-8b99-6074065f9c2a.png?raw=true) If you keep getting redirected to your backend url, head on to your Shuffle folder on your server. + 1. Vim .env ![vim env](https://user-images.githubusercontent.com/31187099/164943744-981638cf-6149-42e1-a455-3d12927ec24c.png) @@ -199,67 +247,108 @@ If you keep getting redirected to your backend url, head on to your Shuffle fold Finally go back to shuffle and use SSO button to login. +If you run multiple Shuffle backend replicas, configure `SHUFFLE_MEMCACHED` so the OIDC login state created before the redirect is available to the backend replica that receives the callback. Without shared cache, use sticky sessions; otherwise callbacks may fail with invalid OpenID state. + ![shuffle SSO](https://user-images.githubusercontent.com/31187099/162689445-8db0766c-6f18-4463-8a92-f6ae62213918.png?raw=true) ### Assigning a Role from Keycloak to Shuffle for a New User -If you want to assign a Shuffle organization role (`admin`, `user`, `org-reader`) from your Keycloak client, you can achieve this using the following method: +If you want to assign a Shuffle organization role (`admin`, `user`, `org-reader`) from your Keycloak client, you can achieve this using **client roles** and a **dedicated protocol mapper** that includes the roles in the ID token. -Steps to Assign Roles +> **Important:** Shuffle reads roles exclusively from the **ID token**, not the access token. Without a mapper configured to add roles to the ID token, Shuffle will not see any role assignments. -1. In your Keycloak client, create three new roles with the names `shuffle-admin`, `shuffle-user`, and `shuffle-org-reader`, as shown in the image below: +#### Step 1: Create Client Roles - ![image](https://github.com/user-attachments/assets/e8a1f344-73c9-453d-b119-aef4643610b4) +In Keycloak, navigate to your Shuffle client and create three roles: -2. After creating the roles, assign them to the users you want. The `shuffle-admin`, `shuffle-user`, and `shuffle-org-reader` roles in Keycloak correspond to the `admin`, `user`, and `org-reader` roles in Shuffle. +**Clients** → *your-shuffle-client* → **Roles** → **Create role** -3. Once the roles are assigned to users, navigate to: +Create the following roles (names must match exactly): - **Client Scopes** → **Roles** → **Mapper** → **Client Roles** +| Keycloak Client Role | Shuffle Role | +|---------------------|--------------| +| `shuffle-admin` | admin | +| `shuffle-user` | user | +| `shuffle-org-reader` | org-reader | - On the **Client Roles** page: +![image](https://github.com/user-attachments/assets/e8a1f344-73c9-453d-b119-aef4643610b4) - - Update the `Token Claim Name` to `roles` - - Enable the option **"Add to ID Token"** so that the roles are included in the response. +#### Step 2: Assign Roles to Users - ![image](https://github.com/user-attachments/assets/09582542-efa1-429b-9d45-e0a3796c6fbd) +Assign one of the client roles to each user who should access Shuffle: - Important Notes: +**Users** → *select user* → **Role mapping** → **Assign role** → Filter by *clients* → select the appropriate `shuffle-*` role -- After completing these settings, if valide role is present in roles claim during sso login then that user will be assign role from sso provider. If no role is assigned, the default role will be `user`. -- You can achieve this behavior starting from Shuffle version 2.0.1 or later. Make sure you are using this version or a newer one +#### Step 3: Add a Protocol Mapper to the Dedicated Client Scope - +Each Keycloak client has a dedicated scope (named `-dedicated`) that is automatically assigned to the client. Add a mapper here to include client roles in the ID token: + +1. Navigate to: **Client Scopes** → **`-dedicated`** → **Mappers** → **Configure a new mapper** (or **Add mapper** → **By configuration**) +2. Select mapper type: **User Client Role** +3. Configure the mapper: + - **Name**: `shuffle-roles` (or any descriptive name) + - **Client ID**: Select your Shuffle client + - **Token Claim Name**: `roles` + - **Claim JSON Type**: `String` + - **Add to ID token**: ✅ **ON** (required) + - **Add to access token**: Optional (not used by Shuffle) + - **Add to userinfo**: Optional (not used by Shuffle) + - **Multivalued**: ✅ **ON** +4. Click **Save** + +![image](https://github.com/user-attachments/assets/09582542-efa1-429b-9d45-e0a3796c6fbd) + +#### Verifying the Configuration + +You can verify that roles appear in the ID token by using Keycloak's built-in token evaluation: + +**Clients** → *your-shuffle-client* → **Client Scopes** → **Evaluate** → select a user → **Generated ID token** + +The token should contain a `roles` claim with the assigned client role, for example: + +```json +{ + "roles": ["shuffle-admin"] +} +``` + +#### Important Notes + +- Shuffle reads roles from three possible claims in the ID token: `roles`, `groups`, and `realm_access.roles`. The recommended approach above uses the `roles` claim via client roles. +- Shuffle updates the user's organization role on every OpenID login based on the verified ID token. If a user previously had `shuffle-admin` but the provider later sends `shuffle-user`, `shuffle-org-reader`, or no Shuffle role, Shuffle revokes the old admin role and applies the current provider role. If no Shuffle role is assigned and role-required mode is off, the default role is `user`. +- If **"Restrict login to users with a valid SSO role"** is enabled in Shuffle's SSO settings, users without one of the three `shuffle-*` roles in their ID token will be denied login entirely. +- You can achieve this behavior starting from Shuffle version 2.0.1 or later. Make sure you are using this version or a newer one ### Azure AD -To use OpenID with Azure AD, Shuffle supports OpenID connect with the use of Client IDs and Client secrets. To set up OpenID Connect with Azure, we use "ID_token" authentication. This entails a few normal steps regarding app creation in Azure App Registration. -1. Set up an app in Azure AD with ID tokens enabled -Go to [app registrations and create a new app](https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade). Make it use "Web" for redirect URI's and direct it to your Shuffle instance at /api/v1/login_openid. From here, make sure to go to "authentication" and enable "ID Tokens" -![image](https://user-images.githubusercontent.com/5719530/169712651-24f481bc-7d90-4b09-b25f-cd1ec6c52c20.png) +To use OpenID with Azure AD, Shuffle supports OpenID Connect with authorization code flow, PKCE `S256`, Client IDs and Client secrets. This entails a few normal steps regarding app creation in Azure App Registration. + +1. Set up an app in Azure AD with authorization code flow + Go to [app registrations and create a new app](https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade). Make it use "Web" for redirect URI's and direct it to your Shuffle instance at `/api/v1/login_openid`. Do not enable implicit grant ID tokens for Shuffle OIDC login. + ![image](https://user-images.githubusercontent.com/5719530/169712651-24f481bc-7d90-4b09-b25f-cd1ec6c52c20.png) 2. Get the Client ID, Client Secret and your Tenant -Start by generating a client secret. Keep it safe, as we'll use it later. -![image](https://user-images.githubusercontent.com/5719530/169712756-a07ae08f-75a6-4174-8c8a-26700fd48654.png) + Start by generating a client secret. Keep it safe, as we'll use it later. + ![image](https://user-images.githubusercontent.com/5719530/169712756-a07ae08f-75a6-4174-8c8a-26700fd48654.png) The client secret and tenant ID can be found in the "Overview" tab: ![image](https://user-images.githubusercontent.com/5719530/169712793-7173a2c5-3450-4a79-8f95-9363b940c293.png) 3. Go to the admin panel in Shuffle and put in the Client ID and Secret -![image](https://user-images.githubusercontent.com/5719530/169712842-83db0618-d1f2-4bcd-ad78-5276f0c0696a.png) + ![image](https://user-images.githubusercontent.com/5719530/169712842-83db0618-d1f2-4bcd-ad78-5276f0c0696a.png) 4. Put in your Tenant ID in the authorization URL -The URL is as such: `https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/authorize`. The Token URL is not strictly required for ID Token auth. -![image](https://user-images.githubusercontent.com/5719530/169712857-fe49855e-da3a-4dca-9fde-59bdc60eaa4d.png) + The Authorization URL is `https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/authorize`. The Token URL is required for authorization code exchange: `https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/token`. + ![image](https://user-images.githubusercontent.com/5719530/169712857-fe49855e-da3a-4dca-9fde-59bdc60eaa4d.png) 5. In the "Token Configuration" tab on the left, click "Add Optional Claim." Select the token type as "ID" and choose "email" as the claim, then click "Add." -![token_configuration](https://github.com/user-attachments/assets/2f3b62b9-23a6-4404-a6c6-2dfd796a0b2d) - -7. Done! Click save and log out. Try your new login based on your Azure AD configuration. - + ![token_configuration](https://github.com/user-attachments/assets/2f3b62b9-23a6-4404-a6c6-2dfd796a0b2d) +6. Done! Click save and log out. Try your new login based on your Azure AD configuration. + PS: When a user signs in, they are granted the access rights of a "user" within the designated organization. The username will be derived from the email address listed in the Azure users list. Therefore, when creating or adding a new user in Azure, ensure that the email field in their profile is populated. If this email field is empty, the user will not be able to log in to the organization. - + ### Other Platforms + As long as you can create an identity and acquire an Entrypoint (IdP) and X509, paste them into the Shuffle fields, and it should work with any SAML/SSO provider. ### Testing SSO @@ -276,111 +365,123 @@ https://github.com/user-attachments/assets/0a927283-d39e-4200-8ba3-654ef6f1b9c1 **Note**: Ensure that the "email" field is included in the SSO response from your SSO provider. If this field is empty, you may encounter errors. The email from your SSO provider will be assigned as the username in Shuffle. +### How to Assign a Role to a New User from an SSO Provider (OpenID Connect) in Shuffle +If you want to assign a role to a user from an SSO provider, you can do so using the following method: -### How to Assign a Role to a New User from an SSO Provider (OpenID Connect) in Shuffle - -If you want to assign a role to a user from an SSO provider, you can do so using the following method: - -1. Create three roles in your SSO application/client: - - `shuffle-admin` - - `shuffle-user` - - `shuffle-org-reader` +1. Create three roles in your SSO application/client: + - `shuffle-admin` + - `shuffle-user` + - `shuffle-org-reader` -2. Assign one of these roles to new users when granting them access to the application/client. - - The `shuffle-admin`, `shuffle-user`, and `shuffle-org-reader` roles in your SSO provider correspond to the `admin`, `user`, and `org-reader` roles in Shuffle. +2. Assign one of these roles to new users when granting them access to the application/client. + - The `shuffle-admin`, `shuffle-user`, and `shuffle-org-reader` roles in your SSO provider correspond to the `admin`, `user`, and `org-reader` roles in Shuffle. -3. Ensure that the `roles` claim is included in the **ID token** response from your SSO provider. +3. Ensure that the `roles` claim is included in the **ID token** response from your SSO provider. -- If none of the specified roles are assigned, or if the `roles` claim is missing, the user will be assigned the default role: **`user`** in Shuffle. +- If none of the specified roles are assigned, or if the `roles` claim is missing, the user will be assigned the default role: **`user`** in Shuffle. -Important Notes +Important Notes -1. Currently, this feature is only available for **OpenID Connect**. Support for **SAML SSO** may be added in the future. -2. After completing these settings, if valide role is present in roles claim during sso login then that user will be assign role from sso provider. If no role is assigned, the default role will be `user` for new users. +1. Currently, this feature is only available for **OpenID Connect**. Support for **SAML SSO** may be added in the future. +2. After completing these settings, if valide role is present in roles claim during sso login then that user will be assign role from sso provider. If no role is assigned, the default role will be `user` for new users. 3. You can achieve this behavior starting from Shuffle version 2.0.1 or later. Make sure you are using this version or a newer one. If you have any questions or need further assistance, please feel free to reach out to us at **[support@shuffler.io](mailto:support@shuffler.io)**. - ### SSO Required for Org + In Shuffle, user can configure whether Single Sign-On (SSO) is **required** or **optional** for an organization. By default, SSO is optional. However, when the **"Require SSO"** option is enabled from the [SSO tab](https://shuffler.io/admin?admin_tab=sso) in the admin panel, all users within that organization must log in through SSO. If SSO is required for an organization and a user logs in using their username and password (not SSO), then switches to a non-SSO organization, they can access it without SSO. However, if they later switch back to the SSO-required organization and the session has changed or expired, they will be required to authenticate via SSO again. ### Auto Provisioning + In Shuffle, users can configure whether **auto-provisioning** is enabled or disabled for an organization. By default, auto-provisioning is **enabled**, meaning that if SSO is configured for the organization, a new user account will be automatically created using the email address provided by the SSO provider—even if the user does not already exist in the organization. However, when the **"Disable Auto Provisioning"** option is enabled from the [SSO tab](https://shuffler.io/admin?admin_tab=sso) in the admin panel, new user accounts will **not** be created automatically. In this case, only users who already exist in the organization will be allowed to log in through SSO. ### Role-Based Restriction -In Shuffle, you can now [assign roles to users](https://shuffler.io/docs/extensions#how-to-assign-a-role-to-a-new-user-from-an-sso-provider-(openid-connect)-in-shuffle) directly from the SSO provider. This functionality is currently supported **only** for SSO providers using **OpenID Connect**. + +In Shuffle, you can now [assign roles to users]() directly from the SSO provider. This functionality is currently supported **only** for SSO providers using **OpenID Connect**. By default, if no valid role is passed from the SSO provider during login, the user is automatically assigned the **user** role. -If you want to restrict login for users who do not receive a valid role from the SSO provider, you can enable the **"Restrict user login if no valid role is assigned"** option from the [SSO tab](https://shuffler.io/admin?admin_tab=sso) in the admin panel. When this option is enabled, only users with a valid role received from the SSO provider will be allowed to log in. +If you want to restrict login for users who do not receive a valid role from the SSO provider, you can enable the **"Restrict user login if no valid role is assigned"** option from the [SSO tab](https://shuffler.io/admin?admin_tab=sso) in the admin panel. When this option is enabled, only users with a valid role received from the SSO provider will be allowed to log in. ### Skip SSO for Admins in Suborganizations + In Shuffle, you can now allow parent organization admins to **skip SSO login** when switching to suborganizations. By default, all users are required to authenticate via SSO when switching to a suborganization if SSO is required for that suborg and the user's session has changed or expired. However, by enabling the **"Skip SSO for Admin"** option from the [SSO tab](https://shuffler.io/admin?admin_tab=sso) in the admin panel, users with an **admin role in the parent organization** can bypass the SSO login when switching to a suborganization, even if SSO is required there and user session is changed or expired. ## Singul -Singul are a new way Shuffle interacts with data, built brick by brick since introducing Shuffle's Integration Layer API in early 2023. The goal of Singul is to enable ourselves and others to be able to perform actions towards a specific API, without necessarily know how to do it specifically for that system. + +Singul are a new way Shuffle interacts with data, built brick by brick since introducing Shuffle's Integration Layer API in early 2023. The goal of Singul is to enable ourselves and others to be able to perform actions towards a specific API, without necessarily know how to do it specifically for that system. As of early 2024, this system is in active development, and we will implement features with it and help third party platforms do the same throughout the next few years. **Example usecases:** + - Listing assets from your asset management system/CMDB to make a list of assets, without needing to know the Assets' API - Blocking an Endpoint without knowing how to use the EDR API. Add this as a button to the list from the previous usecase - Shuffle Notification Workflow: Get notifications directly to your ticketing system with minimal configuration image -The Singul system is based on [generative AI for automatic mapping of fields (Schemaless)](https://github.com/frikky/schemaless), uses Github to [store configurations (Standards)](https://github.com/shuffle/standards), and uses [Shuffle's Integration Layer API](https://shuffler.io/docs/API#integration-layer) to run the actions. +The Singul system is based on [generative AI for automatic mapping of fields (Schemaless)](https://github.com/frikky/schemaless), uses Github to [store configurations (Standards)](https://github.com/shuffle/standards), and uses [Shuffle's Integration Layer API](https://shuffler.io/docs/API#integration-layer) to run the actions. ## AI Agents + AI Agents are in active development as of April 2025. They take input(s) and automatically do the tasks for you, running in unison with [Singul](#singul) to perform tasks. This will be published to everyone as soon as it: + 1. Has an interface allowing interactivity 2. Works with local models + Shuffle AI inference (for open sourcing) -3. Has good reasoning capabilities to handle decisionmaking +3. Has good reasoning capabilities to handle decisionmaking image ## CACAO -CACAO is a standardization framework for playbooks. It is still lacking significantly in capabilities as compared to Shuffle's own workflow system, but may be a good way to handle interoperability. + +CACAO is a standardization framework for playbooks. It is still lacking significantly in capabilities as compared to Shuffle's own workflow system, but may be a good way to handle interoperability. The goals are as follows: + 1. ✅ Ensure [CACAO playbook imports work](https://github.com/shuffle/cacao) -3. Ensure Shuffle Workflow -> CACAO playbook exports work well +2. Ensure Shuffle Workflow -> CACAO playbook exports work well -Shuffle may in the future be based on CACAO playbooks to make workflow documentation work better. +Shuffle may in the future be based on CACAO playbooks to make workflow documentation work better. ## Detection Manager -The Shuffle Detection Manager is a system introduced in beta in December 2024, allowing Shuffle to work with platforms like Tenzir and other systems to help with Detection Engineering. The goal of the system is not to replace actual detection systems themselves, but to offer a centralized way to control Detection rules across tenants and different tools. As an example, **below is a focus on Sigma rules with Tenzir**. The system is tested with Yara rules, Email detection rules and custom rule systems. + +The Shuffle Detection Manager is a system introduced in beta in December 2024, allowing Shuffle to work with platforms like Tenzir and other systems to help with Detection Engineering. The goal of the system is not to replace actual detection systems themselves, but to offer a centralized way to control Detection rules across tenants and different tools. As an example, **below is a focus on Sigma rules with Tenzir**. The system is tested with Yara rules, Email detection rules and custom rule systems. ### Testing Tenzir + Sigma -1. **Rule Manager:** At least One Shuffle org -2. **Job Handler:** An Orborus instance running + +1. **Rule Manager:** At least One Shuffle org +2. **Job Handler:** An Orborus instance running 3. **Detection Handler:** A Tenzir instance running on the same server as Orborus **(no setup needed)** -4. **Log Forwarder:** Any system that can forward logs to Tenzir +4. **Log Forwarder:** Any system that can forward logs to Tenzir To test the Tenzir detection system, it is first important to ensure that your Orborus instance is attached to Shuffle, which can be found on the /admin?tab=Locations path. Below is a BAD instance, where Orborus both says "Stopped" AND Pipelines is crossed out. The first goal is to re-enable these. A bad instance that is not running ### Fixing the pipeline setup + To solve the pipeline issue shown in the previous image, we have to do two things: -1. Start Orborus and get it to the "Running" state. To do this, click the Location in question and copy the command to one of your servers. After this has been done, you should see a the Red "Stopped" part change to a Green "Running" box as in the image below. If this does not occur, reach out to support@shuffler.io. +1. Start Orborus and get it to the "Running" state. To do this, click the Location in question and copy the command to one of your servers. After this has been done, you should see a the Red "Stopped" part change to a Green "Running" box as in the image below. If this does not occur, reach out to support@shuffler.io. 2. Go to /detections/Sigma in the UI, and click "Connect" in the top-right corner. Refresh the page after a minute or so, and the Pipeline system should be showing as green on the [Location page](https://shuffler.io/admin?tab=Locations) and in the top-right corner of the [Detection page](https://shuffler.io/detections/Sigma). If it does not, please reach out to support@shuffler.io. image Tenzir setup configuration: -- **Adding a custom storage folder for Sigma rules:** Mount in the folder you want to control into the Orborus command. Then add the environment variable `SHUFFLE_STORAGE_FOLDER=/tmp/foldername` to Orborus. The default is `/tmp/`. + +- **Adding a custom storage folder for Sigma rules:** Mount in the folder you want to control into the Orborus command. Then add the environment variable `SHUFFLE_STORAGE_FOLDER=/tmp/foldername` to Orborus. The default is `/tmp/`. - **Connecting to an EXTERNAL Tenzir node:** Add the following environment variables to the Orborus command: `SHUFFLE_PIPELINE_URL=http://:5160`. Change the `` to your values (localhost does NOT work). This requires that [the web API is enabled](https://docs.tenzir.com/rest-api) on the node. ### Connecting to a remote Tenzir -If Tenzir is ran separately, follow these steps to set connect to it from Shuffle. The first step is to enable webserver mode. This is for systemd. + +If Tenzir is ran separately, follow these steps to set connect to it from Shuffle. The first step is to enable webserver mode. This is for systemd. 1. `sudo nano /etc/systemd/system/tenzir-node.service` 2. Find the line that says "ExecStart" and add `"--commands=web server --mode=dev --bind=0.0.0.0"` to the end of it @@ -391,10 +492,10 @@ It should look like this: `ExecStart=/opt/tenzir/bin/tenzir-node "--commands=web 4. `systemctl daemon-reload & systemctl restart tenzir-node` 5. Try to connect to it: `curl -XPOST http://localhost:5160/api/v0/ping` - - **Control the Shuffle Tenzir node from Tenzir Cloud**: Go to [Tenzir Cloud](https://app.tenzir.com) and create a node configuration. Download the configuration file, then add the variables found in it to the following environment variables to Orborus: `TENZIR_PLUGINS__PLATFORM__API_KEY=`, `TENZIR_PLUGINS__PLATFORM__CONTROL_ENDPOINT=`, `TENZIR_PLUGINS__PLATFORM__TENANT_ID=` ### Mounting the Tenzir Sigma location into Orborus + 1. You need to mount in the folder that is going to store the sigma rules, controlled from within Shuffle 2. The default location is /tmp/sigma_rules, so to mount it in, use `--volume "/tmp:/tmp"` in the Dockerfile. 3. If you end up changing the storage location, use the `SHUFFLE_STORAGE_FOLDER` environment variable with the full path. @@ -402,19 +503,21 @@ It should look like this: `ExecStart=/opt/tenzir/bin/tenzir-node "--commands=web image -5. Verify if the files actually exist on the server. By default, you can find the rules in the `/tmp/sigma_rules` folder (ls /tmp/sigma_rules). If the folder exists, then it worked. +5. Verify if the files actually exist on the server. By default, you can find the rules in the `/tmp/sigma_rules` folder (ls /tmp/sigma_rules). If the folder exists, then it worked. image **PS: This folder is deleted if rules are globally disabled in the UI.** ### Running the Tenzir Detection pipeline + With detection rules in the `/tmp/sigma_rules` folder, you can now test the Sigma rules directly. To do this, there are two parts: -1. Ingest some kind of logs. We suggest using [Syslog](https://docs.tenzir.com/integrations/syslog/), but almost data format you want is supported. You can enable this by going to the "Triggers" page in Shuffle: [/admin?tab=triggers](/admin?tab=triggers), then clicking `Deploy New Pipeline`, and running the `Syslog listener` pipeline. + +1. Ingest some kind of logs. We suggest using [Syslog](https://docs.tenzir.com/integrations/syslog/), but almost data format you want is supported. You can enable this by going to the "Triggers" page in Shuffle: [/admin?tab=triggers](/admin?tab=triggers), then clicking `Deploy New Pipeline`, and running the `Syslog listener` pipeline. image -It can take up to 60 seconds to deploy, and should look like this in the UI when ready to receive logs. +It can take up to 60 seconds to deploy, and should look like this in the UI when ready to receive logs. image @@ -425,40 +528,45 @@ It can take up to 60 seconds to deploy, and should look like this in the UI when [Sigma Pipelines details](https://docs.tenzir.com/tql2/operators/sigma) ### Running a sample Detection + To run a sample detection requires that the correct rules are in place in the /tmp/sigma_rules folder. 1. [Ensure that the detection pipeline is running](/admin?tab=triggers): -image + image 2. One of the test rules is called `notepad_test.yaml` and works as follows. This should already exist in your sigma_rules folder. + ``` detection: selection: - EventID: 4688 + EventID: 4688 NewProcessName: '*notepad.exe*' Context: Testing condition: selection ``` 3. This detection requires three events to match: EventID, NewProcessName & Context. To trigger this, you may run the following from any machine with connectivity to the Tenzir machine on port 1514. + ``` printf '<165>1 2025-10-06T12:34:56.789Z myhost.example.com myapp 1234 ID47 [huh eventSource="App" EventID="4688" NewProcessName="notepad.exe"] This is a test log message' | nc IP 1514` ``` -4. Find your detection workflow [here](http://localhost:3002/workflows?tab=background_processes). +4. Find your detection workflow [here](http://localhost:3002/workflows?tab=background_processes). image -PS: If you can't see whether it is triggering in the workflow, run this locally to see if it is being ingested: `tenzir 'export live=true'` +PS: If you can't see whether it is triggering in the workflow, run this locally to see if it is being ingested: `tenzir 'export live=true'` 5. Detections are live! Feel free to add or change them on [/detections/sigma](/detections/sigma). ### Storing Tenzir logs in Opensearch + Since we are already ingesting logs using the `import` mechanism, it means we in theory can just route those logs forward. [Tenzir -> Opensearch documentation](https://docs.tenzir.com/integrations/opensearch) Modify the following: + ``` export live=true | to_opensearch "localhost:9200", action="create", index="shuffle_logs", user="admin", passwd="PASSWORD"` ``` @@ -466,22 +574,26 @@ export live=true | to_opensearch "localhost:9200", action="create", index="shuff Additional [to_opensearch docs here](https://docs.tenzir.com/reference/operators/to_opensearch/). - ### Debugging -Running a pipeline manually. This example dumps database content (`export`) into the `/tmp/events.ndjson` file. + Running a pipeline manually. This example dumps database content (`export`) into the `/tmp/events.ndjson` file. + ``` curl http://localhost:5160/api/v0/pipeline/launch -H "Content-type: application/json" -d '{"definition": "export | write_ndjson | save_file \"/tmp/events.ndjson\""}' ``` Listing all pipelines + ``` curl -XPOST http://localhost:5160/api/v0/pipeline/list -H "Content-Type: application/json" ``` Delete a pipeline + ``` curl -XPOST http://localhost:5160/api/v0/pipeline/delete -H "Content-Type: application/json" -d '{"id":"ID"}' -v ``` #### Manually testing pipelines Tenzir: + Ensures pipelines act as they should, and you get a reasonable output ``` @@ -503,6 +615,7 @@ tenzir 'export | to "/var/lib/tenzir/output.json"' ``` #### Network Tests + In case the exports aren't working and you need to figure out what is going wrong. ``` @@ -523,30 +636,32 @@ tenzir '' ``` ## KMS -Shuffle by default allows you to store authentication tokens within Shuffle itself, which are encrypted in the database. Since February 2024, we additionally support the use of external KMS systems to handle authentication, which is based on [Native Actions](https://shuffler.io/docs/extensions#native-actions) and [Schemaless](https://github.com/frikky/schemaless). Native Actions run in the background to perform the "Get KMS key" action, and the run of the app is NOT stored. + +Shuffle by default allows you to store authentication tokens within Shuffle itself, which are encrypted in the database. Since February 2024, we additionally support the use of external KMS systems to handle authentication, which is based on [Native Actions](https://shuffler.io/docs/extensions#native-actions) and [Schemaless](https://github.com/frikky/schemaless). Native Actions run in the background to perform the "Get KMS key" action, and the run of the app is NOT stored. The Shuffle KMS system is built as a third party Key:Value provider. You can reference keys from the KMS in any field marked as "Authentication" in the UI, or from within a Shuffle authentication itself (meaning you can authenticate the authentication..). The way you reference the keys is path-based, starting with `kms/`. Requirements: -* Have an Authentication called **kms shuffle storage** in Shuffle. On the [Auth page](/admin?tab=app_auth), it should clearly stand out as a different type of authentication. -* The Authentication needs to be associated with an App in the IAM category -* The App needs to have an action labeled as "Get KMS key" -* If it's the FIRST translation, it may fail out without internet access to github.com +- Have an Authentication called **kms shuffle storage** in Shuffle. On the [Auth page](/admin?tab=app_auth), it should clearly stand out as a different type of authentication. +- The Authentication needs to be associated with an App in the IAM category +- The App needs to have an action labeled as "Get KMS key" +- If it's the FIRST translation, it may fail out without internet access to github.com When these requirements are fullfilled, you can do the following to use the KMS system: -* Find the required parameters for the action. The first image below shows the parameters IN ORDER for Hashicorp Cloud Platform Vault. -* Use the following format: `kms/field1/field2/field3/field4/...`. This NEEDS to start with `kms/` -* Make sure the Environment is correct. It uses your default environment to connect to the KMS if not otherwise specified on the [Auth page](/admin?tab=app_auth). -image +- Find the required parameters for the action. The first image below shows the parameters IN ORDER for Hashicorp Cloud Platform Vault. +- Use the following format: `kms/field1/field2/field3/field4/...`. This NEEDS to start with `kms/` +- Make sure the Environment is correct. It uses your default environment to connect to the KMS if not otherwise specified on the [Auth page](/admin?tab=app_auth). + image -* Example referencing the "username" in the app name "Jira": `kms/998067a9-33f2-4c4d-bbb6-4a997d784def/2e9a877f-1a89-4394-a242-f2c6d9dd2420/jira/username` -image +- Example referencing the "username" in the app name "Jira": `kms/998067a9-33f2-4c4d-bbb6-4a997d784def/2e9a877f-1a89-4394-a242-f2c6d9dd2420/jira/username` + image If all of this is fulfilled, you can run the workflow, and Shuffle will automatically reference the KMS correctly. **If it fails to authenticate**, you should see a Notification show up like in the following image. image ### Failure handling and Debugging + During the initial configuration of KMS, there are many things that CAN go wrong. After the setup is initially completed and it works, it should not fail again, meaning spending time to set it up properly the first time is well worth it. If you are using Shuffle onprem and have trouble with KMS translations, we suggest setting the environment variable `SHUFFLE_KMS_DEBUG=true` on the shuffle-backend container. This will allow you to test individual keys in ANY field, meaning you can e.g. print out the KMS value that it should translate to. This further means we will not be deleting the KMS execution, and you can see the execution on the [Workflow Runtime Debugger page.](/workflows/debug). **KMS debug mode is NOT enabled on shuffler.io**. @@ -554,13 +669,15 @@ If you are using Shuffle onprem and have trouble with KMS translations, we sugge image If your KMS translation fails, it is most likely due to network connectivity OR translation errors [for the standard](https://github.com/Shuffle/standards/blob/main/translation_standards/get_kms_key.json) in use. After running a translation, you should see the following three file categories in the admin panel: -- translation standards - The available translation standards -- translation input - The data FROM the kms used for translation. Does NOT contain values, only keys. -- translation output - The translation OUTPUT. Includes JSON paths to the correct keys + +- translation standards - The available translation standards +- translation input - The data FROM the kms used for translation. Does NOT contain values, only keys. +- translation output - The translation OUTPUT. Includes JSON paths to the correct keys IF your translation fails, the first area to look at is the "[translation output](/admin?tab=files&category=translation_output)" file category, as this is where the translation for the schema happens. The hash will match the data you have sent in, which can be found in the "translation input" folder. Documentation about schemaless explains [how translations happen](https://github.com/frikky/schemaless?tab=readme-ov-file#example), and how they can be fixed. If you need further help, contact support@shuffler.io A valid KMS translation file should be in the following format: + ``` { "kms_value": "path.in.kms.json" @@ -570,9 +687,10 @@ A valid KMS translation file should be in the following format: **The "kms_key" field is NOT relevant - the ONLY thing that is important is that the kms_value path is correct.** ### Using any KMS + KMS is supported for any system as long as the sections above are covered. It has been tested for the following: -- Microsoft Azure KMS +- Microsoft Azure KMS - Hashicorp Vault - Google Cloud Key Management - AWS Key Management Service @@ -582,16 +700,19 @@ KMS is supported for any system as long as the sections above are covered. It ha - ... and more! Ask if you need help. ## Inbound Webhooks + This section describes inbound webhooks to Shuffle, and how to set them up in many commonly used third-party systems. If your system support outbound Webhooks, it can also forward to Shuffle as a GET or POST request. [More about webhook triggers](/triggers/#webhooks) ### Wazuh -Wazuh is a SIEM platform for security operations. We've used it through their API multiple ways, but were missing an important component; alerting. That's why we've developed a simple alert forwarder from Wazuh to Shuffle. + +Wazuh is a SIEM platform for security operations. We've used it through their API multiple ways, but were missing an important component; alerting. That's why we've developed a simple alert forwarder from Wazuh to Shuffle. [Wazuh extension documentation](https://documentation.wazuh.com/current/user-manual/manager/manual-integration.html) **PS: If you expect more than 10 alerts per second, you should add multiple workflows AND webhook forwarders to Wazuh** These are the steps to set it up: + 1. Create a Workflow which will receive alerts 2. Add a Webhook to the Workflow 3. Configure Wazuh with the Webhook URL @@ -609,7 +730,7 @@ Copy the URL and keep it for the next steps ![Extend Shuffle with Wazuh 2](https://github.com/frikky/shuffle-docs/blob/master/assets/extensions_example_2.png?raw=true) **3. Configure Wazuh with the Webhook URL** -Start by logging into your Wazuh management console with access to edit the ossec.conf file. We'll first start by adding the Shuffle webhook forwarder. +Start by logging into your Wazuh management console with access to edit the ossec.conf file. We'll first start by adding the Shuffle webhook forwarder. 1. Transfer [these integration files](https://github.com/frikky/Shuffle/tree/master/functions/extensions/wazuh) to the server. Ignore the file named "ossec.conf" for now. 2. Move the custom-\* files to the location /var/ossec/integrations @@ -631,6 +752,7 @@ drwxr-x---. 19 root ossec 242 Dec 14 09:40 .. ``` 3. Change the files' ownership and access rights. This MAY be necessary if Wazuh doesn't have root privileges (it shouldn't) + ``` $ chown root:ossec custom-shuffle $ chown root:ossec custom-shuffle.py @@ -638,9 +760,9 @@ $ chmod 750 custom-shuffle $ chmod 750 custom-shuffle.py ``` -4. Configure ossec.conf to forward to Shuffle +4. Configure ossec.conf to forward to Shuffle -Go to the location /var/ossec/etc/ossec.conf (or where the ossec.conf file is). Take the information below (including ) and paste it in. Find the webhook URL from earlier, copy it and add it to the section. +Go to the location /var/ossec/etc/ossec.conf (or where the ossec.conf file is). Take the information below (including ) and paste it in. Find the webhook URL from earlier, copy it and add it to the section. Find more fields like [levels, groups and rule_id here](https://documentation.wazuh.com/current/user-manual/manager/manual-integration.html). @@ -654,11 +776,13 @@ Find more fields like [levels, groups and rule_id here](https://documentation.wa ``` 5. Restart ossec manager. Every time you change ossec.conf, you need to restart the ossec manager. + ``` systemctl restart wazuh-manager.service ``` You should now start seeing data sent from Wazuh into Shuffle which can be used. If data is NOT sent, make sure of a few things: + - If https in the Shuffle URL: is the certificate of Shuffle signed? Default: no. - Check /var/ossec/logs for logs: grep -R custom-shuffle @@ -668,6 +792,7 @@ There are many ways to test the integration, but you can simplify it by setting ![Extend Shuffle with Wazuh 3](https://github.com/frikky/shuffle-docs/blob/master/assets/extensions_example_3.png?raw=true) #### Running custom commands with Wazuh + Active response can be used with the command "Run Command" in the Wazuh app in Shuffle. This requires an Agent (Agent list) and Command ([Active response](https://documentation.wazuh.com/current/getting-started/use-cases/active-response.html) command). Below, we will show how to set up the custom command "reboot". - Wazuh API logs will be on the manager in: /var/ossec/logs/api.log @@ -677,6 +802,7 @@ Active response can be used with the command "Run Command" in the Wazuh app in S The goal with this section is to set up a bash script that can run custom commands from within Shuffle. 1. Log into the Wazuh agent of choice (Linux), and add the following script to the active-response folder. Give it the name "shuffle.sh". Full path: /var/ossec/active-response/bin/shuffle.sh PS: Once done, make sure it's executable: chmod +x /var/ossec/active-response/bin/shuffle.sh + ``` #!/bin/bash # Extra arguments @@ -685,8 +811,8 @@ The goal with this section is to set up a bash script that can run custom comman if ! command -v jq &> /dev/null then echo "jq could not be found - installing" - sudo apt install jq -y - sudo yum install jq -y + sudo apt install jq -y + sudo yum install jq -y fi @@ -695,19 +821,22 @@ CMD=$(echo $INPUT_JSON | jq -r .parameters.alert.cmd) CALLBACK=$(echo $INPUT_JSON | jq -r .parameters.alert.callback) OUTPUT=$($CMD) curl -XPOST $CALLBACK -d """$OUTPUT""" -k -``` +``` 2. Open /var/ossec/etc/shared/ar.conf and ADD the following line to the bottom of the file: + ``` shuffle - shuffle.sh - 0 ``` 3. Restart the wazuh agent + ``` -/var/ossec/bin/wazuh-control restart +/var/ossec/bin/wazuh-control restart ``` 4. Log into Shuffle and import [this public workflow](https://shuffler.io/workflows/44bc4e93-ba6e-4895-8294-424fd2a1d169). Make sure to change the following: + - Activate the Wazuh app if it's not already - Add a Wazuh username & password to the HTTP node - Add the Wazuh URL to the HTTP node, as well as Get_agents and other Wazuh nodes. @@ -716,11 +845,12 @@ shuffle - shuffle.sh - 0 The default from this workflow is that it will reboot the server. ### TheHive + TheHive is a case management platform for and by security professionals. One of their key capabilities is webhooks, which can send realtime updates to a third party system whenever ANYTHING is changed within TheHive (e.g. a new alert or a case task is written). Shuffle has an ideal way of handling this, [outlined in this blogpost (TheHive4)](https://medium.com/shuffle-automation/indicators-and-webhooks-with-thehive-cortex-and-misp-open-source-soar-part-4-f70cde942e59). **PS: There is a difference between TheHive3 and TheHive 4 on how to set this up. We are referring to TheHive4 in this section.** -1. Create a Workflow in Shuffle +1. Create a Workflow in Shuffle 2. Add a Webhook to the Workflow 3. Configure TheHive with the Webhook URL 4. Test the integration @@ -739,10 +869,12 @@ Copy the URL and keep it for the next steps **3. Configure TheHive with the Webhook URL** Configuring TheHive is the only unique step here, and is different between versions. Full documentation about [TheHive4 webhooks can be found here](https://github.com/TheHive-Project/TheHiveDocs/blob/master/TheHive4/Administration/Webhook.md). -As can be seen in the documentation, there are three steps to setting up TheHive webhooks: +As can be seen in the documentation, there are three steps to setting up TheHive webhooks: + 1. Define the webhook forwarder: Find the [application.conf](https://docs.thehive-project.org/thehive/installation-and-configuration/configuration/manage-configuration/) file and scroll to the webhook section. If it doesn't exist, add the following: + ``` notification.webhook.endpoints = [ { @@ -759,13 +891,15 @@ notification.webhook.endpoints = [ Modify it to have the URL you found making the webhook in Shuffle. You can also use https by adding certificate references to wsConfig{} 2. Restart TheHive + ``` systemctl restart THeHive ``` 3. Activate the webhook -Run this curl command (change the URL, username and password), which activates TheHive forwarding to Shuffle. -**PS: Make sure you have access to the organization you want the Webhook for** + Run this curl command (change the URL, username and password), which activates TheHive forwarding to Shuffle. + **PS: Make sure you have access to the organization you want the Webhook for** + ``` curl -XPUT -u thehive_user:thehive_password -H 'Content-type: application/json' thehive_url/api/config/organisation/notification -d ' { @@ -784,7 +918,8 @@ In TheHive UI (NOT CLI), create a new case, or add a comment to an existing case ![Extend Shuffle with TheHive 3](https://github.com/frikky/shuffle-docs/blob/master/assets/extensions_example_3.png?raw=true) -### Logzio +### Logzio + **1. Create a Workflow which will receive alerts** This one is pretty easily explained. Go to Shuffle an make a new Workflow. @@ -802,11 +937,13 @@ After logging into app.logz.io, hover Settings Icon > Settings > click Notificat ![Extend Shuffle with Logz.io](https://github.com/frikky/shuffle-docs/blob/master/assets/extensions_example_4.png?raw=true) Once in the notification endpoint view, click "Add Endpoint". Configure the following elements: -* Type: Custom -* Name: Shuffle (or some other identifier) -* URL: The Webhook URL from step 2 -* Method: POST -* Data (should be autofilled): + +- Type: Custom +- Name: Shuffle (or some other identifier) +- URL: The Webhook URL from step 2 +- Method: POST +- Data (should be autofilled): + ``` { "alert_title": "{{alert_title}}", @@ -824,6 +961,7 @@ Custom data will be parsed into the field "alert_event_samples" Click "Run the test" at the bottom before saving. This allows for it to send a sample payload to Shuffle. If the data payload is configured as in step 3, the custom data will be available as such: + ``` $exec.alert_event_samples ``` @@ -831,6 +969,7 @@ $exec.alert_event_samples ![Extend Shuffle with Logz.io 3](https://github.com/frikky/shuffle-docs/blob/master/assets/extensions_example_6.png?raw=true) ### MISP + MISP, short for Malware Information Sharing Platform, is one of the best Open Source alternatives for Threat Intelligence. For that reason, a lot of our users have wanted a way to handle data in realtime from MISP. What kind of data? Event updates, indicator updates, IDS flag edits, Organization edit etc. That's why we released an extension for Shuffle which can read ZMQ messages from MISP in realtime and send them to a webhook. @@ -838,14 +977,18 @@ That's why we released an extension for Shuffle which can read ZMQ messages from [More information here](https://www.circl.lu/doc/misp/misp-zmq/) **Steps to set it up:** + 1. Enable ZMQ in MISP by going to Server Settings -> Plugins in MISP. Make sure to enable the options for forwarding Events and Attributes. -2. Install pyzmq and redis on the MISP server: +2. Install pyzmq and redis on the MISP server: + ``` pip3 install pyzmq redis ``` + 3. Go to Shuffle and create a new Webhook 4. [Install Docker](https://docs.docker.com/get-docker/) and [docker-compose](https://docs.docker.com/compose/install/) on a host with access to ZMQ (localhost:50000 by default). This can be on the MISP server, but at scale, it preferably shouldn't be. 5. Copy the docker-compose [found here](https://github.com/frikky/Shuffle/tree/master/functions/extensions/misp/docker-compose.yml) and edit the environment variables to point to your ZMQ instance (PS: Don't use localhost, even if it's on the MISP server) + ``` version: '3' services: @@ -859,19 +1002,24 @@ services: - ZMQ_FORWARD_URL= restart: unless-stopped ``` + 6. Run the docker-compose! + ``` docker-compose up -d ``` + 7. Test it by going to your MISP instance and changing an event or attribute ### AWS S3 forwarder + Ever wanted to run an action as soon as something is uploaded to your S3 Bucket? Maybe you just want to build an S3 honeypot? If so, this part is for you. Herein we describe how you can get information about files being uploaded or changed from an S3 bucket into Shuffle. **Basic overview:** User -> Upload to S3 -> S3 triggers Lambda function -> Lambda function triggers Webhook -> File is downloaded from within Shuffle Steps to set it up: + 1. Create a Workflow with Webhook inside Shuffle 2. Create a Lambda function that triggers on S3 changes 3. Configure the workflow to download the file @@ -888,19 +1036,21 @@ Copy the URL and keep it for the next steps ![Extend Shuffle with webhook2](https://github.com/frikky/shuffle-docs/blob/master/assets/extensions_example_2.png?raw=true) **2. Create the lambda function** -- After logging into your AWS, go to [Lambda functions](https://console.aws.amazon.com/lambda/home) and click "Create function" in the top right corner. + +- After logging into your AWS, go to [Lambda functions](https://console.aws.amazon.com/lambda/home) and click "Create function" in the top right corner. - Use "Author from scratch", and type in a name like "Shuffle-forwarding" and make sure to choose Runtime as Python 3.8. Click "Create function" in the bottom right corner. -![Author S3 from scratch](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-2.png?raw=true) + ![Author S3 from scratch](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-2.png?raw=true) - Click "Add trigger" in the window, left of the function. In the next menu find "S3", before choosing the bucket you want and the "Event type". Click "Add trigger" -**PS: The bucket and cloud function have to be in the same location** -![Add trigger](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-3.png?raw=true) + **PS: The bucket and cloud function have to be in the same location** + ![Add trigger](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-3.png?raw=true) - Under Configuration > Environment variables, click "Edit". Add variable with key "SHUFFLE_WEBHOOK", and the value from step 1. Click "Save". -![Configure s3 environment](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-4.png?raw=true) + ![Configure s3 environment](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-4.png?raw=true) + +- Time to add some code. Go to the "code" tab and paste in the code below. Click "deploy". This should now forward the request to Shuffle. -- Time to add some code. Go to the "code" tab and paste in the code below. Click "deploy". This should now forward the request to Shuffle. ``` import json import urllib.parse @@ -910,15 +1060,15 @@ import os def lambda_handler(event, context): # Get the object from the event and show its content type #bucket = event['Records'][0]['s3']['bucket']['name'] - + webhook = os.environ.get("SHUFFLE_WEBHOOK") if not webhook: return "No webhook environment defined: SHUFFLE_WEBHOOK" - + ret = requests.post(webhook, json=event["Records"][0]) if ret.status_code != 200: return "Bad status code for webhook: %d" % ret.status_code - + print("Status code: %d\nData: %s" % (ret.status_code, ret.text)) ``` @@ -931,7 +1081,6 @@ Now all we need is to do is actually download the file. This requires access rig And that's it! All file updates should now come into Shuffle, including a way to download the file. - **4. Extra: Scan the file with Yara** In a lot of cases you would want to analyze the files somehow. But how..? Shuffle has a built in way to run Yara on files, and get results based on built-in rules (or your own). Most of the time, you would also want a response to that search, hence we've created a way to also delete the file if it matches too many Yara rules. ![Basic s3 analysis](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-7.png?raw=true) @@ -939,13 +1088,16 @@ In a lot of cases you would want to analyze the files somehow. But how..? Shuffl To do this, add in the "Yara" node with the action "Analyze file", and put "$get_s3_file.file_id" in the "File Id" field. This should match the file downloaded from the bucket. After this, add another S3 node, and choose the action "Delete file from bucket" with these parameters: + - Bucket Name: $exec.s3.bucket.name - Bucket Path: {{ "$exec.s3.object.key" | url_decode | replace: "+", " " }} -Last but not least, we need a way [to see IF](/docs/workflows#conditions) the file should be deleted. This can be done by clicking the branch between Yara and the S3 delete action > "New Condition" > with the following data: +Last but not least, we need a way [to see IF](/docs/workflows#conditions) the file should be deleted. This can be done by clicking the branch between Yara and the S3 delete action > "New Condition" > with the following data: + ``` {{ $run_yara.matches | size }} > 3 ``` + ![S3 condition yara](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-8.png?raw=true) Done! Whenever a file is downloaded, it will be analyzed by Yara, checking matches, then removing the file if it's more than 3. @@ -955,7 +1107,9 @@ Done! Whenever a file is downloaded, it will be analyzed by Yara, checking match **QRadar send offense webhook to Shuffle.** #### STEP 1: Add the Script to QRadar & Config + > **Bash Code:** + ``` #!/bin/bash # Version 1.0.0 @@ -976,48 +1130,48 @@ EOF What you should do to use it: -1. Log In to QRadar; -2. Go to Admin > Custom Actions > **Define Actions**; +1. Log In to QRadar; +2. Go to Admin > Custom Actions > **Define Actions**; 3. Click **Add**; 4. Fill the Basic Information (name & description); -5. In the Script Configuration set **bash** as interpreter, and import the bash script attached to this message; -6.** For the Script Parameters, add the parameters in the following order:** +5. In the Script Configuration set **bash** as interpreter, and import the bash script attached to this message; 6.** For the Script Parameters, add the parameters in the following order:** ``` -1. [Fixed Property] **shuffle_url** - As value insert the Shuffle Webhook URI, for example https://shuffle.local/api/v1/hooks/webhook_15acc.... -2. [Fixed Property] **authorization** - Insert the Required headers (**SAuthorization**), for example in Shuffle webhook if you set **SAuthorization=s873hn872n_s298ns2-98ns2ns** in the Required headers you authorization value will be "s873hn872n_s298ns2-98ns2ns". *If you change the token name, don't forget to change it in the bash code too*. +1. [Fixed Property] **shuffle_url** - As value insert the Shuffle Webhook URI, for example https://shuffle.local/api/v1/hooks/webhook_15acc.... +2. [Fixed Property] **authorization** - Insert the Required headers (**SAuthorization**), for example in Shuffle webhook if you set **SAuthorization=s873hn872n_s298ns2-98ns2ns** in the Required headers you authorization value will be "s873hn872n_s298ns2-98ns2ns". *If you change the token name, don't forget to change it in the bash code too*. 3. [Network Event Property] **offense_id** - Insert **offense_id** as value ``` -7. Save +7. Save 8. Deploy Changes ![image](https://user-images.githubusercontent.com/21691729/152444978-d360680a-a0b1-40b0-bd04-fa7395cf4d85.png) ![image](https://user-images.githubusercontent.com/21691729/152445000-0d2a1828-ee1d-41b1-a549-5417c9a48c75.png) - #### STEP 2: Create new offense alert rule - -1. Go to Offenses > **Rules**; + +1. Go to Offenses > **Rules**; 2. Click Actions > **New Event Rule**: + ``` Rule Description Apply Shuffle new offense alert on events which are detected by the Local system and when the event QID is one of the following (28250369) Offense Created - + Rule Responses Execute Custom Action QRadar to Shuffle (previous added script) This Rule will be: Enabled ``` -#### QRadar Overview +#### QRadar Overview + - Easy, this way each time a new offense dispatches in QRadar it will send a webhook to Shuffle containing the offense_id, then the webhook node will receive this info and pass it to the next node (QRadar App) that will perform a **get offense** ~~data~~ action using the received offense id as key. - + This way you won't need to execute an api call every x time and save the last offense id. This is a better solution and improves the SLA, coz if you set 1 minute as you x time, you may have 1 minute delay or less, besides you are getting all "ungotten" offenses at once, when you can use shuffle to handle with multiple at the same time if it happen to dispatch more than one offense in QRadar at the same time or with just some seconds of difference. - ### FortiSIEM + FortiSiEM is the SIEM of Fortigate. It has the possibility of notifying Shuffle through a webhook when a rule triggers, which is exatly what this documentation section is for. The main caveat: all data is XML and needs to be transformed with the Shuffle Tools "XML to JSON" formatter. **1. Create a Workflow which will receive alerts** @@ -1033,19 +1187,15 @@ Copy the URL and keep it for the next steps **3. Configure FortiSIEM forwarding** Log into the UI. When inside, go to ADMIN > Settings > Incident Notification. In the "Incident HTTP Notification", paste in the webhook from the previous step. Click "save", then "Test". After the test, check the workflow in Shuffle whether it triggered. If it didn't, your FortiSIEM can't access the Shuffle instance. - ![image](https://user-images.githubusercontent.com/5719530/162085936-58a44de4-0289-4cf9-8f72-7b3229857448.png) - With the Notification Endpoint specified in the previous step, we need to decide what rules to add. By default, we add all of them. To do this, go to ADMIN > Settings > Notification Policy, and add a new policy. In here, select the "Send XML file over HTTP(S) to the destination set in...". This will make sure all alerts are sent to Shuffle. - ![image](https://user-images.githubusercontent.com/5719530/162086144-8bbfb6fa-d512-4c36-81db-f830bcf9c204.png) That's it! It's now time to wait for an alert to actually trigger. When it has, make sure to send Execution Argument from the webhook ($exec) straight into an XML to JSON parser. That way you can use it easily in Shuffle. - - -### Splunk SIEM -Splunk is a SIEM tool for security operations. There are multiple ways to forward the Splunk alerts to external systems. Simplest way to forward splunk alerts to Shuffle is with using webhook. +### Splunk SIEM + +Splunk is a SIEM tool for security operations. There are multiple ways to forward the Splunk alerts to external systems. Simplest way to forward splunk alerts to Shuffle is with using webhook. **Step 1:** First we'll have to create a Shuffle workflow which will recieve alerts from Splunk. Go to /workflows in Shuffle and create a new worfklow. Then, inside workflow editor drag in the webhook from the trigger section in left pane. @@ -1056,7 +1206,7 @@ Splunk is a SIEM tool for security operations. There are multiple ways to forwar **Now that we have webhook running in the Shuffle, Go to your Splunk deployment server and log in.** -**Step 3:** Now we'll have to configure Splunk with the webhook URL. Once logged in, go to the **search and reporting** app and type in the query you want to create an alert for. +**Step 3:** Now we'll have to configure Splunk with the webhook URL. Once logged in, go to the **search and reporting** app and type in the query you want to create an alert for. ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/splunk-alerts-3.png) **Step 4:** Save the search query as an alert. @@ -1070,70 +1220,66 @@ Splunk is a SIEM tool for security operations. There are multiple ways to forwar **You should now start seeing data sent from Splunk into Shuffle which can be used inside workflow for further actions.** - ### Eventlog Analyzer + Note: API integration is unfortunately not supported with EventLog Analyzer. However, If you would like to forward logs from EventLog Analyzer to Shuffle. For more information please follow this [guide](https://www.manageengine.com/products/eventlog/help/StandaloneManagedServer-UserGuide/Configurations/log-forwarder.html) - -### ServicePilot SIEM +### ServicePilot SIEM + ServicePilot is a high-performance analytics platform that supports observability and full-stack monitoring: metrics, traces and logs. You can collect data from many services and sources across your entire IT stack (ITIM, NPM, APM, DEM, SIEM) as well as view details of historical data stored by ServicePilot. Webhook integration service is not provided by the ServicePilot platform. Here am gonna mentioned the steps for how to use ServicePilot app workflow. #### - What this workflow do? + This workflow is help to get all the alerts, objects, events and logs. In this workflow there are two nodes of ServicePilot app. one is for getting all the records from the ServicePilot SIEM app and another one is searching for alerts, objects, events and logs for specific entry. As a result this workflow first excecute the Get_All_Data node and after that it excecute the Get_Specific_Record node and at the last return the result. #### - How the seaching happens? + For searching records from the ServicePilot app we need to write SQL query to search the data. we need to write query as a query parameter in the request url. #### - Steps to create a workflow + **Step 1:** First we'll have to create a Shuffle workflow which will search for events, objects, alerts, and logs from ServicePilot. Go to /workflows in Shuffle and create a new worfklow. Then, inside workflow editor drag in the node for the ServicePilot. - ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/servicepilot-0.0.png) **Step 2:** Click on the ServicePilot node. - ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/servicepilot-1.0.png) **Step 3:** Change the name of the node as Get_All_Data. - ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/servicepilot-1.1.png) **Step 4:** Write Query for searching all the datas from the ServicePilot. - ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/servicepilot-1.2.png) **Step 5:** Inside workflow editor drag in the another node for the ServicePilot and connect it with the existing node. - ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/servicepilot-2.0.png) **Step 6:** Change the name of the node as Get_Specific_Record. - ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/servicepilot-2.1.png) **Step 7:** Write Query for searching specific record from the ServicePilot. - ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/servicepilot-2.2.png) **Finally, Click the excecution button and you should now start seeing data sent from ServicePilot into Shuffle which can be used inside workflow for further actions.** - - + ### ELK/Elastic + Intergrating Elastic with Shuffle will first require you to set up a webhook in Elastic. Below are the steps you will need to follow to ensure your alerts are forwaded into Shuffle immediately. 1. Create a new workflow in Shuffle, bottom left on your screen head to the triggers tab and drag in the webhook into your workflow. Click on it and ensure that the webhook is started. Copy the provided webhook URI and head over to Elastic. - - ![image](https://github.com/user-attachments/assets/e708d1fd-8222-4c47-ac29-7a2d5f62024e) - ![image](https://github.com/user-attachments/assets/98799857-7640-4cf7-aec4-31224a186320) +![image](https://github.com/user-attachments/assets/e708d1fd-8222-4c47-ac29-7a2d5f62024e) + +![image](https://github.com/user-attachments/assets/98799857-7640-4cf7-aec4-31224a186320) 3. Follow Elastic's documentation on setting up the webhook [here](https://www.elastic.co/guide/en/kibana/current/webhook-action-type.html). - When setting up the webhook in Elastic you will have to provide a url to send the info to. Provide the Shuffle webhook URI you copied in step 1 above - You do not need to provide authentication in Elastic while setting up, but just incase you do, remember to do the same in your webhook in Shuffle 4. To finish setting up your connector in Elastic you will be required to provide the body of the webhook connector, we will need to provide a JSON body for the connector. Copy and paste the below in the connector body - ```{"raw": {{context.alerts}}}``` + `{"raw": {{context.alerts}}}` 5. Go ahead and test your connector and then check your workflow executions in Shuffle if you have an incoming execution. - -### Cortex +### Cortex + TBD: Responder executions