diff --git a/bootstrapping-fleet/byok-aws.md b/bootstrapping-fleet/byok-aws.md new file mode 100644 index 0000000..3532d5d --- /dev/null +++ b/bootstrapping-fleet/byok-aws.md @@ -0,0 +1,208 @@ +--- +title: "BYOK with AWS" +parent: Bootstrapping Your Fleet +nav_order: 6 +--- + +# BYOK with AWS: Prerequisites + +This page covers the prerequisites required for using an AWS EKS cluster +through Bring Your Own Kubernetes (BYOK) in Streamtime — sizing, the IAM +role Streamtime needs, and the kubeconfig format Streamtime expects. See +[Advanced Usage (BYOK)](byok.html) for the provider-agnostic walkthrough +of the Streamtime UI flow itself. + +This page assumes you already have a working EKS cluster (OIDC provider +associated, `kubectl` access configured) — that setup is standard EKS +administration. + +--- + +## 1. Recommended EKS sizing for bring-your-own clusters + +One Kafka Unit (KU) is Streamtime's measure of Kafka throughput and +resource need (1 KU = 20 MB/s). The following table can be used as a +reference to size an EKS cluster that will be used to run Kafka using +Streamtime — actual sizing might vary depending on your workload. + +| Cluster capacity | Platform nodes | Kafka / data nodes | Node size (balanced) | +|---|---|---|---| +| 1–3 units | 3 | 3 | 2 vCPU / 8 GB (`m6i.large`) | +| 4+ units | 3 | Match capacity (one data node per unit, minimum 3) | Scale vCPU/memory with capacity | + +Guidance: + +- These sizes assume the cluster is dedicated to Streamtime. Don't pack + unrelated workloads onto the same nodes. +- Platform nodes run monitoring, operators, and the agent. Kafka/data + nodes run your brokers. +- You may combine platform and Kafka nodes into a single node group if + you prefer a simpler layout (for example, 6 nodes for 1–3 units of + capacity). +- Spread nodes across availability zones for resilience. +- No cluster should run with fewer than 3 nodes: for the node group's + autoscaling config, use `minSize=3, desiredSize=3, maxSize=6` as a + starting point, and raise `maxSize` further for higher-capacity tiers. +- Use general-purpose-plus memory-optimized instance families (the `m5`/`m6i` + series) rather than burstable `t`-series instances, which aren't suitable + for sustained Kafka workloads. + +## 2. IAM role for Streamtime + +Streamtime uses a single IRSA (IAM Roles for Service Accounts) role, +trusted by your cluster's OIDC provider, for its in-cluster service +accounts — including the one used to create and write to the S3 bucket +that backs log/metrics storage and Kafka tiered storage. + +Create the role with this trust policy, scoped to the `streamtime-agent`, +`cluster-autoscaler`, and `kafka-fleet-manager-loki` service accounts: + +```json +{ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Principal": { + "Federated": "arn:aws:iam:::oidc-provider/" + }, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringEquals": { + ":aud": "sts.amazonaws.com" + }, + "StringLike": { + ":sub": [ + "system:serviceaccount:streamtime-agent:streamtime-agent*", + "system:serviceaccount:kube-system:cluster-autoscaler*", + "system:serviceaccount:monitoring:kafka-fleet-manager-loki*" + ] + } + } + }] +} +``` + +Attach the AWS-managed `AmazonS3FullAccess` policy, then add the +following inline policy: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + {"Sid": "STSValidation", "Effect": "Allow", + "Action": ["sts:GetCallerIdentity"], "Resource": "*"}, + {"Sid": "EKSCluster", "Effect": "Allow", "Action": "eks:*", + "Resource": "arn:aws:eks:::cluster/"}, + {"Sid": "EKSNodegroups", "Effect": "Allow", "Action": "eks:*", + "Resource": "arn:aws:eks:::nodegroup//*"}, + {"Sid": "EC2Describe", "Effect": "Allow", + "Action": ["ec2:DescribeSubnets", "ec2:DescribeSecurityGroups", + "ec2:DescribeInstances", "ec2:DescribeInstanceTypes"], + "Resource": "*"}, + {"Sid": "IAMEKSRoles", "Effect": "Allow", + "Action": ["iam:PassRole", "iam:GetRole", "iam:ListAttachedRolePolicies", + "iam:ListRolePolicies", "iam:GetRolePolicy"], + "Resource": "arn:aws:iam:::role/streamtime-eks-*"}, + {"Sid": "IAMServiceLinkedRole", "Effect": "Allow", + "Action": ["iam:GetRole", "iam:CreateServiceLinkedRole"], + "Resource": "arn:aws:iam:::role/aws-service-role/eks-nodegroup.amazonaws.com/*"}, + {"Sid": "IAMStoragePolicyManage", "Effect": "Allow", + "Action": ["iam:CreatePolicy", "iam:DeletePolicy", + "iam:GetPolicy", "iam:ListPolicies"], + "Resource": "arn:aws:iam:::policy/streamtime-storage-*"}, + {"Sid": "IAMStoragePolicyAttach", "Effect": "Allow", + "Action": ["iam:AttachRolePolicy", "iam:DetachRolePolicy", + "iam:ListAttachedRolePolicies", "iam:GetRole", + "iam:UpdateAssumeRolePolicy"], + "Resource": "arn:aws:iam:::role/"} + ] +} +``` + +## 3. IP addressing for pods and services + +Size your subnets for your target node count with headroom, not just the +starting count — undersized subnets surface as pods stuck in a `Pending` +state as the cluster grows. As a rough reference point, a production +fleet typically runs on the order of **100–200 pods and services** +combined (Kafka brokers, operators, monitoring, and the agent), scaling +up toward the higher end for larger KU tiers. + +## 4. Load balancer and storage class + +- **Load balancer**: Streamtime exposes its endpoints through a standard + Kubernetes `Service` of type `LoadBalancer`. On EKS this provisions a + load balancer automatically — no separate load balancer controller + needs to be installed. +- **Default storage class**: Streamtime runs several components that + need persistent storage. Your cluster must have a default + `StorageClass` configured — `gp3` is recommended. On EKS this means + the EBS CSI driver addon is installed and a `StorageClass` using the + `gp3` volume type is marked as default. Without one, Streamtime's + storage-backed components will stay stuck in a `Pending` state and + bootstrapping the fleet will not complete. + +## 5. Generating a kubeconfig for the Streamtime Agent's automatic installation + +Streamtime expects a static token in the kubeconfig for user +authentication when using Automatic Installation of the agent. This +kubeconfig is only used to install the Streamtime Agent, and the token +in it should be short-lived. See [Advanced Usage (BYOK)](byok.html) for +more on the Automatic Installation flow. + +Assuming the kubeconfig is set to the EKS cluster, the following commands +can be used to generate a kubeconfig with a static token for an EKS +cluster: + +1. Create a service account and bind it to `cluster-admin`: + ```bash + kubectl create serviceaccount streamtime-admin -n kube-system + kubectl create clusterrolebinding streamtime-admin-binding \ + --clusterrole=cluster-admin \ + --serviceaccount=kube-system:streamtime-admin + ``` +2. Generate a token (shown once — save it): + ```bash + kubectl create token streamtime-admin -n kube-system --duration=24h + ``` +3. Grab the endpoint and CA data: + ```bash + aws eks describe-cluster --name --region --query 'cluster.endpoint' --output text + aws eks describe-cluster --name --region --query 'cluster.certificateAuthority.data' --output text + ``` + +### Step 4: assemble the kubeconfig + +Assemble the kubeconfig in exactly this shape: + +```yaml +apiVersion: v1 +kind: Config +clusters: +- cluster: + server: https:// + certificate-authority-data: + name: +contexts: +- context: + cluster: + user: streamtime-admin + name: -context +current-context: -context +users: +- name: streamtime-admin + user: + token: +``` + +Three rules Streamtime enforces strictly: +- `clusters[].name` must be a short cluster name, **not** the cluster ARN. +- `users[].user` must be a **nested object** containing `token:` — a flat + `user: ` string will fail. +- Spaces only, no tabs, anywhere in the file. + +### Step 5: upload the kubeconfig + +Upload/paste the kubeconfig file as a Secret in the **Agent Management** +section of the fleet. See the [Kubeconfig section](byok.html) of the +BYOK documentation.