Skip to content

Route AlmaLinux 10 x86_64 ALTS tasks to x86-64-v3 workers - #44

Merged
anfimovdm merged 4 commits into
masterfrom
al-el10-x86-64-v3-queue
Oct 27, 2026
Merged

anfimovdm merged 4 commits into
masterfrom
al-el10-x86-64-v3-queue

Conversation

@anfimovdm

@anfimovdm anfimovdm commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Resolves AlmaLinux/build-system#563
Depends on AlmaLinux/alts#236

Summary

Deployment side of sending AlmaLinux 10 x86_64 test tasks only to x86-64-v3 capable workers, plus two ALTS scheduler deployment fixes found while rolling it out.

Changes

  • Route AlmaLinux 10 x86_64 tasks to x86-64-v3 workers
    • playbooks/templates/alts_service_config.yaml.j2: renders queue_arch_overrides for the scheduler only (it's a scheduler setting). Versions are quoted so they stay strings.
    • inventories/alts/group_vars/scheduler.yml: enables the override for AlmaLinux 10, x86_64 → x86_64_v3.
    • inventories/alts/hosts.yml: the example x86_64 worker also consumes docker-x86_64_v3-0, so AlmaLinux 10 tasks have a consumer there.
  • Separate scheduler broker and Redis hosts
    • The scheduler config used rabbitmq_host/alts_redis_host, the same variables as the workers. When the worker-facing addresses are passed as extra vars, they override the scheduler's group_vars.
    • The scheduler now uses scheduler_rabbitmq_host/scheduler_redis_host (renamed in group_vars/scheduler.yml); the workers keep the old names.
  • ansible-core 2.19
    • roles/alts_scheduler/tasks/main.yml: Create scheduler working directory failed because its when: evaluated a string as a boolean.
  • Terraform install on workers
    • roles/alts_terraform/tasks/main.yml: the role downloaded and unpacked Terraform on every run, and in --check mode failed with Source '/tmp/terraform.zip' does not exist because get_url doesn't download in check mode. It now reads the installed version and downloads, unpacks and cleans up only when Terraform is missing or a different version; the unpack is skipped in check mode.

Upgrade notes

  • Inventories that set rabbitmq_host/alts_redis_host for the scheduler need to rename them to scheduler_rabbitmq_host/scheduler_redis_host.
  • Add docker-x86_64_v3-0 to celery_queues only on workers whose CPU supports x86-64-v3 (grep -c avx2 /proc/cpuinfo is non-zero). With the override enabled, AlmaLinux 10 x86_64 tasks wait in RabbitMQ until such a worker consumes the queue.

Rollout

  1. Deploy Schedule AlmaLinux 10 x86_64 tests only on x86-64-v3 capable workers alts#236 with the celery playbook to the workers, x86-64-v3 capable ones with the new queue.
  2. Check the new queue has a consumer.
  3. Run the scheduler playbook.

Testing

  • Rendered the template with the scheduler and worker settings. The scheduler gets scheduler_rabbitmq_host/scheduler_redis_host and queue_arch_overrides; the worker gets rabbitmq_host/alts_redis_host and no override, also with the worker-facing addresses passed as extra vars.
  • ansible-inventory --host resolves the renamed variables and queue_arch_overrides for the scheduler and the new queue for the worker.
  • alts_scheduler.yml in --check --diff mode against production got past the working directory task and showed the expected config diff. Not yet applied.
  • alts_celery.yml passes --syntax-check with the Terraform change; the role itself hasn't been re-run yet.

ansible-core 2.19 requires conditionals to evaluate to a boolean, so testing scheduler_working_directory (a string) directly fails the task.
The scheduler config used rabbitmq_host and alts_redis_host, the same variables as the workers. When the worker-facing addresses are passed as extra vars, they override the scheduler's group_vars, and the scheduler config gets addresses meant for the workers.

The scheduler now uses scheduler_rabbitmq_host and scheduler_redis_host; the workers keep rabbitmq_host and alts_redis_host.

Related: AlmaLinux/build-system#563
The AlmaLinux 10 linux/amd64 image needs an x86-64-v3 CPU. Render the new scheduler setting queue_arch_overrides and enable it in the scheduler group_vars, so AlmaLinux 10 x86_64 tasks go to the docker-x86_64_v3-* queues. Only x86-64-v3 capable workers should list docker-x86_64_v3-0 in celery_queues; the example x86_64 worker does.

Resolves: AlmaLinux/build-system#563
The role downloaded and unpacked Terraform on every run, so it always reported changes. In check mode get_url doesn't download, and the unpack failed with "Source '/tmp/terraform.zip' does not exist". Read the installed version first and download, unpack and clean up only when Terraform is missing or a different version; skip the unpack in check mode.
@anfimovdm
anfimovdm merged commit bd93321 into master Oct 27, 2026
2 checks passed
@anfimovdm
anfimovdm deleted the al-el10-x86-64-v3-queue branch October 27, 2026 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ALTS: schedule AlmaLinux 10 x86_64 tests only on x86-64-v3 capable workers

1 participant