[Docs] Fixes to changelog for version 3.16.0. - #7523
Conversation
764fee6 to
60fb57b
Compare
60fb57b to
9e8aa64
Compare
| - Enforce NFSv4-only on the ParallelCluster-managed NFS server (head node). The NFSv3 client stack (rpcbind, rpc-statd, lockd) are unchanged, so cluster nodes can still mount external NFSv3 servers. | ||
| - Change the default NFS lock manager port from 32768 to 4045. 32768 is in the Linux ephemeral port range (32768–60999), causing sporadic mount failures because of port collision. This only affects nodes that mount an external NFSv3 server; all ParallelCluster managed storage is mounted over NFSv4 and is unaffected. Customers who mount external NFSv3 servers and restrict NFS ports in a firewall must allow TCP/UDP 4045 instead of 32768. | ||
| - The CLI now requires the additional permission `tag:GetResources`, which it uses to resolve login nodes load balancer ARNs by tags (see the related bugfix). | ||
| - Add support for Python 3.13 in the pcluster CLI. |
There was a problem hiding this comment.
Minor: Can we remove "the" to be consistent with previous changelogs?
| - Change the default NFS lock manager (lockd) port from 32768 to 4045, to avoid collisions with the Linux ephemeral port range (32768–60999), which can cause sporadic mount failures. | ||
| This only affects nodes that mount an external NFSv3 server; all ParallelCluster-managed storage is mounted over NFSv4 and is unaffected. | ||
| If you restrict NFS ports in a firewall, you must open TCP/UDP 4045 or NFSv3 mounts will fail. | ||
| - Install the NVIDIA driver, CUDA toolkit, Fabric Manager, NVLSM, and IMEX from the distribution package manager using NVIDIA local repo packages instead of the run file installers. |
There was a problem hiding this comment.
Minor: "run file installers" -> ".run file installers"
| compute nodes, removing the dependency on cfn-hup and cfn-init. | ||
| - Move all ParallelCluster-managed bootstrap files off `/tmp` into a dedicated `/opt/parallelcluster/tmp` | ||
| directory. Therefore, cluster creations and updates work on custom AMIs that mount `/tmp` with `noexec`. Image builds work on the custom AMIs only if GDRcopy installation is skipped. | ||
| directory, so that image builds, cluster creations, and updates work on custom AMIs that mount `/tmp` with noexec. |
There was a problem hiding this comment.
We should keep the exception of GDRcopy described before this PR
| **ENHANCEMENTS** | ||
| - Improve cluster update resiliency on login nodes by reusing the head-node-driven orchestration already in place on compute nodes, | ||
| removing the dependency on cfn-hup and cfn-init. | ||
| - Add the `pcluster-diag` diagnostics tool to ParallelCluster AMIs for running on-demand diagnostic checks on cluster nodes. |
There was a problem hiding this comment.
Minor: Is "the" necessary?
| - Improve resilience of cluster updates on login nodes by reusing the head-node-driven orchestration already in place on | ||
| compute nodes, removing the dependency on cfn-hup and cfn-init. |
There was a problem hiding this comment.
Minor: How about "Improve resilience of cluster updates on login nodes by reusing the head-node-driven orchestration, removing the dependency on cfn-hup and cfn-init."
| - Upgrade NVIDIA driver, Fabric Manager, and IMEX to version 580.173.02 (from 580.126.20). | ||
| - Upgrade CUDA Toolkit to version 13.3.1 (from 13.0.2). | ||
| - Upgrade DCGM to version 4.6.0 (from 4.5.1). | ||
| - In GPU Health Check, skip DCGM diagnostics when NVIDIA MIG is enabled because dcgmi diag does not support MIG. |
There was a problem hiding this comment.
Minor: let's unify the terminology to either "DCGM diagnostics" or dcgmi diag
| - Rdma-core: rdma-core-63.0-1 | ||
| - Open MPI: openmpi40-aws-4.1.7-3 and openmpi50-aws-5.0.9-11 | ||
| - Upgrade mysql-community-client to version 8.4.10 (from 8.4.8). | ||
| - Install the aws-parallelcluster-node package from S3 in all regions instead of PyPI, to support air-gapped and proxied environments. |
There was a problem hiding this comment.
Minor: "to support air-gapped and proxied environments" -> "to support environments without PyPI access"
| Users must now allowlist the CloudFront domain `amazon-efs-utils.aws.com` in their proxy/egress configuration | ||
| when using `build-image` with isolated subnets. |
There was a problem hiding this comment.
Minor: How about
"Users must now allowlist the CloudFront domain amazon-efs-utils.aws.com in their firewall configuration (if exist) when using build-image."
| removing the dependency on cfn-hup and cfn-init. | ||
| - Add the `pcluster-diag` diagnostics tool to ParallelCluster AMIs for running on-demand diagnostic checks on cluster nodes. | ||
| - Improve resilience of cluster creation by retrying EBS volume attachment on transient IMDS connectivity failures. | ||
| - Further reduce transient build-image failures on RHEL and Rocky caused by out-of-sync repo mirrors by resetting metadata upon retry. |
There was a problem hiding this comment.
Minor: let's quote build-image or not quote consistently in this changelog
| - Fix SELinux not being disabled on RHEL-family OSes (kernels >= 6.4) due to a deprecated mechanism being silently | ||
| ignored by newer kernels. | ||
| - Fix `build-image` failure by skipping installation of `fabric-manager`, `gdrcopy`, and `dcgm` if the parent image already has a version installed. | ||
| - Fix an issue where compute nodes are replaced when launching a large number of nodes due to eventual consistency. |
There was a problem hiding this comment.
Minor: "eventual consistency" -> "API eventual consistency"
| - Install the NVIDIA driver, CUDA toolkit, Fabric Manager, NVLSM, and IMEX from the distribution package manager using NVIDIA local repo packages instead of the run file installers. | ||
| - On RHEL-family OSes, install the Xorg driver for DCV GPU acceleration and disable Wayland so that GDM always starts Xorg. | ||
| This is now required after switching NVIDIA driver installation to local repo packages; previously it was needed only on Ubuntu. | ||
| - Upgrade NVIDIA driver, Fabric Manager, and IMEX to version 580.173.02 (from 580.126.20). |
There was a problem hiding this comment.
Nvidia version is wrong
| - Install amazon-efs-utils from the official EFS endpoint instead of building from source. | ||
| Users must now allowlist the CloudFront domain `amazon-efs-utils.aws.com` in their proxy/egress configuration | ||
| when using `build-image` with isolated subnets. | ||
| - Upgrade Cinc Client to version 19.3.14 (from 18.8.54). |
There was a problem hiding this comment.
We have a weird version in the code https://github.com/aws/aws-parallelcluster/blob/develop/cli/src/pcluster/constants.py#L161
However, I believe this code is not used
There was a problem hiding this comment.
This is an interesting catch!
We upgraded the Cinc version to 19.3.14 for the build image, which is set here
But in user data we still use the old version, which is set here https://github.com/aws/aws-parallelcluster/blob/develop/cli/src/pcluster/constants.py#L161
| - Fix `build-image` failure by skipping installation of `fabric-manager`, `gdrcopy`, and `dcgm` if the parent image already has a version installed. | ||
| - Fix an issue where compute nodes are replaced when launching a large number of nodes due to eventual consistency. | ||
| - Fix a race condition where a fleet start was not enough to exit protected mode. | ||
| Even if ParallelCluster was not launching new nodes, issuing the start too early, while some nodes were still coming up, could leave the fleet in protected mode. |
There was a problem hiding this comment.
This is actually not 100% true. See aws/aws-parallelcluster-node#707.
The root cause is not issuing the start too early, but the stop. No matter when issuing cluster.start(), this bug can be hit. As it's hard to explain the root cause, I suggest remove the second paragraph. Should we reconsider the previous changelog Fix an issue where starting the compute fleet may not reliably recover the cluster from protected mode.?
This was what I wrote when found this bug:
-
How to hit the bug (all must hold)
- A job's dynamic nodes keep failing bootstrap, pushing the partition's failure count to the threshold.
- A running job in the same partition delays the entry into protected mode. During this delay, the requeued failing job wakes a fresh batch of nodes, so at the moment protected mode finally enters, those nodes are still powering up and not yet cleaned.
- The fleet is stopped before the next cleanup poll (~60s window), the only chance to reset the dirty nodes is missed.
- After start, the dirty nodes found in one poll ≥
protected_failure_count.
-
Customer impact (if hit)
- After a customer stops and starts the compute fleet in protected mode, the disabled partitions unexpectedly re-enters protected mode shortly (<60s) with no new provisioning failure. In other words, the partitions flips back to INACTIVE, while the customer is expecting UP.
- A second START_REQUESTED would always succeed.
-
Root cause
- When protected mode sets a partition INACTIVE, it does not reset that partition's nodes in the same clustermgtd poll -> cleanup is deferred to the next poll.
- If the fleet is stopped before that next poll arrives, nodes that were still powering up (backing instances terminated due to boot failure, NodeAddr still set) survive stop/start cycle:
cluster.stop()and clustermgtd poll duringSTOPPEDstate skip already-INACTIVE partitions. - On the first poll after start, clustermgtd (counter reset by its restart) counts these leftover nodes as fresh bootstrap failures and re-enters protected mode.
3 participants
Description of changes
Fixes to changelog for version 3.16.0: added missing entries, moved existing ones to the correct sections and fixed grammar.
Tests
Wording approved by PM and SDM.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.