Skip to content

Update Workflows to Version v1.0.2 #305

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
carpentries-bot wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update Workflows to Version v1.0.2 #305

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 78 additions & 30 deletions .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
# Workflow Documentation
# Workbench Workflows Documentation

These GitHub Actions workflows manage and maintain the build processes for lessons using the Carpentries Workbench on GitHub.


## Managing Workflow Updates

Expand All @@ -9,7 +12,11 @@ You can do this locally using your own R and Workbench installation, or via the

### Updating locally

In a terminal/git bash, navigate to the lesson folder where you want to update the workflows.
In a terminal/git bash, navigate to the lesson folder where you want to update the workflows, e.g.:

```bash
cd ~/lessons/shell-novice
```

Then, start an R session and:

Expand Down Expand Up @@ -39,7 +46,22 @@ $ git push origin main
> - Cancel any "01 Maintain: Build and Deploy Site" workflow currently running
> - Run the "02 Maintain: Check for Updated Packages" workflow and merge any PR opened to update the renv lockfile
> - This should automatically run the "03 Maintain: Apply Package Cache" workflow to install packages and build the cache
> - A successful cache buid should then trigger the "01 Maintain: Build and Deploy Site" workflow
> - A successful cache build should then trigger the "01 Maintain: Build and Deploy Site" workflow

#### Updating to a specific workflow release or branch

To test new Workbench workflow features or branches, or to pin to a specific release, the `branch` option can be supplied to `update_github_workflows()`.

For a given branch:

```r
sandpaper::update_github_workflows(branch="develop")
```

Or for a given release version:
```r
sandpaper::update_github_workflows(branch="1.0.1")
```

### Updating using GitHub

Expand Down Expand Up @@ -71,63 +93,90 @@ This will raise a PR with any changes to the workflows that are needed.
If you are happy with the changes made, you can merge the PR into your lesson repository.


## Package Caches for RMarkdown Lessons
## Dependency Images for RMarkdown Lessons

Lessons that use R packages and RMarkdown are built on GitHub using Docker images within these workflows.
These images are based on the [workbench-docker](https://github.com/carpentries/workbench-docker) image.
This base image only contains the Workbench packages to build lessons and not any extra R packages your lesson might need.

In summary, generating a reusable package cache is achieved by running the "02 Maintain: Check for Updated Packages" workflow, and then the "03 Maintain: Apply Package Cache" workflow.
Therefore, there are two steps to perform to resolve the required R packages in a lesson, and then generate a Docker image layer comprising those packages.

The "02 Maintain: Check for Updated Packages" workflow checks for any used R packages and their versions, and if successful, opens a Pull Request to update your lesson's renv lockfile (`renv.lock`).
Once the PR is merged, the "03 Maintain: Apply Package Cache" workflow builds the dependency image layer with `docker build` and publishes it in your user or organisation account Packages area.

> [!NOTE]
> Caching is only relevant for lessons that use Rmd files and renv to manage R packages.
> If you are building basic markdown documents, caching will not apply to you, and the only
> workflow that needs to be run is "01 Maintain: Build and Deploy Site".

### Caching
### Publishing dependency images

The two cache management workflows are separated to ensure that once you have a successful build with a working renv cache, this cache is stored and will be reused by the Workbench Docker container.
This means that lesson builds will be faster once an renv cache is created and reused by the Docker container.
The two dependency workflows are separated to ensure that once you have a successful build with a working renv environment, the resulting dependency image is stored and will be reused by the Workbench Docker container.
This means that lesson builds will be faster and more consistent once a dependency image is created and reused. This is doubly important if you need very specific versions of R packages, i.e. "pinning".

Another major bonus of this setup is that you can keep using this cache indefinitely to build your lesson.
This is important if you need very specific versions of R packages ("pinning").
This setup means that this dependency image layer is available indefinitely:
- to build your lesson, improving robustness and reproducibility
- for others to use as a complete environment in their local builds, teaching environments, or Codespaces

If and when you want to perform an update to the cache, you can re-run the "02 Maintain: Check for Updated Packages" and verify that your lesson still builds with the new packages.
If all looks good, re-run the "03 Maintain: Apply Package Cache" workflow, and this will write a new renv cache file to GitHub.
If and when you want to perform an update to the dependency image, you can re-run the "02 Maintain: Check for Updated Packages" workflow and verify that your lesson still builds with any new packages or changes.
The "03 Maintain: Apply Package Cache" workflow will publish a new repository- or organisation-local GHCR image for the lesson.

In any case, the renv cache is invalidated by new versions of the `renv.lock` file.
As such, the dependency image is invalidated by new versions of the `renv.lock` file.
This happens:
- if you update your lockfile locally by using the `sandpaper::update_cache()` function, and then push it to the lesson repository
- if you update your lockfile locally by using the `sandpaper::update_cache()` and `sandpaper::manage_deps()` functions, and then push it to the lesson repository
- when you run the "02 Maintain: Check for Updated Packages" and there are new packages to install

More information on managing local renv caches for lessons can be found in the [Sandpaper packages vignettes](https://carpentries.github.io/sandpaper/articles/building-with-renv.html).
More information on managing local renv-based lesson dependencies can be found in the [Sandpaper packages vignettes](https://carpentries.github.io/sandpaper/articles/building-with-renv.html).

#### Using different package cache versions
#### Using different dependency image versions

There are times when you may want to go back to a previous renv package cache file:
There are times when you may want to go back to a previous dependency image:
- if you run "02 Maintain: Check for Updated Packages" and "03 Maintain: Apply Package Cache" and the cache generation fails for some reason
- if there is a new R package that produces incorrect or broken lesson output
- a new R version is released and downstream packages fail to build and install, and need updating by the package maintainer(s)

Cache files will have the following name format, where IMAGE is the workbench-docker image version, and HASHSUM is the `renv.lock` lockfile MD5 hash:
Dependency images will have the following name format, where IMAGE is the workbench-docker image version, and HASHSUM is the `renv.lock` lockfile hash:

```
IMAGE HASHSUM
[ | ] [ | ]
v0.2.4_renv-2e499eb706112971b2cffceb49b55a6efe49f3ed75cd6579b10ff224489daca4
```

Copy the hashsum part of the desired cache file you want to use, e.g. `2e499eb706112971b2cffceb49b55a6efe49f3ed75cd6579b10ff224489daca4`.
Copy the hashsum part of the desired image tag you want to use, e.g. `2e499eb706112971b2cffceb49b55a6efe49f3ed75cd6579b10ff224489daca4`.

Then either:
1. Add a repository variable called CACHE_VERSION, and paste in the hash
- Go to ...
2. Run the "01 Maintain: Build and Deploy Site" manually, supplying the CACHE_VERSION input
- Go to ...
2. Run the "01 Maintain: Build and Deploy Site" manually, supplying the `Optional renv cache version override` input

If you have no caches listed, make sure to run the "02 Maintain: Check for Updated Packages" and "03 Maintain: Apply Package Cache" to create a new renv cache file.
If you have no images listed, make sure to run the "02 Maintain: Check for Updated Packages" and "03 Maintain: Apply Package Cache" workflows to publish a new dependency image.

> [!NOTE]
> If you are maintaining an official lesson, caches are saved in an AWS S3 bucket owned by the Carpentries.
> Once a successful cache has been saved, these will be listed in the outputs of the "01 Maintain: Build and Deploy Site" workflow.
> If you are maintaining an official lesson, dependency images are saved to the Carpentries lesson program organisation, e.g. `datacarpentry`, GHCR package namespace.
> Once a successful dependency image has been published, the build workflows will use it automatically when the matching Docker version tag exists, e.g. 'latest' or a specific version.
>
> If you are developing a lesson in your own repository, caches are saved on GitHub.
> You can see available caches by going to the Actions tab, and clicking Caches on the left hand side.
> If you are developing a lesson in your own repository, the dependency images are stored in your repository's GHCR package namespace.
> You can see available images by going to your user or organisation's Packages tab, e.g. `https://github.com/<your_github_username>?tab=packages`.

#### Pruning dependency image layers

Through the lifetime of a lesson, an increasing number of dependency images will be generated.
As the images are public, GitHub currently charges no storage, or network ingress or egress fees to upload or download them.

The "03 Maintain: Apply Package Cache" workflow will automatically check the number of published dependency images and keep each version published plus one underlying untagged digest (if any) for each version.
This means it is possible to roll back easily to a previous version or digest, but not end up with a huge number of dependency images to manage.

In any case, you can manually delete any versions you wish by:
- going to your user or organisation's Packages tab, e.g. `https://github.com/<your_github_username>?tab=packages`.
- selecting the dependency image you want to manage, e.g. `<lesson_repo_name>-deps`
- selecting `Package settings` on the right hand side
- deleting any images

> [!NOTE]
> Deleting these images manually may break your builds.
> Only perform this action if you are sure you want to delete the generated dependency images.
>
> If you mistakenly delete an image, you can rebuild it by rerunning the "03 Maintain: Apply Package Cache" workflow.


## User Settings
Expand Down Expand Up @@ -186,15 +235,14 @@ Repository-level variables for this workflow are:
- This can be set to a specific version number to force all builds to use a given container version
- Default is unset or `latest`


### 04 Maintain: Update Workflow Files (update-workflows.yaml)

There are no repository variables for this workflow.


## Pull Request and Review Management

Because our lessons execute code, pull requests are a security risk for any lesson and thus have security measures associted with them.
Because our lessons execute code, pull requests are a security risk for any lesson and thus have security measures associated with them.
**Do not merge any pull requests that do not pass checks and do not have bots commented on them.**

This series of workflows all go together and are described in the following diagram and the below sections:
Expand Down Expand Up @@ -256,5 +304,5 @@ This emits an artifact that is the pull request number for the next action.

### Remove Pull Request Branch (pr-post-remove-branch.yaml)

Tiggered by `pr-close-signal.yaml`.
Triggered by `pr-close-signal.yaml`.
This removes the temporary branch associated with the pull request (if it was created).
Loading
Loading