repodive-tools

Scripts and tools to gather information about git repositories.

About the Project

This repository is a collection of scripts and tools for a given repodiving effort. Repodiving in this context means going through a git repository and gathering relevant information for a specific purpose.

Agency Mission

We're a group of civic-minded technologists transforming how the federal government delivers healthcare to the American people.

Team Mission

Establish and maintain guidance, policies, practices, and talent pipelines that advance equity, build trust, and amplify impact across CMS, HHS, and Federal Open Source Ecosystems by working and sharing openly.

Core Team

A full list of contributors can be found on https://github.com/DSACMS/repodive-tools/graphs/contributors.

Development and Software Delivery Lifecycle

We follow GitHub Flow with protected branches and pull request reviews. Development includes automated code analysis, security scanning, and adherence to CMS Open Source Policy guidelines. See CONTRIBUTING.md for more details.

Local Development

To run locally, please follow the instructions in CONTRIBUTING.md under Buidling the Project and Building Dependencies.

Scripts Available

Run SCC on repos

Script: run-scc-on-repos.sh

Overview

This script automates running scc across every repository in a GitHub organization. Instead of manually cloning each repository and executing scc one at a time, the script retrieves every repository through the GitHub API, clones any that are not already available locally, runs scc on each repository, and saves the resulting metrics as JSON reports for later analysis.

Prerequisites:

• Access to the configured GitHub Enterprise organization.
• A valid GitHub Enterprise Personal Access Token.
• Required dependencies git, curl, jq, scc.

Usage

1. Install all required dependencies
2. Set the required environment variables, including GITHUB_TOKEN
3. ./run-scc-on-repos.sh <Directory to store GitHub code>

Notes

• The SCC script clones repositories into the directory you specify.
• If the repository already exists in that directory, it will not be cloned again.
• Existing local clones can be reused when running other repodive scripts.
• SCC reports are generated as JSON files and stored in the scc_reports directory.

Gen Gource logs on Repositories

Script: gen-gource-logs-on-repos.sh

Overview

This script retrieves repositories from the configured GitHub organization, clones or updates them locally, generates a Gource custom log for each repository, and formats the logs so they can be combined into a single visualization of repository activity.

• The generated log files contain timestamped repository events (such as file additions, modifications, and deletions) that can later be combined and used by Gource to generate repository activity visualizations.

Prerequisites:

• Access to the configured GitHub organization.
• A valid GitHub Personal Access Token.
• Required dependencies git, curl, jq, gource.

Usage

1. Install all required dependencies
2. Set the required environment variables, including GITHUB_TOKEN
3. ./gen-gource-logs-on-repos.sh <Directory to store GitHub code>

Notes

• The gource script clones repositories into the directory you specify. If the repository already exists in that directory, it will not be cloned again. This is useful for if you want to re-use this directory to run the other scripts on it.
• The script also makes sure to get the latest version of the local clone using git pull.
• Existing local clones can be reused when running other repodive scripts.
• Gource log files are generated for each repository and stored in the gource_logs directory.
• Each generated log is formatted so it can be combined with logs from other repositories for organization-wide visualizations.

Generate a Combined Contributor Breakdown

Scripts: run-contrib-resolution.sh, concat.sh

Overview

The first script generates a contributor report for each cloned Git repository using git shortlog. The second script combines those reports, adds together commit counts for matching contributor names and email addresses, and produces a sorted contributor breakdown across all repositories.

• This script assumes that all of the repositories have been cloned already.

Prerequisites:

• All target repositories must already be cloned into one directory.
• Required dependencies git.
• Required shell utilities, including awk, cat, sort, and mktemp, must be available.

Usage

1. Install all required dependencies
2. ./run-contrib-resolution.sh <Directory with the GitHub Code already there>
3. Enter ctrl+d for any empty records that appear
4. ./concat.sh
5. Look at merged_output.txt and enjoy!

Notes

• run-contrib-resolution.sh saves one contributor report per repository in the shortlog_reports directory.
• Contributor identities are grouped by their exact name and email combination.
• merged_output.txt contains the combined contributor totals.
• shortlog-breakdown.txt contains the same totals sorted from highest to lowest.

Additional Repository Analysis Techniques

Using GitHub's Search Function to Find Project Dependants

A useful feature of GitHub is the search feature that you can find here.

Using the search feature you can search projects that your GitHub account has permission to view and filter the results to your liking.

For example, you can search GitHub for projects that have a specific project as a dependency. This is known as searching for a project's dependants. This is done by using the path: keyword in order to filter the results that have a specific file in the results. Therefore, if you add the parameter path:package.json to your search you will search GitHub for Javascript project dependency files and you can search for projects that have a specific Javascript dependency this way.

Here is an example of searching GitHub for projects that use the package @trussworks/react-uswds:

https://github.com/search?q=%40trussworks%2Freact-uswds+path%3Apackage.json+NOT+is%3Afork&type=code

You might also notice that there are some parameters added such as the parameter NOT is:fork which excludes projects that are a fork of other projects. You can also do additional parameters to further filter the results by categories such as org. To filter by organization you can add org:DSACMS in order to only show projects that belong to that organization.

To read more about the GitHub search feature from GitHub's official documentation you can get to that here

Policies

Open Source Policy

We adhere to the CMS Open Source Policy. If you have any questions, just shoot us an email.

Security and Responsible Disclosure Policy

Submit a vulnerability: Vulnerability reports can be submitted through Bugcrowd. Reports may be submitted anonymously. If you share contact information, we will acknowledge receipt of your report within 3 business days.

For more information about our Security, Vulnerability, and Responsible Disclosure Policies, see SECURITY.md.

Software Bill of Materials (SBOM)

A Software Bill of Materials (SBOM) is a formal record containing the details and supply chain relationships of various components used in building software.

In the spirit of Executive Order 14028 - Improving the Nation’s Cyber Security, a SBOM for this repository is provided here: https://github.com/{{ cookiecutter.project_org }}/{{ cookiecutter.project_repo_name }}/network/dependencies.

For more information and resources about SBOMs, visit: https://www.cisa.gov/sbom.

Public domain

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication as indicated in LICENSE.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request or issue, you are agreeing to comply with this waiver of copyright interest.