Skip to content

fix: report only genuinely current CI state #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fredclausen merged 3 commits into from
Aug 10, 2026
Merged

fix: report only genuinely current CI state #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
990548f
fix: report only genuinely current CI state
fredclausen Aug 10, 2026
9634586
fix: address review findings on CI-state accuracy
fredclausen Aug 10, 2026
3c406d6
fix: write cache entries under the cache key, not the URL
fredclausen Aug 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,8 @@ flag alone.
| `github_repo_pulls_draft` | gauge | `org`, `repo`, `author_kind` |
| `github_pull_created_timestamp_seconds` | gauge | `org`, `repo`, `number`, `author`, `author_kind`, `draft` |
| `github_workflow_run_status` | gauge | `org`, `repo`, `workflow`, `event`, `conclusion` |
| `github_workflow_run_stale` | gauge | `org`, `repo`, `workflow` |
| `github_workflow_enabled` | gauge | `org`, `repo`, `workflow`, `state` |
| `github_workflow_run_timestamp_seconds` | gauge | `org`, `repo`, `workflow` |
| `github_workflow_last_success_timestamp_seconds` | gauge | `org`, `repo`, `workflow` |
| `github_workflow_expected_interval_seconds` | gauge | `org`, `repo`, `workflow` |
Expand All @@ -114,6 +116,19 @@ activity for visibility while alert rules select only `author_kind="human"`.
workflow. An in-flight run reports `conclusion="running"` and does not clear a
previous failure.

## What counts as the current CI state

Getting this right matters more than it sounds. Naively taking the newest run
per workflow produced 38 "failures", of which fewer than half were real:

| Filter | Why |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Workflow must still exist | GitHub keeps run history after a workflow file is deleted. A removed `Update pre-commit hooks` reported a permanent failure across 10 repositories. |
| Identity is the file path | Runs predating a workflow's `name:` report the path instead, splitting one workflow into two series. |
| Branch-state events only | The API's `branch=` filter matches a pull request's *head* branch, so PR runs leak in. A merged PR's last pre-merge failure would otherwise be the branch's CI state forever. |
| Runs age out after 90 days | Some workflows only fire on `pull_request`, leaving a branch-state run many months old. Those report `conclusion="stale"` and set `github_workflow_run_stale`, rather than an unclearable failure. |
| Disabled workflows are kept | A workflow auto-disabled by GitHub after 60 days of inactivity has stopped running silently. That is the fault worth alerting on, so it is reported rather than filtered out. |

## Configuration

TOML file, with `GHCI_`-prefixed environment overrides:
Expand Down
158 changes: 120 additions & 38 deletions github-ci-exporter/src/collector.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ use crate::{
config::Config,
github::{Client, client::RateLimitResource, graphql, rest},
metrics::{
AuthorLabels, Metrics, PullLabels, RepoLabels, ResourceLabels, SkipLabels, WorkflowLabels,
WorkflowStateLabels, author_label,
AuthorLabels, Metrics, PullLabels, RepoLabels, ResourceLabels, SkipLabels,
WorkflowEnabledLabels, WorkflowLabels, WorkflowStateLabels, author_label,
},
model::{Repo, SkipReason},
};
Expand Down Expand Up @@ -45,8 +45,12 @@ const fn estimate_core_requests(monitored: u64) -> u64 {
pub struct WorkflowCache {
/// `owner/name` -> workflow name -> expected interval in seconds.
intervals: HashMap<String, HashMap<String, i64>>,
/// `owner/name` -> whether the repo has any workflow files at all.
has_workflows: HashMap<String, bool>,
/// `owner/name` -> the workflow files currently present in the repo.
///
/// Retained so run history can be intersected against it: GitHub keeps
/// runs of deleted workflows forever, and reporting them shows failures
/// for CI that no longer exists.
workflows: HashMap<String, Vec<rest::Workflow>>,
/// Repositories monitored by the previous cycle, used to size the
/// budget pre-flight check.
monitored_count: u64,
Expand Down Expand Up @@ -139,36 +143,7 @@ pub async fn collect(
let (candidates, mut skipped) =
graphql::partition_repos(discovered, &|name| config.is_denylisted(name), max_age, now);

// Workflow presence. Repos with no workflow files are content-hosting
// repos with no CI signal; they are dropped so the dashboard is not
// padded with permanently-empty rows.
let mut monitored = Vec::with_capacity(candidates.len());
for repo in candidates {
let key = repo.full_name();
let workflows = match rest::list_workflows(client, &repo).await {
Ok(workflows) => workflows,
Err(error) => {
warn!(repo = %repo, %error, "failed to list workflows; keeping repository");
monitored.push(repo);
continue;
}
};

let has_workflows = !workflows.is_empty();
cache.has_workflows.insert(key.clone(), has_workflows);

if !has_workflows && config.skip_repos_without_workflows {
skipped.push((repo, SkipReason::NoWorkflows));
continue;
}

// Resolve cron schedules once per workflow set; they change rarely.
if let std::collections::hash_map::Entry::Vacant(entry) = cache.intervals.entry(key) {
entry.insert(resolve_cron_intervals(client, &repo, &workflows).await);
}

monitored.push(repo);
}
let monitored = resolve_monitored(client, config, cache, candidates, &mut skipped).await;

info!(
monitored = monitored.len(),
Expand All @@ -189,8 +164,25 @@ pub async fn collect(

// Actions runs, one request per repository.
for repo in &monitored {
match rest::fetch_runs(client, repo).await {
Ok(runs) => record_runs(metrics, repo, &runs, cache),
let live = cache
.workflows
.get(&repo.full_name())
.cloned()
.unwrap_or_default();
// Without a workflow set every run would be discarded as orphaned,
// publishing no series at all and making a listing failure look like
// "this repository's CI vanished". Skip the fetch rather than spend a
// request that cannot produce a result.
if live.is_empty() {
warn!(
repo = %repo,
"no workflow set available; skipping run fetch for this cycle"
);
continue;
}
record_workflow_states(metrics, repo, &live);
match rest::fetch_runs(client, repo, &live).await {
Ok(runs) => record_runs(metrics, repo, &runs, cache, now),
Err(error) => warn!(repo = %repo, %error, "failed to fetch workflow runs"),
}
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Expand Down Expand Up @@ -281,6 +273,70 @@ fn record_repo_inventory(metrics: &Metrics, monitored: &[Repo], skipped: &[(Repo
}
}

/// Determines which candidates actually have CI, caching their workflow sets.
///
/// Repositories with no workflow files are content-hosting repos with no CI
/// signal; they are dropped so the dashboard is not padded with permanently
/// empty rows.
async fn resolve_monitored(
client: &Client,
config: &Config,
cache: &mut WorkflowCache,
candidates: Vec<Repo>,
skipped: &mut Vec<(Repo, SkipReason)>,
) -> Vec<Repo> {
let mut monitored = Vec::with_capacity(candidates.len());

for repo in candidates {
let key = repo.full_name();
let workflows = match rest::list_workflows(client, &repo).await {
Ok(workflows) => workflows,
Err(error) => {
// A listing failure is not evidence of absent CI, so the
// repository is kept rather than silently dropped.
warn!(repo = %repo, %error, "failed to list workflows; keeping repository");
monitored.push(repo);
continue;
}
};

if workflows.is_empty() && config.skip_repos_without_workflows {
cache.workflows.remove(&key);
skipped.push((repo, SkipReason::NoWorkflows));
continue;
}
cache.workflows.insert(key.clone(), workflows.clone());

// Cron schedules change rarely, so they are resolved once per set.
if let std::collections::hash_map::Entry::Vacant(entry) = cache.intervals.entry(key) {
entry.insert(resolve_cron_intervals(client, &repo, &workflows).await);
}
Comment on lines +303 to +313

monitored.push(repo);
}

monitored
}

/// Publishes whether GitHub will actually run each workflow.
///
/// A workflow auto-disabled for inactivity has silently stopped running; the
/// `state` label lets an alert distinguish that from a deliberate manual
/// disable.
fn record_workflow_states(metrics: &Metrics, repo: &Repo, workflows: &[rest::Workflow]) {
for workflow in workflows {
metrics
.workflow_enabled
.get_or_create(&WorkflowEnabledLabels {
org: repo.owner.clone(),
repo: repo.name.clone(),
workflow: workflow.name.clone(),
state: workflow.state.as_str().to_owned(),
})
.set(i64::from(workflow.state == rest::WorkflowState::Active));
}
}

/// Resolves each workflow's expected run interval from its cron schedule.
async fn resolve_cron_intervals(
client: &Client,
Expand Down Expand Up @@ -360,18 +416,44 @@ fn record_activity(
}
}

fn record_runs(metrics: &Metrics, repo: &Repo, runs: &rest::RepoRuns, cache: &WorkflowCache) {
fn record_runs(
metrics: &Metrics,
repo: &Repo,
runs: &rest::RepoRuns,
cache: &WorkflowCache,
now: DateTime<Utc>,
) {
let intervals = cache.intervals.get(&repo.full_name());

for run in &runs.latest {
// A run older than the staleness horizon says nothing about the
// current code. Reporting `stale` instead of its original conclusion
// keeps an ancient failure from producing an alert that cannot be
// cleared without an artificial push. `workflow_run_stale` carries the
// fact separately so it stays visible on the dashboard.
let stale = run.is_stale(now);
metrics
.workflow_run_stale
.get_or_create(&WorkflowLabels {
org: repo.owner.clone(),
repo: repo.name.clone(),
workflow: run.workflow.clone(),
})
.set(i64::from(stale));

let conclusion = if stale {
"stale"
} else {
run.conclusion.as_str()
};
metrics
.workflow_run_status
.get_or_create(&WorkflowStateLabels {
org: repo.owner.clone(),
repo: repo.name.clone(),
workflow: run.workflow.clone(),
event: run.event.clone(),
conclusion: run.conclusion.as_str().to_owned(),
conclusion: conclusion.to_owned(),
})
.set(1);

Expand Down
Loading