diff --git a/docs/index.md b/docs/index.md index 2bbe623..71deeae 100644 --- a/docs/index.md +++ b/docs/index.md @@ -62,7 +62,7 @@ steampipe plugin install github | Item | Description| |-------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Credentials | The GitHub plugin supports personal access tokens, fine-grained personal access tokens, and GitHub App installation tokens to authenticate with the GitHub APIs. -| Permissions | You can authenticate using one of the following token types:
    - [Personal access token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) and assign the following scopes:
        - `repo` (all)
        - `read:org`
        - `gist`
        - `read:user`
        - `user:email`
    - [GitHub App installation token](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/authenticating-with-a-github-app-on-behalf-of-a-user#identifying-and-authorizing-users-for-github-apps)
        - [Permissions required for GitHub Apps](https://docs.github.com/en/rest/authentication/permissions-required-for-github-apps?apiVersion=2022-11-28)
    - [Fine-grained personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token)
        - Each table's documentation outlines the permissions needed to query it. +| Permissions | You can authenticate using one of the following token types:
    - [Personal access token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) and assign the following scopes:
        - `gist`
        - `read:org`
        - `read:project`
        - `read:user`
        - `repo` (all)
        - `user:email`
    - [GitHub App installation token](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/authenticating-with-a-github-app-on-behalf-of-a-user#identifying-and-authorizing-users-for-github-apps)
        - [Permissions required for GitHub Apps](https://docs.github.com/en/rest/authentication/permissions-required-for-github-apps?apiVersion=2022-11-28)
    - [Fine-grained personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token)
        - Each table's documentation outlines the permissions needed to query it. | Radius | The GitHub plugin query scope is generally the same as the GitHub API. Usually, this means you can list private resources that you have access to, as well as public resources that you own, or that are owned by organizations to which you belong. The same GitHub APIs are used to get information for public resources, but the public items are returned in list calls (because there would be too many). This has an interesting side effect in Steampipe in that you can sometimes query _a specific item_ by _a specific key column or columns_ that does not show up in a list query.

For example, `select * from github_my_organization` will list details about all the GitHub Organizations to which you belong. `select * from github_organization where login = 'postgres'` will show you the publicly available details about the `postgres` organization, which didn't show up in your first query! It works this way in Steampipe because [that's how it works in the API](https://docs.github.com/en/rest/reference/orgs#list-organizations-for-a-user). While this may seem counter-intuitive at first, it actually can be quite useful. | | Resolution | 1. Credentials in the Steampipe configuration file (`~/.steampipe/config/github.spc`)
2. Credentials specified in environment variables, e.g., `GITHUB_TOKEN`. diff --git a/docs/tables/github_project_v2_item.md b/docs/tables/github_project_v2_item.md new file mode 100644 index 0000000..820036f --- /dev/null +++ b/docs/tables/github_project_v2_item.md @@ -0,0 +1,458 @@ +--- +title: "Steampipe Table: github_project_v2_item - Query GitHub Project (V2) items using SQL" +description: "Allows users to query the items in GitHub Projects (V2), including issues, pull requests and draft issues, with their project field values such as Status." +folder: "Project" +--- + +# Table: github_project_v2_item - Query GitHub Project (V2) items using SQL + +GitHub Projects (V2) track work as items. Each item is an issue, a pull request or a draft issue (a note that only exists in the project), along with the values of the project's fields, such as Status or Iteration. + +## Table Usage Guide + +The `github_project_v2_item` table provides insights into the items in a GitHub Project (V2) owned by an organization or user. As a project manager or developer, explore what's on a project board through this table, including each item's type, status and other custom field values, and the details of the underlying issue, pull request or draft issue. Utilize it to report on project progress, such as which items are in progress, which issues are still open, or what's planned for the current iteration. + +To query this table using a [fine-grained access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token), the following permissions are required: + - Organization permissions: + - Projects (Read-only): Required to access all columns. + +**Important Notes** +- You must specify the `project_number` column, and either the `organization` or the `user_login` column, in a `where` or `join` clause to query the table. +- Each row has a `type` of `ISSUE`, `PULL_REQUEST`, `DRAFT_ISSUE` or `REDACTED`. Columns that don't apply to an item's type are null, e.g. draft issues have no `repository_full_name` or `number`. `REDACTED` items are items the token can't access, and only have the item columns (`id`, `type`, `created_at`, etc.) set. +- The `field_values` column contains the values of the project's custom fields (Text, Number, Date, Single select, Multi select and Iteration), keyed by field name. Built-in fields, such as Title, Assignees, Labels and Milestone, are available as their own columns. +- Archived items are included. Use `is_archived = false` to only list the items shown on the project board. +- This table supports optional quals. Queries with optional quals are optimised to use GitHub's project item filters. Optional quals are supported for the following columns: + - `type` + - `is_archived` + - `repository_full_name` + +## Examples + +### List the items in a project +Explore the issues, pull requests and draft issues in a project, along with their status, to get an overview of the work being tracked. + +```sql+postgres +select + type, + number, + title, + state, + field_values ->> 'Status' as status, + url +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1; +``` + +```sql+sqlite +select + type, + number, + title, + state, + json_extract(field_values, '$.Status') as status, + url +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1; +``` + +### List the items with a specific status +Identify the items that are currently in progress, to see what the team is working on. + +```sql+postgres +select + type, + number, + title, + repository_full_name, + assignees +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1 + and field_values ->> 'Status' = 'In Progress'; +``` + +```sql+sqlite +select + type, + number, + title, + repository_full_name, + assignees +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1 + and json_extract(field_values, '$.Status') = 'In Progress'; +``` + +### Count the items in each status +Summarize the number of items in each status, to understand the overall progress of a project. + +```sql+postgres +select + field_values ->> 'Status' as status, + count(*) as items +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1 + and not is_archived +group by + status +order by + items desc; +``` + +```sql+sqlite +select + json_extract(field_values, '$.Status') as status, + count(*) as items +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1 + and is_archived = 0 +group by + status +order by + items desc; +``` + +### List the open issues in a project +Explore the open issues in a project, along with their labels and assignees, to help with triage. + +```sql+postgres +select + number, + title, + repository_full_name, + labels, + assignees_total_count, + field_values ->> 'Status' as status +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1 + and type = 'ISSUE' + and state = 'OPEN'; +``` + +```sql+sqlite +select + number, + title, + repository_full_name, + labels, + assignees_total_count, + json_extract(field_values, '$.Status') as status +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1 + and type = 'ISSUE' + and state = 'OPEN'; +``` + +### List the draft issues in a project +Discover the draft issues in a project, which haven't been converted to issues yet, along with who created them. + +```sql+postgres +select + title, + author ->> 'login' as author, + content_created_at, + field_values ->> 'Status' as status +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1 + and type = 'DRAFT_ISSUE'; +``` + +```sql+sqlite +select + title, + json_extract(author, '$.login') as author, + content_created_at, + json_extract(field_values, '$.Status') as status +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1 + and type = 'DRAFT_ISSUE'; +``` + +### List the items from a repository +Explore the items in a project that belong to a specific repository. + +```sql+postgres +select + type, + number, + title, + state, + field_values ->> 'Status' as status +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1 + and repository_full_name = 'turbot/steampipe'; +``` + +```sql+sqlite +select + type, + number, + title, + state, + json_extract(field_values, '$.Status') as status +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1 + and repository_full_name = 'turbot/steampipe'; +``` + +### List the items in the current iteration +Identify the items planned for the current iteration, based on an iteration field named `Iteration`. + +```sql+postgres +select + type, + number, + title, + field_values -> 'Iteration' ->> 'title' as iteration, + field_values ->> 'Status' as status +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1 + and (field_values -> 'Iteration' ->> 'start_date')::date <= current_date + and (field_values -> 'Iteration' ->> 'start_date')::date + (field_values -> 'Iteration' ->> 'duration')::int > current_date; +``` + +```sql+sqlite +select + type, + number, + title, + json_extract(field_values, '$.Iteration.title') as iteration, + json_extract(field_values, '$.Status') as status +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1 + and date(json_extract(field_values, '$.Iteration.start_date')) <= date('now') + and date(json_extract(field_values, '$.Iteration.start_date'), '+' || json_extract(field_values, '$.Iteration.duration') || ' days') > date('now'); +``` + +### List the archived items in a project +Discover the items that have been archived from a project. + +```sql+postgres +select + type, + number, + title, + updated_at +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1 + and is_archived; +``` + +```sql+sqlite +select + type, + number, + title, + updated_at +from + github_project_v2_item +where + organization = 'turbot' + and project_number = 1 + and is_archived = 1; +``` + +### List the in progress items across all open projects in an organization +Explore what's in progress across every open project in an organization. The `materialized` keyword makes Postgres list the projects first, then the items in each project. Specify the organization on both tables, since it isn't passed from the `projects` query to the items. + +```sql+postgres +with projects as materialized ( + select + number, + title + from + github_project_v2 + where + organization = 'turbot' + and state = 'open' +) +select + p.number as project_number, + p.title as project_title, + i.type, + i.title, + i.url +from + projects p + join github_project_v2_item i on i.project_number = p.number +where + i.organization = 'turbot' + and i.field_values ->> 'Status' = 'In Progress'; +``` + +```sql+sqlite +select + p.number as project_number, + p.title as project_title, + i.type, + i.title, + i.url +from + github_project_v2 p + join github_project_v2_item i on i.project_number = p.number +where + p.organization = 'turbot' + and p.state = 'open' + and i.organization = 'turbot' + and json_extract(i.field_values, '$.Status') = 'In Progress'; +``` + +### Get the state reason of the closed issues in a project +Explore why each closed issue in a project was closed, by joining with the `github_issue` table for details that aren't in this table. The `materialized` keyword makes Postgres list the project items first, then get each issue. + +```sql+postgres +with items as materialized ( + select + repository_full_name, + number, + title, + field_values + from + github_project_v2_item + where + organization = 'turbot' + and project_number = 1 + and type = 'ISSUE' + and state = 'CLOSED' +) +select + i.number, + i.title, + i.field_values ->> 'Status' as status, + gi.state_reason +from + items i + join github_issue gi on gi.repository_full_name = i.repository_full_name and gi.number = i.number; +``` + +```sql+sqlite +select + i.number, + i.title, + json_extract(i.field_values, '$.Status') as status, + gi.state_reason +from + github_project_v2_item i + join github_issue gi on gi.repository_full_name = i.repository_full_name and gi.number = i.number +where + i.organization = 'turbot' + and i.project_number = 1 + and i.type = 'ISSUE' + and i.state = 'CLOSED'; +``` + +### Get the review status of the open pull requests in a project +Explore the review decision of each open pull request in a project, by joining with the `github_pull_request` table. + +```sql+postgres +with items as materialized ( + select + repository_full_name, + number, + title + from + github_project_v2_item + where + organization = 'turbot' + and project_number = 1 + and type = 'PULL_REQUEST' + and state = 'OPEN' +) +select + i.number, + i.title, + pr.review_decision, + pr.is_draft +from + items i + join github_pull_request pr on pr.repository_full_name = i.repository_full_name and pr.number = i.number; +``` + +```sql+sqlite +select + i.number, + i.title, + pr.review_decision, + pr.is_draft +from + github_project_v2_item i + join github_pull_request pr on pr.repository_full_name = i.repository_full_name and pr.number = i.number +where + i.organization = 'turbot' + and i.project_number = 1 + and i.type = 'PULL_REQUEST' + and i.state = 'OPEN'; +``` + +### List the items in a project owned by a user +Explore the items in a project owned by a specific GitHub user, e.g. one of your own personal projects. + +```sql+postgres +select + type, + number, + title, + field_values ->> 'Status' as status +from + github_project_v2_item +where + user_login = 'octocat' + and project_number = 1; +``` + +```sql+sqlite +select + type, + number, + title, + json_extract(field_values, '$.Status') as status +from + github_project_v2_item +where + user_login = 'octocat' + and project_number = 1; +``` diff --git a/github/issue_pr_utils.go b/github/issue_pr_utils.go index 6b97a20..c30065d 100644 --- a/github/issue_pr_utils.go +++ b/github/issue_pr_utils.go @@ -158,7 +158,6 @@ func appendIssueColumnIncludes(m *map[string]interface{}, cols []string) { (*m)["includeIssueNodeId"] = githubv4.Boolean(slices.Contains(cols, "node_id")) (*m)["includeIssueId"] = githubv4.Boolean(slices.Contains(cols, "id")) (*m)["includeIssueIsReadByUser"] = githubv4.Boolean(slices.Contains(cols, "is_read_by_user")) - (*m)["includeIssueProjectsV2"] = githubv4.Boolean(slices.Contains(cols, "projects_v2") || slices.Contains(cols, "projects_v2_total_count")) } func issueHydrateIsReadByUser(_ context.Context, _ *plugin.QueryData, h *plugin.HydrateData) (interface{}, error) { @@ -473,25 +472,6 @@ func issueHydrateLabels(_ context.Context, _ *plugin.QueryData, h *plugin.Hydrat return issue.Labels.Nodes, nil } -func issueHydrateProjectsV2(_ context.Context, _ *plugin.QueryData, h *plugin.HydrateData) (interface{}, error) { - issue, err := extractIssueFromHydrateItem(h) - if err != nil { - return nil, err - } - if len(issue.ProjectsV2.Nodes) == 0 { - return nil, nil - } - return issue.ProjectsV2.Nodes, nil -} - -func issueHydrateProjectsV2TotalCount(_ context.Context, _ *plugin.QueryData, h *plugin.HydrateData) (interface{}, error) { - issue, err := extractIssueFromHydrateItem(h) - if err != nil { - return nil, err - } - return issue.ProjectsV2.TotalCount, nil -} - func extractIssueCommentFromHydrateItem(h *plugin.HydrateData) (models.IssueComment, error) { if issueComment, ok := h.Item.(models.IssueComment); ok { return issueComment, nil diff --git a/github/models/issue.go b/github/models/issue.go index 1a4fc05..f7113ff 100644 --- a/github/models/issue.go +++ b/github/models/issue.go @@ -49,20 +49,6 @@ type Issue struct { TotalCount int Nodes []BaseUser } `graphql:"assignees(first: 10) @include(if:$includeIssueAssignees)" json:"assignees"` - ProjectsV2 struct { - TotalCount int - Nodes []struct { - Id string `graphql:"id: fullDatabaseId" json:"id"` - NodeId string `graphql:"nodeId: id" json:"node_id"` - Number int `json:"number"` - Title string `json:"title"` - Closed bool `json:"closed"` - ClosedAt NullableTime `json:"closed_at"` - CreatedAt NullableTime `json:"created_at"` - UpdatedAt NullableTime `json:"updated_at"` - Url string `json:"url"` - } - } `graphql:"projectsV2(first: 100) @include(if:$includeIssueProjectsV2)" json:"projects_v2"` } type IssueTemplate struct { diff --git a/github/models/project_v2_item.go b/github/models/project_v2_item.go new file mode 100644 index 0000000..ce3ce0a --- /dev/null +++ b/github/models/project_v2_item.go @@ -0,0 +1,153 @@ +package models + +import "github.com/shurcooL/githubv4" + +// ProjectV2ItemArchivedState isn't defined in the pinned githubv4 version. The +// type name must match the GraphQL enum name, since githubv4 uses it when +// declaring query variables. +type ProjectV2ItemArchivedState string + +const ( + ProjectV2ItemArchivedStateArchived ProjectV2ItemArchivedState = "ARCHIVED" + ProjectV2ItemArchivedStateNotArchived ProjectV2ItemArchivedState = "NOT_ARCHIVED" +) + +type ProjectV2Item struct { + Id string `graphql:"id: fullDatabaseId" json:"id"` + NodeId string `graphql:"nodeId: id" json:"node_id"` + Type githubv4.ProjectV2ItemType `json:"type"` + IsArchived bool `json:"is_archived"` + CreatedAt NullableTime `json:"created_at"` + UpdatedAt NullableTime `json:"updated_at"` + Creator Actor `graphql:"creator @include(if:$includeItemCreator)" json:"creator"` + Content ProjectV2ItemContent `graphql:"content @include(if:$includeItemContent)" json:"content"` + FieldValues struct { + Nodes []ProjectV2ItemFieldValue + } `graphql:"fieldValues(first: 50) @include(if:$includeItemFieldValues)" json:"field_values"` +} + +// ProjectV2ItemContent holds the item's content, one struct per possible type. +// The GraphQL client fills every fragment that shares a JSON key with the +// response, so read the fragment that matches the item's type. +type ProjectV2ItemContent struct { + Issue struct { + Id int `graphql:"id: databaseId"` + NodeId string `graphql:"nodeId: id"` + Number int + Title string + Body string `graphql:"body @include(if:$includeItemBody)"` + Url string + State githubv4.IssueState `graphql:"issueState: state"` + Closed bool + ClosedAt NullableTime + CreatedAt NullableTime + UpdatedAt NullableTime + Author Actor `graphql:"author @include(if:$includeItemAuthor)"` + Repository struct { + NameWithOwner string + } + Assignees ProjectV2ItemAssignees `graphql:"assignees(first: 10) @include(if:$includeItemAssignees)"` + Labels ProjectV2ItemLabels `graphql:"labels(first: 100) @include(if:$includeItemLabels)"` + Milestone Milestone `graphql:"milestone @include(if:$includeItemMilestone)"` + } `graphql:"... on Issue"` + PullRequest struct { + Id int `graphql:"id: databaseId"` + NodeId string `graphql:"nodeId: id"` + Number int + Title string + Body string `graphql:"body @include(if:$includeItemBody)"` + Url string + State githubv4.PullRequestState `graphql:"pullRequestState: state"` + Closed bool + ClosedAt NullableTime + CreatedAt NullableTime + UpdatedAt NullableTime + Author Actor `graphql:"author @include(if:$includeItemAuthor)"` + Repository struct { + NameWithOwner string + } + Assignees ProjectV2ItemAssignees `graphql:"assignees(first: 10) @include(if:$includeItemAssignees)"` + Labels ProjectV2ItemLabels `graphql:"labels(first: 100) @include(if:$includeItemLabels)"` + Milestone Milestone `graphql:"milestone @include(if:$includeItemMilestone)"` + } `graphql:"... on PullRequest"` + DraftIssue struct { + NodeId string `graphql:"nodeId: id"` + Title string + Body string `graphql:"body @include(if:$includeItemBody)"` + CreatedAt NullableTime + UpdatedAt NullableTime + Creator Actor `graphql:"creator @include(if:$includeItemAuthor)"` + Assignees ProjectV2ItemAssignees `graphql:"assignees(first: 10) @include(if:$includeItemAssignees)"` + } `graphql:"... on DraftIssue"` +} + +type ProjectV2ItemAssignees struct { + TotalCount int + Nodes []BasicUser +} + +type ProjectV2ItemLabels struct { + Nodes []Label +} + +// ProjectV2ItemFieldValue only selects the value types of custom fields. Built-in +// fields such as Assignees and Labels come back with just their type name. +type ProjectV2ItemFieldValue struct { + TypeName string `graphql:"typeName: __typename"` + Text struct { + Field ProjectV2FieldRef + Text string + } `graphql:"... on ProjectV2ItemFieldTextValue"` + Number struct { + Field ProjectV2FieldRef + Number float64 + } `graphql:"... on ProjectV2ItemFieldNumberValue"` + Date struct { + Field ProjectV2FieldRef + Date string + } `graphql:"... on ProjectV2ItemFieldDateValue"` + SingleSelect struct { + Field ProjectV2FieldRef + Name string + } `graphql:"... on ProjectV2ItemFieldSingleSelectValue"` + MultiSelect struct { + Field ProjectV2FieldRef + Options []struct { + Name string + } + } `graphql:"... on ProjectV2ItemFieldMultiSelectValue"` + Iteration struct { + Field ProjectV2FieldRef + Title string + StartDate string + Duration int + IterationId string + } `graphql:"... on ProjectV2ItemFieldIterationValue"` + IssueField struct { + Field ProjectV2FieldRef + IssueFieldValue struct { + Text struct { + StringValue string `graphql:"stringValue: value"` + } `graphql:"... on IssueFieldTextValue"` + Date struct { + StringValue string `graphql:"stringValue: value"` + } `graphql:"... on IssueFieldDateValue"` + SingleSelect struct { + StringValue string `graphql:"stringValue: value"` + } `graphql:"... on IssueFieldSingleSelectValue"` + MultiSelect struct { + StringValue string `graphql:"stringValue: value"` + } `graphql:"... on IssueFieldMultiSelectValue"` + Number struct { + NumberValue *float64 `graphql:"numberValue: value"` + } `graphql:"... on IssueFieldNumberValue"` + } + } `graphql:"... on ProjectV2ItemIssueFieldValue"` +} + +type ProjectV2FieldRef struct { + Common struct { + Name string + DataType string + } `graphql:"... on ProjectV2FieldCommon"` +} diff --git a/github/plugin.go b/github/plugin.go index 11d518c..9f0d6b7 100644 --- a/github/plugin.go +++ b/github/plugin.go @@ -59,6 +59,7 @@ func Plugin(ctx context.Context) *plugin.Plugin { "github_package": tableGitHubPackage(), "github_package_version": tableGitHubPackageVersion(), "github_project_v2": tableGitHubProjectV2(), + "github_project_v2_item": tableGitHubProjectV2Item(), "github_pull_request": tableGitHubPullRequest(), "github_pull_request_comment": tableGitHubPullRequestComment(), "github_pull_request_review": tableGitHubPullRequestReview(), diff --git a/github/project_v2_item_utils.go b/github/project_v2_item_utils.go new file mode 100644 index 0000000..ec84fce --- /dev/null +++ b/github/project_v2_item_utils.go @@ -0,0 +1,227 @@ +package github + +import ( + "slices" + "strings" + + "github.com/shurcooL/githubv4" + "github.com/turbot/steampipe-plugin-github/github/models" + "github.com/turbot/steampipe-plugin-sdk/v6/plugin" +) + +// projectV2ItemRow flattens a project item and its content into one row. Content +// columns that don't apply to the item's type are left nil. +type projectV2ItemRow struct { + Id string + NodeId string + Type githubv4.ProjectV2ItemType + IsArchived bool + CreatedAt models.NullableTime + UpdatedAt models.NullableTime + Creator models.Actor + FieldValues map[string]interface{} + ContentId *int + ContentNodeId string + RepositoryFullName string + Number *int + Title string + Body *string + Url string + State string + Closed *bool + ClosedAt models.NullableTime + Author models.Actor + Assignees []models.BasicUser + AssigneesTotalCount *int + Labels map[string]bool + Milestone models.Milestone + ContentCreatedAt models.NullableTime + ContentUpdatedAt models.NullableTime +} + +var projectV2ItemContentColumns = []string{ + "content_id", "content_node_id", "repository_full_name", "number", "title", "body", "url", "state", "closed", + "closed_at", "author", "assignees", "assignees_total_count", "labels", "milestone", "content_created_at", "content_updated_at", +} + +func appendProjectV2ItemColumnIncludes(m *map[string]interface{}, cols []string) { + includeContent := slices.ContainsFunc(cols, func(col string) bool { + return slices.Contains(projectV2ItemContentColumns, col) + }) + (*m)["includeItemCreator"] = githubv4.Boolean(slices.Contains(cols, "creator")) + (*m)["includeItemFieldValues"] = githubv4.Boolean(slices.Contains(cols, "field_values")) + (*m)["includeItemContent"] = githubv4.Boolean(includeContent) + (*m)["includeItemBody"] = githubv4.Boolean(slices.Contains(cols, "body")) + (*m)["includeItemAuthor"] = githubv4.Boolean(slices.Contains(cols, "author")) + (*m)["includeItemAssignees"] = githubv4.Boolean(slices.Contains(cols, "assignees") || slices.Contains(cols, "assignees_total_count")) + (*m)["includeItemLabels"] = githubv4.Boolean(slices.Contains(cols, "labels")) + (*m)["includeItemMilestone"] = githubv4.Boolean(slices.Contains(cols, "milestone")) +} + +// projectV2ItemArchivedStates returns both states unless the query filters on +// is_archived, since the API only returns unarchived items by default. +func projectV2ItemArchivedStates(d *plugin.QueryData) []models.ProjectV2ItemArchivedState { + states := []models.ProjectV2ItemArchivedState{ + models.ProjectV2ItemArchivedStateArchived, + models.ProjectV2ItemArchivedStateNotArchived, + } + if d.Quals["is_archived"] == nil || len(d.Quals["is_archived"].Quals) == 0 { + return states + } + // Postgres still applies every qual to the results, so the first one is + // enough to narrow the request. + q := d.Quals["is_archived"].Quals[0] + archived := q.Value.GetBoolValue() + if q.Operator == "<>" { + archived = !archived + } + if archived { + return states[:1] + } + return states[1:] +} + +// projectV2ItemQuery builds the items filter for the type and +// repository_full_name quals. REDACTED items have no filter, so they're +// filtered after listing. +func projectV2ItemQuery(d *plugin.QueryData) string { + var filters []string + switch d.EqualsQualString("type") { + case string(githubv4.ProjectV2ItemTypeIssue): + // is:issue also matches draft issues + filters = append(filters, "is:issue", "-is:draft") + case string(githubv4.ProjectV2ItemTypePullRequest): + filters = append(filters, "is:pr") + case string(githubv4.ProjectV2ItemTypeDraftIssue): + filters = append(filters, "is:draft") + } + if repo := d.EqualsQualString("repository_full_name"); repo != "" { + filters = append(filters, "repo:"+repo) + } + return strings.Join(filters, " ") +} + +func newProjectV2ItemRow(item models.ProjectV2Item) projectV2ItemRow { + row := projectV2ItemRow{ + Id: item.Id, + NodeId: item.NodeId, + Type: item.Type, + IsArchived: item.IsArchived, + CreatedAt: item.CreatedAt, + UpdatedAt: item.UpdatedAt, + Creator: item.Creator, + FieldValues: projectV2ItemFieldValues(item.FieldValues.Nodes), + } + + switch item.Type { + case githubv4.ProjectV2ItemTypeIssue: + c := item.Content.Issue + row.ContentId = &c.Id + row.ContentNodeId = c.NodeId + row.RepositoryFullName = c.Repository.NameWithOwner + row.Number = &c.Number + row.Title = c.Title + row.Body = &c.Body + row.Url = c.Url + row.State = string(c.State) + row.Closed = &c.Closed + row.ClosedAt = c.ClosedAt + row.Author = c.Author + row.Assignees = c.Assignees.Nodes + row.AssigneesTotalCount = &c.Assignees.TotalCount + row.Labels = projectV2ItemLabels(c.Labels) + row.Milestone = c.Milestone + row.ContentCreatedAt = c.CreatedAt + row.ContentUpdatedAt = c.UpdatedAt + case githubv4.ProjectV2ItemTypePullRequest: + c := item.Content.PullRequest + row.ContentId = &c.Id + row.ContentNodeId = c.NodeId + row.RepositoryFullName = c.Repository.NameWithOwner + row.Number = &c.Number + row.Title = c.Title + row.Body = &c.Body + row.Url = c.Url + row.State = string(c.State) + row.Closed = &c.Closed + row.ClosedAt = c.ClosedAt + row.Author = c.Author + row.Assignees = c.Assignees.Nodes + row.AssigneesTotalCount = &c.Assignees.TotalCount + row.Labels = projectV2ItemLabels(c.Labels) + row.Milestone = c.Milestone + row.ContentCreatedAt = c.CreatedAt + row.ContentUpdatedAt = c.UpdatedAt + case githubv4.ProjectV2ItemTypeDraftIssue: + c := item.Content.DraftIssue + row.ContentNodeId = c.NodeId + row.Title = c.Title + row.Body = &c.Body + row.Author = c.Creator + row.Assignees = c.Assignees.Nodes + row.AssigneesTotalCount = &c.Assignees.TotalCount + row.ContentCreatedAt = c.CreatedAt + row.ContentUpdatedAt = c.UpdatedAt + } + + return row +} + +func projectV2ItemLabels(labels models.ProjectV2ItemLabels) map[string]bool { + m := make(map[string]bool, len(labels.Nodes)) + for _, l := range labels.Nodes { + m[l.Name] = true + } + return m +} + +// projectV2ItemFieldValues maps custom field names to their values. Built-in +// fields (Title, Assignees, Labels, etc.) are skipped since they have their own +// columns. +func projectV2ItemFieldValues(nodes []models.ProjectV2ItemFieldValue) map[string]interface{} { + values := map[string]interface{}{} + for _, v := range nodes { + switch v.TypeName { + case "ProjectV2ItemFieldTextValue": + if v.Text.Field.Common.DataType == "TITLE" { + continue + } + values[v.Text.Field.Common.Name] = v.Text.Text + case "ProjectV2ItemFieldNumberValue": + values[v.Number.Field.Common.Name] = v.Number.Number + case "ProjectV2ItemFieldDateValue": + values[v.Date.Field.Common.Name] = v.Date.Date + case "ProjectV2ItemFieldSingleSelectValue": + values[v.SingleSelect.Field.Common.Name] = v.SingleSelect.Name + case "ProjectV2ItemFieldMultiSelectValue": + options := make([]string, 0, len(v.MultiSelect.Options)) + for _, o := range v.MultiSelect.Options { + options = append(options, o.Name) + } + values[v.MultiSelect.Field.Common.Name] = options + case "ProjectV2ItemFieldIterationValue": + values[v.Iteration.Field.Common.Name] = map[string]interface{}{ + "title": v.Iteration.Title, + "start_date": v.Iteration.StartDate, + "duration": v.Iteration.Duration, + "iteration_id": v.Iteration.IterationId, + } + case "ProjectV2ItemIssueFieldValue": + ifv := v.IssueField.IssueFieldValue + if ifv.Number.NumberValue != nil { + values[v.IssueField.Field.Common.Name] = *ifv.Number.NumberValue + continue + } + for _, s := range []string{ifv.Text.StringValue, ifv.Date.StringValue, ifv.SingleSelect.StringValue, ifv.MultiSelect.StringValue} { + if s != "" { + values[v.IssueField.Field.Common.Name] = s + break + } + } + } + } + if len(values) == 0 { + return nil + } + return values +} diff --git a/github/table_github_issue.go b/github/table_github_issue.go index b520c12..62a4238 100644 --- a/github/table_github_issue.go +++ b/github/table_github_issue.go @@ -64,8 +64,6 @@ func sharedIssueColumns() []*plugin.Column { {Name: "user_did_author", Type: proto.ColumnType_BOOL, Hydrate: issueHydrateUserDidAuthor, Transform: transform.FromValue(), Description: "If true, user authored the issue."}, {Name: "user_subscription", Type: proto.ColumnType_STRING, Hydrate: issueHydrateUserSubscription, Transform: transform.FromValue(), Description: "Subscription state of the user to the issue."}, {Name: "assignees", Type: proto.ColumnType_JSON, Hydrate: issueHydrateAssignees, Transform: transform.FromValue().NullIfZero(), Description: "A list of Users assigned to the issue."}, - {Name: "projects_v2", Type: proto.ColumnType_JSON, Hydrate: issueHydrateProjectsV2, Transform: transform.FromValue().NullIfZero(), Description: "A list of ProjectV2 projects the issue belongs to, with scalar project fields such as id, number, title, closed and url."}, - {Name: "projects_v2_total_count", Type: proto.ColumnType_INT, Hydrate: issueHydrateProjectsV2TotalCount, Transform: transform.FromValue(), Description: "Count of ProjectsV2 the issue is linked to."}, } } @@ -218,6 +216,7 @@ func tableGitHubRepositoryIssueGet(ctx context.Context, d *plugin.QueryData, _ * "issueNumber": githubv4.Int(issueNumber), } appendIssueColumnIncludes(&variables, d.QueryContext.Columns) + appendUserInteractionAbilityForIssue(&variables, d.QueryContext.Columns, d) err := client.Query(ctx, &query, variables) plugin.Logger(ctx).Debug(rateLimitLogString("github_issue", &query.RateLimit)) diff --git a/github/table_github_project_v2_item.go b/github/table_github_project_v2_item.go new file mode 100644 index 0000000..003a1da --- /dev/null +++ b/github/table_github_project_v2_item.go @@ -0,0 +1,174 @@ +package github + +import ( + "context" + "strings" + + "github.com/shurcooL/githubv4" + "github.com/turbot/steampipe-plugin-github/github/models" + + "github.com/turbot/steampipe-plugin-sdk/v6/grpc/proto" + "github.com/turbot/steampipe-plugin-sdk/v6/plugin" + "github.com/turbot/steampipe-plugin-sdk/v6/plugin/transform" +) + +func gitHubProjectV2ItemColumns() []*plugin.Column { + return []*plugin.Column{ + {Name: "organization", Type: proto.ColumnType_STRING, Transform: transform.FromQual("organization"), Description: "The organization that owns the project. Either this or 'user_login' must be provided."}, + {Name: "user_login", Type: proto.ColumnType_STRING, Transform: transform.FromQual("user_login"), Description: "The user that owns the project. Either this or 'organization' must be provided."}, + {Name: "project_number", Type: proto.ColumnType_INT, Transform: transform.FromQual("project_number"), Description: "The number of the project the item belongs to."}, + {Name: "id", Type: proto.ColumnType_INT, Transform: transform.FromField("Id"), Description: "The ID of the project item."}, + {Name: "node_id", Type: proto.ColumnType_STRING, Transform: transform.FromField("NodeId"), Description: "The node ID of the project item."}, + {Name: "type", Type: proto.ColumnType_STRING, Transform: transform.FromField("Type"), Description: "The type of the item: ISSUE, PULL_REQUEST, DRAFT_ISSUE or REDACTED. REDACTED items are items whose content the token can't access, and have no content columns set."}, + {Name: "is_archived", Type: proto.ColumnType_BOOL, Transform: transform.FromField("IsArchived"), Description: "If true, the item is archived."}, + {Name: "created_at", Type: proto.ColumnType_TIMESTAMP, Transform: transform.FromField("CreatedAt").NullIfZero().Transform(convertTimestamp), Description: "The time when the item was added to the project."}, + {Name: "updated_at", Type: proto.ColumnType_TIMESTAMP, Transform: transform.FromField("UpdatedAt").NullIfZero().Transform(convertTimestamp), Description: "The time when the item was last updated."}, + {Name: "creator", Type: proto.ColumnType_JSON, Transform: transform.FromField("Creator").NullIfZero(), Description: "The actor who added the item to the project."}, + {Name: "field_values", Type: proto.ColumnType_JSON, Transform: transform.FromField("FieldValues"), Description: "The item's values for the project's custom fields (e.g. Status, Iteration), keyed by field name (first 50 fields). Built-in fields such as Title, Assignees and Labels are available as their own columns."}, + {Name: "content_id", Type: proto.ColumnType_INT, Transform: transform.FromField("ContentId"), Description: "The ID of the issue or pull request. Null for draft issues."}, + {Name: "content_node_id", Type: proto.ColumnType_STRING, Transform: transform.FromField("ContentNodeId").NullIfZero(), Description: "The node ID of the issue, pull request or draft issue."}, + {Name: "repository_full_name", Type: proto.ColumnType_STRING, Transform: transform.FromField("RepositoryFullName").NullIfZero(), Description: "The full name of the repository of the issue or pull request."}, + {Name: "number", Type: proto.ColumnType_INT, Transform: transform.FromField("Number"), Description: "The number of the issue or pull request."}, + {Name: "title", Type: proto.ColumnType_STRING, Transform: transform.FromField("Title").NullIfZero(), Description: "The title of the issue, pull request or draft issue."}, + {Name: "body", Type: proto.ColumnType_STRING, Transform: transform.FromField("Body"), Description: "The body of the issue, pull request or draft issue."}, + {Name: "url", Type: proto.ColumnType_STRING, Transform: transform.FromField("Url").NullIfZero(), Description: "The URL of the issue or pull request."}, + {Name: "state", Type: proto.ColumnType_STRING, Transform: transform.FromField("State").NullIfZero(), Description: "The state of the issue (OPEN, CLOSED) or pull request (OPEN, CLOSED, MERGED)."}, + {Name: "closed", Type: proto.ColumnType_BOOL, Transform: transform.FromField("Closed"), Description: "If true, the issue or pull request is closed."}, + {Name: "closed_at", Type: proto.ColumnType_TIMESTAMP, Transform: transform.FromField("ClosedAt").NullIfZero().Transform(convertTimestamp), Description: "The time when the issue or pull request was closed."}, + {Name: "author", Type: proto.ColumnType_JSON, Transform: transform.FromField("Author").NullIfZero(), Description: "The actor who authored the issue or pull request, or created the draft issue."}, + {Name: "assignees", Type: proto.ColumnType_JSON, Transform: transform.FromField("Assignees"), Description: "The users assigned to the issue, pull request or draft issue (first 10)."}, + {Name: "assignees_total_count", Type: proto.ColumnType_INT, Transform: transform.FromField("AssigneesTotalCount"), Description: "The number of users assigned to the issue, pull request or draft issue."}, + {Name: "labels", Type: proto.ColumnType_JSON, Transform: transform.FromField("Labels"), Description: "A map of the labels on the issue or pull request (first 100)."}, + {Name: "milestone", Type: proto.ColumnType_JSON, Transform: transform.FromField("Milestone").NullIfZero(), Description: "The milestone of the issue or pull request."}, + {Name: "content_created_at", Type: proto.ColumnType_TIMESTAMP, Transform: transform.FromField("ContentCreatedAt").NullIfZero().Transform(convertTimestamp), Description: "The time when the issue, pull request or draft issue was created."}, + {Name: "content_updated_at", Type: proto.ColumnType_TIMESTAMP, Transform: transform.FromField("ContentUpdatedAt").NullIfZero().Transform(convertTimestamp), Description: "The time when the issue, pull request or draft issue was last updated."}, + } +} + +func tableGitHubProjectV2Item() *plugin.Table { + return &plugin.Table{ + Name: "github_project_v2_item", + Description: "Items (issues, pull requests and draft issues) in a GitHub Project (V2).", + List: &plugin.ListConfig{ + KeyColumns: []*plugin.KeyColumn{ + {Name: "organization", Require: plugin.AnyOf}, + {Name: "user_login", Require: plugin.AnyOf}, + {Name: "project_number", Require: plugin.Required}, + {Name: "type", Require: plugin.Optional}, + {Name: "is_archived", Require: plugin.Optional, Operators: []string{"=", "<>"}}, + {Name: "repository_full_name", Require: plugin.Optional}, + }, + Hydrate: tableGitHubProjectV2ItemList, + }, + Columns: commonColumns(gitHubProjectV2ItemColumns()), + } +} + +// GitHub sometimes fails a page of items with a generic "Something went wrong" +// error when content assignees and field values are fetched together, while +// pages of 10 of the same items succeed. +const projectV2ItemFallbackPageSize = 10 + +type projectV2ItemConnection struct { + PageInfo models.PageInfo + Nodes []models.ProjectV2Item +} + +func tableGitHubProjectV2ItemList(ctx context.Context, d *plugin.QueryData, _ *plugin.HydrateData) (interface{}, error) { + organization := d.EqualsQualString("organization") + user := d.EqualsQualString("user_login") + projectNumber := int(d.EqualsQuals["project_number"].GetInt64Value()) + + maxPageSize := adjustPageSize(100, d.QueryContext.Limit) + pageSize := maxPageSize + // The number of fallback-sized pages left before going back to maxPageSize + fallbackPagesLeft := 0 + variables := map[string]interface{}{ + "owner": githubv4.String(organization), + "number": githubv4.Int(projectNumber), + "pageSize": githubv4.Int(pageSize), + "cursor": (*githubv4.String)(nil), + "archivedStates": projectV2ItemArchivedStates(d), + "query": githubv4.String(projectV2ItemQuery(d)), + } + if organization == "" { + variables["owner"] = githubv4.String(user) + } + appendProjectV2ItemColumnIncludes(&variables, d.QueryContext.Columns) + + client := connectV4(ctx, d) + + for { + var items projectV2ItemConnection + var rateLimit models.RateLimit + var err error + + // The items connection is the same for both owner types, but the root + // query field differs. + if organization != "" { + var query struct { + RateLimit models.RateLimit + Organization struct { + ProjectV2 struct { + Items projectV2ItemConnection `graphql:"items(first: $pageSize, after: $cursor, archivedStates: $archivedStates, query: $query)"` + } `graphql:"projectV2(number: $number)"` + } `graphql:"organization(login: $owner)"` + } + err = client.Query(ctx, &query, variables) + items, rateLimit = query.Organization.ProjectV2.Items, query.RateLimit + } else { + var query struct { + RateLimit models.RateLimit + User struct { + ProjectV2 struct { + Items projectV2ItemConnection `graphql:"items(first: $pageSize, after: $cursor, archivedStates: $archivedStates, query: $query)"` + } `graphql:"projectV2(number: $number)"` + } `graphql:"user(login: $owner)"` + } + err = client.Query(ctx, &query, variables) + items, rateLimit = query.User.ProjectV2.Items, query.RateLimit + } + plugin.Logger(ctx).Debug(rateLimitLogString("github_project_v2_item", &rateLimit)) + if err != nil { + // Fetch the items from the failed page in fallback-sized pages, then go + // back to the full page size. + if strings.Contains(err.Error(), "Something went wrong while executing your query") && pageSize > projectV2ItemFallbackPageSize { + fallbackPagesLeft = (pageSize + projectV2ItemFallbackPageSize - 1) / projectV2ItemFallbackPageSize + pageSize = projectV2ItemFallbackPageSize + variables["pageSize"] = githubv4.Int(pageSize) + plugin.Logger(ctx).Warn("github_project_v2_item", "retrying_with_page_size", pageSize, "api_error", err) + continue + } + plugin.Logger(ctx).Error("github_project_v2_item", "api_error", err) + if strings.Contains(err.Error(), "Could not resolve to a ProjectV2") || + strings.Contains(err.Error(), "Could not resolve to an Organization with the login of") || + strings.Contains(err.Error(), "Could not resolve to a User with the login of") { + return nil, nil + } + return nil, err + } + + for _, item := range items.Nodes { + d.StreamListItem(ctx, newProjectV2ItemRow(item)) + + // Context can be cancelled due to manual cancellation or the limit has been hit + if d.RowsRemaining(ctx) == 0 { + return nil, nil + } + } + + if !items.PageInfo.HasNextPage { + break + } + variables["cursor"] = githubv4.NewString(items.PageInfo.EndCursor) + if fallbackPagesLeft > 0 { + fallbackPagesLeft-- + if fallbackPagesLeft == 0 { + pageSize = maxPageSize + variables["pageSize"] = githubv4.Int(pageSize) + } + } + } + + return nil, nil +} diff --git a/github/table_github_pull_request.go b/github/table_github_pull_request.go index 54ba0c8..d12992e 100644 --- a/github/table_github_pull_request.go +++ b/github/table_github_pull_request.go @@ -205,6 +205,7 @@ func tableGitHubPullRequestGet(ctx context.Context, d *plugin.QueryData, _ *plug "number": githubv4.Int(number), } appendPullRequestColumnIncludes(&variables, d.QueryContext.Columns) + appendUserInteractionAbilityForIssue(&variables, d.QueryContext.Columns, d) err := client.Query(ctx, &query, variables) plugin.Logger(ctx).Debug(rateLimitLogString("github_pull_request", &query.RateLimit))