Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions .eslintignore

This file was deleted.

81 changes: 0 additions & 81 deletions .github/linters/.eslintrc.yml

This file was deleted.

18 changes: 0 additions & 18 deletions .github/linters/.markdown-lint.yml

This file was deleted.

10 changes: 0 additions & 10 deletions .github/linters/.yaml-lint.yml

This file was deleted.

9 changes: 0 additions & 9 deletions .github/linters/tsconfig.json

This file was deleted.

14 changes: 7 additions & 7 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ permissions:

jobs:
test-typescript:
name: TypeScript Tests
name: TypeScript Checks
runs-on: ubuntu-latest

steps:
Expand All @@ -39,12 +39,12 @@ jobs:
run: npm ci
working-directory: eval

- name: Check Format
id: npm-format-check
run: npm run format:check
- name: Check
id: check
run: npm run check
working-directory: eval

- name: Lint
id: npm-lint
run: npm run lint
- name: Test
id: test
run: npm run test
working-directory: eval
20 changes: 4 additions & 16 deletions .github/workflows/linter.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,10 @@ on:

permissions:
contents: read
packages: read
statuses: write

jobs:
lint:
name: Lint Codebase
name: Vite+ Check
runs-on: ubuntu-latest

steps:
Expand All @@ -27,8 +25,6 @@ jobs:
- name: Checkout
id: checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
fetch-depth: 0

- name: Setup Node.js
id: setup-node
Expand All @@ -43,15 +39,7 @@ jobs:
run: npm ci
working-directory: eval

- name: Lint Codebase
id: super-linter
uses: super-linter/super-linter/slim@9e863354e3ff62e0727d37183162c4a88873df41 # v8.6.0
- name: Check
id: check
run: npm run check
working-directory: eval
env:
DEFAULT_BRANCH: main
FILTER_REGEX_EXCLUDE: dist/**/*
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
TYPESCRIPT_DEFAULT_STYLE: prettier
VALIDATE_ALL_CODEBASE: true
VALIDATE_JAVASCRIPT_STANDARD: false
VALIDATE_JSCPD: false
3 changes: 0 additions & 3 deletions .prettierignore

This file was deleted.

15 changes: 0 additions & 15 deletions .prettierrc.json

This file was deleted.

1 change: 1 addition & 0 deletions .tool-versions
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
nodejs 24.15.0
pnpm 10.33.0
39 changes: 30 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ simply include the following step in an action file:

```yaml
- name: Run Evals
uses: step-security/eval-action@v1
uses: step-security/eval-action@v2
with:
api_key: ${{ secrets.BRAINTRUST_API_KEY }}
runtime: node
Expand All @@ -20,12 +20,12 @@ You can configure the following variables:
- `api_key`: Your
[Braintrust API key](https://www.braintrust.dev/docs/api-reference/index#api-reference).
- `root`: The root directory containing your evals (defaults to `'.'`). The root
directory must either have `node` or `python` configured.
directory must have `node`, `python`, or `go` configured.
- `paths`: Specific paths, relative to the root, containing evals you'd like to
run.
- `runtime`: Either `node` or `python`
- `package_manager`: Either `npm`, `pnpm`, or `yarn` for a `node` runtime, or
`pip` or `uv` for a `python` runtime.
- `runtime`: Either `node`, `python`, or `go`
- `package_manager`: Either `npm` or `pnpm` for a `node` runtime, `pip` or `uv`
for a `python` runtime, or `go` for a `go` runtime. You can omit this for Go.
- `use_proxy`: Either `true` or `false`. If set, `OPENAI_BASE_URL` will be set
to `https://braintrustproxy.com/v1`, which will automatically cache repetitive
LLM calls and run your evals faster. Defaults to `true`.
Expand Down Expand Up @@ -61,7 +61,7 @@ jobs:

- name: Setup Node.js
id: setup-node
uses: actions/setup-node@v6
uses: actions/setup-node@v7
with:
node-version: 20

Expand All @@ -74,7 +74,7 @@ jobs:
run: pnpm install

- name: Run Evals
uses: step-security/eval-action@v1
uses: step-security/eval-action@v2
with:
api_key: ${{ secrets.BRAINTRUST_API_KEY }}
runtime: node
Expand All @@ -90,11 +90,32 @@ To see examples of fully configured templates, see the `examples` directory:
- [`node with pnpm`](examples/node/pnpm.yml)
- [`python with pip`](examples/python/pip.yml)
- [`python with uv`](examples/python/uv.yml)
- [`go`](examples/go/go.yml)

## Go evals

The Go runtime executes `go run ${paths}`. To include Go eval results in the PR
comment, print each `ExperimentSummary` as one JSON line after calling
`result.Summarize(ctx)`:

```go
summary, err := result.Summarize(ctx)
if err != nil {
log.Fatal(err)
}
b, err := json.Marshal(summary)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(b))
```

## How it works

The action runs `braintrust eval` and collects experiment results, which are
posted as a comment in the PR alongside a link to Braintrust. For example:
For Node and Python, the action runs `braintrust eval`. For Go, the action runs
`go run` on `paths` from `root`. It collects experiment results emitted as JSONL
and posts them as a comment in the PR alongside a link to Braintrust. For
example:

### Example braintrust eval report

Expand Down
6 changes: 3 additions & 3 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@ inputs:
required: false
default: "."
runtime:
description: "The runtime to use for evals. Valid values: node, python."
description: "The runtime to use for evals. Valid values: node, python, go."
required: true
package_manager:
description:
"The package manager to use for evals. Valid values: npm, pnpm, yarn, pip,
or uv depending on the runtime."
"The package manager to use for evals. Valid values: npm or pnpm for node,
pip or uv for python, or go for Go."
required: false
default: ""
use_proxy:
Expand Down
51 changes: 0 additions & 51 deletions eval/.eslintrc.yml

This file was deleted.

3 changes: 0 additions & 3 deletions eval/.prettierignore

This file was deleted.

247 changes: 132 additions & 115 deletions eval/dist/index.js

Large diffs are not rendered by default.

Loading
Loading