Skip to content

Responses (Surv, Binomial, Multinomial) and the wildcard - #2

Merged
jonathan-taylor merged 2 commits into
mainfrom
responses-wildcard
Oct 26, 2026
Merged

jonathan-taylor merged 2 commits into
mainfrom
responses-wildcard

Conversation

@jonathan-taylor

@jonathan-taylor jonathan-taylor commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

The response of a model, and R's ., for 0.2.0.

Responses. ModelSpec(terms, response=...) and spec.build_response(X) build the response from the same data as the design. A response can be:

  • a column, or a list of columns;
  • Surv(time, status) / Surv(start, stop, status), as R's (keywords stop=, status=, start= also work). Returns a pd.DataFrame of those columns, named as in X: what glmstar's CoxFamily(event_id=..., status_id=..., start_id=...) and lifelines take;
  • Binomial('y') for a binary column: 0/1, boolean, or two levels, with the second level (as in R) or level= counted as a success. Binomial(successes, trials=...) gives either successes and failures (output='counts', R's cbind, which statsmodels' GLM takes) or the proportion and the number of trials (output='proportion': a response and its weights, e.g. for glmstar);
  • Multinomial('class') for the class of each row, as a categorical with the levels seen at fit or one-hot (output='onehot'). Multinomial([...]) takes counts.

Responses learn their levels at fit, and a level unseen at fit raises. The response's columns cannot be used in a term. The response objects are fit as copies, so response is never modified.

The method is build_response, not response, because response is the parameter's name (sklearn stores parameters as attributes). The name also parallels build_submodel and build_sequence.

wildcard, an object rather than the string '.'. It stands for every column of X except the response, and is expanded at fit into expanded_terms_; terms is unchanged.

  • MS([wildcard], response='y') is y ~ .
  • wildcard(pow=2) is .^2, and wildcard(exclude=['x']) is . - x
  • (wildcard, 'g') is .:g
  • pca(wildcard, n_components=3) is a feature built from every column. It can be tuned with terms__0__n_components, and pca()'s name is now optional.

As in R's y ~ x + ., columns (and interactions, in any order) that are already terms are not repeated. pow > 1 is only allowed when the wildcard is itself a term.

fit(X, y) checks that the indices agree when both are pandas objects.

Tests: tests/test_wildcard.py and tests/test_response.py. They check against statsmodels: PHReg on Surv, and binomial GLM with counts versus proportion plus var_weights. The full suite gives 261 passed and 5 strict xfails, and the new tests also pass with warnings as errors.

Docs: a "Responses and the wildcard" page, executed locally with Jupyter Book 2. Sales ~ . gives the same OLS coefficients as R's lm(Sales ~ ., Carseats). The API reference gains the new classes.

🤖 Generated with Claude Code

jonathan-taylor and others added 2 commits October 26, 2026 13:29
ModelSpec(terms, response=...) and build_response(X): the response
from the same data as the design, a column, a list of columns, or Surv,
Binomial or Multinomial. wildcard is R's '.': the columns of X but the
response, expanded at fit into expanded_terms_; wildcard(pow=2) is
'.^2', exclude= is '. - x', and it can be a part of an interaction or a
variable of a feature (pca(wildcard, n_components=3)). fit(X, y) checks
that the indices agree. A docs page, the API reference, and tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The columns of an np.ndarray are np.int64, the terms ints: compare
column identifiers by value, and expand to ints.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jonathan-taylor
jonathan-taylor merged commit c9f9b64 into main Oct 26, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant