Skip to content

spec: grouped operators crash on string labels under pandas 3.0 (ArrowStringArray) #924

Description

@FabianHofmann

Note

This bug report was drafted by an AI agent from a reproduced failure. The
reproduction, traceback and cause below are verbatim from the run.

Building a spec model that groups over a string dimension crashes under
pandas 3.0. The grouped operators are affected: sum(by=…), shift(…, by=…),
sum_back(…, by=…) and at.

pandas 3.0 defaults string data to the Arrow-backed string dtype
(future.infer_string = True), so a lookup whose values are strings arrives as
an ArrowStringArray. linopy/spec/operators.py::_grouped then calls
DataArray.sel(...) on that lookup, and the installed xarray refuses to index
an ArrowStringArray (as_indexable raises TypeError: Invalid array type).

Environment

  • linopy: spec-builder branch
  • pandas 3.0.5, xarray 2026.7.0, numpy 2.4.6, Python 3.12.3

Minimal reproduction

import pandas as pd
import linopy

linopy.options["semantics"] = "v1"
spec = {
    "dimensions": {"t": {"dtype": "str"}, "g": {"dtype": "str"}},
    "lookups": {"grp": {"over": "t", "into": "g"}},
    "parameters": {"v": {"dims": ["t"]}},
    "variables": {
        "x": {"foreach": ["t"], "bounds": {"lower": 0, "upper": 100}},
        "y": {"foreach": ["t"], "bounds": {"lower": -100, "upper": 100}},
    },
    "constraints": {
        "fix": {"foreach": ["t"], "expression": "x == v"},
        "link": {"foreach": ["t"],
                 "expression": "y == shift(x, over=t, offset=1, edge=0, by=grp)"},
    },
    "objective": {"sense": "minimize", "expression": "sum(x)"},
}
data = {
    "t": ["t0", "t1", "t2"], "g": ["g0", "g1"],
    "grp": pd.Series(["g0", "g0", "g1"], index=["t0", "t1", "t2"]),
    "v": pd.Series([1.0, 2.0, 3.0], index=["t0", "t1", "t2"]),
}
linopy.Model.from_spec(spec, data)   # -> TypeError

Cause, confirmed

Adding pd.set_option("future.infer_string", False) before the build makes the
same model build and solve. The trigger is the pandas-3.0 string default
producing an ArrowStringArray for the lookup's values.

Suggested fix

In _grouped (linopy/spec/operators.py:263), take the lookup values off the
Arrow-backed dtype before indexing — build keys from a plain object/numpy
view of the values, or avoid .sel here — so string groups index the same way
int and object groups already do.

Full traceback
Traceback (most recent call last):
  File "repro.py", line 31, in <module>
    linopy.Model.from_spec(spec, data)
  File ".../linopy/model.py", line 510, in from_spec
    return cls(**model_kwargs).add_spec(spec, sources, retain=retain)
  File ".../linopy/model.py", line 494, in add_spec
    self._spec = attach(self, spec, sources, retain)
  File ".../linopy/spec/accessor.py", line 59, in attach
    build(model, bound)
  File ".../linopy/spec/builder.py", line 61, in build
    _constraints(ctx)
  File ".../linopy/spec/builder.py", line 134, in _constraints
    lhs, rhs = evaluate(row.lhs, ctx), evaluate(row.rhs, ctx)
  File ".../linopy/spec/builder.py", line 225, in evaluate
    return operators.shift(
  File ".../linopy/spec/operators.py", line 128, in shift
    groups = _grouped(over, np.asarray(array.indexes[over]), by)
  File ".../linopy/spec/operators.py", line 263, in _grouped
    keys = np.asarray(groups.sel({over: labels}).values, dtype=object)
  File ".../xarray/core/dataarray.py", line 1729, in sel
    ds = self._to_temp_dataset().sel(...)
  File ".../xarray/core/dataset.py", line 3086, in sel
    result = self.isel(indexers=query_results.dim_indexers, drop=drop)
  File ".../xarray/core/dataset.py", line 2939, in isel
    var = var.isel(var_indexers)
  File ".../xarray/core/variable.py", line 1144, in isel
    return self[key]
  File ".../xarray/core/variable.py", line 829, in __getitem__
    indexable = as_indexable(self._data)
  File ".../xarray/core/indexing.py", line 1030, in as_indexable
    raise TypeError(f"Invalid array type: {type(array)}")
TypeError: Invalid array type: <class 'pandas.arrays.ArrowStringArray'>

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingmath-specYAML/text math spec: schema, from_spec/add_spec, operators, expr container

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions