Add pandas 3 support Status: Open. #96

[vrane-tibco]

This pull request introduces significant improvements to how Spotfire metadata is handled in DataFrames, modernizing the approach for compatibility with pandas 3 and simplifying the metadata API. The main changes include a new centralized metadata management module, updates to the public API, and adjustments throughout the codebase to use the new system. Additionally, Python and dependency requirements have been updated to support only Python 3.11+ and newer versions of core libraries.

Metadata Management Modernization

• Introduced a new _metadata.py module that centralizes all Spotfire metadata operations for DataFrames, using pandas 3's df.attrs and providing robust, copy-on-write (CoW) optimizations for performance. This module also handles legacy compatibility and deprecation warnings for old metadata patterns.
• Refactored the public API in public.py to use the new _metadata module for all table and column metadata operations, replacing direct attribute access and legacy patterns with explicit getter/setter functions.
• Updated the SBDF import/export logic (sbdf.pyx) and all internal usages to use the new metadata API, ensuring consistent metadata handling across DataFrame operations and geocoding table support.

Dependency and Compatibility Updates

• Dropped support for Python 3.9 and 3.10, now requiring Python 3.11+ and updating classifiers and build/test configurations accordingly.
• Updated core dependencies: now requires numpy >= 2.0.0 and pandas >= 3.0.0, and always includes tzdata for robust timezone handling in minimal containers.

API and Usability Improvements

• Added new utility functions to the public API for getting/setting table and column metadata, and refactored copy_metadata to use the new centralized logic.

New Public API Functions -

      - spotfire.get_table_metadata(df)
      
      - spotfire.set_table_metadata(df, metadata)
      
      - spotfire.get_column_metadata(df, col)
      
      - spotfire.set_column_metadata(df, col, metadata)

• Improved error handling and deprecation warnings for legacy metadata access patterns, guiding users to the new API.

Bug Fixes and Cleanups

• Fixed pandas value extraction logic in data_function.py to use type checks instead of module string comparisons, improving reliability and future compatibility.
• Updated geocoding table logic to use the new metadata API and improved DataFrame assignment for geometry columns.

These changes modernize metadata handling, maintainability, and ensure compatibility with the latest Python and pandas releases.

[Copilot]

Pull request overview

This PR modernizes Spotfire metadata handling for pandas 3 by centralizing DataFrame metadata operations, updating the public API to use that centralized layer, and aligning SBDF import/export plus tests with the new approach and updated dependency/runtime requirements.

Changes:

• Added a new spotfire/_metadata.py module to manage table metadata, column metadata, and Spotfire types via df.attrs (with legacy fallbacks and CoW-focused optimizations).
• Updated public APIs (spotfire/public.py, spotfire/__init__.py) and SBDF import/export (spotfire/sbdf.pyx) to use _metadata consistently.
• Updated tests and packaging/runtime constraints (Python 3.11+, pandas 3+, numpy 2+, adjusted CI python matrix).

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
spotfire/version.py	Bumps package version to the next major dev series.
spotfire/test/test_sbdf.py	Updates SBDF tests to use _metadata and adjusts expectations for pandas 3 string/NA behavior.
spotfire/test/test_data_function.py	Updates data-function tests to use _metadata-based APIs and removes legacy-warning expectations.
spotfire/test/files/data_function/table_metadata.txt	Removes legacy warning output that should no longer occur with new metadata handling.
spotfire/sbdf.pyx	Refactors SBDF import/export to use _metadata, improves dtype handling, and optimizes metadata/dtype lookup loops.
spotfire/public.py	Adds new public metadata getter/setter functions and routes metadata copying/types/geocoding to _metadata.
spotfire/data_function.py	Updates metadata debug logging to use _metadata and modernizes scalar value extraction checks.
spotfire/_metadata.py	Introduces centralized metadata storage/access layer with CoW dict optimization and legacy/deprecation handling.
spotfire/init.py	Exposes new public metadata functions at the package top level.
pyproject.toml	Updates Python/dependency requirements (Python 3.11+, pandas 3+, numpy 2+) and tooling config.
.github/python-versions.json	Drops Python 3.9/3.10 from the CI test matrix.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.