diff --git a/README.md b/README.md index dd692f7..bfaafa7 100644 --- a/README.md +++ b/README.md @@ -1,248 +1,347 @@ -# Useful Python Utils +
-[![CI](https://github.com/WoLpH/python-utils/actions/workflows/ci.yml/badge.svg)](https://github.com/WoLpH/python-utils/actions/workflows/ci.yml) -[![Coverage Status](https://coveralls.io/repos/WoLpH/python-utils/badge.svg?branch=master)](https://coveralls.io/r/WoLpH/python-utils?branch=master) +python-utils -Python Utils is a collection of small Python functions and classes which make -common patterns shorter and easier. It is by no means a complete collection but -it has served me quite a bit in the past and I will keep extending it. +# ⚡ Python Utils -One of the libraries using Python Utils is Django Utils. +**The fast, fully-typed stdlib helpers you keep rewriting — in one tiny, dependency-light package.** -Documentation is available at: https://python-utils.readthedocs.io/en/latest/ +[![PyPI version](https://img.shields.io/pypi/v/python-utils.svg?logo=pypi&logoColor=white)](https://pypi.python.org/pypi/python-utils) +[![Python versions](https://img.shields.io/pypi/pyversions/python-utils.svg?logo=python&logoColor=white)](https://pypi.python.org/pypi/python-utils) +[![CI](https://github.com/WoLpH/python-utils/actions/workflows/ci.yml/badge.svg)](https://github.com/WoLpH/python-utils/actions/workflows/ci.yml) +[![Coverage Status](https://coveralls.io/repos/github/WoLpH/python-utils/badge.svg?branch=develop)](https://coveralls.io/github/WoLpH/python-utils?branch=develop) +[![Typed](https://img.shields.io/badge/typed-mypy%20%7C%20pyright%20%7C%20pyrefly-blue.svg)](https://github.com/WoLpH/python-utils) +[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff) +[![License](https://img.shields.io/pypi/l/python-utils.svg)](https://github.com/WoLpH/python-utils/blob/develop/LICENSE) +[![Downloads](https://img.shields.io/pypi/dm/python-utils.svg?logo=pypi&logoColor=white)](https://pypi.python.org/pypi/python-utils) + +[**Documentation**](https://python-utils.readthedocs.io/en/latest/) · +[**PyPI**](https://pypi.python.org/pypi/python-utils) · +[**Source**](https://github.com/WoLpH/python-utils) · +[**Issues**](https://github.com/WoLpH/python-utils/issues) + +
+ +--- + +Python Utils is a collection of small, battle-tested functions and classes that +make everyday Python patterns shorter, safer and faster. No sprawling framework, +no heavy dependencies — just the helpers you find yourself re-writing in project +after project, packaged once and typed to the hilt. + +It has powered production code for years (and is used by libraries such as +[Django Utils](https://pypi.python.org/pypi/django-utils2) and +[progressbar2](https://pypi.python.org/pypi/progressbar2)). + +## ✨ Highlights + +- 🪶 **Zero-cost imports** — thanks to [PEP 562][pep562] lazy loading, `import + python_utils` pulls in *nothing* until you actually touch a helper. No + `asyncio`, no `typing_extensions`, until you ask for them. +- ⚡ **Async-native** — `acount`, `abatcher`, and timeout/stall detectors bring + `itertools`-style ergonomics to `async for`. +- 📦 **Smart containers** — self-casting dicts, duplicate-proof lists and a + sliceable deque. +- 🔢 **Forgiving converters** — pull an `int`/`float` out of *any* messy string, + scale bytes to KiB/MiB, remap values between ranges (with `Decimal` precision). +- ⏱️ **Time & retries** — human-readable durations plus timeout generators for + sampling slow APIs without hanging. +- 🎯 **Fully typed & 100% covered** — ships `py.typed`, passes **mypy**, + **basedpyright** *and* **pyrefly** in strict mode, with 100% test coverage. +- 🐍 **Modern & tiny** — Python 3.10+, a single runtime dependency + (`typing_extensions`), BSD-3 licensed. + +## 🗺️ What's inside + +| Module | What you get | +| --- | --- | +| [`converters`](https://python-utils.readthedocs.io/en/latest/) | `to_int` · `to_float` · `to_str` · `to_unicode` · `scale_1024` · `remap` | +| [`formatters`](https://python-utils.readthedocs.io/en/latest/) | `camel_to_underscore` · `apply_recursive` · `timesince` | +| [`time`](https://python-utils.readthedocs.io/en/latest/) | `format_time` · `timeout_generator` · `aio_timeout_generator` · `aio_generator_timeout_detector` | +| [`generators`](https://python-utils.readthedocs.io/en/latest/) | `batcher` · `abatcher` (batch by size **or** time interval) | +| [`aio`](https://python-utils.readthedocs.io/en/latest/) | `acount` · `acontainer` — async `itertools` | +| [`containers`](https://python-utils.readthedocs.io/en/latest/) | `CastedDict` · `LazyCastedDict` · `UniqueList` · `SliceableDeque` | +| [`decorators`](https://python-utils.readthedocs.io/en/latest/) | `listify` · `set_attributes` · `sample` · `wraps_classmethod` | +| [`logger`](https://python-utils.readthedocs.io/en/latest/) | `Logged` · `LoggerBase` (+ `Logurud` via the `loguru` extra) | +| [`import_`](https://python-utils.readthedocs.io/en/latest/) | `import_global` — programmatic `from x import *` | +| [`exceptions`](https://python-utils.readthedocs.io/en/latest/) | `raise_exception` · `reraise` | +| [`terminal`](https://python-utils.readthedocs.io/en/latest/) | `get_terminal_size` — works in shells, IPython & Jupyter | +| [`types`](https://python-utils.readthedocs.io/en/latest/) | handy type aliases (`Number`, `Scope`, `StringTypes`, …) | + +## 📦 Installation -## Links +```bash +pip install python-utils +``` -- The source: https://github.com/WoLpH/python-utils -- Project page: https://pypi.python.org/pypi/python-utils -- Reporting bugs: https://github.com/WoLpH/python-utils/issues -- Documentation: https://python-utils.readthedocs.io/en/latest/ -- My blog: https://wol.ph/ +Optional extras: -## Security contact information +```bash +pip install 'python-utils[loguru]' # loguru-backed logging mixin +``` -To report a security vulnerability, please use the -[Tidelift security contact](https://tidelift.com/security). Tidelift will -coordinate the fix and disclosure. +Python **3.10+** is required. The only runtime dependency is +`typing_extensions` (and it's imported lazily). -## Requirements for installing +## 🚀 Quickstart -Python 3.10 or higher is required. There are no other runtime requirements -beyond `typing_extensions`, which is installed automatically. +```python +import python_utils -## Installation +# Pull a number out of any messy string +python_utils.to_int('listening on port=8080', regexp=True) # 8080 -The package can be installed through `pip` (this is the recommended method): +# Human-readable sizes: (value, power-of-1024) +python_utils.scale_1024(1536, 2) # (1.5, 1) -> 1.5 KiB -```bash -pip install python-utils -``` +# Remap a value between ranges (46% volume -> dB on an AVR) +python_utils.remap(46.0, 0.0, 100.0, -80.0, 10.0) # -38.6 -Or download the latest release from -[PyPI](https://pypi.python.org/pypi/python-utils) or -[GitHub](https://github.com/WoLpH/python-utils). +# "time ago" formatting, Django-style +import datetime +python_utils.timesince(datetime.datetime.now() - datetime.timedelta(seconds=61)) +# '1 minute and 1 second ago' +``` -## Quickstart +Everything is reachable straight off the top-level package (`python_utils.`) +or from its submodule (`python_utils.converters.to_int`) — pick whichever reads +better. Either way, only the modules you touch get imported. -This module makes it easy to execute common tasks in Python scripts such as -converting text to numbers and making sure a string is in unicode or bytes -format. +## 🧰 Examples -## Examples +
+🔢 Converters — numbers out of anything -Automatically converting a generator to a list, dict or other collections using -a decorator: +```python +from python_utils import converters -```pycon ->>> @decorators.listify() -... def generate_list(): -... yield 1 -... yield 2 -... yield 3 -... ->>> generate_list() -[1, 2, 3] +# Extract digits with a built-in or custom regexp +converters.to_int('spam15eggs', regexp=True) # 15 +converters.to_int('nope', default=-1) # -1 +converters.to_float('pi is 3.14', regexp=True) # 3.14 ->>> @listify(collection=dict) -... def dict_generator(): -... yield 'a', 1 -... yield 'b', 2 +# Scale bytes to a sensible unit (value, power) -> 2.0 KiB +converters.scale_1024(2048, 3) # (2.0, 1) ->>> dict_generator() -{'a': 1, 'b': 2} +# Linear remap; pass a Decimal anywhere to keep full precision +converters.remap(500, 0, 1000, 0, 100) # 50 +import decimal +converters.remap(decimal.Decimal('250.0'), 0.0, 1000.0, 0.0, 100.0) +# Decimal('25.0') ``` -### Retrying until timeout +
-To easily retry a block of code with a configurable timeout, you can use the -`time.timeout_generator`: +
+📦 Containers — dicts & lists with super-powers -```pycon ->>> for i in time.timeout_generator(10): -... try: -... # Run your code here -... except Exception as e: -... # Handle the exception +```python +from python_utils import containers + +# Keys and values are cast on the way in +d = containers.CastedDict(int, int) +d['3'] = '4' +d.update({'5': '6'}) +d # {3: 4, 5: 6} + +# A list that silently drops duplicates (or raises, if you prefer) +u = containers.UniqueList(1, 2, 3) +u.append(2) # ignored +u # [1, 2, 3] + +# A deque you can actually slice +s = containers.SliceableDeque([1, 2, 3, 4, 5]) +s[1:4] # SliceableDeque([2, 3, 4]) ``` -### Formatting of timestamps, dates and times - -Easy formatting of timestamps and calculating the time since: - -```pycon ->>> time.format_time('1') -'0:00:01' ->>> time.format_time(1.234) -'0:00:01' ->>> time.format_time(1) -'0:00:01' ->>> time.format_time(datetime.datetime(2000, 1, 2, 3, 4, 5, 6)) -'2000-01-02 03:04:05' ->>> time.format_time(datetime.date(2000, 1, 2)) -'2000-01-02' ->>> time.format_time(datetime.timedelta(seconds=3661)) -'1:01:01' ->>> time.format_time(None) -'--:--:--' - ->>> formatters.timesince(now) -'just now' ->>> formatters.timesince(now - datetime.timedelta(seconds=1)) -'1 second ago' ->>> formatters.timesince(now - datetime.timedelta(seconds=2)) -'2 seconds ago' ->>> formatters.timesince(now - datetime.timedelta(seconds=60)) -'1 minute ago' +
+ +
+⚡ Async helpers — itertools for async for + +```python +from python_utils import aio, generators + +# Async counter (optionally with a delay and a stop value) +async def demo(): + async for i in aio.acount(stop=3): + print(i) # 0, 1, 2 + +# Batch an async stream by size OR time interval — whichever comes first. +# Great for chunking bursty producers without ever stalling a slow loop. +async def batched(): + async for batch in generators.abatcher(aio.acount(stop=10), batch_size=3): + print(batch) # [0, 1, 2], [3, 4, 5], [6, 7, 8], [9] + +# Sync batching too: +list(generators.batcher(range(9), 3)) # [[0, 1, 2], [3, 4, 5], [6, 7, 8]] ``` -### Converting your test from camel-case to underscores +
-```pycon ->>> camel_to_underscore('SpamEggsAndBacon') -'spam_eggs_and_bacon' +
+⏱️ Time & retries — sample slow APIs, format durations + +```python +import datetime +from python_utils import time + +# Loop over a slow operation, but give up after `timeout` seconds +for i in time.timeout_generator(0.1, interval=0.06): + ... # yields 0, 1, 2 then stops + +# Format timedeltas, datetimes and raw seconds uniformly +time.format_time(1) # '0:00:01' +time.format_time(datetime.timedelta(seconds=3661)) # '1:01:01' +time.format_time(datetime.datetime(2000, 1, 2, 3, 4, 5)) # '2000-01-02 03:04:05' +time.format_time(None) # '--:--:--' ``` -### Attribute setting decorator (very useful for the Django admin) +There's also `aio_timeout_generator` (the `async for` twin) and +`aio_generator_timeout_detector`, which fails fast when an async generator +stalls instead of hanging forever. -A convenient decorator to set function attributes using a decorator: +
-```pycon -You can use: ->>> @decorators.set_attributes(short_description='Name') -... def upper_case_name(self, obj): -... return ("%s %s" % (obj.first_name, obj.last_name)).upper() +
+🔤 Formatters — case conversion & friendly timestamps -Instead of: ->>> def upper_case_name(obj): -... return ("%s %s" % (obj.first_name, obj.last_name)).upper() +```python +from python_utils import formatters ->>> upper_case_name.short_description = 'Name' +formatters.camel_to_underscore('SpamEggsAndBacon') # 'spam_eggs_and_bacon' + +# Recursively rewrite every key in a nested dict +formatters.apply_recursive( + formatters.camel_to_underscore, + {'SpamEggs': {'FooBar': 1}}, +) # {'spam_eggs': {'foo_bar': 1}} ``` -This can be very useful for the Django admin as it allows you to have all -metadata in one place. +
-### Scaling numbers between ranges +
+🎀 Decorators — collect generators, tag functions, sample calls -```pycon ->>> converters.remap(500, old_min=0, old_max=1000, new_min=0, new_max=100) -50 +```python +from python_utils import decorators -# Or with decimals: ->>> remap(decimal.Decimal('250.0'), 0.0, 1000.0, 0.0, 100.0) -Decimal('25.0') -``` +# Turn a generator into a concrete collection automatically +@decorators.listify() +def numbers(): + yield 1 + yield 2 + yield 3 -### Get the screen/window/terminal size in characters +numbers() # [1, 2, 3] -```pycon ->>> terminal.get_terminal_size() -(80, 24) -``` +@decorators.listify(collection=dict) +def pairs(): + yield 'a', 1 + yield 'b', 2 -That method supports IPython and Jupyter as well as regular shells, using -`blessings` and other modules depending on what is available. +pairs() # {'a': 1, 'b': 2} -### Extracting numbers from nearly every string +# Attach metadata to a function (handy for the Django admin) +@decorators.set_attributes(short_description='Name') +def upper_case_name(self, obj): + return f'{obj.first_name} {obj.last_name}'.upper() -```pycon ->>> converters.to_int('spam15eggs') -15 ->>> converters.to_int('spam') -0 ->>> number = converters.to_int('spam', default=1) -1 +# Only actually run ~10% of the calls +@decorators.sample(0.1) +def maybe_log(msg): ... ``` -### Doing a global import of all the modules in a package programmatically +
-To do a global import programmatically you can use the `import_global` -function. This effectively emulates a `from ... import *`: +
+📝 Logging — a correctly-named logger on every class ```python -from python_utils.import_ import import_global +from python_utils.logger import Logged -# The following is the equivalent of `from some_module import *` -import_global('some_module') +class MyClass(Logged): + def do_work(self): + self.info('starting %s', 'work') # stdlib %-style logging args + self.error('something went wrong') + +MyClass().do_work() ``` -### Automatically named logger for classes +Prefer [loguru](https://github.com/Delgan/loguru)? Install the extra +(`pip install 'python-utils[loguru]'`) and subclass `Logurud` instead — the same +`self.info(...)` / `self.error(...)` API, backed by loguru so you keep all its +configuration and per-instance context. -Add a correctly named logger to your classes which can be easily accessed: +
-```python -class MyClass(Logged): - def __init__(self): - Logged.__init__(self) +
+🖥️ Terminal & 🧩 misc -my_class = MyClass() +```python +from python_utils import terminal, import_ +from python_utils.exceptions import raise_exception, reraise -# Accessing the logging method: -my_class.error('error') +# Robust terminal size (tries IPython/Jupyter, shutil, blessings, ioctl, tput…) +terminal.get_terminal_size() # e.g. (80, 24) -# With formatting: -my_class.error('The logger supports %(formatting)s', - formatting='named parameters') +# Programmatic `from some_module import *` +import_.import_global('os') -# Or to access the actual log function (overwriting the log formatting can -# be done in the log method) -import logging -my_class.log(logging.ERROR, 'log') +# Build a callable that raises — useful as a default/callback +on_error = raise_exception(ValueError, 'boom') ``` -Alternatively loguru is also supported. It is largely a drop-in replacement for -the logging module which is a bit more convenient to configure. First install -the extra loguru package: +
-```bash -pip install 'python-utils[loguru]' -``` +## ⚡ Performance: lazy by default + +`import python_utils` is intentionally *cheap*. Every submodule and every export +is wired through a [PEP 562][pep562] `__getattr__`, so nothing is imported until +first access — and then it's cached. In particular: + +- Need only the synchronous helpers? `asyncio` is never imported. +- Even `typing_extensions` is deferred, so the import graph stays minimal. ```python -class MyClass(Logurud): - ... +import sys +import python_utils # imports basically nothing extra + +'asyncio' in sys.modules # False +python_utils.acount # now `aio` (and asyncio) load, on demand ``` -Now you can use the `Logurud` class to make functions such as `self.info()` -available. The benefit of this approach is that you can add extra context or -options to your specific loguru instance (i.e. `self.logger`). +See the [performance guide](https://python-utils.readthedocs.io/en/latest/) for +the full story. -### Convenient type aliases and some commonly used types +## 📚 Documentation -```python -# For type hinting scopes such as locals/globals/vars -Scope = Dict[str, Any] -OptionalScope = O[Scope] +Full API reference and guides live at +****. -# Note that Number is only useful for extra clarity since float -# will work for both int and float in practice. -Number = U[int, float] -DecimalNumber = U[Number, decimal.Decimal] +## 🔗 Links -# To accept an exception or list of exceptions -ExceptionType = Type[Exception] -ExceptionsType = U[Tuple[ExceptionType, ...], ExceptionType] +- 📖 Documentation: +- 🐙 Source: +- 📦 PyPI: +- 🐛 Issues: +- ✍️ Author's blog: -# Matching string/bytes types: -StringTypes = U[str, bytes] -``` +## 🔒 Security + +To report a security vulnerability, please use the +[Tidelift security contact](https://tidelift.com/security). Tidelift will +coordinate the fix and disclosure. + +## 🤝 Contributing + +Contributions are very welcome! We keep a strict **100% coverage** bar and run +`ruff`, three type checkers and the full test matrix in CI. See +[CONTRIBUTING.md](https://github.com/WoLpH/python-utils/blob/develop/CONTRIBUTING.md) +to get set up. + +## 📄 License + +BSD-3-Clause — see [LICENSE](https://github.com/WoLpH/python-utils/blob/develop/LICENSE). + +[pep562]: https://peps.python.org/pep-0562/ diff --git a/_python_utils_tests/__init__.py b/_python_utils_tests/__init__.py index e69de29..b035251 100644 --- a/_python_utils_tests/__init__.py +++ b/_python_utils_tests/__init__.py @@ -0,0 +1 @@ +"""Test suite for python_utils.""" diff --git a/_python_utils_tests/test_aio.py b/_python_utils_tests/test_aio.py index 217e146..e7532a8 100644 --- a/_python_utils_tests/test_aio.py +++ b/_python_utils_tests/test_aio.py @@ -1,3 +1,5 @@ +"""Tests for the async helpers in ``python_utils.aio``.""" + import asyncio import pytest @@ -7,9 +9,11 @@ @pytest.mark.asyncio async def test_acount(monkeypatch: pytest.MonkeyPatch) -> None: + """Count with a delay between yields until reaching ``stop``.""" sleeps: types.List[float] = [] async def mock_sleep(delay: float) -> None: + """Record each requested delay instead of sleeping.""" sleeps.append(delay) monkeypatch.setattr(asyncio, 'sleep', mock_sleep) @@ -23,12 +27,16 @@ async def mock_sleep(delay: float) -> None: @pytest.mark.asyncio async def test_acontainer() -> None: + """Collect an async iterable into the requested container.""" + async def async_gen() -> types.AsyncIterable[int]: + """Yield 1, 2, 3 asynchronously.""" yield 1 yield 2 yield 3 async def empty_gen() -> types.AsyncIterable[int]: + """Yield nothing as an async generator.""" if False: yield 1 @@ -52,12 +60,16 @@ async def empty_gen() -> types.AsyncIterable[int]: @pytest.mark.asyncio async def test_adict() -> None: + """Build a dict from an async iterable of key/value pairs.""" + async def async_gen() -> types.AsyncIterable[types.Tuple[int, int]]: + """Yield key/value pairs asynchronously.""" yield 1, 2 yield 3, 4 yield 5, 6 async def empty_gen() -> types.AsyncIterable[types.Tuple[int, int]]: + """Yield no pairs as an async generator.""" if False: yield 1, 2 diff --git a/_python_utils_tests/test_aliases.py b/_python_utils_tests/test_aliases.py index 301f9bd..06c97eb 100644 --- a/_python_utils_tests/test_aliases.py +++ b/_python_utils_tests/test_aliases.py @@ -8,6 +8,7 @@ def test_aliases_do_not_import_typing_extensions() -> None: + """Importing ``_aliases`` must not pull in typing_extensions.""" code = ( 'import sys, python_utils._aliases\n' "assert 'typing_extensions' not in sys.modules\n" @@ -22,6 +23,7 @@ def test_aliases_do_not_import_typing_extensions() -> None: def test_aliases_values() -> None: + """Check the alias values and the ``__all__`` contents.""" from python_utils import _aliases assert _aliases.Number == (int | float) @@ -42,6 +44,7 @@ def test_aliases_values() -> None: def test_types_reexports_aliases_identically() -> None: + """Re-export every ``_aliases`` name identically via ``types``.""" from python_utils import _aliases, types for name in _aliases.__all__: @@ -49,6 +52,7 @@ def test_types_reexports_aliases_identically() -> None: def test_types_still_exposes_typing_extensions_surface() -> None: + """The ``types`` facade still re-exports ``Self``.""" # The facade must keep re-exporting typing_extensions (e.g. Self). # ``hasattr`` (not ``types.Self``) avoids basedpyright's # reportUnknownMemberType, since the wildcard re-export has no static type. diff --git a/_python_utils_tests/test_containers.py b/_python_utils_tests/test_containers.py index a38609d..ddee07e 100644 --- a/_python_utils_tests/test_containers.py +++ b/_python_utils_tests/test_containers.py @@ -1,9 +1,12 @@ +"""Tests for the container types in ``python_utils.containers``.""" + import pytest from python_utils import containers def test_unique_list_ignore() -> None: + """Ignore duplicate appends and block duplicate slice sets.""" a: containers.UniqueList[int] = containers.UniqueList() a.append(1) a.append(1) @@ -17,6 +20,7 @@ def test_unique_list_ignore() -> None: def test_unique_list_raise() -> None: + """Raise on duplicates when ``on_duplicate='raise'``.""" a: containers.UniqueList[int] = containers.UniqueList( *range(20), on_duplicate='raise' ) @@ -32,6 +36,7 @@ def test_unique_list_raise() -> None: def test_sliceable_deque() -> None: + """Support indexing and extended slicing on the deque.""" d: containers.SliceableDeque[int] = containers.SliceableDeque(range(10)) assert d[0] == 0 assert d[-1] == 9 @@ -49,6 +54,7 @@ def test_sliceable_deque() -> None: def test_sliceable_deque_pop() -> None: + """Pop by index and raise ``IndexError`` when out of range.""" d: containers.SliceableDeque[int] = containers.SliceableDeque(range(10)) assert d.pop() == 9 == 9 @@ -65,6 +71,7 @@ def test_sliceable_deque_pop() -> None: def test_sliceable_deque_eq() -> None: + """Compare equal to list, tuple, set, and another deque.""" d: containers.SliceableDeque[int] = containers.SliceableDeque([1, 2, 3]) assert d == [1, 2, 3] assert d == (1, 2, 3) diff --git a/_python_utils_tests/test_decorators.py b/_python_utils_tests/test_decorators.py index 8cf129c..0f8e070 100644 --- a/_python_utils_tests/test_decorators.py +++ b/_python_utils_tests/test_decorators.py @@ -1,3 +1,5 @@ +"""Tests for the decorators in ``python_utils.decorators``.""" + import typing from unittest import mock @@ -10,6 +12,7 @@ @pytest.fixture def random(monkeypatch: pytest.MonkeyPatch) -> mock.MagicMock: + """Patch ``decorators.random.random`` and return the mock.""" random_mock = mock.MagicMock() monkeypatch.setattr( 'python_utils.decorators.random.random', random_mock, raising=True @@ -18,6 +21,7 @@ def random(monkeypatch: pytest.MonkeyPatch) -> mock.MagicMock: def test_sample_called(random: mock.MagicMock) -> None: + """Call the wrapped function when the sampled roll passes.""" demo_function = mock.MagicMock() decorated = decorators.sample(0.5)(demo_function) random.return_value = 0.4 @@ -32,6 +36,7 @@ def test_sample_called(random: mock.MagicMock) -> None: def test_sample_not_called(random: mock.MagicMock) -> None: + """Skip the wrapped function when the sampled roll fails.""" demo_function = mock.MagicMock() decorated = decorators.sample(0.5)(demo_function) random.return_value = 0.5 @@ -42,16 +47,21 @@ def test_sample_not_called(random: mock.MagicMock) -> None: class SomeClass: + """A sample class with classmethods for wrapping tests.""" + @classmethod def some_classmethod(cls, arg: T) -> T: + """Return the argument unchanged (generic classmethod).""" return arg @classmethod def some_annotated_classmethod(cls, arg: int) -> int: + """Return the integer argument unchanged.""" return arg def test_wraps_classmethod() -> None: + """Forward calls through a wrapped classmethod.""" some_class = SomeClass() some_class.some_classmethod = mock.MagicMock() # type: ignore[method-assign] wrapped_method = decorators.wraps_classmethod(SomeClass.some_classmethod)( @@ -62,6 +72,7 @@ def test_wraps_classmethod() -> None: def test_wraps_annotated_classmethod() -> None: + """Forward calls through a wrapped annotated classmethod.""" some_class = SomeClass() some_class.some_annotated_classmethod = mock.MagicMock() # type: ignore[method-assign] wrapped_method = decorators.wraps_classmethod( diff --git a/_python_utils_tests/test_generators.py b/_python_utils_tests/test_generators.py index 39498e0..3550aff 100644 --- a/_python_utils_tests/test_generators.py +++ b/_python_utils_tests/test_generators.py @@ -1,3 +1,5 @@ +"""Tests for the batching helpers in ``python_utils.generators``.""" + import asyncio import pytest @@ -8,6 +10,7 @@ @pytest.mark.asyncio async def test_abatcher() -> None: + """Group an async count into fixed-size batches.""" async for batch in python_utils.abatcher(python_utils.acount(stop=9), 3): assert len(batch) == 3 @@ -17,6 +20,7 @@ async def test_abatcher() -> None: @pytest.mark.asyncio async def test_abatcher_timed() -> None: + """Group async items into batches by time interval.""" batches: types.List[types.List[int]] = [] async for batch in python_utils.abatcher( python_utils.acount(stop=10, delay=0.08), interval=0.1 @@ -29,7 +33,10 @@ async def test_abatcher_timed() -> None: @pytest.mark.asyncio async def test_abatcher_timed_with_timeout() -> None: + """Respect timeouts and propagate errors while batching.""" + async def generator() -> types.AsyncIterator[int]: + """Yield items with sleeps to exercise batch timeouts.""" # Test if the timeout is respected yield 0 yield 1 @@ -58,6 +65,7 @@ async def generator() -> types.AsyncIterator[int]: def test_batcher() -> None: + """Split an iterable into fixed-size batches.""" batch = [] for batch in python_utils.batcher(range(9), 3): assert len(batch) == 3 diff --git a/_python_utils_tests/test_import.py b/_python_utils_tests/test_import.py index 31be2be..e78e117 100644 --- a/_python_utils_tests/test_import.py +++ b/_python_utils_tests/test_import.py @@ -1,12 +1,16 @@ +"""Tests for the import helpers in ``python_utils.import_``.""" + from python_utils import import_, types def test_import_globals_relative_import() -> None: + """Resolve relative imports across several levels.""" for i in range(-1, 5): relative_import(i) def relative_import(level: int) -> None: + """Import ``.formatters`` relatively into a fake module dict.""" locals_: types.Dict[str, types.Any] = {} globals_ = {'__name__': 'python_utils.import_'} import_.import_global('.formatters', locals_=locals_, globals_=globals_) @@ -14,6 +18,7 @@ def relative_import(level: int) -> None: def test_import_globals_without_inspection() -> None: + """Import a module without inspecting the caller frame.""" locals_: types.Dict[str, types.Any] = {} globals_: types.Dict[str, types.Any] = {'__name__': __name__} import_.import_global( @@ -23,6 +28,7 @@ def test_import_globals_without_inspection() -> None: def test_import_globals_single_method() -> None: + """Import only the single named attribute.""" locals_: types.Dict[str, types.Any] = {} globals_: types.Dict[str, types.Any] = {'__name__': __name__} import_.import_global( @@ -35,11 +41,13 @@ def test_import_globals_single_method() -> None: def test_import_globals_with_inspection() -> None: + """Infer the target globals from the caller frame.""" import_.import_global('python_utils.formatters') assert 'camel_to_underscore' in globals() def test_import_globals_missing_module() -> None: + """Ignore ``ImportError`` for a missing module (locals_).""" import_.import_global( 'python_utils.spam', exceptions=ImportError, locals_=locals() ) @@ -47,6 +55,7 @@ def test_import_globals_missing_module() -> None: def test_import_locals_missing_module() -> None: + """Ignore ``ImportError`` for a missing module (globals_).""" import_.import_global( 'python_utils.spam', exceptions=ImportError, globals_=globals() ) diff --git a/_python_utils_tests/test_import_footprint.py b/_python_utils_tests/test_import_footprint.py index bc55d60..abffa08 100644 --- a/_python_utils_tests/test_import_footprint.py +++ b/_python_utils_tests/test_import_footprint.py @@ -31,6 +31,7 @@ def _modules_after_import(target: str) -> set[str]: + """Return ``sys.modules`` after importing ``target`` cleanly.""" code = ( f'import sys, {target}\n' 'import json\n' @@ -49,12 +50,14 @@ def _modules_after_import(target: str) -> set[str]: @pytest.mark.parametrize(('target', 'denied'), FOOTPRINT_CASES) def test_import_footprint(target: str, denied: tuple[str, ...]) -> None: + """Importing ``target`` must not pull in denied modules.""" present = _modules_after_import(target) leaked = [m for m in denied if m in present] assert not leaked, f'{target} eagerly imported {leaked}' def test_bare_import_module_count_under_budget() -> None: + """Keep a bare import under the module-count budget.""" # Coarse bloat tripwire (denylist above is the real guard). Cap tightened # now that __version__ is lazy (importlib.metadata no longer pulled on bare # import). Bump only if a new Python version legitimately adds startup @@ -66,6 +69,7 @@ def test_bare_import_module_count_under_budget() -> None: def test_bare_import_does_not_pull_importlib_metadata() -> None: + """A bare import must not import ``importlib.metadata``.""" # __version__ is resolved lazily; bare import must not call # importlib.metadata.version() (which drags in email/zipfile/json/...). present = _modules_after_import('python_utils') @@ -73,6 +77,7 @@ def test_bare_import_does_not_pull_importlib_metadata() -> None: def test_version_resolves_correctly() -> None: + """Resolve ``__version__`` lazily to a non-empty string.""" import python_utils assert isinstance(python_utils.__version__, str) @@ -92,6 +97,7 @@ def test_version_resolves_correctly() -> None: @pytest.mark.parametrize(('module', 'name'), PUBLIC_CALLABLES_TO_INTROSPECT) def test_get_type_hints_still_resolves(module: str, name: str) -> None: + """Resolve type hints without ``NameError`` for public APIs.""" import importlib import typing diff --git a/_python_utils_tests/test_lazy_imports.py b/_python_utils_tests/test_lazy_imports.py index f5e5c5a..8cc6d8c 100644 --- a/_python_utils_tests/test_lazy_imports.py +++ b/_python_utils_tests/test_lazy_imports.py @@ -14,6 +14,7 @@ def _run_clean(code: str) -> subprocess.CompletedProcess[str]: + """Run ``code`` in a fresh interpreter and return the result.""" # Run in a fresh interpreter: the test session itself has long since # imported asyncio/typing_extensions, so in-process checks are useless. env = {**os.environ, 'PYTHONPATH': os.pathsep.join(sys.path)} @@ -26,6 +27,7 @@ def _run_clean(code: str) -> subprocess.CompletedProcess[str]: def test_package_lazy_attribute_access() -> None: + """Resolve submodule and exported names via ``__getattr__``.""" # Submodule access and exported-name access both resolve via __getattr__. aio = python_utils.aio assert python_utils.aio is aio # repeated access returns the cached module @@ -37,6 +39,7 @@ def test_package_lazy_attribute_access() -> None: def test_bare_import_stays_light() -> None: + """Keep a bare import free of asyncio and typing_extensions.""" # Importing the package must not eagerly pull in heavy/optional deps. result = _run_clean( 'import sys, python_utils\n' @@ -48,6 +51,7 @@ def test_bare_import_stays_light() -> None: def test_importing_time_submodule_avoids_asyncio() -> None: + """Import ``python_utils.time`` without importing asyncio.""" # Importing python_utils.time for its synchronous helpers must not import # asyncio; the async helpers import it lazily inside their own bodies. result = _run_clean( @@ -59,6 +63,7 @@ def test_importing_time_submodule_avoids_asyncio() -> None: def test_first_access_caches_into_module_dict() -> None: + """Cache the first lazy access into the module dict.""" # PEP 562 __getattr__ runs once: the resolved object is cached in the # module namespace so subsequent lookups skip __getattr__ entirely. module = python_utils.time @@ -69,6 +74,7 @@ def test_first_access_caches_into_module_dict() -> None: def test_dir_lists_lazy_submodules() -> None: + """List lazy submodules and ``__all__`` names via ``dir``.""" # Lazy submodules that are not in __all__ (e.g. ``containers`` and # ``exceptions``) must still be discoverable via ``dir``; tools such as # ``import_global`` intersect requested names with ``dir(module)``. @@ -79,6 +85,7 @@ def test_dir_lists_lazy_submodules() -> None: @pytest.mark.asyncio async def test_aio_timeout_generator_default_iterable() -> None: + """Default the iterable to ``aio.acount`` when omitted.""" # With no iterable the generator defaults to ``aio.acount`` -- exercising # the lazy ``aio``/``asyncio`` import and the None-resolution branch. count = 0 diff --git a/_python_utils_tests/test_logger.py b/_python_utils_tests/test_logger.py index 111444a..15122b5 100644 --- a/_python_utils_tests/test_logger.py +++ b/_python_utils_tests/test_logger.py @@ -1,4 +1,6 @@ # mypy: disable-error-code=misc +"""Tests for the loguru mixin in ``python_utils.loguru``.""" + import pytest from python_utils import loguru @@ -7,6 +9,8 @@ def test_logurud() -> None: + """Expose all loguru log-level methods on a subclass.""" + class MyClass(loguru.Logurud): pass diff --git a/_python_utils_tests/test_python_utils.py b/_python_utils_tests/test_python_utils.py index 5a41d4b..558b400 100644 --- a/_python_utils_tests/test_python_utils.py +++ b/_python_utils_tests/test_python_utils.py @@ -1,7 +1,10 @@ +"""Tests for the package metadata in ``python_utils.__about__``.""" + from python_utils import __about__ def test_definitions() -> None: + """Ensure the required metadata fields are defined.""" # The setup.py requires this so we better make sure they exist :) assert __about__.__version__ assert __about__.__author__ diff --git a/_python_utils_tests/test_time.py b/_python_utils_tests/test_time.py index fdbe1fa..da26c9e 100644 --- a/_python_utils_tests/test_time.py +++ b/_python_utils_tests/test_time.py @@ -1,3 +1,5 @@ +"""Tests for the timeout generators in ``python_utils.time``.""" + import asyncio import datetime import itertools @@ -33,6 +35,7 @@ async def test_aio_timeout_generator( iterable: types.AsyncIterable[types.Any], result: int, ) -> None: + """Stop the async generator near the configured timeout.""" i = None async for i in python_utils.aio_timeout_generator( timeout, interval, iterable, maximum_interval=maximum_interval @@ -71,6 +74,7 @@ def test_timeout_generator( ], result: int, ) -> None: + """Stop the sync generator near the configured timeout.""" i = None for i in python_utils.timeout_generator( timeout=timeout, @@ -86,10 +90,12 @@ def test_timeout_generator( @pytest.mark.asyncio async def test_aio_generator_timeout_detector() -> None: + """Raise or exit on per-item and total timeouts.""" # Make pyright happy i = None async def generator() -> types.AsyncGenerator[int, None]: + """Yield 0-9 with increasing sleeps between items.""" for i in range(10): await asyncio.sleep(i / 20.0) yield i @@ -124,9 +130,12 @@ async def generator() -> types.AsyncGenerator[int, None]: @pytest.mark.asyncio async def test_aio_generator_timeout_detector_decorator_reraise() -> None: + """Reraise ``TimeoutError`` on a per-item timeout.""" + # Test regular timeout with reraise @python_utils.aio_generator_timeout_detector_decorator(timeout=0.05) async def generator_timeout() -> types.AsyncGenerator[int, None]: + """Yield with increasing delays to trip the timeout.""" for i in range(10): await asyncio.sleep(i / 100.0) yield i @@ -138,6 +147,7 @@ async def generator_timeout() -> types.AsyncGenerator[int, None]: @pytest.mark.asyncio async def test_aio_generator_timeout_detector_decorator_clean_exit() -> None: + """Exit cleanly when ``on_timeout`` is ``None``.""" # Make pyright happy i = None @@ -146,6 +156,7 @@ async def test_aio_generator_timeout_detector_decorator_clean_exit() -> None: timeout=0.05, on_timeout=None ) async def generator_clean() -> types.AsyncGenerator[int, None]: + """Yield with increasing delays to trip the timeout.""" for i in range(10): await asyncio.sleep(i / 100.0) yield i @@ -160,9 +171,12 @@ async def generator_clean() -> types.AsyncGenerator[int, None]: async def test_aio_generator_timeout_detector_decorator_reraise_total() -> ( None ): + """Reraise ``TimeoutError`` on a total timeout.""" + # Test total timeout with reraise @python_utils.aio_generator_timeout_detector_decorator(total_timeout=0.1) async def generator_reraise() -> types.AsyncGenerator[int, None]: + """Yield with increasing delays to trip the timeout.""" for i in range(10): await asyncio.sleep(i / 100.0) yield i @@ -174,6 +188,7 @@ async def generator_reraise() -> types.AsyncGenerator[int, None]: @pytest.mark.asyncio async def test_aio_generator_timeout_detector_decorator_clean_total() -> None: + """Exit cleanly on total timeout when ``on_timeout`` is ``None``.""" # Make pyright happy i = None @@ -182,6 +197,7 @@ async def test_aio_generator_timeout_detector_decorator_clean_total() -> None: total_timeout=0.1, on_timeout=None ) async def generator_clean_total() -> types.AsyncGenerator[int, None]: + """Yield with increasing delays to trip the timeout.""" for i in range(10): await asyncio.sleep(i / 100.0) yield i diff --git a/docs/_static/banner.svg b/docs/_static/banner.svg new file mode 100644 index 0000000..29648a7 --- /dev/null +++ b/docs/_static/banner.svg @@ -0,0 +1,69 @@ + + python-utils + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + python-utils + + + + + The fast, fully-typed stdlib helpers you keep rewriting. + + + + + + + lazy imports + + + + async-native + + + + fully typed + + + + 100% covered + + + diff --git a/docs/conf.py b/docs/conf.py index 110ee6d..2fe2f05 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -31,9 +31,19 @@ napoleon_numpy_docstring = False autodoc_typehints = 'description' +# `loguru` is an optional dependency (the `loguru` extra); mock it so the docs +# build can import and document `python_utils.loguru` without it installed. +autodoc_mock_imports = ['loguru'] templates_path = ['_templates'] -exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] +exclude_patterns = [ + '_build', + 'Thumbs.db', + '.DS_Store', + # Local skill/session artifacts that are not part of the published docs. + 'superpowers', + 'superpowers/**', +] intersphinx_mapping = {'python': ('https://docs.python.org/3', None)} diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 0000000..b285352 --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,78 @@ +# 🚀 Getting started + +Python Utils is a small collection of typed, dependency-light helpers for the +patterns you keep rewriting: pulling numbers out of strings, batching iterables, +formatting durations, casting dictionaries, and more. + +## Installation + +```bash +pip install python-utils +``` + +Optional extras: + +```bash +pip install 'python-utils[loguru]' # loguru-backed logging mixin +``` + +Python **3.10+** is required. The only runtime dependency is `typing_extensions`, +and even that is imported lazily. + +## Lazy by design + +Everything is reachable straight off the top-level package, but nothing is +imported until you actually use it (see the +[performance guide](guide/performance.md)): + +```python +import python_utils + +python_utils.to_int('port=8080', regexp=True) # 8080 +``` + +You can also import from a submodule when you prefer an explicit namespace — the +result is identical, and only the modules you touch get loaded: + +```python +from python_utils import converters + +converters.to_int('port=8080', regexp=True) # 8080 +``` + +## A five-minute tour + +```python +import datetime +import python_utils + +# Numbers out of anything +python_utils.to_int('spam15eggs', regexp=True) # 15 + +# Human-readable sizes: (value, power-of-1024) +python_utils.scale_1024(2048, 3) # (2.0, 1) -> 2 KiB + +# Remap a value between ranges (Decimal-safe) +python_utils.remap(46.0, 0.0, 100.0, -80.0, 10.0) # -38.6 + +# Friendly durations +python_utils.format_time(datetime.timedelta(seconds=3661)) # '1:01:01' +python_utils.timesince( + datetime.datetime.now() - datetime.timedelta(seconds=61) +) # '1 minute and 1 second ago' + +# Batch an iterable +list(python_utils.batcher(range(9), 3)) # [[0, 1, 2], [3, 4, 5], [6, 7, 8]] +``` + +## Where to next + +- [Async helpers](guide/async.md) — `acount`, `abatcher`, timeout & stall + detectors. +- [Smart containers](guide/containers.md) — `CastedDict`, `UniqueList`, + `SliceableDeque`. +- [Conversions & formatting](guide/conversions-and-formatting.md) — `to_int`, + `scale_1024`, `remap`, `camel_to_underscore`, `timesince`. +- [Performance](guide/performance.md) — how lazy imports keep startup cheap. +- [What's new](whats-new.md) — the v4 modernization. +- The full {doc}`API reference `. diff --git a/docs/guide/async.md b/docs/guide/async.md new file mode 100644 index 0000000..604f95b --- /dev/null +++ b/docs/guide/async.md @@ -0,0 +1,102 @@ +# ⚡ Async helpers + +`python_utils` brings `itertools`-style ergonomics to `async for`, plus tools for +sampling and guarding slow async generators. Because of +[lazy imports](performance.md), `asyncio` is only imported once you actually +touch one of these helpers. + +## `acount` — an async counter + +The async twin of `itertools.count`, with an optional `step`, `delay` (seconds +between yields) and `stop` value: + +```python +import asyncio +from python_utils import aio + +async def main(): + async for i in aio.acount(stop=3): + print(i) # 0, 1, 2 + +asyncio.run(main()) +``` + +## `abatcher` — batch by size *or* time + +`abatcher` wraps an async generator and yields lists of items. Give it a +`batch_size`, an `interval`, or both — it flushes on whichever is reached first. +That makes it ideal for chunking bursty producers without ever stalling a slow +loop. + +Batch purely by size: + +```python +import asyncio +from python_utils import aio, generators + +async def main(): + async for batch in generators.abatcher(aio.acount(stop=10), batch_size=3): + print(batch) # [0, 1, 2], [3, 4, 5], [6, 7, 8], [9] + +asyncio.run(main()) +``` + +Batch by time interval instead (flush at least every ``interval`` seconds). +`interval` accepts a `datetime.timedelta` or a plain number of seconds: + +```python +async for batch in generators.abatcher(source(), interval=0.1): + ... # whatever accumulated in the last 100ms +``` + +After each yield the interval timer resets from the current time, so a slow, +blocking loop never causes a runaway burst. + +There's a synchronous counterpart too: + +```python +from python_utils import generators + +list(generators.batcher(range(9), 3)) # [[0, 1, 2], [3, 4, 5], [6, 7, 8]] +``` + +## `aio_timeout_generator` — sample a slow async source + +The `async for` twin of {func}`~python_utils.time.timeout_generator`: walk an +async iterable until a timeout elapses, sleeping ``interval`` seconds between +items. The `interval` can grow each round (`interval_multiplier`) up to an +optional `maximum_interval`, giving you exponential backoff for free. The default +iterable is `acount`, so you get an async counter out of the box. + +```python +import asyncio +from python_utils import time + +async def main(): + # Yield roughly every 0.06s, for at most 0.1s total. + async for i in time.aio_timeout_generator(timeout=0.1, interval=0.06): + print(i) # 0, 1, ... + +asyncio.run(main()) +``` + +## `aio_generator_timeout_detector` — fail fast on a stalled generator + +Wrap an async generator so that, if it goes quiet for longer than `timeout` +seconds (or exceeds `total_timeout` overall), you find out instead of hanging +forever. By default the underlying `asyncio.TimeoutError` is re-raised; pass an +`on_timeout` callback to handle it your own way. + +```python +from python_utils import time + +async def slow_source(): ... + +guarded = time.aio_generator_timeout_detector(slow_source(), timeout=5) + +# Or as a decorator: +@time.aio_generator_timeout_detector_decorator(timeout=5) +async def producer(): ... +``` + +See the {doc}`API reference <../python_utils>` for the full signatures. diff --git a/docs/guide/containers.md b/docs/guide/containers.md new file mode 100644 index 0000000..98e1fe5 --- /dev/null +++ b/docs/guide/containers.md @@ -0,0 +1,78 @@ +# 📦 Smart containers + +Drop-in replacements for `dict` and `list` that add type casting, uniqueness and +slicing — while staying ordinary `dict`/`list` subclasses you can pass anywhere. + +## `CastedDict` — cast keys and values on the way in + +Give it a key cast and a value cast (either may be `None` to skip). Every +assignment — including `update` and construction — is cast eagerly, so what you +store is already the right type: + +```python +from python_utils import containers + +d = containers.CastedDict(int, int) +d['3'] = '4' +d.update({'5': '6'}) +d # {3: 4, 5: 6} +``` + +Use the subscription syntax to keep type checkers happy: + +```python +d: containers.CastedDict[int, int] = containers.CastedDict(int, int) +``` + +## `LazyCastedDict` — cast values only when read + +Same idea, but values are stored raw and cast on access. Handy when casting is +expensive and you only read a subset of the keys. Note that values are **not** +cached, so the cast runs on every read: + +```python +from python_utils import containers + +d = containers.LazyCastedDict(int, int) +d['3'] = '4' +d.update({'5': '6'}) + +d # {3: '4', 5: '6'} <- values still raw +d[3] # 4 <- cast on access +list(d.values()) # [4, 6] +``` + +## `UniqueList` — a list that rejects duplicates + +By default duplicates are silently ignored; set `on_duplicate='raise'` to turn +them into errors instead. Uniqueness is enforced across `append`, `insert`, +`extend` and item assignment, backed by an internal `set` for O(1) membership +tests: + +```python +from python_utils import containers + +u = containers.UniqueList(1, 2, 3) +u.append(2) # ignored +u # [1, 2, 3] + +strict = containers.UniqueList(1, 2, 3, on_duplicate='raise') +strict.append(2) # ValueError: Duplicate value: 2 +``` + +## `SliceableDeque` — a deque you can slice + +A `collections.deque` keeps fast appends and pops at both ends but doesn't +support slicing. `SliceableDeque` adds it, plus friendly equality against lists, +tuples and sets: + +```python +from python_utils import containers + +s = containers.SliceableDeque([1, 2, 3, 4, 5]) +s[1:4] # SliceableDeque([2, 3, 4]) +s[::-1] # SliceableDeque([5, 4, 3, 2, 1]) +s == [1, 2, 3, 4, 5] # True +``` + +See the {doc}`API reference <../python_utils>` for every method and overload. diff --git a/docs/guide/conversions-and-formatting.md b/docs/guide/conversions-and-formatting.md new file mode 100644 index 0000000..11b1d6a --- /dev/null +++ b/docs/guide/conversions-and-formatting.md @@ -0,0 +1,99 @@ +# 🔢 Conversions & formatting + +Forgiving converters and human-friendly formatters for the messy, real-world data +you get from users, files and APIs. + +## Numbers out of strings + +`to_int` and `to_float` never raise on bad input — they return a `default` +(itself `0` by default). Opt into regexp extraction to pull the first number out +of surrounding text: + +```python +from python_utils import converters + +converters.to_int('abc') # 0 +converters.to_int('spam15eggs', regexp=True) # 15 +converters.to_int('nope', default=-1) # -1 + +converters.to_float('pi is 3.14', regexp=True) # 3.14 +``` + +`regexp=True` uses a sensible built-in pattern; pass your own `str` or compiled +`re.Pattern` for full control (the last capture group is used). + +## Text and bytes + +`to_str` and `to_unicode` normalise between `str` and `bytes` with configurable +encoding and error handling: + +```python +from python_utils import converters + +converters.to_unicode(b'a') # 'a' +converters.to_str('a') # b'a' +``` + +## `scale_1024` — human-readable sizes + +Scale a number by powers of 1024 and get back both the scaled value and the +power, ready to index into a list of prefixes (`['', 'Ki', 'Mi', ...]`): + +```python +from python_utils import converters + +converters.scale_1024(1, 3) # (1.0, 0) -> 1 B +converters.scale_1024(2048, 3) # (2.0, 1) -> 2 KiB +``` + +## `remap` — move a value between ranges + +Linearly remap a value from one range to another. Pass a `decimal.Decimal` +anywhere and the whole computation is done in `Decimal` to avoid floating-point +error; otherwise it follows Python's usual `int`/`float` promotion: + +```python +import decimal +from python_utils import converters + +converters.remap(500, 0, 1000, 0, 100) # 50 +converters.remap(46.0, 0.0, 100.0, -80.0, 10.0) # -38.6 (46% volume -> dB) +converters.remap(decimal.Decimal('250.0'), 0.0, 1000.0, 0.0, 100.0) +# Decimal('25.0') +``` + +## `camel_to_underscore` & `apply_recursive` + +Convert CamelCase to snake_case, and apply any string transform to every key of a +(possibly nested) mapping: + +```python +from python_utils import formatters + +formatters.camel_to_underscore('SpamEggsAndBacon') # 'spam_eggs_and_bacon' + +formatters.apply_recursive( + formatters.camel_to_underscore, + {'SpamEggs': {'FooBar': 1}}, +) # {'spam_eggs': {'foo_bar': 1}} +``` + +## `timesince` & `format_time` + +`timesince` gives a Django-style "time ago" string; `format_time` renders +timedeltas, datetimes, dates and raw seconds uniformly: + +```python +import datetime +from python_utils import formatters, time + +now = datetime.datetime.now() +formatters.timesince(now - datetime.timedelta(seconds=61)) # '1 minute and 1 second ago' + +time.format_time(1) # '0:00:01' +time.format_time(datetime.timedelta(seconds=3661)) # '1:01:01' +time.format_time(datetime.datetime(2000, 1, 2, 3, 4, 5)) # '2000-01-02 03:04:05' +time.format_time(None) # '--:--:--' +``` + +See the {doc}`API reference <../python_utils>` for every parameter. diff --git a/docs/guide/performance.md b/docs/guide/performance.md new file mode 100644 index 0000000..1ee8acc --- /dev/null +++ b/docs/guide/performance.md @@ -0,0 +1,58 @@ +# 🪶 Performance: lazy imports + +`import python_utils` is intentionally *cheap*. The package uses +[PEP 562](https://peps.python.org/pep-0562/) module-level `__getattr__` so that +**nothing** is imported when you import the package — each submodule and each +exported name is loaded on first access, then cached. + +## Why it matters + +Utility libraries are often imported everywhere, including in code paths that only +need one or two synchronous helpers. Eagerly importing every submodule would drag +in `asyncio` (via the async helpers) and other machinery you may never use, +inflating startup time and memory. Lazy loading keeps the import graph minimal. + +In particular: + +- Only need the synchronous helpers? `asyncio` is never imported. +- Even `typing_extensions` is deferred, so the base import stays tiny. + +## See it for yourself + +```python +import sys +import python_utils + +'asyncio' in sys.modules # False +'typing_extensions' in sys.modules # False + +python_utils.acount # touches `aio`, which imports asyncio... +'asyncio' in sys.modules # True +``` + +## How it works + +The package `__init__` maps every public name to the submodule that defines it and +resolves them on demand: + +```python +def __getattr__(name): + # look up which submodule provides `name`, import it lazily, + # cache the result in globals() so __getattr__ runs only once, + # and return it. + ... +``` + +Type checkers and IDEs still see everything: the same names are declared under a +`typing.TYPE_CHECKING` block and listed in `__all__`, so autocompletion and static +analysis work exactly as if the imports were eager. `__dir__` is implemented too, +so `dir(python_utils)` and tab-completion list every lazily-available name. + +## Practical tips + +- Import what you use directly (`python_utils.to_int`) — the first access pays the + one-time import cost for that module only. +- Importing from a submodule (`from python_utils import converters`) is equally + lazy with respect to the *other* submodules. +- The cost is paid once per process; subsequent accesses are plain attribute + lookups. diff --git a/docs/index.rst b/docs/index.rst index 61c7f65..00873db 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,24 +1,69 @@ -Welcome to Python Utils's documentation! -======================================== +Python Utils +============ + +.. image:: _static/banner.svg + :alt: Python Utils — the fast, fully-typed stdlib helpers you keep rewriting + :width: 720px + +.. image:: https://img.shields.io/pypi/v/python-utils.svg + :target: https://pypi.python.org/pypi/python-utils + :alt: PyPI version + +.. image:: https://img.shields.io/pypi/pyversions/python-utils.svg + :target: https://pypi.python.org/pypi/python-utils + :alt: Supported Python versions .. image:: https://github.com/WoLpH/python-utils/actions/workflows/ci.yml/badge.svg - :target: https://github.com/WoLpH/python-utils/actions/workflows/ci.yml + :target: https://github.com/WoLpH/python-utils/actions/workflows/ci.yml + :alt: CI status + +.. image:: https://coveralls.io/repos/github/WoLpH/python-utils/badge.svg?branch=develop + :target: https://coveralls.io/github/WoLpH/python-utils?branch=develop + :alt: Coverage status + +**Python Utils** is a collection of small, battle-tested functions and classes +that make everyday Python patterns shorter, safer and faster. No sprawling +framework, no heavy dependencies — just the helpers you find yourself rewriting +in project after project, packaged once and typed to the hilt. + +Highlights: -.. image:: https://coveralls.io/repos/WoLpH/python-utils/badge.svg?branch=master - :target: https://coveralls.io/r/WoLpH/python-utils?branch=master +- 🪶 **Zero-cost imports** — :doc:`PEP 562 lazy loading ` means + ``import python_utils`` pulls in nothing until you touch a helper. +- ⚡ **Async-native** — ``acount``, ``abatcher`` and timeout/stall detectors for + ``async for``. +- 📦 **Smart containers** — self-casting dicts, duplicate-proof lists, a + sliceable deque. +- 🔢 **Forgiving converters** — numbers out of messy strings, byte scaling, range + remapping with ``Decimal`` precision. +- 🎯 **Fully typed & 100% covered** — ships ``py.typed`` and passes mypy, + basedpyright *and* pyrefly in strict mode. -Contents: +New here? Start with :doc:`getting-started`. .. toctree:: - :maxdepth: 4 + :maxdepth: 2 + :caption: Guides + :hidden: + getting-started + guide/async + guide/containers + guide/conversions-and-formatting + guide/performance + whats-new usage + +.. toctree:: + :maxdepth: 4 + :caption: API reference + :hidden: + python_utils Indices and tables -================== +=================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` - diff --git a/docs/python_utils.rst b/docs/python_utils.rst index 288cac7..47dbc7a 100644 --- a/docs/python_utils.rst +++ b/docs/python_utils.rst @@ -4,10 +4,18 @@ python\_utils package Submodules ---------- -python\_utils\.decorators module +python\_utils\.aio module +------------------------- + +.. automodule:: python_utils.aio + :members: + :undoc-members: + :show-inheritance: + +python\_utils\.containers module -------------------------------- -.. automodule:: python_utils.decorators +.. automodule:: python_utils.containers :members: :undoc-members: :show-inheritance: @@ -20,6 +28,22 @@ python\_utils\.converters module :undoc-members: :show-inheritance: +python\_utils\.decorators module +-------------------------------- + +.. automodule:: python_utils.decorators + :members: + :undoc-members: + :show-inheritance: + +python\_utils\.exceptions module +-------------------------------- + +.. automodule:: python_utils.exceptions + :members: + :undoc-members: + :show-inheritance: + python\_utils\.formatters module -------------------------------- @@ -28,6 +52,14 @@ python\_utils\.formatters module :undoc-members: :show-inheritance: +python\_utils\.generators module +-------------------------------- + +.. automodule:: python_utils.generators + :members: + :undoc-members: + :show-inheritance: + python\_utils\.import\_ module ------------------------------ @@ -44,6 +76,15 @@ python\_utils\.logger module :undoc-members: :show-inheritance: +python\_utils\.loguru module +---------------------------- + +.. automodule:: python_utils.loguru + :members: + :undoc-members: + :show-inheritance: + :exclude-members: logger + python\_utils\.terminal module ------------------------------ @@ -60,11 +101,23 @@ python\_utils\.time module :undoc-members: :show-inheritance: +python\_utils\.types module +--------------------------- + +.. automodule:: python_utils.types + +.. note:: + + ``python_utils.types`` re-exports everything from :mod:`typing`, + :mod:`typing_extensions` and :mod:`types`, plus the lightweight aliases + below (defined in a stdlib-only module so importing them stays cheap): + ``Scope``, ``OptionalScope``, ``Number``, ``DecimalNumber``, + ``ExceptionType``, ``ExceptionsType``, ``StringTypes``, ``delta_type`` and + ``timestamp_type``. The shorthands ``O`` (``Optional``) and ``U`` (``Union``) + are kept for backwards compatibility. + Module contents --------------- .. automodule:: python_utils - :members: - :undoc-members: - :show-inheritance: diff --git a/docs/whats-new.md b/docs/whats-new.md new file mode 100644 index 0000000..8ab55c8 --- /dev/null +++ b/docs/whats-new.md @@ -0,0 +1,49 @@ +# 🎉 What's new in 4.0 + +Version 4.0 is a modernization release: the public helpers you already rely on are +unchanged, but the package underneath is faster to import, stricter about types, +and built with a modern toolchain. + +The authoritative, per-release changelog lives on the +[GitHub releases page](https://github.com/WoLpH/python-utils/releases). The +highlights: + +## 🪶 Lazy imports + +`import python_utils` no longer eagerly imports its submodules. Thanks to +[PEP 562](https://peps.python.org/pep-0562/) lazy loading, nothing is imported +until first access — notably `asyncio` and `typing_extensions` stay out of the +import graph until you use a helper that needs them. See the +[performance guide](guide/performance.md). + +## 🎯 Strict typing on three checkers + +The codebase is verified in strict mode by **mypy**, **basedpyright** *and* +**pyrefly**, and ships `py.typed` so your own type checker picks up the +annotations. Coverage remains at **100%**. + +## 🧰 Modern build & tooling + +- Built with the **`uv_build`** backend. +- Linted and formatted with **ruff**. +- Git hooks via **lefthook**. +- A **tox** matrix across supported interpreters, with split GitHub Actions CI and + PyPI Trusted Publishing. +- The source distribution ships the test suite and `tox.ini`, so downstream + packagers can build and test from the sdist. + +## 🐍 Supported Python versions + +Python **3.10+** is supported, and the package is tested against current CPython +releases as well as PyPI. + +## Handy modules worth a fresh look + +- [Async helpers](guide/async.md) — `acount`, `abatcher`, and the timeout/stall + detectors. +- [Smart containers](guide/containers.md) — `CastedDict`, `LazyCastedDict`, + `UniqueList`, `SliceableDeque`. + +For anything not listed here, consult the +[release notes](https://github.com/WoLpH/python-utils/releases) for the exact +version you're upgrading from. diff --git a/pyproject.toml b/pyproject.toml index 2ae3ce7..42cf539 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -19,7 +19,15 @@ requires-python = '>=3.10' license = 'BSD-3-Clause' license-files = ['LICENSE'] authors = [{ name = 'Rick van Hattem', email = 'wolph@wol.ph' }] -keywords = ['utils', 'utilities', 'helpers', 'typing'] +keywords = [ + 'utils', + 'utilities', + 'helpers', + 'typing', + 'typed', + 'async', + 'lazy-imports', +] classifiers = [ 'Development Status :: 5 - Production/Stable', 'Intended Audience :: Developers', diff --git a/python_utils/__about__.py b/python_utils/__about__.py index e85fd72..e0f6086 100644 --- a/python_utils/__about__.py +++ b/python_utils/__about__.py @@ -14,13 +14,18 @@ from importlib import metadata +#: Distribution name as published on PyPI. __package_name__: str = 'python-utils' +#: Primary author's name. __author__: str = 'Rick van Hattem' +#: Primary author's contact email. __author_email__: str = 'Wolph@wol.ph' +#: One-line description of the package. __description__: str = ( 'Python Utils is a module with some convenient utilities not included ' 'with the standard Python install' ) +#: Canonical project/repository URL. __url__: str = 'https://github.com/WoLpH/python-utils' try: diff --git a/python_utils/__init__.py b/python_utils/__init__.py index 300f54a..f5addab 100644 --- a/python_utils/__init__.py +++ b/python_utils/__init__.py @@ -161,7 +161,18 @@ def __getattr__(name: str) -> _typing.Any: - """Lazily import submodules and their exported names on first access.""" + """Lazily import a submodule or exported name on first access (PEP 562). + + Args: + name: Attribute requested on the ``python_utils`` package. + + Returns: + The imported submodule or object. It is cached in ``globals()`` so + this hook runs only once per name. + + Raises: + AttributeError: If ``name`` is not a known submodule or export. + """ if name in _SUBMODULES: module = _importlib.import_module(f'.{name}', __name__) elif name in _NAME_TO_MODULE: @@ -179,6 +190,7 @@ def __getattr__(name: str) -> _typing.Any: def __dir__() -> list[str]: + """List all eager and lazily-available names (for tab-completion).""" return sorted( set(globals()) | set(__all__) | _SUBMODULES | set(_NAME_TO_MODULE) ) diff --git a/python_utils/_aliases.py b/python_utils/_aliases.py index c0940fc..1f69683 100644 --- a/python_utils/_aliases.py +++ b/python_utils/_aliases.py @@ -23,15 +23,25 @@ 'timestamp_type', ] +#: A namespace mapping, e.g. ``locals()``/``globals()`` (name -> value). Scope = dict[str, Any] +#: A :data:`Scope`, or ``None`` when no namespace is supplied. OptionalScope = Scope | None +#: Any plain (non-decimal) number: an ``int`` or a ``float``. Number = int | float +#: A :data:`Number` or a :class:`decimal.Decimal`, for precise arithmetic. DecimalNumber = Number | decimal.Decimal +#: An exception class (not an instance), e.g. ``ValueError``. ExceptionType = type[Exception] +#: One exception class or a tuple of them, as accepted by ``except``. ExceptionsType = tuple[ExceptionType, ...] | ExceptionType +#: Text-like data: ``str`` or ``bytes``. StringTypes = str | bytes +#: A time interval expressed as a ``timedelta`` or a number of seconds. delta_type = datetime.timedelta | int | float +#: Anything :func:`~python_utils.time.format_time` can render: a duration, a +#: date/datetime, a numeric/str timestamp, or ``None``. timestamp_type = ( datetime.timedelta | datetime.date diff --git a/python_utils/aio.py b/python_utils/aio.py index a58bd28..c315b9b 100644 --- a/python_utils/aio.py +++ b/python_utils/aio.py @@ -1,13 +1,22 @@ -"""Asyncio equivalents to regular Python functions.""" +"""Asyncio equivalents of common synchronous helpers. + +These bring ``itertools``-style ergonomics to ``async for``: ``acount`` is an +async counter, while ``acontainer`` and ``adict`` collect an async iterable +into a concrete container. +""" import asyncio import collections.abc import itertools import typing +#: Numeric type (``int`` or ``float``) produced by ``acount``. _N = typing.TypeVar('_N', int, float) +#: Element type of the async iterables being consumed. _T = typing.TypeVar('_T') +#: Key type when collecting an async iterable of pairs into a dict. _K = typing.TypeVar('_K') +#: Value type when collecting an async iterable of pairs into a dict. _V = typing.TypeVar('_V') @@ -17,7 +26,25 @@ async def acount( delay: float = 0, stop: _N | None = None, ) -> collections.abc.AsyncIterator[_N]: - """Asyncio version of itertools.count().""" + """Async equivalent of ``itertools.count()``. + + Counts from ``start`` in steps of ``step``, sleeping ``delay`` seconds + between values, and stops once ``stop`` (when given) is reached. + + Args: + start: First value to yield. + step: Amount added between successive values. + delay: Seconds to ``asyncio.sleep`` between yields. + stop: Exclusive upper bound; ``None`` counts forever. + + Yields: + The successive counter values. + + >>> async def demo(): + ... return [i async for i in acount(stop=3)] + >>> asyncio.run(demo()) + [0, 1, 2] + """ for item in itertools.count(start, step): # pragma: no branch if stop is not None and item >= stop: break @@ -31,7 +58,8 @@ async def acontainer( iterable: collections.abc.AsyncIterable[_T] | collections.abc.Callable[..., collections.abc.AsyncIterable[_T]], container: type[tuple[_T, ...]], -) -> tuple[_T, ...]: ... +) -> tuple[_T, ...]: + """Overload: collect the async iterable into a ``tuple``.""" @typing.overload @@ -39,7 +67,8 @@ async def acontainer( iterable: collections.abc.AsyncIterable[_T] | collections.abc.Callable[..., collections.abc.AsyncIterable[_T]], container: type[list[_T]] = list, -) -> list[_T]: ... +) -> list[_T]: + """Overload: collect the async iterable into a ``list`` (the default).""" @typing.overload @@ -47,7 +76,8 @@ async def acontainer( iterable: collections.abc.AsyncIterable[_T] | collections.abc.Callable[..., collections.abc.AsyncIterable[_T]], container: type[set[_T]], -) -> set[_T]: ... +) -> set[_T]: + """Overload: collect the async iterable into a ``set``.""" async def acontainer( diff --git a/python_utils/containers.py b/python_utils/containers.py index c2789c4..9318d95 100644 --- a/python_utils/containers.py +++ b/python_utils/containers.py @@ -1,7 +1,8 @@ """ This module provides custom container classes with enhanced functionality. -Classes: +Classes:: + CastedDictBase: Abstract base class for dictionaries that cast keys and values. CastedDict: Dictionary that casts keys and values to specified types. @@ -11,7 +12,8 @@ on duplicates. SliceableDeque: Deque that supports slicing and enhanced equality checks. -Type Aliases: +Type Aliases:: + KT: Type variable for dictionary keys. VT: Type variable for dictionary values. DT: Type alias for a dictionary with keys of type KT and values of type VT. @@ -23,7 +25,8 @@ dictionary. OnDuplicate: Literal type for handling duplicate values in UniqueList. -Usage: +Usage:: + - CastedDict and LazyCastedDict can be used to create dictionaries with automatic type casting. - UniqueList ensures all elements are unique and can raise an error on @@ -78,6 +81,8 @@ #: A type alias for a regular generic type T = typing.TypeVar('T') +#: Argument shapes accepted when updating a casted dict: a mapping, an iterable +#: of key/value pairs, an iterable of mappings, or a keys-and-getitem object. # Kept as `typing.Union` (not PEP 604 `|`): one member is a string forward # reference, and `|` evaluates its operands eagerly, raising `TypeError` on a # `str` operand at runtime. `typing.Union` accepts it as a lazy `ForwardRef`. @@ -88,6 +93,7 @@ '_typeshed.SupportsKeysAndGetItem[KT, VT]', ] +#: Policy for ``UniqueList`` duplicates: silently ``'ignore'`` or ``'raise'``. OnDuplicate = typing.Literal['ignore', 'raise'] @@ -99,7 +105,8 @@ class CastedDictBase(dict[KT, VT], abc.ABC): _key_cast (KT_cast[KT]): Callable to cast dictionary keys. _value_cast (VT_cast[VT]): Callable to cast dictionary values. - Methods: + Methods:: + __init__(key_cast: KT_cast[KT] = None, value_cast: VT_cast[VT] = None, *args: DictUpdateArgs[KT, VT], **kwargs: VT) -> None: Initializes the dictionary with optional key and value casting @@ -211,7 +218,11 @@ class CastedDict(CastedDictBase[KT, VT]): """ def __setitem__(self, key: typing.Any, value: typing.Any) -> None: - """Sets `key` to `cast(value)` in the dictionary.""" + """Cast ``value`` (if a value cast is set) and store it under ``key``. + + The key itself is cast by ``CastedDictBase.__setitem__`` when a key + cast is configured. + """ if self._value_cast is not None: value = self._value_cast(value) @@ -441,14 +452,14 @@ def __contains__(self, item: HT) -> bool: # type: ignore[override] return item in self._set @typing.overload - def __setitem__( - self, indices: typing.SupportsIndex, values: HT - ) -> None: ... + def __setitem__(self, indices: typing.SupportsIndex, values: HT) -> None: + """Overload: assign a single value at an integer index.""" @typing.overload def __setitem__( self, indices: slice, values: collections.abc.Iterable[HT] - ) -> None: ... + ) -> None: + """Overload: assign an iterable of values to a slice.""" def __setitem__( self, @@ -518,7 +529,8 @@ class SliceableDeque(typing.Generic[T], collections.deque[T]): """ A deque that supports slicing and enhanced equality checks. - Methods: + Methods:: + __getitem__(index: typing.SupportsIndex | slice) -> T | 'SliceableDeque[T]': Returns the item or slice at the given index. @@ -531,10 +543,12 @@ class SliceableDeque(typing.Generic[T], collections.deque[T]): """ @typing.overload - def __getitem__(self, index: typing.SupportsIndex) -> T: ... + def __getitem__(self, index: typing.SupportsIndex) -> T: + """Overload: an integer index returns a single item.""" @typing.overload - def __getitem__(self, index: slice) -> 'SliceableDeque[T]': ... + def __getitem__(self, index: slice) -> 'SliceableDeque[T]': + """Overload: a slice returns a new ``SliceableDeque``.""" def __getitem__( self, index: typing.SupportsIndex | slice diff --git a/python_utils/converters.py b/python_utils/converters.py index 37dca0c..ebe57ce 100644 --- a/python_utils/converters.py +++ b/python_utils/converters.py @@ -22,8 +22,11 @@ from python_utils import _aliases +#: Numeric type variable for ``remap`` (any ``int``, ``float`` or ``Decimal``). _TN = typing.TypeVar('_TN', bound=_aliases.DecimalNumber) +#: Accepted ``regexp`` for ``to_int``/``to_float``: a compiled pattern, a +#: pattern string, ``True`` for the built-in digit pattern, or ``None``. _RegexpType: typing.TypeAlias = ( re.Pattern[str] | str | typing.Literal[True] | None ) @@ -287,7 +290,8 @@ def remap( old_max: decimal.Decimal | float, new_min: decimal.Decimal | float, new_max: decimal.Decimal | float, -) -> decimal.Decimal: ... +) -> decimal.Decimal: + """Overload: a ``Decimal`` ``value`` yields a ``Decimal`` result.""" @typing.overload @@ -297,7 +301,8 @@ def remap( old_max: decimal.Decimal | float, new_min: decimal.Decimal | float, new_max: decimal.Decimal | float, -) -> decimal.Decimal: ... +) -> decimal.Decimal: + """Overload: a ``Decimal`` ``old_min`` yields a ``Decimal`` result.""" @typing.overload @@ -307,7 +312,8 @@ def remap( old_max: decimal.Decimal, new_min: decimal.Decimal | float, new_max: decimal.Decimal | float, -) -> decimal.Decimal: ... +) -> decimal.Decimal: + """Overload: a ``Decimal`` ``old_max`` yields a ``Decimal`` result.""" @typing.overload @@ -317,7 +323,8 @@ def remap( old_max: decimal.Decimal | float, new_min: decimal.Decimal, new_max: decimal.Decimal | float, -) -> decimal.Decimal: ... +) -> decimal.Decimal: + """Overload: a ``Decimal`` ``new_min`` yields a ``Decimal`` result.""" @typing.overload @@ -327,7 +334,8 @@ def remap( old_max: decimal.Decimal | float, new_min: decimal.Decimal | float, new_max: decimal.Decimal, -) -> decimal.Decimal: ... +) -> decimal.Decimal: + """Overload: a ``Decimal`` ``new_max`` yields a ``Decimal`` result.""" # Note that float captures both int and float types so we don't need to @@ -339,7 +347,8 @@ def remap( old_max: float, new_min: float, new_max: float, -) -> float: ... +) -> float: + """Overload: all-``float`` (or ``int``) inputs yield a ``float``.""" def remap( # pyright: ignore[reportInconsistentOverload] @@ -365,7 +374,7 @@ def remap( # pyright: ignore[reportInconsistentOverload] This is a great use case example. Take an AVR that has dB values the minimum being -80dB and the maximum being 10dB and you want to convert - volume percent to the equilivint in that dB range + volume percent to the equivalent in that dB range >>> remap(46.0, 0.0, 100.0, -80.0, 10.0) -38.6 @@ -414,6 +423,9 @@ def remap( # pyright: ignore[reportInconsistentOverload] passed parameters are a `float`, otherwise the returned type will be `int`. """ + # Promote every argument to one common type: Decimal if any input is a + # Decimal, otherwise float if any is a float, otherwise int. This preserves + # the caller's precision (see above) instead of silently downcasting. type_: type[_aliases.DecimalNumber] if ( isinstance(value, decimal.Decimal) @@ -456,6 +468,8 @@ def remap( # pyright: ignore[reportInconsistentOverload] # worth it. new_value = (value - old_min) * new_range # type: ignore[operator] # pyright: ignore[reportOperatorIssue, reportUnknownVariableType] + # Integer inputs use floor division to keep the result integral; float and + # Decimal inputs use true division. if type_ is int: new_value //= old_range # pyright: ignore[reportUnknownVariableType] # pyrefly: ignore[unsupported-operation] else: diff --git a/python_utils/decorators.py b/python_utils/decorators.py index ff4e8b6..9bd81bf 100644 --- a/python_utils/decorators.py +++ b/python_utils/decorators.py @@ -54,6 +54,7 @@ def set_attributes( def _set_attributes( function: collections.abc.Callable[_P, _T], ) -> collections.abc.Callable[_P, _T]: + """Attach the captured ``kwargs`` as attributes on ``function``.""" for key, value in kwargs.items(): setattr(function, key, value) return function @@ -121,9 +122,12 @@ def _listify( ..., collections.abc.Iterable[_T] | None ], ) -> collections.abc.Callable[..., collections.abc.Collection[_T]]: + """Materialize ``function``'s result into ``collection``.""" + def __listify( *args: typing.Any, **kwargs: typing.Any ) -> collections.abc.Collection[_T]: + """Call ``function`` and gather its result into ``collection``.""" result: collections.abc.Iterable[_T] | None = function( *args, **kwargs ) @@ -166,8 +170,11 @@ def sample( def _sample( function: collections.abc.Callable[_P, _T], ) -> collections.abc.Callable[_P, _T | None]: + """Wrap ``function`` so it only runs on a sampled fraction of calls.""" + @functools.wraps(function) def __sample(*args: _P.args, **kwargs: _P.kwargs) -> _T | None: + """Run ``function`` with probability ``sample_rate``, else skip.""" if random.random() < sample_rate: return function(*args, **kwargs) else: @@ -192,9 +199,17 @@ def wraps_classmethod( ], collections.abc.Callable[typing.Concatenate[typing.Any, _P], _T], ]: - """ - Like `functools.wraps`, but for wrapping classmethods with the type info - from a regular method. + """Like ``functools.wraps``, but for wrapping classmethods. + + Copies the wrapped method's metadata (name, docstring and annotations) onto + the wrapper, so a classmethod wrapper carries the type information of the + regular method it stands in for. + + Args: + wrapped: The method whose metadata should be copied onto the wrapper. + + Returns: + A decorator that updates its wrapper with ``wrapped``'s metadata. """ def _wraps_classmethod( @@ -202,6 +217,7 @@ def _wraps_classmethod( typing.Concatenate[typing.Any, _P], _T ], ) -> collections.abc.Callable[typing.Concatenate[typing.Any, _P], _T]: + """Copy ``wrapped``'s metadata onto ``wrapper`` and return it.""" # For some reason `functools.update_wrapper` fails on some test # runs but not while running actual code with contextlib.suppress(AttributeError): @@ -215,6 +231,7 @@ def _wraps_classmethod( ), ) if annotations := getattr(wrapped, '__annotations__', {}): + # Drop `self`: the wrapper is a classmethod, so it takes no `self`. annotations.pop('self', None) wrapper.__annotations__ = annotations diff --git a/python_utils/exceptions.py b/python_utils/exceptions.py index 41ec4f5..11feaeb 100644 --- a/python_utils/exceptions.py +++ b/python_utils/exceptions.py @@ -1,7 +1,8 @@ """ This module provides utility functions for raising and reraising exceptions. -Functions: +Functions:: + raise_exception(exception_class, *args, **kwargs): Returns a function that raises an exception of the given type with the given arguments. @@ -30,6 +31,7 @@ def raise_exception( """ def raise_(*args_: typing.Any, **kwargs_: typing.Any) -> typing.Any: + """Raise ``exception_class`` with the captured args.""" raise exception_class(*args, **kwargs) return raise_ diff --git a/python_utils/generators.py b/python_utils/generators.py index eadc22c..20cd9c0 100644 --- a/python_utils/generators.py +++ b/python_utils/generators.py @@ -19,6 +19,7 @@ import python_utils from python_utils import _aliases +#: Element type of the iterables being batched. _T = typing.TypeVar('_T') diff --git a/python_utils/import_.py b/python_utils/import_.py index e1d0dc6..faed863 100644 --- a/python_utils/import_.py +++ b/python_utils/import_.py @@ -21,7 +21,7 @@ class DummyError(Exception): """A custom exception class used as a default for exception handling.""" -# Legacy alias for DummyError +#: Backwards-compatible legacy alias for ``DummyError``. DummyException = DummyError diff --git a/python_utils/logger.py b/python_utils/logger.py index 0c13bf0..f231290 100644 --- a/python_utils/logger.py +++ b/python_utils/logger.py @@ -40,6 +40,8 @@ __all__ = ['Logged'] +#: Accepted values for the logging ``exc_info`` parameter: a bool, an +#: ``(type, value, traceback)`` triple, a bare exception instance, or ``None``. # From the logging typeshed, converted to be compatible with Python 3.8 # https://github.com/python/typeshed/blob/main/stdlib/logging/__init__.pyi _ExcInfoType: typing.TypeAlias = ( @@ -53,11 +55,20 @@ | BaseException | None ) +#: Parameter specification capturing a wrapped logger method's arguments. _P = typing.ParamSpec('_P') +#: Covariant return-type variable for wrapped logger methods. _T = typing.TypeVar('_T', covariant=True) class LoggerProtocol(typing.Protocol): + """Structural type for any ``logging.Logger``-compatible logger. + + Objects that provide these methods (such as a ``logging.Logger`` or a + ``loguru`` logger) can be used as the ``logger`` attribute of a + ``LoggerBase`` subclass. Only the shape matters; no inheritance required. + """ + def debug( self, msg: object, @@ -66,7 +77,8 @@ def debug( stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, - ) -> None: ... + ) -> None: + """Log ``msg`` at ``DEBUG`` level.""" def info( self, @@ -76,7 +88,8 @@ def info( stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, - ) -> None: ... + ) -> None: + """Log ``msg`` at ``INFO`` level.""" def warning( self, @@ -86,7 +99,8 @@ def warning( stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, - ) -> None: ... + ) -> None: + """Log ``msg`` at ``WARNING`` level.""" def error( self, @@ -96,7 +110,8 @@ def error( stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, - ) -> None: ... + ) -> None: + """Log ``msg`` at ``ERROR`` level.""" def critical( self, @@ -106,7 +121,8 @@ def critical( stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, - ) -> None: ... + ) -> None: + """Log ``msg`` at ``CRITICAL`` level.""" def exception( self, @@ -116,7 +132,8 @@ def exception( stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, - ) -> None: ... + ) -> None: + """Log ``msg`` at ``ERROR`` level, including exception information.""" def log( self, @@ -127,7 +144,8 @@ def log( stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, - ) -> None: ... + ) -> None: + """Log ``msg`` at the integer ``level``.""" class LoggerBase(abc.ABC): @@ -161,6 +179,7 @@ class LoggerBase(abc.ABC): def __get_name( # pyright: ignore[reportUnusedFunction] cls, *name_parts: str ) -> str: + """Join the non-empty, stripped ``name_parts`` into a dotted name.""" return '.'.join(n.strip() for n in name_parts if n.strip()) @decorators.wraps_classmethod(logging.Logger.debug) @@ -174,6 +193,7 @@ def debug( stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: + """Log ``msg`` at ``DEBUG`` level on the class logger.""" return cls.logger.debug( # type: ignore[no-any-return] msg, *args, @@ -194,6 +214,7 @@ def info( stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: + """Log ``msg`` at ``INFO`` level on the class logger.""" return cls.logger.info( # type: ignore[no-any-return] msg, *args, @@ -214,6 +235,7 @@ def warning( stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: + """Log ``msg`` at ``WARNING`` level on the class logger.""" return cls.logger.warning( # type: ignore[no-any-return] msg, *args, @@ -234,6 +256,7 @@ def error( stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: + """Log ``msg`` at ``ERROR`` level on the class logger.""" return cls.logger.error( # type: ignore[no-any-return] msg, *args, @@ -254,6 +277,7 @@ def critical( stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: + """Log ``msg`` at ``CRITICAL`` level on the class logger.""" return cls.logger.critical( # type: ignore[no-any-return] msg, *args, @@ -274,6 +298,7 @@ def exception( stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: + """Log ``msg`` at ``ERROR`` level with exception info attached.""" return cls.logger.exception( # type: ignore[no-any-return] msg, *args, @@ -295,6 +320,7 @@ def log( stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: + """Log ``msg`` at the integer ``level`` on the class logger.""" return cls.logger.log( # type: ignore[no-any-return] level, msg, @@ -332,6 +358,7 @@ class Logged(LoggerBase): @classmethod def __get_name(cls, *name_parts: str) -> str: + """Build the dotted logger name via ``LoggerBase``'s joiner.""" return typing.cast( str, LoggerBase._LoggerBase__get_name(*name_parts), # type: ignore[attr-defined] # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType, reportAttributeAccessIssue] diff --git a/python_utils/terminal.py b/python_utils/terminal.py index 554f176..31d21dd 100644 --- a/python_utils/terminal.py +++ b/python_utils/terminal.py @@ -20,9 +20,13 @@ from . import converters +#: A terminal size as ``(width, height)`` in character cells. Dimensions = tuple[int, int] +#: A ``Dimensions`` tuple, or ``None`` if the size could not be determined. OptionalDimensions = Dimensions | None +#: A raw terminal size as ``(width, height)`` strings, before ``int`` casting. _StrDimensions = tuple[str, str] +#: A ``_StrDimensions`` tuple, or ``None`` if the size could not be read. _OptionalStrDimensions = _StrDimensions | None @@ -59,11 +63,13 @@ def get_terminal_size() -> Dimensions: # pragma: no cover # safety we'll always subtract it. return w - 1, h with contextlib.suppress(Exception): + # Fall back to the COLUMNS/LINES environment variables when set. w = converters.to_int(os.environ.get('COLUMNS')) h = converters.to_int(os.environ.get('LINES')) if w and h: return w, h with contextlib.suppress(Exception): + # Try the optional `blessings` library if it happens to be installed. import blessings # type: ignore[import-untyped] terminal = blessings.Terminal() @@ -93,6 +99,11 @@ def get_terminal_size() -> Dimensions: # pragma: no cover def _get_terminal_size_windows() -> OptionalDimensions: # pragma: no cover + """Return the terminal size on Windows via the Win32 console API. + + Returns: + The ``(width, height)`` in cells, or ``None`` if it cannot be read. + """ res = None try: import ctypes @@ -121,6 +132,13 @@ def _get_terminal_size_windows() -> OptionalDimensions: # pragma: no cover def _get_terminal_size_tput() -> OptionalDimensions: # pragma: no cover + """Return the terminal size by shelling out to ``tput``. + + This is mainly needed for Windows Python running under Cygwin's xterm. + + Returns: + The ``(width, height)`` in cells, or ``None`` on any failure. + """ # get terminal width src: http://stackoverflow.com/questions/263890/ try: import subprocess @@ -148,7 +166,21 @@ def _get_terminal_size_tput() -> OptionalDimensions: # pragma: no cover def _get_terminal_size_linux() -> OptionalDimensions: # pragma: no cover + """Return the terminal size on Unix-likes via ``ioctl`` or the environment. + + Tries ``TIOCGWINSZ`` on the standard file descriptors, then the controlling + terminal, then the ``LINES``/``COLUMNS`` environment variables. + + Returns: + The ``(width, height)`` in cells, or ``None`` if every method fails. + """ + def ioctl_gwinsz(fd: int) -> tuple[str, str] | None: + """Query ``fd`` for its window size via the ``TIOCGWINSZ`` ioctl. + + Returns: + The ``(rows, cols)`` window size, or ``None`` if the ioctl fails. + """ try: import fcntl import struct diff --git a/python_utils/time.py b/python_utils/time.py index 3df35c5..2ec037f 100644 --- a/python_utils/time.py +++ b/python_utils/time.py @@ -26,12 +26,15 @@ import python_utils from python_utils import _aliases, exceptions +#: Item type produced by the time/timeout generators. _T = typing.TypeVar('_T') +#: Parameter specification for the timeout-detector decorator's target. _P = typing.ParamSpec('_P') -# There might be a better way to get the epoch with tzinfo, please create -# a pull request if you know a better way that functions for Python 2 and 3 +#: The Unix epoch (1970-01-01) as a naive ``datetime``, used as a reference. +# There might be a better way to get the epoch with tzinfo; please open a +# pull request if you know one. epoch = datetime.datetime(year=1970, month=1, day=1) @@ -87,7 +90,19 @@ def delta_to_seconds(interval: _aliases.delta_type) -> _aliases.Number: def delta_to_seconds_or_none( interval: _aliases.delta_type | None, ) -> _aliases.Number | None: - """Convert a timedelta to seconds or return None.""" + """Convert a timedelta to seconds, passing ``None`` through unchanged. + + Args: + interval: A timedelta or a number of seconds, or ``None``. + + Returns: + The interval in seconds, or ``None`` when ``interval`` is ``None``. + + >>> delta_to_seconds_or_none(datetime.timedelta(seconds=2)) + 2 + >>> delta_to_seconds_or_none(None) is None + True + """ if interval is None: return None else: @@ -164,14 +179,16 @@ def format_time( def _to_iterable( iterable: collections.abc.Callable[[], collections.abc.AsyncIterable[_T]] | collections.abc.AsyncIterable[_T], -) -> collections.abc.AsyncIterable[_T]: ... +) -> collections.abc.AsyncIterable[_T]: + """Async overload: async iterable or factory in, async iterable out.""" @typing.overload def _to_iterable( iterable: collections.abc.Callable[[], collections.abc.Iterable[_T]] | collections.abc.Iterable[_T], -) -> collections.abc.Iterable[_T]: ... +) -> collections.abc.Iterable[_T]: + """Sync overload: sync iterable or factory in, sync iterable out.""" def _to_iterable( @@ -180,6 +197,7 @@ def _to_iterable( | collections.abc.AsyncIterable[_T] | collections.abc.Callable[[], collections.abc.AsyncIterable[_T]], ) -> collections.abc.Iterable[_T] | collections.abc.AsyncIterable[_T]: + """Return ``iterable``, calling it first if it is a zero-arg callable.""" if callable(iterable): return iterable() else: @@ -390,7 +408,19 @@ def aio_generator_timeout_detector_decorator( [collections.abc.Callable[_P, collections.abc.AsyncGenerator[_T, None]]], collections.abc.Callable[_P, collections.abc.AsyncGenerator[_T, None]], ]: - """A decorator wrapper for aio_generator_timeout_detector.""" + """Wrap a generator function with ``aio_generator_timeout_detector``. + + Args: + timeout: Per-item timeout; if a single yield takes longer, + ``on_timeout`` fires. ``None`` disables the per-item check. + total_timeout: Overall timeout across the whole generator. + on_timeout: Callback invoked on a timeout; defaults to re-raising. + **on_timeout_kwargs: Extra keyword arguments passed to ``on_timeout``. + + Returns: + A decorator that wraps an async-generator function so every call is + guarded against stalls. + """ def _timeout_detector_decorator( generator: collections.abc.Callable[ @@ -399,13 +429,14 @@ def _timeout_detector_decorator( ) -> collections.abc.Callable[ _P, collections.abc.AsyncGenerator[_T, None] ]: - """The decorator itself.""" + """Wrap ``generator`` so each call is timeout-guarded.""" @functools.wraps(generator) def wrapper( *args: _P.args, **kwargs: _P.kwargs, ) -> collections.abc.AsyncGenerator[_T, None]: + """Forward the call to ``aio_generator_timeout_detector``.""" return aio_generator_timeout_detector( generator(*args, **kwargs), timeout,