Merge pull request #523 from WilliamK112/codex/docs-imputation-strategies-513

docs(cookbook): choose and audit imputation strategies

Author: JohnnyWilson16

docs/cookbook/imputation-strategies-guide.md

@@ -0,0 +1,268 @@
+---
+title: Choose and audit missing-value imputation
+description: Compare simple and MissForest imputation, configure column rules, protect declared roles, and inspect actual repairs.
+---
+
+# Choose and audit missing-value imputation
+
+Missing values need a decision: retain the gap, fill it from the observed
+column, or estimate it from other features. An imputed value is an estimate,
+not a recovered observation. Start with the simplest strategy that fits your
+data and inspect its effects before using the result downstream.
+
+## Choose a strategy
+
+| `impute` value | Behavior | Tradeoff |
+| --- | --- | --- |
+| `"mean"` | Fill numeric columns with their observed mean. Non-numeric columns are skipped. | Sensitive to extreme values; repeated fills reduce variation. |
+| `"median"` | Fill numeric columns with their observed median. Non-numeric columns are skipped. | More resistant to extreme values, but still a constant estimate. |
+| `"mode"` | Fill from the most frequent observed value, including text columns. | Can reinforce the majority category; inspect ties and rare categories. |
+| `"auto"` | Use median for numeric columns and mode for other columns. | A simple per-column rule, not model selection. |
+| `"missforest"` | Use iterative random-forest regression/classification with other columns as predictors. | Ineligible data can take an audited simple fallback; model-eligible data requires the ML dependencies. |
+
+Simple imputation uses only the observed values of the column being filled.
+FreshData's MissForest-inspired implementation initializes gaps, fits a model
+for each eligible column, and iterates up to `missforest_max_iter`, stopping
+earlier if it converges. More complexity does not guarantee better estimates.
+Compare choices on representative data using validation suited to your task.
+
+`strategy` controls the broader cleaning engine; `impute` and `impute_strategy`
+are explicit missing-value overrides:
+
+| Combination | What to expect |
+| --- | --- |
+| `strategy="conservative"`, `impute=None` | Representation repairs; no decision-engine imputation. |
+| `strategy="balanced"`, `impute=None` | Accuracy-first decisions based on column roles, missingness, and available evidence. Gaps may remain. |
+| `strategy="aggressive"`, `impute=None` | More active filling and column-drop decisions, with default outlier flagging. Capping requires `outlier_action="cap"`. |
+| Any strategy with an explicit `impute` choice | Apply that override to eligible columns; declared ID/target protection still applies. |
+| Any strategy with `impute_strategy={...}` | Rules for the named columns override the global `impute` choice. |
+
+The examples use `"conservative"` to make explicit imputation easy to inspect.
+Other strategies can make additional decisions; none guarantees that every
+gap will be filled. See the [cleaning engine](../cleaning-engine.md).
+
+## Apply global and per-column rules
+
+Fill age with its median and group with its mode. Deliberately request fills
+for declared ID and target columns to verify that those requests are ignored
+and audited. One age and one group are missing at `row-c`.
+
+```python
+import pandas as pd
+import freshdata as fd
+
+frame = pd.DataFrame(
+    {
+        "record_key": ["A", None, "C", "D", "E"],
+        "age": [20.0, 30.0, None, 40.0, 90.0],
+        "group": ["red", "blue", None, "red", "red"],
+        "outcome": [1.0, 0.0, 1.0, None, 0.0],
+    },
+    index=["row-a", "row-b", "row-c", "row-d", "row-e"],
+)
+
+cleaned, report = fd.clean(
+    frame,
+    strategy="conservative",
+    impute="median",                  # global numeric rule
+    impute_strategy={
+        "group": "mode",              # overrides the global rule
+        "record_key": "mode",         # ignored: declared identifier
+        "outcome": "mean",            # ignored: declared target
+    },
+    id_columns=("record_key",),
+    target_column="outcome",
+    fix_dtypes=False,
+    drop_empty_rows=False,
+    drop_empty_columns=False,
+    return_report=True,
+    verbose=False,
+)
+
+assert cleaned.loc["row-c", "age"] == 35.0
+assert cleaned.loc["row-c", "group"] == "red"
+pd.testing.assert_series_equal(cleaned["record_key"], frame["record_key"])
+pd.testing.assert_series_equal(cleaned["outcome"], frame["outcome"])
+print(cleaned[["age", "group"]])
+print(report.warnings)
+```
+
+The warnings identify ignored rules for `record_key` and `outcome`. Declare
+important roles explicitly: explicit simple imputation protects `id_columns`
+and `target_column`, so do not rely on name heuristics to protect arbitrary
+keys or labels. Context-protected columns also remain untouched. All-missing
+columns have no observed fill value, and numeric rules do not apply to text;
+those gaps can remain.
+
+Use the post-normalization names for `impute_strategy`, `id_columns`, and
+`target_column` when the default `column_names=True` renames input headers.
+For `"Customer Age"`, use `"customer_age"`. This is essential for MissForest:
+its role gate matches the normalized column name exactly; declaring a custom
+target under its original spaced name can leave it unprotected. Alternatively,
+use `column_names=False` and declarations matching the unchanged headers.
+Configure these options through `fd.clean` keywords or `fd.CleanConfig`.
+
+## Inspect the audit and the changed cells
+
+Continue with `frame`, `cleaned`, and `report` from the previous example:
+
+```python
+for action in report.actions:
+    if action.step == "impute":
+        print(action.column, action.count, action.description, action.metadata)
+
+# Actions report a column and affected count, not every cell's location.
+# This example keeps the index, columns, and row order, so the frames align.
+pd.testing.assert_index_equal(cleaned.index, frame.index)
+assert all(column in cleaned.columns for column in frame.columns)
+filled = frame.isna() & cleaned[frame.columns].notna()
+repairs = [
+    {
+        "row_position": position,
+        "row_label": frame.index[position],
+        "column": column,
+        "value": cleaned[column].iloc[position],
+    }
+    for column in frame.columns
+    for position in range(len(frame))
+    if filled[column].iloc[position]
+]
+print(repairs)
+assert [(item["row_label"], item["column"]) for item in repairs] == [
+    ("row-c", "age"),
+    ("row-c", "group"),
+]
+
+# Persist the structured aggregate audit when you need a portable record.
+audit_json = report.to_json()
+```
+
+The filled cells are age `35.0` and group `"red"` at `row-c`; ID and target gaps
+remain. Imputation actions report one affected cell per fill and zero-count
+notes for protected columns. `CleanReport` also captures warnings and overall
+before/after missing counts. `columns_imputed` is a decision-engine summary;
+use the actual `impute` actions for explicit overrides.
+
+This comparison identifies missing-to-present repairs. If your plan drops or
+reorders rows or renames columns, align the frames using a stable record
+identity first. It is not a history of every non-missing value change. See
+[audit automated cleaning](audit-automated-data-cleaning.md) and the
+[audit schema](../audit-trail.md) for supported report fields.
+
+## Try model-based imputation
+
+Install the optional ML dependencies:
+
+```bash
+pip install "freshdata-cleaner[ml]"
+```
+
+This separate example has 80 rows and related numeric/categorical predictors.
+It fills eight missing charges with a small forest and at most two iterations.
+No identifier or target column is included in this model example.
+
+```python
+import pandas as pd
+import freshdata as fd
+
+training = pd.DataFrame(
+    {
+        "hours": [float(i % 12) for i in range(80)],
+        "charges": [10 + 2 * (i % 12) + 0.5 * (i % 5) for i in range(80)],
+        "kind": ["standard", "priority"] * 40,
+    }
+)
+training.loc[training.index[::11], "charges"] = float("nan")
+
+estimated, model_report = fd.clean(
+    training,
+    strategy="conservative",
+    impute_strategy={"charges": "missforest"},
+    missforest_n_estimators=10,
+    missforest_max_iter=2,
+    missforest_random_state=42,
+    missforest_add_indicators=False,
+    fix_dtypes=False,
+    return_report=True,
+    verbose=False,
+)
+
+assert estimated["charges"].notna().all()
+for action in model_report.actions:
+    if action.step == "impute" and action.column == "charges":
+        print(action.count, action.metadata)
+        assert action.count == 8
+```
+
+Inspect `selected_model_type`, `iterations`, `converged`, and `fallback_reason`
+in action metadata. With ML dependencies installed, this example reports
+`selected_model_type="regressor"` and `fallback_reason=None`. Reaching the
+iteration limit is recorded separately from convergence. Numerical estimates
+and model diagnostics can vary with dependency versions.
+
+`missforest_min_rows_for_model` defaults to 50. Smaller inputs take a recorded
+simple-imputation fallback. Use the earlier five-row `frame`:
+
+```python
+small, small_report = fd.clean(
+    frame,
+    strategy="conservative",
+    impute_strategy={"age": "missforest"},
+    id_columns=("record_key",),
+    target_column="outcome",
+    missforest_add_indicators=False,
+    fix_dtypes=False,
+    drop_empty_rows=False,
+    drop_empty_columns=False,
+    return_report=True,
+    verbose=False,
+)
+
+assert small.loc["row-c", "age"] == 35.0
+fallback_actions = [
+    action for action in small_report.actions
+    if action.step == "impute" and action.column == "age"
+]
+assert len(fallback_actions) == 1
+print(fallback_actions[0].metadata["fallback_reason"])
+assert "fewer rows" in fallback_actions[0].metadata["fallback_reason"]
+```
+
+Insufficient predictors and ineligible columns can also lead to fallback or
+preservation; inspect the audit rather than assuming a requested model ran.
+If a column is model-eligible but scikit-learn is absent, cleaning raises
+`ImportError` with the ML installation hint. No result or report is returned
+for that failed call; install `freshdata-cleaner[ml]` before retrying.
+
+The following example protects a custom target whose input header is renamed.
+The declaration and imputation rule both use its post-normalization name:
+
+```python
+raw = pd.DataFrame(
+    {
+        "Observed Hours": [float(i % 12) for i in range(80)],
+        "Future Sales": [10 + 2 * (i % 12) + 0.5 * (i % 5) for i in range(80)],
+    }
+)
+raw.loc[raw.index[::11], "Future Sales"] = float("nan")
+
+protected, protected_report = fd.clean(
+    raw,
+    strategy="conservative",
+    impute_strategy={"future_sales": "missforest"},
+    target_column="future_sales",  # matches after header normalization
+    fix_dtypes=False,
+    return_report=True,
+    verbose=False,
+)
+
+pd.testing.assert_series_equal(
+    protected["future_sales"], raw["Future Sales"].rename("future_sales")
+)
+assert protected["future_sales"].isna().sum() == 8
+```
+
+With matching declarations, ID and target gaps are preserved. Role protection
+is not a train/test leakage guarantee: cleaning learns from the supplied frame.
+Isolate training data and validate your downstream workflow before estimating
+values in evaluation data.

docs/cookbook/index.md

@@ -24,6 +24,10 @@ Every recipe is self-contained with runnable code, realistic sample outputs, and
     Role-aware imputation: why identifiers and targets must be preserved while numerics are safely imputed.
     *(Search intent: safe missing value imputation, fillna without corruption)*
 
+- :material-help-box: **[Choose and audit missing-value imputation](imputation-strategies-guide.md)**
+    Compare simple and model-based fills, declare protected roles, and inspect actual repairs.
+    *(Search intent: mean median mode imputation, missforest, per-column imputation)*
+
 - :material-content-copy: **[How to detect and resolve duplicate rows](detect-and-resolve-duplicates.md)**
     Report duplicate clusters without silent row drops, and apply controlled deduplication policies.
     *(Search intent: detect duplicate rows pandas, safe deduplication python)*

docs/fallback-matrix.md

@@ -84,7 +84,6 @@ from pprint import pprint
 
 import polars as pl
 import freshdata as fd
-from freshdata.execution import PlanGenerator
 
 frame = pl.DataFrame({"value": [1, 2]})
 backend = "polars"

mkdocs.yml

@@ -146,6 +146,7 @@ nav:
       - Overview: cookbook/index.md
       - Clean pandas DataFrame: cookbook/clean-pandas-dataframe.md
       - Safely handle missing values: cookbook/safely-handle-missing-values.md
+      - Imputation strategies: cookbook/imputation-strategies-guide.md
       - Detect & resolve duplicates: cookbook/detect-and-resolve-duplicates.md
       - Prepare data for ML: cookbook/prepare-data-for-machine-learning.md
       - Clean Polars DataFrames: cookbook/clean-polars-dataframes.md