ml-01-intro

Comparing train/test split sizes on a small regression dataset.

Project Description

This project is about characterizing machine learning problems: taking a dataset, choosing a target, and deciding what kind of ML question it represents, supervised or unsupervised, classification or regression. It contains two pieces of work, Phase 4 and Phase 5.

Phase 4 is a technical modification of a supervised regression example that uses a small student-performance dataset, where the target is a continuous score predicted from study habits. I modified the example app to train three models using different test sizes (0.2, 0.3, and 0.4) and compared how the train/test split affected the results on that small, 10-row dataset. See the Findings and Visuals section below.

Phase 5 is a custom notebook that applies the same characterization skills to a much larger health dataset. See the My Notebook section below.

My Notebook

This notebook loads diabetes.csv, a health survey with about 70,000 instances and 22 columns, and uses Diabetes_binary as the target. It is a supervised problem, and it should be classification, since the target is a yes/no value, whether a person has diabetes or not.

Every column is stored as a number (1.0 or 0.0), so the notebook's automatic check labeled the problem as regression and called every feature numeric, even though several are really categories. The way the data is stored does not match what it means. My full write-up is in the project documentation below.

• ml_01_gracetulsi.ipynb

Working Files

You'll work with these areas:

• data/raw - raw data for exploration (only if you add a dataset)
• docs/ - project narrative and documentation
• src/mlstudio/ - the app is an example; run only (no need to modify)
• notebooks/ - interactive analysis
• pyproject.toml - update authorship & links
• zensical.toml - update authorship & links

Command Reference

Show command reference

Clone and open

git clone https://github.com/gracecode42/ml-01-intro
cd ml-01-intro
code .

Set up the environment

uv self update
uv python pin 3.14
uv lock --upgrade
uv sync --extra dev --extra docs --upgrade

uvx pre-commit install
uvx pre-commit autoupdate

# git add all files and auto fix them as much as possible while working
git add -A
uvx pre-commit run --all-files
# repeat if changes were made
uvx pre-commit run --all-files

Run my app

uv run python -m mlstudio.app_gracetulsi

Checks and docs

uv run ruff format .
uv run ruff check . --fix
uv run python -m pyright
uv run python -m pytest
uv run python -m zensical build

Save progress

git add -A
git commit -m "describe your change"
git push -u origin main

Findings and Visuals

For Phase 4, I changed the test size and ran three versions, 0.2, 0.3, and 0.4, to see how the split changed the results. I had no expectations for a dataset with only 10 rows.

Test size	Training rows	MAE	R-squared	Predicted score
0.2	8	0.63	1.00	83.5
0.3	7	0.48	1.00	83.4
0.4	6	1.30	0.99	84.1

The 0.4 version trained on 6 rows and had the highest error by far, about a 171% increase over the 0.3 version. Going from 7 training rows to 6, a change of a single instance, produced a much larger error. On this dataset, 6 instances was a small enough training set to make the error jump.

The guidance I started with is that a small dataset can do better with a larger test set like 0.3 or 0.4. I know sample size matters for regression, so fewer training rows means less for the model to learn from. At the same time, you also want a large enough test set to evaluate on, so a larger dataset could give better results on both sides. That raises a question I cannot answer from one small run: how would we even test this? One next step would be to run these same three test sizes on larger datasets and compare.

The coefficient charts changed the most between the three. The 0.4 chart, the one trained on 6 rows, looked very different from the other two, with much larger bars. I am still learning why the coefficients shift like that, but the chart changing so much alongside the jump in error stood out.

Two more observations.

The R-squared barely moved. It was 1.00, 1.00, and 0.99 across all three, even while the error nearly tripled. Looking only at R-squared, all three models look perfect. The MAE, which is in actual score points, is what showed a difference.

The prediction for one student stayed almost the same across all three, 83.5, 83.4, and 84.1, less than a point apart. The error and the charts moved, but the prediction for this one student did not.

With only 10 rows, I am not drawing conclusions about the best test size. What this run gave me was a clear look at how the results change when the training set shrinks, down to a single instance making a visible difference, and a set of questions to test on larger datasets. As a teacher, that is the part I like most. The dataset is small enough that the changes are easy to see, which makes it good for learning.

Project Documentation

Additional project notes and narrative:

docs/index.md

Citation

CITATION.cff

License

MIT