v0.8.2

What's Changed

• fix: multiword broken by @stephantul in #317
• feat: refactor training by @stephantul in #316
• fix: upgrade skeletoken by @stephantul in #318
• fix: Incorrect added token can cause issues when adding token as multiword token by @stephantul in #319
• fix: don't set weights for non-quantized models by @stephantul in #321
• fix: correctly set weights when w is initialized by @stephantul in #322
• feat: convert output to float in distill by @stephantul in #320
• clean up post-processing code by @stephantul in #323
• chore: fix pre-commit by @stephantul in #326
• chore: add exclude-newer by @stephantul in #328
• feat: add frozen-weights option to training by @stephantul in #327
• fix: small typing issue by @stephantul in #329
• chore: Update email by @Pringled in #331
• feat: add trusted publisher by @stephantul in #332

Full Changelog: v0.8.1...v0.8.2

v0.8.1

What's Changed

• fix: Use posix path for Windows compatibility by @stephantul in #314
• increase version to 0.8.1 by @stephantul in #315

Full Changelog: v0.8.0...v0.8.1

v0.8.0

What's Changed

• docs: Added CITATION.cff, updated citation info by @Pringled in #293
• docs: Added modelcard logo by @Pringled in #294
• docs: Update default model for example by @Pringled in #299
• chore: Deprecated Python 3.9 support by @Pringled in #301
• chore: Update CI to run on external PRs, fix test import by @Pringled in #303
• feat: replace tokenizer code with skeletoken by @stephantul in #302
• feat: refactor persistence, remove from_sentence_transformers by @stephantul in #305
• fix: refix tokenizer with added token shenanigans by @stephantul in #304
• fix: if layers == 0, layers were not initialized by @stephantul in #306
• feat: expose random seed, small typing fix by @stephantul in #308
• feat: add util for tokenizer pad id by @stephantul in #310
• docs: Update results to MTEB V2 by @Pringled in #311
• chore: Bump version by @Pringled in #312

Full Changelog: v0.7.0...v0.8.0

v0.7.0

What's Changed

• add support for passing weight to the loss functions by @volker48 in #260
• fix: padding token not recognized, update transformers by @stephantul in #265
• Fix tag train documentation by @Lhemamou in #269
• chore: Added python 3.13 to pyproject and CI by @Pringled in #270
• feat: add classifier freezing by @stephantul in #274
• fix: remove windows tests by @stephantul in #277
• feat: add configurable pad token by @stephantul in #276
• feat: faster loading if model already cached by @stephantul in #278
• feat: add vocabulary quantization by @stephantul in #271
• fix: load faster, make quantization better by @stephantul in #279
• fix: F rule, A rule, update ruff by @stephantul in #281
• feat: Added embedding_dtype and vocabulary_quantization to config by @Pringled in #280
• fix: Disable MPS for Torch versions >=2.8.0 by @Pringled in #287
• feat: Add configurable pooling for distillation by @Pringled in #288
• chore: Deprecate apply_zipf and use_subword parameters by @Pringled in #289
• chore: Rename PoolingType to PoolingMode by @Pringled in #290
• docs: Update main docs by @Pringled in #291
• chore: Bump version by @Pringled in #292

New Contributors

• @volker48 made their first contribution in #260
• @Lhemamou made their first contribution in #269

Deprecation warnings ⚠️

• apply_zipf and use_subword are now officially deprecated from distill

Full Changelog: v0.6.0...v0.7.0

0.6.0

What's Changed

• docs: update chonkie link on tutorial readme by @iaurg in #235
• Fix dates in README.md by @Pringled in #238
• fix: add default arg for push_to_hub by @stephantul in #240
• fix: remove direct dependency on specific hf utils by @stephantul in #244
• feat: smaller tokenizers by @stephantul in #243
• feat: update lock by @stephantul in #246
• feat: allow passing validation set explicitly by @JarbasAl in #245
• docs: Added multilingual results by @Pringled in #247
• fix: distillation for models without card by @JarbasAl in #248
• feat: add supertokenizers by @stephantul in #236
• clean-up print statement by @stephantul in #249
• fix: small typing issue by @stephantul in #250
• docs: Added new logo by @Pringled in #252
• fix: missing unk, fix bug by @stephantul in #251
• bump version by @stephantul in #258
• feat: make normalization dependent on spacing by @stephantul in #259

New Contributors

• @iaurg made their first contribution in #235
• @JarbasAl made their first contribution in #245

Full Changelog: v0.5.0...v0.6.0

v0.5.0

What's Changed

• fix: Updated semantic chunking tutorial by @bhavnicksm in #205
• rewrite backend by @stephantul in #207
• fix bibtex by @stephantul in #208
• feat: Added py.typed file by @Pringled in #214
• fix: pretokenize tokens before checking vocabulary by @stephantul in #215
• feat: add dimensionality during loading by @stephantul in #216
• feat: add quantization by @stephantul in #217
• feat: save load subfolder by @stephantul in #218
• feat: Added quantization for from_sentence_transformers by @Pringled in #219
• feat: faster inference for large vocab by @stephantul in #221
• feat: track token provenance by @stephantul in #222
• fix: typing issues, bug in infernece by @stephantul in #224
• fix: issues with unk and pad by @stephantul in #225
• bug: fix 0 score in evaluate by @stephantul in #226
• fix: precision during training by @stephantul in #228
• fix: issue with unk in unigram by @stephantul in #227
• docs: add info about quantization and dimensionality reduction by @stephantul in #231
• increment version by @stephantul in #232

New Contributors

• @bhavnicksm made their first contribution in #205

Full Changelog: 0.4.1...v0.5.0

0.4.1

What's Changed

• docs: Added training plot, added more training results by @Pringled in #189
• feat: Added min and max epochs to fit by @Pringled in #190
• docs: Update model card template by @Pringled in #192
• feat: Add multilabel classification for training by @Pringled in #191
• feat: Add evaluate function for classifiers by @Pringled in #195
• docs: Added discord badge by @Pringled in #193
• fix: only allows named args in pretrain by @stephantul in #200
• Bump version by @Pringled in #204

Full Changelog: 0.4.0...0.4.1

0.4.0

What's Changed

• Add fittable by @stephantul in #140
• fix scores in readme by @stephantul in #179
• docs: Refactored main docs, added separate docs directory, added training docs by @Pringled in #181
• docs: Update README.md by @Pringled in #183
• Update README.md by @Pringled in #184
• feat: replace 8m by 32m for training by @stephantul in #182
• docs: update scores in README by @stephantul in #186
• docs: Moved training results to results directory, updated docs and description by @Pringled in #187
• Bump version by @Pringled in #188

Full Changelog: v0.3.9...0.4.0

v0.3.9

What's Changed

• docs: Added new model results by @Pringled in #167
• docs: Update plot by @Pringled in #169
• feat: add trust-remote-code option by @stephantul in #173
• feat: Add SIF-like coef by @stephantul in #174
• increase version by @stephantul in #176

Full Changelog: v0.3.8...v0.3.9

v0.3.8

What's Changed

• docs: fix docstrings in distill by @stephantul in #157
• remove unnecessary import by @stephantul in #161
• remove deduplication tutorial by @stephantul in #159
• fix: issue with modernbert tokenizer, add token pattern to _distill by @stephantul in #158
• fix: fix typing issue by @stephantul in #162
• feat: float pca dims by @stephantul in #163
• feat: Add optional embedding normalization to StaticModel loading by @davidberenstein1957 in #164
• feat: Improve distill for modernBERT by @stephantul in #165
• increase version by @stephantul in #166

New Contributors

• @davidberenstein1957 made their first contribution in #164

Full Changelog: v0.3.7...v0.3.8