Repository navigation
Refresh the data science docs, and archive what they replace - #19
Conversation
The published pages described a pipeline two years out of date. The CS2 spec listed 30 channels; the pipeline has published 42 since 2026-08-04. The CS2 dataset page was a find-and-replace of the CS:GO page and still defined CS2 as "Counter-Strike: Global Offensive". The FAQ recommended Python 3.9 and Poetry against a tutorial that now requires 3.14 and uv, pointed at a companion tome dataset last updated in 2022, and said old data may be deleted "at our discretion" when revisions are in fact revoked on a one-year cycle. spec.md and csds_dictionary.csv are generated, not written: the column set, origin, dependents and merge keys come from csgo-ppp's generated reference, which is rendered from a real manifest. Types come from the published manifest where it declares one (227 columns), csgo-rushb's generated column docs (125), the arrow schema measured off 50 real prod matches (284), the same column name resolved elsewhere (118), and the converter source for the eight columns that postdate all of those. Four columns the manifest marks as deleted are excluded. The dataset page, the FAQ and the archive index rest on measurements taken against the production Data Exchange dataset on 2026-08-27: 365 live revisions covering 2025-07-18 onward, 941 revoked; 43 objects per match from 2026-08-04, 31 through 2026-07-31, a mix on 2026-08-02; about 35 MB per match, of which player_vector and player_status are 30 MB. The pages being replaced move under datascience/old/ rather than being deleted, because subscribers hold data that matches them and the retained window still includes revisions the old spec describes. The CS:GO pages keep their original addresses under /datascience/adx/csgo/ so published links still resolve.
There was a problem hiding this comment.
Pull request overview
Updates the Data Science documentation to reflect the current production CS2 pipeline (channel count, retention policy, tooling requirements, and cost guidance), while preserving the replaced 2022/2024-era pages under a dedicated archive so older retained revisions can still be interpreted.
Changes:
- Refreshes the main Data Science FAQ and the CS2 dataset docs (43 objects per match, ~1-year rolling retention, Python/uv tooling, updated cost guidance).
- Regenerates the CS2 spec/data dictionary and documents the Aug 2026 channel-set transition (30 → 42 channels).
- Adds an archived docs section (including CS:GO pages with pinned slugs) and updates sidebar/category ordering.
Reviewed changes
Copilot reviewed 17 out of 21 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/datascience/showcase/category.yml | Adds explicit sidebar ordering for the showcase section. |
| docs/datascience/README.md | Updates the Data Science FAQ to current CS2 dataset, retention, tooling, and cost guidance. |
| docs/datascience/old/README.md | Adds an archive index explaining what’s retained and what changed. |
| docs/datascience/old/datascience-faq.md | Archives the 2022 FAQ with an “Archived page” banner. |
| docs/datascience/old/csgo/csds/spec.md | Archives CS:GO spec with a banner and preserves original route via slug. |
| docs/datascience/old/csgo/csds/revisions.md | Archives CS:GO revision stats with a banner and preserves original route via slug. |
| docs/datascience/old/csgo/csds/README.md | Archives CS:GO dataset page with a banner and preserves original route via slug. |
| docs/datascience/old/csgo/csds/assets/csds_dictionary.csv | Adds archived CS:GO data dictionary asset. |
| docs/datascience/old/csgo/csds/category.yml | Adds sidebar category metadata for archived CS:GO CSDS docs. |
| docs/datascience/old/csgo/category.yml | Adds sidebar category metadata for archived CS:GO section. |
| docs/datascience/old/cs2/csds/spec.md | Archives the 2024 CS2 spec with an “Archived page” banner. |
| docs/datascience/old/cs2/csds/README.md | Archives the 2024 CS2 dataset page with an “Archived page” banner. |
| docs/datascience/old/cs2/csds/assets/csds_dictionary.csv | Adds archived CS2 data dictionary asset. |
| docs/datascience/old/cs2/csds/category.yml | Adds sidebar category metadata for archived CS2 CSDS docs. |
| docs/datascience/old/cs2/category.yml | Adds sidebar category metadata for archived CS2 section. |
| docs/datascience/old/category.yml | Renames/positions the archive section in the sidebar. |
| docs/datascience/adx/cs2/csds/README.md | Updates the live CS2 dataset page with current object counts, retention notes, and corrected glossary entries. |
| docs/datascience/adx/cs2/csds/assets/csds_dictionary.csv | Regenerates the live CS2 data dictionary to match current production schema. |
| docs/datascience/adx/category.yml | Adds explicit sidebar ordering for the ADX section. |
| CHANGELOG.md | Documents the doc refresh + archive move in the changelog. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| This is the CS2 data set page as it stood on 2024-02-24. It describes 31 files per | ||
| match; the data set has published 42 channels plus the index object since | ||
| August 2026. It is kept as a record for anyone working with data downloaded | ||
| before then. The current page is [Competitive CS2 Gameplay](/datascience/adx/cs2/csds/). |
There was a problem hiding this comment.
Good catch, fixed in 9b32f60. The 31-vs-33 contradiction is in the archived page itself (31 in the intro, 33 in the glossary), and is one of the reasons it is being replaced. The banner now says that instead of asserting one number right above text that gives the other.
Three typos carried over from the 2022 FAQ, now that the text around them is being rewritten anyway: "bootstrap you own project", "principals of semver", "part-way though the export process". The archived CS2 banner claimed the page describes 31 files per match. The page says 31 in the intro and 33 in the glossary, which is one of the reasons it is being replaced, so the banner now says that rather than picking one number and reading as a contradiction against the text right below it.
| CS2 has a rating by the [ESRB] of [Mature][esrb ratings] for Blood and Intense Violence. | ||
| CS2 is basically a SWAT team simulator. | ||
| The game includes realistic weapons, bombs, hostages, terrorists, killing, death, and grenades. | ||
| There are no player names, text chat, or voice chat data in the dataset. |
There was a problem hiding this comment.
Fixed in cfc31f4, and this one went further than the sentence. Reading csgo-dsdk scrub_pii.py, "redacted" is four different operations: names and player_chat.text become the literal string "redacted", player_personal.steam_id becomes a per-match alias (A, B, C), player_status.ping becomes 0, and player_info.wins and rank_update.win_count are clamped at 2501. The datasheet answer now says the channels and rows are present and only the values are overwritten, so player_chat shows that somebody typed in a round and not what they typed. Voice data really is absent. The spec preamble explains the distinction and the data dictionary now names the treatment per column instead of saying "Redacted by the PII scrubber" for all of them.
| This is the CS2 channel spec as it stood on 2024-02-24, covering 30 channels. Twelve | ||
| channels have been added since and two columns removed, so use it only to read data | ||
| downloaded before August 2026. The current spec is |
There was a problem hiding this comment.
Both right, fixed in cfc31f4. It is fourteen channels added and two channels removed (item_remove, player_action) for a net twelve, and I wrote "columns" where I meant channels. The banner now keys off the revision rather than the download: it describes revisions through 2026-07-31, which are still retained and still exportable today, with 2026-08-02 mixed and 2026-08-04 onward current.
| A new revision is published every day, and each match in it is 42 data files | ||
| plus an index object. |
There was a problem hiding this comment.
Fixed in cfc31f4. Now reads that revisions from 2026-08-04 onward are 42 data files plus the index and that retained revisions from before then carry 30, with a pointer to the data set page.
| objects, 42 channels plus the index. This page is kept as a record for anyone | ||
| working with data downloaded before then. The current page is |
There was a problem hiding this comment.
Agreed, fixed in cfc31f4. Same correction as on the archived spec: the banner now says the page describes revisions through 2026-07-31, which are still retained and still exportable, rather than keying off when the data was downloaded. That framing was in three banners and the archive index; all four are updated.
The datasheet still said "There are no player names, text chat, or voice chat data in the dataset." That was true when it was written and is not true now: player_chat exists, and the scrubber overwrites values rather than dropping columns. Read off csgo-dsdk scrub_pii.py, redaction is four different operations, so the spec and the dictionary now say which one each column got: names and chat text become the literal string "redacted", steam_id becomes a per-match alias, ping becomes 0, and wins and win_count are clamped at 2501. The chat rows survive, so the data shows that somebody typed in a round and not what they typed. Voice data is genuinely absent. Three places keyed the archive to when a subscriber downloaded data. A pre-August revision is still retained and still exportable today, so that sends exactly the affected reader to the wrong page. They now key off the revision: through 2026-07-31 the archived pages apply, 2026-08-02 is mixed, 2026-08-04 onward is current. The archived spec banner also said twelve channels were added and two columns removed. It is fourteen channels added and two channels removed, item_remove and player_action, for a net twelve. The FAQ claimed every match is 42 files without qualifying it, contradicting the data set page two clicks away.
The data science pages on docs.pureskill.gg describe a pipeline as it stood in
2022 (the FAQ) and 2024 (the dataset pages). This refreshes them against what
production publishes today and moves the pages they replace into an archive
section rather than deleting them.
What was wrong
2026-08-04. Fourteen were added (
bullet_damage,grenade_bounce,grenade_vector,item_dropped,item_refund,molotov_fire,player_chat,player_connect,player_inputs,player_sound,rank_update,score_update,team_change,world_item_vector) and tworemoved (
item_remove,player_action). The spec also still listedplayer_tickandplayer_player_id, retired from every channel.[CS2] - Counter-Strike: Global Offensive, said a match is 31 files in oneplace and 33 in another, and pointed at the CS:GO ESRB entry.
makenew-pyskillrequires>=3.14and uses uv;dsdkandcsgo-dsdkrequire>=3.11.datasets were last updated 2022-06-19.
revoked on a one-year cycle; 941 of 1,306 are already revoked.
How the new spec and dictionary were built
Both files are generated. Nothing in them is hand-typed.
generated
docs/csds/README.md, which is rendered from a real manifest.columns), csgo-rushb's generated column docs (125), the arrow schema measured
off 50 real production matches (284), the same column name resolved from one
of those in another channel (118), and
converters/csds/replay_engineering.pyfor the 8 columns that postdate all of them. Every one of the 762 columns has
a type.
tickandsecondonplayer_infoand
player_personal) are excluded from the dictionary and called out in thespec.
commentfields.Measurements
Taken against the production Data Exchange dataset
(
60a06bd80e9daf907a496ed5cdfd857c) on 2026-08-27:player_vectoris 25 MB andplayer_status5 MBThat last row is why the cost advice changed: skipping two channels cuts a day
to roughly a tenth of the bytes.
Archive
datascience/old/holds the 2022 FAQ, the 2024 CS2 pages and the CS:GO pages,each with a banner saying what it described and when it was current, plus an
index explaining what changed. It is a collapsed sidebar category, last.
The three CS:GO pages carry
slug:frontmatter pinning them to their originaladdresses under
/datascience/adx/csgo/, so links published before this changestill resolve. The live CS2 pages keep their addresses too. Only the archived
CS2 copies get new ones.
The archive is not decoration: the retained window still contains revisions the
old 30-channel spec describes, so a subscriber reading a revision from May 2026
needs it.
Checks
npm run buildpasses withonBrokenLinks: 'throw', and prettier is clean.Verified in the build output that all six archived pages are reachable and that
the CS:GO routes resolve at their original paths.
Not done here
makenew-pyskill's README still says "Python v3.9" while itspyproject.tomlrequires
>=3.14, and tutorial notebook 8 fetches tomes from datasets lastupdated in 2022. Both are that repo's to fix.
the LICENSE say "FPS Critic, Inc." This PR keeps "Inc."; worth settling which
is right.