[2.1] Add a Docker development environment and a baseline forum for upgrade testing by albertlast · Pull Request #9330 · SimpleMachines/SMF

albertlast · 2026-07-30T20:32:05Z

Note

This change was produced by an LLM. The code, the commit messages and this
description were all written by Claude (Anthropic), driven by @albertlast. It has
not yet had human code review.

Everything stated below was verified by running it rather than only reasoning about
it — full installs from an empty database on both engines, and the committed dumps
restored back over the forums that produced them. Even so, please review it as
untrusted work, and as a proposal: whether SMF wants generated fixtures committed to
a release branch is a project decision, not a technical one.

Description

This is a proposal rather than a fix, and it is tooling only — no forum code
changes. Please close it if a bundled dev environment and committed fixtures are not
wanted on the 2.1 branch.

Testing the 2.1 → 3.0 upgrade needs something nobody currently has: a real 2.1 forum,
with content in it, that everyone can start from and that is identical every time.
Building one by hand takes an afternoon of clicking through the installer and produces a
different forum each time, so upgrade work either happens against an empty schema —
which exercises almost none of the migrations — or against whatever private database the
person testing happens to own.

That starting point can only be built here. 3.0 cannot create a 2.1 forum; its installer
writes 3.0 schema. So this branch adds the environment and the builder, and commits the
result:

.docker/baseline/restore.sh --engine mysql --profile small

Roughly ten seconds later there is an installed SMF 2.1.7 forum with 400 members and
6,000 posts, on either engine, identical to the one anyone else restores. Point a 3.0
checkout at that database, run upgrade.php, and the migration has something to
migrate.

The environment half is the same one as #9317, ported to 2.1. Ports are offset
(8180/8181/8125/3407/5533) so both stacks can run at once, which upgrade testing needs.

What makes the scenario a complex one

A stock install exercises almost nothing: SMF 3.0 runs 87 v2_1 migrations and 17
v3_0 ones, and a fresh 2.1 database leaves most of the tables they read completely
empty. Content generation alone does not help much either — the upstream Populate.php
tool fills six tables, and hardly any migration touches them.

So the builder seeds the rest deliberately, and each script names the migrations it
exists to feed:

Seeded	Why it matters to the upgrade
IPv4, IPv6 and absent addresses on messages, members, logins and every log table	one migration per IP-bearing column. On MySQL these are `VARBINARY(16)` and the conversion is real work; on PostgreSQL 2.1 already used `inet` and they should skip. Both engines are built, so both paths are covered
Custom profile fields and their values in 2.1's layout (`themes.variable = 'cust_<name>'`)	that indirection is the whole of `CustomFieldsPart1`–`Part3`
Calendar events — all-day, timed, multi-day, topic-linked — and holidays on the 1004 sentinel	`HolidaysToEvents`, `RecurringEvents`, `EventUids`. A holiday dated any other year is a one-off, and the recurrence conversion never fires
Polls (one open, one expired) with votes, PMs with labels and a filing rule, drafts of both types	`PersonalMessageLabels`, `UserDrafts` and friends
Real attachments on disk and in the table: a normal one, a thumbnail linked to its parent, an avatar	`LegacyAttachments`, `AttachmentSizes`, `AttachmentDirectory`
Bans: IPv4 range, IPv6 range, email mask, member ban, plus logged hits	`Ipv6BanItem`, `Ipv6LogBanned`
Error, action, online, reported, flood, spider and search logs — some rows with backtrace and session, some deliberately without	the remaining IPv6 log migrations, `ErrorLogSession`, `SearchResultsPrimaryKey`
An installed package row, legacy settings (`karmaMode`, `mail_type`, `cookieTime`, `enable_mod_prefs`, `time_offset`), TFA secrets, non-zero time offsets in both signs	`PackageVersion`, `MailType`, `RemoveCookieTime`, `DropModPrefs`, `DropTimeOffset` are all no-ops unless the old values are present
Two MyISAM tables on MySQL	a 2.1 install on any modern server creates everything as InnoDB, so `ConvertToInnoDb` would otherwise find nothing to convert
A utf8mb3 database, matching 2.1's own DDL	this is what makes it a genuine pre-utf8mb4 forum, and converting it is what 3.0's UTF-8 step does. Changing it to utf8mb4 would quietly delete the thing under test
Names differing only by case or homoglyph	`SpoofDetector`

Also seeded, because the alternative is a forum nobody can browse: board access. 2.1's
createBoard() leaves member_groups empty, so every board the generator makes is
admin-only until something opens it up.

Sizes

Profile	Members	Topics	Messages	Dump per engine	Committed
`tiny`	50	~150	600	~0.9 MB	yes
`small`	400	~1,200	6,000	~8 MB	yes
`medium`	2,000	~8,000	40,000	~70 MB	no
`large`	20,000	~40,000	250,000	~450 MB	no

tiny is for a quick upgrade run, small for a realistic one. medium and large
exist as profiles but are gitignored by directory name, so producing one cannot
accidentally add half a gigabyte to the repository.

The committed artifacts total about 17 MB, which is the main thing to weigh in
reviewing this. If that is too much for a release branch, the builder works without them
— make-baseline.sh regenerates everything in one command — and the artifacts can
simply be dropped from the branch.

Installing without a browser

2.1's installer turns out to be drivable with curl: it reads its step from ?step=,
runs steps in order within one request, and stops at the first that needs input. There
is no CSRF token and no server-side progress. Three details are not obvious, and are
documented in install-forum.sh:

DatabasePopulation() returns false even when it succeeds — it pauses to show
its "N duplicate tables ignored" report, and that form posts pop_done to skip past
it. That is why the sequence is five requests, not four.
dbsession and reg_mode must travel with boardurl, because forum settings
returning true falls straight through into database population in the same request.
DeleteInstall() deletes nothing. The congratulations page has a checkbox that
points the browser at ?delete; that request is what removes install.php and the
two schema files.

The driver never sends the stats field, which would register the site with
simplemachines.org.

Populate.php is fetched, not vendored

Content comes from Populate.php in
SimpleMachines/tools. It is downloaded at a
pinned commit and checksummed rather than committed here: it is third-party licensed
(MPL 1.1, with a BSD lorem ipsum generator) and does not belong in an SMF release
branch. Four patches adapt it, each asserting it matched exactly once, so an upstream
change breaks the build loudly instead of quietly producing a different baseline.

Two of those are worth flagging upstream: makeBoards() and makeMessages() both reach
mt_rand(1, 0) on their first item, which PHP 8 rejects outright where PHP 7 silently
swapped the arguments. Populate.php cannot run at all on PHP 8 unpatched.

It does not disturb the existing CI checks

Everything lives under .docker/, which the integrity scripts skip. Every PHP file added
is 7.1-syntax-clean, since the lint job runs phplint over the whole checkout from
7.1 through 8.4 and it is not worth betting on whether it descends into dot directories —
verified by running php -l over all of them under both php:7.1-cli and php:8.4-cli.
shellcheck -S warning is clean on every shell script. No file under .github/ is
touched.

Outside .docker/ and compose.yaml the only tracked changes are .gitignore and
.gitattributes: the former because 2.1 ignores *.sql, *.gz and attachments/ and
would otherwise swallow the entire deliverable, the latter to mark the dumps -text -diff so line-ending rewriting cannot corrupt their hex-encoded IP columns.

Verification

Both engines, both profiles, built from an empty database and then restored from the
committed files:

MySQL 8.4 and PostgreSQL 17, PHP 8.2, full headless install each time.
Every dump restored back over the database that produced it, with checksums matching
the manifest and all 41 recorded table counts matching.
check-coverage.php — 78 assertions that the columns the migrations read are
actually filled, not merely that rows exist — passes on both engines.
The restored forum browses: index, boards, topics and calendar all return 200, with
zero deprecation notices in the response body.

That coverage check exists because it was needed. The custom field definitions were
silently absent on PostgreSQL for several builds: db_insert() builds its ON CONFLICT
clause only from key columns that also appear in the column list, and id_field is a
sequence the caller does not supply, so PostgreSQL received ON CONFLICT () DO NOTHING,
rejected it as a syntax error, and SMF logged that rather than raising it. Row counts
looked healthy — a stock install ships four custom fields of its own — and only reading
the data showed it. make-baseline.sh now runs the check before dumping.

Notes for review

Credentials are admin / baseline and smf / smf / smf, deliberately trivial
and published; the whole stack is throwaway and binds only to localhost. Generated
members cannot log in at all, and every address is @example.com, which RFC 2606
reserves.
$auth_secret and $image_proxy_secret are pinned by the builder. They live in
Settings.php and not in the database, so a restored dump paired with fresh random
secrets would invalidate every session and two-factor backup code inside it.
MySQL is pinned to 8.4 and must not float to 9: MySQL 9 removes the utf8/utf8mb3
aliases 2.1's DDL is written against.
Regenerating writes a new artifact directory and bumps CURRENT; it never
overwrites, so a test can pin a revision.
.docker/baseline/README.md documents the cross-branch recipe for consuming these from
a 3.0 branch — git archive from a pinned commit, so the artifacts have exactly one
home and no copy can drift.

Issues References (Fixes|Related|Closes)

Related: [3.0] Add a Docker development environment for MySQL and PostgreSQL #9317 (the 3.0 Docker environment this is the 2.1 counterpart of)

Ports the environment from the 3.0 branch, with the changes 2.1 needs: - The installer loads its schema from install_2-1_<type>.sql, and looks for it next to install.php rather than in other/. The entrypoint now stages both engines' files alongside install.php; without them the Welcome step marks an engine unsupported. - The writable list is 2.1's: agreement.txt at the root rather than under Languages/, and no Sources. Directories and files are created separately, because creating everything with mkdir -p would turn a missing agreement.txt into a directory. - No composer. 2.1 has no runtime vendor/ directory; its only dependency is a dev-only build-tools package fetched from git, so installing it at boot cost a slow, network-dependent first run and bought nothing. - MySQL is pinned to utf8mb3, matching what 2.1's own DDL creates. This is deliberate rather than an oversight: it is what makes the environment a genuine pre-utf8mb4 forum, which is what a 3.0 upgrade has to convert. - PHP defaults to 8.2. 2.1 supports 7.1 through 8.4, but 8.3 and later fill the log with deprecation notices on every page. Every port is offset from the 3.0 environment's, so both stacks can run at the same time -- which upgrade testing needs. Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

Testing the upgrade needs a starting point: a real, installed, filled 2.1 forum that everybody can restore in seconds. This builds one, for both database engines, in a single command: .docker/baseline/make-baseline.sh --engine both --profile small Per engine it resets, installs, populates, seeds, dumps and then restores the dump -- because a dump that has never been restored is a guess. Installing headlessly turned out to be possible because 2.1's installer is nearly stateless: ?step=N, then it runs steps in order in one request and stops at the first needing input. Three things are not obvious, and are documented in install-forum.sh: DatabasePopulation returns false even when it succeeds (its report posts pop_done to skip past it), dbsession and reg_mode must travel with boardurl because forum settings falls straight through into population, and DeleteInstall deletes nothing -- a separate GET ?delete does that. Content comes from upstream Populate.php, fetched at a pinned commit and checksummed rather than vendored: it is third-party licensed and does not belong in an SMF release branch. Four asserted patches adapt it; two of them fix mt_rand(1, 0) calls that are fatal on PHP 8 and would otherwise stop it running at all. Populate only builds six tables, though, and almost none of the 3.0 migrations touch them. So extras/ seeds what they do care about -- IPv4, IPv6 and absent addresses, custom field values in their 2.1 layout, polls, PMs, drafts, attachments, calendar holidays dated year 0004, bans, logs, legacy settings, and MyISAM tables that a modern install would never produce. Each script names the migrations it exists to exercise. Every PHP file here is 7.1-syntax-clean, since 2.1's CI lints the whole checkout from 7.1 through 8.4. Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

A grep pattern written as $'\r' is quoted differently by different shells, and one that silently becomes the empty pattern matches every line -- so the guard was either useless or would have failed every dump. Counting the bytes with tr says what it means. Also corrects a comment in run-extras.php: re-running after a *finished* script is free, but a script that failed part way never wrote its marker, and repeating it would insert its rows twice. Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

An installed, populated SMF 2.1.7 forum, dumped for both engines at two sizes, plus the manifest that lets a restore be checked rather than merely attempted: .docker/baseline/restore.sh --engine mysql --profile small tiny 50 members, ~150 topics, 600 messages ~1 MB per engine small 400 members, ~1200 topics, 6000 messages ~8 MB per engine Both were produced by make-baseline.sh and then restored from the committed files, on both engines, before being committed. The dumps are marked -text -diff: they are byte-exact generated blobs, and rewriting their line endings would corrupt the hex-encoded binary IP columns. medium and large profiles exist in the builder but are gitignored -- at 70 and 450 MB they are generated on demand. Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

The baseline's package-log seeding writes it, and so does the package manager on any real install. It belongs inside the artifacts, not in the checkout. Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

Two things were wrong with what 40-calendar.php produced, and both defeated the point of the script. The recurring holidays were dated year 0004. SMF 2.1's sentinel for "every year" is 1004 -- getHolidayRange() rewrites the range bounds to that year to find them, and it is the column default. A holiday dated 0004 is simply a holiday in the year 4: SMF never shows it, and an upgrade would carry it across as a one-off. Which means the baseline had no recurring holiday at all, and HolidaysToEvents would have had nothing to convert. Verified after the fix: "Baseline yearly holiday" now resolves on 2026-12-25 and "Baseline new year" on 2027-01-01. The calendar itself was also left switched off, which is how 2.1 ships. That left the baseline describing a state no real forum is in -- four events and three holidays on a feature nobody could have reached to create them. Artifacts regenerated for both profiles and both engines, and restored from the committed files afterwards. Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

The three baseline custom field definitions never reached PostgreSQL. db_insert() builds its ON CONFLICT clause only from key columns that also appear in the column list, and id_field is a sequence the script deliberately does not supply -- so PostgreSQL got `ON CONFLICT () DO NOTHING`, rejected it as a syntax error, and SMF logged that rather than raising it. The script carried on and reported three fields inserted. MySQL, with its INSERT IGNORE, was fine, so the two engines' baselines disagreed. It survived every check we had. Row counts looked healthy because a stock install supplies four custom fields of its own, and the field *values* were written by a later query that worked, so the tables were all non-empty. Only reading the data showed it. So: use a plain insert, count what actually arrived rather than what was asked for, and add check-coverage.php -- 78 assertions that the columns the 3.0 migrations read are genuinely filled, not merely present. make-baseline.sh now runs it before dumping, on the grounds that a thin artifact is worth catching while the forum that produced it still exists. Two things it caught immediately, both since fixed: SMF's MySQL driver leaves the connection with no default schema, so DATABASE() is empty and every information_schema check silently matched nothing. Audited the other ten ignore/replace inserts; all name key columns that are present in their column lists. Artifacts regenerated and checked on both engines: 78 checks, 0 failures. Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

A comment line beginning with the linter's own name is parsed as another directive, so the explanation underneath the disable turned into a parse error and stopped the file being checked at all. Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

check-eof.php does descend into dot directories -- the assumption that the integrity scripts skip .docker/ was wrong, and only check-smf-index and check-smf-license behave that way. CI caught it on the first push. Its rules are specific: a file must end with `?>` at the very last byte, with no trailing whitespace, and with a blank line between that and the preceding `}` or `;`. That is what every other PHP file in the repository does. Verified by running the real build-tools scripts against the branch rather than guessing: check-eof, check-smf-license, check-smf-languages and check-version all pass. Re-linted on 7.1 and re-ran the generator afterwards -- 78 coverage checks, 0 failures. Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

albertlast · 2026-07-30T21:29:39Z

something i ever wanted,
a source for migration test,
so maybe a little bit confusing, why this pr target branche 2.1,
but in his core of motivation it's a feature for 3.0.
since the biggest userbase wouldn't be install 3.0 fresh, they would like to migrate from 2.1,
this pr build the ground work for migration test from 2.1 to 3.0.

albertlast · 2026-08-01T07:38:51Z

note for my futue me,
i don't like that the admin account is factor enabled, it make acessing the admin account harder, also since it not mention,
on the other hand i like that admin account got als two factor enabled for testing the upgrade process since it provide more complexity

albertlast added 9 commits July 30, 2026 20:28

Ignores Packages/installed.list

2269368

The baseline's package-log seeding writes it, and so does the package manager on any real install. It belongs inside the artifacts, not in the checkout. Signed-off-by: albertlast <mathiaspapealbert@hotmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[2.1] Add a Docker development environment and a baseline forum for upgrade testing - #9330

[2.1] Add a Docker development environment and a baseline forum for upgrade testing#9330
albertlast wants to merge 9 commits into
SimpleMachines:release-2.1from
albertlast:docker-dev-env-2.1

albertlast commented Jul 30, 2026

Uh oh!

albertlast commented Jul 30, 2026

Uh oh!

albertlast commented Aug 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

albertlast commented Jul 30, 2026

Description

What makes the scenario a complex one

Sizes

Installing without a browser

Populate.php is fetched, not vendored

It does not disturb the existing CI checks

Verification

Notes for review

Issues References (Fixes|Related|Closes)

Uh oh!

albertlast commented Jul 30, 2026

Uh oh!

albertlast commented Aug 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant