Skip to content

Fix bulk add writing wrong values and dropping rows and columns - #1172

Open
labkey-martyp wants to merge 8 commits into
release26.7-SNAPSHOTfrom
26.7_fb_bulk_add_date_parse
Open

Fix bulk add writing wrong values and dropping rows and columns#1172
labkey-martyp wants to merge 8 commits into
release26.7-SNAPSHOTfrom
26.7_fb_bulk_add_date_parse

Conversation

@labkey-martyp

@labkey-martyp labkey-martyp commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Rationale

Fixes a set of bugs that let the EHR bulk add importer write wrong or incomplete data without reporting anything, in the UI, the browser console, or the server log.

Because nothing fails, the damage is only discoverable by diffing the loaded data back against the source, so an import can appear clean while carrying shifted dates, missing dates, lookup values swapped for unrelated ones, whole columns absent because their header was spelled differently, and — on tables whose key is assigned by the server — only the last of the pasted rows. The reporting added here echoes pasted text back to the user, so it is escaped; an import commonly starts from a spreadsheet someone else produced.

The same work also covers the opposite failure, where the importer refused input that was perfectly good or blamed the wrong thing for it, since a rejection nobody can act on costs the user as much as a silent one.

Related Pull Requests

None.

Changes

  • Every date column is now parsed the same way as the form's primary date field. Previously the others were read as UTC, so in a negative-offset timezone they landed a day early, and a date falling on the epoch was discarded as empty.
  • Lookup and project values now require an exact match rather than resolving on a leading substring, so an input can no longer be replaced by an unrelated record that merely starts with the same text. Matching stays case-insensitive, ignores surrounding whitespace, and accepts either the display value or the stored key.
  • A value that cannot be resolved to a lookup, a project, or a date is now reported against its row rather than being written through as raw text or left silently empty, as is a lookup whose values have not loaded. The import stops so the source can be corrected first.
  • Pasted text is escaped wherever it appears in those error messages.
  • Column headers are now identified the same way the rest of the product identifies a pasted column, by a field's name, label, or any of its import aliases, ignoring case and surrounding whitespace. Previously only the exact field name or a single alias was tried, so a header spelled any other way dropped its whole column without a word.
  • A header naming no field in the form, or two headers naming the same one, is now reported and the import stops before any row is read. Note the behavior change: a spreadsheet carrying extra columns the form does not have will no longer import until those are removed, where previously they were ignored.
  • Trimming the pasted block no longer strips tabs, only spaces and line endings. Stripping tabs discarded the empty trailing cells of the final row, which then looked truncated and was rejected even though every value in it was correct.
  • A row is measured against the last column a required field occupies rather than against the count of required fields, which bore no relation to where those fields sit. A row missing only optional trailing cells is no longer rejected for it.
  • A problem belonging to the header row — a required column absent, or a header row naming nothing at all — is reported once instead of restated as a missing value on every one of up to 250 rows.
  • Pasting several rows into a table keyed on a server-assigned column no longer collapses them onto one record, which previously left only the last row's values.

Date columns other than the form's primary date field were read as UTC, so imported dates landed a day early or vanished when they hit the epoch, and lookup values matched on a prefix instead of exactly, letting one code silently store another.
The bulk add importer silently stored raw text for unmatched lookups and dropped unparseable dates, and resolved project names by prefix. All three now surface a per-row error.
Rows keyed solely on a server-assigned identity column have an empty key before save, so every new row after the first resolved to the first and collapsed onto it. Such tables could not add more than one row per save.
@labkey-martyp labkey-martyp changed the title Fix bulk add parsing dates as UTC and lookups by prefix Fix bulk add writing wrong values and dropping new rows on save Jul 27, 2026
Error messages rendered pasted cell values as HTML, a lookup store with no records loaded silently wrote raw text through instead of reporting it, and exact matching rejected whitespace-padded cells and pasted key values that previously imported.
Header cells were matched exactly and case-sensitively, so a padded or recased header silently dropped its whole column; they are now normalized, and a header matching no field, or two naming the same one, is reported. Columns handled before the field loop are skipped by name rather than truthiness so a failed project or date is not resolved a second time by a path that disagrees, and unknown-project errors echo the pasted value instead of its zero-padded form.
Header matching covered only a field's name and first import alias, so a header spelled as the field's label, or as any later alias, resolved to nothing -- and since an unrecognized header now stops the import, the recognized-name set was also vouching for aliases the matcher could not actually place, dropping those columns silently. Resolution now runs through LABKEY.ext4.Util.resolveFieldNameFromLabel for both, keyed on the resolved field so duplicates are caught per field rather than per header spelling. An exact name match is tried first because that helper breaks on its first name/caption/label hit and would otherwise let a field merely labelled the same win on config order.
@labkey-martyp labkey-martyp changed the title Fix bulk add writing wrong values and dropping new rows on save Fix bulk add writing wrong values and dropping rows and columns Jul 27, 2026
Trimming the pasted block with Ext4.String.trim() counted a tab as whitespace, so the last row lost its empty trailing cells and read as truncated even though every value in it was correct; spaces and line endings are still trimmed, since a stray one at either end parses as a junk row or is mistaken for the header row. A row is now measured against the last column a required field maps to rather than the number of required fields, which was unrelated to where those fields sit, and a missing required column or an unusable header row is reported once instead of as a missing value on every row. Header resolution also prefers importable fields for exact names, matching the order already used for display names, so a field the importer skips cannot claim a name another query declares.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant