Report generation performance overhaul: pre-compiled OOXML cache, image renditions, list-numbering fixes#142
Merged
Merged
Conversation
…ration
Report generation for 400+ vuln assessments was spending ~28s in
XHTMLImporterImpl.convert() calls. This change pre-compiles each
vulnerability's desc/rec/details fields into serialized OOXML at save
time, so report generation can skip the importer entirely for cached
fields.
- Add cachedDescXml/cachedRecXml/cachedDetailsXml + hash fields to
Vulnerability DAO
- Add DocxPrecompiler that converts HTML to OOXML at save time, leaving
assessment-level ${...} variables as literal text in <w:t> nodes
- Embed image bytes as data URIs in cached XML (r:embed) so snippets
are self-contained; at report time, create real image parts in the
report package and remap to valid rIds
- Add wrapHTMLFromCache fast path in DocxUtils.checkTables and
setFindings that uses cached XML when hash matches, applies
assessment vars via string replacement, falls back to live
conversion on cache miss or when report extensions are active
- Hook precompiler into vuln save paths (assessments API and
AddVulnerability action) as background threads
- Add startup migration in AppBootstrapListener to pre-compile all
vulnerabilities in open assessments
- Add DocxUtils2 raw-XML experiment and parity/perf test harness
- Cache version (v5) invalidates prior caches on format changes
…w cache
- Short-circuit Extensions.updateReport() before cloning when text has no
${ placeholder
- Remove isExtended() guard in wrapHTMLFromCache so cache is always used
- Skip updateReport() in replacement() for content without ${ placeholders
- Short-circuit per-paragraph in updateDocWithExtensions
- Run extensions at pre-compile time in DocxPrecompiler
- Force-initialize lazy Assessment.answers collection
- Clear cache on assessment finalize
- Bump cache version to v6
Images: - Store a report-ready rendition on Image at upload time (downscaled or re-encoded once); report paths embed it with no decode work. Startup backfill prepares existing images; FACTION_REPORT_IMAGE_MAX_WIDTH changes invalidate automatically. Pre-compiled OOXML cache (DocxPrecompiler v2): - Cached fields are now self-contained: list numbering definitions are captured from the scratch package and carried in the payload with tokenized numId references, then re-created under freshly allocated ids at report time. Fixes bullet lists rendering as continued decimal numbers when cached XML referenced ids owned by the report template. - Cache version bump (v7) invalidates all previous-format rows; a guard additionally refuses numbering references without carried definitions. - Type-3 (HTML) custom fields are cached too (cachedCfXml, one marker-delimited entry per field with its own hash). - Misnested-list repair (hoistMisnestedLists) now runs at compile time. Report-time (DocxUtils): - vulnTable/checkTables path converts fields in batches with per-vuln placeholder keys (was one importer call per field) and serves cache hits without conversion. - wrapHTMLFromCache: variable substitution scans gated on token presence; deepCopy round trip removed; image parts for known formats created directly instead of docx4j's probing createImagePart. - replaceAssessment runs before findings are inserted so its whole-document marshal covers only the template; per-field paths now also resolve asmtStart/asmtEnd and assessment-level text custom fields inside finding content. Tests: list numbering verified live, cached, and via vulnTable (bullet numFmt resolution per item, poisoned-legacy-cache rejection, cached image part integrity); assessment-variable parity across both paths; image rendition lifecycle.
normalizeNamespaces scanned the xmlns prefix with an off-by-one that
read the usual single-char "w" prefix as an empty string, so the
"w".equals(prefix) early-return never fired and the blind
replace(" :", " w:") ran over the whole marshalled XML — turning every
" :" in user text into " w:" (":80" rendered as "w:80").
The prefix scan is fixed and every replacement is now anchored to XML
structure: element names via '<'/'</' (which cannot appear unescaped in
text), attribute names via their trailing '="'. The regex-fallback
paragraph splitter's prefix rename is anchored the same way. Cache
version bumps to v8 since v7 rows may carry the corruption baked in.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Overhauls DOCX report generation performance for large (400+ vuln) assessments and fixes several list-numbering / namespace correctness bugs that surfaced along the way. Previously report generation spent ~28s in
XHTMLImporterImpl.convertper assessment; this change moves that conversion to save time by caching self-contained OOXML snippets.What changed
Pre-compiled OOXML cache (
DocxPrecompiler)${...}variables as literal text in<w:t>nodes.r:embed) in the cached XML so snippets are self-contained; at report time real image parts are created in the report package and remapped to valid rIds.hoistMisnestedLists) runs at compile time.List-numbering correctness
normalizeNamespacesoff-by-one fixed that was reading thewprefix as empty and running a blindreplace(" :", " w:")over the whole marshalled XML, corrupting:80→w:80in user text. Element/attribute replacements are now anchored to XML structure.Image renditions
Imageat upload time (downscaled/re-encoded once viaReportImageScaler); report paths embed it with no decode work. Startup backfill prepares existing images and auto-invalidates onFACTION_REPORT_IMAGE_MAX_WIDTHchanges.Report-time (
DocxUtils) optimizationsvulnTable/checkTablesconverts fields in batches with per-vuln placeholder keys (was one importer call per field) and serves cache hits without conversion.wrapHTMLFromCache: variable substitution gated on token presence;deepCopyround trip removed; image parts for known formats created directly instead of docx4j's probingcreateImagePart.replaceAssessmentruns before findings are inserted so its whole-document marshal covers only the template.updateReportbefore cloning when text has no${placeholder; run extensions at pre-compile time; clear cache on assessment finalize.Tests
DocxListNumberingTest— list numbering verified live, cached, and via vulnTable (bullet numFmt resolution per item, poisoned-legacy-cache rejection, cached image part integrity).DocxAssessmentVarsParityTest— assessment-variable parity across both paths.DocxUtils2ParityTest— DocxUtils2 raw-XML experiment parity.ReportImageScalerTest— image rendition lifecycle.DocxPerfHarness/DocxUtilsBatchIntegrationTest— perf + batch integration.