[Ev3nt]

Restrict computed indexes over system tables

Problem

Currently, Firebird allows creating computed indexes on system tables. There is even a dedicated test for this case, core_5746_test.py (#2171). The test passes without any problems and everything seems fine, but an attempt to reconnect to the database causes a hang and the server crashes due to a stack overflow.

The culprit is the following statement:

create descending index systable_calc_idx on rdb$relations computed by ( 1 + rdb$format );

Root cause

1. When connecting to a database, a query is made against the RDB$AUTH_MAPPING table.
2. To do this, scan_relation is called, which looks up this table by ID by querying RDB$RELATIONS.
3. BTR_all retrieves the list of indexes.
4. For the systable_calc_idx index, BTR_description is called. Since this is an expression index, it invokes MET_lookup_index_expression.
5. MET_lookup_index_expression calls scan_relation to look up the RDB$RELATIONS table by ID.
6. From there, steps 2-5 repeat recursively until the stack overflows.

Notes

This problem does not occur in version 6 with the merged metadata cache.

Fix

Completely prohibit the creation of computed indexes on system tables.

[aafemt]

This is not a fix. Break the recursion instead.

[Ev3nt]

This is not a fix. Break the recursion instead.

Do we really want to let users create computed indexes over system tables? 🤨😬

[aafemt]

I would say yes. This is one of the ways to improve performance of big databases.

[dyemanov]

I would say yes. This is one of the ways to improve performance of big databases.

Do you know a single case when a computed index is used for that purpose? This requires knowledge of Firebird internal queries.

[aafemt]

System tables are accessed by user applications as well. That's where indexes matching application's needs are useful.

[Ev3nt]

@aafemt, standard indexes over system tables are one thing, and they are already perfectly allowed. But computed expressions are a completely different story.

To safely break the recursion instead of prohibiting the index, we would have to inject complex, fragile checks into MET_lookup_index_expression() or the relation scanning logic just to handle a highly theoretical edge case.

Can you provide at least one real-world, production-proven scenario where an application actually needs a expression-based computed index (like COMPUTED BY (1 + RDB$FORMAT)) over the core system dictionary?

Unless there is a practical use case that outweighs the architectural complexity of breaking this loop, blocking this at the DDL level is the cleanest way to protect the server from a trivial stack overflow crash.

[aafemt]

To safely break the recursion instead of prohibiting the index, we would have to inject complex, fragile checks into MET_lookup_index_expression() or the relation scanning logic just to handle a highly theoretical edge case.

Why to inject fragile checks when you can inject a dumb loading of non-computed indexes only during the initial table scan and delay loading of computed ones as much as possible? That's what for flags REL_being_scanned and REL_rescan are supposed to be used, right?

[sim1984]

This is not a fix. Break the recursion instead.

Do we really want to let users create computed indexes over system tables? 🤨😬

No

[Ev3nt]

That's what for flags REL_being_scanned and REL_rescan are supposed to be used

Yes, managed to make it work with REL_being_scanned / REL_rescan — deferred loading of the computed index expression, no stack overflow, index stays functional. I have a .diff with the changes and test script if needed.