Use of the aai and/or aao record types does not work correctly

[pheest]

I am adding new loopback tests (i.e. write to the aao record VAL field and scan using the OUT field to propagate to the aai record having the same FTVL).

The test fails usually with the values read back being discrepant.
Sometimes it passes.
Sometimes it delivers an access violation error code 0XC0000005.
Sometimes the test locks up.

[pheest]

This issue is caused by use of record caching in db.py.
The issue is not restricted to use of array types, but is generic to all types.

When a record instance is created by calling getRecord(), a reference to the record is stored in the global _rec_cache variable, and a reference is also returned.
Subsequent calls to getRecord() then return the reference stored in _rec_cache, rather than creating a new instance from scratch.

This usage is aimed at reducing the run-time overhead of re-creating records that are used repeatedly - typically those that are referenced at local scope.

This concept fails because resources that are held by the record are de-allocated by the destructor when the record instance leaves scope.

Any subsequent attempt to re-use the already-created record consequently fails, typically resulting in segmentation fault or unpredictable behaviour. This is reproducible on both Windows and Linux.

There are no circumstances in which the cached record capability can be safely used.

[mdavidsaver]

This concept fails because resources that are held by the record are de-allocated by the destructor when the record instance leaves scope.

I am having trouble parsing this statement. Which resources? (just memory?) Which destructor? Which scope?

typically resulting in segmentation fault or unpredictable behaviour. This is reproducible on both Windows and Linux.

Can you provide a stack trace?

Do the test cases added in test_db.py trigger SIGSEGV if the change in db.py is reverted?

I am wondering if a crash as you describe might actually be due to missing ref. count manipulation elsewhere.

[pheest]

Hi @mdavidsaver,

I have reliably reproduced the issue on Windows and Linux by copy'n'pasting a duplicate of the existing test 'def test_link(self):' and running the tests. This causes the 'src' and 'tgt' scalar records to be re-used in local scope, and fails on the second instance.
I would invite you to do the same (using the existing code).

I also reproduced the issue on Windows using my newly added array tests which re-use the 'rec:aai:l' and 'rec:aai:f' records. I did not get around to re-verifying the matter on Linux however.

The change in db.py fixes the issue by removing the caching behaviour. This fully addresses the issue.

[mdavidsaver]

I am able to reproduce with https://github.com/mdavidsaver/pyDevSup-dev/tree/refs/heads/ci using python3-dbg from Debian 13. So the crash involves an array field feeding some invalid pointers into numpy..

(gdb) py-bt
Traceback (most recent call first):
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 989, in fillFormat
    finite_vals = data[isfinite(data)]
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 985, in __init__
    self.fillFormat(data)
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 455, in <lambda>
    'float': lambda: FloatingFormat(
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 516, in _get_format_function
    return formatdict['float']()
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 583, in _array2string
    format_function = _get_format_function(data, **options)
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 557, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 784, in array2string
    return _array2string(a, options, separator, prefix)
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 1591, in _array_repr_implementation
    lst = array2string(arr, max_line_width, precision, suppress_small,
  File "/usr/lib/python3/dist-packages/numpy/_core/arrayprint.py", line 1674, in array_repr
    return _array_repr_implementation(
  File "/usr/lib/python3/dist-packages/numpy/testing/_private/utils.py", line 243, in build_err_msg
    r = r_func(a)
  File "/usr/lib/python3/dist-packages/numpy/testing/_private/utils.py", line 801, in assert_array_compare
    msg = build_err_msg([x, y],
  File "/usr/lib/python3/dist-packages/numpy/testing/_private/utils.py", line 1172, in assert_array_almost_equal
    assert_array_compare(compare, actual, desired, err_msg=err_msg,
  File "/usr/lib/python3/dist-packages/numpy/_utils/__init__.py", line 86, in wrapper
    return fun(*args, **kwargs)
  File "/home/dev/epics/pyDevSup/python3.13d/linux-x86_64/devsup/test/test_db.py", line 120, in array_float_test
    assert_array_almost_equal(tgt.VAL, [])
  File "/home/dev/epics/pyDevSup/python3.13d/linux-x86_64/devsup/test/test_db.py", line 229, in test_aao_float
    self.array_float_test(src, tgt)
...

Thread 1 "python3-dbg" received signal SIGSEGV, Segmentation fault.
Download failed: Invalid argument.  Continuing without source file ./string/../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S.
__memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:265
warning: 265    ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory
(gdb) bt
#0  __memcpy_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:265
#1  0x00007ffff56643c9 in memmove (__dest=<optimized out>, __src=<optimized out>, __len=<optimized out>)
    at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:36
#2  _contig_to_contig (context=<optimized out>, args=<optimized out>, dimensions=<optimized out>, __NPY_UNUSED_TAGGEDstrides=<optimized out>, 
    __NPY_UNUSED_TAGGEDdata=<optimized out>) at ../numpy/_core/src/multiarray/lowlevel_strided_loops.c.src:384
#3  0x00007ffff56e81b4 in raw_array_assign_array (ndim=<optimized out>, shape=<optimized out>, dst_dtype=<optimized out>, 
    dst_data=<optimized out>, dst_strides=<optimized out>, src_dtype=<optimized out>, src_data=<optimized out>, src_strides=0x7fffffff3e30)
    at ../numpy/_core/src/multiarray/array_assign_array.c:150
#4  0x00007ffff56e8ca7 in PyArray_AssignArray (dst=0x7ffff45269c0, src=0x7ffff4526940, wheremask=0x0, casting=NPY_NO_CASTING)
    at ../numpy/_core/include/numpy/ndarraytypes.h:1508
#5  0x00007ffff56fe4b0 in PyArray_CopyInto (dst=dst@entry=0x7ffff45269c0, src=src@entry=0x7ffff4526940)
    at ../numpy/_core/src/multiarray/ctors.c:2900
#6  0x00007ffff56f6d75 in PyArray_CastToType (arr=0x7ffff4526940, dtype=0x7ffff5f7aec0 <DOUBLE_Descr>, is_f_order=is_f_order@entry=0)
    at ../numpy/_core/src/multiarray/convert_datatype.c:317
#7  0x00007ffff57f4c4b in check_for_trivial_loop (ufuncimpl=0x7ffff553ce10, op=0x7fffffff4960, dtypes=0x7fffffff4d60, 
    casting=NPY_SAME_KIND_CASTING, buffersize=<optimized out>) at ../numpy/_core/src/umath/ufunc_object.c:808
#8  PyUFunc_GenericFunctionInternal (ufunc=0x7ffff6e439b0, ufuncimpl=0x7ffff553ce10, operation_descrs=0x7fffffff4d60, op=0x7fffffff4960, 
    casting=NPY_SAME_KIND_CASTING, order=NPY_KEEPORDER, wheremask=<optimized out>) at ../numpy/_core/src/umath/ufunc_object.c:2230
#9  ufunc_generic_fastcall (ufunc=<optimized out>, args=<optimized out>, len_args=<optimized out>, kwnames=<optimized out>, 
    outer=<optimized out>) at ../numpy/_core/src/umath/ufunc_object.c:4530
#10 0x00000000004b95d1 in _PyObject_VectorcallTstate (tstate=0xafbe80 <_PyRuntime+299040>, 
    callable=callable@entry=<numpy.ufunc at remote 0x7ffff6e439b0>, args=args@entry=0x7ffff7f929d8, nargsf=9223372036854775809, 
    kwnames=kwnames@entry=0x0) at ../Include/internal/pycore_call.h:168
#11 0x00000000004b96a1 in PyObject_Vectorcall (callable=callable@entry=<numpy.ufunc at remote 0x7ffff6e439b0>, 
    args=args@entry=0x7ffff7f929d8, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at ../Objects/call.c:327
#12 0x00000000005d2525 in _PyEval_EvalFrameDefault (tstate=0xafbe80 <_PyRuntime+299040>, frame=0x7ffff7f928f0, throwflag=0)
...

Looking at C frame 4 I see an apparently bad data = 0x61 pointer in src.

(gdb) p *(PyArrayObject_fields*)dst
$9 = {ob_base = {{ob_refcnt = 1, ob_refcnt_split = {1, 0}}, ob_type = 0x7ffff5f857a0 <PyArray_Type>}, 
  data = 0x12455d0 " \313\343\367\377\177", nd = 1, dimensions = 0x1243f30, strides = 0x1243f38, base = 0x0, 
  descr = 0x7ffff5f7aec0 <DOUBLE_Descr>, flags = 1287, weakreflist = 0x0, _buffer_info = 0x0, 
  mem_handler = <PyCapsule at remote 0x7ffff6ddc600>}

(gdb) p *(PyArrayObject_fields*)src
$6 = {ob_base = {{ob_refcnt = 32, ob_refcnt_split = {32, 0}}, ob_type = 0x7ffff5f857a0 <PyArray_Type>}, 
  data = 0x61 <error: Cannot access memory at address 0x61>, nd = 1, dimensions = 0x1243fb0, strides = 0x1243fb8, base = 0x0, 
  descr = 0x7ffff5f7aec0 <DOUBLE_Descr>, flags = 3, weakreflist = 0x0, _buffer_info = 0x0, mem_handler = 0x0}

[mdavidsaver]

Ah, RR to the rescue. _rec_cache needs to be cleared by IOCHelper after each test cycle. testdbCleanup() frees all PDB records, which invalidates the DBADDR held by the python Record and Field objects.