linecache.cache and linecache.getlines() are not documented

[zzzeek]

Documentation

linecache.cache is not underscored, and is widely used throughout the python ecosystem to provide source context for dynamically generated Python functions. Examples include attrs

https://github.com/python-attrs/attrs/blob/764bf92a1c96abe4615b59f3e5a7b738b0340a94/src/attr/_make.py#L230-L263

as well as ipython:

https://github.com/ipython/ipython/blob/19f9ae0a863c21cff52fa07c74b18fc5b482d9c3/IPython/core/compilerop.py#L138-L182

a code search on github reports over 700 files accessing this attribute directly and assuming it's a dict:

https://github.com/search?q=linecache.cache+language%3APython&type=code&p=1

the utility of linecache.cache is significant, in that it allows us to provide real source lines for generated code that comes up in stacktraces, and which is also walkable in pdb. The linecache.lazycache() documented API almost provides this functionality, but is explicitly blocked for string-oriented "filenames" like <string>.

Without linecache.cache:

broken_function="""\
def go():
    print("step1")
    print("step2")
    x = 1 / 0
    print("step3")
"""

code = compile(broken_function, "<custom go() function>", "exec")

exec(code, env := {})

env["go"]()

this code produces a stack trace that sees the generated function as opaque:

$ python test3.py
step1
step2
Traceback (most recent call last):
  File "/home/classic/dev/sqlalchemy/test3.py", line 14, in <module>
    env["go"]()
    ~~~~~~~~~^^
  File "<custom go() function>", line 4, in go
ZeroDivisionError: division by zero

however add linecache and suddenly we get much more detail:

import linecache

broken_function = """\
def go():
    print("step1")
    print("step2")
    x = 1 / 0
    print("step3")
"""

filename = "<custom go() function>"
code = compile(broken_function, filename, "exec")

entry = (
    len(broken_function),
    None,
    broken_function.splitlines(True),
    filename,
)
linecache.cache[filename] = entry

exec(code, env := {})

env["go"]()

now we get introspection into the generated function itself:

$ python test3.py
step1
step2
Traceback (most recent call last):
  File "/home/classic/dev/sqlalchemy/test3.py", line 24, in <module>
    env["go"]()
    ~~~~~~~~~^^
  File "<custom go() function>", line 4, in go
    x = 1 / 0
        ~~^~~
ZeroDivisionError: division by zero

The None used as the second entry of the tuple is so that checkcache() skips asserting that the filename is actually a file; this is commented by the ipython project in the link above.

There is one documented API that is close to this use case linecache.lazycache(), but it's explicitly disallowed from working with the universal convention for generated code, a name in angle brackets ( see

cpython/Lib/linecache.py

Lines 224 to 226 in af4ee51

	def _make_lazycache_entry(filename, module_globals):
	if not filename or (filename.startswith('<') and filename.endswith('>')):
	return None

), and also for this use case would require us to fabricate a loader object with a get_source() method and a module_globals dict carrying __name__.

from my perspective it seems that the behavior of compile() could be improved directly by providing options to populate the linecache, or alternatively that lazycache() could accommodate the angle-bracket names that generated code actually uses, however neither seems to be an option today. Hence we see widespread direct manipulation of linecache.cache, and this technique has by now escaped containment and is being recommended by robots such as Claude.

I am seeking clarification on why this attribute has remained undocumented for so long, if this is in fact public API, or if it's in some kind of "public, but we really don't like people doing this" sort of limbo (and if so, is there an opening for actual features to be proposed).

[picnixz]

I do not think the cache should be documented but we cannot make it private I guess. I would rather habe a function for altering the cache directly (bypassing any other approaches) rather than directly altering it.

[zzzeek]

well what is the meaning of "public attribute but dont document it" ? that seems awfully ambiguous ? why not make a decision - like, deprecate it. or add a new function, or whatever. it's causing me problems in its current state.