Skip to content

gh-159207: Store Tachyon samples in bounded columnar blocks #160313

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ivonastojanovic wants to merge 3 commits into
base: main
Choose a base branch
from
+2,419 −1,025
Open

gh-159207: Store Tachyon samples in bounded columnar blocks #160313

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8828b33
Store Tachyon samples in bounded columnar blocks
ivonastojanovic Oct 20, 2026
8865260
Clean up binary format tests
ivonastojanovic Oct 22, 2026
442fc17
fixup! Store Tachyon samples in bounded columnar blocks
ivonastojanovic Oct 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
296 changes: 198 additions & 98 deletions InternalDocs/profiling_binary_format.md
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion Lib/profiling/sampling/binary_collector.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ def get_stats(self):
"""Get encoding statistics.

Returns:
Dict with encoding statistics including repeat/full/suffix/pop-push
Dict with encoding statistics including repeat/full/suffix/reuse
record counts, frames written/saved, and compression ratio.
"""
return self._writer.get_stats()
Expand Down
8 changes: 4 additions & 4 deletions Lib/profiling/sampling/sample.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -411,22 +411,22 @@ def _print_binary_stats(self, collector):
repeat_samples = stats.get('repeat_samples', 0)
full_records = stats.get('full_records', 0)
suffix_records = stats.get('suffix_records', 0)
pop_push_records = stats.get('pop_push_records', 0)
reuse_records = stats.get('reuse_records', 0)
total_records = stats.get('total_records', 0)

if total_records > 0:
repeat_pct = repeat_records / total_records * 100
full_pct = full_records / total_records * 100
suffix_pct = suffix_records / total_records * 100
pop_push_pct = pop_push_records / total_records * 100
reuse_pct = reuse_records / total_records * 100
else:
repeat_pct = full_pct = suffix_pct = pop_push_pct = 0
repeat_pct = full_pct = suffix_pct = reuse_pct = 0

print(f" Records: {total_records:,}")
print(f" RLE repeat: {repeat_records:,} ({ANSIColors.GREEN}{repeat_pct:.1f}%{ANSIColors.RESET}) [{repeat_samples:,} samples]")
print(f" Full stack: {full_records:,} ({full_pct:.1f}%)")
print(f" Suffix match: {suffix_records:,} ({suffix_pct:.1f}%)")
print(f" Pop-push: {pop_push_records:,} ({pop_push_pct:.1f}%)")
print(f" Stack reuse: {reuse_records:,} ({reuse_pct:.1f}%)")

frames_written = stats.get('total_frames_written', 0)
frames_saved = stats.get('frames_saved', 0)
Expand Down
694 changes: 660 additions & 34 deletions Lib/test/test_profiling/test_sampling_profiler/test_binary_format.py
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions Lib/test/test_profiling/test_sampling_profiler/test_cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -954,15 +954,15 @@ def test_cli_replay_reader_errors_exit_cleanly(self):
mock.patch("sys.argv", ["profiling.sampling.cli", "replay", profile]),
mock.patch(
"profiling.sampling.cli.BinaryReader",
side_effect=ValueError("Unsupported format version 2"),
side_effect=ValueError("Unsupported format version 3"),
),
):
with self.assertRaises(SystemExit) as cm:
main()

self.assertEqual(
str(cm.exception),
"Error: Unsupported format version 2",
"Error: Unsupported format version 3",
)

def test_cli_replay_propagates_recorded_mode(self):
Expand Down
2 changes: 1 addition & 1 deletion Makefile.pre.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3452,7 +3452,7 @@ MODULE__CTYPES_MALLOC_CLOSURE=@MODULE__CTYPES_MALLOC_CLOSURE@
MODULE__ELEMENTTREE_DEPS=$(srcdir)/Modules/pyexpat.c @LIBEXPAT_INTERNAL@
MODULE__HASHLIB_DEPS=$(srcdir)/Modules/hashlib.h $(srcdir)/Modules/_openssl_mem.h
MODULE__IO_DEPS=$(srcdir)/Modules/_io/_iomodule.h
MODULE__REMOTE_DEBUGGING_DEPS=$(srcdir)/Modules/_remote_debugging/_remote_debugging.h $(srcdir)/Modules/_remote_debugging/gc_stats.h
MODULE__REMOTE_DEBUGGING_DEPS=$(srcdir)/Modules/_remote_debugging/_remote_debugging.h $(srcdir)/Modules/_remote_debugging/binary_io.h $(srcdir)/Modules/_remote_debugging/gc_stats.h

# HACL*-based cryptographic primitives
MODULE__MD5_DEPS=$(srcdir)/Modules/hashlib.h $(LIBHACL_MD5_HEADERS) $(LIBHACL_MD5_LIB_@LIBHACL_LDEPS_LIBTYPE@)
Expand Down
3 changes: 3 additions & 0 deletions Misc/NEWS.d/next/Library/2026-10-20-12-49-32.gh-issue-159207.Lccezg.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
The :mod:`profiling.sampling` binary output format now stores samples in
independent blocks. This is a breaking format change: Python 3.16 does not
support reading profiles written in the previous format.
Loading
Loading