Skip to content

DM-53098 : Convert ctrl_bps_htcondor to use htcondor2 - #101

Merged
tcjennings merged 8 commits into
mainfrom
tickets/DM-53098
Oct 8, 2026
Merged

tcjennings merged 8 commits into
mainfrom
tickets/DM-53098

Conversation

@tcjennings

@tcjennings tcjennings commented Aug 20, 2026 •

Copy link
Copy Markdown
Contributor

Migrates htcondor -> htcondor2 for HTCondor Python bindings 24.0.x (LTS) and beyond.

Supports both 24.0.x and 25.0.x LTS client packages. (closes #71)

For 24.0.x only, the legacy SecMan.ping method is used. In 24.x and beyond, SecMan is deprecated and removed. The ping function has been added to the module level (htcondor2.ping) except in 24.0.x.

Because htcondor2 job events do not read or parse a "Toe" tag from event entries, the JobCompleted handlers have been refactored into a single class definition to support the new Job Event ClassAd standard.

Migrates project management to uv. Updates GHA workflows for uv and security (closes #107) (closes #18).

Todo

  • Validate DagmanConfigurator params
  • Refactor and deprecate "ToE"-related logic

Checklist

  • ran Jenkins
  • added a release note for user-visible changes to doc/changes

@codecov

codecov Bot commented Aug 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 79.08745% with 55 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.19%. Comparing base (69b3ae7) to head (541434d).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
python/lsst/ctrl/bps/htcondor/lssthtc.py 53.19% 21 Missing and 1 partial ⚠️
python/lsst/ctrl/bps/htcondor/htcondor_compat.py 47.05% 18 Missing ⚠️
python/lsst/ctrl/bps/htcondor/htcondor_service.py 66.66% 7 Missing and 1 partial ⚠️
python/lsst/ctrl/bps/htcondor/common_utils.py 58.82% 6 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #101      +/-   ##
==========================================
- Coverage   88.41%   88.19%   -0.22%     
==========================================
  Files          21       23       +2     
  Lines        5067     5059       -8     
  Branches      504      502       -2     
==========================================
- Hits         4480     4462      -18     
- Misses        499      510      +11     
+ Partials       88       87       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@tcjennings
tcjennings force-pushed the tickets/DM-53098 branch 4 times, most recently from dafde1d to 57ad5ae Compare August 20, 2026 21:23
@tcjennings
tcjennings force-pushed the tickets/DM-53098 branch 2 times, most recently from 3606836 to 89678b5 Compare September 3, 2026 15:38
@tcjennings tcjennings changed the title DM:53098 : Convert ctrl_bps_htcondor to use htcondor2 DM-53098 : Convert ctrl_bps_htcondor to use htcondor2 Sep 10, 2026
@tcjennings
tcjennings force-pushed the tickets/DM-53098 branch 2 times, most recently from 9a23aae to 5bc5233 Compare September 11, 2026 15:59
@tcjennings
tcjennings marked this pull request as ready for review September 11, 2026 16:01
@tcjennings

Copy link
Copy Markdown
Contributor Author

This PR will require a repository admin to change the configuration for "required status checks" in branch protection to require only the bare test status check, which is a summary job for the entire test matrix.

@tcjennings
tcjennings force-pushed the tickets/DM-53098 branch 3 times, most recently from c4342bc to c32d8bc Compare September 11, 2026 16:24
@tcjennings
tcjennings force-pushed the tickets/DM-53098 branch 4 times, most recently from f3d702f to dc368ac Compare September 14, 2026 19:07
Comment thread python/lsst/ctrl/bps/htcondor/htcondor_compat.py Outdated
raise HTCondorException("Unable to locate daemon.") from e
except HTCondorIOError as e:
raise HTCondorException("Unable to connect to daemon.") from e
secman = None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this assignment here? Seems necessary.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's intended here is to clear any references to objects in the legacy htcondor module and "unload" the module, but this is probably better done with del statements.

Comment thread python/lsst/ctrl/bps/htcondor/htcondor_compat.py
Comment thread python/lsst/ctrl/bps/htcondor/lssthtc.py
Comment thread python/lsst/ctrl/bps/htcondor/lssthtc.py Outdated
Comment thread tests/test_handlers.py Outdated
@tcjennings
tcjennings force-pushed the tickets/DM-53098 branch 6 times, most recently from 144ed13 to 68c585d Compare October 1, 2026 18:23
- use `htcondor2` imports instead of `htcondor`
- correct patch objects in tests
- refactor `htcondor2`-induced changes
- log htcondor package version in wms service prepare
- write a fake ToE for events.
- mark ToE-related classes deprecated
- removes deprecated JobStatus enum in favor of `htcondor.JobStatus`
- update dagman configuration model
- for htcondor 24.0 LTS, use the old ping API
- pin htcondor to 24|25
- add matrix test for htcondor versions
- add towncrier fragment
- move to uv
- update github actions
- run ci tests in a python/htcondor matrix
- reduce `JobCompleted*Handler` classes to a single `JobCompletedHandler`
- remove ToE generation and expectation code
- improve type handling in handler tests
- import specific objects from `classad2`
- incidental docstring and typing improvement
- add an explicit "queue 1" to Submit ads when using `htcondor==24.0.*`
- use `itemdata=None` when submitting to a Schedd.
- improve docstrings
- clean up object references in legacy ping with `del`
- improve exception catching
- bump codeql action to 4.38.2
- rebase against main and migrate new work to `htcondor2`
@tcjennings
tcjennings merged commit a77d303 into main Oct 8, 2026
21 of 23 checks passed
@tcjennings
tcjennings deleted the tickets/DM-53098 branch October 8, 2026 17:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ANN: Please migrate away from pytest-openfiles

2 participants