Repository navigation
Resolve OpenSSL from the conda environment in the foundation image - #10032
quantconnect-kronos[bot] wants to merge 4 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the trailing LD_LIBRARY_PATH reference so the only change is the conda lib directory going first, and add a test asserting that the OpenSSL .NET loads by soname is the python environment's. It fails with the current library path, where .NET maps the system libssl.so.3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The failing That's the same result as my local run on the current image. With This is the situation discussed in #8896: a test added with a foundation change fails until the image is rebuilt. I haven't re-run the job, since it would fail the same way, and I haven't weakened the test. Options:
Which do you prefer? |
There was a problem hiding this comment.
The diagnosis is right, and the regression test catches the bug: it fails on the published quantconnect/lean:foundation image with exactly the #10014 symptom (.NET has mapped the system libssl.so.3). The Cryptography test is fine too. I'm requesting changes because putting the whole /opt/miniconda3/lib/ first does much more than #10014 needs. I checked this in the current foundation image, comparing today's LD_LIBRARY_PATH with the new one.
Requested change: narrow the library path to OpenSSL (you asked about this in the description)
LD_LIBRARY_PATH is searched before a library's own RUNPATH, and every process in the image inherits it. With 599 entries in the conda lib directory, the change moves a lot of libraries outside OpenSSL, for both Python wheels and system binaries:
- ortools ends up running two versions of its solver libraries.
ortools/.libs/libortools.so.9usesRUNPATH $ORIGIN. With the new path,libClp.so.1,libClpSolver.so.1andlibOsiClp.so.1load from conda's coincbc install (Clp 1.14.10, which pulls in conda'slibCoinUtils.so.3andlibOsi.so.1). ortools' own Cbc,libCoinUtils.so.2andlibOsi.so.0stay loaded in the same process./proc/self/mapsafter a CBC and a CLP solve shows both CoinUtils and both Osi. My small LP/MIP gave the same objective values both ways, but ortools' Cbc was built against its bundled Clp 1.17.10, so this is an ABI mix that could give wrong results or crash on other models. - Other libraries that move:
dglandthundergbmswitch to conda'slibgomp. 554RUNPATHlibraries in site-packages (PyQt6, tensorflow, pyarrow, jaxlib, the nvidia libraries, …) now resolvelibstdc++,libgcc_s,libzand others from conda. Xvfb, wkhtmltopdf (reports) and the IB Gateway JRE'slibawt_xawt.soswitch to conda'slibX11/libxcb.apt-get,wget,curl,sshand systempython3switch to conda'slibstdc++,libz,libzstd,libcurl, krb5 and so on. The build steps after thisENVline (apt-get,add-apt-repository,dpkg,wget) also run with the new path. Rendering a PDF with wkhtmltopdf under Xvfb, a JRE AWT window under Xvfb,apt-get,add-apt-repositoryandwgetall still worked for me. Butcurlnow prints/opt/miniconda3/lib/libcurl.so.4: no version information available (required by curl)on every call, including when an algorithm runs it as a subprocess.
A directory holding only links to conda's libssl.so.3 and libcrypto.so.3, placed first, fixes #10014 and changes nothing else. Both libraries depend only on libc/libdl/libpthread and have RPATH $ORIGIN. I tried it: a .NET HttpClient against several HTTPS hosts then maps /opt/miniconda3/lib/libssl.so.3 and libcrypto.so.3, ICU stays at the system's 70, and the HTTPS requests and culture formatting give the same results as today. /proc/self/maps shows the resolved paths, so DotNetLoadsPythonEnvironmentOpenSsl keeps working as written. Please also update the end-to-end checks in the description for the narrowed variant, run with the new path.
CI
Python Virtual Environmentsfails only onDotNetLoadsPythonEnvironmentOpenSsl(OpenSSL loaded outside of /opt/miniconda3: /opt/miniconda3/lib/libcrypto.so.3, /usr/lib/x86_64-linux-gnu/libssl.so.3), with 138 passing. CI runs against the published foundation image, which this Dockerfile change doesn't touch until the image is rebuilt. That matches the earlier guidance on #8896, so I don't count it against the change.- Every other check passed.
Benchmarkswas still running when I submitted.
Other notes, not blocking
DockerfileLeanFoundationARMsets noLD_LIBRARY_PATH, as the description says. It probably needs the same fix in its own pull request.- Images built from the foundation that append to
LD_LIBRARY_PATHinherit the change. No other repository needs an update.
Risk: I set risk:high. This changes how libraries resolve for every process in the foundation image and every image built from it, so the blast radius is wide. It also hasn't been verified on a rebuilt image. Narrowing it to OpenSSL would remove most of that.
| ENV XLA_FLAGS=--xla_gpu_cuda_data_dir=/opt/miniconda3/ | ||
| ENV LD_LIBRARY_PATH=/opt/miniconda3/lib/python3.11/site-packages/nvidia/nvjitlink/lib/:/opt/miniconda3/lib/python3.11/site-packages/nvidia/cuda_nvrtc/lib/:$LD_LIBRARY_PATH | ||
| # conda's lib first: .NET loads libssl/libcrypto by soname before python, and conda extensions then bind to that same copy, which must be conda's | ||
| ENV LD_LIBRARY_PATH=/opt/miniconda3/lib/:/opt/miniconda3/lib/python3.11/site-packages/nvidia/nvjitlink/lib/:/opt/miniconda3/lib/python3.11/site-packages/nvidia/cuda_nvrtc/lib/:$LD_LIBRARY_PATH |
There was a problem hiding this comment.
This puts all 599 entries of the conda lib directory ahead of every library's RUNPATH and ahead of the system libraries, for every process in the image. For example, ortools' libortools.so.9 (RUNPATH $ORIGIN) now loads conda's Clp 1.14.10/libCoinUtils.so.3 next to its own bundled Cbc/libCoinUtils.so.2. Could this point at a directory holding only links to /opt/miniconda3/lib/libssl.so.3 and libcrypto.so.3 instead? That fixes #10014 by itself: I checked that .NET then maps conda's OpenSSL while ICU and everything else stay as they are today.
There was a problem hiding this comment.
Done in 93004d0: conda's libssl.so.3 and libcrypto.so.3 are now linked into /opt/conda-openssl/lib/, and only that directory goes first. With it, the engine maps conda's OpenSSL while ICU stays at 70 and libstdc++ at 6.0.30, as today. ortools maps only its bundled Clp/CoinUtils/Osi, and curl prints no warning.
Link conda's libssl.so.3 and libcrypto.so.3 into their own directory and put that first in LD_LIBRARY_PATH instead of the whole conda lib directory, so every other library keeps resolving as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
I narrowed the change as requested, in 93004d0. I re-ran everything in an image built from the current foundation with the new
I updated the description to match. The Python Virtual Environments check will keep failing on |
|
On 93004d0, Python Virtual Environments (run) fails the same way as on the previous commit, and only there: The workflow runs against the published Syntax, Research Regression and Report Generator passed. Build & Test, Regression Tests and Benchmarks are still running. |
|
Benchmarks failed on 93004d0 (run) on one C# benchmark: This change only edits Python Virtual Environments still fails only on |
There was a problem hiding this comment.
93004d0 does what I asked for in my last review. Only conda's libssl.so.3 and libcrypto.so.3 are linked into /opt/conda-openssl/lib/, and only that directory goes first in LD_LIBRARY_PATH. Every other library resolves as it does today. I'm approving.
What I checked on this commit, in an image built from the current quantconnect/lean:foundation with the PR's RUN and ENV lines on top:
- The fix:
DotNetLoadsPythonEnvironmentOpenSslandCryptographypass with the change. On the current imageDotNetLoadsPythonEnvironmentOpenSslfails with the #10014 symptom (/usr/lib/x86_64-linux-gnu/libssl.so.3). A small .NET program doing HTTPS maps/opt/miniconda3/lib/libssl.so.3/libcrypto.so.3instead of the system copies. - The test:
/proc/self/mapsshows the resolved paths, so the symlinked directory still counts as undersys.prefix. The CI run ofPythonPackagesTestsuses the conda interpreter, not one of the venvs, so the prefix check holds there. - One side effect the description doesn't mention: the OpenSSL that .NET and the system tools load now reads its configuration from conda's
OPENSSLDIR(/opt/miniconda3/ssl) instead of/usr/lib/ssl./usr/bin/openssl version -dshows the switch. So .NET's root certificate store comes from conda's CA bundle (143 roots, a current Mozilla list) instead of the image'sca-certificates20211016 (127 roots). About 40 newer roots are added, such as ISRG Root X2. Roots Mozilla has since removed are dropped, including GlobalSign Root CA R1/R2, Baltimore CyberTrust, Go Daddy and Starfield Class 2, TrustCor and AAA Certificate Services. This is the same store Python'ssslalready uses in this process. I compared .NETHttpClientGETs against about 60 public HTTPS endpoints of brokerages, data providers, PyPI, GitHub and notification services, in the current image and with the change. The results were identical: every reachable host succeeded in both, and none failed on TLS.wgetandcurlstill work. I don't think this needs a code change, but please add it to the description so whoever rebuilds the foundation knows the engine's trust store moves with it. See the inline comment.
CI
Python Virtual Environmentsfails only onDotNetLoadsPythonEnvironmentOpenSsl, with 138 of 139 passing. This is the new regression test running against the published foundation image, which won't include this Dockerfile change until it's rebuilt. A maintainer said in #8896 that this is expected for a foundation fix, so I don't count it against the change. I reproduced the same pass/fail split locally, as described above.Benchmarksfailed on its re-run:EmptySingleSecuritySecondEquityBenchmark(C#) ran at 291k data points per second against a reference of at least 322.2k. The first run of the same job gave 358k. This pull request changes no code that the engine or that benchmark runs: only the Dockerfile, which CI doesn't build, and two tests inPythonPackagesTests. So this is runner noise, not the change.- Every other check passed.
Other repositories: none need an update. Images built on the foundation that append to LD_LIBRARY_PATH inherit the change. As before, DockerfileLeanFoundationARM probably needs the same fix in its own pull request.
Risk: risk:high stays. The change affects how libraries and the OpenSSL configuration resolve for every process in the foundation image and in every image built from it. It also only takes effect, and only gets CI coverage, once the foundation image is rebuilt, which CI doesn't do. An end-to-end check on the rebuilt image before it's published would lower it.
| # conda's OpenSSL first: .NET loads libssl/libcrypto by soname before python, and conda extensions then bind to that same copy, | ||
| # which must be conda's. Only these two are linked so every other library keeps resolving as before | ||
| RUN mkdir -p /opt/conda-openssl/lib && ln -s /opt/miniconda3/lib/libssl.so.3 /opt/miniconda3/lib/libcrypto.so.3 /opt/conda-openssl/lib/ | ||
| ENV LD_LIBRARY_PATH=/opt/conda-openssl/lib/:/opt/miniconda3/lib/python3.11/site-packages/nvidia/nvjitlink/lib/:/opt/miniconda3/lib/python3.11/site-packages/nvidia/cuda_nvrtc/lib/:$LD_LIBRARY_PATH |
There was a problem hiding this comment.
Not blocking: with conda's libcrypto loaded first, .NET and the system tools also take conda's OPENSSLDIR (/opt/miniconda3/ssl). The engine's trusted root store then becomes conda's current Mozilla bundle instead of the image's 2021 ca-certificates. I saw no difference across about 60 HTTPS endpoints. Could the description say so, so the foundation rebuild accounts for it?
|
The Benchmarks re-run (attempt 2) failed again on the same benchmark: Why I don't think this change causes it:
I'm leaving it there, without another re-run. |
There was a problem hiding this comment.
The narrowed OpenSSL-only directory fixes the reported import failure, but the new loader test has one reproducible regression with Lean's supported Python virtual environments. Please correct the prefix it checks; details and reproduction are inline.
Verified on this head
dotnet build Tests/QuantConnect.Tests.csproj -c Releasesucceeds.- Both new tests pass with the Dockerfile change applied. Without it,
DotNetLoadsPythonEnvironmentOpenSslfails on the systemlibssl.so.3, while the Fernet test passes, matching CI. - Broader package runs were not clean: one encountered TensorFlow import failures that also reproduce without the change; with foundation-equivalent library paths, all 138 assertions pass but the host ends with
Test Run Aborted, also seen without the change. I do not count these as a clean full-suite pass. - With the fix applied and
python-venvpointing at a fresh system-site-packages venv, Fernet passes but the loader test fails despite both OpenSSL libraries correctly resolving to conda. - Tested end to end: the original algorithm import fails before the change; afterward, a backtest and paper-trading initialization verify Fernet, Python HTTPS and
self.download, and basic numpy/scipy/pandas computations. The backtest places an order. OpenSSL resolves to conda while ICU and libstdc++ remain the system copies. The paper-trading test was stopped. - A synthetic CA installed with
update-ca-certificatesremains trusted by .NET HTTPS after the change. In this image, .NET falls back to/etc/ssl/certsbecause conda's certificate directory is empty; the configuration-directory change does not simply replace the entire .NET trust store.
CI: I inspected both failing checks. Python Virtual Environments fails only on the new loader test against the published foundation image, which does not yet contain this Dockerfile change; the local before/after result confirms the author's diagnosis. Benchmarks reports 291k versus a 322.2k threshold for the C# empty-security benchmark. This PR changes no engine or benchmark runtime code, and the only commit since the previous passing benchmark changes the Dockerfile that CI does not build. These failures are separate from the virtual-environment regression above. All other checks passed.
Other repositories: no companion source updates are required; consumers that append to the inherited library path retain the fix. DockerfileLeanFoundationARM remains unchanged, and I have not verified the fix on ARM.
Risk: retaining risk:high. Even narrowed to two libraries, this changes process-wide OpenSSL resolution and configuration for the foundation and its derived images. Verification of the amended existing images is useful evidence, but is not a full foundation rebuild.
| using (Py.GIL()) | ||
| { | ||
| using var sys = Py.Import("sys"); | ||
| pythonPrefix = sys.GetAttr("prefix").As<string>(); |
There was a problem hiding this comment.
[P2] Check the base interpreter prefix, not the activated venv prefix. AssemblyInitialize activates the configured python-venv, and PythonInitializer.TryInitPythonVirtualEnvironment() deliberately sets sys.prefix to that venv while keeping PYTHONNET_PYDLL and native libraries in the base installation. I reproduced this with the Dockerfile fix applied: create python -m venv /lean-testenv --system-site-packages, set "python-venv": "/lean-testenv" in the test configuration, and run the two new tests. Fernet passes, but this test fails with OpenSSL loaded outside of /lean-testenv: /opt/miniconda3/lib/libcrypto.so.3, /opt/miniconda3/lib/libssl.so.3. Those are exactly the desired libraries. Please compare against the base interpreter's prefix (for example, sys.base_prefix) so the regression assertion also accepts correctly configured virtual environments.
There was a problem hiding this comment.
Done in 193f07a: the test now compares against sys.base_prefix. I reproduced your case first: with the fix applied and python-venv set to a --system-site-packages venv, the old check failed with OpenSSL loaded outside of /lean-testenv: /opt/miniconda3/lib/libcrypto.so.3, /opt/miniconda3/lib/libssl.so.3. With base_prefix it passes.
A configured python virtual environment sets sys.prefix to the venv while native libraries stay in the base installation, so check sys.base_prefix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed the review in 193f07a. I ran
As a control, the previous |
|
On 193f07a, Python Virtual Environments (run) fails only on It now reports Syntax, Report Generator and Research Regression passed. Build & Test, Regression Tests and Benchmarks are still running. |
There was a problem hiding this comment.
193f07a fixes the remaining request from the other review: DotNetLoadsPythonEnvironmentOpenSsl now compares against sys.base_prefix. I'm approving.
What I checked on this commit
- Only the test changed since 93004d0, which I approved. The Dockerfile is the same.
PythonInitializer.TryInitPythonVirtualEnvironment(Common/Python/PythonInitializer.cs:275-281) setssys.prefix/sys.exec_prefixto the venv but leavessys.base_prefixalone. So with a configuredpython-venv, the check now compares against the base installation, which is where conda'slibssl/libcryptolive. Without a venv,base_prefixequalsprefix, so the regression check is as strong as before. CI shows that: it still reports the system/usr/lib/x86_64-linux-gnu/libssl.so.3on the published image.- The developer's before/after table covers all four cases (with and without the change, with and without a venv), and the control run reproduced the failure the other reviewer reported. I have nothing more to ask for here.
A correction to my earlier note on the trust store (not blocking): the other review found that conda's /opt/miniconda3/ssl/certs is empty, and I confirmed that in the foundation image. In their test, .NET still trusted a CA added with update-ca-certificates. So the switch to conda's OPENSSLDIR adds conda's cacert.pem to what .NET trusts, and doesn't replace the system store as my inline comment said. The description could still mention that the OpenSSL configuration directory moves to conda's, but it's optional.
CI
Python Virtual Environmentsfails only onDotNetLoadsPythonEnvironmentOpenSsl, with 138 of 139 passing:OpenSSL loaded outside of /opt/miniconda3: /opt/miniconda3/lib/libcrypto.so.3, /usr/lib/x86_64-linux-gnu/libssl.so.3. This is the new regression test catching #10014 on the published foundation image, which won't include this Dockerfile change until it's rebuilt. That's expected for a foundation fix, as discussed in #8896, and the developer and both reviewers got the matching pass with the change applied. I don't count it against the change.- Build & Test, Regression Tests, Benchmarks, Syntax, Research Regression and Report Generator all passed on this commit. Benchmarks passing confirms the earlier failure was runner noise.
Other repositories: none need an update. Images built on the foundation inherit the LD_LIBRARY_PATH. DockerfileLeanFoundationARM probably needs the same fix in its own pull request.
Risk: risk:high stays. The change affects how OpenSSL and its configuration resolve for every process in the foundation image and every image built from it. CI only covers it once the image is rebuilt. An end-to-end check on the rebuilt foundation before it's published, with the loader test going green there, is what would lower it.
There was a problem hiding this comment.
193f07a resolves my previous request. The only change since my review is the switch from sys.prefix to sys.base_prefix and its explanation. PythonInitializer.TryInitPythonVirtualEnvironment changes the active prefixes, not the base installation, so this now accepts correctly configured virtual environments without weakening detection of the system OpenSSL. No remaining blocking findings.
Verified on this head
dotnet build Tests/QuantConnect.Tests.csproj -c Releasesucceeds with 0 errors.- Ran
DotNetLoadsPythonEnvironmentOpenSslandCryptographyin separate test hosts for all four combinations below, using the repository's test data and a fresh system-site-packages venv configured throughpython-venv.
| Loader settings | No venv | Configured venv |
|---|---|---|
| Original | Loader test fails on system libssl.so.3; Fernet passes |
Same result |
| Dockerfile change applied | Both pass | Both pass |
The configured-venv run confirms sys.prefix is /lean-testenv while sys.base_prefix remains /opt/miniconda3. Both successful runs finish normally. Initial attempts using a much larger data directory timed out during discovery before executing tests; the results above use the repository fixtures. I did not repeat the full package suite or the end-to-end runs from my preceding review for this test-only amendment; the Dockerfile and runtime behavior are unchanged.
CI: I checked the current head's failing logs. Python Virtual Environments fails only on DotNetLoadsPythonEnvironmentOpenSsl, with 138 passing, reporting the same system libssl.so.3 as the original-loader controls above. Its workflow uses the published foundation image and does not build this Dockerfile. This is the existing image defect caught by the regression test, pending the foundation rebuild, consistent with the maintainer's guidance in #8896. All other active checks, including Benchmarks, passed; API Tests is skipped.
Other repositories: no companion source updates are required. Derived images retain the fix through the inherited library path. ARM remains unchanged and was not tested.
Risk: retaining risk:high. Even narrowed to two libraries, the Dockerfile changes process-wide OpenSSL resolution and configuration for the foundation and its derived images. Testing with the loader change applied is not a full foundation rebuild; an end-to-end check and passing loader regression test on the fully rebuilt foundation remain important before publishing it.
Closes #10014
Description
The engine's .NET runtime loads
libssl.so.3/libcrypto.so.3by soname before the algorithm's Python module is imported. Conda's lib directory isn't onLD_LIBRARY_PATH, so .NET gets the system OpenSSL 3.0.2, which is builtno-idea. Conda extensions loaded later bind to that already-loaded copy instead of the conda OpenSSL 3.6.0 they were built against, socryptography's_rust.abi3.sofails withundefined symbol: EVP_idea_ecb.DockerfileLeanFoundation: symlinks conda'slibssl.so.3andlibcrypto.so.3into/opt/conda-openssl/lib/and puts that directory first inLD_LIBRARY_PATH.RPATH $ORIGIN.LD_LIBRARY_PATHentries stay as they are, including the trailing$LD_LIBRARY_PATH. Images built on the foundation that append toLD_LIBRARY_PATHinherit the change.PythonPackagesTests:Cryptography: a Fernet encrypt/decrypt round trip. This is the commit from Fix OpenSSL loading for Python cryptography #10026, cherry-picked with its author kept.DotNetLoadsPythonEnvironmentOpenSsl: has .NET load its OpenSSL, then asserts that every mappedlibssl/libcryptolives under the Python installation's prefix (sys.base_prefix, so a configuredpython-venvis handled too). It runs first in the fixture because a Python extension loading conda's libssl earlier hides the problem.Both tests run in the Python Virtual Environments workflow against the published
quantconnect/lean:foundationimage.DotNetLoadsPythonEnvironmentOpenSslfails there until the image is rebuilt with this change.DockerfileLeanFoundationARMsets noLD_LIBRARY_PATH, so it may hit the same conflict. It isn't changed here.Testing
I tested in an image built from the current
quantconnect/lean:foundationwith this change'sRUNandENVlines added on top:PythonPackagesTests, CI's filter (139 tests)DotNetLoadsPythonEnvironmentOpenSsl,OpenSSL loaded outside of /opt/miniconda3: /opt/miniconda3/lib/libcrypto.so.3, /usr/lib/x86_64-linux-gnu/libssl.so.3(same as this PR's CI run)TestCategory!=TravisExclude&TestCategory!=ResearchRegressionTests)In the same image:
curl https://…returns 200 with no library warnings./usr/bin/openssl versionreportsOpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.6.0 1 Oct 2025).libClp/libCoinUtils.so.2/libOsi.so.0.I also tested end to end with this change, with a Python algorithm in a backtest and in live paper trading:
ImportError: ... _rust.abi3.so: undefined symbol: EVP_idea_ecb, withssl.OPENSSL_VERSIONreporting 3.0.2.cryptographyandsslboth reportOpenSSL 3.6.0 1 Oct 2025.libssl.so.3/libcrypto.so.3, while ICU (libicuuc.so.70.1) andlibstdc++.so.6.0.30stay the system ones, as today.urllib,requestsandself.downloadworks, and .NET culture and string handling give the same results.🤖 Generated with Claude Code