Mental model
Building OpenSSL from source takes three steps: ./config (which features, installed where), make (compile), make install (copy). PQC does not need a fourth step, because from OpenSSL 3.5 onward ML-KEM, ML-DSA and SLH-DSA are part of the default provider; no separate flag or module is needed. But installing with a package manager (as you will see in the RHEL/RockyLinux example) is often simpler and less error-prone than building from source. This lesson shows both paths as they were actually tested.
Correction: there is no special PQC flag
The early research notes reviewed while preparing this lesson suggested three different ./config flags on three different days: one used no flag at all, one said enable-pqc, one said enable-pq. All three were trying to do the same thing (build OpenSSL 3.5 with PQC support), but all three differed, and two of them (enable-pqc, enable-pq) are not real ./config options. No such flag is defined in OpenSSL’s official INSTALL.md.
The command sequence in this lesson was actually run on 2026-09-04 with no special flags (only shared, a standard option that builds the libraries as shared objects), and the result was verified. That proves none of the three contradictory instructions was needed. But producing that proof also walked into an interesting trap, described below, because it is exactly the kind of error this module is meant to teach.
A version mismatch, caught live
Right after the build above, running /opt/openssl-3.5/bin/openssl version printed: OpenSSL 3.5.1 1 Jul 2025 (Library: OpenSSL 3.5.7 9 Jun 2026). Two different versions on one line. The CLI itself (3.5.1) is the version you just built; the “Library” in parentheses (3.5.7) is the shared library actually loaded at run time. They should match, and they did not.
Why: when libssl-dev was installed in the build environment, the system package manager had already installed a libssl3 runtime library as a dependency (here Debian trixie’s own 3.5.7). When the newly built openssl CLI ran without LD_LIBRARY_PATH set, it found the system’s libcrypto.so.3 through the operating system’s standard library search path and used that, not its own library just installed under /opt/openssl-3.5/lib.
The fix is one line:
LD_LIBRARY_PATH=/opt/openssl-3.5/lib /opt/openssl-3.5/bin/openssl version
# OpenSSL 3.5.1 1 Jul 2025 (Library: OpenSSL 3.5.1 1 Jul 2025) <- now they matchAfter this fix, ML-DSA-65 key generation worked without problems, really using the newly built library. This is not an accidental find but a deliberately recorded teaching example: looking at a CLI’s version output does not guarantee which library actually runs. In production, this kind of mismatch can mean believing “we moved to PQC” while silently still using an old library.
RHEL / RockyLinux 9: with the package manager, but carefully
On Debian/Ubuntu with apt things are relatively simple. The RHEL family (tested on RockyLinux 9, binary compatible with RHEL 9) is more instructive: on a fresh image, openssl version -a shows 3.0.7MEASURED, with no PQC, built with a Red Hat FIPS patch. Just running dnf install openssl (which upgrades the existing install to the current repository version) moves it to 3.5.5MEASURED, and on that version ML-DSA-65 key generation really works.
What this means: on a RHEL 9-based system, the answer to “which OpenSSL version is installed” changes depending on when the image was built and when packages were last updated. For an inventory (see M12), this is another example of why “the image name is X” is not enough on its own.
WSL2 and macOS/Homebrew
WSL2 (Windows Subsystem for Linux 2) is a virtual machine running a real Linux kernel. An Ubuntu or Debian distribution inside it behaves exactly like the apt-based steps above, because at the level of these commands there is no difference between “native Linux” and “Linux inside WSL2”. This lesson gives no separate WSL2 sequence because none is needed: open an Ubuntu shell in WSL2 and run the steps from the apt section as they are.
macOS (Homebrew). The Homebrew install on the machine used for this course was checked: the openssl@3 formula (alias openssl@3.6) provides version 3.6.2, which is above the 3.5+ threshold and supports ML-KEM, ML-DSA and SLH-DSA natively. If you already use Homebrew, brew install openssl@3 (or brew upgrade openssl@3 to update) is enough; no separate source build is needed.
The /opt/oqs static-lib/no-DSO error: the real text
When loading a provider (for example oqsprovider) into OpenSSL, OpenSSL looks for a shared library (.so file, a “dynamic shared object” or DSO). If an installation has only a static library (.a file) and no .so, loading fails. This scenario was actually reproduced in this lesson (with an empty .a file and a missing .so), and OpenSSL’s real error is given in full in the lab section above.
When you see this error, check two things: (1) does the OPENSSL_MODULES environment variable really point at the directory with the .so file, and (2) is there really a .so file in that directory, or only an .a. But first ask: do you actually need this provider? For ML-KEM, ML-DSA and SLH-DSA, no: the default provider is enough. Only for an algorithm OpenSSL does not support yet (for example FN-DSA/Falcon, see M4).