OpenSSL in Red Hat Enterprise Linux 10: From engines to providers

OpenSSL is a popular cryptographical toolkit with more than 20 years of history. For a long time, the only way to extend it was by using an "engine", which defines how a cryptographic algorithm is computed. This could include hardware devices and even new algorithms not included in the main library, but as OpenSSL evolved it became evident that the engines API was limiting. A new pluggable system, called a "provider", was introduced.

What is a provider

A provider, in OpenSSL terms, is a unit of code that provides one or more implementations of cryptographic operations, making new algorithms available. It's also a dynamically loaded plugin that can be loaded during OpenSSL initialization, as long as it's specified in the OpenSSL configuration file. It can also be loaded on demand, which is vital when you're using legacy (insecure) algorithms when interacting with a very old system, or when you're experimenting with a very new algorithm.

The provider concept was introduced into OpenSSL during the development of version 3.0. It was inevitable that some new concept would get implemented to reach one of the main goals of the OpenSSL 3.0 design: Maintainable FIPS-140-3 certified modules. The previous approach of implementing numerous runtime checks was hardly sustainable, and FIPS requirements are a moving target, so it was important to implement a more robust model.

The engine API didn't allow separating FIPS and non-FIPS algorithms and implementations in a robust manner, nor did it ensure that the data didn’t cross the FIPS boundary back and forth, breaking FIPS requirements. The provider API was designed from scratch, and the engine API has been deprecated. Deprecated APIs may be removed in future major releases.

What has changed

OpenSSL 3.0 was introduced in Red Hat Enterprise Linux (RHEL) 9. It came with three providers covering all the main use cases, including the FIPS module.

In RHEL 9, there were no other mature providers, so we had no valid replacement for the engines that were regularly used in RHEL (the most important one being the PKCS#11 engine). We supported engines for the lack of any other option. However, since then we've worked hard to provide a replacement option for existing engines. In RHEL 10, we have a new PKCS#11 provider. We've also added several more providers.

Modifying applications with explicit support for engines

For ABI compatibility reasons, we are unable to compile OpenSSL without engine support. But in the forthcoming RHEL 10, we're removing header file declarations of engine-specific functions. By enforcing the definition of OPENSSL_NO_ENGINE, we disable engine support in applications that are properly detecting engine support at build time. We found a low number of applications that use engines in a way that requires significant code changes. However, most applications respect the OPENSSL_NO_ENGINE definition and don't require any code changes.

Some applications include the openssl/engine.h header file even when they support the OPENSSL_NO_ENGINE definition. In RHEL 10, we're changing the engine.h header to be a dummy, empty file. Some applications have custom mechanisms to detect the presence of engines, but it's often incorrect. Some expect that engines are always available in OpenSSL, but not in OpenSSL forks. All these applications require changes, both upstream and downstream.

We introduced the same change in Fedora. The community did not approve the idea of just removing the engine.h file there, so we moved it to a separate package and defined the OPENSSL_NO_ENGINE symbol only when that header is not installed.

Notable providers in RHEL 10

RHEL 10 has a new PKCS#11 provider. It fully encompasses the functionality of the old PKCS#11 engine, and allows tuning for some specific tokens. It’s extensively tested by internal and external customers and has no major problems we're aware of.

We have also included a TPM2 provider, another hardware-backed solution. TPM2 is a chip present in most modern computers. It establishes that the boot process starts from a trusted combination of hardware and software, and helps establish a trusted execution environment. The TPM2 provider allows users to operate this chip and use its keys through the OpenSSL API.

Finally, we've added the OQS provider, which brings post-quantum algorithms to OpenSSL. The post-quantum cryptography transition is one of the biggest cryptographic challenges of the RHEL 10 lifetime. This provider allows end users and developers to conduct experiments with post-quantum cryptography until OpenSSL adds native support.

Conclusion

A major version change is generally a good time to drop deprecated functionality. We are not aware of any common scenarios previously covered by engines that are not covered by providers. Applications now do not need to care about implementation details. Instead, code can be written in a way that transparently uses a key managed by a provider, as opposed to previous use of engines requiring custom code.