Use service mesh and mTLS to establish secure routes and TLS termination

The first two articles in this series covered establishing secure routes and TLS termination and gave an example of using wildcard certificates to secure end-to-end TLS.

This article will explore another approach to securing TLS and illustrate it with a command-line example. It uses Red Hat OpenShift Service Mesh (based on the Istio project), which includes a reverse proxy called Ingress-Gateway implemented by Envoy.

[ Learn how to build a flexible foundation for your organization. Download An architect's guide to multicloud infrastructure. ]

What are mTLS and service mesh?

Mutual Transport Layer Security (mTLS) is a protocol that allows two parties to authenticate each other using certificates. It can be used with more general protocols, such as Internet Key Exchange (IKE), Secure Shell (SSH), and Transport Layer Security (TLS) and can be implemented without making changes to the application or service code. In a service mesh, mTLS is typically handled by the infrastructure, using sidecar proxies to establish secure connections between workloads.

Istio is a service mesh that can securely provision strong identities to every workload using X.509 certificates. Istio agents, which run alongside Envoy proxies, work with istiod to automate the rotation of keys and certificates at scale. The diagram and summary below illustrate the Istio identity-provisioning flow, as described in the Istio docs.

1. istiod offers a gRPC service to take certificate signing requests (CSRs).
2. When started, the Istio agent creates the private key and CSR and then sends the CSR with its credentials to istiod for signing.
3. The certificate authority (CA) in istiod validates the credentials carried in the CSR. Upon success, it signs the CSR to generate the certificate.
4. When a workload starts, Envoy requests the certificate and key from the Istio agent in the same container via the Envoy secret discovery service (SDS) API.
5. The Istio agent sends the certificates received from istiod and the private key to Envoy via the Envoy SDS API.
6. The Istio agent monitors the expiration of the workload certificate. The above process repeats periodically for certificate and key rotation.

By default, mTLS in OpenShift Service Mesh is enabled and set to permissive mode, where the sidecars in Service Mesh accept both plain-text traffic and connections encrypted using mTLS.

Enabling mTLS across the mesh at the control plane level secures all the traffic in the service mesh without rewriting hosted applications and workloads. You can secure namespaces in the mesh at the data-plane level in the ServiceMeshControlPlane resource. To customize traffic encryption connections, namespaces at the application level should be configured with PeerAuthentication and DestinationRule resources.

Within Istio, the Ingress-Gateway always operates in reencrypt mode. A company-signed certificate must be supplied to the Ingress-Gateway. You can use cert-manager to do this because the Ingress-Gateway consumes certificates from secrets.

High-level architecture

The components above form the following general architecture:

You can configure the internal connections in the mesh to use mTLS. The Istio Certificate Authority automatically generates certificates to support mTLS connections and injects them into the application pods. In this case, the use of mTLS carries an additional benefit since it allows administrators to create role-based access control (RBAC) rules in the OpenShift cluster to specify which client can connect to what services.

[ Simplify your security operations center. ]

Enable strict mTLS across the service mesh

Because hosted workloads do not communicate with outside services, you can quickly enable mTLS across the mesh without communication interruptions. Do that by setting spec.security.dataPlane.mtls to true in the ServiceMeshControlPlane resource. The Operator then creates the required resources.

Here is an example:

apiVersion: maistra.io/v2
kind: ServiceMeshControlPlane
spec:
  version: v2.2
  security:
    dataPlane:
      mtls: true

Furthermore, you can create a policy to configure mTLS more granularly for individual services. To see how to configure sidecars for incoming or outgoing connections for specific services, please refer to OpenShift Service Mesh's documenation.

Page 5 of the U.S, Department of Defense Enterprise DevSecOps Reference Design requires:

• Use of a service mesh within the Kubernetes orchestrator to manage all east-west networks.
• Mandatory adoption of the Sidecar Container Security Stack (SCSS) to implement zero trust down to the container/function level, also providing behavior protection.

The same document calls out service mesh on page 21 for its ability to "...enforce zero trust mTLS traffic for east/west traffic."

[ Learn the basics in the Kubernetes cheat sheet. ]

Wrap up

The next and final part in this series will examine the use of Keycloak, an open source tool for single sign-on (SSO) and identity and access management. It provides many features and capabilities, including support for end-to-end TLS and secure protocols and algorithms for encrypting data in transit. Keycloak, which is the basis for Red Hat Single Sign-On, is a viable alternative to most of the proprietary solutions on the market today, such as Okta, Auth0, or OneLogin.