Introduction

In today’s interconnected world, communication faces evolving security threats. From sensitive financial transactions in online banking to secure data transmissions in the automobile industry, ensuring trust and authenticity between businesses is becoming more and more critical. This is where Mutual Transport Layer Security (mTLS) can be an option to offer enhanced security through advanced cryptographic handshake protocols.

While traditional TLS encrypts data in transit, mTLS verifies the identities of both the client and server before establishing a connection, utilizing digital certificates issued by a trusted authority. This two-way authentication creates a mutual trust relationship, guaranteeing that only authorized parties can exchange information.

For businesses dealing with complex applications, managing mTLS infrastructure can be daunting. Manually building and maintaining custom solutions often involve substantial developer resources, updates, and infrastructure investments. Moreover, self-created or third-party solutions can increase time and management overhead, diverting focus from core business objectives.

Figure 1: TLS connection workflow

Figure 2: mTLS connection workflow

To help our customers improve their security posture without additional management overhead, in 2023 we launched native support for mTLS within Application Load Balancer (ALB), which eliminated the need for such complex custom solutions by allowing you to offload client authentication to the load balancer itself. As a result, developers gained valuable time and resources to focus on application developments, while enjoying a fully managed, scalable, and cost-effective solution for securing their environment.

In this post, we will show you how you can seamlessly use ALB’s native mTLS capabilities with Amazon Elastic Kubernetes Service (Amazon EKS) workloads, making use of a recently released features of the AWS Load Balancer Controller (version 2.7) that added support for (mTLS) Mutual Transport Layer Security on Kubernetes Ingress.

Solution overview

This post is dedicated to ensuring secure communications between Kubernetes workloads with mTLS in Amazon EKS. It covers the following components:

• AWS Load Balancer Controller: A Kubernetes controller to help manage Elastic Load Balancers for the Kubernetes cluster
• ExternalDNS: A Kubernetes component that allows for automatically creating and managing DNS records for services exposed externally with supported DNS providers such as Amazon Route 53. It facilitates external communication with services running inside the Kubernetes cluster by resolving the service’s hostname to the external IP address of the Kubernetes cluster.
• Sample Application Deployment: Deploy a sample workload for an mTLS-enabled service. This workload encompasses the deployment of a sample application configured for mutual TLS (mTLS) within a Kubernetes environment, with a specific focus on Amazon EKS.

Figure 3: The red lines indicates the implementation of mTLS with both client and server presenting their certificate to each other and using a private certificate authority (CA) to authenticate clients before granting access.

Prerequisites

• You have an Amazon EKS cluster, otherwise create a new cluster.
• You have properly installed and configured latest versions of AWS Command Line Interface (AWS Command Line Interface [AWS CLI]), kubectl, and Helm CLI
• A custom domain name to test your sample application
• A certificate managed by AWS Certificate Manager

Walkthrough

Step 1: Make a self-signed certificate

To test the mutual authentication on Application Load Balancer, follow the step-by-step instructions below to make a self-signed CA bundle and client certificate with x509v3 extensions using OpenSSL:

1. Create the private certificate authority (CA) private and public keys:

2. Prepare X.509 Extensions configuration file. Save the content below into a file named custom_openssl.cnf

3. Generate the Client Key, and Certificate and Sign with the CA Certificate:

For a production environment, you can implement mutual authentication on ALB using a CA bundle with roots and/or intermediate certificates generated by AWS Private Certificate Authority as the source of trust to validate your client certificates. Follow the step-by-step instructions to make a CA bundle and client certificate using AWS Private Certificate Authority.

Step 2: Create a trust store

To create a trust store, we will use the CA certificate as the certificate bundle. This will require creating an Amazon Simple Storage Service (Amazon S3) bucket, then uploading your CA bundle to the Amazon S3 bucket. You complete this task with the following commands:

Expected result:

Figure 4: Trust Store created for ALB on AWS Console

Step 2: Install ALB Controller

The controller in your cluster needs access to the AWS ALB/NLB APIs with AWS Identity and Access Management (AWS IAM) permissions. Refer to the steps in the documentation to configure the recommended AWS IAM roles for service accounts (IRSA) for the controller. To install Application Load Balancer Controller, you can use Helm. Retrieve the VPC automatically created for the cluster and substitute values below with your own:

For mTLS support in ALB provisioned by Kubernetes Ingress resource, you will need to install the version 2.7 or later version of the AWS Load Balancer Controller. Install the AWS Load Balancer Controller with the Helm command below:

Step 3: Install and Setup External DNS (Optional)

You may create a DNS record in Amazon Route53 for your application. Refer to the documentation to setup and manage records in Route 53 that point to controller deployed ALBs. Alternatively, you can set up ExternalDNS in your Kubernetes cluster to create DNS record in Amazon Route53 on your behalf, see Setting up ExternalDNS for services on AWS (on the GitHub website) and Set up ExternalDNS. Be sure to use 0.11.0 version or higher of ExternalDNS.

1. Download sample external-dns

2. Open the downloaded external-dns.yaml file and edit the –domain-filter flag to include your hosted zone(s). The following example is for a hosted zone com:

3. Deploy the downloaded external-dns.yaml file:

4. Verify it deployed successfully.

Step 4: Deploy a sample application

1. Prepare your environment by declaring the variables below:

2. Copy and paste the content below into a file named mtls.yaml to create a sample workload:

In this example, we specified four HTTPS listeners with ports 80, 443, 8080, and 8443 for the ingress. A new attribute to enable connection logs for your application loadbalancer is also included in the annotation. When you enable connection logs, you must specify an Amazon S3 bucket for the connection logs. For more information, refer to the Amazon S3 bucket requirements here.

Deploy the manifest:

After few minutes, run the following command to verify the ingress created:

Expected output:

5. Run a curl command to test connectivity to the application:

You should see a 200 HTTP response with TLS verification happening once. See CERT verify in the example output:

Step 5: Associate a trust store to the Ingress Resource

mTLS for ALBs provides two options for validating X.509v3 client certificates: passthrough and verify mode. For passthrough mode, you can configure the listener to accept any certificate(s) from the client. For verify mode, you will need to create a new Trust Store, upload your CA bundle, and attach the Trust Store to your listener that is configured to verify client certificates.

Edit the ingress annotation section in the previous mtls.yaml file and add the annotation below to associate the trust store created in previous step:

In this example, the listener HTTPS:80 will be set to passthrough mode, the listener HTTPS:443 will be set to verify mode and will be associated with the provided trust-store-arn arn:aws:elasticloadbalancing:trustStoreArn. The remaining listeners HTTPS:8080 and HTTPS:8443 will be set to default mTLS mode (i.e., off).

Run the following command to apply the changes from the manifest:

Verify the ALB created from Amazon Elastic Compute Cloud (Amazon EC2) Console:

Figure 5: ALB details page on Amazon EC2 Console

Run the curl command again with the client certificate and key.

You should see a 200 HTTP response with TLS verification happening twice. See CERT verify in the example output:

Cleaning up

To avoid incurring future charges, you should delete the resources created during this tutorial. You can delete the resources with the following command:

Conclusion

In this post, we showed you how to enable mutual TLS at ALB created by a Kubernetes Ingress resource for an application running in Amazon EKS using a self-signed generated certificates. Companies have improved their security posture by using Mutual TLS implementation to add an extra layer of security to their Kubernetes workload traffic and to prevent various kinds of attacks.