Introduction

Amazon Elastic Kubernetes Services (Amazon EKS) provides excellent abstraction from managing the Kubernetes control plane and data plane nodes that are responsible for operating and managing a cluster. AWS offers managed Amazon Machine Images, or AMIs, for Amazon Linux 2, Bottlerocket, and Windows Server. Many customers have requirements, or simply prefer, to use Red Hat Enterprise Linux (RHEL) for all their Linux machines. Although RHEL isn’t officially supported for Amazon EKS, customers can build Amazon EKS worker nodes on RHEL using the scripts in this post.

An Amazon EKS cluster has built-in add-ons for networking: The Amazon Virtual Private Cloud (Amazon VPC) container network interface (CNI) plugin for Kubernetes, CoreDNS and kube-proxy components. Each of these components plays a vital role for Kubernetes networking. Of these components, kube-proxy is responsible for maintaining the network rules that allow network communication to your Pods from network sessions inside or outside of your cluster. IPVS and iptables are two proxy modes for kube-proxy on Linux, with the default being iptables. While it is most common and acceptable to use the default mode, there are known limitations on iptables that can ultimately impacts a cluster’s performance.

With RHEL 8.6, Red Hat transitioned from iptables to nftables for network filtering. This presents a challenge when running Kubernetes components such as the kube-proxy daemonset, which at the time of writing doesn’t support nftables on RHEL. One work around for this challenge is to use IP Virtual Server (IPVS).

IPVS can enhance the performance of a Kubernetes cluster as it was created for load balancing. It uses hash tables for efficient data structuring, making it more suitable for use in larger clusters. To enable IPVS, you must configure this at the Kubernetes cluster level as well as on the individual worker nodes. In this post, we’ll show you how to configure your Amazon EKS cluster for IPVS networking. We’ll also demonstrate how to build RHEL 8.x or RHEL 9.x worker nodes and join them to your cluster.

Note: There is a network routing issue caused by the nm-cloud-setup service that comes preinstalled on RHEL machines. In this guide, we will disable this service and reboot the EC2 instances as recommended by Red Hat in the following KB article.

Prerequisites

This walkthrough requires the following prerequisites (versions of each are listed as of the time of writing).

• AWS Command Line Interface (AWS CLI) (walkthrough version 2.13.21)
• eksctl (walkthrough version 0.160.0)
• kubectl (walkthrough version 1.28.1)
• git (walkthrough version 2.40.1)
• HashiCorp Packer (walkthrough version 1.9.4)

Walkthrough

First, we’ll create an Amazon EKS cluster without any workers nodes and configure IPVS networking. Next, we’ll build out the Amazon EKS worker nodes on RHEL. And finally, we’ll join the Amazon EKS worker nodes to the cluster.

Amazon EKS cluster creation and configuration

Create an Amazon EKS cluster without any nodes.

Edit the kube-proxy-config ConfigMap to use IPVS. This can be done using the Amazon EKS VPC CNI add-on using instructions from this post.

Set the ipvs → scheduler value to your preferred method. Example: rr

• rr: round-robin
• lc: least connection
• dh: destination hashing
• sh: source hashing
• sed: shortest expected delay
• nq: never queue

Set the mode to “ipvs”. Save and exit.

With IPVS enabled at the cluster level, your worker nodes need to have the proper kernel modules installed and loaded. This is taken care of by the installation script referenced in the next portion of the walkthrough, but you can enable these modules manually if you don’t want to run the installation script.

Building the RHEL Worker Node AMI

We’ll use Packer to build the worker nodes. Packer can be setup on your local workstation with Application Programming Interface (API) keys for AWS credentials. Packer can also be setup on an Amazon Elastic Compute Cloud (Amazon EC2) instance with an AWS Identity and Access Management (AWS IAM) Instance Profile with the necessary permissions, or it can be setup in AWS CloudShell. We recommend using AWS CloudShell because it is free to use and is preconfigured with the same credentials of the user logged into the AWS Management Console.

To setup Packer in AWS CloudShell, you can run the following commands.

Next, clone the GitHub repository for building Amazon EKS RHEL AMIs and cd into the directory.

Launch the Packer build process using a Make command similar to one of the ones below, specifying your Kubernetes cluster version as well as any other custom values you desire.

Example basic command:

Example command for building a customized Defense Information Systems Agency (DISA) Security Technical Implementation Guide (STIG) compliant AMI, owned by a specific AWS Account in AWS GovCloud us-gov-east-1 Region, with binaries stored in a private Amazon Simple Storage Service (Amazon S3) bucket, an AWS IAM instance profile attached, and using AWS Systems Manager Session Manager for Packer terminal access:

The build process can take anywhere from 5 to 20 minutes and should provide an output similar to the one below at the end. Copy down the resulting AMI ID for use in the next steps.

Now that you have successfully built a custom AMI, it’s time to join worker nodes to your cluster. In the root of the directory of the GitHub repository cloned at the beginning of the walkthrough, there are bash (Linux) and zsh (MacOS) scripts that you can execute to join nodes to your cluster. These scripts use eksctl to create Amazon EKS-managed node groups. Modify these scripts with any custom tags or labels as desired.

Note: These scripts contain custom Amazon EC2 User Data scripts that disable the nm-cloud-setup service to resolve the issue with this service mentioned earlier in the post.

Script syntax:

Example command:

At this point, we want to perform some validation of our Amazon EKS cluster. First, we’ll verify our worker nodes have joined the cluster.

Next, we’ll verify kube-system namespace pods are running.

Note: We want to confirm that coredns is working as expected as this is one of the main issues seen when moving from iptables to nftables or IPVS.

Lastly, we’ll verify IPVS entries are present on one of our worker nodes.

Note that the first entry is the Transmission Control Protocol (TCP) entry for the Kubernetes ClusterIP service (from kubectl get service kubernetes).

The second and third entries are the TCP and User Datagram Protocol (UDP) entries for the kube-dns service (from kubectl get service kube-dns -n kube-system). The endpoints are the coredns pod IPs.

Cleaning up

To avoid incurring future charges, delete the two AWS CloudFormation Stacks created in the steps above. You’ll need to delete the Managed Node Group stack first, and then you can delete the Amazon EKS cluster stack.

Conclusion

In this post, we showed you how to create Amazon EKS worker nodes that run on RHEL 8 and RHEL 9 Amazon EC2 instances and how to run your Amazon EKS clusters in IPVS networking mode.