Managing applications across multiple Kubernetes clusters is complex, especially when those clusters span different environments or even cloud providers. One powerful and secure solution combines Google Kubernetes Engine (GKE) fleets and, Argo CD, a declarative, GitOps continuous delivery tool for Kubernetes. The solution is further enhanced with Connect Gateway and Workload Identity.

This blog post guides you in setting up a robust, team-centric multi-cluster infrastructure with these offerings. We use a sample GKE fleet with application clusters for your workloads and a control cluster to host Argo CD. To streamline authentication and enhance security, we leverage Connect Gateway and Workload Identity, enabling Argo CD to securely manage clusters without the need to manage cumbersome Kubernetes Services Accounts.

On top of this, we incorporate GKE Enterprise Teams to manage access and resources, helping to ensure that each team has the right permissions and namespaces within this secure framework.

Finally, we introduce the fleet-argocd-plugin, a custom Argo CD generator designed to simplify cluster management within this sophisticated setup. This plugin automatically imports your GKE Fleet cluster list into Argo CD and maintains synchronized cluster information, making it easier for platform admins to manage resources and for application teams to focus on deployments.

Follow along as we:

• Create a GKE fleet with application and control clusters

• Deploy Argo CD on the control cluster, configured to use Connect Gateway and Workload Identity

• Configure GKE Enterprise Teams for granular access control

• Install and leverage the fleet-argocd-plugin to manage your secure, multi-cluster fleet with team awareness

By the end, you'll have a powerful and automated multi-cluster system using GKE Fleets, Argo CD, Connect Gateway, Workload Identity, and Teams, ready to support your organization's diverse needs and security requirements. Let's dive in!

Set up multi-cluster infrastructure with GKE fleet and Argo CD

Setting up a sample GKE fleet is a straightforward process:

1. Enable the required APIs in the desired Google Cloud Project. We use this project as the fleet host project.

a. gcloud SDK must be installed, and you must be authenticated via gcloud auth login.

2. Create application clusters and register them under your fleet host project.

3. Set up teams on your fleet. Let’s say you have one frontend team with a webserver namespace. 

a. With fleet teams and fleet Namespace, you can control which team accesses specific namespaces on specific clusters.

4. Now, set up Argo CD and deploy it to the control cluster. Create a new GKE cluster as your application and enable Workload Identity on it.

5. Install the Argo CD CLI to interact with the Argo CD API server. Version 2.8.0 or higher is required. Detailed installation instructions can be found via the CLI installation documentation. 

6. Deploy Argo CD on the control cluster.

Customize the Argo CD generator

Now you’ve got your GKE fleet up and running, and you’ve installed Argo CD on the control cluster. In Argo CD, application clusters are registered with the control cluster by storing their credentials (like API server address and authentication details) as Kubernetes Secrets within the Argo CD namespace. We've got a way to make this whole process a lot easier!

The fleet-argocd-plugin is a customized Argo CD plugin generator that takes the hassle out of cluster management by: 

• Automatically importing your GKE fleet cluster list into Argo CD and setting up the cluster secret objects for each application cluster 

• Keeping an eye on your fleet's status on Google Cloud, making sure your Argo CD cluster list is always in sync and up-to-date

Now, let’s see how to build and configure the Argo CD generator. 

7. Install fleet-argocd-plugin on your control cluster. 

a. In this demo, we use Cloud Build to build and deploy the fleet-argocd-plugin.

8. To make sure the fleet-argocd-plugin works as it should, give it the right permissions for fleet management. 

a. Create an IAM service account in your Argo CD control cluster and grant it the appropriate permissions. The setup follows the official onboarding guide of GKE Workload Identity Federation.

b. You also need to allow the Google Compute Engine service account to access images from your artifacts repository.

9. Run the fleet plugin on your Argo CD control cluster!

Demo time

Let's do a quick check to make sure the GKE fleet and Argo CD are playing nicely together. You should see that the secrets for your application clusters have been automatically generated.

Demo 1: Automatic fleet management in Argo CD

Okay, let's see how this works! We'll use the guestbook example app. First, we deploy it to the clusters that the frontend team uses. You should then see the guestbook app running on your application clusters, and you won't have to deal with any cluster secrets manually!

Demo 2: Evolving your fleet is easy with fleet-argocd-plugin

Suppose you decide to add another cluster to the frontend team. Create a new GKE cluster and assign it to the frontend team. Then, check to see if your guestbook app has been deployed on the new cluster.

Closing thoughts

In this blog post, we’ve shown you how to combine the power of GKE fleets, Argo CD, Connect Gateway, Workload Identity, and GKE Enterprise Teams to create a robust and automated multi-cluster platform. By leveraging these tools, you can streamline your Kubernetes operations, enhance security, and empower your teams to efficiently manage and deploy applications across your fleet.

However, this is just the beginning! There's much more to explore in the world of multi-cluster Kubernetes. Here are some next steps to further enhance your setup:

• You can find the source code for the fleet-argocd-plugin on GitHub.

• Deep dive into GKE Enterprise Teams: Explore the advanced features of GKE Enterprise Teams to fine-tune access control, resource allocation, and namespace management for your teams. Learn more in the official documentation.

• Secure your clusters with Connect Gateway: Delve deeper into Connect Gateway and Workload Identity to understand how they simplify and secure authentication to your clusters, eliminating the need for VPNs or complex network configurations. Check out this blog post for a detailed guide.

• Master advanced deployment strategies: Explore advanced deployment strategies with Argo CD, such as blue/green deployments, canary releases, and automated rollouts, to achieve zero-downtime deployments and minimize risk during updates. This blog post provides a great starting point.

As you continue your journey with multi-cluster Kubernetes, remember that GKE fleets and Argo CD provide a solid foundation for building a scalable, secure, and efficient platform. Embrace the power of automation, GitOps principles, and team-based management to unlock the full potential of your Kubernetes infrastructure.