Securing cloud credentials has emerged as a challenge on the scale of Moby Dick: It presents an enormous problem, and simple solutions remain elusive. Credential security problems are also widespread: More than 69% of cloud compromises were caused by credential issues, including weak passwords, no passwords, and exposed APIs, according to Google Cloud’s Q3 2023 Threat Horizons Report. However, unlike Moby Dick, this story may have a happy ending. Organizations can get started on reducing the risk they face from credential-related compromises by protecting service accounts.

Service accounts are essential tools in cloud management. They make API calls on behalf of applications and they rely on IAM roles to make those calls.They also make an appealing target for attackers to establish initial access in a cloud environment.

“Nearly 65% of alerts across organizations were related to risky use of service accounts. These accounts have associated permissions where, if compromised, could lead to attackers gaining persistence and subsequently using this access for privilege escalation in cloud environments,” we wrote in the report.

This assessment follows a similar conclusion from our Q1 2023 Threat Horizons Report, where we detailed how attackers can abuse service account keys — and what organizations can do to stop them. That same report noted that 68% of service accounts had overly permissive roles, service account keys are often found hardcoded on public repositories, and project owners did not take corrective action after Google attempted to contact them in 42% of leaked key incidents.

Here are some mitigation tips that can help you avoid creating service account keys at scale, monitor for key usage, and respond to alerts quickly. We strongly encourage you to assess all of the techniques highlighted in the Q1 2023 report.

Prevent service account key creation

This section assumes that you have the necessary permissions to manage Organization Policies, guardrails that set broad yet unbendable limits for cloud engineers before they start creating and working with resources. To learn more, see creating and managing organization policies.

One of the best practices for managing service account keys is to use Organization Policy constraints to prevent creating new service account keys, and allow exceptions only for projects that have demonstrated that they cannot use a more secure alternative.

The easiest way to enforce this Organization Policy Constraint is in your Google Cloud console.

Before proceeding, ensure that you are logged in to the correct Google account. If you want to enforce this constraint across the entire Organization, be sure to select the Organization in the Resource Manager explorer:

Then click on “Manage Policy”:

And enforce the policy:

Subsequently, any time a user attempts to create a service account key, the user will be presented with the following error:

Find instances of Org Policy Changes and Service Account Key Creation

Now that service account key creation is disabled, the next step is to ensure that no one has altered this policy and covertly created new service account keys.

You can find evidence of policy changes using the Google Cloud Operations Suite. These next two examples assume that all logs are stored in a centralized Google Cloud Project and that you have the required permissions to query logs and set alerts. To learn more, see:

• Routing and Storage Overview
• Collate and route organization-level logs to supported destinations
• Build queries by using the Logging query language
• Configure log-based alerts

Visit the Logs Explorer. Use the Refine scope button in the Action toolbar and select the Scope that represents the location of your Organization’s centralized logs.

Use the following query to find instances of the “iam.disableServiceAccountKeyCreation” Organization Constraint being changed:

Log entry of an Org Policy Change Affecting “iam.disableServiceAccountKeyCreation”

To create an alert based on this query, proceed to the “Create Alerts for Rapid Response” section.

Find instances of service account key creation

Visit the Logs Explorer. Use the Refine scope button in the Action toolbar and select the Scope that represents the location of your Organization’s centralized logs.

Use the following query to find instances of a new service account key:

This log entry signifies that:

1. At some point, the Org Policy Constraint preventing service account key creation was disabled
2. Someone created a new service account key

Create alerts for rapid response

Google Cloud Operations Suite

The previous sections described how you can find instances of either new service account keys or Organization Policy Changes. To promote rapid and reactive response, create alerts so you may respond quickly to such events.

This section assumes you understand the following:

• How Alerting Works
• Create and Manage Notification Channels
• Manage Incidents for Log-Based Alerts

Click on “Create Alert”

Name the Alert and confirm that the logs to include in the alert match the query in the Log Explorer. Click “Preview Logs'' to confirm the entries are desirable. Set the notification frequency and autoclose duration.

Choose the notification channel. If you need to create a notification channel, follow these instructions.

Below is an example of an alert, sent via email, listing an incident triggered after I created a service account key.

Clicking “View Incident” directs me to the Alerting page:

Clicking on the incident directs me to a view listing:

• the condition
• the log query
• the set of logs matching the query
• the date and time the incident was triggered

Next steps: Build a collaborative incident management process

In examining the risks of using service account keys, enforcing Organization Policies that limit service account key creation at scale, and using continuous monitoring to detect policy violations, we have shown that you can better manage one of the biggest potential risks in your environment.

We highly encourage everyone to read “Build a Collaborative Incident Management Process,” which can help your organization operationalize responses to this and other types of incidents. Besides, who needs the drama of chasing a great white whale? Ditching service account keys is a much simpler way to boost your cloud security.