This post is an extension of our Container Image Signing blog series. In our first post, we discussed the motivations and fundamental concepts behind cryptographic signing for containers.

Introduction

Organizations today are adding additional security measures to their software development lifecycles (SDLC) due to compliance, governance, or executive requirements. For containerized applications, one such security measure is using cryptographic signatures or attestations to validate the authenticity of the images. Today, organizations building container images for their applications can use AWS Signer to cryptographically sign their images. However, other types of metadata files such as Common Vulnerabilities and Exposure (CVE) scan results, a Software Bill of Materials (SBOM), or a Helm chart may also be included with the image that require the authenticity and provenance of a signature.

This post explains how these additional metadata files, or artifacts, can be signed and verified with AWS Signer alongside the image they correspond to. First, we’ll define what these artifacts are and how to work with them. Then, we’ll examine how an organization may sign and verify artifacts in a typical SDLC, and demonstrate how to get started with this process with a reference architecture.

OCI artifacts

The Open Container Initiative (OCI) is a project founded in 2015 to standardize how container images are used. Specifically, the OCI Image Format specification defines how images are built and organized, including the image manifest and layered OCI artifacts.

According to Docker, “OCI artifacts are any arbitrary files related to a software application.” To distribute OCI artifacts to and between OCI-compliant repositories such as Amazon Elastic Container Registry (Amazon ECR), we’ll be using ORAS, an open-source tool that provides a CLI to accomplish such tasks. This post will not be covering the fundamentals of OCI and its specification. Further reading of the history and terms associated with the OCI can be found in the ORAS documentation. To sign and verify OCI artifacts, we can use AWS Signer, a fully managed code-signing service. AWS Signer offers an image signing feature that uses the open source Notation CLI to sign any OCI artifact present in any OCI registry.

Solution overview

Organizational use cases with signing OCI artifacts

Let’s say we’re an organization that is planning on adopting image signing within our development pipelines. We should consider what other OCI artifacts we plan on including alongside our image. Just as importantly, we should ensure that security is involved throughout the lifecycle of the image from initial build to deployment. For the latter requirement, using signatures in an OCI repository can be beneficial, as they address a few containers security best practices:

• A method of maintaining trust as vulnerabilities and configuration requirements change
• Validation to ensure images are tamper-free
• Ensuring images and artifacts that meet organizational compliance requirements are approved

Therefore, we should be selecting, signing, and generating all related OCI artifacts for an image that can extend trust and improve visibility throughout an SDLC.

Let’s consider a typical development process and explore how our organization can adopt such practices. You can also refer to this security blog for other details on adopting AWS Signer in a pipeline. In an initial development pipeline, developers may be customizing an image for use as outlined in the following figure:

Figure 1. A development software pipeline used to build and sign container images.

1. In an initial stage, developers may pull public images for initial testing and customization to their local machine.
2. Once ready for building, they can upload their build specification files to a shared code repository, which in this case is AWS CodeCommit.
3. A change in the shared repository triggers a pipeline using AWS CodeBuild, where the build specification files are used to build and sign an image with AWS Signer.
4. The image is uploaded to an OCI-compliant image repository, which in this case is Amazon Elastic Container Registry (Amazon ECR). Note, AWS Signer, along with the underlying Notation CLI client can be used with any OCI 1.0 compatible registry. Refer to the Notary FAQ here.

In this example, we sign the container image to provide initial approval (step 3), but what other types of files could provide further insight into the security posture of this image?

• Security: The pipeline should be able to verify the software packages the image uses and any associated vulnerabilities. The image itself should also be created in a pipeline with attested execution reports. Artifacts like SBOMs and CVE scans provide visibility of the security posture to understand the components of the image and make informed decisions on determining risk. Signing such artifacts offers provenance, which ensures integrity of artifacts since its creation. Consider referring to our first post and Amazon Elastic Kubernetes Service (Amazon EKS) Best Practices Guide for more information about using SBOMs and other image security recommendations.
• Compliance: An organization may have legal obligations to adhere to. In this case, the organization can include license files or compliance certifications.
• Deployment: DevOps and development teams may require additional files that define how an image is used and deployed. For some projects, a readme file can suffice. For Kubernetes projects, a Helm chart can be included as well.

To establish the same proof of trust associated with the image, we should also sign the image’s OCI artifacts. As additional artifacts are generated and added to the repository, they form links of related artifacts described by the associated manifest. For additional reading, refer to our blog on OCI Artifact Support in Amazon ECR.

Figure 2. Various OCI artifacts linked with a specific property within their respective manifests.

After the generated OCI image and artifacts are pushed to a central OCI repository, we can still take further actions according to our organizational security needs. For instance, other teams within the organization can include additional OCI artifacts and corresponding signature to serve as a proof of approval through their team’s workflow. Perhaps a security team has accepted the results of a penetration test, or a compliance team requires certain license files to be packaged. This process is outlined in the following architecture:

Figure 3. Successive deployment pipelines used to do further signing and testing of images.

1. A downstream continuous integration pipeline ingests the OCI image and artifacts so that further testing and validation can occur.
2. The deployment environment pulls the image and artifacts from the image repository and verifies the signatures.
3. Upon signature verification, the image is deployed into a container orchestrator, such as Amazon Elastic Container Service (Amazon ECS), for testing, such as runtime testing, integration testing, etc. You can refer to this blog on integrating Signer with ECS.
4. Other artifacts and signatures are added as needed. In architecture shown, the organization uses AWS Lambda to attach and sign other required artifacts with a team’s signature.

Prerequisites

• An AWS account with Administrator permissions
• AWS Command Line Interface configured to build and deploy CDK
• NodeJS 14.x or later installed
• CDK installed

Walkthrough

Getting started with OCI artifact signing on AWS

In an environment where an application is just beginning development, there may be fewer approval workflows and OCI artifacts required in the pipeline. Still, we can incorporate container signing best practices with a foundational reference architecture and scale it to meet organizational needs. In the example following architecture, a small application is being developed and pushed through to a final deployment. There is an initial development pipeline where an image and its artifacts are built and signed with the development team’s signature. Once deployment is approved, they can start a production pipeline where the signatures are automatically verified and deployed.

 Figure 4. A sample image pipeline in a SDLC with a dev and prod pipeline.

To try this out yourself, you can deploy the AWS Cloud Development Kit (AWS CDK) application linked on GitHub here, which consists of a sample webserver that is built, signed, verified, and deployed to compute infrastructure via two AWS CodePipeline continuous integration and continuous delivery/development (CI/CD) pipelines. An AWS Signer signing profile and Amazon ECS on AWS Fargate cluster is also deployed alongside permissions for the pipelines to execute relevant AWS application programming interface (API) calls.

The development pipeline consists of two steps where an initial AWS CodeCommit repository stores a Dockerfile pushed by developers. In the build phase, the image and associated OCI artifacts are generated, signed, and pushed to an external repository.

Figure 5. The build phase, where an image alongside its OCI artifacts are built.

The buildspec in AWS CodeBuild installs the relevant libraries and builds the image. To generate an SBOM and CVE scan results, we are using Anchore’s open-source Syft and Grype tools as shown in the following commands. Then, we use ORAS to push the json files to a separate Amazon ECR repository. As a side note, Amazon Inspector also offers native capabilities to generate an SBOM and package vulnerability findings as well.

ORAS does the work of attaching the SBOM and CVE artifacts to the original image and including a manifest so that we can sign it. Finally, we use the AWS Signer plugin alongside the Notation CLI to sign the image and artifacts in the Amazon ECR repository:

Note: This post uses images that are signed and verified using image tags. As shown in the following, Notation does warn that using tags may pose a risk.

Once the build phase has finished, we can also use ORAS to visualize the artifacts in the Amazon ECR repository:

Figure 6. A tree representation of an OCI image and its artifacts.

So far, we’ve built, signed and pushed our image and its related OCI artifacts (e.g. SBOM and CVE scan results) to Amazon ECR. Now, we’re ready for a hypothetical deployment where we verify the signatures included in the repository and push the image to a production environment. In a separate production pipeline, our hypothetical organization has implemented a manual approval step so a human can approve a change before continuing deployment to a production environment. Once approved, a build stage is initialized where the image and associated artifact signatures are verified using the notation verify command. Then, the image is deployed into an Amazon ECS on AWS Fargate cluster.

Figure 7. The production pipeline awaiting manual approval for a deployment.

After deploying the CDK application, a URL for the load balanced Amazon ECS application will be shown in the CDK output. After the pipelines have finished executing, we can connect to the URL on a browser and see that the web app has been launched.

Figure 8. The running web application on Amazon ECS.

To see how changes flow through the pipelines, you can commit a new change to the Dockerfile in the AWS CodeCommit repository created by the CDK application. As the image and artifacts are being built in the pipeline, you can view live logs of the AWS CodeBuild step as it is running to see the outputs of each command. One important step in the build stage is where AWS CodeBuild imports the Notation trust policy via the notation policy import command, which includes attributes such as the AWS Signer signing profiles used to verify signatures.

The logs for the build step in the production pipeline can also be viewed as well by selecting View Logs in the VerifyImage CodeBuild step. Note that the manual step must be approved first so your changes can be deployed. You can also re-release changes in the pipeline to redeploy an image in the source Amazon ECR repository. However, be mindful of the signature validity period, because if the signature expires or is revoked, the verification will fail and the image will not be deployed. For this CDK application, the signature validity period is set to 7 days as seen in the AWS Signer profile and CDK application stack.

Cleaning up

To avoid incurring future charges, follow the following steps:

1. Navigate to the ECR repository which should be named signer-workflow-ecr and delete all the artifacts.
2. Delete the CDK application by running the following command: cdk destroy

Conclusion

In this post, we showed you the value of including other OCI artifacts and signing them in an SDLC. We discussed the organizational use cases for generating various forms of OCI artifacts. We then explored how an organization can include image signing within its own SLDCs. Then, to see the concepts we discussed in action, we deployed a reference architecture demonstrating OCI artifact signing and verification using AWS Signer image signing capabilities. For more information about AWS Signer, you can refer to the documentation.