Just Trust Me: Solving the first secret problem in pipelines with modern authentication protocols

Secret Sprawl and the Path to Production

When an idea escapes the confines of you IDE and needs to be shared with the world, it often needs to communicate with third party services to be deployed. For example, you might want to leverage a cloud service provider like AWS, Azure, or GCP. You might leverage infrastructure as code to programmatically describe the desired configuration for running your application. As more managed services enter the mix, the need to authenticate between these systems continues to grow.

All this complexity leads to a sprawl of secrets. These might include:

• API keys


• Access and secret keys


• Username and passwords


• other credential materials

To help combat this sprawl, various secret management platforms like HashiCorp Vault, AWS secrets manager, or Azure Key Vault were created. These systems help to centrally manage and audit the use of secrets. While this avoids secrets being injected to various locations in different providers, a first secret is needed to authenticate to these secret management platforms.

Historically this has meant setting a static secret in the local secret storage of the platform in use. For example, GitHub provides a mechanism to store secrets that can be leveraged while running actions. Kubernetes similarly offers its secrets resource as a means to provide secrets at runtime to a container.

A Better Way

In recent years, a path has emerged to allow for dynamic secrets, reducing the risk of a credential being compromised and long lived. While this does not completely eliminate the risk of the credentials being misused, it is a step in the right direction.

For deployment pipelines, many major vendors such as Gitlab and GitHub support Open ID Connect (OIDC), as a means of authenticating to external systems. This allows the CI/CD system to act as an identity provider that the external system can trust.

This approach uses JSON Web Tokens (JWTs) as a means to authenticate. The target system verifies the token and further validates attributes of the token as needed.

A visual of the authentication flow is provided by Gitlab's documentation

You may want to limit a production deployment to your main branch, to ensure code is reviewed before being able to leverage that level of access. [https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services#configuring-the-role-and-trust-policy) are provided in the documentation for the configuration of various providers. Once authentication is complete, the identity is governed by the authorization logic of the system (ex: AWS IAM policies, or HashiCorp Vault policies).

To allow your pipelines to leverage these dynamic credentials, you will need to update your pipeline definitions. For example, GitHub requires the following permission to be granted to allow for the token generation to occur

permissions:

  id-token: write # This is required for requesting the JWT

Further documentation on this configuration can be found here

The OIDC approach is already built into a number of third-party providers. Due to it leveraging a standard authentication protocol, it could also be used in custom built API authentication/authorization logic. If your pipeline needs to emit an event to (or fetch data from) a custom API, you could validate the token using common language libraries or authorization systems like Open Policy Agent (OPA). This helps avoid the need for another API key.

The pipeline identity also has uses outside of authenticating to external systems. Github built an additional feature around artifact attestations which leverages the identity to sign artifacts for later verification. This would allow a user to verify that a file/program was created by a specific repository's pipeline.

Conclusion

The first secret problem has been an ongoing challenge in the technology space, but innovative approaches as outlined above help move us forward toward more secure solutions. Leveraging standard protocols makes it even easier to be more widely adopted. I am excited to see continued progress in this space.