When working with Azure DevOps, there's a lot of options and configurations to tailor the service exactly to the needs of your organization. Part of the responsibilities that lie on the ones that managed these pipelines is to ensure that you don't spill the beans - or in other words, leak any sensitive data.
With Azure DevOps, you can get sensitive data like Connection Strings, Secrets, API Keys and whatever else you may classify as sensitive, directly from an Azure Key Vault.
In this post we'll explore how to configure Azure DevOps to fetch Secrets directly from Azure Key Vault during a build, instead of configuring those secrets on the actual build pipeline, which to be honest, perhaps shouldn't be there in the first place.
Please excuse my lack of artistic creativeness in this illustration, but the idea is to visualize the flow of what we want to achieve:
- Azure DevOps build Pipeline
- Authorized as an Azure AD application
- Has permissions (and access policies) to Get and List secrets from an Azure Key Vault
Voila. No credentials in your Azure DevOps build pipelines ever again.
Pre-requisites
- An Azure DevOps account and pipelines.
- An Azure Key Vault with secrets you want to use in your pipelines.
Azure DevOps Variable Groups and Key Vaults
I love the concept of a Variable group in Azure DevOps. It helps me to create re-usable configurations of variables that I can use from multiple different pipelines.
But, what about the Azure Key Vault task?
Great question - read more about that one here.
Use this task in a build or release pipeline to download secrets such as authentication keys, storage account keys, data encryption keys, .PFX files, and passwords from an Azure Key Vault instance. The task can be used to fetch the latest values of all or a subset of secrets from the vault, and set them as variables that can be used in subsequent tasks of a pipeline. The task is Node-based, and works with agents on Linux, macOS, and Windows
My main takeaway when it comes to choosing between a Variable Group and an Azure Key Vault Task is:
- Use Variable Groups for reusable configuration, to be used across multiple pipelines.
- Use the Azure Key Vault Task for single-purpose access to a vault from a specific pipeline.
In this post, we'll look at the Variable Groups, because I have multiple pipelines in various setups that all need access to secrets from one or more of my vaults, and I need the re-usability across all my pipelines that target the same system(s).
With that out of the way, let's kick things off.
Ensure you have Secrets in your Azure Key Vault
My Azure Key Vault already contains my required secrets. You should ensure that this is done before continuing, to make the following steps easier, because when you later try to add variables to the Variable Group, it'll look for Secrets from the connected vault.
Here's an example of what Secrets I have, so that we can relate that to the variables later:
Great - we have our Key Vault, it contains Secrets, and we're ready to start connecting our Azure DevOps service to make use of these secrets so that we can avoid having plain text and avoid having sensitive configurations flying around.
Configure a Variable Group to connect to an Azure Key Vault
It's time to actually create our Variable Group, which can be done easily from the UI in Azure DevOps. Here's how we start.
Go to "Pipelines" and then "Library" and "Add variable group":
Next, populate the data as you see fit and select your Subscription and Vault from the options available (e.g. from the tenants you're connected to):
It will ask you to Authorize the connection so that Azure DevOps has permissions to Get and List secrets from the given vault. Once you've clicked "Authorize" you should see an empty section of Variables.
Next up, click "Add" under Variables and see that the list populates with the same variables as you have Secrets in your Key Vault. Remember from the previous section, where we had two Secrets in our vault? That's the ones we're looking at as Variables now in Azure DevOps.
Tick the boxes of each Secret you want to bring into your pipe, and click OK.
Hit Save on the Variable Group.
Configure a Pipeline to make use of the new Variable Group
When we've reached this point, we have a variable group configured to use our Azure Key Vault Secrets as per the previous section in this post. Now we're ready to see if we can start utilizing them from the Azure DevOps pipelines. Here's how.
Go to the pipeline you want to make changes to. Select "Variables" and then "Link variable group":
All variable groups show up, and you can select the one you have just created. The parentheses show the number of linked Secrets, which correctly corresponds to the two ones I created previously:
Once done, you can expand the variable group to ensure the Secrets are listed with the values hidden:
From this point, whatever variables you have in your variable group can be accessed from your build pipe as $(VariableNameHere), and to exemplify this with my own examples, it could be used as $(NewsServiceApiKey).
With multiple stages in your pipeline, you can also define various scopes for the variables under "Pipeline variables", but for the purpose of demonstrating this functionality, I'm going with this flat approach.
Configure a build task to use the variables, coming straight from Secrets in Azure Key Vault
At this point we're ready to test the variables out from our pipelines. In the following example I'm just adding a simple PowerShell task that outputs them in the log - but DevOps is smart enough to know that this is a bad idea, and hence it'll just redact the actual values. It proves that the link works however, and we can now make use of them in any task down the pipe - we have success.
In a sample PowerShell task, I'm just adding this dummy inline script to prove that the linked variables work:
Write-Host "Hi, I'm Tobias."
Write-Host "I'm accessing sensitive data in my pipeline"
Write-Host "Here's an example: "
Write-Host "=========="
Write-Host "Send Grid API Key: $(SendGridApiKey)"
Write-Host "Cogni News Service: $(NewsServiceApiKey)"
Write-Host "=========="
Write-Host "Did I mention these comes from the Azure Key Vault?"
Write-Host "Thanks for tuning in. See you in the next blog post"
There's two key ingredients in the above snippet. It's $(SendGridApiKey) and $(NewsServiceApiKey).
What's great about DevOps though is that it's obfuscating the values so it's not showing up in the log. This is a good thing, because logging secrets isn't good practice and you shouldn't be doing that - I'm just making the demonstration as simple as I can so we can grasp how the variables are linked all the way from the Azure Key Vault, through our Variable Group, to our Pipeline Tasks 💪
In the job on DevOps, we can also see that there's a new step "Download secrets: ci-buildpipe" which takes care of linking the Secrets from the vault directly into my build pipe. It's amazingly smooth:
Starting: Download secrets: ci-buildpipe
==============================================================================
Task : Azure Key Vault
Description : Download Azure Key Vault secrets
Version : 1.155.3
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-key-vault
==============================================================================
SubscriptionId: REDACTED-GUID.
Key vault name: ci-buildpipe.
Downloading secret value for: NewsServiceApiKey.
Downloading secret value for: SendGridApiKey.
Finishing: Download secrets: ci-buildpipe
Done - that's a wrap for this post, and I hope you can walk away with more insights into how to connect your Azure Key Vault Secrets to your Azure DevOps pipelines, and avoid having any credentials in any other service, configuration, variables or code. A single place to manage them securely - Azure Key Vault.
Security considerations
Remember when you clicked "Authorize" to authorize DevOps to access our Key Vault to list and get secrets? It actually creates a Service Principal in your directory that you should be aware of, and also assigns an Access Policy to your Key Vault with "Get" and "List" permissions. Let's take a look.
Azure AD Application
When you connect your Azure DevOps service to your underlying subscription and it asks you to manage access and authorization, it creates an application in your directory. Here's mine, obviously slightly obfuscated 😂
Should you want to at some point clean things up, don't forget to take a look at the apps here.
So, we have the application automatically created for us - we can see that the key vault policies have been added too.
Azure Key Vault Access Policies added for the new app
The new application has Get and List permissions, but no changes or deletions can be made. It has automatically been granted this access policy when we clicked "Authorize" and said that it was OK. You might want to be aware of this, so you know where this is coming from if you do a security audit of your vaults.
Summary and Links
That's a wrap. A fairly straight forward way to connect your Azure Key Vault secrets to your Azure DevOps pipelines without spilling any secrets.
I hope this can help someone who's tasked with increasing the security awareness of their Azure DevOps Pipelines, as well as increase the general security posture across the board. Don't go light on these things - enforce security configuration and avoid sensitive data in your configurations.
- Azure DevOps: Variable Groups
- Azure DevOps: Azure Key Vault Task - Not what I used in this post, but interesting to learn more about.
Thanks for reading - please leave a comment. Tobi, out.
Comments