When working with Azure DevOps, there's a lot of options and configurations to tailor the service exactly to the needs of your organization. Part of the responsibilities that lie on the ones that managed these pipelines is to ensure that you don't spill the beans - or in other words, leak any sensitive data.
With Azure DevOps, you can get sensitive data like Connection Strings, Secrets, API Keys, and whatever else you may classify as sensitive. You can get them directly from an Azure Key Vault, instead of configuring them on your build pipeline.
With Azure DevOps, you can get sensitive data like Connection Strings, Secrets, API Keys, and whatever else you may classify as sensitive. You can get them directly from an Azure Key Vault, instead of configuring them on your build pipeline.
Please excuse my lack of artistic creativeness in this illustration. Still, the idea is to visualize the flow of what we want to achieve:
- Azure DevOps build pipeline
- Authorized as an Azure AD application.
- Has permissions (and access policies) to Get and List secrets from an Azure Key Vault
Voila. No credentials in your Azure DevOps build pipelines ever again.
Pre-requisites
- An Azure DevOps account and pipelines
- An Azure Key Vault with secrets you want to use in your pipelines
Azure DevOps Variable Groups and Key Vaults
I love the concept of a Variable group in Azure DevOps. It helps me to create re-usable configurations of variables that I can use from multiple different pipelines.
But, what about the Azure Key Vault task?
Great question - read more about that one here.
Use this task in a build or release pipeline to download secrets such as authentication keys, storage account keys, data encryption keys, .PFX files, and passwords from an Azure Key Vault instance. The task can be used to fetch the latest values of all or a subset of secrets from the vault, and set them as variables that can be used in subsequent tasks of a pipeline. The task is Node-based, and works with agents on Linux, macOS, and Windows
My main takeaway when it comes to choosing between a Variable Group and an Azure Key Vault Task is:
- Use Variable Groups for configuration re-used across multiple pipelines
- Use the Azure Key Vault Task for single-purpose access to a vault from a specific pipeline
In this post, we'll look at the Variable Groups, because I have multiple pipelines in various setups that all need access to secrets from one or more of my vaults. I need the re-usability across all my builds that target the same system(s).
With that out of the way, let's kick things off.
Ensure you have Secrets in your Azure Key Vault
My Azure Key Vault already contains my required secrets. Ensure that this gets done before continuing to make the following steps easier. When we try to add variables to the Variable Group, it'll look for Secrets from the connected vault.
Here's an example of what Secrets I have so that we can relate that to the variables later:
Great - we have our Key Vault. It contains Secrets, and we're ready to start connecting our Azure DevOps service. We can now make use of these secrets to avoid having plain text or sensitive configurations flying around.
Configure a Variable Group to connect to an Azure Key Vault
It's time to create our Variable Group, which we do from the UI in Azure DevOps in this case. Here's how we start.
Go to "Pipelines" and then "Library" and "Add variable group":
Next, populate the data as you see fit and select your Subscription and Vault from the options available (e.g., from the tenants that are connected):
It will ask you to Authorize the connection so that Azure DevOps has permission to Get and List secrets from the given vault. Once you've clicked "Authorize" you should see an empty section of Variables.
Next up, click "Add" under Variables and see that the list populates with the same variables as you have Secrets in your Key Vault. Remember from the previous section, where we had two Secrets in our vault? That's the ones we're looking at as Variables now in Azure DevOps.
Tick the boxes of each Secret you want to bring into your pipe, and click OK.
Hit Save on the Variable Group.
Configure a Pipeline to make use of the new Variable Group
When we've reached this point, we have a variable group configured to use our Azure Key Vault Secrets as per the previous section in this post. Now we're ready to see if we can start utilizing them from the Azure DevOps pipelines. Here's how.
Go to the pipeline where you want to make changes. Select "Variables" and then "Link variable group":
All variable groups show up, and you can select the one you have just created. The parentheses show the number of linked Secrets, which correctly corresponds to the two ones I created previously:
Once done, you can expand the variable group to ensure the Secrets exist with the values hidden:
From this point, whatever variables you have in your variable group can be accessed from your build pipe as $(VariableNameHere), and to exemplify this with my own examples, it could be used as $(NewsServiceApiKey).
With multiple stages in your pipeline, you can also define various scopes for the variables under "Pipeline variables," but to demonstrate this functionality, I'm going with this flat approach.
Configure a build task to use the variables, coming straight from Secrets in Azure Key Vault
At this point, we're ready to test the variables out in our pipelines. In the following example, I'm adding a simple PowerShell task that outputs them in the log - but DevOps is smart enough to know that this is a bad idea. Hence, it will redact the actual values. It proves that the link works, and we can now make use of them in any task down the pipe - we have success.
In a sample PowerShell task, I'm just adding this dummy inline script to prove that the linked variables work:
Write-Host "Hi, I'm Tobias."
Write-Host "I'm accessing sensitive data in my pipeline"
Write-Host "Here's an example: "
Write-Host "=========="
Write-Host "Send Grid API Key: $(SendGridApiKey)"
Write-Host "Cogni News Service: $(NewsServiceApiKey)"
Write-Host "=========="
Write-Host "Did I mention these comes from the Azure Key Vault?"
Write-Host "Thanks for tuning in. See you in the next blog post"
There are two key ingredients in the above snippet. It's $(SendGridApiKey) and $(NewsServiceApiKey).
What's great about DevOps though, is that it's obfuscating the values, so it's not showing up in the log. Obfuscation is a good thing because logging secrets isn't good practice. I'm making the demonstration as simple as I can so we can grasp how the variables are linked from the Azure Key Vault, through our Variable Group, to our Pipeline Tasks 💪
In the job on DevOps, we can also see that there's a new step, "Download secrets: ci-buildpipe" which takes care of linking the Secrets from the vault directly into my build pipe. It's amazingly smooth:
Starting: Download secrets: ci-buildpipe
==============================================================================
Task : Azure Key Vault
Description : Download Azure Key Vault secrets
Version : 1.155.3
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-key-vault
==============================================================================
SubscriptionId: REDACTED-GUID.
Key vault name: ci-buildpipe.
Downloading secret value for: NewsServiceApiKey.
Downloading secret value for: SendGridApiKey.
Finishing: Download secrets: ci-buildpipe
That's a wrap for the actual configuration and testing that it works. Now let us move on to a couple of other essential topics that we should know about before we dig deeper into this.
Security considerations
Remember when you clicked "Authorize" to authorize DevOps to access our Key Vault to list and get secrets? It creates a Service Principal in your directory that you should be aware of, and also assigns an Access Policy to your Key Vault with "Get" and "List" permissions. Let's take a look.
Azure AD Application
When you connect your Azure DevOps service to your underlying subscription, and it asks you to manage access and authorization, it creates an application in your directory. Here's mine, obviously slightly obfuscated. 😂
Should you want to, at some point, clean things up, don't forget to take a look at the apps here.
So, we have the application automatically created for us. In essence, we can see that the Key Vault policies exist, too.
Azure Key Vault Access Policies added for the new app
The new application has Get and List permissions to Secrets, but no changes or deletions are allowed. It is automatically granted with access policy when we clicked "Authorize" and said that it was OK. You might want to be aware of this, so you know where this is coming from if you do a security audit of your vaults.
Summary and Links
That's a wrap - a straight forward way to connect your Azure Key Vault secrets to your Azure DevOps pipelines without spilling any secrets.
I hope this can help someone tasked with increasing the security awareness of their Azure DevOps Pipelines, as well as increase the general security posture across the board. Don't go light on these things - enforce security configuration and avoid sensitive data in your systems.
- Azure DevOps: Variable Groups
Thanks for reading - please leave a comment. Tobi, out.
Comments