Azure Functions and secure Configuration with built-in integration to Azure Key Vault
Azure Functions are usually tied to an Azure Storage Account by using App Settings. Unfortunately, when launching a new Function App project in Visual Studio, or watching demos and examples online, the connection string usually is in App Settings in plain text.
In this post I'm sharing a quick tip on how to protect sensitive configuration values in App Settings by using Secrets from a Key Vault, and you can even reference the default Storage Account connection string this way, completely avoiding any type of sensitive data in App Settings, from scratch.
Microsoft have some good documentation (links in the bottom) on this, and I'm sharing my additional thoughts and experiences around it. I have production workloads that processes hundreds of millions, or aggregated it's in the billions, of transactions to and from Storage Accounts - a key thing I wanted to learn is if the Key Vault References will have any impact on the performance, cost or scalability of the solutions. Tag along.
Why use a Key Vault with Functions by default?
The same way that you don't put a post-it note with a password, you don't put credentials in plain text. This is declared by the security communities at large, and also recommended by Microsoft's practices and guidelines.
There are no valid reasons for having credentials, connection strings, or other type of sensitive data in plain text or available for anyone else to read. With proper DevSecOps and SDL routines, you're staying ahead of the game by protecting the secrets in a Vault.
This also means that even a Global Reader user in your Azure Subscription cannot see the connection strings, Client Secrets, API Keys and whatever else you have in App Settings, which they otherwise could.
Configuring the Key Vault and Secrets for a Function App
We need:
- An Azure Function App
- An Azure Key Vault
- An Azure Storage Account + a Connection String (or other applicable sensitive credential you want to work with)
Grant the Function App access to the Azure Key Vault
By using Access Policies on the Azure Key Vault, we can grant access to the Azure Function App, and if it's using Managed Identity it can do this without credentials anywhere in configuration. This is recommended.
Usually I work with User Assigned Managed Identity, because I can control the lifecycle of that identity better than with a System Assigned identity. However, as of this writing, the Key Vault reference integration only works with System Assigned Managed Identities.
So let's do that:
- Create a System Assigned Managed Identity
- Grant it permissions to the Azure Key Vault with "Get" permissions on Secrets
First we head to the Function App -> Platform Features -> Identity and enable the System Assigned Managed Identity;
Next up, we need to grant this identity access to the Key Vault Secrets. So head on over to the Key Vault, and select "Access Policies", then grant your new identity the "Get" permissions for "Secrets":
We have prepared our Function App by enabling System Managed Identity, and we have ensured that this identity can Read secrets from the Azure Key Vault.
Moving on.
Configure secrets and link to App Settings
In this example, we'll be using a simple Azure Storage Account and the connection string from there, as secrets in the Azure Key Vault.
Here, I'm adding three secrets to my Azure Key Vault:
- AzureWebJobsStorage
- AzureStorageKey
- AzureStorageAccountName
For each secret, I will copy the Secret uri from the Key Vault:
Heading over to my Function App, I can now modify the Configuration section and add references to these secrets, by wrapping the Uri in the @Microsoft.KeyVault() reference:
@Microsoft.KeyVault(SecretUri=https://functionvaultdemos.vault.azure.net/secrets/StorageConnectionString/de3d6ce54e4a43c687ccdd1683e2a75f)
As per the documentation, you can use any of these formats with the @Microsoft.KeyVault approach:
- SecretUri=secretUri
- VaultName=vaultName;SecretName=secretName;SecretVersion=secretVersion
I'm opting for the full Uri because it's simply easier to manage for me.
We have added Application Configuration values that are linked to Azure Key Vault secrets. We can verify this by looking at the "Source" column:
If you now click one of these configuration values, you'll see that there's additional properties displayed to verify that it is indeed connected to a vault secret:
The great thing here is that we now have NO connection strings or sensitive data in our Function App configuration at all.
Get configuration values from .NET Core (C#)
There are plenty of ways to connect to an Azure Key Vault from code, which can also be done using Managed Identity to support a fully passwordless approach.
However, if you don't want to pull in additional resources and dependencies in your code, you can also rely on the ConfigurationManager.AppSettings approach, since you can now link the configuration value to an actual Secret in your Key Vault.
The code is simple, and works just like any other C# configuration fetch:
// For demo purposes I've redacted the local config file and rely
// only on Environment variables, which the App Settings are in Azure.
var config = new ConfigurationBuilder()
.AddEnvironmentVariables()
.Build();
var key = config["AzureStorageKey"];
var name = config["AzureStorageAccountName"];
log.LogInformation($"AzureStorageAccountName:{name??"null"}");
log.LogInformation($"AzureStorageKey:{key?.Substring(0,5) ?? "null"}");
For demo purposes to verify in the logs that the connection works, I have added some logging statements to log the value of the configuration keys as per above snippet. If things are connected properly, the values should display the Secret value.
Verify that the setup works
Next up, we just need to ensure that the Function still works and that it can properly access the Key Vault and the secrets.
Going over to my Function(s) and looking into the logs, I should see the values of the above configurations, which are pulled directly from the Key Vault rather than plain-text configurations.
What about performance at scale, and what about caching of the Secrets?
In order to test how this scales, which is extremely important for my scenarios, I need to ensure that it can handle hundreds of millions of requests with no added overhead.
Update 2019-12-14:
On Twitter, Jeff Hollan replied with a bit more insights into the below explorations, to confirm that the secrets are cached when the functions are instantiated, which means we don't have to do a server round-trip to the vault on every request for a secret.
Yes securely cached in the instance after it starts up so you shouldn’t hit throttles by accessing secret value lots in something like a function app
— Jeff Hollan (@jeffhollan) December 14, 2019
This means that the below assumptions about metrics and how often (or how little) the vault is being called, stays true.
End of update.
My initial thought was to try and figure out whether or not the added references to the Key Vault causes regular round-trips to the Vault, or if it inherently caches the secrets in batched requests, etc.
When I now look at the Metrics of the Key Vault vs. the Storage Account, I have millions of requests to the Storage Account, but there's only a few hundred requests to the Key Vault to grab the Secret.
As I have now learnt, the secrets are cached securely when a function app instantiates. This should mean that there's no exponential impact on the Key Vault, and performance shouldn't be an issue moving to this solution either.
Here's the Storage Account being hit by the Function executions in the last 1 day:
Here's the Key Vault, almost without activity, except for the very irregular round-trip to fetch the Secrets when a function app is started:
I'm glad to learn that we can move into a more secure development and infrastructure environment with no additional complexities like performance, cost or throttling due to heavy load on the functions.
Things to note from above: In my example I've built a fairly dumb logic without any transient fault handling, which is why we see some ClientOtherError in the screenshots. These are caused by 429 (Throttling) and other intermittent failures. In a production cloud design, you should always handle transient faults and ensure that the code is resilient to these things. For my simple blog post here though, I've skipped that part.
Summary
Following suit with best practices and security guidelines from Microsoft, and in general having a good SDL (Security Development Lifecycle) is key to a long-term security posture in the cloud.
There are no reasons why you should leave credentials or other sensitive data in plain text in configuration values. This includes the Function Apps' configurations to their underlying storage accounts, of course.
With the integration options available for natively referencing secrets in a secured Key Vault, we're one step ahead again.
Setting this up was pretty trivial, and it's also well documented from Microsoft. There's still a couple of known caveats that I'm waiting for more information about, which are listed below.
Known issue: Consider Managed Identity type
Most of the time I use User Assigned Managed Identity, where I can fully control the lifecycle of the identity and assign it across my resources as I need.
As of this writing (Dec 12th, 2019), the Key Vault reference does NOT support User Assigned Managed Identities. It only supports System Assigned Managed Identities.
I've opened a GitHub issue asking about any roadmap for GA, here:
Known issue: Consider networking
In all of my production workloads, I have networking around all my resources so I can control the traffic using Virtual Networks, delegated Subnets and Network Security Groups. This allows me to restrict access to the Storage, Key Vault, Function and other resources both for external and internal users and systems.
However, as of this writing (Dec 12th, 2019), the Key Vault reference is NOT supported for resources that are using networking.
I've opened a GitHub issue asking about any roadmap for GA, here:
Links
Recent comments