Azure Container Instances (ACI) and Secrets - Creating secret volumes and consume secrets using C# .NET Core

Managing secrets in any type of application should be treated with the utmost respect. Especially if the said secrets are confidential data, connection details to systems or anything else that would be risky to not keep safe. In a couple of my previous blog posts, I've talked about Azure Container Instances and how you can do things like run your private C# .NET Core apps in ACI, and using Secure Environment Variables with ACI.

This post aims to take the Secrets-part further by exploring yet another option for storing secrets when running ACI. This time, we're talking about mounting a Secret volume that your container accesses during runtime.

Background

As a follow up to the previous posts, I would like to expand further on the topics of Secrets and how to maintain them in your Azure Container Instances. With this post, we'll investigate how we can use the --secrets and --secrets-mount-path flags with the Azure CLI to create a new container instance with a mount to a Secret volume.

What's the benefit with a secret volume?

• Any data that resides in a secret volume is entirely stored in a memory.
• Secret volumes are based on tmpfs, a temporary file storage facility that lives in the RAM of your container group only.
• When your container dies, your secrets die with it.
• You can easily consume the secrets from inside the container, as they will be mounted to /yourMountPath/secrets - so if you follow my example below, you can read them using file operations and read them right out of the mount path.

What's the main differences between a secret volume and secure environment variables?

Good and valid question, thanks for asking!

Secure environment variables

• One key/value pair per secret, but no categorization or folder structure.
• Easy to specify both with the Azure CLI and with a YAML definition file.
• Are consumed in your apps by reading the container's environment variables. In C#, this is done with Environment.GetEnvironmentVariable("yourVariableName").

Secure volume

• Easy to specify both with the Azure CLI and with a YAML definition file.
• Secrets exist as a mounted folder, where each secret has its own folder and the contents of that folder is the value of your secret.
• Multiple secret volumes can be mounted, to create some type of structure.
• Are consumed in your apps by reading the mounted path. In C# this is done with for example System.IO classes.

Are there limitations depending on if you use the CLI exclusively, or use YAML?

Another great question, thanks for asking!

One of the main differences I've stumbled on during my experience working with these things:

• CLI: You can specify one secret volume mount which would contain all your secrets.
• YAML: You can specify multiple secret volume mounts, and you can put secrets in different logical buckets this way.

How to create a Secret Volume for Azure Container Instances

Let's get our hands dirty. Now we'll create a Secret volume and secrets in two varieties. On one hand, we'll use the Azure CLI exclusively, and on the other we'll define a YAML file to mount multiple secret volumes and mount them.

Create a Secret volume for ACI using the Azure CLI

In the Azure CLI, there's the az create container command that I've mentioned in my other posts. There's two flags you can specify, --secrets and --secrets-mount-path. We'll make use of those to create a set of secrets, and then mount them as a secret volume in our container group.

Here's how it works.

az container create 
  -n aci-demo-app-with-secrets-mount 
  -g demos 
  --image "acrdemomagic.azurecr.io/aci-demo-with-secret-mount-app:latest" 
  --registry-username "acrdemomagic" 
  --registry-password "YOUR_ACR_PASSWORD" 
  --secrets storage-account-name="Sample Account Name" storage-account-key="Sample value" app-subscription-key="Yet another example value." app-license-type="Some other secret value here, as an example."
  --secrets-mount-path "/mountpath/app-secrets"

The last two lines above declares a set of secrets, and then where these secrets will be accessible from inside the container, as a mounted path, available at /mountpath/app-secrets.

See further down in this post to learn how to fetch these values using C# with .NET Core.

Create multiple secret volumes for ACI using YAML + CLI

With the previous approach it's super quick and convenient to shoot out a new container with some az container create logic. In the following example, we're instead using a yaml definition file and will specify multiple mount paths to hold different types of secrets.

• Mind that the values of the secrets in a yaml file has to be Base64 encoded.

My aci-demo-with-secret-volume-mount.yaml file:

apiVersion: 2018-10-01
name: aci-demo-app-with-secrets-mount
location: westeurope
type: Microsoft.ContainerInstance/containerGroups
properties:
  osType: Linux
  restartPolicy: Always
  containers:
  - name: aci-demo-app-with-secrets-mount
    properties:
      image: YOUR_ACR_REPO_HERE.azurecr.io/aci-demo-app-with-secrets-mount:latest
      resources:
        requests:
          cpu: 2.0
          memoryInGB: 2.0
      volumeMounts:
      - name: 'azure-resources-secrets'
        mountPath: '/mounts/azure-resources-secrets'
      - name: 'app-secrets'
        mountPath: '/mounts/app-secrets'
  volumes:
  - name: 'azure-resources-secrets'
    secret:
      storage-account-name: 'bXkgc2VjcmV0IG51bWJlciAx'
      storage-account-key: 'bXkgc2VjcmV0IG51bWJlciAy'
      aad-app-clientid: 'bXkgc2VjcmV0IG51bWJlciAz'
  - name: 'app-secrets'
    secret:
      app-subscription-key: 'M2VmNTNhZTgtY2E5OS00MjNjLWJlMTUtNWIxODE4YWJjNWM0'
      app-deployment-type: 'TGljZW5zZVR5cGUuUHJv'
  imageRegistryCredentials:
  - server: YOUR_ACR_REPO_HERE.azurecr.io
    username: YOUR_ACR_NAME_HERE
    password: YOUR_ACR_PASSWORD_HERE

As previously, we can now easily shoot this up into our Azure subscription:

az container create -g demo -f aci-demo-with-secret-volume-mount.yaml

Things to note, below.

volumeMounts:

We define the volume mounts. In my case, I'm showcasing two different mounts with different paths and names as below:

• azure-resources-secrets : /mounts/azure-resources-secrets
• app-secrets : /mounts/app-secrets

volumes:

Under the volumes param, we have secret which is the config that holds all the secrets per volumeMount, and it uses the name property to target the correct mount path.

• azure-resources-secrets: I define three new secrets here.
• app-secrets: I define two new secrets here.

These are now logically separated, but in theory they're all still accessible from inside the container - so it's more out of convenience than anything else.

Pro tip: If you're using Azure DevOps, you can automate pretty much everything around this - including dynamically replacing values inside the yaml files. But that's for another day.

Verify secret mounts using the Azure CLI

Using any of the aforementioned create-approaches, you would get a json response when the request is returned successfully, which would look something like this:

{
  "id": "/subscriptions/YOUR_GUID/resourceGroups/demos/providers/Microsoft.ContainerInstance/containerGroups/aci-demo-app-with-secrets-mount",
  "identity": null,
  "kind": null,
  "location": "westeurope",
  "managedBy": null,
  "name": "aci-demo-app-with-secrets-mount",
  "plan": null,
  "properties": {
    "containers": [
      {
        "name": "aci-demo-app-with-secrets-mount",
        "properties": {
          "environmentVariables": [],
          "image": "acrdemomagic.azurecr.io/aci-demo-app-with-secrets-mount:latest",
          "instanceView": {
            "currentState": {
              "detailStatus": "",
              "startTime": "2019-02-20T20:42:04Z",
              "state": "Running"
            },
            "events": [
              {
                "count": 1,
                "firstTimestamp": "2019-02-20T20:41:46Z",
                "lastTimestamp": "2019-02-20T20:41:46Z",
                "message": "pulling image \"acrdemomagic.azurecr.io/aci-demo-app-with-secrets-mount:latest\"",
                "name": "Pulling",
                "type": "Normal"
              },
              {
                "count": 1,
                "firstTimestamp": "2019-02-20T20:41:54Z",
                "lastTimestamp": "2019-02-20T20:41:54Z",
                "message": "Successfully pulled image \"acrdemomagic.azurecr.io/aci-demo-app-with-secrets-mount:latest\"",
                "name": "Pulled",
                "type": "Normal"
              },
              {
                "count": 1,
                "firstTimestamp": "2019-02-20T20:42:04Z",
                "lastTimestamp": "2019-02-20T20:42:04Z",
                "message": "Created container",
                "name": "Created",
                "type": "Normal"
              },
              {
                "count": 1,
                "firstTimestamp": "2019-02-20T20:42:04Z",
                "lastTimestamp": "2019-02-20T20:42:04Z",
                "message": "Started container",
                "name": "Started",
                "type": "Normal"
              }
            ],
            "restartCount": 0
          },
          "ports": [],
          "resources": {
            "requests": {
              "cpu": 2.0,
              "memoryInGB": 2.0
            }
          },
          "volumeMounts": [
            {
              "mountPath": "/mounts/azure-resources-secrets",
              "name": "azure-resources-secrets"
            },
            {
              "mountPath": "/mounts/app-secrets",
              "name": "app-secrets"
            }
          ]
        }
      }
    ],
    "imageRegistryCredentials": [
      {
        "server": "acrdemomagic.azurecr.io",
        "username": "acrdemomagic"
      }
    ],
    "instanceView": {
      "events": [],
      "state": "Running"
    },
    "osType": "Linux",
    "provisioningState": "Succeeded",
    "restartPolicy": "Always",
    "volumes": [
      {
        "name": "azure-resources-secrets",
        "secret": {}
      },
      {
        "name": "app-secrets",
        "secret": {}
      }
    ]
  },
  "resourceGroup": "demos",
  "sku": null,
  "tags": null,
  "type": "Microsoft.ContainerInstance/containerGroups"
}

Key points:

• volumeMounts are created.
• volumes are mounted, but no secrets are displayed.

Verify secret mounts using the Azure Portal

From the Azure Portal, navigate to your container group and check the container:

Read Secret volumes using C# with .NET Core

Awesome, you've come this far. Why stop now?

In this section we'll take a look at a small sample code that does a basic set of tasks:

1. Get the secret mount(s)
2. Iterate all secrets
3. Log them in the console

You're right, this is not for production scenarios - never log sensitive data in any telemetry, console logs, or any type of exception handling you've got going on. Sensitive data is called sensitive for a reason - so the example above is just to prove that the code works, then you can do whatever you want with it - except leak the data ;-)

Basic C# app in .NET Core that fetches the secrets

Note, if you need guidance on how to get your C# app from your box to the cloud, I've already discussed this topic.

The code is available for download, including the docker files and yaml files et al, on GitHub. Here's the main block of code, which is easy enough to understand and that reads the secrets mounted to the containers.

public class Program
{
    // name: azure-resources-secret, path: /mounts/azure-resources-secrets
    private const string AzureSecretsMountPath = "/mounts/azure-resources-secrets";

    // name: app-secrets, path: /mounts/app-secrets
    private const string AppSecretsMountPath = "/mounts/app-secrets";

    static void Main(string[] args)
    {
        Console.WriteLine("Hello World");
        Thread.Sleep(1000);
        while (true)
        {
           ReadSecretMountVolume(AzureSecretsMountPath);
           ReadSecretMountVolume(AppSecretsMountPath);
           Thread.Sleep(5000);
        }
    }
    
    private static void ReadSecretMountVolume(string mountPath)
    {
        Console.WriteLine($"Processing: {mountPath}");

       var secretFolders = Directory.GetDirectories(mountPath);
       foreach (var folder in secretFolders)
       {
           var allSecretFiles = Directory.GetFiles(folder);
           foreach (var f in allSecretFiles)
           {
               Console.WriteLine($"Secret '{f}' has value '{File.ReadAllText(f)}'");
           }
        }
    }
}

Key points in code:

• Every Secret volume mount is a directory
• Every directory has sub-directories, which is the names of the secrets
• List all files under the sub-directory and read it - this is the value of the secret.

Verify the logs in the portal

To quickly ensure that it works, the demo code just outputs the secrets to the console, and they will show up in the logs like this:

So we can see that we found our /mounts/app-secrets and /mounts/azure-resources-secrets mounts easily, and foreach of the directories inside of these mounts, we grab the secrets and list their values.

Summary

Hey, you made it to the end. This makes me very happy!

We have just walked through these things:

• Benefits with a secret volume
• Difference between secure environment variable and secure volume
• Limitations (that I've experienced) with CLI vs. YAML
• Creating secret volumes for ACI
• Verifying secret volumes for ACI
• Consuming/Reading the secrets using C# with .NET Core
• Verified the logs of our app to ensure that the app does indeed pull out the correct data from the secrets mounts!

Enjoy, and PLEASE leave a COMMENT if you found it useful <3