Best Practices for security in Azure Container Registry

We've transitioned over the years into more modern ways to build and deploy our systems, and the Azure Container Registry has started to play a more important role for our private repositories every day.

In this post I'll talk about some of my experiences and recommendations for using the Azure Container Registry in production, and at scale, with a mindset of zero-trust and being security aware.

There's a lot of angles for any good practice. This post focuses around a healthy security posture.

Secure your Azure Container Registry

Previously, I wrote about Embracing a Security Development Lifecycle (SDL) for Azure, and Automate Azure DevOps code security analysis with the Microsoft Security Code Analysis extensions.

Embracing security as a daily necessity rather than an after-thought is important. We can strengthen our security posture by using a SDL as per my previous post (link above), and we can even automate a lot of security code analysis using various tools (also linked above).

However, even if we run code analysis and we have a great SDL routine in place in our organization, we also need to tend to the deployed assets, and their infrastructure. Container images goes into the Azure Container Registry, so it definitely makes sense to ensure that your registries are tight. This is especially true if your registries contain images with Intellectual Property, etc.

Tip 1. Enable Azure Security Center

With Security Center, you can get automatic scans of your registries, and it's actually very helpful.

Here's an example of what it can look like, pulled from one of my older workloads:

Clicking one of the items brings out the very detailed explanation and links, including the public CVE's.

This information can now help you made the decision whether you want to take action or not. Ideally, when there's any type of security flaws, you would want to remediate that as soon as you can.

Tip 2. Disable the Admin Account on your Container Registry

While it's a very convenient way to allow access into your ACR, or Azure Container Registry, it comes at a price. The price isn't monetay, but a single username/pass combination to access everything in your Container Registry.

To add to the above, you also have only a single username appearing in the audit logs - it can be hard to understand what user or system really initiated the request to your registry without additional thorough logging, as the username in the log will just be the admin user, not the initial caller.

This can of course be mitigated by disabled the admin account, and enable RBAC and Azure AD authorization instead. Managed identities, service principals or users - anything that suits your architecture and infrastructure, but that comes out more traceable in the end.

Option 1: Disable ACR Admin User from the Azure Portal

From the Azure Portal, navigate to your Container Registry and select "Access Keys", then disable from here:

Option 2: Disable ACR Admin User from Azure CLI

Another option is to use the Azure CLI to disable the admin user. Here's how:

az acr update -n myacr --admin-enabled false

Option 3: Disable ACR ADmin User from Azure PowerShell

From Azure PowerShell, you can also disable the admin user:

Update-AzContainerRegistry -Name  'myacr' -ResourceGroupName 'rg-weu-myacr' -DisableAdminUser

Tip 3. Authorize using Service Principals

So, we disabled the Admin User, but we still have some type of headless application running as a service or automated task, and that still needs to access the Azure Container Registry. What do we do?

One option is to use Managed Identities - we'll get to that later. The other option is to use Service Principals.

Microsoft have documented this very well here: Azure Container Registry authentication with service principals.

Key reasons for most of the scenarios where I've used this:

• Headless: Services and apps running automated tasks, without any interactive auth flows.
• Granular access rights, exactly how you want them, instead of a "one key to rule them all".
• Read-only scenarios easily supported, where the principal never can modify or push images.

Tip 4. Authorize using Managed Identity

If you, like me, use Azure Functions, Azure Container Instances, Azure Web Apps or other services that have support for identities, it will ease your mind to know that the Azure Container Registry have great support for managed identities as well.

Personally, I prefer managed identities over Service Principals, and especially the user-assigned managed identities. They reside inside my resource group(s), and don't pollute the Azure Active Directory with additional principals.

Regardless if you use System Assigned or User Assigned Managed Identity, ACR supports it.

Key reasons for using it:

• Granular access control with RBAC.
• Enables a specific service or resource direct access to what it needs, enabling granular access controls for various services with different identities, like "This Azure Container Instance should have Reader access only".
• This identity, tied to a given resource, can also be assigned permissions to other resources that support managed identities, giving you an easy way to manage access for that one resource without creating additional principals, etc.

Tip 5. Stay on top of RBAC (Role-Based Access Control)

With most modern services in Azure, you can easily define granular access rights using RBAC. Azure Container Registry is no exception to this, and you can enforce strict access rules using RBAC.

So we created a service principal, or used Managed Identities - how do we control what they can access? The answer is RBAC.

Key takeaways from my experiences:

• Assign the 'Reader' role to identities/users/principals who should only pull images, but never modify or make other changes.
• Stay on top of your RBAC assignments; Ensure there's no delegated access, and that there's no inherited access for certain accounts with a lot of privileges.

Microsoft provides a pretty nice overview of the Azure Container Roles and permissions.

Tip 6. Use Webhooks to stay on top of Vulnerabilities

Webhooks for ACR are awesome. You can subscribe to events that happen inside the Azure Container Registry, instead of scheduling a poll-job that goes in to check things every now and then.

This enables you to more easily do event-based scans for vulnerabilities, which has also been highlighted as a clear benefit, especially if you are tying in third-party tools and services for enhanced scanning.

Microsoft mentions their Vulnerability Scanning Update here, while also making clear that the Azure Security Center will have some of these capabilities.

If you're using a third-party vulnerability scanner, it can tie into events in your ACR rather than schedule analysis. I like this option of event-based actions rather than having gaps from when an event happened to the next scan.

Ideally, image vulnerability scans should be configured though webhooks directly on a push-event, which means you can immediately remediate any security issues or vulnerabilities it finds, before putting it to use in production.

For more insights into how ACR can work with webhooks, Thorsten Hans have a great post about it.

Tip 7. Make use of Content Trust

Docker comes with a model called "Content trust", which enables you to enforce signing of images that are pushed and pulled. We usually talk about securing the infrastructure and code.

With content trust, the idea is to verify the source and integrity of the images, ensuring that they are the images we expect. You can sign your images as you're pushing them to the registry, and can then configure the clients to only pull signed images, coming from a trusted source with verified data integrity.

There are some steps involved for a full-cycle content trust scenario to work, and they've done a great job documenting this in the official Microsoft docs: Content trust in Azure Container Registry.

Tip 8. Build new container images as base images are updated

If you're basing your container images from any available base image, there is a large chance that these images will be updated at some point, some more regular than others.

A base image is when your dockerfile points to a parent image to base the new image on. For example, if you are (like me) running .NET Core cabable docker images, Microsoft have their base images here: https://hub.docker.com/r/microsoft/dotnet/

These images gets regular updates, and as they update you should update your own image to use the latest version too - and of course test accordingly.

With the Azure Container Registry, you get something called "Tasks" that can help automate this process - it supports automated build execution when the base image is updated.

Read "Tutorial: Automate container image builds when a base image is updated in an Azure container registry".

The bottom line is that I don't allow outdated or vulnerable images in my system, and ensuring that they're always up to date can help mitigate any threats from known (or unknown) vulnerabilities.

Tip 9. Store sensitive configuration data in an Azure Key Vault

Something I've seen in the field, is Azure Container Registries configured with IAM and access using Service Principals. This is great, and allows for a granular control with RBAC.

Unfortunately it's not uncommon to see services that connect to ACR to store the credentials to this said Service Principal in plain text, or other non-protected resources like application configuration (app.config, project.json, ...) or simply in Environment Variables (Configuration on an Azure App Service / Function App, etc).

It defeats the whole purpose of securing resources if we put the credentials to the resources in plain text somewhere. This can easily be mitigated by making use of an Azure Key Vault.

The scenario can then be like this:

1. Your service/app/daemon uses Managed Identity to access the Key Vault
2. Reads the Service Principal credentials from the Secrets in the vault
3. Use these credentials to access the ACR
4. Does its thing.

By putting the sensitive data inside of your Azure Key Vault, you're protecting the credentials at rest as well - which isn't the case if it goes into a configuration repository that isn't inherently secured for sensitive data.

Tip 10. Review Activity Logs for your Container Registry

With the Azure Container Registry, you also get a great audit log in the "Activity log" menu option in the Azure Portal. Reviewing these logs regularly can help to stay on top of things more easily, and explore and find erroneous, suspicious and anomalous activity.

Tip 11. Use the Firewall

In the past, I wrote Protecting your Azure Container Registry by denying all requests except from allowed IP addresses, which shows how to use Virtual Network rules with your Azure Container Registry.

There's some guidance available from Microsoft on this topic now, on docs: Restrict access to an Azure container registry using an Azure virtual network or firewall rules.

With a virtual network or firewall rule in place, you can more easily control the entire traffic flow. I use, and enforce, this setup with most of my infrastructure where applicable.

I've already covered this scenario from various angles, which are available from the links above and in the end of this post.

Good things to read up about if you're choosing to put ACR behind a network:

• Virtual Networks and how they work, specifically with ACR.
• Subnets, and how these work.
• Service Endpoints, and when/why to use them.

Tip 12 (Preview). Repository-scoped permissions

Here's one of my favorite up-and-coming features, which enables even more granular control. Instead of scoping RBAC at the entire ACR level, you can scope the permissions directly to a repository inside of the ACR.

This enables even more granular control of access, and you can have a single ACR for a use case that have multiple images, where different access control is required - so an identity that have Reader access can only read from a given repository, instead of the entire registry and all repositories inside of it.

Known limitations (2020-02-07 Preview):

• You cannot scope access to an Azure AD object like a Service Principal or Managed Identity
• This is only available in the Premium tier of ACR

While I have yet to actually try this out (because it's preview, and I can't use that in my production workloads), here's what Microsoft says about it:

This article shows how to create and use an access token that has  permissions to perform actions on only specific repositories in a  registry. With an access token, you can provide users or services with  scoped, time-limited access to repositories to pull or push images or  perform other actions.

Read more about this feature in the Microsoft Docs: Repository-scoped permissions in Azure Container Registry.

Summary and Links

Only a few moments after publishing this post, I realized I've missed a point or two. Hence the list is no longer a "Top 10" but rather just a list of my recommended best practices. 🚀

This wraps up a post I've had in my mind for some time. It's easy to spin up new resources, but after a while - and especially if you manage more than a few small environments, it becomes a hassle to stay on top of things.

I hope this post from my experiences can help clarify the plans and paths forward for someone looking to get their hands dirty with ACR. It's plenty of fun, and doing it right from start enables you to more easily stay on top of things down the road - and of course there will be more green and less red lights in Azure Security Center in the end.

Additional reading from the public docs:

• RBAC Roles and Permissions (Microsoft)
• ACR and Security Center integration (Microsoft)
• Authenticate with an Azure Container Registry (Microsoft)
• Authenticate to ACR with Managed Identities (Microsoft)
• Repository-scoped Permissions for ACR (Microsoft)
• Content trust in Azure Container Registry (Microsoft)
• Content trust and Azure DevOps (Microsoft)

More reading:

• Azure Container Registry Unleashed – ACR up and running (Thorsten Hans)
• Content trust in Docker (Docker)