How to populate Azure Storage Accounts with demo data

If you, like me, love diving into features like monitoring and observability, discovery, insights and analytics, and more, you also know that testing new capabilities out without robust underlying data might be a challenge. Nobody loves an empty dashboard.

So, I built a tool to help you populate storage accounts with data.

GitHub - Zimmergren/azure-storage-demo-populator: A simple command line tool to populate Azure storage accounts with demo and dummy data for training and educational purposes.

A simple command line tool to populate Azure storage accounts with demo and dummy data for training and educational purposes. - Zimmergren/azure-storage-demo-populator

GitHubZimmergren

Here's a few of the reasons why I chose to build this:

• Exploring observability and data insights: You want to see dashboards with real data, capacity, trends of usage, transactions and more.
• Demonstrating batch vs single operations: Uploading one blob versus thousands shows how metrics behave differently. You can keep tabs on what monitoring and telemetry looks like for each type of operation.
• Training and demos: Nobody likes empty dashboards. Populated accounts make dashboards and metrics more engaging, and easier to understand how the capabilities can be used.
• Experimenting safely: With dummy data you can practice lifecycle policies, monitoring queries, and cleanup processes without risk. But please remember to only target specific demo/test accounts that you can easily discard.

In this post, you'll see why you might want to populate storage accounts with dummy data, how this small demo tool can help, and what you will see once it is running.

Using the Azure storage account populator tool

I created a small console app that connects to one or more storage accounts, creates random containers, tables, queues, and shares, and then continuously uploads blobs, inserts rows, and pushes messages.

It runs in bursts; sometimes steady, sometimes very fast. Over time this generates millions of transactions, enough to light up dashboards. With these intermittent dynamic changes to pace and number of items to upload, dashboards will also look more realistic.

Running the tool

From the project folder, you can run:

dotnet run

You can also specify a speed profile:

dotnet run slow
dotnet run medium
dotnet run fastest

If you just run it, it'll use a random speed profile and switch things up intermittently, which is what I found interesting.

See the results

Once the tool is running, you can see the results in Azure in various ways. For example:

• Metrics: requests, latency, success rates, and throttling.
• Azure Storage Discovery: aggregated view of capacity, transactions, and throughput across accounts. However, be advised that there's a several hour-long delay for Azure Storage Discovery to populate new data, so you might want to check back there later.
• Storage Insights in Azure Monitor: built-in dashboards for accounts, containers, tables, queues, and shares.

Here's a quick example of what the result could look like, running at intermittent speeds for about half an hour.

+1,000,000 transactions, casually adding demo content to the storage accounts.

After an hour or two, the results are indistinguishable from a busy production-like workload, if you look purely at the transaction data and metrics.

Considerations

Before you start hammering away with this tool, consider this.

• Lifecycle management: The tool creates thousands or millions of blobs and rows, queue messages, and files. The easiest cleanup is to delete the demo storage account. Therefore, ensure you target only demo accounts.
• Costs: Storage and transactions are inexpensive, but not free. Expect some small cost if you leave the tool running for very long periods.
• Scalability limits: The tool will push hard; throttling and retries are expected. This is useful for the demo, but should not be aimed at production accounts.
• Multiple accounts: The tool can target several accounts in parallel and switch between them, giving you a wide horizontal spread of activity.
• Exposed secrets: Connection strings are hardcoded today, which is very inadvisable. I would recommend using managed identities and Azure Key Vault for storing secrets, but I built this for a quick demo purpose only – feel free to modify the source code any way you want to fit your needs.

Summary

If you want to see how Azure Storage Discovery and other monitoring features behave under load and with realistic transactions, you need accounts filled with activity. A small console tool that continuously generates dummy blobs, tables, queues, and files is a simple way to achieve this.

You get:

• Accounts that look busy and production-like.
• Realistic metrics and dashboards to explore.
• A safe environment for demos, training, and testing.

Enjoy.