Cloud cost allocation made easy

Skip tedious tagging in favor of this simple but robust AWS account strategy that makes neat, shareable reports quickly.

Imagine you’re on the path to maturing your startup, and the finance team needs you to build reliable and correct cost attribution. You know, things like cost of goods sold (COGS) and contribution margin. The problem is that you have everything in a huge AWS account. Your best hope is a tedious and error-prone project to identify and tag every resource. After that’s done, you’ll probably still be chasing resources that fell through the cracks or finding new things that were not tagged properly.

Welcome to the challenge of cost allocation on AWS. 

I’ve been there, leading an entire team dedicated to cloud costs at Confluent. We had to do it the hard way because we didn’t do a multi-account strategy early. I have an easier way, but it takes a bit of forethought. 

TLDR: You should have lots of AWS accounts.

When you should switch to multiple accounts

When an engineering team is just a few people in the early days, it may be hard to imagine a time when tagging will be anything other than a small annoyance. However, your future self and (hopefully growing) team may beg you to consider the following:

• It’s hard to enforce tagging to ensure every team that creates AWS resources is doing it properly.
• Sometimes tags don’t propagate into subordinate resources.
• Some APIs don’t allow you to tag them at creation time.
• Someone could change a tag later.

Or perhaps, you’re a relatively young startup with a single product. You’re seeing some traction, and you’ve been around the block enough to know you want to get a few things right before it’s too late. You like the idea of isolating your environments, and you might even have SOC2 in your future. You know you’re going to have to start allocating costs separately for production, pre-production, and different products.

Likewise, your startup may be debuting its second, third, or fourth product, and it will run mostly on its own stack. You want to set it up in its own AWS accounts. Perhaps this can be the model for future products and even a migration of your existing systems.

But truly, you could just be super sick of chasing down tags and cost leaks, and want to reap the other benefits of a multi-account strategy. So you’re ready to start migrating parts of your system into their own accounts.

The setup

We’ll be focusing on the account setup to create isolation, so you can view costs by account. Of course, you’ll also have to set up your applications or migrate them into those accounts. How to do that is very specific to your particular setup, so we won’t be covering that here.

Step 1: Separate development from production

Create AWS accounts for development, staging, and production at a very minimum. In Substrate we call these “environments,” and each environment can even be multiple AWS accounts with a shared VPC.

If you have existing systems, the easiest thing is to create the new accounts or environments for development and staging. Then migrate them out to those new accounts, leaving production where it is. You probably don’t want to take production down, but you can coordinate an outage for development and staging environments to cleanly migrate them.

If you’re not using Substrate, create these accounts however you prefer. Otherwise, follow along and adapt the commands to your environment.

We can create new environments for development and staging, and while we’re at it, we’ll put our app servers and db servers in separate domains called app and db.

Run:

substrate setup

Then answer “no” when asked if your list of environments is correct, and follow the prompts. Next, create your domains with these commands:

substrate account create --domain app --environment staging

substrate account create --domain db --environment staging

substrate account create --domain app --environment development

substrate account create --domain db --environment development

Now you can build your staging and development environments in the new domain or environment accounts.

Step 2: Separate non-development and non-production workloads

Identify any other non-production systems. These could be things like a data warehouse or other analytics tools that are important to your business but that probably don’t qualify as COGS. Migrate these to their own account or environment.

Perhaps you have some analytics or business intelligence use case that needs to be able to copy production data into a data-warehouse, but that system is not COGS, so you want to attribute costs separately. Setting up the isolation is easy with Substrate. Run:

substrate account create --domain analytics --environment production

Now you can start building up your analytics stack, and because Substrate created the new account in the shared production environment VPC, your ETLs can still talk to the production databases.

Step 3: Separate different production applications

As soon as you create a new application, or product, put it in its own account. If you already have multiple applications or products, work up a migration plan to put them into their own accounts. If you’ve been reading along, you already know how to do this with Substrate. Simply create a new domain, and you’re ready to go. If your new application is called payments, run this:

substrate account create --domain payments --environment production

You should also setup a staging and development environment for it, so run these:

substrate account create --domain payments --environment development

substrate account create --domain payments --environment staging

Now you’re ready to start building out that new payments application in your new domains.

Step 4: View costs by account in AWS Cost Explorer

In your organization’s management account, go to Billing and Cost Management and then Cost Explorer. On the right side of Cost Explorer, go to Report Parameters”. You may have to expand it if it’s not visible. Then select “Linked account” in the Dimension dropdown menu:

Now you should have a view similar to this one showing costs by account, and you didn’t have to tag a single thing!

The takeaway

On AWS, go multi-account early and often. Separate pre-production frnoom production. Separate different production applications from each other.

Use AWS Cost Explorer to view your costs by account! (You can even give your friends on the finance team access to AWS Cost Explorer.)

You can make neat reports quickly, slicing by application, because each application is in its own account. Easy!