Networking

Substrate manages VPCs for you for each environment and quality pair, which it shares into the service account for each domain in that environment and quality. This serves to balance isolation between domains with practical connectivity necessary to deliver a higher-level service.

If you’ve chosen to use multiple qualities, in search of incremental change to AWS resources, Substrate will peer the VPCs for each quality in an environment to create a flat network. This enables e.g. a fleet of web servers deployed to both beta and gamma quality to reach a database cluster running just at gamma quality.

If you’ve chosen to use multiple regions, Substrate will peer all the VPCs in all your regions that share the same environment. This, too, creates a big flat network and removes a lot of friction from using both multiple AWS accounts and multiple regions.

To make all of this happen, Substrate actively manages the CIDR prefixes assigned to all your VPCs and ensures they do not overlap.

All your VPCs exist in your network account and are managed by Terraform code that is generated by substrate setup in root-modules/network. You can re-run this Terraform code anytime by re-running substrate setup.

Subnets

Substrate chooses the newest three availability zones in each region and creates a public and a private subnet in each of those availability zones. Each public subnet is an IPv4 /22; each private subnet is an IPv4 /20. Each subnet also assigns IPv6 addresses.

The subnets come with unsurprising route tables. Public subnets get Internet Gateways. Private subnets get IPv6 Egress-Only Internet Gateways and, if you opt in, IPv4 NAT Gateways.

In your Terraform code, you can use module.substrate.public_subnet_ids and module.substrate.private_subnet_ids to get a set(string) of those three subnets to use in other Terraform resources that create clusters, define autoscaling groups, and so on.

NAT Gateways

NAT Gateways, when configured zonally as Substrate does, cost about $100 per environment/quality per region per month. Thus, the substrate.nat-gateways file is available to control whether they’re provisioned at all. If they’re not, your private subnets are limited to IPv6-only outbound connectivity. In practice, this is almost entirely sufficient, with the one glaring hole being access to https://github.com.

VPC Endpoints

Substrate automatically configures the gateway-style VPC Endpoints for DynamoDB and S3. These two VPC Endpoints are free and can dramatically cut network transit costs you incur.

Interface-style VPC Endpoints (which are newer) are available for most other AWS services, however they come at a cost and so are not created by default. Many won’t be worth the cost if usage is low.

However, some AWS services aren’t yet available over IPv6, so these services will be unavailable from private subnets without either a VPC Endpoint or a NAT Gateway.

The interaction between VPC Endpoints and security groups in a shared VPC can be confusing. We’ve verified that creating both the endpoint and its security group in your network account is the best path to follow.

Security groups

Substrate’s VPCs and subnets are defined in the network account and shared into your service accounts. Security groups, though, exist in your service accounts. They can allow whatever traffic they want — CIDR prefixes and security group IDs from other domains being the most common sources.

In your Terraform code, you can use module.substrate.vpc_id to define security groups.

Creating your own VPCs

If for some reason you don’t want to use the Substrate-managed VPCs, you’re free to create your own directly in your service accounts. If you do so, and these VPCs are meant to interact with services in Substrate-managed VPCs, consider using Private Link or VPC Peering to more cost-effectively and securely integrate these VPCs.

Beware, though, that there’s a per-byte cost for crossing a VPC boundary, even within an availability zone. It can add up.