-quality "..." is now optional in organizations that have only one line in substrate.qualities, which can allow substrate assume-role, substrate create-account, and substrate create-admin-account invocations to be a little shorter. Note that substrate accounts -format shell will continue to generate the longer version with an explicit -quality option.-number to substrate create-account which allows you to tag an existing organization account with a domain, environment, and quality, generate Terraform modules for it, and manage its basic IAM roles just like any other Substrate-managed account.s3:GetBucketLocation, s3:GetObject*, s3:ListBucket*, and s3:PutObject*.Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.
After upgrading Substrate, you should run sh <(substrate accounts -format shell -no-apply), review what Terraform plans to do, and then run sh <(substrate accounts -auto-approve -format shell) to apply the changes.
modules/common/global and modules/common/regional which will be instantiated in every new service account. See below for directions on opting existing service accounts into this new default behavior.modules/deploy/global and modules/deploy/regional as blank slates so folks can add resources (e.g. AWS ECR repositories) to their deploy account in every region.substrate credentials -format json to be compatible with the AWS SDK’s credential_process directive. Specifically, rename AccessKeyID to AccessKeyId and Expires to Expiration, as documented in Sourcing credentials with an external process.substrate credentials -no-open to give folks the opportunity to choose which browser window opens the Credential Factory authorization.substrate credentials as a credential_process for the AWS CLI/SDK. Substrate never intended to read ~/.aws/config or ~/.aws/credentials but was doing so inadvertantly; it no longer reads either of these files.If you wish to instantiate the new common modules in your existing service accounts, take the following steps for each domain:
Add the following block to modules/domain/global/main.tf:
module "common" {
providers = {
aws = aws
aws.us-east-1 = aws.us-east-1
}
source = "../../common/global"
}
Add the following block to modules/domain/regional/main.tf:
module "common" {
providers = {
aws = aws
aws.network = aws.network
}
source = "../../common/regional"
}
Run substrate create-account -domain domain -environment environment -quality quality for each domain service account.
These modules aren’t being instantiated in existing service accounts automatically because Substrate can’t guarantee that’s safe.
Upgrade Substrate by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.
After upgrading Substrate, you should run sh <(substrate accounts -format shell -no-apply), review what Terraform plans to do, and then run sh <(substrate accounts -auto-approve -format shell) to apply the changes.
"*" in your admin account.substrate credentials failures they may encounter.substrate credentials to ensure we get more meaningful error messages and fewer 504 Gateway Timeout responses. This will manifest as Terraform lowering the Intranet Lambda functions’ timeouts.arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs to your admin account’s Administrator role. This will manifest as Terraform destroying module.intranet.aws_iam_role_policy_attachment.admin-cloudwatch in your admin account.substrate credentials (whether due to bugs, the process being interrupted, or something else).Get the 2022.09 release by running substrate upgrade and following its prompts. If your copy of substrate is writeable, this will be all you need to do to upgrade.
After upgrading Substrate, you at least need to run substrate create-admin-account -quality quality to update your Intranet. Even better would be to run sh <(substrate accounts -format shell -no-apply), review what Terraform plans to do, and then run sh <(substrate accounts -auto-approve -format shell) to apply the changes.
-console option to substrate assume-role in order to keep that command completely decoupled from the Intranet.)-no-apply to substrate accounts -format shell to enable you to review all the changes Terraform is planning to make across all your accounts at once. (After you review, you could swap -no-apply for -auto-approve or run terraform apply in each root module directly.)substrate credentials. There’s a theoretical security benefit to doing so but mostly this prevents forgotten shells from running up your Lambda and CloudWatch bill while muddling your Intranet’s logs.substrate upgrade command, to be put into service with the 2022.09 release. (More details and updated documentation will accompany that release.)RoleSessionName in the sts:AssumeRole API.Upgrade Substrate as in the updated installation manual:
tar xf substrate-version-commit-OS-ARCH.tar.gz -C ~/bin --strip-components 2 substrate-version-commit-OS-ARCH/bin/substrateEach released version and commit is offered in four binary formats; choose the appropriate one for your system.
OSis one of “darwin” or “linux” andARCHis one of “amd64” or “arm64”.You can install Substrate wherever you like. If
~/bindoesn’t suit you, just ensure the directory where you install it is on yourPATH.
After upgrading Substrate, the best idea is to run sh <(substrate accounts -format shell -no-apply), review what Terraform proposes to do, and then run sh <(substrate accounts -auto-approve -format shell) to ensure your code and your AWS organization don’t diverge. If you need a minimal upgrade process, it’s substrate create-admin-account -quality quality to update your Intranet.
Advance notice of an upcoming change: Next month’s release will delete an old EC2 security group that was used by the Instance Factory until late 2021. Beware that, if you have any Instance Factory instances from 2021 or earlier, you’ll have to change their security group or terminate them before upgrading next month.
substrate program in bin/ from optional extra programs in opt/bin/.substrate so previously installed symbolic links will continue to function).substrate accounts -format shell output with set -e -x so the shell will stop on error and print the commands as they’re executed.-auto-approve to substrate accounts -format shell to enable one-command, non-interactive upgrades for the most daring among us.module.substrate.cidr_prefix to the other outputs of the convenience module.substrate that’s automatically available in domain modules.substrate create-account in 2022.06 failed to create accounts but now it’s back. (It had one job!)Upgrade Substrate as in the updated installation manual:
tar xf substrate-version-commit-OS-ARCH.tar.gz -C ~/bin --strip-components 2 substrate-version-commit-OS-ARCH/bin/substrateEach released version and commit is offered in four binary formats; choose the appropriate one for your system.
OSis one of “darwin” or “linux” andARCHis one of “amd64” or “arm64”.You can install Substrate wherever you like. If
~/bindoesn’t suit you, just ensure the directory where you install it is on yourPATH.
After upgrading Substrate:
substrate bootstrap-management-accountsubstrate bootstrap-network-accountsubstrate bootstrap-deploy-accountsubstrate create-admin-account -quality quality for each of your admin accountssubstrate create-account -domain domain -environment environment -quality quality for each of your service accountsIf your shell supports process substitution, you can run sh <(substrate accounts -format shell) to run all of these, in the proper order, in one command. As of this release you can make this non-interactive as sh <(substrate accounts -auto-approve -format shell) but this is not recommended as it forgoes your opportunity to object before Terraform applies changes.
substrate bootstrap-* and substrate create-*account) now prevent Substrate downgrades by checking the tags on the accounts themselves.aws_subnet_ids everywhere except in the generated modules/peering-connection directory, which will take a little more time.substrate credentials to occasionally hang forever in 2022.05.After upgrading Substrate:
substrate bootstrap-management-accountsubstrate bootstrap-network-accountsubstrate bootstrap-deploy-accountsubstrate create-admin-account -quality quality for each of your admin accountssubstrate create-account -domain domain -environment environment -quality quality for each of your service accountsIf your shell supports process substitution, you can upgrade Terraform and then run sh <(substrate accounts -format shell) to run all of these, in the proper order, in one command.
InstanceFactory-arm64 or InstanceFactory-x86_64, if the one matching the requested instance type is defined. See customizing EC2 instances from the Instance Factory for details and an example.cloudtrail:DeleteTrail to the (short) list of APIs that are denied by the Substrate-managed service control policy on your management account.modules/ tree, instead letting all the version constraints come from the root module.organizations:DescribeOrganization to the IAM policy attached to the CredentialFactory IAM user so that it can orient itself fully.After upgrading Substrate:
substrate bootstrap-management-accountsubstrate create-admin-account -quality quality for each of your admin accountssubstrate bootstrap-management-account.--profile option to the standard AWS CLI. See using AWS CLI profiles for details.modules/substrate Terraform code.TooManyRequestsException from AWS Organizations during Terraform runs.SubstrateVersion tag on your AWS accounts themselves when Substrate tries to create them and finds that they already exist.root-modules/deploy to avoid a race during substrate bootstrap-deploy-account.After upgrading Substrate:
substrate bootstrap-management-accountsubstrate create-admin-account -quality quality for each of your admin accountsroot-modules/deploy.substrate.environments and substrate.qualities on the Intranet’s Accounts page and in the output of substrate accounts and substrate root-modules.After upgrading Substrate:
substrate bootstrap-deploy-accountsubstrate-create-admin-account -quality="..." for each of your admin accountssubstrate-create-admin-account and substrate-create-account can proceed smoothly when you create your 11th account.SubstrateVersion tag from Terraform-managed resources. It hasn’t been as helpful here as it is on Substrate-managed resources. Plus, Terraform plans are much easier to read without it.You must upgrade to Terraform 1.1.6 in order to use Substrate 2022.02. Terraform 1.1.6 may be found here:
After upgrading Substrate, do the following to land the Terraform upgrade and remove the SubstrateVersion tags:
substrate-bootstrap-network-accountsubstrate-bootstrap-deploy-accountsubstrate-create-admin-account -quality="..." for each of your admin accountssubstrate-create-account -domain="..." -environment="..." -quality="..." for each of your service accounts-role="..." option to substrate-assume-role now defaults to OrganizationAdministrator, Auditor, DeployAdministrator, or NetworkAdministrator for the special accounts and Administrator for admin and service accounts (or Auditor pretty much across the board, if you begin in the Auditor role). This should save a great deal of typing.After upgrading Substrate:
substrate-create-admin-account -quality="..."~/.aws/credentials under any circumstances. Since we never use this file in a Substrate-managed AWS organization, reading this file can only serve to “cross the streams” with a legacy AWS account.The upgrade process this month is much more involved that most. As such, we’ll talk in Slack about when you’re going to perform the upgrade to ensure support’s available in the moment.
Before upgrading Substrate, audit your Terraform modules for resources in global modules that aren’t from global AWS serivces by copying the following program to audit.sh in your Substrate repository and running sh audit.sh.
set -e
substrate root-modules |
grep "/global\$" |
while read DIRNAME
do
echo "$DIRNAME" >&2
terraform -chdir="$DIRNAME" state pull >"$DIRNAME/audit.tfstate"
grep -F ":us-east-1:" "$DIRNAME/audit.tfstate" || :
rm -f "$DIRNAME/audit.tfstate"
echo >&2
done
Every resource this program identifies needs to be modified before proceeding. The most likely modification is to add provider = aws.us-east-1 to resources in the Terraform code that manages them.
Block all your coworkers from making Terraform changes however you usually do (announcing in Slack, deactivating CI/CD jobs, taking state file locks, etc.) and move your global state files from us-east-1 to your default region by copying the following program to mv-state.sh in your Substrate repository and running sh mv-state.sh.
set -e
DEFAULT_REGION="$(cat "substrate.default-region")"
PREFIX="$(cat "substrate.prefix")"
if [ "$DEFAULT_REGION" = "us-east-1" ]
then exit # nothing to do
fi
eval $(substrate-assume-role -role="DeployAdministrator" -special="deploy")
substrate root-modules |
grep "/global\$" |
while read DIRNAME
do
echo "$DIRNAME" >&2
aws s3 cp "s3://$PREFIX-terraform-state-us-east-1/$DIRNAME/terraform.tfstate" "s3://$PREFIX-terraform-state-$DEFAULT_REGION/$DIRNAME/terraform.tfstate"
aws s3 rm "s3://$PREFIX-terraform-state-us-east-1/$DIRNAME/terraform.tfstate"
echo >&2
done
Once you’ve run this program, there’s a provider to thread through the tree of Terraform modules before you can upgrade to Substrate 2021.12.
Add the following four lines in the domain module stanzas in root-modules/*/*/*/global/main.tf:
providers = {
aws = aws
aws.us-east-1 = aws.us-east-1
}
Add the following three lines below aws = { in modules/*/global/versions.tf except modules/lambda-function/global/versions.tf, modules/substrate/global/versions.tf, and your own modules:
configuration_aliases = [
aws.us-east-1,
]
(I regret not being able to provide a patch(1) file for these operations but the contents of versions.tf post-Terraform 1.0 are too unpredictable to do so safely.)
Now you can upgrade Substrate. Don’t release your block just yet, though.
After upgrading Substrate:
substrate-bootstrap-deploy-accountsubstrate-create-admin-account -quality="..." for each of your admin accountssubstrate-create-account -domain="..." -environment="..." -quality="..." for each of your service accountsOnce all of these have run successfully, ensure all your coworkers upgrade Substrate and unblock Terraform changes.
I regret the complexity of this upgrade process but feel it is, on balance, less risky than attempting to hide all this motion behind automation. Thanks for your patience.
Cookie headers to HTTP(S) services wired into the Intranet by the experimental new modules/intranet/regional/proxy module. The theoretical security benefit of not exposing raw cookies (and instead exposing identity) is not remotely worth the loss in functionality it cost.substrate-intranet Lambda function.substrate-credentials opens in your browser are relative links and were missing the ../ prefix, which has now been added.archive provider.-no-cloudwatch from substrate-bootstrap-management-account, substrate-create-admin-account, and substrate-create-account in favor of just actually detecting when it’s necessary and not doing it when it’s not.Before upgrading Substrate, if you’re using Google as your IdP:
After upgrading Substrate:
substrate-create-admin-account -quality="..." to upgrade your Intranet./accounts page now logs you into the AWS Console and assumes the specified role without requiring you to have already been logged in.-console option to substrate-assume-role likewise now logs into the AWS Console and assumes the specified role without requiring you to have already been logged in.Administrator and Auditor roles may now be augmented by adding a JSON-encoded assume role policy in substrate.Administrator.assume-role-policy.json and/or substrate.Auditor.assume-role-policy.json.substrate-whoami output now also includes your IAM role ARN.cd or set SUBSTRATE_ROOT when they try to eval $(substrate-credentials) from outside the Substrate repository.modules/intranet/regional/proxy that makes it easy to put SSO in front of internal websites and HTTP APIs. See protecting internal websites for more information and an example.s3:PutObjectAcl so that it’s possible for all authorized principals to upload objects with the bucket-owner-full-control canned ACL.substrate-intranet.zip from the substrate binary during Terraform runs in root-modules/admin/*/* instead of only during substrate-create-admin-account. This makes it far less painful for mulitple teammates to work in the same Substrate repository and for CI/CD systems to apply Terraform changes.substrate-create-admin-account and substrate-create-account to fail every time they were used to actually create an account.-no-cloudwatch to substrate-bootstrap-management-account, substrate-create-admin-account, and substrate-create-account that skips the slow process of managing all the roles necessary for cross-account CloudWatch sharing. (Useful if you’re certain you’ve not created a new account and you’re in a hurry.)After upgrading Substrate:
substrate-bootstrap-deploy-account to fix the bucket policy so that all authorized principals in the organization can upload to the deploy artifact bucket(s).substrate-create-admin-account -quality="..." to upgrade your Intranet and Auditor roles. Note well this will produce a fair number of new resources; this is step one in a multi-month process of brining some naming consistency to Substrate-managed resources in IAM, Lambda, and other AWS services.substrate-assume-role form rather than their new substrate assume-role form) on MacOS.If you’re upgrading from 2021.08, follow the upgrade instructions from 2021.09. If you already upgraded to 2021.09, there are no further upgrade steps.
This release changes the interactive interface to substrate-bootstrap-network-account and substrate-create-admin-account to make them easier to run in CI. If you are automating these commands by providing yes and no answers on standard input, this release will break your automation; you should run these commands interactively first to see what’s changed. The details of what’s changed are listed in the usual format below.
substrate binary with symbolic links replacing the substrate-* binaries from previous releases. This can mostly be considered a no-op but note that now Substrate commands may be also be invoked as substrate subcommand. This is not a deprecation notice for the original invocation style.-fully-interactive, -minimally-interactive, and -non-interactive to all Substrate commands. -fully-interactive is almost identical (see below) to the behavior of 2021.08 and earlier releases. -minimally-interactive is the new default and removes the incessant “is this correct? (yes/no)” dialogs, which I thought would be welcome but turned out to be annoying. -non-interactive will never prompt for input and will instead exit with a non-zero status if input is required.substrate-create-admin-account by providing yes and no answers on standard input, this change will break your automation; you should run this command interactively first to see what’s changed.substrate-create-admin-account and substrate-create-account to prevent errant creation of new AWS accounts (which are tedious to delete in case creation was a mistake) plus a new -create option to suppress that confirmation.cloudtrail:CreateTrail API. Substrate creates a multi-region, organization-wide trail early in its initialization. This policy prevents additional trails from being created because they are excessively expensive and redundant.substrate.accounts.txt, substrate-accounts, and the Intranet’s /accounts page.-format=shell to substrate-accounts, which enumerates AWS accounts as shell commands to the various substrate-bootstrap-* and substrate-create-* commands. This is useful for driving CI/CD of Terraform changes. It’s also useful for automating Substrate upgrades.substrate-root-modules, which enumerates every Substrate-managed Terraform root module in a sensible order. This, too, is useful for driving CI/CD of Terraform changes.root-modules/deploy/global root module under the management of substrate-bootstrap-deploy-account. Substrate doesn’t manage any resources there but you’re free to.bucket-owner-full-control canned ACL.substrate-whoami is invoked with a / in the pathname (i.e. as ~/bin/substrate-whoami).After upgrading Substrate:
substrate-bootstrap-management-account to update your organization’s Service Control Policy.substrate-bootstrap-deploy-account to reconfigure the deploy buckets in S3 and generate the global root module.substrate-create-admin-account -quality="..." to add the e-mail address column to your Intranet’s /accounts page.substrate-apigateway-authorizer, substrate-credential-factory, and substrate-instance-factory into substrate-intranet. This is a no-op listed here for transparency. It’s a prerequisite step towards unifying all the Substrate tools as subcommands of substrate, thereby reducing the size and complexity of the Substrate distribution.archive and external Terraform providers by embedding substrate-intranet.zip directly in substrate-create-admin-account. Dependence on these providers will be made optional in a subsequent release.substrate-assume-role (which will have changed if invoked with SUBSTRATE_ROOT set) before forking and executing a child process.substrate-whoami to make it easy to learn the domain, environment, and quality of the AWS account your current credentials operate on.-format=json to substrate-accounts to make it easier to enumerate and act programatically on every AWS account in your organization. See enumerating all your AWS accounts for an example.After upgrading Substrate:
substrate-bootstrap-management-account to grant substrate-whoami the permissions it needs.substrate-bootstrap-network-account to remove coarse-grained organization-wide VPC sharing.substrate-create-admin-account -quality="..." to upgrade your Intranet./accounts page now opens the AWS Console in new browser tabs as it probably always should have.archive, aws, and external providers rather than all of versions.tf. This opens the door to Substrate users adding (and version constraining) additional providers. See additional Terraform providers for an example.aws provider >= 3.49.0.Manager and SubstrateVersion using the default_tags facility of the AWS provider. If you encounter the following error, remove Manager and SubstrateVersion (if present) from the indicated resources and re-run.Error: "tags" are identical to those in the "default_tags" configuration block of the provider: please de-duplicate and try again
SUBSTRATE_ROOT environment variable, if set, rather than always proceeding in whatever the working directory was when invoked.You must upgrade to Terraform 1.0.2 in order to use Substrate 2021.07. Terraform 1.0.2 may be found here:
After upgrading Terraform and Substrate:
substrate-bootstrap-network-account and substrate-bootstrap-deploy-account to complete the Terraform 1.0.2 upgrade there. Note well that tags and tags_all output will be somewhat confusing but will ultimately do the right thing.substrate-create-admin-account and substrate-create-account to complete the Terraform 1.0.2 upgrade for each of your admin and service accounts. Here, too, note well that tags and tags_all output will be somewhat confusing but will ultimately do the right thing.substrate-apigateway-authenticator and substrate-apigateway-index into substrate-intranet. This is a no-op listed here for transparency.>= version constraints on providers (but not Terraform itself).You must upgrade to Terraform 0.15.5 in order to use Substrate 2021.06. Terraform 0.15.5 may be found here:
After upgrading Terraform and Substrate:
substrate-bootstrap-network-account and substrate-bootstrap-deploy-account to complete the Terraform 0.15.5 upgrade there.substrate-create-admin-account -quality="..." to update your Intranet.substrate-create-account -domain="..." -environment="..." -quality="..." for all your service accounts to tag your shared VPCs.If you’ve added any stub provider blocks to your modules, leave them in place for now and accept the deprecation warning. Terraform only allows one required_providers block and that is now managed by Substrate. A future release will accommodate these additional providers.
After upgrading:
substrate-bootstrap-network-account to fix S3 routes.substrate-create-admin-account -quality="..." to enable direct CloudWatch access and make Instance Factory improvements./accounts to the Intranet with links to assume the Administrator and Auditor roles in all your accounts in the AWS Console.-console to substrate-assume-role which attempts to open the AWS Console’s role switching screen in your web browser with all the values filled in.substrate-create-terraform-module which creates the directory structure (with the global and regional pattern), providers, and Substrate metadata for a new Terraform module.After upgrading, run substrate-create-admin-account -quality="..." to add /accounts to your Intranet.
substrate-bootstrap-network-account now creates peering connections between all VPCs in all regions for each environment across all valid qualities.Administrator role in admin accounts that prevented Instance Factory instances from seamlessly assuming the role.You must upgrade to Terraform 0.14.7 in order to use Substrate 2021.03. Terraform 0.14.7 may be found here:
After upgrading:
rm -f -r root-modules/network/*/peering and remove these files from version control.substrate-bootstrap-network-account to peer all your VPCs that should be peered.substrate-create-admin-account -quality="..." to fix Instance Factory IAM roles, following the Google SAML setup guide if Google is your IdP to also get 12-hour AWS Console sessions.-format from substrate-credentials to substrate-assume-role per request from a customer. Now credentials can be had with or without the export prefix or as JSON a la aws sts assume-role itself.root-modules/admin/*’s awkward dependency on finding GOBIN in the environment. The generated Makefile in each root module remains, however.substrate-assume-role and substrate-credentials now (better) tolerate being invoked from subdirectories of your Substrate repository.aws.network provider was incorrectly added to global modules and thus should have been expected in module stanzas.export AWS_ACCESS_KEY_ID=... line when substrate-assume-role is given a command to execute directly.intranet/regional module to facilitate adding more resources to these APIs.You must upgrade to Terraform 0.13.6 in order to use Substrate 2021.02. Terraform 0.13.6 may be found here:
You must run substrate-create-admin-account for each of your admin accounts before you’ll be able to use eval $(substrate-credentials) to streamline your use of the Credential Factory.
You must run substrate-bootstrap-management-account in order to re-tag your former master account as your management account. (This rename follows AWS’ own renaming.)
Contact hello@src-bin.com for prior release notes.