Source & Binary

Substrate release notes

2021.12

The upgrade process this month is much more involved that most. As such, we’ll talk in Slack about when you’re going to perform the upgrade to ensure support’s available in the moment.

Before upgrading Substrate, audit your Terraform modules for resources in global modules that aren’t from global AWS serivces by copying the following program to audit.sh in your Substrate repository and running sh audit.sh.

set -e

substrate root-modules |
grep "/global\$" |
while read DIRNAME
do
    echo "$DIRNAME" >&2
    terraform -chdir="$DIRNAME" state pull >"$DIRNAME/audit.tfstate"
    grep -F ":us-east-1:" "$DIRNAME/audit.tfstate" || :
    rm -f "$DIRNAME/audit.tfstate"
    echo >&2
done

Every resource this program identifies needs to be modified before proceeding. The most likely modification is to add provider = aws.us-east-1 to resources in the Terraform code that manages them.

Block all your coworkers from making Terraform changes however you usually do (announcing in Slack, deactivating CI/CD jobs, taking state file locks, etc.) and move your global state files from us-east-1 to your default region by copying the following program to mv-state.sh in your Substrate repository and running sh mv-state.sh.

set -e

DEFAULT_REGION="$(cat "substrate.default-region")"
PREFIX="$(cat "substrate.prefix")"

if [ "$DEFAULT_REGION" = "us-east-1" ]
then exit # nothing to do
fi

eval $(substrate-assume-role -role="DeployAdministrator" -special="deploy")

substrate root-modules |
grep "/global\$" |
while read DIRNAME
do
    echo "$DIRNAME" >&2
    aws s3 cp "s3://$PREFIX-terraform-state-us-east-1/$DIRNAME/terraform.tfstate" "s3://$PREFIX-terraform-state-$DEFAULT_REGION/$DIRNAME/terraform.tfstate"
    aws s3 rm "s3://$PREFIX-terraform-state-us-east-1/$DIRNAME/terraform.tfstate"
    echo >&2
done

Once you’ve run this program, there’s a provider to thread through the tree of Terraform modules before you can upgrade to Substrate 2021.12.

(I regret not being able to provide a patch(1) file for these operations but the contents of versions.tf post-Terraform 1.0 are too unpredictable to do so safely.)

Now you can upgrade Substrate. Don’t release your block just yet, though.

After upgrading Substrate:

  1. substrate-bootstrap-deploy-account
  2. substrate-create-admin-account -quality="..." for each of your admin accounts
  3. substrate-create-account -domain="..." -environment="..." -quality="..." for each of your service accounts

Once all of these have run successfully, ensure all your coworkers upgrade Substrate and unblock Terraform changes.

I regret the complexity of this upgrade process but feel it is, on balance, less risky than attempting to hide all this motion behind automation. Thanks for your patience.

2021.11

Before upgrading Substrate, if you’re using Google as your IdP:

  1. Add an additional custom attribute as follows:
    1. Visit https://admin.google.com/ac/customschema in a browser (or visit https://admin.google.com, click Users, click More, and click Manage custom attributes)
    2. Click the AWS section
    3. In the blank bottom row, enter “RoleName” for Name, select “Text” for Info type, select “Visible to user and admin” for Visibility, select “Single Value” for No. of values
    4. Click SAVE
  2. Visit https://admin.google.com/ac/users and set the RoleName attribute in the AWS category to “Administrator” for every user authorized to use AWS.
  3. Visit https://console.cloud.google.com/apis/library/admin.googleapis.com, confirm the selected project is the one that contains your Intranet’s OAuth OIDC configuration (its name will be listed next to "Google Cloud Platform" in the header), and click ENABLE.

After upgrading Substrate:

  1. Run substrate-create-admin-account -quality="..." to upgrade your Intranet.

2021.10

After upgrading Substrate:

  1. Run substrate-bootstrap-deploy-account to fix the bucket policy so that all authorized principals in the organization can upload to the deploy artifact bucket(s).
  2. Run substrate-create-admin-account -quality="..." to upgrade your Intranet and Auditor roles. Note well this will produce a fair number of new resources; this is step one in a multi-month process of brining some naming consistency to Substrate-managed resources in IAM, Lambda, and other AWS services.

2021.09.28

If you’re upgrading from 2021.08, follow the upgrade instructions from 2021.09. If you already upgraded to 2021.09, there are no further upgrade steps.

2021.09

This release changes the interactive interface to substrate-bootstrap-network-account and substrate-create-admin-account to make them easier to run in CI. If you are automating these commands by providing yes and no answers on standard input, this release will break your automation; you should run these commands interactively first to see what’s changed. The details of what’s changed are listed in the usual format below.

After upgrading Substrate:

  1. Run substrate-bootstrap-management-account to update your organization’s Service Control Policy.
  2. Run substrate-bootstrap-deploy-account to reconfigure the deploy buckets in S3 and generate the global root module.
  3. Run substrate-create-admin-account -quality="..." to add the e-mail address column to your Intranet’s /accounts page.

2021.08

After upgrading Substrate:

  1. Run substrate-bootstrap-management-account to grant substrate-whoami the permissions it needs.
  2. Run substrate-bootstrap-network-account to remove coarse-grained organization-wide VPC sharing.
  3. Run substrate-create-admin-account -quality="..." to upgrade your Intranet.

2021.07

You must upgrade to Terraform 1.0.2 in order to use Substrate 2021.07. Terraform 1.0.2 may be found here:

After upgrading Terraform and Substrate:

  1. Run substrate-bootstrap-network-account and substrate-bootstrap-deploy-account to complete the Terraform 1.0.2 upgrade there. Note well that tags and tags_all output will be somewhat confusing but will ultimately do the right thing.
  2. Run substrate-create-admin-account and substrate-create-account to complete the Terraform 1.0.2 upgrade for each of your admin and service accounts. Here, too, note well that tags and tags_all output will be somewhat confusing but will ultimately do the right thing.

2021.06

You must upgrade to Terraform 0.15.5 in order to use Substrate 2021.06. Terraform 0.15.5 may be found here:

After upgrading Terraform and Substrate:

  1. Run substrate-bootstrap-network-account and substrate-bootstrap-deploy-account to complete the Terraform 0.15.5 upgrade there.
  2. Run substrate-create-admin-account -quality="..." to update your Intranet.
  3. Run substrate-create-account -domain="..." -environment="..." -quality="..." for all your service accounts to tag your shared VPCs.

If you’ve added any stub provider blocks to your modules, leave them in place for now and accept the deprecation warning. Terraform only allows one required_providers block and that is now managed by Substrate. A future release will accommodate these additional providers.

2021.05

After upgrading:

2021.04

After upgrading, run substrate-create-admin-account -quality="..." to add /accounts to your Intranet.

2021.03

You must upgrade to Terraform 0.14.7 in order to use Substrate 2021.03. Terraform 0.14.7 may be found here:

After upgrading:

  1. rm -f -r root-modules/network/*/peering and remove these files from version control.
  2. substrate-bootstrap-network-account to peer all your VPCs that should be peered.
  3. substrate-create-admin-account -quality="..." to fix Instance Factory IAM roles, following the Google SAML setup guide if Google is your IdP to also get 12-hour AWS Console sessions.

2021.02

You must upgrade to Terraform 0.13.6 in order to use Substrate 2021.02. Terraform 0.13.6 may be found here:

2021.01 and prior releases

Contact hello@src-bin.com for prior release notes.