Serverless multi-account deployments with CodePipeline & CodeBuild
Overview
This post will cover building a serverless based deployment pipeline on AWS with the following features:
- Cross-account deployments with CodePipeline and CodeBuild
- Approval actions
- CodeStar Connections (Github Version 2 connection)
- Pipeline monitoring (notifications)
- Integrating CodeArtifact
- Leveraging SSM
- Running automated tests
- Built with infrastructure as code (IaC)
Get the Code
Everything covered in this article is built with CloudFormation and available in this Github repo as a working demo.
High level Architecture of Pipeline
IAM cross-account roles
The key to cross account deployment is using IAM cross-account roles with CodeBuild, so that is a good place to start.
To provide account isolation, CodePipeline and CodeBuild will run in a separate CI account. From the CI account, the application will be deployed to multiple target accounts, i.e., dev, test, prod.
For this to work, the CodeBuild service will need access to an IAM role in each target account capable of creating the required resources. Since CodeBuild runs under an IAM role, each cross-account role will need to have the CodeBuild role as a trusted entity so that it can be assumed by CodeBuild. And the CodeBuild role, will need the ability to assume each cross-account role.
In CloudFormation it looks like this.
CodeBuild will dynamically assume a cross-account role for each of the target accounts in the buildspec.yml file. The details of buildspec will be discussed a bit further down.
Building out the Pipeline
CodePipeline will transition through multiple stages to complete the deployments in each account.
Source Stage
The initial stage of the pipeline is the Source stage. This stage will be responsible for cloning the source repository whenever new commits are pushed to the repo and storing the source code in an artifact that can be passed to each stage. This is an important note, as we only want the repo cloned one time and the source then passed to each build stage for a single pipeline lifecycle. This will be accomplished using OutputArtifacts and InputArtifacts.
Setting up the Source Connection
This CodePipeline setup uses the newer recommended GitHub Version 2 connection which uses a CodeStarSourceConnection to manage the integration between GitHub and CodePipeline. (GitHub Version 1 connection relied on a GitHub personal access token used as an OAuthToken.)
Adding the following CloudFormation config will create a new CodeStarSourceConnection that can be used by the CodePipeline Source stage.
The CodePipeline Source stage can then be configured to use the CodeStarConnection.
Note: From the CloudFormation docs on CodeStarConnections
A connection created through CloudFormation is in PENDING status by default. You can make its status AVAILABLE by updating the connection in the console.
Therefore, once the pipeline (and connection) has been created, a one-time step of going into to the CodePipeline console and editing the Source stage action is required.
Editing the source stage
From the CodePipeline edit Source stage action screen, Choose the “Connect to GitHub” button.
The Connect to GitHub dialog should open to allow the creation of the GitHub connection app.
After completing the Connect to GitHub process, there should be a green success box signifying that the connection is successful.
Lastly, in the GitHub account that was used, under Settings/Applications, the AWS Connector for GitHub should be visible.
Build Stages
In the CodePipeline config, each Build stage will need to be wired up to the corresponding CodeBuild project (for each account) using the ProjectName property under Configuration. As mentioned, the InputArtifacts property will be used to import the Source stage’s OutputArtifacts.
Approval Actions
Rarely should pipelines automate the deployment to every account on each code push to the repo, that could become very disruptive. There are likely users testing in one or more of the accounts, so having the ability to control when a specific version is released to an account will be important.
CodePipeline approval actions provide a gating mechanism for a manual, one-click, approval to allow or reject the pipeline to proceed to the next stage.
This example also uses the optional Configuration setting property to send approval events to an SNS topic (this will be used later). Also provided, is the option for a custom link that will show up in the approval dialog.
Refer to the CodePipeline CloudFormation docs for details on all the available properties.
Setting Up CodeBuild Projects
CodeBuild is where the work happens. It is in the CodeBuild build projects that the application gets packaged and deployed to a target account.
Some of the key things to note in the CodeBuild project definition is the ServiceRole, which is the role that CodeBuild runs under, and the EnvironmentalVariables that will be available for use in the buildspec.yml file at build time.
Reference the CodeBuild CloudFormation docs for details on all the available properties.
Setting up the buildspec
At build time, CodeBuild will need access to the ARN of the cross-account IAM role for each target account, so that it can assume that role and perform the deployment. This action will take place in the buildspec.yml file, which generally lives in the root of the project and is read in and executed by CodeBuild.
To make the cross-account role ARN accessible to the buildspec, the ARN’s for each target account need to be stored somewhere.
There are a couple of places to do this:
- CodeBuild environment variable
- Systems Manager Parameter Store (SSM)
Storing the cross-account role in SSM
SSM params that store each cross account role ARN will need to be available in the CI account for the pipeline to access. By embedding the STAGE name in the SSM param name (e.g, /myapp/dev/deploy_role), we can use the $STAGE CodeBuild environmental variable (assigned to each project) as a way to dynamically identify the right role for the target account.
We can then use the aws ssm get-parameter cli command to get the role in the buildspec pre_build phase. Notice the usage of the environment variable $STAGE in the SSM param name.
Note: On line 4, the installation of the jq library and its usage on line 6 as a clean way to pull things out of the JSON returned from the get-parameter call.
Storing the Cross-account role as a CodeBuild environment variable
For this to work, a CodeBuild environment variable will need to be created in each project and its value set to the ARN of the cross-account role for that target account. The variable will be accessible in the buildspec by using the $ notation, e.g., $DEPLOY_ROLE.
Assuming the cross-account role and deploying the application
Now, as the pipeline moves through each build stage, the ARN of the cross-account role will be accessed (either via SSM or CodeBuild environment variable) and passed to the aws sts assume-role command. That API call will return a set of temporary credentials which can now be used to deploy resources in each of the target accounts.
This example uses the Serverless Framework to deploy service named app.
Note: Once the role is assumed, the key, secret and token of the assumed role (returned in the JSON response) will need to be set in the CodeBuild container environment before it can be used properly. Notice again the usage of jq to pull the values out of the returned JSON. Also note that we are running sls deploy from the local node_modules/.bin. This allows the managing of Serverless versions to be maintained in package.json devDependencies and not in the buildspec.
Reference the buildspec docs for details on all the available settings.
Pipeline Monitoring
Now that the app can deployed to each target account, getting notifications on build success and failure, as well as when an approval is pending will be desired.
A sane way for managing pipeline notifications is to build a small Serverless monitoring app in the CI account.
CodePipeline Approval Notifications
In each approval stage of CodePipeline, the NotificationArn property has been set to an SNS topic. A lambda subscriber is all that is needed to get those events. The lambda can be used to format and forward a message to a Slack webhook.
CodeBuild Status Notifications
As CodeBuild moves through its lifecycle, it emits events to CloudWatch. Using a lambda to listen for the SUCCEEDED and FAILED events will provide the ability to send success and fail notifications to Slack.
Check out the monitor service in the GitHub repo for specific details on monitoring and sending formatted events to Slack.
AWS CodeArtifact
Although not required, this working example has integrated AWS CodeArtifact as its artifact repository. Recently, CodeArtifact has added CloudFormation support making for a clean IaC implementation.
Created here are two CodeArtifact repositories under a single CodeArtifact Domain. The artifacts-repo repository is wired up to the upstream-arifacts-repo, which is in turn, wired up to the public npm registry.
To allow CodeBuild to use the CodeArtifact repository requires adding a single line command to the pre_build phase of the buildspec.yml file to authenticate with CodeArtifact.
By default the login token is valid for 12 hours, but that can be controlled by passing the — duration-seconds flag.
See CodeArtifact authentication docs for more details.
Running Automated Tests
An easy way to set up automated testing with a Node based serverless application is with an npm script.
This example uses the mocha testing framework to run all test files located in the test directory that end with -test.js.
To run this automated with the pipeline, npm run test can be executed in any of the phases within the buildspec.yml.
A nice way to control running tests, per-account, is to use an environment variable as the command:
For the accounts where tests should be run, the value of the CodeBuild environment variable TEST_CONDITION is set to npm run test. And setting the value to echo not running tests for this account, for the accounts where tests should not run.
Finished Pipeline
Conclusion
There is definitely a bit to be unpacked here, especially if some of these tools are unfamiliar. Check out the GitHub repo to help fill in the areas I may not have covered in detail.
Have fun, go build!
