AWS Application Services

Tutorial to build a NodeJs Serverless application using the AWS Serverless Application Model.

Tutorial Objectives:

1. Learn to build Serverless application using AWS SAM

2. Learn to fully automate builds and deployments by building a CI/CD pipeline using the AWS CDK.

3. Learn to run a Serverless application locally using the SAM CLI.

Prerequisites:

1. Download and extract the necessary code zip file from here.

Step 1: CREATE A CLOUD9 WORKSPACE

Navigate to the Cloud9 console: https://console.aws.amazon.com/cloud9 Choose Create environment.

Once you navigate to the Cloud9 console, click on the create environment button:

Chose a name for your environment.

Leave the default configuration, we don’t need a heavy server for this workshop.

Once you confirm creation, after a couple of minutes, your environment should look like the following image:

(Optional) If you prefer a dark theme, choose the theme in the top menu: View > Themes > Cloud9 > Cloud 9 Night.

Congratulations you have successfully created your Cloud9 Environment.

Step 2: UPGRADE SAM CLI

We have put together a bootstrap script that will make the upgrade easier for you. Download it by running the following command from your Cloud9 terminal. Download the bootstrap.sh file provided in the prerequisite. Upload the bootstrap.sh file to your cloud9 environment and run the following commands.

chmod +x bootstrap.sh
./bootstrap.sh

THIS MAY TAKE A FEW MINUTES TO COMPLETE.

Example Output:

To verify the new version, Run the following command:

sam --version 

You should see SAM CLI, version 0.43.0 or greater.

Step 3: Create a new SAM Application

To initialize a new SAM Application Project, run the following command.

sam init

It will prompt for project configuration parameters:

Type 1 to select AWS Quick Start Templates.

Type 1 to select Zip as the package type.

Type 1 to select the NodeJs14.x runtime.

Leave sam-app for the project name

Type 1 to select the Hello World Example.

Project should now be initialized. You should see a new folder sam-app created with a basic Hello World scaffolding.

Step 4: RUN PROJECT LOCALLY

Install Dependencies

Before we run the application locally, it’s a common practice to install third-party libraries or dependencies that your application might be using.

cd sam-app/hello-world

And install the dependencies:

npm install

Run Using SAM-CLI In the terminal, run the following command from the root directory of the sam-app folder:

cd ~/environment/sam-app
sam local start-api --port 8080

To test our endpoint, we have 2 options

Option A) : Using CURL:

Now, we’re going to test our endpoint using curl.

First, without killing the running process, open a new terminal

Test your endpoint by running a CURL command that triggers an HTTP GET request.

curl http://localhost:8080/hello

The output should look like this

Option B) : Using a browser window

From the Cloud9 Menu Bar, Select Tools> Preview > Preview Running Application.

As you can see, it responded with the message “Hello World”.

Now, While the app is still running, open the file sam-app/hello-world/app.js and change the code to make the application say “hello my friend”

Save the file and refresh your Cloud9 browser tab.

Let’s run a Unit Test on our application, Navigate to the sam-app/hello-world folder and run the test command

cd ~/environment/sam-app/hello-world
npm run test

Don’t worry! This test is supposed to fail

Let’s fix our unit test now.

Locate the file sam-app/hello-world/tests/unit/test-handler.js and change the expected value from “hello world” to “hello my friend”

Your file should look like this:

Run the test again

npm run test

This test should pass.

Step 5: Manually Deploy to AWS

Building the app:

To build our SAM project we’re going to use sam build command. Navigate to sam-app folder and run the following command.

cd ~/environment/sam-app
sam build

The Build should be completed without any errors. Also, make sure you’ve made hidden files visible.

Deploy the application Now, we’re going to deploy our application manually on AWS with the help of sam deploy command

sam deploy --guided

This command is likely to ask for some input parameters. Say “Y” to everything and keep everything else as default

Confirm your deployment by entering “Y” again

Once the deployment is complete it should look like this,

Step 6: Create a pipeline

Creating a git repository First, we’re going to create a remote git repository using AWS CodeCommit. Run the following command to create the git repository.

aws codecommit create-repository --repository-name sam-app

You’ll see the following output. Copy the value of cloneUrlHttp, you will need it later.

Configure the credentials for CodeCommit with following command. Don’t forget to replace “Replace with your name” and replace_with_your_email@example.com with your own name and email

git config --global user.name "Replace with your name"
git config --global user.email "replace_with_your_email@example.com"

Locate your sam-app/.gitignore file (Make sure you have enabled to show hidden file as mentioned at the start of Step 5). And paste the following snippet there.

.aws-sam/
packaged.yaml

Your gitignore file should look like this

Navigate to your root directory of sam-app and run following commands to commit your changes.

cd ~/environment/sam-app
git init
git add .
git commit -m "Initial commit"

Push the code

Push your code to your remote repository with the following command

git remote add origin codecommit://sam-app
git push -u origin master

Verify your commit in CodeCommit

From your AWS Management console, Navigate to AWS CodeCommit. And verify the changes are actually reflected there.

Setting up a CDK Project

First, we’re going to initialize our project with the following command within our sam-app directory where the pipeline code with reside.

cd ~/environment/sam-app
mkdir pipeline
cd pipeline

Initialize a new CDK project within the pipeline folder by running the following command:

cdk init --language typescript

Now install the CDK modules that we will be using to build a pipeline:

npm install --save @aws-cdk/aws-codedeploy @aws-cdk/aws-codebuild
npm install --save @aws-cdk/aws-codecommit @aws-cdk/aws-codepipeline-actions
npm install --save @aws-cdk/aws-s3

After a few seconds, the pipeline folder will be created in our directory.

Run this command to resize your Cloud9 volume

wget https://cicd.serverlessworkshops.io/assets/resize.sh
chmod +x resize.sh
./resize.sh 20

Modify stack name

In your sam-app/pipeline/bin/pipeline.ts file change the project name to sam-app-cicd and make sure you save the file.

Building the CDK project

Even though we haven’t wrote any code yet, let’s get familiar with how to build and deploy a CDK project, as you will be doing it multiple times in this workshop and you should get comfortable with the process. Start by building the project with the following command:

cd ~/environment/sam-app/pipeline
npm run build

Now, we’re going to deploy our project.

cdk deploy

Our output should look like this,

Optionally, you can open the CloudFormation console and check if your CloudFormation stack has been created.

Artifacts Bucket

Every Code Pipeline needs an artifacts bucket, also known as Artifact Store. CodePipeline will use this bucket to pass artifacts to the downstream jobs and its also where SAM will upload the artifacts during the build process.

Make sure you are editing the pipeline-stack file with .ts extension This file is located at sam-app/pipeline/lib/pipeline-stack.ts

Paste the following code in the pipeline-stack.ts file

// lib/pipeline-stack.ts

import * as cdk from '@aws-cdk/core';
import s3 = require('@aws-cdk/aws-s3');
import codecommit = require('@aws-cdk/aws-codecommit');
import codepipeline = require('@aws-cdk/aws-codepipeline');
import codepipeline_actions = require('@aws-cdk/aws-codepipeline-actions');
import codebuild = require('@aws-cdk/aws-codebuild');

export class PipelineStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // The code that defines your stack goes here
    const artifactsBucket = new s3.Bucket(this, "ArtifactsBucket");
  }
}

Now, again build and deploy the application.

cd ~/environment/sam-app/pipeline
npm run build
cdk deploy

The output should look like this

Note: If you get an error during the build process, make sure all @aws-cdk dependencies in the package.json file have the same version number, if not, fix it (by giving same version number to every @aws-cdk dependency), delete the node_modules folder and run npm install. More info: Visit this GitHub thread

Now, Open sam-app/pipeline/lib/pipeline-stack.ts file. And paste the following snippet after your bucket definition

// Import existing CodeCommit sam-app repository
const codeRepo = codecommit.Repository.fromRepositoryName(
  this,
  'AppRepository', // Logical name within CloudFormation
  'sam-app' // Repository name
);
// Pipeline creation starts
const pipeline = new codepipeline.Pipeline(this, 'Pipeline', {
  artifactBucket: artifactsBucket
});
// Declare source code as an artifact
const sourceOutput = new codepipeline.Artifact();
// Add source stage to pipeline
pipeline.addStage({
  stageName: 'Source',
  actions: [
    new codepipeline_actions.CodeCommitSourceAction({
      actionName: 'CodeCommit_Source',
      repository: codeRepo,
      output: sourceOutput,
    }),
  ],
});

Please refer the pipeline-stack-1.txt file from the extracted zip package to confirm no errors are made.

Adding the build stage We’re going to add a build stage for our project. For this we’ll be using AWS CodeBuild service.

// Declare build output as artifacts
const buildOutput = new codepipeline.Artifact();
// Declare a new CodeBuild project
const buildProject = new codebuild.PipelineProject(this, 'Build', {
  environment: { buildImage: codebuild.LinuxBuildImage.AMAZON_LINUX_2_2 },
  environmentVariables: {
    'PACKAGE_BUCKET': {
      value: artifactsBucket.bucketName
    }
  }
});
// Add the build stage to our pipeline
pipeline.addStage({
  stageName: 'Build',
  actions: [
    new codepipeline_actions.CodeBuildAction({
      actionName: 'Build',
      project: buildProject,
      input: sourceOutput,
      outputs: [buildOutput],
    }),
  ],
});

Please refer the pipeline-stack-2.txt file from the extracted zip package to confirm no errors are made. Note: If you get any errors at this stage, Go to sam-app/pipeline/package.json and make sure all sure all @aws-cdk dependencies in the package.json file have the same version number, if not, fix it (by giving same version number to every @aws-cdk/core dependency), After that run the command

npm install –-save @aws-cdk/core

More info: Visit this GitHub thread e.g: Once during the execution. We faced this error.

Now navigate to sam-app/pipeline/package.json and see if the version numbers of all the @aws-cdk dependencies are same or not.

As you can see in the above picture, the version of @aws-cdk/core is 1.135.0 whereas other dependencies are running at 1.137.0 In this case we’ll change the version of @aws-cdk/core here and will run the aforementioned npm install --save @aws-cdk/core command.

Now, again build and deploy application.

cd ~/environment/sam-app/pipeline
npm run build
cdk deploy

Navigate to AWS CodePipeline and click on your newly created pipeline.

The build step must have failed. But don’t worry this is expected. We haven’t specified the commands to run during the build yet.

Buildspec file:

To provide commands during the build process we need create and code a buildspec file.

Right-click on your sam-app folder and click on new file.

Name your file buildspec.yml This file will contain the build commands we need to provide during the build phase.

Now, Copy and paste the code from the next page into your buildspec.yml file

Take a not that the yml files are based on indentations, so an error in the indent could result in an error. Hence we suggest to carefully paste the code without any errors/changes.

# ~/environment/sam-app/buildspec.yml

version: 0.2
phases:
  install:
    runtime-versions:
      nodejs: 12
    commands:
      - pip3 install --upgrade aws-sam-cli
      - sam --version
      - cd hello-world
      - npm install
  
  pre_build:
    commands:
      - npm run test

  build:
    commands:
      - cd ..
      - sam build

  post_build:
    commands:
      - sam package --s3-bucket $PACKAGE_BUCKET --output-template-file packaged.yaml

artifacts:
  discard-paths: yes
  files:
    - packaged.yaml

Save the file. It should look like this.

Push the changes to your repository and this time the CodePipeline should automatically get to build and deploy phase.

cd ~/environment/sam-app
git add .
git commit -m "Added buildspec.yml"
git push

Go to CodeBuild and see if your build has succeeded or not. This time, it should

Deploy stage

The Deploy Stage is where your SAM application and all its resources are created an in an AWS account. The most common way to do this is by using CloudFormation ChangeSets to deploy. This means that this stage will have 2 actions: the CreateChangeSet and the ExecuteChangeSet.

Add the code from the next page to your pipeline-stack.ts file

// Deploy stage
pipeline.addStage({
  stageName: 'Dev',
  actions: [
    new codepipeline_actions.CloudFormationCreateReplaceChangeSetAction({
      actionName: 'CreateChangeSet',
      templatePath: buildOutput.atPath("packaged.yaml"),
      stackName: 'sam-app',
      adminPermissions: true,
      changeSetName: 'sam-app-dev-changeset',
      runOrder: 1
    }),
    new codepipeline_actions.CloudFormationExecuteChangeSetAction({
      actionName: 'Deploy',
      stackName: 'sam-app',
      changeSetName: 'sam-app-dev-changeset',
      runOrder: 2
    }),
  ],
});

Please refer the pipeline-stack-3.txt file from the extracted zip package to confirm no errors are made.

Run the following commands within your pipeline directory.

cd ~/environment/sam-app/pipeline
npm run build
cdk deploy

Triggering the release

Navigate to your pipeline and you will see the Deploy stage has been added, however, it is currently grayed out because it hasn’t been triggered. Let’s just trigger a new run of the pipeline manually by clicking the Release Change button from your CodePipeline.

Once the Dev looks like gets succeeded push the changes to your CodeCommit repository

git add .
git commit -m "CI/CD Pipeline definition"
git push

Congratulations! You have created a CI/CD pipeline for a Serverless application!

Step 7: Canary Deployments

A Canary Deployment is a technique that reduces the risk of deploying a new version of an application by slowly rolling out the changes to a small subset of users before rolling it out to the entire customer base.

In this step you will learn how to implement gradual deployments with AWS SAM, AWS CloudFormation and AWS CodeDeploy with just a few lines of configuration.

Update the SAM Template:

Open sam-app/template.yaml and add the following lines under HelloWorldFunction properties section.

AutoPublishAlias: live
DeploymentPreference:
    Type: Canary10Percent5Minutes

Please check the indentation as it is very important in YAML file

Validate the SAM template

Run the following command on your terminal:

cd ~/environment/sam-app
sam validate

If the template is correct, you will see template.yaml is a valid SAM Template. If you see an error, then you likely have an indentation issue on the YAML file. Double check and make sure it matches the screenshot shown above.

Finally, In the terminal, run the following commands from the root directory of your sam-app project.

cd ~/environment/sam-app
git add .
git commit -m "Canary deployments with SAM"
git push

Define a Cloudwatch alarm

Add the following lines to the DeploymentPreference section of the HelloWorldFunction definition.

Alarms:
    - !Ref CanaryErrorsAlarm

And then add the following alarm definition to the template.yaml file in the Resources section after the HelloWorldFunction definition.

CanaryErrorsAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmDescription: Lambda function canary errors
    ComparisonOperator: GreaterThanThreshold
    EvaluationPeriods: 2
    MetricName: Errors
    Namespace: AWS/Lambda
    Period: 60
    Statistic: Sum
    Threshold: 0
    Dimensions:
      - Name: Resource
        Value: !Sub "${HelloWorldFunction}:live"
      - Name: FunctionName
        Value: !Ref HelloWorldFunction
      - Name: ExecutedVersion
        Value: !GetAtt HelloWorldFunction.Version.Version

Please refer the template-yaml.txt file from the extracted zip package to confirm no errors are made especially in the indentations.

Validate the SAM template Run the following command on your terminal:

cd ~/environment/sam-app
sam validate

The output should look like this

If you see any error, that’s probably because of the indentation and we recommend to copy the whole file from template-yaml.txt which is in the zip package you downloaded from Prerequisites.

Finally, In the terminal, run the following commands from the root directory of your sam-app project.

cd ~/environment/sam-app
git add .
git commit -m "Canary deployments with SAM"
git push

Verify in CodeDeploy

Wait for your pipeline to get to the deployment stage (ExecuteChangeSet) and when you see it In Progress. Navigate to the CodeDeploy console to watch the deployment progress.

Navigate to the AWS CodeDeploy console and after a couple of minutes, you should see a new deployment in progress. Click on the Deployment to see the details.

The deployment status shows that 10% of the traffic has been shifted to the new version (aka The Canary). CodeDeploy will hold the remaining percentage until the specified time interval has ellapsed, in this case we specified the interval to be 5 minutes.

Shortly after the 5 minutes, the remaining traffic should be shifted to the new version:

Rollbacks Introduce an error on purpose: Lets break the Lambda function on purpose so that the CanaryErrorsAlarm gets triggered during deployment. Update the lambda code in sam-app/hello-world/app.js to throw an error on every invocation, like this:

let response;
exports.lambdaHandler = async (event, context) => {
    throw new Error("This will cause a deployment rollback");
    // try {
    //     response = {
    //         'statusCode': 200,
    //         'body': JSON.stringify({
    //             message: 'hello my friend with canaries',
    //         })
    //     }
    // } catch (err) {
    //     console.log(err);
    //     return err;
    // }
    // return response
};

Make sure to update the unit test, otherwise the build will fail. Comment out every line in the sam-app/hello-world/tests/unit/test-handler.js file:

// 'use strict';

// const app = require('../../app.js');
// const chai = require('chai');
// const expect = chai.expect;
// var event, context;

// describe('Tests index', function () {
//     it('verifies successful response', async () => {
//         const result = await app.lambdaHandler(event, context)

//         expect(result).to.be.an('object');
//         expect(result.statusCode).to.equal(200);
//         expect(result.body).to.be.an('string');

//         let response = JSON.parse(result.body);

//         expect(response).to.be.an('object');
//         expect(response.message).to.be.equal("hello my friend with canaries");
//     });
// });

Push the changes

cd ~/environment/sam-app
git add .
git commit -m "Breaking the lambda function on purpose"
git push

Wait for the deployment to start

While the deployment is running, you need to generate traffic to the new Lambda function to make it fail and trigger the CloudWatch Alarm. In a real production environment, your users will likely generate organic traffic to the canary function, so you may not need to do this.

In your terminal, run the following command to invoke the Lambda function:

sudo yum install -y jq
aws lambda invoke --function-name \
$(aws lambda list-functions | jq -r -c '.Functions[] | select( .FunctionName | contains("sam-app-HelloWorldFunction")).FunctionName'):live \
--payload '{}' \
response.json

This will be the output

There will be a new file response.json created. It contains the response of the lambda invocation. If you open it, you may see the the response of the old Lambda version, or you may see the new one that causes an error.

Remember: During deployment, only 10% of the traffic will be routed to the new version. So, keep on invoking your lambda many times. 1 out of 10 invocations should trigger the new broken lambda, which is what you want to cause a rollback.

Here is a command that invokes your function 15 times in a loop. Feel free to run it in your terminal.

counter=1
while [ $counter -le 15 ]
do
    aws lambda invoke --function-name \
    $(aws lambda list-functions | jq -r -c '.Functions[] | select( .FunctionName | contains("sam-app-HelloWorldFunction")).FunctionName'):live \
    --payload '{}' \
    response.json
    sleep 1
    ((counter++))
done

Since 1/10 invokes go to the broken lambda, out of 10 you’d see one response with a Unhandled exception

Go to your CodePipeline and confirm that the Deployment actually failed.

Step 8: Clean up.

Empty and delete both the buckets created by SAM CLI and CodePipeline.

Now that the buckets are empty, we can delete the Cloudformation stacks:

Delete the CF stacks in following sequence and please wait for each to complete before going to next one.

Delete sam-app

aws cloudformation delete-stack --stack-name sam-app

Delete sam-app-cicd

aws cloudformation delete-stack --stack-name sam-app-cicd

Delete aws-sam-cli-managed-default

aws cloudformation delete-stack --stack-name aws-sam-cli-managed-default

You can also delete the CF Stacks by navigation to the Cloud Formation management console and deleting the stacks.

Finally Delete the cloud 9 environment.