Welcome to the home for CI CD Lab Series using AWS Developer Tools!
This blog covers an overview of AWS CI CD Services, 3 use cases leveraging AWS CI CD Services, Template files used in CodeBuild, CodeDeploy and Serverless Application Deployment.
How do you debug you build or deployment failures? And how do you rollback in case of failure.
Details around Generic Pipeline and the metrics to gauge DevOps
When you wanted to write about AWS Developer Tools / Code Services many use cases around flashed and puzzled what to write, where to start & when to stop.
Since it sounded little difficult to put the vast knowledge into a frame, decided to coherently sketch them as a blog with hands-on lab so that Cloud enthusiasts can extract the essence of it with the fresh experience.
Introduction
Software Release Phases vs AWS CI CD Services
What-so-ever may be the software development methodology (Agile or Feature Driven Development) the company follows for their software development, the ultimate goal is take the code changes from source to production faster than ever which ideally encompasses 4 phases – Source, Build, Test & Deploy. This forms software release phases in general.
Let’s see where does the AWS CI CD Services fits in the above software release phases.
If you are wondering why am I using the term AWS CI CD Services instead of AWS Developer Tools, its just because in this session, we will be talking only about AWS CodeCommit, AWS CodeBuild, AWS CodeDeploy & AWS CodePipeline but not AWS Code-Star, AWS Cloud-9 & AWS X-ray.
- Source phase is where the complete Code Management happens comprising check-in source code, peer review, version control and the AWS CodeCommit is all around that.
- In Build phase, you will compile the code, run unit test or create container images and the AWS CodeBuild is designed for that.
- In Test phase, you will perform the Integration, UI, Load, Stress Test by integrating with various third-party tools.
- In Deploy phase, the package generated in the build phase will be deployed to the target environment, be it Dev, QA, PreProd or Prod.
- AWS CodeDeploy service is here to make your life simple where deploying to thousands of servers is just as easy as deploying to 1.
- AWS CodePipeline is an end-to-end solution, which comes handy to glue all these services. CodePipeline enables you to rapidly deliver features and updates with high quality through the automation of your build, test, and release process.
AWS CI CD Services - Bird's Eye View
This Lab Series is not AWS Developer Tools 101 course and we will be talking more about how AWS developer tools can be leveraged to create a powerful and scalable developer environment that can help you to get the code changes from the source to production faster than ever.
1. AWS CodeCommit
- AWS CodeCommit provides a fully-managed source control service, this is where your code is stored and managed securely.
- It's powered using Git, so if you're familiar with Git already, well, you already know 90% of CodeCommit.
- The key feature of CodeCommit is that it's backed by AWS technology. which means your code repositories can easily scale up and down based on their size.
- In CodeCommit both data in-transit & at-rest are encrypted using AWS-managed keys. You cannot use a customer-managed key in AWS KMS to encrypt or decrypt data in AWS CodeCommit repositories.
- There is no size limit on the repos and also no restrictions on the number of repos that can be created.
- You can have Post Commit hooks to call out to SNS / Lambda Service.
- All that you do with any traditional source control management software like create a commit, view a commit, compare two commits, create / delete a tag, pull request for code reviews are all very much possible in AWS CodeCommit
- I really like the DAG (directed acyclic graph) based commit visualizer to view the commits made to a repository (barring the fast forward ones).
- You can configure an IAM Policy to Limit Pushes and Merges to a Branch (like ready-only user & to specific repo)
2. AWS CodeBuild
- AWS CodeBuild is a fully managed build service in the cloud that compiles your source code, runs unit tests, and produces artifacts that are ready to deploy.
- AWS CodeBuild eliminates the need to provision, manage, and scale your own build servers.
- AWS CodeBuild scales automatically to meet peak build requests. You pay only for the number of build minutes you consume.
- It provides pre-packaged build environments for the most popular programming languages (.Net core, Python, Ruby, Java, Golang, PHP, node.js, Docker, Android, Base). Please note that only .Net core is supported and not .Net Framework.
- CodeBuild supports build tools such as Apache Maven, Gradle, and more. You can also customize your own environment by creating a Docker image and uploading it to the Amazon EC2 Container Registry or the Docker Hub registry.
- Java, .Net, Go, iOS languages can be compiled whereas PHP, Ruby, Python, Node.js are interpreted
- In a nutshell, CodeBuild takes raw code, probably straight from CodeCommit, runs some tests & build options and creates a neatly-wrapped deployment package for us and store in S3 zip file or as individual files.
3. AWS CodeDeploy
- AWS CodeDeploy is a deployment service that automates code deployments to any instance, including Amazon EC2 instances and instances running on-premises or serverless lambda functions.
- AWS CodeDeploy makes it easier for you to rapidly release new features, helps you avoid downtime during deployment, and handles the complexity of updating your applications.
- Once the automated deployment is configured & tested for one instance, the complexity of updating the applications is completely off your shoulders
- With suitable deployment type, you can completely avoid downtime and also configure to rollback automatically, in case of failure.
- You can deploy a nearly unlimited variety of application content, such as code, serverless AWS Lambda functions, web and configuration files, executables, packages, scripts, multimedia files, and so on.
- AWS CodeDeploy works with various systems for configuration management, source control, continuous integration, continuous delivery, and continuous deployment
- Production Integrations page on CodePipline will tell you the pre-built integrations for popular third-party services that are certified by the partners
AWS CodeDeploy Components
Here you go with the components of AWS CodeDeploy
Application: An application is a collection of software and configuration to be deployed to a group of instances. Typically, the instances in the group run the same software. For example, if you have a large distributed system, the web tier will likely constitute one application and the data tier another application.
Revision: A revision is a specific version of deployable content, such as source code, post-build artifacts, web pages, executable files, and deployment scripts, along with an AppSpec file. The AWS CodeDeploy Agent can access a revision from GitHub or an Amazon S3 bucket.
Deployment Group: A deployment group is for grouping EC2 instances or AWS Lambda functions in a CodeDeploy deployment.
appspec.yml is configuration file which specifies the files to be copied and scripts to executed as part of deployment
Deployment configuration: A set of deployment rules that defines the deployment success and failure conditions used by AWS CodeDeploy during a deployment.
For example, if your application needs at least 50% of the instances in a deployment group to be up and serving traffic, you can specify that in your deployment configuration so that a deployment does not cause downtime. If no deployment configuration is associated with either the deployment or the deployment group, then by default AWS CodeDeploy will deploy to one instance at a time.
There are two types of compute platform on which AWS CodeDeploy deploys an application namely EC2/On-Premises & AWS Lambda.
In an EC2/On-Premises deployment, a deployment group is a set of individual instances targeted for a deployment.
Deployment is an action of deploying new revision into a instance.
Deployment type is nothing but the method used to make the latest application revision available onto the instances in a deployment group. There are 2 types of deployment – In-place deployment & Blue / Green Deployment.
Instance Setup: Instance setup is key here:
-
Verify your instance has an IAM instance profile and verify the permissions allows it to participate in AWS CodeDeploy deployment
-
Tag the instance - or make sure it is in Auto Scaling Group
-
Install the agent (can be automated)
-
Verify the agent is running
Here you go with the components of AWS CodeDeploy
Application: An application is a collection of software and configuration to be deployed to a group of instances. Typically, the instances in the group run the same software. For example, if you have a large distributed system, the web tier will likely constitute one application and the data tier another application.
Revision: A revision is a specific version of deployable content, such as source code, post-build artifacts, web pages, executable files, and deployment scripts, along with an AppSpec file. The AWS CodeDeploy Agent can access a revision from GitHub or an Amazon S3 bucket.
Deployment Group: A deployment group is for grouping EC2 instances or AWS Lambda functions in a CodeDeploy deployment.
appspec.yml is configuration file which specifies the files to be copied and scripts to executed as part of deployment
Deployment configuration: A set of deployment rules that defines the deployment success and failure conditions used by AWS CodeDeploy during a deployment.
For example, if your application needs at least 50% of the instances in a deployment group to be up and serving traffic, you can specify that in your deployment configuration so that a deployment does not cause downtime. If no deployment configuration is associated with either the deployment or the deployment group, then by default AWS CodeDeploy will deploy to one instance at a time.
There are two types of compute platform on which AWS CodeDeploy deploys an application namely EC2/On-Premises & AWS Lambda.
In an EC2/On-Premises deployment, a deployment group is a set of individual instances targeted for a deployment.
Deployment is an action of deploying new revision into a instance.
Deployment type is nothing but the method used to make the latest application revision available onto the instances in a deployment group. There are 2 types of deployment – In-place deployment & Blue / Green Deployment.
Instance Setup: Instance setup is key here:
- Verify your instance has an IAM instance profile and verify the permissions allows it to participate in AWS CodeDeploy deployment
- Tag the instance - or make sure it is in Auto Scaling Group
- Install the agent (can be automated)
- Verify the agent is running
4. AWS CodePipeline
- Releasing software to the market at a fast pace is at the core of every business. Releasing feature changes & bug fixes to production using traditional approach is often error prone.
- AWS CodePipeline is a continuous delivery service you can use to model, visualize, and automate the steps required to release your software. With AWS CodePipeline, you model the full release process from building your code, deploying to pre-production environments, testing your application and releasing it to production.
- AWS supports many third-party software solutions that you can plug and play into different stages in your development workflow. This can be especially helpful when you only need to use AWS tools for certain pieces of your workflow and wanted to reuse some of your existing implementations.
- Automating your build, test, and release process allows you to test each code change and catch bugs while they are small and easy to fix. You can assure production quality of your code because each change goes through your standardized release process.”
- There are two ways in which you can detect the changes in source - periodic checks from & change-detection. The most recommended way is change-detection.
- There are two ways in which you can detect the changes in source - periodic checks from & change-detection. The most recommended way is change-detection.
AWS CodePipeline Components
There are 3 main components of CodePipeline
- Stages – The pipeline consists of multiple stages. In your organization, you can have multiple stages such as source, build, test, staging and production.
- Action – Each stage consists of a series of actions which specify what must be done on the artifact. Each stage can only perform one revision of the source code at a time. All of the artifacts are stored in an S3 bucket. For example, the code which needs to be built will be copied to the S3 bucket for the pipeline. This becomes the input artifact.
- Transition – A transition is an action which is performed when one stage is moved to the next in the pipeline. An approval action can also put in place so that if a manual approval is required before it goes to the next stage.
- Important Note: The output of one stage becomes the input of the other and is stored in S3 bucket - either AWS default (say something like `codepipeline-us-east-1-162146562936`) and to the custom S3 bucket of your choice. So when you delete a CodePipeline, make sure to clean-up this artifact content as I do not find them being cleaned up when a CodePipeline is deleted.
This might cost you, if the size of your source code and / or artifacts are in GBs, as the objects will be billed at S3 Standard storage class price.
- Stages – The pipeline consists of multiple stages. In your organization, you can have multiple stages such as source, build, test, staging and production.
- Action – Each stage consists of a series of actions which specify what must be done on the artifact. Each stage can only perform one revision of the source code at a time. All of the artifacts are stored in an S3 bucket. For example, the code which needs to be built will be copied to the S3 bucket for the pipeline. This becomes the input artifact.
- Transition – A transition is an action which is performed when one stage is moved to the next in the pipeline. An approval action can also put in place so that if a manual approval is required before it goes to the next stage.
- Important Note: The output of one stage becomes the input of the other and is stored in S3 bucket - either AWS default (say something like `codepipeline-us-east-1-162146562936`) and to the custom S3 bucket of your choice. So when you delete a CodePipeline, make sure to clean-up this artifact content as I do not find them being cleaned up when a CodePipeline is deleted. This might cost you, if the size of your source code and / or artifacts are in GBs, as the objects will be billed at S3 Standard storage class price.
AWS CodePipeline Actions
Here is the CodePipeline Actions in detail, to see what all we can do as part of pipeline and what third-party tools integration are supported.
- At present we have only Manual Approval Type
- You can get the source for your build from Amazon S3, AWS CodeCommit or GitHub
- Build configuration from your existing Jenkins or AWS CodeBuild or Solono CI can be integrated with CodePipeline.
- You can run your test suites from wide variety of tools, like Jenkins, DeviceFarm, CodeBuild, Blazemeter, Ghost Inspector etc
- Deployment providers can be AWS ECS, AWS CodeDeploy, AWS CloudFormation, AWS ElasticBean Stack & AWS Opswork Stacks
- You can invoke AWS Lambda as part of CodePipeline
If you are wondering why they are numbered, simple; if you go back and check AWS documentation few months down the line (after Sep 2018), the numbers would have increased due to evolving integrations. Please refer integrations page for details.
- At present we have only Manual Approval Type
- You can get the source for your build from Amazon S3, AWS CodeCommit or GitHub
- Build configuration from your existing Jenkins or AWS CodeBuild or Solono CI can be integrated with CodePipeline.
- You can run your test suites from wide variety of tools, like Jenkins, DeviceFarm, CodeBuild, Blazemeter, Ghost Inspector etc
- Deployment providers can be AWS ECS, AWS CodeDeploy, AWS CloudFormation, AWS ElasticBean Stack & AWS Opswork Stacks
- You can invoke AWS Lambda as part of CodePipeline
Here you go with the well designed Lab Series!
What will I learn in this Lab Series ?
You will learn the following that are bare minimum requirement for this Lab Series- Creating IAM Group by attaching existing policies and defining in-line policies
- Create IAM User & Service Role
- Create S3 Bucket
- Create a repo in AWS CodeCommit, add source & commit the changeset
- Build Sample Java Application using AWS CodeBuild, taking source from CodeCommit
- Create Security Group, Instance Profile Role, EC2 Instance from console
- Create CodeDeploy to install pre-requisites & deploy Java Web Application
- Create CodeDeploy to spin-up EC2 instance & deploy Java Web Application
- Create & Execute CloudFormation template to create lambda function
- Create AWS CodePipeline to glue AWS CodeCommit, AWS CodeBuild, AWS CodeDeploy
- Integrating Manual Approval with AWS CodePipeline
Disclaimer:
- Note that the entire AWS Developer Tools - CI CD LAB have been created & validated in my personal AWS Account. Feel free to add comments, if you find any issues.
- Though this LAB material will cover the AWS resources very well within your monthly free limits, ensure to tear down the unused resources to avoid surprises.
- AWS CodeCommit: Anyone with an AWS account can get started with AWS CodeCommit for free. Your account gets 5 active users per month for free (within limits), after which you pay $1 per additional active user per month. There are no upfront fees or commitments.
- AWS CodeBuild: Only pay by minute for the compute resources you use
- AWS CodeDeploy: There is no additional charge for code deployments to Amazon EC2 instances through AWS CodeDeploy. You pay $0.02 per on-premises instance update using AWS CodeDeploy.
- AWS CodePipeline: You pay only for what you use. AWS CodePipeline costs $1 per active pipeline* per month. To encourage experimentation, pipelines are free for the first 30 days after creation.
No comments:
Post a Comment