Integrating with GitHub Actions – Amazon CodeGuru in your DevSecOps Pipeline

Posted on by Mark FeehilyComments Off on Integrating with GitHub Actions – Amazon CodeGuru in your DevSecOps Pipeline

Many organizations have adopted DevOps practices to streamline and automate software delivery and IT operations. A DevOps model can be adopted without sacrificing security by using automated compliance policies, fine-grained controls, and configuration management techniques. However, one of the key challenges customers face is analyzing code and detecting any vulnerabilities in the code pipeline due to a lack of access to the right tool. Amazon CodeGuru addresses this challenge by using machine learning and automated reasoning to identify critical issues and hard-to-find bugs during application development and deployment, thus improving code quality.

We discussed how you can build a CI/CD pipeline to deploy a web application in our previous post “Integrating with GitHub Actions – CI/CD pipeline to deploy a Web App to Amazon EC2”. In this post, we will use that pipeline to include security checks and integrate it with Amazon CodeGuru Reviewer to analyze and detect potential security vulnerabilities in the code before deploying it.

Amazon CodeGuru Reviewer helps you improve code security and provides recommendations based on common vulnerabilities (OWASP Top 10) and AWS security best practices. CodeGuru analyzes Java and Python code and provides recommendations for remediation. CodeGuru Reviewer detects a deviation from best practices when using AWS APIs and SDKs, and also identifies concurrency issues, resource leaks, security vulnerabilities and validates input parameters. For every workflow run, CodeGuru Reviewer’s GitHub Action copies your code and build artifacts into an S3 bucket and calls CodeGuru Reviewer APIs to analyze the artifacts and provide recommendations. Refer to the code detector library here for more information about CodeGuru Reviewer’s security and code quality detectors.

With GitHub Actions, developers can easily integrate CodeGuru Reviewer into their CI workflows, conducting code quality and security analysis. They can view CodeGuru Reviewer recommendations directly within the GitHub user interface to quickly identify and fix code issues and security vulnerabilities. Any pull request or push to the master branch will trigger a scan of the changed lines of code, and scheduled pipeline runs will trigger a full scan of the entire repository, ensuring comprehensive analysis and continuous improvement.

Solution overview

The solution comprises of the following components:

GitHub Actions – Workflow Orchestration tool that will host the Pipeline.

AWS CodeDeploy – AWS service to manage deployment on Amazon EC2 Autoscaling Group.

AWS Auto Scaling – AWS service to help maintain application availability and elasticity by automatically adding or removing Amazon EC2 instances.

Amazon EC2 – Destination Compute server for the application deployment.

Amazon CodeGuru – AWS Service to detect security vulnerabilities and automate code reviews.

AWS CloudFormation – AWS infrastructure as code (IaC) service used to orchestrate the infrastructure creation on AWS.

AWS Identity and Access Management (IAM) OIDC identity provider – Federated authentication service to establish trust between GitHub and AWS to allow GitHub Actions to deploy on AWS without maintaining AWS Secrets and credentials.

Amazon Simple Storage Service (Amazon S3) – Amazon S3 to store deployment and code scan artifacts.

The following diagram illustrates the architecture:

Figure 1. Architecture Diagram of the proposed solution in the blog

Developer commits code changes from their local repository to the GitHub repository. In this post, the GitHub action is triggered manually, but this can be automated.
GitHub action triggers the build stage.
GitHub’s Open ID Connector (OIDC) uses the tokens to authenticate to AWS and access resources.
GitHub action uploads the deployment artifacts to Amazon S3.
GitHub action invokes Amazon CodeGuru.
The source code gets uploaded into an S3 bucket when the CodeGuru scan starts.
GitHub action invokes CodeDeploy.
CodeDeploy triggers the deployment to Amazon EC2 instances in an Autoscaling group.
CodeDeploy downloads the artifacts from Amazon S3 and deploys to Amazon EC2 instances.

Prerequisites

This blog post is a continuation of our previous post – Integrating with GitHub Actions – CI/CD pipeline to deploy a Web App to Amazon EC2. You will need to setup your pipeline by following instructions in that blog.

After completing the steps, you should have a local repository with the below directory structure, and one completed Actions run.

Figure 2. Directory structure

To enable automated deployment upon git push, you will need to make a change to your .github/workflow/deploy.yml file. Specifically, you can activate the automation by modifying the following line of code in the deploy.yml file:

From:

workflow_dispatch: {}

To:

#workflow_dispatch: {}
push:
branches: [ main ]
pull_request:

Solution walkthrough

The following steps provide a high-level overview of the walkthrough:

Create an S3 bucket for the Amazon CodeGuru Reviewer.
Update the IAM role to include permissions for Amazon CodeGuru.
Associate the repository in Amazon CodeGuru.
Add Vulnerable code.
Update GitHub Actions Job to run the Amazon CodeGuru Scan.
Push the code to the repository.
Verify the pipeline.
Check the Amazon CodeGuru recommendations in the GitHub user interface.

1. Create an S3 bucket for the Amazon CodeGuru Reviewer

When you run a CodeGuru scan, your code is first uploaded to an S3 bucket in your AWS account.

Note that CodeGuru Reviewer expects the S3 bucket name to begin with codeguru-reviewer-.

You can create this bucket using the bucket policy outlined in this CloudFormation template (JSON or YAML) or by following these instructions.

2.  Update the IAM role to add permissions for Amazon CodeGuru

Locate the role created in the pre-requisite section, named “CodeDeployRoleforGitHub”.
Next, create an inline policy by following these steps. Give it a name, such as “codegurupolicy” and add the following permissions to the policy.

{
“Version”: “2012-10-17″,
“Statement”: [
{
“Action”: [
“codeguru-reviewer:ListRepositoryAssociations”,
“codeguru-reviewer:AssociateRepository”,
“codeguru-reviewer:DescribeRepositoryAssociation”,
“codeguru-reviewer:CreateCodeReview”,
“codeguru-reviewer:DescribeCodeReview”,
“codeguru-reviewer:ListRecommendations”,
“iam:CreateServiceLinkedRole”
],
“Resource”: “*”,
“Effect”: “Allow”
},
{
“Action”: [
“s3:CreateBucket”,
“s3:GetBucket*“,
“s3:List*“,
“s3:GetObject”,
“s3:PutObject”,
“s3:DeleteObject”
],
“Resource”: [
“arn:aws:s3:::codeguru-reviewer-*“,
“arn:aws:s3:::codeguru-reviewer-*/*”
],
“Effect”: “Allow”
}
]
}

3.  Associate the repository in Amazon CodeGuru

Follow the instructions here to associate your repo – https://docs.aws.amazon.com/codeguru/latest/reviewer-ug/create-github-association.html

Figure 3. Associate the repository

At this point, you will have completed your initial full analysis run. However, since this is a simple “helloWorld” program, you may not receive any recommendations. In the following steps, you will incorporate vulnerable code and trigger the analysis again, allowing CodeGuru to identify and provide recommendations for potential issues.

4.  Add Vulnerable code

Create a file application.conf
at /aws-codedeploy-github-actions-deployment/spring-boot-hello-world-example

Add the following content in application.conf file.

db.default.url=”postgres://test-ojxarsxivjuyjc:ubKveYbvNjQ5a0CU8vK4YoVIhl@ec2-54-225-223-40.compute-1.amazonaws.com:5432/dcectn1pto16vi?ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory”

db.default.url=${?DATABASE_URL}

db.default.port=”3000″

db.default.datasource.username=”root”

db.default.datasource.password=”testsk_live_454kjkj4545FD3434Srere7878″

db.default.jpa.generate-ddl=”true”

db.default.jpa.hibernate.ddl-auto=”create”

5. Update GitHub Actions Job to run Amazon CodeGuru Scan

You will need to add a new job definition in the GitHub Actions’ yaml file. This new section should be inserted between the Build and Deploy sections for optimal workflow.
Additionally, you will need to adjust the dependency in the deploy section to reflect the new flow: Build -> CodeScan -> Deploy.
Review sample GitHub actions code for running security scan on Amazon CodeGuru Reviewer.

codescan:
needs: build
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
security-events: write

steps:

– name: Download an artifact
uses: actions/[email protected]
with:
name: build-file

– name: Configure AWS credentials
id: iam-role
continue-on-error: true
uses: aws-actions/[email protected]
with:
role-to-assume: ${{ secrets.IAMROLE_GITHUB }}
role-session-name: GitHub-Action-Role
aws-region: ${{ env.AWS_REGION }}

– uses: actions/[email protected]
if: steps.iam-role.outcome == ‘success’
with:
fetch-depth: 0

– name: CodeGuru Reviewer
uses: aws-actions/[email protected]
if: ${{ always() }}
continue-on-error: false
with:
s3_bucket: ${{ env.S3bucket_CodeGuru }}
build_path: .

– name: Store SARIF file
if: steps.iam-role.outcome == ‘success’
uses: actions/[email protected]
with:
name: SARIF_recommendations
path: ./codeguru-results.sarif.json

– name: Upload review result
uses: github/codeql-action/[email protected]
with:
sarif_file: codeguru-results.sarif.json

– run: |

echo “Check for critical volnurability”
count=$(cat codeguru-results.sarif.json | jq ‘.runs[].results[] | select(.level == “error”) | .level’ | wc -l)
if (( $count > 0 )); then
echo “There are $count critical findings, hence stopping the pipeline.”
exit 1
fi

Refer to the complete file provided below for your reference. It is important to note that you will need to replace the following environment variables with your specific values.

S3bucket_CodeGuru
AWS_REGION
S3BUCKET

name: Build and Deploy

on:
#workflow_dispatch: {}
push:
branches: [ main ]
pull_request:

env:
applicationfolder: spring-boot-hello-world-example
AWS_REGION: us-east-1 # <replace this with your AWS region>
S3BUCKET: *<Replace your bucket name here>*
S3bucket_CodeGuru: codeguru-reviewer-<*replacebucketnameher*> # S3 Bucket with “codeguru-reviewer-*” prefix

jobs:
build:
name: Build and Package
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
– uses: actions/[email protected]
name: Checkout Repository

– uses: aws-actions/[email protected]
with:
role-to-assume: ${{ secrets.IAMROLE_GITHUB }}
role-session-name: GitHub-Action-Role
aws-region: ${{ env.AWS_REGION }}

– name: Set up JDK 1.8
uses: actions/[email protected]
with:
java-version: 1.8

– name: chmod
run: chmod -R +x ./.github

– name: Build and Package Maven
id: package
working-directory: ${{ env.applicationfolder }}
run: $GITHUB_WORKSPACE/.github/scripts/build.sh

– name: Upload Artifact to s3
working-directory: ${{ env.applicationfolder }}/target
run: aws s3 cp *.war s3://${{ env.S3BUCKET }}/

– name: Artifacts for codescan action
uses: actions/[email protected]
with:
name: build-file
path: ${{ env.applicationfolder }}/target/*.war

codescan:
needs: build
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
security-events: write

steps:

– name: Download an artifact
uses: actions/[email protected]
with:
name: build-file

– name: Configure AWS credentials
id: iam-role
continue-on-error: true
uses: aws-actions/[email protected]
with:
role-to-assume: ${{ secrets.IAMROLE_GITHUB }}
role-session-name: GitHub-Action-Role
aws-region: ${{ env.AWS_REGION }}

– uses: actions/[email protected]
if: steps.iam-role.outcome == ‘success’
with:
fetch-depth: 0

– name: CodeGuru Reviewer
uses: aws-actions/[email protected]
if: ${{ always() }}
continue-on-error: false
with:
s3_bucket: ${{ env.S3bucket_CodeGuru }}
build_path: .

– name: Store SARIF file
if: steps.iam-role.outcome == ‘success’
uses: actions/[email protected]
with:
name: SARIF_recommendations
path: ./codeguru-results.sarif.json

– name: Upload review result
uses: github/codeql-action/[email protected]
with:
sarif_file: codeguru-results.sarif.json

– run: |

echo “Check for critical volnurability”
count=$(cat codeguru-results.sarif.json | jq ‘.runs[].results[] | select(.level == “error”) | .level’ | wc -l)
if (( $count > 0 )); then
echo “There are $count critical findings, hence stopping the pipeline.”
exit 1
fi
deploy:
needs: codescan
runs-on: ubuntu-latest
environment: Dev
permissions:
id-token: write
contents: read
steps:
– uses: actions/[email protected]
– uses: aws-actions/[email protected]
with:
role-to-assume: ${{ secrets.IAMROLE_GITHUB }}
role-session-name: GitHub-Action-Role
aws-region: ${{ env.AWS_REGION }}
– run: |
echo “Deploying branch ${{ env.GITHUB_REF }} to ${{ github.event.inputs.environment }}”
commit_hash=`git rev-parse HEAD`
aws deploy create-deployment –application-name CodeDeployAppNameWithASG –deployment-group-name CodeDeployGroupName –github-location repository=$GITHUB_REPOSITORY,commitId=$commit_hash –ignore-application-stop-failures

6.  Push the code to the repository:

Remember to save all the files that you have modified.
To ensure that you are in your git repository folder, you can run the command:

git remote -v

The command should return the remote branch address, which should be similar to the following:

[email protected] GitActionsDeploytoAWS % git remote -v
origin g[email protected]:<username>/GitActionsDeploytoAWS.git (fetch)
origin g[email protected]:<username>/GitActionsDeploytoAWS.git (push)

To push your code to the remote branch, run the following commands:

git add .
git commit -m “Adding Security Scan”
git push

Your code has been pushed to the repository and will trigger the workflow as per the configuration in GitHub Actions.

7.  Verify the pipeline

Your pipeline is set up to fail upon the detection of a critical vulnerability. You can also suppress recommendations from CodeGuru Reviewer if you think it is not relevant for setup. In this example, as there are two critical vulnerabilities, the pipeline will not proceed to the next step.
To view the status of the pipeline, navigate to the Actions tab on your GitHub console. You can refer to the following image for guidance.

Figure 4. GitHub Actions pipeline

To view the details of the error, you can expand the “codescan” job in the GitHub Actions console. This will provide you with more information about the specific vulnerabilities that caused the pipeline to fail and help you to address them accordingly.

Figure 5. Codescan actions logs

8. Check the Amazon CodeGuru recommendations in the GitHub user interface

Once you have run the CodeGuru Reviewer Action, any security findings and recommendations will be displayed on the Security tab within the GitHub user interface. This will provide you with a clear and convenient way to view and address any issues that were identified during the analysis.

Figure 6. Security tab with results

Clean up

To avoid incurring future charges, you should clean up the resources that you created.

Empty the Amazon S3 bucket.
Delete the CloudFormation stack (CodeDeployStack) from the AWS console.

Delete codeguru Amazon S3 bucket.

Disassociate the GitHub repository in CodeGuru Reviewer.
Delete the GitHub Secret (‘IAMROLE_GITHUB’)

Go to the repository settings on GitHub Page.
Select Secrets under Actions.
Select IAMROLE_GITHUB, and delete it.

Conclusion

Amazon CodeGuru is a valuable tool for software development teams looking to improve the quality and efficiency of their code. With its advanced AI capabilities, CodeGuru automates the manual parts of code review and helps identify performance, cost, security, and maintainability issues. CodeGuru also integrates with popular development tools and provides customizable recommendations, making it easy to use within existing workflows. By using Amazon CodeGuru, teams can improve code quality, increase development speed, lower costs, and enhance security, ultimately leading to better software and a more successful overall development process.

In this post, we explained how to integrate Amazon CodeGuru Reviewer into your code build pipeline using GitHub actions. This integration serves as a quality gate by performing code analysis and identifying challenges in your code. Now you can access the CodeGuru Reviewer recommendations directly within the GitHub user interface for guidance on resolving identified issues.

About the author:

Mahesh Biradar

Mahesh Biradar is a Solutions Architect at AWS. He is a DevOps enthusiast and enjoys helping customers implement cost-effective architectures that scale.

Suresh Moolya

Suresh Moolya is a Senior Cloud Application Architect with Amazon Web Services. He works with customers to architect, design, and automate business software at scale on AWS cloud.

Shikhar Mishra

Shikhar is a Solutions Architect at Amazon Web Services. He is a cloud security enthusiast and enjoys helping customers design secure, reliable, and cost-effective solutions on AWS.

Setting up a secure CI/CD pipeline in a private Amazon Virtual Private Cloud with no public internet access

Posted on by Mark FeehilyComments Off on Setting up a secure CI/CD pipeline in a private Amazon Virtual Private Cloud with no public internet access

With the rise of the cloud and increased security awareness, the use of private Amazon VPCs with no public internet access also expanded rapidly. This setup is recommended to make sure of proper security through isolation. The isolation requirement also applies to code pipelines, in which developers deploy their application modules, software packages, and other dependencies and bundles throughout the development lifecycle. This is done without having to push larger bundles from the developer space to the staging space or the target environment. Furthermore, AWS CodeArtifact is used as an artifact management service that will help organizations of any size to securely store, publish, and share software packages used in their software development process.

We’ll walk through the steps required to build a secure, private continuous integration/continuous development (CI/CD) pipeline with no public internet access while maintaining log retention in Amazon CloudWatch. We’ll utilize AWS CodeCommit for source, CodeArtifact for the Modules and software packages, and Amazon Simple Storage Service (Amazon S3) as artifact storage.

Prerequisites

The prerequisites for following along with this post include:

An AWS Account
A Virtual Private Cloud (Amazon VPC)
A CI/CD pipeline – This can be CodePipeline, Jenkins or any CI/CD tool you want to integrate CodeArtifact with, we will use CodePipeline in our walkthrough here.

Solution walkthrough

The main service we’ll focus on is CodeArtifact, a fully managed artifact repository service that makes it easy for organizations of any size to securely store, publish, and share software packages used in their software development process. CodeArtifact works with commonly used package managers and build tools, such as Maven and Gradle (Java), npm and yarn (JavaScript), pip and twine (Python), or NuGet (.NET).

Users push code to CodeCommit, CodePipeline will detect the change and start the pipeline, in CodeBuild the build stage will utilize the private endpoints and download the software packages needed without the need to go over the internet.

The preceding diagram shows how the requests remain private within the VPC and won’t go through the Internet gateway, by going from CodeBuild over the private endpoint to CodeArtifact service, all within the private subnet.

The requests will use the following VPC endpoints to connect to these AWS services:

CloudWatch Logs endpoint (for CodeBuild to put logs in CloudWatch)
CodeArtifact endpoints

AWS Security Token Service (AWS STS) endpoint
Amazon Simple Storage Service (Amazon S3) endpoint

Walkthrough

Create a CodeCommit Repository:

Navigate to your CodeCommit Console then click on Create repository

Figure 2. Screenshot: Create repository button.

Type in name for the repository then click Create

Figure 3. Screenshot: Repository setting with name shown as “Private” and empty Description.

Scroll down and click Create file

Figure 4. Create file button.

Copy the example buildspec.yml file and paste it to the editor

Example buildspec.yml file:

version: 0.2
phases:
install:
runtime-versions:
nodejs: 16

commands:
– export AWS_STS_REGIONAL_ENDPOINTS=regional
– ACCT=`aws sts get-caller-identity –region ${AWS_REGION} –query Account –output text`
– aws codeartifact login –tool npm –repository Private –domain private –domain-owner ${ACCT}
– npm install
build:
commands:
– node index.js

Name the file buildspec.yml, type in your name and your email address then Commit changes

Figure 5. Screenshot: Create file page.

Create CodeArtifact

Navigate to your CodeArtifact Console then click on Create repository
Give it a name and select npm-store as public upsteam repository

Figure 6. Screenshot: Create repository page with Repository name “Private”.

For the Domain Select this AWS account and enter a domain name

Figure 7. Screenshot: Select domain page.

Click Next then Create repository

Figure 8. Screenshot: Create repository review page.

Create a CI/CD using CodePipeline

Navigate to your CodePipeline Console then click on Create pipeline

Figure 9. Screenshot: Create pipeline button.

Type a name, leave the Service role as “New service role” and click next

Figure 10. Screenshot: Choose pipeline setting page with pipeline name “Private”.

Select AWS CodeCommit as your Source provider
Then choose the CodeCommit repository you created earlier and for branch select main then click Next

Figure 11. Screenshot: Create pipeline add source stage.

For the Build Stage, Choose AWS CodeBuild as the build provider, then click Create Project

Figure 12. Screenshot: Create pipeline add build stage.

This will open new window to create the new Project, Give this project a name

Figure 13. Screenshot: Create pipeline create build project window.

 Scroll down to the Environment section: select pick Managed image,
For Operating system select “Amazon Linux 2”,
Runtime “Standard” and
For Image select the aws/codebuild/amazonlinux2-x86+64-standard:4.0
For the Image version: Always use the latest image for this runtime version
Select Linux for the Environment type
Leave the Privileged option unchecked and set Service Role to “New service role”

Figure 14. Screenshot: Create pipeline create build project, setting up environment window.

Expand Additional configurations and scroll down to the VPC section, select the desired VPC, your Subnets (we recommend selecting multiple AZs, to ensure high availability), and Security Group (the security group rules must allow resources that will use the VPC endpoint to communicate with the AWS service to communicate with the endpoint network interface, default VPC security group will be used here as an example)

Figure 15. Screenshot: Create pipeline create build project networking window.

Scroll down to the Buildspec and select “Use a buildspec file” and type “buildspec.yml” for the Buildspec name

Figure 16. Screenshot: Create pipeline create build project buildspec window.

Select the CloudWatch logs option you can leave the group name and stream empty this will let the service use the default values and click Continue to CodePipeline

Figure 17. Screenshot: Create pipeline create build project logs window.

This will create the new CodeBuild Project, update the CodePipeline page, now you can click Next

Figure 18. Screenshot: Create pipeline add build stage window.

 Since we are not deploying this to any environment, you can skip the deploy stage and click “Skip deploy stage”

Figure 19. Screenshot: Create pipeline add deploy stage.

Figure 20. Screenshot: Create pipeline skip deployment stage confirmation.

After you get the popup click skip again you’ll see the review page, scroll all the way down and click Create Pipeline

Create a VPC endpoint for Amazon CloudWatch Logs. This will enable CodeBuild to send execution logs to CloudWatch:

Navigate to your VPC console, and from the navigation menu on the left select “Endpoints”.

Figure 21. Screenshot: VPC endpoint.

 click Create endpoint Button.

Figure 22. Screenshot: Create endpoint.

For service Category, select “AWS Services”. You can set a name for the new endpoint, and make sure to use something descriptive.

Figure 23. Screenshot: Create endpoint page.

From the list of services, search for the endpoint by typing logs in the search bar and selecting the one with com.amazonaws.us-west-2.logs.
This walkthrough can be done in any region that supports the services. I am going to be using us-west-2, please select the appropriate region for your workload.

Figure 24. Screenshot: create endpoint select services with com.amazonaws.us-west-2.logs selected.

Select the VPC that you want the endpoint to be associated with, and make sure that the Enable DNS name option is checked under additional settings.

Figure 25. Screenshot: create endpoint VPC setting shows VPC selected.

Select the Subnets where you want the endpoint to be associated, and you can leave the security group as default and the policy as empty.

Figure 26. Screenshot: create endpoint subnet setting shows 2 subnet selected and default security group selected.

Select Create Endpoint.

Figure 27. Screenshot: create endpoint button.

Create a VPC endpoint for CodeArtifact. At the time of writing this article, CodeArifact has two endpoints: one is for API operations like service level operations and authentication, and the other is for using the service such as getting modules for our code. We’ll need both endpoints to automate working with CodeArtifact. Therefore, we’ll create both endpoints with DNS enabled.

In addition, we’ll need AWS Security Token Service (AWS STS) endpoint for get-caller-identity API call:

Follow steps a-c from the steps that were used from the creating the Logs endpoint above.

a. From the list of services, you can search for the endpoint by typing codeartifact in the search bar and selecting the one with com.amazonaws.us-west-2.codeartifact.api.

Figure 28. Screenshot: create endpoint select services with com.amazonaws.us-west-2.codeartifact.api selected.

Follow steps e-g from Part 4.

Then, repeat the same for com.amazon.aws.us-west-2.codeartifact.repositories service.

Figure 29. Screenshot: create endpoint select services with com.amazonaws.us-west-2.codeartifact.api selected.

Enable a VPC endpoint for AWS STS:

Follow steps a-c from Part 4

a. From the list of services you can search for the endpoint by typing sts in the search bar and selecting the one with com.amazonaws.us-west-2.sts.

Figure 30.Screenshot: create endpoint select services with com.amazon.aws.us-west-2.codeartifact.repositories selected.

Then follow steps e-g from Part 4.

Create a VPC endpoint for S3:

Follow steps a-c from Part 4

a. From the list of services you can search for the endpoint by typing sts in the search bar and selecting the one with com.amazonaws.us-west-2.s3, select the one with type of Gateway

Then select your VPC, and select the route tables for your subnets, this will auto update the route table with the new S3 endpoint.

Figure 31. Screenshot: create endpoint select services with com.amazonaws.us-west-2.s3 selected.

Now we have all of the endpoints set. The last step is to update your pipeline to point at the CodeArtifact repository when pulling your code dependencies. I’ll use CodeBuild buildspec.yml as an example here.

Make sure that your CodeBuild AWS Identity and Access Management (IAM) role has the permissions to perform STS and CodeArtifact actions.

Navigate to IAM console and click Roles from the left navigation menu, then search for your IAM role name, in our case since we selected “New service role” option in step 2.k was created with the name “codebuild-Private-service-role” (codebuild-<BUILD PROJECT NAME>-service-role)

Figure 32. Screenshot: IAM roles with codebuild-Private-service-role role shown in search.

From the Add permissions menu, click on Create inline policy

Search for STS in the services then select STS

Figure 34. Screenshot: IAM visual editor with sts shown in search.

Search for “GetCallerIdentity” and select the action

Figure 35. Screenshot: IAM visual editor with GetCallerIdentity in search and action selected.

Repeat the same with “GetServiceBearerToken”

Figure 36. Screenshot: IAM visual editor with GetServiceBearerToken in search and action selected.

Click on Review, add a name then click on Create policy

Figure 37. Screenshot: Review page and Create policy button.

You should see the new inline policy added to the list

Figure 38. Screenshot: shows the new in-line policy in the list.

For CodeArtifact actions we will do the same on that role, click on Create inline policy

Figure 39. Screenshot: attach policies.

Search for CodeArtifact in the services then select CodeArtifact

Figure 40. Screenshot: select service with CodeArtifact in search.

Search for “GetAuthorizationToken” in actions and select that action in the check box

Figure 41. CodeArtifact: with GetAuthorizationToken in search.

Repeat for “GetRepositoryEndpoint” and “ReadFromRepository”

Click on Resources to fix the 2 warnings, then click on Add ARN on the first one “Specify domain resource ARN for the GetAuthorizationToken action.”

Figure 42. Screenshot: with all selected filed and 2 warnings.

You’ll get a pop up with fields for Region, Account and Domain name, enter your region, your account number, and the domain name, we used “private” when we created our domain earlier.

Figure 43. Screenshot: Add ARN page.

Then click Add

Repeat the same process for “Specify repository resource ARN for the ReadFromRepository and 1 more”, and this time we will provide Region, Account ID, Domain name and Repository name, we used “Private” for the repository we created earlier and “private” for domain

Figure 44. Screenshot: add ARN page.

Note it is best practice to specify the resource we are targeting, we can use the checkbox for “Any” but we want to narrow the scope of our IAM role best we can.

Navigate to CodeCommit then click on the repo you created earlier in step1

Figure 45. Screenshot: CodeCommit repo.

Click on Add file dropdown, then Create file button

Paste the following in the editor space:

{
“dependencies”: {
“mathjs”: “^11.2.0”
}
}

Name the file “package.json”

Add your name and email, and optional commit message

Repeat this process for “index.js” and paste the following in the editor space:

const { sqrt } = require(‘mathjs’)
console.log(sqrt(49).toString())

Figure 46. Screenshot: CodeCommit Commit changes button.

This will force the pipeline to kick off and start building the application

Figure 47. Screenshot: CodePipeline.

This is a very simple application that gets the square root of 49 and log it to the screen, if you click on the Details link from the pipeline build stage, you’ll see the output of running the NodeJS application, the logs are stored in CloudWatch and you can navigate there by clicking on the link the View entire log “Showing the last xx lines of the build log. View entire log”

Figure 48. Screenshot: Showing the last 54 lines of the build log. View entire log.

We used npm example in the buildspec.yml above, Similar setup will be used for pip and twine,

For Maven, Gradle, and NuGet, you must set Environment variables and change your settings.xml and build.gradle, as well as install the plugin for your IDE. For more information, see here.

Cleanup

Navigate to VPC endpoint from the AWS console and delete the endpoints that you created.

Navigate to CodePipeline and delete the Pipeline you created.

Navigate to CodeBuild and delete the Build Project created.

Navigate to CodeCommit and delete the Repository you created.

Navigate to CodeArtifact and delete the Repository and the domain you created.

Navigate to IAM and delete the Roles created:

For CodeBuild: codebuild-<Build Project Name>-service-role

For CodePipeline: AWSCodePipelineServiceRole-<Region>-<Project Name>

Conclusion

In this post, we deployed a full CI/CD pipeline with CodePipeline orchestrating CodeBuild to build and test a small NodeJS application, using CodeArtifact to download the application code dependencies. All without going to the public internet and maintaining the logs in CloudWatch.

About the author:

MJ Kubba

MJ Kubba is a Solutions Architect who enjoys working with public sector customers to build solutions that meet their business needs. MJ has over 15 years of experience designing and implementing software solutions. He has a keen passion for DevOps and cultural transformation.