AWS Now Supports Credentials-fetcher for gMSA on Amazon Linux 2023

In Q1 of 2023, AWS announced the release of the group Managed Service Account (gMSA) credentials-fetcher daemon, with initial support on Amazon Linux 2023, Fedora Linux 36, and Red Hat Enterprise Linux 9. The credentials-fetcher daemon, developed by AWS, is an open source project under the Apache 2.0 License. This release solves a 10-year, longstanding challenge affecting domain connected Linux machines. Until now, Linux users couldn’t use Microsoft Active Directory (Microsoft AD) gMSA and thus have missed out on the improved security and flexibility that gMSA offers over standard service accounts. With the release of the credentials-fetcher daemon, organizations now gain all of gMSA’s benefits without being tied to Windows based hosts.

In this blog post, we explain the use case for credentials-fetcher and give simple instructions for using an Active Directory domain joined Linux server with gMSA. We also demonstrate the interaction with other domain joined services such as Amazon Relational Database Service (Amazon RDS) for Microsoft SQL Server.  The new capabilities of credentials-fetcher pave the way for additional use cases, such as using a Linux host in Amazon Elastic Container Service (Amazon ECS) clusters with gMSA. AWS is committed to using the credentials-fetcher open source project in the AWS cloud, though users may choose to run the service elsewhere. The utility of the service is not limited to AWS. The credentials-fetcher daemon can be leveraged on any supported distribution of Linux and in any environment that meets the Microsoft Active Directory version requirement. This includes on-premise environments, hosted data centers, and other cloud providers.

Solution overview

Organizations running Windows workloads hosted in on-premises data centers use Microsoft AD to authenticate users and services to shared resources over the network. As these organizations migrate workloads into Windows-based environments on AWS and on other clouds, customers traditionally use the domain-join model to access Microsoft AD from Windows instances. In addition, organizations that use Windows containers to scale their applications and reduce their total cost of ownership (TCO) have used gMSAs for Active Directory access by providing Kerberos tickets for container-hosts.

As customers modernize their Windows and Microsoft SQL Server workloads to Linux-based platforms, they still need to authenticate the migrated applications through the organization’s existing Microsoft AD. Although customers can use the domain-join methodology to connect Linux instances to Microsoft AD, it requires a number of steps that traditionally include security limitations. The current method involves a sidecar architecture that fails to periodically rotate passwords, unlike gMSA on Windows containers, thus inducing a security risk of password exposure. Organizations with stringent security postures have not adopted this method on Linux containers and have been waiting for a “gMSA on Windows containers”-like experience on Linux containers.  Active Directory gMSAs have been technically infeasible for customers on Linux-based environments, until today.

A brief introduction to gMSA

Windows-based server infrastructure commonly uses Microsoft Active Directory to facilitate authentication and authorization between users, computers, and other computer network resources. Traditionally, enterprise applications running on Windows platforms use either manually managed accounts used as service accounts or Managed Service Accounts (MSA) for authentication and authorization. The use of manually managed service accounts brings with it the overhead of service account password management, including manually updating the password and updating the password on all servers. It also introduces increased security risks as these accounts typically have elevated privileges and are not tied to a specific user, which creates challenges for attributing activity when auditing the account. For this reason, password management of these accounts is critical.

In contrast, Managed Service Accounts don’t have any password management overhead; the passwords for these type of accounts are automatically rotated and updated on your servers. They are also limited to a single computer account, which means they can’t be used on more than one computer, and cannot be used for interactive logons. A Group Managed Service Account (gMSA) is a special type of service account which augments the functionality; its identity can be shared across multiple computers without needing to know the password. Computers should be part of a Microsoft Active Directory domain, which manages these service accounts to make use of them. Although Windows containers cannot join a domain like an instance, they can still use gMSA identity for authentication and authorization.

Credentials-fetcher’s potential scenarios

With the addition of the credentials-fetcher daemon, more organizations can use gMSA. This gives customers more options if they’re more familiar with Linux, they’re looking to save on licensing costs, and/or looking to improve their security posture. Customers can now associate Linux machines to a gMSA and take advantage of the authentication and authorization between members of that group managed security account. Environments hosted on domain joined, gMSA associated Linux machines running .NET applications or running in Linux containers can now use the gMSA to authenticate between their own domains and other services like Microsoft SQL Server.

Scenario 1: A Microsoft .NET application is running in Docker containers, with the hosts on a Microsoft Active Directory domain joined Amazon Elastic Computer Cloud (Amazon EC2) Linux server. The Linux application server is added as members of the gMSA group. The gMSA account is granted permissions to the domain joined Microsoft SQL Server or Amazon RDS for Microsoft SQL Server database.

Scenario 2: A Microsoft .NET application is running in Docker containers and Microsoft SQL server running in its own Docker container, with the hosts on a Microsoft Active Directory domain joined Amazon EC2 Linux server.  The Linux host servers of the application containers and Microsoft SQL Server container are added as members of the gMSA group. The gMSA account is granted permissions to the Microsoft SQL Server instance database running in a container.

Scenario 3: A Microsoft .NET application is running on an Amazon Elastic Container Service (Amazon ECS) cluster, hosted on a Microsoft Active Directory domain. The Linux servers within the Amazon ECS cluster are added as members of the gMSA group. The gMSA account is granted permissions to the domain joined Microsoft SQL Server or Amazon RDS for Microsoft SQL Server database.

Here is a visualization of the featured use-case scenarios.

Figure 1 Different use-case scenarios with Credentials-fetcher

Implementing the environment

This section will walk you through the prerequisites, environment setup and the installation steps for the credentials-fetcher daemon’s use cases.

Prerequisites

You have properly installed and configured the AWS Command Line Interface (AWS CLI) and PowerShell core on your workstation. We’ve chosen to use the AWS CLI for these steps so that the end-to-end workflow can be demonstrated.
This blog post as of April 4th, 2023 requires an install of Fedora Linux 36 or newer and the latest Amazon Linux 2023 AMI.
This blog post references AWS Managed Microsoft Active Directory, but it will also work with other self-managed Microsoft Active Directory scenarios as long as the Linux machines are able to be domain joined.
You have installed Amazon Relational Database Service (Amazon RDS) instance that is joined to the domain.
You have elevated administrative Active Directory credentials to configure instances to join a domain and create a Microsoft AD security group.
You have accessed to the credentials-fetcher GitHub package for the installation of the latest daemon and updated instructions.

Environment Setup for gMSA on Linux use cases

Figure 2 Credentials-fetcher running in Fedora Linux Server.

All instructions are assuming the use of the Fedora Linux 36 distro, which has been made available during the time of the blog creation. We plan to add gMSA support for additional Linux distributions in the future.

1.     Set up AWS Managed Microsoft Active Directory or Self-hosted Active Directory.

Active Directory setup:  You will set up domain-join from Linux instance to the AD domain. The Linux instance is part of the AD Security group that has access to gMSA account as configured by AD administrator.
AWS Managed Microsoft Active Directory can be deployed using this AWS CloudFormation template.

2.     Create a gMSA account as a Microsoft AD administrator.

Example: Replace ‘LinuxAppFarm’, ‘LinuxFarm01$’ and ‘CORP.EXAMPLE.COM’ with your own gMSA and domain names, respectively. Three Linux instances are displayed in this example: LinuxInstance01$, LinuxInstance02$ and LinuxInstance03$.

# Create the AD group
New-ADGroup -Name “LinuxAppFarm” -SamAccountName “LinuxAppFarm” -GroupScope DomainLocal

# Create the gMSA
New-ADServiceAccount -Name “gmsamachines” -DnsHostName “gmsamachines.CORP.EXAMPLE.COM” -ServicePrincipalNames “host/LinuxAppFarm”, “host/LinuxAppFarm.CORP.EXAMPLE.COM” -PrincipalsAllowedToRetrieveManagedPassword “LinuxAppFarm”

# Add your Linux Instance or containers to the AD group Add-ADGroupMember -Identity “LinuxAppFarm” -Members “LinuxInstances01$”, “LinuxInstances02$”, “LinuxInstances03$”, “MSSQLRDSIntance$”

3.     Verify and test the gMSA account.

PowerShell
# Get the current computer’s group membership
Test-ADServiceAccount gmsamachines

# Get the current computer’s group membership
Get-ADComputer $env:LinuxInstances01 | Get-ADPrincipalGroupMembership | Select-Object DistinguishedName

# Get the groups allowed to retrieve the gMSA password and Change “gmsamachines” for your own gMSA name
(Get-ADServiceAccount gmsamachines -Properties PrincipalsAllowedToRetrieveManagedPassword).PrincipalsAllowedToRetrieveManagedPassword.

Additional reference detailed instructions can be found in this guide to getting started with group managed service accounts.

4.     Create a credentialspec associated with a gMSA account:

Install powershell CredentialSpec module and create CredentialSpec

PowerShell
Install-Module CredentialSpec

New-CredentialSpec -AccountName LinuxAppFarm // Replace ‘LinuxAppFarm’ with your own gMSA Group

You will find the credentialspec in the directory ‘C:Program DataDockerCredentialspecsLinuxAppFarm_CredSpec.json’

5.     Obtain and deploy the supported Fedora Linux 36 version or newer supported AMI (AWS Public Cloud download.)

6.     Manually join your Linux system to the Microsoft Active Directory domain using the following command:

#Install realmd and configure DNS resolver for the Active Directory domain
sudo dnf install realmd sssd oddjob oddjob-mkhomedir adcli krb5-workstation samba-common-tools -y
sudo systemctl stop systemd-resolved
sudo systemctl disable systemd-resolved
sudo unlink /etc/resolv.conf

#Add your DNS nameserver IP and domain name to the resolv.conf and save
sudo nano /etc/resolv.conf

nameserver 10.0.0.20
search corp.example.com

#Join the Linux Server to the realm/domain case-sensitive

Replace (upper-case) realm account and domain name indicated by <bold text>  with the UPN of domain user and FQDN of domain name. Remove < and > in your final command.

Auto-join is not currently supported until the Amazon Linux 2022 distro is updated with the new rpm.

Microsoft SQL Server and Amazon RDS for Microsoft SQL Server can be added for Kerberos database authentication.

Microsoft SQL and Amazon RDS for Microsoft SQL Server must be joined to the AWS Managed Microsoft AD Domain.

See instructions on how to connect Amazon RDS for Microsoft SQL Server to the Microsoft Active Directory domain.

For the highest recommended security, constrained Kerberos delegation for gMSA should be applied to the accounts for any service access.

Set-ADAccountControl -Identity <TestgMSA$> -TrustedForDelegation $false -TrustedToAuthForDelegation $false
Set-ADServiceAccount -Identity TestgMSA$ -Clear ‘msDS-AllowedToDelegateTo’

Detailed instructions can be found here.

7.     Invoke the AddkerberosLease API with the credentialsspec input as shown in following command. This step is important to allow the credentials-fetcher to make a connection to Microsoft Active Directory. The gMSA account is then used for authentication.
Use this command with Fedora Linux only: (grpc_cli is not available on Amazon Linux)

#Replace gMSA group name, netbios name and DNS names in the command (Bold text)
grpc_cli call unix:/var/credentials-fetcher/socket/credentials_fetcher.sock AddKerberosLease “credspec_contents: ‘{“CmsPlugins”:[“ActiveDirectory”],”DomainJoinConfig”:{“Sid”:”S-1-5-21-1445507628-2856671781-3529916291″,”MachineAccountName”:”gmsamachines”,”Guid”:”af602f85-d754-4eea-9fa8-fd76810485f1″,”DnsTreeName”:”corp.example.com”,”DnsName”:”corp.example.com”,”NetBiosName”:”DEMOCORP”},”ActiveDirectoryConfig”:{“GroupManagedServiceAccounts”:[{“Name”:”gmsamachines”,”Scope”:”corp.example.com”},{“Name”:”gmsamachines”,”Scope”:”DEMOCORP”}]}}'”

Response example: (Note the response for use with your Docker application container)
path to kerberos ticket : /var/credentials-fetcher/krbdir/726837743cc966c7b4da/WebApp01

8.     Invoke the Delete kerberosLease API with lease id input as shown here. Set unique identifier lease_id, associated to the request. The deleted_kerberos_file_paths are paths associated to the Kerberos tickets deleted corresponding to the gMSA accounts.
Use this command from the Linux host:

#Delete Kerberos Lease sample command
grpc_cli call unix:/var/credentials-fetcher/socket/credentials_fetcher.sock DeleteKerberosLease “lease_id: ‘${response_lease_id_from_add_kerberos_lease}'”

Installation of credentials-fetcher on supported Linux distros

In this basic use case for testing the credentials-fetcher rpm, the following architecture is assumed for the purposes of this blog.

An AWS Managed Microsoft Active Directory joined Linux Application Server.
An AWS Managed Microsoft Active Directory joined RDS for Microsoft SQL.
gMSA account established for account credentials in an AWS Managed Microsoft Active Directory.

Fedora Linux 36 Server setup:

Deploy the “Fedora cloud based image for AWS public cloud” located here, to your AWS account.
Credentials-fetcher is packaged and included as part of the standard Fedora package repositories. Install the credentials-fetcher rpm by typing the command:

sudo dnf install credentials-fetcher -y

How to use credentials-fetcher per scenario

In these instructions, we will demonstrate the use of credentials-fetcher with an ASP.NET application and Amazon RDS for Microsoft SQL Server. A Microsoft SQL Server container scenario will also be demonstrated as an additional use case.

Scenario 1:  Using .NET Core container application on Linux with Amazon RDS for Microsoft SQL Server backend

Figure 3 Using .NET Core container application on Linux with Amazon RDS for Microsoft SQL Server backend

Once the environment prerequisites have been met, you can install Docker and a repository in preparation for deploying a .NET Core application to a container on the Linux server or servers.

1.     Set up the repository.

Install the dnf-plugins-core package (which provides the commands to manage your DNF repositories) and set up the repository.

sudo dnf -y install dnf-plugins-core

sudo dnf config-manager
–add-repo
https://download.docker.com/linux/fedora/docker-ce.repo

2.     Install Docker Engine and verify credentials-fetcher is installed and started.

Install the latest version of Docker Engine, containerd, and Docker Compose and start the Docker daemon:

sudo dnf install docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo systemctl start docker

sudo dnf install credentials-fetcher
sudo systemctl start credentials-fetcher

Additional reference detailed instructions on how to install Docker engine can be found here.

1. Create a kerberos ticket – associated to the gMSA account as described in step 8 of “Environment Setup for gMSA on Linux use cases”

Take a note of the response generated by the Kerberos ticket creation.

2.     Leverage the Docker File for environment variables

FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build-image
WORKDIR /src
COPY *.csproj ./
RUN dotnet restore
COPY . ./
RUN dotnet publish -c Release -o out
FROM mcr.microsoft.com/dotnet/aspnet:5.0
WORKDIR /app
EXPOSE 80
COPY —from=build-image /src/out ./
RUN apt-get -qq update &&
apt-get -yqq install krb5-user &&
apt-get -yqq clean

ENV KRB5CCNAME=/var/credentials-fetcher/krbdir/krb5cc

ENTRYPOINT [“dotnet”, “WebApp.dll”]

Example: /var/credentials-fetcher/krbdir/726837743cc966c7b4da/WebApp01

3.     Build a Docker image for your application on the Linux server:

docker build -t <your_image_name>

4.     Run Docker with bind mount to the kerberos ticket, ensure the environment variable KRB5CCNAME (see example docker file above) is pointing to the destination location of the bind mount inside the application container.

sudo docker run -p 80:80 -d -it —name webapp1 —mount type=bind,source=/var/credentials-fetcher/krbdir/726837743cc966c7b4da/WebApp01,target=/var/credentials-fetcher/krbdir {docker_image}

5.     Add Amazon RDS for Microsoft SQL Server to the gMSA group with the following commands:

#Add the AD join SQL RDS server to the gMSA account
Set-ADServiceAccount -Identity “gmsamachines” -PrincipalsAllowedToRetrieveManagedPassword “mssqlrdsname$”

6.     Download and install the SQL Management Tool (SSMS) on a management Windows machine that is a member of the service account group to test the Amazon RDS for Microsoft SQL Server connections. The .NET application will have access to each other and the Amazon RDS for Microsoft SQL Server instance.

7.     Log in to the Amazon RDS for Microsoft SQL Server instance and apply the gMSA service account with the desired permissions required by the .NET application.

Scenario 2:  Using .NET Core container application on Linux with a Microsoft SQL Server Container

Figure 4 Using .NET Core container application on Linux with a Microsoft SQL Server Container.

As with Scenario 1, the same steps to install Docker on Linux and deploy your application will apply. The difference will be the deployment of Microsoft SQL Server in a container and the confirmation that the server operates as expected running on Linux and leveraging gMSA for authentication.

1.     As with the first scenario, install and run the credentials-fetcher on the Linux server or servers you are deploying your .NET application containers to.

2.     Deploy a Microsoft SQL Server 2022 container on one of the Linux servers in your gMSA group.

Leverage the Docker file example in “Use Case 1” environment KRB5CCNAME from the Microsoft SQL Server container.
Reference “Use Case 1” for details on verifying docker file KRB5CCNAME.

Run the following command on your Linux server to install the latest Microsoft SQL Server 2022 Docker container available. Replace yourStrong(!)Password with a strong password in the command.

sudo docker run -e “ACCEPT_EULA=Y” -e “SA_PASSWORD=yourStrong(!)Password” -p 1433:1433 -d mcr.microsoft.com/mssql/server:2022-latest —mount type=bind,source=/var/credentials-fetcher/krbdir/726837743cc966c7b4da/WebApp01,target=/var/credentials-fetcher/krbdir {SQL_docker_image}

Verify that the Microsoft SQL Server Docker container is running with the following command:

sudo docker ps

Additional reference details for deploying a Microsoft SQL Server container on Linux can be found here.

3.     Download and install the SQL Management Tool (SSMS) on a management Windows machine that is a member of the service account group to test the Microsoft SQL Server connection.

4.     Log in to the Microsoft SQL Server instance and apply the gMSA service account with the desired permissions required by the .NET application.

Conclusion

Linux containers have become a key modernization destination for customers running .NET workloads. gMSA for Linux containers will help organizations and the overall Microsoft administrator and developer community to access AD from applications and services hosted on Linux containers using the service account authentication model.

gMSA is a managed account that provides automatic password management, service principal name (SPN) management, and the ability to delegate management to administrators over multiple servers or instances. This unblocks a range of modernization use cases around identity using Microsoft AD, such as, connecting .NET Core applications hosted on Linux containers with SQL Server authenticating over Microsoft AD, and securely opening up access to network resources from applications running with service accounts. Based on the capabilities and customer usefulness, credentials-fetcher is positioned to be extended into services such as Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS).

This service feature extends support for non-Windows container applications that require gMSA for Microsoft AD Authentication. AWS is dedicated to continuing development and support for the credentials-fetcher daemon open source project. We believe that open source is good for everyone and we are committed to bringing the value of open source to our customers, and the operational excellence of AWS to open source communities. Contributions and feedback are welcome.

Flatlogic Admin Templates banner

Unit Testing AWS Lambda with Python and Mock AWS Services

When building serverless event-driven applications using AWS Lambda, it is best practice to validate individual components.  Unit testing can quickly identify and isolate issues in AWS Lambda function code.  The techniques outlined in this blog demonstrates unit test techniques for Python-based AWS Lambda functions and interactions with AWS Services.

The full code for this blog is available in the GitHub project as a demonstrative example.

Example use case

Let’s consider unit testing a serverless application which provides an API endpoint to generate a document.  When the API endpoint is called with a customer identifier and document type, the Lambda function retrieves the customer’s name from DynamoDB, then retrieves the document text from DynamoDB for the given document type, finally generating and writing the resulting document to S3.

Figure 1. Example application architecture

Amazon API Gateway provides an endpoint to request the generation of a document for a given customer.  A document type and customer identifier are provided in this API call.
The endpoint invokes an AWS Lambda function that generates a document using the customer identifier and the document type provided.
An Amazon DynamoDB table stores the contents of the documents and the users name, which are retrieved by the Lambda function.
The resulting text document is stored to Amazon S3.

Our testing goal is to determine if an isolated “unit” of code works as intended. In this blog, we will be writing tests to provide confidence that the logic written in the above AWS Lambda function behaves as we expect. We will mock the service integrations to Amazon DynamoDB and S3 to isolate and focus our tests on the Lambda function code, and not on the behavior of the AWS Services.

Define the AWS Service resources in the Lambda function

Before writing our first unit test, let’s look at the Lambda function that contains the behavior we wish to test.  The full code for the Lambda function is available in the GitHub repository as src/sample_lambda/app.py.

As part of our Best practices for working AWS Lambda functions, we recommend initializing AWS service resource connections outside of the handler function and in the global scope.  Additionally, we can retrieve any relevant environment variables in the global scope so that subsequent invocations of the Lambda function do not repeatedly need to retrieve them.  For organization, we can put the resource and variables in a dictionary:

_LAMBDA_DYNAMODB_RESOURCE = { “resource” : resource(‘dynamodb’),
“table_name” : environ.get(“DYNAMODB_TABLE_NAME”,”NONE”) }

However, globally scoped code and global variables are challenging to test in Python, as global statements are executed on import, and outside of the controlled test flow.  To facilitate testing, we define classes for supporting AWS resource connections that we can override (patch) during testing.  These classes will accept a dictionary containing the boto3 resource and relevant environment variables.

For example, we create a DynamoDB resource class with a parameter “boto3_dynamodb_resource” that accepts a boto3 resource connected to DynamoDB:

class LambdaDynamoDBClass:
def __init__(self, lambda_dynamodb_resource):
self.resource = lambda_dynamodb_resource[“resource”]
self.table_name = lambda_dynamodb_resource[“table_name”]
self.table = self.resource.Table(self.table_name)

Build the Lambda Handler

The Lambda function handler is the method in the AWS Lambda function code that processes events. When the function is invoked, Lambda runs the handler method. When the handler exits or returns a response, it becomes available to process another event.

To facilitate unit test of the handler function, move as much of logic as possible to other functions that are then called by the Lambda hander entry point.  Also, pass the AWS resource global variables to these subsequent function calls.  This approach enables us to mock and intercept all resources and calls during test.

In our example, the handler references the global variables, and instantiates the resource classes to setup the connections to specific AWS resources.  (We will be able to override and mock these connections during unit test.)

Then the handler calls the create_letter_in_s3 function to perform the steps of creating the document, passing the resource classes.  This downstream function avoids directly referencing the global context or any AWS resource connections directly.

def lambda_handler(event: APIGatewayProxyEvent, context: LambdaContext) -> Dict[str, Any]:

global _LAMBDA_DYNAMODB_RESOURCE
global _LAMBDA_S3_RESOURCE

dynamodb_resource_class = LambdaDynamoDBClass(_LAMBDA_DYNAMODB_RESOURCE)
s3_resource_class = LambdaS3Class(_LAMBDA_S3_RESOURCE)

return create_letter_in_s3(
dynamo_db = dynamodb_resource_class,
s3 = s3_resource_class,
doc_type = event[“pathParameters”][“docType”],
cust_id = event[“pathParameters”][“customerId”])

Unit testing with mock AWS services

Our Lambda function code has now been written and is ready to be tested, let’s take a look at the unit test code!   The full code for the unit test is available in the GitHub repository as tests/unit/src/test_sample_lambda.py.

In production, our Lambda function code will directly access the AWS resources we defined in our function handler; however, in our unit tests we want to isolate our code and replace the AWS resources with simulations.  This isolation facilitates running unit tests in an isolated environment to prevent accidental access to actual cloud resources.

Moto is a python library for Mocking AWS Services that we will be using to simulate AWS resource our tests.  Moto supports many AWS resources, and it allows you to test your code with little or no modification by emulating functionality of these services.

Moto uses decorators to intercept and simulate responses to and from AWS resources.  By adding a decorator for a given AWS service, subsequent calls from the module to that service will be re-directed to the mock.

@moto.mock_dynamodb
@moto.mock_s3

Configure Test Setup and Tear-down

The mocked AWS resources will be used during the unit test suite.  Using the setUp() method allows you to define and configure the mocked global AWS Resources before the tests are run.

We define the test class and a setUp() method and initialize the mock AWS resource.  This includes configuring the resource to prepare it for testing, such as defining a mock DynamoDB table or creating a mock S3 Bucket.

class TestSampleLambda(TestCase):
def setUp(self) -> None:
dynamodb = boto3.resource(“dynamodb”, region_name=”us-east-1″)
dynamodb.create_table(
TableName = self.test_ddb_table_name,
KeySchema = [{“AttributeName”: “PK”, “KeyType”: “HASH”}],
AttributeDefinitions = [{“AttributeName”: “PK”,
“AttributeType”: “S”}],
BillingMode = ‘PAY_PER_REQUEST’

s3_client = boto3.client(‘s3’, region_name=”us-east-1″)
s3_client.create_bucket(Bucket = self.test_s3_bucket_name )

After creating the mocked resources, the setup function creates resource class object referencing those mocked resources, which will be used during testing.

mocked_dynamodb_resource = resource(“dynamodb”)
mocked_s3_resource = resource(“s3”)
mocked_dynamodb_resource = { “resource” : resource(‘dynamodb’),
“table_name” : self.test_ddb_table_name }
mocked_s3_resource = { “resource” : resource(‘s3’),
“bucket_name” : self.test_s3_bucket_name }
self.mocked_dynamodb_class = LambdaDynamoDBClass(mocked_dynamodb_resource)
self.mocked_s3_class = LambdaS3Class(mocked_s3_resource)

Test #1: Verify the code writes the document to S3

Our first test will validate our Lambda function writes the customer letter to an S3 bucket in the correct manner.  We will follow the standard test format of arrange, act, assert when writing this unit test.

Arrange the data we need in the DynamoDB table:

def test_create_letter_in_s3(self) -> None:

self.mocked_dynamodb_class.table.put_item(Item={“PK”:”D#UnitTestDoc”,
“data”:”Unit Test Doc Corpi”})
self.mocked_dynamodb_class.table.put_item(Item={“PK”:”C#UnitTestCust”,
“data”:”Unit Test Customer”})

Act by calling the create_letter_in_s3 function.  During these act calls, the test passes the AWS resources as created in the setUp().

test_return_value = create_letter_in_s3(
dynamo_db = self.mocked_dynamodb_class,
s3=self.mocked_s3_class,
doc_type = “UnitTestDoc”,
cust_id = “UnitTestCust”
)

Assert by reading the data written to the mock S3 bucket, and testing conformity to what we are expecting:

bucket_key = “UnitTestCust/UnitTestDoc.txt”
body = self.mocked_s3_class.bucket.Object(bucket_key).get()[‘Body’].read()

self.assertEqual(test_return_value[“statusCode”], 200)
self.assertIn(“UnitTestCust/UnitTestDoc.txt”, test_return_value[“body”])
self.assertEqual(body.decode(‘ascii’),”Dear Unit Test Customer;nUnit Test Doc Corpi”)

Tests #2 and #3: Data not found error conditions

We can also test error conditions and handling, such as keys not found in the database.  For example, if a customer identifier is submitted, but does not exist in the database lookup, does the logic handle this and return a “Not Found” code of 404?

To test this in test #2, we add data to the mocked DynamoDB table, but then submit a customer identifier that is not in the database.

This test, and a similar test #3 for “Document Types not found”, are implemented in the example test code on GitHub.

Test #4: Validate the handler interface

As the application logic resides in independently tested functions, the Lambda handler function provides only interface validation and function call orchestration.  Therefore, the test for the handler validates that the event is parsed correctly, any functions are invoked as expected, and the return value is passed back.

To emulate the global resource variables and other functions, patch both the global resource classes and logic functions.

@patch(“src.sample_lambda.app.LambdaDynamoDBClass”)
@patch(“src.sample_lambda.app.LambdaS3Class”)
@patch(“src.sample_lambda.app.create_letter_in_s3”)
def test_lambda_handler_valid_event_returns_200(self,
patch_create_letter_in_s3 : MagicMock,
patch_lambda_s3_class : MagicMock,
patch_lambda_dynamodb_class : MagicMock
):

Arrange for the test by setting return values for the patched objects.

patch_lambda_dynamodb_class.return_value = self.mocked_dynamodb_class
patch_lambda_s3_class.return_value = self.mocked_s3_class

return_value_200 = {“statusCode” : 200, “body”:”OK”}
patch_create_letter_in_s3.return_value = return_value_200

We need to provide event data when invoking the Lambda handler.  A good practice is to save test events as separate JSON files, rather than placing them inline as code. In the example project, test events are located in the folder “tests/events/”. During test execution, the event object is created from the JSON file using the utility function named load_sample_event_from_file.

test_event = self.load_sample_event_from_file(“sampleEvent1”)

Act by calling the lambda_handler function.

test_return_value = lambda_handler(event=test_event, context=None)

Assert by ensuring the create_letter_in_s3 function is called with the expected parameters based on the event, and a create_letter_in_s3 function return value is passed back to the caller.  In our example, this value is simply passed with no alterations.

patch_create_letter_in_s3.assert_called_once_with(
dynamo_db=self.mocked_dynamodb_class,
s3=self.mocked_s3_class,
doc_type=test_event[“pathParameters”][“docType”],
cust_id=test_event[“pathParameters”][“customerId”])

self.assertEqual(test_return_value, return_value_200)

Tear Down

The tearDown() method is called immediately after the test method has been run and the result is recorded.  In our example tearDown() method, we clean up any data or state created so the next test won’t be impacted.

Running the unit tests

The unittest Unit testing framework can be run using the Python pytest utility.  To ensure network isolation and verify the unit tests are not accidently connecting to AWS resources, the pytest-socket project provides the ability to disable network communication during a test.

pytest -v –disable-socket -s tests/unit/src/

The pytest command results in a PASSED or FAILED status for each test.  A PASSED status verifies that your unit tests, as written, did not encounter errors or issues,

Conclusion

Unit testing is a software development process in which different parts of an application, called units, are individually and independently tested. Tests validate the quality of the code and confirm that it functions as expected. Other developers can gain familiarity with your code base by consulting the tests. Unit tests reduce future refactoring time, help engineers get up to speed on your code base more quickly, and provide confidence in the expected behaviour.

We’ve seen in this blog how to unit test AWS Lambda functions and mock AWS Services to isolate and test individual logic within our code.

AWS Lambda Powertools for Python has been used in the project to validate hander events.   Powertools provide a suite of utilities for AWS Lambda functions to ease adopting best practices such as tracing, structured logging, custom metrics, idempotency, batching, and more.

Learn more about AWS Lambda testing in our prescriptive test guidance, and find additional test examples on GitHub.  For more serverless learning resources, visit Serverless Land.

About the authors:

Tom Romano

Tom Romano is a Solutions Architect for AWS World Wide Public Sector from Tampa, FL, and assists GovTech and EdTech customers as they create new solutions that are cloud-native, event driven, and serverless. He is an enthusiastic Python programmer for both application development and data analytics. In his free time, Tom flies remote control model airplanes and enjoys vacationing with his family around Florida and the Caribbean.

Kevin Hakanson

Kevin Hakanson is a Sr. Solutions Architect for AWS World Wide Public Sector based in Minnesota. He works with EdTech and GovTech customers to ideate, design, validate, and launch products using cloud-native technologies and modern development practices. When not staring at a computer screen, he is probably staring at another screen, either watching TV or playing video games with his family.