LocalStack : The zero cost aws cloud subscription

Development using aws cloud services requires a subscription. Development and testing can sometimes be a costly affair for some organizations incurring expenses to the tune of around 1000 USD. LocalStack is a more cheaper way of developing and testing code locally before it is deployed to aws cloud.

Architecture

Objective

The objective of this post is demonstrate the following using LocalStack :

LocalStack in Developer Machine

  1. Create an s3 bucket
  2. Create an SQS queue
  3. Setup an event in s3 such that when a file is loaded into the s3 bucket a notification is sent to sqs queue
  4. Create an AWS step function workflow consisting of
    • A lambda that picks up a file from a s3 bucket and delivers it to a SQS queue and
    • the data in the file is inserted into to DynamoDB

LocalStack in Continuous Integration

  1. Write some pytest unit testcases for a lambda
  2. Build the zip and upload it to LocalStack to create the lambda and
  3. Run the pytest unit test cases and generate report. 

Deploying Application to AWS

  1. Spin up a Jenkins in docker that will then
  2. Download the code for Lambda from git
  3. Run a ci/cd pipeline to deploy the artifacts to s3
  4. Run the unit test cases

Reference :

More information can be found at :

https://localstack.cloud/ or

https://github.com/localstack/localstack

All the steps in this post use the free Community Edition of LocalStack. There are also the Pro and Enterprise editions available at a small subscription fee that come with more powerful AWS services like RDS and AWS IoT.

Setting up :

Install and Configure aws cli

D:\localstack> aws configure --profile stack-profile
AWS Access Key ID [None]: covid
AWS Secret Access Key [None]: covid
Default region name [None]: us-east-1
Default output format [None]: json

Install and Configure docker

D:\localstack> docker --version
Docker version 19.03.12, build 48a66213fe

Create the following docker-compose.yml file

version: '2.1'

services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: localstack/localstack
    ports:
      - "4566-4599:4566-4599"
      - "${PORT_WEB_UI-8080}:${PORT_WEB_UI-8080}"
    environment:
      - SERVICES=${SERVICES- }
      - DEBUG=${DEBUG- }
      - DATA_DIR=${DATA_DIR- }
      - PORT_WEB_UI=${PORT_WEB_UI- }
      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- }
      - KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
      - DOCKER_HOST=unix:///var/run/docker.sock
      - HOST_TMP_FOLDER=${TMPDIR}
    volumes:
      - "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
docker-compose up -d

LocalStack in Developer Machine

Create a s3 bucket

aws --profile stack-profile --endpoint-url=http://localhost:4566 s3 mb s3://mulan

Create a sqs queue

aws --profile stack-profile --endpoint-url=http://localhost:4566 sqs create-queue --queue-name mulan

Add an event notification configuration to the s3 bucket using notification.json file

aws --profile stack-profile --endpoint-url=http://localhost:4566 s3api put-bucket-notification-configuration --bucket mulan --notification-configuration file://notification.json

Contents of notification.json file

{
    "QueueConfigurations": [
        {
            "QueueArn": "arn:aws:sqs:us-east-1:000000000000:mulan",
            "Events": [
                "s3:ObjectCreated:*"
            ]
        }
    ]
}

Get the arn above from :

aws --profile stack-profile --endpoint-url=http://localhost:4566 sqs get-queue-attributes --queue-url http://localhost:4566/queue/mulan --attribute-names All

Add a file test.csv to the s3 bucket

aws --profile stack-profile --endpoint-url=http://localhost:4566 s3 cp test.csv s3://mulan

Check if a message is received in sqs using the command

aws --profile stack-profile --endpoint-url=http://localhost:4566 sqs receive-message --queue-url http://localhost:4566/queue/mulan

AWS Step function workflow

Create a lambda.py file with the following code

import urllib.parse
import boto3
import json

# print('Loading function')

HOST = "http://127.0.0.1"
# Get the service resource
# To production it's not necessary inform the "endpoint_url" and "region_name"
s3 = boto3.client('s3',
                  endpoint_url= HOST + ":4566",
                  region_name="us-east-1")
sqs = boto3.client('sqs',
                  endpoint_url= HOST + ":4566",
                  region_name="us-east-1")

def handler(event, context):
    # print("Received event: " + json.dumps(event, indent=2))

    # Get the object from the event and show its content type
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
    url_queue = HOST + ":4566/queue/lambda-tutorial"

    try:

        response = s3.get_object(Bucket=bucket, Key=key)

        deb = {
            "request_id": response['ResponseMetadata']['RequestId'],
            "queue_url": url_queue,
            "key": key,
            "bucket": bucket,
            "message": "aws lambda with localstack..."
        }

        print("#########################################################")
        print("Send Message")
        #Send message to SQS queue
        response = sqs.send_message(
                QueueUrl=deb["queue_url"],
                MessageBody=json.dumps(deb)
        )

        print("response: {}".format(response))

        print("#########################################################")
        print("Receive 10 Messages From SQS Queue")
        response = sqs.receive_message(
            QueueUrl=deb["queue_url"],
            MaxNumberOfMessages=10,
            VisibilityTimeout=0,
            WaitTimeSeconds=0
        )

        print("#########################################################")
        print("Read All Messages From Response")
        messages = response['Messages']
        for message in messages:
            print("Message: {}".format(message))

        print("Final Output: {}".format(json.dumps(response)))
        return json.dumps(response)

    except Exception as e:
        print(e)
        raise e

zip the above lambda.py file into a papi-handler.zip

Create a new lambda function in aws using the command

aws --profile stack-profile --endpoint-url=http://localhost:4566 lambda create-function --function-name papi-handler --runtime python3.8 --handler lambda.handler --memory-size 128 --zip-file fileb://papi-handler.zip --role arn:aws:iam::123456:role/irrelevant
aws --profile stack-profile --endpoint-url=http://localhost:4566 lambda invoke --function-name papi-handler
aws --profile stack-profile --endpoint-url=http://localhost:4566 s3 mb s3://tutorial
aws --profile stack-profile --endpoint-url=http://localhost:4566 s3api put-object --bucket tutorial --key lambda
aws --profile stack-profile --endpoint-url=http://localhost:4566 s3 cp ./test/files/ s3://tutorial/lambda/ --recursive
curl -v http://localhost:4572/tutorial
aws --profile stack-profile --endpoint-url=http://localhost:4566 sqs create-queue --queue-name lambda-tutorial

Create AWS Step function

aws stepfunctions --profile stack-profile --endpoint-url=http://localhost:4566 create-state-machine --definition "{\"Comment\": \"Localstack step function example\",\"StartAt\": \"HelloWorld1\",\"States\": {\"HelloWorld1\": {\"Type\": \"Task\",\"Resource\": \"arn:aws:lambda:us-east-1:000000000000:function:papi-handler\",\"End\": true}}}}}" --name "HelloWorld1" --role-arn "arn:aws:iam::000000000000:role/a-role"

Execute aws step function

aws stepfunctions --profile stack-profile --endpoint-url=http://localhost:4566 start-execution --state-machine arn:aws:states:us-east-1:000000000000:stateMachine:HelloWorld1 --name test
aws stepfunctions --profile stack-profile --endpoint-url=http://localhost:4566 describe-execution --execution-arn arn:aws:states:us-east-1:000000000000:execution:HelloWorld1:test

Note : The above steps contain some minor bugs. I plan to resolve them and update the page over the next couple of days/months whenever I find time off from work.

DynamoDB functions :

#Create a table in DynamoDB
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb create-table --table-name mulan_table  --attribute-definitions AttributeName=first,AttributeType=S AttributeName=second,AttributeType=N --key-schema AttributeName=first,KeyType=HASH AttributeName=second,KeyType=RANGE --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

#List all tables in DynamoDB
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb list-tables

#Describe a table in DynamoDB
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb describe-table --table-name mulan_table

#Put item into a table
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb put-item --table-name test_table  --item "{\"first\":{\"S\":\"Eddy\"},\"second\":{\"N\":\"1542\"}}"

#Put another item
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb put-item --table-name test_table  --item "{\"first\":{\"S\":\"James\"},\"second\":{\"N\":\"6097\"}}"

#Scan a table
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb scan --table-name mulan

#Perform get item on a table
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb get-item --table-name test_table  --key "{\"first\":{\"S\":\"Eddy\"},\"second\":{\"N\":\"1542\"}}"

#Perform query on a table
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb query --table-name test_table --projection-expression "#first, #second" --key-condition-expression "#first = :value" --expression-attribute-values "{\":value\" : {\"S\":\"James\"}}" --expression-attribute-names "{\"#first\":\"first\", \"#second\":\"second\"}"

LocalStack in Continuous Integration

In this section we shall write the following :

Lambda.py :  A simple hello world lambda

Testutils.py : A utility to create the lambda function

Test_lambda.py : A utility that runs the unit test cases on lambda.py

  • Create a lambda.py as per the source code provided below
import logging

LOGGER = logging.getLogger()
LOGGER.setLevel(logging.INFO)


def handler(event, context):
    LOGGER.info("I've been called!")
    return {
        "message": "Hello LocalStack pytest!"
    }
  • Create a testutils.py as per the source code provided below
import json
import os
from zipfile import ZipFile

import boto3
import botocore

CONFIG = botocore.config.Config(retries={'max_attempts': 0})
LAMBDA_ZIP = './lambda.zip'


def get_lambda_client():
    return boto3.client(
        'lambda',
        aws_access_key_id='covid',
        aws_secret_access_key='covid',
        region_name='us-east-1',
        endpoint_url='http://localhost:4574',
        config=CONFIG
    )


def create_lambda_zip(function_name):
    with ZipFile(LAMBDA_ZIP, 'w') as z:
        z.write(function_name + '.py')


def create_lambda(function_name):
    lambda_client = get_lambda_client()
    create_lambda_zip(function_name)
    with open(LAMBDA_ZIP, 'rb') as f:
        zipped_code = f.read()
    lambda_client.create_function(
        FunctionName=function_name,
        Runtime='python3.6',
        Role='role',
        Handler=function_name + '.handler',
        Code=dict(ZipFile=zipped_code)
    )


def delete_lambda(function_name):
    lambda_client = get_lambda_client()
    lambda_client.delete_function(
        FunctionName=function_name
    )
    os.remove(LAMBDA_ZIP)


def invoke_function_and_get_message(function_name):
    lambda_client = get_lambda_client()
    response = lambda_client.invoke(
        FunctionName=function_name,
        InvocationType='RequestResponse'
    )
    return json.loads(
        response['Payload']
        .read()
        .decode('utf-8')
    )

Create a test_lambda.py as per the source code provided below :

import testutils
from unittest import TestCase


class Test(TestCase):

    @classmethod
    def setup_class(cls):
        print('\r\nSetting up the class')
        testutils.create_lambda('lambda')

    @classmethod
    def teardown_class(cls):
        print('\r\nTearing down the class')
        testutils.delete_lambda('lambda')

    def test_that_lambda_returns_correct_message(self):
        payload = testutils.invoke_function_and_get_message('lambda')
        self.assertEqual(payload['message'], 'Hello LocalStack pytest!')

Run

pytest -s .

To check the output below

Deploying Application to AWS

For deploying the application to aws we need to first setup the CI/CD process locally. Lets start with Jenkins

Spinup Jenkins in a docker container locally

sudo yum install python3
sudo yum install python3-pip
sudo pip3 install -U pytest
pip3 install pytest-html
sudo pip3 install pylint


git clone https://github.com/4OH4/jenkins-docker.git

cd jenkins-docker

docker build -t jenkins-docker .

docker run -it -p 8080:8080 -p 50000:50000 -v jenkins_home:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock --restart unless-stopped jenkins-docker

Create a new pipeline job

  1. Click New Item on your Jenkins home page, enter a name for your (pipelinejob, select Pipeline, and click OK.
  2. In the Script text area of the configuration screen configure the below script.
pipeline {
	environment {

		AWS_ID = credentials("awsaccesskey")
		AWS_ACCESS_KEY_ID = "XXXXXXXXXXXXXX"
		AWS_SECRET_ACCESS_KEY = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

	}

	agent any

	stages {
		stage('checkout') {
			steps {
				git branch: 'master', credentialsId: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx', url: 'https://github.com/mulan/cfcode.git'

			}
		}


		stage('pylint') {
			steps {
				sh ''
				'
				cd $WORKSPACE / hello_world

					/
					usr / local / bin / pylint app.py > pylintreport.txt--exit - zero

				''
				'


			}
		}

		stage('validate cloudformation template') {
			steps {
				sh ''
				'
				aws--region us - east - 1 cloudformation validate - template--template - body file: //template.yaml


					''
				'

			}
		}

		stage('zip the sourcefiles') {
			steps {
				sh ''
				'
				zip - r hello_world.zip hello_world


				''
				'

			}
		}


		stage('push the sourfiles to S3') {
			steps {
				sh ''
				'

				aws s3 cp hello_world.zip s3: //mulan-folder/hello_world.zip


					''
				'

			}
		}

		stage('SamPackage') {
			steps {
				sh ''
				'

				aws--region us - east - 1 cloudformation package--template template.yaml--s3 - bucket awsdeploy001--output - template - file template.packaged.yml

				''
				'

			}
		}

		stage('deploy') {
			steps {
				sh ''
				'

				aws--region us - east - 1 cloudformation deploy--template - file template.packaged.yml--stack - name mymulanstack--capabilities CAPABILITY_IAM

				''
				'

			}
		}
		stage('unitest') {
			steps {
				sh ''
				'

				cd $WORKSPACE / hello_world


					/
					usr / local / bin / pytest test_handler.py--html = Report.html

				''
				'

			}
		}

	}
}

Use the below template.yaml file for the above pipeline

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  AWS

  Sample SAM Template for AWS

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: "s3://mulan-folder/hello_world.zip"
      Handler: app.lambda_handler
      Runtime: python3.8
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

Check the stage view and trigger the Jenkins job as per screenshots below

After the jenkins job succeeds the reports are published in the following format

pylint report :

************* Module app
app.py:14:0: C0301: Line too long (166/100) (line-too-long)
app.py:25:0: C0301: Line too long (118/100) (line-too-long)
app.py:1:0: C0114: Missing module docstring (missing-module-docstring)
app.py:6:19: W0613: Unused argument 'event' (unused-argument)
app.py:6:26: W0613: Unused argument 'context' (unused-argument)

--------------------------------------------------------------------
Your code has been rated at -6.67/10 (previous run: -6.67/10, +0.00)

Test report

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: