LocalStack : The zero cost aws cloud subscription

Development using aws cloud services requires a subscription. Development and testing can sometimes be a costly affair for some organizations incurring expenses to the tune of around 1000 USD. LocalStack is a more cheaper way of developing and testing code locally before it is deployed to aws cloud.

Architecture

Objective

The objective of this post is demonstrate the following using LocalStack :

LocalStack in Developer Machine

Create an s3 bucket
Create an SQS queue
Setup an event in s3 such that when a file is loaded into the s3 bucket a notification is sent to sqs queue
Create an AWS step function workflow consisting of
- A lambda that picks up a file from a s3 bucket and delivers it to a SQS queue and
- the data in the file is inserted into to DynamoDB

LocalStack in Continuous Integration

Write some pytest unit testcases for a lambda
Build the zip and upload it to LocalStack to create the lambda and
Run the pytest unit test cases and generate report.

Deploying Application to AWS

Spin up a Jenkins in docker that will then
Download the code for Lambda from git
Run a ci/cd pipeline to deploy the artifacts to s3
Run the unit test cases

Reference :

More information can be found at :

https://localstack.cloud/ or

https://github.com/localstack/localstack

All the steps in this post use the free Community Edition of LocalStack. There are also the Pro and Enterprise editions available at a small subscription fee that come with more powerful AWS services like RDS and AWS IoT.

Setting up :

Install and Configure aws cli

D:\localstack> aws configure --profile stack-profile
AWS Access Key ID [None]: covid
AWS Secret Access Key [None]: covid
Default region name [None]: us-east-1
Default output format [None]: json

Install and Configure docker

D:\localstack> docker --version
Docker version 19.03.12, build 48a66213fe

Create the following docker-compose.yml file

version: '2.1'

services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: localstack/localstack
    ports:
      - "4566-4599:4566-4599"
      - "${PORT_WEB_UI-8080}:${PORT_WEB_UI-8080}"
    environment:
      - SERVICES=${SERVICES- }
      - DEBUG=${DEBUG- }
      - DATA_DIR=${DATA_DIR- }
      - PORT_WEB_UI=${PORT_WEB_UI- }
      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- }
      - KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
      - DOCKER_HOST=unix:///var/run/docker.sock
      - HOST_TMP_FOLDER=${TMPDIR}
    volumes:
      - "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

docker-compose up -d

LocalStack in Developer Machine

Create a s3 bucket

aws --profile stack-profile --endpoint-url=http://localhost:4566 s3 mb s3://mulan

Create a sqs queue

aws --profile stack-profile --endpoint-url=http://localhost:4566 sqs create-queue --queue-name mulan

Add an event notification configuration to the s3 bucket using notification.json file

aws --profile stack-profile --endpoint-url=http://localhost:4566 s3api put-bucket-notification-configuration --bucket mulan --notification-configuration file://notification.json

Contents of notification.json file

{
    "QueueConfigurations": [
        {
            "QueueArn": "arn:aws:sqs:us-east-1:000000000000:mulan",
            "Events": [
                "s3:ObjectCreated:*"
            ]
        }
    ]
}

Get the arn above from :

aws --profile stack-profile --endpoint-url=http://localhost:4566 sqs get-queue-attributes --queue-url http://localhost:4566/queue/mulan --attribute-names All

Add a file test.csv to the s3 bucket

aws --profile stack-profile --endpoint-url=http://localhost:4566 s3 cp test.csv s3://mulan

Check if a message is received in sqs using the command

aws --profile stack-profile --endpoint-url=http://localhost:4566 sqs receive-message --queue-url http://localhost:4566/queue/mulan

AWS Step function workflow

Create a lambda.py file with the following code

import urllib.parse
import boto3
import json

# print('Loading function')

HOST = "http://127.0.0.1"
# Get the service resource
# To production it's not necessary inform the "endpoint_url" and "region_name"
s3 = boto3.client('s3',
                  endpoint_url= HOST + ":4566",
                  region_name="us-east-1")
sqs = boto3.client('sqs',
                  endpoint_url= HOST + ":4566",
                  region_name="us-east-1")

def handler(event, context):
    # print("Received event: " + json.dumps(event, indent=2))

    # Get the object from the event and show its content type
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
    url_queue = HOST + ":4566/queue/lambda-tutorial"

    try:

        response = s3.get_object(Bucket=bucket, Key=key)

        deb = {
            "request_id": response['ResponseMetadata']['RequestId'],
            "queue_url": url_queue,
            "key": key,
            "bucket": bucket,
            "message": "aws lambda with localstack..."
        }

        print("#########################################################")
        print("Send Message")
        #Send message to SQS queue
        response = sqs.send_message(
                QueueUrl=deb["queue_url"],
                MessageBody=json.dumps(deb)
        )

        print("response: {}".format(response))

        print("#########################################################")
        print("Receive 10 Messages From SQS Queue")
        response = sqs.receive_message(
            QueueUrl=deb["queue_url"],
            MaxNumberOfMessages=10,
            VisibilityTimeout=0,
            WaitTimeSeconds=0
        )

        print("#########################################################")
        print("Read All Messages From Response")
        messages = response['Messages']
        for message in messages:
            print("Message: {}".format(message))

        print("Final Output: {}".format(json.dumps(response)))
        return json.dumps(response)

    except Exception as e:
        print(e)
        raise e

zip the above lambda.py file into a papi-handler.zip

Create a new lambda function in aws using the command

aws --profile stack-profile --endpoint-url=http://localhost:4566 lambda create-function --function-name papi-handler --runtime python3.8 --handler lambda.handler --memory-size 128 --zip-file fileb://papi-handler.zip --role arn:aws:iam::123456:role/irrelevant

aws --profile stack-profile --endpoint-url=http://localhost:4566 lambda invoke --function-name papi-handler

aws --profile stack-profile --endpoint-url=http://localhost:4566 s3 mb s3://tutorial

aws --profile stack-profile --endpoint-url=http://localhost:4566 s3api put-object --bucket tutorial --key lambda

aws --profile stack-profile --endpoint-url=http://localhost:4566 s3 cp ./test/files/ s3://tutorial/lambda/ --recursive

curl -v http://localhost:4572/tutorial

aws --profile stack-profile --endpoint-url=http://localhost:4566 sqs create-queue --queue-name lambda-tutorial

Create AWS Step function

aws stepfunctions --profile stack-profile --endpoint-url=http://localhost:4566 create-state-machine --definition "{\"Comment\": \"Localstack step function example\",\"StartAt\": \"HelloWorld1\",\"States\": {\"HelloWorld1\": {\"Type\": \"Task\",\"Resource\": \"arn:aws:lambda:us-east-1:000000000000:function:papi-handler\",\"End\": true}}}}}" --name "HelloWorld1" --role-arn "arn:aws:iam::000000000000:role/a-role"

Execute aws step function

aws stepfunctions --profile stack-profile --endpoint-url=http://localhost:4566 start-execution --state-machine arn:aws:states:us-east-1:000000000000:stateMachine:HelloWorld1 --name test

aws stepfunctions --profile stack-profile --endpoint-url=http://localhost:4566 describe-execution --execution-arn arn:aws:states:us-east-1:000000000000:execution:HelloWorld1:test

Note : The above steps contain some minor bugs. I plan to resolve them and update the page over the next couple of days/months whenever I find time off from work.

DynamoDB functions :

#Create a table in DynamoDB
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb create-table --table-name mulan_table  --attribute-definitions AttributeName=first,AttributeType=S AttributeName=second,AttributeType=N --key-schema AttributeName=first,KeyType=HASH AttributeName=second,KeyType=RANGE --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

#List all tables in DynamoDB
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb list-tables

#Describe a table in DynamoDB
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb describe-table --table-name mulan_table

#Put item into a table
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb put-item --table-name test_table  --item "{\"first\":{\"S\":\"Eddy\"},\"second\":{\"N\":\"1542\"}}"

#Put another item
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb put-item --table-name test_table  --item "{\"first\":{\"S\":\"James\"},\"second\":{\"N\":\"6097\"}}"

#Scan a table
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb scan --table-name mulan

#Perform get item on a table
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb get-item --table-name test_table  --key "{\"first\":{\"S\":\"Eddy\"},\"second\":{\"N\":\"1542\"}}"

#Perform query on a table
aws --profile stack-profile --endpoint-url=http://localhost:4566 dynamodb query --table-name test_table --projection-expression "#first, #second" --key-condition-expression "#first = :value" --expression-attribute-values "{\":value\" : {\"S\":\"James\"}}" --expression-attribute-names "{\"#first\":\"first\", \"#second\":\"second\"}"

LocalStack in Continuous Integration

In this section we shall write the following :

Lambda.py : A simple hello world lambda

Testutils.py : A utility to create the lambda function

Test_lambda.py : A utility that runs the unit test cases on lambda.py

Create a lambda.py as per the source code provided below

import logging

LOGGER = logging.getLogger()
LOGGER.setLevel(logging.INFO)


def handler(event, context):
    LOGGER.info("I've been called!")
    return {
        "message": "Hello LocalStack pytest!"
    }

Create a testutils.py as per the source code provided below

import json
import os
from zipfile import ZipFile

import boto3
import botocore

CONFIG = botocore.config.Config(retries={'max_attempts': 0})
LAMBDA_ZIP = './lambda.zip'


def get_lambda_client():
    return boto3.client(
        'lambda',
        aws_access_key_id='covid',
        aws_secret_access_key='covid',
        region_name='us-east-1',
        endpoint_url='http://localhost:4574',
        config=CONFIG
    )


def create_lambda_zip(function_name):
    with ZipFile(LAMBDA_ZIP, 'w') as z:
        z.write(function_name + '.py')


def create_lambda(function_name):
    lambda_client = get_lambda_client()
    create_lambda_zip(function_name)
    with open(LAMBDA_ZIP, 'rb') as f:
        zipped_code = f.read()
    lambda_client.create_function(
        FunctionName=function_name,
        Runtime='python3.6',
        Role='role',
        Handler=function_name + '.handler',
        Code=dict(ZipFile=zipped_code)
    )


def delete_lambda(function_name):
    lambda_client = get_lambda_client()
    lambda_client.delete_function(
        FunctionName=function_name
    )
    os.remove(LAMBDA_ZIP)


def invoke_function_and_get_message(function_name):
    lambda_client = get_lambda_client()
    response = lambda_client.invoke(
        FunctionName=function_name,
        InvocationType='RequestResponse'
    )
    return json.loads(
        response['Payload']
        .read()
        .decode('utf-8')
    )

Create a test_lambda.py as per the source code provided below :

import testutils
from unittest import TestCase


class Test(TestCase):

    @classmethod
    def setup_class(cls):
        print('\r\nSetting up the class')
        testutils.create_lambda('lambda')

    @classmethod
    def teardown_class(cls):
        print('\r\nTearing down the class')
        testutils.delete_lambda('lambda')

    def test_that_lambda_returns_correct_message(self):
        payload = testutils.invoke_function_and_get_message('lambda')
        self.assertEqual(payload['message'], 'Hello LocalStack pytest!')

Run

pytest -s .

To check the output below

Deploying Application to AWS

For deploying the application to aws we need to first setup the CI/CD process locally. Lets start with Jenkins

Spinup Jenkins in a docker container locally

sudo yum install python3
sudo yum install python3-pip
sudo pip3 install -U pytest
pip3 install pytest-html
sudo pip3 install pylint


git clone https://github.com/4OH4/jenkins-docker.git

cd jenkins-docker

docker build -t jenkins-docker .

docker run -it -p 8080:8080 -p 50000:50000 -v jenkins_home:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock --restart unless-stopped jenkins-docker

Create a new pipeline job

Click New Item on your Jenkins home page, enter a name for your (pipeline) job, select Pipeline, and click OK.
In the Script text area of the configuration screen configure the below script.

pipeline {
	environment {

		AWS_ID = credentials("awsaccesskey")
		AWS_ACCESS_KEY_ID = "XXXXXXXXXXXXXX"
		AWS_SECRET_ACCESS_KEY = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

	}

	agent any

	stages {
		stage('checkout') {
			steps {
				git branch: 'master', credentialsId: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx', url: 'https://github.com/mulan/cfcode.git'

			}
		}


		stage('pylint') {
			steps {
				sh ''
				'
				cd $WORKSPACE / hello_world

					/
					usr / local / bin / pylint app.py > pylintreport.txt--exit - zero

				''
				'


			}
		}

		stage('validate cloudformation template') {
			steps {
				sh ''
				'
				aws--region us - east - 1 cloudformation validate - template--template - body file: //template.yaml


					''
				'

			}
		}

		stage('zip the sourcefiles') {
			steps {
				sh ''
				'
				zip - r hello_world.zip hello_world


				''
				'

			}
		}


		stage('push the sourfiles to S3') {
			steps {
				sh ''
				'

				aws s3 cp hello_world.zip s3: //mulan-folder/hello_world.zip


					''
				'

			}
		}

		stage('SamPackage') {
			steps {
				sh ''
				'

				aws--region us - east - 1 cloudformation package--template template.yaml--s3 - bucket awsdeploy001--output - template - file template.packaged.yml

				''
				'

			}
		}

		stage('deploy') {
			steps {
				sh ''
				'

				aws--region us - east - 1 cloudformation deploy--template - file template.packaged.yml--stack - name mymulanstack--capabilities CAPABILITY_IAM

				''
				'

			}
		}
		stage('unitest') {
			steps {
				sh ''
				'

				cd $WORKSPACE / hello_world


					/
					usr / local / bin / pytest test_handler.py--html = Report.html

				''
				'

			}
		}

	}
}

Use the below template.yaml file for the above pipeline

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  AWS

  Sample SAM Template for AWS

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: "s3://mulan-folder/hello_world.zip"
      Handler: app.lambda_handler
      Runtime: python3.8
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

Check the stage view and trigger the Jenkins job as per screenshots below

After the jenkins job succeeds the reports are published in the following format

pylint report :

************* Module app
app.py:14:0: C0301: Line too long (166/100) (line-too-long)
app.py:25:0: C0301: Line too long (118/100) (line-too-long)
app.py:1:0: C0114: Missing module docstring (missing-module-docstring)
app.py:6:19: W0613: Unused argument 'event' (unused-argument)
app.py:6:26: W0613: Unused argument 'context' (unused-argument)

--------------------------------------------------------------------
Your code has been rated at -6.67/10 (previous run: -6.67/10, +0.00)

Test report

2 thoughts on “LocalStack : The zero cost aws cloud subscription”

Ambi says:

May 17, 2022 at 6:50 pm

Hi! Thanks for this blog, very helpful !
I am trying to run my localstack scripts ((written in Java) in jenkins but unable to figure out the flow. Do you also have any reference for java (I have zero python knowledge 😦 )

LikeLike

1. Raghu Tirnellai says:
  
  August 17, 2022 at 10:27 am
  
  Inorder to work with localstack you would need to have more knowledge on aws cli rather than python. The code to use the aws services can still be written in java. For example lets say you need to write a pipeline that
  1. pulls data from a s3 bucket,
  2. pushes the data to a sqs queue and
  3. reads the same data at the other end of the queue , performs some transformations and
  4. writes the transformed data to a Mysql table in RDS.
  
  Inorder to spin up the s3 , sqs and RDS services in localstack you need aws cli knowledge,
  Inorder to write the code to pull data in s3, write the producer and consumer for sqs, and insert data into mysql , you have aws sdk in java available.
  Therefore you dont need knowledge of python. Nevertheless python is a very important language when it comes to cloud based development and for data analytics. Learning python will definitely help. You might have a inertia initially when you move over from java to python but once you are fully into python you might never want to go back to java and c#.
  
  LikeLike

Architecture

Objective

LocalStack in Developer Machine

LocalStack in Continuous Integration

Deploying Application to AWS

Reference :

Setting up :

LocalStack in Developer Machine

Create a s3 bucket

Create a sqs queue

Add an event notification configuration to the s3 bucket using notification.json file

Add a file test.csv to the s3 bucket

Check if a message is received in sqs using the command

AWS Step function workflow

Create AWS Step function

Execute aws step function

DynamoDB functions :

LocalStack in Continuous Integration

Deploying Application to AWS

Spinup Jenkins in a docker container locally

Create a new pipeline job

Check the stage view and trigger the Jenkins job as per screenshots below

Share this:

Like this:

Related

2 thoughts on “LocalStack : The zero cost aws cloud subscription”

Leave a Reply Cancel reply