06.05, Katowice AWS Summit Poland
12 min read

Say "hello" to a CloudFormation stack via API Gateway

Sometimes you need to postpone your custom, long-term ideas for short-term, already running ones.

Sometimes you need to postpone your custom, long-term ideas for short-term already running ones. The KISS (Keep It Simple, Stupid) principle might be your good friend when you know you should follow it. For one of our clients we had to deliver a rapid way of creating a CloudFormation stack and checking its state, whether it has failed or completed.

(In case you’re not familiar with AWS CloudFormation — it allows to build AWS resources using templates. A template is a configuration file (YAML or JSON) for provisioning all your AWS resources.)

In case of success they would be developed further and integrated with a third party, non-AWS tool. The main challenge was that the solution had to be delivered at short notice (just one day). The customer wanted to follow Infrastructure as Code principle for their environment and to have an open door for third-party tool integration which would be a kind of ordering portal for its internal customers. Cool — we didn’t waste time and rolled up our sleeves to come up with an idea.

Call me and I’ll tell you the truth

Together, we decided that it was time to explore AWS and redesign our client’s existing architecture. We needed the ability to expose an AWS service without knowing anything about it. While they now have a proper CI/CD pipeline, the story started from something like in the picture below:

We went in tandem with AWS API Gateway as a ‘front door’ for functionality from our back-end service. Moreover, together with AWS Lambda as our two functions, API Gateway forms the app-facing part of the AWS serverless infrastructure. The end user only sees the API endpoint URLs which they’re facing via proper HTTP calls. At the end of the line there’s a CloudFormation service reached via Lambdas. The smaller, the simpler and therefore functions have been separated to serve only one task.

Serverless for rapid implementations

To make it all versionable and easy to maintain, we used Serverless Framework for provisioning. Check out our article about getting started with the Serverless Framework, if you missed it.

I remember the nauseating sensation when I deployed a serverless environment for the first time. CloudFormation/Terraform are nice alternatives but trust me, once you start using Serverless Framework, you won’t look back.

A snippet from serverless.yml file to present the main functionality:

serverless.yml yaml
functions:  cf-creator-get:    name: ${self:custom.app_acronym}-cf-get    description: Checks the state of CloudFormation Stack    handler: handler_get.lambda_handler    role: CloudFormationRole    # environment:    #   region: ${self:custom.region}    tags:      Name: ${self:custom.app_acronym}-cf-get      Project: serverless      Environment: dev    events:      - http:        path: /state/{stackname}        method: get        private: true        request:          parameters:            paths:              stackname: true  cf-creator-post:    name: ${self:custom.app_acronym}-cf-post    description: Create CloudFormation stacks    handler: handler_post.lambda_handler    role: CloudFormationRole    # environment:    #   region: ${self:custom.region}    tags:      Name: ${self:custom.app_acronym}-cf-post      Project:  serverless      Environment: dev    events:      - http:          path: /create          method: post          private: true

NOTE: without the private flag set to true, anyone would be able to post data to our API endpoint, which is a bad idea. With a setting of true, our API endpoint requires an API key defined in this part:

provider:  name: aws  apiKeys:    - ${self:custom.app_acronym}-apikey  runtime: python2.7  region: eu-west-1  memorySize: 128  timeout: 60 # optional, in seconds  versionFunctions: true  tags: # Optional service wide function tags    Owner: chaosgears    ContactPerson: chaosgears    Project: serverless    Environment: dev

Then Serverless Framework is going to handle the rest of the creation process.

For simplicity and for testing purposes, I’ve added a cloudformation:* action within the following policies. For actual production environments you should follow the rule of least privilege and allow only strictly necessary actions. This way it’s more feasible to control and maintain internal calls among services.

resources:  Resources:    CloudFormationRole:      Type: AWS::IAM::Role      Properties:        AssumeRolePolicyDocument:          Version: "2012-10-17"          Statement:            - Effect: Allow              Principal:                Service:                  - lambda.amazonaws.com              Action:                - sts:AssumeRole        Path: /        Policies:          -            PolicyName: Serverless-CF-Creator            PolicyDocument:              Version: "2012-10-17"              Statement:                -                  Effect: Allow                  Action:                    - lambda:ListFunctions                    - lambda:InvokeFunction                  Resource:                    - "*"                -                  Effect: Allow                  Action:                    - s3:*                  Resource:                    - "arn:aws:s3:::${self:custom.s3}"                    - "arn:aws:s3:::${self:custom.s3}/*"                -                  Effect: Allow                  Action:                    - cloudformation:*                  Resource:                    - "*"                -                  Effect: Allow                  Action:                    - logs:CreateLogStream                    - logs:PutLogEvents                    - logs:PutRetentionPolicy                    - logs:CreateLogGroup                    - logs:DescribeLogStreams                    - logs:DeleteLogGroup                  Resource:                    - "arn:aws:logs:*:*:*"

Backend brothers

Some of our attentive reader noticed that we hadn’t analyzed Lambdas’ code and how they take care of events they’re receiving. I’ve created a small class called CF_Invoker to collect two methods for creation and checking state respectively:

class CF_Invoker(object):    def __init__(self, service):        try:            self.client = boto3.client(service)        except ClientError as err:            logging.critical("----ClientError: {0}".format(err))    def cf_create_stack(self, stackname, templateurl='https://s3-xxx.amazonaws.com/mybucket/out/output.yaml'):        try:            response = self.client.create_stack(                StackName=stackname,                TemplateURL=templateurl,                DisableRollback=False,                TimeoutInMinutes=10,                Capabilities=['CAPABILITY_IAM']            )            print(response)            if response['ResponseMetadata']['HTTPStatusCode'] < 300:                logging.info("---CloudFormation creation SUCCEDED: {0}".format(json.dumps(response)))                return {                    "statusCode": response['ResponseMetadata']['HTTPStatusCode'],                    "body": "Successfully created STACKNAME: {0}, STACKID: {1}".format(stackname, response['StackId'])                }            else:                logging.critical("---Unexpected error: {0}".format(json.dumps(response)))                return {                    "statusCode": response['ResponseMetadata']['HTTPStatusCode'],                    "body": "Error: {0}".format(response['ResponseMetadata']['HTTPStatusCode'])                }        except ValueError as err:            logging.critical("----Value error: {0}".format(err))        except ClientError as err:            logging.critical("----Client error: {0}".format(err))    def cf_delete_stack(self, stackname):        try:            response = self.client.delete_stack(                StackName=stackname            )            print(response)            if response['ResponseMetadata']['HTTPStatusCode'] < 300:                logging.info("---CloudFormation deletion SUCCEDED: {0}".format(json.dumps(response)))                return {                    "statusCode": response['ResponseMetadata']['HTTPStatusCode'],                    "body": "Successfully deleted STACKNAME: {0}".format(stackname)                }            else:                logging.critical("---Unexpected error: {0}".format(json.dumps(response)))                return {                    "statusCode": response['ResponseMetadata']['HTTPStatusCode'],                    "body": "Error: {0}".format(response['ResponseMetadata']['HTTPStatusCode'])                }        except ValueError as err:            logging.critical("----Value error: {0}".format(err))        except ClientError as err:            logging.critical("----Client error: {0}".format(err))    def cf_stack_state(self, stackname):        try:            response = self.client.describe_stacks(StackName=stackname)            if response['ResponseMetadata']['HTTPStatusCode'] < 300:                logging.info("----CloudFormation STACK STATUS: {0}".format(response['Stacks'][0]['StackStatus']))                return {                    "statusCode": response['ResponseMetadata']['HTTPStatusCode'],                    "body": "CloudFormation STACKNAME: {0}, STATE: {1}".format(stackname, response['Stacks'][0]['StackStatus'])                }            else:                logging.critical("---Unexpected error: {0}".format(json.dumps(response)))                return {                    "statusCode": response['ResponseMetadata']['HTTPStatusCode'],                    "body": "CloudFormation STACKNAME: {0}, STATE: {1}".format(stackname, response['Stacks'][0]['StackStatus'])                }        except ClientError as err:            logging.critical("----Client error: {0}".format(err))

Then, having a class prepared, two following handlers have been added. They’re taking event info gathered from API Gateway and do their jobs. For me it was also a nice way to test HTTP requests with embedded parameters, forwarded by API Gateway and finally served by Lambdas. One thing to be added is that returned response value pasted in handler functions is being return to API Gateway which is then forwarded to the end user.

import loggingimport jsonfrom cf_invoker import CF_Invokerdef lambda_handler(event, context):   logger = logging.getLogger()   logger.setLevel(logging.INFO)   logger.info("Event: {0}".format(event))   payload = json.loads(event['body'])   cf = CF_Invoker('cloudformation')   response = cf.cf_create_stack(payload['stackname'],payload['templateurl'])   return response

… and …

import loggingfrom cf_invoker import CF_Invokerdef lambda_handler(event, context):   logger = logging.getLogger()   logger.setLevel(logging.INFO)   logger.info("Event: {0}".format(event))   payload = event['pathParameters']   cf = CF_Invoker('cloudformation')   response = cf.cf_stack_state(payload['stackname'])   return response

To deploy all of that you just need to type using necessary AWS profile bound with your AWS account:

sls plugin install -n serverless-python-requirementssls deploy --aws-profile AWS_PROFILE

After two or three minutes you’ll get you stack deployed on particular AWS region and, of course, your Lambdas, API key and API Gateway. Everything implemented via single file.

Service Informationservice: articlestage: devregion: eu-west-1stack: article-devapi keys:  article-apikey: YOUR_NEW_API_KEYendpoints:  GET - https://API_ENDPOINT/dev/state/{stackname}  POST - https://API_ENDPOINT/dev/createfunctions:  cf-creator-get: article-dev-cf-creator-get  cf-creator-post: article-dev-cf-creator-postServerless: Publish service to Serverless Platform...Service successfully published! Your service details are available at:https://platform.serverless.com/services/USERNAME/chaosgears

Call my API

Great! We’ve gone through the deployment part, but generally the usage of Serverless Framework saves your time and makes life much easier. To test whether everything works fine, just type during the creation of the CF Stack:

curl -H "x-api-key: YOUR_NEW_API_KEY" -d '{"stackname": "YOUR_STACK_NAME", "templateurl":"https://S3_HTTPS_URL/S3_BUCKETNAME/PATH/FILENAME.yaml"}' https://API_ENDPOINT/dev/create

You should receive a response like the following if CloudFormation stack has been deployed properly:

Successfully created STACKNAME:: YOUR_STACK_NAME, STACKID: arn:aws:cloudformation:xxxx

To get the state of a new CF stack via API_ENDPOINT type:

curl -H "x-api-key: YOUR_NEW_API_KEY" https://API_ENDPOINT/dev/state/YOUR_STACK_NAME

The response you get should look like this:

CloudFormation STACKNAME: YOUR_STACK_NAME, STATE: STACK_STATE

And finally, to delete the stack:

curl -H "x-api-key: YOUR_NEW_API_KEY" -d '{"stackname": "YOUR_STACK_NAME"}' https://API_ENDPOINT/dev/delete

The response you get should look like this:

Successfully deleted STACKNAME: YOUR_STACK_NAME

Where this path leads…

Maybe our example doesn’t fit in all your needs but take a look at it from a different angle. We’ve combined 3 AWS services, fully serverless (it doesn’t mean that it’s self-service) and we were able to set up the infrastructure with a simple HTTP POST request. Let’s picture people having no idea about AWS services but are eager to learn and have a strong resolve to automate some bottlenecked processes of provisioning new AWS environments after ordering them in their internal portal.

Obviously serverless doesn’t solve all your problems — sometimes an event-based pattern brings more challenges than a standard approach, but examples like ours show that it’s a good starting point if you never worked with serverless and want to find something easy to test and develop your integration idea.

Let's talk about your project

We'd love to answer your questions and help you thrive in the cloud.