AWS

Say ‘Hello’ to CloudFormation Stack via API Gateway

By 11/22/2018 No Comments

Simplicity Over Perfection

Sometimes you need to postpone your custom, long-term ideas for short-term already running ones. The principle called KISS (Keep It Simple, Stupid), personally I’m a big fan of simplicity, might be your good friend when you know you should follow it. For one of our customer’s we had to deliver a rapid the way of creating the CloudFormation Stack and checking its state, whether it has failed or completed. (In case you’re not familiar with the AWS CloudFormation service – it allows to build AWS resources using templates. A template is a configuration file (YAML or JSON) for provisioning all your AWS resources.)

In case of success they would be developed further and integrated with a third-party, non AWS tool. The main challenge was that the solution had to be delivered at short notice (just one day). The customer wanted to follow Infrastructure as Code principle for their environment and to have an open door for third-party tool integration which would be a kind of ordering portal for its internal customers. Cool – we didn’t waste our time and rolled up our sleeves to come up with the idea.

‘Call me and I’ll tell you the truth’

Together, we decided that this is the time for AWS exploration and redesign but the real question how to expose AWS service without knowing anything about it. Now they’ve got a full CI/CD pipeline but the story started from something like in the picture below:

We went in tandem with AWS API Gateway as a ‘front door’ for functionality from our back-end service. Moreover, together with AWS Lambda as our two functions, the API Gateway forms the app-facing part of the AWS serverless infrastructure. The end user only sees the API endpoint URLs which they’re facing via proper HTTP calls. At the end of the line there’s a CloudFormation service reached via Lambdas. The smaller, the simpler and therefore functions have been separated to serve only one task.

cf-creator-get - reached via HTTP GET calls, returns STATE of CloudFormation Stack
cf-creator-post - reached via HTTP POST calls, creates new CloudFormation Stack

Serverless for Rapid Implementations

To make it all versionable and easy to maintain Serverless Framework was used for provisioning. In case you missed my article about first steps with this awesome framework (Link) check it out. I remember the nauseating sensation when I had been deploying serverless environment for the first time. CloudFormation/Terraform are nice alternatives but trust me, you won’t use anything else the day you start using Serverless Framework (of course they’ve got some areas in the development stage but anyway it’s worth of diving in to). A snippet from serverless.yml file to present the main functionality:

functions:
  cf-creator-get:
    name: ${self:custom.app_acronym}-cf-get
    description: Checks the state of CloudFormation Stack
    handler: handler_get.lambda_handler
    role: CloudFormationRole
    # environment:
    #   region: ${self:custom.region}
    tags:
      Name: ${self:custom.app_acronym}-cf-get
      Project: serverless
      Environment: dev
    events:
      - http:
          path: /state/{stackname}
          method: get
          private: true
          request:
            parameters:
              paths:
                stackname: true
  cf-creator-post:
    name: ${self:custom.app_acronym}-cf-post
    description: Create CloudFormation stacks
    handler: handler_post.lambda_handler
    role: CloudFormationRole
    # environment:
    #   region: ${self:custom.region}
    tags:
      Name: ${self:custom.app_acronym}-cf-post
      Project:  serverless
      Environment: dev
    events:
      - http:
          path: /create
          method: post
          private: true

NOTE: without a private flag set to true, anyone will be able to post data to our API Endpoint, which is a bad idea. In case of true value set, our API Endpoint requires a API Key created in part:

provider:
  name: aws
  apiKeys:
    - ${self:custom.app_acronym}-apikey
  runtime: python2.7
  region: eu-west-1
  memorySize: 128
  timeout: 60 # optional, in seconds
  versionFunctions: true
  tags: # Optional service wide function tags
    Owner: chaosgears
    ContactPerson: chaosgears
    Project: serverless
    Environment: dev

Then Serverless is going to handle the rest of the creation process.

For simplicity I’ve added cloudformation:* action in following Policy just for testing purposes; for in production-like environments keep the rule of least privilege and allow only the necessary actions. This way it’s more feasible to control and maintain internal calls among services.

resources:
  Resources:
    CloudFormationRole:
      Type: AWS::IAM::Role
      Properties:
        AssumeRolePolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - lambda.amazonaws.com
              Action:
                - sts:AssumeRole
        Path: /
        Policies:
          -
            PolicyName: Serverless-CF-Creator
            PolicyDocument:
              Version: "2012-10-17"
              Statement:
                -
                  Effect: Allow
                  Action:
                    - lambda:ListFunctions
                    - lambda:InvokeFunction
                  Resource:
                    - "*"
                -
                  Effect: Allow
                  Action:
                    - s3:*
                  Resource:
                    - "arn:aws:s3:::${self:custom.s3}"
                    - "arn:aws:s3:::${self:custom.s3}/*"
                -
                  Effect: Allow
                  Action:
                    - cloudformation:*
                  Resource:
                    - "*"
                -
                  Effect: Allow
                  Action:
                    - logs:CreateLogStream
                    - logs:PutLogEvents
                    - logs:PutRetentionPolicy
                    - logs:CreateLogGroup
                    - logs:DescribeLogStreams
                    - logs:DeleteLogGroup
                  Resource:
                    - "arn:aws:logs:*:*:*"

‘Backend Brothers’

Some of our attentive reader noticed that we hadn’t analyzed Lambdas’ code and how they take care of events they’re receiving. I’ve created a small class called CF_Invoker to collect two methods for creation and checking state respectively:

class CF_Invoker(object):
    def __init__(self, service):
        try:
            self.client = boto3.client(service)
        except ClientError as err:
            logging.critical("----ClientError: {0}".format(err))
def cf_create_stack(self, stackname, templateurl='https://s3-xxx.amazonaws.com/mybucket/out/plik_wyjsciowy.yaml'):
        try:
            response = self.client.create_stack(
                    StackName=stackname,
                    TemplateURL=templateurl,
                    DisableRollback=False,
                    TimeoutInMinutes=10,
                    Capabilities=[
                        'CAPABILITY_IAM'
                    ]
                    )
            print(response)
            if response['ResponseMetadata']['HTTPStatusCode'] < 300:
             logging.info("---CloudFormation creation SUCCEDED: {0}".format(json.dumps(response)))
                return { "statusCode": response['ResponseMetadata']['HTTPStatusCode'],
                         "body": "Successfully created STACKNAME: {0}, STACKID: {1}".format(stackname, response['StackId'])
                         }
            else:
                logging.critical("---Unexpected error: {0}".format(json.dumps(response)))
                return { "statusCode": response['ResponseMetadata']['HTTPStatusCode'],
                         "body": "Error: {0}".format(response['ResponseMetadata']['HTTPStatusCode'])
                         }
        except ValueError as err:
            logging.critical("----Value error: {0}".format(err))
        except ClientError as err:
            logging.critical("----Client error: {0}".format(err))
  def cf_delete_stack(self, stackname):
        try:
            response = self.client.delete_stack(
                    StackName=stackname
                    )
            print(response)
            if response['ResponseMetadata']['HTTPStatusCode'] < 300:
                logging.info("---CloudFormation deletion SUCCEDED: {0}".format(json.dumps(response)))
                return { "statusCode": response['ResponseMetadata']['HTTPStatusCode'],
                         "body": "Successfully deleted STACKNAME: {0}".format(stackname)
                         }
            else:
                logging.critical("---Unexpected error: {0}".format(json.dumps(response)))
                return { "statusCode": response['ResponseMetadata']['HTTPStatusCode'],
                         "body": "Error: {0}".format(response['ResponseMetadata']['HTTPStatusCode'])
                         }
        except ValueError as err:
            logging.critical("----Value error: {0}".format(err))
        except ClientError as err:
            logging.critical("----Client error: {0}".format(err))

  def cf_stack_state(self, stackname):
        try:
            response = self.client.describe_stacks(StackName=stackname)
            if response['ResponseMetadata']['HTTPStatusCode'] < 300:
                logging.info("----CloudFormation STACK STATUS: {0}".format(response['Stacks'][0]['StackStatus']))
                return { "statusCode": response['ResponseMetadata']['HTTPStatusCode'],
                         "body": "CloudFormation STACKNAME: {0}, STATE: {1}".format(stackname, response['Stacks'][0]['StackStatus'])
                         }
            else:
                logging.critical("---Unexpected error: {0}".format(json.dumps(response)))
                return { "statusCode": response['ResponseMetadata']['HTTPStatusCode'],
                         "body": "CloudFormation STACKNAME: {0}, STATE: {1}".format(stackname, response['Stacks'][0]['StackStatus'])
                         }
        except ClientError as err:
            logging.critical("----Client error: {0}".format(err))

Then, having a class prepared, two following handlers have been added. They’re taking event info gathered from API Gateway and do their jobs. For me it was also a nice way to test HTTP requests with embedded parameters, forwarded by API Gateway and finally served by Lambdas. One thing to be added is that returned response value pasted in handler functions is being return to API Gateway which is then forwarded to the end user.

import logging
import json
from cf_invoker import CF_Invoker
def lambda_handler(event, context):
   logger = logging.getLogger()
   logger.setLevel(logging.INFO)
   logger.info("Event: {0}".format(event))
   payload = json.loads(event['body'])
   cf = CF_Invoker('cloudformation')
   response = cf.cf_create_stack(payload['stackname'],payload['templateurl'])
   return response
and
import logging
from cf_invoker import CF_Invoker
def lambda_handler(event, context):
   logger = logging.getLogger()
   logger.setLevel(logging.INFO)
   logger.info("Event: {0}".format(event))
   payload = event['pathParameters']
   cf = CF_Invoker('cloudformation')
   response = cf.cf_stack_state(payload['stackname'])
   return response

To deploy all of that you just need to type using necessary AWS profile bound with your AWS account:

sls plugin install -n serverless-python-requirements
sls deploy --aws-profile AWS_PROFILE

After two or three minutes you’ll get you stack deployed on particular AWS region and, of course, your Lambdas, API key and API Gateway. Everything implemented via single file.

Service Information
service: article
stage: dev
region: eu-west-1
stack: article-dev
api keys:
  article-apikey: YOUR_NEW_API_KEY
endpoints:
  GET - https://API_ENDPOINT/dev/state/{stackname}
  POST - https://API_ENDPOINT/dev/create
functions:
  cf-creator-get: article-dev-cf-creator-get
  cf-creator-post: article-dev-cf-creator-post
Serverless: Publish service to Serverless Platform...
Service successfully published! Your service details are available at:
https://platform.serverless.com/services/USERNAME/chaosgears

‘Call my API’

Great! We’ve gone through the deployment part, but generally the usage of Serverless Framework saves your time and makes life much easier. To test whether everything works fine, just type during the creation of the CF Stack:

curl -H "x-api-key: YOUR_NEW_API_KEY" -d '{"stackname": "YOUR_STACK_NAME", "templateurl": 
"https://S3_HTTPS_URL/S3_BUCKETNAME/PATH/FILENAME.yaml"}' https://API_ENDPOINT/dev/create

You should receive a response like the following if CloudFormation stack has been deployed properly:

Successfully created STACKNAME:: YOUR_STACK_NAME, STACKID: arn:aws:cloudformation:xxxx

To get the state of a new CF stack via API_ENDPOINT type:

curl -H "x-api-key: YOUR_NEW_API_KEY" https://API_ENDPOINT/dev/state/YOUR_STACK_NAME

the response you get should look like this:

CloudFormation STACKNAME: YOUR_STACK_NAME, STATE: STACK_STATE

And finally, to delete the stack:

curl -H "x-api-key: YOUR_NEW_API_KEY" -d '{"stackname": "YOUR_STACK_NAME"}' https://API_ENDPOINT/dev/delete

the response you get should look like this:

Successfully deleted STACKNAME: YOUR_STACK_NAME

Where this path leads…

Maybe our example doesn’t fit in all your needs but take a look at it from a different angle. We’ve combined 3 AWS services, fully serverless (it doesn’t mean that its self-service) and we were able to setup the infrastructure by simple HTTP POST with some attached variables. Let’s image people having no idea about AWS services but are eager to learn and have a strong resolve to automate some bottlenecked processes of provisioning new AWS environments after ordering them in their internal portal. Obviously serverless doesn’t solve all your problems, sometimes event-based pattern brings many more challenges than a typical scenario but an example like ours shows that it’s a good starting point if you never worked with serverless and want to find something easy to test and develop your idea of integration.