Sometimes you need to postpone your custom, long-term ideas for short-term, already running ones.
Sometimes you need to postpone your custom, long-term ideas for short-term already running ones. The principle called KISS (Keep It Simple, Stupid), personally I’m a big fan of simplicity, might be your good friend when you know you should follow it. For one of our customer’s we had to deliver a rapid way of creating the CloudFormation Stack and checking its state, whether it has failed or completed. (In case you’re not familiar with the AWS CloudFormation service — it allows to build AWS resources using templates. A template is a configuration file (YAML or JSON) for provisioning all your AWS resources.)
In case of success they would be developed further and integrated with a third-party, non-AWS tool. The main challenge was that the solution had to be delivered at short notice (just one day). The customer wanted to follow Infrastructure as Code principle for their environment and to have an open door for third-party tool integration which would be a kind of ordering portal for its internal customers. Cool — we didn’t waste our time and rolled up our sleeves to come up with the idea.
Together, we decided that this is the time for AWS exploration and redesign but the real question how to expose AWS service without knowing anything about it. Now they’ve got a full CI/CD pipeline but the story started from something like in the picture below:
We went in tandem with AWS API Gateway as a ‘front door’ for functionality from our back-end service. Moreover, together with AWS Lambda as our two functions, the API Gateway forms the app-facing part of the AWS serverless infrastructure. The end user only sees the API endpoint URLs which they’re facing via proper HTTP calls. At the end of the line there’s a CloudFormation service reached via Lambdas. The smaller, the simpler and therefore functions have been separated to serve only one task.
To make it all versionable and easy to maintain Serverless Framework was used for provisioning. In case you missed my article about first steps with this awesome framework (Link) check it out. I remember the nauseating sensation when I had been deploying serverless environment for the first time. CloudFormation/Terraform are nice alternatives but trust me, you won’t use anything else the day you start using Serverless Framework (of course they’ve got some areas in the development stage but anyway it’s worth of diving in to). A snippet from serverless.yml file to present the main functionality:
functions:
cf-creator-get:
name: ${self:custom.app_acronym}-cf-get
description: Checks the state of CloudFormation Stack
handler: handler_get.lambda_handler
role: CloudFormationRole
# environment:
# region: ${self:custom.region}
tags:
Name: ${self:custom.app_acronym}-cf-get
Project: serverless
Environment: dev
events:
- http:
path: /state/{stackname}
method: get
private: true
request:
parameters:
paths:
stackname: true
cf-creator-post:
name: ${self:custom.app_acronym}-cf-post
description: Create CloudFormation stacks
handler: handler_post.lambda_handler
role: CloudFormationRole
# environment:
# region: ${self:custom.region}
tags:
Name: ${self:custom.app_acronym}-cf-post
Project: serverless
Environment: dev
events:
- http:
path: /create
method: post
private: trueNOTE: without a private flag set to true, anyone will be able to post data to our API Endpoint, which is a bad idea. In case of true value set, our API Endpoint requires a API Key created in part:
provider:
name: aws
apiKeys:
- ${self:custom.app_acronym}-apikey
runtime: python2.7
region: eu-west-1
memorySize: 128
timeout: 60 # optional, in seconds
versionFunctions: true
tags: # Optional service wide function tags
Owner: chaosgears
ContactPerson: chaosgears
Project: serverless
Environment: devThen Serverless Framework is going to handle the rest of the creation process.
For simplicity I’ve added cloudformation:* action in following Policy just for testing purposes; for in production-like environments keep the rule of least privilege and allow only the necessary actions. This way it’s more feasible to control and maintain internal calls among services.
resources:
Resources:
CloudFormationRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
Path: /
Policies:
-
PolicyName: Serverless-CF-Creator
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: Allow
Action:
- lambda:ListFunctions
- lambda:InvokeFunction
Resource:
- "*"
-
Effect: Allow
Action:
- s3:*
Resource:
- "arn:aws:s3:::${self:custom.s3}"
- "arn:aws:s3:::${self:custom.s3}/*"
-
Effect: Allow
Action:
- cloudformation:*
Resource:
- "*"
-
Effect: Allow
Action:
- logs:CreateLogStream
- logs:PutLogEvents
- logs:PutRetentionPolicy
- logs:CreateLogGroup
- logs:DescribeLogStreams
- logs:DeleteLogGroup
Resource:
- "arn:aws:logs:*:*:*"Some of our attentive reader noticed that we hadn’t analyzed Lambdas’ code and how they take care of events they’re receiving. I’ve created a small class called CF_Invoker to collect two methods for creation and checking state respectively:
1class CF_Invoker(object):
2 def __init__(self, service):
3 try:
4 self.client = boto3.client(service)
5 except ClientError as err:
6 logging.critical("----ClientError: {0}".format(err))
7def cf_create_stack(self, stackname, templateurl='https://s3-xxx.amazonaws.com/mybucket/out/plik_wyjsciowy.yaml'):
8 try:
9 response = self.client.create_stack(
10 StackName=stackname,
11 TemplateURL=templateurl,
12 DisableRollback=False,
13 TimeoutInMinutes=10,
14 Capabilities=[
15 'CAPABILITY_IAM'
16 ]
17 )
18 print(response)
19 if response['ResponseMetadata']['HTTPStatusCode'] < 300:
20 logging.info("---CloudFormation creation SUCCEDED: {0}".format(json.dumps(response)))
21 return { "statusCode": response['ResponseMetadata']['HTTPStatusCode'],
22 "body": "Successfully created STACKNAME: {0}, STACKID: {1}".format(stackname, response['StackId'])
23 }
24 else:
25 logging.critical("---Unexpected error: {0}".format(json.dumps(response)))
26 return { "statusCode": response['ResponseMetadata']['HTTPStatusCode'],
27 "body": "Error: {0}".format(response['ResponseMetadata']['HTTPStatusCode'])
28 }
29 except ValueError as err:
30 logging.critical("----Value error: {0}".format(err))
31 except ClientError as err:
32 logging.critical("----Client error: {0}".format(err))
33 def cf_delete_stack(self, stackname):
34 try:
35 response = self.client.delete_stack(
36 StackName=stackname
37 )
38 print(response)
39 if response['ResponseMetadata']['HTTPStatusCode'] < 300:
40 logging.info("---CloudFormation deletion SUCCEDED: {0}".format(json.dumps(response)))
41 return { "statusCode": response['ResponseMetadata']['HTTPStatusCode'],
42 "body": "Successfully deleted STACKNAME: {0}".format(stackname)
43 }
44 else:
45 logging.critical("---Unexpected error: {0}".format(json.dumps(response)))
46 return { "statusCode": response['ResponseMetadata']['HTTPStatusCode'],
47 "body": "Error: {0}".format(response['ResponseMetadata']['HTTPStatusCode'])
48 }
49 except ValueError as err:
50 logging.critical("----Value error: {0}".format(err))
51 except ClientError as err:
52 logging.critical("----Client error: {0}".format(err))
53
54 def cf_stack_state(self, stackname):
55 try:
56 response = self.client.describe_stacks(StackName=stackname)
57 if response['ResponseMetadata']['HTTPStatusCode'] < 300:
58 logging.info("----CloudFormation STACK STATUS: {0}".format(response['Stacks'][0]['StackStatus']))
59 return { "statusCode": response['ResponseMetadata']['HTTPStatusCode'],
60 "body": "CloudFormation STACKNAME: {0}, STATE: {1}".format(stackname, response['Stacks'][0]['StackStatus'])
61 }
62 else:
63 logging.critical("---Unexpected error: {0}".format(json.dumps(response)))
64 return { "statusCode": response['ResponseMetadata']['HTTPStatusCode'],
65 "body": "CloudFormation STACKNAME: {0}, STATE: {1}".format(stackname, response['Stacks'][0]['StackStatus'])
66 }
67 except ClientError as err:
68 logging.critical("----Client error: {0}".format(err))Then, having a class prepared, two following handlers have been added. They’re taking event info gathered from API Gateway and do their jobs. For me it was also a nice way to test HTTP requests with embedded parameters, forwarded by API Gateway and finally served by Lambdas. One thing to be added is that returned response value pasted in handler functions is being return to API Gateway which is then forwarded to the end user.
1import logging
2import json
3from cf_invoker import CF_Invoker
4def lambda_handler(event, context):
5 logger = logging.getLogger()
6 logger.setLevel(logging.INFO)
7 logger.info("Event: {0}".format(event))
8 payload = json.loads(event['body'])
9 cf = CF_Invoker('cloudformation')
10 response = cf.cf_create_stack(payload['stackname'],payload['templateurl'])
11 return response
12and
13import logging
14from cf_invoker import CF_Invoker
15def lambda_handler(event, context):
16 logger = logging.getLogger()
17 logger.setLevel(logging.INFO)
18 logger.info("Event: {0}".format(event))
19 payload = event['pathParameters']
20 cf = CF_Invoker('cloudformation')
21 response = cf.cf_stack_state(payload['stackname'])
22 return responseTo deploy all of that you just need to type using necessary AWS profile bound with your AWS account:
sls plugin install -n serverless-python-requirements
sls deploy --aws-profile AWS_PROFILEAfter two or three minutes you’ll get you stack deployed on particular AWS region and, of course, your Lambdas, API key and API Gateway. Everything implemented via single file.
Service Information
service: article
stage: dev
region: eu-west-1
stack: article-dev
api keys:
article-apikey: YOUR_NEW_API_KEY
endpoints:
GET - https://API_ENDPOINT/dev/state/{stackname}
POST - https://API_ENDPOINT/dev/create
functions:
cf-creator-get: article-dev-cf-creator-get
cf-creator-post: article-dev-cf-creator-post
Serverless: Publish service to Serverless Platform...
Service successfully published! Your service details are available at:
https://platform.serverless.com/services/USERNAME/chaosgearsGreat! We’ve gone through the deployment part, but generally the usage of Serverless Framework saves your time and makes life much easier. To test whether everything works fine, just type during the creation of the CF Stack:
curl -H "x-api-key: YOUR_NEW_API_KEY" -d '{"stackname": "YOUR_STACK_NAME", "templateurl":
"https://S3_HTTPS_URL/S3_BUCKETNAME/PATH/FILENAME.yaml"}' https://API_ENDPOINT/dev/createYou should receive a response like the following if CloudFormation stack has been deployed properly:
Successfully created STACKNAME:: YOUR_STACK_NAME, STACKID: arn:aws:cloudformation:xxxxTo get the state of a new CF stack via API_ENDPOINT type:
curl -H "x-api-key: YOUR_NEW_API_KEY" https://API_ENDPOINT/dev/state/YOUR_STACK_NAMEthe response you get should look like this:
CloudFormation STACKNAME: YOUR_STACK_NAME, STATE: STACK_STATEAnd finally, to delete the stack:
curl -H "x-api-key: YOUR_NEW_API_KEY" -d '{"stackname": "YOUR_STACK_NAME"}' https://API_ENDPOINT/dev/deletethe response you get should look like this:
Successfully deleted STACKNAME: YOUR_STACK_NAMEMaybe our example doesn’t fit in all your needs but take a look at it from a different angle. We’ve combined 3 AWS services, fully serverless (it doesn’t mean that its self-service) and we were able to setup the infrastructure by simple HTTP POST with some attached variables. Let’s image people having no idea about AWS services but are eager to learn and have a strong resolve to automate some bottlenecked processes of provisioning new AWS environments after ordering them in their internal portal. Obviously serverless doesn’t solve all your problems, sometimes event-based pattern brings many more challenges than a typical scenario but an example like ours shows that it’s a good starting point if you never worked with serverless and want to find something easy to test and develop your idea of integration.
We'd love to answer your questions and help you thrive in the cloud.
