ArticleAWSBOT

How we met “Charlize” – story about our bot

By 05/14/2019 No Comments

“Small teams = not enough time for everything”

Like everyone else, we launched our startup with a bunch of people eager to help other companies to use cloud technology as an accelerator for their innovative ideas. We followed the rule of automation, as much as possible, to save some time and do not repeat ourselves. Contrary to the previous jobs it was not only a technological shift but probably more important a culture one. After the first year, we realized that the main enemy is the time, which always flies against you. Honestly saying the more time you have, the one you can spend on innovation and internal development, the bigger chance that you’ll get success. We’re coming to the point where you probably ask me how to save time in a startup, which develops its event-driven app internally and support external teams as an augmented team in different kinds of AWS projects. In our case, such remedy was to automate tiny workflows, tasks, actions which we had to repeat from time to time. We continuously improve already coded ones and add new ones. It is the moment where I’d like to highlight one point – “It doesn’t work for everyone,” but a proper selection of tasks which we’re capable of automating was a key to new areas.

“Small team using serverless architecture to implement non-human team member”

So, I brought you to the point where me and my teammates decided to leverage AWS services and combine them with Slack to build a bot, we’ve called her Charlize – do not ask my why, still don’t know that, which was going to act as a team member. We thought that instead of simply asking other colleagues about some information regarding AWS environments or requesting someone to suddenly invoke a Lambda via API, restart an instance or do some other job. We’re humans who are always the bottleneck in processes, no matter how big the team is. Each company working with technology struggles with failures and we’ve noticed that we had been repeating same “remedy” tasks when something crushed. Then comes Chalize and now I’ll lead you through its deployment so maybe one day you’re gonna build your own.

“Small team building its bot – add app”

Slack app creation consists of 3 main steps:

1.Go to https://api.slack.com/apps and click the big green Create New App button. Enter a name for your app and click the next prominent green button.

2.The Basic Information link on the left hand side of your app’s settings page contains information you’ll need, such as the Client ID and Client Secret, to authenticate OAuth requests for your app

Basic Information (Source: https://api.slack.com/apps/OUR_APP_ID/general?)

3. Then it’s time to setup a Redirect URL for your app(picture below). This is the endpoint for Slack to send a unique temporary code to your server during a user’s installation. Your server will then send back this code, along with your Client ID and Client Secret, so that we know we can trust you. The Redirect URL must be publicly accessible and secure. If you want to run your server locally.

Source: https://api.slack.com/apps/OUR_APP_ID/oauth?

In our case we used API Gateway as our entry point to AWS backend with Install Lambda function(serverless.yml snippet):

functions:
  install:
    name: ${self:custom.app_function}-install
    description: Install Slack integration
    handler: gear_install.handler
    role: LambdaRole
    environment:
      tablename: ${self:custom.app_function}-tokens-${self:custom.stage}
    events:
      - http:
          path: /install
          method: get

ADD TO SLACK request (Source: https://api.slack.com/docs)

 

Important note from slack page regaring ADD SLACK button:

Note: the incoming-webhook scope is designed to allow you to request permission to post content into the user’s Slack workspace. It intentionally does not include any read privileges, making it perfect for services that want to send posts or notifications into Slack workspaces that might not want to give read access to messages.”

Below our config:

ADD TO SLACK request (Source: https://api.slack.com/docs)

 

So what happens when user clicks “Add to Slack” button on a webpage?

When a user clicks your Add to Slack button, a request is sent to Slack’s servers. The Client ID sent via the button click is validated and a code is sent as a GET request to your Redirect URL. The code is in a JSON object named query of the request, i.e. req.query.code.

We’ve coded a Lambda function which leverages AWS SSM Parameter store to keep securely CLIENT ID and CLIENT SECRET, sends a request with those values and CODE to https://slack.com/api/oauth.access URL as its payload.

The story behind CODE is that Slack sends a temporary CODE, which you will need to exchange it to a more permanent token. The code expires 10 minutes after release. Finally, token metadata is being put into AWS Dynamodb table via class method put_dynamo_items(self, item, team, team_id, token, bot).

class SlackArmyDynamo(gearDynamo):

    def put_dynamo_items(self, item, team, team_id, token, bot):
        try:
            response = self.table.put_item(
                Item={
                    'team_id': team_id,
                    'team': team,
                    'customer_id': item,
                    'date': datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
                    'bot': bot,
                    'token': token
                })
            return response
        except ClientError as err:
            logger.critical("----Client error: {0}".format(err))
            logger.critical(
                "----HTTP code: {0}".format(err.response['ResponseMetadata']['HTTPStatusCode']))

def get_param(item):
    try:
        ssm = boto3.client('ssm')
        parameter = ssm.get_parameter(Name=item, WithDecryption=True)
        param = str(parameter['Parameter']['Value'])
        return param
    except ClientError as err:
        logger.critical("----Client error: {0}".format(err))
        logger.critical(
            "----HTTP code: {0}".format(err.response['ResponseMetadata']['HTTPStatusCode']))


def get_token(code, tablename):
    logger.info("------Getting token..")
    if code == '':
        ("------Code value is null")
        output = {
            "statusCode": 400,
            "body": "---Error. Code value is null"
            }
        return output
    else:
        url = 'https://slack.com/api/oauth.access'
        payload = {
              "client_id": get_param('CLIENT_ID'),
              "client_secret": get_param('CLIENT_SECRET'),
              "code" : code
        }
        data = urllib.parse.urlencode(payload).encode("utf-8")
        req = urllib.request.Request(url)
        info = urllib.request.urlopen(req, data)
        logger.info("------Getting info from url: %s", info.geturl())
        response = json.loads(info.read().decode('utf-8'))
        if response['ok'] == False:
            logger.error("------Problem with the token: %s", response['error'])
            output = {
                    "statusCode": 400,
                    "body": response['error']
                    }
            return output
        else:
            logger.info("------Putting token into the dynamodb table: %s", tablename)
            table = SlackArmyDynamo(tablename)
            table.put_dynamo_items(item=response['user_id'], team=response['team_name'], team_id=response['team_id'], token=response['access_token'], bot=response['bot'] )
            output = {
                    "statusCode": 200,
                    "body": "Token validated. Put into Dynamodb"
                    }
            return output


def handler(event, context):
    logger.info("------Event: {0}".format(event))
    code = event['queryStringParameters']['code']
    tablename = os.environ['tablename']
    token_response = get_token(code, tablename=tablename)
    return token_response

Response after “Add to Slack” request looks like that:

{'ok': True, 'access_token': 'TOKEN', 'scope': 'identify,bot', 'user_id': 'USER_ID', 'team_name': 'Chaos Gears', 'team_id': 'TEAM_ID', 'bot': {'bot_user_id': 'BOT_USER_ID', 'bot_access_token': 'BOT_ACCESS_TOKEN'}}

In case of response you encounter code expiration message:  {‘ok’: False, ‘error’: ‘code_expired’}:

Now we can use this token to make API calls.

“Small team building its bot – API calls, functions etc.”

First of all our “Charlize” bot has to be notified about events in Slack to make its job done. URL in our case is also an API Gateway endpoint. Slack sends HTTP POST requests to this URL when events occur with a challenge parameter, and API endpoint must respond with the challenge value. In our case Lambda called “gear_event” was responsible for taking care of this challenge exchange. More on that below in following section.

Lambda “gear_event” makes verification before call any other internal “action” Lambda and sends this challenge value.

events:
    name: ${self:custom.app_function}-events
    description: Provides verification before invoking internal functions
    handler: gear_events.handler
    role: LambdaRole
    environment:
      tablename: ${self:custom.app_function}-tokens-${self:custom.stage}
      function_ec2: ${self:custom.app_function}-ec2-actions
      table_instances: ${self:custom.t_instances}
      region: ${self:custom.region}
    events:
      - http:
          path: /events
          method: post

Snippet of the “gear_event” function responsible for sending back challenge value:

def get_challenge(body):
    logging.info("------Checking event challenge value from slack")
    if body['type'] == 'url_verification':
        logging.info("------Sending challenge back to Slack")
        response = {
            "statusCode": 200,
            "body": body['challenge']
        }
        return response

We’ve selected “app.mention” event for bot calling, so whenever our team member uses “@charlize” in slack channel, the whole process is being invoked:

Subscribe to event (Source: https://api.slack.com/apps/OUR_APP_ID/event-subscriptions?)

 

TL:TR

  • def token_verification – checking whether proper token values exists in HTTP request payload coming through API Gateway
  • def sendResponse – sending response to Slack(messages you see in the channel)
  • class Lambda(object) – small class containing method for Lambda async invocation. We need that to call “actions” Lambdas.
  • def get_service_action  – it’s checking the command received in slack channel, translates it into any known action(if there is any), if needed searching for EC2 instances metadata in Dynamodb table. This table contains running instances from customer’s environment(One of our task for being automated)

So simply saying “Charlize” is able to list you all running instances by just asking:

  • def import_data – at the beginning we wanted to store all metadata which we need in the file to make it as simple as possible(we’ve already pushed that into Dynamodb). The content of the file is presented below and contains actions which are eligible to invoke with specific “states” because it regards AWS EC2 service.
    {
        "services": [
            {
                "name": "ec2",
                "actions": ["check", "stop", "kill", "terminate", "restart", "reboot", "find"],
                "states": ["stopped", "running", "terminated"]
            }
        ]
    }
    

FUNCTIONS in “gear_events”:

def token_verification(body, param='VERIFICATION_TOKEN'):
    logging.info("------Checking Verification Token")
    if body['token'] != get_param(param):
        raise ValueError('InvalidToken')
    else:
        response = {
            "statusCode": 200,
            "body": "TokenVerified"
        }
        return response


def get_teamid(body, tablename):
    table = gearDynamo(tablename)
    logging.info("------Getting info from dynamodb")
    item = table.get_item('team_id', body['team_id'])
    return item['Item']



def sendResponse(body, text, data):
    params = {
            "attachments": [
            {
                "title": "Charlize's response",
                "author_name": "ChaosGears",
                "text": text,
                "color": "#2eb886"
            }],
            'token': body['bot_access_token'],
            'channel': body['event']['channel'],
            }
    url = 'https://slack.com/api/chat.postMessage'
    logging.info("------Requesting: '%s'", url)
    data = urllib.parse.urlencode(params).encode("utf-8")
    req = urllib.request.Request(url)
    info = urllib.request.urlopen(req, data)
    logging.info("------Getting info from url: %s", info.geturl())
    response = json.loads(info.read().decode('utf-8'))
    print(response)

def import_data(filename, dirname):
    if os.path.isfile(os.path.join(dirname, filename)) and os.access(os.path.join(dirname, filename), os.R_OK):
        with open(os.path.join(dirname,filename), 'r') as file:
            inputt = json.load(file)
        file.close()
        return inputt
    else:
        print('File ' + str(filename) + ' does not exist')

def get_service_action(body, tablename, t_instances):
    message = body['event']['text']
    logging.info("------Got the message from Slack: '%s'", message)
    botUserId = get_teamid(body, tablename)['bot']['bot_user_id']
    botAccessToken = get_teamid(body, tablename)['bot']['bot_access_token']
    command = (re.split(('<@'+str(botUserId)+'>'), message))[1].lower()
    default_reply = "Excuse me Sir, I didn't understand the command: " + command 
    print(command)
    temp, flag = action_selector(command, t_instances)
    response = {
        "command": command,
        "bot_user_id": botUserId,
        "bot_access_token": botAccessToken,
        "data": temp
    }
    for item in response.keys():
        body = response
    logging.info("------Bot %s is mentioned in: %s", botUserId, message)
    if len(temp) == 0 and flag == '1':
        sendResponse(body, text=default_reply, data=command)
        return body
    elif len(temp) == 0 and flag == '5':
        text = "Excuse me Sir. Instance you've mentioned is not recognized"
        sendResponse(body, text=text, data=command)
        return body
    elif len(temp) == 0 and flag == '2':
        text = "Excuse me Sir. No AWS service found in the command"
        sendResponse(body, text=text, data=command)
        return body
    else:
        text = "Hello Sir. I've got the command:"+ str(command)
        sendResponse(body, text=text, data=command)
        return body


class Lambda(object):

    def __init__(self, region, service='lambda'):
        try:
            self.region = region
            self.client = boto3.client(service, self.region)
        except ClientError as err:
            logging.error("------ClientError: %s", err)

    def invoke_function(self, functioname, payload, invoke_type='Event'):
        try:
            self.client.invoke(FunctionName=functioname, InvocationType=invoke_type, Payload=json.dumps(payload))
        except ClientError as err:
            logging.error("------ClientError: %s", err)


def handler(event, context):
    logger.info("------Event: {0}".format(event))
    body = json.loads(event['body'])
    tablename = os.environ['tablename']
    region = os.environ['region']
    t_instances = os.environ['table_instances']
    function_ec2 = os.environ['function_ec2']
    if token_verification(body)['statusCode'] == 200 and body['type'] != 'url_verification':
        response = {
            "statusCode": 200
        }
        data = get_service_action(body, tablename, t_instances)
        if len(data['data']) > 0:
            logging.info("------Command successfully extracted")
            if data['data'][0] == 'ec2':
                function = Lambda(region)
                logging.info("------Invoking Lambda function: %s", function_ec2)
                function.invoke_function(functioname=function_ec2, payload=data)
                return response
            else:
                logging.info("------AWS service not binded with any function")
                return response
        else:
            logging.info("------Problem with extraction of the data")
            return response
    elif token_verification(body)['statusCode'] == 200:
        logging.info("------Verification of the url slack challenge")
        response = get_challenge(body)
        return response

“@Charlize do something for me”

We’ve got events function being a simple selector so it’s high time to introduce an example of “actions” function.

action-ec2:
  name: ${self:custom.app_function}-ec2-actions
  description: Provides actions regarding EC2 service
  handler: gear_ec2.handler
  environment:
    regions: ${self:custom.regions}
    tagkey: ${self:custom.tagkey}
    tagvalue: ${self:custom.tagvalue}
    table_roles: ${self:custom.t_roles}
    table_instances: ${self:custom.t_instances}
    dest_account: ${self:custom.dest_account}
  role: EC2Role

with EC2Role snippet which contains cross-account IAM Role which is being assumed whenever we want to do something on customer’s account. IAM Roles with account number and customer’s name are stored in Dynamodb which we call to get the name of the role.

Pretty simple defined IAM role for “action” Lambda via Serverless.

Policies:
        -
          PolicyName: GearSlack-Army-EC2
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              -
                Effect: Allow
                Action:
                  - sts:AssumeRole
                Resource:
                  - arn:aws:iam::${self:custom.dest_account}:role/${self:custom.dest_rolename}
              -
                Effect: Allow
                Action:
                  - dynamodb:PutItem
                  - dynamodb:GetItem
                Resource:
                  - arn:aws:dynamodb:${self:custom.region}:*:table/${self:custom.app_function}-roles-${self:custom.stage}
                  - arn:aws:dynamodb:${self:custom.region}:*:table/${self:custom.app_function}-instances-${self:custom.stage}

Right now “action” Lambda is able to find instances and reboot them because that’s what we needed that time. Here’s the returning message being sent to Slack after action taken:

def sendResponse(body, text, data):
    if data != '':
        params = {
            "attachments": [
            {
                "title": "Charlize is saying:",
                "text": text,
                "color": "#2eb886",
                "fields": [
                {
                    "title": "Response",
                    "value": data,
                    "short": "false"
                }]
            }],
            'token': body['bot_access_token'],
            'channel': body['event']['channel'],
            }
    else:
        params = {
            "attachments": [
            {
                "title": "Charlize is saying:",
                "text": text,
                "color": "#2eb886"
            }],
            'token': body['bot_access_token'],
            'channel': body['event']['channel'],
            }
    url = 'https://slack.com/api/chat.postMessage'
    logging.info("------Requesting: '%s'", url)
    data = urllib.parse.urlencode(params).encode("utf-8")
    req = urllib.request.Request(url)
    info = urllib.request.urlopen(req, data)
    logging.info("------Getting info from url: %s", info.geturl())
    response = json.loads(info.read().decode('utf-8'))
    print(response)

Then asking Charlize:

    • @charlize find number of running EC2 instances”
    • @charlize check number of stopped ec2 instances”
    • @charlize find running ec2 instances”
    • “@charlize find running ec2 instances in customer environment”

and get the response from channel. For acknowledgment that Charlize is getting the right message from the Slack channel we’ve added first response where “I’ve got the command: COMMAND_USED” is being returned.

An example of asking for stopped instances. Actually none was found

An example of asking for running instances. Dictionary returned

See you then “Charlize”..it’s not the end, just the beginning

We’ve showed one task that Charlize is capable of doing and saves us time, which in our case means a lot. Obviously it’s just a small grain of sand on the desert of ideas. Just imagine what your customised bot might be useful for. Start with simple task like we did and then add new “skills”. You’re going to experience a lot of fun and gain new member in your team able to do task you team is tired of.