ArticleAWScomputeFargateLambdaserverless

Fargate, powerful boost to serverless computing

So, we meet again

We haven’t seen each other for a long time, but now, finally, I return to you with a new, fresh portion of serverless knowledge. Last time we focused mainly on lambda function but, as with everything else, it contains some drawbacks that need to be overcome by other services. That’s why I decided to introduce you to this quite new service. Take your time and make yourself comfortable, so we can start our trip through AWS.

Serverless, serverless and serverless

In my previous articles I shared a lot of information linked with the abovementioned issue, but somewhere along the line I realized that I’ve said nothing about what it actually is. So, in reference to AWS description, serverless allows you to create and run applications without having to wonder about server management. It may seem like a slight simplification, but you need to realize the amount of problems it solves. Behind the server management hide system pathing, operating system issues or capacity provisioning.

With serverless, AWS take care of every aspect so you can focus on core functionality, and that simply means your application. The other benefit is high scalability guaranteed by AWS. When you work with on-premise systems and the demands continue to grow, you need to replace weaker components or add additional ones, so they would handle your application demands. When you use cloud without serverless approach, you need to add extra ec2 instance or replace it with other instance type. While using serverless, you don’t need to worry about it because when demands grow, AWS handle everything concerned with providing the right amount of compute capacity.

Services that can be used to create serverless application: Lambda, S3, DynamoDB, API Gateway, Kinesis, Fargate and many more. As you can see, services cover a lot of IT fields like, Storage, Analytics, Api, Compute, etc. But today we will focus on the last field – compute and fargate…

Fargate, let me introduce our Readers. Dear Readers, that’s Fargate

Fargate was launched on 2 April 2019. It’s quite new service but helps to overcome unresolved or really difficult to handle problems, using lambda function. It can work both with ECS (Elastic Container Service) and EKS (Elastic Kubernetes service). But why I became interested in that service when I had a great tool to execute serverless calculations?

The biggest problem with lambda is its execution time, which (after recent update) is limited to 15 minutes. As Lambda divides your code into smaller pieces and distributes it between more lambdas, time limit sounds like a good choice. But often our calculations take more than 15 minutes. Take for example big data processing or copying objects between buckets, but not only one object or some objects like it was shown in previous articles. Imagine moving large number of huge objects with size expressed in GB or even TB.

Is it possible to overcome this with lambda functions? We can run parallel functions, each copying pieces of objects. Unfortunately, making a correct division in every case may prove difficult, especially when you need to make pieces of data simultaneously and dynamically set the right number of lambda functions that can vary depending on case.

The other possible way is to set only one lambda for copying data and wake up other one to continue the task at the end of execution time of the first function. In that case, we can find it difficult to save context and retrieve it during the process of copying. It would be necessary to save key of last copied object and pass it to the next lambda function. This solution can cause errors or incomplete tasks if the key was not properly passed.

That’s why I started to look for other tools that could help me finish this task. And soon I found fargate. A long execution time fits this case perfectly, while maintaining the capacity to run multiple tasks at once provide the ability to copy multiple buckets at the same time. Depending on the amount of data to copy, we can run our task for minutes, hours or even days (!!!). And it is still a serverless solution, so it charges us only for the time we use it.

Image, first step to run fargate

Unfortunately, to take full advantage of fargate’s capabilities we need to use container. So, before you go any further, you need to create image of your application. First of all, you must install docker and then create simple file which contains information required to create the right image.  

FROM python:alpine3.6

COPY . ~/Desktop/python_image/image

WORKDIR ~/Desktop/python_image/image

RUN pip install -r requirements.txt

CMD python ./migrator.py

First, we define the language in which our application was written. I’m a python freak, so my app uses that technology. Next, we copy all modules of our app, and move it to the workdir which is set next line. The second to last line is really important. In it we define modules that must be installed to make our application run properly. The last line defines module used to start our app. And that’s all. With this file we can create our first image. Run the command:

Sudo docker build .

And if the code doesn’t contain any faults, we should get output of image tag by random number used to identify the right image.

ECR, storage for your images

Ok, we have our image but we need to pass it to aws. Elastic Container Registry can help with that, by providing a place to store your images. Assuming you have one registry created, we can try to push our app to it. There are few steps we need to make to achieve it

1. Get credentials from ecr service (Your user need to have access to ecr to receive access token)
aws ecr get-login –no-include-email –region eu-west-1

 

2. Log into ecr by using credentials received in previous step

docker login -u AWS -p **** https://************.dkr.ecr.eu-west-1.amazonaws.com

You should put received token after -p flag

In the url before .dkr you should put account number

 If everything went ok, you should receive login success respond

 

3. Tag our image with corresponding registry name and sufix latest

sudo docker tag ******* s3-migrator-repo:latest

After tag, you should put image name

 

4. Link image and registry together

sudo docker tag s3-migrator-repo:latest ************.dkr.ecr.eu-west-1.amazonaws.com/s3-migrator-repo:latest

Before .dkr you should give an account number

 

5. Push image to registry

sudo docker push ************.dkr.ecr.eu-west-1.amazonaws.com/s3-migrator-repo:latest

Before .dkr you should type an account number

If everything went well, your image should now be in your AWS ECR registry with tag latest. Next, you need to create container and link it with your image. When you are ready you can move to the next step.

Run Fargate Task

Creating and running a task with fargate is quite similar to the steps you need to take when doing the same thing with EC2, though there are some differences. First, you need to log into your management console, move to ECS and get inside cluster that holds your app. Then, change option to “tasks” and hit “Create Task”.

Next menu gives you a lot of options, but the most important thing to remember is changing the launch type to fargate. Task definition, platform version and cluster should immediately correspond to the right cluster and task, so you can leave them as they are, and instead focus on VPC and security groups options. To run tasks with fargate you need to have defined vpc with subnet. In reference to AWS documentation you need to have:

a) specify ENABLED for Auto-assign public IP when launching the task, for public subnets

b) specify DISABLED for Auto-assign public IP when launching the task, and configure a NAT Gateway in your VPC to route requests to the internet, for private subnets

Instead of using events [‘body] to pass arguments, like we did with lambda function, we can use advanced options inside task definition, and define environmental variables that can be used later by our application.

If need be, we can add one more to our task by opening Advanced Option -> Container Overrides -> Environment variables overrides. Add Environment Variable allows us to add as many arguments as we want with Key Value pattern, and that same key may be later used to get data from variable. In the Python code we can get retrieve environment variable by using:

variable = os.environ[‘variable_name’]

Where variable_name must correspond to the appropriate key name.

Container Overrides offer one more interesting option, namely to override container role. We can use it to set additional privileges to container, and that allows us to use other aws services from the running task. It works the same way lambda function roles do, so you should know how useful it could be. It also removes the need of hard coding your credentials inside a code, which makes our app safer. Hard coded credentials are strongly discouraged by AWS.

Run Fargate task with Lambda

It is possible, of course, and we will use lambda and boto3 to achieve it. Example of creating a task with fargate is shown below:

def run_task(data):
    client = boto3.client('ecs')
    response = client.run_task(
    cluster='cluster_name', 
    launchType = 'FARGATE',
    taskDefinition='task_name', 
    count = 1,
    platformVersion='LATEST',
    networkConfiguration={
        'awsvpcConfiguration': {
            'subnets': [
                'subnet_name' 
                
            ],
            'assignPublicIp': 'ENABLED'
        }  
    }

As usual, we create boto3 client to get access to service, in our case we need to use ecs. Method run_task allows us to run task in the same way like we did in management console. All arguments correspond to the one in AWS management console. But I want to highlight launchType which must be set to FARGATE. We are able to pass environmental variable to the container in a way shown below: 

overrides={
    	'containerOverrides': [
        	{
        	'name': 'cluster_name',    
        	'environment': [
               	 
                	{
                    	'name': 'variable_name',
                    	'value': 'variable_value'
                	}
          	
            	]    
        	}
        	]

We can retrieve it inside the task the same way we did, using management console.

That’s all for now

But don’t worry. As I said earlier, fargate is a rather new service, so despite the fact it has already provided great solution for lambda’s problem, I’m sure there will be much more to discover throughout its development. I hope this short article will help you with your long processing apps and I strongly believe that fargate will stay with you for a while longer. Wait for next parts which will appear shortly as I delve deeper inside this fantastic service. May the Fargate be with you.