Have you ever had short lived containers like the following use cases:
- Batch and ETL (Extract, Transform & Load) Jobs.
- Database backups and replication.
- Machine Learning algorithms for generation of learning and training models.
- Integration & Sanity tests.
- Web scrapers & crawlers.
And you were wondering how you can deploy your container periodically or in response to an event ? The answer is by using Lambda itself, the idea is by making a Lambda function trigger a deployment of your container from the build server. The following figure illustrates how this process can be implemented:
I have wrote a simple application in Go to simulate a short time process using sleep method:
As Go is a complied language, I have used Docker multi-stage build feature to build a lightweight Docker image with the following Dockerfile:
Next, I have a simple CI/CD workflow in Jenkins, the following is the Jenkinsfile used to build the pipeline:
An example of the pipeline execution is given as follows:
Now, all changes to the application will trigger a new build on Jenkins which will build the new Docker image, push the image to a private registry and deploy the new Docker image to the Swarm cluster:
If you issue the “docker service logs APP_NAME” on one of the cluster managers, your application should be working as expected:
Now our application is ready, let’s make execute everyday at 8am using a Lambda function. The following is the entrypoint (handler) that will be executed on each invocation of the function:
It uses the Jenkins API to trigger the deployment process job.
Now the function is defined, use the shell script below to create the following:
- Build a deployment package (.zip file).
- Create an IAM role with permissions to push logs to CloudWatch.
- Create a Go based Lambda function from the deployment package.
- Create a CloudWatch Event rule that will be executed everyday at 8am.
- Make the CloudWatch Event invoke the Lambda function.
As a result, a Lambda function will be created as follows:
To test it out, you can invoke it manually either from the Lambda Console or using the following AWS CLI command:
A new deployment should be triggered in Jenkins and your application should be deployed once again:
That’s it, it was a quick example on how you can use Serverless with Containers, you can go further and use Lambda functions to scale out/scale in your services in your Swarm/Kubernetes cluster by using either CloudWatch events for expected scheduled events (Holidays, Black Friday …) or other AWS managed services like API Gateway in response to incoming client requests.