One of the benefits of serverless is the pay-per-use pricing model you get from the platform. That is, if your code doesn’t run, you don’t pay for them!
Combined with the simplified deployment flow (compared with applications running in containers or VMs) it has enabled many teams to make use of temporary CloudFormation stacks.
In this post, let’s talk about two ways you should use temporary CloudFormation stacks, and why.
Disclaimer: this shouldn’t be taken as a prescription. It’s a general approach that has pros and cons, which we will discuss along the way.
Temporary stacks for feature branches
It’s common for teams to have multiple AWS accounts, one for each environment. While there doesn’t seem to be a consensus on how to use these environments, I tend to follow these conventions:
is shared by the team, this is where the latest development changes are deployed and tested end-to-end. This environment is unstable by nature, and shouldn’t be used by other teams.
is where other teams can integrate with your team’s work. This environment should be fairly stable so to not slow down other teams.
should closely resemble production, and would often contain dumps of production data. This is where you can stress test your release candidate in a production-like environment.
- and then there’s
Having one account per environment is considered best practice. Some of my AWS friends would go even further and have one AWS account per developer.
Personally, I have never felt the need to have one account per developer. After all, there is some overhead for having an AWS account. Instead, I usually settle for one AWS account per team per environment.
However, for a particular project, I would often have a separate deployment CloudFormation stack per developer (but in the same dev account). This is especially useful when I’m working on feature branches.
When I start work on a new feature, I’m still feeling my way towards the best solution for the problem. The codebase is still unstable and many bugs haven’t been ironed out yet. Deploying my half-baked changes to the dev environment can be quite disruptive:
- it risks destabilising the team’s shared environment
- it’d overwrite other features the team is working on
- team members would be fighting over (not literally!) who gets to deploy their feature branch to the shared environment
. Feature toggles won’t help you in this case.
. Using the Serverless framework, that is as easy as running the command
sls deploy -s dev-my-feature
. This would deploy all the Lambda functions, API Gateway and any other related resources (DynamoDB, etc.) in its own CloudFormation stack. I would be able to test my work-in-progress feature in a live AWS environment.
account since it’s only used by the team. When the developer is done with the feature, the temporary stack can be easily removed by running
sls remove -s dev-my-feature
However, since these temporary stacks are an extension of your feature branch. They exhibit the same problems when you have long-lived feature branches. Namely, they get out-of-synch with other systems they need to integrate with. Both in terms of the events coming into your function, such as the payloads from SQS/SNS/Kinesis, etc. As well as data your function depends on, such as those that reside in DynamoDB.
While it’s not a problem with serverless technologies per se, but I find teams move faster when they use serverless. Which means the problems with long-lived feature branches become more prominent and noticeable as well.
Don’t leave feature branches hanging around for more than, say, a week. If the work is large and takes longer to implement, then break it up into smaller features. And when you’re working on a feature branch, integrate from master regularly (no less than once per day).
running localstack vs talking to AWS
The main downsides are:
- you need an internet connection
- deploying to AWS is slower than running code locally, which slows down the feedback loop
The internet access argument is only relevant for a handful of people who spend most of their time on the road. I travel more than most and do a lot of work while I’m at airports, and internet access is rarely a problem for me.
sls invoke local
to run my functions locally while talking to the real AWS services.
Temporary stacks for end-to-end tests
Speaking of testing, another common use of temporary CloudFormation stacks is for running end-to-end tests.
One of the common problems with these tests is that you need to insert test data into a live, shared AWS environment. As a rule of thumb I always:
- insert the data a test case needs before the test
- delete the data after the test finishes
However, despite our best intentions, mistakes happen and sometimes we deliberately cut corners to gain agility in the short-term. Over time, these shared environments are filled with test data, which at times interfere with normal operations.
As a counter-measure, many teams would employ cron jobs to wipe these environments from time-to-time.
An emerging practice to combat these challenges is to create a temporary CloudFormation stack during the CI/CD pipeline instead. The temporary stack is used to execute the end-to-end tests and destroyed afterwards.
This way, there is no need to clean up test data, either as part of your test fixture or with cron jobs.
You should weigh the benefits of this approach against the delay it adds to your CI/CD pipeline and decide if it’s right for a project. Personally, I think it’s a great approach and I’d encourage more teams to adopt it. However, the more data you have to clean up in external systems (that is, systems that are not provisioned as part of the temporary stack) the less useful it becomes.
In summary, here are two ways you can use temporary CloudFormation stacks to improve your development flow for serverless applications:
- use temporary stacks for feature branches to avoid destabilising shared environments
- use temporary stacks during CI/CD pipeline to run end-to-end tests to remove the overhead of cleaning up test data
These approaches should not be considered as a prescription. You need to consider their pros and cons and see they fit with your constraints and how your teams work.
In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. Including basic concepts, HTTP and event triggers, activities, design patterns and best practices.
Come learn about operational BEST PRACTICES for AWS Lambda: CI/CD, testing & debugging functions locally, logging, monitoring, distributed tracing, canary deployments, config management, authentication & authorization, VPC, security, error handling, and more.
You can also get 40% off the face price with the code ytcui.