If TDD is Zen, then TDD + Serverless is Nirvana – Hacker Noon

The choice of a particular database is out of the scope of this article. Because MongoDB is one of the most popular NoSQL databases, we will open an account at MongoDB Atlas.

Select a provider and a region that suit your needs. Since our code will be running on AWS Lambda, it makes sense to select AWS as the provider, and the same region where we will deploy our lambdas.

Check any additional settings and choose a name for your cluster. Wait for a few minutes for it to provision.

When it is ready, open the Security tab of the cluster and add a new user.

Since we are just testing, enter a username/password of your choice and select “Read and write to any database”.

IMPORTANT: When you are ready for production, you should apply again the Least Privilege Principle and restrict privileges to just your app’s database. You should also create different users for production and staging environments.

Keep the user and password and get back to the cluster overview. Click on the “Connect” button. Next we need to whitelist the IP addresses allowed to connect to the database. Unfortunately, on AWS Lambda we have no predictable way to determine the IP address that will connect to MongoDB Atlas. So the only choice is to go for “Allow access from anywhere”.

Finally, click on “Connect your application”, choose version 3.6 and copy the URL string for later.

Connection URL

Let’s code!

Enough accounts, now let’s get our hands dirty. Open the console and create a NodeJS project on folder my-api:

serverless create --template aws-nodejs --path my-api
cd my-api

Let’s invoke the default function:

serverless invoke local -f hello
(output)
{
"statusCode": 200,
"body": "{"message":"Go Serverless v1.0! Your function executed successfully!","input":""}"
}

Ok, running. Let’s create thepackage.json and add a few dependencies:

npm init -y
npm i mongodb
npm i -D serverless-offline serverless-mocha-plugin

Next, let’s define our environment. The serverless CLI just created the file serverless.yml for us. Clean it up and edit serverless.yml like this:

A few things to note here:

  • We are using NodeJS 8.10 in order to get the modern Javascript goodies.
  • We have defined the same region that we previously chose on MongoDB Atlas.
  • A hello function is added by default in handlers.js
  • We are adding two plugins at the bottom (from the dependencies we installed before). In a Daft Punk fashion, they will help us develop better, faster, stronger.

Now we can start listening for incoming HTTP requests on our local computer:

serverless offline start

Serverless offline listening on port 3000

So if we open our browser and visit http://localhost:3000, we will get the output of the hello function, which is attached to the / path by default. The output is a message, in addition to the HTTP headers and parameters that the function gets.

The cool thing is that if we edit handlers.js so that hello returns something different, you’ll notice that refreshing the browser will show the updated content. Live reload API out of the box!

Before we jump into our TDD environment, let’s tidy up a bit our project and arrange our files and routes.

mkdir handlers test
rm handler.js
touch handlers/users.js

The routes we will be supporting are:

GET /users
GET /users/
POST /users
PUT /users/
DELETE /users/

So let’s edit serverless.yml and define them. Edit the functions: block to contain these lines:

As you can see, each key inside the functions block is the name of a Lambda function. Note that handler: handlers/users.list would translate into: 
Use the list JS function inside handlers/user.js .

Time to TDD!

The serverless CLI provides a command to add new tests for each function. Let’s check it:

serverless create test -f listUsers
serverless create test -f getUser
serverless create test -f addUser
serverless create test -f updateUser
serverless create test -f removeUser

The test folder now contains a dummy spec file for each of our Lambda functions. As all our user-related functions are pointing to handlers/users.js maybe it’s better that the specs keep the same structure.

So let’s discard the isolated specs rm test/* , combine them into a single file and code the full user spec in test/users.spec.js .

As you can see, instead of importing just one function wrapper, we import them all and define test cases for the whole set.

You may also note that we are not performing HTTP requests. Rather we have to pass the parameters as they would be received by the function. If you are interested to spec like an HTTP client, check this library.

What happens now if we run the test suite?

serverless invoke test

You guessed it! Booom. And that’s because… we haven’t coded yet the handlers! But as you know: in TDD, specs are coded before the application logic.

So, as our runtime is NodeJS 8, we can take advantage of ES6/ES7 features to write cleaner code in our lambda functions. Let’s implement the (still undefined) functions in handlers/users.js :

Note that unlike a traditional NodeJS server, the connection to the database needs to be opened and closed at each execution. This is due to the nature of how serverless works: there is no process running all the time. Instead, instances are created and destroyed when external events occur.

Also note that all routes make sure to always close the DB connection. Otherwise, the internal NodeJS event loop would keep the process alive until the timeout was reached, and you might incur in higher charges.

And finally, note that thanks to using NodeJS v8, we can return our response instead of using callbacks with error-first parameter.

So now, our specs are ready, our implementation is there. The moment of truth:

Test execution

Yay! Here is our first serverless API. As you may see, local execution times are not particularly impressive, but keep the following in mind:

  • Our goal is not about single request speed. Rather, we aim for concurrent massive scalability, easy maintainability and future-proof code.
  • Latencies are mainly due to the time spent by our local computer connecting to the remote database. Tests with 3~4 DB requests will experience much higher latencies than when code is running within the datacenter.
  • We are using the lightest implementation possible (mongodb instead of mongoose, and plain JS instead of Express/Connect on top of Serverless).
  • If you switch to a local MongoDB server, running the tests will take around 80ms.

We’ll check performance back when our code is deployed.

Secret management

We are almost ready to deploy, but before we need to deal with an important aspect: keeping credentials out of the codebase.

Luckily for us, Serverless suports Simple Systems Manager (SSM) since version 1.22. This means that we can store key/value data to our IAM user and have it automatically retrieved whenever Serverless needs to resolve a secret.

So, first of all let’s get back to our handlers/users.js file, copy the current URL string and replace:

const uri = "mongodb+srv://lambda:[email protected]"

with:

const uri = process.env.MONGODB_URL

Next, let’s add an environment block to provider in serverless.yml :

provider:
name: aws
runtime: nodejs8.10
stage: prod
region: eu-central-1
environment:
MONGODB_URL: ${ssm:MY_API_MONGODB_URL~true}

This will bind the MONGODB_URL environment variable to the MY_API_MONGODB_URL SSM key at deploy time and ~true will decrypt the contents.

Finally, let’s grab the string we just copied and store the credential in our SSM:

pip install awscli    # install the AWS CLI if necessary
aws configure # confirm the key/secret, define your region
aws ssm put-parameter --name MY_API_MONGODB_URL --type SecureString --value "mongodb+srv://lambda:[email protected]/my-app?retryWrites=true"

If you are part of a bigger team, read here.

NOTE: At the time of writing, there is an issue with the serverless CLI retrieving SSM variables. If you encounter warnings or errors, check here and here.

Deploying

Stay with me, we are almost done! Let’s deploy our code to the cloud. We will update serverless.yml before. Then:

serverless deploy

Ready! You can also manage them here (select the appropriate region).

If you call the URL corresponding to the listUsers function you will see that the latency takes under one third of what it was taking from out computer.

This happens because now, the lambda function and the DB server are in the same region (i.e. datacenter), so connection latencies between them are considerably lower. Our round trip to Amazon will always be there, but now DB connections will not.

Production

As already commented during the article, when your API is ready to ship:

  • Remove the administrative permissions from your IAM user and refer to this page for insights on how to grant fine grained privileges.
  • Restrict the privileges of the DB user to only the database of your application, and not just any.

Deployments should only be made by project maintainers. The rest of developers shouldn’t need to configure any IAM Lambda credentials.

Cleaning

Deploying a Lambda function will involve (at least) three different services from Amazon. If you intend to wipe an existing Lambda function you need to:

  • Delete the function from AWS Lambda
  • Delete the corresponding bucket from S3
  • Delete the corresponding stack from CloudFormation

read original article here