Many clients have asked me “how do I record custom metrics from Lambda?”.
Generally speaking, you can either:
- Publish custom metrics synchronously – e.g. send them at the end of an invocation.
- Publish custom metrics asynchronously by writing them to stdout first and then extracting them from CloudWatch Logs.
Problems with sending custom metrics synchronously
The synchronous approach adds latency to invocations. This can be especially problematic when those extra milliseconds are experienced by our users. For example, if the user is waiting for an API response.
Individually, the delay might be negligible. CloudWatch metrics typically respond within tens of milliseconds. That is acceptable to most. But they can quickly compound when functions call one another via API Gateway.
Moreover, services are most fragile around their integration points – i.e. when they make network calls to other services. Publishing metrics to CloudWatch introduces another integration point that you need to harden.
If CloudWatch experiences an outage, surely you would still want your system to stay up, right? Similarly, if CloudWatch experiences elevated response time then you wouldn’t want your functions to timeout as a result!
Hence why I generally prefer to record custom metrics asynchronously, even though this approach also has its drawbacks:
- There is an additional delay in seeing the most recent metric data.
- It’s sending more data to CloudWatch Logs, which has a cost implication
- It introduces complexity because you need something to process the logs and turn them into metrics.
Sending custom metrics asynchronously
Instead, you can use a Lambda function.
Or you can deploy it as part of a CloudFormation stack with AWS SAM:
Recording custom metrics
Once deployed, you would be able to record custom metrics by writing to stdout in this format:
| | | |
These messages would be processed and published as custom metrics in CloudWatch metrics. All without adding latency to your invocations!
Parse the REPORT messages at the end of an invocation and turn Billed Duration, Memory Size and Memory Used into metrics.
In this course, we’ll cover everything you need to know to use AWS Step Functions service effectively. Including basic concepts, HTTP and event triggers, activities, design patterns and best practices.
Come learn about operational BEST PRACTICES for AWS Lambda: CI/CD, testing & debugging functions locally, logging, monitoring, distributed tracing, canary deployments, config management, authentication & authorization, VPC, security, error handling, and more.
You can also get 40% off the face price with the code ytcui.