Invoke HTTP from AWS Lambda without Waiting
What is the fastest way to invoke a HTTP/REST URL from an AWS Lambda?
The answer is pretty straight forward if you need a response. But what if you don’t want to wait for a reply?
Non-Waiting HTTP
Consider the case of a Lambda function that after processing, accumulates some metrics and then emits those back to a REST aggregator service for processing. If the metrics are counters, it may not matter if the metrics request fails occasionally, as the next request will send the updated counter totals. In cases like these, it will be faster not to wait for the HTTP response.
So how would we do this in NodeJS?
Let’s look first at a naive example request (without error handling) where we wait for the response.
Just for this example, we’ll create an async “request” node function that returns a promise that we can wait on.
Our Lambda would then look like:
To invoke our HTTP request inside a Lambda, we use “await” which issues the request and then waits for the response.
If the metric aggregator service takes 950 milliseconds to process the request and return a status code we will be billed for an additional second on every invocation. During this wait time, our Lambda function is asleep, but AWS is still billing us for that time. With AWS Lambda, you are billed for elapsed time, not for utilized CPU time. While Lambda is extraordinarily cheap, with high transaction volumes, these short waits of 950 milliseconds can add up to a significant bill.
Don’t Wait
So what happens if we simply do not call “await” and thus not wait on the response from our HTTP request?
Strange things happen.
Sometimes the request is sent and sometimes the request is not.
Sometimes the request is received immediately by the metrics aggregator and sometimes the request is received after our Lambda next runs. What is happening?
Freezing Lambda Containers
Lambda functions run inside an AWS Firecracker container. When you return from your Lambda function, AWS immediately freezes the container and its global state. When the Lambda is next invoked, it will thaw the container to service the new invocation.
If we send a HTTP request and that request is not fully sent over the network, if we return from the Lambda function, AWS will immediately suspend our Lambda container AND the partially sent request will be suspended as well. The request will remain frozen until the Lambda is next invoked and the Node event loop will then resume processing and the request will be fully transmitted.
How to Solve?
We could try a short sleep to give time for the request to be sent? But how long should we wait.
This hardly seems reliable.
Wait for Request Transmission
The correct solution is to use the Node req.end(,,callback) API and wait until the request is fully sent, but not wait for the response to be received. Here is a sample:
Notice that the request is resolved via the end callback on the “req” object and not on the “res” object in the previous example.
This modified request function should be invoked by our Lambda with “await”. In this case, we are not waiting for the HTTP response, but rather for the request to be fully sent. This is much faster than the 750 milliseconds to receive our metrics response, typically under 20 milliseconds.
Alternative Design Patterns
There are many other excellent ways to avoid waiting and blocking in Lambda functions such as using Step functions, SQS queues and directly invoking Lambda functions as Events without waiting. Consider the best approach for your app, but if you must use HTTP and you don’t need to wait for a response, consider the technique above to lower your wait time and AWS bill.
EmbedThis
Our EmbedThis Ioto service uses this technique. We needed the Ioto service to be exceptionally fast and not wait for any REST/HTTP API requests.
References
Here are some other good reads about Lambda and AWS asynchronous programming.
- Node event loop freezing:
https://levelup.gitconnected.com/avoiding-the-pitfalls-of-async-node-js-functions-in-aws-lambda-941220582e7a - Lambda Billing:
https://www.jeremydaly.com/serverless-tip-dont-overpay-when-waiting-on-remote-api-calls - Async programming:
https://read.acloud.guru/save-time-and-money-with-aws-lambda-using-asynchronous-programming-3548ea65f751 - Perils of step functions:
https://blog.scottlogic.com/2018/06/19/step-functions.html - Invoke Lambda async:
https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html - Lambda invocation overview:
https://docs.aws.amazon.com/lambda/latest/dg/lambda-invocation.html
Originally published at https://www.sensedeep.com on September 6, 2020.
