With Node.js v12+, native source maps can be applied to stack traces. Support for source maps is currently behind a flag and can be enabled in any Node.js v12+ Lambda function via the
NODE_OPTIONS environment variable:
The corresponding source map for your Lambda function must be uploaded alongside your handler code. If you are not already generating source maps, you’ll need to enable this with your bundler of choice.
Let’s look at a real-life example that I encountered recently. I picked up a ticket to investigate a pretty obscure production bug, where a Lambda function was returning an error response via API Gateway, with the message
Cannot read property 'val' of null. To begin with, this is not a particularly helpful error message! And the stack trace in CloudWatch Logs is not much more help:
TypeError: Cannot read property 'val' of null at Object.eval (eval at compileRestore (/var/task/src/handler/index.js:19131:20), <anonymous>:299:77) at Object.eval (eval at redactor (/var/task/src/handler/index.js:19016:18), <anonymous>:1195:10) at Pino.asJson (/var/task/src/handler/index.js:19991:45) at Pino.write (/var/task/src/handler/index.js:19646:28) at Pino.LOG (/var/task/src/handler/index.js:19914:21) at module.exports.__webpack_modules__.27750.exports.default (/var/task/src/handler/index.js:92651:24) at processTicksAndRejections (internal/process/task_queues.js:97:5)
serverless-webpack Serverless Framework plugin. Looking at this stack trace, I can just about tell that the error has something to do with the
pino logger. Beyond that, it is going to be extremely difficult to pinpoint the root cause of the error.
Source maps to the rescue!
Enabling native Node.js source maps on AWS Lambda requires two things: running Node.js with source maps enabled and generating (and deploying) source maps for your Lambda function code.
If you are using the Serverless Framework, your function configuration will need to look something like this:
functions: hello-world: name: hello-world handler: src/handler/index environment: NODE_OPTIONS: --enable-source-maps
After enabling source maps, you will notice corresponding
With source maps deployed and enabled, the stack trace for the same error now looks like this:
TypeError: Cannot read property 'val' of null at Object.eval (eval at compileRestore (/var/task/src/handler/webpack:/my-service/node_modules/@some-org/logger/node_modules/fast-redact/lib/restorer.js:16:1), <anonymous>:299:77) at Object.eval (eval at redactor (/var/task/src/handler/webpack:/my-service/node_modules/@some-org/logger/node_modules/fast-redact/lib/redactor.js:9:1), <anonymous>:1195:10) at Pino.asJson (/var/task/src/handler/webpack:/my-service/node_modules/@some-org/logger/node_modules/pino/lib/tools.js:115:1) at Pino.write (/var/task/src/handler/webpack:/my-service/node_modules/@some-org/logger/node_modules/pino/lib/proto.js:148:1) at Pino.LOG (/var/task/src/handler/webpack:/my-service/node_modules/@some-org/logger/node_modules/pino/lib/tools.js:38:1) at module.exports.__webpack_modules__.27750.exports.default (/var/task/src/handler/webpack:/my-service/src/index.js:98:1) at processTicksAndRejections (internal/process/task_queues.js:97:5)
From this source map-enhanced stack trace, I can now see that the error originated in the
fast-redact package, which is a dependency of
pino. Now that I have the file and line reference for the origin of the error I can now begin to isolate, replicate and, hopefully, fix the bug. 🙏