Writing logs for ESDIS Metrics

info

This feature is only available for Cumulus deployments in NGAP environments.

prerequisite

You must configure your Cumulus deployment to deliver your logs to the correct shared logs destination for ESDIS metrics.

Log messages delivered to the ESDIS metrics logs destination conforming to an expected format will be automatically ingested and parsed to enable helpful searching/filtering of your logs via the ESDIS metrics Kibana dashboard.

Expected log format

The ESDIS metrics pipeline expects a log message to be a JSON string representation of an object (dict in Python or map in Java). An example log message might look like:

{
  "level": "info",
  "executions": "arn:aws:states:us-east-1:000000000000:execution:MySfn:abcd1234",
  "granules": "[\"granule-1\",\"granule-2\"]",
  "message": "hello world",
  "sender": "greetingFunction",
  "stackName": "myCumulus",
  "timestamp": "2018-10-19T19:12:47.501Z"
}

A log message can contain the following properties:

• executions: The AWS Step Function execution name in which this task is executing, if any
• granules: A JSON string of the array of granule IDs being processed by this code, if any
• level: A string identifier for the type of message being logged. Possible values:
  • debug
  • error
  • fatal
  • info
  • warn
  • trace
• message: String containing your actual log message
• parentArn: The parent AWS Step Function execution ARN that triggered the current execution, if any
• sender: The name of the resource generating the log message (e.g. a library name, a Lambda function name, an ECS activity name)
• stackName: The unique prefix for your Cumulus deployment
• timestamp: An ISO-8601 formatted timestamp
• version: The version of the resource generating the log message, if any

None of these properties are explicitly required for ESDIS metrics to parse your log correctly. However, a log without a message has no informational content. And having level, sender, and timestamp properties is very useful for filtering your logs. Including a stackName in your logs is helpful as it allows you to distinguish between logs generated by different deployments.

Using Cumulus Message Adapter libraries

If you are writing a custom task that is integrated with the Cumulus Message Adapter, then some of language specific client libraries can be used to write logs compatible with ESDIS metrics.

The usage of each library differs slightly, but in general a logger is initialized with a Cumulus workflow message to determine the contextual information for the task (e.g. granules, executions). Then, after the logger is initialized, writing logs only requires specifying a message, but the logged output will include the contextual information as well.

• cumulus-message-adapter-java: AdapterLogger.java
• cumulus-message-adapter-python: cumulus_logger.py

Writing logs using custom code

Any code that produces logs matching the expected log format can be processed by ESDIS metrics.

Node.js

Cumulus core provides a @cumulus/logger library that writes logs in the expected format for ESDIS metrics.