Cumulus Data Management Types
What Are The Cumulus Data Management Types
Monitoring Best Practices
This document intends to provide a set of recommendations and best practices for monitoring the state of a deployed Cumulus and diagnosing any issues.
S3 Server Access Logging
Via AWS Console
Cloudwatch Retention
Our lambdas dump logs to AWS CloudWatch. By default, these logs exist indefinitely. However, there are ways to specify a duration for log retention.
Setting S3 Lifecycle Policies
This document will outline, in brief, how to set data lifecycle policies so that you are more easily able to control data storage costs while keeping your data accessible. For more information on why you might want to do this, see the 'Additional Information' section at the end of the document.
Collection Cost Tracking and Storage Best Practices
Organizing your data is important for metrics you may want to collect. AWS S3 storage and cost metrics are calculated at the bucket level, so it is easy to get metrics by bucket. You can get storage metrics at the key prefix level, but that is done through the CLI, which can be very slow for large buckets. It is very difficult to estimate costs at the prefix level.
Configuration of Tasks
The cumulus module exposes values for configuration for some of the provided archive and ingest tasks. Currently the following are available as configurable variables: