Cumulus API

The Cumulus API allows developers to interact with the Cumulus Framework, such as monitoring status or creating, editing, and deleting records. This is the same API that powers the Cumulus dashboard.

By utilizing this API, a developer can integrate with the Cumulus framework in any language or environment; although interacting with Cumulus through the Cumulus dashboard may be appropriate for many end users, for some use cases it's best to have the flexibility of a web-accessible API.

The API accepts and responds with JSON payloads at various HTTPS endpoints.

In order to use these endpoints, you must include authentication information in your HTTPS request; authentication is explained in the following section.

The following table lists the query string parameters that can be used with most of the Cumulus API endpoints. {fieldName} is a stand-in for any of the fields in the record, and for nested objects dot notation can be used; for example, valid fieldNames include: pdrName, status, and recipe.processStep.description.

query string parameter description
limit={number} number of records to be returned by the API call; default is 1, maximum is 100
page={number} page number, 1-indexed; default is 1
sort_by={fieldName} which field to sort by; default is timestamp
order={asc|desc} whether to sort in asc or desc order
sort_key[]={-fieldName1}&sort_key[]={fieldName2} One or more sort keys can be specified using the sort_key[] parameter. The order used impacts searching. Fields can be prepended with a - to sort in descending order or a + to sort in ascending. Ascending order is the default. The + must be escaped with %2B
prefix={value} startsWith search of the granuleId, status, pdrName, name, error, id and msg fields
infix={value} includes search of the granuleId, status, pdrName, name, error, id and msg fields
fields={fieldName1, fieldName2} which fields to return, separated by a comma
{fieldName}={value} exact value match for the given field
{fieldName}__from={number} for numeric fields, field value must be greater than the given number
{fieldName}__to={number} for numeric fields, field value must be less than the given number
{fieldName}__not={value} field does not match the given value
{fieldName}__in={value1, value2} field matches one of the values in the comma-separated list
{fieldName}__exists={true|false} field exists or doesn't exist in the record
q="fieldName:[1 TO 2] AND fieldName2:[3 TO 4]" arbitrary Apache Lucene query syntax, not needed for most uses of the API; if the q parameter is used, all other query parameters will be ignored, besides limit, page, and fields

Versioning

The Cumulus API is versioned and the current version is v1. Retrieve the latest API version from Cumulus.

To use any version, include the version number in the path before the query endpoint. {API_URL}/{version}/{query}

GET
/version

Example Request

$ curl https://example.com/version

Example Response

{
    "response_version": "v1",
    "api_version": "1.1.1"
}

Token

Returns a bearer token using oAuth with Earthdata Login service. The token will be returned as a JWT (JSON Web Token).

GET
/token

Query Parameters

query string parameter description
state={string} The URI to redirect to after if oAuth was successful

Example request

$ curl https://example.com/token

Example response

{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDY1MDk3ODczIn0.SxFtZ7dqp9KsUSn1uTXhWis8Il8Hig8mwLANGU3cXhY"}

Refresh token

Refreshes a bearer token received from oAuth with Earthdata Login service. The token will be returned as a JWT (JSON Web Token).

POST
/refresh

Request body

parameter type required description
token string true The JWT received from the /token endpoint to refresh

Example request

$ curl --request POST https://example.com/refresh --header 'Content-Type: application/json' --data '{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDY1MDk3ODczIn0.SxFtZ7dqp9KsUSn1uTXhWis8Il8Hig8mwLANGU3cXhY"}'

Example response

{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDcxMjk4ODEzIn0.vO6RlSRo47kkH15_muoUYNvv74fRzFxs7FlmVaarHlc"}

Delete token

Delete the record for an access token received from oAuth with Earthdata Login service.

token is the JWT containing access token information to delete

DELETE
/tokenDelete/{token}

Example request

$ curl --request DELETE https://example.com/tokenDelete/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDY1MDk3ODczIn0.SxFtZ7dqp9KsUSn1uTXhWis8Il8Hig8mwLANGU3cXhY

Example response

{"message": "Token record was deleted"}

Authorization header

When a request is made to the Cumulus API, it must contain a bearer token generated by the API.

The token is generated after the user login with Earthdata Login service.

The token is included in requests using the Authorization header.

If no token is provided, the Cumulus API server will respond with an error, requesting credentials. If an incorrect token is provided, the server will respond with a separate error noting this.

Authentication example

#! /bin/sh

ORIGIN=$(dirname $CUMULUS_BASEURL)
LOGIN_URL="$CUMULUS_BASEURL/token"

# create a base64 hash of your login credentials
AUTH=$(printf "$EARTHDATA_USERNAME:$EARTHDATA_PASSWORD" | base64)

# Request the Earthdata url with client id and redirect uri to use with Cumulus
AUTHORIZE_URL=$(curl -s -i ${LOGIN_URL} | grep location | sed -e "s/^location: //");

# Request an authorization grant code
TOKEN_URL=$(curl -s -i -X POST \
  -F "credentials=${AUTH}" \
  -H "Origin: ${ORIGIN}" \
  ${AUTHORIZE_URL%$'\r'} | grep Location | sed -e "s/^Location: //")

# Request the token through the CUMULUS API url that's returned from Earthdata
# Response is a JSON object of the form { token: String }
# This uses the cli tool jq to parse the JSON and get the token string
# More info on jq: https://stedolan.github.io/jq/
TOKEN=$(curl -s ${TOKEN_URL%$'\r'} | jq -r '.message.token')

echo $TOKEN

List providers

List providers in the Cumulus system.

GET
/providers

Example request

$ curl https://example.com/providers --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "daac-cumulus",
        "table": "provider",
        "limit": 1,
        "page": 1,
        "count": 2
    },
    "results": [
        {
            "id": "HTTP_MODIS",
            "globalConnectionLimit": 10,
            "protocol": "http",
            "host": "https://data.modis.gov/",
            "timestamp": 1508861082226
        }
    ]
}

Retrieve provider

Retrieve a single provider.

GET
/providers/{id}

Example request

$ curl https://example.com/providers/HTTP_MODIS --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "createdAt": 1508861081785,
    "id": "HTTP_MODIS",
    "host": "https://data.modis.gov/",
    "globalConnectionLimit": 10,
    "updatedAt": 1508861081785,
    "protocol": "http"
}

Create provider

Create a provider. For more information on creating providers and the contents of a request see the Cumulus setup documentation.

Overview of the schema fields:

Field Value Description
id string provider id/name
protocol "s3"|"http"|"https"|"ftp" file transfer (sync) protocol
host string provider host endpoint
port number provider host port
globalConnectionLimit number limit to number of concurrent connections
username string provider connection username
password string provider connection password
privateKey string filename assumed to be in s3://bucketInternal/stackName/crypto
cmKeyId string AWS KMS Customer Master Key arn or alias
allowedRedirects Array<string> Only hosts in this list will have the provider username/password forwarded for authentication. Entries should be specified as host.com or host.com:7000 if redirect port is different than the provider port.
certificateUri string Optional SSL Certificate S3 URI for custom or self-signed SSL (TLS) certificate
POST
/providers

Example request

$ curl --request POST https://example.com/providers --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
    "host": "https://www.example.gov",
    "id": "MY_DAAC_SATELLITE",
    "protocol": "http",
    "globalConnectionLimit": 10
}'

Example response

{
    "message": "Record saved",
    "record": {
        "createdAt": 1491941727851,
        "host": "https://www.example.gov",
        "id": "MY_DAAC_SATELLITE",
        "protocol": "http",
        "globalConnectionLimit": 10,
        "timestamp": 1513956151186
    }
}

Update/replace provider

Update/replace an existing provider. Expects payload to specify the entire provider object, and will completely replace the existing provider with the specified payload. For a field reference see "Create provider".

Returns status 200 on successful replacement, 400 if the id property in the payload does not match the corresponding value in the resource URI, or 404 if there is no provider with the specified ID.

PUT
/providers/{id}

Example request

$ curl --request PUT https://example.com/providers/MY_DAAC_SATELLITE --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
  "id": "MY_DAAC_SATELLITE",
  "host": "https://www.example.co.uk",
  "globalConnectionLimit": 10,
  "protocol": "http"
}'

Example successful response

{
  "createdAt": 1491941727851,
  "id": "MY_DAAC_SATELLITE",
  "host": "https://www.example.co.uk",
  "globalConnectionLimit": 10,
  "updatedAt": 1513956150733,
  "protocol": "http",
  "timestamp": 1513956555713
}

Delete provider

Delete a provider from Cumulus. The related PDRs and granules remain in the Cumulus and CMR systems.

DELETE
/providers/{id}

Example request

$ curl --request DELETE https://example.com/providers/MY_DAAC_SATELLITE --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Record deleted"
}

List collections

List collections in the Cumulus system.

If the query includes a value of true for getMMT, cumulus will search the CMR for each collection and insert the link to the Metadata Management Tool (MMT) into each result under MMTLink

If the query includes a value of true for includeStats, cumulus will compute and include the granule aggregation statistics in each collection's return value.

GET
/collections

Example request

$ curl https://example.com/collections --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "collection",
        "limit": 1,
        "page": 1,
        "count": 3
    },
    "results": [
        {
            "name": "MOD11A1",
            "version": "006",
            "dataType": "MOD11A1",
            "process": "modis",
            "provider_path": "/",
            "granuleId": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}$",
            "granuleIdExtraction": "(MOD11A1\\..*)\\.hdf",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
            "files": [
                {
                    "bucket": "protected",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
                },
                {
                    "bucket": "private",
                    "regex": "^BROWSE\\.MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
                    "sampleFileName": "BROWSE.MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
                },
                {
                    "bucket": "private",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf\\.met$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf.met"
                },
                {
                    "bucket": "protected",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.cmr\\.xml$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.cmr.xml"
                },
                {
                    "bucket": "public",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_2\\.jpg$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_2.jpg"
                },
                {
                    "bucket": "public",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_1\\.jpg$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_1.jpg"
                }
            ],
            "timestamp": 1513020427284,
            "createdAt": 1510761441174,
            "updatedAt": 1513020427162,
            "edpa": true,
            "some_other_field": "field",
            "duplicateHandling": "skip",
        }
    ]
}

List collections with active granules

List collections in the Cumulus system that have active associated granules.

If time parameters are specified, the query will return collections that have granules that have been updated in that time frame.

If the query includes a value of true for getMMT, cumulus will search the CMR for each collection and insert the link to the Metadata Management Tool (MMT) into each result under MMTLink

If the query includes a value of true for includeStats, cumulus will compute and include the granule aggregation statistics in each collection's return value.

GET
/collections/active

Example request

$ curl https://example.com/collections/active?includeStats=true --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "collection",
        "limit": 1,
        "page": 1,
        "count": 3
    },
    "results": [
        {
            "name": "MOD11A1",
            "version": "006",
            "dataType": "MOD11A1",
            "process": "modis",
            "provider_path": "/",
            "granuleId": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}$",
            "granuleIdExtraction": "(MOD11A1\\..*)\\.hdf",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
            "files": [
                {
                    "bucket": "protected",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
                },
                {
                    "bucket": "private",
                    "regex": "^BROWSE\\.MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
                    "sampleFileName": "BROWSE.MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
                },
                {
                    "bucket": "private",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf\\.met$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf.met"
                },
                {
                    "bucket": "protected",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.cmr\\.xml$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.cmr.xml"
                },
                {
                    "bucket": "public",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_2\\.jpg$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_2.jpg"
                },
                {
                    "bucket": "public",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_1\\.jpg$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_1.jpg"
                }
            ],
            "timestamp": 1513020427284,
            "createdAt": 1510761441174,
            "updatedAt": 1513020427162,
            "edpa": true,
            "some_other_field": "field",
            "duplicateHandling": "skip",
            "stats": {
                "running": 0,
                "completed": 6,
                "failed": 1,
                "total": 7
            }
        }
    ]
}

Retrieve collection

Retrieve a single collection.

GET
/collections/{name}/{version}

Example request

$ curl https://example.com/collections/MOD11A1/006 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "process": "modis",
    "granuleIdExtraction": "(MOD11A1\\..*)\\.hdf",
    "version": "006",
    "dataType": "MOD11A1",
    "some_other_field": "field",
    "createdAt": 1510761441174,
    "edpa": true,
    "name": "MOD11A1",
    "duplicateHandling": "skip",
    "provider_path": "/",
    "files": [
        {
            "bucket": "protected",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
            "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$"
        },
        {
            "bucket": "private",
            "sampleFileName": "BROWSE.MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
            "regex": "^BROWSE\\.MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$"
        },
        {
            "bucket": "private",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf.met",
            "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf\\.met$"
        },
        {
            "bucket": "protected",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.cmr.xml",
            "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.cmr\\.xml$"
        },
        {
            "bucket": "public",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_2.jpg",
            "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_2\\.jpg$"
        },
        {
            "bucket": "public",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_1.jpg",
            "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_1\\.jpg$"
        }
    ],
    "updatedAt": 1513020427162,
    "granuleId": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}$",
    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
}

Create collection

Create a collection. For more information on creating collections and the contents of a request see the Cumulus setup documentation. A collection generally includes, but is not limited to, the fields listed below.

Overview of the schema fields:

Field Value Description
name string collection name
version string collection version
dataType string matches collection with PDR datatype
duplicateHandling "replace"|"version"|"skip"|"error" duplicate handling protocol
process string process choice step variable
provider_path string path of remote files to sync
granuleId string (regex) regex to match granule IDs
granuleIdExtraction string (regex) regex to extract ID from files
sampleFileName string sample filename for granule ID
url_path string s3 prefix template
files array array of file specification objects
-- file.bucket string file destination bucket
-- file.regex string (regex) regex to match file names
-- file.sampleFileName string sample filename
-- file.url_path string s3 prefix template
-- file.fileType string granule file fileType mapping
POST
/collections

Example request

$ curl --request POST https://example.com/collections --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
  "name": "MOD09GQ",
  "version": "006",
  "dataType": "MOD09GQ",
  "process": "modis",
  "duplicateHandling": "replace",
  "provider_path": "cumulus-test-data/pdrs",
  "granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
  "granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
  "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
  "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
  "files": [
    {
      "bucket": "protected",
      "regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
      "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
      "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
    }
  ]
}'

Example response

{
    "message": "Record saved",
    "record": {
        "name": "MOD09GQ",
        "version": "006",
        "dataType": "MOD09GQ",
        "duplicateHandling": "replace",
        "process": "modis",
        "provider_path": "cumulus-test-data/pdrs",
        "granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
        "granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
        "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
        "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
        "files": [
            {
                "bucket": "protected",
                "regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
                "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
                "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
            }
        ],
        "createdAt": 1491946535919
    }
}

Update/replace collection

Update/replace an existing collection. Expects payload to specify the entire collection object, and will completely replace the existing collection with the specified payload. For a field reference see "Create collection".

Returns status 200 on successful replacement, 400 if the name or version property in the payload does not match the corresponding value in the resource URI, or 404 if there is no collection with the specified name and version.

PUT
/collections/{name}/{version}

Example request

$ curl --request PUT https://example.com/collections/MOD09GQ/006 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
  "name": "MOD09GQ",
  "version": "006",
  "dataType": "MOD09GQ",
  "duplicateHandling": "error",
  "newNeededField": "myCustomFieldValue",
  "process": "modis",
  "provider_path": "new_path/test-data",
  "granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
  "granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
  "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
  "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
  "files": [
    {
      "bucket": "protected",
      "regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
      "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
      "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
    }
  ]
}'

Example successful response

{
  "name": "MOD09GQ",
  "version": "006",
  "dataType": "MOD09GQ",
  "duplicateHandling": "error",
  "newNeededField": "myCustomFieldValue",
  "process": "modis",
  "provider_path": "new_path/test-data",
  "granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
  "granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
  "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
  "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
  "files": [
    {
      "bucket": "protected",
      "regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
      "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
      "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
    }
  ],
  "createdAt": 1491946535919,
  "updatedAt": 1514304825894
}

Delete collection

Delete a collection from Cumulus, but not from CMR. All related granules in Cumulus must have already been deleted from Cumulus.

DELETE
/collections/{name}/{version}

Example request

$ curl --request DELETE https://example.com/collections/MOD09GQ/006 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Record deleted"
}

List granules

List granules in the Cumulus system. If the query includes a value of true for getRecoveryStatus, recoveryStatus will be included in each granule's return value when applicable.

GET
/granules

Example request

$ curl https://example.com/granules --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "granule",
        "limit": 1,
        "page": 1,
        "count": 8
    },
    "results": [
        {
            "granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
            "pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
            "collectionId": "MOD11A1___006",
            "status": "completed",
            "provider": "LP_TS2_DataPool",
            "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:7f071dae1a93c9892272b7fd5",
            "files": [
                {
                    "bucket": "cumulus-devseed-protected",
                    "checksum": 964704694,
                    "key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
                    "fileSize": 1447347,
                    "fileType": "data",
                    "checksumType": "CKSUM",
                    "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
                },
                {
                    "bucket": "cumulus-devseed-private",
                    "checksum": 121318124,
                    "key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met",
                    "fileSize": 22559,
                    "fileType": "metadata",
                    "checksumType": "CKSUM",
                    "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met"
                },
                {
                    "bucket": "cumulus-devseed-private",
                    "checksum": 2188150664,
                    "key": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
                    "fileSize": 18118,
                    "fileType": "data",
                    "checksumType": "CKSUM",
                    "fileName": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
                },
                {
                    "bucket": "cumulus-devseed-public",
                    "key": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg",
                    "fileType": "browse",
                    "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg"
                },
                {
                    "bucket": "cumulus-devseed-protected",
                    "key": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml",
                    "fileType": "metadata",
                    "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml"
                },
                {
                    "bucket": "cumulus-devseed-public",
                    "key": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg",
                    "fileType": "browse",
                    "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg"
                }
            ],
            "error": null,
            "createdAt": 1513020455831,
            "timestamp": 1513020462156,
            "published": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
            "duration": 6.325,
            "cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS"
        }
    ]
}

Retrieve granule

Retrieve a single granule. If the query includes a value of true for getRecoveryStatus, the returned granule will include recoveryStatus when applicable.

GET
/granules/{granuleId}

Example request

$ curl https://example.com/granules/MOD11A1.A2017137.h20v17.006.2017138085755 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
    "pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
    "collectionId": "MOD11A1___006",
    "status": "completed",
    "provider": "LP_TS2_DataPool",
    "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:7f071dae1a93c9892272b7fd5",
    "files": [
        {
            "bucket": "cumulus-devseed-protected",
            "checksum": 964704694,
            "key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
            "fileSize": 1447347,
            "checksumType": "CKSUM",
            "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
        },
        {
            "bucket": "cumulus-devseed-private",
            "checksum": 121318124,
            "key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met",
            "fileSize": 22559,
            "checksumType": "CKSUM",
            "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met"
        },
        {
            "bucket": "cumulus-devseed-private",
            "checksum": 2188150664,
            "key": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
            "fileSize": 18118,
            "checksumType": "CKSUM",
            "fileName": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
        },
        {
            "bucket": "cumulus-devseed-protected",
            "key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
            "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
        },
        {
            "bucket": "cumulus-devseed-public",
            "key": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg",
            "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg"
        },
        {
            "bucket": "cumulus-devseed-protected",
            "key": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml",
            "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml"
        },
        {
            "bucket": "cumulus-devseed-public",
            "key": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg",
            "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg"
        }
    ],
    "error": null,
    "createdAt": 1513020455831,
    "timestamp": 1513020462156,
    "published": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
    "duration": 6.325,
    "cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
    "_id": "MOD11A1.A2017137.h20v17.006.2017138085755"
}

Create Granule

Create a granule. A granule can have the following fields.

Field Type Required Description
beginningDateTime string no The time when the granule's temporal coverage begins
cmrLink string no link to CMR
collectionId string yes Collection associated with the granule
createdAt integer no Time granule record was created (now)
duration number no Ingest duration milliseconds
endingDateTime string no The time when the graule's temporal coverage ends
error object no The error details for this granule
execution string no Step Function Execution URL
files array no Files associated with the granule
granuleId string yes Granule ID
lastUpdateDateTime string no The date/time that data provider last updated the granule info on data provider's database
pdrName string no PDR associated with the granule
processingEndDateTime string no Time processing of granule ends. Usually a StepFunction's stop time
processingStartDateTime string no Time processing of granule began. Usually a StepFunction's start time
productVolume number no Sum of the granule's file's sizes in bytes
productionDateTime string no The date and time a specific granule was produced by a PGE
provider string no Granule's provider
published boolean no If granule is published to CMR
queryFields object no Arbitrary query fields assigned to the granule
status string yes Ingest status of the granule one of ['running', 'completed', 'failed']
timeToArchive number no Time to post to CMR in seconds
timeToPreprocess number no Time to sync granule in seconds
timestamp integer no Timestamp for granule record (now)
updatedAt integer no Update Time for granule (now)
POST
/granules

Example request

$ curl --request POST https://example.com/granules \
  --header 'Authorization: Bearer ReplaceWithToken' \
  --header 'Content-Type: application/json' \
  --data '{
    "granuleId": "granuleId.A20200113.006.1005",
  "collectionId": "alpha___006",
  "status": "running",
  "beginningDateTime": "1995-01-01T00:00:00.000Z",
  "cmrLink": "https://mmt.uat.earthdata.nasa.gov/collections/C1237256734-CUMULUS",
  "createdAt": 1631547286190,
  "duration": 60000,
  "endingDateTime": "2021-09-13T00:00:00.000Z",
  "error": {},
  "execution": "https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestStepFunction2:cff1266e-ef36-664f-ff00-3a4d26bd1735",
  "files": [
    {
      "bucket": "stack-protected",
      "key": "granuleId.A20200113.006.1005.hdf",
      "fileName": "granuleId.A20200113.006.1005.hdf"
    }
  ],
  "lastUpdateDateTime":"2021-09-12T15:10:01.000Z",
  "pdrName": "aPdrName",
  "processingEndDateTime": "2020-10-13T23:59:59.999Z",
  "processingStartDateTime": "2020-10-13T00:00:00.000Z",
  "productVolume": 59632353,
  "productionDateTime": "2020-10-14T15:40:05.546Z",
  "provider": "s3-local",
  "published": true,
  "queryFields": {"custom": "values can go here"},
  "timeToArchive": 5001,
  "timeToPreprocess": 3240,
  "timestamp":1631547675248,
  "updatedAt":1631547675248
  }'

Example response

{
    "message": "Successfully wrote granule with Granule Id: granuleId.A20200113.006.1005"
}

Update or replace granule

Update/replace an existing granules. Expects payload to contain the modified parts of the granule and the existing granule values will be overwritten by the modified portions. The same fields are available as are for creating a granule..

Returns status 200 on successful replacement, 404 if the granuleId can not be found in the database, or 400 when the granuleId in the payload does not match the corresponding value in the resource URI.

PUT
/granules/{granuleId}

Example request

$ curl --request PUT https://example.com/granules/granuleId.A19990103.006.1000 \
  --header 'Authorization: Bearer ReplaceWithToken' \
  --header 'Content-Type: application/json' \
  --data '{
  "granuleId": "granuleId.A20200113.006.1005",
  "files": [
    {
      "bucket": "stack-protected",
      "key": "granuleId.A20200113.006.1005.hdf",
      "fileName": "granuleId.A20200113.006.1005.hdf"
    },
    {
      "bucket": "stack-protected",
      "key": "granuleId.A20200113.006.1005.jpg",
      "fileName": "granuleId.A20200113.006.1005.jpg"
    }
  ],
  "duration": 1000,
  "status": "completed"
  }'

Example response

{
    "message": "Successfully updated granule with Granule Id: granuleId.A20200113.006.1005"
}

Associate execution

Associate an execution with a granule. Returns status 200 on successful association, 404 if the granule or execution can not be found in the database, or 400 when the granuleId in the payload does not match the corresponding value in the resource URI.

The request should have the following fields.

Field Type Description
collectionId string Collection associated with the granule (e.g. <name>___<version> where <name> is the collection name and <version> is the collection version)
granuleId string Granule ID
executionArn string Execution arn
POST
/granules/{granuleId}/executions

Example request

$ curl --request POST https://example.com/granules/granuleId.A19990103.006.1000/executions \
  --header 'Authorization: Bearer ReplaceWithToken' \
  --header 'Content-Type: application/json' \
  --data '{
  "granuleId": "granuleId.A19990103.006.1000",
  "collectionId": "MOD09GQ___006",
  "executionArn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735"
  }'

Example response

{
    "message": "Successfully associated execution arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 with granule granuleId granuleId.A19990103.006.1000 collectionId MOD09GQ___006"
}

Reingest granule

Reingest a granule. This causes the granule to re-download to Cumulus from source, and begin processing from scratch. Reingesting a granule will overwrite existing granule files.

You trigger the reingest by posting with the data's action set to reingest.

A reingest request may also include one optional parameters, either executionArn or workflowName. Use of one of these will cause the reingest occur, but before running the ingest, a different original payload is found from existing executions and used in place of the most recent execution. This has the result of allowing you to reingest a granule with any of it's previous inputs. If executionArn is specified, the originalMessage is pulled directly from that execution. When workflowName is specified, the database is search to find all of the executions with that workflowName that were run on the granuleid, and the most recent originalMessage is pulled and used for the reingest. Remember only to supply either executionArn or workflowName, if both are present, workflowName is ignored and executionArn is used to determing the input message to the reingest.

A warning message is included in the response if the collection's duplicateHandling is not set to 'replace'.

PUT
/granules/{granuleId}

Example request

$ curl --request PUT https://example.com/granules/MOD11A1.A2017137.h20v17.006.2017138085755
       --header 'Authorization: Bearer ReplaceWithTheToken'
       --header 'Content-Type: application/json'
       --data '{"action": "reingest",
               ["executionArn": "arn:aws:states:us-east-1:123456789012:execution:stack-lambdaName:9da47a3b-4d85-4599-ae78-dbec2e042520"],
               ["workflowName": "TheWorkflowName"] }'

Example response

{
    "granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
    "action": "reingest",
    "status": "SUCCESS",
    "warning": "The granule files may be overwritten"
}

Apply workflow to granule

Apply the named workflow to the granule. Workflow input will be built from template and provided entire Cumulus granule record as payload.

PUT
/granules/{granuleid}

Example request

$ curl --request PUT https://example.com/granules/MOD11A1.A2017137.h19v16.006.2017138085750 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{ "action": "applyWorkflow", "workflow": "inPlaceWorkflow" }'

Example response

{
    "granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
    "action": "applyWorkflow inPlaceWorkflow",
    "status": "SUCCESS"
}

Move a granule

Move a granule from one location on S3 to another. Individual files are moved to specific locations by using a regex that matches their filenames.

PUT
/granules/{granuleId}

Example request

$ curl --request PUT https://example.com/granules/MOD11A1.A2017137.h19v16.006.2017138085750 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{ "action": "move", "destinations": [{ "regex": ".*.hdf$", "bucket": "s3-bucket", "filepath": "new/filepath/" }]}'

Example response

{
  "granuleId": "MOD11A1.A2017137.h19v16.006.2017138085750",
  "action": "move",
  "status": "SUCCESS"
}

Remove granule from CMR

Remove a Cumulus granule from CMR.

PUT
/granules/{granuleId}

Example request

$ curl --request PUT https://example.com/granules/MOD11A1.A2017137.h19v16.006.2017138085750 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"action": "removeFromCmr"}'

Example response

{
    "granuleId": "MOD11A1.A2017137.h19v16.006.2017138085750",
    "action": "removeFromCmr",
    "status": "SUCCESS"
}

Delete granule

Delete a granule from Cumulus. It must already be removed from CMR.

DELETE
/granules/{granuleId}

Example request

$ curl --request DELETE https://example.com/granules/1A0000-2016121001_002_001 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "detail": "Record deleted"
}

Bulk Operations

Apply a workflow to the granules provided. Granules can be sent as a list of IDs or as an Elasticsearch query to the Metrics' Elasticsearch.

Overview of the request fields:

Field Required Value Description
workflow Y string Worfklow to be applied to all granules
queueUrl N string URL of SQS queue to use for scheduling granule workflows (e.g. https://sqs.us-east-1.amazonaws.com/12345/queue-name)
ids yes - if query not present Array<string> List of IDs to process. Required if there is no Elasticsearch query provided
query yes - if ids not present Object Query to Elasticsearch to determine which Granules to go through given workflow. Required if no IDs are given.
index yes - if query is present string Elasticsearch index to search with the given query

Use the Retrieve async operation endpoint with the id in the response to determine the status of the async operation.

POST
/granules/bulk

Example request with Elasticsearch query generated by Kibana:

$ curl --request POST \
    https://example.com/granules/bulk --header 'Authorization: Bearer ReplaceWithTheToken' \
  --header 'Content-Type: application/json' \
  --data '{
        "workflowName": "HelloWorldWorkflow",
        "index": "index-in-es",
        "query": {
            "size": 500,
            "query": {
                "filter": [
                    {
                        "bool": {
                            "filter": [
                                {
                                    "bool": {
                                        "should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
                                        "minimum_should_match": 1
                                    }
                                },
                                {
                                    "bool": {
                                        "should": [{"match": {"status": "FAILED"}}],
                                        "minimum_should_match": 1
                                    }
                                }
                            ]
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    }'

Example request with given Granule IDs:

curl -X POST
  https://example.com/granules/bulk --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"ids": ["MOD09GQ.A2016358.h13v04.006.2016360104606"], "workflowName": "HelloWorldWorkflow"}'

Example response

{
    "createdAt": 1574730504000,
    "id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
    "updatedAt": 1574730504762,
    "status": "RUNNING",
    "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/d481e76e-f5fc-9c1c-2411-fa13779b111a"
}

Bulk Delete

Bulk delete the provided granules. Granules can be sent as a list of IDs or as an Elasticsearch query to the Metrics' Elasticsearch.

Overview of the request fields:

Field Required Value Description
forceRemoveFromCmr N bool Whether to remove published granules from CMR before deletion. You must set this value to true to do bulk deletion of published granules, otherwise deleting them will fail.
ids yes - if query not present Array<string> List of IDs to process. Required if there is no Elasticsearch query provided
query yes - if ids not present Object Query to Elasticsearch to determine which Granules to delete. Required if no IDs are given.
index yes - if query is present string Elasticsearch index to search with the given query

Use the Retrieve async operation endpoint with the id in the response to determine the status of the async operation.

POST
/granules/bulkDelete

Example request with Elasticsearch query generated by Kibana:

$ curl --request POST \
    https://example.com/granules/bulkDelete --header 'Authorization: Bearer ReplaceWithTheToken' \
  --header 'Content-Type: application/json' \
  --data '{
        "forceRemoveFromCmr": true,
        "index": "index-in-es",
        "query": {
            "size": 500,
            "query": {
                "filter": [
                    {
                        "bool": {
                            "filter": [
                                {
                                    "bool": {
                                        "should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
                                        "minimum_should_match": 1
                                    }
                                },
                                {
                                    "bool": {
                                        "should": [{"match": {"status": "FAILED"}}],
                                        "minimum_should_match": 1
                                    }
                                }
                            ]
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    }'

Example request with given Granule IDs:

curl -X POST
  https://example.com/granules/bulkDelete --header 'Authorization: Bearer ReplaceWithTheToken' --data '{"ids": ["MOD09GQ.A2016358.h13v04.006.2016360104606"], "forceRemoveFromCmr": true}'

Example response

{
    "createdAt": 1574730504000,
    "id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
    "updatedAt": 1574730504762,
    "status": "RUNNING",
    "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/d481e76e-f5fc-9c1c-2411-fa13779b111a"
}

Bulk Reingest

Reingest the granules provided. Granules can be sent as a list of IDs or as an Elasticsearch query to the Metrics' Elasticsearch.

Overview of the request fields:

Field Required Value Description
ids yes - if query not present Array<string> List of IDs to process. Required if there is no Elasticsearch query provided
query yes - if ids not present Object Query to Elasticsearch to determine which Granules to be reingested. Required if no IDs are given.
index yes - if query is present string Elasticsearch index to search with the given query
workflowName no string optional workflow name that allows different workflow and initial input to be used during reingest. See below.

An optional data parameter of workflowName is also available to allow you to override the input message to the reingest. If workflowName is specified, the original message is pulled directly by finding the most recent execution of the workflowName assocated with the granuleId.

Use the Retrieve async operation endpoint with the id in the response to determine the status of the async operation.

POST
/granules/bulkReingest

Example request with Elasticsearch query generated by Kibana:

$ curl --request POST \
    https://example.com/granules/bulkReingest --header 'Authorization: Bearer ReplaceWithTheToken' \
  --header 'Content-Type: application/json' \
  --data '{
        "index": "index-in-es",
        "query": {
            "size": 500,
            "query": {
                "filter": [
                    {
                        "bool": {
                            "filter": [
                                {
                                    "bool": {
                                        "should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
                                        "minimum_should_match": 1
                                    }
                                },
                                {
                                    "bool": {
                                        "should": [{"match": {"status": "FAILED"}}],
                                        "minimum_should_match": 1
                                    }
                                }
                            ]
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    }'

Example request with given Granule IDs and optional workflow parameter:

curl -X POST
  https://example.com/granules/bulkReingest
  --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json'
  --data '{
        "ids": ["MOD09GQ.A2016358.h13v04.006.2016360104606"],
        "workflow": "workflowName",
        }'

Example response

{
    "createdAt": 1574730504000,
    "description": "Bulk granule reingest run on 2 granules",
    "id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
    "operationType": "Bulk Granule Reingest",
    "status": "RUNNING",
    "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/d481e76e-f5fc-9c1c-2411-fa13779b111a",
    "updatedAt": 1574730504762
}

List PDRs

List PDRs in the Cumulus system.

GET
/pdrs

Example request

$ curl https://example.com/pdrs --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "pdr",
        "limit": 1,
        "page": 1,
        "count": 8
    },
    "results": [
        {
            "pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
            "collectionId": "MOD11A1___006",
            "status": "failed",
            "provider": "LP_TS2_DataPool",
            "progress": 0,
            "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:ACCOUNT:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:6ef0c52f83c549db58b3a1e50",
            "PANSent": false,
            "PANmessage": "N/A",
            "stats": {
                "processing": 0,
                "completed": 0,
                "failed": 0,
                "total": 0
            },
            "createdAt": 1514305411204,
            "timestamp": 1514305424036,
            "duration": 12.832
        }
    ]
}

Retrieve PDR

Retrieve a single PDR.

GET
/pdrs/{pdrName}

Example request

$ curl https://example.com/pdrs/7970bff5-128a-489f-b43c-de4ad7834ce5.PDR --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
    "collectionId": "MOD11A1___006",
    "status": "failed",
    "provider": "LP_TS2_DataPool",
    "progress": 0,
    "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:ACCOUNT:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:6ef0c52f83c549db58b3a1e50",
    "PANSent": false,
    "PANmessage": "N/A",
    "stats": {
        "processing": 0,
        "completed": 0,
        "failed": 0,
        "total": 0
    },
    "createdAt": 1514305411204,
    "timestamp": 1514305424036,
    "duration": 12.832,
    "_id": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR"
}

Delete PDR

Delete a PDR from Cumulus. Its granules will remain, and the PDR may be re-discovered and re-ingested/re-processed from scratch in the future.

DELETE
/pdrs/{pdrName}

Example request

$ curl --request DELETE https://example.com/pdrs/good_25grans.PDR --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Record deleted"
}

List rules

List rules in the Cumulus system.

GET
/rules

Example request

$ curl https://example.com/rules --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "rule",
        "limit": 1,
        "page": 1,
        "count": 7
    },
    "results": [
        {
            "name": "repeat",
            "workflow": "DiscoverPdrs",
            "provider": "local",
            "collection": {
                "name": "AST_L1A",
                "version": "003"
            },
            "rule": {
                "type": "scheduled",
                "value": "rate(5 minutes)"
            },
            "timestamp": 1511232462534,
            "state": "DISABLED"
        }
    ]
}

Retrieve rule

Retrieve a single rule.

GET
/rules/{name}

Example request

$ curl https://example.com/rules/repeat --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "workflow": "DiscoverPdrs",
    "collection": {
        "name": "AST_L1A",
        "version": "003"
    },
    "updatedAt": 1511232462507,
    "createdAt": 1510903518741,
    "provider": "local",
    "name": "repeat",
    "rule": {
        "type": "scheduled",
        "value": "rate(5 minutes)"
    },
    "state": "DISABLED"
}

Create rule

Create a rule. For more information on creating rules and the contents of a request see the Cumulus setup documentation.

Overview of the schema fields:

Field Value Description
name string rule name (letters, numbers, underscores only)
state "DISABLED"|"ENABLED" rule state (default: ENABLED)
workflow string name of workflow started by the rule
rule Object rule object
-- rule.type "onetime"|"scheduled"|"kinesis"|"sns"|"sqs" rule trigger type
-- rule.value onetime: N/A
scheduled: cron-type or rate expression
kinesis: Kinesis stream ARN
sns: SNS topic ARN
sqs: SQS queue URL
required value differs by type
-- rule.arn string kinesis scheduled event arn
-- rule.logEventArn string kinesis scheduled log event arn
provider string provider record provided to workflow (optional)
collection Object collection record provided to workflow (optional)
-- collection.name string collection name
-- collection.version string collection version
meta Object contents to add to workflow input's meta field
tags array Optional tags (for search)
POST
/rules

Example request

$ curl --request POST https://example.com/rules --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
    "workflow": "DiscoverPdrs",
    "collection": {
        "name": "AST_L1A",
        "version": "003"
    },
    "provider": "local",
    "name": "repeat_test",
    "rule": {
        "type": "scheduled",
        "value": "rate(5 minutes)"
    },
    "meta": { "publish": false },
    "state": "DISABLED"
}'

Example response

{
    "message": "Record saved",
    "record": {
        "workflow": "DiscoverPdrs",
        "collection": {
            "name": "AST_L1A",
            "version": "003"
        },
        "createdAt": 1510903518741,
        "provider": "local",
        "name": "repeat_test",
        "rule": {
            "type": "scheduled",
            "value": "rate(5 minutes)"
        },
        "meta": { "publish": false },
        "state": "DISABLED",
    }
}

Update/replace rule

Update/replace an existing rule. Expects payload to specify the entire rule object, and will completely replace the existing rule with the specified payload. For a field reference see "Create rule".

Returns status 200 on successful replacement, 400 if the name property in the payload does not match the corresponding value in the resource URI, or 404 if there is no rule with the specified name.

Special case:

Field Value Description
action "rerun" rerun rule (onetime rule only)
PUT
/rules/{name}

Example request

$ curl --request PUT https://example.com/rules/repeat_test --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
  "name": "repeat_test",
  "workflow": "DiscoverPdrs",
  "collection": {
    "name": "AST_L1A",
    "version": "003"
  },
  "provider": "local",
  "rule": {
    "type": "scheduled",
    "value": "rate(5 minutes)"
  },
  "state": "ENABLED"
}'

Example successful response

{
  "name": "repeat_test",
  "workflow": "DiscoverPdrs",
  "collection": {
    "name": "AST_L1A",
    "version": "003"
  },
  "updatedAt": 1521755265130,
  "createdAt": 1510903518741,
  "provider": "local",
  "rule": {
    "type": "scheduled",
    "value": "rate(5 minutes)"
  },
  "state": "ENABLED"
}

Rerun request (onetime-rule special case)

$ curl --request PUT https://example.com/rules/my_onetime_rule --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"action": "rerun"}'

Delete rule

Delete a rule from Cumulus.

DELETE
/rules/{name}

Example request

$ curl --request DELETE https://example.com/rules/repeat_test --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Record deleted"
}

Summary

Retrieve a summary of statistics around the granules in the system. The collections returned are the number of distinct collections for the granules active during the given time period, defaulted to the last day if none is specified.

GET
/stats

Example Request

$ curl https://example.com/stats --header 'Authorization: Bearer ReplaceWithTheToken'

Example success response

{
    "errors": {
        "dateFrom": "1970-01-18T12:36:59+00:00",
        "dateTo": "2017-12-26T04:38:15+00:00",
        "value": 2,
        "aggregation": "count",
        "unit": "error"
    },
    "collections": {
        "dateFrom": "1970-01-01T12:00:00+00:00",
        "dateTo": "2017-12-26T04:38:15+00:00",
        "value": 3,
        "aggregation": "count",
        "unit": "collection"
    },
    "processingTime": {
        "dateFrom": "1970-01-18T12:36:59+00:00",
        "dateTo": "2017-12-26T04:38:15+00:00",
        "value": null,
        "aggregation": "average",
        "unit": "second"
    },
    "granules": {
        "dateFrom": "1970-01-18T12:36:59+00:00",
        "dateTo": "2017-12-26T04:38:15+00:00",
        "value": 8,
        "aggregation": "count",
        "unit": "granule"
    }
}

Count

Count the value frequencies for a given field, for a given type of record in Cumulus. Requires the following query parameters, and may include the regular filter parameters:

query string parameter description
type={providers|collections|granules|pdrs|logs} type of Cumulus record to query
field={fieldName} which field to count frequencies for; no default
GET
/stats/aggregate

Example request

curl 'https://example.com/stats/aggregate?field=status&type=pdrs' --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "meta": {
    "name": "cumulus-api",
    "count": 52,
    "field": "status.keyword"
  },
  "count": [
    {
      "key": "failed",
      "count": 43
    },
    {
      "key": "completed",
      "count": 5
    },
    {
      "key": "parsed",
      "count": 3
    }
  ]
}

List logs

Note: This endpoint will only work if logs are being delivered to the Metrics system. Otherwise it will return a 400 Error.

List processing logs from the Cumulus engine. A log's level field should be specified by their string value and be one of:

  • fatal (60)
  • error (50)
  • warn (40)
  • info (30)
  • debug (20)
  • trace (10)
GET
/logs

Example request

$ curl https://example.com/logs?limit=5 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "logs",
        "limit": 5,
        "page": 1,
        "count": 750
    },
    "results": [
        {
            "pid": 1,
            "hostname": "ip-10-29-4-186",
            "name": "cumulus",
            "level": 50,
            "msg": "https://ba008ffc.ngrok.io/ not found",
            "file": "discover-pdrs/index.js",
            "type": "Error",
            "stack": "HostNotFound: https://ba008ffc.ngrok.io/ not found\n    at discover.discover.then.catch.e (/var/task/index.js:9006:22)\n    at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
            "message": "https://ba008ffc.ngrok.io/ not found",
            "v": 1,
            "timestamp": 1513881407867
        },
        {
            "pid": 1,
            "hostname": "ip-10-29-4-186",
            "name": "cumulus",
            "level": 50,
            "file": "discover-pdrs/index.js",
            "message": "Received a 404 error from undefined. Check your endpoint!",
            "details": {
                "host": "ba008ffc.ngrok.io",
                "path": "/",
                "port": "",
                "protocol": "https",
                "uriPath": "/",
                "url": "https://ba008ffc.ngrok.io/",
                "depth": 1,
                "fetched": true,
                "status": "notfound",
                "stateData": {
                    "requestLatency": 66,
                    "requestTime": 66,
                    "contentLength": 34,
                    "contentType": "text/plain",
                    "code": 404,
                    "headers": {
                        "content-length": "34",
                        "connection": "close",
                        "content-type": "text/plain"
                    }
                },
                "id": 0
            },
            "v": 1,
            "timestamp": 1513881407867
        },
        {
            "pid": 1,
            "hostname": "ip-10-29-4-186",
            "name": "cumulus",
            "level": 50,
            "msg": "Cannot read property 'provider_path' of undefined",
            "file": "discover-pdrs/index.js",
            "type": "Error",
            "stack": "TypeError: Cannot read property 'provider_path' of undefined\n    at HttpDiscover.Discover (/var/task/index.js:10098:33)\n    at HttpDiscover._class (/var/task/index.js:96368:255)\n    at new HttpDiscover (/var/task/index.js:10342:241)\n    at handler (/var/task/index.js:8968:23)\n    at Promise (/var/task/cumulus-sled/index.js:82:5)\n    at invokeHandler (/var/task/cumulus-sled/index.js:72:10)\n    at then.then.then (/var/task/cumulus-sled/index.js:112:14)\n    at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
            "v": 1,
            "timestamp": 1513881031694
        },
        {
            "pid": 1,
            "hostname": "ip-10-29-4-186",
            "name": "cumulus",
            "level": 50,
            "msg": "Provider info not provided",
            "file": "discover-pdrs/index.js",
            "type": "Error",
            "stack": "ProviderNotFound: Provider info not provided\n    at handler (/var/task/index.js:8962:20)\n    at Promise (/var/task/cumulus-sled/index.js:82:5)\n    at invokeHandler (/var/task/cumulus-sled/index.js:72:10)\n    at then.then.then (/var/task/cumulus-sled/index.js:112:14)\n    at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
            "message": "Provider info not provided",
            "v": 1,
            "timestamp": 1513879654040
        },
        {
            "pid": 1,
            "hostname": "ip-10-29-4-186",
            "name": "cumulus",
            "level": 50,
            "msg": "Provider info not provided",
            "file": "discover-pdrs/index.js",
            "type": "Error",
            "stack": "ProviderNotFound: Provider info not provided\n    at handler (/var/task/index.js:8962:20)\n    at Promise (/var/task/cumulus-sled/index.js:82:5)\n    at invokeHandler (/var/task/cumulus-sled/index.js:72:10)\n    at then.then.then (/var/task/cumulus-sled/index.js:112:14)\n    at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
            "message": "Provider info not provided",
            "v": 1,
            "timestamp": 1513879435010
        }
    ]
}

Example request for errors

$ curl https://example.com/logs?level=error --header 'Authorization: Bearer ReplaceWithTheToken'

Example response for errors

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "logs",
        "limit": 1,
        "page": 1,
        "count": 181
    },
    "results": [
        {
            "level": 50,
            "msg": "Unexpected HTTP status code: 404",
            "pid": 1,
            "hostname": "ip-10-26-125-174",
            "name": "cumulus",
            "file": "sync-granule/index.js",
            "type": "Error",
            "stack": "Error: Unexpected HTTP status code: 404\n    at ClientRequest.<anonymous> (/var/task/sync-granule/index.js:125136:29)\n    at ClientRequest.g (events.js:292:16)\n    at emitOne (events.js:96:13)\n    at ClientRequest.emit (events.js:188:7)\n    at HTTPParser.parserOnIncomingClient (_http_client.js:473:21)\n    at HTTPParser.parserOnHeadersComplete (_http_common.js:99:23)\n    at TLSSocket.socketOnData (_http_client.js:362:20)\n    at emitOne (events.js:96:13)\n    at TLSSocket.emit (events.js:188:7)\n    at readableAddChunk (_stream_readable.js:176:18)",
            "code": 404,
            "v": 1,
            "timestamp": 1515446810226
        }
    ]
}

Retrieve Logs

Note: This endpoint will only work if logs are being delivered to the Metrics system. Otherwise it will return a 400 Error.

Retrieve all logs from a specific execution.

GET
/logs/{executionName}

Example request

$ curl https://example.com/logs/93e5e049-89aa-47e5-900d-0eec87327260?limit=5 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "test-src-integration",
        "table": "logs",
        "limit": 5,
        "page": 1,
        "count": 10
    },
    "results": [
        {
            "level": 30,
            "executions": "93e5e049-89aa-47e5-900d-0eec87327260",
            "timestamp": "2018-08-15T19:23:57.752Z",
            "sender": "test-src-integration-PostToCmr",
            "version": "$LATEST",
            "message": "Published MOD09GQ.A8009365.7JQhni.006.6594428626875 to the CMR. conceptId: G1223210348-CUMULUS",
            "RequestId": "05712fb1-5593-4640-b5a3-a99db41d8003"
        },
        {
            "level": 30,
            "executions": "93e5e049-89aa-47e5-900d-0eec87327260",
            "timestamp": "2018-08-15T19:23:42.054Z",
            "sender": "test-src-integration-SyncGranuleNoVpc",
            "version": "$LATEST",
            "message": "uploaded s3://cumulus-test-sandbox-internal/file-staging/test-src-integration/MOD09GQ___006/MOD09GQ.A8009365.7JQhni.006.6594428626875.hdf",
            "RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
        },
        {
            "level": 30,
            "executions": "93e5e049-89aa-47e5-900d-0eec87327260",
            "timestamp": "2018-08-15T19:23:41.913Z",
            "sender": "test-src-integration-SyncGranuleNoVpc",
            "version": "$LATEST",
            "message": "Finishing downloading s3://cumulus-test-sandbox-internal/cumulus-test-data/pdrs/MOD09GQ.A8009365.7JQhni.006.6594428626875.hdf",
            "RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
        },
        {
            "level": 30,
            "executions": "93e5e049-89aa-47e5-900d-0eec87327260",
            "timestamp": "2018-08-15T19:23:41.888Z",
            "sender": "test-src-integration-SyncGranuleNoVpc",
            "version": "$LATEST",
            "message": "uploaded s3://cumulus-test-sandbox-internal/file-staging/test-src-integration/MOD09GQ___006/MOD09GQ.A8009365.7JQhni.006.6594428626875_ndvi.jpg",
            "RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
        },
        {
            "level": 30,
            "executions": "93e5e049-89aa-47e5-900d-0eec87327260",
            "timestamp": "2018-08-15T19:23:41.886Z",
            "sender": "test-src-integration-SyncGranuleNoVpc",
            "version": "$LATEST",
            "message": "uploaded s3://cumulus-test-sandbox-internal/file-staging/test-src-integration/MOD09GQ___006/MOD09GQ.A8009365.7JQhni.006.6594428626875.hdf.met",
            "RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
        }
    ]
}

Get Granule CSV file

Get a CSV file of all the granule in the Cumulus database.

GET
/granule-csv

Example request

$ curl https://example.com/granule-csv --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

"granuleUr","collectionId","createdAt","startDateTime","endDateTime","status","updatedAt","published"
"MOD14A1.A9506271.IvEJsu.006.8359924290786","MOD14A1___006","2020-05-18T20:15:54.525Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-05-18T20:16:02.473Z",false
"MYD13Q1.A9663671.0zkwKH.006.9812354158395","MYD13Q1___006","2020-07-06T19:46:19.957Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-07-06T19:46:57.054Z",true

List executions

List executions.

GET
/executions

Example request

$ curl https://example.com/executions?limit=3 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "meta": {
    "name": "cumulus-api",
    "stack": "test-src-integration",
    "table": "execution",
    "limit": 3,
    "page": 1,
    "count": 447
  },
  "results": [
    {
      "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
      "duration": 1.693,
      "createdAt": 1534443719571,
      "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
      "name": "08aa40f8-7d91-41b4-986e-d12ce495aec5",
      "error": {},
      "type": "IngestGranule",
      "collectionId": "MOD09GQ___006",
      "tasks": {},
      "status": "running",
      "timestamp": 1534443721559,
      "updatedAt": 1534443721265
    },
    {
      "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
      "duration": 1.718,
      "createdAt": 1534443706867,
      "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
      "name": "bee18782-e987-4bd4-b726-41669e489e2f",
      "error": {},
      "type": "IngestGranule",
      "collectionId": "MOD09GQ___006",
      "tasks": {},
      "status": "running",
      "timestamp": 1534443709043,
      "updatedAt": 1534443708585
    },
    {
      "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
      "duration": 29.397,
      "createdAt": 1534443668185,
      "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
      "name": "55d62b27-50cd-4cc1-81f9-e425cf10532e",
      "error": {
        "Cause": "None",
        "Error": "Unknown Error"
      },
      "type": "IngestGranule",
      "collectionId": "MOD09GQ___006",
      "tasks": {},
      "status": "completed",
      "timestamp": 1534443698322,
      "updatedAt": 1534443697582
    }
  ]
}

Retrieve Execution

Retrieve details for a specific execution.

GET
/executions/{executionArn}

Example request

$ curl https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
    "updatedAt": 1534441782302,
    "status": "completed",
    "timestamp": 1534441782302,
    "tasks":
    {
        "MoveGranuleStep":
        {
            "name": "test-src-integration-MoveGranules",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-MoveGranules",
            "version": "$LATEST"
        },
        "ProcessingStep":
        {
            "name": "test-src-integration-FakeProcessing",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-FakeProcessing",
            "version": "$LATEST"
        },
        "StopStatus":
        {
            "name": "test-src-integration-SfSnsReport",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-SfSnsReport",
            "version": "$LATEST"
        }
    },
    "createdAt": 1534441752026,
    "duration": 30.276,
    "name": "cff1266e-ef36-664f-a649-3a4d26bd1735",
    "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
    "collectionId": "MOD09GQ___006",
    "error":
    {
        "Error": "Unknown Error",
        "Cause": "None"
    },
    "type": "IngestGranule",
    "originalPayload": { "somePayloadKey": "object containing payload at execution start" },
    "finalPayload": { "somePayloadKey" : "object containing the last reported payload from an execution" }
}

Retrieve Execution Status

Retrieve details and status of a specific execution. This also returns execution history and details of the state machine when the execution exists in Step Function API.

GET
/executions/status/{executionArn}

Example request

$ curl https://example.com/executions/status/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl6sgv:d0a6584b-bea6-476e-a745-c1feb2ad00b2 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "execution":
    {
        "executionArn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv:d0a6584b-bea6-476e-a745-c1feb2ad00b2",
        "stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv",
        "name": "d0a6584b-bea6-476e-a745-c1feb2ad00b2",
        "status": "SUCCEEDED",
        "startDate": "2018-08-16T18:39:32.209Z",
        "stopDate": "2018-08-16T18:39:59.089Z",
        "input":
        {
            "foo": "input"
        },
        "output":
        {
            "bar": "output"
        }
    },
    "executionHistory":
    {
        "events":
        [
            {
                "taskName": "foo bar 1"
            },
            {
                "taskName": "foo bar 2"
            }
        ]
    },
    "stateMachine":
    {
        "stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv",
        "name": "TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv",
        "status": "ACTIVE",
        "definition":
        {
            "Comment": "Ingest Granule",
            "StartAt": "First State",
            "States":
            {
                "First State": {}
            }
        },
        "roleArn": "arn:aws:iam::123456789012:role/test-src-integration-steprole",
        "creationDate": "2018-06-11T16:08:17.533Z"
    }
}

Search Executions By Granules

Return executions associated with specific granules. Granules can be sent as a list of granule objects containing granuleIds and collectionIds or as an Elasticsearch query to the Metrics' Elasticsearch.

Overview of the request fields:

Field Required Value Description
granules yes - if query not present Array<Object> List of granules to process. Each granule must contain a granuleId and collectionId
-- granule.granuleId yes - if using granules string GranuleId for a granule
-- granule.collectionId yes - if using granules string CollectionId for a granule
query yes - if granules not present Object Query to Elasticsearch to determine which Granules with which to search. Required if no IDs are given.
index yes - if query is present string Elasticsearch index to search with the given query
POST
/executions/search-by-granules

Example request with given Granules:

curl -X POST
  https://example.com/executions/search-by-granules?limit=3 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"granules": [{ "granuleId":"MOD09GQ.A2016358.h13v04.006.2016360104606", "collectionId": "MOD09GQ___006"}]}'

Example request with Elasticsearch query generated by Kibana:

$ curl --request POST \
    https://example.com/executions/search-by-granules?limit=3 --header 'Authorization: Bearer ReplaceWithTheToken' \
  --header 'Content-Type: application/json' \
  --data '{
        "index": "index-in-es",
        "query": {
            "size": 500,
            "query": {
                "filter": [
                    {
                        "bool": {
                            "filter": [
                                {
                                    "bool": {
                                        "should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
                                        "minimum_should_match": 1
                                    }
                                },
                                {
                                    "bool": {
                                        "should": [{"match": {"collectionId": "MOD09GQ__006"}}],
                                        "minimum_should_match": 1
                                    }
                                }
                            ]
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    }'

Example response

{
   "meta": {
       "count": 3
    },
    "results": [
        {
            "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
            "duration": 1.693,
            "createdAt": 1534443719571,
            "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
            "name": "08aa40f8-7d91-41b4-986e-d12ce495aec5",
            "error": {},
            "type": "IngestGranule",
            "collectionId": "MOD09GQ___006",
            "tasks": {},
            "status": "running",
            "timestamp": 1534443721559,
            "updatedAt": 1534443721265
        },
        {
            "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
            "duration": 1.718,
            "createdAt": 1534443706867,
            "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
            "name": "bee18782-e987-4bd4-b726-41669e489e2f",
            "error": {},
            "type": "IngestGranule",
            "collectionId": "MOD09GQ___006",
            "tasks": {},
            "status": "running",
            "timestamp": 1534443709043,
            "updatedAt": 1534443708585
        },
        {
            "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
            "duration": 29.397,
            "createdAt": 1534443668185,
            "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
            "name": "55d62b27-50cd-4cc1-81f9-e425cf10532e",
            "error": {
            "Cause": "None",
            "Error": "Unknown Error"
            },
            "type": "IngestGranule",
            "collectionId": "MOD09GQ___006",
            "tasks": {},
            "status": "completed",
            "timestamp": 1534443698322,
            "updatedAt": 1534443697582
        }
    ]
}

Workflows By Granules

Return the workflows that have run on specific granules. If multiple granules are specified, it will return the intersection of workflows that have run on ALL the granules. That is, only a workflow that has been run on every inputted granule will be returned

Overview of the request fields:

Field Required Value Description
granules yes - if query not present Array<Object> List of granules to process. Each granule must contain a granuleId and collectionId
-- granule.granuleId yes - if using granules string GranuleId for a granule
-- granule.collectionId yes - if using granules string CollectionId for a granule
query yes - if granules not present Object Query to Elasticsearch to determine which Granules with which to search. Required if no IDs are given.
index yes - if query is present string Elasticsearch index to search with the given query
POST
/executions/workflows-by-granules

Example request with given Granules:

curl -X POST
  https://example.com/executions/workflows-by-granules --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"granules": [{ "granuleId":"MOD09GQ.A2016358.h13v04.006.2016360104606", "collectionId": "MOD09GQ___006"}]}'

Example request with Elasticsearch query generated by Kibana:

$ curl --request POST \
    https://example.com/executions/workflows-by-granules --header 'Authorization: Bearer ReplaceWithTheToken' \
  --header 'Content-Type: application/json' \
  --data '{
        "index": "index-in-es",
        "query": {
            "size": 500,
            "query": {
                "filter": [
                    {
                        "bool": {
                            "filter": [
                                {
                                    "bool": {
                                        "should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
                                        "minimum_should_match": 1
                                    }
                                },
                                {
                                    "bool": {
                                        "should": [{"match": {"collectionId": "MOD09GQ__006"}}],
                                        "minimum_should_match": 1
                                    }
                                }
                            ]
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    }'

Example response

[
    "DiscoverGranules",
    "IngestGranules"
]

Create execution

Create an execution. An execution generally includes the fields listed below.

Overview of the schema fields:

Field Type Required Description
arn string yes execution arn (this field is unique)
asyncOperationId string no id of the associated async operation
collectionId string no collectionId of the associated collection (e.g. <name>___<version> where <name> is the collection name and <version> is the collection version)
cumulusVersion string no cumulus version for the execution (e.g. 9.0.0)
duration number no duration of the execution
error object no the error details in case of a failed execution
execution string no the execution page url on AWS console
finalPayload object no the final payload of this execution
name string yes execution name
originalPayload object no the original payload for this execution
parentArn string no arn of the parent execution
status string yes the execution status, possible values: running, completed, failed, unknown
tasks object no the tasks for this execution
type string no the workflow name, e.g. IngestGranule
timestamp number no the timestamp for this execution
POST
/executions

Example request

$ curl --request POST https://example.com/executions --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
    "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
    "asyncOperationId": "c4ece1f9-dca0-42f8-be77-da43b9b25918",
    "collectionId": "MOD09GQ___006",
    "cumulusVersion": "1.2.3",
    "error": {},
    "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
    "name": "cff1266e-ef36-664f-a649-3a4d26bd1735",
    "originalPayload": { "somePayloadKey": "object containing payload at execution start" },
    "parentArn": "arn:aws:states:us-east-1:123456789012:execution:test-src-integration-ParsePdr:59edd6a4-5187-4865-b3c3-71ebcf76d7a1",
    "status": "running",
    "tasks":
    {
        "MoveGranuleStep":
        {
            "name": "test-src-integration-MoveGranules",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-MoveGranules",
            "version": "$LATEST"
        },
        "ProcessingStep":
        {
            "name": "test-src-integration-FakeProcessing",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-FakeProcessing",
            "version": "$LATEST"
        },
        "StopStatus":
        {
            "name": "test-src-integration-SfSnsReport",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-SfSnsReport",
            "version": "$LATEST"
        }
    },
    "type": "IngestGranule",
    "timestamp": 1513020462156
}'

Example response

{
    "message": "Successfully wrote execution with arn arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735"
}

Update or replace execution

Update/replace an existing execution. Expects payload to specify the entire execution object, and will completely replace the existing execution with the specified payload. For a field reference see "Create execution".

Returns status 200 on successful replacement, 400 if the arn property in the payload does not match the corresponding value in the resource URI, or 404 if there is no execution with the specified arn.

PUT
/executions/{executionArn}

Example request

$ curl --request PUT https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
    "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
    "asyncOperationId": "c4ece1f9-dca0-42f8-be77-da43b9b25918",
    "collectionId": "MOD09GQ___006",
    "cumulusVersion": "1.2.3",
    "duration": 30.276,
    "error": {},
    "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
    "finalPayload": { "somePayloadKey" : "object containing the last reported payload from an execution" },
    "name": "cff1266e-ef36-664f-a649-3a4d26bd1735",
    "originalPayload": { "somePayloadKey": "object containing payload at execution start" },
    "parentArn": "arn:aws:states:us-east-1:123456789012:execution:test-src-integration-ParsePdr:59edd6a4-5187-4865-b3c3-71ebcf76d7a1",
    "status": "completed",
    "tasks":
    {
        "MoveGranuleStep":
        {
            "name": "test-src-integration-MoveGranules",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-MoveGranules",
            "version": "$LATEST"
        },
        "ProcessingStep":
        {
            "name": "test-src-integration-FakeProcessing",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-FakeProcessing",
            "version": "$LATEST"
        },
        "StopStatus":
        {
            "name": "test-src-integration-SfSnsReport",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-SfSnsReport",
            "version": "$LATEST"
        }
    },
    "type": "IngestGranule",
    "timestamp": 1513020462156
}'

Example response

{
    "message": "Successfully updated execution with arn arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735"
}

Delete execution

Delete an execution from Cumulus.

DELETE
/executions/{executionArn}

Example request

$ curl --request DELETE https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Record deleted"
}

List workflows

List workflows in the Cumulus system.

GET
/workflows

Example request

$ curl https://example.com/workflows --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

[
    {
        "name": "TestLambdaVersionWorkflow",
        "template": "s3://cumulus-test-sandbox-internal/cumulus/workflows/TestLambdaVersionWorkflow.json",
        "definition": {
            "Comment": "Tests Lambda update after redeploy",
            "StartAt": "StartStatus",
            "States": {
                "StartStatus": {
                    "Type": "Task",
                    "Resource": "${SfSnsReportLambdaAliasOutput}",
                    "Next": "WaitForDeployment"
                },
                "WaitForDeployment": {
                    "Type": "Task",
                    "Resource": "${WaitForDeploymentLambdaAliasOutput}",
                    "Next": "VersionUpTest"
                },
                "VersionUpTest": {
                    "Type": "Task",
                    "Resource": "${VersionUpTestLambdaAliasOutput}",
                    "Next": "StopStatus"
                },
                "StopStatus": {
                    "Type": "Task",
                    "Resource": "${SfSnsReportLambdaAliasOutput}",
                    "Catch": [
                        {
                            "ErrorEquals": [
                                "States.ALL"
                            ],
                            "Next": "WorkflowFailed"
                        }
                    ],
                    "End": true
                },
                "WorkflowFailed": {
                    "Type": "Fail",
                    "Cause": "Workflow failed"
                }
            }
        }
    },
    {
        "name": "HelloWorldWorkflow",
        "template": "s3://cumulus-test-sandbox-internal/cumulus/workflows/HelloWorldWorkflow.json",
        "definition": {
            "Comment": "Returns Hello World",
            "StartAt": "StartStatus",
            "States": {
                "StartStatus": {
                    "Type": "Task",
                    "Resource": "${SfSnsReportLambdaAliasOutput}",
                    "Next": "HelloWorld"
                },
                "HelloWorld": {
                    "Type": "Task",
                    "Resource": "${HelloWorldLambdaAliasOutput}",
                    "Next": "StopStatus"
                },
                "StopStatus": {
                    "Type": "Task",
                    "Resource": "${SfSnsReportLambdaAliasOutput}",
                    "Catch": [
                        {
                            "ErrorEquals": [
                                "States.ALL"
                            ],
                            "Next": "WorkflowFailed"
                        }
                    ],
                    "End": true
                },
                "WorkflowFailed": {
                    "Type": "Fail",
                    "Cause": "Workflow failed"
                }
            }
        }
    }
]

Retrieve workflow

Retrieve a single workflow.

GET
/workflow/{name}

Example request

$ curl https://example.com/workflows/HelloWorldWorkflow --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "name": "HelloWorldWorkflow",
    "template": "s3://cumulus-test-sandbox-internal/cumulus/workflows/HelloWorldWorkflow.json",
    "definition": {
        "Comment": "Returns Hello World",
        "StartAt": "StartStatus",
        "States": {
            "StartStatus": {
                "Type": "Task",
                "Resource": "${SfSnsReportLambdaAliasOutput}",
                "Next": "HelloWorld"
            },
            "HelloWorld": {
                "Type": "Task",
                "Resource": "${HelloWorldLambdaAliasOutput}",
                "Next": "StopStatus"
            },
            "StopStatus": {
                "Type": "Task",
                "Resource": "${SfSnsReportLambdaAliasOutput}",
                "Catch": [
                    {
                        "ErrorEquals": [
                            "States.ALL"
                        ],
                        "Next": "WorkflowFailed"
                    }
                ],
                "End": true
            },
            "WorkflowFailed": {
                "Type": "Fail",
                "Cause": "Workflow failed"
            }
        }
    }
}

List async operations

Async operations are long-running requests serviced by the Cumulus API.

These tend to be bulk operations. Examples include bulk granule operations and replaying ingest notifications.

This endpoint lists async operations in the Cumulus system.

GET
/asyncOperations

Example request

$ curl https://example.com/asyncOperations --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "asyncOperation",
        "limit": 1,
        "page": 1,
        "count": 8
    },
    "results": [
        {
          "output": "{\"deletedGranules\":[\"granule-id-f02a53418f\"]}",
          "createdAt": 1591384094512,
          "taskArn": "arn:aws:ecs:us-east-1:111111111111:task/d481e76e-f5fc-9c1c-2411-fa13779b111a",
          "description": "Bulk granule deletion",
          "operationType": "Bulk Granule Delete",
          "id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
          "status": "SUCCEEDED",
          "updatedAt": 1591384094512,
          "timestamp": 1591384095235
        }
    ]
}

Retrieve async operation

Retrieve information about a single async operation. Useful for determining the status of an async operation.

GET
/asyncOperations/{id}

Example request

$ curl https://example.com/asyncOperations/0eb8e809-8790-5409-1239-bcd9e8d28b8e --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
  "updatedAt": 1574730504762,
  "status": "RUNNING",
  "taskArn": "arn:aws:ecs:us-east-1:111111111111:task/d481e76e-f5fc-9c1c-2411-fa13779b111a",
  "description": "Bulk granule deletion",
  "operationType": "Bulk Granule Delete"
}

Ingest Replays

The Cumulus API supports requests to replay ingest notifications. The schema below describes the expected fields in a replay request

Field Type Required Description
type string required Currently only accepts kinesis.
kinesisStream string for type kinesis Any valid kinesis stream name (not ARN)
kinesisStreamCreationTimestamp * optional Any input valid for a JS Date constructor. For reasons to use this field see AWS documentation on StreamCreationTimestamp.
endTimestamp * optional Any input valid for a JS Date constructor. Messages newer than this timestamp will be skipped.
startTimestamp * optional Any input valid for a JS Date constructor. Messages will be fetched from the Kinesis stream starting at this timestamp. Ignored if it is further in the past than the stream's retention period.

Example Request

$ curl -X POST https://example.com/replays --header 'Authorization: Bearer ReplaceWithTheToken' --data '{ "type: "kinesis", "kinesisStream": "my-stream", "endTimestamp": 1567890123456, "startTimestamp": "2019-08-31T22:22:03.456Z"}'

Example Response

{
  "asyncOperationId": "208a463a-e096-4dd9-b006-e09345319ae6"
}

Retrieve schema

Retrieve the data schema for a particular type of Cumulus record.

This schema describes the expected format of a record's JSON object when retrieving from Cumulus, as well as a summary of what each field may contain. The schema response can also be used to determine which fields are required when creating a new record using the API.

Supported type values are provider, collection, granule, and pdr.

GET
/schemas/{type}

Example request

$ curl https://example.com/schemas/provider --header 'Authorization: Bearer ReplceWithTheToken'

Example response

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "title": "Provider Object",
  "description": "Keep the information about each ingest endpoint",
  "type": "object",
  "properties": {
    "name": {
      "title": "Title",
      "description": "A title for the provider record",
      "type": "string",
      "pattern": "^([\\w\\d_\\-]*)$"
    },
    "providerName": {
      "title": "Provider, e.g. MODAPS",
      "description": "Name of the SIP",
      "type": "string"
    },
    "protocol": {
      "title": "Protocol",
      "type": "string",
      "enum": [
        "http",
        "ftp"
      ],
      "default": "http"
    },
    "host": {
      "title": "Host",
      "type": "string"
    },
    "path": {
      "title": "Path to the PDR/files folder",
      "type": "string"
    },
    "config": {
      "title": "Configuration",
      "type": "object",
      "properties": {
        "username": {
          "type": "string"
        },
        "password": {
          "type": "string"
        },
        "port": {
          "type": "string"
        }
      }
    },
    "status": {
      "title": "Status",
      "type": "string",
      "enum": [
        "ingesting",
        "stopped",
        "failed"
      ],
      "default": "stopped",
      "readonly": true
    },
    "isActive": {
      "title": "Is Active?",
      "type": "boolean",
      "default": false,
      "readonly": true
    },
    "regex": {
      "type": "object",
      "patternProperties": {
        "^([\\S]*)$": {
          "type": "string"
        }
      },
      "readonly": true
    },
    "lastTimeIngestedAt": {
      "title": "Last Time Ingest from the Provider",
      "type": "number",
      "readonly": true
    },
    "createdAt": {
      "type": "number",
      "readonly": true
    },
    "updatedAt": {
      "type": "number",
      "readonly": true
    }
  },
  "required": [
    "name",
    "providerName",
    "protocol",
    "host",
    "path",
    "isActive",
    "status",
    "createdAt",
    "updatedAt"
  ]
}

List reconciliation reports

List reconciliation reports in the Cumulus system.

GET
/reconciliationReports

Example request

$ curl https://example.com/reconciliationReports --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "reconciliationReport",
        "limit": 2,
        "page": 1,
        "count": 7
    },
    "results": [
        {
            "createdAt": 1589824212102,
            "name": "inventoryReport-20200518T175012101",
            "location": "s3://lpdaac-internal/lpdaac-cumulus/reconciliation-reports/inventoryReport-20200518T175012101.json",
            "type": "Inventory",
            "status": "Generated",
            "updatedAt": 1589824218825,
            "timestamp": 1589824219329
        },
        {
            "createdAt": 1589596344305,
            "name": "ModisInternalReport",
            "location": "s3://lpdaac-internal/lpdaac-cumulus/reconciliation-reports/ModisInternalReport.json",
            "type": "Internal",
            "status": "Generated",
            "updatedAt": 1589596352010,
            "timestamp": 1589596352356
        }
    ]
}

Retrieve reconciliation report

Retrieve a single reconciliation report.

GET
/reconciliationReports/{name}

Example request

$ curl https://example.com/reconciliationReports/inventoryReport-20190305T153430508 --header 'Authorization: Bearer ReplaceWithTheToken'

Example responses

{
  "presignedS3Url": "https://example.amazonaws.com/example",
  "data": "report data, see examples below for each report type"
}
{
  "presignedS3Url": "https://example.amazonaws.com/example",
  "data": "Error: Report examplereport exceeded maximum allowed payload size"
}
Inventory Report
{
    "reportStartTime": "2019-03-05T15:34:30.508Z",
    "reportEndTime": "2019-03-05T15:34:37.243Z",
    "status": "SUCCESS",
    "error": null,
    "reportType": "Inventory",
    "filesInCumulus": {
        "okCount": 40,
        "okCountByGranule": {
          "MOD09GQ.A2016358.h13v04.006.2016360104606": 10,
          "MYD13Q1.A2017297.h19v10.006.2017313221303": 10,
          "MOD09GQ.A3518809.ln_rVr.006.7962927138074": 10,
          "MOD09GQ.A8768252.HC4ddD.006.2077696236118": 5,
          "MOD09GQ.A8722843.GTk5A3.006.4026909316904": 5
        },
        "onlyInS3": [
            "s3://cumulus-test-sandbox-protected/MOD09GQ.A2016358.h13v04.006.2016360104606.cmr.xml",
            "s3://cumulus-test-sandbox-private/BROWSE.MYD13Q1.A2017297.h19v10.006.2017313221201.hdf"
        ],
        "onlyInDynamoDb": [
            {
                "uri": "s3://cumulus-test-sandbox-protected/MOD09GQ.A2016358.h13v04.006.2016360104606.hdf",
                "granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"
            },
            {
                "uri": "s3://cumulus-test-sandbox-private/MYD13Q1.A2017297.h19v10.006.2017313221303.hdf.met",
                "granuleId": "MYD13Q1.A2017297.h19v10.006.2017313221303"
            }
        ]
    },
    "collectionsInCumulusCmr": {
        "okCount": 1,
        "onlyInCumulus": [
            "L2_HR_PIXC___000"
        ],
        "onlyInCmr": [
            "MCD43A1___006",
            "MOD14A1___006"
        ]
    },
    "granulesInCumulusCmr": {
        "okCount": 3,
        "onlyInCumulus": [
            {
                "granuleId": "MOD09GQ.A3518809.ln_rVr.006.7962927138074",
                "collectionId": "MOD09GQ___006"
            },
            {
                "granuleId": "MOD09GQ.A8768252.HC4ddD.006.2077696236118",
                "collectionId": "MOD09GQ___006"
            }
        ],
        "onlyInCmr": [
            {
                "GranuleUR": "MOD09GQ.A0002421.oD4zvB.006.4281362831355",
                "ShortName": "MOD09GQ",
                "Version": "006"
            },
            {
                "GranuleUR": "MOD09GQ.A0007443.H16AHq.006.7078190437865",
                "ShortName": "MOD09GQ",
                "Version": "006"
            }
        ]
    },
    "filesInCumulusCmr": {
        "okCount": 11,
        "onlyInCumulus": [
            {
                "fileName": "MOD09GQ.A8722843.GTk5A3.006.4026909316904.jpeg",
                "uri": "s3://cumulus-test-sandbox-public/MOD09GQ___006/MOD/MOD09GQ.A8722843.GTk5A3.006.4026909316904.jpeg",
                "granuleId": "MOD09GQ.A8722843.GTk5A3.006.4026909316904"
            }
        ],
        "onlyInCmr": [
            {
                "URL": "https://example.com/MYD13Q1.A2017297.h19v10.006.2017313221202.hdf",
                "Type": "GET DATA",
                "GranuleUR": "MOD09GQ.A4675287.SWPE5_.006.7310007729190"
            },
            {
                "URL": "https://cumulus-test-sandbox-public.s3.amazonaws.com/MOD09GQ___006/MOD/MOD09GQ.A8722843.GTk5A3.006.4026909316904_ndvi.jpg",
                "Type": "GET DATA",
                "GranuleUR": "MOD09GQ.A8722843.GTk5A3.006.4026909316904"
            }
        ]
    }
}
Granule Not Found Report
{
  "createStartTime": "2020-08-28T18:33:50.525Z",
  "createEndTime": "2020-08-28T18:34:16.646Z",
  "status": "SUCCESS",
  "reportType": "Granule Not Found",
  "filesInCumulus": {
    "okCount": 20,
    "okCountByGranule": {
      "MOD09GQ.A2770300.W1_V5Z.006.7853319756315": 2,
      "MOD09GQ.A7682091.QtZjhI.006.1218763030745": 3,
      "MOD09GQ.A9536857.bV1q4A.006.0980635705136": 3,
      "MOD09GQ.A7790080.xLPpTZ.006.2292724247258": 3,
      "MOD09GQ.A3769578.tlqq7j.006.4434181201872": 3,
      "MOD09GQ.A5423294.RFOrMI.006.8677601711674": 3,
      "MOD09GQ.A8837603.fb2Vhw.006.3950746589733": 3
    },
    "onlyInS3": [
      "s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593122280799/MOD09GQ.A6921412.znLBqe.006.9673856807617.hdf.met",
      "s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593122597517/MOD09GQ.A1201557.d6tP2Y.006.7482431709753.hdf.met",
      "s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593198955197/MOD09GQ.A7375038.gm1irM.006.1317280710645.hdf.met",
      "s3://cumulus-sandbox-protected/MOD09GQ___006/2017/MOD/test-data-1593122280799/MOD09GQ.A6921412.znLBqe.006.9673856807617.hdf",
      "s3://cumulus-sandbox-protected/MOD09GQ___006/2017/MOD/test-data-1593122597517/MOD09GQ.A1201557.d6tP2Y.006.7482431709753.hdf",
      "s3://cumulus-sandbox-protected/MOD09GQ___006/2017/MOD/test-data-1593122881446/MOD09GQ.A2708681.CFkGhW.006.7154000014360.hdf"
    ],
    "onlyInDynamoDb": [
      {
        "uri": "s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593178175944/MOD09GQ.A2770300.W1_V5Z.006.7853319756315.hdf.met",
        "granuleId": "MOD09GQ.A2770300.W1_V5Z.006.7853319756315"
      }
    ]
  },
  "collectionsInCumulusCmr": {
    "okCount": 0,
    "onlyInCumulus": [
      "MOD09GQ_test-test-data-1593122280799___006",
      "MOD09GQ_test-test-data-1593122597517___006",
      "MOD09GQ_test-test-data-1593122881446___006",
      "MOD09GQ_test-test-data-1593178175944___006",
      "MOD09GQ_test-test-data-1593198739991___006",
      "MOD09GQ_test-test-data-1593198955197___006",
      "MOD09GQ_test-test-data-1593199179316___006",
      "MOD09GQ_test-test-data-1593199379860___006",
      "MOD09GQ_test-test-data-1593199548454___006",
      "MOD09GQ_test-test-data-1593199742089___006",
      "MOD09GQ_test-test-data-1593199934835___006"
    ],
    "onlyInCmr": [
      "A2_RainOcn_NRT___0",
      "A2_SI12_NRT___0",
      "A2_SI25_NRT___0",
      "A2_SI6_NRT___0",
      "AST_L1A___003",
      "CMR44JK___001",
      "L2_HR_PIXC___000",
      "L2_HR_PIXC___1",
      "MCD43A1___006",
      "MOD09GQ___006",
      "MOD11A1___006",
      "MOD14A1___006",
      "MUR-JPL-L4-GLOB-v4.1___1",
      "MYD09A1___006",
      "MYD13A1___006",
      "MYD13Q1___006",
      "hs3avaps___1",
      "hs3wwlln___1"
    ]
  },
  "granulesInCumulusCmr": {
    "okCount": 0,
    "onlyInCumulus": [],
    "onlyInCmr": []
  },
  "filesInCumulusCmr": {
    "okCount": 0,
    "onlyInCumulus": [],
    "onlyInCmr": []
  }
}
Internal Report
{
  "reportType": "Internal",
  "createStartTime": "2020-08-28T19:01:50.636Z",
  "createEndTime": "2020-08-28T19:02:08.211Z",
  "reportStartTime": "2020-01-28T18:21:31.119Z",
  "reportEndTime": "2020-08-28T18:21:31.119Z",
  "status": "SUCCESS",
  "collections": {
    "okCount": 10,
    "withConflicts": [
      {
        "es": {
          "createdAt": 1591902043153,
          "granuleId": "^.*$",
          "sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5",
          "dataType": "L2_HR_PIXC",
          "name": "L2_HR_PIXC",
          "files": [
            {
              "bucket": "private",
              "regex": ".*.h5$",
              "sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5"
            }
          ],
          "granuleIdExtraction": "^(.*)(\\.h5|\\.cmr)",
          "reportToEms": true,
          "version": "000",
          "duplicateHandling": "error",
          "updatedAt": 1591902043153,
          "timestamp": 1591902043840
        },
        "db": {
          "duplicateHandling": "error",
          "granuleIdExtraction": "^(.*)(\\.h5|\\.cmr)",
          "version": "000",
          "dataType": "L2_HR_PIXC",
          "files": [
            {
              "bucket": "private",
              "sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5",
              "regex": ".*.h5$"
            }
          ],
          "updatedAt": 1591902043153,
          "createdAt": 1591902043153,
          "reportToEms": false,
          "granuleId": "^.*$",
          "sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5",
          "name": "L2_HR_PIXC"
        }
      }
    ],
    "onlyInEs": [
      "MCD43A1___006",
      "MOD09GQ___006"
    ],
    "onlyInDb": [
      "MYD13QA___005"
    ]
  },
  "granules": {
    "okCount": 77,
    "withConflicts": [
      {
        "es": {
          "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:IngestAndPublishGranule:edb21d63-b62d-42e9-8879-f39d53314f16",
          "processingEndDateTime": "2020-05-18T19:49:04.769Z",
          "published": false,
          "error": {},
          "productVolume": 253949,
          "timeToPreprocess": 0,
          "duration": 1.005,
          "createdAt": 1589831343764,
          "granuleId": "MOD14A1.A4313371.RQSeyH.006.7795385650931",
          "processingStartDateTime": "2020-05-18T19:49:03.831Z",
          "provider": "s3_provider",
          "timeToArchive": 0,
          "files": [
            {
              "bucket": "cumulus-test-sandbox-internal",
              "fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
              "size": 233840,
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
              "type": "data",
              "key": "test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf"
            },
            {
              "bucket": "cumulus-test-sandbox-internal",
              "fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
              "size": 14297,
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
              "type": "metadata",
              "key": "test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met"
            },
            {
              "bucket": "cumulus-test-sandbox-internal",
              "fileName": "BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
              "size": 5812,
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
              "type": "browse",
              "key": "test-data/files/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg"
            }
          ],
          "collectionId": "MOD14A1___006",
          "status": "running",
          "timestamp": 1589831367756,
          "updatedAt": 1589831344769
        },
        "db": {
          "published": true,
          "endingDateTime": "2017-11-08T23:59:59Z",
          "status": "completed",
          "timestamp": 1589831361039,
          "createdAt": 1589831343764,
          "processingEndDateTime": "2020-05-18T19:49:20.023Z",
          "productVolume": 258297,
          "timeToPreprocess": 0.263,
          "timeToArchive": 0.942,
          "productionDateTime": "2018-07-19T12:01:01Z",
          "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:IngestAndPublishGranule:edb21d63-b62d-42e9-8879-f39d53314f16",
          "files": [
            {
              "bucket": "cumulus-test-sandbox-protected",
              "key": "MOD14A1___006/2017/MOD/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
              "size": 233840,
              "fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
              "type": "data"
            },
            {
              "bucket": "cumulus-test-sandbox-private",
              "key": "MOD14A1___006/MOD/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
              "size": 14297,
              "fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
              "type": "metadata"
            },
            {
              "bucket": "cumulus-test-sandbox-public",
              "key": "MOD14A1___006/BRO/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
              "size": 5812,
              "fileName": "BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
              "type": "browse"
            },
            {
              "bucket": "cumulus-test-sandbox-protected-2",
              "key": "MOD14A1___006/MOD/MOD14A1.A4313371.RQSeyH.006.7795385650931.cmr.xml",
              "size": 4348,
              "fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.cmr.xml",
              "type": "metadata"
            }
          ],
          "cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1234771093-CUMULUS",
          "processingStartDateTime": "2020-05-18T19:49:03.831Z",
          "updatedAt": 1589831361039,
          "beginningDateTime": "2017-10-24T00:00:00Z",
          "provider": "s3_provider",
          "granuleId": "MOD14A1.A4313371.RQSeyH.006.7795385650931",
          "collectionId": "MOD14A1___006",
          "duration": 17.275,
          "error": {
            "Error": "Unknown Error",
            "Cause": "None"
          },
          "lastUpdateDateTime": "2018-04-25T21:45:45.524053"
        }
      }
    ],
    "onlyInEs": [
      {
        "granuleId": "MYD13Q1.A0323210.Yujnv6.006.1973526114346",
        "collectionId": "MYD13Q1___006",
        "provider": "s3_provider",
        "createdAt": 1589913461833,
        "updatedAt": 1589913522990
      }
    ],
    "onlyInDb": [
      {
        "granuleId": "MYD13Q1.A5822875.qHI8hK.006.8698738069249",
        "collectionId": "MYD13Q1___006",
        "provider": "s3_provider",
        "createdAt": 1598552797312,
        "updatedAt": 1598552827611
      }
    ]
  }
}
ORCA Backup Report
{
  "collectionIds": [
    "MYD13Q1___006",
    "MOD09GQ___006"
  ],
  "createEndTime": "2022-02-11T19:13:41.986Z",
  "createStartTime": "2022-02-11T19:13:41.153Z",
  "granuleIds": [
    "MYD13Q1.A3194547.tnYsne.006.8400707913298",
    "MYD13Q1.A2655880.GOWVT9.006.3531712476486",
    "MOD09GQ.A8858216.Y6HJnu.006.6168319936421"
  ],
  "providers": [
    "modas_provider",
    "s3_provider"
  ],
  "reportEndTime": "2022-02-11T19:12:25.000Z",
  "reportType": "ORCA Backup",
  "status": "SUCCESS",
  "granules": {
    "okCount": 0,
    "cumulusCount": 2,
    "orcaCount": 2,
    "okFilesCount": 5,
    "conflictFilesCount": 6,
    "withConflicts": [
      {
        "okFilesCount": 4,
        "granuleId": "MYD13Q1.A3194547.tnYsne.006.8400707913298",
        "collectionId": "MYD13Q1___006",
        "provider": "s3_provider",
        "createdAt": 1644606673827,
        "updatedAt": 1644606718024,
        "conflictFiles": [
          {
            "fileName": "MYD13Q1.A3194547.tnYsne.006.8400707913298.cmr.xml",
            "bucket": "cumulus-test-sandbox-protected-2",
            "key": "MYD13Q1___006/MYD/MYD13Q1.A3194547.tnYsne.006.8400707913298.cmr.xml",
            "reason": "shouldBeExcludedFromOrca"
          },
          {
            "fileName": "BROWSE.MYD13Q1.A3194547.tnYsne.006.8400707913298.1.jpg2",
            "bucket": "cumulus-test-sandbox-public",
            "key": "MYD13Q1___006/BRO/BROWSE.MYD13Q1.A3194547.tnYsne.006.8400707913298.1.jpg2",
            "reason": "onlyInCumulus"
          },
          {
            "fileName": "BROWSE.MYD13Q1.A3194547.tnYsne.006.8400707913298.1.jpg",
            "bucket": "cumulus-test-sandbox-public",
            "key": "MYD13Q1___006/BRO/BROWSE.MYD13Q1.A3194547.tnYsne.006.8400707913298.1.jpg",
            "orcaBucket": "cumulus-test-sandbox-orca-glacier",
            "reason": "onlyInOrca"
          }
        ]
      }
    ],
    "onlyInCumulus": [
      {
        "okFilesCount": 1,
        "granuleId": "MYD13Q1.A2655880.GOWVT9.006.3531712476486",
        "collectionId": "MYD13Q1___006",
        "provider": "s3_provider",
        "createdAt": 1644606673656,
        "updatedAt": 1644606683360,
        "conflictFiles": [
          {
            "fileName": "MYD13Q1.A2655880.GOWVT9.006.3531712476486.hdf",
            "bucket": "cumulus-test-sandbox-protected",
            "key": "MYD13Q1___006/2017/MYD/MYD13Q1.A2655880.GOWVT9.006.3531712476486.hdf",
            "reason": "onlyInCumulus"
          },
          {
            "fileName": "BROWSE.MYD13Q1.A2655880.GOWVT9.006.3531712476486.hdf",
            "bucket": "cumulus-test-sandbox-private",
            "key": "MYD13Q1___006/BRO/BROWSE.MYD13Q1.A2655880.GOWVT9.006.3531712476486.hdf",
            "reason": "onlyInCumulus"
          },
          {
            "fileName": "BROWSE.MYD13Q1.A2655880.GOWVT9.006.3531712476486.1.jpg",
            "bucket": "cumulus-test-sandbox-public",
            "key": "MYD13Q1___006/BRO/BROWSE.MYD13Q1.A2655880.GOWVT9.006.3531712476486.1.jpg",
            "reason": "onlyInCumulus"
          },
          {
            "fileName": "MYD13Q1.A2655880.GOWVT9.006.3531712476486.cmr.xml",
            "bucket": "cumulus-test-sandbox-protected-2",
            "key": "MYD13Q1___006/MYD/MYD13Q1.A2655880.GOWVT9.006.3531712476486.cmr.xml",
            "reason": "onlyInCumulus"
          }
        ]
      }
    ],
    "onlyInOrca": [
      {
        "granuleId": "MOD09GQ.A8858216.Y6HJnu.006.6168319936421",
        "provider": "s3_provider",
        "collectionId": "MOD09GQ___006",
        "createdAt": 1643920768281,
        "conflictFiles": [
          {
            "bucket": "cumulus-test-sandbox-protected",
            "key": "MOD09GQ___006/2017/MOD/MOD09GQ.A8858216.Y6HJnu.006.6168319936421.hdf",
            "fileName": "MOD09GQ.A8858216.Y6HJnu.006.6168319936421.hdf",
            "orcaBucket": "cumulus-test-sandbox-orca-glacier",
            "reason": "onlyInOrca"
          },
          {
            "bucket": "cumulus-test-sandbox-public",
            "key": "MOD09GQ___006/MOD/MOD09GQ.A8858216.Y6HJnu.006.6168319936421_ndvi.jpg",
            "fileName": "MOD09GQ.A8858216.Y6HJnu.006.6168319936421_ndvi.jpg",
            "orcaBucket": "cumulus-test-sandbox-orca-glacier",
            "reason": "onlyInOrca"
          },
          {
            "bucket": "cumulus-test-sandbox-protected-2",
            "key": "MOD09GQ___006/MOD/MOD09GQ.A8858216.Y6HJnu.006.6168319936421.cmr.xml",
            "fileName": "MOD09GQ.A8858216.Y6HJnu.006.6168319936421.cmr.xml",
            "orcaBucket": "cumulus-test-sandbox-orca-glacier",
            "reason": "onlyInOrca"
          }
        ]
      }
    ]
  }
}
Granule Inventory Report
"granuleUr","collectionId","createdAt","startDateTime","endDateTime","status","updatedAt","published"
"MOD14A1.A9506271.IvEJsu.006.8359924290786","MOD14A1___006","2020-05-18T20:15:54.525Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-05-18T20:16:02.473Z",false
"MYD13Q1.A9663671.0zkwKH.006.9812354158395","MYD13Q1___006","2020-07-06T19:46:19.957Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-07-06T19:46:57.054Z",true

Create reconciliation report

Create a new reconciliation report.

POST
/reconciliationReports

Request body

parameter value required description
reportName string false Report name.
reportType "Granule Inventory"|"Granule Not Found"|"Internal"|"Inventory"|"ORCA Backup" false Report type (default Inventory)
startTimestamp string false Any input valid for a JS Date contstructor. Data older than this will be ignored in the generated report.
endTimestamp string false Any input valid for a JS Date contstructor. Data newer than this will be ignored in the generated report.
collectionId [string | array] false collectionId (or array of collectionIds) for comparison of collection and granule holdings.
granuleId [string | array] false granuleId (or array of granuleIds) for use for comparison of collection and granule holdings.
provider [string | array] false provider name (or array of providers) for comparison of granule holdings
status string false status filter for Granule Inventory reports

NOTE: Adding a startTimestamp or endTimestamp value to the POST request will result in one way comparisons for some fields for Inventory and Granule Not Found reports.

NOTE: Adding a granuleId input will result in an one way report for collections for Inventory and Granule Not Found reports.

NOTE: Inventory and Granule Not Found reports only alllow one of the parameters (collectionId, granuleId, provider) in same request.

NOTE: Granule Inventory reports can be filtered with the following parameters: (collectionId, granuleId, status). If granuleId is a string or single value array, it will search for granules with granuleIds containing that substring.

Example request

$ curl --request POST https://example.com/reconciliationReports --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
  "reportName": "ModisInventoryReport",
  "reportType": "Inventory",
  "startTimestamp": 1269993600000,
  "endTimestamp": 1350000000000,
  "collectionId": "MOD09GQ___006",
  "provider": "MODIS",
  "granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"
}'

Example response

{
  "createdAt": 1623256106616,
  "updatedAt": 1623256106616,
  "id": "bb7059f4-0cd8-4205-8857-fb0e6b68b3e4",
  "status": "RUNNING",
  "taskArn": "arn:aws:ecs:us-east-1:123456789012:task/stackname-CumulusECSCluster/656b2c8203c4488e9feffff38beb4be5",
  "description": "Create Reconciliation Report",
  "operationType": "Reconciliation Report"
}

Delete reconciliation report

Delete a reconciliation report from Cumulus.

DELETE
/reconciliationReports/{name}

Example request

$ curl --request DELETE https://example.com/reconciliationReports/inventoryReport-20180620T205838883 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Report deleted"
}

Create EMS reports

Create EMS reports and send them to EMS. For more information on EMS reporting and types of reports see the Cumulus EMS Reporting documentation.

Overview of the request fields:

Field Required Value Description
reportType Y "metadata"|"ingest"|"distribution" type of report
startTime N string report startTime in format YYYY-MM-DD or YYYY-MM-DDTHH:mm:ss
If startTime and endTime are not specified, the reports for previous day will be generated
endTime N string report endTime in format YYYY-MM-DD or YYYY-MM-DDTHH:mm:ss
collectionId N string collectionId of the report records
This field can be specified when report type is metadata or ingest
POST
/ems

Example requests

$ curl https://example.com/ems --header 'Authorization: Bearer ReplaceWithTheToken' --data '{
    "reportType": "metadata"
}'
$ curl https://example.com/ems --header 'Authorization: Bearer ReplaceWithTheToken' --data '{
    "reportType": "ingest",
    "startTime": "2019-06-27",
    "endTime": "2019-06-29",
    "collectionId": "MOD14A1___006"
}'

Example response

{
    "message": "Reports are being generated",
    "status": 202
}

Instance Metadata

The Cumulus API can provide information about its configuration.

GET requests to the instance metadata endpoint return a json object with information about how the Cumulus stack is configured. It returns the CMR provider and environment as well as the stackName (prefix).

GET
/instanceMeta

Example Request

$ curl https://example.com/instanceMeta --header 'Authorization: Bearer ReplaceWithTheToken'

Example Response

{
  "cmr": {
    "provider": "CUMULUS",
    "environment": "UAT"
  },
  "cumulus": {
    "stackName": "cumulus-stack-prefix"
  }
}

Elasticsearch

These endpoints provide an interface into the Elasticsearch SDK to perform functions like Elasticsearch reindex.

Cumulus uses index aliases to allow reindexing with no downtime.

Reindex

The reindex command creates a new index and reindexes the source index to the new, destination index. This uses the Elasticsearch reindex functionality, but includes setting up the new index.

Note that there will now be two copies of your index. Your Cumulus instance will still point to the old copy until you perform the change-index function described below.

An alias should not be specified unless you have a specific alias configured. If a source index is not specified, it will default to the index from the alias. If you want to name the destination index something particular, you can specify a name, otherwise the destination index name will default to 'cumulus-year-month-day' with today's date (e.g. cumulus-4-12-2019). If you specify a destination index name, it must be an index that does not already exist in your cluster. The destination index must be different than the source index or the operation will fail.

Since reindex can be a long running operation, the reindex-status endpoint must be queried to see if the operation has completed.

Overview of request fields:

parameter value required description
aliasName string false Alias to reindex from. Will be used if sourceIndex is not provided. Will default to the default alias if not provided.
sourceIndex string false Index to reindex from
destIndex string true Index to reindex to

Example request

$ curl --request POST https://example.com/elasticsearch/reindex --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{ "message": "Reindexing from cumulus-4-4-2019 to cumulus-4-12-2019. Check the reindex-status endpoint for status." }

Example request

$ curl --request POST https://example.com/elasticsearch/reindex --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
  "aliasName": "cumulus-alias",
  "sourceIndex": "cumulus-4-4-2019",
  "destIndex": "cumulus-new-index"
}'

Example response

{ "message": "Reindexing from cumulus-4-4-2019 to cumulus-new-index. Check the reindex-status endpoint for status." }

Reindex Status

Reindexing can be a long running operation, so use this endpoint to get the status of your reindex and the status of your indices.

reindexStatus.nodes will be populated if there is a reindex operation currently running, nodes will be empty if not.

The elasticsearchStatus shows the number of documents in each index which can be used as a sanity check for the reindex operation before switching over to the new index. Note that the active index will continue to index logs, so the number of documents in the active index will be slightly higher and continue to grow.

Example request

$ curl https://example.com/elasticsearch/reindex-status --header 'Authorization: Bearer ReplaceWithTheToken'

Example responses

{
    "reindexStatus": {
        "nodes": {
            "TKDzeZ_TQVWkwKZYtCtv_g": {
                "name": "TKDzeZ_",
                "roles": [
                    "master",
                    "data",
                    "ingest"
                ],
                "tasks": {
                    "TKDzeZ_TQVWkwKZYtCtv_g:546691": {
                        "node": "TKDzeZ_TQVWkwKZYtCtv_g",
                        "id": 546691,
                        "type": "transport",
                        "action": "indices:data/write/reindex",
                        "start_time_in_millis": 1555073365557,
                        "running_time_in_nanos": 75424890,
                        "cancellable": true
                    }
                }
            }
        }
    },
    "indexStatus": {
        "_shards": {
            "total": 32,
            "successful": 16,
            "failed": 0
        },
        "_all": {
            "primaries": {
                "docs": {
                    "count": 52213,
                    "deleted": 0
                }
            },
            "total": {
                "docs": {
                    "count": 52213,
                    "deleted": 0
                }
            }
        },
        "indices": {
            ".kibana": {
                "primaries": {
                    "docs": {
                        "count": 1,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 1,
                        "deleted": 0
                    }
                }
            },
            "cumulus-2019-4-12": {
                "primaries": {
                    "docs": {
                        "count": 25569,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 25569,
                        "deleted": 0
                    }
                }
            },
            "cumulus-2019-4-9": {
                "primaries": {
                    "docs": {
                        "count": 25644,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 25644,
                        "deleted": 0
                    }
                }
            }
        }
    }
}
{
    "reindexStatus": {
        "nodes": {}
    },
    "indexStatus": {
        "_shards": {
            "total": 22,
            "successful": 11,
            "failed": 0
        },
        "_all": {
            "primaries": {
                "docs": {
                    "count": 51210,
                    "deleted": 0
                }
            },
            "total": {
                "docs": {
                    "count": 51210,
                    "deleted": 0
                }
            }
        },
        "indices": {
            ".kibana": {
                "primaries": {
                    "docs": {
                        "count": 1,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 1,
                        "deleted": 0
                    }
                }
            },
            "cumulus-2019-4-12": {
                "primaries": {
                    "docs": {
                        "count": 25569,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 25569,
                        "deleted": 0
                    }
                }
            },
            "cumulus-2019-4-9": {
                "primaries": {
                    "docs": {
                        "count": 25640,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 25640,
                        "deleted": 0
                    }
                }
            }
        }
    }
}

Change Index

Change index switches the Elasticsearch index to point to the new index, rather than the current index. You may choose to delete your current index during this operation using the deleteSource parameter, which defaults to false. If you are using the default index created by Cumulus, this will switch your Cumulus instance to the new index and you will see those changes immediately via the API and dashboard. If newIndex does not exist, an index with that name will be created.

currentIndex and newIndex are required parameters.

In the context of reindex, you'd call change-index, following reindex completion to start using the new index with no downtime.

Overview of the request fields:

parameter value required description
aliasName string false Alias to use for newIndex. Will be the default index if not provided
currentIndex string true Index to change the alias from
newIndex string true Index to change alias to
deleteSource boolean false If set to true it will delete the index provided in currentIndex

Example request

$ curl --request POST https://example.com/elasticsearch/change-index --header 'Content-Type: application/json' --header 'Authorization: Bearer ReplaceWithTheToken' --data '{
  "aliasName": "cumulus-alias",
  "currentIndex": "cumulus-12-4-2019",
  "newIndex": "cumulus-4-12-2019",
  "deleteSource": false
}'

Example response

{ "message": "Reindex success - alias cumulus now pointing to cumulus-4-12-2019" }

Index from Database

In case of corruption of your Elasticsearch index, you can reindex your data from the database. This will include collections, executions, granules, pdrs, providers, and rules. Logs will not be indexed since they are not stored in the database.

You can specify an index (should be empty) or if no index name is specified, a default with the format of 'cumulus-year-month-day' with today's date (e.g. cumulus-4-12-2019) will be used. If the specified index does not exist, it will be created.

You may optionally specify the following request fields:

parameter value required description
postgresResultPageSize number false Number of records to pull per database query for index to Elasticsearch Defaults 1000
postgresConnectionPoolSize number false Max number of connections to postgres database to allocate. Defaults to 10
esRequestConcurrency number false Maximum concurrency of writes to Elasticsearch Defaults to 10
indexName string false Name of index to re-index to. Defaults to cumulus-dd-mm-yyyy

It is recommended that workflow rules be turned off, as any data ingested into the database during this operation cannot be guaranteed to make it into the new index. Following the completion of the index, you will need to use the change index operation to switch your Elasticsearch to point to the new instance.

Indexing is an async operation, so an operation id will be returned. You can query the status using the asyncoperations GET endpoint with the operation id.

Example request

$ curl --request POST https://example.com/elasticsearch/index-from-database --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
  "indexName": "recovery-index"
}'

Example response

{"message":"Indexing database to recovery-index. Operation id: 8f4d35ba-c858-40ae-93f0-4465f54a911d"}

Example request - operation status

$ curl --header 'Authorization: Bearer ReplaceWithTheToken' https://example.com/asyncoperations/8f4d35ba-c858-40ae-93f0-4465f54a911d

Example response - operation status

{
   "id":"8f4d35ba-c858-40ae-93f0-4465f54a911d",
   "status":"SUCCEEDED",
   "taskArn":"arn:aws:ecs:us-east-1:XXXXXXXXXXXX:task/dc41ff2b-e610-4f7a-8c63-48b48ceb7160",
   "output":"\"Index from database complete\""
}

Indices Status

Use the indices-status endpoint to view information about your elasticsearch indices. This endpoint will return the information for the indices call in the Elasticsearch API.

Example request

$ curl  https://example.com/elasticsearch/indices-status --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

yellow open .kibana           yxBUTswHSr2ecec8y1szEg 1 1      1   0   3.1kb   3.1kb
yellow open cumulus-2019-6-21 ELpdxYVKSc-VvNxlNjmMUA 5 1    700 225   2.5mb   2.5mb
yellow open cumulus-2019-6-24 lmylLkTsQJWIVYFJcftMqQ 5 1    700   0     4mb     4mb
yellow open cumulus           jQLXsb8yS4aEmLAAyHcmCw 5 1 217298 588 124.3mb 124.3mb

Current Index

Get the current aliased index being used by the Cumulus Elasticsearch instance.

Example request

$ curl  https://example.com/elasticsearch/current-index --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

["cumulus-2019-6-24"]

Serve the dashboard from a bucket

Serve the dashboard from an S3 bucket.

This is a way to serve the Cumulus dashboard in a browser from an S3 bucket without making the bucket or files public.

To use this:

  • Your dashboard bucket must be in the bucket definitions in your Cumulus terraform.tfvars, otherwise you will not be able to access the bucket.
  • Deploy the dashboard to your bucket using the instructions in the Cumulus dashboard README.
  • In a browser, use the example request below, but replace example.com with your Cumulus backend API and dashboard-bucket with your bucket name.
https://example.com/dashboard/dashboard-bucket/index.html
GET
/dashboard/{bucket}/{key}

Example request

ORCA

This endpoint authenticates and forwards requests to the ORCA private API, and returns the response from the ORCA API. Please refer to ORCA API reference on how to use ORCA API.

POST
/orca

Example request

$ curl --request POST https://example.com/orca/recovery/granules --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json'  --data '{
  "granule_id": "MOD14A1.061.H5V12.2020312.141531789"
}'

Migration Counts

This endpoint triggers a run of the postgres-migration-count-tool as an async operation of type Migration Count Report.

For more information on the tool, see the README

The query uses the following optional parameters:

parameter description
cutoffSeconds Sets the number of seconds prior to this execution to 'cutoff' reconciliation queries. This allows in-progress or other in-flight operations time to complete and propagate to ElasticSearch/Dynamo/Postgres. Default is 3600
dbConcurrency Sets max number of parallel collections reports the script will run at a time. Default 20
dbMaxPool Sets the maximum number of connections the database pool has available. Modifying this may result in unexpected failures. Default is 20
reportPath Sets the path location for the tool to write a copy of the lambda payload to S3
reportBucket Sets the bucket used for reporting. If this argument is used, a reportPath must be set to generate a report
POST
/migrationCounts

Example request

$curl -X POST https://$API_URL/dev/migrationCounts -d 'reportBucket=someBucket&reportPath=someReportPath&cutoffSeconds=60&dbConcurrency=20&dbMaxPool=20' --header 'Authorization: Bearer $TOKEN'

Example response

{"id":"7ccaed31-756b-40bb-855d-e5e6d00dc4b3","status":"RUNNING","taskArn":"arn:aws:ecs:us-east-1:AWSID:task/$PREFIX-CumulusECSCluster/123456789","description":"Migration Count Tool ECS Run","operationType":"Migration Count Report"}

Recover cumulus messages

Endpoint provides a mechanism for recovery of S3 sfEventSqsToDbRecords dead letter objects (created as described in the Core Documentation). The endpoint will invoke an async operation that will attempt to process all of the objects in the specified location.

The endpoint by default will process all records contained in the S3 objects from the default storage location on the S3 system bucket under the prefix of <stackName>/dead-letter-archive/sqs/. However, it is likely useful to process a subset of those objects, which you can do by moving the desired subset of objects to a new path on S3 and then specifying that new path using the path parameter to the endpoint request.

The query uses the following optional parameters:

parameter description
bucket The bucket to read records from. Defaults to the Core system bucket
path The S3 prefix (path) to read DLQ records from. Defaults to <stackName>/dead-letter-archive/sqs/
POST
/deadLetterArchive/recoverCumulusMessages

Example request

curl -X POST https://cumulus.podaac.sit.earthdata.nasa.gov/deadLetterArchive/recoverCumulusMessages --header "Authorization: Bearer $TOKEN" --header 'Content-Type: application/json' --data '{
    "bucket": "some-cumulus-bucket",
    "path": "some/path/to/records/"
}'

Example response

{
    "createdAt":1646861517957,
    "updatedAt":1646861517957,
    "id":"a0cd2cf0-a677-e82a-27d1-0a09271aa37d",
    "status":"RUNNING",
    "taskArn":"arn:aws:ecs:us-west-2:xxxxxxxxxx:task/stack-CumulusECSCluster/{SHA}",
    "description":"Dead-Letter Processor ECS Run",
    "operationType":"Dead-Letter Processing" 
}
Show examples in:
Cumulus API