Cumulus API

The Cumulus API allows developers to interact with the Cumulus Framework, such as monitoring status or creating, editing, and deleting records. This is the same API that powers the Cumulus dashboard.

By utilizing this API, a developer can integrate with the Cumulus framework in any language or environment; although interacting with Cumulus through the Cumulus dashboard may be appropriate for many end users, for some use cases it's best to have the flexibility of a web-accessible API.

The API accepts and responds with JSON payloads at various HTTPS endpoints.

In order to use these endpoints, you must include authentication information in your HTTPS request; authentication is explained in the following section.

The following table lists the query string parameters that can be used with most of the Cumulus API endpoints. {fieldName} is a stand-in for any of the fields in the record, and for nested objects dot notation can be used; for example, valid fieldNames include: pdrName, status, and recipe.processStep.description.

query string parameter description
limit={number} number of records to be returned by the API call; default is 1, maximum is 100
page={number} page number, 1-indexed; default is 1
sort_by={fieldName} which field to sort by; default is timestamp
order={asc|desc} whether to sort in asc or desc order
sort_key[]={-fieldName1}&sort_key[]={fieldName2} One or more sort keys can be specified using the sort_key[] parameter. The order used impacts searching. Fields can be prepended with a - to sort in descending order or a + to sort in ascending. Ascending order is the default. The + must be escaped with %2B
prefix={value} startsWith search of the granuleId, status, pdrName, name, error, id and msg fields
infix={value} includes search of the granuleId, status, pdrName, name, error, id and msg fields
fields={fieldName1, fieldName2} which fields to return, separated by a comma
{fieldName}={value} exact value match for the given field
{fieldName}__from={number} for numeric fields, field value must be greater than the given number
{fieldName}__to={number} for numeric fields, field value must be less than the given number
{fieldName}__not={value} field does not match the given value
{fieldName}__in={value1, value2} field matches one of the values in the comma-separated list
{fieldName}__exists={true|false} field exists or doesn't exist in the record
q="fieldName:[1 TO 2] AND fieldName2:[3 TO 4]" arbitrary Apache Lucene query syntax, not needed for most uses of the API; if the q parameter is used, all other query parameters will be ignored, besides limit, page, and fields

Versioning

The Cumulus API is versioned and the current version is v1. Retrieve the latest API version from Cumulus.

To use any version, include the version number in the path before the query endpoint. {API_URL}/{version}/{query}

GET
/version

Example Request

$ curl https://example.com/version

Example Response

{
    "response_version": "v1",
    "api_version": "1.1.1"
}

Token

Returns a bearer token using oAuth with Earthdata Login service. The token will be returned as a JWT (JSON Web Token).

GET
/token

Query Parameters

query string parameter description
state={string} The URI to redirect to after if oAuth was successful

Example request

$ curl https://example.com/token

Example response

{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDY1MDk3ODczIn0.SxFtZ7dqp9KsUSn1uTXhWis8Il8Hig8mwLANGU3cXhY"}

Refresh token

Refreshes a bearer token received from oAuth with Earthdata Login service. The token will be returned as a JWT (JSON Web Token).

POST
/refresh

Request body

parameter type required description
token string true The JWT received from the /token endpoint to refresh

Example request

$ curl --request POST https://example.com/refresh --header 'Content-Type: application/json' --data '{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDY1MDk3ODczIn0.SxFtZ7dqp9KsUSn1uTXhWis8Il8Hig8mwLANGU3cXhY"}'

Example response

{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDcxMjk4ODEzIn0.vO6RlSRo47kkH15_muoUYNvv74fRzFxs7FlmVaarHlc"}

Delete token

Delete the record for an access token received from oAuth with Earthdata Login service.

token is the JWT containing access token information to delete

DELETE
/tokenDelete/{token}

Example request

$ curl --request DELETE https://example.com/tokenDelete/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDY1MDk3ODczIn0.SxFtZ7dqp9KsUSn1uTXhWis8Il8Hig8mwLANGU3cXhY

Example response

{"message": "Token record was deleted"}

Authorization header

When a request is made to the Cumulus API, it must contain a bearer token generated by the API.

The token is generated after the user login with Earthdata Login service.

The token is included in requests using the Authorization header.

If no token is provided, the Cumulus API server will respond with an error, requesting credentials. If an incorrect token is provided, the server will respond with a separate error noting this.

Authentication example

#! /bin/sh

ORIGIN=$(dirname $CUMULUS_BASEURL)
LOGIN_URL="$CUMULUS_BASEURL/token"

# create a base64 hash of your login credentials
AUTH=$(printf "$EARTHDATA_USERNAME:$EARTHDATA_PASSWORD" | base64)

# Request the Earthdata url with client id and redirect uri to use with Cumulus
AUTHORIZE_URL=$(curl -s -i ${LOGIN_URL} | grep location | sed -e "s/^location: //");

# Request an authorization grant code
TOKEN_URL=$(curl -s -i -X POST \
  -F "credentials=${AUTH}" \
  -H "Origin: ${ORIGIN}" \
  ${AUTHORIZE_URL%$'\r'} | grep Location | sed -e "s/^Location: //")

# Request the token through the CUMULUS API url that's returned from Earthdata
# Response is a JSON object of the form { token: String }
# This uses the cli tool jq to parse the JSON and get the token string
# More info on jq: https://stedolan.github.io/jq/
TOKEN=$(curl -s ${TOKEN_URL%$'\r'} | jq -r '.message.token')

echo $TOKEN

List providers

List providers in the Cumulus system.

GET
/providers

Example request

$ curl https://example.com/providers --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "daac-cumulus",
        "table": "provider",
        "limit": 1,
        "page": 1,
        "count": 2
    },
    "results": [
        {
            "id": "HTTP_MODIS",
            "globalConnectionLimit": 10,
            "protocol": "http",
            "host": "https://data.modis.gov/",
            "timestamp": 1508861082226
        }
    ]
}

Retrieve provider

Retrieve a single provider.

GET
/providers/{id}

Example request

$ curl https://example.com/providers/HTTP_MODIS --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "createdAt": 1508861081785,
    "id": "HTTP_MODIS",
    "host": "https://data.modis.gov/",
    "globalConnectionLimit": 10,
    "updatedAt": 1508861081785,
    "protocol": "http"
}

Create provider

Create a provider. For more information on creating providers and the contents of a request see the Cumulus setup documentation.

Overview of the schema fields:

Field Value Description
id string provider id/name
protocol "s3"|"http"|"https"|"ftp" file transfer (sync) protocol
host string provider host endpoint
port number provider host port
globalConnectionLimit number limit to number of concurrent connections
username string provider connection username
password string provider connection password
privateKey string filename assumed to be in s3://bucketInternal/stackName/crypto
cmKeyId string AWS KMS Customer Master Key arn or alias
POST
/providers

Example request

$ curl --request POST https://example.com/providers --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
    "host": "https://www.example.gov",
    "id": "MY_DAAC_SATELLITE",
    "protocol": "http",
    "globalConnectionLimit": 10
}'

Example response

{
    "message": "Record saved",
    "record": {
        "createdAt": 1491941727851,
        "host": "https://www.example.gov",
        "id": "MY_DAAC_SATELLITE",
        "protocol": "http",
        "globalConnectionLimit": 10,
        "timestamp": 1513956151186
    }
}

Update/replace provider

Update/replace an existing provider. Expects payload to specify the entire provider object, and will completely replace the existing provider with the specified payload. For a field reference see "Create provider".

Returns status 200 on successful replacement, 400 if the id property in the payload does not match the corresponding value in the resource URI, or 404 if there is no provider with the specified ID.

PUT
/providers/{id}

Example request

$ curl --request PUT https://example.com/providers/MY_DAAC_SATELLITE --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
  "id": "MY_DAAC_SATELLITE",
  "host": "https://www.example.co.uk",
  "globalConnectionLimit": 10,
  "protocol": "http"
}'

Example successful response

{
  "createdAt": 1491941727851,
  "id": "MY_DAAC_SATELLITE",
  "host": "https://www.example.co.uk",
  "globalConnectionLimit": 10,
  "updatedAt": 1513956150733,
  "protocol": "http",
  "timestamp": 1513956555713
}

Delete provider

Delete a provider from Cumulus. The related PDRs and granules remain in the Cumulus and CMR systems.

DELETE
/providers/{id}

Example request

$ curl --request DELETE https://example.com/providers/MY_DAAC_SATELLITE --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Record deleted"
}

List collections

List collections in the Cumulus system.

GET
/collections

Example request

$ curl https://example.com/collections --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "collection",
        "limit": 1,
        "page": 1,
        "count": 3
    },
    "results": [
        {
            "name": "MOD11A1",
            "version": "006",
            "dataType": "MOD11A1",
            "process": "modis",
            "provider_path": "/",
            "granuleId": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}$",
            "granuleIdExtraction": "(MOD11A1\\..*)\\.hdf",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
            "files": [
                {
                    "bucket": "protected",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
                },
                {
                    "bucket": "private",
                    "regex": "^BROWSE\\.MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
                    "sampleFileName": "BROWSE.MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
                },
                {
                    "bucket": "private",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf\\.met$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf.met"
                },
                {
                    "bucket": "protected",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.cmr\\.xml$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.cmr.xml"
                },
                {
                    "bucket": "public",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_2\\.jpg$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_2.jpg"
                },
                {
                    "bucket": "public",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_1\\.jpg$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_1.jpg"
                }
            ],
            "timestamp": 1513020427284,
            "createdAt": 1510761441174,
            "updatedAt": 1513020427162,
            "edpa": true,
            "some_other_field": "field",
            "duplicateHandling": "skip",
            "stats": {
                "running": 0,
                "completed": 6,
                "failed": 1,
                "total": 7
            }
        }
    ]
}

List collections with active granules

List collections in the Cumulus system that have active associated granules.

If time parameters are specified, the query will return collections that have granules that have been updated in that time frame.

GET
/collections/active

Example request

$ curl https://example.com/collections/active --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "collection",
        "limit": 1,
        "page": 1,
        "count": 3
    },
    "results": [
        {
            "name": "MOD11A1",
            "version": "006",
            "dataType": "MOD11A1",
            "process": "modis",
            "provider_path": "/",
            "granuleId": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}$",
            "granuleIdExtraction": "(MOD11A1\\..*)\\.hdf",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
            "files": [
                {
                    "bucket": "protected",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
                },
                {
                    "bucket": "private",
                    "regex": "^BROWSE\\.MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
                    "sampleFileName": "BROWSE.MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
                },
                {
                    "bucket": "private",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf\\.met$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf.met"
                },
                {
                    "bucket": "protected",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.cmr\\.xml$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.cmr.xml"
                },
                {
                    "bucket": "public",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_2\\.jpg$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_2.jpg"
                },
                {
                    "bucket": "public",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_1\\.jpg$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_1.jpg"
                }
            ],
            "timestamp": 1513020427284,
            "createdAt": 1510761441174,
            "updatedAt": 1513020427162,
            "edpa": true,
            "some_other_field": "field",
            "duplicateHandling": "skip",
            "stats": {
                "running": 0,
                "completed": 6,
                "failed": 1,
                "total": 7
            }
        }
    ]
}

Retrieve collection

Retrieve a single collection.

GET
/collections/{name}/{version}

Example request

$ curl https://example.com/collections/MOD11A1/006 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "process": "modis",
    "granuleIdExtraction": "(MOD11A1\\..*)\\.hdf",
    "version": "006",
    "dataType": "MOD11A1",
    "some_other_field": "field",
    "createdAt": 1510761441174,
    "edpa": true,
    "name": "MOD11A1",
    "duplicateHandling": "skip",
    "provider_path": "/",
    "files": [
        {
            "bucket": "protected",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
            "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$"
        },
        {
            "bucket": "private",
            "sampleFileName": "BROWSE.MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
            "regex": "^BROWSE\\.MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$"
        },
        {
            "bucket": "private",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf.met",
            "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf\\.met$"
        },
        {
            "bucket": "protected",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.cmr.xml",
            "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.cmr\\.xml$"
        },
        {
            "bucket": "public",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_2.jpg",
            "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_2\\.jpg$"
        },
        {
            "bucket": "public",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_1.jpg",
            "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_1\\.jpg$"
        }
    ],
    "updatedAt": 1513020427162,
    "granuleId": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}$",
    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
    "stats": {
        "running": 0,
        "completed": 6,
        "failed": 1,
        "total": 7
    }
}

Create collection

Create a collection. For more information on creating collections and the contents of a request see the Cumulus setup documentation. A collection generally includes, but is not limited to, the fields listed below.

Overview of the schema fields:

Field Value Description
name string collection name
version string collection version
dataType string matches collection with PDR datatype
duplicateHandling "replace"|"version"|"skip"|"error" duplicate handling protocol
process string process choice step variable
provider_path string path of remote files to sync
granuleId string (regex) regex to match granule IDs
granuleIdExtraction string (regex) regex to extract ID from files
sampleFileName string sample filename for granule ID
url_path string s3 prefix template
files array array of file specification objects
-- file.bucket string file destination bucket
-- file.regex string (regex) regex to match file names
-- file.sampleFileName string sample filename
-- file.url_path string s3 prefix template
-- file.fileType string granule file fileType mapping
POST
/collections

Example request

$ curl --request POST https://example.com/collections --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
  "name": "MOD09GQ",
  "version": "006",
  "dataType": "MOD09GQ",
  "process": "modis",
  "duplicateHandling": "replace",
  "provider_path": "cumulus-test-data/pdrs",
  "granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
  "granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
  "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
  "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
  "files": [
    {
      "bucket": "protected",
      "regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
      "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
      "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
    }
  ]
}'

Example response

{
    "message": "Record saved",
    "record": {
        "name": "MOD09GQ",
        "version": "006",
        "dataType": "MOD09GQ",
        "duplicateHandling": "replace",
        "process": "modis",
        "provider_path": "cumulus-test-data/pdrs",
        "granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
        "granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
        "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
        "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
        "files": [
            {
                "bucket": "protected",
                "regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
                "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
                "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
            }
        ],
        "createdAt": 1491946535919
    }
}

Update/replace collection

Update/replace an existing collection. Expects payload to specify the entire collection object, and will completely replace the existing collection with the specified payload. For a field reference see "Create collection".

Returns status 200 on successful replacement, 400 if the name or version property in the payload does not match the corresponding value in the resource URI, or 404 if there is no collection with the specified name and version.

PUT
/collections/{name}/{version}

Example request

$ curl --request PUT https://example.com/collections/MOD09GQ/006 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
  "name": "MOD09GQ",
  "version": "006",
  "dataType": "MOD09GQ",
  "duplicateHandling": "error",
  "newNeededField": "myCustomFieldValue",
  "process": "modis",
  "provider_path": "new_path/test-data",
  "granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
  "granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
  "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
  "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
  "files": [
    {
      "bucket": "protected",
      "regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
      "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
      "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
    }
  ]
}'

Example successful response

{
  "name": "MOD09GQ",
  "version": "006",
  "dataType": "MOD09GQ",
  "duplicateHandling": "error",
  "newNeededField": "myCustomFieldValue",
  "process": "modis",
  "provider_path": "new_path/test-data",
  "granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
  "granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
  "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
  "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
  "files": [
    {
      "bucket": "protected",
      "regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
      "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
      "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
    }
  ],
  "createdAt": 1491946535919,
  "updatedAt": 1514304825894
}

Delete collection

Delete a collection from Cumulus, but not from CMR. All related granules in Cumulus must have already been deleted from Cumulus.

DELETE
/collections/{name}/{version}

Example request

$ curl --request DELETE https://example.com/collections/MOD09GQ/006 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Record deleted"
}

List granules

List granules in the Cumulus system.

GET
/granules

Example request

$ curl https://example.com/granules --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "granule",
        "limit": 1,
        "page": 1,
        "count": 8
    },
    "results": [
        {
            "granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
            "pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
            "collectionId": "MOD11A1___006",
            "status": "completed",
            "provider": "LP_TS2_DataPool",
            "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:433612427488:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:7f071dae1a93c9892272b7fd5",
            "files": [
                {
                    "bucket": "cumulus-devseed-protected",
                    "path": "/TEST_B/Cumulus/MODIS/DATA",
                    "checksumValue": 964704694,
                    "filename": "s3://cumulus-devseed-protected/MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
                    "fileSize": 1447347,
                    "fileType": "data",
                    "checksumType": "CKSUM",
                    "name": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
                    "url_path": ""
                },
                {
                    "bucket": "cumulus-devseed-private",
                    "path": "/TEST_B/Cumulus/MODIS/DATA",
                    "checksumValue": 121318124,
                    "filename": "s3://cumulus-devseed-private/MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met",
                    "fileSize": 22559,
                    "fileType": "metadata",
                    "checksumType": "CKSUM",
                    "name": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met",
                    "url_path": ""
                },
                {
                    "bucket": "cumulus-devseed-private",
                    "path": "/TEST_B/Cumulus/MODIS/DATA",
                    "checksumValue": 2188150664,
                    "filename": "s3://cumulus-devseed-private/BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
                    "fileSize": 18118,
                    "fileType": "data",
                    "checksumType": "CKSUM",
                    "name": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
                    "url_path": ""
                },
                {
                    "bucket": "cumulus-devseed-protected",
                    "filename": "s3://cumulus-devseed-protected/MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
                    "fileType": "data",
                    "name": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
                },
                {
                    "bucket": "cumulus-devseed-public",
                    "filename": "s3://cumulus-devseed-public/MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg",
                    "fileType": "browse",
                    "name": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg"
                },
                {
                    "bucket": "cumulus-devseed-protected",
                    "granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
                    "filename": "s3://cumulus-devseed-protected/MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml",
                    "fileType": "metadata",
                    "name": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml"
                },
                {
                    "bucket": "cumulus-devseed-public",
                    "filename": "s3://cumulus-devseed-public/MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg",
                    "fileType": "browse",
                    "name": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg"
                }
            ],
            "error": null,
            "createdAt": 1513020455831,
            "timestamp": 1513020462156,
            "published": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
            "duration": 6.325,
            "cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS"
        }
    ]
}

Retrieve granule

Retrieve a single granule.

GET
/granules/{granuleId}

Example request

$ curl https://example.com/granules/MOD11A1.A2017137.h20v17.006.2017138085755 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
    "pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
    "collectionId": "MOD11A1___006",
    "status": "completed",
    "provider": "LP_TS2_DataPool",
    "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:433612427488:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:7f071dae1a93c9892272b7fd5",
    "files": [
        {
            "bucket": "cumulus-devseed-protected",
            "path": "/TEST_B/Cumulus/MODIS/DATA",
            "checksumValue": 964704694,
            "filename": "s3://cumulus-devseed-protected/MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
            "fileSize": 1447347,
            "checksumType": "CKSUM",
            "name": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
            "url_path": ""
        },
        {
            "bucket": "cumulus-devseed-private",
            "path": "/TEST_B/Cumulus/MODIS/DATA",
            "checksumValue": 121318124,
            "filename": "s3://cumulus-devseed-private/MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met",
            "fileSize": 22559,
            "checksumType": "CKSUM",
            "name": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met",
            "url_path": ""
        },
        {
            "bucket": "cumulus-devseed-private",
            "path": "/TEST_B/Cumulus/MODIS/DATA",
            "checksumValue": 2188150664,
            "filename": "s3://cumulus-devseed-private/BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
            "fileSize": 18118,
            "checksumType": "CKSUM",
            "name": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
            "url_path": ""
        },
        {
            "bucket": "cumulus-devseed-protected",
            "filename": "s3://cumulus-devseed-protected/MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
            "name": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
        },
        {
            "bucket": "cumulus-devseed-public",
            "filename": "s3://cumulus-devseed-public/MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg",
            "name": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg"
        },
        {
            "bucket": "cumulus-devseed-protected",
            "granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
            "filename": "s3://cumulus-devseed-protected/MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml",
            "name": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml"
        },
        {
            "bucket": "cumulus-devseed-public",
            "filename": "s3://cumulus-devseed-public/MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg",
            "name": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg"
        }
    ],
    "error": null,
    "createdAt": 1513020455831,
    "timestamp": 1513020462156,
    "published": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
    "duration": 6.325,
    "cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
    "_id": "MOD11A1.A2017137.h20v17.006.2017138085755"
}

Reingest granule

Reingest a granule. This causes the granule to re-download to Cumulus from source, and begin processing from scratch. Reingesting a granule will overwrite existing granule files.

A warning message is included in the response if the collection's duplicateHandling is not set to 'replace'.

PUT
/granules/{granuleId}

Example request

$ curl --request PUT https://example.com/granules/MOD11A1.A2017137.h20v17.006.2017138085755 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"action": "reingest"}'

Example response

{
    "granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
    "action": "reingest",
    "status": "SUCCESS",
    "warning": "The granule files may be overwritten"
}

Apply workflow to granule

Apply the named workflow to the granule. Workflow input will be built from template and provided entire Cumulus granule record as payload.

PUT
/granules/{granuleid}

Example request

$ curl --request PUT https://example.com/granules/MOD11A1.A2017137.h19v16.006.2017138085750 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{ "action": "applyWorkflow", "workflow": "inPlaceWorkflow" }'

Example response

{
    "granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
    "action": "applyWorkflow inPlaceWorkflow",
    "status": "SUCCESS"
}

Move a granule

Move a granule from one location on S3 to another. Individual files are moved to specific locations by using a regex that matches their filenames.

PUT
/granules/{granuleId}

Example request

$ curl --request PUT https://example.com/granules/MOD11A1.A2017137.h19v16.006.2017138085750 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{ "action": "move", "destinations": [{ "regex": ".*.hdf$", "bucket": "s3-bucket", "filepath": "new/filepath/" }]}'

Example response

{
  "granuleId": "MOD11A1.A2017137.h19v16.006.2017138085750",
  "action": "move",
  "status": "SUCCESS"
}

Remove granule from CMR

Remove a Cumulus granule from CMR.

PUT
/granules/{granuleId}

Example request

$ curl --request PUT https://example.com/granules/MOD11A1.A2017137.h19v16.006.2017138085750 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"action": "removeFromCmr"}'

Example response

{
    "granuleId": "MOD11A1.A2017137.h19v16.006.2017138085750",
    "action": "removeFromCmr",
    "status": "SUCCESS"
}

Delete granule

Delete a granule from Cumulus. It must already be removed from CMR.

DELETE
/granules/{granuleId}

Example request

$ curl --request DELETE https://example.com/granules/1A0000-2016121001_002_001 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "detail": "Record deleted"
}

Bulk Operations

Apply a workflow to the granules provided. Granules can be sent as a list of IDs or as an Elasticsearch query to the Metrics' Elasticsearch.

Overview of the request fields:

Field Required Value Description
workflow Y string Worfklow to be applied to all granules
queueName N string Queue to process Granules in
ids Y - if no query Array<string> List of IDs to process. Required if there is no Elasticsearch query provided
query Y - if no ids Object Query to Elasticsearch to determine which Granules to go through given workflow. Required if no IDs are given.
index Y - if query is present string Elasticsearch index to search with the given query

Use the Retrieve async operation endpoint with the id in the response to determine the status of the async operation.

POST
/granules/bulk

Example request with Elasticsearch query generated by Kibana:

$ curl --request POST \
    https://example.com/granules/bulk --header 'Authorization: Bearer ReplaceWithTheToken' \
  --data '{
        "workflowName": "HelloWorldWorkflow",
        index: "index-in-es",
        query": {
            "size": 500,
            "query": {
                "filter": [
                    {
                        "bool": {
                            "filter": [
                                {
                                    "bool": {
                                        "should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
                                        "minimum_should_match": 1
                                    }
                                },
                                {
                                    "bool": {
                                        "should": [{"match": {"status": "FAILED"}}],
                                        "minimum_should_match": 1
                                    }
                                }
                            ]
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    }'

Example request with given Granule IDs:

curl -X POST
  https://example.com/granules/bulk --header 'Authorization: Bearer ReplaceWithTheToken' --data '{"ids": ["MOD09GQ.A2016358.h13v04.006.2016360104606"], "workflowName": "HelloWorldWorkflow"}'

Example response

{
    "createdAt": 1574730504000,
    "id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
    "updatedAt": 1574730504762,
    "status": "RUNNING",
    "taskArn": "arn:aws:ecs:us-east-1:111111111111:task/d481e76e-f5fc-9c1c-2411-fa13779b111a"
}

Bulk Delete

Bulk delete the provided granules. Granules can be sent as a list of IDs or as an Elasticsearch query to the Metrics' Elasticsearch.

Overview of the request fields:

Field Required Value Description
forceRemoveFromCmr N bool Whether to remove published granules from CMR before deletion. You must set this value to true to do bulk deletion of published granules, otherwise deleting them will fail.
ids Y - if no query Array<string> List of IDs to process. Required if there is no Elasticsearch query provided
query Y - if no ids Object Query to Elasticsearch to determine which Granules to delete. Required if no IDs are given.
index Y - if query is present string Elasticsearch index to search with the given query

Use the Retrieve async operation endpoint with the id in the response to determine the status of the async operation.

POST
/granules/bulkDelete

Example request with Elasticsearch query generated by Kibana:

$ curl --request POST \
    https://example.com/granules/bulkDelete --header 'Authorization: Bearer ReplaceWithTheToken' \
  --data '{
        "forceRemoveFromCmr": true,
        index: "index-in-es",
        query": {
            "size": 500,
            "query": {
                "filter": [
                    {
                        "bool": {
                            "filter": [
                                {
                                    "bool": {
                                        "should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
                                        "minimum_should_match": 1
                                    }
                                },
                                {
                                    "bool": {
                                        "should": [{"match": {"status": "FAILED"}}],
                                        "minimum_should_match": 1
                                    }
                                }
                            ]
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    }'

Example request with given Granule IDs:

curl -X POST
  https://example.com/granules/bulkDelete --header 'Authorization: Bearer ReplaceWithTheToken' --data '{"ids": ["MOD09GQ.A2016358.h13v04.006.2016360104606"], "forceRemoveFromCmr": true}'

Example response

{
    "createdAt": 1574730504000,
    "id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
    "updatedAt": 1574730504762,
    "status": "RUNNING",
    "taskArn": "arn:aws:ecs:us-east-1:111111111111:task/d481e76e-f5fc-9c1c-2411-fa13779b111a"
}

List PDRs

List PDRs in the Cumulus system.

GET
/pdrs

Example request

$ curl https://example.com/pdrs --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "pdr",
        "limit": 1,
        "page": 1,
        "count": 8
    },
    "results": [
        {
            "pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
            "collectionId": "MOD11A1___006",
            "status": "failed",
            "provider": "LP_TS2_DataPool",
            "progress": 0,
            "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:433612427488:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:6ef0c52f83c549db58b3a1e50",
            "PANSent": false,
            "PANmessage": "N/A",
            "stats": {
                "processing": 0,
                "completed": 0,
                "failed": 0,
                "total": 0
            },
            "createdAt": 1514305411204,
            "timestamp": 1514305424036,
            "duration": 12.832
        }
    ]
}

Retrieve PDR

Retrieve a single PDR.

GET
/pdrs/{pdrName}

Example request

$ curl https://example.com/pdrs/7970bff5-128a-489f-b43c-de4ad7834ce5.PDR --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
    "collectionId": "MOD11A1___006",
    "status": "failed",
    "provider": "LP_TS2_DataPool",
    "progress": 0,
    "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:433612427488:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:6ef0c52f83c549db58b3a1e50",
    "PANSent": false,
    "PANmessage": "N/A",
    "stats": {
        "processing": 0,
        "completed": 0,
        "failed": 0,
        "total": 0
    },
    "createdAt": 1514305411204,
    "timestamp": 1514305424036,
    "duration": 12.832,
    "_id": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR"
}

Delete PDR

Delete a PDR from Cumulus. Its granules will remain, and the PDR may be re-discovered and re-ingested/re-processed from scratch in the future.

DELETE
/pdrs/{pdrName}

Example request

$ curl --request DELETE https://example.com/pdrs/good_25grans.PDR --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Record deleted"
}

List rules

List rules in the Cumulus system.

GET
/rules

Example request

$ curl https://example.com/rules --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "rule",
        "limit": 1,
        "page": 1,
        "count": 7
    },
    "results": [
        {
            "name": "repeat",
            "workflow": "DiscoverPdrs",
            "provider": "local",
            "collection": {
                "name": "AST_L1A",
                "version": "003"
            },
            "rule": {
                "type": "scheduled",
                "value": "rate(5 minutes)"
            },
            "timestamp": 1511232462534,
            "state": "DISABLED"
        }
    ]
}

Retrieve rule

Retrieve a single rule.

GET
/rules/{name}

Example request

$ curl https://example.com/rules/repeat --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "workflow": "DiscoverPdrs",
    "collection": {
        "name": "AST_L1A",
        "version": "003"
    },
    "updatedAt": 1511232462507,
    "createdAt": 1510903518741,
    "provider": "local",
    "name": "repeat",
    "rule": {
        "type": "scheduled",
        "value": "rate(5 minutes)"
    },
    "state": "DISABLED"
}

Create rule

Create a rule. For more information on creating rules and the contents of a request see the Cumulus setup documentation.

Overview of the schema fields:

Field Value Description
name string rule name (letters, numbers, underscores only)
state "DISABLED"|"ENABLED" rule state (default: ENABLED)
workflow string name of workflow started by the rule
rule Object rule object
-- rule.type "onetime"|"scheduled"|"kinesis"|"sns"|"sqs" rule trigger type
-- rule.value onetime: N/A
scheduled: cron-type or rate expression
kinesis: Kinesis stream ARN
sns: SNS topic ARN
sqs: SQS queue URL
required value differs by type
-- rule.arn string kinesis scheduled event arn
-- rule.logEventArn string kinesis scheduled log event arn
provider string provider record provided to workflow (optional)
collection Object collection record provided to workflow (optional)
-- collection.name string collection name
-- collection.version string collection version
meta Object contents to add to workflow input's meta field
tags array Optional tags (for search)
POST
/rules

Example request

$ curl --request POST https://example.com/rules --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
    "workflow": "DiscoverPdrs",
    "collection": {
        "name": "AST_L1A",
        "version": "003"
    },
    "provider": "local",
    "name": "repeat_test",
    "rule": {
        "type": "scheduled",
        "value": "rate(5 minutes)"
    },
    "meta": { "publish": false },
    "state": "DISABLED"
}'

Example response

{
    "message": "Record saved",
    "record": {
        "workflow": "DiscoverPdrs",
        "collection": {
            "name": "AST_L1A",
            "version": "003"
        },
        "createdAt": 1510903518741,
        "provider": "local",
        "name": "repeat_test",
        "rule": {
            "type": "scheduled",
            "value": "rate(5 minutes)"
        },
        "meta": { "publish": false },
        "state": "DISABLED",
    }
}

Update/replace rule

Update/replace an existing rule. Expects payload to specify the entire rule object, and will completely replace the existing rule with the specified payload. For a field reference see "Create rule".

Returns status 200 on successful replacement, 400 if the name property in the payload does not match the corresponding value in the resource URI, or 404 if there is no rule with the specified name.

Special case:

Field Value Description
action "rerun" rerun rule (onetime rule only)
PUT
/rules/{name}

Example request

$ curl --request PUT https://example.com/rules/repeat_test --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
  "name": "repeat_test",
  "workflow": "DiscoverPdrs",
  "collection": {
    "name": "AST_L1A",
    "version": "003"
  },
  "provider": "local",
  "rule": {
    "type": "scheduled",
    "value": "rate(5 minutes)"
  },
  "state": "ENABLED"
}'

Example successful response

{
  "name": "repeat_test",
  "workflow": "DiscoverPdrs",
  "collection": {
    "name": "AST_L1A",
    "version": "003"
  },
  "updatedAt": 1521755265130,
  "createdAt": 1510903518741,
  "provider": "local",
  "rule": {
    "type": "scheduled",
    "value": "rate(5 minutes)"
  },
  "state": "ENABLED"
}

Rerun request (onetime-rule special case)

$ curl --request PUT https://example.com/rules/my_onetime_rule --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"action": "rerun"}'

Delete rule

Delete a rule from Cumulus.

DELETE
/rules/{name}

Example request

$ curl --request DELETE https://example.com/rules/repeat_test --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Record deleted"
}

Summary

Retrieve a summary of statistics around the granules in the system. The collections returned are the number of distinct collections for the granules active during the given time period, defaulted to the last day if none is specified.

GET
/stats

Example Request

$ curl https://example.com/stats --header 'Authorization: Bearer ReplaceWithTheToken'

Example success response

{
    "errors": {
        "dateFrom": "1970-01-18T12:36:59+00:00",
        "dateTo": "2017-12-26T04:38:15+00:00",
        "value": 2,
        "aggregation": "count",
        "unit": "error"
    },
    "collections": {
        "dateFrom": "1970-01-01T12:00:00+00:00",
        "dateTo": "2017-12-26T04:38:15+00:00",
        "value": 3,
        "aggregation": "count",
        "unit": "collection"
    },
    "processingTime": {
        "dateFrom": "1970-01-18T12:36:59+00:00",
        "dateTo": "2017-12-26T04:38:15+00:00",
        "value": null,
        "aggregation": "average",
        "unit": "second"
    },
    "granules": {
        "dateFrom": "1970-01-18T12:36:59+00:00",
        "dateTo": "2017-12-26T04:38:15+00:00",
        "value": 8,
        "aggregation": "count",
        "unit": "granule"
    }
}

Count

Count the value frequencies for a given field, for a given type of record in Cumulus. Requires the following query parameters, and may include the regular filter parameters:

query string parameter description
type={providers|collections|granules|pdrs|logs} type of Cumulus record to query
field={fieldName} which field to count frequencies for; no default
GET
/stats/aggregate

Example request

curl 'https://example.com/stats/aggregate?field=status&type=pdrs' --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "meta": {
    "name": "cumulus-api",
    "count": 52,
    "field": "status.keyword"
  },
  "count": [
    {
      "key": "failed",
      "count": 43
    },
    {
      "key": "completed",
      "count": 5
    },
    {
      "key": "parsed",
      "count": 3
    }
  ]
}

List logs

List processing logs from the Cumulus engine. A log's level field should be specified by their string value and be one of:

  • fatal (60)
  • error (50)
  • warn (40)
  • info (30)
  • debug (20)
  • trace (10)
GET
/logs

Example request

$ curl https://example.com/logs?limit=5 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "logs",
        "limit": 5,
        "page": 1,
        "count": 750
    },
    "results": [
        {
            "pid": 1,
            "hostname": "ip-10-29-4-186",
            "name": "cumulus",
            "level": 50,
            "msg": "https://ba008ffc.ngrok.io/ not found",
            "file": "discover-pdrs/index.js",
            "type": "Error",
            "stack": "HostNotFound: https://ba008ffc.ngrok.io/ not found\n    at discover.discover.then.catch.e (/var/task/index.js:9006:22)\n    at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
            "message": "https://ba008ffc.ngrok.io/ not found",
            "v": 1,
            "timestamp": 1513881407867
        },
        {
            "pid": 1,
            "hostname": "ip-10-29-4-186",
            "name": "cumulus",
            "level": 50,
            "file": "discover-pdrs/index.js",
            "message": "Received a 404 error from undefined. Check your endpoint!",
            "details": {
                "host": "ba008ffc.ngrok.io",
                "path": "/",
                "port": "",
                "protocol": "https",
                "uriPath": "/",
                "url": "https://ba008ffc.ngrok.io/",
                "depth": 1,
                "fetched": true,
                "status": "notfound",
                "stateData": {
                    "requestLatency": 66,
                    "requestTime": 66,
                    "contentLength": 34,
                    "contentType": "text/plain",
                    "code": 404,
                    "headers": {
                        "content-length": "34",
                        "connection": "close",
                        "content-type": "text/plain"
                    }
                },
                "id": 0
            },
            "v": 1,
            "timestamp": 1513881407867
        },
        {
            "pid": 1,
            "hostname": "ip-10-29-4-186",
            "name": "cumulus",
            "level": 50,
            "msg": "Cannot read property 'provider_path' of undefined",
            "file": "discover-pdrs/index.js",
            "type": "Error",
            "stack": "TypeError: Cannot read property 'provider_path' of undefined\n    at HttpDiscover.Discover (/var/task/index.js:10098:33)\n    at HttpDiscover._class (/var/task/index.js:96368:255)\n    at new HttpDiscover (/var/task/index.js:10342:241)\n    at handler (/var/task/index.js:8968:23)\n    at Promise (/var/task/cumulus-sled/index.js:82:5)\n    at invokeHandler (/var/task/cumulus-sled/index.js:72:10)\n    at then.then.then (/var/task/cumulus-sled/index.js:112:14)\n    at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
            "v": 1,
            "timestamp": 1513881031694
        },
        {
            "pid": 1,
            "hostname": "ip-10-29-4-186",
            "name": "cumulus",
            "level": 50,
            "msg": "Provider info not provided",
            "file": "discover-pdrs/index.js",
            "type": "Error",
            "stack": "ProviderNotFound: Provider info not provided\n    at handler (/var/task/index.js:8962:20)\n    at Promise (/var/task/cumulus-sled/index.js:82:5)\n    at invokeHandler (/var/task/cumulus-sled/index.js:72:10)\n    at then.then.then (/var/task/cumulus-sled/index.js:112:14)\n    at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
            "message": "Provider info not provided",
            "v": 1,
            "timestamp": 1513879654040
        },
        {
            "pid": 1,
            "hostname": "ip-10-29-4-186",
            "name": "cumulus",
            "level": 50,
            "msg": "Provider info not provided",
            "file": "discover-pdrs/index.js",
            "type": "Error",
            "stack": "ProviderNotFound: Provider info not provided\n    at handler (/var/task/index.js:8962:20)\n    at Promise (/var/task/cumulus-sled/index.js:82:5)\n    at invokeHandler (/var/task/cumulus-sled/index.js:72:10)\n    at then.then.then (/var/task/cumulus-sled/index.js:112:14)\n    at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
            "message": "Provider info not provided",
            "v": 1,
            "timestamp": 1513879435010
        }
    ]
}

Example request for errors

$ curl https://example.com/logs?level=error --header 'Authorization: Bearer ReplaceWithTheToken'

Example response for errors

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "logs",
        "limit": 1,
        "page": 1,
        "count": 181
    },
    "results": [
        {
            "level": 50,
            "msg": "Unexpected HTTP status code: 404",
            "pid": 1,
            "hostname": "ip-10-26-125-174",
            "name": "cumulus",
            "file": "sync-granule/index.js",
            "type": "Error",
            "stack": "Error: Unexpected HTTP status code: 404\n    at ClientRequest.<anonymous> (/var/task/sync-granule/index.js:125136:29)\n    at ClientRequest.g (events.js:292:16)\n    at emitOne (events.js:96:13)\n    at ClientRequest.emit (events.js:188:7)\n    at HTTPParser.parserOnIncomingClient (_http_client.js:473:21)\n    at HTTPParser.parserOnHeadersComplete (_http_common.js:99:23)\n    at TLSSocket.socketOnData (_http_client.js:362:20)\n    at emitOne (events.js:96:13)\n    at TLSSocket.emit (events.js:188:7)\n    at readableAddChunk (_stream_readable.js:176:18)",
            "code": 404,
            "v": 1,
            "timestamp": 1515446810226
        }
    ]
}

Retrieve Logs

Retrieve all logs from a specific execution.

GET
/logs/{executionName}

Example request

$ curl https://example.com/logs/93e5e049-89aa-47e5-900d-0eec87327260?limit=5 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "test-src-integration",
        "table": "logs",
        "limit": 5,
        "page": 1,
        "count": 10
    },
    "results": [
        {
            "level": 30,
            "executions": "93e5e049-89aa-47e5-900d-0eec87327260",
            "timestamp": "2018-08-15T19:23:57.752Z",
            "sender": "test-src-integration-PostToCmr",
            "version": "$LATEST",
            "message": "Published MOD09GQ.A8009365.7JQhni.006.6594428626875 to the CMR. conceptId: G1223210348-CUMULUS",
            "RequestId": "05712fb1-5593-4640-b5a3-a99db41d8003"
        },
        {
            "level": 30,
            "executions": "93e5e049-89aa-47e5-900d-0eec87327260",
            "timestamp": "2018-08-15T19:23:42.054Z",
            "sender": "test-src-integration-SyncGranuleNoVpc",
            "version": "$LATEST",
            "message": "uploaded s3://cumulus-test-sandbox-internal/file-staging/test-src-integration/MOD09GQ___006/MOD09GQ.A8009365.7JQhni.006.6594428626875.hdf",
            "RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
        },
        {
            "level": 30,
            "executions": "93e5e049-89aa-47e5-900d-0eec87327260",
            "timestamp": "2018-08-15T19:23:41.913Z",
            "sender": "test-src-integration-SyncGranuleNoVpc",
            "version": "$LATEST",
            "message": "Finishing downloading s3://cumulus-test-sandbox-internal/cumulus-test-data/pdrs/MOD09GQ.A8009365.7JQhni.006.6594428626875.hdf",
            "RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
        },
        {
            "level": 30,
            "executions": "93e5e049-89aa-47e5-900d-0eec87327260",
            "timestamp": "2018-08-15T19:23:41.888Z",
            "sender": "test-src-integration-SyncGranuleNoVpc",
            "version": "$LATEST",
            "message": "uploaded s3://cumulus-test-sandbox-internal/file-staging/test-src-integration/MOD09GQ___006/MOD09GQ.A8009365.7JQhni.006.6594428626875_ndvi.jpg",
            "RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
        },
        {
            "level": 30,
            "executions": "93e5e049-89aa-47e5-900d-0eec87327260",
            "timestamp": "2018-08-15T19:23:41.886Z",
            "sender": "test-src-integration-SyncGranuleNoVpc",
            "version": "$LATEST",
            "message": "uploaded s3://cumulus-test-sandbox-internal/file-staging/test-src-integration/MOD09GQ___006/MOD09GQ.A8009365.7JQhni.006.6594428626875.hdf.met",
            "RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
        }
    ]
}

Get Granule CSV file

Get a CSV file of all the granule in the Cumulus database.

GET
/granule-csv

Example request

$ curl https://example.com/granule-csv --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

"granuleUr","collectionId","createdAt","startDateTime","endDateTime","status","updatedAt","published"
"MOD14A1.A9506271.IvEJsu.006.8359924290786","MOD14A1___006","2020-05-18T20:15:54.525Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-05-18T20:16:02.473Z",false
"MYD13Q1.A9663671.0zkwKH.006.9812354158395","MYD13Q1___006","2020-07-06T19:46:19.957Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-07-06T19:46:57.054Z",true

List executions

List executions.

GET
/executions

Example request

$ curl https://example.com/executions?limit=3 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "meta": {
    "name": "cumulus-api",
    "stack": "test-src-integration",
    "table": "execution",
    "limit": 3,
    "page": 1,
    "count": 447
  },
  "results": [
    {
      "arn": "arn:aws:states:us-east-1:596205514284:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
      "duration": 1.693,
      "createdAt": 1534443719571,
      "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:596205514284:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
      "name": "08aa40f8-7d91-41b4-986e-d12ce495aec5",
      "error": {},
      "type": "IngestGranule",
      "collectionId": "MOD09GQ___006",
      "tasks": {},
      "status": "running",
      "timestamp": 1534443721559,
      "updatedAt": 1534443721265
    },
    {
      "arn": "arn:aws:states:us-east-1:596205514284:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
      "duration": 1.718,
      "createdAt": 1534443706867,
      "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:596205514284:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
      "name": "bee18782-e987-4bd4-b726-41669e489e2f",
      "error": {},
      "type": "IngestGranule",
      "collectionId": "MOD09GQ___006",
      "tasks": {},
      "status": "running",
      "timestamp": 1534443709043,
      "updatedAt": 1534443708585
    },
    {
      "arn": "arn:aws:states:us-east-1:596205514284:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
      "duration": 29.397,
      "createdAt": 1534443668185,
      "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:596205514284:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
      "name": "55d62b27-50cd-4cc1-81f9-e425cf10532e",
      "error": {
        "Cause": "None",
        "Error": "Unknown Error"
      },
      "type": "IngestGranule",
      "collectionId": "MOD09GQ___006",
      "tasks": {},
      "status": "completed",
      "timestamp": 1534443698322,
      "updatedAt": 1534443697582
    }
  ]
}

Retrieve Execution

Retrieve details for a specific execution.

GET
/executions/{executionName}

Example request

$ curl https://example.com/executions/cff1266e-ef36-664f-a649-3a4d26bd1735 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:596205514284:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
    "updatedAt": 1534441782302,
    "status": "completed",
    "timestamp": 1534441782302,
    "tasks":
    {
        "MoveGranuleStep":
        {
            "name": "test-src-integration-MoveGranules",
            "arn": "arn:aws:lambda:us-east-1:596205514284:function:test-src-integration-MoveGranules",
            "version": "$LATEST"
        },
        "ProcessingStep":
        {
            "name": "test-src-integration-FakeProcessing",
            "arn": "arn:aws:lambda:us-east-1:596205514284:function:test-src-integration-FakeProcessing",
            "version": "$LATEST"
        },
        "StopStatus":
        {
            "name": "test-src-integration-SfSnsReport",
            "arn": "arn:aws:lambda:us-east-1:596205514284:function:test-src-integration-SfSnsReport",
            "version": "$LATEST"
        }
    },
    "createdAt": 1534441752026,
    "duration": 30.276,
    "name": "cff1266e-ef36-664f-a649-3a4d26bd1735",
    "arn": "arn:aws:states:us-east-1:596205514284:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
    "collectionId": "MOD09GQ___006",
    "error":
    {
        "Error": "Unknown Error",
        "Cause": "None"
    },
    "type": "IngestGranule",
    "originalPayload": { "somePayloadKey": "object containing payload at execution start" },
    "finalPayload": { "somePayloadKey" : "object containing the last reported payload from an execution" }
}

Retrieve Execution Status

Retrieve details and status of a specific execution. This also returns execution history and details of the state machine when the execution exists in Step Function API.

GET
/executions/status/{executionArn}

Example request

$ curl https://example.com/executions/status/arn:aws:states:us-east-1:596205514284:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl6sgv:d0a6584b-bea6-476e-a745-c1feb2ad00b2 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "execution":
    {
        "executionArn": "arn:aws:states:us-east-1:596205514284:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv:d0a6584b-bea6-476e-a745-c1feb2ad00b2",
        "stateMachineArn": "arn:aws:states:us-east-1:596205514284:stateMachine:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv",
        "name": "d0a6584b-bea6-476e-a745-c1feb2ad00b2",
        "status": "SUCCEEDED",
        "startDate": "2018-08-16T18:39:32.209Z",
        "stopDate": "2018-08-16T18:39:59.089Z",
        "input":
        {
            "foo": "input"
        },
        "output":
        {
            "bar": "output"
        }
    },
    "executionHistory":
    {
        "events":
        [
            {
                "taskName": "foo bar 1"
            },
            {
                "taskName": "foo bar 2"
            }
        ]
    },
    "stateMachine":
    {
        "stateMachineArn": "arn:aws:states:us-east-1:596205514284:stateMachine:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv",
        "name": "TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv",
        "status": "ACTIVE",
        "definition":
        {
            "Comment": "Ingest Granule",
            "StartAt": "First State",
            "States":
            {
                "First State": {}
            }
        },
        "roleArn": "arn:aws:iam::596205514284:role/test-src-integration-steprole",
        "creationDate": "2018-06-11T16:08:17.533Z"
    }
}

List workflows

List workflows in the Cumulus system.

GET
/workflows

Example request

$ curl https://example.com/workflows --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

[
    {
        "name": "TestLambdaVersionWorkflow",
        "template": "s3://cumulus-test-sandbox-internal/cumulus/workflows/TestLambdaVersionWorkflow.json",
        "definition": {
            "Comment": "Tests Lambda update after redeploy",
            "StartAt": "StartStatus",
            "States": {
                "StartStatus": {
                    "Type": "Task",
                    "Resource": "${SfSnsReportLambdaAliasOutput}",
                    "Next": "WaitForDeployment"
                },
                "WaitForDeployment": {
                    "Type": "Task",
                    "Resource": "${WaitForDeploymentLambdaAliasOutput}",
                    "Next": "VersionUpTest"
                },
                "VersionUpTest": {
                    "Type": "Task",
                    "Resource": "${VersionUpTestLambdaAliasOutput}",
                    "Next": "StopStatus"
                },
                "StopStatus": {
                    "Type": "Task",
                    "Resource": "${SfSnsReportLambdaAliasOutput}",
                    "Catch": [
                        {
                            "ErrorEquals": [
                                "States.ALL"
                            ],
                            "Next": "WorkflowFailed"
                        }
                    ],
                    "End": true
                },
                "WorkflowFailed": {
                    "Type": "Fail",
                    "Cause": "Workflow failed"
                }
            }
        }
    },
    {
        "name": "HelloWorldWorkflow",
        "template": "s3://cumulus-test-sandbox-internal/cumulus/workflows/HelloWorldWorkflow.json",
        "definition": {
            "Comment": "Returns Hello World",
            "StartAt": "StartStatus",
            "States": {
                "StartStatus": {
                    "Type": "Task",
                    "Resource": "${SfSnsReportLambdaAliasOutput}",
                    "Next": "HelloWorld"
                },
                "HelloWorld": {
                    "Type": "Task",
                    "Resource": "${HelloWorldLambdaAliasOutput}",
                    "Next": "StopStatus"
                },
                "StopStatus": {
                    "Type": "Task",
                    "Resource": "${SfSnsReportLambdaAliasOutput}",
                    "Catch": [
                        {
                            "ErrorEquals": [
                                "States.ALL"
                            ],
                            "Next": "WorkflowFailed"
                        }
                    ],
                    "End": true
                },
                "WorkflowFailed": {
                    "Type": "Fail",
                    "Cause": "Workflow failed"
                }
            }
        }
    }
]

Retrieve workflow

Retrieve a single workflow.

GET
/workflow/{name}

Example request

$ curl https://example.com/workflows/HelloWorldWorkflow --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "name": "HelloWorldWorkflow",
    "template": "s3://cumulus-test-sandbox-internal/cumulus/workflows/HelloWorldWorkflow.json",
    "definition": {
        "Comment": "Returns Hello World",
        "StartAt": "StartStatus",
        "States": {
            "StartStatus": {
                "Type": "Task",
                "Resource": "${SfSnsReportLambdaAliasOutput}",
                "Next": "HelloWorld"
            },
            "HelloWorld": {
                "Type": "Task",
                "Resource": "${HelloWorldLambdaAliasOutput}",
                "Next": "StopStatus"
            },
            "StopStatus": {
                "Type": "Task",
                "Resource": "${SfSnsReportLambdaAliasOutput}",
                "Catch": [
                    {
                        "ErrorEquals": [
                            "States.ALL"
                        ],
                        "Next": "WorkflowFailed"
                    }
                ],
                "End": true
            },
            "WorkflowFailed": {
                "Type": "Fail",
                "Cause": "Workflow failed"
            }
        }
    }
}

List async operations

Async operations are long-running requests serviced by the Cumulus API.

These tend to be bulk operations. Examples include bulk granule operations and replaying ingest notifications.

This endpoint lists async operations in the Cumulus system.

GET
/asyncOperations

Example request

$ curl https://example.com/asyncOperations --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "asyncOperation",
        "limit": 1,
        "page": 1,
        "count": 8
    },
    "results": [
        {
          "output": "{\"deletedGranules\":[\"granule-id-f02a53418f\"]}",
          "createdAt": 1591384094512,
          "taskArn": "arn:aws:ecs:us-east-1:111111111111:task/d481e76e-f5fc-9c1c-2411-fa13779b111a",
          "description": "Bulk granule deletion",
          "operationType": "Bulk Granule Delete",
          "id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
          "status": "SUCCEEDED",
          "updatedAt": 1591384094512,
          "timestamp": 1591384095235
        }
    ]
}

Retrieve async operation

Retrieve information about a single async operation. Useful for determining the status of an async operation.

GET
/asyncOperations/{id}

Example request

$ curl https://example.com/asyncOperations/0eb8e809-8790-5409-1239-bcd9e8d28b8e --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
  "updatedAt": 1574730504762,
  "status": "RUNNING",
  "taskArn": "arn:aws:ecs:us-east-1:111111111111:task/d481e76e-f5fc-9c1c-2411-fa13779b111a",
  "description": "Bulk granule deletion",
  "operationType": "Bulk Granule Delete"
}

Ingest Replays

The Cumulus API supports requests to replay ingest notifications. The schema below describes the expected fields in a replay request

Field Type Required Description
type string required Currently only accepts kinesis.
kinesisStream string for type kinesis Any valid kinesis stream name (not ARN)
kinesisStreamCreationTimestamp * optional Any input valid for a JS Date constructor. For reasons to use this field see AWS documentation on StreamCreationTimestamp.
endTimestamp * optional Any input valid for a JS Date constructor. Messages newer than this timestamp will be skipped.
startTimestamp * optional Any input valid for a JS Date constructor. Messages will be fetched from the Kinesis stream starting at this timestamp. Ignored if it is further in the past than the stream's retention period.

Example Request

$ curl -X POST https://example.com/replays --header 'Authorization: Bearer ReplaceWithTheToken' --data '{ "type: "kinesis", "kinesisStream": "my-stream", "endTimestamp": 1567890123456, "startTimestamp": "2019-08-31T22:22:03.456Z"}'

Example Response

{
  "asyncOperationId": "208a463a-e096-4dd9-b006-e09345319ae6"
}

Retrieve schema

Retrieve the data schema for a particular type of Cumulus record.

This schema describes the expected format of a record's JSON object when retrieving from Cumulus, as well as a summary of what each field may contain. The schema response can also be used to determine which fields are required when creating a new record using the API.

Supported type values are provider, collection, granule, and pdr.

GET
/schemas/{type}

Example request

$ curl https://example.com/schemas/provider --header 'Authorization: Bearer ReplceWithTheToken'

Example response

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "title": "Provider Object",
  "description": "Keep the information about each ingest endpoint",
  "type": "object",
  "properties": {
    "name": {
      "title": "Title",
      "description": "A title for the provider record",
      "type": "string",
      "pattern": "^([\\w\\d_\\-]*)$"
    },
    "providerName": {
      "title": "Provider, e.g. MODAPS",
      "description": "Name of the SIP",
      "type": "string"
    },
    "protocol": {
      "title": "Protocol",
      "type": "string",
      "enum": [
        "http",
        "ftp"
      ],
      "default": "http"
    },
    "host": {
      "title": "Host",
      "type": "string"
    },
    "path": {
      "title": "Path to the PDR/files folder",
      "type": "string"
    },
    "config": {
      "title": "Configuration",
      "type": "object",
      "properties": {
        "username": {
          "type": "string"
        },
        "password": {
          "type": "string"
        },
        "port": {
          "type": "string"
        }
      }
    },
    "status": {
      "title": "Status",
      "type": "string",
      "enum": [
        "ingesting",
        "stopped",
        "failed"
      ],
      "default": "stopped",
      "readonly": true
    },
    "isActive": {
      "title": "Is Active?",
      "type": "boolean",
      "default": false,
      "readonly": true
    },
    "regex": {
      "type": "object",
      "patternProperties": {
        "^([\\S]*)$": {
          "type": "string"
        }
      },
      "readonly": true
    },
    "lastTimeIngestedAt": {
      "title": "Last Time Ingest from the Provider",
      "type": "number",
      "readonly": true
    },
    "createdAt": {
      "type": "number",
      "readonly": true
    },
    "updatedAt": {
      "type": "number",
      "readonly": true
    }
  },
  "required": [
    "name",
    "providerName",
    "protocol",
    "host",
    "path",
    "isActive",
    "status",
    "createdAt",
    "updatedAt"
  ]
}

List reconciliation reports

List reconciliation reports in the Cumulus system.

GET
/reconciliationReports

Example request

$ curl https://example.com/reconciliationReports --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "reconciliationReport",
        "limit": 2,
        "page": 1,
        "count": 7
    },
    "results": [
        {
            "createdAt": 1589824212102,
            "name": "inventoryReport-20200518T175012101",
            "location": "s3://lpdaac-internal/lpdaac-cumulus/reconciliation-reports/inventoryReport-20200518T175012101.json",
            "type": "Inventory",
            "status": "Generated",
            "updatedAt": 1589824218825,
            "timestamp": 1589824219329
        },
        {
            "createdAt": 1589596344305,
            "name": "ModisInternalReport",
            "location": "s3://lpdaac-internal/lpdaac-cumulus/reconciliation-reports/ModisInternalReport.json",
            "type": "Internal",
            "status": "Generated",
            "updatedAt": 1589596352010,
            "timestamp": 1589596352356
        }
    ]
}

Retrieve reconciliation report

Retrieve a single reconciliation report.

GET
/reconciliationReports/{name}

Example request

$ curl https://example.com/reconciliationReports/inventoryReport-20190305T153430508 --header 'Authorization: Bearer ReplaceWithTheToken'

Example responses

Inventory Report
{
    "reportStartTime": "2019-03-05T15:34:30.508Z",
    "reportEndTime": "2019-03-05T15:34:37.243Z",
    "status": "SUCCESS",
    "error": null,
    "type": "Inventory",
    "filesInCumulus": {
        "okCount": 40,
        "okCountByGranule": {
          "MOD09GQ.A2016358.h13v04.006.2016360104606": 10,
          "MYD13Q1.A2017297.h19v10.006.2017313221303": 10,
          "MOD09GQ.A3518809.ln_rVr.006.7962927138074": 10,
          "MOD09GQ.A8768252.HC4ddD.006.2077696236118": 5,
          "MOD09GQ.A8722843.GTk5A3.006.4026909316904": 5
        },
        "onlyInS3": [
            "s3://cumulus-test-sandbox-protected/MOD09GQ.A2016358.h13v04.006.2016360104606.cmr.xml",
            "s3://cumulus-test-sandbox-private/BROWSE.MYD13Q1.A2017297.h19v10.006.2017313221201.hdf"
        ],
        "onlyInDynamoDb": [
            {
                "uri": "s3://cumulus-test-sandbox-protected/MOD09GQ.A2016358.h13v04.006.2016360104606.hdf",
                "granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"
            },
            {
                "uri": "s3://cumulus-test-sandbox-private/MYD13Q1.A2017297.h19v10.006.2017313221303.hdf.met",
                "granuleId": "MYD13Q1.A2017297.h19v10.006.2017313221303"
            }
        ]
    },
    "collectionsInCumulusCmr": {
        "okCount": 1,
        "onlyInCumulus": [
            "L2_HR_PIXC___000"
        ],
        "onlyInCmr": [
            "MCD43A1___006",
            "MOD14A1___006"
        ]
    },
    "granulesInCumulusCmr": {
        "okCount": 3,
        "onlyInCumulus": [
            {
                "granuleId": "MOD09GQ.A3518809.ln_rVr.006.7962927138074",
                "collectionId": "MOD09GQ___006"
            },
            {
                "granuleId": "MOD09GQ.A8768252.HC4ddD.006.2077696236118",
                "collectionId": "MOD09GQ___006"
            }
        ],
        "onlyInCmr": [
            {
                "GranuleUR": "MOD09GQ.A0002421.oD4zvB.006.4281362831355",
                "ShortName": "MOD09GQ",
                "Version": "006"
            },
            {
                "GranuleUR": "MOD09GQ.A0007443.H16AHq.006.7078190437865",
                "ShortName": "MOD09GQ",
                "Version": "006"
            }
        ]
    },
    "filesInCumulusCmr": {
        "okCount": 11,
        "onlyInCumulus": [
            {
                "fileName": "MOD09GQ.A8722843.GTk5A3.006.4026909316904.jpeg",
                "uri": "s3://cumulus-test-sandbox-public/MOD09GQ___006/MOD/MOD09GQ.A8722843.GTk5A3.006.4026909316904.jpeg",
                "granuleId": "MOD09GQ.A8722843.GTk5A3.006.4026909316904"
            }
        ],
        "onlyInCmr": [
            {
                "URL": "https://example.com/MYD13Q1.A2017297.h19v10.006.2017313221202.hdf",
                "Type": "GET DATA",
                "GranuleUR": "MOD09GQ.A4675287.SWPE5_.006.7310007729190"
            },
            {
                "URL": "https://cumulus-test-sandbox-public.s3.amazonaws.com/MOD09GQ___006/MOD/MOD09GQ.A8722843.GTk5A3.006.4026909316904_ndvi.jpg",
                "Type": "GET DATA",
                "GranuleUR": "MOD09GQ.A8722843.GTk5A3.006.4026909316904"
            }
        ]
    }
}
Granule Not Found Report
{
  "createStartTime": "2020-08-28T18:33:50.525Z",
  "createEndTime": "2020-08-28T18:34:16.646Z",
  "status": "SUCCESS",
  "reportType": "Granule Not Found",
  "filesInCumulus": {
    "okCount": 20,
    "okCountByGranule": {
      "MOD09GQ.A2770300.W1_V5Z.006.7853319756315": 2,
      "MOD09GQ.A7682091.QtZjhI.006.1218763030745": 3,
      "MOD09GQ.A9536857.bV1q4A.006.0980635705136": 3,
      "MOD09GQ.A7790080.xLPpTZ.006.2292724247258": 3,
      "MOD09GQ.A3769578.tlqq7j.006.4434181201872": 3,
      "MOD09GQ.A5423294.RFOrMI.006.8677601711674": 3,
      "MOD09GQ.A8837603.fb2Vhw.006.3950746589733": 3
    },
    "onlyInS3": [
      "s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593122280799/MOD09GQ.A6921412.znLBqe.006.9673856807617.hdf.met",
      "s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593122597517/MOD09GQ.A1201557.d6tP2Y.006.7482431709753.hdf.met",
      "s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593198955197/MOD09GQ.A7375038.gm1irM.006.1317280710645.hdf.met",
      "s3://cumulus-sandbox-protected/MOD09GQ___006/2017/MOD/test-data-1593122280799/MOD09GQ.A6921412.znLBqe.006.9673856807617.hdf",
      "s3://cumulus-sandbox-protected/MOD09GQ___006/2017/MOD/test-data-1593122597517/MOD09GQ.A1201557.d6tP2Y.006.7482431709753.hdf",
      "s3://cumulus-sandbox-protected/MOD09GQ___006/2017/MOD/test-data-1593122881446/MOD09GQ.A2708681.CFkGhW.006.7154000014360.hdf"
    ],
    "onlyInDynamoDb": [
      {
        "uri": "s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593178175944/MOD09GQ.A2770300.W1_V5Z.006.7853319756315.hdf.met",
        "granuleId": "MOD09GQ.A2770300.W1_V5Z.006.7853319756315"
      }
    ]
  },
  "collectionsInCumulusCmr": {
    "okCount": 0,
    "onlyInCumulus": [
      "MOD09GQ_test-test-data-1593122280799___006",
      "MOD09GQ_test-test-data-1593122597517___006",
      "MOD09GQ_test-test-data-1593122881446___006",
      "MOD09GQ_test-test-data-1593178175944___006",
      "MOD09GQ_test-test-data-1593198739991___006",
      "MOD09GQ_test-test-data-1593198955197___006",
      "MOD09GQ_test-test-data-1593199179316___006",
      "MOD09GQ_test-test-data-1593199379860___006",
      "MOD09GQ_test-test-data-1593199548454___006",
      "MOD09GQ_test-test-data-1593199742089___006",
      "MOD09GQ_test-test-data-1593199934835___006"
    ],
    "onlyInCmr": [
      "A2_RainOcn_NRT___0",
      "A2_SI12_NRT___0",
      "A2_SI25_NRT___0",
      "A2_SI6_NRT___0",
      "AST_L1A___003",
      "CMR44JK___001",
      "L2_HR_PIXC___000",
      "L2_HR_PIXC___1",
      "MCD43A1___006",
      "MOD09GQ___006",
      "MOD11A1___006",
      "MOD14A1___006",
      "MUR-JPL-L4-GLOB-v4.1___1",
      "MYD09A1___006",
      "MYD13A1___006",
      "MYD13Q1___006",
      "hs3avaps___1",
      "hs3wwlln___1"
    ]
  },
  "granulesInCumulusCmr": {
    "okCount": 0,
    "onlyInCumulus": [],
    "onlyInCmr": []
  },
  "filesInCumulusCmr": {
    "okCount": 0,
    "onlyInCumulus": [],
    "onlyInCmr": []
  }
}
Internal Report
{
  "reportType": "Internal",
  "createStartTime": "2020-08-28T19:01:50.636Z",
  "createEndTime": "2020-08-28T19:02:08.211Z",
  "reportStartTime": "2020-01-28T18:21:31.119Z",
  "reportEndTime": "2020-08-28T18:21:31.119Z",
  "status": "SUCCESS",
  "collections": {
    "okCount": 10,
    "withConflicts": [
      {
        "es": {
          "createdAt": 1591902043153,
          "granuleId": "^.*$",
          "sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5",
          "dataType": "L2_HR_PIXC",
          "name": "L2_HR_PIXC",
          "files": [
            {
              "bucket": "private",
              "regex": ".*.h5$",
              "sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5"
            }
          ],
          "granuleIdExtraction": "^(.*)(\\.h5|\\.cmr)",
          "reportToEms": true,
          "version": "000",
          "duplicateHandling": "error",
          "updatedAt": 1591902043153,
          "timestamp": 1591902043840
        },
        "db": {
          "duplicateHandling": "error",
          "granuleIdExtraction": "^(.*)(\\.h5|\\.cmr)",
          "version": "000",
          "dataType": "L2_HR_PIXC",
          "files": [
            {
              "bucket": "private",
              "sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5",
              "regex": ".*.h5$"
            }
          ],
          "updatedAt": 1591902043153,
          "createdAt": 1591902043153,
          "reportToEms": false,
          "granuleId": "^.*$",
          "sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5",
          "name": "L2_HR_PIXC"
        }
      }
    ],
    "onlyInEs": [
      "MCD43A1___006",
      "MOD09GQ___006"
    ],
    "onlyInDb": [
      "MYD13QA___005"
    ]
  },
  "granules": {
    "okCount": 77,
    "withConflicts": [
      {
        "es": {
          "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:IngestAndPublishGranule:edb21d63-b62d-42e9-8879-f39d53314f16",
          "processingEndDateTime": "2020-05-18T19:49:04.769Z",
          "published": false,
          "error": {},
          "productVolume": 253949,
          "timeToPreprocess": 0,
          "duration": 1.005,
          "createdAt": 1589831343764,
          "granuleId": "MOD14A1.A4313371.RQSeyH.006.7795385650931",
          "processingStartDateTime": "2020-05-18T19:49:03.831Z",
          "provider": "s3_provider",
          "timeToArchive": 0,
          "files": [
            {
              "bucket": "cumulus-test-sandbox-internal",
              "fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
              "size": 233840,
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
              "type": "data",
              "key": "test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf"
            },
            {
              "bucket": "cumulus-test-sandbox-internal",
              "fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
              "size": 14297,
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
              "type": "metadata",
              "key": "test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met"
            },
            {
              "bucket": "cumulus-test-sandbox-internal",
              "fileName": "BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
              "size": 5812,
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
              "type": "browse",
              "key": "test-data/files/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg"
            }
          ],
          "collectionId": "MOD14A1___006",
          "status": "running",
          "timestamp": 1589831367756,
          "updatedAt": 1589831344769
        },
        "db": {
          "published": true,
          "endingDateTime": "2017-11-08T23:59:59Z",
          "status": "completed",
          "timestamp": 1589831361039,
          "createdAt": 1589831343764,
          "processingEndDateTime": "2020-05-18T19:49:20.023Z",
          "productVolume": 258297,
          "timeToPreprocess": 0.263,
          "timeToArchive": 0.942,
          "productionDateTime": "2018-07-19T12:01:01Z",
          "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:IngestAndPublishGranule:edb21d63-b62d-42e9-8879-f39d53314f16",
          "files": [
            {
              "bucket": "cumulus-test-sandbox-protected",
              "key": "MOD14A1___006/2017/MOD/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
              "size": 233840,
              "fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
              "type": "data"
            },
            {
              "bucket": "cumulus-test-sandbox-private",
              "key": "MOD14A1___006/MOD/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
              "size": 14297,
              "fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
              "type": "metadata"
            },
            {
              "bucket": "cumulus-test-sandbox-public",
              "key": "MOD14A1___006/BRO/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
              "size": 5812,
              "fileName": "BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
              "type": "browse"
            },
            {
              "bucket": "cumulus-test-sandbox-protected-2",
              "key": "MOD14A1___006/MOD/MOD14A1.A4313371.RQSeyH.006.7795385650931.cmr.xml",
              "size": 4348,
              "fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.cmr.xml",
              "type": "metadata"
            }
          ],
          "cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1234771093-CUMULUS",
          "processingStartDateTime": "2020-05-18T19:49:03.831Z",
          "updatedAt": 1589831361039,
          "beginningDateTime": "2017-10-24T00:00:00Z",
          "provider": "s3_provider",
          "granuleId": "MOD14A1.A4313371.RQSeyH.006.7795385650931",
          "collectionId": "MOD14A1___006",
          "duration": 17.275,
          "error": {
            "Error": "Unknown Error",
            "Cause": "None"
          },
          "lastUpdateDateTime": "2018-04-25T21:45:45.524053"
        }
      }
    ],
    "onlyInEs": [
      {
        "granuleId": "MYD13Q1.A0323210.Yujnv6.006.1973526114346",
        "collectionId": "MYD13Q1___006",
        "provider": "s3_provider",
        "createdAt": 1589913461833,
        "updatedAt": 1589913522990
      }
    ],
    "onlyInDb": [
      {
        "granuleId": "MYD13Q1.A5822875.qHI8hK.006.8698738069249",
        "collectionId": "MYD13Q1___006",
        "provider": "s3_provider",
        "createdAt": 1598552797312,
        "updatedAt": 1598552827611
      }
    ]
  }
}

Create reconciliation report

Create a new reconciliation report.

POST
/reconciliationReports

Request body

parameter value required description
reportName string false Report name.
reportType "Inventory"|"Internal"|"GranuleNotFound" false Report type (default Inventory)
startTimestamp string false Any input valid for a JS Date contstructor. Data older than this will be ignored in the generated report.
endTimestamp string false Any input valid for a JS Date contstructor. Data newer than this will be ignored in the generated report.
collectionId [string array] false
provider string false provider name for comparison of granule holdings
granuleId string false granuleId for comparison of granule holdings

NOTE: Adding a startTimestamp or endTimestamp value to the POST request will result in one way comparisons for some fields.

Example request

$ curl --request POST https://example.com/reconciliationReports --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
  "reportName": "ModisInventoryReport",
  "reportType": "Inventory",
  "startTimestamp": 1269993600000,
  "endTimestamp": 1350000000000,
  "collectionId": "MOD09GQ___006",
  "provider": "MODIS",
  "granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"
}'

Example response

{
    "message": "Report is being generated",
    "status": 202
}

Delete reconciliation report

Delete a reconciliation report from Cumulus.

DELETE
/reconciliationReports/{name}

Example request

$ curl --request DELETE https://example.com/reconciliationReports/inventoryReport-20180620T205838883 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Report deleted"
}

Create EMS reports

Create EMS reports and send them to EMS. For more information on EMS reporting and types of reports see the Cumulus EMS Reporting documentation.

Overview of the request fields:

Field Required Value Description
reportType Y "metadata"|"ingest"|"distribution" type of report
startTime N string report startTime in format YYYY-MM-DD or YYYY-MM-DDTHH:mm:ss
If startTime and endTime are not specified, the reports for previous day will be generated
endTime N string report endTime in format YYYY-MM-DD or YYYY-MM-DDTHH:mm:ss
collectionId N string collectionId of the report records
This field can be specified when report type is metadata or ingest
POST
/ems

Example requests

$ curl https://example.com/ems --header 'Authorization: Bearer ReplaceWithTheToken' --data '{
    "reportType": "metadata"
}'
$ curl https://example.com/ems --header 'Authorization: Bearer ReplaceWithTheToken' --data '{
    "reportType": "ingest",
    "startTime": "2019-06-27",
    "endTime": "2019-06-29",
    "collectionId": "MOD14A1___006"
}'

Example response

{
    "message": "Reports are being generated",
    "status": 202
}

Instance Metadata

The Cumulus API can provide information about its configuration.

GET requests to the instance metadata endpoint return a json object with information about how the Cumulus stack is configured. It returns the CMR provider and environment as well as the stackName (prefix).

GET
/instanceMeta

Example Request

$ curl https://example.com/instanceMeta --header 'Authorization: Bearer ReplaceWithTheToken'

Example Response

{
  "cmr": {
    "provider": "CUMULUS",
    "environment": "UAT"
  },
  "cumulus": {
    "stackName": "cumulus-stack-prefix"
  }
}

Elasticsearch

These endpoints provide an interface into the Elasticsearch SDK to perform functions like Elasticsearch reindex.

Cumulus uses index aliases to allow reindexing with no downtime.

Reindex

The reindex command creates a new index and reindexes the source index to the new, destination index. This uses the Elasticsearch reindex functionality, but includes setting up the new index.

Note that there will now be two copies of your index. Your Cumulus instance will still point to the old copy until you perform the change-index function described below.

An alias should not be specified unless you have a specific alias configured. If a source index is not specified, it will default to the index from the alias. If you want to name the destination index something particular, you can specify a name, otherwise the destination index name will default to 'cumulus-year-month-day' with today's date (e.g. cumulus-4-12-2019). If you specify a destination index name, it must be an index that does not already exist in your cluster.

Since reindex can be a long running operation, the reindex-status endpoint must be queried to see if the operation has completed.

Example request

$ curl --request POST https://example.com/elasticsearch/reindex --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{ "message": "Reindexing from cumulus-4-4-2019 to cumulus-4-12-2019. Check the reindex-status endpoint for status." }

Example request

$ curl --request POST https://example.com/elasticsearch/reindex --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
  "aliasName": "cumulus-alias",
  "sourceIndex": "cumulus-4-4-2019",
  "destIndex": "cumulus-new-index"
}'

Example response

{ "message": "Reindexing from cumulus-4-4-2019 to cumulus-new-index. Check the reindex-status endpoint for status." }

Reindex Status

Reindexing can be a long running operation, so use this endpoint to get the status of your reindex and the status of your indices.

reindexStatus.nodes will be populated if there is a reindex operation currently running, nodes will be empty if not.

The elasticsearchStatus shows the number of documents in each index which can be used as a sanity check for the reindex operation before switching over to the new index. Note that the active index will continue to index logs, so the number of documents in the active index will be slightly higher and continue to grow.

Example request

$ curl https://example.com/elasticsearch/reindex-status --header 'Authorization: Bearer ReplaceWithTheToken'

Example responses

{
    "reindexStatus": {
        "nodes": {
            "TKDzeZ_TQVWkwKZYtCtv_g": {
                "name": "TKDzeZ_",
                "roles": [
                    "master",
                    "data",
                    "ingest"
                ],
                "tasks": {
                    "TKDzeZ_TQVWkwKZYtCtv_g:546691": {
                        "node": "TKDzeZ_TQVWkwKZYtCtv_g",
                        "id": 546691,
                        "type": "transport",
                        "action": "indices:data/write/reindex",
                        "start_time_in_millis": 1555073365557,
                        "running_time_in_nanos": 75424890,
                        "cancellable": true
                    }
                }
            }
        }
    },
    "indexStatus": {
        "_shards": {
            "total": 32,
            "successful": 16,
            "failed": 0
        },
        "_all": {
            "primaries": {
                "docs": {
                    "count": 52213,
                    "deleted": 0
                }
            },
            "total": {
                "docs": {
                    "count": 52213,
                    "deleted": 0
                }
            }
        },
        "indices": {
            ".kibana": {
                "primaries": {
                    "docs": {
                        "count": 1,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 1,
                        "deleted": 0
                    }
                }
            },
            "cumulus-2019-4-12": {
                "primaries": {
                    "docs": {
                        "count": 25569,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 25569,
                        "deleted": 0
                    }
                }
            },
            "cumulus-2019-4-9": {
                "primaries": {
                    "docs": {
                        "count": 25644,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 25644,
                        "deleted": 0
                    }
                }
            }
        }
    }
}
{
    "reindexStatus": {
        "nodes": {}
    },
    "indexStatus": {
        "_shards": {
            "total": 22,
            "successful": 11,
            "failed": 0
        },
        "_all": {
            "primaries": {
                "docs": {
                    "count": 51210,
                    "deleted": 0
                }
            },
            "total": {
                "docs": {
                    "count": 51210,
                    "deleted": 0
                }
            }
        },
        "indices": {
            ".kibana": {
                "primaries": {
                    "docs": {
                        "count": 1,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 1,
                        "deleted": 0
                    }
                }
            },
            "cumulus-2019-4-12": {
                "primaries": {
                    "docs": {
                        "count": 25569,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 25569,
                        "deleted": 0
                    }
                }
            },
            "cumulus-2019-4-9": {
                "primaries": {
                    "docs": {
                        "count": 25640,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 25640,
                        "deleted": 0
                    }
                }
            }
        }
    }
}

Change Index

Change index switches the Elasticsearch index to point to the new index, rather than the current index. You may choose to delete your current index during this operation using the deleteSource parameter, which defaults to false. If you are using the default index created by Cumulus, this will switch your Cumulus instance to the new index and you will see those changes immediately via the API and dashboard.

currentIndex and newIndex are required parameters.

In the context of reindex, you'd call change-index, following reindex completion to start using the new index with no downtime.

Example request

$ curl --request POST https://example.com/dev/elasticsearch/change-index --header 'Content-Type: application/json' --header 'Authorization: Bearer ReplaceWithTheToken' --data '{
  "aliasName": "cumulus-alias",
  "currentIndex": "cumulus-12-4-2019",
  "newIndex": "cumulus-4-12-2019",
  "deleteSource": false
}'

Example response

{ "message": "Reindex success - alias cumulus now pointing to cumulus-4-12-2019" }

Index from Database

In case of corruption of your Elasticsearch index, you can reindex your data from the database. This will include collections, executions, granules, pdrs, providers, and rules. Logs will not be indexed since they are not stored in the database.

You can specify an index (should be empty) or if no index name is specified, a default with the format of 'cumulus-year-month-day' with today's date (e.g. cumulus-4-12-2019) will be used. If the specified index does not exist, it will be created.

It is recommended that workflow rules be turned off, as any data ingested into the database during this operation cannot be guaranteed to make it into the new index. Following the completion of the index, you will need to use the change index operation to switch your Elasticsearch to point to the new instance.

Indexing is an async operation, so an operation id will be returned. You can query the status using the asyncoperations GET endoint with the operation id.

Example request

$ curl --request POST https://example.com/elasticsearch/index-from-database --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
  "indexName": "recovery-index"
}'

Example response

{"message":"Indexing database to recovery-index. Operation id: 8f4d35ba-c858-40ae-93f0-4465f54a911d"}

Example request - operation status

$ curl --header 'Authorization: Bearer ReplaceWithTheToken' https://example.com/asyncoperations/8f4d35ba-c858-40ae-93f0-4465f54a911d

Example response - operation status

{
   "id":"8f4d35ba-c858-40ae-93f0-4465f54a911d",
   "status":"SUCCEEDED",
   "taskArn":"arn:aws:ecs:us-east-1:XXXXXXXXXXXX:task/dc41ff2b-e610-4f7a-8c63-48b48ceb7160",
   "output":"\"Index from database complete\""
}

Indices Status

Use the indices-status endpoint to view information about your elasticsearch indices. This endpoint will return the information for the indices call in the Elasticsearch API.

Example request

$ curl  https://example.com/elasticsearch/indices-status --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

yellow open .kibana           yxBUTswHSr2ecec8y1szEg 1 1      1   0   3.1kb   3.1kb
yellow open cumulus-2019-6-21 ELpdxYVKSc-VvNxlNjmMUA 5 1    700 225   2.5mb   2.5mb
yellow open cumulus-2019-6-24 lmylLkTsQJWIVYFJcftMqQ 5 1    700   0     4mb     4mb
yellow open cumulus           jQLXsb8yS4aEmLAAyHcmCw 5 1 217298 588 124.3mb 124.3mb

Current Index

Get the current aliased index being used by the Cumulus Elasticsearch instance.

Example request

$ curl  https://example.com/elasticsearch/current-index --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

["cumulus-2019-6-24"]

Serve the dashboard from a bucket

Serve the dashboard from an S3 bucket.

This is a way to serve the Cumulus dashboard in a browser from an S3 bucket without making the bucket or files public.

To use this:

  • Your dashboard bucket must be in the bucket definitions in your Cumulus terraform.tfvars, otherwise you will not be able to access the bucket.
  • Deploy the dashboard to your bucket using the instructions in the Cumulus dashboard README.
  • In a browser, use the example request below, but replace example.com with your Cumulus backend API and dashboard-bucket with your bucket name.
https://example.com/dashboard/dashboard-bucket/index.html
GET
/dashboard/{bucket}/{key}

Example request

Show examples in:
Cumulus API