The Cumulus API allows developers to interact with the Cumulus Framework, such as monitoring status or creating, editing, and deleting records. This is the same API that powers the Cumulus dashboard.
By utilizing this API, a developer can integrate with the Cumulus framework in any language or environment; although interacting with Cumulus through the Cumulus dashboard may be appropriate for many end users, for some use cases it's best to have the flexibility of a web-accessible API.
The API accepts and responds with JSON payloads at various HTTPS endpoints.
In order to use these endpoints, you must include authentication information in your HTTPS request; authentication is explained in the following section.
The following table lists the query string parameters that can be used with most of the Cumulus API endpoints. {fieldName} is a stand-in for any of the fields in the record, and for nested objects dot notation can be used; for example, valid fieldNames include: pdrName, status, and recipe.processStep.description.
| query string parameter | description |
|---|---|
limit={number} |
Number of records to be returned by the API call; default is 10. A value of null will return all records. |
page={number} |
page number, 1-indexed; default is 1 |
sort_by={fieldName} |
which field to sort by; default is timestamp |
order={asc|desc} |
whether to sort in asc or desc order |
sort_key[]={-fieldName1}&sort_key[]={fieldName2} |
One or more sort keys can be specified using the sort_key[] parameter. The order used impacts searching. Fields can be prepended with a - to sort in descending order or a + to sort in ascending. Ascending order is the default. The + must be escaped with %2B |
prefix={value} |
startsWith search of the Providers by name, Collections by name, Granules by granuleId, PDRs by pdrName, Rules by name, Executions by arn, Async Operations by id, Reconciliation Reports by name |
infix={value} |
includes search of the Providers by name, Collections by name, Granules by granuleId, PDRs by pdrName, Rules by name, Executions by arn, Async Operations by id, Reconciliation Reports by name |
fields={fieldName1, fieldName2} |
which fields to return, separated by a comma |
{fieldName}={value} |
exact value match for the given field |
{fieldName}__from={number} |
for numeric fields, field value must be greater than the given number |
{fieldName}__to={number} |
for numeric fields, field value must be less than the given number |
{fieldName}__not={value} |
field does not match the given value |
{fieldName}__in={value1, value2} |
field matches one of the values in the comma-separated list |
{fieldName}__exists={true|false} |
field exists or doesn't exist in the record |
q="fieldName:[1 TO 2] AND fieldName2:[3 TO 4]" |
arbitrary Apache Lucene query syntax, not needed for most uses of the API; if the q parameter is used, all other query parameters will be ignored, besides limit, page, and fields |
The Cumulus API is versioned and the current version is v2. Retrieve the latest API version from Cumulus.
To use any version, include the version number in the path before the query endpoint. {API_URL}/{version}/{query}
$ curl https://example.com/version
{
"response_version": "v2",
"api_version": "15.0.0"
}
Returns a bearer token using oAuth with Earthdata Login service. The token will be returned as a JWT (JSON Web Token).
| query string parameter | description |
|---|---|
state={string} |
The URI to redirect to after if oAuth was successful |
$ curl https://example.com/token
{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDY1MDk3ODczIn0.SxFtZ7dqp9KsUSn1uTXhWis8Il8Hig8mwLANGU3cXhY"}
Refreshes a bearer token received from oAuth with Earthdata Login service. The token will be returned as a JWT (JSON Web Token).
| parameter | type | required | description |
|---|---|---|---|
token |
string | true |
The JWT received from the /token endpoint to refresh |
$ curl --request POST https://example.com/refresh --header 'Content-Type: application/json' --data '{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDY1MDk3ODczIn0.SxFtZ7dqp9KsUSn1uTXhWis8Il8Hig8mwLANGU3cXhY"}'
{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDcxMjk4ODEzIn0.vO6RlSRo47kkH15_muoUYNvv74fRzFxs7FlmVaarHlc"}
Delete the record for an access token received from oAuth with Earthdata Login service.
token is the JWT containing access token information to delete
$ curl --request DELETE https://example.com/tokenDelete/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDY1MDk3ODczIn0.SxFtZ7dqp9KsUSn1uTXhWis8Il8Hig8mwLANGU3cXhY
{"message": "Token record was deleted"}
When a request is made to the Cumulus API, it must contain a bearer token generated by the API.
The token is generated after the user login with Earthdata Login service.
The token is included in requests using the Authorization header.
If no token is provided, the Cumulus API server will respond with an error, requesting credentials. If an incorrect token is provided, the server will respond with a separate error noting this.
#! /bin/sh
ORIGIN=$(dirname $CUMULUS_BASEURL)
LOGIN_URL="$CUMULUS_BASEURL/token"
# create a base64 hash of your login credentials
AUTH=$(printf "$EARTHDATA_USERNAME:$EARTHDATA_PASSWORD" | base64)
# Request the Earthdata url with client id and redirect uri to use with Cumulus
AUTHORIZE_URL=$(curl -s -i ${LOGIN_URL} | grep location | sed -e "s/^location: //");
# Request an authorization grant code
TOKEN_URL=$(curl -s -i -X POST \
-F "credentials=${AUTH}" \
-H "Origin: ${ORIGIN}" \
${AUTHORIZE_URL%$'\r'} | grep Location | sed -e "s/^Location: //")
# Request the token through the CUMULUS API url that's returned from Earthdata
# Response is a JSON object of the form { token: String }
# This uses the cli tool jq to parse the JSON and get the token string
# More info on jq: https://stedolan.github.io/jq/
TOKEN=$(curl -s ${TOKEN_URL%$'\r'} | jq -r '.message.token')
echo $TOKEN
List providers in the Cumulus system.
$ curl https://example.com/providers --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "daac-cumulus",
"table": "providers",
"limit": 1,
"page": 1,
"count": 2
},
"results": [
{
"id": "HTTP_MODIS",
"globalConnectionLimit": 10,
"maxDownloadTime": 300,
"protocol": "http",
"host": "https://data.modis.gov/",
"timestamp": 1508861082226
}
]
}
Retrieve a single provider.
$ curl https://example.com/providers/HTTP_MODIS --header 'Authorization: Bearer ReplaceWithTheToken'
{
"createdAt": 1508861081785,
"id": "HTTP_MODIS",
"host": "https://data.modis.gov/",
"globalConnectionLimit": 10,
"maxDownloadTime": 300,
"updatedAt": 1508861081785,
"protocol": "http"
}
Create a provider. For more information on creating providers and the contents of a request see the Cumulus setup documentation.
Overview of the schema fields:
| Field | Value | Description |
|---|---|---|
id |
string |
provider id/name |
protocol |
"s3"|"http"|"https"|"ftp" |
file transfer (sync) protocol |
host |
string |
provider host endpoint |
port |
number |
provider host port |
globalConnectionLimit |
number |
limit to number of concurrent connections. This is the maximum number of connections Cumulus compatible ingest lambdas are expected to make to a provider. Defaults to unlimited |
maxDownloadTime |
number |
Maximum download time in seconds for all granule files on a sync granule task. The timeout is used together with globalConnectionLimit to limit concurrent downloads. |
username |
string |
provider connection username |
password |
string |
provider connection password |
privateKey |
string |
filename assumed to be in s3://bucketInternal/stackName/crypto |
cmKeyId |
string |
AWS KMS Customer Master Key arn or alias |
allowedRedirects |
Array<string> |
Only hosts in this list will have the provider username/password forwarded for authentication. Entries should be specified as host.com or host.com:7000 if redirect port is different than the provider port. |
certificateUri |
string |
Optional SSL Certificate S3 URI for custom or self-signed SSL (TLS) certificate |
$ curl --request POST https://example.com/providers --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
"host": "https://www.example.gov",
"id": "MY_DAAC_SATELLITE",
"protocol": "http",
"globalConnectionLimit": 10,
"maxDownloadTime": 300
}'
{
"message": "Record saved",
"record": {
"createdAt": 1491941727851,
"host": "https://www.example.gov",
"id": "MY_DAAC_SATELLITE",
"protocol": "http",
"globalConnectionLimit": 10,
"maxDownloadTime": 300,
"timestamp": 1513956151186
}
}
Update/replace an existing provider. Expects payload to specify the entire provider object, and will completely replace the existing provider with the specified payload. For a field reference see "Create provider".
Returns status 200 on successful replacement, 400 if the id property in the
payload does not match the corresponding value in the resource URI, or 404 if
there is no provider with the specified ID.
$ curl --request PUT https://example.com/providers/MY_DAAC_SATELLITE --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
"id": "MY_DAAC_SATELLITE",
"host": "https://www.example.co.uk",
"globalConnectionLimit": 10,
"maxDownloadTime": 300,
"protocol": "http"
}'
{
"createdAt": 1491941727851,
"id": "MY_DAAC_SATELLITE",
"host": "https://www.example.co.uk",
"globalConnectionLimit": 10,
"maxDownloadTime": 300,
"updatedAt": 1513956150733,
"protocol": "http",
"timestamp": 1513956555713
}
Delete a provider from Cumulus. The related PDRs and granules remain in the Cumulus and CMR systems.
$ curl --request DELETE https://example.com/providers/MY_DAAC_SATELLITE --header 'Authorization: Bearer ReplaceWithTheToken'
{
"message": "Record deleted"
}
List collections in the Cumulus system.
If the query includes a value of true for getMMT, cumulus will search the CMR for each collection and insert the link to the Metadata Management Tool (MMT) into each result under MMTLink
If the query includes a value of true for includeStats, cumulus will compute and include the granule aggregation statistics in each collection's return value.
$ curl https://example.com/collections --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "collections",
"limit": 1,
"page": 1,
"count": 3
},
"results": [
{
"name": "MOD11A1",
"version": "006",
"dataType": "MOD11A1",
"process": "modis",
"provider_path": "/",
"granuleId": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}$",
"granuleIdExtraction": "(MOD11A1\\..*)\\.hdf",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
"files": [
{
"bucket": "protected",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
},
{
"bucket": "private",
"regex": "^BROWSE\\.MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
"sampleFileName": "BROWSE.MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
},
{
"bucket": "private",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf\\.met$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf.met"
},
{
"bucket": "protected",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.cmr\\.xml$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.cmr.xml"
},
{
"bucket": "public",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_2\\.jpg$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_2.jpg"
},
{
"bucket": "public",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_1\\.jpg$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_1.jpg"
}
],
"timestamp": 1513020427284,
"createdAt": 1510761441174,
"updatedAt": 1513020427162,
"edpa": true,
"some_other_field": "field",
"duplicateHandling": "skip",
}
]
}
List collections in the Cumulus system that have active associated granules.
If time parameters are specified, the query will return collections that have granules that have been updated in that time frame.
If the query includes a value of true for getMMT, cumulus will search the CMR for each collection and insert the link to the Metadata Management Tool (MMT) into each result under MMTLink
If the query includes a value of true for includeStats, cumulus will compute and include the granule aggregation statistics in each collection's return value.
$ curl https://example.com/collections/active?includeStats=true --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "collections",
"limit": 1,
"page": 1,
"count": 3
},
"results": [
{
"name": "MOD11A1",
"version": "006",
"dataType": "MOD11A1",
"process": "modis",
"provider_path": "/",
"granuleId": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}$",
"granuleIdExtraction": "(MOD11A1\\..*)\\.hdf",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
"files": [
{
"bucket": "protected",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
},
{
"bucket": "private",
"regex": "^BROWSE\\.MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
"sampleFileName": "BROWSE.MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
},
{
"bucket": "private",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf\\.met$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf.met"
},
{
"bucket": "protected",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.cmr\\.xml$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.cmr.xml"
},
{
"bucket": "public",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_2\\.jpg$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_2.jpg"
},
{
"bucket": "public",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_1\\.jpg$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_1.jpg"
}
],
"timestamp": 1513020427284,
"createdAt": 1510761441174,
"updatedAt": 1513020427162,
"edpa": true,
"some_other_field": "field",
"duplicateHandling": "skip",
"stats": {
"running": 0,
"completed": 6,
"failed": 1,
"total": 7
}
}
]
}
Retrieve a single collection.
$ curl https://example.com/collections/MOD11A1/006 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"process": "modis",
"granuleIdExtraction": "(MOD11A1\\..*)\\.hdf",
"version": "006",
"dataType": "MOD11A1",
"some_other_field": "field",
"createdAt": 1510761441174,
"edpa": true,
"name": "MOD11A1",
"duplicateHandling": "skip",
"provider_path": "/",
"files": [
{
"bucket": "protected",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$"
},
{
"bucket": "private",
"sampleFileName": "BROWSE.MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
"regex": "^BROWSE\\.MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$"
},
{
"bucket": "private",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf.met",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf\\.met$"
},
{
"bucket": "protected",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.cmr.xml",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.cmr\\.xml$"
},
{
"bucket": "public",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_2.jpg",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_2\\.jpg$"
},
{
"bucket": "public",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_1.jpg",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_1\\.jpg$"
}
],
"updatedAt": 1513020427162,
"granuleId": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
}
Create a collection. For more information on creating collections and the contents of a request see the Cumulus setup documentation. A collection generally includes, but is not limited to, the fields listed below.
Overview of the schema fields:
| Field | Value | Description |
|---|---|---|
name |
string |
collection name |
version |
string |
collection version |
dataType |
string |
matches collection with PDR datatype |
duplicateHandling |
"replace"|"version"|"skip"|"error" |
duplicate handling protocol |
process |
string |
process choice step variable |
provider_path |
string |
path of remote files to sync |
granuleId |
string (regex) |
regex to match granule IDs |
granuleIdExtraction |
string (regex) |
regex to extract ID from files |
sampleFileName |
string |
sample filename for granule ID |
url_path |
string |
s3 prefix template |
files |
array |
array of file specification objects |
-- file.bucket |
string |
file destination bucket |
-- file.regex |
string (regex) |
regex to match file names |
-- file.sampleFileName |
string |
sample filename |
-- file.url_path |
string |
s3 prefix template |
-- file.fileType |
string |
granule file fileType mapping |
$ curl --request POST https://example.com/collections --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
"name": "MOD09GQ",
"version": "006",
"dataType": "MOD09GQ",
"process": "modis",
"duplicateHandling": "replace",
"provider_path": "cumulus-test-data/pdrs",
"granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
"granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"files": [
{
"bucket": "protected",
"regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
}
]
}'
{
"message": "Record saved",
"record": {
"name": "MOD09GQ",
"version": "006",
"dataType": "MOD09GQ",
"duplicateHandling": "replace",
"process": "modis",
"provider_path": "cumulus-test-data/pdrs",
"granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
"granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"files": [
{
"bucket": "protected",
"regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
}
],
"createdAt": 1491946535919
}
}
Update/replace an existing collection. Expects payload to specify the entire collection object, and will completely replace the existing collection with the specified payload. For a field reference see "Create collection".
Returns status 200 on successful replacement, 400 if the name or version
property in the payload does not match the corresponding value in the resource
URI, or 404 if there is no collection with the specified name and version.
$ curl --request PUT https://example.com/collections/MOD09GQ/006 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
"name": "MOD09GQ",
"version": "006",
"dataType": "MOD09GQ",
"duplicateHandling": "error",
"newNeededField": "myCustomFieldValue",
"process": "modis",
"provider_path": "new_path/test-data",
"granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
"granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"files": [
{
"bucket": "protected",
"regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
}
]
}'
{
"name": "MOD09GQ",
"version": "006",
"dataType": "MOD09GQ",
"duplicateHandling": "error",
"newNeededField": "myCustomFieldValue",
"process": "modis",
"provider_path": "new_path/test-data",
"granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
"granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"files": [
{
"bucket": "protected",
"regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
}
],
"createdAt": 1491946535919,
"updatedAt": 1514304825894
}
Delete a collection from Cumulus, but not from CMR. All related granules in Cumulus must have already been deleted from Cumulus.
$ curl --request DELETE https://example.com/collections/MOD09GQ/006 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"message": "Record deleted"
}
List granules in the Cumulus system.
If the query includes a value of true for getRecoveryStatus, recoveryStatus will be included in each granule's return value when applicable.
If the query string parameters include a value of true for includeFullRecord, any associated files and executions will be included in each granule's return value. The default value is false.
For requests without filters, if the query string parameters include a value of false for estimateTableRowCount, the returned count will be the actual exact count, otherwise count will be an estimated count. The default value is true.
$ curl https://example.com/granules?includeFullRecord=true --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "granules",
"limit": 1,
"page": 1,
"count": 8,
},
"results": [
{
"granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
"pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
"collectionId": "MOD11A1___006",
"status": "completed",
"provider": "LP_TS2_DataPool",
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:7f071dae1a93c9892272b7fd5",
"files": [
{
"bucket": "cumulus-devseed-protected",
"checksum": 964704694,
"key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
"fileSize": 1447347,
"fileType": "data",
"checksumType": "CKSUM",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
},
{
"bucket": "cumulus-devseed-private",
"checksum": 121318124,
"key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met",
"fileSize": 22559,
"fileType": "metadata",
"checksumType": "CKSUM",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met"
},
{
"bucket": "cumulus-devseed-private",
"checksum": 2188150664,
"key": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
"fileSize": 18118,
"fileType": "data",
"checksumType": "CKSUM",
"fileName": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
},
{
"bucket": "cumulus-devseed-public",
"key": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg",
"fileType": "browse",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg"
},
{
"bucket": "cumulus-devseed-protected",
"key": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml",
"fileType": "metadata",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml"
},
{
"bucket": "cumulus-devseed-public",
"key": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg",
"fileType": "browse",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg"
}
],
"error": null,
"createdAt": 1513020455831,
"timestamp": 1513020462156,
"published": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
"duration": 6.325,
"cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS"
}
]
}
Retrieve a single granule. Two routes are currently available. The preferred query includes both a Collection ID and a Granule ID to identify a unique granule.
Please note -- Querying by the Granule ID alone (e.g. GET /granules/{granuleId}) is supported but may be deprecated in the future.
If the query includes a value of true for getRecoveryStatus, the returned granule will include recoveryStatus when applicable.
$ curl https://example.com/granules/MOD11A1.A2017137.h20v17.006.2017138085755 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
"pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
"collectionId": "MOD11A1___006",
"status": "completed",
"provider": "LP_TS2_DataPool",
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:7f071dae1a93c9892272b7fd5",
"files": [
{
"bucket": "cumulus-devseed-protected",
"checksum": 964704694,
"key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
"fileSize": 1447347,
"checksumType": "CKSUM",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
},
{
"bucket": "cumulus-devseed-private",
"checksum": 121318124,
"key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met",
"fileSize": 22559,
"checksumType": "CKSUM",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met"
},
{
"bucket": "cumulus-devseed-private",
"checksum": 2188150664,
"key": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
"fileSize": 18118,
"checksumType": "CKSUM",
"fileName": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
},
{
"bucket": "cumulus-devseed-protected",
"key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
},
{
"bucket": "cumulus-devseed-public",
"key": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg"
},
{
"bucket": "cumulus-devseed-protected",
"key": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml"
},
{
"bucket": "cumulus-devseed-public",
"key": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg"
}
],
"error": null,
"createdAt": 1513020455831,
"timestamp": 1513020462156,
"published": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
"duration": 6.325,
"cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
"_id": "MOD11A1.A2017137.h20v17.006.2017138085755"
}
Create a granule. A granule can have the following fields.
| Field | Type | Required | Description |
|---|---|---|---|
beginningDateTime |
string |
no |
The time when the granule's temporal coverage begins |
cmrLink |
string |
no |
link to CMR |
collectionId |
string |
yes |
Collection associated with the granule |
createdAt |
integer |
no |
Time granule record was created (now) |
duration |
number |
no |
Ingest duration milliseconds |
endingDateTime |
string |
no |
The time when the granule's temporal coverage ends |
error |
object |
no |
The error details for this granule |
execution |
string |
no |
Step Function Execution URL |
files |
array |
no |
Files associated with the granule |
granuleId |
string |
yes |
Granule ID |
lastUpdateDateTime |
string |
no |
The date/time that data provider last updated the granule info on data provider's database |
pdrName |
string |
no |
PDR associated with the granule |
processingEndDateTime |
string |
no |
Time processing of granule ends. Usually a StepFunction's stop time |
processingStartDateTime |
string |
no |
Time processing of granule began. Usually a StepFunction's start time |
productVolume |
number |
no |
Sum of the granule's file's sizes in bytes |
productionDateTime |
string |
no |
The date and time a specific granule was produced by a PGE |
provider |
string |
no |
Granule's provider |
published |
boolean |
no |
If granule is published to CMR |
queryFields |
object |
no |
Arbitrary query fields assigned to the granule |
status |
string |
yes |
Ingest status of the granule one of ['running', 'completed', 'failed'] |
timeToArchive |
number |
no |
Time to post to CMR in seconds |
timeToPreprocess |
number |
no |
Time to sync granule in seconds |
timestamp |
integer |
no |
Timestamp for granule record (now) |
updatedAt |
integer |
no |
Update Time for granule (now) |
$ curl --request POST https://example.com/granules \
--header 'Authorization: Bearer ReplaceWithToken' \
--header 'Content-Type: application/json' \
--data '{
"granuleId": "granuleId.A20200113.006.1005",
"collectionId": "alpha___006",
"status": "running",
"beginningDateTime": "1995-01-01T00:00:00.000Z",
"cmrLink": "https://mmt.uat.earthdata.nasa.gov/collections/C1237256734-CUMULUS",
"createdAt": 1631547286190,
"duration": 60000,
"endingDateTime": "2021-09-13T00:00:00.000Z",
"error": {},
"execution": "https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestStepFunction2:cff1266e-ef36-664f-ff00-3a4d26bd1735",
"files": [
{
"bucket": "stack-protected",
"key": "granuleId.A20200113.006.1005.hdf",
"fileName": "granuleId.A20200113.006.1005.hdf"
}
],
"lastUpdateDateTime":"2021-09-12T15:10:01.000Z",
"pdrName": "aPdrName",
"processingEndDateTime": "2020-10-13T23:59:59.999Z",
"processingStartDateTime": "2020-10-13T00:00:00.000Z",
"productVolume": 59632353,
"productionDateTime": "2020-10-14T15:40:05.546Z",
"provider": "s3-local",
"published": true,
"queryFields": {"custom": "values can go here"},
"timeToArchive": 5001,
"timeToPreprocess": 3240,
"timestamp":1631547675248,
"updatedAt":1631547675248
}'
{
"message": "Successfully wrote granule with Granule Id: granuleId.A20200113.006.1005"
}
Replace an existing granule. Expects payload to contain the modified parts of the granule and the existing granule values will be overwritten by the modified portions. Any unspecified values will be removed, and appropriate fields will be replaced with defaults. Executions associated will not be modified if not specified. The same fields are available as are for creating a granule..
Returns status 200 on successful update, 201 on new granule creation, 404 if
the granuleId can not be found in the database, or 400 when the granuleId in
the payload does not match the corresponding value in the resource URI.
Please note -- In versions of CUMULUS prior to release v16, PUT endpoint was identical to PATCH requests.
$ curl --request PUT https://example.com/granules/COLLECTION___VERSION/granuleId.A19990103.006.1000 \
--header 'Authorization: Bearer ReplaceWithToken' \
--header 'Content-Type: application/json' \
--header 'Cumulus-API-Version: 2'\ --data '{
"granuleId": "granuleId.A20200113.006.1005",
"files": [
{
"bucket": "stack-protected",
"key": "granuleId.A20200113.006.1005.hdf",
"fileName": "granuleId.A20200113.006.1005.hdf"
},
{
"bucket": "stack-protected",
"key": "granuleId.A20200113.006.1005.jpg",
"fileName": "granuleId.A20200113.006.1005.jpg"
}
],
"duration": 1000,
"status": "completed"
}'
{
"message": "Successfully updated granule with Granule Id: granuleId.A20200113.006.1005, CollectionId: COLLECTION___VERSION"
}
Update an existing granule. Expects payload to contain the modified
parts of the granule as the existing granule values will be overwritten by the
modified portions. Unspecified keys will be retained. Keys set to null
will be removed. Executions will not be disassociated from the granule via
null deletion. The same fields are available as are for creating a
granule..
Returns status 200 on successful update, 201 on new granule creation, 404 if
the granuleId can not be found in the database, or 400 when the granuleId in
the payload does not match the corresponding value in the resource URI.
Please note -- Querying by the Granule ID alone (e.g. PATCH /granules/{granuleId}) is supported but may be deprecated in the future.
$ curl --request PATCH https://example.com/granules/granuleId.A19990103.006.1000 \
--header 'Authorization: Bearer ReplaceWithToken' \
--header 'Content-Type: application/json' \
--header 'Cumulus-API-Version: 2' \
--data '{
"granuleId": "granuleId.A20200113.006.1005",
"files": [
{
"bucket": "stack-protected",
"key": "granuleId.A20200113.006.1005.hdf",
"fileName": "granuleId.A20200113.006.1005.hdf"
},
{
"bucket": "stack-protected",
"key": "granuleId.A20200113.006.1005.jpg",
"fileName": "granuleId.A20200113.006.1005.jpg"
}
],
"duration": 1000,
"status": "completed"
}'
{
"message": "Successfully updated granule with Granule Id: granuleId.A20200113.006.1005"
}
Associate an execution with a granule. Returns status 200 on successful association, 404 if the granule or execution can not be found in the database, or 400 when the granuleId in the payload does not match the corresponding value in the resource URI.
The request should have the following fields.
| Field | Type | Description |
|---|---|---|
collectionId |
string |
Collection associated with the granule (e.g. <name>___<version> where <name> is the collection name and <version> is the collection version) |
granuleId |
string |
Granule ID |
executionArn |
string |
Execution arn |
$ curl --request POST https://example.com/granules/granuleId.A19990103.006.1000/executions \
--header 'Authorization: Bearer ReplaceWithToken' \
--header 'Content-Type: application/json' \
--header 'Cumulus-API-Version: 2'\
--data '{
"granuleId": "granuleId.A19990103.006.1000",
"collectionId": "MOD09GQ___006",
"executionArn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735"
}'
{
"message": "Successfully associated execution arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 with granule granuleId granuleId.A19990103.006.1000 collectionId MOD09GQ___006"
}
Reingest a granule. This causes the granule to re-download to Cumulus from source, and begin processing from scratch. Reingesting a granule will overwrite existing granule files.
You trigger the reingest by posting with the data's action set to reingest.
A reingest request may also include one optional parameters, either
executionArn or workflowName. Use of one of these will cause the reingest
occur, but before running the ingest, a different original payload is found
from existing executions and used in place of the most recent execution. This
has the result of allowing you to reingest a granule with any of it's previous
inputs. If executionArn is specified, the originalMessage is pulled directly
from that execution. When workflowName is specified, the database is search
to find all of the executions with that workflowName that were run on the
granuleid, and the most recent originalMessage is pulled and used for the
reingest. Remember only to supply either executionArn or workflowName, if
both are present, workflowName is ignored and executionArn is used to determining
the input message to the reingest.
A warning message is included in the response if the collection's duplicateHandling is not set to 'replace'.
$ curl --request PATCH https://example.com/granules/MOD11A1.A2017137.h20v17.006.2017138085755
--header 'Authorization: Bearer ReplaceWithTheToken'
--header 'Content-Type: application/json'
--header 'Cumulus-API-Version: 2'\
--data '{"action": "reingest",
["executionArn": "arn:aws:states:us-east-1:123456789012:execution:stack-lambdaName:9da47a3b-4d85-4599-ae78-dbec2e042520"],
["workflowName": "TheWorkflowName"] }'
{
"granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
"action": "reingest",
"status": "SUCCESS",
"warning": "The granule files may be overwritten"
}
Apply the named workflow to the granule. Workflow input will be built from template and provided entire Cumulus granule record as payload.
$ curl --request PATCH https://example.com/granules/MOD11A1.A2017137.h19v16.006.2017138085750 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --header 'Cumulus-API-Version: 2'\ --data '{ "action": "applyWorkflow", "workflow": "inPlaceWorkflow" }'
{
"granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
"action": "applyWorkflow inPlaceWorkflow",
"status": "SUCCESS"
}
Move a granule from one location on S3 to another. Individual files are moved to specific locations by using a regex that matches their filenames.
$ curl --request PATCH https://example.com/granules/MOD11A1.A2017137.h19v16.006.2017138085750 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --header 'Cumulus-API-Version: 2'\
--data '{ "action": "move", "destinations": [{ "regex": ".*.hdf$", "bucket": "s3-bucket", "filepath": "new/filepath/" }]}'
{
"granuleId": "MOD11A1.A2017137.h19v16.006.2017138085750",
"action": "move",
"status": "SUCCESS"
}
Remove a Cumulus granule from CMR.
$ curl --request PATCH https://example.com/granules/MOD11A1.A2017137.h19v16.006.2017138085750 --header 'Cumulus-API-Version: 2'\
--header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --header 'Cumulus-API-Version: 2'\ --data '{"action": "removeFromCmr"}'
{
"granuleId": "MOD11A1.A2017137.h19v16.006.2017138085750",
"action": "removeFromCmr",
"status": "SUCCESS"
}
Delete a granule from Cumulus. It must already be removed from CMR.
Please note -- Querying by the Granule ID alone (e.g. DELETE /granules/{granuleId}) is supported but may be deprecated in the future.
$ curl --request DELETE https://example.com/granules/1A0000-2016121001_002_001 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"detail": "Record deleted"
}
Apply a workflow to the granules provided. Granules can be sent as a list of IDs or as an Elasticsearch query to the Metrics' Elasticsearch.
Overview of the request fields:
| Field | Required | Value | Description |
|---|---|---|---|
workflow |
Y |
string |
Workflow to be applied to all granules |
granules |
yes - if query not present |
Array<object> |
List of Granules to process e.g. { granuleId, collectionId }. Required if there is no Elasticsearch query provided |
query |
yes - if granules not present |
Object |
Query to Elasticsearch to determine which Granules to go through given workflow. Required if no Granules are given. |
index |
yes - if query is present |
string |
Elasticsearch index to search with the given query |
knexDebug |
N |
bool |
Sets knex PostgreSQL connection pool/query debug output. Defaults to false |
queueUrl |
N |
string |
URL of SQS queue to use for scheduling granule workflows (e.g. https://sqs.us-east-1.amazonaws.com/12345/queue-name) |
Use the Retrieve async operation endpoint with the id in the response to determine the status of the async operation.
$ curl --request POST \
https://example.com/granules/bulk --header 'Authorization: Bearer ReplaceWithTheToken' \
--header 'Content-Type: application/json' \
--data '{
"workflowName": "HelloWorldWorkflow",
"index": "index-in-es",
"query": {
"size": 500,
"query": {
"filter": [
{
"bool": {
"filter": [
{
"bool": {
"should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [{"match": {"status": "FAILED"}}],
"minimum_should_match": 1
}
}
]
}
}
],
"should": [],
"must_not": []
}
}
}'
curl -X POST
https://example.com/granules/bulk --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"granules": [ { "granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606", "collectionId": "MOD11A1___006" } ], "workflowName": "HelloWorldWorkflow"}'
{
"id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e"
}
Bulk delete the provided granules. Granules can be sent as a list of IDs or as an Elasticsearch query to the Metrics' Elasticsearch.
Overview of the request fields:
| Field | Required | Value | Description |
|---|---|---|---|
granules |
yes - if query not present |
Array<object> |
List of Granules to process e.g. { granuleId, collectionId }. Required if there is no Elasticsearch query provided |
index |
yes - if query is present |
string |
Elasticsearch index to search with the given query |
query |
yes - if granules not present |
Object |
Query to Elasticsearch to determine which Granules to delete. Required if no Granules are given |
concurrency |
N |
integer |
Sets the granule concurrency for the bulk deletion operation. Defaults to 10 |
forceRemoveFromCmr |
N |
bool |
Whether to remove published granules from CMR before deletion. You must set this value to true to do bulk deletion of published granules, otherwise deleting them will fail. |
knexDebug |
N |
bool |
Sets knex PostgreSQL connection pool/query debug output. Defaults to false |
maxDbConnections |
N |
integer |
Sets the maximum database connections to allocate for the operation. Defaults to concurrency value |
Use the Retrieve async operation endpoint with the id in the response to determine the status of the async operation.
$ curl --request POST \
https://example.com/granules/bulkDelete --header 'Authorization: Bearer ReplaceWithTheToken' \
--header 'Content-Type: application/json' \
--data '{
"forceRemoveFromCmr": true,
"index": "index-in-es",
"query": {
"size": 500,
"query": {
"filter": [
{
"bool": {
"filter": [
{
"bool": {
"should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [{"match": {"status": "FAILED"}}],
"minimum_should_match": 1
}
}
]
}
}
],
"should": [],
"must_not": []
}
}
}'
curl -X POST
https://example.com/granules/bulkDelete --header 'Authorization: Bearer ReplaceWithTheToken' --data '{"granules": [ { "granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606", "collectionId": "MOD11A1___006" } ], "forceRemoveFromCmr": true}'
{
"id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e"
}
Reingest the granules provided. Granules can be sent as a list of IDs or as an Elasticsearch query to the Metrics' Elasticsearch.
Overview of the request fields:
| Field | Required | Value | Description |
|---|---|---|---|
granules |
yes - if query not present |
Array<object> |
List of Granules to process e.g. { granuleId, collectionId }. Required if there is no Elasticsearch query provided |
index |
yes - if query is present |
string |
Elasticsearch index to search with the given query |
query |
yes - if ids not present |
Object |
Query to Elasticsearch to determine which Granules to be reingested. Required if no Granules are given. |
knexDebug |
N |
bool |
Sets knex postgreSQL connection pool/query debug output. Defaults to false |
workflowName |
no |
string |
optional workflow name that allows different workflow and initial input to be used during reingest. See below. |
An optional data parameter of workflowName is also available to allow you to override the input message to the reingest. If workflowName is specified, the original message is pulled directly
by finding the most recent execution of the workflowName associated with the granuleId.
Use the Retrieve async operation endpoint with the id in the response to determine the status of the async operation.
$ curl --request POST \
https://example.com/granules/bulkReingest --header 'Authorization: Bearer ReplaceWithTheToken' \
--header 'Content-Type: application/json' \
--data '{
"index": "index-in-es",
"query": {
"size": 500,
"query": {
"filter": [
{
"bool": {
"filter": [
{
"bool": {
"should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [{"match": {"status": "FAILED"}}],
"minimum_should_match": 1
}
}
]
}
}
],
"should": [],
"must_not": []
}
}
}'
curl -X POST
https://example.com/granules/bulkReingest
--header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json'
--data '{
"granules": [ { "granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606", "collectionId": "MOD11A1___006" } ],
"workflow": "workflowName",
}'
{
"id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e"
}
Updates a batch of existing granules' linked collection (collectionId) in postgres and ES. Expects payload to contain a list of granules and a new collectionId to update them to.
This endpoint will fail if non-existant granuleIds are provided, it can only be used to change existing granules' collectionId in postgres.
Returns status 200 on successful update, 404 if the granuleId can not be found in the database,
or 400 for datastore write or validation errors.
If a write failure occurs, the endpoint is idempotent so the operation can be re-run and should correct the discrepancy.
$ curl --request PATCH https://example.com/granules/bulkPatchGranuleCollection \
--header 'Authorization: Bearer ReplaceWithTheToken' \
--header 'Content-Type: application/json' \
--header 'Cumulus-API-Version: 2' \
--data '{
"apiGranules": [{
"granuleId": "granuleId.A20200113.006.1005",
"collectionId: "collectionId.A20200113.006",
"files": [...],
"duration": 1000,
"status": "completed"
}, {
"granuleId": "granuleId.A20200113.006.1006",
"collectionId: "collectionId.A20200113.006",
"files": [...],
"duration": 1000,
"status": "completed"
},...,
{
"granuleId": "granuleId.A20200113.006.1100",
"collectionId: "collectionId.A20200113.006",
"files": [...],
"duration": 1000,
"status": "completed"
}}],
"collectionId": "collectionId.B31311224.007",
}'
{ "message": "Successfully wrote granules with Granule Id: ['granuleId.A20200113.006.1005', 'granuleId.A20200113.006.1006',...,'granuleId.A20200113.006.1100'], Collection Id: 'collectionId.B31311224.007'" }
Update a batch of granules. Expects payload to contain a list of the modified
granules as the existing granule values will be overwritten by the
modified portions. Please see the Update Granule endpoint for additional details. Configuration for dbConcurrency, a configurable postgres database concurrency for the request, and dbMaxPool, the maximum number of postgres connections the request can make, is provided
for improved database performance tuning.
Returns status 200 on successful update, 201 on new granule creation, 404 if
the granuleId can not be found in the database, or 400 for database write or validation errors.
Please Note: If this endpoint fails on granule update for a batch, some of the batched granules may not be updated. This endpoint does not provide granular update results for each granule (as a feature design boundary) or other retry logic, that update is anticipated in future releases.
If a write failure occurs, as this is an update, correct the failure and re-attempt the batch write to resolve. When a failure occurs, a potential resolution includes re-running the endpoint.
$ curl --request PATCH https://example.com/granules/bulkPatch \
--header 'Authorization: Bearer ReplaceWithTheToken' \
--header 'Content-Type: application/json' \
--header 'Cumulus-API-Version: 2' \
--data '{
"apiGranules": [{
"granuleId": "granuleId.A20200113.006.1005",
"collectionId: "collectionId.B31311224.007",
"files": [
{
"bucket": "stack-protected",
"key": "granuleId.B31311224.007.1005.hdf",
"fileName": "granuleId.B31311224.007.1005.hdf"
},
{
"bucket": "stack-protected",
"key": "granuleId.B31311224.007.1005.jpg",
"fileName": "granuleId.B31311224.007.1005.jpg"
}
],
"duration": 1000,
"status": "completed"
},
{
"granuleId": "granuleId.A20200113.006.1006",
"collectionId: "collectionId.B31311224.007",
"files": [
{
"bucket": "stack-protected",
"key": "granuleId.B31311224.007.1006.hdf",
"fileName": "granuleId.B31311224.007.1006.hdf"
},
{
"bucket": "stack-protected",
"key": "granuleId.B31311224.007.1006.jpg",
"fileName": "granuleId.B31311224.007.1006.jpg"
},
{
"bucket": "stack-protected",
"key": "granuleId.B31311224.007.1006.txt",
"fileName": "granuleId.B31311224.007.1006.txt"
}
],
"duration": 1000,
"status": "completed"
},...,
{
"granuleId": "granuleId.A20200113.006.1100",
"collectionId: "collectionId.B31311224.007",
"files": [
{
"bucket": "stack-protected",
"key": "granuleId.B31311224.007.1100.hdf",
"fileName": "granuleId.B31311224.007.1100.hdf"
},
{
"bucket": "stack-protected",
"key": "granuleId.B31311224.007.1100.jpg",
"fileName": "granuleId.B31311224.007.1100.jpg"
}
],
"duration": 1000,
"status": "completed"
}],
dbConcurrency: 10,
dbMaxPool: 20,
}'
{ "message": "Successfully patched Granules" }
List PDRs in the Cumulus system.
$ curl https://example.com/pdrs --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "pdrs",
"limit": 1,
"page": 1,
"count": 8
},
"results": [
{
"pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
"collectionId": "MOD11A1___006",
"status": "failed",
"provider": "LP_TS2_DataPool",
"progress": 0,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:ACCOUNT:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:6ef0c52f83c549db58b3a1e50",
"PANSent": false,
"PANmessage": "N/A",
"stats": {
"processing": 0,
"completed": 0,
"failed": 0,
"total": 0
},
"createdAt": 1514305411204,
"timestamp": 1514305424036,
"duration": 12.832
}
]
}
Retrieve a single PDR.
$ curl https://example.com/pdrs/7970bff5-128a-489f-b43c-de4ad7834ce5.PDR --header 'Authorization: Bearer ReplaceWithTheToken'
{
"pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
"collectionId": "MOD11A1___006",
"status": "failed",
"provider": "LP_TS2_DataPool",
"progress": 0,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:ACCOUNT:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:6ef0c52f83c549db58b3a1e50",
"PANSent": false,
"PANmessage": "N/A",
"stats": {
"processing": 0,
"completed": 0,
"failed": 0,
"total": 0
},
"createdAt": 1514305411204,
"timestamp": 1514305424036,
"duration": 12.832,
"_id": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR"
}
Delete a PDR from Cumulus. Its granules will remain, and the PDR may be re-discovered and re-ingested/re-processed from scratch in the future.
$ curl --request DELETE https://example.com/pdrs/good_25grans.PDR --header 'Authorization: Bearer ReplaceWithTheToken'
{
"message": "Record deleted"
}
List rules in the Cumulus system.
$ curl https://example.com/rules --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "rules",
"limit": 1,
"page": 1,
"count": 7
},
"results": [
{
"name": "repeat",
"workflow": "DiscoverPdrs",
"provider": "local",
"collection": {
"name": "AST_L1A",
"version": "003"
},
"rule": {
"type": "scheduled",
"value": "rate(5 minutes)"
},
"timestamp": 1511232462534,
"state": "DISABLED"
}
]
}
Retrieve a single rule.
$ curl https://example.com/rules/repeat --header 'Authorization: Bearer ReplaceWithTheToken'
{
"workflow": "DiscoverPdrs",
"collection": {
"name": "AST_L1A",
"version": "003"
},
"updatedAt": 1511232462507,
"createdAt": 1510903518741,
"provider": "local",
"name": "repeat",
"rule": {
"type": "scheduled",
"value": "rate(5 minutes)"
},
"state": "DISABLED"
}
Create a rule. For more information on creating rules and the contents of a request see the Cumulus setup documentation.
Overview of the schema fields:
| Field | Value | Required | Description |
|---|---|---|---|
name |
string |
yes |
rule name (letters, numbers, underscores only) |
state |
"DISABLED"|"ENABLED" |
yes |
rule state (default: ENABLED) |
workflow |
string |
yes |
name of workflow started by the rule |
rule |
Object |
yes |
rule object |
-- rule.type |
"onetime"|"scheduled"|"kinesis"|"sns"|"sqs" |
yes |
rule trigger type |
-- rule.value |
onetime: N/Ascheduled: cron-type or rate expressionkinesis: Kinesis stream ARNsns: SNS topic ARNsqs: SQS queue URL |
no |
required value differs by type |
-- rule.arn |
string |
no |
kinesis scheduled event arn |
-- rule.logEventArn |
string |
no |
kinesis scheduled log event arn |
collection |
Object |
no |
collection record provided to workflow |
-- collection.name |
string |
yes |
collection name |
-- collection.version |
string |
yes |
collection version |
executionNamePrefix |
string |
no |
Execution Name Prefix |
meta |
Object |
no |
contents to add to workflow input's meta field |
payload |
string |
no |
input payload to be used in onetime and scheduled rules |
provider |
string |
no |
provider record provided to workflow |
queueUrl |
string |
no |
queue URL for Scheduled Executions |
tags |
array |
no |
tags (for search) |
$ curl --request POST https://example.com/rules --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
"workflow": "DiscoverPdrs",
"collection": {
"name": "AST_L1A",
"version": "003"
},
"provider": "local",
"name": "repeat_test",
"rule": {
"type": "scheduled",
"value": "rate(5 minutes)"
},
"meta": { "publish": false },
"state": "DISABLED"
}'
{
"message": "Record saved",
"record": {
"workflow": "DiscoverPdrs",
"collection": {
"name": "AST_L1A",
"version": "003"
},
"createdAt": 1510903518741,
"provider": "local",
"name": "repeat_test",
"rule": {
"type": "scheduled",
"value": "rate(5 minutes)"
},
"meta": { "publish": false },
"state": "DISABLED"
}
}
Replace an existing rule. Expects payload to specify the entire rule object, and will completely replace the existing rule with the specified payload. For a field reference see "Create rule".
Returns status 200 on successful replacement, 400 if the name property in the
payload does not match the corresponding value in the resource URI, or 404 if
there is no rule with the specified name.
Special case:
| Field | Value | Description |
|---|---|---|
action |
"rerun" |
rerun rule (onetime rule only), and rule record is not replaced |
$ curl --request PUT https://example.com/rules/repeat_test \
--header 'Authorization: Bearer ReplaceWithToken' \
--header 'Content-Type: application/json' \
--header 'Cumulus-API-Version: 2' \
--data '{
"name": "repeat_test",
"workflow": "DiscoverPdrs",
"collection": {
"name": "AST_L1A",
"version": "003"
},
"provider": "local",
"rule": {
"type": "scheduled",
"value": "rate(5 minutes)"
},
"state": "ENABLED"
}'
{
"name": "repeat_test",
"workflow": "DiscoverPdrs",
"collection": {
"name": "AST_L1A",
"version": "003"
},
"updatedAt": 1521755265130,
"createdAt": 1510903518741,
"provider": "local",
"rule": {
"type": "scheduled",
"value": "rate(5 minutes)"
},
"state": "ENABLED"
}
onetime-rule special case)$ curl --request PUT https://example.com/rules/my_onetime_rule \
--header 'Authorization: Bearer ReplaceWithToken' \
--header 'Content-Type: application/json' \
--header 'Cumulus-API-Version: 2' \
--data '{
"name": "repeat_test",
"workflow": "DiscoverPdrs",
"collection": {
"name": "AST_L1A",
"version": "003"
},
"provider": "local",
"rule": {
"type": "scheduled",
"value": "rate(5 minutes)"
},
"state": "ENABLED",
"action": "rerun"
}'
Update an existing rule. Expects payload to contain the modified and/or additional
fields of the rule and the existing rule values will be overwritten by the
modified portions. Unspecified keys will be retained. Keys set to null
will be removed. For a field reference see "Create rule".
Returns status 200 on successful update, 400 if the name property in the
payload does not match the corresponding value in the resource URI, or 404 if
there is no rule with the specified name.
Special case:
| Field | Value | Description |
|---|---|---|
action |
"rerun" |
rerun rule (onetime rule only), and rule record is not updated |
$ curl --request PATCH https://example.com/rules/repeat_test \
--header 'Authorization: Bearer ReplaceWithToken' \
--header 'Content-Type: application/json' \
--header 'Cumulus-API-Version: 2' \
--data '{
"name": "repeat_test",
"workflow": "NewDiscoverPdrs",
"meta": {
"additionalKey": "additionalKeyValue"
},
"state": "ENABLED"
}'
{
"name": "repeat_test",
"workflow": "NewDiscoverPdrs",
"collection": {
"name": "AST_L1A",
"version": "003"
},
"updatedAt": 1521755265130,
"createdAt": 1510903518741,
"provider": "local",
"rule": {
"type": "scheduled",
"value": "rate(5 minutes)"
},
"meta": {
"publish": false,
"additionalKey": "additionalKeyValue"
},
"state": "ENABLED"
}
onetime-rule special case)$ curl --request PATCH https://example.com/rules/my_onetime_rule \
--header 'Authorization: Bearer ReplaceWithToken' \
--header 'Content-Type: application/json' \
--header 'Cumulus-API-Version: 2' \
--data '{"action": "rerun"}'
Delete a rule from Cumulus.
$ curl --request DELETE https://example.com/rules/repeat_test --header 'Authorization: Bearer ReplaceWithTheToken'
{
"message": "Record deleted"
}
Retrieve a summary of statistics around the granules in the system. The collections returned are the number of distinct collections for the granules active during the given time period, defaulted to the last day if none is specified.
$ curl https://example.com/stats --header 'Authorization: Bearer ReplaceWithTheToken'
{
"errors": {
"dateFrom": "1970-01-18T12:36:59+00:00",
"dateTo": "2017-12-26T04:38:15+00:00",
"value": 2,
"aggregation": "count",
"unit": "error"
},
"collections": {
"dateFrom": "1970-01-01T12:00:00+00:00",
"dateTo": "2017-12-26T04:38:15+00:00",
"value": 3,
"aggregation": "count",
"unit": "collection"
},
"processingTime": {
"dateFrom": "1970-01-18T12:36:59+00:00",
"dateTo": "2017-12-26T04:38:15+00:00",
"value": null,
"aggregation": "average",
"unit": "second"
},
"granules": {
"dateFrom": "1970-01-18T12:36:59+00:00",
"dateTo": "2017-12-26T04:38:15+00:00",
"value": 8,
"aggregation": "count",
"unit": "granule"
}
}
Count the value frequencies for a given field, for a given type of record in Cumulus. Requires the following query parameters, and may include the regular filter parameters:
| query string parameter | description |
|---|---|
type={providers|collections|granules|pdrs|logs} |
type of Cumulus record to query |
field={fieldName} |
which field to count frequencies for; no default |
curl 'https://example.com/stats/aggregate?field=status&type=pdrs' --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"count": 52,
"field": "status.keyword"
},
"count": [
{
"key": "failed",
"count": 43
},
{
"key": "completed",
"count": 5
},
{
"key": "parsed",
"count": 3
}
]
}
Note: This endpoint will only work if logs are being delivered to the Metrics system. Otherwise it will return a 400 Error.
List processing logs from the Cumulus engine. A log's level field should be specified by their string value and be one of:
$ curl https://example.com/logs?limit=5 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "logs",
"limit": 5,
"page": 1,
"count": 750
},
"results": [
{
"pid": 1,
"hostname": "ip-10-29-4-186",
"name": "cumulus",
"level": 50,
"msg": "https://ba008ffc.ngrok.io/ not found",
"file": "discover-pdrs/index.js",
"type": "Error",
"stack": "HostNotFound: https://ba008ffc.ngrok.io/ not found\n at discover.discover.then.catch.e (/var/task/index.js:9006:22)\n at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
"message": "https://ba008ffc.ngrok.io/ not found",
"v": 1,
"timestamp": 1513881407867
},
{
"pid": 1,
"hostname": "ip-10-29-4-186",
"name": "cumulus",
"level": 50,
"file": "discover-pdrs/index.js",
"message": "Received a 404 error from undefined. Check your endpoint!",
"details": {
"host": "ba008ffc.ngrok.io",
"path": "/",
"port": "",
"protocol": "https",
"uriPath": "/",
"url": "https://ba008ffc.ngrok.io/",
"depth": 1,
"fetched": true,
"status": "notfound",
"stateData": {
"requestLatency": 66,
"requestTime": 66,
"contentLength": 34,
"contentType": "text/plain",
"code": 404,
"headers": {
"content-length": "34",
"connection": "close",
"content-type": "text/plain"
}
},
"id": 0
},
"v": 1,
"timestamp": 1513881407867
},
{
"pid": 1,
"hostname": "ip-10-29-4-186",
"name": "cumulus",
"level": 50,
"msg": "Cannot read property 'provider_path' of undefined",
"file": "discover-pdrs/index.js",
"type": "Error",
"stack": "TypeError: Cannot read property 'provider_path' of undefined\n at HttpDiscover.Discover (/var/task/index.js:10098:33)\n at HttpDiscover._class (/var/task/index.js:96368:255)\n at new HttpDiscover (/var/task/index.js:10342:241)\n at handler (/var/task/index.js:8968:23)\n at Promise (/var/task/cumulus-sled/index.js:82:5)\n at invokeHandler (/var/task/cumulus-sled/index.js:72:10)\n at then.then.then (/var/task/cumulus-sled/index.js:112:14)\n at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
"v": 1,
"timestamp": 1513881031694
},
{
"pid": 1,
"hostname": "ip-10-29-4-186",
"name": "cumulus",
"level": 50,
"msg": "Provider info not provided",
"file": "discover-pdrs/index.js",
"type": "Error",
"stack": "ProviderNotFound: Provider info not provided\n at handler (/var/task/index.js:8962:20)\n at Promise (/var/task/cumulus-sled/index.js:82:5)\n at invokeHandler (/var/task/cumulus-sled/index.js:72:10)\n at then.then.then (/var/task/cumulus-sled/index.js:112:14)\n at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
"message": "Provider info not provided",
"v": 1,
"timestamp": 1513879654040
},
{
"pid": 1,
"hostname": "ip-10-29-4-186",
"name": "cumulus",
"level": 50,
"msg": "Provider info not provided",
"file": "discover-pdrs/index.js",
"type": "Error",
"stack": "ProviderNotFound: Provider info not provided\n at handler (/var/task/index.js:8962:20)\n at Promise (/var/task/cumulus-sled/index.js:82:5)\n at invokeHandler (/var/task/cumulus-sled/index.js:72:10)\n at then.then.then (/var/task/cumulus-sled/index.js:112:14)\n at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
"message": "Provider info not provided",
"v": 1,
"timestamp": 1513879435010
}
]
}
$ curl https://example.com/logs?level=error --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "logs",
"limit": 1,
"page": 1,
"count": 181
},
"results": [
{
"level": 50,
"msg": "Unexpected HTTP status code: 404",
"pid": 1,
"hostname": "ip-10-26-125-174",
"name": "cumulus",
"file": "sync-granule/index.js",
"type": "Error",
"stack": "Error: Unexpected HTTP status code: 404\n at ClientRequest.<anonymous> (/var/task/sync-granule/index.js:125136:29)\n at ClientRequest.g (events.js:292:16)\n at emitOne (events.js:96:13)\n at ClientRequest.emit (events.js:188:7)\n at HTTPParser.parserOnIncomingClient (_http_client.js:473:21)\n at HTTPParser.parserOnHeadersComplete (_http_common.js:99:23)\n at TLSSocket.socketOnData (_http_client.js:362:20)\n at emitOne (events.js:96:13)\n at TLSSocket.emit (events.js:188:7)\n at readableAddChunk (_stream_readable.js:176:18)",
"code": 404,
"v": 1,
"timestamp": 1515446810226
}
]
}
Note: This endpoint will only work if logs are being delivered to the Metrics system. Otherwise it will return a 400 Error.
Retrieve all logs from a specific execution.
$ curl https://example.com/logs/93e5e049-89aa-47e5-900d-0eec87327260?limit=5 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "test-src-integration",
"table": "logs",
"limit": 5,
"page": 1,
"count": 10
},
"results": [
{
"level": 30,
"executions": "93e5e049-89aa-47e5-900d-0eec87327260",
"timestamp": "2018-08-15T19:23:57.752Z",
"sender": "test-src-integration-PostToCmr",
"version": "$LATEST",
"message": "Published MOD09GQ.A8009365.7JQhni.006.6594428626875 to the CMR. conceptId: G1223210348-CUMULUS",
"RequestId": "05712fb1-5593-4640-b5a3-a99db41d8003"
},
{
"level": 30,
"executions": "93e5e049-89aa-47e5-900d-0eec87327260",
"timestamp": "2018-08-15T19:23:42.054Z",
"sender": "test-src-integration-SyncGranuleNoVpc",
"version": "$LATEST",
"message": "uploaded s3://cumulus-test-sandbox-internal/file-staging/test-src-integration/MOD09GQ___006/MOD09GQ.A8009365.7JQhni.006.6594428626875.hdf",
"RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
},
{
"level": 30,
"executions": "93e5e049-89aa-47e5-900d-0eec87327260",
"timestamp": "2018-08-15T19:23:41.913Z",
"sender": "test-src-integration-SyncGranuleNoVpc",
"version": "$LATEST",
"message": "Finishing downloading s3://cumulus-test-sandbox-internal/cumulus-test-data/pdrs/MOD09GQ.A8009365.7JQhni.006.6594428626875.hdf",
"RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
},
{
"level": 30,
"executions": "93e5e049-89aa-47e5-900d-0eec87327260",
"timestamp": "2018-08-15T19:23:41.888Z",
"sender": "test-src-integration-SyncGranuleNoVpc",
"version": "$LATEST",
"message": "uploaded s3://cumulus-test-sandbox-internal/file-staging/test-src-integration/MOD09GQ___006/MOD09GQ.A8009365.7JQhni.006.6594428626875_ndvi.jpg",
"RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
},
{
"level": 30,
"executions": "93e5e049-89aa-47e5-900d-0eec87327260",
"timestamp": "2018-08-15T19:23:41.886Z",
"sender": "test-src-integration-SyncGranuleNoVpc",
"version": "$LATEST",
"message": "uploaded s3://cumulus-test-sandbox-internal/file-staging/test-src-integration/MOD09GQ___006/MOD09GQ.A8009365.7JQhni.006.6594428626875.hdf.met",
"RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
}
]
}
Get a CSV file of all the granule in the Cumulus database.
$ curl https://example.com/granule-csv --header 'Authorization: Bearer ReplaceWithTheToken'
"granuleUr","collectionId","createdAt","startDateTime","endDateTime","status","updatedAt","published" "MOD14A1.A9506271.IvEJsu.006.8359924290786","MOD14A1___006","2020-05-18T20:15:54.525Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-05-18T20:16:02.473Z",false "MYD13Q1.A9663671.0zkwKH.006.9812354158395","MYD13Q1___006","2020-07-06T19:46:19.957Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-07-06T19:46:57.054Z",true
List executions.
If the query string parameters include a value of true for includeFullRecord, any associated async operations and parent execution will be included in each executions's return value. The default value is false.
For requests without filters, if the query string parameters include a value of false for estimateTableRowCount, the returned count will be the actual exact count, otherwise count will be an estimated count. The default value is true.
$ curl https://example.com/executions?limit=3 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "test-src-integration",
"table": "executions",
"limit": 3,
"page": 1,
"count": 447
},
"results": [
{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
"duration": 1.693,
"createdAt": 1534443719571,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
"name": "08aa40f8-7d91-41b4-986e-d12ce495aec5",
"error": {},
"type": "IngestGranule",
"collectionId": "MOD09GQ___006",
"tasks": {},
"status": "running",
"timestamp": 1534443721559,
"updatedAt": 1534443721265
},
{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
"duration": 1.718,
"createdAt": 1534443706867,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
"name": "bee18782-e987-4bd4-b726-41669e489e2f",
"error": {},
"type": "IngestGranule",
"collectionId": "MOD09GQ___006",
"tasks": {},
"status": "running",
"timestamp": 1534443709043,
"updatedAt": 1534443708585
},
{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
"duration": 29.397,
"createdAt": 1534443668185,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
"name": "55d62b27-50cd-4cc1-81f9-e425cf10532e",
"error": {
"Cause": "None",
"Error": "Unknown Error"
},
"type": "IngestGranule",
"collectionId": "MOD09GQ___006",
"tasks": {},
"status": "completed",
"timestamp": 1534443698322,
"updatedAt": 1534443697582
}
]
}
Retrieve details for a specific execution.
$ curl https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
"updatedAt": 1534441782302,
"status": "completed",
"timestamp": 1534441782302,
"tasks":
{
"MoveGranuleStep":
{
"name": "test-src-integration-MoveGranules",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-MoveGranules",
"version": "$LATEST"
},
"ProcessingStep":
{
"name": "test-src-integration-FakeProcessing",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-FakeProcessing",
"version": "$LATEST"
},
"StopStatus":
{
"name": "test-src-integration-SfSnsReport",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-SfSnsReport",
"version": "$LATEST"
}
},
"createdAt": 1534441752026,
"duration": 30.276,
"name": "cff1266e-ef36-664f-a649-3a4d26bd1735",
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
"collectionId": "MOD09GQ___006",
"error":
{
"Error": "Unknown Error",
"Cause": "None"
},
"type": "IngestGranule",
"originalPayload": { "somePayloadKey": "object containing payload at execution start" },
"finalPayload": { "somePayloadKey" : "object containing the last reported payload from an execution" }
}
Retrieve details and status of a specific execution. This also returns execution history and details of the state machine when the execution exists in Step Function API.
$ curl https://example.com/executions/status/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl6sgv:d0a6584b-bea6-476e-a745-c1feb2ad00b2 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"presignedS3Url": "https://example.amazonaws.com/example",
"data": "Execution status data, see example below"
}
{
"presignedS3Url": "https://example.amazonaws.com/example",
"data": "Error: Execution Status for execution123 exceeded maximum allowed payload size"
}
{
"execution":
{
"executionArn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv:d0a6584b-bea6-476e-a745-c1feb2ad00b2",
"stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv",
"name": "d0a6584b-bea6-476e-a745-c1feb2ad00b2",
"status": "SUCCEEDED",
"startDate": "2018-08-16T18:39:32.209Z",
"stopDate": "2018-08-16T18:39:59.089Z",
"input":
{
"foo": "input"
},
"output":
{
"bar": "output"
}
},
"executionHistory":
{
"events":
[
{
"taskName": "foo bar 1"
},
{
"taskName": "foo bar 2"
}
]
},
"stateMachine":
{
"stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv",
"name": "TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv",
"status": "ACTIVE",
"definition":
{
"Comment": "Ingest Granule",
"StartAt": "First State",
"States":
{
"First State": {}
}
},
"roleArn": "arn:aws:iam::123456789012:role/test-src-integration-steprole",
"creationDate": "2018-06-11T16:08:17.533Z"
}
}
Return executions associated with specific granules. Granules can be sent as a list of granule objects containing granuleIds and collectionIds or as an Elasticsearch query to the Metrics' Elasticsearch.
Overview of the request fields:
| Field | Required | Value | Description |
|---|---|---|---|
granules |
yes - if query not present |
Array<Object> |
List of granules to process. Each granule must contain a granuleId and collectionId |
-- granule.granuleId |
yes - if using granules |
string |
GranuleId for a granule |
-- granule.collectionId |
yes - if using granules |
string |
CollectionId for a granule |
query |
yes - if granules not present |
Object |
Query to Elasticsearch to determine which Granules with which to search. Required if no IDs are given. |
index |
yes - if query is present |
string |
Elasticsearch index to search with the given query |
curl -X POST
https://example.com/executions/search-by-granules?limit=3 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"granules": [{ "granuleId":"MOD09GQ.A2016358.h13v04.006.2016360104606", "collectionId": "MOD09GQ___006"}]}'
$ curl --request POST \
https://example.com/executions/search-by-granules?limit=3 --header 'Authorization: Bearer ReplaceWithTheToken' \
--header 'Content-Type: application/json' \
--data '{
"index": "index-in-es",
"query": {
"size": 500,
"query": {
"filter": [
{
"bool": {
"filter": [
{
"bool": {
"should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [{"match": {"collectionId": "MOD09GQ__006"}}],
"minimum_should_match": 1
}
}
]
}
}
],
"should": [],
"must_not": []
}
}
}'
{
"meta": {
"count": 3
},
"results": [
{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
"duration": 1.693,
"createdAt": 1534443719571,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
"name": "08aa40f8-7d91-41b4-986e-d12ce495aec5",
"error": {},
"type": "IngestGranule",
"collectionId": "MOD09GQ___006",
"tasks": {},
"status": "running",
"timestamp": 1534443721559,
"updatedAt": 1534443721265
},
{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
"duration": 1.718,
"createdAt": 1534443706867,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
"name": "bee18782-e987-4bd4-b726-41669e489e2f",
"error": {},
"type": "IngestGranule",
"collectionId": "MOD09GQ___006",
"tasks": {},
"status": "running",
"timestamp": 1534443709043,
"updatedAt": 1534443708585
},
{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
"duration": 29.397,
"createdAt": 1534443668185,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
"name": "55d62b27-50cd-4cc1-81f9-e425cf10532e",
"error": {
"Cause": "None",
"Error": "Unknown Error"
},
"type": "IngestGranule",
"collectionId": "MOD09GQ___006",
"tasks": {},
"status": "completed",
"timestamp": 1534443698322,
"updatedAt": 1534443697582
}
]
}
Return the workflows that have run on specific granules. If multiple granules are specified, it will return the intersection of workflows that have run on ALL the granules. That is, only a workflow that has been run on every inputted granule will be returned
Overview of the request fields:
| Field | Required | Value | Description |
|---|---|---|---|
granules |
yes - if query not present |
Array<Object> |
List of granules to process. Each granule must contain a granuleId and collectionId |
-- granule.granuleId |
yes - if using granules |
string |
GranuleId for a granule |
-- granule.collectionId |
yes - if using granules |
string |
CollectionId for a granule |
query |
yes - if granules not present |
Object |
Query to Elasticsearch to determine which Granules with which to search. Required if no IDs are given. |
index |
yes - if query is present |
string |
Elasticsearch index to search with the given query |
curl -X POST
https://example.com/executions/workflows-by-granules --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"granules": [{ "granuleId":"MOD09GQ.A2016358.h13v04.006.2016360104606", "collectionId": "MOD09GQ___006"}]}'
$ curl --request POST \
https://example.com/executions/workflows-by-granules --header 'Authorization: Bearer ReplaceWithTheToken' \
--header 'Content-Type: application/json' \
--data '{
"index": "index-in-es",
"query": {
"size": 500,
"query": {
"filter": [
{
"bool": {
"filter": [
{
"bool": {
"should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [{"match": {"collectionId": "MOD09GQ__006"}}],
"minimum_should_match": 1
}
}
]
}
}
],
"should": [],
"must_not": []
}
}
}'
[
"DiscoverGranules",
"IngestGranules"
]
Create an execution. An execution generally includes the fields listed below.
Overview of the schema fields:
| Field | Type | Required | Description |
|---|---|---|---|
arn |
string |
yes |
execution arn (this field is unique) |
asyncOperationId |
string |
no |
id of the associated async operation |
collectionId |
string |
no |
collectionId of the associated collection (e.g. <name>___<version> where <name> is the collection name and <version> is the collection version) |
cumulusVersion |
string |
no |
cumulus version for the execution (e.g. 9.0.0) |
duration |
number |
no |
duration of the execution |
error |
object |
no |
the error details in case of a failed execution |
execution |
string |
no |
the execution page url on AWS console |
finalPayload |
object |
no |
the final payload of this execution |
name |
string |
yes |
execution name |
originalPayload |
object |
no |
the original payload for this execution |
parentArn |
string |
no |
arn of the parent execution |
status |
string |
yes |
the execution status, possible values: running, completed, failed, unknown |
tasks |
object |
no |
the tasks for this execution |
type |
string |
no |
the workflow name, e.g. IngestGranule |
timestamp |
number |
no |
the timestamp for this execution |
$ curl --request POST https://example.com/executions --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
"asyncOperationId": "c4ece1f9-dca0-42f8-be77-da43b9b25918",
"collectionId": "MOD09GQ___006",
"cumulusVersion": "1.2.3",
"error": {},
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
"name": "cff1266e-ef36-664f-a649-3a4d26bd1735",
"originalPayload": { "somePayloadKey": "object containing payload at execution start" },
"parentArn": "arn:aws:states:us-east-1:123456789012:execution:test-src-integration-ParsePdr:59edd6a4-5187-4865-b3c3-71ebcf76d7a1",
"status": "running",
"tasks":
{
"MoveGranuleStep":
{
"name": "test-src-integration-MoveGranules",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-MoveGranules",
"version": "$LATEST"
},
"ProcessingStep":
{
"name": "test-src-integration-FakeProcessing",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-FakeProcessing",
"version": "$LATEST"
},
"StopStatus":
{
"name": "test-src-integration-SfSnsReport",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-SfSnsReport",
"version": "$LATEST"
}
},
"type": "IngestGranule",
"timestamp": 1513020462156
}'
{
"message": "Successfully wrote execution with arn arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735"
}
Update/replace an existing execution. Expects payload to specify the entire execution object, and will completely replace the existing execution with the specified payload. For a field reference see "Create execution".
Returns status 200 on successful replacement, 400 if the arn
property in the payload does not match the corresponding value in the resource
URI, or 404 if there is no execution with the specified arn.
$ curl --request PUT https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
"asyncOperationId": "c4ece1f9-dca0-42f8-be77-da43b9b25918",
"collectionId": "MOD09GQ___006",
"cumulusVersion": "1.2.3",
"duration": 30.276,
"error": {},
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
"finalPayload": { "somePayloadKey" : "object containing the last reported payload from an execution" },
"name": "cff1266e-ef36-664f-a649-3a4d26bd1735",
"originalPayload": { "somePayloadKey": "object containing payload at execution start" },
"parentArn": "arn:aws:states:us-east-1:123456789012:execution:test-src-integration-ParsePdr:59edd6a4-5187-4865-b3c3-71ebcf76d7a1",
"status": "completed",
"tasks":
{
"MoveGranuleStep":
{
"name": "test-src-integration-MoveGranules",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-MoveGranules",
"version": "$LATEST"
},
"ProcessingStep":
{
"name": "test-src-integration-FakeProcessing",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-FakeProcessing",
"version": "$LATEST"
},
"StopStatus":
{
"name": "test-src-integration-SfSnsReport",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-SfSnsReport",
"version": "$LATEST"
}
},
"type": "IngestGranule",
"timestamp": 1513020462156
}'
{
"message": "Successfully updated execution with arn arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735"
}
Delete an execution from Cumulus.
$ curl --request DELETE https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"message": "Record deleted"
}
collectionIdThis request, when given an existing collectionId will trigger a bulk action that will take the following actions:
collectionId (in a single query with a limit specified by esBatchSize in the payload)Remove all records from the postgreSQL database matching the collectionId (in a single query with a limit specified by dbBatchSize in the payload)
parent_cumulus_id field constraints, the child record will have that field set to null.$ curl --request POST \
https://example.com/executions/bulk-delete-by-collection --header 'Authorization: Bearer ReplaceWithTheToken' \
--header 'Content-Type: application/json' \
--data '{
"collectionId": "COLLECTION_NAME___COLLECTION_VERSION",
"esBatchSize": 100000,
"dbBatchSize": 50000
}'
{
"id": "12345678-2222-3333-4444-1234567890ab"
}
List workflows in the Cumulus system.
$ curl https://example.com/workflows --header 'Authorization: Bearer ReplaceWithTheToken'
[
{
"name": "TestLambdaVersionWorkflow",
"template": "s3://cumulus-test-sandbox-internal/cumulus/workflows/TestLambdaVersionWorkflow.json",
"definition": {
"Comment": "Tests Lambda update after redeploy",
"StartAt": "StartStatus",
"States": {
"StartStatus": {
"Type": "Task",
"Resource": "${SfSnsReportLambdaAliasOutput}",
"Next": "WaitForDeployment"
},
"WaitForDeployment": {
"Type": "Task",
"Resource": "${WaitForDeploymentLambdaAliasOutput}",
"Next": "VersionUpTest"
},
"VersionUpTest": {
"Type": "Task",
"Resource": "${VersionUpTestLambdaAliasOutput}",
"Next": "StopStatus"
},
"StopStatus": {
"Type": "Task",
"Resource": "${SfSnsReportLambdaAliasOutput}",
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"Next": "WorkflowFailed"
}
],
"End": true
},
"WorkflowFailed": {
"Type": "Fail",
"Cause": "Workflow failed"
}
}
}
},
{
"name": "HelloWorldWorkflow",
"template": "s3://cumulus-test-sandbox-internal/cumulus/workflows/HelloWorldWorkflow.json",
"definition": {
"Comment": "Returns Hello World",
"StartAt": "StartStatus",
"States": {
"StartStatus": {
"Type": "Task",
"Resource": "${SfSnsReportLambdaAliasOutput}",
"Next": "HelloWorld"
},
"HelloWorld": {
"Type": "Task",
"Resource": "${HelloWorldLambdaAliasOutput}",
"Next": "StopStatus"
},
"StopStatus": {
"Type": "Task",
"Resource": "${SfSnsReportLambdaAliasOutput}",
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"Next": "WorkflowFailed"
}
],
"End": true
},
"WorkflowFailed": {
"Type": "Fail",
"Cause": "Workflow failed"
}
}
}
}
]
Retrieve a single workflow.
$ curl https://example.com/workflows/HelloWorldWorkflow --header 'Authorization: Bearer ReplaceWithTheToken'
{
"name": "HelloWorldWorkflow",
"template": "s3://cumulus-test-sandbox-internal/cumulus/workflows/HelloWorldWorkflow.json",
"definition": {
"Comment": "Returns Hello World",
"StartAt": "StartStatus",
"States": {
"StartStatus": {
"Type": "Task",
"Resource": "${SfSnsReportLambdaAliasOutput}",
"Next": "HelloWorld"
},
"HelloWorld": {
"Type": "Task",
"Resource": "${HelloWorldLambdaAliasOutput}",
"Next": "StopStatus"
},
"StopStatus": {
"Type": "Task",
"Resource": "${SfSnsReportLambdaAliasOutput}",
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"Next": "WorkflowFailed"
}
],
"End": true
},
"WorkflowFailed": {
"Type": "Fail",
"Cause": "Workflow failed"
}
}
}
}
Async operations are long-running requests serviced by the Cumulus API.
These tend to be bulk operations. Examples include bulk granule operations and replaying ingest notifications.
This endpoint lists async operations in the Cumulus system.
$ curl https://example.com/asyncOperations --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "async_operations",
"limit": 1,
"page": 1,
"count": 8
},
"results": [
{
"output": "{\"deletedGranules\":[\"granule-id-f02a53418f\"]}",
"createdAt": 1591384094512,
"taskArn": "arn:aws:ecs:us-east-1:111111111111:task/d481e76e-f5fc-9c1c-2411-fa13779b111a",
"description": "Bulk granule deletion",
"operationType": "Bulk Granule Delete",
"id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
"status": "SUCCEEDED",
"updatedAt": 1591384094512,
"timestamp": 1591384095235
}
]
}
Retrieve information about a single async operation. Useful for determining the status of an async operation.
$ curl https://example.com/asyncOperations/0eb8e809-8790-5409-1239-bcd9e8d28b8e --header 'Authorization: Bearer ReplaceWithTheToken'
{
"id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
"updatedAt": 1574730504762,
"status": "RUNNING",
"taskArn": "arn:aws:ecs:us-east-1:111111111111:task/d481e76e-f5fc-9c1c-2411-fa13779b111a",
"description": "Bulk granule deletion",
"operationType": "Bulk Granule Delete"
}
The Cumulus API supports requests to replay ingest notifications. The schema below describes the expected fields in a replay request
| Field | Type | Required | Description |
|---|---|---|---|
type |
string | required | Currently only accepts kinesis. |
kinesisStream |
string | for type kinesis |
Any valid kinesis stream name (not ARN) |
kinesisStreamCreationTimestamp |
* | optional | Any input valid for a JS Date constructor. For reasons to use this field see AWS documentation on StreamCreationTimestamp. |
endTimestamp |
* | optional | Any input valid for a JS Date constructor. Messages newer than this timestamp will be skipped. |
startTimestamp |
* | optional | Any input valid for a JS Date constructor. Messages will be fetched from the Kinesis stream starting at this timestamp. Ignored if it is further in the past than the stream's retention period. |
$ curl -X POST https://example.com/replays --header 'Authorization: Bearer ReplaceWithTheToken' --data '{ "type: "kinesis", "kinesisStream": "my-stream", "endTimestamp": 1567890123456, "startTimestamp": "2019-08-31T22:22:03.456Z"}'
{
"asyncOperationId": "208a463a-e096-4dd9-b006-e09345319ae6"
}
Retrieve the data schema for a particular type of Cumulus record.
This schema describes the expected format of a record's JSON object when retrieving from Cumulus, as well as a summary of what each field may contain. The schema response can also be used to determine which fields are required when creating a new record using the API.
Supported type values are provider, collection, granule, and pdr.
$ curl https://example.com/schemas/provider --header 'Authorization: Bearer ReplceWithTheToken'
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Provider Object",
"description": "Keep the information about each ingest endpoint",
"type": "object",
"properties": {
"name": {
"title": "Title",
"description": "A title for the provider record",
"type": "string",
"pattern": "^([\\w\\d_\\-]*)$"
},
"providerName": {
"title": "Provider, e.g. MODAPS",
"description": "Name of the SIP",
"type": "string"
},
"protocol": {
"title": "Protocol",
"type": "string",
"enum": [
"http",
"ftp"
],
"default": "http"
},
"host": {
"title": "Host",
"type": "string"
},
"path": {
"title": "Path to the PDR/files folder",
"type": "string"
},
"config": {
"title": "Configuration",
"type": "object",
"properties": {
"username": {
"type": "string"
},
"password": {
"type": "string"
},
"port": {
"type": "string"
}
}
},
"status": {
"title": "Status",
"type": "string",
"enum": [
"ingesting",
"stopped",
"failed"
],
"default": "stopped",
"readonly": true
},
"isActive": {
"title": "Is Active?",
"type": "boolean",
"default": false,
"readonly": true
},
"regex": {
"type": "object",
"patternProperties": {
"^([\\S]*)$": {
"type": "string"
}
},
"readonly": true
},
"lastTimeIngestedAt": {
"title": "Last Time Ingest from the Provider",
"type": "number",
"readonly": true
},
"createdAt": {
"type": "number",
"readonly": true
},
"updatedAt": {
"type": "number",
"readonly": true
}
},
"required": [
"name",
"providerName",
"protocol",
"host",
"path",
"isActive",
"status",
"createdAt",
"updatedAt"
]
}
List reconciliation reports in the Cumulus system.
$ curl https://example.com/reconciliationReports --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "reconciliation_reports",
"limit": 2,
"page": 1,
"count": 7
},
"results": [
{
"createdAt": 1589824212102,
"name": "inventoryReport-20200518T175012101",
"location": "s3://lpdaac-internal/lpdaac-cumulus/reconciliation-reports/inventoryReport-20200518T175012101.json",
"type": "Inventory",
"status": "Generated",
"updatedAt": 1589824218825,
"timestamp": 1589824219329
},
{
"createdAt": 1589596344305,
"name": "ModisInternalReport",
"location": "s3://lpdaac-internal/lpdaac-cumulus/reconciliation-reports/ModisInternalReport.json",
"type": "Internal",
"status": "Generated",
"updatedAt": 1589596352010,
"timestamp": 1589596352356
}
]
}
Retrieve a single reconciliation report.
$ curl https://example.com/reconciliationReports/inventoryReport-20190305T153430508 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"presignedS3Url": "https://example.amazonaws.com/example",
"data": "report data, see examples below for each report type"
}
{
"presignedS3Url": "https://example.amazonaws.com/example",
"data": "Error: Report examplereport exceeded maximum allowed payload size"
}
{
"reportStartTime": "2019-03-05T15:34:30.508Z",
"reportEndTime": "2019-03-05T15:34:37.243Z",
"status": "SUCCESS",
"error": null,
"reportType": "Inventory",
"filesInCumulus": {
"okCount": 40,
"okCountByGranule": {
"MOD09GQ.A2016358.h13v04.006.2016360104606": 10,
"MYD13Q1.A2017297.h19v10.006.2017313221303": 10,
"MOD09GQ.A3518809.ln_rVr.006.7962927138074": 10,
"MOD09GQ.A8768252.HC4ddD.006.2077696236118": 5,
"MOD09GQ.A8722843.GTk5A3.006.4026909316904": 5
},
"onlyInS3": [
"s3://cumulus-test-sandbox-protected/MOD09GQ.A2016358.h13v04.006.2016360104606.cmr.xml",
"s3://cumulus-test-sandbox-private/BROWSE.MYD13Q1.A2017297.h19v10.006.2017313221201.hdf"
],
},
"collectionsInCumulusCmr": {
"okCount": 1,
"onlyInCumulus": [
"L2_HR_PIXC___000"
],
"onlyInCmr": [
"MCD43A1___006",
"MOD14A1___006"
]
},
"granulesInCumulusCmr": {
"okCount": 3,
"onlyInCumulus": [
{
"granuleId": "MOD09GQ.A3518809.ln_rVr.006.7962927138074",
"collectionId": "MOD09GQ___006"
},
{
"granuleId": "MOD09GQ.A8768252.HC4ddD.006.2077696236118",
"collectionId": "MOD09GQ___006"
}
],
"onlyInCmr": [
{
"GranuleUR": "MOD09GQ.A0002421.oD4zvB.006.4281362831355",
"ShortName": "MOD09GQ",
"Version": "006"
},
{
"GranuleUR": "MOD09GQ.A0007443.H16AHq.006.7078190437865",
"ShortName": "MOD09GQ",
"Version": "006"
}
]
},
"filesInCumulusCmr": {
"okCount": 11,
"onlyInCumulus": [
{
"fileName": "MOD09GQ.A8722843.GTk5A3.006.4026909316904.jpeg",
"uri": "s3://cumulus-test-sandbox-public/MOD09GQ___006/MOD/MOD09GQ.A8722843.GTk5A3.006.4026909316904.jpeg",
"granuleId": "MOD09GQ.A8722843.GTk5A3.006.4026909316904"
}
],
"onlyInCmr": [
{
"URL": "https://example.com/MYD13Q1.A2017297.h19v10.006.2017313221202.hdf",
"Type": "GET DATA",
"GranuleUR": "MOD09GQ.A4675287.SWPE5_.006.7310007729190"
},
{
"URL": "https://cumulus-test-sandbox-public.s3.amazonaws.com/MOD09GQ___006/MOD/MOD09GQ.A8722843.GTk5A3.006.4026909316904_ndvi.jpg",
"Type": "GET DATA",
"GranuleUR": "MOD09GQ.A8722843.GTk5A3.006.4026909316904"
}
]
}
}
{
"createStartTime": "2020-08-28T18:33:50.525Z",
"createEndTime": "2020-08-28T18:34:16.646Z",
"status": "SUCCESS",
"reportType": "Granule Not Found",
"filesInCumulus": {
"okCount": 20,
"okCountByGranule": {
"MOD09GQ.A2770300.W1_V5Z.006.7853319756315": 2,
"MOD09GQ.A7682091.QtZjhI.006.1218763030745": 3,
"MOD09GQ.A9536857.bV1q4A.006.0980635705136": 3,
"MOD09GQ.A7790080.xLPpTZ.006.2292724247258": 3,
"MOD09GQ.A3769578.tlqq7j.006.4434181201872": 3,
"MOD09GQ.A5423294.RFOrMI.006.8677601711674": 3,
"MOD09GQ.A8837603.fb2Vhw.006.3950746589733": 3
},
"onlyInS3": [
"s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593122280799/MOD09GQ.A6921412.znLBqe.006.9673856807617.hdf.met",
"s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593122597517/MOD09GQ.A1201557.d6tP2Y.006.7482431709753.hdf.met",
"s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593198955197/MOD09GQ.A7375038.gm1irM.006.1317280710645.hdf.met",
"s3://cumulus-sandbox-protected/MOD09GQ___006/2017/MOD/test-data-1593122280799/MOD09GQ.A6921412.znLBqe.006.9673856807617.hdf",
"s3://cumulus-sandbox-protected/MOD09GQ___006/2017/MOD/test-data-1593122597517/MOD09GQ.A1201557.d6tP2Y.006.7482431709753.hdf",
"s3://cumulus-sandbox-protected/MOD09GQ___006/2017/MOD/test-data-1593122881446/MOD09GQ.A2708681.CFkGhW.006.7154000014360.hdf"
],
},
"collectionsInCumulusCmr": {
"okCount": 0,
"onlyInCumulus": [
"MOD09GQ_test-test-data-1593122280799___006",
"MOD09GQ_test-test-data-1593122597517___006",
"MOD09GQ_test-test-data-1593122881446___006",
"MOD09GQ_test-test-data-1593178175944___006",
"MOD09GQ_test-test-data-1593198739991___006",
"MOD09GQ_test-test-data-1593198955197___006",
"MOD09GQ_test-test-data-1593199179316___006",
"MOD09GQ_test-test-data-1593199379860___006",
"MOD09GQ_test-test-data-1593199548454___006",
"MOD09GQ_test-test-data-1593199742089___006",
"MOD09GQ_test-test-data-1593199934835___006"
],
"onlyInCmr": [
"A2_RainOcn_NRT___0",
"A2_SI12_NRT___0",
"A2_SI25_NRT___0",
"A2_SI6_NRT___0",
"AST_L1A___003",
"CMR44JK___001",
"L2_HR_PIXC___000",
"L2_HR_PIXC___1",
"MCD43A1___006",
"MOD09GQ___006",
"MOD11A1___006",
"MOD14A1___006",
"MUR-JPL-L4-GLOB-v4.1___1",
"MYD09A1___006",
"MYD13A1___006",
"MYD13Q1___006",
"hs3avaps___1",
"hs3wwlln___1"
]
},
"granulesInCumulusCmr": {
"okCount": 0,
"onlyInCumulus": [],
"onlyInCmr": []
},
"filesInCumulusCmr": {
"okCount": 0,
"onlyInCumulus": [],
"onlyInCmr": []
}
}
{
"reportType": "Internal",
"createStartTime": "2020-08-28T19:01:50.636Z",
"createEndTime": "2020-08-28T19:02:08.211Z",
"reportStartTime": "2020-01-28T18:21:31.119Z",
"reportEndTime": "2020-08-28T18:21:31.119Z",
"status": "SUCCESS",
"collections": {
"okCount": 10,
"withConflicts": [
{
"es": {
"createdAt": 1591902043153,
"granuleId": "^.*$",
"sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5",
"dataType": "L2_HR_PIXC",
"name": "L2_HR_PIXC",
"files": [
{
"bucket": "private",
"regex": ".*.h5$",
"sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5"
}
],
"granuleIdExtraction": "^(.*)(\\.h5|\\.cmr)",
"reportToEms": true,
"version": "000",
"duplicateHandling": "error",
"updatedAt": 1591902043153,
"timestamp": 1591902043840
},
"db": {
"duplicateHandling": "error",
"granuleIdExtraction": "^(.*)(\\.h5|\\.cmr)",
"version": "000",
"dataType": "L2_HR_PIXC",
"files": [
{
"bucket": "private",
"sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5",
"regex": ".*.h5$"
}
],
"updatedAt": 1591902043153,
"createdAt": 1591902043153,
"reportToEms": false,
"granuleId": "^.*$",
"sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5",
"name": "L2_HR_PIXC"
}
}
],
"onlyInEs": [
"MCD43A1___006",
"MOD09GQ___006"
],
"onlyInDb": [
"MYD13QA___005"
]
},
"granules": {
"okCount": 77,
"withConflicts": [
{
"es": {
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:IngestAndPublishGranule:edb21d63-b62d-42e9-8879-f39d53314f16",
"processingEndDateTime": "2020-05-18T19:49:04.769Z",
"published": false,
"error": {},
"productVolume": 253949,
"timeToPreprocess": 0,
"duration": 1.005,
"createdAt": 1589831343764,
"granuleId": "MOD14A1.A4313371.RQSeyH.006.7795385650931",
"processingStartDateTime": "2020-05-18T19:49:03.831Z",
"provider": "s3_provider",
"timeToArchive": 0,
"files": [
{
"bucket": "cumulus-test-sandbox-internal",
"fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
"size": 233840,
"source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
"type": "data",
"key": "test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf"
},
{
"bucket": "cumulus-test-sandbox-internal",
"fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
"size": 14297,
"source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
"type": "metadata",
"key": "test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met"
},
{
"bucket": "cumulus-test-sandbox-internal",
"fileName": "BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
"size": 5812,
"source": "s3://cumulus-test-sandbox-internal/test-data/files/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
"type": "browse",
"key": "test-data/files/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg"
}
],
"collectionId": "MOD14A1___006",
"status": "running",
"timestamp": 1589831367756,
"updatedAt": 1589831344769
},
"db": {
"published": true,
"endingDateTime": "2017-11-08T23:59:59Z",
"status": "completed",
"timestamp": 1589831361039,
"createdAt": 1589831343764,
"processingEndDateTime": "2020-05-18T19:49:20.023Z",
"productVolume": 258297,
"timeToPreprocess": 0.263,
"timeToArchive": 0.942,
"productionDateTime": "2018-07-19T12:01:01Z",
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:IngestAndPublishGranule:edb21d63-b62d-42e9-8879-f39d53314f16",
"files": [
{
"bucket": "cumulus-test-sandbox-protected",
"key": "MOD14A1___006/2017/MOD/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
"size": 233840,
"fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
"source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
"type": "data"
},
{
"bucket": "cumulus-test-sandbox-private",
"key": "MOD14A1___006/MOD/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
"size": 14297,
"fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
"source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
"type": "metadata"
},
{
"bucket": "cumulus-test-sandbox-public",
"key": "MOD14A1___006/BRO/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
"size": 5812,
"fileName": "BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
"source": "s3://cumulus-test-sandbox-internal/test-data/files/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
"type": "browse"
},
{
"bucket": "cumulus-test-sandbox-protected-2",
"key": "MOD14A1___006/MOD/MOD14A1.A4313371.RQSeyH.006.7795385650931.cmr.xml",
"size": 4348,
"fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.cmr.xml",
"type": "metadata"
}
],
"cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1234771093-CUMULUS",
"processingStartDateTime": "2020-05-18T19:49:03.831Z",
"updatedAt": 1589831361039,
"beginningDateTime": "2017-10-24T00:00:00Z",
"provider": "s3_provider",
"granuleId": "MOD14A1.A4313371.RQSeyH.006.7795385650931",
"collectionId": "MOD14A1___006",
"duration": 17.275,
"error": {
"Error": "Unknown Error",
"Cause": "None"
},
"lastUpdateDateTime": "2018-04-25T21:45:45.524053"
}
}
],
"onlyInEs": [
{
"granuleId": "MYD13Q1.A0323210.Yujnv6.006.1973526114346",
"collectionId": "MYD13Q1___006",
"provider": "s3_provider",
"createdAt": 1589913461833,
"updatedAt": 1589913522990
}
],
"onlyInDb": [
{
"granuleId": "MYD13Q1.A5822875.qHI8hK.006.8698738069249",
"collectionId": "MYD13Q1___006",
"provider": "s3_provider",
"createdAt": 1598552797312,
"updatedAt": 1598552827611
}
]
}
}
{
"collectionIds": [
"MYD13Q1___006",
"MOD09GQ___006"
],
"createEndTime": "2022-02-11T19:13:41.986Z",
"createStartTime": "2022-02-11T19:13:41.153Z",
"granuleIds": [
"MYD13Q1.A3194547.tnYsne.006.8400707913298",
"MYD13Q1.A2655880.GOWVT9.006.3531712476486",
"MOD09GQ.A8858216.Y6HJnu.006.6168319936421"
],
"providers": [
"modas_provider",
"s3_provider"
],
"reportEndTime": "2022-02-11T19:12:25.000Z",
"reportType": "ORCA Backup",
"status": "SUCCESS",
"granules": {
"okCount": 0,
"cumulusCount": 2,
"orcaCount": 2,
"okFilesCount": 5,
"conflictFilesCount": 6,
"withConflicts": [
{
"okFilesCount": 4,
"granuleId": "MYD13Q1.A3194547.tnYsne.006.8400707913298",
"collectionId": "MYD13Q1___006",
"provider": "s3_provider",
"createdAt": 1644606673827,
"updatedAt": 1644606718024,
"conflictFiles": [
{
"fileName": "MYD13Q1.A3194547.tnYsne.006.8400707913298.cmr.xml",
"bucket": "cumulus-test-sandbox-protected-2",
"key": "MYD13Q1___006/MYD/MYD13Q1.A3194547.tnYsne.006.8400707913298.cmr.xml",
"reason": "shouldBeExcludedFromOrca"
},
{
"fileName": "BROWSE.MYD13Q1.A3194547.tnYsne.006.8400707913298.1.jpg2",
"bucket": "cumulus-test-sandbox-public",
"key": "MYD13Q1___006/BRO/BROWSE.MYD13Q1.A3194547.tnYsne.006.8400707913298.1.jpg2",
"reason": "onlyInCumulus"
},
{
"fileName": "BROWSE.MYD13Q1.A3194547.tnYsne.006.8400707913298.1.jpg",
"bucket": "cumulus-test-sandbox-public",
"key": "MYD13Q1___006/BRO/BROWSE.MYD13Q1.A3194547.tnYsne.006.8400707913298.1.jpg",
"orcaBucket": "cumulus-test-sandbox-orca-glacier",
"reason": "onlyInOrca"
}
]
}
],
"onlyInCumulus": [
{
"okFilesCount": 1,
"granuleId": "MYD13Q1.A2655880.GOWVT9.006.3531712476486",
"collectionId": "MYD13Q1___006",
"provider": "s3_provider",
"createdAt": 1644606673656,
"updatedAt": 1644606683360,
"conflictFiles": [
{
"fileName": "MYD13Q1.A2655880.GOWVT9.006.3531712476486.hdf",
"bucket": "cumulus-test-sandbox-protected",
"key": "MYD13Q1___006/2017/MYD/MYD13Q1.A2655880.GOWVT9.006.3531712476486.hdf",
"reason": "onlyInCumulus"
},
{
"fileName": "BROWSE.MYD13Q1.A2655880.GOWVT9.006.3531712476486.hdf",
"bucket": "cumulus-test-sandbox-private",
"key": "MYD13Q1___006/BRO/BROWSE.MYD13Q1.A2655880.GOWVT9.006.3531712476486.hdf",
"reason": "onlyInCumulus"
},
{
"fileName": "BROWSE.MYD13Q1.A2655880.GOWVT9.006.3531712476486.1.jpg",
"bucket": "cumulus-test-sandbox-public",
"key": "MYD13Q1___006/BRO/BROWSE.MYD13Q1.A2655880.GOWVT9.006.3531712476486.1.jpg",
"reason": "onlyInCumulus"
},
{
"fileName": "MYD13Q1.A2655880.GOWVT9.006.3531712476486.cmr.xml",
"bucket": "cumulus-test-sandbox-protected-2",
"key": "MYD13Q1___006/MYD/MYD13Q1.A2655880.GOWVT9.006.3531712476486.cmr.xml",
"reason": "onlyInCumulus"
}
]
}
],
"onlyInOrca": [
{
"granuleId": "MOD09GQ.A8858216.Y6HJnu.006.6168319936421",
"provider": "s3_provider",
"collectionId": "MOD09GQ___006",
"createdAt": 1643920768281,
"conflictFiles": [
{
"bucket": "cumulus-test-sandbox-protected",
"key": "MOD09GQ___006/2017/MOD/MOD09GQ.A8858216.Y6HJnu.006.6168319936421.hdf",
"fileName": "MOD09GQ.A8858216.Y6HJnu.006.6168319936421.hdf",
"orcaBucket": "cumulus-test-sandbox-orca-glacier",
"reason": "onlyInOrca"
},
{
"bucket": "cumulus-test-sandbox-public",
"key": "MOD09GQ___006/MOD/MOD09GQ.A8858216.Y6HJnu.006.6168319936421_ndvi.jpg",
"fileName": "MOD09GQ.A8858216.Y6HJnu.006.6168319936421_ndvi.jpg",
"orcaBucket": "cumulus-test-sandbox-orca-glacier",
"reason": "onlyInOrca"
},
{
"bucket": "cumulus-test-sandbox-protected-2",
"key": "MOD09GQ___006/MOD/MOD09GQ.A8858216.Y6HJnu.006.6168319936421.cmr.xml",
"fileName": "MOD09GQ.A8858216.Y6HJnu.006.6168319936421.cmr.xml",
"orcaBucket": "cumulus-test-sandbox-orca-glacier",
"reason": "onlyInOrca"
}
]
}
]
}
}
"granuleUr","collectionId","createdAt","startDateTime","endDateTime","status","updatedAt","published" "MOD14A1.A9506271.IvEJsu.006.8359924290786","MOD14A1___006","2020-05-18T20:15:54.525Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-05-18T20:16:02.473Z",false "MYD13Q1.A9663671.0zkwKH.006.9812354158395","MYD13Q1___006","2020-07-06T19:46:19.957Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-07-06T19:46:57.054Z",true
Create a new reconciliation report.
| parameter | value | required | description |
|---|---|---|---|
reportName |
string |
false |
Report name. |
reportType |
"Granule Inventory"|"Granule Not Found"|"Internal"|"Inventory"|"ORCA Backup" |
false |
Report type (default Inventory) |
startTimestamp |
string |
false |
Any input valid for a JS Date contstructor. Data older than this will be ignored in the generated report. |
endTimestamp |
string |
false |
Any input valid for a JS Date contstructor. Data newer than this will be ignored in the generated report. |
collectionId |
[string | array] |
false |
collectionId (or array of collectionIds) for comparison of collection and granule holdings. |
granuleId |
[string | array] |
false |
granuleId (or array of granuleIds) for use for comparison of collection and granule holdings. |
provider |
[string | array] |
false |
provider name (or array of providers) for comparison of granule holdings |
status |
string |
false |
status filter for Granule Inventory reports |
NOTE: Adding a startTimestamp or endTimestamp value to the POST request will result in one way comparisons for some fields for Inventory and Granule Not Found reports.
NOTE: Adding a granuleId input will result in an one way report for collections for Inventory and Granule Not Found reports.
NOTE: Inventory and Granule Not Found reports only allow one of the parameters (collectionId, granuleId, provider) in same request.
NOTE: Granule Inventory reports can be filtered with the following parameters: (collectionId, granuleId, status). If granuleId is a string or single value array, it will search for granules with granuleIds containing that substring.
$ curl --request POST https://example.com/reconciliationReports --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
"reportName": "ModisInventoryReport",
"reportType": "Inventory",
"startTimestamp": 1269993600000,
"endTimestamp": 1350000000000,
"collectionId": "MOD09GQ___006",
"provider": "MODIS",
"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"
}'
{
"id": "bb7059f4-0cd8-4205-8857-fb0e6b68b3e4"
}
Delete a reconciliation report from Cumulus.
$ curl --request DELETE https://example.com/reconciliationReports/inventoryReport-20180620T205838883 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"message": "Report deleted"
}
The Cumulus API can provide information about its configuration.
GET requests to the instance metadata endpoint return a json object with information about how the Cumulus stack is configured. It returns the CMR provider and environment as well as the stackName (prefix).
$ curl https://example.com/instanceMeta --header 'Authorization: Bearer ReplaceWithTheToken'
{
"cmr": {
"provider": "CUMULUS",
"environment": "UAT"
},
"cumulus": {
"stackName": "cumulus-stack-prefix"
}
}
Serve the dashboard from an S3 bucket.
This is a way to serve the Cumulus dashboard in a browser from an S3 bucket without making the bucket or files public.
To use this:
terraform.tfvars, otherwise you will not be able to access the bucket.example.com with your Cumulus backend API and dashboard-bucket with your bucket name.https://example.com/dashboard/dashboard-bucket/index.html
This endpoint authenticates and forwards requests to the ORCA private API, and returns the response from the ORCA API. Please refer to ORCA API reference on how to use ORCA API.
$ curl --request POST https://example.com/orca/recovery/granules --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
"granuleId": "MOD14A1.061.H5V12.2020312.141531789"
}'
Endpoint provides a mechanism for recovery of S3 sfEventSqsToDbRecords dead letter objects (created as described in the Core Documentation). The endpoint will invoke an async operation that will attempt to process all of the objects in the specified location.
The endpoint by default will process all records contained in the S3 objects from the default storage location on the S3 system bucket under the prefix of <stackName>/dead-letter-archive/sqs/. However, it is likely useful to process a subset of those objects, which you can do by moving the desired subset of objects to a new path on S3 and then specifying that new path using the path parameter to the endpoint request.
The query uses the following optional parameters:
| parameter | description |
|---|---|
| bucket | The bucket to read records from. Defaults to the Core system bucket |
| path | The S3 prefix (path) to read DLQ records from. Defaults to <stackName>/dead-letter-archive/sqs/ |
| batchSize | Specifies how many DLA objects to read from S3 and hold in memory. Defaults to 1000 |
| concurrency | Specifies how many messages to process at the same time. Defaults to 30 |
| dbMaxPool | Specifies how many database connections to allow the process to utilize. Defaults to 30. Process should at minimum the value set for concurrency |
curl -X POST https://cumulus.podaac.sit.earthdata.nasa.gov/deadLetterArchive/recoverCumulusMessages --header "Authorization: Bearer $TOKEN" --header 'Content-Type: application/json' --data '{
"bucket": "some-cumulus-bucket",
"path": "some/path/to/records/"
}'
{
"createdAt":1646861517957,
"updatedAt":1646861517957,
"id":"a0cd2cf0-a677-e82a-27d1-0a09271aa37d",
"status":"RUNNING",
"taskArn":"arn:aws:ecs:us-west-2:xxxxxxxxxx:task/stack-CumulusECSCluster/{SHA}",
"description":"Dead-Letter Processor ECS Run",
"operationType":"Dead-Letter Processing"
}