Introduction

The MISR Toolkit is a simplified programming interface to access MISR L1B2, L2, and MISR-HR, conventional and ancillary data products. It is an interface built upon HDF-EOS that knows about MISR data products. It has the ability to:

Specify regions to read based on geographic location and extent or the more traditional path and block range
Map between path, orbit, block, time range and geographic location
Automatically stitch, unpack and unscale MISR data while reading
Perform coordinate conversions between lat/lon, SOM x/y, block/line/sample and line/sample of a data plane, which means geolocation can be computed instantly without referring to an ancillary data set lookups
Retrieve pixel acquistion time from L1B2 product files
Read a slice of a multi-dimensional field into an 2-D data plane (eg. RetrAppMask[0][5])
Convert MISR product files to IDL ENVI files *Note: Recent versions of ENVI no longer support SOM projections

Users Guide and Interface Documents

The MISR Toolkit Users Guide gives a good introduction and is a good place to start
The MISR Toolkit Python Interface gives a good summary of the Python bindings
The MISR Data Production Specification describes all MISR data products

Function Summary

MISR Toolkit Routine Summary Table describes the core C interface
MISR Toolkit C Data Structure Summary describes the C data structures
MISR Toolkit IDL Function Index describes the IDL interface
MISR Toolkit Python Interface describes the Python interface
MISR Toolkit Command-line Utilities describes the command-line interface

Architecture and Language Support

CentOS 7 or later (64-bit)

C functions (source only)
IDL bindings (source only)
Python bindings (source only)

MacOS X 10.14.6 or later (Intel)

C functions (source only)
IDL bindings (source only)
Python bindings (source only)

Windows 10 (64-bit)

C functions (dll & source)
IDL bindings (dll & source - pre-built dll's require IDL 8.2)
Python bindings (dll & source - pre-built dll's require Python 2.7)

Availability

Source code and Microsoft Windows dll's are available at Releases on the MISR Toolkit GitHub

Third Party Library Dependencies

The MISR Toolkit depends on the following libraries. Download links are provided for your reference, but it is highly recommended that you use the script "download_libraries" in the source bundle "scripts" directory. Python users will still need to download NumPy 1.15 or later and install it according to NumPy instructions.

HDF-EOS2.18v1.00 Download
hdf4.2.10 Download
hdf5-1.8.16 Download
netcdf-4.4.0 Download
jpegsrc.v6b Download
zlib-1.2.5 Download
NumPy 1.15 or later Python module Download

Installation Instructions

Linux and MacOS X

The recommended location for the HDF-EOS/HDF libraries and the MISR Toolkit is /usr/local/hdfeoslibs and /usr/local/Mtk-1.5.X respectively, because this location provides convenient access for multiple user accounts. It is not necessary, however, to install in /usr/local. The MISR Toolkit and HDF-EOS can be installed anywhere. Of course installing in /usr/local requires root or sudo privileges. Use the sudo commands below if you are a sudoer or simply become root before installing the HDF-EOS/HDF libraries and before the "make install" step for the MISR Toolkit.

1) Create a working directory

mkdir Mtk_tmp
cd Mtk_tmp

2) Extract Mtk-src-1.5.X.tar.gz

tar xzvf Mtk-src-1.5.X.tar.gz (if not done already)

3) Extract Mtk testdata (substitute Mtk-testdata-1.5.x with your version)

tar xzvf Mtk-testdata-1.5.x.tar.gz

4) Download HDF-EOS/HDF, JPEG and ZLIB libraries

cd Mtk-src-1.5.X
scripts/download_libraries
Hit "return" for defaults

5) Install HDF-EOS/HDF, JPEG and ZLIB libraries (using the following script is necessary because it applies patches which are required for some systems, like Mac Universal Binaries, Linux and Linux64)

The next step requires root privileges to install into /usr/local/hdfeoslibs, otherwise choose another location and disregard sudo
sudo scripts/install_hdf+hdfeos
Hit "return" for defaults (/usr/local/hdfeoslibs)

6) Build, test and install Mtk (substitute Mtk-src-1.5.X with your version)

Setup your HDF/HDFEOS environment
- for csh:
  - source <your-step5-path>/bin/hdfeos_env.csh
- for bash:
  - source <your-step5-path>/bin/hdfeos_env.sh
Set MTK_INSTALLDIR environment variable (ex. /usr/local/Mtk-1.5.X)
- for csh:
  - setenv MTK_INSTALLDIR <your-path>/Mtk-1.5.X
- for bash:
  - export MTK_INSTALLDIR=<your-path>/Mtk-1.5.X
Set IDL_DIR environment variable (if applicable)
- for csh:
  - setenv IDL_DIR <path-to-idl>/harris/idl
- for bash:
  - export IDL_DIR=<path-to-idl>/harris/idl
To build, test and install (choose which libraries to build)
- Everything - C library, command-line utilities, IDL and Python
  - make
  - make testall (If you have Mtk testdata installed and would like to test C, IDL and Python)
  - The next step requires root privileges to install into /usr/local/Mtk-1.5.X, otherwise choose another location and disregard sudo
  - sudo make install
- Or - C library and command-line utilites
  - make lib
  - make cmdutil
  - make test (If you have Mtk testdata installed and would like to test only C routines)
  - The next step requires root privileges to install into /usr/local/Mtk-1.5.X, otherwise choose another location and disregard sudo
  - sudo make install
- Or - IDL library
  - make idl
  - make testidl (If you have Mtk testdata installed and would like to test only IDL routines - requires user interaction)
  - The next step requires root privileges to install into /usr/local/Mtk-1.5.X, otherwise choose another location and disregard sudo
  - sudo make install
- Or - Python library
  - make python
  - make testpython (If you have Mtk testdata installed and would like to test only python routines)
  - The next step requires root privileges to install into /usr/local/Mtk-1.5.X, otherwise choose another location and disregard sudo
  - sudo make install
Other build targets and options
- make help (for other build targets)
- make clean (to clean everything)
If the build complains about .d files (This occurs when the HDF-EOS environment is not set)
- scripts/rmdepend.sh (to remove dependency files)
- make

7) To use Mtk

The HDF/HDFEOS environment is already set above
Setup your Mtk environment (pick any):
- For csh:
  - source $MTK_INSTALLDIR/bin/Mtk_c_env.csh for C
  - source $MTK_INSTALLDIR/bin/Mtk_idl_env.csh for IDL
  - source $MTK_INSTALLDIR/bin/Mtk_python_env.csh for Python (Don't need to do if installed in site-packages (see step 8))
- For bash:
  - . $MTK_INSTALLDIR/bin/Mtk_c_env.sh for C
  - . $MTK_INSTALLDIR/bin/Mtk_idl_env.sh for IDL
  - . $MTK_INSTALLDIR/bin/Mtk_python_env.sh for Python (Don't need to do if installed in site-packages (see step 8))

For C examples: $MTK_INSTALLDIR/examples/C
For IDL examples: $MTK_INSTALLDIR/examples/idl
For Python examples: $MTK_INSTALLDIR/examples/python
For C command-line utilities source code: Mtk-src-1.5.X/src
For IDL tests source code: Mtk-src-1.5.X/wrappers/idl

8) Optional - You may want to install the MisrToolkit into you Python site-packages directory

cd Mtk-src-1.5.X/wrappers/python
sudo python setup.py install
setenv LD_LIBRARY_PATH $MTK_INSTALLDIR/lib

9) Optional - After installing the Mtk_tmp directory, it's contents is not needed, unless for reference and may be removed

cd ../..
rm -rf Mtk_tmp

Windows

See win32/README.txt in the source or binary bundle.

Documentation

Documentation Availability

HTML documentation for the supported interfaces is available in the doc directory.

Documentation Regeneration

The target to remake documentation is "make doc"
The C documentation, including this page, is created using doxygen. A doxygen config file template is at doc/Doxyfile.template

The Python documentation is generated using the python's mkhowto tool, which was removed from recent Python distributions and pdfTeX.
You must sepearely install pdfTeX (for example by installing the TeX Live distribution) before using make doc to rebuild python documentation.
Scripts have been provided to allow the download of a compatible distribution and extraction of the required files.

 
  $ cd $MTK_INSTALLDIR
  $ scripts/download_mkhowto
  $ scripts/install_mkhowto

You must then set PYDOCDIR environment variable (ex. ~/Mtk_tmp/Mtk_thirdparty/pythonDocTools)

for csh:
- setenv PYDOCDIR <your-path>/Mtk_thirdparty/pythonDocTools

for bash:
- export PYDOCDIR=<your-path>/Mtk_thirdparty/pythonDocTools

You should then be able to make doc successfully.

Examples

How to use the command-line utilities

        $ source <HDFEOSDIR>/bin/<ARCH>/hdfeos_env.csh
        $ source $MTK_INSTALLDIR/bin/Mtk_c_env.csh
        $ MtkLatLonToPathList --dd=-75.345,169.89
          Lat/Lon: -75.345000 169.890000
        Path list: 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224

        $ MtkOrbitToTimeRange 29058
        2005-06-04T17:41:17Z
        2005-06-04T19:20:10Z

        $ MtkMisrToEnvi --entire-file --hdffilename=../Mtk_testdata/in/MISR_AM1_GRP_ELLIPSOID_GM_P037_O029058_AA_F03_0024.hdf \
        > --gridname=BlueBand --fieldname="Blue Brf" --envifilename=out
        path = 37
        start block = 1
        end block = 140

How to build using core C MisrToolkit library

Compile, link and run (click foo.c, bar.c and biz.c for source code) or see examples/C/Makefile

        $ source <HDFEOSDIR>/bin/<ARCH>/hdfeos_env.csh
        $ source $MTK_INSTALLDIR/bin/Mtk_c_env.csh
        $ gcc $MTK_CFLAGS -c foo.c
        $ gcc $MTK_CFLAGS -c bar.c
        $ gcc $MTK_CFLAGS -c biz.c
        $ gcc -o baz foo.o bar.o biz.o $MTK_LDFLAGS
        $./baz

        Example 1:
        path = 37
        resolution = 275
        block, line, sample = 60,  256.0, 1024.0
        lat_dd, lon_dd = 38.130890, -110.848247
        lat deg, min, sec = 38:07:51.21
        lon deg, min, sec = -110:50:53.69
        b, l, s = 60,  256.0, 1024.0

        Example2:
        starttime = 2002-02-02T02:00:00Z
        endtime = 2002-05-02T02:00:00Z
        Pathlist = 35 36 37 38 39 
        Orbitlist for Path 35 = 11379 11612 11845 12078 12311 12544 
        Orbitlist for Path 36 = 11481 11714 11947 12180 12413 
        Orbitlist for Path 37 = 11350 11583 11816 12049 12282 12515 
        Orbitlist for Path 38 = 11452 11685 11918 12151 12384 
        Orbitlist for Path 39 = 11321 11554 11787 12020 12253 12486

        Example 3:
        region center lat/lon (dd) = (32.200000, -114.500000)
                 region extent (m) = (200000.000000, 100000.000000)

                      Reading file = /Users/ber/Development/Toolkit/Mtk/../Mtk_testdata/in/MISR_AM1_GRP_ELLIPSOID_GM_P037_O029058_AA_F03_0024.hdf
                      Reading grid = RedBand
                     Reading field = Red Brf
           number of lines/samples = (768, 384)
                input lat/lon (dd) = (32.200000, -114.500000)
                       line/sample = (383.293396, 191.933014)
                     Brf[383][192] = 0.357760
                 block/line/sample = (65, 210.292969, 413.932983)
                      lat/lon (dd) = (32.200000, -114.500000)
                     lat/lon (dms) = (32:11:59.999887, -114:30:0.000085)
                           SOM x/y = (16529918.187500, 307019.078125)
              pixel time[383][192] = 2005-06-04T18:21:45.415768Z

                      Reading file = /Users/ber/Development/Toolkit/Mtk/../Mtk_testdata/in/MISR_AM1_AS_LAND_P037_O029058_F06_0017.hdf
                      Reading grid = SubregParamsLnd
                     Reading field = LandHDRF[3][4]
           number of lines/samples = (192, 96)
                input lat/lon (dd) = (32.200000, -114.500000)
                       line/sample = (95.198349, 47.108253)
                      HDRF[95][47] = 0.208766
                 block/line/sample = (65, 52.198242, 103.108246)
                      lat/lon (dd) = (32.200000, -114.500000)
                     lat/lon (dms) = (32:11:59.999887, -114:30:0.000085)
                           SOM x/y = (16529918.184044, 307019.077821)

        Worked like champ!

See MISR Toolkit Routine Summary Table and MISR Toolkit Users Guide for more C examples

How to use the IDL MisrToolkit library

See examples/idl/filelist_by_timerange.pro

    $ source $MTK_INSTALLDIR/bin/Mtk_idl_env.sh
        $ idl
        IDL> help,/dlm
            ....
            MTK - MISR Toolkit (not loaded)
            Version: 1.2.0, Build Date: Nov 08 2006, Source: Jet Propulsion Laboratory
            ....
        IDL> filelist_by_timerange,'2005-01-01T00:00:00','2005-01-01T00:59:59'
              Start time = 2005-01-01T00:00:00
                End time = 2005-01-01T00:59:59
        Number of orbits =            2
              Orbit list =        26804       26805
               0
        orbit =        26804 path =           88
        MISR_AM1_GRP_ELLIPSOID_GM_P088_O026804_DA_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P088_O026804_CA_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P088_O026804_BA_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P088_O026804_AA_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P088_O026804_AN_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P088_O026804_AF_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P088_O026804_BF_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P088_O026804_CF_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P088_O026804_DF_F03_0023.hdf
        MISR_AM1_GP_GMP_P088_O026804_F03_0013.hdf
        MISR_AM1_TC_STEREO_P088_O026804_F07_0012.hdf
               1
        orbit =        26805 path =          104
        MISR_AM1_GRP_ELLIPSOID_GM_P104_O026805_DA_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P104_O026805_CA_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P104_O026805_BA_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P104_O026805_AA_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P104_O026805_AN_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P104_O026805_AF_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P104_O026805_BF_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P104_O026805_CF_F03_0023.hdf
        MISR_AM1_GRP_ELLIPSOID_GM_P104_O026805_DF_F03_0023.hdf
        MISR_AM1_GP_GMP_P104_O026805_F03_0013.hdf
        MISR_AM1_TC_STEREO_P104_O026805_F07_0012.hdf
        IDL> help,/dlm
            ....
            MTK - MISR Toolkit (loaded)
            Version: 1.2.0, Build Date: Nov 08 2006, Source: Jet Propulsion Laboratory
            ....

See MISR Toolkit IDL Function Index

How to use the Python MisrToolkit library

See examples/python/readmisr.py

    $ source $MTK_INSTALLDIR/bin/Mtk_python_env.sh
        $ python
        >>> import MisrToolkit as Mtk
        >>> import os

        >>> region = Mtk.MtkRegion(37,40,42)
        >>> filename = os.getenv('MTKHOME') + '/../Mtk_testdata/in/MISR_AM1_GRP_ELLIPSOID_GM_P037_O029058_AA_F03_0024.hdf'
        >>> gridname = 'RedBand'
        >>> fieldname = 'Red Brf'
        >>> d = Mtk.MtkFile(filename).grid(gridname).field(fieldname).read(region)
        >>> n = d.data()
        >>> n.shape
        >>> m = d.mapinfo()

        >>> type(d)
        >>> type(n)
        >>> type(m)

        >>> help(d)
        >>> help(n)
        >>> help(m)

See MISR Toolkit Python Interface

Known Problems

The IDL routine MTK_FIND_FILELIST() does not work properly.
By default OS X swallows click-events on change of focus.
- If IDL test windows on MacOS X 10.14 don't respond to clicks, do this command "defaults write org.x.x11 wmclickthrough -bool true"
Then, once at a shell prompt, restart X11/XQuartz and retry "make testidl".
IDL is not compatible with XQuartz versions greater than 2.7.9 If IDL tests return an error of "Error: attempt to add non-widget child "dsm" to parent "idl" which supports only widgets"" then you can either downgrade to XQuartz 2.7.9 or follow the procedure described at goo.gl/RvXOXy to add /opt/X11/lib/flat_namespace to the DYLD_LIBRARY_PATH set by your idl launcher script (e.g. /Applications/harris/idl87/bin/idl)