Modis Chlorophyll product command line creation steps

Modis Chlorophyll product command line creation steps

Command line is usually quicker to perform, and avoid any GUI issues I am having in SeaDAS 7.0. I corrupt window default setting with ease. Once I have repeatable command line steps, automation scripts are simple to create.

1. identify clear days via http://oceancurrent.imos.org.au/ (Australia)
2. Download from Ocean Color Level 1 and 2 Browser
1. if area  available in a single swarth download the L1A_LAC file
2. otherwise download L0_LAC file.
3. For L0_LAC files
• Concatenate
• cat A2011098040500.L0_LAC A2011098041000.L0_LAC > A2011098.L0_LAC
• result is A2011098.L0_LAC
• Create L1A_LAC
• modis_L1A.py A2011098.L0_LAC
• result is A2011098040500.L1A_LAC
4. Create GEO file for L1A file
• modis_GEO.py A2011098040500.L1A_LAC
• If input product result of concatenation consider if -r option is required.
• result
• A2011098040500.GEO
5. Extract area slightly large than area of interest. Further subsection performed later.
• modis_L1A_extract.py A2011098040500.L1A_LAC  -n -40 -s -45 -w 142 -e 147 --verbose --log
• results include
• A2011098045000.L1A_LAC.sub
• A2011098045000.GEO.sub
6. Create L1B file for area of interest
• modis_L1B.py A2011098sub.L1A_LAC 
• I renamed my files 
• A2011098sub.L1A_LAC
• A2011098sub.GEO
• Results include
• A2011098sub.L1B_HKM
• A2011098sub.L1B_LAC
• A2011098sub.L1B_QKM
7. Adjust the names in my l2gen par files

for example my gsm.par

ifile=A2011098sub.L1B_LAC
geofile=A2011098sub.GEO
ofile=gsm.hdf
l2prod="adg_443_gsm,bbp_443_gsm,chl_gsm"

      8. Run l2gen par=gsm.par

Only need 3 dates at the moment so using SeaDAS 7.0 GUI and ERMapper.

l2gen with giop

I asked this on the forum

l2gen giop algorithm error

My aim is to work with General IOP algorithm.

I am starting with basic parameter sets.

* l2gen with l2prod=chlor_a ran sucessfully
* l2gen with l2prod="a_443_giop,bb_443_giop,aph_vvv_giop,chl_giop" is unsuccessful

The end of the log is below.

----

Setting pure water aw and bbw for GIOP.
412 4.640989e-03 3.327229e-03
.
.
.
2130 2.512733e+03 2.768182e-06

dyld: lazy symbol binding failed: Symbol not found: __ZNKSt5ctypeIcE13_M_widen_initEv
  Referenced from: /Applications/seadas-7.0/ocssw/run/bin/macosx_intel/l2gen
  Expected in: /usr/lib/libstdc++.6.dylib

dyld: Symbol not found: __ZNKSt5ctypeIcE13_M_widen_initEv
  Referenced from: /Applications/seadas-7.0/ocssw/run/bin/macosx_intel/l2gen
  Expected in: /usr/lib/libstdc++.6.dylib


Advised to run otool

-bash: otool: command not found

Google response
otool command moved in Mountain Lion

I have worked with xcode before, just not since my last rebuild. Mucking around with Objective-C
--> Choose to install xcode 4.6.2

Used this post as a guide
i installed xcode but i still dont have gcc in my terminal

After completing the command tool install I tried l2gen again 
--> NO CHANGE

  otool -L /Applications/seadas-7.0/ocssw/run/bin/macosx_intel/l2gen 


2130 2.512733e+03 2.768182e-06
dyld: lazy symbol binding failed: Symbol not found: __ZNKSt5ctypeIcE13_M_widen_initEv
  Referenced from: /Applications/seadas-7.0/ocssw/run/bin/macosx_intel/l2gen
  Expected in: /usr/lib/libstdc++.6.dylib

dyld: Symbol not found: __ZNKSt5ctypeIcE13_M_widen_initEv
  Referenced from: /Applications/seadas-7.0/ocssw/run/bin/macosx_intel/l2gen
  Expected in: /usr/lib/libstdc++.6.dylib

Trace/BPT trap: 5

Suggested Disagnois

>uname -a

Darwin MacBook-Pro.local 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan  6 22:37:10 PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64

>otool -L /Applications/seadas-7.0/ocssw/run/bin/macosx_intel/l2gen 

/Applications/seadas-7.0/ocssw/run/bin/macosx_intel/l2gen:

/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)

/opt/local/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.17.0)

/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1094.0.0)

/opt/local/lib/gcc45/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

Alternative avenue explored. 

I did have some success when I ran it against another version of l2gen. This  version was from SeaDAS 7.0 beta, and I have performed a patch, as ftp://seadasuk.pml.ac.uk/seadas/seadas/updates/update_01/README.txt for macosx_intel.

otool -L /projects/SD/ocssw/run/bin/macosx_intel/l2gen

/projects/SD/ocssw/run/bin/macosx_intel/l2gen:

/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)

/opt/local/lib/gcc45/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.14.0)

WhiteG response

WhiteG response refers to 'port' command which I don't have. Checked out MacPorts software,  well beyond my usual unix dabbling. 

Investigated solution not requiring MacPorts to be downloaded.

Did not gain much insight. 

Installed MacPorts, unlike 'otool', the port command is not a diagnostic but a program. 

Ran

sudo port -v selfupdate

sudo port install gcc47

* l2gen with l2prod="a_443_giop,bb_443_giop,aph_vvv_giop,chl_giop" SUCCESSFUL

Build ceres, beam and seaDAS from source

Just a list of some issues

beam-core
fails tests if there is a space in the class path

beam-gpf 
can't find tools.jar, not in the expected location.
pom.xml or MacOSX profile wrong
I don't have java 1.7.0 installed

MODIS Aqua L2 Standard Product Notes

More of my adventures

Between 2 OS, multiple data formats and multiple software tools that metadata including to the names of layer/bands in the files is either getting removed or not displayed.

Blogger is accessible from both OS so I am writing some notes here.

Name	Index
aot 869	1
angstrom	2
Rrs 412	3
Rrs 443	4
Rrs 469	5
Rrs 488	6
Rrs 531	7
Rrs 547	8
Rrs 555	9
Rrs 645	10
Rrs 667	11
Rrs 678	12
chlor a (OC3)	13
Kd 490	14
pic	15
poc	16
cdom index	17
ipar	18
nflh	19
par	20
l2 flags	21
longitude	22
latitude	23

MODIS Aqua level 1A to Level 1B

Create GEO files

modis_GEO.py

I am using the command line even though I have SeaDAS-7.0 beta installed.

There are path issues going on for me 

/Users/user 1/Desktop/Project
modis_GEO.py MYD01.A2007265.0530.005.2013122114904.hdf 

/Users/user 1/Desktop/Project
modis_GEO.py MYD01.A2007044.0425.005.2013122114642.hdf

>dir

MYD01.A2007044.0425.005.2013122114642
MYD01.A2007044.0425.005.2013122114642.GEO

Appears that l2gen can utilise Level 1A SeaWifs but not modis.

Next 

Create Level 1B files

l1bgen

In 

/Applications/seadas-7.0-beta/Processing/run/bin/macosx_intel

there is 

l1bgen_modisa

In the seaDAS-7.0-beta GUI modis_L1b.py is used.

Standard Install has no window is opened when called

Personal SeaDAS built from GIT master branch 4 May 2013

On selecting modis_l1bgen error regarding input path. Possible because I have a space in the path. 

Move my data to a short path with no spaces and checked the permissions, 

Error message indicates a dislike for my input file name.

Shortening

MYD01A2007044.hdf

MYD01A2007044.hdf.GEO

MYD01A2007265.hdf

MYD01A2007265.hdf.GEO

Still getting error the null pointer exception is coming back to a file currently in flux on the development . Not worth pursuing, and I do not believe my system is not sufficiently standard to provide useful  provide feedback to the developer.

Back to command line in SeaDAS 7.0 beta processing module.

Having file names with multiple '.' is definitely an issue for the modis_L1b.py script

Trying the GUI once more after renaming

MYD01A2007044.GEO

MYD01A2007044.hdf

MYD01A2007265.GEO

MYD01A2007265.hdf

Still not able to run,

Using the names above with the command line. 

/projects/chldata>

modis_L1B.py MYD01A2007265.hdf 

MYD01A2007265.GEO

MYD01A2007265.L1B_HKM

MYD01A2007265.L1B_LAC

MYD01A2007265.L1B_QKM

MYD01A2007265.hdf

Sweet

SeaDAS 7.0 beta setup and configuration files

Further notes on my Adventures processing MODIS files

After starting SeaDAS 7.0 beta, via the icon, I opened a terminal window so that I could tail the log.

The first line was

/Users/Kaffkar/seadas/config/seadas.env: No such file or directory

I recently had to reinstall my operating system, as some windows on my bootcamp partition thought destroying the start software would be sensible.

The result is a slightly odd setup where I have one user and 2 user directories

performed a

find . -name seadas* 

and only the expected results were returned.

>env

TERM_PROGRAM=Apple_Terminal
TERM=xterm-256color
SHELL=/bin/bash
TMPDIR=/var/folders/kt/h45k9r0s2lv2x6yd7tn296vr0000gn/T/
Apple_PubSub_Socket_Render=/tmp/launch-jdUITF/Render
TERM_PROGRAM_VERSION=309
TERM_SESSION_ID=34319DE3-BBFF-416A-8949-1E4870045241
USER=Kaffkar
COMMAND_MODE=unix2003
SSH_AUTH_SOCK=/tmp/launch-YsRd2W/Listeners
__CF_USER_TEXT_ENCODING=0x1F5:0:0
Apple_Ubiquity_Message=/tmp/launch-MQI2sO/Apple_Ubiquity_Message
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
PWD=/Users/Kaffkar 1
LANG=en_AU.UTF-8
SHLVL=1
HOME=/Users/Kaffkar 1
LOGNAME=MacGill
_=/usr/bin/env

>java -version

java version "1.6.0_43"

Java(TM) SE Runtime Environment (build 1.6.0_43-b01-447-11M4203)

Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01-447, mixed mode)

>l2gen

-bash: l2gen: command not found

I am only a casual unix user but it appears both SeaDAS Gui may have been installed correctly but not the SeaDAS 

I have been referencing

HOW TO INSTALL SeaDAS 6.4
Creating True Color RGB in SeaDAS 7.0
How do I make true-color images from MODIS or SeaWiFS

2 years ago I was a Senior Java Developer, and utilised Maven to build, test and create packages so the files in GitHub actually make some sense.

SeaDAS GUI Source

my .bashrc file contains the reference

source /Users/Kaffkar/seadas/config/seadas.env

This needs to be change.
Trying this.



export OCSSWROOT=/Applications/seadas-7.0-beta/Processing
source /Applications/seadas-7.0-beta/Processing/OCSSW_bash.env

Closed all terminals and the SeaDAS Gui
Restarted SeaDAS
Opened a terminal

>l2gen

Command recognised

Tried l1bgen through GUI. No luck, at least the command line is working
Tried l1brsgen through GUI

>l1brsgen ifile=MYD021KM.A2003359.1225.005.2010067090806.hdf ofile=MYD021KMrgb.ppm geofile=MYD03.A2003359.1225.005.2010067073507.hdf subsamp=4 outmode=2 atmocor=1

This worked, now I just have to work out what it did.

SeaDAS 7.0-beta

My machine

OSX 10.8.3
Processor 2.2 GHz Intel Core i7
Memory 4 GB 1333 MHz DDR3

Install of SeaDAS 7.0-beta
Online Install

• Base files
• SeaWiFS
• Modis Aqua

Tips and Tricks

• Directories are not relative, after install so don't move the install around

Updates

I was experience so GUI problems, one suggestion was that SeaWiFS module needed to be installed for l2gen windows to configure.

Downloaded everything to ensure this was not the problem.
Still having issues. They are not consistent so have reported a bug yet.

Atmospheric Correction

Another post about my adventures with MODIS data

MODIS aqua appear to have at least 3 post atmospheric correction product

LW_nnn : The water-leaving radiance.

nLw_nnn : Normalised water leaving radiance

Rrs_nn remote sensing refectance

A200xxxxxxxxxxx.L2_LAC_OC

includes these Remote sensing reflectances

Rrs 412 --> Band 8

Rrs 443 --> Band 9

Rrs 469 ---> Band 3

Rrs 488  --> Band 10

Rrs 531 --> Band 11

Rrs 547 --> Band 12

Rrs 555 --> Band4, Band 12

Rrs 645 --> Band 1

Rrs 667 --> Band 13

Rrs 678  --> Band 14

Rrs_nnn = nLw_nn/ F0_nnn;

F0_nnn = extraterrestrial solar irradiance (constant provided in metadata )

Think I need a greater understanding of data available and the processing so I can make reasonable choices.

MODIS Aqua RGB with SeaDAS 7.0 Beta

Another post recording my adventures with MODIS data

Aim :  Create true colour RGB image from MODIS Level 1B

MODIS Level 1B

• Calibrated
• Not atmospherically corrected

Why 

• Atmospheric Correction required for Chlorophyll Algorithms
• RGB image required in write up

Download data Reverb NASA

• MYD021KM.A2003359.1225.005.2003360215150.hdf
• MYD02HKM.A2003359.1225.005.2003360215150.hdf
• MYD02QKM.A2003359.1225.005.2003360215150.hdf
• MYD03.A2003359.1225.005.2003360210518.hdf

• MYD021KM - reflective bands at 1km
• MYD02HKM - reflective bands at 500m.
• MYD02QKM - reflective bands at 250m.
• MYD03 - geolocation’ information required by MODIS Oceans Level 2 
  (i.e., latitude and longitude) for any remapping. 

I know all the above products can be generated from MODIS Level 1A, today I am being trying to be directed. and shall not investigate that area.

Interest link discussing how the different resolution reflective product work with the 1km resolution geolocation file. iCare MODIS_geolocation wiki

Tried to run l2gen from SeaDAS GUI -->  Issue
Known and Fixed Issues

This should have been fixed I believe.

Reinstalled SeaDAS with all modules, --> Issue
Bugger will have to understand the command line.

Back one step
Let's try l2brgen - the gui is throwing NullPointerExceptions. in the good old AWT Event Queue

DId manage to reproject through the GUI which something

WIMsoft trial

A post regarding my adventures with MODIS data

Summary

• Saw images in Window environment
• unable to achieve goal with WIM evaluation software
• WAM required to handle atmospheric correction

---

Aim :  Create true colour RGB image from MODIS Level 1B

MODIS Level 1B

• Calibrated
• Not atmospherically corrected

Why 

• Atmospheric Correction required for Chlorophyll Algorithms
• RGB image required in write up
• Familiarization with WIM software

Exercise : Exercises_modis_250m.pdf

Download data Reverb NASA

Create a control file
Control specifies

• input directory , data file 1 line
• input files, in this case, 4 lines
• flags indicating bands to process, 1 line
• output direction, line 1

Specifies the band to process

Selecting Dates

The west coast of tasmania is often cloudy.

Finding dates with little cloud cover was necessary.

MODIS Chlorophyl

Using knowledge about upwelling and the low resolution data available here I was able to select 2 dates approximately 6 months apart.

Once I selected a date, I downloaded the A200xxxxxxxxxxx.L2_LAC_OC.bz2. This has all the band I am interested and a bit more.

Now I need to investigate

• if and how the atomsphere has been accounted for.
• reporjection

I have chosen to use MODIS (Moderate Resolution Imaging Spectroradiometer) Aqua Data

http://modis.gsfc.nasa.gov/

Why

• Publicly available
• Temporal and Spatial resolution
• Sensor bands appropriate for ocean colour
• Range of algorithms

MODIS  Band Summary

8	405–420	1000m	Ocean Color/ Phytoplankton/ Biogeochemistry
9	438–448	1000m
10	483–493	1000m
11	526–536	1000m
12	546–556	1000m
13	662–672	1000m
14	673–683	1000m
15	743–753	1000m
16	862–877	1000m

I have reviewed some algorithms for chlorophyl, the following 7 bands were utilised, and I will need to download the data.

9	438–448	B
10	483–493	B
11	526–536	G
12	546–556	G. Mention of re-centring, not certain how this will effect me
13	662–672	R
14	673–683	R aligned with Band 13. Further research required to understand
15	743–753	NIR

Band 13 /Band 14 

Installing QGIS 1.8.0-2 on Mountain Loin

Recently rebuilt my macbook pro,. Today I am installing QGIS . This is how I installed it. Perhaps there are better ways.

QGIS Mac OSX

Sites I referenced

Eospansite Installing qgis 1.6on macosx 
(Version 20130108)
Eospansite Installing gdal on macosx


QGIS installation requires the command line. It and it's dependencies are not packaged up in a single dmg.

Download from kyngchaos

• QGIS-1.8.0-2.dmg
• Dependencies
• GDAL_Complete-1.9.dmg
• GSL_Framework-1.15-2.dmg
• Optional
• GRASS-6.4.2-5-Snow.dmg
• dependencies
• FreeType_Framework-2.4.10-1.dmg
• PIL-1.1.7-4.dmg
• cairo_Framework-1.12.2-1.dmg
• R-2.15.3.pkg (local mirror)

Install order

• GDAL Complete
• Both gdal.pkg and numpy.pkg
• GSL Framework
• FreeType Framework
• Cairo Framework
• PIL
• R
• GRASS
• GRASS and GRASS GDAL plugin
• QGIS

A chapter in this book was referenced by a paper I read. 

It is a good intro to different chlorophyl algorithms (Sometimes call products)

It is in the Unimelb ERC library

Advances in environmental remote sensing : sensors, algorithms, and applications

edited by Qihao Weng.

628.0284 ADVA

I am entering my final semester of a a Master's program.

The main component is a 6 month research project.

Marine phytoplankton are the base of the marine food web and play a key role in the ocean carbon cycle.  These single cell plants found in the upper layers of the global oceans.
The main factors affecting phytoplankton growth are sunlight, water movement, temperature and nutrients. Phytoplankton, in turn impacts all marine trophic levels, atmosphere carbon levels and ocean acidity. Monitoring and quantifying phytoplankton assists scientists in understand, and model, marine ecosystems and climate change.

The scale of the oceans, means in-situ monitoring, with the temporal and spatially density required for statistically significance is not feasible. Satellite remote sensing can provide this level coverage, a number of ocean colour sensors, such as MODIS, MERIS, and SeaWiFS have been deployed.

Chlorophyl, is a suitable measurement indicator of the amount of phytoplankton, in the ocean. Algorithms which model chlorophyl level in open water (Class 1) from ocean color sensor are have been established. These algorithms do not perform well in optical complex water and coastal water. Optically complex water contains active sediment and nutrients components, which contribute to response in the same wavelength range as the chlorophyl, utilise in the open water algorithms.