2009-08-12: SeaWiFS Processing and Data Analysis

From Datafedwiki

Jump to: navigation, search

Back to DataFed Development Events

This page will discuss the process of learning how to use SeaWiFS including the data acquisition process, preprocessing, and data analysis. In addition, discussion on the background of satellite measurements will be covered.

Contents

[edit] Instructions for Acquiring and Preprocessing Data

The instructions from Varun Yadav's report on SeaWiFS Data Processing are straightforward, but require some extra information for new people to smoothly transition into SeaWiFS data acquisition and preprocessing. Once I have successfully preprocessed a batch of files, I will rewrite these instructions.

SeaWIFs data is processed using the software ENVI 3.5

[edit] Basic online SeaWIFs data ordering process

  1. Make sure you have applied for authorization to order data from http://oceancolor.gsfc.nasa.gov/cgi/apply.pl?page=du
  2. Go to the website: http://oceancolor.gsfc.nasa.gov/
  3. Under "Data Access" click on "Level 1 and 2 Browser"
  4. On the upper left corner of the screen click on "TC" to turn on true color instead of the chlorophyll view
  5. Select the time period of interest (usually, order a month of data at a time). You can choose a time period by clicking on the months in the “Mission” table or you can customize your time period by clicking on the date directly above the world picture and inputting a time period.
  6. Uncheck the MODIS box in the upper left table and check the LAC and MLAC boxes. If you change your time period of interest the check boxes will return to MODIS and uncheck LAC and MLAC.
  7. Specify the boundary conditions on the far right side of the page as: N: 55.0, S: 25, E: -65, W: -160
  8. Click on "find swaths" below
  9. Wait for images to load (it may take a while)
  10. Select the swaths by clicking on the series of asterisks (****) above the images. Upon clicking, the (****) will become a “Yes” highlighted in green. One may click on the image to check the region it covers and the quality of the image. Desirable swaths are ones that cover a large portion of the US. If the large swaths miss the coasts, ordering a swath that does not cover much land area but gives information about the coasts is not a bad idea. Also, order a swath that is past the Western US and in the Pacific if possible. This extra data is necessary for the Rayleigh correction.
  11. Click on "Order Data" at the top of your screen
  12. Fill in your login information
  13. Check the box "Do extract" so that the data is extracted according to the indicated coordinates shown (this extraction makes the file sizes smaller and slightly more manageable)
  14. Check the box next to Level 1
  15. Uncheck final 3 boxes at the bottom at your discretion
  16. Click on Review Order
  17. Click on Submit Order
  18. You will get an email in your mailbox in a few moments prior to the order, reply back to this email as soon as you get it to begin the staging process.
  19. After all the data has been staged you will receive a notification email and a link to the ftp site

[edit] Download and Extraction of Data

The data can be downloaded from the staging ftp site either directly through internet explorer or mozilla or through FTP from the command line (instructions are available in the notification email) Place all the files along with bzip2.exe in a folder. Access the folder from command line and unzip the files (command bzip2.exe -d *.bz2) to obtain the HDF files.

[edit] Using FTP

I tried using the SQLbox for my FTP downloads, but the host kept closing the connection. I think the downloads were going too slowly on that machine. On other computers though, the downloads are very fast (around a minute per bz2 file). I will use a flash drive to transfer the downloads to the SQLbox.

  • Update: The SQLbox downloads are no longer slow. No need to use a flash drive. I ended up mapping a network drive to the SQLbox from another computer. From the other computer, I download the data directly into the SQLbox through the network drive. I think this is a little easier on the SQLbox, so most of its resources can go into processing the pictures.

The Ocean Color FTP server requires users to download data using their FTP client in passive mode. Windows does not support this in its command line, so you need another program to download data using FTP. NcFTPis a useful FTP client that will solve this issue. Here is an example of how to use NcFTP to download a batch of data:

  1. Open ncFTP
  2. Choose a directory you want the files to be downloaded to using the “lcd” command. The default directory is the Desktop. You can make a folder on the Desktop and download the data into that folder by typing in “lcd Folder name”. You can also change directories into a network drive by typing "lcd Y:". This will change directories to the Y drive.
  3. Connect to the ocean color FTP site using the “open” command. For example: open ftp://ocdist6.sci.gsfc.nasa.gov.
  4. Check if passive mode is on using the “passive” command. The command will return an on or off value.
  5. Change directories on the FTP site to the desired folder (requested_files) using the “cd” command. Example: “cd 50e000b0948aa947” followed by “cd requested files”
  6. Download one file using the “get” command or multiple files using the “mget” command. To download all files type in “mget *.*”.


Here is an example session of downloading a batch of files.

NcFTP 3.2.3 (Jul 28, 2009) by Mike Gleason (http://www.NcFTP.com/contact/).

ncftp> lcd September2008

ncftp> passive

passive off

ncftp> passive

passive on

ncftp> open ftp://ocdist6.sci.gsfc.nasa.gov

Resolving ocdist6.sci.gsfc.nasa.gov...

Connecting to 169.154.128.62...

192.168.2.56 FTP server ready

Logging in...

Anonymous access granted, restrictions apply

Logging in...

Logged in to ocdist6.sci.gsfc.nasa.gov.

Current remote directory is /.

ncftp / > cd 50b020d0a383a748

ncftp /50b020d0a383a748 > cd requested_files

ncftp ...83a748/requested_files > ls

S2008118183159.L1A_MLAC.x.hdf.bz2 S2008118214959.L1A_MLAC.x.hdf.bz2

S2008118201037.L1A_MLAC.x.hdf.bz2 S2008118232943.L1A_MLAC.x.hdf.bz2

ncftp ...83a748/requested_files > mget *.*

S2008118183159.L1A_MLAC.x.hdf.bz2: 24.43 MB 1.31 MB/s

S2008118201037.L1A_MLAC.x.hdf.bz2: 25.53 MB 1.15 MB/s

S2008118214959.L1A_MLAC.x.hdf.bz2: 25.44 MB 1.26 MB/s

S2008118232943.L1A_MLAC.x.hdf.bz2: 14.31 MB 1.31 MB/s

ncftp ...83a748/requested_files >

[edit] Preprocessing Data using ENVI 3.5

Preprocessing of the data involves geometric corrections, splicing, cropping, scattering angle normalization and rayleigh correction of the data files.

The following steps need to be followed to obtain the processed files.

  1. Starting the License manager
    1. If the computer has restarted, then the license manager needs to be restarted too for ENVI to be operational. Open LMTools located in Start\Programs\RSI ENVI 4.3. Locate the "start/Stop/Reread" tab and click on "Start Server". Note that even though you are using the license manager from ENVI 4.3, you will be using ENVI 3.5 for the preprocessing.
  2. Open ENVI 3.5 from the start menu. IDL will open with it.
  3. Select Seawif -> Preprocessing
  4. Specify the directories for all of the outputs. The defaults are
Output Directory C:\SeaWiFS\Preprocessing\FinalPreprocessed
Temp Geometric Output Directory C:\SeaWiFS\Preprocessing\tmpGeometry
Temp L1b Output Directory C:\SeaWiFS\Preprocessing\tmpGeoreferenced
Temp Points Output Directory C:\SeaWiFS\Preprocessing\tmpWarp_points
Temp Geometry Output Directory C:\SeaWiFS\Preprocessing\Georef_Geom
Temp Spliced L1b Output Directory C:\SeaWiFS\Preprocessing\Spliced
Temp Rayleigh Corrected Output Directory C:\SeaWiFS\Preprocessing\Raycorr
JPEG Output Directory C:\SeaWiFS\Preprocessing\JPEG

5. Specify the boundary conditions (For US, N: 52.0, S: 24, E: -65, W: -160) and click OK.
Note: Specifying the correct boundary conditions is essential to the rest of the process. If these coordinates are wrong, you will have to rerun them, which is extremely time consuming. One way to check if the coordinates are correct is to look at the dimensions of the JPEG files. As of September 28, 2009, 3600 x 1680 is the size of the JPEGs that we use.

6. Browse to the location of extracted HDF files and select all the files together by holding the SHIFT key and clicking on the first and last file (files for the entire month can be given for analysis at the same time).
Note: The HDF files must be in the SeaWiFS directory when you preprocess them. If they are not, IDL will give an error message stating the HDF files are invalid.
7. A new window pops up for the calibration file. Browse to C:\Seawifs and enter the calibration file SEAWIFS_SENSOR_CAL.TBL.
8. Another new window opens for elevation file. Browse to C:\Seawifs\Elevation Data and enter the elevation file NAmDEMReplacedValue.

Processing time varies over 40 to 120 min for each day.

[edit] Process to Preprocessing

[edit] 2009-08-11

  • Went to the Ocean Color web page to order data
  • Found data and applied for authorization to order it

[edit] 2009-08-12

  • Downloaded data. Something was wrong with the FTP download, so I used Mozilla and downloaded each file one by one. I do not recommend doing this because it just takes too long. Hopefully I can figure out what was wrong with the FTP download, so I can download everything at once. I will write instructions for that if I can get it to work.
  • Extracted the data
  • Tried using ENVI and IDL to preprocess the data, but an error saying the HDF files were invalid came up. I am unsure why this message has come up. Erin and I suspect the satellite was not fully operational and may have uploaded corrupt files. I ordered data that is currently on the SQLbox and is working to see if that will work. I am hoping to start the preprocessing ASAP.

[edit] 2009-08-13

  • I am preprocessing a day of data from April 2008. I don't know why this day is working. After looking at all of the HDF files that were working from Varun's work, I noticed that they were all bigger in size than the ones I downloaded. I decided it was worth a shot, so I found the biggest HDF files I could find from April 2008 and tried running those. It worked! Could it be a size issue? Maybe the invalid HDF files are incomplete? Maybe that is why they are smaller than the ones that work. I need to test this out a little more, but I am glad I got at least one day to work.
  • I rewrote the instructions that Varun wrote. They are a bit more thorough I think and hold the reader's hand a little more. If anyone else would need to do this, I don't think they will have much trouble. Additions to the instructions will include the definitive cause of the preprocessing problems and how to download the ordered files using FTP.
  • I think I found my mistake. As embarrassing as it is, I think I input West as -65 and East at -125. This might be the reason behind the small HDF files and why the computer is sending so many errors. The correct bounding box is West -125 and East -65. I should probably redo April 2008 now.

[edit] 2009-08-17

I processed my first picture! Unfortunately it did not crop/splice correctly. Varun's report says that he increased the bounding box to -160 West, but that did not work either. I wonder if he ordered the data with the increased bounding box. I think if that's the case then I will just always order data at -160 West.

The first picture with the 55, -120, -65, 25 Bounding box

The second picture with the 55, -160, -65, 25 Bounding box

[edit] 2009-08-18

3 more days from April 2008 are in. The cropping/splicing worked for these! However, there seems to be a problem with the Rayleigh correction for the Western US. I am happy to see a continuous conterminous country though.


[edit] 2009-08-25

Preprocessing is up and running. 2 days are now preprocessed, and they look much better than the previous ones. The extension of the bounding box to -160 west seems to have corrected the Rayleigh scattering correction errors. There is an undesirable portion outside of the US, however. Obviously, setting the bounding box to -160 west on ENVI should include that region. Will the fix still work if I set the bounding box on ENVI to -125? Varun said he remembers setting ENVI's bounding box to -160 and Rudy said Varun had a file that chopped the extra portion of the West US out.




Specifying the ENVI bounding box to -125 seems to have worked well. The Rayleigh scattering correction seems to be working and there is no splicing. These days are 2 of the days run previously in April.


[edit] 2009-08-28: April 2008 on Vigor

The first month that I have successfully preprocessed is now on vigor for everyone to see. April 2008 preprocessed pictures look good. Some pictures have holes in them because of missing data while other pictures are a little slanted. The slanted pictures still have the data though. One picture that I am worried about is April 25, 2008. There seems to be a portion of the Southwest US that does not look right.


In addition to the pictures being available on vigor, a list of preprocessed pictures and the "Final Preprocessed" ENVI Standard files is being updated every hour. This file will let us know the preprocessing is working.

[edit] 2009-09-02: Bad Pictures

September 2008 did not go as well as I would have liked. 9 pictures came out that did not look right to me:

I looked at the data, and I think days 257, 260, and 263 should be rerun. This might fix them. The other days look similar to the HDF files, but I do not know why the bright streak appears on 257, 260, and 263. Picture 254 spliced florida incorrectly. After looking at the data, I see that clouds are completely covering Florida and the swaths do not cover very much of Florida, so the algorithm may be having trouble with that.

[edit] 2009-09-14

The first round of SeaWiFS preprocessing is complete. There are days where the data did not come out right, but I will try reordering the data and rerunning them to see if they can be fixed. Also, There are some days with missing swaths. I will check them to see if I possibly missed some swaths or the computer did not process them.

[edit] Data Coverage

The minimum images for some of the days still have cloud pixels. These arise from large gaps in data where the only pixels available for the processing are cloud pixels. This section will document the days we do have, the days with missing swaths, and the reasons why some data are missing.

[edit] 2008 Data Coverage

The following days are available for 2008:

  • 95-118
  • 122-128
  • 130-145
  • 147- 149
  • 151-165
  • 167-170
  • 172
  • 175-178
  • 182
  • 232-274
  • 277-366

The following days are missing:

  • 001-094
  • 119-121
  • 129
  • 146
  • 150
  • 166
  • 171
  • 173-174
  • 181
  • 183-231
  • 275-276

[edit] 2009 Data Coverage

The following days are available for 2009:

  • 1-113
  • 168-174
  • 176-182
  • 199-210
  • 212-231

The availability will change as more days are available on the ocean color website.

The following days are missing:

  • 114-167
  • 175
  • 183-198
  • 211
  • 242-365

[edit] Available Days Missing Swaths

The following days are available but missing swaths:

Day Swath Missing
2008095 East half of US
100 Western US
116 Texas area
128 Midwest + West
143 Everything East of California
144 East half of US
145 Texas area
151 Eastern half of US
167 Midwest + East
170 Texas area
172 Midwest + East
182 Texas area + East
232 East half of US
241 Texas area
257 Far East coast
260 East
274 West
277 Midwest + East
289 East
306 East
326 West
327 Western Midwest
328 West
329 Western Midwest
333 Midwest + West
334 Midwest + West
338 Midwest
340 Eastern Midwest
343 Midwest
344 West half of US
349 West half of US
350 West
352 West
353 Midwest
356 Western Midwest
358 Midwest
360 East
2009 007 Midwest + East
008 West
009 West + Midwest
011 East
038 East
097 East
176 Midwest + East
182 West Half
199 East half
200 East
201 Midwest
212 Midwest + East
213 West Coast + Texas area

From the missing swaths and data gaps, we can expect minimum images to be lacking in quality for Julian days 2008 166 - 231. Images from 183 - 231 will be extremely inaccurate since no new data is available for the processing algorithms to compute.

Images from 2009 114-167 will be inaccurate since this month is missing.

Julian days 2008 143 - 151 may be inaccurate because of the 4 days of missing swaths.

Julian days 2008 326 - 360 may be inaccurate because many contain missing swaths.

Julian days 2009 007 - 011 may be inaccurate because of the 4 days of missing swaths.

[edit] Data Outages

The SeaWiFS satellite did experience some downtime for various reasons.

On January 4, 2008, the satellite was experiencing a telemetry anomaly. Operations continued on April 9, 2008.

On July 2, 2008, the satellite was not in imaging mode. It returned to routine operation on August 20, 2008.

April 24 - June 16 2009 have no data. No forum posts on the data outage could be found.

[edit] List of Days to Rerun

These are days that have bright streaks in them or have missing data that can be filled in. There is also a blur effect on some of the 2009 days (218, 225, 228). Also, many of the 2009 days are missing the west coast. According to the Ocean Color website, California stations were having trouble with some of the data. I will have to look every once in a while to see if the data is available.

The method used to get rid of these streaks is to process these pictures without the swath containing the blue streak (or blur).

Number Picture Swath Removed
1 2008 285 S2008285172545
2 2008 289 S2008289182946
3 2008 257 S2008257181836
4 2008 260 This picture was not bad during the rerun
5 2008 263 S2008263172535
6 2008 306 S2008306182519
7 2008 326 S2008326220240
8 2008 327 S2008327210425
9 2008 328 S2008328214450
10 2008 329 S2008329204632
11 2008 333 S2008333201048
12 2008 334 S2008334222954
13 2008 338 S2008338201526
14 2008 340 S2008340195732
15 2008 343 S2008343201956
16 2008 344 S2008344210019, S2008344223901
17 2008 349 S2008349210448
18 2008 350 S20083500214510
19 2008 352 S2008352212711
20 2008 353 S2008353220732
21 2008 356 S2008356205112
22 2008 358 S2008358203314
23 2008 360 S2008360183335
24 2009 008 S2009008212624
25 2009 010 S2009010224700
26 2009 038 S2009038183251
27 2009 066 S2009066173654
28 2009 097 S2009097184339
29 2009 200 S2009200185328
30 2009 218 S2009218174054
31 2009 222 S2009222185229
32 2009 225 This picture was not bad during the rerun
33 2009 227 S2009227185648
34 2009 228 This picture was not bad during the rerun
35 2008 342 S2008342225658
36 2009 239 S2009239184729

[edit] Minimum Image Processing

To run the minimum image processing,

  1. Select the create minimum image option from the SeeWiFS tab in ENVI 3.5
  2. Select the Angle Corrected files you want to process
    • Note: These files do not need to be in the SeaWiFS directory like the preprocessing files do. They can be anywhere.
    • Warning: Make sure no other files except Angle Corrected files are in this directory. This includes folders.
  3. Choose the output directories of all files (use defaults)
    • Note: Make sure these output directories are empty. FinalMinImage, JPEG, and enlarged cloudmasks folders do not need to be empty, but all others involved in the processing should be empty before you start.

Even if you give the wrong coordinates for the Angle Corrected preprocessing, the minimum image processing will still work. However, you will not be able to do calculations with past data since the data are not correlated in any way.

[edit] Changes to Code

The minimum image processing had a few problems throughout the running. The following code changes were made to fix the problems:

Interactive_SeaWiFS_MinImage:

lastDayNeeded = long(dates[total_days-1])+time_window

Changed to

lastDayNeeded = long(unique_days[total_days-1])+time_window

Line 180

dates[i]

Changed to

unique_days[i]

create_anchor_points_smr:

Added line after dates_used=daylist_tile[j,*]

if max(daylist_tile) EQ 0 then CONTINUE

[edit] Running AOT Images in ENVI 4.6

Running the AOT images is the shortest run of the 3 processes. It is also run in a newer version of ENVI.

  1. Open ENVI 4.6 (Not 3.5)
  2. Select Seawif -> Create AOT Images
  3. Choose if you want the median filter applied (0 or 1). Usually, you will want to type in 1.
  4. Select the daily reflectance files for processing. These are the ENVI AngleCorrected files created from the preprocessing (i.e. not the jpegs).
  5. Select the minimum image directory that contains the minimum image ENVI files corresponding to the AngleCorrected days
  6. Select the destination directory for the ENVI files that are going to be generated. Seawifs_AOT/AOT_Med_1000 is the directory I chose.
  7. Select the jpeg directory for the jpegs that are going to be generated. A folder with the year of the data as the name is a good place.

[edit] Data Organization

The data stored for the preprocessing is a bit scattered. We need to standardize the method in which we store the files. Right now, organizing the files by parameter seems like a good method. More details need to be worked out though.

The following sections will discuss what data are on the four external harddrives we have currently, what is missing, and what the future could look like in terms of organizing this data. The sections refer to the labels placed on each of the harddrives. A text file of the directories including subdirectories of each harddrive is linked.

[edit] Reflectance 1999-2003

Contains a folder with elevation data. One elevation file that has the same name we currently use in the preprocessing and another elevation file that is named Modified followed by the same name. An HDR file is associated with each of the files.

An L1a folder, but it only has an exe file in it called gzip.exe.

4 Empty folders with a similarly themed name: Fall_SON_Ref, Spring_MAM_Ref, Summer_JJA_Ref, Winter_DJF_Ref.

5 folders starting with the year followed by the word reflectance. Each of these folders have the ENVI standard files for that particular year that are output during the preprocessing and put into the final preprocessed folder. The folders also contain 2 folders with pictures: one titled Reflectance_600 and one Reflectance_jpg. I don't know how they differ, but they are not the same file size.

  • The 2003_reflectance folder only has reflectance_600 pictures. Reflectance_jpg is missing.

[edit] Minimage 1998-2004

Contains the ENVI standard files for each year.

Contains the Minimum images for all 7 years.

  • 2002 and 2003 do NOT have the minimum images in their folders.

1999 folder has another folder named 99CalSmoke_MinImage. Folder contains ENVI standard files and minimum images. I suppose this is a zoomed in minimum image.

[edit] AOT 1998-2004

Contains several folders of [Year]_AOT_MED. These folders contain AOTB1_1000, AOTB1_1500 and AOTColor folders with jpgs in them.

  • The 2002_AOT_Med folder only contains AOTColor images.

Contains 1999_AOT_CA Subset folder.

  • Contains 99CaliSmokeAOT folder which has jpgs in them
  • Contains 99CalSmokeAOT_22904 folder which has ENVI standard files, a bluetiff folder containing tif files, and a jpg folder containing jpgs.

2000_AOT_Med has additional AOTB1_1000_Med2000 and AOTB1_1500_Med2000 folders containing jpgs. It also has the other AOT folders shared between the other years.

Contains AOT Red Filt folder.

  • Contains AOT_Median folder with jpgs and ENVI standard files
  • Contains JPGS_AOT_NoMed
  • Contains ENVI Standard files for 1998-2004.

[edit] Refl 1998 Refl 2004

Contains Reflectance for 1998 and 2004 years but nothing in between.

Contains IDL code, other projects in it. See the text file for all the files in it.

[edit] Aggregate AOT

Contains L1a data for 1998 and 1999.

Contains ENVI files for the year 2007 days 274 to 304

Aggregate AOT folder with 5 subfolders

  • 4 yr AVG
    • AOTpSTD
    • Count
    • filter_test
    • PercData
    • Std_deviation
    • Sum
  • Freq_dist
    • 4 yr
    • 5 yr
  • Masks
  • Percentile
  • temp

Most of these folders contain ENVI files associated with their parameter. There is a month ENVI file for each month of the year along with a seasonal one for every 3 months. See the text file of all the files for more information.

[edit] Possible Organization

I believe it will be easy to organize these by parameter. Anything that does not conform to the naming schemes in the year folders will be put into the proper parameter if applicable. Parameters will be listed soon.

After looking at the files and parameters we have, I think a good way to organize the data is as follows.

  • General Parameter (AOT, Reflectance)
    • File Type (ENVI folder and jpg folder)
    • Special Projects (99 CalSmoke)

Parameters could be Reflectance, AOT, Aggregate AOT, MinImage, and L1A.

Subcategories for each parameter:

  • Reflectance
    • ENVI
      • Year
    • 350
    • 600
  • MinImage
    • ENVI
      • Year
    • 350
    • 600
  • AOT
    • ENVI
      • Year
    • AOTB1_1000
    • AOTB1_1500
    • AOTColor
    • AOTB1_1000_MED_2000
    • AOTB1_1500_MED_2000
    • AOT RedFilt
      • AOT_Median
        • ENVI
        • jpg


  • Experimental AOT
    • AOT
      • ENVI
      • Old AOT
        • ENVI
        • jpg
    • AOTpSTD
      • ENVI
    • Count
      • ENVI
      • Old Count
        • ENVI
        • jpg
    • filter_test
      • ENVI
    • PercData
      • ENVI
      • Old_percent
        • ENVI
        • jpg
    • Std_Deviation
      • log_normal
      • Normal
        • ENVI
        • Old_STD
          • ENVI
    • Sum
      • ENVI
      • Old_sum
        • ENVI
        • jpg
    • freq_dist
      • 4 yr
        • ENVI
      • 5 yr
        • ENVI
    • Masks
    • Percentile
  • L1a
    • Year
      • HDF Files
  • 99CaliSmoke
    • MinImage
      • ENVI
    • AOT
      • ENVI
      • bluetiff
      • jpg
        • AOT
        • AOT_2294

In the subcategories will be the jpgs or ENVI files. Two options for organizing these files with this setup are putting all of the information into the folders as-is or making year folders for each and putting the files into their respective year folders. I like the idea of having the L1A files in yearly folders and possibly the ENVI files that span over many years. The jpg files, though, I think should stay in the same folder without breaking them down by year.

[edit] Identifying Processing Files

All of the JPGs for Reflectance 350 and 600, minimages 350, AOT Color, AOT 1000, and AOT 1500 are on the SeaWiFS Dataset. We need to identify all of the ENVI output files associated with these JPGs to expose the data we used. Also, we should identify the L1A files used as inputs.

The reflectance ENVI files (final preprocessed files) are all accounted for. 1998 and 2004 are on the external harddrive Refl 1998 2004. 1999-2003 pics are on the Reflectance 1999-2003 harddrive. 2005-2007 are currently on the SQLbox in Seawifs/Final Preprocessed. Minimum image ENVI files for 1998-2004 are on the external harddrive MinImage 1998-2004. 2005-2007 ENVI files are currently on the SQLbox in Seawifs/MinImageProcessing/FinalMinImage. The AOT ENVI files are on the external harddrive AOT 1998-2004 in the RedFilt folder. 2005-2007 ENVI AOT files are currently on the SQLbox in the Seawifs_AOT folder.

The L1A files are incomplete. 1998 and 1999 are on the external harddrive Aggregate AOT in the form of bz2 files. 2000, 2001, and 2004 cannot be found. 2002 is incomplete on SQLbox. 2003 has one month on the SQLbox in the folder AlaskaL1A. This may not be data on the whole US. 2006-2007 are all on the SQL box in Seawifs/L1a.

[edit] Exposing Raw Data with WCS

The next step in this process is to expose the raw data that ENVI+IDL is outputting. We have the raw input data with the HDF files that are downloaded from the ocean color website. Exposing the output, however, is very desirable. The goal of getting the raw data is to create a WCS interface with the output, so that other people may run other IDL processes on that data through web services. Determining what additional service processing users will do is something for the future. The issue now is preparing the output data for the WCS interface. The output files of the preprocessing are ENVI Standard files. As of right now, I do not know what we can do with those files, but I am looking into it. If I can save those ENVI Standard files into something else or find something that extracts data from the ENVI Standard files, we should be able to make the data WCS-ready shortly after. I am hoping to find something soon.

[edit] References

These are the references I have used thus far in obtaining a better background in satellite measurements. I also plan on using the references in the Aerosol Measurements Book Chapter.

Title Notes
2008-07-01_Satellite-Based_Aerosol_Measurements_Book_Chpt Rudy's Satellite Measurement book chapter. Gives a very nice background.
SeaWiFS_Processing_Research_Rotation_Blog Varun's work on this project
SeaWiFS_Processing_Instructions_on_Wiki
Sean Raffuse's Thesis (need link)
Satellite-derived aerosol optical depth over dark water from MISR and MODIS: Comparisons with AERONET and implications for climatological studies This paper helped me realize the kind of data analysis and comparisons that I may be able to make in the future
Operational remote sensing of tropospheric aerosol over land from EOS moderate resolution imaging spectroradiometer

[edit] Future Reading

Title Notes
Satellite Sensing of Aerosol Absorption
Measurements of the aerosol optical thickness and the path radiance- IMPLICATION ON AEROSOL REMOTE SENSING AND ATMOSPHERIC CORRECTIONS
Reflectance quantities in optical remote sensing—definitions and case studies

[edit] Questions, Comments, and Notes

While reading these papers and book chapters, I do have a few questions on the subject.

The Coretrieval process:

  1. Data approximates the earth's surface reflectance
  2. That data then allows the approximation of aerosol scattering and absorption
  3. The aerosol approximation refines the surface data


Non-Lambertian reflectors exhibit different apparent brightness depending on the angle of the sun, target, and sensor.


The pixels from the satellite pictures are classified as vegetation, soil, water, or clouds using reference spectra.


Bright spots on the satellite images reduce the quality of aerosol retrievals.


Filtering is a little confusing for me. Spike filters, spectral variability filters, and jump filters?

What is spatial filtering?

These papers mention satellite counts. I gather it is a measurement of some sort, but what is it counting?

Satellite have mirror sides and are calibrated differently. Why don't they just make the sides the same? Is there something else to it?

I don't understand internal mixing. How does the particle size increase but the count stays the same? Do molecules attach to them?

Externally mixed particles do not interact with each other physically or chemically. Do they never touch? Or does this mean that upon colliding no coagulation occurs? I'm not sure.

Fine particles in urban, industrial regions exhibit increased mass median diameter with increasing concentration. This allows the formulation of dynamic aerosol models that directly relate the spectral and angular scattering characteristics to aerosol concentration.

  • I'm not sure I fully understand how to relate these. The data a satellite retrieves on the reflectance of the earth's surface will not be the true reflectance.
  • After correcting that value for Rayleigh scattering, and if you do know the actual surface reflectance, you can determine the amount of scattering due to aerosols.
  • This relates the AOT to the concentration with a known extinction coefficient, which you can then relate to mass using the density of the aerosols. Why is this important for the mass median diameter? The mass average diameter would change as well.
  • I think I need to know more about the models for my question to be answered

Image:PhaseFunctionvsScatteringAngle.jpg

This graph shows how higher scattering angles drastically decrease the phase function. At low angles, the phase functions are very similar. At about 60 degrees, the difference becomes more and more noticeable. Higher extinction coefficients see more intense phase function changes. At 0.014 Bext, the phase function stays fairly constant. Why is this? Why does the 0.023 Bext line cross over the 0.014 line after 150 degrees? I need help wrapping my head around all of this.

I would like to discuss Mie Theory a bit. It just doesn't make sense to me why the phase function would be elongated in the forward direction.

Tha phase function is the ratio of the energy scattered into a direction to the average energy scattered in all directions.

There are many different aerosol types that have their own sources, transport and optical characteristics. They should be treated as a separate aerosols with different parameters for models.

The ratio of the aerosol scattering phase function to the surface reflectance is important in determining whether or not aerosols will increase or decrease the apparent reflectance. At low ratios, the reflectance is diminished. High ratios increase the reflectance. For higher and higher optical thickness, the apparent reflectance approaches the value of the aerosol scattering phase function.

When the surface reflectance and aerosol scattering phase function are equal, aerosol detection is impossible. So is there a way to estimate the aerosols present? Over land, surface reflectance and aerosol scattering phase function are similar.

The Bidirectional Diffuse Reflectance Function (BDRF) states that over land, the angular dependence of the reflected radiation depends both on the angle of the incoming radiation as well as the angle of reflected radiation. So the angular dependence of reflected radiation depends on the angle of reflected radiation? This confuses me.

Wouldn't it be useful to use data from multiple datasets? I would imagine you could get more accurate results or at least have something to compare results with.

The reason we can't rely solely on ground measurements for tracking aerosols is they cannot help assess trends with global aerosols, because of the short lifetime of aerosols. Satellites can. Ground measurements are more accurate, but satellites have the ability to see things on a much larger scale.

Retrieved from "http://datafedwiki.wustl.edu/index.php/2009-08-12:_SeaWiFS_Processing_and_Data_Analysis"
Personal tools
Workspaces
Clicky Web Analytics