Dataset Descriptions
From Datafedwiki
The FASTNET project now has access to an array of datasets that allow the basic documentation of natural aerosol events. The list of datasets and some of their features are summarized in Table 1. The datasets include both near real-time and historical observations. It is anticipated that following this first-year pilot project additional chemical, meteorological, emissions, and GIS datasets will be added to the FASTNET collection.
[edit] Table 1. FASTNET datasets and characteristics
| Dataset/ Time | Provider | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 00 | 01 | 02 | 03 | 04 | 05 | Start Date | End Date | Time Resolution | type | |||||||||||||||||||
| EPA/STI | 7/1/2002 | now | hour | cache | aerosol | |||||||||||||||||||||||||||||||||||||
| NOAA/PSWC | 7/30/1998 | now | hour | cache | meteorol | |||||||||||||||||||||||||||||||||||||
| NOAA/STI | 5/1/2003 | now | hour | cache | aerosol | |||||||||||||||||||||||||||||||||||||
| NOAA/NCDC | 4/19/1997 | now | hour | meteorol | ||||||||||||||||||||||||||||||||||||||
| DOD/NRL | 1/1/2001 | now | 6 hour | aerosol | ||||||||||||||||||||||||||||||||||||||
| USDA/USFS | 10/1/2003 | now | day | fire | ||||||||||||||||||||||||||||||||||||||
| NASA/USFS | 8/1/2003 | now | day | fire | ||||||||||||||||||||||||||||||||||||||
| GOES | 7/16/2004 | now` | .5hour | archived | aerosol | |||||||||||||||||||||||||||||||||||||
| NASA/GSFC | 7/25/1996 | now | day | aerosol | ||||||||||||||||||||||||||||||||||||||
| NASA/CAPITA | 1/1/2000 | 12/31/2002 | day | aerosol | ||||||||||||||||||||||||||||||||||||||
| NPS/CIRA | 3/1/1988 | 1/1/2003 | 3 day | cache | aerosol | |||||||||||||||||||||||||||||||||||||
| NPS/CIRA | 1/1/1988 | 5/31/2002 | 6 hour | meteorol | ||||||||||||||||||||||||||||||||||||||
| MWH | 7/19/2004 | now | day | archived | aerosol | |||||||||||||||||||||||||||||||||||||
| NPS | 4/5/2004 | now | day | archived | aerosol |
The near-realtime data are available within hours to a day following the observations. In Table 1 the top eight datasets fall in this category as indicated by "now" in the end date column. The time resolution of these near-realtime data ranges from an hour to once a day. In the paragraphs below there is a brief description of each dataset with emphasis on its application for the FASTNET project.
AIRNOW is an EPA project to gather and distribute near-realtime data from several hundred continuous PM2.5 and ozone monitors. From the point of view of the FASTNET project the AIRNOW system establishes the spatial and temporal patterns of the key pollutants PM2.5 and ozone. The AIRNOW dataset is provided to the FASTNET project by EPA using a web-based interface.
SURF_MET is a dataset gathered by the Automated... , which in 1996 replaced the human observers for routine monitoring of surface weather parameters. Currently, there are about 1,200 SURF_MET stations recording weather data every minute. The SURF_MET dataset is archived at the Plymoth State Weather Center and accessible to the DataFederation through a web-based interface The SURF_MET parameters include relative humidity, wind direction and speed, as well as the light scattering coefficient,Bext, obtained by the ASOS forward scattering visibility sensor (actually Bscat). The 1,200 station visibility data are particularly useful for FASTNET for delineating the spatial-temporal extent of hazy air-masses from dust, smoke, or anthropogenic haze events. A major limitation of this SURF_MET visibility data is the truncation of high visibility at ten miles which precludes the detection of haze events with greater than ten mile visibility.
ASOS_STI represents a 250-station subset of the national ASOS SURF_MET dataset. The unique feature of this data is that the visibility sensor values are available in an un-truncated, high resolution form. This dataset indicates that the sensitivity of the ASOS sensor is about 25 miles, which allows spatial-temporal analysis of haze events using the in situ ASOS sensors. The ASOS_STI dataset is provided to the FASTNET project by Sonoma Technology Inc. using a web-based interface.
NEXTRAD is a standard dataset of radar reflectivity images for the U.S. The hourly NEXTRAD radar maps are provided to FASTNET by the National Climatic Data Center. In FASTNET the radar data are used to delineate the spatial-temporal pattern of precipitation.
NAAPS is a model dataset generated by the Global Aerosol Model of the Naval Research Laboratory. Assimilated parameters include windblown dust, biomass smoke, and sulfate aerosols. The output parameters are expressed as surface concentration and vertical optical thickness for each aerosol component. The six-hourly model output is driven by forecasted winds generated by a global circulation model. For the FASTNET project the NAAPS model output for North America provides the only forecast aerosol pattern and it is also a very useful tool for assessing intercontinental transport from Asia and Africa. The NAAPS output images are accessed by Datafed from the NRL-NAAPS data archive.
HMS_Fire and MODIS_Fire location maps are provided by NOAA's Satellite Services Division (http://gp16.wwb.noaa.gov/FIRE/fire.html). Both the MODIS and HMS fire location data are available for freely open ftp download from ftp://gp16.wwb.noaa.gov/pub/FIRE/. Only the most recent 400 datasets are archived. CAPITA has written and maintains a SQL Server Data Transformation Service for dailyaccess and storage of the ftp accessible ASCII data files. The fire location data are useful to the FASTNET community in analyzing the spatial and temporal distribution of detected fires and in identifying source locations for smoke plumes.MODIS_Fire products are received by the Satellite Services Divion from NOAA's MODIS Near Real Time Processing System in NOAA's Information Processing Division (http://www.osdpd.noaa.gov/MODIS/index.html). The MODIS instrument flies onboard the NASA TERRA and AQUA satellite, and the fire algorithm was developed by the MODIS Fire and Thermal Anomalies team (http://modis-fire.gsfc.nasa.gov/).
HMS (Hazard Mapping System) is a fire location product derived by NOAA satellite analysts manually integrating data from various automated fire detection algorithms using GOES, AVHRR and MODIS images. The result is a quality controlled display of the locations of fires and significant smoke plumes detected by meteorological satellites. The HMS product is generated twice a day (approximately 4pm and 11pm Eastern time).
GASP (GOES Aerosol/Smoke Product) which provides near real-time aerosol optical thickness using the GOES image. http://orbit-net.nesdis.noaa.gov/crad3/gasp/RealTime.html
TOMS_AI is an aerosol related dataset derived from the Total Ozone Monitoring Satellite (TOMS) Sensor. The TOMS aerosol index arises from absorbing aerosols such as dust and smoke in the upper layers of the atmosphere. In the FASTNET project the TOMS images are used to identify qualitatively delineate major smoke or dust events, particularly those originating in Africa, Central America, or Asia. The TOMS_AI images are accessed by DataFed directly from the TOMS project website.
SEAWiFS is a high resolution (1km) satellite dataset derived from the eight wavelength SEAWiFS sensor. The dataset also includes the aerosol reflectance over the conterminous U.S. For FASTNET the high resolution aerosol pattern derived from SEAWiFS is useful compliment to the surface-based monitoring data. The daily SEAWiFS images for the U.S. are provided by CAPITA for the FASTNET project. The daily images for 2000, 2001, and 2002...
VIEWS_CHEM is an aerosol chemistry dataset integrated for the Regional Planning Organizations (RPOs) by Colorado State/CIRA (?). The VIEWS chemical database is arguably the most valuable data source for quantification of dust, smoke and anthropogenic haze in different parts of the country. The VIEWS dataset is used by numerous analysis tools for the identification of dust, smoke and anthropogenic haze in different parts of the country. The VIEWS chemical dataset is provided to Datafed through a collaborative arrangement.
ATAD is an extensive back-trajectory database created at CIRA for each of the IMPROVE monitoring sites for each day between 1988 and May 2002. The ATAD dataset is used primarily in conjunction with the VIEWS/IMPROVE chemical data to establish the airmass histories associated with certain chemical conditions. Within FASTNET the ATAD trajectory database is used as input to the Combine Aerosol Transport Tool (CATT). ATAD data were downloaded from CIRA/VIEWS website.
MWH Webcams and NPS Webcams are just a few examples of webcams available on the internet. The NPS webcam console is created similarly to other consoles because the NPS archives its images. However, for MWH cams the images are being saved out hourly and archived on the DataFed server.
[edit] FASTNET Data Caching
All the datasets used in the FASTNET project are cataloged, accessed, and distributed through the Datafed.net infrastructure. Table 1. shows the list of agencies and organizations through which the datasets are provided to the FASTNET project. In most cases these providers themselves are simply passing data through as part of the value-adding data processing chain. Similarly, Datafed.net is designed to pass the data to other applications such as the FASTNET project, and other value-adding activities. For some datasets, such as ASOS_STI, SURF_MET, VIEWS_CHEM, ATAD the data are accessed through special arrangements between Datafed and the provider. In other cases e.g. NEXTRAD, NAAPS, TOMS_AI, and HMS_Fire the data are accessed from the provider's website without special arrangements with the provider.
In distributed systems, such as DataFed, the data are maintained by their respective providers in their native environment. Users access the distributed data through the �wrappers� and mediators provided as federation services. An additional federation service is caching, i.e. local storage or pre-calculation of frequently requested queries. At this writing, the FASTNET datasets cached in the DataFed system are AIRNOW, SURF_MET, ASOS_STI, and VIEWS.
Numeric Data Caching. The above listed datasets are point monitoring data that are updated hourly or daily by their providers. The updates are generally individual files of spatial pattern containing the most recent measurements. This local storage is convenient for the incremental maintenance of the database but it is inefficient for data access, particularly for temporal data views. In DataFed the caching consists of densely packed numeric �data cubes� suitable for fast and efficient queries for spatial and temporal views. The data cubes are updated hourly, daily or intermittently as dictated by the data availability and user-need. The daily cache updates is monitored through the cache status console (link here). The time series show the number of valid stations for each hour/day in the dataset. A drop in number of stations indicates problems in data acquisition, transmission or caching.
Image Caching.The creation of some of the data views is time consuming. For example spatial interpolation and contouring of monitoring data may take 20-30 seconds, which makes interactive data browsing impractical. For frequently used views, the data are pre-rendered and stored as images for fast retrieval and interactive browsing. The FASTNET browsing tools including the Analysts Consoles and Animator are set up to use the cashed images. For a subset of views, the image caching is performed automatically following the update of the numeric data cubes. The image datasets are from distributed providers, the data are fetched from the provider and passed to the user on the fly. The cached image datasets for the FASTNET project include: AIRNOW: pmfine, super
ASOS_STI: DryVisMean, VisMean
SURF_MET: Fbext, RH, RHbext, T, TD
SURF_MET_WIND: Wind
