Datafedwiki:Sandbox

DataFed: Mediated Web Services for Distributed Air Quality Data Access and Processing.  

Rudolf B. Husar and Kari Hoijarvi

Washington University, CAPITA

1 Brookings Drive, St. Louis, MO 63130

Developments in air quality monitoring and data dissemination technologies offer outstanding opportunities to fulfill the information needs of agile air quality management. The data from surface-based air pollution monitoring networks now provides routinely high-grade, spatio-temporal and chemical patterns throughout the US for PM25 and ozone. Satellite sensors with sub-kilometer-scale spatial resolution now provide real-time snapshots which depict the global pattern of haze, smoke and dust in stunning detail. The ‘terabytes’ of data from these surface and remote sensors can now be stored, processed and delivered in near-real time. The instantaneous ‘horizontal’ diffusion of information via the Internet also permits, in principle, the delivery of the right information to the right people at the right place and time. Standardized computer-computer communication languages and Service-Oriented Architectures (SOA) facilitates flexible processing of raw data into high-grade ‘actionable’ knowledge. Last but not least, the World Wide Web has opened the way to generous sharing of data and tools leading to faster knowledge creation through collaborative analysis and virtual workgroups.

Nevertheless, air quality analysts face significant hurdles, most notably the exponentially growing “data deluge”.  In this report we summarize recent developments in DataFed, (Husar and Poirot, 2005) an infrastructure for real-time integration and web-based delivery of distributed monitoring data. The federated data system, DataFed, (http://datafed.net) aims to support air quality management and science by more effective use of relevant data. Building on the emerging pattern of the Internet itself, DataFed assumes that datasets and new data processing services will continue to emerge spontaneously and be maintained autonomously on the Internet.

The key roles of the DataFed federation infrastructure are to (1) facilitate registration of the distributed data in a user-accessible catalog; (2) ensure data interoperability based on physical dimensions of space and time; (3) provide a set of basic tools for data exploration and analysis. The federated datasets can be queried, by simply specifying a latitude-longitude window for spatial views, time range for time views, etc. This universal access is accomplished by ‘wrapping’ the heterogeneous data (Fig. 1), a process that turns data access into a standardized web service, callable through well-defined Internet protocols.

Figure.1. Data Access Protocols and Adapters. The electric adapter is a good analogue of the DataFed software adapters. 

The result of this ‘wrapping’ process is an array of homogeneous, virtual datasets that can be queried by spatial and temporal attributes and processed into higher-grade data products. The rich structure and semantics of Earth Science data means that any given dataset can be accessed through multiple protocols. In general, each client and server is capable of communicating through a subset of protocols.  Thus, loose coupling between data access and processing services involves choices and negotiations. The main topics of client-server negotiation are the selection of a shared data access protocol and a choice of returned data format.

The Service Oriented Architecture (SOA) of DataFed allows, in principle, to build web-applications by connecting the web service components (e.g. services for data access, transformation, fusion, rendering, etc.) in Lego-like assembly. The generic web-tools created in this fashion include catalogs for data discovery, browsers for spatial-temporal exploration, multi-view consoles, animators, multi-layer overlays, etc.(Fig. 2).

Figure 2. Main user interfaces to DataFed: Catalog for finding and selecting data, Viewer for exploration, and Consoles for data analysis and presentation.

The analysis of air quality data through agile information system requires (1) seamless data access (2) reusable data processing component to access, filter, aggregate, and fuse distributed data; (3) service-oriented framework that facilitates the publish-find-bind and chain web service model for application building. Such <a href="http://en.wikipedia.org/wiki/Loose_coupling">loose coupling </a>can only be achieved if the service components adhere to strict standard protocols. Achieving such loosely coupled service oriented architectures has been the stated goal of numerous national and international Earth Science programs including <a href="http://www.earthobservations.org/index.html">GEO</a>.

The most important step toward service oriented Earth Science information systems is the adaptation of strongly typed standards for finding and accessing data. Given such standards-based interface, providers of data and services can publish their data resources and users can find suitable data in formal catalogs. Most importantly, formal protocols allow snap-like binding i.e. data access, between the server and the client operations.

Current Earth Science data systems do not yet allow such flexible, user-defined data processing. An attractive development in this regard is the emergence and the broad acceptance of geospatial standards coordinated internationally by <a href="http://www.opengeospatial.org/" target="_blank" title="http://www.opengeospatial.org/">Open Geospatial Consortium</a> (OGC). In the Earth Sciences similar development led to a standard for binary-encoded Earth Science data, through the <a href="http://www.unidata.ucar.edu/software/netcdf/" target="_blank" title="http://www.unidata.ucar.edu/software/netcdf/">netCDF encoding</a>, augmented by the <a href="http://www.cgd.ucar.edu/cms/eaton/cf-metadata/" target="_blank" title="http://www.cgd.ucar.edu/cms/eaton/cf-metadata/">CF Conventions</a>.  Air quality monitoring data are best accessed through the Web Coverage Service (WCS). The spatial pattern of sampling locations can be conveniently represented by the Web Feature Services (WFS). We have found and demonstrated that for the air quality applications WCS is a well-suited protocol for point/station, image and gridded data Still, substantial hurdles are ahead before we reach a 'snap'-like WCS client-server interoperability.

The paper will describe the DataFed approach in more detail and provide examples of its real-world applications. This paper is being written using the collaborative wiki page, (<a href="http://datafedwiki.wustl.edu/index.php/2007-07-25:_IGARSS07_Barcelona">http://datafedwiki.wustl.edu/index.php/2007-07-25:_IGARSS07_Barcelona</a>). Comments and suggestions are welcome.

Husar, R.B. and R.; Poirot: DataFed and Fastnet: Tools for Agile Air Quality Analysis; Environmental Manager 2005, September, 39-41 (<a href="http://capita.wustl.edu/capita/capitareports/050601AWMA_FASTNET/Submitted/EM_DataFed_FASTNET_050720.pdf">pdf</a>)

Husar, R.B. S. R. Falke and K. Hoijarvi: 

 Interoperability of Web Service-Based Data Access and Processing: Experience Using the DataFed System. ESTO Meeting, 2006, <a href="http://esto.nasa.gov/conferences/ESTC2006/papers/a6p2.pdf">Paper A6P2</a>