Architecture of MADIS Data Processing and Distribution at FSL

Abstract

The NOAA Forecast Systems Laboratory's (FSL) Meteorological Assimilation Data Ingest System (MADIS) (Miller, et al. 2005) supplies qual-ity controlled (QC) observational data to a large and growing segment of the meteorological com-munity, including operational National Weather Service modeling and forecasting users. The re-quirements for this system have dramatically ex-panded to accommodate the increasing volume and types of data handled, as well as the growing user base. Today, the system ingests and merges data from dozens of sources, and supplies the resultant quality controlled products to hundreds of users via several standard protocols. Both real-time and saved MADIS datasets are made avail-able. While it needs to be highly reliable, the sys-tem must be sized for the CPU-intensive QC analyses, and must provide controlled access to the data so that proprietary datasets are only available to appropriate users. Methods have also been developed for keeping track of the large, di-verse user community and consistently responding to specific user needs, for example, notification of significant events. To ensure reliability of MADIS, the system runs within the FSL Central Facility, a controlled environment maintained by FSL's Information and Technology Services, which provides system ad-ministration, networking, configuration manage-ment, data acquisition/distribution, and monitoring support (Lipschutz and MacDermaid, 2005). MADIS comprises a distributed architecture for ingest, processing, and data distribution functions. In addition, in a recent architectural advance, the various functional hosts are arranged in pairs us-ing High-Availability (HA) Linux to provide auto-mated failover in case of system failure. Real-time data distribution methods include FTP, Local Data Manager (LDM), and the Web-based OPeNDAP (OPen source project for Network Data Access Protocol). Special consideration for protecting pro-prietary datasets has added an extra layer of complexity, but is well handled by the configura-tion methods implemented on the LDM and FTP servers. In this paper, we review the requirements and architecture of the operational implementation of MADIS at FSL. We describe our use of High-Availability Linux toward achieving high system reliability, and consider our implementation of the data distribution methods. Finally, we also discuss our user support infrastructure.