Skip to main content
U.S. flag

An official website of the United States government

Dot Gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.


The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Architecture of MADIS Data Processing and Distribution at FSL


The NOAA Forecast Systems Laboratory's (FSL) Meteorological Assimilation Data Ingest System (MADIS) (Miller, et al. 2005) supplies qual-ity controlled (QC) observational data to a large and growing segment of the meteorological com-munity, including operational National Weather Service modeling and forecasting users. The re-quirements for this system have dramatically ex-panded to accommodate the increasing volume and types of data handled, as well as the growing user base. Today, the system ingests and merges data from dozens of sources, and supplies the resultant quality controlled products to hundreds of users via several standard protocols. Both real-time and saved MADIS datasets are made avail-able. While it needs to be highly reliable, the sys-tem must be sized for the CPU-intensive QC analyses, and must provide controlled access to the data so that proprietary datasets are only available to appropriate users. Methods have also been developed for keeping track of the large, di-verse user community and consistently responding to specific user needs, for example, notification of significant events. To ensure reliability of MADIS, the system runs within the FSL Central Facility, a controlled environment maintained by FSL's Information and Technology Services, which provides system ad-ministration, networking, configuration manage-ment, data acquisition/distribution, and monitoring support (Lipschutz and MacDermaid, 2005). MADIS comprises a distributed architecture for ingest, processing, and data distribution functions. In addition, in a recent architectural advance, the various functional hosts are arranged in pairs us-ing High-Availability (HA) Linux to provide auto-mated failover in case of system failure. Real-time data distribution methods include FTP, Local Data Manager (LDM), and the Web-based OPeNDAP (OPen source project for Network Data Access Protocol). Special consideration for protecting pro-prietary datasets has added an extra layer of complexity, but is well handled by the configura-tion methods implemented on the LDM and FTP servers. In this paper, we review the requirements and architecture of the operational implementation of MADIS at FSL. We describe our use of High-Availability Linux toward achieving high system reliability, and consider our implementation of the data distribution methods. Finally, we also discuss our user support infrastructure.

Article / Publication Data
Available Metadata
Fiscal Year
Published On
January 01, 2005

This publication was presented at the following:

21st Int. Conf. on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanograp
American Meteorolgical Society
Conference presentation


Not available


Authors who have authored or contributed to this publication.