MADIS – Meteorological Assimilation Data Ingest System – SOWTS


MADIS – Meteorological Assimilation Data Ingest System

MADIS is a meteorological observational database and data delivery system that provides observations that cover the globe

MADIS ingests data from NOAA data sources and non-NOAA providers, decodes the data then encodes all of the observational data into a common format with uniform observational units and time stamps. Quality checks are conducted and the integrated data sets are stored along with a series of flags indicating the results of the various QC checks. MADIS provides several methods for users to access the data to meet their needs. Users can request data from July of 2001, which is when MADIS was first available to the public, to the present.

To provide these services, MADIS leverages partnerships with international agencies; federal, state, and local agencies (e.g., states’ Departments of Transportation); universities; volunteer networks; and the private sector (e.g., airlines, railroads) to integrate and quality check observations from their stations with those of NOAA.

MADIS runs operationally at the NWS National Centers for Environmental Prediction (NCEP) Central Operations (NCO) as part of the Integrated Dissemination Project (IDP). All MADIS data is being added to NOAA’s data archive that is run by the National Environmental Satellite, Data, and Information Service (NESDIS) National Climatic Data Center (NCDC). Research and development efforts are being provided by NOAA’s Office of Oceanic and Atmospheric Research (OAR) Earth System Research Laboratory (ESRL) Global Systems Division (GSD). 

MADIS meteorological surface variables are quality-controlled by the Quality Control and Monitoring System, which collects statistics on observational errors encountered for NWS sea-level pressure, potential temperature, dewpoint, surface wind, and altimeter setting. The system provides the total number of observations for each variable, the number of observations that failed the QC checks, the station names for the failed observations, and the error and threshold values for each of the failed observations. The error is defined as the difference between the QC analysis value and the observed value, as computed in the spatial consistency check.

Statistics are calculated for all stations. Stations from different networks are kept statistically separate. Specifically, the following stratifications are maintained: “ASOS”, “OTHER-MTR”, (non-ASOS METARs), “MARITIME”, “SAO”, “CRN”, “USHCN-M”, and “NEPP”. Mesonets are stratified by provider. For example, “MAP” for Multi-Agency Profiler. The statistics are available in the form of hourly QC messages. The following figure shows the MAP hourly message for 8 July 2004 at 1600 UTC.


             MAP HOURLY QUALITY CONTROL MESSAGE  (PAGE   1 OF   1)                        

  *********************************************************************************
  *  8-JUL-2004 *   SLP    * POT TEMP *  DEW PNT *    DD    *    FF    *   ALT    *
  *  1600 UTC   *   (MB)   * (DEG F)  *  (DEG F) *   (DEG)  *  (KNTS)  *   (MB)   *
  *********************************************************************************
  * TOTAL OBS   *      0   *      5   *      5   *      5   *      5   *      5   *
  *   QST OBS   *      0   *      1   *      0   *      0   *      0   *      5   *
  * PERCENT QST *   0.00   *  20.00   *   0.00   *   0.00   *   0.00   * 100.00   *
  *********************************************************************************
  *    ASTOR    *          *          *          *          *          *1007(   2)*
  *    CCDNH    *          *          *          *          *          *1002(   2)*
  *    CCOCA    *          *   7(   6)*          *          *          *1000(   4)*
  *    PSENH    *          *          *          *          *          *1002(   2)*
  *    PYMMA    *          *          *          *          *          *1002(   2)*
  *             *          *          *          *          *          *          *
  *             *          *          *          *          *          *          *
  *             *          *          *          *          *          *          *
  *             *          *          *          *          *          *          *
  *********************************************************************************
  ERROR = ANALYSIS - OBSERVATION            OB ERROR ( ERROR THRESHOLD )

Hourly QC message for Multi-Agency Profiler stations on 8 July 2004 at 1600 UTC. The stations listed had variables that were found bad by the spatial consistency check.

Statistics for the total number of observations (“TOTAL OBS”), the total number of observations that failed the QC checks (“QST OBS”), and the percentage of failed observations (“PERCENT QST”) are given at the top of each page of the hourly message. “QST” represents “questionable” observations. Errors and threshold values for the failed observations are listed in the columns. In ASOS hourly messages, the stations are also stratified by NWS region.

Note that threshold values are not given for the surface wind errors in the above figure. Wind observations are tested by computing observation errors and threshold values for each of the u and v components of the wind. However, observation errors are converted to polar coordinates before display in the QC messages. If either of the u or v components fail, both direction and speed errors are computed.

Stations listed in the QC messages are either in error due to hardware or software failure, or are unrepresentative of the observation scale and, as such, are susceptible to diurnal, mesoscale, and terrain effects. To help distinguish between the two, daily, weekly, and monthly (4-week) summaries of the hourly QC messages are also provided. The summaries include the percentage of failed observations and the average error and rms error of the failed observations for individual stations and for all stations combined. The following figure shows the daily NOAA Profiler Network (NPN) QC message for 16 May 1997.


          NPN   DAILY QUALITY CONTROL MESSAGE (PAGE   1 OF   1)                

  *********************************************************************************
  *16-MAY-1997*   SLP    * POT TEMP *  DEW PNT *     DD     *    FF    *   ALT    *
  * 00-23 UTC *   (MB)   * (DEG F)  *  (DEG F) *    (DEG)   *  (KNTS)  *   (MB)   *
  *********************************************************************************
  * TOTAL OBS *    336   *    336   *    301   *     336    *    336   *     0    *
  *  QST OBS  *      0   *      0   *     11   *       0    *      0   *     0    *
  *PERCENT QST*   0.00   *   0.00   *   3.65   *    0.00    *   0.00   *  0.00    *
  *********************************************************************************
  *    PRCO2  *          *          *21/-20/ 85*            *          *          *
  *           *          *          *          *            *          *          *
  *           *          *          *          *            *          *          *
  *           *          *          *          *            *          *          *
  *           *          *          *          *            *          *          *
  *           *          *          *          *            *          *          *
  *           *          *          *          *            *          *          *
  *           *          *          *          *            *          *          *
  *           *          *          *          *            *          *          *
  *********************************************************************************
  ERROR= ANALYSIS-OBSERVATION           RMS ERROR/MEAN ERROR/PERCENT QST

Daily QC message for NOAA Profiler Network (NPN) surface stations on 16 May 1997.

As with the hourly messages, all stations in the domain are used to calculate the statistics reported at the top of each page, but only stations that have failed the QC checks (at least once in the reporting period) are listed in the individual statistics. Stations with large percentages of failed observations are most likely experiencing hardware or software failures. For example, the QC message in the figure shows PRCO2 (Purcell, OK) as reporting bad dewpoint temperature observations 85% of the time. The rms errors for the station are also identical to the absolute value of the mean error, an indication that a persistent bias exists in the observations. The fact that the error is negative further indicates that the observations are biased high. With the information, the Profiler Control Center in Boulder was able to determine that the surface dewpoint sensor at Purcell had failed. The sensor has since been fixed, and the percentage of dewpoint observations failing the QC checks is back to zero.

Daily, weekly, and monthly summaries include only those stations with observations that have failed more than 25% of the time.

MADIS Quality Control

MADIS observations are quality-controlled during data processing and these results are made available to users. Observations in the MADIS database are stored with a series of flags indicating the quality of the observation from a variety of perspectives (e.g. temporal consistency and spatial consistency), or more precisely, a series of flags indicating the results of various quality control (QC) checks. MADIS users and their applications can then inspect the flags and decide whether or not to use the observation.

The QC procedures are, for the most part, provided by the NWS Techniques Specification Package (TSP) 88-21-R2 (1994).

Two categories of QC checks, static and dynamic, are described in the TSP for a variety of observation types, including most of the observations available in the different MADIS datasets. The static QC checks are single-station, single-time checks which, as such, are unaware of the previous and current meteorological or hydrologic situation described by other observations and grids. Checks falling into this category include validity, internal consistency, and vertical consistency. Although useful for locating extreme outliers in the observational database, the static checks can have difficulty with statistically reasonable, but invalid data. To address these difficulties, the TSP also describes dynamic checks which refine the QC information by taking advantage of other available hydrometeorological information. Examples of dynamic QC checks include position consistency, temporal consistency, and spatial consistency.

The TSP also describes single-character “data descriptors” for each observation, which are intended to provide an overall opinion of the quality of the observation by combining the information from the various QC checks. Algorithms used to compute the data descriptor are a function of the types of QC checks applied to the observation, and the sophistication of those checks. Level 1 QC checks are considered the least sophisticated, level 3 the most sophisticated checks.

Information, Data, Maps & Graphics Disclaimer - http://www.shelbyohwx.com/site-usage-polices/information-disclaimer-sup/
Shelby Ohio Weather Station - Copyright © 2010-2018