Forecast Assessment For The New York 2008 Convective Weather Project

Abstract

This study assesses the performance of the Enhanced Convective Forecast (ECF) and its associated model product, the Weather Research and Forecasting model (WRF) composite reflectivity, both of which were used as experimental forecasts during the New York 2008 Convective Weather Project. The evaluation compares the performance of the ECF and WRF directly against that of the Collaborative Convective Forecast Product (CCFP), which is the current operational baseline. Additionally, the analysis evaluates the quality of the experimental products when used to supplement CCFP as outlined in the project’s concept of use. Accounting for resolution and domain differences between the products, the study adopts two approaches for the direct comparison of the ECF and WRF to the operational baseline. The first, a sector-based method, scores the products on a “grid” of air traffic sectors, allowing for displacement of the forecast on a user-relevant scale. The second, a resolution-based approach, measures performance on a number of regularly spaced grids, ranging from 48 km down to the native resolution of the observation field, 4 km. The analysis of the supplemental relationship of ECF and WRF with respect to CCFP relies on a decomposition approach that identifies agreement and disagreement between the forecasts. Within each subdomain, the supplemental forecasts are characterized with various measures (e.g., structure of convection) to determine the value added to CCFP. Results from the direct comparison of the products indicate the following. • By all measures in the sector-based approach, ECF performed poorly relative to the CCFP baseline, clearly due to the small size and limited number of individual ECF polygons within any given forecast. Overall, the WRF and CCFP showed similar skill (as measured by the Heidke Skill Score), but the forecasts have different strengths and weaknesses. The WRF forecast often misplaced convection relative to its occurrence, but effectively predicted the correct number of sectors significantly covered by hazardous weather. CCFP, while overforecasting, shows value by locating most of the sectors with significant coverage. • As in the case of the 4- and 6-h leads, the ECF failed to show any appreciable skill in forecasting convective coverage of air traffic sectors at 8-h and 10-h leads. The WRF forecast, however, appears to retain some skill, which degrades by a factor of two at these longer lead times. Used cautiously, the longer lead times of the WRF forecast may provide utility to planners. • In addition to confirming the overall indications found in the sector-based approach, the resolution-based analysis suggests that the ECF and WRF products don’t perform well at high resolution. Even at the most coarse resolution (48 km) studied, median performance values are very low. Results from the analysis of the supplemental relationship indicate the following. • When the forecasts agree on the presence of convection: The WRF simulated reflectivity appears to add more value than ECF by better forecasting the amount of hazardous convection in and around a CCFP polygon. Additionally, the WRF seems to better indicate the structure of convection within a CCFP polygon, as measured by the “center of mass” of the convection, and the distributions of the sizes and shapes of convective objects. • When ECF and WRF forecast convection outside of CCFP polygons: Overall, neither supplemental forecast seems to effectively identify significant convection outside of a CCFP polygon. Additionally, in most cases, these regions appear to contain isolated convection that may not meet minimum CCFP criteria to warrant a polygon. More research would need to be done to completely quantify this observation. • When only CCFP forecasts convection: Rarely – in less than 1% of the area of the domain -- does a CCFP polygon exist without some associated ECF or WRF forecast of convection. These cases almost always contain only “clipped” CCFP polygons that are found along the edge of the verification domain, likely a result of the difference in the product domains and granularity. • When CCFP and the supplementary forecast agree on a forecast of ‘no convection’: The forecast combinations appear to accurately predict ‘no convection.’ For forecasts issued in the study period, the combined regions of ‘no convection’ have median convective coverage of less than 1%. Rarely did both CCFP and its supplement miss significant convection; together, in this subdomain, they appear very trustworthy for use in air traffic planning.