Aligning Forecast Verification With User-specific Needs: An Example For Aviation

Abstract

Brier and Allen (1951) state that there are three primary purposes for performing objective evaluations of forecast quality: administrative, scientific, and economic. The separation of administrative purposes from scientific purposes is not particularly significant in practice. The primary difference is that administrative verification is closely linked to operational forecasting systems while verification of developing forecast systems relies, or should rely, more heavily on scientific, or diagnostic, approaches. In this broad view, the information needs of both the producers and consumers are not explicitly separated. Clearly, forecast producers have different needs from verification than consumers and therefore benefit from different verification information than forecast consumers. Modern forecasting systems are often comprised solely of computer algorithms rather than human-generated content. For these types of systems where algorithms can be changed and re-run on identical inputs, verification can guide the design and implementation of the algorithms and lead the producers to improve the product before it becomes operational. For the forecast consumer, the goal is no longer about improving the forecast as much as possible but rather about making the best decisions possible using the available information. The obvious difficulty with separating the problem into producer and consumer views is the clear identification of all distinct consumers and their particular needs for a forecasting system and the subsequent verification. Consumer-oriented verification can be seen as a cross-cutting approach that delves significantly into both the administrative and scientific applications of verification information targeted at a specific user or set of users with similar information needs.