How to Assess the Quality of Space Weather Forecasts?
As space weather forecasting matures more attention is being given to how we verify and validate those forecasts. A simple approach is to classify a set of forecasts of space weather events as: (a) the forecast event occurs (a hit), (b) does not occur (a false alarm), (c) does occur but is not forecast (a miss), or (d) no event is forecast and none occurs. The counts of forecasts in each class are then formed into a contingency table from which a wide range of metrics can be derived (example here). This simple approach is sensitive to any quantitative thresholds used to identify an event; for example, does the particle flux exceed some level. If an event falls 1 per cent below the threshold it will be counted as false alarm. Thus, we need more nuanced metrics that take account of this and other cases where thresholds are narrowly missed.