Expert: Philip A. Stahl - 5/16/2007

Question
Can you explain a bit about the Brier P-score for predicting flares and how it would actually work?

Answer
Hello,

The Brier P-Score is a method of statistical flare forecast evaluation developed  in 1950 as a “proper” assessment technique for flare prediction.  By way of comparison, an “improper” method would be illustrated if a forecaster were to issue a ‘no flare’ forecast (say for major flares) every day of the year and only 5 events occurred. Then, by improperly counting the ‘no flare’ days as actual events a 99 percent success rate could be arrived at.

The standard Brier P-Score is defined:

P =  1/ M  SIGMA^k  k=1  SIGMA^M i=1 (f_ik – O_ik)^2

where P is the verification score, M is the number of forecasts made, k is the number of categories for each forecast occasion, and f is the forecast probability with range 0- 1 in each category.

The observation is denoted by the letter O and may be zero (0) (event i in category k does not occur) or 1 (event does occur).  Mathematically, the smaller the forecaster’s score the greater his skill – since the less difference between what is forecast and observed (the squared term at the end)

A more refined modification due to Saunders (1963) takes into account more factors than the simplified version above, but we will focus on the simpler version.

Now, as to a specific application. Consider the interval April 5 – 11, 1980 when I actually made ex post facto predictions that were later checked using the P-Score. The results are tabulated as follows and these are for “major SID flares”. E.g. flares that produced an SID event or sudden ionospheric disturbance, of at least importance ‘2’ on a 0-2 scale.

Date    4/5    4/6   4/7   4/8   4/9   4/10   4/11

Obs.   2   1   2   0   0   2   0

Pred.   0   1   2   1   0   1   0

f_ik   0   0.2   0.5   0.2   0   0.1   0    =    1.0

The P-score that resulted was 0.48 from this example.

Again, this is raw and just to show how the basic score works. As I noted there are ways to refine it. More recently (1979) Simon and Smith (Solar –Terrestrial Predictions Proceedings, Vol. II, p. 311) have noted that forecast accuracy can be fundamentally limited by Poisson statistics, e.g. the type that yield the Posson distribution:

P(N) =  exp(- S) S^N/ N!

where is the mean rate of SID flares per day and N denotes the number of flares associated with the particular class.

It is possible, if such considerations had been applied to the example above, the P-score would have been significantly improved – since fewer predicted flares would have been assigned on those days when fewer occurred (were actually observed).