Run enough long range models in parallel and somewhere in the not-too-distant future you’ll find a chart showing you almost whatever you want to see. We’ve found one, a ‘would be’ Hurricane Gordon – and it’s really something…
We mentioned this one last week. It’s a shot from our 16 day forecast and it shows a hurricane that isn’t yet even in embryonic form creating a striking chart for US East Coast surfers on Sunday, August 19th, 2012. As well as giving us a chance to run this ‘fantasy’ chart again it gives us a great chance to talk about probabilities and long term forecasting. How do we even start checking out whether there could be a glimpse of reality here worth considering?
Why computer simulations struggle
A ‘model’ is a computer ‘simulation’ of the future, which we’ve written more about here. A model depends on two things for its long term accuracy. One is the starting conditions, we need to know exactly what is happening right now to predict the future. While we have satellite data of amazing quality there are always going to be small variations between reality and our assumptions. Secondly the model is, essentially, a stack of mathematical equations based on scientific research that aim to simplify VERY COMPLEX physical processes. We have to simplify in order to predict, otherwise our forecast would arrive long, long after the actual storm had happened!
These two factors have markedly different impacts on the accuracy of swell and wind models. Swell models are ‘weakly non-linear and dissipative’ – this means that inaccuracies tend to disappear over time – a swell called 5ft shy of reality arrives 1ft shy of forecast at your beach. It’s an issue, but one that gives us a predictable margin of error and means that, regardless which model you consult for your forecasts: if the storm has already happened the predicted swell is very, very similar. Swell models typically get it wrong when wind models got the start point wrong and rarely be a greater degree than the original error.
Wind models are very different. They’re highly ‘non-linear’. This means a tiny error in the next 24 hours becomes a massively unpredictable variation over 16 days. That error could be in the mathematical equations, or in the starting conditions. This is the ‘chaos effect’ you might have heard of, the idea that a butterfly flapping its wings in the US causes a Typhoon in Japan. It’s a pretty extreme example but fitting for our purposes here! Because we need a wind forecast to make a swell forecast, it means any forecast for storms that haven’t actually happened yet will be A LOT less accurate than one for storms which have. In an already chaotic model Tropical Storm formation and track suffers an even greater margin of error.
One way to counter this variability is to check a number of different models. We’ll start by cautioning this DOESN’T mean checking a number of different surf forecasting sites. Almost all surf forecasting websites use the same model in the short-term (The reliable GFS/NWW3 combination) and that means that, by and large, any significant differences in accuracy are in the presentation of the data. Instead you need to head directly to sources of differing raw model data for example:
And look for a consensus view. Do bear in mind that lots of people telling you the same thing doesn’t make it more probable – the average of a lot of completely uninformed guesses is another uninformed guess (Feynman’s emperor’s nose parable is a perfect example), however our models aren’t completely uninformed. Generally we’d consider cases where models were in agreement to represent atmospheric scenarios that were more stable (ie. more linear) and less likely to vary. It’s reasonable to assume there’s a correlation between model agreement and probability of a particular scenario occurring.
So how does this affect our perfect Hurricane pictured above? We’ll as of right now the limited number of differing models operating at the range we’re talking about shows no consensus, only the GFS is showing this storm in the range we’re talking about.
Comparing different models compares different mathematical equations AND different starting data. As we’ve mentioned both can be an issue. A really robust approach to testing the latter is called ‘ensemble modelling’. Running a series of models at the same time using subtly different starting data. Let’s imagine you’re trying to forecast the swell at your local spot and all you know is the wind speed in the centre of the storm from a satellite reading. You can create a rough idea of the size of arriving swell from this. Now lets assume that that satellite is accurate to about 10% of the actual wind speed and the reading is 40mph. You could do your calculations for 40mph, 36mph and 44mph and see how this changes things. Now as we’ve explained above for a swell model if the error at start is around the 10% mark, it’s roughly reasonable to assume the error in the swell height will be similar. But the atmospheric model is very chaotic, by running (in our case) 20 simultaneous models with slightly different starting conditions we can often highlight huge changes in the prediction over a longer timeframe.
So for our Hurricane pictured above a dig into the data of 20 different variations of our normal model shows that around 80% of variations show this storm forming. That’s a reasonably positive sign in favour of the storm, or at least the models confidence in it. The chart below is one from a new MSW product that gives you that probability in graphical form. Here the Blue/Green blob at the bottom of the chart represents not the size of the storm, but the probability of a storm generating large swell at this location in 10 days time:
Model Run Variation
Anyone familiar with MSW will appreciate that, particularly in the long term, things can change. In fact we update every forecast for every spot four times a day, running the latest observed data back through the model and updating the output. Sometimes we’ll see huge jumps from ‘run’ to ‘run’ (a run is a single model case) and other times a lot less so. Typically the shorter the forecast timeframe the less variation. Again it’s reasonable to assume there’s a level of confidence inferred by changing starting points not significantly affecting the outcome.
We’ve been talking about this storm since Friday last week and it’s appeared on every model run since then. That’s 12 model runs consistently showing this same storm. Another positive indicator in favour.
The ‘common sense’ test
If the heir to a Nigerian fortune emails you offering to exchange his inheritance for a $1000 cheque you don’t need a degree in statistics to smell a rat. It’s utterly improbable. We call this the ‘common sense’ test – but what we really mean is comparing a predicted outcome to the range of historic possibility. Are 50ft seas likely in the Great Lakes in August? Can a long period swell find its way to French Mediterranean beaches? Do we get significant Atlantic Hurricanes in August?
Yes we can and do get significant Atlantic Hurricanes in August! The chart below shows just the last 20 years worth of Hurricane Category storms in August originating in the Atlantic. That’s 14 storms in 20 years – roughly a 70% chance of a comparable storm (failing to allow for conditions making grouping of storms into particular seasons likely). That’s not the same as a 70% probability of this very specific forecast scenario – but neither is it reason to discount it.
We’ve mentioned judging a prediction against past outcomes, but there’s also a quick sanity check on outcomes vs past predictions worth throwing into the mix. How reliably have past forecasts in this range predicted events of this type?
For our storm now we have limited data, with just over a year of 16 day data accumulated. However we do have one shining example of the possibilities of this system, having spotted a 16 day chart very much like this one for Hurricane last year. Hurricane Katia kindly followed our long range forecast track hitting NY with perfect timing for the 2011 Quiksilver Pro New York.
So there you have it – Hurricane Gordon: Some good reasons to have faith, and plenty of reasons to maintain a healthy level of skepticism. As always we’ll continue to update here on MSW and full storm tracking will be available when (and if) Hurricane Gordon materializes.