Open Avalanche Project--Improving Avy Forecasting with Machine Learning

[scottcha]

Hey maggots, it's been awhile since I've been around here. Glad to see things haven't changed.
I just launched a new project https://openavalancheproject.org .

I've been thinking about this idea for years and I finally set down to build it over the past year. Basically, I connected the historical weather data up to the forecast histories to teach an algorithm to predict the avy danger. Similar technology is used to train your Netflix recommendations or self-driving cars.

Currently, I'm just using NWAC data and am only forecasting one day but I eventually want to expand to cover any problems, aspects, and multiple days in the future. We should also be able to do some cool things on the forecast like more frequent updates during storms, increased forecast resolution and cover areas without avy centers and what not. I also want to figure out how to best partner with the local avy centers to enable them with this technology. All the code and data is also available.

Let me know your thoughts!

[LightRanger]

Very interesting. I'm a dentist, not a coder, but thanks for doing this stuff. Will be cool to watch it develop.

Sent from my SM-G930V using Tapatalk

[Sirshredalot]

I'm more of an R guy than a Python guy, but I will check out the repo when I have a free minute.

Thanks for getting this project rolling.

[abraham]

You should talk to cmor, or at least check out his winterscience.com site. There's some information about it on here, and he did a presentation at the nwac snow science event two years ago(?) The video is on YouTube

[scottcha]

Thanks for the feedback guys. I have checked out winterscience and do know about his work but would be worth having another followup with him.

[rod9301]

How many years of days were you able to get?

Sent from my Moto G (5) Plus using TGR Forums mobile app

[cmor]

Your confusion matrices look insanely accurate. The Above treeline forecast says it matches nwacs forecast 97% of the time. However today it's forecasting low and nwac is at considerable.

I think there is a problem in how you are splitting the data. It's likely mixing rows it shouldn't, for example the model can learn off of snoqualmie's forecast for a day and then validate the Stevens forecast for the same day. Since they are highly correlated it makes for an easy win. You could split by day, but even day to day it's highly correlated. I would train on 14-17 and then predict 18 and validate. That's how I've validated some of my similar work with nwac's data. Mixing more than that and you start to over-fit.

There's been a bunch of papers trying to doing similar work that have been presented at issw, you could check them out. I would say the reception has been mixed at best for those guys. They struggle to be accurate, and even if someone could get a decent model going there is a lot of bias against the idea. Eventually it will be part of the forecasting work flow. Some day.

[scottcha]

Yeah, I agree with the concern about the last couple of days of forecasts (I went ahead and put a statement on the site stating they are incorrect). I need to look in more to why and it could be several factors but my theory is that its a pretty complicated forcast for the last couple of days and my model isn't picking up on the combination of elements going in to the forecast.

I did double check about the train/test split. I'm pretty sure its ok. Every forecast point is entirely independent for every day and only knows about the weather and snow data at that point and nothing else. Previous day forecasts or other regions forecasts are not part of the training for a point. I did do an experiement once and allowed the previous day forecast to be a part of the training and as expected got a big bump in accuracy but I was concerned that in the long term this would lead to increasing innaccuracies via a feedback loop. The jupyter notebook should be pretty easy to clone and repro if you want to check it out and let me know if you see anything of concern: https://github.com/scottcha/OpenAvalancheProject/blob/master/ML/XGBoostV1TrainingNotebook.ipynb


I do like the idea though of training on a few years and then using another year for testing. I'll attempt that and publish what the results are.

I have seen a few of the papers you mentioned (eventually once I clean it up I'll put my whole list of references up on my repro) but will check what happened as the latest ISSW. I agree that an approach including ML will definitely be part of the forecasting though I think we just need to figure out the right way to do it.

[scottcha]

Quick update here. @cmor you were totally right. There was overfitting due to how I did the train/test split as well as a few other issues I corrected over the summer. I wrote a more in depth analysis here: https://blog.openavalancheproject.or...eason-preview/ but the tldr is there is lots of work left to do but I do think its something we can get to a point where it provides value to the community.