Substorm Onset Prediction using Machine Learning Classified Auroral Images

In our publication we classify all sky images and use the classified images to predict whether a substorm will occur within 15 minutes after a 30-minute interval of processed image data has been shown to a classifier. Our data is archived on the NIRD Research Data Archive. It is licensed under CC-BY 4.0 and available for at least 10 years after publication. The archive with accompanying information can be accessed here: https://doi.org/10.11582/2022.00070
On this website we will provide and describe the code we used in our publication for replication of our results.

1. Setup

Make sure all dependencies are installed:

   sudo apt-get install python3 wget p7zip

Download our code:

   mkdir ~/SubstormOnsetPrediction
   cd ~/SubstormOnsetPrediction
   wget http://tid.uio.no/SOP/data/code.7z
   7z x code.7z

Create a conda environment from the environment file we provide. This may take a few moments. Afterwards, activate the environment, to use it. You can find installation instructions for conda here:https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html

   conda env create -f environment.yml
   conda activate OnsetPrediction

Download our fully preprocessed data archives and extract them:

   wget https://ns9999k.webs.sigma2.no/10.11582_2022.00070/data.7z -P
   7z x data.7z -o./data

After downloading code and data, your folder structure should look like this:

   .
   ├── data
   │   ├── images
   │   ├── magn
   │   ├── other
   │   └── themis
   ├── dataHandler
   │   ├── DataClasses
   │   │   ├── datasets.py
   │   │   ├── __init__.py
   │   │   ├── TimeFrame.py
   │   │   └── TimeSplit.py
   │   ├── helpers
   │   │   ├── common.py
   │   │   ├── constants.py
   │   │   ├── hmmclasses.py
   │   │   ├── __init__.py
   │   │   └── logger.py
   │   ├── __init__.py
   │   └── processors
   │       ├── DataHandler.py
   │       ├── __init__.py
   │       ├── Provider.py
   │       ├── SD.py
   │       └── SubstormDetection.py
   ├── environment.yml
   ├── index.html
   ├── LICENSE
   └── README.md

2. Execution:

Run Python

   python

Import our library and start processing

   from dataHandler.processors.SubstormDetection import SubstormDetection
   sd = SubstormDetection()
   sd.binned_substorm_regression(False)
   sd.image_progression_before_storm()
   sd.get_images_for_workflow()

This will create the figures we show in our publication and save them in ./data/images.

3. Retrieval of original data

All data used for this project is available under open licenses. We provide information on how to obtain the original data here, and make our processed data available alongside. Information on how to obtain the processed data can be found in 1.4.
The all sky imager data is obtained from THEMIS. A description of the data can be found here and the original data here
The list of substorms is obtained through SuperMAG and can be downloaded here

Questions

If you have any questions or remarks, please send me an e-mail.

References

The data is archived here:
https://doi.org/10.11582/2022.00070
If you have not already done so, please read our publication based on this data:
https://doi.org/10.1002/essoar.10512391.2

If you use any part of this library or our publication, you can cite us the following way:

@article{10.1002/essoar.10512391.2,
   author = {Sado, Pascal and Clausen, Lasse Boy Novock and Miloch, Wojciech Jacek and Nickisch, Hannes},
   title = {Substorm Onset Prediction using Machine Learning Classified Auroral Images},
   journal = {Earth and Space Science Open Archive},
   pages = {15},
   year = {2022},
   DOI = {10.1002/essoar.10512391.2},
   url = {https://doi.org/10.1002/essoar.10512391.2}
}

Acknowledgements and Copyright

The source code in this library is licensed under a BSD-2-Clause License. Unless stated otherwise, all data contained in the datasets we provide ourselves alongside this publication under the links above are licensed under a Creative Commons Attribution-NonCommercial 4.0 License (CC BY-NC 4.0). The copyright for the all-sky imager data, some data-files are derived from, remains with the original copyright holder, NASA / the THEMIS project. The copyright of the substorm data remains with their respective authors: https://supermag.jhuapl.edu/substorms/?tab=about