In my usual scrolling of the Python-o-sphere I came across the project Scalable Offshore Wind Analysis With Pangeo by D. O’Callaghan, and S. McBreen from Earth Observation Offshore (EOOffshore) School of Physics, in University College Dublin (UCD)
I thought this may be of great interest to the group - it sure is to me.
The project uses a combination of xarray, Dask and Zarr to enable scalable wind data processing that generates power estimates for offshore renewable energy assessment for the Irish Continental Shelf region. This has involved the creation of a new wind data catalog using Intake and Zarr, which features up to 21 years of available data products from various providers.
Details of the current project outputs are available at https://eooffshore.github.io . This includes a collection of Jupyter notebooks describing:
- Data retrieval and Zarr store creation
- Area Of Interest (AOI) assessment - the Irish Continental Shelf region
- A prototype interactive wind atlas, created using HoloViz libraries
I think the thing that interests me the most is the Approach here. We have large quantities of data and the approach as far as I can see uses Zarr, Intake, xarray & Dask to allow this to be used directly - is this something we think can be used for even larger scale data sets, without dumbing down to the statistics? @oriol I know when we started PYWRAM this is something you were particularly interested to explore. @neildavis / @bjarketol NEWA is used here as an example (no 4 below) - is Zarr something the DTU team have looked at ?
the processed data sets in Zarr format for 5 models can be found below:
- EOOffshore: ASCAT Wind Data for the Irish Continental Shelf Region | Zenodo
- EOOffshore: CCMP v0.2.1.NRT Wind Data for the Irish Continental Shelf Region | Zenodo
- EOOffshore: ERA5 Wind Data for the Irish Continental Shelf Region | Zenodo
- EOOffshore: New European Wind Atlas (NEWA) Data for the Irish Continental Shelf Region | Zenodo
- EOOffshore: Sentinel-1 Wind Data for the Irish Continental Shelf Region | Zenodo
Further details of the above data sets can be found here: Irish Continental Shelf Data Sets (eooffshore.github.io)
The original post by Derek is from Offshore Wind Analysis with Pangeo - Science - Pangeo.