An intresting article on how to use the Python CDS API to downlaod and read ERA5 Data into memory with some worked examples.
Read ERA5 into Memory using Python
1 Like
Does CDS encourage people to read data directly into memory? Under current circumstances, I would think they would prefer not. Reading into memory implies you are not downloading a file, which implies that if you need the same data later, you will have to make another CDS request, rather than read from a local file you previously downloaded. Considering that CDS is often overwhelmed with requests, I wonder if they would prefer that users download files and re-use that data, rather than making new requests into memory for potentially the same data that was requested previously.
It’s a good point @markstoelinga they probably do prefer us to download and reuse. That’s how we have always done it also.
I was wondering if there would be any speed benefits with this method as I have experienced some lengthy download times from the climate data store in the past ?
@Mdavid800 My guess is that there would be no speed benefit. The wait in the queue would be the same, as well as the processing of the data request, and the data transfer. The only thing that would be faster would be on your machine, where the data is placed directly into memory rather than written to disk…but either of those would be more than able to keep up with the download speed.
1 Like