Weather4cast Data Access

[Aleksandra Gruca (network admin)]

Dear Competitors,

The Weather4cast competition task is to predict quantitatively future high resolution rainfall events from lower resolution satellite radiances.

The competition data comprise of 7 regions for 2019 and 2020 years of training data for the Core Challenge.

In addition, we provide test data for 3 additional regions in 2 years to assess spatial transfer learning, plus test data for all 10 regions in a third year to assess temporal and/or spatial transfer learning.

Last but not least, we now also provide static data with latitude, longitude, and topological height!

You can now clone our github repository and download the data as described in the Starter Kit repo from the data links below.

Learn more about the competition data structure from the Starter Kit and hit the road running with the provided baseline model! Write to us on the forums if you have any questions or suggestions!

Data structure

• The HRIT data are available by download via sftp.
• The OPERA data are provided as two ZIP archive files, one each for 2019 and 2020.
• The static data are provided as a ZIP archive file.

For the larger HRIT files, we provide an sftp repository.

Download links are available on the Weather4cast Data Download page.

After downloading all the data your final directory tree structure for individual files (HRIT, OPERA) should look as follows:

+-- data --
    +-- 2019 --
        +-- HRIT --
            +-- boxi_0015.test.reflbt0.ns.h5 - 182M
            +-- boxi_0015.train.reflbt0.ns.h5 - 16G
            +-- boxi_0015.val.reflbt0.ns.h5 - 1.7G
            +-- boxi_0034.test.reflbt0.ns.h5 - 187M
            +-- boxi_0034.train.reflbt0.ns.h5 - 16G
            +-- boxi_0034.val.reflbt0.ns.h5 - 1.7G
            +-- boxi_0076.test.reflbt0.ns.h5 - 179M
            +-- boxi_0076.train.reflbt0.ns.h5 - 16G
            +-- boxi_0076.val.reflbt0.ns.h5 - 1.7G
            +-- roxi_0004.test.reflbt0.ns.h5 - 303M
            +-- roxi_0004.train.reflbt0.ns.h5 - 26G
            +-- roxi_0004.val.reflbt0.ns.h5 - 2.8G
            +-- roxi_0005.test.reflbt0.ns.h5 - 303M
            +-- roxi_0005.train.reflbt0.ns.h5 - 26G
            +-- roxi_0005.val.reflbt0.ns.h5 - 2.8G
            +-- roxi_0006.test.reflbt0.ns.h5 - 310M
            +-- roxi_0006.train.reflbt0.ns.h5 - 26G
            +-- roxi_0006.val.reflbt0.ns.h5 - 2.8G
            +-- roxi_0007.test.reflbt0.ns.h5 - 291M
            +-- roxi_0007.train.reflbt0.ns.h5 - 25G
            +-- roxi_0007.val.reflbt0.ns.h5 - 2.8G
            +-- roxi_0008.test.reflbt0.ns.h5 - 266M
            +-- roxi_0009.test.reflbt0.ns.h5 - 312M
            +-- roxi_0010.test.reflbt0.ns.h5 - 297M
        +-- OPERA --
            +-- boxi_0015.train.rates.crop.h5 - 134M
            +-- boxi_0015.val.rates.crop.h5 - 13M
            +-- boxi_0034.train.rates.crop.h5 - 128M
            +-- boxi_0034.val.rates.crop.h5 - 16M
            +-- boxi_0076.train.rates.crop.h5 - 75M
            +-- boxi_0076.val.rates.crop.h5 - 7.1M
            +-- roxi_0004.train.rates.crop.h5 - 299M
            +-- roxi_0004.val.rates.crop.h5 - 32M
            +-- roxi_0005.train.rates.crop.h5 - 271M
            +-- roxi_0005.val.rates.crop.h5 - 27M
            +-- roxi_0006.train.rates.crop.h5 - 236M
            +-- roxi_0006.val.rates.crop.h5 - 28M
            +-- roxi_0007.train.rates.crop.h5 - 64M
            +-- roxi_0007.val.rates.crop.h5 - 7.7M
    +-- 2020 --
        +-- HRIT --
            +-- boxi_0015.test.reflbt0.ns.h5 - 307M
            +-- boxi_0015.train.reflbt0.ns.h5 - 31G
            +-- boxi_0015.val.reflbt0.ns.h5 - 2.7G
            +-- boxi_0034.test.reflbt0.ns.h5 - 301M
            +-- boxi_0034.train.reflbt0.ns.h5 - 31G
            +-- boxi_0034.val.reflbt0.ns.h5 - 2.8G
            +-- boxi_0076.test.reflbt0.ns.h5 - 294M
            +-- boxi_0076.train.reflbt0.ns.h5 - 31G
            +-- boxi_0076.val.reflbt0.ns.h5 - 2.7G
            +-- roxi_0004.test.reflbt0.ns.h5 - 310M
            +-- roxi_0004.train.reflbt0.ns.h5 - 31G
            +-- roxi_0004.val.reflbt0.ns.h5 - 2.7G
            +-- roxi_0005.test.reflbt0.ns.h5 - 303M
            +-- roxi_0005.train.reflbt0.ns.h5 - 31G
            +-- roxi_0005.val.reflbt0.ns.h5 - 2.7G
            +-- roxi_0006.test.reflbt0.ns.h5 - 304M
            +-- roxi_0006.train.reflbt0.ns.h5 - 31G
            +-- roxi_0006.val.reflbt0.ns.h5 - 2.8G
            +-- roxi_0007.test.reflbt0.ns.h5 - 298M
            +-- roxi_0007.train.reflbt0.ns.h5 - 30G
            +-- roxi_0007.val.reflbt0.ns.h5 - 2.7G
            +-- roxi_0008.test.reflbt0.ns.h5 - 260M
            +-- roxi_0009.test.reflbt0.ns.h5 - 299M
            +-- roxi_0010.test.reflbt0.ns.h5 - 303M
        +-- OPERA --
            +-- boxi_0015.test.reflbt0.ns.h5 - 307M
            +-- boxi_0015.test.reflbt0.ns.h5 - 307M
            +-- boxi_0015.train.rates.crop.h5 - 229M
            +-- boxi_0015.val.rates.crop.h5 - 22M
            +-- boxi_0034.train.rates.crop.h5 - 273M
            +-- boxi_0034.val.rates.crop.h5 - 27M
            +-- boxi_0076.train.rates.crop.h5 - 103M
            +-- boxi_0076.val.rates.crop.h5 - 12M
            +-- roxi_0004.train.rates.crop.h5 - 383M
            +-- roxi_0004.val.rates.crop.h5 - 32M
            +-- roxi_0005.train.rates.crop.h5 - 353M
            +-- roxi_0005.val.rates.crop.h5 - 32M
            +-- roxi_0006.train.rates.crop.h5 - 266M
            +-- roxi_0006.val.rates.crop.h5 - 22M
            +-- roxi_0007.train.rates.crop.h5 - 107M
            +-- roxi_0007.val.rates.crop.h5 - 16M
    +-- 2021 --
        +-- HRIT --
            +-- boxi_0015.test.reflbt0.ns.h5 - 305M
            +-- boxi_0034.test.reflbt0.ns.h5 - 304M
            +-- boxi_0076.test.reflbt0.ns.h5 - 304M
            +-- roxi_0004.test.reflbt0.ns.h5 - 306M
            +-- roxi_0005.test.reflbt0.ns.h5 - 303M
            +-- roxi_0006.test.reflbt0.ns.h5 - 309M
            +-- roxi_0007.test.reflbt0.ns.h5 - 299M
            +-- roxi_0008.test.reflbt0.ns.h5 - 262M
            +-- roxi_0009.test.reflbt0.ns.h5 - 306M
            +-- roxi_0010.test.reflbt0.ns.h5 - 308M
        +-- OPERA --    to be predicted!
    +-- static --
        +-- boxi_0015.HRIT.static.lat-long.h5 - 676K
        +-- boxi_0015.HRIT.static.topo.h5 - 116K
        +-- boxi_0015.OPERA.static.lat-long.h5 - 604K
        +-- boxi_0015.OPERA.static.topo.h5 - 12K
        +-- boxi_0034.HRIT.static.lat-long.h5 - 652K
        +-- boxi_0034.HRIT.static.topo.h5 - 176K
        +-- boxi_0034.OPERA.static.lat-long.h5 - 572K
        +-- boxi_0034.OPERA.static.topo.h5 - 20K
        +-- boxi_0076.HRIT.static.lat-long.h5 - 644K
        +-- boxi_0076.HRIT.static.topo.h5 - 116K
        +-- boxi_0076.OPERA.static.lat-long.h5 - 560K
        +-- boxi_0076.OPERA.static.topo.h5 - 12K
        +-- roxi_0004.HRIT.static.lat-long.h5 - 680K
        +-- roxi_0004.HRIT.static.topo.h5 - 64K
        +-- roxi_0004.OPERA.static.lat-long.h5 - 588K
        +-- roxi_0004.OPERA.static.topo.h5 - 12K
        +-- roxi_0005.HRIT.static.lat-long.h5 - 684K
        +-- roxi_0005.HRIT.static.topo.h5 - 92K
        +-- roxi_0005.OPERA.static.lat-long.h5 - 592K
        +-- roxi_0005.OPERA.static.topo.h5 - 16K
        +-- roxi_0006.HRIT.static.lat-long.h5 - 656K
        +-- roxi_0006.HRIT.static.topo.h5 - 168K
        +-- roxi_0006.OPERA.static.lat-long.h5 - 576K
        +-- roxi_0006.OPERA.static.topo.h5 - 20K
        +-- roxi_0007.HRIT.static.lat-long.h5 - 636K
        +-- roxi_0007.HRIT.static.topo.h5 - 88K
        +-- roxi_0007.OPERA.static.lat-long.h5 - 552K
        +-- roxi_0007.OPERA.static.topo.h5 - 16K
        +-- roxi_0008.HRIT.static.lat-long.h5 - 600K
        +-- roxi_0008.HRIT.static.topo.h5 - 144K
        +-- roxi_0008.OPERA.static.lat-long.h5 - 604K
        +-- roxi_0008.OPERA.static.topo.h5 - 16K
        +-- roxi_0009.HRIT.static.lat-long.h5 - 648K
        +-- roxi_0009.HRIT.static.topo.h5 - 180K
        +-- roxi_0009.OPERA.static.lat-long.h5 - 564K
        +-- roxi_0009.OPERA.static.topo.h5 - 20K
        +-- roxi_0010.HRIT.static.lat-long.h5 - 660K
        +-- roxi_0010.HRIT.static.topo.h5 - 132K
        +-- roxi_0010.OPERA.static.lat-long.h5 - 608K
        +-- roxi_0010.OPERA.static.topo.h5 - 20K

• This topic was modified 1 year ago by wpadmmiriam.
• This topic was modified 1 year ago by Aleksandra Gruca (network admin).
• This topic was modified 1 year ago by Aleksandra Gruca (network admin).
• This topic was modified 1 year ago by Aleksandra Gruca (network admin).
• This topic was modified 1 year ago by Aleksandra Gruca (network admin).
• This topic was modified 1 year ago by Aleksandra Gruca (network admin).
• This topic was modified 1 year ago by Aleksandra Gruca (network admin).
• This topic was modified 1 year ago by Aleksandra Gruca (network admin).
• This topic was modified 1 year ago by wpadmmiriam.
• This topic was modified 1 year ago by wpadmmiriam.
• This topic was modified 1 year ago by wpadmdkreil.
• This topic was modified 1 year ago by wpadmdkreil.
• This topic was modified 1 year ago by wpadmdkreil.
• This topic was modified 1 year ago by Aleksandra Gruca (network admin).
• This topic was modified 1 year ago by Aleksandra Gruca (network admin).

[rpablos]

Hello,

Thank you for the detailed instructions! Authentication seems to be failing for me when connecting to the SFTP server. I have tried the password provided (including the exclamation mark). Could you please help?

Thanks in advance,
Rafa

[Aleksandra Gruca (network admin)]

Hi Rafa,

Thanks for letting us know!

We fixed the problem and we also updated the password on the Data Download website.
Please try again! All should work now.

[rpablos]

It is perfect now, thanks a lot!

[ajkmr]

Hello. ‘static.zip’ file does not contain files for ‘boxi’ regions. It has files only for ‘roxi’ regions.

[jyoteesh]

Hi W4C23 team, Thanks for arranging the things for the competition. We have looked at the topography provided in the static data folder of ‘roxi’ files. We find that the high-res opera data is not exactly in the center of the low-res HRIT data (see the plot in the below link). Can you provide us the lat lon coordinates for the “HRIT reflbt” and “OPERA rates.crop” data? One more question: in “OPERA rates.crop” data, large values like -8888000 exist. Do those refer to “no data” or zero rain rate? Please provide us with clarification. Thank you.

https://drive.google.com/file/d/1y-F-YvXLzr5oQgMKKUL6EoHFP_rYGJBp/view?usp=share_link

[Aleksandra Gruca (network admin)]

Hello ajkmr!

Thanks a lot for letting is know.

We have now included also boxis region into the static.zip data file. Please download it again!

[ajkmr]

Hello Aleksandra, Thanks a lot for updating static.zip file!

[Aleksandra Gruca (network admin)]

Hi Jyoteesh,

We thoroughly checked the regions crops for static data and it seems that it is correct – the centre of the OPERA data is exactly a centre of the satellite data. Please check if your plotting code is correct.

Please find below are our plots of the static data for the roxi-04 region:

EDIT – the longitudinal coordinates for the satellite picture are wrong – please see the conversation below

Radar static data for the roxi-04 region:

Satellite static data for the roxi-04 region:

• This reply was modified 11 months, 3 weeks ago by Aleksandra Gruca (network admin).
• This reply was modified 11 months, 3 weeks ago by Aleksandra Gruca (network admin).
• This reply was modified 11 months, 3 weeks ago by Aleksandra Gruca (network admin).
• This reply was modified 11 months, 3 weeks ago by Aleksandra Gruca (network admin).

[Aleksandra Gruca (network admin)]

Hi Jyoteesh,

Thank you very much for your questions about the negative values in OPERA rates.crop files!

In addition to measured rain rates you can find additional values in OPERA rate files:
-8888000 – it means there is no signal for the particular pixel in the area that is covered by radars (that can be due to some device measurement errors)
-9999000 – it means there is no coverage of radar over this particular regions (this will happen mostly on the sea surface)

For the leaderboard scoring we map -8888000 values to zero, and the -9999000 values are masked out from the evaluation.

• This reply was modified 11 months, 3 weeks ago by Aleksandra Gruca (network admin).

[harishbaki]

Hello,

I appreciate your prompt response and for providing the visual plots. However, upon a closer examination, I’ve observed a few discrepancies in the plots. Specifically, it seems that the latitude coordinates in both figures do not align correctly when centered. Additionally, while the geographical region depicted in the plots bears a resemblance to the British Isles, the latitude and longitude values do not correspond to the geographical coordinates typically associated with that region (typically ranging from 50N 10W to 60N 2E).

Could you kindly provide some clarification regarding the actual geographic region represented in these plots? Your assistance in resolving these discrepancies would be greatly appreciated.

Thank you for your attention to this matter.

[Aleksandra Gruca (network admin)]

Hi Harishbaki,

Thanks a lot for your post.

Just to clarify – we do not provide the exact location of the regions of interests.

Indeed there was a mistake in my plot above regarding the latitude values. Thank you for bringing that into my attention and please find the corrected version below:
Radar static data for the Roxi-04 region:

Satellite static data for the Roxi-04 region:

Please also note that visually, due to the projection of the pixels to longitude and latitude coordinates, the region of interest is no longer at the center. Even though it is still central when we take pixel coordinates into account.

So, for comparison, below I have plotted only topological information for Roxi-004, showing the region of interest on the satellite plot with the red rectangle. Just to remind you, in OPERA super resolution, the region of interest has a size of 252×252, while in satellite resolution, the region of interest has a size of 42×42. The whole satellite picture with the context has a size of 252×252

Radar topology only for the Roxi-04 region:

Satellite topology only for the Roxi-04 region:

Hope this helps!

[hexm]

Hello, I want to ask how to download the data quickly. When using the remote server I cannot connect to the sftp server due to the “time out” error, and by using FileZilla to transport the data is too slow.
Thank you for your attention

[Author]

Posts