I'm running a Python command + API to access ECMWF (European Centre for Medium range Weather Forecast) data server (called MARS) and download some files (weather data). I launch the Python code with shell (I'm using csh) doing ./python_script.py in the shell
It runs fine if I download a single file (year 1999 with range(1999, 2000) in my script, instead of range(1998, 2000) in the below failing example of Python script). Now I want to download many of them and thus make a loop over years.
My problem is that it seems the Python script doesn't wait for the shell command(s) / API to be finished and continue on the next year. It results in errors. The file are produced but with zero size.
I'd like to know if I can specify the Python script to wait to find some key words in the shell window before proceeding to the next for/loop step.
I know I'm using some specific API in that case and could probably find another API-specific solution to do so, but it seems easier to me to identify some print out in the shell.
It could be for instance "Transfer rate" which seems to appear in the shell window only when the job has been done, see the log (last successful lines) I've saved from shell (with ./python_script.py >& log_file.log.
MY PYTHON CODE IS:
#!/usr/bin/env python
for year in range(1998, 2000):
from ecmwfapi import ECMWFDataServer
server = ECMWFDataServer()
server.retrieve({
"class": "e2",
"dataset": "era20c",
"date": '%d-07-01/%d-07-02' % (year,year),
"domain": "g",
"area" : "12/-72/-67/22",
"grid" : "1.0/1.0",
"expver": "1",
"param": "214.140/233.140",
"step": "3/9/15/21",
"format" : "netcdf",
"stream": "wave",
"target": '/home/nicolas/hycom/hycom_data/ECMWF/ERA20C/forecast/%d/test_era20c_wave_set1.nc' % (year),
"time": "06",
"type": "fc",
})
MY LOG LAST LINES FOR ONE FILE DOWNLOADING ONLY (SUCCESSFUL):
2016-02-13 16:00:21 Request is complete
2016-02-13 16:00:21 Transfering 239.441 Kbytes into /home/nicolas/hycom/hycom_data/ECMWF/ERA20C/forecast/1999/test_era20c_wave_set1.nc
2016-02-13 16:00:21 From http://stream.ecmwf.int/data/atls04/data/data01/scratch/_grib2netcdf-atls04-95e2cf679cd58ee9b4db4dd119a05a8d-JLUk0w.nc
2016-02-13 16:00:28 Transfer rate 32.6278 Kbytes/s
There's no need to import in the loop, also it might be indentaiton problem. If your script is as you provided it, then the retrieve is not in the loop. Indentation in python matters s lot.
Try rewriting the script this way:
#!/usr/bin/env python
from ecmwfapi import ECMWFDataServer
server = ECMWFDataServer()
for year in range(1998, 2000):
server.retrieve({
"class": "e2",
"dataset": "era20c",
"date": '%d-07-01/%d-07-02' % (year,year),
"domain": "g",
"area" : "12/-72/-67/22",
"grid" : "1.0/1.0",
"expver": "1",
"param": "214.140/233.140",
"step": "3/9/15/21",
"format" : "netcdf",
"stream": "wave",
"target": '/home/nicolas/hycom/hycom_data/ECMWF/ERA20C/forecast/%d/test_era20c_wave_set1.nc' % (year),
"time": "06",
"type": "fc",
})