How do I parse a metar csv file?

[ManojAgrawal]

Hello All,
I have a metar csv file downloaded from https://mesonet.agron.iastate.edu/request/download.phtml?network=CO_ASOS

which has the structure

station valid metar
DEN 12/31/16 23:53 KDEN 310053Z 33014KT 10SM FEW120 SCT180 SCT250...
DEN 1/1/17 0:53 KDEN 010053Z 02007KT 10SM FEW150 M03/M07 A2973...
DEN 1/1/17 1:53 KDEN 010153Z 01005KT 10SM CLR M03/M07 A2974 RM...
DEN 1/1/17 2:53 KDEN 010253Z 00000KT 10SM CLR M03/M07 A2974 RM...
DEN 1/1/17 3:53 KDEN 010353Z 18005KT 10SM CLR M06/M08 A2974 RM...

How do I use metpy to run this file and create a dataframe for all the rows in one go? Currently, I am iterating through all the rows and appending parsed data into a dataframe one by one which is taking a lot of time.

i tried io.parse_metar_to_dataframe(den_df['metar']) but that doesn't work.

thanks,
Manoj Agrawal

[dopplershift]

Yeah, you definitely don't want to keep appending to a dataframe one-by-one because that (I believe) will result in making a copy of the entire dataframe every time you do that. This leads to O(N^2) behavior. Here's what MetPy's code does, which should work for your case:

import contextlib

from metpy.io.metar import ParseError
import pandas as pd

metars = []
for metar in df['metar']:
    with contextlib.suppress(ParseError):
        # Parse the string of text and assign to values within the named tuple
        metars.append(parse_metar(metar, year=year, month=month))
df = pd.DataFrame(metars)

Appending to a Python list and then turning that into a DataFrame will avoid the O(N^2) behavior.

We have an internal function _metars_to_dataframe() that just takes an iterable and does what you want, but we never had a use case to make it public before. I'm wondering if we should make that public...

[ManojAgrawal]

Looks like I still have to iterate over my csv file and parse metars one by one. Just that instead of appending to a dataframe I append to a list and then convert to a dataframe. Am I right?

Isn't there a way that I can pass the dataframe of metars and get a dataframe of parsed metars back?

[ManojAgrawal]

ah okay. yes, that 2017 and 12 was just to test how to input the year and month. will need to figure out to pass it dynamically. thanks again! :-)

[akrherz]

I'm the author of the mesonet download page you referenced. When you export the data from there, pick the UTC timezone for the valid timestamp. Then when you load the CSV into pandas, that valid column can be used to provide metpy the year and month values as it will be a datetime-ish object.

[ManojAgrawal]

Hi Daryl,

I have the valid time stamp in the csv file and I am passing metars as suggested by Ryan as below

all_metars = ' '.join(line.split(',')[4] for line in open('/Users/magrawal/repos/TMI/data/FAA/metar_den2.csv', 'rt'))
df = metpy.io.parse_metar_file(StringIO(all_metars))

my csv file contains 3 yrs metar data and I have no idea how to pass year and month in the above code unless I break the file into groups of year and month.

thanks,

[dopplershift]

You need to expand that loop to parse the date/time string, or at least the month/year off it. At this point it's probably easier to read the code if you do it this way:

import contextlib

from metpy.io.metar import ParseError
import pandas as pd

metars = []
for line in open('/Users/magrawal/repos/TMI/data/FAA/metar_den2.csv', 'rt'):
    _, dt_str, metar = line.split(',')
    date = dt_str.split(' ')[0]
    _, month, year = date.split('/')
    with contextlib.suppress(ParseError):
        metars.append(parse_metar(metar, year=year, month=month))
df = pd.DataFrame(metars)

[ManojAgrawal]

Thanks Ryan!