How to retrieve data in a more time efficient manner using GetOrders?

[sktong]

Hi,

Our business requires to retrieve at least 1 year order data from API and I try to use GetOrders as follows:

@throttle_retry()
@load_all_pages()
def load_all_orders(**kwargs):
"""
a generator function to return all pages, obtained by NextToken
"""
return Orders(credentials=credentials).get_orders(**kwargs)

orders=[]
for page in load_all_orders(LastUpdatedAfter=(datetime(2021,4,1,0,0,0)).isoformat(),
LastUpdatedBefore=(datetime(2022,4,1,0,0,0)).isoformat()):
for order in page.payload.get('Orders'):
orders.append(order)
orders = pd.DataFrame(orders)

I am aiming to save the data into a data frame. But I found that the code above does not work for me: either too slow to retrieve the result or experience the timeout error. Can anyone tell me how I can retrieve the data in such a long time window and save it into data frame more efficiently? Thanks so much

[IP050]

Pull 'old' data, until month ago and store into json/csv file. If you run into timeout errors, this way you can just split it into different timeframes and extend the json/csv file. You will only need to do this once.

Create dataframe based on file, do request on recent data with api, extend your dataframe with new data and extend csv file with new data or overwrite current df.

You can build your own logic based on timeframes, there's no need to constantly pull years worth of data(data that is probably definitive in values). For orders you could build logic not based on time but on status for example. Up to you, good luck.

[sktong]

Great ones. Appreciate it.