Paginated API request for asset

[dagsir[bot]]

Hi everyone, Dagster newbie here. What's the best way to handle a paginated API request for an asset?

Should I loop through all pages within the asset

@asset
def messages():
  next_page = None
  has_more = True
  all_data = []

  while has_more:
    response = fetch_data_from_api(page = next_page)
    all_data = all_data.concat(response.data)
    next_page = response.next_page:
    if not next_page:
      has_more = False

  store_data(all_data)

Or is there a way to somehow store the next page cursor as context where Dagster knows to call messages again with new new cursor?

---

The question was originally asked in Dagster Slack.

[jamiedemaria]

Looping through all of the pages is certainly a good option/approach. if there is a reasonable way to partition the data (ie on a date field) then you could also look into partitioning your asset and only fetch + store data for a particular partition when that partition is materialized https://docs.dagster.io/concepts/partitions-schedules-sensors/partitions