some requests to soundxyz's API fail, are we rate limited?

[TimDaub]

• see this crawl: https://github.com/neume-network/data/runs/7089365507?check_suite_focus=true#step:7:855

(node:1887) UnhandledPromiseRejectionWarning: Error: FetchError: request to https://sound.xyz/api/metadata/38/1/14 failed, reason: connect ETIMEDOUT 104.26.1.213:443
    at Worker.<anonymous> (file:///home/runner/work/data/data/src/strategies/src/lifecycle.mjs:151:13)
    at Worker.emit (events.js:400:28)
    at MessagePort.<anonymous> (internal/worker.js:236:53)
    at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:399:24)
    at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26)
(node:1887) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 121)
(node:1887) UnhandledPromiseRejectionWarning: Error: Error: Request unsuccessful with status: 400
    at Worker.<anonymous> (file:///home/runner/work/data/data/src/strategies/src/lifecycle.mjs:151:13)
    at Worker.emit (events.js:400:28)
    at MessagePort.<anonymous> (internal/worker.js:236:53)
    at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:399:24)
    at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26)

[il3ven]

I am also getting the same error locally. Sometimes I also get a DNS error.

Error: soundxyz-get-tokenuri: FetchError: request to https://www.sound.xyz/api/metadata/55/1/18 failed, reason: getaddrinfo ENOTFOUND www.sound.xyz
    at Worker.<anonymous> (file:///Users/vaibhav/repos/il3ven/music-os/core/src/strategies/src/lifecycle.mjs:152:13)
    at Worker.emit (node:events:527:28)
    at MessagePort.<anonymous> (node:internal/worker:232:53)
    at MessagePort.[nodejs.internal.kHybridDispatch] (node:internal/event_target:643:20)
    at MessagePort.exports.emitMessage (node:internal/per_context/messageport:23:28)

As a solution we can implement retry functionality in extraction-worker.

[TimDaub]

As a solution we can implement retry functionality in extraction-worker.

as long as we don't manage retry state ourselves (because that'd become mad complex), I'd be fine with that. We're using

extraction-worker/package.json

Line 63 in 57eccc6

"better-queue": "3.8.10",

, I'll transfer this issue over.

[il3ven]

We can use this for retry. https://github.com/diamondio/better-queue#retry

maxRetries: 3,
retryDelay: 1000