Skip to content

Commit

Permalink
Manual download zst conversion needs --stdout flag
Browse files Browse the repository at this point in the history
  • Loading branch information
ginic committed Mar 13, 2024
1 parent 0bbd433 commit 73fcb3e
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions dvc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ stages:
foreach: ${download_months}
do:
desc: Download Reddit comments data for the month ${item}.
cmd: curl https://files.pushshift.io/reddit/comments/RC_${item}.zst | unzstd --long=31 | bzip2 > ${comments_dir}/RC_${item}.bz2
cmd: curl https://files.pushshift.io/reddit/comments/RC_${item}.zst | unzstd --long=31 | bzip2 > ${comments_dir}/RC_${item}.bz2
outs:
- ${comments_dir}/RC_${item}.bz2

Expand All @@ -14,7 +14,7 @@ stages:
foreach: ${manual_download_months}
do:
desc: Converting manually downloaded Reddit comments to bzip for month ${item}.
cmd: unzstd --long=31 ${manual_downloads_dir}/comments/RC_${item}.zst | bzip2 > ${comments_dir}/RC_${item}.bz2
cmd: unzstd --stdout --long=31 ${manual_downloads_dir}/comments/RC_${item}.zst | bzip2 > ${comments_dir}/RC_${item}.bz2
outs:
- ${comments_dir}/RC_${item}.bz2

Expand Down

0 comments on commit 73fcb3e

Please sign in to comment.