Skip to content

Latest commit

 

History

History
132 lines (114 loc) · 30.8 KB

README.md

File metadata and controls

132 lines (114 loc) · 30.8 KB
Raw

ytfeed

Build Status GitHub GitHub tag (latest by date) GitHub All Releases codecov GoDoc Go Report Card FOSSA Status Donate

Automatic Youtube video and stream archiver.

Configuration

Set configuration by setting up environment variables.

Environment Variable Description Default Value Is Required
YTFEED_YOUTUBE_API_KEY Youtube Data API Key. true
YTFEED_VERIFICATION_TOKEN Verification token used to subscribe and unsubscribe topics. true
YTFEED_VERIFICATION_SECRET Hmac secret used to subscribe and unsubscribe topics. true
YTFEED_RESUB_CALLBACK_ADDR Callback address to ytfeed. true
YTFEED_RESUB_TARGET_ADDR The subscription page of pubsubhubbub. https://pubsubhubbub.appspot.com/subscribe
YTFEED_RESUB_TOPIC The topic the subscription should subscribe to, for example https://www.youtube.com/xml/feeds/videos.xml?channel_id=mychannelid, can be space separated for multiple topics, required. true
YTFEED_RESUB_INTERVAL The interval between resubscription. 72h
YTFEED_STORAGE_BACKEND The storage backend, required. Must be one of disk, gcs, or s3. true
YTFEED_S3_ENDPOINT The S3 compliant server endpoint, required if YTFEED_STORAGE_BACKEND is s3.
YTFEED_S3_ACCESS_KEY_ID The secret access key id for the S3 compliant server, required if YTFEED_STORAGE_BACKEND is s3 .
YTFEED_S3_SECRET_ACCESS_KEY The secret access key id for the S3 compliant server, required if YTFEED_STORAGE_BACKEND is s3 .
YTFEED_S3_BUCKET_NAME The bucket name for the S3 compliant server, required if YTFEED_STORAGE_BACKEND is s3 .
YTFEED_GCS_CREDENTIAL_JSON_FILE_PATH The JSON credential file for GCS, only used if YTFEED_STORAGE_BACKEND is gcs .
YTFEED_GCS_BUCKET_NAME The bucket name for GCS, required if YTFEED_STORAGE_BACKEND is gcs .
YTFEED_DISK_DIRECTORY The disk directory path, required if YTFEED_STORAGE_BACKEND is disk.
YTFEED_FILENAME_TEMPLATE The filename template. The usable variables are .ChannelID, .VideoID, .Published, .Title, .PublishedYear, .PublishedMonth, .PublishedDay, .PublishedHour, .PublishedMinute, .PublishedSecond, .PublishedNanosecond, .PublishedTimeZone, .PublishedTimeZoneOffsetSeconds, .VideoQuality, .VideoExtension, and .Author. {{.ChannelID}}/{{.PublishedYear}}/{{.PublishedMonth}}/{{.PublishedDay}}/{{.PublishedTimeZone}}/{{.VideoID}}.{{.VideoExtension}}
YTFEED_HOST The host address. :8123
YTFEED_VIDEO_FORMAT_QUALITY The quality of the video to download, must be one of 1080, 720, 640, 480, 360, 240, or 144. 720
YTFEED_VIDEO_FORMAT_EXTENSION The extension of the video to download. webm
YTFEED_VIDEO_DOWNLOAD_RETRY_DELAY Delay time when retrying, set to activate retries. Must be Golang time duration string. Example: 5m
YTFEED_VIDEO_DOWNLOAD_MAX_RETRIES Maximum retries before giving up. 5
YTFEED_REDIS_ADDR Redis address, required if you want to publish the data to Redis PubSub.
YTFEED_REDIS_USERNAME
YTFEED_REDIS_PASSWORD
YTFEED_REDIS_CHANNEL Redis publish channel. ytfeed
YTFEED_REDIS_DB
YTFEED_REDIS_MAX_RETRIES
YTFEED_REDIS_DIAL_TIMEOUT
YTFEED_REDIS_WRITE_TIMEOUT
YTFEED_REDIS_READ_TIMEOUT
YTFEED_REDIS_POOL_SIZE
YTFEED_REDIS_MIN_IDLE_CONNS
YTFEED_REDIS_MAX_CONN_AGE
YTFEED_REDIS_POOL_TIMEOUT
YTFEED_REDIS_IDLE_TIMEOUT
YTFEED_REDIS_IDLE_CHECK_FREQUENCY
YTFEED_BOLTDB_PATH Set this to a file path if you want to activate stream scheduler.
YTFEED_STREAM_SCHEDULER_RETRY_DELAY Retry delay of the scheduler. 1m
YTFEED_AMQP_DSN AMQP DSN, required if you want to publish the data to AMQP broker.
YTFEED_AMQP_EXCHANGE ytfeed
YTFEED_AMQP_KEY schedule
YTFEED_AMQP_PUBLISH_MANDATORY true
YTFEED_AMQP_PUBLISH_IMMEDIATE false
YTFEED_AMQP_EXCHANGE_KIND topic
YTFEED_AMQP_EXCHANGE_DURABLE true
YTFEED_AMQP_EXCHANGE_INTERNAL false
YTFEED_AMQP_EXCHANGE_AUTO_DELETE false
YTFEED_AMQP_EXCHANGE_NO_WAIT false

Example of fairly common configuration is:

export YTFEED_VERIFICATION_TOKEN='sometoken'
export YTFEED_VERIFICATION_SECRET='somesecret' 
export YTFEED_RESUB_TOPIC="https://www.youtube.com/xml/feeds/videos.xml?channel_id=channelid"
export YTFEED_RESUB_CALLBACK_ADDR='https://my.server.addr/path'
export YTFEED_STORAGE_BACKEND='s3'
export YTFEED_S3_ENDPOINT='my.s3.endpoint'
export YTFEED_S3_ACCESS_KEY_ID='acceskeyid'
export YTFEED_S3_SECRET_ACCESS_KEY='secretaccesskey'
export YTFEED_S3_BUCKET_NAME='ytfeed'
export YTFEED_YOUTUBE_API_KEY='myyoutubeapikey'
export YTFEED_RESUB_INTERVAL='24h'

export YTFEED_VIDEO_FORMAT_QUALITY='720'
export YTFEED_VIDEO_FORMAT_EXTENSION='webm'
export YTFEED_VIDEO_DOWNLOAD_RETRY_DELAY='5m'

export YTFEED_BOLTDB_PATH='/home/ytfeed/schedule.db'
export YTFEED_STREAM_SCHEDULER_WORKER_INTERVAL='1m'

Building

Your ol' plain go build cmd/ytfeed/main.go

Note

  • To run s3 test or complete suite test, you need to run test_s3.sh for it to start a Minio server. You can stop it or delete it after testing by running test_s3_teardown.sh
  • You also need to set YTFEED_YOUTUBE_API_KEY, YTFEED_YOUTUBE_VIDEO_ID, and YTFEED_YOUTUBE_VIDEO_URL to do a complete suite test.
  • You can redo failed download by sending a POST request with the XML message in the log as the body.
  • If you are planning to download live broadcast, the downloaded video will be in the format mp4 regardless of your extension config.
  • If you send SIGINT while ytfeed is downloading video, it will only kills the immediate youtube-dl process, you have to make sure ffmpeg process youtube-dl spawned is also killed and the resulting temporary directory and file removed.
  • If the program got killed because of OOM, you can turn on swap file if you cannot raise the machine's memory. Or you can PR me a better way to handle the upload.
  • If you want to use stream scheduler, make sure file that is pointed at YTFEED_BOLTDB_PATH is already exists. You can create the file using touch $YTFEED_BOLTDB_PATH command.

Before Using

  • You MUST install youtube-dl first and MUST be available at $PATH.
  • You MUST install ffmpeg too for this to work.
  • You MUST* run the program first and make sure it could be reached from the internet, then go here to subscribe to a channel.
  • From the page in the URL above:
    • Fill Callback URL field with the address of your program, for example http://my.ip.address:8123. This must be the same as the value of YTFEED_RESUB_CALLBACK_ADDR.
    • Fill Topic URL field to be the same as your YTFEED_RESUB_TOPIC.
    • Choose Verify type field as synchronous.
    • Choose Mode field as subscribe.
    • Fill Verify token field to be the same as your YTFEED_VERIFICATION_TOKEN.
    • Fill HMAC secret field to be the same as your YTFEED_VERIFICATION_SECRET.
    • Let Lease seconds field to be empty.
    • Press Do it button.
    • There should be a log that says something like INFO[0635] Subscribed to topic https://www.youtube.com/xml/feeds/videos.xml?channel_id=channelid with challenge 4776706698370686416.

Usage

Set the required environment variables and run the binary like ./ytfeed and you'll see log messages if it runs.

How to Contribute

  • Keep your code super simple and clean.
  • Add comments to your code if you have to explain what your code is doing.
  • Make sure you have Commitizen CLI installed and your commit message must be written through git cz.
  • Pull request to branch other than development will be rejected.