Internetarchive-YouTube

Poetry-build Supported Python versions PEP8

🚀 GitHub Action and CLI tool to archive YouTube channels by automatically uploading an entire YouTube channel to archive.org in few clicks.

📌 Global Requirements

🔧 Usage

⚡️ GitHub Action: Getting Started

  1. Fork this repository.

  2. Enable the workflows in your fork.

  1. Create a backend database (or JSON bin).

  2. Add your Archive.org credentials to the repository’s actions secrets:
    • ARCHIVE_USER_EMAIL
    • ARCHIVE_PASSWORD
  3. Add a list of the channels you want to archive as a CHANNELS secret to the repository’s actions secrets:

The CHANNELS secret should be formatted like this example:

CHANNEL_NAME: CHANNEL_URL
FOO: FOO_CHANNEL_URL
FOOBAR: FOOBAR_CHANNEL_URL
SOME_CHANNEL: SOME_CHANNEL_URL

Don’t add any quotes around the name or the URL, and make sure to keep one space between the colon and the URL.

  1. Add the database secret(s) to the repository’s Actions secrets:

If you picked option 1 (MongoDB), add this secret:

If you picked option 2 (JSON bin), add this additional secret:

  1. (optional) You can add command line options other than the defaults by creating a secret called CLI_OPTIONS and adding the options to the secret. See the CLI: Getting Started for a list of all the available options.

  2. Run the workflow under Actions manually, or wait for it to run automatically every 6 hours.

That’s it! 🎉

🧑‍💻 CLI: Getting Started

Requirements:

⬇️ Installation:

pip install internetarchive-youtube

Then login to internetarchive:

ia configure

🗃️ Backend database:

export MONGODB_CONNECTION_STRING=mongodb://username:password@host:port

# or add it to your shell configuration file:
echo "MONGODB_CONNECTION_STRING=$MONGODB_CONNECTION_STRING" >> "$HOME/.$(basename $SHELL)rc"
source "$HOME/.$(basename $SHELL)rc"

export JSONBIN_KEY=xxxxxxxxxxxxxxxxx

# or add it to your shell configuration file:
echo "JSONBIN_KEY=$JSONBIN_KEY" >> "$HOME/.$(basename $SHELL)rc"
source "$HOME/.$(basename $SHELL)rc"

⌨️ Usage:

usage: ia-yt [-h] [-p PRIORITIZE] [-s SKIP_LIST] [-f] [-t TIMEOUT] [-n] [-a] [-c CHANNELS_FILE] [-S] [-C] [-m] [-T THREADS] [-k] [-i IGNORE_VIDEO_IDS]

options:
  -h, --help            show this help message and exit
  -p PRIORITIZE, --prioritize PRIORITIZE
                        Comma-separated list of channel names to prioritize when processing videos.
  -s SKIP_LIST, --skip-list SKIP_LIST
                        Comma-separated list of channel names to skip.
  -f, --force-refresh   Refresh the database after every video (Can slow down the workflow significantly, but is useful when running multiple concurrent
                        jobs).
  -t TIMEOUT, --timeout TIMEOUT
                        Kill the job after n hours (default: 5.5).
  -n, --no-logs         Don't print any log messages.
  -a, --add-channel     Add a channel interactively to the list of channels to archive.
  -c CHANNELS_FILE, --channels-file CHANNELS_FILE
                        Path to the channels list file to use if the environment variable `CHANNELS` is not set (default: ~/.yt_channels.txt).
  -S, --show-channels   Show the list of channels in the channels file.
  -C, --create-collection
                        Creates/appends to the backend database from the channels list.
  -m, --multithreading  Enables processing multiple videos concurrently.
  -T THREADS, --threads THREADS
                        Number of threads to use when multithreading is enabled. Defaults to the optimal maximum number of workers.
  -k, --keep-failed-uploads
                        Keep the files of failed uploads on the local disk.
  -i IGNORE_VIDEO_IDS, --ignore-video-ids IGNORE_VIDEO_IDS
                        Comma-separated list or a path to a file containing a list of video ids to ignore.

🏗️ Creating A Backend Database

📝 Notes