Content Pipeline (content-pipeline.js)

The content pipeline is the acquisition → preparation → delivery chain. It turns raw internet audio into loudness-matched, metadata-clean, radio-ready files and pushes them to the broadcast node.

Stages

Stage details

1. Discovery & download

The pipeline selects fetch targets from the Content Manager catalogs and dynamic discovery terms, then downloads with yt-dlp using --embed-metadata so ID3 tags land inside the file. The download archive (yt_download_archive.txt) prevents re-fetching the same video ID.

2. Sanitize

yt-dlp stores the raw YouTube video title in the TIT2 tag. The pipeline re-derives clean {artist, title} pairs by stripping:

• | separators (e.g. Artist | Song)
• / feature splits (e.g. Artist / Artist2)
• [Official ...], (Official Audio), (...) decorations
• known label / channel suffixes

See Registry & Metadata for the exact rules and the standalone sanitizer tool.

3. Validate

Tracks are checked against curated artist catalogs — only established stars are accepted for a category. Gemini AI resolves ambiguous cases (see Gemini AI). Label regex filters screen out obvious non-song material. Since 2026-08-11 a _BAD_TERMS blocklist additionally bans AI-generated music at every gate (yt-dlp --reject-title pre-download, post-download BAD_TITLE, scrubCategory, deriveMeta re-check) so AI-slop tracks can never re-enter the library (see Architecture).

4. Loudness

Every accepted asset is normalized with ffmpeg loudnorm to EBU R128 (−14 LUFS / −1.5 dBTP / 11 LRA) so the whole log sits at broadcast level.

5. Registry

All results are cached in meta_registry.json keyed by file, holding clean artist/title, category, energy, and analysis fields. The scheduler reads from this registry instead of re-probing files.

6. Sync

radio-engine.js ships playlists and media to 10.10.8.230:/home/sms/radio over HTTP pull (one ssh per file only runs the remote curl; sizes are verified). See Media Sync for the full mechanism. sync-media-full.js is the one-shot full-mirror exception (scp, user session).

Category map & voice tags

• LIB maps each category to its directory under MEDIA_ROOT including a staging area where new downloads land before promotion.
• Two voice tags are referenced as station identity markers:
  • voice/loklok2.mp3
  • voice/loklokwhisper.mp3

CLI wrapper (scripts/run-content-pipeline.js)

Operators drive the pipeline through a thin CLI wrapper:

Flag	Meaning
--sweepers	Fetch/process sweepers
--mashups	Fetch/process mashups
--sfx	Fetch/process sound effects
--songs=N	Fetch/process N songs
--ads=N	Fetch/process N ads
--no-fetch	Skip downloads (process existing downloads only)
--no-purge	Keep originals after processing (skip cleanup)
--count	Report counts then exit
--ids	Operate on specific video IDs

See Scripts & Tools for the full script inventory.