Troubleshooting

Stutter on short non-song items (stingers/sfx/sweepers)

Symptom: the station stutters specifically around short items (stingers 0.24–0.31s, sweepers 3.5–9s). Root cause: Liquidsoap cross(duration=2.0) pre-buffers 2s of the incoming track; a 0.24s stinger under a 2s window is buffer starvation. Check the remote log:

grep -c 'End of track reached while buffering' /home/sms/radio/radio.log

0 expected. Non-zero means the rotation predates the liq_cross_duration fix — rebuild it and restart liq if it was frozen (below).

Fix (deployed 2026-08-15): the scheduler annotates every short/imaging M3U entry with annotate:type=<type>,liq_cross_duration=<d>:<path> (CROSS_OVERRIDE: sfx/shorts 0.2s, jingles/sweepers/ads/news 0.5s; music keeps 2s). Verify the deployed rotation carries it:

grep -c liq_cross_duration /home/sms/radio/playlists/main.m3u   # ~51

See Liquidsoap + Equations §12.1.

Rotation frozen on-air (inode-freeze trap)

Symptom: the rotation plays the same content for hours even though the engine logs successful pushes, and Reloading playlist never appears after a push. Cause: Liquidsoap watches the m3u via inotify on a specific inode; any manual remote rewrite/mv of main.m3u replaces that inode, so pushed updates land on a file liq never watches. Check:

stat -c %i /home/sms/radio/playlists/main.m3u   # compare to liq's last reload time

Fix: restart Liquidsoap so it re-arms the watch on the live inode:

kill <liq-pid> && setsid nohup liquidsoap /home/sms/radio/radio_worker.liq \
  -- main /home/sms/radio/playlists/main.m3u hackme localhost 8005 \
  > /home/sms/radio/radio.log 2>&1 < /dev/null &

Never hand-edit the remote m3u — always go through the engine's dd-notrunc push; after any manual remote edit, restart liq.

Repeating songs on air

The station's anti-repeat behaviour is enforced by the stable rotation model (see Anti-repeat & M3U rotation). If repeats come back, check in order:

1. Is the M3U being rewritten constantly? The rotation should only be regenerated at ~85% consumed. In the scheduler log look for Rotation stable (n/120 consumed) — skipping rewrite between regens. If you see Updated M3U with N tracks every few minutes, an old build is deployed (or finalizeSync is re-uploading unchanged content — see #2).
2. Is the engine re-uploading unchanged M3Us? radio-engine.js must log m3u unchanged — skipping remote push on unchanged batches. A re-upload reloads the watched file and rewinds the stream.
3. Is a purge archiving rotation files? A content refresh must never archive nowPlaying, the queue, lastTrackPlayed, or the next 40 rotation tracks (liveProtectedPaths). If liq logs Fetch failed after a refresh, files got archived mid-rotation.
4. Was the just-played track excluded from regen? The regen's safety valve must never re-add the exact track that just played. Check the log for excludePath behaviour on the degenerate "all tracks on cooldown" case.
5. Is the local m3u ever empty? An empty overwrite (Fetch failed: empty) stalls liq; the scheduler refuses to write an empty buffer.

Dead air (nothing playing)

Check in order:

1. Is Liquidsoap running? pgrep -fa radio_worker.liq. Restart via ./start_liquidsoap.sh (or let watch_radio.sh do it) if not.
2. Is there a fresh playlist? ls -lt /home/sms/radio/playlists/ — the M3U must exist and reference files that exist under media/. Mismatched paths are the #1 cause of dead air.
3. Do files exist remotely? Compare the playlist paths to media/. Re-run sync-media-full.js from the local node if files are missing.
4. Is the master stream healthy? curl -s http://localhost:8005/status-json.xsl on the production node, or curl -sI http://localhost:8005/main (expect 200 + bytes).

HLS dead but Icecast works

The HLS path is a separate ffmpeg bridge:

1. curl -sI https://api.mediahubnetwork.net/loklok/index.m3u8 — expect a 302 cookieCheck redirect, then a 200 master (same-origin cookie flow; see Streaming).
2. Check /home/sms/radio/loklok_bridge.log — restart ./start_loklok_bridge.sh if the bridge died. The bridge uses -reconnect so transient Icecast errors are retried inside the process; if the bridge restarts, MediaMTX recreates the HLS muxer (new segment-hash prefix + sequence reset) → connected players 404 and play stale audio — they self-heal via the muxer-generation tracker in index.html, but verify the page is current (hard-refresh, see #6).
3. Confirm MediaMTX is up (pgrep -fa mediamtx) — it ingests RTMP on :19350 and serves HLS on :19351.
4. 401 on audio1_stream.m3u8 — session/cookie issue: the page and HLS URL must be same-origin (api.mediahubnetwork.net); a cross-origin files.mediahubnetwork.net URL makes the cookie third-party and breaks every segment request.
5. _HLS_msn=… → 400 loop — this MediaMTX build rejects all LL-HLS blocking-reload requests; the player must run lowLatencyMode:false. A stale page running old JS shows the 400 forever.
6. "Repeating the same segment" / stale page — browsers cache the player page; force Ctrl+F5. radio_server.py serves / with Cache-Control: no-store so this is a one-time hard-refresh after a fix, not a recurring maintenance step.

Go-Live / live ingest problems

1. WHIP server down — curl -s http://127.0.0.1:8089/whip-info (expect JSON with STUN/TURN). Restart ./start_whip.sh or let watch_whip.sh (cron */5) do it.
2. Mic page won't connect — check TURN: the server mints credentials with HMAC-SHA1 using TURN_SECRET; verify the coturn static-auth-secret matches and that ports 3478/5349 are reachable.
3. Audio never reaches air — confirm the FIFO is flowing: ls -l /home/sms/radio/whep_live.wav (should be a named pipe) and that the engine's LiveManager logged a restream into harbor :8006. The .liq switch gives channel_live priority automatically.
4. Duplicate live streams in /api/station/live — the engine dedupes by url + title + normalized ingest; if duplicates appear the running engine predates the dedupe fix (restart server.js).
5. Go live stays on the mix — the WHIP server re-registers on every new audio track; restarting the WHIP server is safe. If the channel never switches, the restream process (ffmpeg → channel_live) died — check the engine's live monitor log.

Scheduler / playout stalls

• A category under 15 tracks triggers the safety fallback (supplement from arabic_hits). If a niche category is empty, the fallback still keeps the station alive — but investigate why discovery produced nothing.
• Check the scheduler log under LOG_ROOT for selection warnings.

YouTube fetch failures

• Run the smoke test: node scripts/test-youtube-fetch.js.
• Confirm yt-dlp is current (yt-dlp -U), network is up, and yt_download_archive.txt is not full of failed IDs.
• A full clean reload is retag_library.js (destructive — archives the library and clears the archive file first).

Sync failures (local → prod)

• Can you SSH as sms@10.10.8.230? The engine uses ConnectTimeout=15 + keepalives and retries 3×.
• Disk full on the target (df -h /home/sms/radio).
• The engine logs the failed file — check the scheduler/pipeline log for the offending path.

Web / Socket.io issues

• Is server.js up on the local node (port 5000 / public api.mediahubnetwork.net)?
• Frontends expect Socket.io on the same origin — a reverse-proxy that does not forward WebSocket upgrade requests will break live events.
• GET /ping returns a timestamp; GET /api/status returns now-playing. If these work but events don't, the socket transport is the problem.

Player not playing (Next.js)

• HLS endpoint reachable? curl -I https://api.mediahubnetwork.net/loklok/index.m3u8 (expect 302 → 200 cookie flow). If not, the ffmpeg bridge or MediaMTX is down (see above).
• Player process alive? Check /home/sms/radio/radio-web/start.log; the watchdog restarts it every 3s on exit.
• Desktop browsers need the hls.js fallback path (Safari/iOS use native HLS); Chrome in particular needs hls.js. The primary PWA bundles hls.js 1.5.15 and an OGG fallback — if audio is silent but the stream loads, the browser cannot decode the format and the fallback path should have kicked in.

Loudness / level issues

• All assets are normalized to EBU R128 (−14 LUFS) at pipeline time. Files added manually (outside the pipeline) are NOT normalized — re-run the pipeline or normalize manually with ffmpeg loudnorm.

Registry corruption

• Rebuild: scripts/reprobe_registry.js → scripts/sanitize_registry.js → scripts/retag_files.js. See Retag tools.

Checkpoints

Check	How
Control plane alive	GET /api/status
Content pipeline healthy	GET /api/content/status
Station snapshot	GET /api/station/state (rotation, decks, live)
Master stream	https://files.mediahubnetwork.net/main
HLS stream	https://api.mediahubnetwork.net/loklok/index.m3u8 (302 cookie → 200)
Crossfade stutter warnings	grep -c 'End of track reached while buffering' /home/sms/radio/radio.log (expect 0)
M3U override present	grep -c liq_cross_duration /home/sms/radio/playlists/main.m3u (~51)
WHIP server	curl -s http://10.10.8.230:8089/whip-info
As-run advancing	tail -f /home/sms/radio/logs/
Liveness	GET /ping