Production Runbook — 10.10.8.230 (/home/sms/radio)

The broadcast node runs the live chain. Access: SSH as sms.

The chain

Liquidsoap (radio_worker.liq) ──▶ Icecast :8005/main  (OGG)
                                       │  ffmpeg bridge (AAC 256k → FLV)
                                       ▼
                                MediaMTX :19350/19351  (HLS)
                                       ▲
          WHIP server :8089 ──▶ FIFO whep_live.wav ──▶ ffmpeg ─▶ harbor :8006
                                                          (channel_live switch)

Crontab (all services self-heal)

Schedule	Script	Purpose
*/3 * * * *	watch_radio.sh	Liquidsoap worker watchdog
*/5 * * * *	watch_whip.sh	WHIP server watchdog
@reboot sleep 20	start_loklok_bridge.sh	HLS ffmpeg bridge after reboot
@reboot sleep 25	radio-web/start-player.sh	Next.js player after reboot
@reboot sleep 30	watch_whip.sh	WHIP server after reboot
@reboot sleep 40	watch_chat.sh	Long-poll chat server after reboot
@reboot sleep 45	video_feed/watch_video.sh	LOKLOK TV renderer + ticker after reboot
1 17 1,30 * * + @reboot sleep 120	turn renew	TURN credential cron + refresh
every 12h	update_turn_creds.py	coturn static-auth-secret rotation

Start Liquidsoap

The worker is started by the launch script:

cd /home/sms/radio
./start_liquidsoap.sh

Which runs:

liquidsoap radio_worker.liq -- main /home/sms/radio/playlists/main.m3u hackme localhost 8005

Argument	Value	Meaning
Channel	main	The main broadcast channel
Playlist	/home/sms/radio/playlists/main.m3u	Starting playlist (rotation continues from the staged M3Us)
Source password	hackme	Icecast source password
Icecast host	localhost	Local Icecast
Icecast port	8005	Source mount port

Watchdog — watch_radio.sh

watch_radio.sh keeps the worker alive (run it from cron):

1. Dedupe — if multiple liquidsoap …radio_worker.liq processes exist, kill all but the first.
2. Restart — if none exist, relaunch liquidsoap radio_worker.liq -- main …/main.m3u hackme localhost 8005 with output to /home/sms/radio/radio.log.
3. Health — if healthy, curl http://127.0.0.1:8005/main and log the bytes served to /home/sms/radio/watch_radio.log. It never kills a healthy worker (prevents restart loops).

Start the HLS bridge

The HLS path needs the ffmpeg bridge running:

cd /home/sms/radio
./start_loklok_bridge.sh &

The watchdog loops, keeping an ffmpeg process alive that converts the Icecast OGG stream to AAC 256k and pushes it to MediaMTX. Log: /home/sms/radio/loklok_bridge.log. The input uses -reconnect so transient Icecast errors never restart the process (a restart recreates the MediaMTX muxer and 404s connected players — see Streaming).

Changing the bridge bitrate requires killing the watchdog too (pkill -9 -f start_loklok_bridge.sh + relaunch via the @reboot cron command) — the running watchdog re-spawns ffmpeg from its old in-memory loop.

Start the LOKLOK TV video channel

The branded 1080p video program lives in /home/sms/radio/video_feed/:

cd /home/sms/radio/video_feed
setsid nohup bash watch_video.sh < /dev/null &

watch_video.sh supervises both daemons: video_bridge.sh (the ffmpeg renderer → MediaMTX loklokvideo) and video_ticker.js (the text-file feeder), restarting either on death. Logs to watch_video.log.

Verify it is live:

mpv rtsp://files.mediahubnetwork.net:19354/loklokvideo          # cookie-free
curl -s http://localhost:19351/loklokvideo/index.m3u8           # HLS master

Edit headlines any time: echo "…" >> /home/sms/radio/video_feed/news.txt. See LOKLOK TV for the full design and gotchas.

Start the Next.js player

The player is served on port 4477 by a watchdog:

cd /home/sms/radio/radio-web
./start-player.sh &

start-player.sh loops next start -p 4477, logging to /home/sms/radio/radio-web/start.log, and restarts after 3 seconds if the process exits.

Start the WHIP server

The mobile live-ingest server runs from whip_server.py (aiohttp + aiortc) in the whip-venv/ virtualenv:

cd /home/sms/radio
./start_whip.sh        # launches whip_server.py :8089

watch_whip.sh (cron */5) restarts it if it dies and logs health (HTTP 200) to /home/sms/radio/watch_whip.log.

Verify it is up:

curl -s http://127.0.0.1:8089/whip-info   # { stun: [...], turn: [...], live: ... }

The mobile page is at http://10.10.8.230:8089/go-live. Restarting the WHIP server after a code change is safe — the Windows engine's LiveManager dedupes pipe/WebRTC sources, so re-registration keeps exactly one stream.

Do not kill WHIP with a plain pkill -f whip_server.py if a second stale instance may exist — prefer pkill -9 and let watch_whip.sh relaunch a single fresh instance.

Verify the stream is live

• Icecast status: curl -s http://localhost:8005/status-json.xsl (or the public mount).
• Public master stream: https://files.mediahubnetwork.net/main.
• Public HLS: https://api.mediahubnetwork.net/loklok/index.m3u8 (302 cookie redirect → 200; same-origin with the page — see Streaming).
• Now playing: GET /api/status on the control plane.
• WHIP server: curl -s http://127.0.0.1:8089/whip-info.
• Live takeover: connect a source to the harbor source:hackme@10.10.8.230:8006/channel_live (see Live ingest).

Updating content

Content is pushed from the local node over HTTP pull (the engine runs a remote curl against the token-gated /api/sync/* endpoints — see Media Sync); sync-media-full.js is the one-shot full-mirror exception (scp, run from a user session with keys).

After a sync:

ls -lt /home/sms/radio/playlists/   # newest M3U should be staged
ls -lt /home/sms/radio/media/       # new files present
tail -f /home/sms/radio/logs/       # as-run log advancing

Logs

Log	Location
As-run (air-check)	/home/sms/radio/logs/
Liquidsoap output	/home/sms/radio/radio.log
Worker watchdog	/home/sms/radio/watch_radio.log
HLS bridge	/home/sms/radio/loklok_bridge.log
WHIP server	/home/sms/radio/whip_server.log
WHIP watchdog	/home/sms/radio/watch_whip.log
Player watchdog	/home/sms/radio/radio-web/start.log
Video feed	/home/sms/radio/video_feed/watch_video.log (+ .ffmpeg.pid / .node.pid)

See also

• Broadcast chain
• Troubleshooting