Rotation Operations Guide
A practical, task-oriented guide to the on-air rotation — how it lives, how it is rebuilt, and how to operate it safely without breaking the stream. This is the operator's counterpart to Rotation Science & Comparison (the theory) and Smart Scheduler (the code).
Who this is for: anyone who may ever need to touch the station's playlist — a weekend engineer answering a call, a content operator who wants to force a song, or a developer debugging "why is the same song playing again." It assumes you know what the rotation is but not necessarily how it behaves; by the end you should be able to operate it safely and explain to a colleague why each rule exists.
1. The mental model
The on-air rotation is one file: playlists/main.m3u on the production node
(/home/sms/radio/playlists/main.m3u). Liquidsoap watches it with
reload_mode="watch" and restarts from its head on every in-place write.
Everything in this guide follows from that single fact.
Think of the file as the state and the write as the commit. The scheduler commits a new rotation only when it is genuinely warranted (consumed, or the daypart changed), and the delivery path is designed so that a commit produces exactly one reload — never zero (frozen rotation), never two (double-replay of the head). Every operational rule in this guide reduces to: commit rarely, commit atomically, commit once.
| Property | Value |
|---|---|
| Rotation length | 120 tracks (ROTATION.BUFFER_MAX) |
| Rebuild trigger | ≥85% consumed OR daypart changed (see §4) |
| Rebuild excludes | Tracks aired within MUSIC_FRESH_MS (3 h) |
| File size (remote) | Exactly 32768 bytes (FIXED_M3U_SIZE) |
| Remote write method | dd if=tmp of=final conv=notrunc bs=1M (one IN_MODIFY) |
| Reloads per push | Exactly 1 (verified) |
2. When does the rotation rebuild?
rotationNeedsRefresh() returns true only when:
- The buffer is empty.
rotationConsumed / length >= REFRESH_FRACTION(0.85).- The daypart profile changed since the rotation was built (daypart-stamped rebuild, 2026-08-15).
Any other updateM3U() call short-circuits with Rotation stable — skipping rewrite. This stability is deliberate: rewriting the file makes Liquidsoap
reload and rewind, which is exactly how "same songs repeating" happened
historically.
Why 85% and not 100%? The rebuild is a multi-step operation (build → push → verify → reload) that takes real time. The 15% margin (~18 tracks ≈ over an hour of airtime) guarantees the rotation can never run dry mid-rebuild. Treat this number as a latency budget, not a threshold to be tuned toward 100%.
What triggers a rebuild and what doesn't:
| Event | Rebuild? |
|---|---|
| 85% of rotation consumed | Yes |
| Daypart boundary crossed (02:00, 05:00, 11:00, 16:00, 23:00 Cairo) | Yes, within 60 s |
| Server restarts | Yes (fresh boot rebuilds) |
| Content refresh purges files | Yes (resyncSchedulerAfterRefresh forces updateM3U(true)) |
POST /api/station/rotation/rebuild | Yes (force bypasses freshness) |
| Heartbeat ticks, pool-health checks | No (short-circuit: "Rotation stable") |
3. Common operations
3.1 Force a rebuild now
From the control plane (local node):
curl -X POST http://localhost:5000/api/station/rotation/rebuild
or use the MCP tool rebuild_rotation. This calls updateM3U(true) (force
bypasses the freshness gate), rebuilds from current decks, and pushes to the
remote. Verify:
ssh sms@10.10.8.230 "wc -c /home/sms/radio/playlists/main.m3u" # expect 32768
ssh sms@10.10.8.230 "grep -c '^.*m3u8\|mp3' /home/sms/radio/playlists/main.m3u"
The engine logs Rotation verified on remote (N files present, size-checked).
When to use it: after adding/removing content, after a ban/scrub, or when you need the new daypart or a new imaging element on air now. When not to use it: don't rebuild "just to be fresh" on a cadence — each rebuild rewinds the stream to a new head, which is the exact mechanism that replays songs if done casually.
3.2 Skip the current track
curl -X POST http://localhost:5000/api/station/rotation/rebuild
(plus the skip_track MCP tool) — rebuilds fresh so the head cannot be the
just-played track.
Semantics: the rebuild excludes the just-played track (via lastTrackPlayed
and MUSIC_FRESH_MS), so the new head is guaranteed different from what just
aired. A skip is therefore not "advance one line" — it is "regenerate from
scratch with the current track disqualified." Use it for genuinely bad tracks;
for anything else, the rotation will fix itself at the next natural rebuild.
3.3 Check rotation health
curl -s http://localhost:5000/api/station/state | C:\nvm4w\nodejs\node.exe -e "let d='';process.stdin.on('data',c=>d+=c).on('end',()=>{const s=JSON.parse(d);console.log(JSON.stringify({nowPlaying:s.nowPlaying,queue:s.queue&&s.queue.length,rotation:s.rotation&&s.rotation.length,decks:s.decks},null,2))})"
3.4 The 60-second daypart watchdog
A watchdog in server.js ([M3U] Daypart watchdog fired) compares the stamped
daypart key against the live key every 60 s and calls rotateBatch() within a
minute of any boundary crossing. This guarantees the 02:00 deep-night English
gate takes effect even if the rotation is only 10% consumed. There is nothing to
operate — it self-heals.
Why it exists: before the stamp, a rotation built at 23:59 would keep airing its mixed content through the entire English-only window until 85% consumption (~5 hours later) — the exact "Arabic leaks at 04:50" incident that produced this watchdog. It is the safety net behind the daypart key: even if every other rebuild trigger misses, this probe catches the boundary within a minute.
4. The daypart rules
The rotation is stamped with the profile that built it. The key mirrors
_pickMusicCategory exactly:
Deep night (02–05) is a hard English-only gate; the other windows apply
weight multipliers (see Equations §3). If you set custom
SCHEDULER_CONFIG.DAYPARTS, the key becomes
custom:start-end:hard:weights and the same boundary logic applies.
5. The critical traps (read before touching the remote)
5.1 Never hand-edit main.m3u on the remote
Any rewrite/mv of the remote m3u replaces the inode Liquidsoap watches.
watch mode uses inotify on the old inode, so the watch never fires again
and the rotation silently freezes (pushed updates land on an inode liq never
watches). This exact incident froze the rotation on 2026-08-15.
- If you must edit the remote m3u by hand, restart Liquidsoap afterwards so it re-arms the watch on the live inode.
- The only sanctioned write path is the engine's dd-notrunc push.
5.2 Do not use mv -f for the m3u
mv = new inode = broken watch. cat tmp > final = two IN_MODIFY events =
double reload = head-replay. Only dd conv=notrunc (single write syscall, inode
preserved, file pre-padded to 32768 bytes) is correct.
5.3 Failed experiment history — do not retry
| Attempt | Result |
|---|---|
reload_mode="poll" | Reloads ~1/s unconditionally — catastrophic |
reload_mode="off" | Also ~1/s + manual telnet bursts |
mv-rename m3u | Breaks inotify → frozen rotation |
Both reverted to watch. Remote backups: radio_worker.liq.bak.*,
radio_worker.liq.pollbad.*.
5.4 Anti-repeat state persists across restarts
E:\radionew\models\weights\scheduler_antirepeat.json holds trackHistory
(pruned to 72 h), lastTrackPlayed, a 40-entry _recentOnAir,
_lastMusicArtist, and rotationConsumed. It is saved on every
markOnAirTrack() and rotation write — so a server restart never wipes
cooldowns (the root cause of restart-replay). Do not delete it casually; a
clean-slate deletion is a deliberate operation (done once, 2026-08-07, to clear
a polluted file).
5.5 A rebuild is not "the rotation fixes itself forever"
The rotation is self-healing in mechanism (the consumption trigger, the
daypart watchdog, the pool-health monitor all fire without a human) but it is
not infallible. If a deck is empty or a category is banned to nothing, the
rebuild will happily produce a thin or weird rotation. When you see something
odd on air, first check the decks (/api/station/state shows sizes), not
just the rotation — an empty deck is a content problem, and no amount of
rebuilding fixes a content problem.
6. Verifying a healthy rotation
| Check | Command / endpoint | Expect |
|---|---|---|
| Remote size | ssh sms@10.10.8.230 "wc -c playlists/main.m3u" | 32768 |
| No duplicate lines | ssh … "sort playlists/main.m3u | uniq -d | wc -l" | 0 |
| Signature jingle present | ssh … "grep -c loklok1_mixdown playlists/main.m3u" | ~7 per rotation |
| One reload per push | grep 'Reloading playlist' /home/sms/radio/radio.log | 1 per recent push |
| No cross warnings | grep 'End of track reached while buffering' radio.log | 0 (after liq restart) |
| Short-item override | ssh … "grep -c liq_cross_duration playlists/main.m3u" | 51 (stochastic) |
| Remote files exist | Engine log Rotation verified on remote (… size-checked) | no MISS |
7. Rotation tuning cheat-sheet
Edit SCHEDULER_CONFIG.ROTATION (or use the MCP set_rotation tool):
| Knob | Default | Effect of raising |
|---|---|---|
BUFFER_MAX | 120 | Longer rotation = fewer rebuilds = more stable but staler |
REFRESH_FRACTION | 0.85 | Fewer rewrites = fewer reloads = less rewind risk |
MUSIC_FRESH_MS | 3 h | Longer freshness = more variety per rebuild, thinner pools |
COOLDOWN_PERIOD | 90 m | Longer = fewer repeats, fewer eligible tracks |
ELEMENT_COOLDOWN | 10 m | Imaging recycle rate |
SIG_JINGLE_MIN/MAX_SLOTS | 2/5 | Signature jingle cadence (~every 30–45 min) |
Tuning logic, not just values: every knob is a trade, never a free
improvement. Raise BUFFER_MAX and you get fewer reloads but a staler rotation
(the head is older relative to the library). Raise MUSIC_FRESH_MS and you get
more variety per rebuild but thinner candidate pools (a deck with 30 eligible
tracks and a 3 h window may have only ~20 left). Raise COOLDOWN_PERIOD and you
reduce repeats at the cost of reusing a smaller working set. Change one knob at
a time, verify against the §6 checklist, and give the station a full daypart
cycle before judging.
8. Failure signatures (diagnostic table)
Symptom → likely cause → first check. Keep this list handy; most "the radio is broken" reports reduce to one of these rows.
| Symptom | Likely cause | First check |
|---|---|---|
| "Same songs repeating" | Rotation rewritten too often (or double-reload) | grep 'Reloading playlist' radio.log — expect 1 per push |
| "Rotation frozen" (nothing changes for hours) | m3u inode replaced (manual edit / mv) | ls -i main.m3u vs the inode liq watches → restart liq |
| "Arabic leaks at 04:50" | Stale rotation from a previous daypart | /api/station/state → check stamped daypart; trigger rebuild |
| "The rotation is only 5 tracks" | Deep-night English gate with a thin deck | Check english deck size; a short buffer is correct in the gate |
| "Empty deck / one-track playlist" | Deck starved below cap | /api/station/state deck sizes → run content refresh |
| "Crossfade warnings" | Short item missing liq_cross_duration | grep liq_cross_duration main.m3u → should be ~51 |
| "On-air shows stale metadata" | Engine died / metadata webhook down | Engine log; guard respawns in ~2 min |
| "No reload on push" | Watch broke (inode) or liq not running | watch_radio.sh health probe; restart liq |
See also
- Rotation Science & Comparison — the theory.
- Smart Scheduler — the implementation.
- Equations & Math Reference — the mathematics.
- Production runbook — the broadcast node.