Colin Calnan  /  Builds  /  Automated video · Applied AI

16 whole gigs in, 323 song reels out

  • 17 · Dancing in the Dark
  • Reel frame: Colin with his guitar at Intrigue, caption Oh, stand, stand by me01 · Stand By Me
  • Reel frame: Colin under the gazebo with the vineyard behind, caption Some people call me the space cowboy, yeah20 · The Joker
  • Reel frame: a close-up of Colin singing, caption Those were the best days of my life16 · Summer of ’69
That gig, at Intrigue in September 2026. Every tile is a reel it cut and captioned from the one video.
What it is
I film every gig on my phone. I press go and it finds each song, names it, and cuts a reel with the real lyrics.
What it proves
I put AI to work on real video, with checks that stop the wrong song’s lyrics ever reaching the screen.
Measured
At one 83-minute gig it found 20 of 20 songs and named 19 of them right.
Status
In use since May 2026; older footage back to 2022 reeled too
Built
One Saturday evening in May 2026: working by 6:22 pm, last fix 10:10 pm
Made with
Claude, a speech-to-text tool, a free lyrics library
How
Claude Code wrote the code; I set the rules and checked the results

How it works

Three steps from one video to a reel per song

  1. Find the songs by how loud the room is

    I guessed the chatter between songs would be louder than me. It’s the other way round.

    The Intrigue gig's loudness over 83 minutes, with a dashed cut-off line and 20 yellow bands, one for each song found. 0 min 20 min 40 min 60 min 80 min How loud the room was, minute by minute The Intrigue gig's loudness over 83 minutes, with a dashed cut-off line and 20 yellow bands, one for each song found. 0 min 40 min 80 min How loud the room was
    The real Intrigue gig. Each yellow band, loud for over 90 seconds and above the dashed cut-off, is a song it found. The dips are me talking.

  2. Name each song from the words

    Shazam can’t recognise a live cover, so I dropped it the first night. Now a speech-to-text tool writes down a minute of each song, mistakes and all, and Claude names the song from that.

    Intrigue, the song folder it madefirst 8 of 20first 5 of 20
    song_01_Ben_E_King_-_Stand_By_Me.MOV
song_02_U2_-_Where_The_Streets_Have_No_Name.MOV
song_03_Barenaked_Ladies_-_If_I_Had_1000000.MOV
song_04_Matt_Nathanson_-_Come_On_Get_Higher.MOV
song_05_Guns_N_Roses_-_Sweet_Child_O_Mine.MOV
song_06_John_Travolta_Olivia_Newton-John_-_Youre_the_One_That_I_Want.MOV
song_07_Creedence_Clearwater_Revival_-_Have_You_Ever_Seen_the_Rain.MOV
song_08_Johnny_Nash_-_I_Can_See_Clearly_Now.MOV
    The folder reads like my set list. Song 4 is the one it got wrong: I sang Accidentally in Love.

  3. Put the real lyrics on screen

    Each reel starts just before the line that repeats most, usually the chorus. Computers mishear lots of sung words, so the captions now use the song’s published lyrics, timed to my voice within a tenth of a second.

    May: what the computer heardA May reel's caption over the floor by Colin's feet: without a spot This gun's
    Now: the real lyricA September reel's caption: without a spark
    The same Dancing in the Dark line at two gigs. “Spot” was the computer’s guess; “spark” is the song.

What it unlocked

One song, sung across ten gigs

With every gig split and named, I could ask for one song across all of them. Claude found the songs I’d played at seven or more gigs and made 16 reels overnight while I slept, one line per gig.

Six frames from one reel of Take It Easy, each from a different gig: Sandhill Wines, a private party at sunset, Gallagher's Canyon, Cadence, Ex Nihilo Vineyards and Folktale Cider. Each frame has a yellow venue label and the lyric line sung in that room.
Take It Easy across ten gigs, each line a different night. With sound.
A contact sheet of every take of Take It Easy, four frames from each of 13 gigs. BB Hair Studio is marked Last resort: back to camera. A private party in July is marked Skipped: speaker in the way.
Every take on one sheet: a blocked lens skipped, a party with kids left off.
The finished reel, each line a different night. With sound.

Proof

What I measured

3 min 15 sA one-hour gig split, named and made into reels in October, after my speed fixes and on a new Mac. Just splitting a two-hour gig into songs by hand would take me most of a Saturday.About 38 min perhour of video, AugustAbout 3 min 15 s per hour, October
41% of words wrongafter Claude tried to fix what the computer heard, up from 25%. On Maneater it wrote the record’s chorus, not the verse I sang.25% wrong as heard41% after Claude

Found and fixed. The words now come from the published lyrics, never from the AI. The tool that lines lyrics up with my voice will line up any song’s lyrics, even the wrong song’s, so two checks run first: enough shared words, and timings that agree. If either fails, the reel shows what the computer heard.

Words shared with what I sang: the right song's lyrics share 59 to 92 percent, a wrong song's 4 to 9 percent. Words the lyrics share with what I sang Right song59–92% Wrong song4–9% 0%50%100% Words shared with what I sang: the right song's lyrics share 59 to 92 percent, a wrong song's 4 to 9 percent. Words shared with what I sang Right song 59–92% Wrong song 4–9% 0%100%
Measured on three songs, each against every set of lyrics.

AI and me

Who did what

  • The AI (Claude). Wrote nearly all the code; names each song from misheard words.

  • Simple rules, no AI. Loudness, the chorus, both lyric checks.

  • Me. The gigs, the look, the rules, and my ears: I heard a beat missing at every cut in the overnight reels, and had it fixed.

Still breaks

Not fixed yet

  • Some rooms. One cut-off doesn’t fit every room: at one gig two songs ran together and I split them by hand.

  • Misnamed songs. Now and then, like song 4 above.

  • My cousin’s copy. Can’t draw captions yet: it needs my Mac’s fonts.

The full story

Why

I've filmed every gig for years on a phone clipped to the speaker stand. Splitting a two-hour video into songs by hand would take most of a Saturday, before naming anything or making a single reel.

The first evening

Built in Claude Code over one Saturday evening in May on the footage from Predator Ridge the night before: 28 songs found and 23 reels cut.

TimeWhat happened
6:22 pmFirst working version: split, name, cut reels.
6:29 pmReels sped up by only listening to 90 seconds of each song.
9:18 pmEvery step made safe to stop and restart.
9:19 pmShazam removed. It never matched a live acoustic cover, because it matches studio recordings.
10:10 pmFixed a cut-short copy of the audio that had hidden half the gig: 15 of 28 songs found until then.

How it decides

Loudness is measured over 15-second windows so a quiet verse doesn't split a song. Whisper writes down a 60-second sample of each song (trying later if it lands on banter), and Claude Sonnet names it, told the words will be wrong but the phrases close. The chorus is the most repeated 4 to 8 word phrase; with no repeat, the reel starts a quarter of the way in. Song files are cut without re-encoding, so nothing loses quality.

Captions, handed back and won back

From late May I switched captions off and captioned by hand in InShot, because the misheard words were wrong too often to post. In September I measured it. Against lyrics I typed by hand for three songs, every speech model I tried got 21% to 27% of words wrong, close to the published best for sung lyrics. Asking Claude to correct the words made it worse, 25% to 41%.

So the words come from LRCLIB, a free open lyrics library that had 66 of the 68 songs in my set, and a forced aligner places each word on the audio. On Maneater the speech model heard "by the power of a JQR"; the reel now says "by the purr of a Jaguar". The two checks are measured, not guessed: right songs share 0.59 to 0.92 of their words with what was heard, wrong songs 0.04 to 0.09; the right verse drifts 0.05 to 0.15 seconds from the transcript, a wrong verse 6 to 18. When a wider test found one wrong pairing inside the timing limit, the limit was tightened to 1.0 second.

Other fixes that round: captions break where the lyric line breaks (one read "door So many have"); a title card shows the song and venue for five seconds, never the date, because a date makes a late post look stale; and captions are drawn by the tool itself after a routine update left my video converter unable to draw text at all.

Stand By Me

For four months "Stand By Me" came out as Me_by_Ben_E_King_-_Stand, at four gigs. Claude had answered correctly every time. My code split its reply at the first " by ", which is inside the title. It now trusts the quote marks, and the four files are renamed.

Speed

A background-priority setting made it nine times slower on the same test, 172.7 seconds against 18.5, not the "few percent" its comment claimed; I measured it and turned it off. The blurred background behind each reel was computed at full size; blurring a thumbnail looks the same and took a 30-second reel from 78.5 seconds to 21.3. In August a gig took about 75 minutes for 26 reels on my old Intel Mac. In October, on the new M5 Pro's graphics chip, a one-hour, 13-song gig split, named and made into reels end to end in 3 minutes 15 seconds, and the bigger speech model got more words right: 14, 35 and 9% wrong on three clips against 23, 46 and 20%.

One song, many gigs

The overnight reels came from one sentence before bed: make as many as you can find, they can run all night. Claude counted songs across the named files (16 songs, 151 takes), placed every lyric line in every take, picked one take per line with no venue twice in a row, and checked each finished shot by listening again. It left out a birthday party with kids in shot without being told, because that rule was already written down. The reels run five to ten gigs each.

They read perfectly and sounded wrong to me: a beat missing at every cut. Claude measured all 93 cuts and found half a second of music lost at a typical one, because each side of a cut used a different guess of when the next line started. Now both sides cut at the same moment in the music, found from my voice alone, and the guitar strums are lined up. A beat tracker and chord matching were tried first and didn't hold up on live takes.

Runs

Predator Ridge (28 songs, 23 reels), Britannia, Ex Nihilo twice (26 of 26 in June), Folktale, Sandhill, Nighthawk Road, Cadence (where the cut-off welded two songs into a 9 minute 47 second clip), the airport private gig, Intrigue twice, Upper Mission (25 of 25 named), BB Hair Studio, Gallagher’s Canyon, the Food Bank market and Rose Hill in October, and every older folder of clips I’d already cut, back to 2022. On disk today: 323 reels from the 16 whole gigs it split, and 65 more from those older clips.

Tools

  • Claude Code
  • Claude Sonnet
  • Python
  • librosa
  • ffmpeg
  • Whisper large-v3 (MLX)
  • faster-whisper
  • MMS forced alignment
  • LRCLIB
  • Pillow
  • Demucs
  • Gradio
  • Docker
Next build: my shoe on a kick pedal at a gig