How OnKey works

• A Mac app: a Rust engine (audio, models, scoring) inside a Tauri window with a React interface. engine.rs runs everything.
• Everything happens on your Mac. Models run with ONNX Runtime, on Core ML (GPU and Neural Engine) where it helps.
• Audio is analysed at 16 kHz in 10 ms frames, the unit every number below uses (dsp.rs, FRAME_S).
• A song counts as learned once 70% of it has been heard (READY_COVERAGE in store.rs). A partly heard song keeps learning on later plays.

• Capture uses Core Audio process taps (macOS 14.2+), with your mic, at your output device's own rate (usually 48 kHz), on one clock (capture.swift, capture.rs).
• Only the processes of the music app are tapped, chosen by its bundle id (source.rs).
• Everything is resampled to 16 kHz mono for analysis. Release builds never write the music's audio anywhere. Only your own voice is kept with each take.
• Permissions: system audio recording and the microphone. Controlling Spotify and Music (what's playing and where, "Sing it again", skip, and pause) is a separate permission, asked in setup, and "Listen only" turns it off. Controlling the YouTube tab is off until you switch it on.

• Separation: UVR MDX-Net Karaoke 2 (ONNX, on Core ML), run on 20 s chunks with 2 s of context either side. The music is held in memory while the song plays and separated after it ends, at a low priority. One song is learned at a time. Others wait their turn ("Waiting to learn"), and learning pauses while you sing a learned song with the window showing (learn.rs, mdx.rs).
• First play: up to its first 90 s are separated as they play, 5 s at a time, until the lyric lines are lined up with the singer. The rest waits for the song to end. This is skipped on 8 GB Macs and while another song is being learned. Across my library, 60% of first-play lines were within 0.3 s of the singing this way, against 13% at LRCLIB's own times.
• Memory: with 8 GB, one model is loaded at a time and unloaded after its step. Below 32 GB, the learning models (about 1.5 GB) are unloaded once a song is learned, unless Settings → Keep learning models ready is on (lowmem.rs).
• Pitch: CREPE "full" on the separated vocal, 16 kHz, 10 ms hop, 1024-sample windows, 50 to 1100 Hz. A frame counts as sung when its confidence is over 0.45 (pitch.rs).
• Notes: GAME (OpenVPI) transcribes notes from the vocal. Each syllable bar takes its note, and bars with no sung note under them are dropped (game.rs, syllables.rs).
• Lyrics: synced lyrics from LRCLIB (looked up by artist, title, album, and length, then by name if that finds nothing). Lines are re-placed by aligning the whole song's words with wav2vec 2.0 (CTC), and split into syllables with the CMU pronouncing dictionary (lyrics.rs, ctc.rs).
• Measured: on 18 hand-marked songs, 98% of sung notes are covered by a bar, 92% on the right note, and 73% starting within 0.1 s. 94% of lyric lines are within 0.3 s.
• Kept with the song: the pitch line, the bars, a rhythm fingerprint, a Shazam signature and the lyrics' timing, plus the LRCLIB entry those times were made for (kept on the Mac, never uploaded), so the same words show on the same bars every play. The music and the separated vocal aren't kept.

• Fingerprint: how much the sound rises in each of 8 bands from about 60 Hz to 7 kHz, every 10 ms (FP_BANDS in dsp.rs).
• Placement: one tracker places every play (tracker.rs, after Arzt and Widmer's "any-time" music tracker). Each check matches the last few seconds of music, at most 8 s, against the fingerprint, to within a 10 ms frame, and keeps the best few places as weighted candidates, so a repetitive intro that matches a beat off doesn't fool it. The music app's position and listening's answers only nudge the candidates near them. The shown place moves to another candidate only when that one clearly leads over about 4 s of music.
• Measured: replaying 153 real plays with the music app's position wrong (0.73 s or 1.7 s off, or a skip landing 3.5 s away), the shown place was within 0.1 s of the music about 91% of the time. The rules before managed 75%, 0%, and 0%.
• Listening mode (always on, listen.rs, recognize.rs): after 3 s of sound it asks, in order, OnKey's own catalogue of learned songs (ShazamKit, on the Mac, exact to about 10 ms), the fingerprints, and then Shazam's catalogue. It checks again every 3 s for learned songs and about every 12 s for songs only Shazam knows. Shazam gets a signature of at most 5 s of the music app's sound, never the mic. A new Shazam answer needs a second one that agrees. A pause is 0.15 s of digital silence.
• First test of Shazam's catalogue: it named 25 of 30 test songs from their first 5 s, in a median 0.24 s.

• Live pitch: CREPE "tiny", on your echo-cancelled mic, every 10 ms, run on Core ML's CPU kernels: the pitch dot is 2 to 4 ms behind your voice, and singing a learned song costs OnKey 15 to 19% of one CPU core on my Mac (pitch.rs).
• Echo canceller: a partitioned frequency-domain adaptive filter: 16 partitions of 256 samples at 16 kHz, so it models 256 ms of echo (bleed.rs). Some outputs put the echo later than that, so every 2 s the echo's delay is measured (GCC-PHAT, up to 800 ms) and the song is fed to the filter that late. It only adapts while the song is clearly reaching the mic, and if its output comes out louder than the mic, the mic is passed through unchanged.
• Measured: built-in MacBook speakers (echo about 24 ms late): 20 dB or more removed. External speakers with the echo about 290 ms late: 8 to 14 dB, up from 1 to 2 dB before the delay was measured.
• Limit: 8 to 14 dB still leaves some of a loud song in the mic, and that can be mistaken for singing. Headphones are more reliable.

For now the scoring is a bit lenient on purpose. It’s set for beginners like me, so harder modes to really test good singers may come later.

The numbers in your report are explained in Scoring explained.

• Credit: clamp((150 − |error|) / 100, 0, 1), with the error in cents folded into ±600 so octaves don't count (credit, fold in dsp.rs).
• Timing slack: each 0.5 s window gets the one shift within ±80 ms (TOL = 8 frames) that lines you up best (tolerant_error).
• Your delay: after the song, a search from 0.3 s early to 1.5 s late finds the lag that fits best (analysis.rs).
• Notes hit: held within 100 cents for at least half the note, ignoring its first 80 ms. Notes under 0.15 s aren't counted.
• Score: 100 × accuracy × (0.3 × min(1, sung / 0.1) + 0.7 × min(1, sung / 0.4)): full credit from 40% of the singer's part sung.
• Checked against experts: on 20 rated recordings, notes hit agrees with expert rankings at 0.77 and the score at 0.61 (experts agree with each other at 0.88).

• Stored in ~/Library/Application Support/audio.onkey.app/: songs/ (pitch line, bars, fingerprint, lyric timings and a Shazam signature, saved for every learned song in either mode), takes/ (mic.m4a: your microphone, AAC mono at about 64 kbps, about 0.5 MB a minute, all kept up to 2 GB in total, plus scores and the report with the lyric lines) and records/ (logs and a summary of each play, numbers only, kept 30 days, at most 200 MB). Deleting a song or take moves it to the Trash.
• Network: Shazam (listening mode, can be turned off) for songs not yet learned, LRCLIB lyric lookups (unless Show lyrics is off), iTunes Search for covers, iTunes Lookup for a recognised song's length, the cover images (from Apple, or YouTube's thumbnail), and a daily check of GitHub Releases for updates. No account, no analytics, and no server of mine.
• More in Privacy & legal.