[MusicTools dev note] The Feature I Threw Away: Detecting Chords from an MP3

Not every feature makes it into MusicTools. This one I built in a day, tried on real music, and threw away before it ever reached a commit. Here’s what it was, and why it didn’t make it.

What happened: a tester asked for chords from an MP3

In October a tester sent me a list of requests: show the bar being played, add ritardando, allow tempo and meter changes in the middle of a song, and find the chords in an MP3. The first three went into a new Song Form editor. The fourth was a project of its own.

Before any code, I decided where it would live. I didn’t want a fourth tab at the bottom of the screen, because three look best. I didn’t want to rename the Recordings tab either; it has its own job. So the plan was a built-in “Imported” folder inside Recordings, with an Import Audio button where the record button sits, and a Find Chords action on recordings as well.

How I built it: beats, chroma and 24 templates

The detector was plain TypeScript, in four steps:

1. Beat tracking. Find where the beats are, since chords usually change on a beat.
2. Chroma. For each beat, measure how much of each of the 12 pitch classes (C, C♯, D and so on up to B) is sounding, with every octave folded together.
3. Templates. Compare that against 24 chord templates, a major and a minor triad on each of the 12 roots, and score how well each one fits.
4. Viterbi. Pick the most likely run of chords through the whole song, so a single noisy beat can’t flip the chord back and forth.

It was tested on songs I synthesized: chords I’d chosen, played by simple synth voices. It passed.

What happened next: real songs

Chords were never a core feature of MusicTools. Still, if they could be built easily, there was no reason to leave them out, so I gave it a quick try: build it fast, test it myself on real music once it was finished, and go no further if it didn’t work.

Then I tried it on real music files, and the chords it found didn’t match well. So that was the end of it. I discarded all of it before any of it was committed, and the app has no trace of it.

Why: what real songs have that test songs don’t

The code below rebuilds steps 2 and 3 in about fifty lines: a chord is synthesized, its chroma measured, and the 24 templates scored, best three first. Save it as chords.mjs and run node chords.mjs:

// chords.mjs: run it with `node chords.mjs`
const RATE = 22050;
const NAMES = ['C', 'C#', 'D', 'D#', 'E', 'F', 'F#', 'G', 'G#', 'A', 'A#', 'B'];
const hz = (midi) => 440 * 2 ** ((midi - 69) / 12);

// A chord the way a simple synth plays it: each note with a few overtones.
function play(notes, seconds = 0.5) {
  const out = new Float32Array(Math.round(RATE * seconds));
  for (const note of notes)
    for (let h = 1; h <= 6; h++)
      for (let i = 0; i < out.length; i++)
        out[i] += Math.sin((2 * Math.PI * hz(note) * h * i) / RATE) / h;
  return out;
}

// Chroma: how much of each of the 12 pitch classes is in the sound, octaves folded together.
function chroma(samples) {
  const bins = new Array(12).fill(0);
  for (let note = 36; note < 96; note++) { // C2 to B6: one Goertzel filter per note
    const k = 2 * Math.cos((2 * Math.PI * hz(note)) / RATE);
    let s1 = 0, s2 = 0;
    for (const x of samples) [s1, s2] = [x + k * s1 - s2, s1];
    bins[note % 12] += s1 * s1 + s2 * s2 - k * s1 * s2;
  }
  return bins;
}

// 24 templates: a major and a minor triad on each of the 12 roots.
const TEMPLATES = NAMES.flatMap((name, root) =>
  [['', 4], ['m', 3]].map(([kind, third]) => ({
    name: name + kind,
    notes: [root, (root + third) % 12, (root + 7) % 12],
  })),
);

// The three templates that fit best, scored 0 to 1.
function guess(samples) {
  const c = chroma(samples);
  const total = Math.hypot(...c);
  return TEMPLATES
    .map(({ name, notes }) => ({ name, score: notes.reduce((s, n) => s + c[n], 0) / Math.sqrt(3) / total }))
    .sort((a, b) => b.score - a.score)
    .slice(0, 3)
    .map(({ name, score }) => `${name} ${score.toFixed(2)}`)
    .join('  ');
}

// MIDI note numbers: 60 is middle C.
const tries = {
  'C major (C E G)': [60, 64, 67],
  'A minor (A C E)': [57, 60, 64],
  'Am7 (A C E G)': [57, 60, 64, 67],
  'Fmaj7 (F A C E)': [53, 57, 60, 64],
  'Dm7 (D F A C)': [50, 53, 57, 60],
};
for (const [label, notes] of Object.entries(tries)) console.log(label.padEnd(18), guess(play(notes)));

// A window that catches a chord change: a quarter second of C, then a quarter second of G.
const change = new Float32Array([...play([60, 64, 67], 0.25), ...play([55, 59, 62], 0.25)]);
console.log('C, then G'.padEnd(18), guess(change));

This is what it prints:

C major (C E G)    C 1.00  Em 0.69  Cm 0.67
A minor (A C E)    Am 1.00  C 0.69  A 0.68
Am7 (A C E G)      C 0.88  Am 0.86  Em 0.62
Fmaj7 (F A C E)    Am 0.89  F 0.85  C 0.63
Dm7 (D F A C)      F 0.89  Dm 0.84  Am 0.65
C, then G          G 0.85  Em 0.79  C 0.76

The first two lines are what a synthetic test looks like: a plain triad, found with a perfect score. The rest is what real music brings.

• Four-note chords hold two triads. Am7 is A, C, E and G, which contains both A minor (A C E) and C major (C E G), and the templates pick C. Fmaj7 comes out as A minor, and Dm7 as F. Songs are full of sevenths, sixths and added notes.
• A window that catches a chord change reads as neither chord. Here, half a second holding a C and then a G comes out as G, with E minor close behind. Everything leans on finding the beats, and every beat that’s missed blurs two chords together.
• Real recordings carry what my synth didn’t: drums, a bass line, a singer, notes passing between chords, and instruments whose overtones light up pitch classes that aren’t in the chord at all.

What I’d keep

A quick try with a clear way out. Chords were a nice-to-have, so the deal was simple: if it worked on real music, it could stay, and if it didn’t, it would go. It didn’t, so it went, and nothing half-working ever reached anyone using the app.

And passing tests on songs I made myself proved very little; real songs settled it. If I try chords again, it starts with real recordings whose chords are known, and it will likely need more than 24 triad templates. Where the feature would live can stay as it was: the Imported folder and the button where the record button sits still make sense to me.