Building a Sample Manager: Collections, HPSS, and Getting Serious About Key Detection

This one covers a fair bit more ground than usual, from building out collections and the library UI properly, through to a proper deep dive into whether the key detection was actually any good, which it turned out wasn’t as much as I’d hoped.

Collections

Samples can now be grouped into user-defined collections, each with its own highlight colour, and a sample can sit in as many collections as it needs to. It’s a standard many-to-many setup: collections and samples tables, joined by a collection_samples table with a composite primary key on the pair so the same sample can’t be added to the same collection twice, and cascading deletes on both sides so removing a collection or a sample cleans up memberships automatically without leaving anything orphaned.

The upload modal got a type-to-filter collection picker, showing matching collections as you type and selected ones as removable chips, and the same pattern got reused for a proper bulk-edit toolbar that flies in whenever samples are selected in the list, letting you rename, re-collect, or delete several at once. Every modal in the app now shares the same shape: a focus trap, Escape to close, a submitting/error state tracked through use:enhance, and a backdrop click to dismiss.

Recently played tracking went in alongside this: samples now carry a lastPlayedAt and a playCount. Deciding what actually counts as a play took a bit of thought. A short one-shot ending naturally counts immediately, but anything longer needs a qualifying timer (a second of continuous playback) so scrubbing past something doesn’t inflate its count, and a flag stops pause/resume from double-counting the same play. The dashboard and sidebar both show a live “recently played” list that updates the moment a play is reported, without a page reload.

Teaching the backend to actually understand the audio

The bigger piece of work this round was on the FastAPI side. Two goals going in: tell percussive samples apart from melodic ones, and improve key detection, since key matters even for drums (is this kick actually in tune with the track?).

The first piece was a harmonic/percussive ratio using librosa’s HPSS, which splits a signal into a harmonic component and a percussive one. Sum the energy of each and divide to get a ratio from 0 (fully percussive) to 1 (fully harmonic). The first version used a stricter margin setting copied from an example, which turned out to quietly discard a chunk of the signal as an unlabelled residual and skewed everything toward “harmonic”. Removing the margin fixed it, but the results held a genuine surprise: kicks read as strongly harmonic (0.868), because a kick is mostly a sustained low sine body once the initial click passes, and HPSS reads sustained tone as harmonic regardless of whether a human would call it “melodic”. So the ratio actually measures tonal-vs-noisy rather than melodic-vs-percussive, which is arguably more useful for a sample manager anyway, and the labels got renamed to match: tonal / noisy rather than the original percussive/melodic framing.

# Using Librosa to extract harmonic & percusive elemnets from the sample
y_harmonic, y_percussive = librosa.effects.hpss(y)
harm_ratio = harmonic_ratio(y_harmonic, y_percussive)

# Gaining a harmonic ratio based off the provided extracts

def harmonic_ratio(y_harmonic: np.ndarray, y_percussive: np.ndarray) -> float | None:
    harm_energy = np.sum(np.square(y_harmonic))
    perc_energy = np.sum(np.square(y_percussive))
    
    total = harm_energy + perc_energy
    
    if total < 1e-10:
        return None

    ratio = harm_energy / total

    return float(ratio)

Finding out the key detection wasn’t actually that good

Rather than just eyeballing a few results and calling it done, I built a proper labelled test set, 40 samples with known keys pulled from filenames, and a scoring script that runs every combination of raw vs harmonic input, four different chroma extraction methods, and six different key profile templates (including Krumhansl, the one already in use, and a few alternatives tuned for electronic music) against MIREX-style scoring, which gives partial credit for a fifth or a relative key rather than marking everything as flat right or wrong.

The results were humbling. The existing setup, harmonic input plus Krumhansl profiles, actually performed worse than just using the raw signal, and swapping to a profile called bgate (tuned for electronic music, which fits drum & bass reasonably well) gave a real improvement across three separate test rounds. Confidence scores turned out to be almost useless for flagging bad results too, a drum loop with no real tonal content could score higher confidence than a correctly identified melodic sample, since the correlation maths gives every input a moderate score “for free” against smooth profile curves.

The best setup lands around 55% exact tonic accuracy, with fifth/fourth confusion (mistaking a key for its dominant or subdominant) being the single biggest error category, which is a known limitation of averaging chroma over a whole file rather than something a quick tweak fixes. That’s about as far as template matching goes; anything meaningfully better would mean a different approach entirely, like a trained model rather than profile correlation.

What’s next

The test set needs to grow, particularly with melodic loops that already have drums mixed in, since that’s the actual case the harmonic separation is meant to help with and the current set doesn’t really test it. Distinguishing one-shots from loops is still open, duration alone doesn’t work since some one-shots in the library run over 40 seconds long, so onset counting is the next thing to try. There’s also root-note detection for tonal one-shots like kicks and 808s using pyin rather than key profiles, which is a different problem to melodic key detection but sits right next to it. Template matching has more or less hit its ceiling at around 55% tonic accuracy, so it’s worth properly researching other approaches to key detection, a trained model rather than profile correlation, before sinking more time into tuning the current method further.

Alongside that, it’s time to actually get the vector database set up and start planning how sample data gets plotted into it, a separate embeddings table keyed by sample so different models can coexist without touching the samples table itself, starting with the handcrafted librosa features already being extracted before layering in a proper learned embedding model later. This is also the point to move the current analysis pipeline off a directly awaited API call and into proper background jobs, which opens the door to re-analysing samples on demand, useful for every time the key detection or feature extraction improves, rather than only ever running once on upload.