stem-separated music visuals
The first SoundSol audio-reactive pieces listened mostly to level, bass, mids and treble. That makes speech feel alive, but music is richer than loudness. A track has drums, bass, pads, chords, melody, vocals, air, room and transitions. The research question is how a browser visual system can listen to those layers without becoming heavy, delayed or gimmicky.
fast features plus local stem splitting
The production audio engine still extracts instant kick-like low transients, snare-like upper-mid hits, hat-like high-band transients, beat pulse, spectral flux, harmonic body, vocal-like presence and rough chroma. The lab now also has a local file mode that actually separates uploaded audio in the browser into playable drums, bass, vocals and other stems using spectral masking.
The practical architecture is hybrid: instant mic features make visuals respond live; browser stem separation gives research-grade control for uploaded songs without sending files to a server.
Demucs is next, spectral masking is live now
Browser stem separation is possible now. The shipped lab starts with a lighter in-browser DSP separator: decode the chosen file locally, build an STFT spectrogram, apply harmonic/percussive median masks, then render four WAV stems. This is not Demucs-quality AI separation, but it is real client-side separation with playable outputs.
Demucs Web remains the next heavier layer: HTDemucs through ONNX Runtime Web can split drums, bass, other and vocals with better isolation, but the model is large, needs WebGPU/WASM, and may require cross-origin isolation headers. That should stay optional until the research layer proves value.
stem-aware visual lab
Two layers run here: realtime Web Audio estimates for live mic control, plus an upload mode that performs local browser stem separation and renders playable drums, bass, vocals and other stems. It is not Demucs-quality yet, but it is a real client-side split.
drum / transient plate
kick, snare, hats and spectral flux push the contour lattice.
harmonic body plate
bass, low mids, chord body and rough chroma steer the bloom.
vocal / texture plate
vocal-presence estimate, upper mids and air drive the swarm.
Useful for live mic input because it reacts instantly. AI separation can sit behind this later as a slower, deeper layer.
Rough pitch-class energy. This is where melody and chords can begin affecting colour and geometry.
upload a track, split it locally
This runs in your browser on the first 35 seconds of a local audio file. It uses STFT analysis, harmonic/percussive median masking and frequency masks to render four playable WAV stems: drums, bass, vocals and other. No file is uploaded.
- [01]Demucs WebBrowser-based HTDemucs via ONNX Runtime Web with WebGPU/WASM acceleration. Four stems: drums, bass, other, vocals.
- [02]Demucs in JavaScriptJavaScript and ONNX port of Meta Demucs. Useful as a file-based separation reference and performance benchmark.
- [03]ONNX Runtime WebBrowser inference layer for WASM/WebGPU model execution.
- [04]Web Audio APIThe low-latency browser layer used for microphone capture, FFT, onset estimates and musical control signals.