Neural Networks in Browser?
Hum-to-Tab: Building a Browser-Native Pitch-to-Tablature App in Under 24 Hours
A technical journal of the architecture, pivots, and one weird CREPE bug.
What got built
A Progressive Web App that captures audio from your microphone, detects pitch with a neural network running in the browser, segments frames into discrete notes, estimates the musical key, solves an ergonomic guitar-tablature fingering, and renders the result as standard tab notation (VexFlow) with ASCII and MusicXML export.
It's deployed at luttrell.ai/hum-2-tab — single nginx vhost, Let's Encrypt TLS, service worker for offline use. No backend. Everything, including ML inference, runs on the user's device.
The codebase is 8 TypeScript packages in a pnpm monorepo with ~111 passing tests, wired up in roughly 18 working hours.
Stack at a glance
The pipeline
mic → MediaRecorder → Blob → decodeAudioData → Float32Array │ Web Worker boundary │ ┌──────────────────────┐ ▼ │ resample to 16 kHz │ (linear interpolation) └──────────┬───────────┘ │ ┌──────────▼───────────┐ │ frame: 1024 samples, │ (10ms hop, mean/std normalize) │ 60% overlap │ └──────────┬───────────┘ │ ┌──────────▼───────────┐ │ CREPE-tiny ONNX │ (batched, 64 frames at a time) │ → 360-bin softmax │ └──────────┬───────────┘ │ ┌──────────▼───────────┐ │ decodeFrame │ (peak + sub-octave correction) │ → MIDI + confidence │ └──────────┬───────────┘ │ ┌──────────▼───────────┐ │ segment │ (median smooth, energy onsets, │ → discrete notes │ pitch-jump boundaries) └──────────┬───────────┘ │ ┌──────────▼───────────┐ │ analyze key │ (Krumhansl-Schmuckler over │ → tonic + mode │ 24 major/minor profiles) └──────────┬───────────┘ │ ┌──────────▼───────────┐ │ solveTab │ (Viterbi DP over │ → fretboard tab │ (string, fret, hand-position)) └──────────┬───────────┘ │ back to main thread │ ▼ VexFlow renderer Each stage is a pure function in its own package, with unit tests. The packages are core-pitch (CREPE inference and decode), core-segmentation (frames to notes), core-theory (key estimation), core-tab-solver (Viterbi DP), and core-exporters (ASCII and MusicXML output). Pipeline composition lives in apps/web/src/workers/transcribe.worker.ts.
Running ONNX in the browser
The pitch model is the hardest part. Three things made it work.
ORT-Web with WASM threads
Multi-threaded inference uses SharedArrayBuffer, which requires the headers Cross-Origin-Opener-Policy set to same-origin and Cross-Origin-Embedder-Policy set to require-corp. Both are set in nginx and verified per-request. Without them, ORT silently falls back to single-threaded mode at roughly 3 times slower.
MIME type matters
Ubuntu's nginx doesn't ship the application/wasm MIME type by default. The browser refuses to streaming-compile a .wasm file served as application/octet-stream. One missing line caused a "Failed to execute compile on WebAssembly" error that took a debug pass to track down.
Vite optimization avoidance
The ORT-Web prebuilt ESM bundle imports its own .wasm companion. If Vite's dependency optimizer rewrites that import, the bundle breaks at runtime. The fix is to exclude onnxruntime-web from optimizeDeps in vite.config.ts.
The WASM is huge
The threaded variant is roughly 26 MB. Workbox's runtime cache uses CacheFirst with a 10 MB max-per-entry override for .wasm files, so subsequent loads are instant offline.
The CREPE story
CREPE is a 1024-sample monophonic pitch estimator outputting a 360-bin softmax over roughly 32.7 Hz to 1975 Hz. We use the "tiny" variant at about 2 MB float32 ONNX. Three CREPE-specific bugs ate real time.
Quantized model output flat-zero
The first file we tried was about 513 KB — an INT8 conversion that returned 0.007 max activation on a perfect 440 Hz sine. Worthless. Resolved by converting fresh from the Keras h5 weights with tf2onnx at opset 17, producing a 1.9 MB float32 ONNX. Confidence on the same signal: 0.93.
Octave doubling on low E
Open low E (E2, 82 Hz) sits near the bottom of CREPE's training distribution, and on a real guitar the second harmonic at 165 Hz often has more energy than the fundamental. CREPE picks the louder peak — and reports E3 instead of E2. Fixed in the frame decoder with sub-octave correction: after finding the peak bin, look 60 bins below (one octave); if there is at least 40% activation there and the peak is below 1000 Hz, prefer the lower octave. This sits inside pitch decoding, before any downstream stage sees the wrong note.
Isolated octave errors
Occasionally CREPE produces a single short, low-confidence frame an octave above the real note. A separate corrector in segmentation looks for notes that are 10 to 14 semitones above the local 4-neighbour median, and shorter than the median duration, and less confident — only then does it drop them by 12. Conjunctive criteria so legitimate octave leaps survive.
The tab solver
Mapping MIDI notes to fretboard positions is the most algorithmically interesting part. E4 has five legal fingerings: high E open, B fret 5, G fret 9, D fret 14, A fret 19. Picking the right one means picking for the whole phrase, not one note at a time.
The core is Viterbi shortest-path DP. The state at each step started as a (string, fret) pair and evolved through several iterations.
v1 — Cost-only
Hand-travel, hand-stretch, position-anchor. Worked for hummed melodies. Failed on real-guitar input: descending scales jumped strings, repeated notes migrated up the neck.
v2 — Position-aware state
State became a (string, fret, hand-position) triple, where hand-position is the fret the index finger sits at. For a fretted note at fret f, valid positions are from max(1, f minus 3) to min(f, 19) — a 4-fret reach matching standard guitar pedagogy. Open strings inherit the previous position; they don't reset it.
The cost function collapsed to four terms: position-shift, same-string-quick (same string within 150 ms), open-string bonus (only when the previous note was also open or there is no previous, so a fretted lick doesn't substitute open strings mid-phrase), and a high-fret penalty above fret 12.
v3 — Open-far-from-position penalty
Even with the contextual bonus, open strings were essentially free: they preserved position and had zero position-shift cost. The algorithm would happily inject an open high E mid-phrase from 12th position. Added a penalty that scales linearly with distance from the established position — frets 1 through 4 free, fret 5 and beyond at 0.5 per fret beyond 4. Cleared up the "open string injected from a high position" pathology.
v4 — Initial position bias
Viterbi doesn't know where to start without a tiebreaker. Multiple equal-cost paths through different starting positions tie, and the candidate-list ordering decides arbitrarily — usually putting the first note on the wrong string. A small bias (0.1 per fret of distance from fret 5, the conventional 5th position) on the first note's hand-position resolves this without affecting subsequent decisions.
The result handles a 5th-position blues lick, a descending scale, a melody that crosses strings deliberately, and a low-E single-string riff — all without per-genre tuning.
Notable decisions
Pure-function packages
Each pipeline stage is a separate package with no I/O, no side effects, and no DOM dependencies. The CLI consumes the same modules as the web app. Tests run against synthetic inputs — segmentation gets a contrived array of pitch frames, the solver gets a contrived array of notes. This pays off massively when iterating on the cost function: a 3-second test run replaces a 30-second record-decode-listen cycle.
Web Worker for the pipeline
The audio buffer is transferred zero-copy via postMessage with the buffer in the transferables list. The main thread stays responsive while CREPE runs 400 or more frames through WASM. Progress messages stream back at four checkpoints into an Ionic progress bar.
No backend
Eliminates an entire category of complexity: no auth, no rate limiting, no storage costs, no GDPR. The full app is static files behind a CDN. The cost of CREPE inference is paid by the user's CPU — which is what they want, since their voice never leaves their device.
Service worker via Workbox
PWA installable on iOS and Android. Subsequent loads are network-free. After the first session, total download for a recording-and-transcribing session is essentially zero.
What didn't get built
Lessons
Outtake
Repo layout (8 packages):
hum-to-tab/ ├── packages/ │ ├── types/ # shared TS interfaces │ ├── core-pitch/ # CREPE inference + decode │ ├── core-segmentation/ # frames → notes │ ├── core-theory/ # key estimation │ ├── core-tab-solver/ # Viterbi tab DP │ ├── core-exporters/ # ASCII + MusicXML │ ├── cli/ # Node CLI prototype │ └── native/audio-capture/ # iOS Capacitor plugin (deferred) ├── apps/ │ └── web/ # Vue + Ionic + Vite PWA └── docs/ # build sequence (00-10) + this journal Tests: 111 unit tests across all packages, all passing.