The Mathematics of Musical Timbre

Timbre, often defined negatively as "everything about a sound that is not pitch, loudness, or duration," is the quality that distinguishes a violin from a clarinet playing the same note at the same volume. Hermann von Helmholtz's foundational 1863 work "On the Sensations of Tone" established that timbre arises from the pattern of overtones (harmonics) produced by a vibrating body, and that different instruments produce characteristically different overtone ratios. A clarinet, because of its cylindrical bore and single-reed excitation mechanism, suppresses even-numbered harmonics, producing a hollow, reedy tone. A violin's bowed string produces a rich harmonic series with both even and odd partials, yielding a brighter, more complex sound.

The systematic perceptual study of timbre began with multidimensional scaling (MDS) experiments in the 1970s. Grey (1977) asked listeners to rate the dissimilarity of pairs of instrument tones, then used MDS to position the instruments in a low-dimensional perceptual space such that acoustically similar instruments were geometrically close. The resulting two- and three-dimensional solutions had axes interpretable in acoustic terms: one axis corresponded to spectral centroid (brightness), another to attack time (percussive vs. sustained onset), and a third to spectral flux (how rapidly the overtone structure changes over the note duration).

The connection between these psychoacoustic findings and computational audio analysis is compelling: if the perceptual space is determined by measurable spectral features, then automatic extraction of those features from audio should allow computational reconstruction of the perceptual space. This has obvious applications in music information retrieval, automatic instrument recognition, and synthesis design.

We compute spectral features from isolated instrument recordings from the University of Iowa Musical Instrument Samples database, which provides high-quality recordings of orchestral instruments at multiple pitches and dynamics under controlled conditions. For each note at each pitch and dynamic level, we extract the following features from the sustain portion of the note (0.2 to 0.8 of note duration, to exclude onset transient and decay): attack time (measured as the time from onset to 90% of peak amplitude), spectral centroid (frequency weighted by amplitude across the spectrum), spectral spread (the second central moment of the spectral distribution), spectral skewness (asymmetry of the spectral envelope), and spectral irregularity (the average absolute difference between successive partial amplitudes, divided by their mean, quantifying how smoothly the harmonics fall off).

For each instrument, we average these features across pitches in the mid-range (approximately E3 to E5) where all instruments are well-represented, producing a single feature vector per instrument. We then construct a pairwise Euclidean dissimilarity matrix in normalized feature space and apply metric MDS using the SMACOF algorithm to obtain a 2D representation of instrument relationships. We compare the MDS solution to the perceptual timbre space from Grey (1977) by finding the optimal affine alignment (Procrustes analysis) between the two configurations.