Take a spoken word and a string orchestra. Feed both through an FFT simultaneously. Multiply their magnitude spectra bin by bin. Run an inverse FFT. The result: an orchestra that speaks, with vowel formants imprinting themselves onto the string harmonics. That is spectral cross-synthesis, and it produces sounds that cannot come from any other process.
What It Is, Precisely
Cross-synthesis uses the Short-Time Fourier Transform (STFT) to decompose two audio signals into their frequency-domain representations, combine their spectral properties, and reconstruct a new signal via the inverse STFT. Unlike time-domain mixing, where you sum waveforms, you are operating on magnitude and phase data for hundreds of individual frequency bins, frame by frame.
Two principal approaches exist:
Magnitude multiplication. For each frequency bin k in each time frame n, multiply the magnitudes of the two input signals:
|Y[k,n]| = |A[k,n]| * |B[k,n]|Signal A is the "modulator" (providing the spectral shape) and B is the "carrier" (whose phases are retained for reconstruction). The output inherits the dynamic spectral envelope of A and the harmonic structure of B. Where A has energy, the output is bright. Where A is quiet across a frequency band, the output is suppressed there.
Spectral envelope imposition. Extract the smooth spectral envelope of A using LPC or cepstral smoothing, divide B's spectrum by its own spectral envelope to flatten it, then multiply by A's envelope. This creates a harder "stamp" of one sound's character, closer to how a channel vocoder works but happening entirely in the frequency domain rather than through a filter bank.
The distinction matters in practice. Magnitude multiplication blends the two sounds; envelope imposition erases one character and replaces it with another.
Why the Carrier's Phase Is Kept
In both approaches, the output phase comes from the carrier (B). This is not arbitrary. The auditory system uses phase relationships primarily for spatial localization, not timbre. Pitch and temporal structure are encoded in phase, not magnitude. By keeping the carrier's phase and reshaping only the magnitudes, you preserve the perceived pitch and rhythmic attack of B while the spectral coloring comes from A.
The classic demonstration: use a drum loop as carrier (lots of transient energy, rhythmic structure) with a string pad as modulator (dense harmonic content in the midrange). The output is a drum loop with a string-like harmonic body. It moves rhythmically like percussion but resonates like strings. Replace the modulator with a spoken voice and you get drums that audibly suggest vowel shapes, the attack of each hit articulating a new formant configuration.
This is impossible with EQ or filtering. Filters modify a single signal. Cross-synthesis imposes the time-varying spectral structure of one sound onto the energy distribution of another, frame by frame, across up to thousands of frequency bins.
History: From Speech Compression to Sound Design
James Flanagan and Roger Golden published the phase vocoder in the Bell System Technical Journal in November 1966. The goal was speech compression for telephone transmission: by encoding phase derivatives and magnitudes separately, they could transmit intelligible speech at reduced bandwidth. The connection to music was not the original intent.
Researchers at Stanford's Center for Computer Research in Music and Acoustics (CCRMA) adapted the phase vocoder for music in the 1970s, exploring time-scale modification and cross-synthesis as compositional tools. Pierre Boulez had established IRCAM in Paris in 1970 (the institute's main building opened in 1977), and spectral processing became central to how IRCAM composers approached timbre. Their AudioSculpt software allowed direct manipulation of STFT frames: you could select a region of the spectrogram, copy it, superimpose it on another sound, and render the result. That is manual cross-synthesis.
The technique stayed largely in academic computer music until real-time STFT processing became feasible in ordinary DAWs, which took until the 2010s at the earliest.
Modern Implementations
Ableton Live 11 (February 23, 2021) shipped Spectral Resonator and Spectral Time, two STFT-based devices that brought spectral processing into production workflows without requiring Max/MSP or SuperCollider. Spectral Resonator decomposes the input into partials and rebuilds them tuned to MIDI pitch input. With its MIDI sidechain, it functions as a spectral cross-synthesizer where a synthesizer serves as the modulator, imposing a pitch-quantized spectral structure onto any incoming audio.
Max/MSP exposes raw STFT access through the pfft~ object, where you can write cross-synthesis patches in a few dozen lines. SuperCollider provides FFT and IFFT unit generators. Both give you direct access to the complex spectral frames, so the code implementing magnitude multiplication is straightforward:
import numpy as np
def spectral_cross_synthesis(modulator_frames, carrier_frames):
output_frames = []
for A, B in zip(modulator_frames, carrier_frames):
# Combine: magnitude from A, phase from B
magnitude = np.abs(A) * np.abs(B)
phase = np.angle(B)
combined = magnitude * np.exp(1j * phase)
output_frames.append(combined)
return output_framesA real implementation also handles windowing (Hann or Kaiser windows), overlap-add reconstruction, and normalization. But the core operation is those four lines.
The Time-Frequency Trade-off
Every spectral cross-synthesis setup hits the same wall: FFT size. A 4096-point FFT at 44.1 kHz gives approximately 10.8 Hz frequency resolution but 92.9 ms time resolution. A 512-point FFT gives 86 Hz resolution and 11.6 ms time resolution. Cross-synthesis on fast transients, consonants, or drum attacks requires short windows that sacrifice the frequency resolution needed to distinguish harmonics cleanly. Cross-synthesis on vowels or sustained tones benefits from long windows that would smear those transients.
This is the Gabor uncertainty principle: frequency resolution and time resolution cannot both be maximized. The FFT size is not a preference, it is a constraint on what each signal can "tell" the synthesis about itself per frame. For speech cross-synthesis, a 2048-point window (about 46 ms at 44.1 kHz) is a reasonable default: long enough to capture vowel formant structure, short enough not to blur adjacent phonemes into each other.
Artifacts and How to Manage Them
Magnitude multiplication introduces phase inconsistencies between adjacent frames. The carrier's phase evolves according to B's natural phase trajectory, but the magnitude is being shaped by A. When A changes quickly (a sharp transient in the modulator), the imposed magnitude changes faster than the carrier's phase can track smoothly. This causes pre-echo: a faint smear before a transient. Taming it requires either longer hop sizes (at the cost of more temporal blur) or phase correction algorithms that re-estimate consistent phases for the output magnitudes.
Overlap-add windowing with at least 75% overlap reduces most audible flamming. The remaining artifacts are often usable as texture, which is why spectral cross-synthesis sounds the way it does: slightly metallic, somewhat unreal, inherently tied to the motion of both source materials.
What Makes It Worth Learning
The argument for spectral cross-synthesis is simple: it creates timbral hybrids that no combination of EQ, saturation, modulation, or time-domain convolution can replicate. You are not filtering a sound. You are using one sound as a dynamic, frequency-resolved mold for another. A piano played through a waterfall, a snare drum given the formant structure of a bassoon, a synthesizer pad that breathes with the attack transients of a hi-hat, these textures have no other origin.
The tools are more accessible than they were five years ago. If you have Ableton Live 11 or later, you already have Spectral Resonator. If you have Max/MSP or SuperCollider, pfft~ and FFT UGens put the raw algorithm in reach. What requires the most work is developing an ear for which source combinations yield useful results, and for that there is no shortcut other than running material through the process.