This page is an interactive exploration where sound is essential.
Please turn your volume on!
Or does it merely compute the patterns of vibration and fake a voice?
Your voice is a miracle of vibrating vocal cords, a dancing tongue, and the breath of your lungs. A machine's "voice" is a sum of sine waves, a spectrogram pattern, and a neural network's prediction. Same sound, entirely different origins. Let's listen to the mathematics.
00 โ WHAT IS SOUND
Sound is pressure changes in air โ fast vibrations make high pitch, slow vibrations make low pitch. All sound is ultimately a wave. Adjust the sliders below and hear it for yourself.
01 โ THE ANATOMY OF SOUND
In 1822, Joseph Fourier proved a revolutionary fact: any complex waveform can be decomposed into a sum of simple sine waves. A piano's middle C is not one frequency โ it's the sum of a fundamental frequency and dozens of harmonics.
Why the Fourier Transform matters for AI. โ Speech recognition, music generation, TTS โ the first step of every audio AI is decomposing sound into frequency components. Transforming a complex time-domain waveform into a clean frequency-domain spectrum. This is how a machine "sees" sound.
02 โ THE FINGERPRINT OF SOUND
A spectrogram is a 3D map: time (horizontal) ร frequency (vertical) ร intensity (brightness). Even the same syllable looks different for every speaker and every emotion. AI reads these "photographs of sound."
"Humans hear sound. Machines see it."
The mel spectrogram scales the frequency axis nonlinearly to match human hearing: wide bins for low frequencies, narrow for high. Exactly how our ears perceive pitch. This is the standard input format for speech AI.
03 โ VOICE AS NUMBERS
To handle continuous air vibrations, a computer must sample them โ measuring amplitude tens of thousands of times per second and converting it to an array of numbers. CD quality is 44,100 Hz. Phone calls: 8,000 Hz.
04 โ HOW A MACHINE SPEAKS
Modern TTS works in three stages: text analysis โ spectrogram generation โ waveform reconstruction. Each stage uses a neural network.
Pre-recorded phoneme fragments stitched together. Robotic voice. Used in GPS navigation.
Google DeepMind. Predicts audio samples one at a time. First natural-sounding AI voice. But extremely slow.
VALL-E, Bark, ElevenLabs. Clone anyone with 3 seconds of audio. Real-time synthesis. Nearly indistinguishable.
05 โ VOICE CLONING
Modern voice cloning captures a speaker's characteristics from just 3 seconds of audio โ timbre, pace, intonation, even emotion. Technically, the speaker's voice is compressed into a speaker embedding vector and injected as a condition into the TTS model.
06 โ MACHINE COMPOSITION
Music AI โ Suno, Udio, MusicGen โ creates music from text prompts. Internally, it converts audio to tokens (via codecs) and predicts the next token with a Transformer. Exactly the same principle from Part II, applied to sound.
The key insight of music AI. โ Next-token prediction for text (Part II) and next-token prediction for music are mathematically identical. The only difference is what the tokens represent โ text AI predicts words; music AI predicts audio codec tokens. Same Transformer, different language.
07 โ THE CONNECTION
| Human | Machine | |
|---|---|---|
| ๐ซ | Source โ Vocal cord vibration + lung air | Source โ Sine wave synthesis / neural vocoder |
| ๐ | Articulation โ Tongue, lips, oral cavity | Articulation โ Filter coefficients / mel spectrogram |
| ๐ง | Language โ Broca's area, Wernicke's area | Language โ Transformer encoder |
| ๐ญ | Emotion โ Autonomic nervous system, hormones | Emotion โ Prosody embedding vector |
| ๐ถ | Learning โ Years of imitation and feedback | Learning โ Tens of thousands of hours of speech data |
| ๐ต | Singing โ Emotion + musical training | Singing โ Audio codec token prediction |
A machine's voice has no breath.
No tremor, no crack, no laughter woven into words.
Only sums of sine waves, spectrogram patterns,
and astonishingly precise probabilistic predictions.
Perhaps that is not speaking.
But your ears can no longer
tell the difference.