Simple Transformer TTS
From a blog series to a single self-contained Colab notebook — smoke-tested on a free T4 GPU.
One notebook, one toy voice
ake a transformer that learns to say one thing — hello, world — and fold its whole repository into a single notebook.
The base — Transformer-TTS (arXiv 1809.08895), deliberately simplified: letters instead of phonemes, Griffin-Lim instead of a vocoder, one speaker, one dataset.
This experiment — reassembles tttzof351's Medium series and repo as 34 inlined cells, adds Drive persistence, Kaggle download and wandb logging, then smoke-tests the loop on a free GPU in minutes.
LJSpeech, delivered through Kaggle
kaggle.json uploaded through the Colab files panel → ~/.kaggle, chmod 600.
Wavs — the mirror mathurinache/the-lj-speech-dataset, a ≈ 3.8 GB zip unpacked to /content/LJSpeech-1.1/wavs.
Metadata — tttzof351/ljspeech-meta, the two-column csv the repo's dataset.py expects.
Archives are deleted after unzip to keep the Colab disk breathing.
The source LJSpeech files ship with gaps, and dataset.py reads exactly two columns: wav and text_norm.
A repair cell keeps every run alive: use a correct csv if present; else rebuild /content/metadata.csv from the pipe-separated wav | text | text_norm file, skipping bad lines, then assert.
Text to letters, audio to mel
The article's simplification: no espeak or g2p phonemizer — plain characters carry the pronunciation load.
A 44-symbol alphabet: EOS + punctuation and space + a–z + accented letters + typographic quotes.
text_to_seq lowercases, maps to ids, appends EOS — embedding table sized at 2 × 44.
Spectrogram — n_fft 2048 · hop 256 · win 1024, power 2, at 22.05 kHz.
Mel — MelScale projects 1025 stft bins onto 128 perceptual bands.
dB + normalize — amplitude_to_DB, clipped to [−4, 4]; min_level_db = −100, missing in the original repo, restored here so norm / denorm round-trip.




Padding, shifting, and when to stop
Texts and mels are padded to the longest member; per-item lengths travel alongside. Stop-token targets mark the true end — 1 on the last real mel frame, 0 across the padding — so the model learns a termination signal.
The decoder receives [SOS, mel_target[:, :-1]]: a zero start-of-sequence frame followed by every frame except the last. At each position the model predicts the next mel frame — an autoregressive loop wearing training wheels.
mask_from_seq_lengths turns lengths into a boolean mask, fed as key_padding_mask so attention never reads the padded frames as speech.
A small autoregressive transformer
Pre-norm transformer blocks: 4-head self-attention at d = 256, feed-forward 1024, dropout 0.1, residual around both halves; a conv embedding (kernel 3) fronts the character ids.
The same block shape, but causal: an attention mask hides the future, while the padding mask hides the fill. It attends to the encoded text and to its own mel frames so far.
A mel projection refined by a postnet (d = 1024, kernel 5), plus a stop-token head that decides when the utterance ends. block patterns credit NVIDIA Tacotron2
One loss, three terms, gentle clipping
L = MSE(dec, mel) + MSE(postnet, mel) + r_gate · BCE(stop)
Mel frames pay twice — before and after the postnet.
Optimizer — Adam, lr 2e-4, grad clip 1.0, seed 42.
Batch — 8, for the smoke run.
Rhythm — log every 30 steps; test loss + checkpoint every 120.
Losses, mel heatmaps and sample audio stream to wandb.
From repo scripts to one notebook
A run sized for minutes
Sampled from metadata — small enough to cache in RAM.
A pocket test split for the test-loss signal.
Down from the repo's 32 — gentler on a shared T4.
A full train → test → checkpoint cycle, bounded.
Loss, mel heatmaps, audio land in wandb at each tick.
Five checkpoints across the run — enough to see motion.
Render a real target mel beside the untrained model's prediction — shapes, masks and plumbing verified at zero training cost.
T4 GPU runtime required · the full ≈ 3.8 GB dataset still downloads once · a wandb login gates the run.
The curve comes down
Both curves fall smoothly from ≈ 1.6 and settle near 1.14 — no spikes, no divergence, and the 16-clip test curve tracks the training curve almost exactly. That equality is a plumbing signal, not a generalization claim; the subset is too small to say more.
What it sounds like at step 480
Sample audio, step 480 — hum and texture; words not yet.
What the smoke test proves — and what it doesn't
Every stage ran on the T4: Kaggle download → metadata repair → dataset + collate → model → loss → backward → wandb → checkpointing → inference → mp3 export.
The loss behaved — smooth decay, stable at batch 8, no NaNs, no divergence.
The pipeline is sound enough to scale: nothing in the loop needs rearchitecting before a longer run.
Train = test = 1.1403. With only 16 held-out clips, the test loss is a sanity signal, not a generalization estimate.
Near-identical curves say more about the tiny subset — the model has capacity to spare over 256 clips — than about learning to speak.
A falling curve here certifies the machinery; it does not yet certify a voice.
The reference run only flattens near test ≈ 0.23 after roughly 400,000 steps.
600 / 400,000 ≈ 0.15 % of the journey — the smoke test is the first breath of a long descent.
1.14 is a healthy place to be this early; it is not a place to stop and listen.
Limits, and the road ahead
vocoder Griffin-Lim rebuilds phase by iteration — a metallic ceiling no amount of training lifts; a neural vocoder is the modern fix.
letters Character-only input leaves pronunciation edge cases to chance — no phonemizer in the loop.
subset 256 clips invite memorization rather than speech; and LJSpeech is one speaker, by design.
test set Sixteen clips cannot steer hyperparameters — only a full run can.
i. Scale to the full 13,100 clips, batch back to 32.
ii. Train ≥ 100k steps with Drive checkpoints through runtime resets.
iii. Add a phonemizer to the text front-end.
iv. Swap Griffin-Lim for a HiFi-GAN vocoder.
v. Compare runs in wandb — and listen at every milestone.