Experiment report · Colab smoke test
A toy text-to-speech transformer, rebuilt for one notebook

Simple Transformer TTS

From a blog series to a single self-contained Colab notebook — smoke-tested on a free T4 GPU.

13,100
LJSpeech clips · one speaker
256
smoke subset · 16 held out
600
steps — one bounded cycle
1.1403
final loss · train = test
Mission
02 / 13

One notebook, one toy voice

ake a transformer that learns to say one thing — hello, world — and fold its whole repository into a single notebook.

The base — Transformer-TTS (arXiv 1809.08895), deliberately simplified: letters instead of phonemes, Griffin-Lim instead of a vocoder, one speaker, one dataset.

This experiment — reassembles tttzof351's Medium series and repo as 34 inlined cells, adds Drive persistence, Kaggle download and wandb logging, then smoke-tests the loop on a free GPU in minutes.

arXiv 1809.08895· SimpleTransfromerTTS repo· Medium series, 2023
The audio loop, as torchaudio sees it
torchaudio transform map
Waveform → spectrogram → mel for training; InverseMelScale + Griffin-Lim walk the same map backwards at synthesis.
Dataset
03 / 13

LJSpeech, delivered through Kaggle

13,100
text ↔ audio pairs
≈ 24 h
single-speaker audio
2.6 GB
wavs on disk
22.05 kHz
mono — asserted per clip
Kaggle delivery, per run

kaggle.json uploaded through the Colab files panel → ~/.kaggle, chmod 600.

Wavs — the mirror mathurinache/the-lj-speech-dataset, a ≈ 3.8 GB zip unpacked to /content/LJSpeech-1.1/wavs.

Metadata — tttzof351/ljspeech-meta, the two-column csv the repo's dataset.py expects.

Archives are deleted after unzip to keep the Colab disk breathing.

Metadata repair — closing a gap

The source LJSpeech files ship with gaps, and dataset.py reads exactly two columns: wav and text_norm.

A repair cell keeps every run alive: use a correct csv if present; else rebuild /content/metadata.csv from the pipe-separated wav | text | text_norm file, skipping bad lines, then assert.

full ≈ 3.8 GB dataset still downloads once only the 256-clip subset trains
Front-ends
04 / 13

Text to letters, audio to mel

Text front-end — letters, not phonemes

The article's simplification: no espeak or g2p phonemizer — plain characters carry the pronunciation load.

A 44-symbol alphabet: EOS + punctuation and space + a–z + accented letters + typographic quotes.

text_to_seq lowercases, maps to ids, appends EOS — embedding table sized at 2 × 44.

Audio front-end — three hops, one domain

Spectrogram — n_fft 2048 · hop 256 · win 1024, power 2, at 22.05 kHz.

Mel — MelScale projects 1025 stft bins onto 128 perceptual bands.

dB + normalize — amplitude_to_DB, clipped to [−4, 4]; min_level_db = −100, missing in the original repo, restored here so norm / denorm round-trip.

waveform
1 · the raw waveform
power spectrogram
2 · power spectrogram
mel spectrogram
3 · projected onto 128 mel bands
mel in dB, normalized
4 · dB + normalized — the target
Batching
05 / 13

Padding, shifting, and when to stop

Collate — everything padded to the batch max

Texts and mels are padded to the longest member; per-item lengths travel alongside. Stop-token targets mark the true end — 1 on the last real mel frame, 0 across the padding — so the model learns a termination signal.

Teacher forcing — the input is the target, shifted

The decoder receives [SOS, mel_target[:, :-1]]: a zero start-of-sequence frame followed by every frame except the last. At each position the model predicts the next mel frame — an autoregressive loop wearing training wheels.

Masking — padding stays invisible

mask_from_seq_lengths turns lengths into a boolean mask, fed as key_padding_mask so attention never reads the padded frames as speech.

padding illustration
The article's padding picture — 1s are content, 0s are fill; lengths locate the boundary.
Model
06 / 13

A small autoregressive transformer

Encoder — reads the sentence

Pre-norm transformer blocks: 4-head self-attention at d = 256, feed-forward 1024, dropout 0.1, residual around both halves; a conv embedding (kernel 3) fronts the character ids.

Decoder — speaks one frame at a time

The same block shape, but causal: an attention mask hides the future, while the padding mask hides the fill. It attends to the encoded text and to its own mel frames so far.

Two outputs per frame

A mel projection refined by a postnet (d = 1024, kernel 5), plus a stop-token head that decides when the utterance ends. block patterns credit NVIDIA Tacotron2

model architecture
The repo's architecture — embedding, encoder–decoder attention, and the mel + stop heads.
Training
07 / 13

One loss, three terms, gentle clipping

The objective

L = MSE(dec, mel) + MSE(postnet, mel) + r_gate · BCE(stop)

Mel frames pay twice — before and after the postnet.

Run knobs

Optimizer — Adam, lr 2e-4, grad clip 1.0, seed 42.

Batch — 8, for the smoke run.

Rhythm — log every 30 steps; test loss + checkpoint every 120.

Losses, mel heatmaps and sample audio stream to wandb.

What a converged run looks like — the reference
full-run loss curves
The original repo's full run on all 13k clips — hundreds of thousands of steps before the curves flatten (wandb). The smoke test below borrows its shape, not its budget.
Colab tailoring
08 / 13

From repo scripts to one notebook

AspectOriginal setupThis notebookVerdict
code layout
Seven separate .py scripts, driven from a Colab session
34 cells — every script inlined, hyperparams to train() in one file
consolidated
persistence
Checkpoints and logs to local /content dirs — lost on runtime reset
Google Drive mount; params + train_logs survive resets
kept
dataset
LJSpeech-1.1 source files ship with gaps
Kaggle mirrors for wavs and metadata, fetched per run
mirrored
metadata
dataset.py silently expects wav + text_norm columns
Repair cell auto-detects or rebuilds metadata.csv, then asserts
repaired
monitoring
tensorboard logs, sampled figures
wandb streams losses, mel heatmaps, sample audio
adapted
scale
Full 13k clips, batch 32, days of training
256 clips, batch 8, 600 steps — minutes on a free T4
smoke-sized
Smoke-test setup
09 / 13

A run sized for minutes

i.
256 clips, seed 42

Sampled from metadata — small enough to cache in RAM.

ii.
16 held-out clips

A pocket test split for the test-loss signal.

iii.
Batch 8

Down from the repo's 32 — gentler on a shared T4.

iv.
600 steps, hard stop

A full train → test → checkpoint cycle, bounded.

v.
Log every 30 steps

Loss, mel heatmaps, audio land in wandb at each tick.

vi.
Test + save every 120

Five checkpoints across the run — enough to see motion.

Pre-flight check, before any training

Render a real target mel beside the untrained model's prediction — shapes, masks and plumbing verified at zero training cost.

Ground rules

T4 GPU runtime required · the full ≈ 3.8 GB dataset still downloads once · a wandb login gates the run.

Results
10 / 13

The curve comes down

1.1403
final train = test loss · step 600
Reading the decay

Both curves fall smoothly from ≈ 1.6 and settle near 1.14 — no spikes, no divergence, and the 16-clip test curve tracks the training curve almost exactly. That equality is a plumbing signal, not a generalization claim; the subset is too small to say more.

train loss, wandb
Train loss — wandb, 600 steps, 256-clip subset.
test loss, wandb
Test loss — the 16 held-out clips, logged to the same wandb run.
Results
11 / 13

What it sounds like at step 480

predicted mel at step 480
Predicted mel at step 480 (wandb) — energy gathering into voiced bands, not yet formed speech.

Sample audio, step 480 — hum and texture; words not yet.

The bar to clear — the original after its full run
original Hello World mel and stop token
The repo's "Hello, World" — clean harmonic tracks and a decisive stop-token transition (top panel). That is what another few hundred thousand steps buys.
Discussion
12 / 13

What the smoke test proves — and what it doesn't

Proven, end to end

Every stage ran on the T4: Kaggle download → metadata repair → dataset + collate → model → loss → backward → wandb → checkpointing → inference → mp3 export.

The loss behaved — smooth decay, stable at batch 8, no NaNs, no divergence.

The pipeline is sound enough to scale: nothing in the loop needs rearchitecting before a longer run.

Read the numbers with care

Train = test = 1.1403. With only 16 held-out clips, the test loss is a sanity signal, not a generalization estimate.

Near-identical curves say more about the tiny subset — the model has capacity to spare over 256 clips — than about learning to speak.

A falling curve here certifies the machinery; it does not yet certify a voice.

Where 600 steps sits

The reference run only flattens near test ≈ 0.23 after roughly 400,000 steps.

600 / 400,000 ≈ 0.15 % of the journey — the smoke test is the first breath of a long descent.

1.14 is a healthy place to be this early; it is not a place to stop and listen.

Discussion
13 / 13

Limits, and the road ahead

What still holds it back

vocoder  Griffin-Lim rebuilds phase by iteration — a metallic ceiling no amount of training lifts; a neural vocoder is the modern fix.

letters  Character-only input leaves pronunciation edge cases to chance — no phonemizer in the loop.

subset  256 clips invite memorization rather than speech; and LJSpeech is one speaker, by design.

test set  Sixteen clips cannot steer hyperparameters — only a full run can.

Next on the roadmap

i.  Scale to the full 13,100 clips, batch back to 32.

ii.  Train ≥ 100k steps with Drive checkpoints through runtime resets.

iii.  Add a phonemizer to the text front-end.

iv.  Swap Griffin-Lim for a HiFi-GAN vocoder.

v.  Compare runs in wandb — and listen at every milestone.

It says hello, world — after enough steps.
01 / 13
editing · text autosaves locally · ⌘S / Ctrl+S exports the file