A Deep Dive into Phonemic Transcription Metrics

An Introduction

The International Phonetic Alphabet (IPA) provides a useful phonetic alphabet that can detail a speaker's pronunciation. It is also the vocabulary of the Koel Labs phonetic transcription models, enabling us to model accent variation and develop amazing applications for language learning! In recent years, predicting these phonemic transcriptions from audio has become a popular machine learning task. But how do we calculate the accuracy of these models?

At Koel Labs, we use two key metrics to evaluate phonemic transcription models:

1. Phonemic Error Rate (PER): The classic "how many mistakes did you make?" metric
2. Weighted Phone Feature Error Rate (WPFER): A smarter approach that weights each mistake by how acoustically similar the sounds are

Why Traditional Metrics Fall Short: A Tale of Three Words

Let's say we're trying to transcribe the word "Bop". Our model could make different types of mistakes.

Consider two models making different predictions:

• Model 1 predicts: "Pop"
• Model 2 predicts: "Sop"

From a linguistics perspective, these mistakes are not created equal:

• 'B' and 'P' are like cousins. They're both plosive bilabial consonants, made by stopping airflow with your lips. The only difference is that 'B' is voiced (your vocal cords vibrate) and 'P' isn't.
• 'B' and 'S', on the other hand, are more like distant relatives. 'S' is a fricative alveolar consonant, made by forcing air between your tongue and the ridge behind your upper teeth: a completely different sound.

This is where traditional PER falls short. It calculates errors based on simple substitutions, deletions, and insertions. In our example:

"Bop" → "Pop": 1 substitution = 33.33% error rate
"Bop" → "Sop": 1 substitution = 33.33% error rate

That's like saying someone who just missed the bullseye did as poorly as someone who hit the wall next to the dartboard, which makes for misleading evaluations.

Weighted Phone Feature Error Rate

WPFER addresses this, powered by the Panphon library. Instead of treating each phoneme as completely different or identical, it represents them as a sequence of features, things like:

• Is it voiced?
• Where in the mouth is it made?
• How is the air released?

Each phoneme becomes a feature vector, something like:

B: [+voiced, +bilabial, +plosive, -fricative, ...]
P: [-voiced, +bilabial, +plosive, -fricative, ...]
S: [-voiced, +alveolar, -plosive, +fricative, ...]

When we measure the distance between these vectors, we get a much more nuanced view. PER scores both errors identically, but WPFER pulls them apart:

PER("Bop" → "Pop") = 0.33     WPFER("Bop" → "Pop") = 0.006  // b→p: just voicing
PER("Bop" → "Sop") = 0.33     WPFER("Bop" → "Sop") = 0.060  // b→s: place AND manner

WPFER sums the feature-weighted cost of every edit and divides by the length of the ground truth, with a normalization factor (one over twice the total feature weight) that keeps it on a small, PER-comparable scale. Identical transcriptions score 0, and the further apart the sounds, the higher the number. That's why the "Sop" error lands about ten times higher than the near-miss "Pop", even though PER can't tell them apart. You can find the exact implementation in our ML repo.

Why This Matters

When you're teaching a model to transcribe speech, you want it to understand that predicting a similar sound is better than predicting a completely different one. This is especially important because different models might use different phoneme vocabularies, with some using 40 symbols and others up to 400.

Traditional PER might unfairly favor models that happen to use the exact same phoneme set as your ground truth data, even if other models are making more linguistically sensible predictions. WPFER helps level the playing field by considering phonetic similarity.

The Takeaway

Reporting WPFER alongside PER gives a fuller picture of how a model is really doing. It measures not just whether you picked the exact right symbol, but how close you came to the right sound, which is what matters for pronunciation feedback.

You can compare both metrics across open models on our IPA transcription leaderboard:

For the full details, including how we use WPFER to train and evaluate our models, see our paper, Scaling Human and G2P Supervision for Robust Phonetic Transcription.