Total: 1
The best ASR models on the Hugging Face Open ASR Leaderboard have 5-7% WER on standard benchmarks, yet their reliability on nonstandard speech remains uncertain. We present the first leaderboard-anchored evaluation of leading open models spanning encoder-decoder (Whisper, CrisperWhisper), transducer (Parakeet), speech-augmented LM (Canary), and speech-aware LLM (Granite) architectures on stuttered speech and dysarthric speech. Using curated subsets of FluencyBank, SEP-28k, and the UIUC Speech Accessibility Project dataset, we evaluated transcripts with WER, CER, F1 and BERTScore metrics. The average WER increases 2-5× over leaderboard results, with extreme cases reaching 24× inflation from hallucination bursts. Overall, Whisper-Large-v3 generalizes most robustly, while Parakeet shows complementary strengths on dysarthric speech. The results expose a substantial generalization gap and underscore the urgent need for accessibility-focused ASR evaluation.