2512.09610

Total: 1

#1 ImageTalk: Designing a Multimodal AAC Text Generation System Driven by Image Recognition and Natural Language Generation [PDF] [Copy] [Kimi] [REL]

Authors: Boyin Yang, Puming Jiang, Per Ola Kristensson

People living with Motor Neuron Disease (plwMND) frequently encounter speech and motor impairments that necessitate a reliance on augmentative and alternative communication (AAC) systems. This paper tackles the main challenge that traditional symbol-based AAC systems offer a limited vocabulary, while text entry solutions tend to exhibit low communication rates. To help plwMND articulate their needs about the system efficiently and effectively, we iteratively design and develop a novel multimodal text generation system called ImageTalk through a tailored proxy-user-based and an end-user-based design phase. The system demonstrates pronounced keystroke savings of 95.6%, coupled with consistent performance and high user satisfaction. We distill three design guidelines for AI-assisted text generation systems design and outline four user requirement levels tailored for AAC purposes, guiding future research in this field.

Subjects: Human-Computer Interaction , Artificial Intelligence , Computer Vision and Pattern Recognition

Publish: 2025-12-10 12:57:55 UTC