Higher sem. TÖI. Jan-Louis Kruger: Reading in Multimodal Contexts: From Subtitles to ASR-Supported
Simultaneous Interpreting

Reading in multimodal settings is rarely linear. In subtitle viewing, gaze alternates between image and text, producing scanning, corrective refixations, crossovers and frequent regressions. Two early-stage latencies are especially informative: (i) time to first fixation (delay from subtitle onset to first glance) and (ii) processing latency (time to begin progressive reading after initial corrections). Both reflect the cost of programming a saccade from the image to the subtitle and locating a viable launch point for reading. Since subtitle reading is also supported by information in the soundtrack and visual cues on the image, eye movement patterns in this context differ substantially from the linear patterns characteristic of static text reading. In particular, readers lean towards skimming and keyword checking.

We extend this lens to simultaneous interpreting (SI) with on-screen automatic speech recognition (ASR). In SI with ASR, the interpreter must reconcile multiple latencies along the pipeline (from speech onset through ASR processing to display) before integrating the information in the transcript with the speech and their own production. Prior work suggests that latency can shift attention allocation across channels and reshape timing and strategy, raising practical questions about when ASR helps and when it hinders.

Our study asks whether interpreters can exploit ASR output in simulated naturalistic conditions and how latency modulates that benefit. We analyse non-numeral tokens and numerals separately, given ASR’s tendency to delay multiword spoken numerals rendered as single digits (e.g., “three hundred and seventy” → “370”), which inflates onset delays. We predict that longer word-level transcript latency will increase skipping and reduce local reading depth (fixation count and dwell) in ASR trials, while at the global level longer latency will increase regression rate, lengthen forward saccades, reduce linearity, and raise global skipping; at the performance level, we test whether ASR presence improves accuracy and whether latency degrades it relative to 0-latency.

We use a within-subjects design with three conditions: No-ASR, ASR-0 (0-latency transcript), and ASR-Lat (natural latency), counterbalanced across three ~600-word talks. Word-level latencies in ASR-Lat come from the natural empirical distribution of observed (non-numeral) latencies, with a monotonic display constraint and a minimum inter-word gap to prevent visual “bursts.” Numerals retain their natural latencies; they are never advanced and are only lifted to preserve order. Outcomes include trial-level interpreting accuracy (primary), ASR-only local gaze (e.g. skip, fixations, dwell) for non-numerals, ASR-only global gaze (regression rate, forward saccade length, linearity, global skipping), and numeral correctness modelled against display onset relative to speech and spoken-span length.

By bridging evidence on subtitle reading with SI with ASR, we characterize how multimodal reading routines adapt under realistic, time-locked constraints and identify latency regimes under which ASR is likely to be usable, ignorable, or disruptive for professional interpreters. Findings will inform technology design (e.g., stability vs. speed trade-offs), interpreter training, and methodological standards for studying human–AI collaboration in naturalistic settings.