Most captioning viewers are hearing people. If you are a native speaker of the language being captioned, you sit there and follow the speech and captions simultaneously. You do that for all kinds of programming, including how-to shows, dramas, music videos, porn, or whatever else. The captions appear and disappear (or scroll or paint on) under someone else’s control.
But it turns out that your understanding of instructional programming may actually be impaired by simultaneously reading text and hearing the same words.
A paper by Slava Kalyuga et al, “When redundant on-screen text in multimedia technical instruction can interfere with learning,” conducted several experiments with teenage learners involving simultaneous or sequential presentation of visible and spoken text. Generally, the subjects had to sit there and think harder when the two media were presented simultaneously. Presenting words and speech one after another resulted in better test scores.
The trainees’ control over the pace of instruction could be an important factor influencing our results. For example, [another study] showed that replacing on-screen text with audio narration (the modality effect) was effective only when the pace of instruction was set by the time of the narration and students had no control over the pacing. […]
The instructional format based on an auditory-only presentation of text was significantly more efficient than the concurrent audio and visual presentation format…. These results indicate that a redundancy effect was obtained under conditions that required learners in the concurrent text group to read and listen to the same text simultaneously for a limited time and that required learners in the auditory-only text group to listen to identical auditory explanations for an equal amount of time without reading them. Having the same information presented in two modes simultaneously is less effective than when it is presented in one mode alone under conditions in which the pacing of instruction is controlled by the system.
So:
- Does this same effect apply to watching instructional programs on TV?
- Does it apply to watching TV at all? Do the moving images interfere with the effect?
- How does the effect vary according to program type?
- Does the same thing apply to hard-of-hearing people? (I doubt it applies to totally-deaf people.) How about ESL learners?