Predictive perception: multisensory consequences of one’s own actions
Funding: DFG
When planning an action, we generate predictions about the sensory consequences of that action. These predictions enable us to achieve more efficient stimulus processing and discriminate self-generated from externally-generated stimuli. Such predictive processes are thus essential to establish meaningful interactions with the environment. Previous research has investigated action predictions using unimodal action consequences. However, most actions lead to multisensory consequences. For example, when knocking on a door, we receive visual, auditory, tactile, and proprioceptive feedback. The (neural) mechanisms underlying such multisensory predictions are unclear.
The goal of the PACT project is to investigate the mechanisms underlying the prediction of multisensory consequences of one’s own actions. The investigations focus on three different aspects: 1) supramodal vs. unimodal effects of action-sensory feedback matching processing, 2) the neural correlates of supramodal predictive mechanisms and their consequences and 3) the investigation of predictive mechanisms for tool-use actions and the perception of related multisensory consequences.
Our results from the first funding period support the existence and relevance of multisensory predictive mechanisms on the behavioral and neural levels. In fMRI studies, a BOLD suppression for consequences of active compared to passive hand movements in auditory, visual and sensorimotor brain regions could be demonstrated. The relevance of cerebellum, angular gyrus and temporal lobe for predictive or comparator functions could be validated by parametric analyses. However, modulation of behavioral delay detection performance highlights a differentiation between continuous action feedback (e.g., seeing the hand moving) and discrete action outcomes (e.g., dot on a screen after a button press), as we found better performance (enhancement) for discrete action outcomes (active>passive) and reduced performance (behavioral suppression) for continuous natural video feedback (passive>active). Furthermore, we identified striking individual differences in multisensory action outcome monitoring suggesting a compensatory function of multisensory processing in subjects with poor perceptual abilities. Variation in multisensory processing was further related to individual differences in self-reported gesture perception and production during everyday life situations.
These new findings raise a number of questions. In the second founding period, we will therefore (1) directly compare processing of continuous feedback and discrete outcomes of actions in comprehensive investigations and (2) examine the source of individual differences in multisensory action feedback/outcome processing. In addition, (3) supramodal recalibration of the forward model based on temporal prediction errors is a new research focus. The investigation of cross-modal adaptive processes within the action-perception cycle will i) allow us to directly test our hypothesis of multisensory predictive mechanisms, ii) help to understand the relevance of the cerebellum for recalibrating the forward model (comparator function) and iii) demonstrate the flexibility of an action-perception cycle in a context of predictable delays.
Who’s working on PACT?
Principal investigators: Benjamin Straube, Tilo Kircher
Postdocs: Bianca van Kemenade, Eleftherios Kavroulakis
Former PhD students: Mareike Pazen
Former associated PhD student: Lukas Uhlmann
Former medical student: Benjamin Schmalenbach
Student assistant
Further information
Cooperations regarding action and perception
Related publications
van Kemenade, B.M., Arikan, B.E., Podranski, K., Steinsträter, O., Kircher, T., & Straube, B. (2019). Distinct roles for the cerebellum, angular gyrus and middle temporal gyrus in action-feedback monitoring. Cerebral Cortex, 29(4), 1520–1531. https://doi.org/10.1093/cercor/bhy048. IF: 5.437
Lubinus, C., Einhäuser, W., Schiller, F., Kircher, T., Straube, B.* & van Kemenade, B.M.* (2022). Action-based predictions affect visual perception, neural processing, and pupil size, regardless of temporal predictability. NeuroImage, in press. IF: 7.4
Pazen, M., Uhlmann, L., van Kemenade, B.M., Steinsträter, O., Straube, B. & Kircher, T. (accepted). Predictive perception of self-generated movements: Commonalities and differences in the neural processing of tool and hand actions. NeuroImage IF: 5.426
Arikan, B.E., van Kemenade, B.M., Podranski, K., Steinsträter, O., Straube, B., & Kircher, T. (accepted). Perceiving your hand moving: BOLD suppression in sensory cortices and the role of the cerebellum in the detection of feedback delays. Journal of Vision. IF: 2.089
Krala, M., van Kemenade, B., Straube, B., Kircher, T. & Bremmer, F. (accepted). Predictive coding in a multisensory path integration task: an fMRI study. Journal of Vision. IF: 2.266
van Kemenade, B. M., Kircher, T., Arikan, B. E., & Straube, B. (2016). Predicting the sensory consequences of one’s own action: First evidence for multisensory facilitation. Attention, Perception, & Psychophysics. http://doi.org/10.3758/s13414-016-1189-1
Straube, B., van Kemenade, B.M., Arikan, B.E., Fiehler, K., Leube, D., Harris, L.R., & Kircher, T. (2017). Predicting the multisensory consequences of one’s own action: BOLD suppression in auditory and visual cortices. PlosOne. Doi:1371/journal.pone.0169131 IF: 3.057
van Kemenade, B.M., Arikan, E.A., Kircher, T., & Straube, B. (in press). The angular gyrus is a supramodal comparator area in action-outcome monitoring. Brain Structure and Functioning. IF: 5.811
Straube, B., Schülke, R., Drewing, K., Kircher, T., & Kemenade, B. M. van. (2017). Hemispheric differences in the processing of visual consequences of active vs. passive movements: a transcranial direct current stimulation study. Experimental Brain Research, 1–10. http://doi.org/10.1007/S00221-017-5053-X IF: 1.917
Arikan, B.E., van Kemenade, B.M., Straube, B., Harris, L.H., & Kircher, T. (accepted). Voluntary and involuntary movements widen the window of subjective simultaneity. i-Perception IF: 1.051
Schmalenbach, S. B., Billino, J., Kircher, T., Van Kemenade, B., & Straube, B. (2017). Links between gestures and multisensory processing: individual differences suggest a compensation mechanism. Frontiers in Psychology, 8, 1828. https://doi.org/10.3389/FPSYG.2017.01828 IF: 2.323
Translational Neuroimaging Marburg 