We present neural models of one of humans’ most astonishing cognitive feats: the ability to interpret linguistic instructions in order to perform novel tasks with just a few practice trials. Models are trained on a set of commonly studied psychophysical tasks, and receive linguistic instructions embedded by transformer architectures pre-trained on natural language processing. We found that the resulting neural representations capture the semantic structure of interrelated tasks even for novel tasks, allowing for the composition of practiced skills in unseen settings. Finally, we also demonstrate how this model can generate a linguistic description of a task it has identified using motor feedback. To our knowledge, this is the first experimentally testable neural model of how language can structure sensorimotor representations to allow for task compositionality.
Organizer
Aaron Wong
a.wong@erasmusmc.nl