880419 :Multimodality & Communication (BDM/CD)


Voertaal Engels
Werkvorm: 2 small group seminars per week, one literature discussion seminar, one assignment seminar (Collegerooster)
Tentamenvorm: tentamen (50%) + paper (50%) (Tentamenrooster)
Studielast:6 ECTS credits
Inschrijving:This course has a maximum capacity of 40 participants. If you filled out the survey you received recently (“selecting master courses CIS Fall semester 2017) you will be enrolled automatically. Other students: see specifics.
Blackboard informatieLink to Blackboard (Als u de melding 'Guest are not allowed in this course' krijgt, dient u nog bij Blackboard in te loggen)


dr. J. Schilperoord (coordinator)

dr. N.T. Cohn

Doel van de cursus (alleen in het Engels beschikbaar)

The aim of the course is for students to gain insight into the ways people process and understand multimodal information: information that is coded in different modalities, such as visual and verbal language, and which can relate to each other in different ways: complementary, similar, differently. More in particular, we examine the characteristics of verbal descriptions - texts casted in verbal language - people produce of brief narrative comics - texts casted in visual language - and compare the two types of texts on various types of characteristics: structural and regarding content. To meet these ends, we study several texts on multimodality and narrative structures in verbal and visual messages.

After the course the students are able to: 

  • Describe, analyze and compare relevant literature on multimodality
  • Interpret and evaluate relevant experimental research on multimodality. 
  • Describe and evaluate main stream theoretical accounts of multimodality in cognition and communication
  • Analyze multimodal narrative texts
  • Apply and evaluate different research methods
  • Design a research project  question and research question in the field of multimodality
  • Interpret and analyze experimental data 

This course contributes to student’s skills concerning

  • Processing, interpreting and evaluating scientific literature
  • Autonomous acquisition of new knowledge and broadening existing knowledge
  • Designing new research questions on the basis of existing literature, application of knowledge to other fields (transfer knowledge).

This course contributes to student’s skills to:

  • Motivate research questions on the basis of literature and preceding research.
  • Discuss theories and apply them to relevant data
  • Analyze and search for appropriate (multimodal) material
  • Write a scientific paper.

Inhoud van de cursus (alleen in het Engels beschikbaar)

All student activities during this course will center on designing, conducting and reporting about an experiment, which you carry out in small groups (4 or 5 students). The course starts by making students familiar with the so-called Visual narrative Grammar, developed by Neil Cohn. VNG is a model of the way people process and interpret sequential visual messages: comics. It distinguishes various types of panels and types of relations between panels. Students will be taught and train themselves to analyze short, 5 or 6 panel comics, noticeably Charley Brown comics, according to VNG. In addition, each groups asks a number of participants to provide verbal descriptions of the same comics. Participants will be given the assignment to produce short verbal narratives on the basis of a certain comic strip in a way that makes the text understandable and likeable for readers who do not have access to the visual variant of the narrative. The experimental and analytical task will be to compare the two versions - one casted in visual language and several others in verbal language - according to various narrative criteria:           
1. the overall structure of the narratives, the ways the two types of messages mark structural aspects of the narratives,
2. the content of the narratives
3. processing factors, noticeably the temporal structure of the production of verbal narratives viz a viz the visual input.

Each week contains a 2 hour literature seminar, and a 2 hour seminar on analyzing verbal and visual narratives and designing the experiment. Attending this course requires willingness and eagerness to engage in the following activities: 

Literature study

  • The individual study of research articles (essays, experimental papers, corpus research papers), contemplating its merits and reflecting on it with fellow participants. 
  • Participation in weekly literature seminars where literature (1, 2 or 3 articles) will be discussed. In the first seminar(s) we will focus on the different types of questions which can be asked about literature. Gradually, we make a shift from questions prepared by supervisors to self-imposed questions which guide studying papers. So, questions of various types, from comprehension over application to evaluation and finally to synthesis, will be prepared by students and discussed in groups under the supervision of the teacher. 


  • Participation in weekly seminars where analytical issues are discussed and practiced, and all matters having to do with designing and conducting the group experiment.

Preparation of seminars also includes meetings with colleague students to prepare the seminars.  

Bijzonderheden (alleen in het Engels beschikbaar)

This course has a maximum capacity of 40 participants. If you filled out the survey you received recently (selecting master courses CIS Fall semester 2017) you will be enrolled automatically. All other students have to send an e-mail to: mastercoursesCIW@uvt.nl and will be enrolled if places are available.

The final mark for the course is based on: (1)  an individual final exam on the required reading (50%); (2) the group paper about the experiment. To complete the course, both the mark for the final exam and the paper must be minimally a pass.

Verplichte literatuur

  1. Selection of about 12 articles and tutorials.

Aanbevolen literatuur

  1. Some assignments require also studying and searching additional literature

Gewenste voorkennis

Apart form an analytical mindset and interest in the wonders of multimodal communication, it is highly recommended for participants to be intrigued by and attracted to comics and all kinds of other forms of visual communication.

Vereiste voorkennis


Mogelijk interessant voor