These guidelines specify a procedure for annotating non-manual elements in visual communication data.

• Visual communication data are videos or motion capture files used to investigate visual communication between people. Visual communication includes multimodal communication (speaking and gesturing at the same time), silent gesture (gesturing alone, often in a lab setting), or sign language (e.g., Dutch Sign Language).

• Non-manual elements are facial expressions, and head and body movements that people make while communicating. Manual elements are hand movements and configurations.

You are currently reading the FIRST PRODUCT version of the guidelines, published July 2025.

In Section 2 and Section 3, we discuss the two basic principles for annotating non-manual elements according to the present guidelines.

In Section 4, we briefly introduce the annotation software ELAN.

In Section 5, we present the ELAN annotation template that can be used by annotators.

In Section 6, we discuss all annotation labels that are included in the annotation template.

In Section 7, we offer some general background information and recommendations for researchers.

In research on visual communication, ELAN is frequently used as annotation software. This is why our annotation template makes use of ELAN.

In principle, it is also possible for researchers to use the annotation guidelines with different annotation software. The basic principles that are discussed in Section 1 and 2, and the annotation labels described in Section 6 are described in a way that is independent of any particular features of the software that is used for annotation.

Section 4 is not relevant for researchers who use annotation software other than ELAN, because this section exclusively discusses ELAN. Section 5 remains partially applicable, but the tutorial sections about ELAN are not relevant in this case.

Example annotations based on the current version of the guidelines will be made available in the future.

Example annotations based on the previous (prototype) version of the guidelines can be downloaded here (under Files > Example annotations ).

It is very important that you thoroughly absorb the information in Sections 2 to 5 before you start annotating.

These chapters contain quite a lot of information so studying them requires some time investment, but this is essential for the annotation process.

There are two general strategies for annotating a certain segment in a video:

• Annotating based on form
• Annotating based on comparison

In principle, you are meant to annotate based on form .

However, often this is easier said than done! When you annotate, you will often come across non-manual configurations or movements which fall in a certain gray area (e.g., the eyebrows are not fully raised but not really neutral either), consequently making them difficult to label. A major challenge is to determine what to do in such cases. We have developed a concrete guideline for this scenario, which we discuss in this section.

In principle, you annotate based on form . As an example, below you see two video stills from a fragment in which a signer is telling a short story in Sign Language of the Netherlands. We are interested in annotating the configuration of the eyelids in this example. In our annotation template, we make annotations for this body part on the tier ‘s.eyelids’. This tier includes four possible annotation labels: neutral , squinted , widened , closed (see Section 6 for descriptions).

If you had to annotate these video stills purely based on form, which labels would you opt for?

In all likelihood, you will agree with our label choice:

In these cases, we are able to select the annotation labels squinted and neutral with reasonable certainty.

But how would you label the following video still?:

It is very well possible that you are hesitating here between the labels squinted and neutral .* Put differently, if we were to ask ten different people to choose an annotation label for this image, then it is likely that some will choose squinted (because the eyelids are squeezed together a little bit), while others will opt for neutral (because the eyelids are only squeezed together the tiniest bit).

*Perhaps you even considered the label widened , but be cautious: the eyelids are not widened here; it is the eyebrows that give the possible illusion of widened eyelids.

The example given in the previous subsection is difficult to label based on form because it appears to fall in between two annotation values. In such cases, you probably automatically switch to a different strategy: instead of looking purely at form , you start comparing what you see with other cases which you already labeled and which you felt a lot more certain about:

The problem is that annotating based on comparison is a lot more subjective, substantially increasing the likelihood that different annotators make different choices.

If you were to order the three video stills introduced above on a sort of scale from obviously squinted to obviously neutral , it is well possible that you would do it something like this:

But it is equally possible that you see it as something like this:

In the first scenario, you lean more toward the label squinted , and in the second scenario to neutral . It might even happen that you would go for the label squinted on one day, but feel more for the label neutral on another. Not ideal, if the aim is to annotate consistently and reliably!

At the same time, this problem is simply unavoidable when annotating non-manual elements. After all, we are trying to stick categorical labels on things that are not categorical at all, but rather gradual .

This is why below, we shall make explicit how you as an annotator are expected to handle this issue while annotating.

First of all, it is very important for annotators to realize and acknowledge that non-manual elements simply are sometimes (or maybe even often) difficult to label.

We therefore ask annotators to indicate for every annotation how certain they are about the label that they chose. This is done on the annotation tier ‘confidence.about.label’ (see Section 5 ).

You always choose between the labels high , medium and low on this tier.

The basic principle is: the more you can rely purely on form while choosing an annotation label, the higher your confidence in that label; the more you are forced to resort to comparison with other annotation labels, the lower your confidence.

By indicating your confidence in the label that you choose, it becomes possible to take that information on board when analyzing your data later. Researchers may, for instance, decide to only focus on annotations that received a high confidence score, and ignore all the other annotations.

In Section 6, we describe all possible annotation labels for the different tiers. We will only provide examples that are as obvious as possible, i.e., examples that you can label purely based on form . In terms of confidence scores, these are thus examples that we would give the score high .

We purposely do not give examples of cases which we would ourselves give a confidence score of medium or low , because there would be a fair ( medium ) or even considerable ( low ) chance that different annotators label these examples differently.

(Note: We plan to evaluate, based on a test data set, if confidence scores and inter-rater agreement are indeed correlated like this in future work.

• Choose, where possible, annotation labels purely based on form . In Section 6, we specify which label fits with which form.
• If it is not possible to annotate (purely) based on form, then (also) use the comparison strategy.
• Confidence scores are an indication of how certain you are about the labels that you selected. Degree of certainty, in turn, is correlated with the annotation strategy/strategies you used:

• We make a distinction between simplex and complex events.

• An example of a simplex event is ‘eyebrows raised'. Such an event involves a single target configuration , and the transition toward this target configuration in this case is a simple, linear transition.
• Another example of a simplex event is 'head circling'. Unlike in the case of ‘eyebrows raised', the transition that leads from the initial configuration to the final target configuration is non-linear in this case. Still, the event cannot be broken up in a meaningful way into a sequence of even more elementary constituent events. Therefore, we still consider it to be a simplex event.
• An example of a complex event is ‘head shake’. Such an event involves a sequence of simplex events (e.g. rotating the head to the left, then to the right, and then back to a neutral position).
• The distinction between simplex and complex events is also made in the annotation template: we distinguish between simplex event tiers and complex event tiers . The name of a simplex event tier always starts with s. and the name of a complex event tier always starts with c.
• In the subsections below, we will give general definitions of simplex and complex events, discuss the choices that researchers have w.r.t. the annotation of simplex and complex events, and the advantages of drawing a distinction between these two types of events.

What are simplex events?

• Definition : a simplex event is an event that involves single target configuration (e.g. raised eyebrows) and cannot be broken up in a meaningful way into a sequence of more elementary events.

• The transition toward the target configuration can be fast, but it can also be slow and gradual (e.g. a slow gradual eyebrow raise across an entire sentence).
• The transition toward the target configuration is often linear (e.g. in the case of an eyebrow raise or a rotation of the head to the left or right) but can also be non-linear (e.g. when the head or the eyeballs make a circling movement).
  

The label neutral is always available on simplex event tiers

• You can always select the label neutral on a simplex event tier.
• The reason for this is that a neutral configuration is also a possible target configuration.

• For a detailed description of the annotation labels for all tiers, see Section 6 .

Where does a simplex event begin and end?

• A simplex event always includes the transition toward the target configuration.

• So it begins when the relevant articulator starts to transition from the previous event's target configuration toward the current event's target configuration.

• This means that you start a new annotation at the moment the configuration begins to change.

In the example below, for instance, the eyebrows go from a neutral configuration to a raised configuration. You annotate this on the 's.eyebrows' tier as follows:

What are complex events?

• Definition : a complex event always consists of a particular sequence of simplex events .
• For instance, a headshake may consist of three simplex events:
1. A rotation of the head to the left,
2. Then a rotation of the head to the right,
3. And finally a rotation of the head back to a neutral position.
• Another example: an eyeblink consists of two simplex events:
1. Closure of the eyelids,
2. And then re-opening of the eyelids.

The label neutral is never available on complex event tiers

• The label neutral is never available on a complex event tier.
• This is because, even if a complex event ends with a neutral configuration (this is typically the case for many complex events, e.g., headshakes and eyeblinks), the label that is assigned to the event never just describes the final configuration but rather the way in which this final configuration was reached.

Where does a complex event begin and end?

• Just like a simplex event, a complex event also begins when the relevant articulator starts to transition from the previous event's target configuration towards a new configuration.

• It ends when the articulator reaches the final target configuration.
• For instance, the annotation for a blink ('c.eyeblink' tier) starts when the eyelids begin to close, and ends when the eyes are open again.

Complex event tiers are typically sparsely annotated

• On complex event tiers, there are usually many unlabeled segments. A label is only assigned when the relevant type of complex event occurs.

• For example, the 'c.eyeblink' tier only contains short labeled segments when eyeblinks occur. Segments in between eyeblinks are left unlabeled.

Simplex event tiers are densely annotated

• Simplex event tier, on the other hand, are densely annotated.
• In fact, there are only two reasons why a given segment on a simplex event tier would not be labeled, namely:
1. It may be that the configuration of the relevant articulator is invisible; for instance, a hand in front of the face may make it impossible to determine the configuration of the eyebrows, or closed eyelids make it impossible to determine the eye gaze direction.
2. It may be that the relevant articulator is engaged in a complex event and that it is sufficient for the purposes of your research project to annotate the complex event on the complex event tier without also annotating the constituent simplex events on the simplex event tier; for instance, in the case of a headshake, it may not be necessary to annotate the simplex constituents of the event but only the complex sequential event as a whole.
• If neither of these situations applies, a label should always be assigned on a simplex event tier.

Simplex event tiers are Time Subdivision tiers

• In light of the above considerations, we will assume that simplex event tiers are always Time Subdivision tiers.

• This means that all segments of a given annotation unit must be assigned a certain label on such tiers; no segment can be left unlabeled (see the ELAN manual or ELANport for more information on Time Subdivision and other tier types).

To make this possible, also in the situations indicated above, our ELAN template ensures that the following two labels are always available on simplex event tiers:

• Occluded
  This label is used whenever the relevant articulator is occluded (invisible) for some reason (e.g., a hand in front of the face, or eyelids closed so that eye gaze direction cannot be seen).
• Complex event
  This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. a headshake) which is annotated on the corresponding complex event tier.

Complex event tiers are not Time Subdivision tiers

• Unlike simplex event tiers, complex event tiers are not of type Time Subdivision tiers but rather of type Included In .
• This means that not all segments of a given annotation unit need to be labeled; typically, only some segments are labeled (again, see the ELAN manual or ELANport for more information on Time Subdivision and other tier types).

• The fact that simplex and complex events involving the same articulator are annotated on separate tiers gives researchers flexibility .
• For a given study, researchers may choose (for each relevant articulator) to only annotate simplex events, to only annotate complex events, or to do both.
• If they decide to annotate both simplex and complex events for a certain articulator, they still have a further choice to make: namely, whenever the relevant articulator is engaged in a sequential complex event , researchers can choose between:
1. Annotating the complex event holistically on the complex event tier (e.g., ‘nodding’) and annotating all simplex events in the sequence on the simplex event tier as well (e.g. ‘up’ - ‘neutral’ - ‘up’ - ‘neutral’ - ‘up’), or
2. Annotating the complex event holistically on the complex event tier (‘nodding’), and using a dedicated label ‘complex event’ on the simplex event tier, just to indicate that there is a complex event taking place.

Depending on the choices researchers make, they adopt one of the following four annotation styles (per articulator):

1. Simplex only: Only annotate simplex events.
2. Complex only : Only annotate complex events.
3. Both, but sequential events only labeled holistically : Annotate both simplex and complex events; in the case of a sequential complex event, label the event holistically on the complex event tier and use a dedicated label 'complex event' on the simplex event tier.
4. Both, sequential events labeled holistically and piecemeal : Annotate both simplex and complex events; in the case of a sequential complex event, label the event holistically on the complex event tier and also annotate all simplex events in the sequence on the simplex event tier.

These annotation styles are summarized in the following diagram.

Making a distinction between simplex and complex events, and including separate simplex and complex event tiers in the annotation template has a number of advantages.

Flexibility and efficiency

• Many studies focus only on simplex events, or only on complex events.
• The fact that the ANM annotation template includes separate simplex and complex event tiers gives researchers the flexibility to use only those tiers that are really relevant for the purposes of their study.
• This also makes the annotation process more efficient: annotators don’t have to do more work than necessary.

Transparency w.r.t. subjective interpretation

• It is impossible to specify univocally when a sequence of simplex events (e.g. rotating the head to the left, then to the right, and then back to a neutral position) should be seen as constituting a complex event (a headshake) rather than just a series of simplex events which incidentally mirrors the pattern of a particular type of complex event.
• As a consequence, annotating sequential complex events always requires some level of subjective interpretation on the annotator’s part as to whether a given temporal progression of simplex events constitutes a certain complex event or not.
• Annotating simplex events does not require such interpretation and is therefore more objective (though it may of course still involve other subjective choices).
• Having separate simplex and complex event tiers makes it easier for researchers to recognize which annotations require this kind of interpretation on the part of the annotator, and thus require extra caution in the analysis.
• Also, it gives researchers who do not want annotators to make such type of decisions the option to ask annotators to consider simplex event tiers only. Complex events (e.g. headshakes) are then always just annotated as sequences of simplex events (e.g. left - right - neutral) and never holistically.

Tier type distinction increases annotation quality and uniformity

• Simplex event tiers are always of the type Time Subdivision, while complex event tiers are always of the type Included In.
• We expect that this clearcut differentiation in terms of tier type, and especially the fact that simplex event tiers are always Time Subdivision tiers, will enhance annotation quality and uniformity of annotations across projects.
• This is because on simplex event tiers, annotators will have to label each segment of each annotation unit -- it is impossible to skip or forget to label certain segments.

• ELAN is an annotation tool that is often used in research on sign languages and multimodal communication.
• ELAN can be downloaded here .
• The full ELAN documentation is available here .
• Short ELAN video tutorials for beginners – in Dutch Sign Language (NGT) and English (voice-over/subtitled) – are available on ELANport .
• A quick guide to using ELAN is available here .

• Annotations in ELAN can take the form of a sentence, word, gloss, or comment.
• It is possible to create a list of possible annotation labels – a Controlled Vocabulary – in advance. This allows annotators to efficiently pick labels from a list.
• Annotations can be added on multiple layers; these are called tiers . Tiers are hierarchically organized. Annotations that are added to tiers, are temporally aligned with the video data.
• Annotations can be saved as text documents with the extension .eaf. A set of tiers can be saved as a template in an .etf file. Accompanying Controlled Vocabularies are saved as .ecv files.

To facilitate the annotation of non-manual elements according to the ANM guidelines, we have developed an ELAN annotation template. The template and supplementary materials can be downloaded here .

• Annotation template (.etf)
• Go to Files > Annotation template
• Controlled Vocabularies (.ecv)
• Go to Files > Annotation template
• Example annotations (.eaf)
• Go to Files > Example annotations
• Example video (.mp4)
• Go to Files > Example annotations

• For use, download the template (.etf file) and the accompanying Controlled Vocabularies (.ecv file) and save them on your computer.
• A tutorial video explaining how to work with the annotation template is available here . It will be helpful to watch this video tutorial before continuing to read the information below.
• If you are going to annotate a new video file, then go to File > New… in ELAN. Then select the desired video files. Make sure that ‘Media’ is checked and press ‘>>’. Do the same for the template file (select ‘Template’ first):

• If this hasn’t already happened automatically, also load the Controlled Vocabularies via Edit > Edit Controlled Vocabularies… > External CV > Browse… Find the folder in which you saved the .ecv file, select the file, and click OK.

Now you see something like this:

The annotation template has three levels:

• Level 1 : One tier on which the relevant annotation units in the video are indicated.
• Level 2 : A set of 21 tiers on which all non-manual elements are annotated.
• Level 3 : A set of tiers on which annotators provide additional information about the labels that were selected on Level 2 tiers.

• Level 1 : There is only one tier on this level. This tier is called 'non-manuals.annotation-units'. On this tier, it is indicated which segments of the video contain non-manual elements that should be annotated. Each such segment receives a label.
• Level 2 : There are 21 tiers on Level 2. These are all daughter tiers of the main tier 'non-manuals.annotation-units'. On every Level-2 tier, a different non-manual element is annotated. Which element that is, is indicated by the tier name (e.g., 's.eyebrows'). A distinction is made between simplex event tiers and complex event tiers (see Section 3 ).
• Level 3 : Every Level-2 tier has at least three daughter tiers on Level 3. On these tiers, confidence scores and other additional information is given about the Level-2 annotations.

Marking and labeling relevant video segments

• There is one Level-1 tier: 'non-manuals.annotation-units'.
• Annotations on this tier have two functions :
1. To indicate the scope of the relevant annotation units;
2. To provide every annotation unit with a label.

What is the scope of an annotation unit?

• The scope of an annotation unit depends on the type of study. For example:
• If the study focuses on non-manual elements accompanying the lexical signs in a given sign language, the scope of each annotation unit may be a lexical sign .
• If the study focuses on the non-manual visual marking of questions, the scope of each annotation unit may be an entire sentence .
• What the scope of annotation units in a particular study should be determined by the researchers prior to data annotation.
• The researchers should also precisely define when, for instance, a sentence or sign begins and ends.

What kind of label does an annotation unit receive?

• Every annotation unit gets its own label.
• Labels on the 'non-manuals.annotation-units' tier can be chosen freely. Therefore, there is no Controlled Vocabulary for this tier. What kind of labels are used, is determined by the researchers.
• Labels can be used, for instance, to number annotation units, to indicate the experimental condition, to refer to the participant (number) in the video, or a combination of these.

Two examples

• Example 1 : You are annotating data for an experiment that investigates singular and plural forms of signs in Catalan Sign Language. You are annotating at the level of the manual sign.
• Possible labels could be: plural_1_P01, plural_2_P01, singular_1_P01, singular_2_P01, etc., for participant 1, plural_1_P02, plural_2_P02, singular_1_P02, singular_2_P02, etc., for participant 2.
• Example 2 : You are annotating a short monologue in Polish. The study concerns the non-manual marking of different sentence types (declaratives, interrogatives, imperatives). You are annotating at sentence level and you number each sentence. Your annotations could look as follows in this case:

Adding annotations on the 'non-manuals.annotation-units' tier

• Determine where your annotation unit should start and end by watching the video material. Select precisely that part by dragging your mouse. The selected part turns blue/purple:

• Then on the 'non-manuals.annotation-units' tier, double click on the selected part and add your text of choice. Result:

Labeling non-manual elements

• On Level-2 tiers, you annotate all non-manual elements you observe and which are relevant for the study.
• Every element is annotated on a separate tier. The element is indicated by the tier name (e.g., s.eyebrows ).
• In total, there are 21 Level-2 tiers.
• Every Level-2 tier is a daughter tier of the main tier 'non-manuals.annotation-units' (Level 1) and has three daughter tiers in turn (Level 3).
• For background on how we arrived at the proposed level-2 tier set, see Section 7 .

Simplex event tiers and complex event tiers

• We make a distinction between tiers that describe simplex events and tiers that describe complex events .
• Simplex event tiers start with an s (e.g., 's.eyelids').
• Complex event tiers start with a c (e.g., 'c.eyeblink').
• See Section 3 for a detailed explanation of the distinction between simplex events and complex events .

Hiding irrelevant tiers in ELAN

• For many studies, not all tiers in the template will be relevant.
• For instance, if your study is about headshakes and headnods, you may only need the tiers 'c.head.shakes' and 'c.head.nods'.
• Tiers irrelevant for your study can be hidden in ELAN.
• Right-click on the tier structure (left) and select Show / Hide More…
• Uncheck the irrelevant tiers.

Choosing annotation labels

• Once you have made your tier selection, you can start adding annotation labels on each tier.
• In Section 6 , we show for each Level-2 tier which annotation labels there are. With the use of images and videos, we illustrate when these labels apply.
• Below we offer some general recommendations and instructions for choosing the right label.
• Remember that you annotate based on form where possible, and based on comparison where necessary (see Section 2 ).

Annotating per video or per tier?

• We recommend that you add annotations per tier , not per video .
• That is, if you have a data set which contains multiple videos, then pick a tier first (e.g., 's.eyebrows') and add annotations on it for all videos in the data set.
• Then move on to the next tier, and add annotations on this tier for all videos. And so forth.

Annotating simplex events

• All simplex event tiers are of the tier type Time Subdivision.
• This means that annotations on these tiers must together always cover the entire scope of the annotation units on the Level-1 tier.
• An example:

• To add annotation labels to simplex event tiers, first select the annotation unit you would like to annotate (on the Level-1 tier).
• Then double click on the Level-2 tier you would like to annotate. A list of possible annotation labels (the Controlled Vocabulary ) appears.

• Select the label you would like to use and hit Enter :

• If you would like to annotate multiple labels within the same annotation unit, then right-click on the first annotation you already added and select New Annotation After (or Alt+shift+N [Windows]).

• Choose the correct label and also make sure that your annotations have the right scope. Do this by hovering your mouse over the boundary between the two annotations. Then press Alt and drag the boundary to the left or right with your mouse. This should result in something like this:

• As discussed in Section 3, the following two labels are always available on simplex event tiers:
  
  Occluded
  This label is used whenever the relevant articulator is occluded (invisible) for some reason (e.g., a hand in front of the face, or eyelids closed so that eye gaze direction cannot be seen).
  
  Complex event
  This label is used indicate that the relevant articulator is engaged in a complex event (e.g. a headshake) which is annotated on the corresponding complex event tier.

Annotating complex events

• All complex event tiers are of the tier type Included In.
• This means that annotations you add on these tiers must always fall within the scope of the annotation units on the Level-1 tier.
• An example:

• To add an annotation on a complex event tier, select the segment you wish to label.

• Click on the selected part, and select the correct label from the Controlled Vocabulary .
• If there are no other relevant events, you just leave the rest of the tier empty. It is possible that there are annotation units to which you do not need to add any annotations at all on the various complex event tiers.
• For example: as long as the person on the video does not blink their eyes, no annotation needs to be added on the 'c.eyeblink' tier.

Adding extra information

• On the Level-3 tiers, you provide extra information about the annotations you added on Level 2.
• Every Level-2 tier has at least three daughter tiers on Level 3:
1. 'confidence.about.label'
2. 'exclusion'
3. 'comments'
• Especially the 'confidence.about.label' tier is important; see the discussion about annotation strategies in Section 2 for details.
• In some cases, there is a fourth daughter tier on Level 3, namely 'sidedness'. This is only the case for Level-2 tiers for events which can be either one- or two-sided, such as shoulder and eyebrow raises.

Relation between Level-2 and Level-3 tiers

• Level-3 tiers are all Symbolic Association tiers.
• This means that, for every annotation on a Level-2 tier, you can add at most one annotation on every daughter tier at Level 3.
• For instance, you cannot give two different confidence scores to the same Level-2 annotation.

Tier: 'confidence.about.label'

• For every annotation on Level 2, you should indicate on the Level-3 tier 'confidence.about.label' how certain you are about the label you picked. There are three options:
• High : You are as good as certain that you chose the most fitting label. You chose the label purely based on form .
• Medium : You believe you chose the most fitting label, but there is a chance that a different label is more appropriate in the end. You chose the label based on a combination of form and comparison .
• Low : You are in doubt about the label you picked; it is quite probable that a different label is actually more fitting. You picked the label purely based on comparison .
• It is important to add confidence scores to your annotations. This information will be used when the data is analyzed.
• The researcher may, for instance, choose to only consider annotations with high or medium confidence scores, while leaving the annotations with low scores out of the analysis.
• The expectation is that, if two annotators both give a high confidence score to an annotation, they are also more likely to agree about the label. In other words, annotations with a high confidence score are expected to be more reliable.
• If the same annotation procedure, including the use of confidence scores, is used in multiple studies, then the results of these studies can be compared with each other more easily.

Tier: 'exclusion'

• On the 'exclusion' tier, you can indicate that the given segment should be excluded from the analysis.
• You use this tier, for instance, when it is evident that the non-manual configuration that you have annotated on Level 2 has no communicative function or a communicative function that is irrelevant for the purposes of your study.
• For instance, the speaker or signer may move their body forward because they are picking up a coffee cup, or they may turn their head because someone suddenly enters the room.
• Such movements are plausibly communicatively irrelevant.
• Still, they are annotated on the relevant Level 2 tiers, because these tiers are only concerned with form – not with function.
• The 'exclusion' tier makes it possible to exclude these annotations from further analysis.
• Another example: you may be interested in headnods as polarity/confirmation markers. But headnods may also serve other purposes. For instance, they may mark clause boundaries. On Level 2, you will annotate all headnods, no matter their function. But the 'exclusion' tier can be used to exclude boundary marking headnods from further analysis.
• The 'exclusion' tier does not have a Controlled Vocabulary : you can freely choose the text you want to add, for instance, briefly explaining why the segment at hand should be excluded from the analysis.
• You may also leave this tier empty.
  

Tier: 'comments'

• For every Level-2 annotation, there is the option to add a comment on the 'comments' tier.
• The 'comments' tier does not have a Controlled Vocabulary : you may freely choose what sort of text to add.
• You may also leave this tier empty.

Tier: 'sidedness'

• For Level-2 tiers like s.eyebrows there is a Level-3 tier 'sideness', which allows you to indicate, for instance, whether an eyebrow raise only involves one eyebrow rather than both, and whether this eyebrow is the left one or the right one.
• For simplex event tiers at Level 2, the 'sidedness' sub-tier always has a controlled vocabulary with three possible values: one-sided, one-sided.left, and one-sided.right .
• For complex event tiers at Level 2, the 'sidedness' sub-tier always has a controlled vocabulary with four possible values: one-sided, one-sided.left, one-sided.right, and alternating .
• The label alternating can be used, for instance, when the shoulders alternatingly move up and down (using the label mp.raising on the c.shoulders tier at Level 2).
• If the relevant event (e.g. a shoulder or eyebrow raise) is two-sided, then no label needs to be selected on the 'sidedness' tier. This is considered to be the default case.

As mentioned above, we have created a hands-on tutorial video explaining how to work with the annotation template. This tutorial video is available here . It may be helpful to watch this video (again) to review all the information about using the template given above.

• Simplex event tier

• Labels: neutral – raised – lowered – inner-raised – occluded – complex event

Choose the label neutral if the eyebrows are in their neutral position. In a neutral position, there is no activation of the eyebrow muscles, see the following images:

Tip : You can establish the neutral eyebrow position by looking at other frames that show the same person, earlier or later in the video. When the person relaxes their eyebrows, they are in a neutral position.

Choose the label raised if the eyebrow muscles are activated and the brows are raised in their entirety (so both the inner and outer parts), as in the following images:

Tip: Horizontal wrinkles often appear on the forehead when the eyebrows are raised. This may help, but be cautious: this does not apply to every person!

Note: If only one eyebrow is raised while the other remains neutral, annotate this using the label raised on the s.eyebrows tier, and add an annotation on the third-level tier sidedness.s.eyebrows to specify the asymmetry.

Choose the label lowered if the eyebrow muscles are activated and the eyebrows are lowered in their entirety (so both the inner and outer parts), as in the following images:

Tip: Wrinkles often appear in between the eyebrows when they are lowered.

Pay attention to the difference between inner-raised and lowered eyebrows: Both eyebrow positions often result in wrinkles between the brows, but an inner-raised position often also results in horizontal wrinkles on the forehead. This is almost never the case when the brows are lowered.

Note: If only one eyebrow is lowered while the other remains neutral, annotate this using the label lowered on the s.eyebrows tier, and add an annotation on the third-level tier sidedness.s.eyebrows to specify the asymmetry.

Choose the label inner-raised if the inner parts of the brows are raised. The outer parts of the brows are either in a neutral position or lowered, as in the following images:

Tip: When the inner part of the brows is raised, this often results in wrinkles in between the brows and horizontal wrinkles on the forehead.
Pay attention: Not every person is capable of raising only the inner parts of their brows.
Note: If only one eyebrow is inner-raised while the other remains neutral, annotate this using the label inner-raised on the s.eyebrows tier, and add an annotation on the third-level tier sidedness.s.eyebrows to specify the asymmetry.

Use the label occluded whenever the relevant articulator is occluded (invisible) for some reason (e.g., a hand or object in front of the face, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the eyebrows are raising ) which is annotated on the corresponding complex event tier.

• Complex event tier

• Labels: sg.raising, mp.raising, sg.lowering, mp.lowering, sg.inner-raising, mp.inner-raising

Use the label sg.raising if the eyebows are raised and immediately lowered again just once (not repeatedly), as in the video below:

Note: A sg.raising event is a sequential complex event, involving a sequence of two simplex eyebrow events, for example raised and neutral. These simplex events can be annotated on the s.eyebrows tier.

Note : If only one eyebrow is raising once , while the other remains neutral (as shown in the video below), annotate this using the label sg.raising on the c.eyebrows tier, and add an annotation on the third-level tier sidedness.c.eyebrows to specify the asymmetry.

ADD VIDEO

Use the label mp.raising if the eyebows are raised and immediately lowered again, at least two times in a row, as in the video below:

Note: A mp.raising event is a sequential complex event, involving a sequence of multiple simplex eyebrow events (for example raised, neutral, raised, neutral, etc.). These simplex events can be annotated on the s.eyebrows tier.

Note: If only one eyebrow is raising repeatedly , while the other remains neutral, or if the eyebrows are alternating, annotate this using the label mp.raising on the c.eyebrows tier, and add an annotation on the third-level tier sidedness.c.eyebrows to specify the asymmetry.

Use the label sg.lowering if the eyebows are lowered and immediately return again just once (not repeatedly), as in the video below:

Note : A sg.lowering event is a sequential complex event, involving a sequence of two simplex eyebrow events, for example lowered and neutral . These simplex events can be annotated on the s.eyebrows tier.

Note : If only one eyebrow is lowering once , while the other remains neutral, annotate this using the label sg.lowering on the c.eyebrows tier, and add an annotation on the third-level tier sidedness.c.eyebrows to specify the asymmetry.

Use the label mp.lowering if the eyebows are lowered and immediately return again, at least two times in a row, as in the video below:

Note : A mp.lowering event is a sequential complex event, involving a sequence of multiple simplex eyebrow events (for example lowered, neutral, lowered, neutral, etc.). These simplex events can be annotated on the s.eyebrows tier.

Note: If only one eyebrow is lowering repeatedly , while the other remains neutral, or if the eyebrows are alternating, annotate this using the label mp.lowering on the c.eyebrows tier, and add an annotation on the third-level tier sidedness.c.eyebrows to specify the asymmetry.

Use the label sg.inner-raising if the eyebows are inner-raised and immediately return again just once (not repeatedly), as in the video below:

Note: A sg.inner-raising event is a sequential complex event, involving a sequence of two simplex eyebrow events, for example inner-raised and neutral . These simplex events can be annotated on the s.eyebrows tier.

Note: If only one eyebrow is inner-raising once , while the other remains neutral, annotate this using the label sg.inner-raising on the c.eyebrows tier, and add an annotation on the third-level tier sidedness.c.eyebrows to specify the asymmetry.

Use the label mp.inner-raising if the eyebows are inner-raised and immediately return again, at least two times in a row, as in the video below:

Note: A mp.inner-raising event is a sequential complex event, involving a sequence of multiple simplex eyebrow events (for example inner-raised, neutral, inner-raised, neutral, etc.). These simplex events can be annotated on the s.eyebrows tier.

Note: If only one eyebrow is inner-raising repeatedly , while the other remains neutral, or if the eyebrows are alternating, annotate this using the label mp.inner-raising on the c.eyebrows tier, and add an annotation on the third-level tier sidedness.c.eyebrows to specify the asymmetry.

• Simplex event tier

• Labels: neutral – squinted – widened – closed – occluded – complex event

Choose the label neutral if the eyelids are relaxed and in their neutral position, as in the following images:

Tip: You can establish the neutral eyelids position by looking at other frames that show the same person, earlier or later in the video. When the person relaxes their eyelids, they are in a neutral position.
Pay attention: The position of the eyelids is independent of the eyegaze direction. If the eyegaze direction is upwards/downwards, the eyelids may seem widened/squinted. However, if the eyelids are relaxed, you should still choose the label neutral .

Choose the label squinted if the eyelids are tense and squeezed. In this state, the eyeballs are less visible, as seen in the following images:

Tip: Small wrinkles often (but not always) appear around the eyes if the eyelids are squinted.

Note: If the eyelids of only one of the eyes are squinted, while the eyelids of the other eye stay open (as shown in the image below), annotate this using the label squinted on the s.eyelids tier, and add an annotation on the third-level tier sidedness.s.eyelids to specify the asymmetry.

Choose the label widened if the eyes are wide open. The eyeballs are clearly visible in this case, as in the following images:

Choose the label closed when the eyes are closed, i.e., the eyelids are fully pressed against each other; see the following image:

Pay attention : The label closed on the s.eyelids tier marks a simplex event. This is different from the label blink on the c.eyelids tier, which concerns a complex event (the eyes closing and opening again, i.e., blinking).

Note: If the eyelids of only one of the eyes are closed, while the eyelids of the other eye stay open (as shown in the image below), annotate this using the label closed on the s.eyelids tier, and add an annotation on the third-level tier sidedness.s.eyelids to specify the asymmetry.

Use the label occluded whenever the relevant articulator is occluded (invisible) for some reason (e.g., a hand or object in front of the face, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the eyelids are blinking ) which is annotated on the corresponding complex event tier.

• Complex event tier

• Labels: sg.blinking – mp.blinking – sg.squinting – mp.squinting – sg.widening – mp.widening

Choose the label sg . blinking if the eyelids of both eyes close and immediately open again once , as in the video below:

Pay attention: Only use the label sg.blinking if the eyelids close and open once . If the eyelids close and open repeatedly, use the label mp.blinking .

Note : A sg.blinking event is a sequential complex event, involving a sequence of two simplex eyelid events, for example closed and neutral . These simplex events can be annotated on the s.eyelids tier.

Note: If the eyelids of only one of the eyes close and open once , while the eyelids of the other eye stay open (as shown in the video below), annotate this using the label sg.blinking on the c.eyelids tier, and add an annotation on the third-level tier sidedness.c.eyelids to specify the asymmetry.

Choose the label mp.blinking if both eyes close and immediately open again repeatedly , as in the video below:

Pay attention: Use the label mp.blinking only if the eyelids close and open repeatedly . If the eyelids close and open only once, use the label sg.blinking .

Note : A mp.blinking event is a sequential complex event, involving a sequence of multiple simplex eyelid events (for example closed, neutral, closed, neutral, etc). These simplex events can be annotated on the s.eyelids tier.

Note: If the eyelids of only one of the eyes close and open repeatedly , while the eyelids of the other eye stay open (as shown in the video below), or if the eyelids of both eyes are alternating, annotate this using the label mp.blinking on the c.eyelids tier, and add an annotation on the third-level tier sidedness.c.eyelids to specify the asymmetry.

Use the label sg.squinting if the eyelids are squinted and immediately return again once , as in the video below:

Pay attention: Only use the label sg.squinting if the eyelids are squinted and return again once . If the eyelids are squinted and return repeatedly, use the label mp.squinting .

Note: A sg.squinting event is a sequential complex event, involving a sequence of two simplex eyelid events, for example squinted and neutral . These simplex events can be annotated on the s.eyelids tier.

Note: If the eyelids of only one of the eyes are squinting once , while the eyelids of the other eye stay open (as shown in the video below), annotate this using the label sg.squinting on the c.eyelids tier, and add an annotation on the third-level tier sidedness.c.eyelids to specify the asymmetry.

Use the label mp.squinting if the eyelids are squinted and immediately return again repeatedly, as in the video below:

Pay attention: Only use the label mp.squinting if the eyelids are squinted and return again repeatedly . If the eyelids are squinted and return only once, use the label sg.squinting .

Note: A mp.squinting event is a sequential complex event, involving a sequence of multiple simplex eyelid events (for example squinted, neutral , squinted , neutral , etc.). These simplex events can be annotated on the s.eyelids tier.

Note: If the eyelids of only one of the eyes is squinting repeatedly , while the eyelids of the other eye stay open (as shown in the video below), or if the eyelids of both eyes are alternating, annotate this using the label mp.squinting on the c.eyelids tier, and add an annotation on the third-level tier sidedness.c.eyelids to specify the asymmetry.

Use the label sg.widening if the eyelids are widened and immediately return again once , as in the video below:

Pay attention: Only use the label sg.widening if the eyelids are widened and return again once . If the eyelids are widened and return repeatedly, use the label mp.widening .

Note: A sg.widening event is a sequential complex event, involving a sequence of two simplex eyelid events, for example widened and neutral . These simplex events can be annotated on the s.eyelids tier.

Use the label mp.widening if the eyelids are widened and immediately return again repeatedly , as in the video below:

Pay attention: Only use the label mp.widening if the eyelids are widened and return again repeatedly . If the eyelids are widened and return only once, use the label sg.widening .

Note: A mp.widening event is a sequential complex event, involving a sequence of multiple simplex eyelid events (for example widened, neutral , widened , neutral , etc.). These simplex events can be annotated on the s.eyelids tier.

• Simplex event tier
• Labels: neutral – left – right – up – down – left-up – left-down – right-up – right-down

Important general note : always determine the direction of the eyegaze relative to the individual's head, not relative to the camera. For example, when the head is turned to the left and the individual gazes straight into the camera, then use the label right , not neutral .

Use the label neutral if the eyegaze is directed straight ahead, as in the image below:

Use the label left if the eyegaze is directed to the signer's or speaker's left, as in the image below:

Use the label right if the eyegaze is directed to the signer's or speaker's right, as in the image below:

Use the label up if the eyegaze is directed straight upwards, as in the image below:

Use the label down if the eyegaze is directed straight downwards, as in the image below:

Use the label left-up if the eyegaze is directed to the signer's or speaker's upper left, as in the image below:

Use the label left-down if the eyegaze is directed to the signer's or speaker's lower left, as in the image below:

Use the label right-up if the eyegaze is directed to the signer's or speaker's upper right, as in the image below:

Use the label right-down if the eyegaze is directed to the signer's or speaker's lower right, as in the image below:

Use the label occluded if the gaze direction cannot be determined for some reason (e.g., a hand or object in front of the face, or the person steps out of frame). For the s.eyeballs tiers, this also applies to cases in which the eyelids are closed.

Closed eyelids should be annotated on the s.eyelids tier; see the label closed on that tier for more information. See the image below for an example of closed eyes.

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the eyeballs are circling) which is annotated on the corresponding complex event tier.

• Complex event tier
• Labels: sg.circling – mp.circling – sg.darting – mp.darting – sg.tracking-hand – mp.tracking-hand

Use the label sg.circling if the eyeballs make a circling movement once . This may be a full or half circle, as in the video below:

Note: This is a sequential complex event, involving a sequence of eyeballs.direction simplex events (e.g., lower-left , left, upper-left, up, upper-right, right, lower-right, down, neutral for a full circle, or left, upper-left, up, upper-right, right, neutral for a half circle, depending on where the eye circle starts and ends), which can be annotated on the s.eyeballs.direction tier.

Use the label mp.circling if the eyeballs make a circling movement (full or half circles) multiple consecutive times , as in the video below:

Note: This is a sequential complex event, involving a sequence of eyeballs.direction simplex events (e.g., repetitions of the sequence lower-left, left, upper-left, up, upper-right, right, lower-right, down, neutral for full circles, or left, upper-left, up, upper-right, right, neutral for half circles, depending on where the eye circle starts and ends), which can be annotated on the s.eyeballs.direction tier.

Use the label sg.darting if the speaker or signer’s gaze direction moves from side to side once in a short time span, as in the video below:

Note: This is a sequential complex event, involving a sequence of eyeballs.direction simplex events (e.g., left, neutral, right, neutral , depending on where the eye dart starts and ends), which can be annotated on the s.eyeballs.direction tier.

Use the label mp.darting if the speaker or signer’s gaze direction moves from side to side multiple consecutive times , in a short time span, as in the video below:

Note : This is a sequential complex event, involving a sequence of eyeballs.direction simplex events (e.g., repetitions of the sequence left, neutral, right, neutral, depending on where the eye dart starts and ends), which can be annotated on the s.eyeballs.direction tier.

Use the label sg.tracking-hand if the speaker or signer’s gaze direction follows their hand's or hands' movement once , as in the video below:

Note: This is a sequential complex event, involving a sequence of eyeballs.direction simplex events (e.g., left, neutral, right, neutral, depending on the direction in which the hand(s) and eyeballs move), which can be annotated on the s.eyeballs.direction tier.

Use the label mp.tracking-hand if the speaker or signer’s gaze direction follows their hand's or hands' movement multiple consecutive times , as in the video below:

Note: This is a sequential complex event, involving a sequence of eyeballs.direction simplex events (e.g., repetitions of the sequence left, neutral, right, neutral, depending on the direction in which the hand(s) and eyeballs move), which can be annotated on the s.eyeballs.direction tier.

• Simplex event tier

• Labels: camera – researcher – hand – referent – other-participant – space – occluded – complex event

Use the label camera if the gaze of the speaker or signer is directed into the camera or one of the camera's, as in the image below:

Tip: If there are multiple camera angles, annotate camera if the speaker or signer is looking into one of them.

Use the label researcher if the speaker or signer’s gaze is directed toward the researcher (who is not an active conversation partner), as in the following images and video:

Tip: Determine the researcher’s position in the room to decide if the speaker or signer’s gaze is directed toward them. If the researcher is standing behind the camera, the speaker or signer may look over the camera.
If there is no researcher present, this label does not apply.

Use the label hand if the signer or speaker's gaze is directed toward one or both of their hands, as in the following image:

Use the label referent if the signer or speaker's gaze is directed toward a referent localized in the signing or gesture space, as in the following image/video:

Use the label other-participant if the speaker or signer’s gaze is directed toward another research participant, as in the image below:

Tip: The other participant in the study may be the interlocutor in a conversation, for instance. Determine their position in the room to decide if the speaker or signer’s gaze is directed at them. If there are multiple other participants, and the signer is looking at one of them, choose this label. If there are no other participants, this label does not apply.

Use the label space if the speaker or signer’s gaze is directed at a point in space, as in the following images:

Tip: If the speaker or signer is not looking towards the researcher, the camera, their hand(s), a referent or another participant (if present), but into space, use the label space .

Use the label occluded if the gaze fixation cannot be determined for some reason (e.g., a hand or object in front of the face, or the person steps out of frame). For the s.eyeballs tiers, this is the case in particular if the eyelids are closed.

Closed eyelids should be annotated on the s.eyelids tier; see the label closed on that tier for more information. See the image below for an example of closed eyes.

NOTE: there is no complex event tier for eyeballs.fixation.

• Simplex event tier

• Labels: neutral – wrinkled – flared – occluded – complex event

Use the label neutral if the nose is in its neutral position, that is, if there is no activation of the muscles in the nose, as in the following images:

Tip: You can establish what the neutral position of the nose is by looking at other frames that show the same person, earlier or later in the video. If the person relaxes their nose, it is in a neutral position.

Use the label wrinkled if the muscles in the nose are activated and the nostrils are wrinkled, as in the following images:

Tip: Small wrinkles often appear around the nose if the muscles in the nose are activated and the nose is wrinkled, especially in between the eyebrows and below the eyes; the nostrils are often somewhat higher and wider.

Use the label flared if the nostrils are widened, as in the following images:

ADD IMAGES

Use the label occluded whenever the relevant articulator is occluded (invisible) for some reason (e.g., a hand or object in front of the face, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the nose is wrinkling ) which is annotated on the corresponding complex event tier.

• Complex event tier

• Labels: sg.wrinkling – mp.wrinkling – sg.flaring – mp.flaring

Use the label sg.wrinkling if the muscles in the nose are activated to wrinkle the nostrils and then relax again, within a short time span, and only in case this happens once (rather than repeatedly), as in the following videos:

Note: This is a sequential complex event. It involves a sequence of two simplex events, for example wrinkled and neutral , which can be annotated on the s.nose tier.

Use the label mp.wrinkling if the muscles in the nose are activated and then relax again multiple consecutive times , within a short time span, as in the videos below:

Note: This is a sequential complex event, involving a sequence of simplex events (for example wrinkled, neutral, wrinkled, neutral, etc.) which can be annotated on the s.nose tier.

Use the label sg.flaring if the nostrils are widened and then relax again, within a short time span, and only in case this happens once (rather than repeatedly), as in the following videos:

ADD VIDEO

Note: This is a sequential complex event. It involves a sequence of two simplex events, for example flared and neutral , which can be annotated on the s.nose tier.

Use the label mp.flaring if the nostrils are widened and then relax again multiple consecutive times , within a short time span, as in the video below:

ADD VIDEO

Note: This is a sequential complex event, involving a sequence of simplex events (for example flared, neutral, flared, neutral, etc.) which can be annotated on the s.nose tier.

This part of the guidelines is still under development.

1: Annotate based on form, not on meaning

• It is important to keep in mind that annotations should be based on form, not on meaning.
• For example: you may associate shaking of the head (c.head.shaking) with negation, but sometimes, it may have a different meaning.
• When annotating, focus on the form only. So label headshakes not only when they express negation, but also when they may seem to fulfil a different function.

2: Each tier is associated with a different direction/axis of movement

• The head can move in different directions / along different axes: it can be 'shifted' forward or backward, it can be tilted to the left or to the right, it can be rotated to the left or to the right (as in a headshake), and it can be rotated up or down (as in a head nod).
• Each head tier is associated with a different direction/axis of movement.
• For example: if the head is tilted to the left, this is annotated on the s.head.tilted.left-right tier with the label tilted-left .
• At the same time, the head may be rotated up; this is annotated on the s.head.rotated.up-down tier with the label rotated-up .
• Meanwhile, it may well be that the head is not shifted forward or backward; this is annotated on the s.head.forward-backward tier using the label neutral .

So keep a close eye on which direction should be annotated on each tier!

3: Multiple camera views may help!

It can be quite tricky to establish that e.g., the head is shifted forward based on a frontal camera view only. Of course, you may not always have control over which camera view(s) are available for your data, but in case you are collecting data yourself and you want to make annotations on (a subset of) the head tiers, consider a multiple-camera set-up so that annotators can rely on footage from more than just one angle.

Simplex event tier

Labels: neutral – forward – backward – occluded

Use the label neutral if the head is in its neutral position (so not forward or backward), as in the following images:

Tip: This occurs frequently if the person does not say or sign anything (though this is not always the case).

Use the label forward if the head is shifted forward, as in the following images:

Tip: To determine whether the head is forward, it can be helpful to view the signer from a side view, if available.

Use the label backward if the head is pulled backward, as in the following images:

Tip: To determine whether the head is backward, it can be helpful to view the signer from a side view, if available.

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the head is repeatedly shifting forward and back) which is annotated on the corresponding complex event tier.

• Simplex event tier

• Labels: neutral – tilted-left – tilted-right – occluded – complex event

Use the label neutral if the head is in its neutral position (so not tilted to the left or the right), as in the following images:

Use the label tilted-left if the head is tilted to the left (the left ear towards the left shoulder). This refers to the left from the speaker or signer’s perspective (not from the annotator’s perspective), as in the following images:

Use the label tilted-right if the head is tilted to the right (the right ear towards the right shoulder). This refers to the right from the speaker or signer’s perspective (not from the annotator’s perspective), as in the following images:

Use the label occluded whenever the relevant articulator is occluded (invisible) for some reason (e.g., a hand or object in front of the face, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the head is bobbling ) which is annotated on the corresponding complex event tier.

• Simplex event tier

• Labels: neutral – rotated-up – rotated-down – occluded – complex event

Use the label neutral if the head is in its neutral position (so not rotated up or down), as in the following images:

Use the label rotated-up if the head is rotated up. The back of the head is directed towards the neck and the nose sticks up, as in the following images:

Note: The label rotated-up marks a simplex event. This simplex event may be part of a complex nodding event. Such events can be annotated on the c.head.nods tier.

Use the label rotated-dow n if the head is rotated down. The chin is directed towards the throat and the nose points downwards, as in the following images and video:

Note : The label rotated-down marks a simplex event. This simplex event may be part of a complex nodding event. Such complex events can be annotated on the c.head.nods tier.

Use the label occluded whenever the relevant articulator is occluded (invisible) for some reason (e.g., a hand or object in front of the face, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the head is nodding ) which is annotated on the corresponding complex event tier.

• Simplex tier: Head rotated left/right

• Labels: neutral – rotated-left – rotated-right – occluded – complex event

Use the label neutral if the head is in its neutral position (so not rotated to the left or right), as in the following images:

Use the label rotated-left if the head is rotated to the left. This refers to the left from the speaker or signer’s perspective (not from the annotator’s perspective), as in the following images:

Use the label rotated-right if the head is rotated to the right. This refers to the right from the speaker or signer’s perspective (not from the annotator’s perspective), as in the following images:

Use the label occluded whenever the relevant articulator is occluded (invisible) for some reason (e.g., a hand or object in front of the face, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the head is shaking ) which is annotated on the corresponding complex event tier.

• Simplex event tier

• Labels: neutral – circle-left – circle-right – occluded – complex event

Use the label neutral if the head is in its neutral position (so not making a circling motion), as in the following images:

Use the label circle-left if the head is making a circling motion that starts toward the left (the left ear towards the left shoulder). This refers to the left from the speaker or signer’s perspective (not from the annotator’s perspective), as in the following video:

Use the label circle-right if the head is making a circling motion that starts toward the right (the right ear towards the right shoulder). This refers to the right from the speaker or signer’s perspective (not from the annotator’s perspective), as in the following video:

Use the label occluded whenever the relevant articulator is occluded (invisible) for some reason (e.g., a hand or object in front of the face, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the head is circling ) which is annotated on the corresponding complex event tier.

• Simplex event tier

• Labels: neutral – swing-left – swing-right – occluded – complex event

Use the label neutral if the head is in its neutral position (so not making a swinging motion), as in the following images:

Use the label swing-left if the head is making a swinging motion (with an arc) to the left (the left ear towards the left shoulder). This refers to the left from the speaker or signer’s perspective (not from the annotator’s perspective), as in the following videos:

Use the label swing-left if the head is making a swinging motion (with an arc) to the right (the right ear towards the right shoulder). This refers to the right from the speaker or signer’s perspective (not from the annotator’s perspective), as in the following images:

Use the label occluded whenever the relevant articulator is occluded (invisible) for some reason (e.g., a hand or object in front of the face, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the head is swinging ) which is annotated on the corresponding complex event tier.

• Simplex event tier

• Labels: neutral – slide-left – slide-right – occluded – complex event

Use the label neutral if the head is in its neutral position (so not making a sliding motion), as in the following images:

Use the label slide-left if the head slides toward the left (the left ear towards the left shoulder). This refers to the left from the speaker or signer’s perspective (not from the annotator’s perspective). The head is kept upright and does not tilt sideways, as in the following images:

Use the label slide-right if the head slides toward the right (the right ear towards the right shoulder). This refers to the right from the speaker or signer’s perspective (not from the annotator’s perspective). The head is kept upright and does not tilt sideways, as in the following images:

Use the label occluded whenever the relevant articulator is occluded (invisible) for some reason (e.g., a hand or object in front of the face, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the head is circling ) which is annotated on the corresponding complex event tier.

• Complex event tier

• Labels: sg.forward – mp.forward – sg.backward – mp.backward

Use the label sg.forward if the head shifts forward and back again just once , as in the following video’s:

Note: Only use the label sg.forward if the head shifts forward and back just once ; if it shifts forward and back repeatedly use the label mp.forward .

Use the label mp.forward if the head shifts forward and back again multiple times, as in the following video’s:

Note: Only use the label mp.forward if the head shifts forward and back again repeatedly, otherwise use the label sg.forward .

Use the label sg.backward if the head shifts backward and back again just once , as in the following video’s:

Note: Only use the label sg.backward if the head shifts backward and back just once; if it shifts backward and back repeatedly use the label mp.backward .

Use the label mp.backward if the head shifts backward and back multiple times, as in the video below:

Note: Only use the label mp.backward if the head shifts backward and back again repeatedly, otherwise use the label sg.backward .

• Complex event tier

• Labels: sg.nodding – mp.nodding

Use the label sg.nodding if the head makes a nodding movement (e.g., down and immediately back up) just once , as in the following videos:

Note: Only use the label sg.nodding if the head makes a nodding movement just once ; if it nods repeatedly use the label mp.nodding .

Use the label mp.nodding if the head makes multiple nodding movements (down and back up), as in the video below:

Note: Only use the label mp.nodding if the head nods repeatedly, otherwise use the label sg.nodding .

• Complex event tier

• Labels: sg.shaking – mp.shaking

Use the label sg.shaking if the head moves once to the left or right and immediately back again, as in the video below:

Note : Only use the label sg.shaking if the head moves to the left or right and back just once . If the head moves back and forth multiple times use the label mp.shaking .

Use the label mp.shaking if the head moves back and forth to the right or to the left multiple times, as in the video below:

Note: Only use the label mp.shaking if the head moves back and forth multiple times ; otherwise use the label sg.shaking .

• Complex event tier

• Labels: sg.bobbling – mp.bobbling

Use the label sg.bobbling if the head tilts to the left or right and immediately returns to a neutral position just once , as in the videos below:

Note : Use the label sg.bobbling only if the movement is made just once ; if it is made multiple times use the label mp.bobbling .

Use the label mp.bobbling if the head repeatedly tilts to the left or right and immediately returns to an upright position, as in the video:

Note : Use the label mp.bobbling only if the movement is made repeatedly ; if it is made just once use the label sg.bobbling .

• Complex event tier

• Labels: mp.circling

Use the label mp.circling if the head repeatedly makes a circling motion left or right, as in the video:

Note : Because the start and end point of a (non-linear) simplex circling event are the same, the label sg.circling is not applicable (there is no return to the starting position). The c.head.circling tier therefore only includes the label mp.circling . Use this label only if the circling motion of the head is made repeatedly . In case there is only one circling event, make an annotation on the s.head.circle tier instead.

• Complex event tier

• Labels: sg.swinging – mp.swinging

Use the label sg.swinging if the head swings (with an arc) to the left or right and immediately returns to a neutral position just once , as in the video below:

Note : Use the label sg.swinging only if the movement is made just once ; if it is made multiple times use the label mp.swinging .

Use the label mp.swinging if the head repeatedly swings (with an arc) to the left or right and immediately returns to an upright position, as in the video:

Note : Use the label mp.swinging only if the movement is made repeatedly ; if it is made just once use the label sg.swinging .

• Complex event tier

• Labels: sg.sliding – mp.sliding

Use the label sg.sliding if the head slides to the left or right (the head is kept upright and does not tilt sideways) and immediately returns to a neutral position just once , as in the video below:

Note : Use the label sg.sliding only if the movement is made just once ; if it is made multiple times use the label mp.sliding .

Use the label mp.sliding if the head repeatedly slides to the left or right (the head is kept upright and does not tilt sideways) and immediately returns to a neutral position, as in the video:

Note : Use the label mp.sliding only if the movement is made repeatedly ; if it is made just once use the label sg.sliding .

• Simplex event tier

• Labels: neutral – raised – opposed – occluded – complex event

Use the label neutral if the shoulders are in the typical rested position for the subject, as in the images below:

Tip : This configuration of the shoulders typically occurs when the person is not saying or signing anything (although this is not always the case).

Use the label raised if the shoulders are higher than in the neutral position, as in the images and video below:

Pay attention: The label raised on the s.shoulders tier marks a simplex event. This is different from the labels shrug and shrugging on the c.shoulders tier, which mark complex events.

Note : If the shoulders are raised on only one side, while the shoulder on the other side stays put (as seen in the image below), annotate this using the label raised on the s.shoulders.raised tier, and add an annotation on the third-level tier sidedness.s.shoulders to specify the asymmetry.

Use the label opposed if one shoulder is held higher than the neutral position while the other shoulder is held lower than the neutral position, as in the image below:

Note: If it is relevant which side is higher and which side is lower, this can be annotated in the comments.

Use the label occluded whenever the relevant articulator is not directly visible for some reason (e.g., a hand or object in front of the body, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the shoulders are shrugging ) which is annotated on the corresponding complex event tier.

Simplex event tier

Labels: neutral – rotated-forward – rotated-backward – opposed – occluded – complex event

Use the label neutral if the shoulders are in the typical rested position for that subject, and the shoulders are not rotated forward or backward, as in the following image:

Use the label rotated-forward if the shoulders complete a rotating movement at which the shoulders move forward at the highest point of the rotation. By forward we mean forward for the person in the video (so not for the annotator), as illustrated in the video.

Note : If the shoulders are rotated forward on only one side, while the shoulder on the other side stays put (as seen in the video below), annotate this using the label rotated-forward on the s.shoulders.rotated.forward - backward tier, and add an annotation on the third-level tier sidedness.s.shoulders to specify the asymmetry.

Use the label rotated-backward if the shoulders complete a rotating movement at which the shoulders move backwards at the highest point of the rotation. By backward we mean backward for the person in the video (so not for the annotator), as illustrated in the video.

Note : If the shoulders are rotated backward on only one side, while the shoulder on the other side stays put (as seen in the video below), annotate this using the label rotated-backward on the s.shoulders.rotated.forward - backward tier, and add an annotation on the third-level tier sidedness.s.shoulders to specify the asymmetry.

Use the label opposed if there is a shoulder roll where the left and right shoulders are simultaneously rolling in opposite directions.

If it is relevant which side is rotating forward and which side is rotating backwards, this can be annotated in the comments.

Use the label occluded whenever the relevant articulator is not directly visible for some reason (e.g., an object in front of the body, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the torso is rotating left and back) which is annotated on the corresponding complex event tier.

Simplex event tier

Labels: neutral – forward – backward – occluded – complex event

Use the label neutral if the shoulders are in the typical rested position for that subject, and the shoulders are not bent forward or backward, as in the images below:

Use the label forward if the shoulders are shifted forward with respect to the neutral position, as in the following images:

Tip : To determine whether the shoulders are shifted forward, it helps to view the person from the side. So if a side view video is available, please use it.

Note : If the shoulders are in a forward position on only one side, while the shoulder on the other side stays put (as seen in the image below), annotate this using the label forward on the s.shoulders.forward - backward tier, and add an annotation on the third-level tier sidedness.s.shoulders to specify the asymmetry.

Use the label backward if the shoulders are shifted backward with respect to the neutral position, as in the following image:

Tip : To determine whether the shoulders are shifted backward, it helps to view the person from the side. So if a side view video is available, please use it.

Note : If the shoulders are in a backward position on only one side, while the shoulder on the other side stays put (as seen in the image below), annotate this using the label backward on the s.shoulders.forward - backward tier, and add an annotation on the third-level tier sidedness.s.shoulders to specify the asymmetry.

Use the label occluded whenever the relevant articulator is not directly visible for some reason (e.g., an object in front of the body, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the shoulders are shifted forward and back) which is annotated on the corresponding complex event tier.

• Complex event tier

• Labels: sg.shrugging – mp.shrugging, alternating

Use the label sg.shrugging if the shoulders move up and come down again just once , as in the videos below:

Note: this movement is relative to the position that the shoulders were in before. It is possible to shrug while the shoulders are in a raised position on the s.shoulders.raised tier.

Note : Only use the label sg.shrugging if the shoulders move up and back down just once , otherwise use the label mp.shrugging on the c.shoulders.shrugging tier.

Note : An sg.shrugging event is a sequential complex event, involving a sequence of simplex shoulder events (e.g., raised, neutral). These simplex events can be annotated on the s.shoulders.raised tier.

Note : If the shoulders are raised and move back down on only one side, while the shoulder on the other side stays put (as seen in the video below), annotate this using the label sg.shrugging on the c.shoulders.shrugging tier, and add an annotation on the third-level tier sidedness.s.shoulders to specify the asymmetry.

Use the label mp.shrugging if the shoulders repeatedly move up and down, as in these videos:

Pay attention : Only use the label mp.shrugging if the shoulders move up and come down repeatedly . If this happens only once, use the label sg.shrugging .

Note : The label mp.shrugging marks a sequential complex event, involving a sequence of multiple simplex events (e.g., raised, neutral, raised, neutral, etc.). These simplex events can be annotated on the s.shoulders.raised tier.

Note : If the shoulders are raised and move back down repeatedly on only one side, while the shoulder on the other side stays put (as seen in the video below), annotate this using the label mp.shrugging on the c.shoulders.shrugging tier, and add an annotation on the third-level tier sidedness.s.shoulders to specify the asymmetry.

Use the label alternating if the shoulders alternate which side moves up and which side moves down, as in this video:

Complex event tier

Labels: mp.rotating-forward – mp.rotating-backward, mp.opposed

Use the label mp.rotating-forward if the shoulders rotate forward and back again multiple times, as in the following video:

Note : Only use the label mp.rotating-forward if the shoulders rotate forwards repeatedly , otherwise use the label rotated-forward on the s.shoulders.rotated.forward - backward tier.

Note : An mp.rotating-forward event is a sequential complex event, involving a sequence of multiple simplex shoulder events (e.g., rotating-forward, rotating-forward, etc.). These simplex events can be annotated on the s.shoulders.rotating.forward - backward tier.

Note : If the shoulders rotates forward on only one side, while the shoulder on the other side stays put (as seen in the video below), annotate this using the label rotating-forward on the c.shoulders.rotating.forward - backward tier, and add an annotation on the third-level tier sidedness.s.shoulders to specify the asymmetry.

Use the label mp.rotating-backward if the shoulders rotate backward and back multiple times, as in the video below:

Note : Only use the label mp.rotating-backward if the shoulders rotate backward repeatedly , otherwise use the label rotated-backward on the s.shoulders.rotated.forward - backward tier.

Note : An mp.rotating-backward event is a sequential complex event, involving a sequence of multiple simplex shoulder events (e.g., rotating-backward, rotating-backward, etc.). These simplex events can be annotated on the s.shoulders.rotating.forward - backward tier.

Note : If the shoulders rotates backward on only one side, while the shoulder on the other side stays put (as seen in the video below), annotate this using the label rotating-backward on the c.shoulders.rotating.forward - backward tier, and add an annotation on the third-level tier sidedness.s.shoulders to specify the asymmetry.

Use the label mp.opposed if one shoulder rotates forward while the other shoulder rotates backwards multiple times, as in the following video:

Note : Only use the label mp.opposed if the shoulders rotate in an opposite direction repeatedly , otherwise use the label opposed on the s.shoulders.rotated.forward - backward tier.

Note : An mp.opposed event is a sequential complex event, involving a sequence of multiple simplex shoulder events (e.g., opposed, opposed, etc.). These simplex events can be annotated on the s.shoulders.rotating.forward - backward tier.

Complex event tier

Labels: sg.forward – mp.forward – sg.backward – mp.backward

Use the label sg.forward if the shoulders shift forward and back again just once , as in the following video:

Note : Only use the label sg.forward if the shoulders shift forward and back just once ; if they shift forward and back repeatedly use the label mp.forward .

Note : A sg.forward event is a sequential complex event, involving a sequence of two simplex shoulder events, e.g., forward and neutral. These simplex events can be annotated on the s.shoulders.forward - backward tier.

Note : If the shoulders move forward and back on only one side, while the shoulder on the other side stays put (as shown in the video below), annotate this using the label sg.forward on the c.shoulders.forward - backward tier, and add an annotation on the third-level tier sidedness.c.shoulders to specify the asymmetry.

Use the label mp.forward if the shoulders shift forward and back again multiple times, as in the following video:

Note : Only use the label mp.forward if the shoulders shift forward and back again repeatedly , otherwise use the label sg.forward .

Note : An mp.forward event is a sequential complex event, involving a sequence of multiple simplex shoulder events (e.g., forward, neutral, forward, neutral, etc.). These simplex events can be annotated on the s.shoulders.forward - backward tier.

Note : If the shoulders move forward and back on only one side, while the shoulder on the other side stays put, annotate this using the label mp.forward on the c.shoulders.forward - backward tier, and add an annotation on the third-level tier sidedness.c.shoulders to specify the asymmetry.

Use the label sg.backward if the shoulders shift backward and back again just once , as in the following video:

Note : Only use the label sg.backward if the shoulders shift backward and back just once ; if they shift backward and back repeatedly use the label mp.backward .

Note : An mp.backward event is a sequential complex event, involving a sequence of multiple simplex shoulder events, e.g., backward and neutral. These simplex events can be annotated on the s.shoulders.forward-backward tier.

Note : If the shoulders move backward and back on only one side, while the shoulder on the other side stays put (as shown in the video below), annotate this using the label sg.backward on the c.shoulders.forward - backward tier, and add an annotation on the third-level tier sidedness.c.shoulders to specify the asymmetry.

Use the label mp.backward if the shoulders shift backward and back multiple times, as in the video below:

Note : Only use the label mp.backward if the shoulders shift backward and back again repeatedly , otherwise use the label sg.backward .

Note : An mp.backward event is a sequential complex event, involving a sequence of multiple simplex shoulder events (e.g., backward, neutral, backward, neutral, etc.). These simplex events can be annotated on the s.shoulders.forward - backward tier.

Note : If the shoulders move backward and back on only one side, while the shoulder on the other side stays put, annotate this using the label mp.backward on the c.shoulders.forward - backward tier, and add an annotation on the third-level tier sidedness.c.shoulders to specify the asymmetry.

Different tiers for different directions/axes

• There are four main tier types for the torso. Three of these pertain to different directions/axes, and one of these pertains to a non-linear movement.
• For example: if the torso is tilted to the left, this is annotated with the label left on the tier s.torso.tilted.left-right.
• It is possible that the torso is otherwise in a neutral position; for instance, if it is not bent forward or backward, but only tilted to the left, then you use the label neutral on the tier s.torso.forward-backward.

So pay attention to the specific direction/axis that is relevant for a given tier!

Difference between Torso movements and Chest movements

• There are tiers describing movements of the torso and tiers describing movements of the chest.
• Distinguish between these by examining the spine: typically, the entire spine is straight for torso movements.
• For example, if the person is leaning their entire torso forwards (keeping the spine relatively straight), we would annotate forward on the s.torso.forward-backward tier.

Simplex event tier

Labels: neutral – forward – backward – occluded – complex event

Use the label neutral if the torso is in the typical rested position for that subject, and the torso is not bent forward or backward, as in the images below:

Tip: This often occurs when the person is not signing or speaking (although this is not always the case).

Use the label forward if the torso is bent forward with respect to the neutral position, as in the following image and video:

Tip : To determine whether the torso is bent forward, it helps to view the person from the side. So if a side view video is available, please use it.

Use the label backward if the torso is bent backward with respect to the neutral position, as in the following image and video:

Tip : To determine whether the torso is bent backward, it helps to view the person from the side. So if a side view video is available, please use it.

Use the label occluded whenever the relevant articulator is not directly visible for some reason (e.g., an object in front of the body, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the torso is moving forward and back) which is annotated on the corresponding complex event tier.

• Simplex event tier

• Labels: neutral – tilted-left – tilted-right – occluded – complex event

Use the label neutral if the torso is in its neutral position (so not tilted to the left or the right), as in the following images:

Use the label tilted-left if the torso is tilted to the left. This refers to the left from the speaker or signer’s perspective (not from the annotator’s perspective), as in the following images:

Use the label tilted-right if the torso is tilted to the right. This refers to the right from the speaker or signer’s perspective (not from the annotator’s perspective), as in the following images:

Use the label occluded whenever the relevant articulator is occluded (invisible) for some reason (e.g., an object in front of the body, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the torso is tilting ) which is annotated on the corresponding complex event tier.

Simplex event tier

Labels: neutral – rotated-left – rotated-right – occluded – complex event

Use the label neutral if the torso is in the orientation of the typical rested position for that subject, and the torso is not rotated to the left or to the right, as in the following images:

Tip: This often occurs when the person is not signing or speaking (although this is not always the case).

Use the label rotated-left if the torso is rotated to the left with respect to the neutral position. By left we mean left for the person in the video (so not for the annotator), as illustrated in the image.

Tip : In sign language, the torso is often rotated to indicate a role-shift.

Use the label rotated-right if the torso is rotated to the right with respect to the neutral position. By right we mean right for the person in the video (so not for the annotator), as illustrated in the image.

Tip : In sign language, the torso is often rotated to indicate a role-shift.

Use the label occluded whenever the relevant articulator is not directly visible for some reason (e.g., an object in front of the body, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the torso is rotating left and back) which is annotated on the corresponding complex event tier.

• Simplex event tier

• Labels: neutral – circle-left – circle-right – occluded – complex event

Use the label neutral if the torso is in the typical rested position for that subject (so not making a circling motion), as in the following images:

Use the label circle-left if the torso is making a circling motion that starts toward the left. This refers to the left from the speaker or signer’s perspective (not from the annotator’s perspective), as in the following video:

Use the label circle-right if the torso is making a circling motion that starts toward the right. This refers to the right from the speaker or signer’s perspective (not from the annotator’s perspective).

Use the label occluded whenever the relevant articulator is not directly visible for some reason (e.g., an object in front of the body, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the torso is circling ) which is annotated on the corresponding complex event tier.

Complex event tier

Labels: sg.forward – mp.forward – sg.backward – mp.backward

Use the label sg.forward if the torso shifts forward and back again just once , as in the following video:

Note : Only use the label sg.forward if the torso shifts forward and back just once ; if it shifts forward and back repeatedly use the label mp.forward .

Note : The label sg.forward marks a sequential complex event, involving a sequence of simplex events (e.g., forward, neutral). These simplex events can be annotated on the s.torso.forward-backward tier.

Use the label mp.forward if the torso shifts forward and back again multiple times, as in the following video:

Note : Only use the label mp.forward if the torso shifts forward and back again repeatedly , otherwise use the label sg.forward .

Note : The label mp.forward marks a sequential complex event, involving a sequence of multiple simplex events (e.g., forward, neutral, forward, neutral, etc.). These simplex events can be annotated on the s.torso.forward-backward tier.

Use the label sg.backward if the torso shifts backward and back again just once , as in the following video:

Note : Only use the label sg.backward if the torso shifts backward and back just once ; if it shifts backward and back repeatedly use the label mp.backward .

Note : The label sg.backward marks a sequential complex event, involving a sequence of simplex events (e.g., backward, neutral). These simplex events can be annotated on the s.torso.forward-backward tier.

Use the label mp.backward if the torso shifts backward and back multiple times, as in the video below:

Note : Only use the label mp.backward if the torso shifts backward and back again repeatedly , otherwise use the label sg.backward .

Note : The label mp.backward marks a sequential complex event, involving a sequence of multiple simplex events (e.g., backward, neutral, backward, neutral, etc.). These simplex events can be annotated on the s.torso.forward-backward tier.

Complex event tier

Labels: sg.tilting-left – mp.tillting-left – sg.tilting-right – mp.tilting-right

Use the label sg.tilting-left if the torso tilts left and back again just once , as in the following video:

Note : Only use the label sg.tilting-left if the torso tilts left and back just once ; if it tilts left and back repeatedly use the label mp.tilting-left .

Note : The label sg.tilting-left marks a sequential complex event, involving a sequence of simplex events (e.g., tilting-left, neutral). These simplex events can be annotated on the s.torso.tilting.left - right tier.

Use the label mp.tilting-left if the torso tilts left and back again multiple times, as in the following video:

Note : Only use the label mp.tilting-left if the torso tilts left and back again repeatedly , otherwise use the label sg.tilting-left .

Note : The label mp.tilting-left marks a sequential complex event, involving a sequence of simplex events (e.g., tilting-left, neutral, tilting-left, neutral, etc.). These simplex events can be annotated on the s.torso.tilted.left - right tier.

Use the label sg.tilting-right if the torso tilts right and back again just once , as in the following video:

Note : Only use the label sg.tilting-right if the torso tilts right and back just once ; if it tilts right and back repeatedly use the label mp.tilting-right .

Note : The label sg.tilting-right marks a sequential complex event, involving a sequence of simplex events (e.g., tilting-right, neutral). These simplex events can be annotated on the s.torso.tilted.left - right tier.

Use the label mp.tilting-right if the torso tilts right and back multiple times, as in the video below:

Note : Only use the label mp.tilting-right if the torso tilts right and back again repeatedly, otherwise use the label sg.tilting-right .

Note : The label mp.tilting-right marks a sequential complex event, involving a sequence of simplex events (e.g., tilting-right, neutral, tilting-right, neutral, etc.). These simplex events can be annotated on the s.torso.tilted.left - right tier.

Complex event tier

Labels: sg.rotating-left – mp.rotating-left – sg.rotating-right – mp.rotating-right

Use the label sg.rotating-left if the torso rotates left and back again just once , as in the following video:

Note : Only use the label sg.rotating-left if the torso rotates left and back just once ; if it rotates left and back repeatedly use the label mp.rotating-left .

Note : The label sg.rotating-left marks a sequential complex event, involving a sequence of simplex events (e.g., rotating-left, neutral). These simplex events can be annotated on the s.torso.rotated.left - right tier.

Use the label mp.rotating-left if the torso rotates left and back again multiple times, as in the following video’s:

Note : Only use the label mp.rotating-left if the torso rotates left and back again repeatedly , otherwise use the label sg.rotating-left .

Note : The label mp.rotating-left marks a sequential complex event, involving a sequence of simplex events (e.g., rotating-left, neutral, rotating-left, neutral, etc.). These simplex events can be annotated on the s.torso.rotated.left - right tier.

Use the label sg.rotating-right if the torso rotates right and back again just once , as in the following video:

Note : Only use the label sg.rotating-right if the torso rotates right and back just once ; if it rotates right and back repeatedly use the label mp.rotating-right .

Note : The label sg.rotating-right marks a sequential complex event, involving a sequence of simplex events (e.g., rotating-right, neutral). These simplex events can be annotated on the s.torso.rotated.left - right tier.

Use the label mp.rotating-right if the torso rotates right and back multiple times, as in the video below:

Note : Only use the label mp.rotating-right if the torso rotates right and back again repeatedly , otherwise use the label sg.rotating-right .

Note : The label mp.rotating-right marks a sequential complex event, involving a sequence of simplex events (e.g., rotating-right, neutral, rotating-right, neutral, etc.). These simplex events can be annotated on the s.torso.rotated.left - right tier.

• Complex event tier

• Labels: mp.circling

Use the label mp.circling if the torso repeatedly makes a circling motion left or right, as in the video below:

Note : Because the start and end point of a (non-linear) simplex circling event are the same, the label sg.circling is not applicable (there is no return to the starting position). The c.torso.circling tier therefore only includes the label mp.circling . Use this label only if the circling motion of the torso is made repeatedly . In case there is only one circling event, make an annotation on the s.torso.circle tier instead.

Difference between Torso movements and Chest movements

• There are tiers describing movements of the torso and tiers describing movements of the chest.
• Distinguish between these by examining the spine: typically, the spine is curved for chest movements. left on the tier s.chest.tilted.left-right.
• For example, if the person is puffing their chest forwards (resulting in a curved spine), we would annotate forward on the s.chest.forward-backward tier.

So pay attention to the specific direction/axis that is relevant for a given tier!

Simplex event tier

Labels: neutral – forward – backward – occluded – complex event

Use the label neutral if the chest is in the typical rested position for that subject, and the chest is not bent forward or backward, as in the images below:

Use the label forward if the chest is bent forward with respect to the neutral position. By forward we mean forward from the perspective of the subject, so the chest is puffed outwards, as in the images below:

Use the label backward if the torso is bent backward with respect to the neutral position. By backward we mean backward from the perspective of the subject, so the chest is hunched inwards, as in the images below:

Use the label occluded whenever the relevant articulator is not directly visible for some reason (e.g., an object in front of the body, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the chest is moving forward and back) which is annotated on the corresponding complex event tier.

Simplex event tier

Labels: neutral – shifted-left – shifted-right – occluded – complex event

Use the label neutral if the chest is in the typical rested position for that subject, and the chest is not shifted to the left or to the right, as in the following images:

Use the label shifted-left if the chest is shifted to the left with respect to the neutral position. This typically involves the shoulders maintaining an equal height. By left we mean left for the person in the video (so not for the annotator), as in the following images:

Use the label shifted-right if the chest is shifted to the right with respect to the neutral position. This typically involves the shoulders maintaining an equal height. By right we mean right for the person in the video (so not for the annotator), as in the following images:

Use the label occluded whenever the relevant articulator is not directly visible for some reason (e.g., an object in front of the body, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the chest is shifting left and back) which is annotated on the corresponding complex event tier.

• Simplex event tier

• Labels: neutral – circle-left – circle-right – occluded – complex event

Use the label neutral if the chest is in the typical rested position for that subject (so not making a circling motion), as in the following image:

Use the label circle-left if the chest is making a circling motion that starts toward the left. This refers to the left from the speaker or signer’s perspective (not from the annotator’s perspective), as in the following video:

Use the label circle-right if the chest is making a circling motion that starts toward the right. This refers to the right from the speaker or signer’s perspective (not from the annotator’s perspective).

Use the label occluded whenever the relevant articulator is not directly visible for some reason (e.g., an object in front of the body, or the person steps out of frame).

This label is used to indicate that the relevant articulator is engaged in a complex event (e.g. the chest is circling ) which is annotated on the corresponding complex event tier.

Complex event tier

Labels: sg.forward – mp.forward – sg.backward – mp.backward

Use the label sg.forward if the chest shifts forward and back again just once , as in the following video:

Note : Only use the label sg.forward if the chest shifts forward and back just once ; if it shifts forward and back repeatedly use the label mp.forward .

Note : The label sg.forward marks a sequential complex event, involving a sequence of simplex events (e.g., forward, neutral). These simplex events can be annotated on the s.chest.forward - backward tier.

Use the label mp.forward if the chest shifts forward and back again multiple times, as in the following video:

Note : Only use the label mp.forward if the chest shifts forward and back again repeatedly , otherwise use the label sg.forward .

Note : The label mp.forward marks a sequential complex event, involving a sequence of simplex events (e.g., forward, neutral, forward, neutral, etc.). These simplex events can be annotated on the s.chest.forward - backward tier.

Use the label sg.backward if the chest shifts backward and back again just once , as in the following video:

Note : Only use the label sg.backward if the chest shifts backward and back just once ; if it shifts backward and back repeatedly use the label mp.backward .

Note : The label sg.backward marks a sequential complex event, involving a sequence of simplex events (e.g., backward, neutral). These simplex events can be annotated on the s.chest.forward - backward tier.

Use the label mp.backward if the chest shifts backward and back multiple times, as in the video below:

Note : Only use the label mp.backward if the chest shifts backward and back again repeatedly , otherwise use the label sg.backward .

Note : The label mp.backward marks a sequential complex event, involving a sequence of simplex events (e.g., backward, neutral, backward, neutral, etc.). These simplex events can be annotated on the s.chest.forward - backward tier.

Complex event tier

Labels: sg.shifting-left – mp.shifting-left – sg.shifting-right – mp.shifting-right

Use the label sg.shifting-left if the chest shifts left and back again just once , as in the following video:

Note : Only use the label sg.shifting-left if the chest shifts left and back just once ; if it shifts left and back repeatedly use the label mp.shifting-left .

Note : The label sg.shifting-left marks a sequential complex event, involving a sequence of simplex events (e.g., shifted-left, neutral). These simplex events can be annotated on the s.chest.shifted.left - right tier.

Use the label mp.shifting-left if the chest shifts left and back again multiple times, as in the following video:

Note : Only use the label mp.shifitng-left if the ches shifts left and back again repeatedly , otherwise use the label sg.shifting-left .

Note : The label mp.shifting-left marks a sequential complex event, involving a sequence of simplex events (e.g., shifted-left, neutral, shifted-left, neutral, etc.). These simplex events can be annotated on the s.chest.shifted.left - right tier.

Use the label sg.shifting-right if the chest shifts right and back again just once , as in the following video:

Note : Only use the label sg.shifting-right if the chest shifts right and back just once ; if it shifts right and back repeatedly use the label mp.shifting-right .

Note : The label sg.shifting-right marks a sequential complex event, involving a sequence of simplex events (e.g., shifted-right, neutral). These simplex events can be annotated on the s.chest.shifted.left - right tier.

Use the label mp.shifting-right if the chest shifts right and back multiple times, as in the video below:

Note : Only use the label mp.shifting-right if the chest shifts right and back again repeatedly , otherwise use the label sg.shifting-right .

Note : The label mp.shifting-right marks a sequential complex event, involving a sequence of simplex events (e.g., shifted-right, neutral, shifted-right, neutral, etc.). These simplex events can be annotated on the s.chest.shifted.left - right tier.

• Complex event tier

• Labels: mp.circling

Use the label mp.circling if the chest repeatedly makes a circling motion left or right, as in the video below:

Note : Because the start and end point of a (non-linear) simplex circling event are the same, the label sg.circling is not applicable (there is no return to the starting position). The c.chest.circling tier therefore only includes the label mp.circling . Use this label only if the circling motion of the chest is made repeatedly . In case there is only one circling event, make an annotation on the s.chest.circle tier instead.

The inventory of level-2 tiers and the annotation labels available on each tier were determined based on the following five principles:

• One body part per tier – Every level-2 tier describes events involving one specific body part / articulator, e.g., the eyelids, the nose, or the head.
• Simplex and complex events on separate tiers – Every level-2 tier either has a set of labels describing simplex events or a set of labels describing complex events; simplex and complex events are never mixed on a single tier.
• Labels are exclusive – The labels within one tier should describe events which cannot co-occur. Events that can co-occur should be labeled on different tiers (e.g., ‘head forward’ and ‘head rotated down’ have to be annotated on separate tiers, because it is possible for someone’s head to simultaneously be in a forward position and rotated down).
• Labels are exhaustive – The labels within one tier must exhaustively cover the set of possibilities for the relevant non-manual feature. On the 's.torso.forward-backward' tier, for instance, this includes ‘torso forward, ‘torso backward’ and ‘neutral’.
• Labels are contrastive – The labels within one tier have to be sufficiently contrastive. On the 's.torso.forward-backward' tier, for instance, one could in theory make a distinction between ‘torso slightly forward’ and ‘torso substantially forward’, but this difference is not contrastive enough, which increases the chance that annotators will often disagree, thus making annotations less consistent.

In case you are analyzing already existing data, you are dependent on the material that is available. But if you are going to collect new data, then we have the following recommendations for video recording, which will make the annotation process easier and more reliable:

• Ask participants not to wear glasses, hats, or other accessories that may obscure part of the face, if possible.
• Use multiple cameras to obtain at least one frontal and one side view. This facilitates the annotation of, in particular, the position of the head and torso.
• Use the frontal recording to also prepare a version of the video which zooms in on the face. Use this video to annotate the facial tiers (eyes, nose, mouth). For this it is very important to record at high resolution (preferably at least 4K). That way, even if you zoom in on the face, the image quality will remain high.