Skip to content

An R package for analyzing scanpath patterns in eye movements

Notifications You must be signed in to change notification settings

tmalsburg/scanpath

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Scanpath – An R Package for Analyzing Scanpaths

https://zenodo.org/badge/doi/10.5281/zenodo.31800.svg

An R package for analyzing scanpaths in eye movement data. The package includes a simple toy dataset and example code. Consult von der Malsburg & Vasishth (2011) for the details of this analysis method. The manual of the package can be found here and a PDF-version of this page here.

News

[2021-07-09]
Fixed a bug triggered when factors were used as trial identifiers.
[2018-02-22]
Added an interactive demo app that shows how the scanpath measure works.
[2018-02-21]
Added new functions rscasim and plot_alignment. See section ’How the sausage is made’ below for details. Also note that the order of parameters for plot_scanpaths has changed for consistency with ggplot2.
[2018-02-20]
The new default is no normalization of scasim scores. Improved documentation and examples for find.fixation and match.scanpath. Minor improvement in functions for plotting scanpaths.
[2018-01-30]
Version 1.06 doesn’t logarithmize fixation durations when calculating scanpath similarities. (In previous versions, when normalize="durations" was used, the normalization was done using non-log-transformed durations, which could in some rare cases break the triangle inequality.)

Install

To install the latest version of the package, execute the following commands:

library("devtools");
install_github("tmalsburg/scanpath/scanpath", dependencies=TRUE)

Usage example

The code shown below can also be found in the file README.R. Open that file in RStudio and play with it as you read through this mini-tutorial.

Let’s have a look at the toy data that is included in the package:

library(tidyverse)
library(magrittr)
library(scanpath)
data(eyemovements)
head(eyemovements)
subjecttrialwordxyduration
Anne1146384319
Anne13131388147
Anne1210638688
Anne13165387156
Anne14186386244
Anne15264388193

Plotting scanpaths

To get a sense of what is going on in this data set, we create a series of plots. For this purpose, we use the function plot_scanpaths from the scanpath package. In the first plot below, each panel shows the data from one trial. There are three participants which are coded by color. The data is from a sentence reading task. The x-axis shows words and the y-axis time within trial in milliseconds.

plot_scanpaths(eyemovements, duration ~ word | trial, subject)

Plots/scanpaths.png

We can see that the participants differ in their reading speed. Also we see that each participant read the sentence more or less straight from left to right (trials: 1, 4, 7), or with a short regressions from the end of the sentence to its beginning (trials: 2, 5, 8), or with a long regression from the end of the sentence (trials: 3, 6, 9).

In the next plot, we use the fixations’ x- and y-coordinates. Each circle is a fixation and the size of the circle represents the duration of the corresponding fixation.

plot_scanpaths(eyemovements, duration ~ x + y | trial, subject)

Plots/scanpaths2.png

The function plot_scanpaths returns a ggplot object. This means that we add more elements to the plot before rendering. For example, we can labels the fixations with their index and change the limits of the axes:

plot_scanpaths(eyemovements, duration ~ x + y | trial, subject) +
  geom_text(aes(label=i), vjust=2.5, show.legend=FALSE, size=3) +
  xlim(0, 600) + ylim(284, 484)

Plots/scanpaths3.png

Extracting subsets of fixations or sub-scanpaths

In many analyses, it is not desirable to analyze the complete scanpaths as recorded during the experiment but to analyze some subset of the fixations. For instance, in a reading experiment, we might want to investigate how readers responded to a certain word and not care about what happened earlier. The scanpath package offers two functions that can be used to easily pinpoint and extract the fixations of interest: find.fixation and match.scanpath.

The function find.fixation identifies fixations that match a set of criteria which can be specified using regular expressions. For instance, the following code finds fixations on word 6:

idx <- find.fixation(eyemovements$word, eyemovements$trial, "6")
eyemovements[idx,]
subjecttrialwordxyduration
Anne16330381290
Anne26330381290
Anne36330381290
Anne36320381189
Udi46330381319
Udi56330381319
Udi66330381319
Udi66320381208
Gustave76330381348
Gustave86330381348
Gustave96330381348
Gustave96320381227

Finding these fixations could also have been achieved with a subset operation. However, if have more complex criteria for the fixations we’re interested in, things can get rather tricky. For instance, a subset is not enough when we’re only interested in the second fixation on word 6 in each trial. The following code extracts only those:

idx <- find.fixation(eyemovements$word, eyemovements$trial, "6", nth=2)
eyemovements[idx,]
subjecttrialwordxyduration
Anne36320381189
Udi66320381208
Gustave96320381227

Regular expressions also allow us to specify the context in which the fixations of interest appear. For instance the code below finds fixations on word 3 but only those that are followed by fixations on word 4:

idx <- find.fixation(eyemovements$word, eyemovements$trial, "34")
eyemovements[idx,]
subjecttrialwordxyduration
Anne13165387156
Anne23165387156
Anne33165387156
Udi43165387172
Udi53165387172
Udi63165387172
Gustave73165387187
Gustave83165387187
Gustave93165387187

Here, we find fixations on word 3 that are preceded by fixations on word 1:

idx <- find.fixation(eyemovements$word, eyemovements$trial, "1(3)", subpattern=1)
eyemovements[idx,]
subjecttrialwordxyduration
Anne13131388147
Anne23131388147
Anne33131388147
Udi43131388162
Udi53131388162
Udi63131388162
Gustave73131388176
Gustave83131388176
Gustave93131388176

The following code finds fixations on the last word but only of those that are not directly preceded by fixations on words 4 to 7:

idx <- find.fixation(eyemovements$word, eyemovements$trial, "[^4-7](8)", subpattern=1)
eyemovements[idx,]
subjecttrialwordxyduration
Anne28492382143
Udi58492382157
Gustave88492382172

The function match.scanpath works similarly but can be used to identify not just individual fixations but sequences of fixations (let’s call them scanpathlets). For example, the following code finds scanpathslets spanning words 6, 7, and 8 but only those that directly preceded by a fixation on word 4:

idx <- match.scanpath(eyemovements$word, eyemovements$trial, "4([678]+)", subpattern=1)
scanpathlets <- eyemovements[idx,]
plot_scanpaths(scanpathlets, duration~word|trial)

Plots/scanpathslets.png

See the documentation of find.fixation and match.scanpath for more details and examples.

Calculating scanpath dissimilarities

Next, we calculate the pair-wise similarities of the nine scanpaths in the dataset using the scasim measure. A simplifying intuition is that the measure quantifies the time that was spent looking at different things (or at the same things but in different order). For a precise definition see von der Malsburg & Vasishth (2011).

d1 <- scasim(eyemovements, duration ~ x + y | trial, 512, 384, 60, 1/30)
round(d1)
123456789
10454112921771713954359801670
245406756712639418895261216
311296750134693832015641201641
42176711346050012422187631509
571726393850007427182631009
613959413201242742014601005321
74358891564218718146005451355
8980526120176326310055450810
9167012166411509100932113558100

Like the function plot_scanpaths, the function scasim takes a data frame and a formula as parameters. The formula specifies which columns in the data frame should be used for the calculations. To account for distortion due to visual perspective, the comparison of the scanpaths is carried out in visual field coordinates (latitude and longitude). In order to transform the pixel coordinates provided by the eye-tracker to visual field coordinates, the scasim function needs some extra information. The first is the position of the gaze when the participant looked straight ahead (512, 384, in the present case), the distance of the eyes from the screen (60 cm), and the size of one pixel in the unit that was used to specify the distance from the screen (1/30). Finally, we have to specify a normalization procedure. normalize=FALSE means that we don’t want to normalize. See the documentation of the scasim function for details.

The time that was spent looking at different things of course depends on the duration of the two compared trials. (total duration of the two compared scanpaths constitutes an upper bound). This means that two long scanpaths may have a larger dissimilarity than two shorter scanpaths even if they look more similar. Depending on the research question, this may be undesirable. One way to get rid of the trivial influence of total duration is to normalize the dissimilarity scores. For example, we can divide them by the total duration of the two compared scanpaths:

d2 <- scasim(eyemovements, duration ~ x + y | trial, 512, 384, 60, 1/30,
             normalize="durations")
round(d2*100)
123456789
109215142591828
290121351517919
3211202415527199
451324092141424
514515901213415
6251552112024154
791727413240921
818919144159012
9281992415421120

The number are smaller now and can be interpreted as the percentage of time that was spent looking at different things.

Maps of scanpath space

The numbers in the matrix above capture a lot of information about the scanpath variance in the data set. However, dissimilarity scores are somewhat tricky to analyze. One problem is that these values have strong statistical dependencies. When we change one scanpath, this affects n dissimilarity scores. This has to be kept in mind when doing inferential stats directly on the dissimilarity scores. While there are solutions for this, it is typically more convenient to produce a representation of scanpath variance that is free from this problem. One such representation is what we call the “map of scanpath space.” On such a map, every point represents a scanpath and the distances on the map reflect the dissimilarities according to our scanpath measure, i.e. the dissimilarity scores in the matrix above.

The method for calculating these maps is called multi-dimensional scaling and one simple version of the general idea is implemented in the function cmdscale (see also isoMDS in the MASS package).

map <- cmdscale(d2)
round(map, 2)
V1V2
1-0.12-0.07
2-0.01-0.06
30.12-0.08
4-0.110
5-0.010.01
60.120
7-0.110.07
800.07
90.130.07

The table above contains two numbers for each scanpath in the data set. These numbers (V1 and V2) determine a scanpath’s location in the two-dimensional scanpath space created by cmdscale. How many dimensions we need is an empirical question.

Below is a plot showing the map of scanpaths:

map <- map %*% matrix(c(1, 0, 0, -1), 2)  # flip y-axis
plot(map, cex=4)
text(map, labels=rownames(map))

Plots/map_of_scanpath_space.png

Interestingly, the scanpaths are arranged in the same way as in the plot of the data at the top. Participants are arranged vertically and reading patterns are horizontally. This suggests that scasim not just recovered these two different kinds of information (reading speed and reading strategy) but also that it can distinguish between them.

To test how well this map represents the original dissimilarity scores, we can calculate the pair-wise differences on the map and compare them to the pair-wise scasim scores:

d2.dash <- as.matrix(dist(map))
plot(d2, d2.dash)
abline(0, 1)

Plots/fit_of_map.png

This plot suggests that the map preserves the variance in dissimilarity scores really well. Given this very good fit of the map, it appears that two dimensions were sufficient to describe the scanpath variance that is captured by scasim. This is not surprising because the scanpaths in the toy data set were designed to vary with respect to two properties: 1.) The speed of the reader, and 2.) whether there was a regression back to the beginning of the sentence and how long it was.

The benefit of the map representation is that it has much weaker statistical dependencies and that it is much more suitable for all kinds of analyses. For example, we can choose among a large number of clustering algorithms to test whether there are groups of similar scanpaths in a data set. Below, we use the simple k-means algorithm to illustrate this:

set.seed(4)
clusters <- kmeans(map, 3, iter.max=100)
plot(map, cex=4, col=clusters$cluster, pch=19)
text(map, labels=rownames(map), col="white")
points(clusters$centers, col="blue", pch=3, cex=4)

Plots/clusters.png

In this plot, color indicates to which cluster a scanpath belongs and the crosses show the center of each cluster. We see that the clusters correspond to the different reading patterns and that participants are ordered according to their reading speed within the clusters.

Apart from cluster analyses there are many other ways to analyze scanpath variance. See the articles listed below for more details.

How the sausage is made

For educational purposes, the package also includes a pure-R implementation of the scasim measure in the form of the function rscasim. This function calculates the similarity of two scanpaths and returns the alignment of fixations obtained with the Needleman-Wunsch algorithm.

s <- subset(eyemovements, trial==1)
t <- subset(eyemovements, trial==9)
alignment <- rscasim(s, t, duration ~ x + y | trial,
                     512, 384, 60, 1/30)
round(alignment)
stcost
114
2229
3318
4431
5549
6639
7758
8828
9930
101034
111155
NA12146
NA13222
NA14151
NA15216
NA16227
NA17161
NA18172

Each row in the table above describes one edit operation. The columns s and t contain the indices of the fixations involved in the edit and the column cost shows the cost of the edit. The sum of the values in the cost column is the total dissimilarity of the two scanpaths.

If both s and t contain an index, this means that two fixations were matched. If either column contains an NA, that means that a fixation in one scanpath had no matching counterpart in the other scanpath. The alignment can be visualized with the function plot_alignment:

plot_alignment(s, t, alignment, duration ~ x + y | trial, 10, 10)

Plots/alignment.png

References

  • von der Malsburg, T., & Vasishth, S. (2011). What is the scanpath signature of syntactic reanalysis? Journal of Memory and Language, 65(2), 109–127. https://dx.doi.org/10.1016/j.jml.2011.02.004
  • von der Malsburg, T., Kliegl, R., & Vasishth, S. (2015). Determinants of scanpath regularity in reading. Cognitive Science, 39(7), 1675–1703. https://dx.doi.org/10.1111/cogs.12208
  • von der Malsburg, T., & Vasishth, S. (2013). Scanpaths reveal syntactic underspecification and reanalysis strategies. Language and Cognitive Processes, 28(10), 1545–1578. https://dx.doi.org/10.1080/01690965.2012.728232
  • von der Malsburg, T., Vasishth, S., & Kliegl, R. (2012). Scanpaths in reading are informative about sentence processing. In P. B. Michael Carl, & K. K. Choudhary, Proceedings of the First Workshop on Eye-tracking and Natural Language Processing (pp. 37–53). Mumbai, India: The COLING 2012 organizing committee. https://tmalsburg.github.io/MalsburgEtAl2012Coling.pdf
  • Parshina, O., Sekerina, I., Lopukhina, A., & von der Malsburg, Titus (2022). Monolingual and bilingual reading strategies in Russian: An exploratory scanpath analysis. Reading Research Quarterly, 57(2). https://dx.doi.org/10.1002/rrq.414