Engineering Generalizable Features for Cognitive Performance from Eye-Tracking Data Through Machine Learning

Modern user-adaptive systems rely on understanding the current cognitive capabilities of users [1, 2, 3]. However, many such systems rely primarily on tailor-made solutions to detect such capabilities [4, 5, 6]. By moving towards generalizable solutions for detecting cognitive performance, we can decrease our reliance on the time-consuming development of such detection systems.

Cognition describes the different workings of our mental capacities. Weintraub et al. [7] identified executive function, episodic memory, language, processing speed, working memory as the most critical cognition subdomains for health, success in school and work, and independence in daily functioning. The National Institutes of Health (NIH) Toolbox Cognition Batteries is a well-established set of tests of cognition [7]. While the tests in the NIH Toolbox are thorough and well tested, they are also quite involved and time-consuming. As such, an alternative to the Toolbox is needed for the evaluation of cognitive performance. Cognitive workload, measured and assessed using eye-tracking technology, could serve as an alternative [8, 9].

Cognitive workload describes the level of mental resources that one person requires to perform a task and naturally falls within the purview of cognitive performance. Cognitive load theory, first proposed by John Sweller, establishes the three domains of cognitive load as intrinsic, extraneous, and germane. Intrinsic is concerned with the complexities in the information being learned. Extraneous cognitive load is the load that stems from the presentation of the instructional material. Lastly, germane cognitive load is concerned with the learner’s personal characteristics and the amount of attention that is being applied when learning [10, 11, 12]. Cognitive workload has also been shown to influence one’s eye movements [9, 13, 14].

Our project will include data gathered from studies handling many different cognitive tasks. We will argue that each task has some measure, such as a score, that correlates to the cognitive performance for that task.

We will follow the methodology of Sharma et al. to develop generalizable features from gaze data [24]. In the process, we will also develop and present a machine learning platform to effectively run experiments to investigate generalizability and produce features from gaze data.

Our research questions are as follows:

When finding the most generalizable features, we will necessarily discover features that exhibit a high degree of context-specificity.

The thesis will tackle the research questions through the following research objectives:

In this section, we will introduce the reader to a few key terms from the eye-tracking field.

The term fixation refers to when the eye holds steady on an object or a position. Fixations usually reflect attention to a stimulus [9]. When discussing fixations, we will often refer to the fixation duration, which is the amount between the start of the fixation and the end of the fixation.

Saccades are small rapid eye movements that occur between two fixations. Cognitive information is not processed during saccadic eye movements. However, studies have shown that they still can provide information about viewing behavior and indicate changes in cognition [9, 28]. We will primarily refer to the measures saccade duration and saccade length. Saccade length is the euclidean distance between the two fixations. Saccade duration is the time that the eye movements take, the time between the two fixations.

Pupillary changes are changes in pupil size. We will most commonly refer to pupil diameter, which is the size in pixels or millimeters that the pupil has at a point in time. Changes to the pupil diameter can serve as a reliable proxy of mental effort [9].

Our introduction introduces the motivation for the work, our research questions, some key terminology, and this outline.

The related work chapter situates our work in relation to the established literature and provides the theoretical background that outlines our assumptions.

This chapter outlines how we approach engineering generalizable features from gaze data and presents the datasets used in our experiments.

Our chapter on implementation presents the requirements for our platform, describes the architecture we propose to meet these requirements, and presents details on how we implement that architecture, including how and which features we engineer and how we evaluate our pipelines.

This chapter presents the evaluations of our pipelines and provides assistance in interpreting the presented materials.

The discussion delves further into the results, presents our interpretations of the patterns displayed by the results, and discusses the results in relation to the literature. The chapter also discusses the limitations of our work and suggests further work.

At last, the conclusion presents a summary of the work and the contributions of the work.

Gaze data from a human can be associated with cognition [29]. Multiple studies have been conducted to find relationships between eye-tracking and cognitive performance. Literature concerning gaze data and cognitive load is highly relevant for our work, as cognitive load can be a proxy between a task and a subject’s performance [8]. Zhang et al. [30] classified a driver’s cognitive load in two classes, using the direction of the subject’s gaze and the mean and standard deviation of change in pupil dilation. They achieved significant results using a decision tree classifier. Haapalainen et al. [8] used multiple physiological sensors and combinations of their signals to determine the usefulness of these signals in assessing cognitive load. Their results show that pupillometry was one of the least valuable features for their problem. Steichen et al. [31] investigated how to infer cognitive abilities from gaze data and showed that aggregated features from gaze data could predict a subject’s perceptual speed and visual working memory reliably. Contrary to Haapalainen et al. [8], Toker et al. [32] show that pupil dilation has a significant effect on predictions of confusion and skill acquisition when included in the feature set proposed by Steichen et al. [31]. Chen and Epps [33] performed binary classification on cognitive load using a Gaussian mixture model with pupil dilation and blink frequency. Their work showed promising results for the automatic prediction of cognitive load with gaze data.

There are also novel features engineered from gaze data. Duchowski et al. [14] engineered a measure of the frequency of pupil diameter oscillation which captures an indicator of cognitive load, called Index of Pupillary activity (IPA). IPA was proposed as an alternative to the existing Index of Cognitive Activity (ICA) since ICA is not available to the public. Later they proposed an improvement to IPA called The Low High Index of Pupillary Activity (LHIPA) [13], which can discriminate cognitive load in several experiments where IPA fails to do so. Sharma et al. [34] used heatmaps generated from the fixtures of subjects to predict students' performance. Features from the heatmaps were extracted with a pre-trained VGG19 model. They showed that they could predict cognitive performance with an error rate of less than 5%.

While research on gaze data has achieved promising results predicting cognitive workload and performance, no work has addressed inferring cognitive performance between contexts from gaze data.

While universalism and context-sensitivity in information systems research remain essential research topics, [22, 23] generalizability continues to be a chief concern in machine learning research. It is named as one of the primary goals in a slew of fields ranging from healthcare analytics [35, 36], business research [37], finance [38] to psychiatry [39]

Turney has worked to provide a formal definition of context-sensitive features within concept learning [40, 41, 42]. His work is strengthened by John et al. [43] and others. While our view of context is tangential to his, the definition of context-sensitive features as features only relevant given a set context is still applicable to our work. We look to engineer generalizable features that provide predictive power from data gathered in a context different from the feature’s origin.

Bouchard et al. work with technologies for ambient intelligence to increase the autonomy of the elderly through enhanced tracking to assist medical personnel. Their work with Bluetooth beacons revealed a feature based on the received signal strength indication timeseries that generalize between contexts made up of different hardware, different hardware configurations, and different floor plans [44].

Ferentzi et al. investigated whether information gained from a single interoceptive modality (mode of understanding one’s own body) can be generalized to other modalities. [45] They investigated a group of students' ability to count their heartbeats, sense their gastric fullness, sense bitterness, pain threshold, proprioceptive sensitivity (ability to position their limbs), and sense of balance. They argue that their findings strongly support that interoceptive accuracy assessed with a single modality cannot be generalized to other modalities.

Feature generalizability has seen some popularity in Brain-Computer Interfaces (BCI) research. Kim et al. demonstrate generalizable features generated using a simple deep neural network that decodes learning strategies from electroencephalography (EEG) signals gathered when subjects execute a learning strategy task [46]. Nurse et al. also showed an approach including a high-dimension neural network used to do real-time EEG classification for use in BCI. In this approach, a neural network acted as both feature extractor and classifier, and the technique was shown to generalize [47].

Kidzinski et al. discuss how the bias-variance trade-off is a central concern when studying generalizability in the context of Massive Open Online Courses (MOOCs) [48]. The bias-variance trade-off refers to the fact that one cannot achieve both zero variance and zero bias in practice. A model with a significant bias captures the general trend of the data but does not fit the sample data points closely. On the other hand, variance is how similarly the model captures the testing dataset compared to the training dataset. Kidzinski et al. argue that much of the research on MOOCs focuses on the specific course being studied and, as such, models do not generalize to other courses. Their view is that the bias-variance trade-off is central when assessing the generalizability of MOOC models [48].

Recently feature generalizability has also been shown to have relevance when predicting driver’s intentions at intersections in an automotive context [49], when using ML for personality assessment [50], in speech enhancement systems [51], in face-based mind wandering detection [52], and in music information retrieval [26].

Machine learning platforms and pipelines are essential in many research projects. In bioinformatics, Guzzetta et al. present an L1L2 based machine learning pipeline that is effective for fitting quantitative phenotypes from genome data [53]. Fountain-Jones et al. demonstrate how a flexible ensemble pipeline can predict pathogen exposure in animal ecology. To demonstrate their pipeline, they model pathogen exposure risk in two different viruses that infect African lions [54]. Sutton et al. have developed PhysOnline, a pipeline built on the Apache Spark platform. PhysOnline uses streaming physiological data and can be used for real-time classification in the biomedical and health informatics field [55]. Shaikh et al. tackle the complicated task of creating a machine learning pipeline that can ensure fairness in decision-making. The system can understand policies written in natural language and will assist users in ensuring that set fairness policies are followed [56].

There has been some research that focuses on the pipelines themselves. Olson and Moore present TPOT, a tree-based tool for optimizing pipelines, which has recently shown much promise [57]. This system exists in the new and promising field of AutoML, which seeks to automate the design of pipelines for machine learning, democratizing machine learning further. Mohr et al. have contributed to predicting pipeline runtimes to improve one of the significant limitations of current AutoMl technology [58]. The systems we have available today take a considerable amount of time to arrive at good pipeline designs if the case is even slightly complex. Effectively predicting pipeline runtimes would enable us to significantly decrease the cost of AutoML systems, as one could ignore pipelines that would not complete within the systems timeout limits.

There have also been several attempts at creating more generally applicable platforms for machine learning. López García et al. presents the DEEP-Hybrid-DataCloud framework as "a distributed architecture to provide machine learning practitioners with a set of tools and cloud services to cover the whole machine learning development cycle" [59]. The DEEP framework represents an important step in democratizing machine learning and deep learning for all research areas. Ribeiro et al. present an architecture to provide scalable machine learning capabilities as a service, along with an open-source implementation [60]. Kraska et al. have developed MLBase, which includes a simple declarative language for specifying machine learning tasks to be used by both researchers and end-users [61]. The available resources for machine learning such as proposed architectures, open-source and commercial platforms and pipelines, tooling, and other supporting technologies have increased significantly in the last couple of years. Machine learning tools to process large-scale data for decision assistance, automation, and interactive data analytics are growing and will continue to be essential for research and business [62].

We have been working with three different datasets gathered and published by other researchers. They are all gathered during cognitive tasks and, as such, are suitable for investigating cognitive performance. The selected datasets differ in context and what parts of the spectrum of cognitive processes they cover. We have used the shorthand names EMIP, CSCW, and Fractions to refer to the datasets. They are further described in the following section.

Our work seeks to investigate generalizability between specific contexts; thus, we must be aware of our contextual biases. We have selected datasets that we consider to cover a significant spectrum of cognitive processes.

EMIP is an individual task that is about reading and understanding programming code. We hypothesize that the task in EMIP relies primarily on three of the cognitive subdomains deemed most critical by Weintraub et al. [7]. Reading and understanding code is a trained skill reliant heavily on one’s understanding of language. Understanding the entirety of a class requires keeping all functions of the class in one’s working memory. The post-test is organized so that one needs to remember these functions for a short time after reading the code. As with almost all cognitive tasks, attention is a critical part of performing well when reading and understanding code.

CSCW is a task where participants collaborate in creating a concept map from a video they have watched individually. Naturally, language will be an important part of any collaborative work as one needs to express one’s understanding of the content to one collaborator. Executive function, specifically planning, is essential for creating concept maps. Concept maps include tying different pieces of information together in a complete whole. Before starting the concept map, they viewed a video explaining the concept they were to map. In order to remember information presented in the videos, the cognitive subdomain episodic memory is at work. Again, attention is essential when viewing a video for learning and successful collaboration with another party.

The Fractions dataset also stems from a collaborative task. Students work together to learn about equivalent fractions in an ITS. For the collaborative aspect, attention and language are again important.

All cognitive tasks likely include some aspect of all cognitive subdomains. What we intend with this section is to illustrate how our three datasets cover tasks that rely more heavily on five of the six cognitive subdomains presented by Weintraub et al. [7] as most important. Of the six cognitive subdomains, our datasets do not include tasks that rely heavily on processing speed. Processing speed is an important factor in good collaboration, but we did not consider this subdomain to be as central in any of the tasks and thus will not claim to cover it with these datasets.

This subsection explains how we achieved goals 1 & 2 of creating a platform for generating generalizable features.

This subsection explains how we achieved goal 3.

We will present the flow of the feature extraction process and discuss the specific features we extract. The features are organized into three sections:

This section explains how we solved goal 4 of creating our platform.

By a pipeline, we mean a specific combination of datasets, feature groups, and methods for reducing the feature space (dimensionality reduction or feature selection). We will refer to these parts as pipeline components.

This section will outline how we achieve the fifth goal of our platform:

Models produced by our pipelines are evaluated in two ways, with out-of-sample testing and out-of-study testing. Out-of-sample testing uses a subset of the in-study dataset to evaluate the predictive power of the model. Out-of-study testing uses the dataset designated as out-of-study to evaluate generalizability. This subsection explains how we evaluate the results of each test.

This section explains how we reached the sixth goal of our platform.

Our reproducibility strategy primarily consists of four components. The version-control tool, git; the machine learning management tool, comet.ml; the python package management tool, poetry; and google cloud platform.

We calculate a specific baseline for each dataset combination. The baseline is equivalent to predicting the mean of the labels from the in-study dataset. We then quantify this baseline by the NRMSE from those predictions. The calculation of the baseline is done separately for each dataset combination. Our pipelines are evaluated against the baseline for the given dataset combination.

Pseudocode for the in-study baseline:

Our work focuses on engineering generalizable features. This section outlines the results from the out-of-sample testing, which does not indicate generalizability. However, the section is presented to the reader to benchmark the range of predictive power that our features exhibit in more traditional tasks.

We evaluate the generalizability of the pipelines with the out-of-study dataset. We compare the errors from making predictions on the out-of-study dataset to the out-of-sample results using FGI, explained in Section 4.4.2. In order to identify the most generalizable pipelines, we need to filter out pipelines that perform poorly in out-of-sample testing. We do this by filtering out the pipelines that do not perform better than the baseline. From 216 pipelines, 72 beat the baseline.

As represented in Figure 7, 2-to-1 pipelines are, in general, more generalizable than 1-to-1 pipelines. This is in line with the results of [24]. Our focus is generalizability, and as such, we will only refer to the 2-to-1 pipelines in the following sections.

Table 8 shows the most generalizable pipelines. The first and most prevalent factor for generalizable pipelines is which datasets were designated in-study datasets for that pipeline. When the in-study dataset combines Fractions and CSCW (fractions_cscw), and EMIP is the out-of-study dataset, the pipelines are more generalizable. Nine of the ten more generalizable pipelines contain this combination of datasets.

LASSO is the most represented method of feature space reduction among the more generalizable pipelines. We note that more pipelines with LASSO beat the baseline, which might explain the trend we are seeing. LASSO is also included in the two pipelines with the best FGI. PCA performs very well for the feature groups All and heatmap; this might be explained by the fact that these are the largest feature groups with hundreds of thousands of values.

Seven of our twelve feature groups are represented among the ten most generalizable pipelines. However, three feature groups show up more than once, All, Heatmaps, and GARCH. This indicates that they might be more generalizable feature groups. It should also be noted that GARCH is also the only pipeline with an in-study dataset that is not fractions_cscw of the ten most generalizable pipelines.

The bottom third of the filtered baselines contain pipelines that are more context-specific.

Again we can see that the dataset combination is an important factor in what pipelines exhibit context-sensitivity. LASSO is more represented among the more context-specific pipelines. As with the generalizable pipelines, there are more pipelines with LASSO that beat the baseline. For the feature groups, we observe some differences from the generalizable pipelines. ARMA, LHIPA, Gaze Characteristics, and Pupil Diameter are present in the context-sensitive pipelines but not in the more generalizable pipelines. This indicates that these feature groups are more likely to produce context-sensitive pipelines. Heatmaps, Spectral Histogram, Hidden Markov Models, All, and Saccade Length appear in generalizable and context-sensitive pipelines. This further supports the observation that dataset combination is an essential variable for generalizability.

Our goal was to identify generalizable pipelines, and for that, we have the FGI measure described in Section 4.4.2. FGI describes how similar the distribution of errors from out-of-sample testing and out-of-study testing and is a measure of generalizability [24]. However, it is not enough that the two distributions are similar. Generalizability should be a measure of usable predictive power in contexts other than the training context. Random guesses would create identical distributions of error in out-of-sample testing and out-of-study testing, and as such, would have a good FGI value. To counteract this, we chose to disregard all pipelines that do not outperform their respective baselines for the purpose of measuring generalizability.

As shown in Figure 7, 2-to-1 pipelines tend to generalize better than 1-to-1 pipelines. Combining two datasets from two separate contexts for training introduces more variability to the dataset. By introducing more variability, we are optimizing for both bias and variance, which is crucial for generalizability [48]. These results align with the findings of Sharma et al. [24]. When we generate a feature that can predict cognitive performance in two separate contexts, that feature has mutual information with cognitive performance in those two contexts. Such a feature likely describes some non-contextual aspects of cognitive performance, meaning it should have higher mutual information with cognitive performance in a third context. In further discussions, we will focus on results from the 2-to-1 pipelines.

Table 8 indicates that pipelines that combine Fractions and CSCW for training and use EMIP for testing are more generalizable than pipelines where EMIP was included in the training set. The EMIP dataset consists of gaze data recorded of people completing individual tasks. All datasets include information from one or more context-specific tasks, but the Fractions and CSCW also have a collaborative aspect. We theorize that pipelines trained on EMIP do worse at describing this collaborative aspect. The variability introduced by the interactions that come with collaborative work assists those pipelines in generalizing. There is a degree of division of labor in collaboration, which gives rise to individual tasks even in the collaborative context; as such, there is mutual information even with cognitive performance in individual tasks even when training on collaborative tasks. Pinpointing the effectiveness of training on collaborative data when generalizing to contexts without a collaborative aspect is a promising area for more experimentation.

Table 8 presents our more generalizable pipelines. In this section, we will explain why we believe the feature groups used in those pipelines generalize.

Table 9 presents our more context-sensitive pipelines. In this section, we will explain why we believe the feature groups exhibit context-sensitivity.

In Section 3.2, we outline how we believe our datasets are representative of a significant portion of human cognition. However, it would be presumptuous to say that three datasets from three different contexts could represent all of the cognitive processes. Our goal has been to generalize between our three contexts, and we believe that our methods provide meaningful insights into how one could create generalizable features for other contexts. We do not mean to say that our features will generalize to any context. Nevertheless, this is a first step that provides evidence on how gaze-related features provide a certain level of generalizability across three distinct and commonly employed contexts.

Table 8 and Table 9 show some indications that datasets from individual tasks generalize poorly to contexts that include collaborative work. Had individual work been better represented in our data, we might be able to say more about how individual tasks generalize in general. Ideally, we should have had at least one more dataset for individual tasks.

Our work assumes that cognitive performance can be characterized by labels in our datasets and represented in gaze data. For our approach, we need an object, quantifiable metric to assess cognitive performance, but as with many other things in cognition, the reality is likely more complex.

For complete external repeatability, we would ideally publish the data we used to perform our experiments. However, the scope of our thesis project was such that it would be impossible to gather our own data to perform the analyses we have performed. As a result, we had to turn to generous researchers who allowed us to work with their data, which in turn means that the data is not ours to share.

Due to the considerable effort put into creating our experimental platform, it would be possible to expand the different pipeline components we test greatly. In our work, we tested 22 features in 12 feature groups, three datasets in 9 combinations, two methods for reducing the feature space, and a single ensemble classifier. While our tested features are exhaustive, we limited how many feature-space reduction methods we worked with and tested only a single ensemble classifier. It would be possible to investigate the effects of other variants of these pipeline components on generalizability in further work.

While we can identify feature groups that can produce generalizable pipelines, we do not know how the individual features in each group affect the generalizability. It is also likely that combinations of features from different groups would create very generalizable pipelines.

The Hidden Markov Models (HMM) are included in both Table 8 and Table 9. That HMMs generalize seems counter-intuitive, especially given that ARMA does not generalize (see Section 6.6.1). At its core, the transition matrix of HMM represents a discrete version of ARMA. ARMA models how previous values in a timeseries affect the current value, while HMMs describe how previous states affect the current state. What dataset was used might be a significant contributing factor to why HMMs either generalize or exhibit context-specificity; However, more research is needed to draw any conclusions.

Our code is available at https://github.com/s0lvang/ideal-pancake under the MIT License [105]. If there are any questions or bugs regarding the code, please open an issue. The text of this thesis can be found at https://github.com/aslakhol/thesis.

Figure 8 shows the density for all the values in the datasets and signals we use. The data is normalized, as explained in Section 4.1.