skip to main content
research-article

Cognitive Heat: Exploring the Usage of Thermal Imaging to Unobtrusively Estimate Cognitive Load

Published: 11 September 2017 Publication History

Abstract

Current digital systems are largely blind to users’ cognitive states. Systems that adapt to users’ states show great potential for augmenting cognition and for creating novel user experiences. However, most approaches for sensing cognitive states, and cognitive load specifically, involve obtrusive technologies, such as physiological sensors attached to users’ bodies. This paper present an unobtrusive indicator of the users’ cognitive load based on thermal imaging that is applicable in real-world. We use a commercial thermal camera to monitor a person’s forehead and nose temperature changes to estimate their cognitive load. To assess the effect of different levels of cognitive load on facial temperature we conducted a user study with 12 participants. The study showed that different levels of the Stroop test and the complexity of reading texts affect facial temperature patterns, thereby giving a measure of cognitive load. To validate the feasibility for real-time assessments of cognitive load, we conducted a second study with 24 participants, we analyzed the temporal latency of temperature changes. Our system detected temperature changes with an average latency of 0.7 seconds after users were exposed to a stimulus, outperforming latency in related work that used other thermal imaging techniques. We provide empirical evidence showing how to unobtrusively detect changes in cognitive load in real-time. Our exploration of exposing users to different content types gives rise to thermal-based activity tracking, which facilitates new applications in the field of cognition-aware computing.

References

[1]
Yomna Abdelrahman. 2013. Thermal imaging for interactive surfaces. Master’s thesis.
[2]
Yomna Abdelrahman, Mohamed Khamis, Stefan Schneegass, and Florian Alt. 2017. Stay cool! understanding thermal attacks on mobile-based user authentication. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 3751--3763.
[3]
Yomna Abdelrahman, Pascal Knierim, and Albrecht Schmidt. 2017. Snake View: Exploring Thermal Imaging as a Vision Extender in Mountains. In (UbiMount) ubiquitous computing in the mountains.
[4]
Yomna Abdelrahman, Pascal Knierim, Pawel Wozniak, Niels Henze, and Albrecht Schmidt. 2017. See through the Fire: Evaluating the Augmentation of Visual Perception of Firefighters Using Depth and Thermal Cameras. In (WAHM) Workshop on Ubiquitous Technologies for Augmenting the Human Mind.
[5]
Yomna Abdelrahman, Alireza Sahami Shirazi, Niels Henze, and Albrecht Schmidt. 2015. Investigation of Material Properties for Thermal Imaging-Based Interaction. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15). ACM, New York, NY, USA, 15--18.
[6]
Jackson Beatty. 1982. Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological bulletin 91, 2 (1982), 276.
[7]
Kirsch C. (Producer) Moxham J. B. (Producer) Alusi G. (Producer) Berkovitz, B. (Producer) and T. (Producer) Cheesman. 2013. 3D Head and Neck Anatomy with Special Senses and Basic Neuroanatomy. {DVD-ROM}. United Kingdom: Primal Pictures.
[8]
Katie Crowley, Aidan Sliney, Ian Pitt, and Dave Murphy. 2010. Evaluating a Brain-Computer Interface to Categorise Human Emotional Response. In ICALT. 276--278.
[9]
Mai ElKomy, Yomna Abdelrahman, Markus Funk, Tilman Dingler, Albrecht Schmidt, and Slim Abdennadher. 2017. ABBAS: An Adaptive Bio-sensors Based Assistive System. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2543--2550.
[10]
Douglas C Engelbart. 1962. Augmenting human intellect: a conceptual framework (1962). Technical Report for the Air Force Office of Scientific Research (1962).
[11]
Veronika Engert, Arcangelo Merla, Joshua A Grant, Daniela Cardone, Anita Tusche, and Tania Singer. 2014. Exploring the use of thermal infrared imaging in human stress research. PloS one 9, 3 (2014), e90782.
[12]
Stephen H. Fairclough. 2009. Fundamentals of Physiological Computing. Interact. Comput. 21, 1- 2 (Jan. 2009), 133--145.
[13]
Ismael Fernández-Cuevas, Joao Carlos Bouzas Marins, Javier Arnáiz Lastras, Pedro María Gómez Carmona, Sergio Piñonosa Cano, Miguel Ángel García-Concepción, and Manuel Sillero-Quintana. 2015. Classification of factors influencing the use of infrared thermography in humans: A review. Infrared Physics 8 Technology 71 (2015), 28--55.
[14]
Markus Funk, Tilman Dingler, Jennifer Cooper, and Albrecht Schmidt. 2015. Stop helping me-I’m bored!: why assembly assistance needs to be adaptive. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers. ACM, 1269--1273.
[15]
Rikke Gade and Thomas B Moeslund. 2014. Thermal cameras and applications: a survey. Machine vision and applications 25, 1 (2014), 245--262.
[16]
Marc Germain, Michel Jobin, and Michel Cabanac. 1987. The effect of face fanning during recovery from exercise hyperthermia. Canadian journal of physiology and pharmacology 65, 1 (1987), 87--91.
[17]
Nitesh Goyal and Susan R. Fussell. 2017. Intelligent Interruption Management using Electro Dermal Activity based Physiological Sensor on Collaborative Sensemaking. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 16.
[18]
M Haak, S Bos, S Panic, and LJM Rothkrantz. 2009. Detecting stress using eye blinks and brain activity from EEG signals. Proceeding of the 1st driver car interaction and interface (DCII 2008) (2009), 35--60.
[19]
Jennifer A Healey and Rosalind W Picard. 2005. Detecting stress during real-world driving tasks using physiological sensors. Intelligent Transportation Systems, IEEE Transactions on 6, 2 (2005), 156--166.
[20]
Javier Hernandez, Pablo Paredes, Asta Roseway, and Mary Czerwinski. 2014. Under pressure: sensing stress of computer users. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 51--60.
[21]
Sarah M Hosni, Mahmoud E Gadallah, Sayed F Bahgat, and Mohamed S AbdelWahab. 2007. Classification of EEG signals using different feature extraction techniques for mental-task BCI. In Computer Engineering 8 Systems, 2007. ICCES’07. International Conference on. IEEE, 220--226.
[22]
Curtis S Ikehara and Martha E Crosby. 2005. Assessing cognitive load with physiological sensors. In Proceedings of the 38th annual hawaii international conference on system sciences. IEEE, 295a--295a.
[23]
Stephanos Ioannou, Sjoerd Ebisch, Tiziana Aureli, Daniela Bafunno, Helene Alexi Ioannides, Daniela Cardone, Barbara Manini, Gian Luca Romani, Vittorio Gallese, and Arcangelo Merla. 2013. The autonomic signature of guilt in children: a thermal infrared imaging study. PloS one 8, 11 (2013), e79440.
[24]
Stephanos Ioannou, Vittorio Gallese, and Arcangelo Merla. 2014. Thermal infrared imaging in psychophysiology: potentialities and limits. Psychophysiology 51, 10 (2014), 951--963.
[25]
Giulio Jacucci, Anna Spagnolli, Jonathan Freeman, and Luciano Gamberini. 2014. Symbiotic Interaction: A Critical Definition and Comparison to other Human-Computer Paradigms. Springer International Publishing, Cham, 3--20.
[26]
SD Jenkins and RDH Brown. 2014. A correlational analysis of human cognitive activity using Infrared Thermography of the supraorbital region, frontal EEG and self-report of core affective state. QIRT.
[27]
Jihun Kang and Kari Babski-Reeves. 2008. Detecting Mental Workload Fluctuation during Learning of a Novel Task Using Thermography. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 52, 19 (2008), 1527--1531.
[28]
J Kang, JA McGinley, G McFadyen, and K Babski-Reeves. 2006. Determining learning level and effective training times using thermography. In Proceedings of Army Science Conference, Orlando, Florida, USA.
[29]
Hisanori Kataoka, Hiroshi Kano, Hiroaki Yoshida, Atsuo Saijo, Masashi Yasuda, and Masato Osumi. 1998. Development of a skin temperature measuring system for non-contact stress evaluation. In Engineering in Medicine and Biology Society, 1998. Proceedings of the 20th Annual International Conference of the IEEE, Vol. 2. IEEE, 940--943.
[30]
Koji Kuraoka and Katsuki Nakamura. 2011. The use of nasal skin temperature measurements in studying emotion in macaque monkeys. Physiology 8 behavior 102, 3 (2011), 347--355.
[31]
Hindra Kurniawan, Alexey V Maslov, and Mykola Pechenizkiy. 2013. Stress detection from speech and galvanic skin response signals. In Computer-Based Medical Systems (CBMS), 2013 IEEE 26th International Symposium on. IEEE, 209--214.
[32]
Elise Labbé, Nicholas Schmidt, Jonathan Babin, and Martha Pharr. 2007. Coping with stress: the effectiveness of different types of music. Applied psychophysiology and biofeedback 32, 3-4 (2007), 163--168.
[33]
Eric Larson, Gabe Cohn, Sidhant Gupta, Xiaofeng Ren, Beverly Harrison, Dieter Fox, and Shwetak Patel. 2011. HeatWave: Thermal Imaging for Surface User Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, New York, NY, USA, 2565--2574.
[34]
Robert W Levenson. 1988. Emotion and the autonomic nervous system: A prospectus for research on autonomic specificity. (1988).
[35]
Joseph CR Licklider. 1960. Man-computer symbiosis. IRE transactions on human factors in electronics 1 (1960), 4--11.
[36]
Barbara Manini, Daniela Cardone, Sjoerd JH Ebisch, Daniela Bafunno, Tiziana Aureli, and Arcangelo Merla. 2013. Mom feels what her child feels: thermal signatures of vicarious autonomic response while watching children in a stressful situation. Front. Hum. Neurosci 7 (2013).
[37]
TV McCaffrey, RD McCook, and RD Wurster. 1975. Effect of head skin temperature on tympanic and oral temperature in man. Journal of Applied Physiology 39, 1 (1975), 114--118.
[38]
Arcangelo Merla and Gian Luca Romani. 2007. Thermal signatures of emotional arousal: a functional infrared imaging study. In 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 247--249.
[39]
Takakazu Oka, Kae Oka, and Tetsuro Hori. 2001. Mechanisms and mediators of psychological stress-induced rise in core temperature. Psychosomatic medicine 63, 3 (2001), 476--486.
[40]
Calvin KL Or and Vincent G Duffy. 2007. Development of a facial skin temperature-based methodology for non-intrusive mental workload measurement. Occupational Ergonomics 7, 2 (2007), 83--94.
[41]
Avinash Parnandi, Beena Ahmed, Eva Shipp, and Ricardo Gutierrez-Osuna. 2013. Chill-Out: Relaxation training through respiratory biofeedback in a mobile casual game. In Mobile Computing, Applications, and Services. Springer, 252--260.
[42]
Avinash Parnandi, Youngpyo Son, and Ricardo Gutierrez-Osuna. 2013. A control-theoretic approach to adaptive physiological games. In Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on. IEEE, 7--12.
[43]
Ioannis Pavlidis and James Levine. 2002. Thermal image analysis for polygraph testing. IEEE Engineering in Medicine and Biology Magazine 21, 6 (2002), 56--64.
[44]
Ioannis Pavlidis, Panagiotis Tsiamyrtzis, Dvijesh Shastri, Avinash Wesley, Yan Zhou, Peggy Lindner, Pradeep Buddharaju, Rohan Joseph, Anitha Mandapati, Brian Dunkin, et al. 2012. Fast by nature-how stress patterns define human experience and performance in dexterous tasks. Scientific Reports 2 (2012).
[45]
Michael Kai Petersen, Carsten Stahlhut, Arkadiusz Stopczynski, Jakob Eg Larsen, and Lars Kai Hansen. 2011. Smartphones get emotional: mind reading images and reconstructing the neural sources. In International Conference on Affective Computing and Intelligent Interaction. Springer, 578--587.
[46]
Bastian Pfleging, Drea K. Fekety, Albrecht Schmidt, and Andrew L. Kun. 2016. A Model Relating Pupil Diameter to Mental Workload and Lighting Conditions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA.
[47]
Rosalind W. Picard. 1997. Affective Computing. MIT Press, Cambridge, MA, USA.
[48]
Colin Puri, Leslie Olson, Ioannis Pavlidis, James Levine, and Justin Starren. 2005. StressCam: non-contact measurement of users’ emotional states through thermal imaging. In CHI’05 extended abstracts on Human factors in computing systems. ACM, 1725--1728.
[49]
B. A. Rajoub and R. Zwiggelaar. 2014. Thermal Facial Analysis for Deception Detection. IEEE Transactions on Information Forensics and Security 9, 6 (June 2014), 1015--1023.
[50]
Alireza Sahami Shirazi, Yomna Abdelrahman, Niels Henze, Stefan Schneegass, Mohammadreza Khalilbeigi, and Albrecht Schmidt. 2014. Exploiting Thermal Reflection for Interactive Systems. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 3483--3492.
[51]
Albrecht Schmidt. 2017. Augmenting Human Intellect and Amplifying Perception and Cognition. IEEE Pervasive Computing 16, 1 (2017), 6--10.
[52]
Stefan Schneegass, Bastian Pfleging, Nora Broy, Frederik Heinrich, and Albrecht Schmidt. 2013. A data set of real world driving to assess driver workload. In Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 150--157.
[53]
Nandita Sharma and Tom Gedeon. 2012. Objective measures, sensors and computational techniques for stress recognition and classification: A survey. Computer methods and programs in biomedicine 108, 3 (2012), 1287--1301.
[54]
Dvijesh Shastri, Arcangelo Merla, Panagiotis Tsiamyrtzis, and Ioannis Pavlidis. 2009. Imaging facial signs of neurophysiological responses. IEEE Transactions on Biomedical Engineering 56, 2 (2009), 477--484.
[55]
Dvijesh Shastri, Manos Papadakis, Panagiotis Tsiamyrtzis, Barbara Bass, and Ioannis Pavlidis. 2012. Perinasal imaging of physiological stress and its affective potential. IEEE Transactions on Affective Computing 3, 3 (2012), 366--378.
[56]
Yu Shi, Natalie Ruiz, Ronnie Taib, Eric Choi, and Fang Chen. 2007. Galvanic skin response (GSR) as an index of cognitive load. In CHI’07 extended abstracts on Human factors in computing systems. ACM, 2651--2656.
[57]
Alireza Sahami Shirazi, Markus Funk, Florian Pfleiderer, Hendrik Glück, and Albrecht Schmidt. 2012. MediaBrain: Annotating Videos based on Brain-Computer Interaction. In Mensch 8 Computer. 263--272.
[58]
Alireza Sahami Shirazi, Mariam Hassib, Niels Henze, Albrecht Schmidt, and Kai Kunze. 2014. What’s on your mind?: mental task awareness using single electrode brain computer interfaces. In Proceedings of the 5th Augmented Human International Conference. ACM, 45.
[59]
Rajita Sinha, William R Lovallo, and Oscar A Parsons. 1992. Cardiovascular differentiation of emotions. Psychosomatic Medicine 54, 4 (1992), 422--435.
[60]
John Stemberger, Robert S Allison, and Thomas Schnell. 2010. Thermal imaging as a way to classify cognitive workload. In Computer and Robot Vision (CRV), 2010 Canadian Conference on. IEEE, 231--238.
[61]
J Ridley Stroop. 1935. Studies of interference in serial verbal reactions. Journal of Experimental Psychology: General 121, 1 (1935), 15.
[62]
John Sweller, Paul Ayres, and Slava Kalyuga. 2011. Measuring Cognitive Load. Springer New York, New York, NY, 71--85.
[63]
Boris M Velichkovsky and John Paulin Hansen. 1996. New technological windows into mind: There is more in eyes and brains for human-computer interaction. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 496--503.
[64]
Paul Viola and Michael J Jones. 2004. Robust real-time face detection. International journal of computer vision 57, 2 (2004), 137--154.
[65]
Zixuan Wang, Jinyun Yan, and Hamid Aghajan. 2012. A framework of personal assistant for computer users by analyzing video stream. In Proceedings of the 4th Workshop on Eye Gaze in Intelligent Human Machine Interaction. ACM, 14.
[66]
Mark Weiser. 1991. The computer for the 21st century. Scientific american 265, 3 (1991), 94--104.
[67]
Robert B Zajonc, Sheila T Murphy, and Marita Inglehart. 1989. Feeling and facial efference: implications of the vascular theory of emotion. Psychological review 96, 3 (1989), 395.
[68]
Zhen Zhu, Panagiotis Tsiamyrtzis, and Ioannis Pavlidis. 2007. Forehead thermal signature extraction in lie detection. In 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 243--246.

Cited By

View all
  • (2025)Facial thermal imaging: A systematic review with guidelines and measurement uncertainty estimationMeasurement10.1016/j.measurement.2024.115879242(115879)Online publication date: Jan-2025
  • (2024)Between Symbols and Particles: Investigating the Complexity of Learning Chemical EquationsEducation Sciences10.3390/educsci1406057014:6(570)Online publication date: 26-May-2024
  • (2024)Hearing the World: A Pilot Study Design on Spatial Audio for the Visually ImpairedProceedings of the 27th International Academic Mindtrek Conference10.1145/3681716.3689442(244-248)Online publication date: 8-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 1, Issue 3
September 2017
2023 pages
EISSN:2474-9567
DOI:10.1145/3139486
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 September 2017
Accepted: 01 June 2017
Revised: 01 May 2017
Received: 01 February 2017
Published in IMWUT Volume 1, Issue 3

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Thermal Imaging
  2. Thermal latency
  3. cognitive load

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

  • German Research Foundation within the SimTech Cluster of Excellence
  • Amplify project which received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme
  • Victorian State Government and Microsoft through their contributions to the Microsoft Research Centre for Social Natural User Interfaces (SocialNUI)

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)206
  • Downloads (Last 6 weeks)25
Reflects downloads up to 28 Oct 2024

Other Metrics

Citations

Cited By

View all
  • (2025)Facial thermal imaging: A systematic review with guidelines and measurement uncertainty estimationMeasurement10.1016/j.measurement.2024.115879242(115879)Online publication date: Jan-2025
  • (2024)Between Symbols and Particles: Investigating the Complexity of Learning Chemical EquationsEducation Sciences10.3390/educsci1406057014:6(570)Online publication date: 26-May-2024
  • (2024)Hearing the World: A Pilot Study Design on Spatial Audio for the Visually ImpairedProceedings of the 27th International Academic Mindtrek Conference10.1145/3681716.3689442(244-248)Online publication date: 8-Oct-2024
  • (2024)TimelyTale: A Multimodal Dataset Approach to Assessing Passengers' Explanation Demands in Highly Automated VehiclesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785448:3(1-60)Online publication date: 9-Sep-2024
  • (2024)Reading Between the HeatProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314417:4(1-30)Online publication date: 12-Jan-2024
  • (2024)Interrupting for Microlearning: Understanding Perceptions and Interruptibility of Proactive Conversational Microlearning ServicesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642778(1-21)Online publication date: 11-May-2024
  • (2024)Smart Plants on Wheels: Enhancing Indoor Productivity Using Smart Plants and Autonomous Ground DronesIEEE Pervasive Computing10.1109/MPRV.2024.341667723:3(31-38)Online publication date: 1-Jul-2024
  • (2024)Democratizing EEG: Embedding Electroencephalography in a Head-Mounted Display for Ubiquitous Brain-Computer InterfacingInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2388368(1-25)Online publication date: 19-Aug-2024
  • (2024)A Bibliometric Analysis of Cognitive Load Sensing Methodologies and Its ApplicationsDigital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management10.1007/978-3-031-61060-8_9(113-134)Online publication date: 29-Jun-2024
  • (2024)Introducing machine‐learning‐based data fusion methods for analyzing multimodal data: An application of measuring trustworthiness of microenterprisesStrategic Management Journal10.1002/smj.359745:8(1597-1629)Online publication date: 5-Mar-2024
  • Show More Cited By

View Options

Get Access

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media