Toward an Intuitive Device for Construction Hazard Recognition Management: Eye Fixation–Related Potentials in Reinvestigation of Hazard Recognition Performance Prediction
Publication: Journal of Construction Engineering and Management
Volume 150, Issue 5
Abstract
Developing real-time automatic intuitive devices based on electroencephalography (EEG) to improve hazard recognition performance (HRP) is essential in construction engineering management. However, existing studies generally developed devices based on unimodal data, failing to distinguish the role played by brain activities in different visual areas and ignoring the possible existence of other psychological mechanisms. Therefore, this study aims to use multimodal data based on brain-eye synergy to predict HRP and reveal the cognitive mechanism of hazard recognition to develop intuitive hazard recognition devices. Accordingly, we designed experiments to investigate this problem: (1) 53 construction workers completed a 120-trial hazard recognition task in a laboratory environment (), (2) an EEG recorder collected the participants’ EEG activities and an eye-tracking device recorded the data of eye movement, and (3) the power spectral density (PSD) of EEG was calculated and then imported into Matlab2018a in combination with EEGLAB for analysis. The results showed that the prediction results of hazard recognition have a time-variant effect. For fall-related hazards, EEG signals within 0–320 ms after the onset were significant in predicting HRP; for electric-related and fire-related hazards, EEG signals within 560–640 ms after the starting point were the most valuable. Also, compared with fall-related hazards, it took a significantly longer time to recognize electric-related and fire-related hazards, which consumed more attentional resources. Finally, the frontal lobe requires more attentional resources to finish cognitive activities than the occipital lobe to finish visual information processing activities, and the occipital lobe needs to process visual information first, and then transmit it to the frontal lobe for analysis and judgment. For the body of knowledge, this study, finding the time-variant effect of HRP, improves the theory of construction hazard recognition from an eye-brain synergy perspective and provides guidance for developing intuitive hazard recognition devices and improving management performance.
Practical Applications
This study combined eye-movement data and EEG data to predict the performance of hazard recognition using multimodal data. Not only did this study reveal the mechanism of hazard recognition, but it also proposed suggestions for project management and device development. Managers are suggested to implement refined management and regularly monitor the physiological signals of employees. Some construction companies started to use smart devices to measure workers’ arousal, emotional valence, emotional regulation ability, fatigue, workload, or safety attention level before work every day. And it is recommended to provide safety reminders to employees based on whether there are any abnormalities in their physiological indicators. In addition to project management, the results of this study can serve the development or optimization of hazard recognition equipment. Developing devices cannot only consider EEG information, but also combine EEG information with eye-movement information. Meanwhile, developers can also consider different types of hazards and different brain regions to improve device performance.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or codes that support the findings of this study are available from the corresponding author upon reasonable request (data of EEG signals; data of the information of participants).
Acknowledgments
The authors thank the National Natural Science Foundation of China (No. 51878382) for supporting this study.
Author contributions: Jiaming Wang completed the main work of the thesis. Mingxuan Liang finished data analysis and Pin-Chao Liao directed this research as the supervisor.
References
Albert, A., M. R. Hallowell, M. Skaggs, and B. Kleiner. 2017. “Empirical measurement and improvement of hazard recognition skill.” Saf. Sci. 93 (Mar): 1–8. https://doi.org/10.1016/j.ssci.2016.11.007.
Angelidis, A., M. Hagenaars, D. van Song, W. van der Does, and P. Putman. 2018. “Do not look away! Spontaneous frontal EEG theta/beta ratio as a marker for cognitive control over attention to mild and high threat.” Int. J. Behav. Med. 25 (May): S66–S66. https://doi.org/10.1016/j.biopsycho.2018.03.002.
Boucsein, W., A. Haarmann, and F. Schaefer. 2007. “Combining skin conductance and heart rate variability for adaptive automation during simulated IFR flight.” In Proc., Engineering Psychology and Cognitive Ergonomics, 639–647. Berlin: Springer.
Bulling, A., J. A. Ward, H. Gellersen, and G. Troster. 2011. “Eye movement analysis for activity recognition using electrooculography.” IEEE Trans. Pattern Anal. Mach. Intell. 33 (4): 741–753. https://doi.org/10.1109/TPAMI.2010.86.
Calvo, R. A., and S. D’Mello. 2010. “Affect detection: An interdisciplinary review of models, methods, and their applications.” IEEE Trans. Affective Comput. 1 (1): 18–37. https://doi.org/10.1109/T-AFFC.2010.1.
Chen, J., J. E. Taylor, and S. Comu. 2017. “Assessing task mental workload in construction projects: A novel electroencephalography approach.” J. Constr. Eng. Manage. 143 (8): 04017053. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001345.
Cheon, B., C. K. Kim, and D. S. Kwon. 2022. “Intuitive endoscopic robot master device with image orientation correction.” Int. J. Med. Rob. Comput. Assisted Surg. 18 (5): e2415. https://doi.org/10.1002/rcs.2415.
Davidson, P. R., R. D. Jones, and M. T. R. Peiris. 2007. “EEG-based lapse detection with high temporal resolution.” IEEE Trans. Biomed. Eng. 54 (5): 832–839. https://doi.org/10.1109/TBME.2007.893452.
Degno, F., and S. P. Liversedge. 2020. “Eye movements and fixation-related potentials in reading: A review.” Vision 4 (1): 11. https://doi.org/10.3390%2Fvision4010011.
Devillez, H., N. Guyader, and A. Guérin-Dugué. 2015. “An eye fixation-related potentials analysis of the P300 potential for fixations onto a target object when exploring natural scenes.” J. Vision 15 (13): 20. https://doi.org/10.1167/15.13.20.
Dimigen, O., and B. V. Ehinger. 2021. “Regression-based analysis of combined EEG and eye-tracking data: Theory and applications.” J. Vision 21 (1): 3. https://doi.org/10.1167/jov.21.1.3.
D’Mello, S. K., and J. Kory. 2015. “A review and meta-analysis of multimodal affect detection systems.” ACM Comput. Surv. 47 (3): 1–36. https://doi.org/10.1145/2682899.
Eoh, H. J., M. K. Chung, and S. H. Kim. 2005. “Electroencephalographic study of drowsiness in simulated driving with sleep deprivation.” Int. J. Ind. Ergon. 35 (4): 307–320. https://doi.org/10.1016/j.ergon.2004.09.006.
Frey, A., B. Lemaire, L. Vercueil, and A. Guerin-Dugue. 2018. “An eye fixation-related potential study in two reading tasks: Reading to memorize and reading to make a decision.” Brain Topogr. 31 (4): 640–660. https://doi.org/10.1007/s10548-018-0629-8.
Fu, R. R., H. Wang, and W. B. Zhao. 2016. “Dynamic driver fatigue detection using hidden Markov model in real driving condition.” Expert Syst. Appl. 63 (Nov): 397–411. https://doi.org/10.1016/j.eswa.2016.06.042.
Gao, R., B. Mu, S. Lyu, H. Wang, and C. Yi. 2022. “Review of the application of wearable devices in construction safety: A bibliometric analysis from 2005 to 2021.” Buildings 12 (3): 344. https://doi.org/10.3390/buildings12030344.
Golparvar, A. J., and M. K. Yapici. 2018. “Graphene-coated wearable textiles for EOG-based human-computer interaction.” In Proc., 2018 IEEE 15th Int. Conf. on Biomedical and Health Informatics (BHI) and the Wearable and Implantable Body Sensor Networks (BSN), 189–192. New York: IEEE.
Guo, B. H. W., Y. Zou, Y. H. Fang, Y. M. Goh, and P. X. W. Zou. 2021. “Computer vision technologies for safety science and management in construction: A critical review and future research directions.” Saf. Sci. 135 (Mar): 105130. https://doi.org/10.1016/j.ssci.2020.105130.
Hasanzadeh, S., B. Esmaeili, and M. D. Dodd. 2017. “Measuring the impacts of safety knowledge on construction workers’ attentional allocation and hazard detection using remote eye-tracking technology.” J. Manage. Eng. 33 (5): 04017024. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000526.
Healy, G., and A. F. Smeaton. 2011. “Eye fixation related potentials in a target search task.” In Proc., 2011 Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (EMBC), 4203–4206. New York: IEEE.
Hu, Z., W. T. Chan, H. Hu, and F. Xu. 2023. “Cognitive factors underlying unsafe behaviors of construction workers as a tool in safety management: A review.” J. Constr. Eng. Manage. 149 (3): 03123001. https://doi.org/10.1061/JCEMD4.COENG-11820.
Humphreys, H. C., Y. M. Choi, and W. J. Book. 2023. “Advanced patient transfer assist device with intuitive interaction control.” Assist. Technol. 35 (1): 3–13. https://doi.org/10.1080/10400435.2017.1396564.
Jeelani, I., A. Albert, and K. Han. 2020. “Improving safety performance in construction using eye-tracking, visual data analytics, and virtual reality.” In Proc., Construction Research Congress 2020: Safety, Workforce, and Education, 395–404. Reston, VA: ASCE.
Jeon, J., and H. B. Cai. 2021. “Classification of construction hazard-related perceptions using: Wearable electroencephalogram and virtual reality.” Autom. Constr. 132 (Dec): 103975. https://doi.org/10.1016/j.autcon.2021.103975.
Kamienkowski, J. E., M. J. Ison, R. Q. Quiroga, and M. Sigman. 2012. “Fixation-related potentials in visual search: A combined EEG and eye tracking study.” J. Vision 12 (7): 4 https://doi.org/10.1167/12.7.4.
Kazai, K., and A. Yagi. 2003. “Comparison between the lambda response of eye-fixation-related potentials and the P100 component of pattern-reversal visual evoked potentials.” Cognit. Affective Behav. Neurosci. 3 (1): 46–56. https://doi.org/10.3758/CABN.3.1.46.
Kelly, S. P., and R. G. O’Connell. 2013. “Internal and external influences on the rate of sensory evidence accumulation in the human brain.” J. Neurosci. 33 (50): 19434–19441. https://doi.org/10.1523/JNEUROSCI.3355-13.2013.
Kosmyna, N., C. Morris, T. Nguyen, S. Zepf, J. Hernandez, P. Maes, and M. Assoc Comp. 2019. “AttentivU: Designing EEG and EOG compatible glasses for physiological sensing and feedback in the car.” In Proc., AUTOMOTIVEUI’19: Proc., 11th ACM Int. Conf. On Automotive User Interfaces and Interactive Vehicular Applications, 355–368. New York: Association for Computing Machinery.
Kubota, Y., W. Sato, M. Toichi, T. Murai, T. Okada, A. Hayashi, and A. Sengoku. 2001. “Frontal midline theta rhythm is correlated with cardiac autonomic activities during the performance of an attention demanding meditation procedure.” Cognit. Brain Res. 11 (2): 281–287. https://doi.org/10.1016/S0926-6410(00)00086-0.
Lin, C. T., R. C. Wu, S. F. Liang, W. H. Chao, Y. J. Chen, and T. P. Jung. 2005. “EEG-based drowsiness estimation for safety driving using independent component analysis.” IEEE Trans. Circuits Syst. I Regul. Pap. 52 (12): 2726–2738. https://doi.org/10.1109/TCSI.2005.857555.
Lopez-Duran, N. L., R. Nusslock, C. George, and M. Kovacs. 2012. “Frontal EEG asymmetry moderates the effects of stressful life events on internalizing symptoms in children at familial risk for depression.” Psychophysiology 49 (4): 510–521. https://doi.org/10.1111/j.1469-8986.2011.01332.x.
Ma, J. X., L. C. Shi, and B. L. Lu. 2010. “Vigilance estimation by using electrooculographic features.” In Proc., 2010 Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (EMBC), 6591–6594. New York: IEEE.
Ma, Q., X. Bai, G. Pei, and Z. Xu. 2018. “The hazard perception for the surrounding shape of warning signs: Evidence from an event-related potentials study.” Front. Neurosci. 12 (Nov): 824. https://doi.org/10.3389/fnins.2018.00824.
Mainsah, B. O., G. Reeves, L. M. Collins, and C. S. Throckmorton. 2017. “Optimizing the stimulus presentation paradigm design for the P300-based brain-computer interface using performance prediction.” J. Neural. Eng. 14 (4): 046025. https://doi.org/10.1088/1741-2552/aa7525.
Morillas-Romero, A., M. Tortella-Feliu, X. Bornas, and P. Putman. 2015. “Spontaneous EEG theta/beta ratio and delta-beta coupling in relation to attentional network functioning and self-reported attentional control.” Cognit. Affective Behav. Neurosci. 15 (3): 598–606. https://doi.org/10.3758/s13415-015-0351-x.
Morris, T. L., and J. C. Miller. 1996. “Electrooculographic and performance indices of fatigue during simulated flight.” Biol. Psychol. 42 (3): 343–360. https://doi.org/10.1016/0301-0511(95)05166-X.
Nasution, D., T. Henny, F. Harumy, E. Haryanto, F. Fachrizal, F. Julham, and A. Turnip. 2015. “A Classification Method for Prediction of Qualitative Properties of Multivariate EEG-P300 Signals.” In Proc., Int. Conf. on Automation, Cognitive Science, Optics, Micro Electro-Mechanical System, and Information Technology. New York: IEEE.
Noghabaei, M., K. Han, and A. Albert. 2021. “Feasibility study to identify brain activity and eye-tracking features for assessing hazard recognition using consumer-grade wearables in an immersive virtual environment.” J. Constr. Eng. Manage. 147 (9): 04021104. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002130.
Pavithran, P. G., K. Arunkumar, N. P. G. Seshadri, B. K. Singh, V. Mahesh, and B. Geethanjal. 2019. “Index of theta/alpha ratio to quantify visual-spatial attention in dyslexics using electroencephalogram.” In Proc., 2019 5th Int. Conf. on Advanced Computing & Communication Systems (ICACCS), 417–422. New York: IEEE.
Putman, P., J. van Peer, I. Maimari, and S. van der Werff. 2010. “EEG theta/beta ratio in relation to fear-modulated response-inhibition, attentional control, and affective traits.” Biol. Psychol. 83 (2): 73–78. https://doi.org/10.1016/j.biopsycho.2009.10.008.
Schleiger, E., N. Sheikh, T. Rowland, A. Wong, S. Read, and S. Finnigan. 2014. “Frontal EEG delta/alpha ratio and screening for post-stroke cognitive deficits: The power of four electrodes.” Int. J. Psychophysiol. 94 (1): 19–24. https://doi.org/10.1016/j.ijpsycho.2014.06.012.
Shang, Q., Y. Huang, and Q. Ma. 2015. “Hazard levels of warning signal words modulate the inhibition of return effect: Evidence from the event-related potential P300.” Exp. Brain Res. 233 (9): 2645–2653. https://doi.org/10.1007/s00221-015-4335-4.
Shi, L. C., R. N. Duan, and B. L. Lu. 2013. “A robust principal component analysis algorithm for EEG-based vigilance estimation.” In Proc., 2013 35th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (EMBC), 6623–6626. New York: IEEE.
Vařeka, L., and S. Ladouce. 2021. “Prediction of navigational decisions in the real-world: A visual P300 event-related potentials brain-computer interface.” Int. J. Hum.-Comput. Interact. 37 (14): 1375–1389. https://doi.org/10.1080/10447318.2021.1888510.
Wang, M. H., Y. Y. Zhao, and P. C. Liao. 2022. “EEG-based work experience prediction using hazard recognition.” Autom. Constr. 136 (Apr): 104151. https://doi.org/10.1016/j.autcon.2022.104151.
Xu, Q., H.-Y. Chong, and P.-C. Liao. 2019. “Exploring eye-tracking searching strategies for construction hazard recognition in a laboratory scene.” Saf. Sci. 120 (Dec): 824–832. https://doi.org/10.1016/j.ssci.2019.08.012.
Zander, T. O., C. Kothe, S. Jatzev, M. Gaertner, and A. Nijholt. 2010. “Enhancing human-computer interaction with input from active and passive brain-computer interfaces.” In Brain-computer interfaces: Applying our minds to human-computer interaction, 181–199. Dordrecht, Netherlands: Springer.
Zhang, J., B. Wang, C. Zhang, Y. Xiao, and M. Y. Wang. 2019. “An EEG/EMG/EOG-based multimodal human-machine interface to real-time control of a soft robot hand.” Front. Neurorobot. 13 (Mar): 7. https://doi.org/10.3389/fnbot.2019.00007.
Zhou, X., and P.-C. Liao. 2023. “EEG-based performance-driven adaptive automated hazard alerting system in security surveillance support.” Sustainability 15 (6): 4812. https://doi.org/10.3390/su15064812.
Zhou, X. S., Y. N. Hu, P. C. Liao, and D. Zhang. 2021. “Hazard differentiation embedded in the brain: A near-infrared spectroscopy-based study.” Autom. Constr. 122 (Feb): 103473. https://doi.org/10.1016/j.autcon.2020.103473.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 150 • Issue 5 • May 2024
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Feb 16, 2023
Accepted: Nov 27, 2023
Published online: Feb 28, 2024
Published in print: May 1, 2024
Discussion open until: Jul 28, 2024
ASCE Technical Topics:
- Analysis (by type)
- Business management
- Construction engineering
- Construction equipment
- Construction management
- Disaster risk management
- Disasters and hazards
- Employment
- Engineering fundamentals
- Equipment and machinery
- Human and behavioral factors
- Labor
- Personnel management
- Power spectral density
- Practice and Profession
- Risk management
- Statistical analysis (by type)
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.