Efficient Online Engagement Analytics Algorithm Toolkit That Can Run on Edge

Abstract

The rapid expansion of video conferencing and remote works due to the COVID-19 pandemic has resulted in a massive volume of video data to be analyzed in order to understand the audience engagement. However, analyzing this data efficiently, particularly in real-time, poses a scalability challenge as online events can involve hundreds of people and last for hours. Existing solutions, especially open-sourced contributions, usually require dedicated and expensive hardware, and are designed as centralized cloud systems. Additionally, they may also require users to stream their video to remote servers, which raises privacy concerns. This paper introduces scalable and efficient computer vision algorithms for analyzing face orientation and eye blink in real-time on edge devices, including Android, iOS, and Raspberry Pi. An example solution is presented for proctoring online meetings, workplaces, and exams. It analyzes audiences on their own devices, thus addressing scalability and privacy issues, and runs at up to 30 fps on a Raspberry Pi. The proposed face orientation detection algorithm is extremely simple, efficient, and able to estimate the head pose in two degrees of freedom, horizontal and vertical. The proposed Eye Aspect Ratio (EAR) with simple adaptive threshold demonstrated a significant improvement in terms of false positives and overall accuracy compared to the existing constant threshold method. Additionally, the algorithms are implemented and open sourced as a toolkit with modular, cross-platform MediaPipe Calculators and Graphs so that users can easily create custom solutions for a variety of purposes and devices.

1. Introduction

Scholars have been predicting the rise of video conferences and virtual events for many decades. In 2020, COVID-19 accelerated video conferencing, online education, and online conducts to a whole new level and many solutions such as Zoom, Microsoft Teams and other video meeting software have emerged. Zoom, for instance, increased from 10 million daily users in December, 2019 to 300 million in April, 2020 [1], especially due to COVID-19-driven travel constraints in countries around the world. Video conferencing today has become the norm and it helps bring the remote partners much closer than personal meetings.

1.1. The Future of Workplaces and Meetings

Additionally, the COVID-19 pandemic has resulted in a rapid shift towards remote work for many organizations and employees. According to a report by Organization for Economic Co-operation and Development (OECD), industries that are highly digitalized, such as information and communication services, professional, scientific and technical services, and financial services, saw the greatest increase in teleworking, with over 50% of employees working remotely on average [2]. This trend has been driven by the need to reduce the spread of the virus by limiting in-person contact, as well as by the increasing availability of technology that enables remote work.

Although the trend was largely driven by the pandemic, it is likely to shape the way we work in the future. One reason for this is that online meetings and remote work offer a number of benefits that are likely to continue to be valued even after the pandemic subsides [3]. They can also enable more flexible and agile ways of working, which can increase productivity and enable organizations to respond more quickly to changing needs and conditions [4]. In addition, online meetings, classes and remote work can reduce the need for travel, which can save time and money and reduce the environmental impact of work [5,6].

Another reason is that the shift towards online meetings and remote work has required organizations to invest in new technologies and infrastructure as well as cybersecurity measures that are likely to continue to be used even after the pandemic subsides and will continue to pay dividends in terms of increased productivity and efficiency [7].

1.2. The Problems

A qualitative study has shown that remote work can lead to feelings of isolation and disconnection from colleagues and the broader organization [8]. This can be particularly challenging for those who are not used to working remotely or who lack strong social connections outside of work [9]. Additionally, a study [10] found that students also have difficulties with the shift from hands-on laboratory experiences to online classes, which resulted in loss of communication among peers, which is important for understanding, engagement and persistence.

Given the nature of video conferences, the audience, especially the meeting hosts, suffers from the stress which is coined as “Zoom Fatigue”. According to the studies [11], it can be accounted for the fact that the audience have to turn on their cameras and have close engagement with others for long hours and as for the meeting hosts, online engagement creates social isolation in [12]. Additionally, having the audience unnecessarily turn on their video feed all the time can have an adverse effect on the engagement itself as well as on privacy concerns.

Bailenson [11] made theoretical arguments on four causes for Zoom fatigue: close eye contact, cognitive load due to harder non-verbal cues, self-consciousness from looking at one’s self image and reduced mobility.

Additionally, he and other researchers [13] conducted an empirical study on the scale (called Zoom Exhaustion and Fatigue Scale, ZEF) where they outlined general facts regarding Zoom Fatigue. Also, the same researchers [14] conducted an empirical test on five non-verbal causes, similar to Bailenson’s [11] arguments. From [11,13,14], one can argue that video-first online conference settings significantly contributes to Zoom fatigue and the fatigue leads to negative attitudes towards online meeting, making matters worse and disrupting the engagement as a whole.

A study in Thailand [15], for instance, revealed that teachers have difficulties with gauging student engagement. One potential solution to the stress associated with video sharing is to instead share engagement analytics such as face orientation and eye blink, which can address the causes of Zoom Fatigue by reducing the need for video sharing. For meeting hosts too, this can provide assurance of audience attentiveness, even without visual access to the attendees’ faces.

1.3. Understanding Audience

Having said that, video plays a major role in evaluating the progress of the online meeting as it reveals how an audience engages with the content, giving the meeting hosts insight and confidence. For the employers, examiners, teachers, and online meeting hosts, the need to understand the behavior and attentiveness of the online audience naturally arises.

First, understanding the behavior and attentiveness of the audience can help to improve the effectiveness of the meeting or exam. One study [16] observed that individuals who pay attention to the information are more likely to retain it and perform better on tasks compared to those who are likely to be distracted (also [17,18]). By using video analytics to track attentiveness, employers, examiners, and teachers can identify any issues that may be impacting the ability of the audience to engage with the material, and take steps to address them.

Second, understanding the behavior and attentiveness of the audience can help to ensure the integrity of the meeting or exam. For example, in an online exam setting, video analytics can be used to detect and prevent cheating by identifying behaviors that are inconsistent with an individual working alone [19].

1.4. Concerns

As online meetings and exams become increasingly reliant on video analytics to understand the behavior and attentiveness of the audience, it is important to consider the concerns that the audience might have about these analytics being applied to their video feed.

One concern that the audience might have is about privacy [20]. With online meetings and remote work, it is often necessary to share personal information and data, such as one’s location, webcam feed, and computer screen. This can raise concerns about data security and the risk of data breaches and the policies to protect privacy. Many people are understandably concerned about their data being collected, shared, or used in ways that they did not consent to. A qualitative study conducted in Japan found that students who participated in online classes were comfortable with showing their faces on camera, but they were hesitant to be recorded due to concerns that the recordings might be shared publicly [21]. To address this concern, it is important for online meeting hosts and employers to provide clear information about how the analytics will be used, and to give the audience control over their data. This might include options for opt-in or opt-out, as well as clear information about what data will be collected and how it will be used.

There is also a risk that the analytics could be biased and used for nefarious purposes, such as surveillance or control. To address this concern, it is important for online meeting hosts and employers to be transparent about how the analytics will be used and to ensure that they are not being used for purposes that are not in the best interests of the audience.

1.5. Existing Proctoring Solutions

The authors of [22] conducted an extensive review of 29 premium, commercial exam proctoring solutions, evaluating the features and platforms of each solution. These solutions often utilize multi-modal systems, which may involve multiple cameras or specialized hardware, and are usually centralized web platforms. Most of these solutions offer advanced performance and are not open source. They typically take control of the examinee’s computer system while executing analytics. However, these solutions may raise concerns about privacy and ethics due to their complexity and lack of transparency [20]. A systematic review on similar software is provided in [6,23].

Likewise, many transparent solutions (with research published) utilize a cloud-based multi-modal approach, incorporating data from multiple sources. One example of such a system [24] uses a wearable camera and microphone, as well as a webcam, to evaluate various factors such as gaze, speech, text, and phone screen presence. Other research [25] has focused on analyzing behavior such as facial expressions, eye and mouth movements to detect cheating. In conclusion, proctoring automation requires a variety of factors and imposes certain constraints on examinees in order to operate accurately.

Additionally, the authors researched open-source tools for facial analytics in proctoring. The open-source projects, in general, can be categorized into general and proctoring-specific. General tools such as OpenCV and Tensorflow can be used as a foundation to build proctoring solutions but may lack specificity, requiring extra effort for proctoring purposes. Proctoring-specific tools, on the other hand, come as a complete package or solution with dashboard, etc. They usually are highly specialized and do not offer much customization. They may not be portable as they are implemented in Python [26], and for web-based deployment [27,28]. Some proctoring features normally require dedicated hardware such as Graphical Processing Unit (GPU) to perform optimally in real-time, adding further constraints on scalability and portability. Being open-source project, they may not provide all of the features and infrastructure that premium proctoring services offer.

1.6. A Toolkit for Proctoring

The aim of this paper is to propose an open-source initiative to make proctoring-related algorithms accessible, portable, scalable and customizable, as a toolkit. Instead of creating a complete proctoring solution, the authors aim to promote customizablility, transparency, and alleviate concerns about proctoring tools, as discussed in [20]. Additionally, the proposed algorithms are designed to run directly on consumer devices, which can be called edge, in real-time, eliminating the need to send video streams or recorded videos to a remote server. This helps protect the privacy of end users. For organizations and facilitators, it can reduce the need for dedicated hardware and infrastructure, as the algorithms can extract features at the edge. This addresses scalability issues.

As of the writing, the proposed toolkit is able to detect face orientation and eye blink. Those features are important for a variety of purposes, including proctoring, as they can provide valuable insights into the behavior and attentiveness of individuals. For example, detecting face orientation can help to determine whether someone is looking directly at the screen or camera, which can be useful for detecting cheating during online exams or for ensuring that employees are paying attention during online meetings. Eye blink detection can also be useful for assessing attentiveness, as prolonged periods of inactivity or low blink rate can be indicators of fatigue or lack of engagement.

Similar to the proposed toolkit, the OpenFace Toolkit [29] is an open-source project that offers libraries and tools for facial behavior analysis. It is written in C++ and Python and is designed to be cross-platform, utilizing the CMake build system to run on Windows, Linux, and macOS. However, certain features based on dlib and TBB may not be compatible with mobile devices such as iOS and Android, and some prior knowledge of CMake is required to build for these platforms. In contrast, the authors’ toolkit uses the out-of-the-box build configuration provided by the MediaPipe Framework to easily support a wider range of devices.

1.7. MediaPipe

MediaPipe is an opensource framework developed by Google for building cross-platform multimodal machine learning pipelines [30]. It allows developers to build and deploy solutions for a variety of applications, including augmented reality, video analytics, and gesture recognition.

One of the key features of MediaPipe is its modular design, which allows developers to easily incorporate a wide range of machine learning models and algorithms into their pipelines. MediaPipe is built on the concept of “Calculators”, which are modular components that can be easily combined to create custom pipelines for a variety of applications. This makes it easy to customize and extend existing solutions, or to build new ones from scratch. The proposed algorithms have been implemented as separate Calculator components, allowing them to be easily integrated into the MediaPipe ecosystem.

MediaPipe also includes support for a variety of input and output modalities, including video, audio, and sensor data. This makes it well-suited for applications that require the integration of multiple data streams, such as those found in augmented reality, video analytics, multi-modal proctoring or the online engagement analytics.

In addition to its flexibility and modularity, MediaPipe is also highly efficient and scalable. It is designed to run on a variety of platforms, including mobile devices and edge devices, and is optimized for performance and real-time processing. This allows developers to bring machine learning to the edge, meaning that models can be run directly on device hardware rather than being required to be run on a remote server or cloud infrastructure. This makes it possible to build and deploy models that are highly efficient and responsive, making it an attractive choice for a wide range of custom applications. The proposed algorithms by the authors, for instance, use the landmarks detected by FaceMesh solution [31] of MediaPipe framework.

1.8. Contributions

The main contribution of the paper is the initiative for an open-source proctoring toolkit [32], which supports face orientation, eye blink detection and other trivial analytics. The authors proposed an extremely simple yet efficient face orientation detection approach. Additionally, the authors improved the F1 score and accuracy of the existing Eye Aspect Ratio (EAR) thresholding approach on eye blink detection with a simple non-linear regression-based adaptive threshold. Last but not least, they statistically proved that face orientation indeed affects the perceived eye closeness; given the adaptive threshold based on face orientation, a single eyelid distance can outperform EAR thresholding method.

2. Related Works

The proposed toolkit is comprised of landmark standardization, face orientation, and eye blink detection; the following sub-sections explain the justifications.

2.1. Landmark Standardization

The most straight-forward standardization is the scaling normalization where the coordinates are normalized into a value range of 0 and 1, based on the max and min values (Equation (1)). However, this method can correlate to the face size compared to z-score standardization.

$$x_{norm}=\frac{x-x_{min}}{x_{max}}$$

(1)

Z-score standardization is a statistical method that transforms a variable to have a mean of 0 and a standard deviation of 1. This can be useful for comparing values of the variable across different samples or populations, or for comparing the variable to a normal distribution.

In the context of facial analysis, z-score standardization can be applied to facial landmarks, which are points on the face that correspond to specific features such as the eyes, nose, and mouth. By standardizing the coordinates of these landmarks, it is possible to compare the relative positions of the landmarks across different faces or to detect subtle changes.

Research in [33] evaluated the effect of standardizing imaging features on the accuracy of a radiomics analysis for predicting the histology of non-small cell lung cancer (NSCLC). A total of 476 imaging features were extracted from each database and standardized using min-max normalization, z-score normalization, and whitening from principal component analysis. The results showed that all of the standardization strategies improved the accuracy of the histology prediction, with the highest accuracy achieved using z-score normalization.

Another case of using z-score standardization as explained in [34] occurs in measuring telomere lengths across different studies or runs, because raw values of rTL (Relative Telomere Length) can be influenced by technical and contextual factors, and are not directly comparable. Z-score standardization helped to mitigate this problem by expressing the data in standard deviations, which are directly comparable between studies.

Likewise, un-normalized landmarks can be influenced by many factors such as input image dimension, size of the face, and detected location. This can affect the robustness of algorithms that are designed to work with various devices and people. In MediaPipe Facemesh [31], the landmarks are normalized using the width and height of the image, which again may not be ideal for comparative purposes.

In the context of machine learning, comparing K-means clustering results on infectious diseases datasets showed that the z-score standardization method is more effective and efficient than the min-max and decimal scaling standardization methods [35]. A study in [36], on the other hand, pointed out that conventional z-score standardization may not be optimal for certain clustering analyses, along with the performances of modified z-score standardizations.

Overall, [33,34,35,36], strongly suggest that z-score is the optimal, robust standardization for meta analysis of facial landmarks.

2.2. Head Pose Estimation

Proctoring involves identifying whether an audience is looking straight at the computer screen or not, in order to assess their behavior, attentiveness, and integrity during online conduct. This is often achieved by analyzing the head pose, or the Euler angles (yaw, pitch, and roll) of the head in 3D space. Many studies have focused on estimating head pose, especially from visual data, for a variety of applications, including driver fatigue detection [37,38,39]. Head pose has also been studied for proctoring purposes in previous research [40,41,42].

Deep learning-based approaches for estimating head pose can be found in previous research [38,40,43]. The authors of [44], for instance, used a CNN [45] to detect the head pose. However, they took a different approach; they z-score standardize the yaw and pitch to accommodate different settings based on an assumption that they follow normal distribution, as in the case of face landmarks in this paper. Other studies have used non-deep learning approaches, such as localized gradient orientation (LGO) histograms with support vector regressors (SVRs), to detect not only head pose but also position in 3D space [39]. In another proctoring solution [42], the head pose was estimated using an extended version of the Constrained Local Model.

The survey in [46] documents several methods for estimating head pose, including geometric methods that use facial landmarks. For example, ref. [41] described a method for calculating head pose from cylindrical and ellipsoidal face models (proposed in [47]).

An example of a method for estimating head pose using facial landmarks can be found in the work of [48]. They used Euler angles to estimate head pose by solving the Point-n-Perspective (PnP) problem with methods such as POSIT (Pose from Orthography and Scaling with ITerations). This solution requires a reference 3D face model and camera parameters, and is most optimal when the real camera parameters and the input face closely resemble the references [45,49]. Another study [37] used Efficient PnP (EPnP) for head pose estimation in driver fatigue detection.

This paper, however, presents an algorithm that estimates face orientation using two degrees of freedom (horizontal and vertical) for proctoring purposes. A similar approach for detecting face orientation can be found in a previous study [50], which used five separate AdaBoost classifiers to classify five discrete face orientations (left, right, up, down, straight) directly from raw pixels. In contrast, the authors of this paper used only a single nose landmark to evaluate continuous horizontal and vertical orientations. The algorithm is reliable as long as the face landmark detection is accurate, which is ensured by the underlying MediaPipe FaceMesh solution [31], which has been demonstrated to be accurate and is widely used in production.

2.3. Eye Blink Detection

Eye blink detection is important in various fields of research, including facial anti-spoofing, physical and mental health assessment, and human-computer interaction. By accurately detecting and analyzing eye blinks, researchers can gain insights into a person’s cognitive and emotional states, detect potential security threats, and design more intuitive and user-friendly interfaces.

A vision-based human–computer interface that uses eye blinks as control commands on a consumer computer is described in [51]. The detection of Deepfakes using eye blink patterns, as demonstrated in [52,53], highlights the utility of eye blink detection for proctoring purposes. Research on drowsiness detection [54] and eye fatigue detection [55] show that it is also useful for detecting physical and mental health problems and analyzing attentiveness and behavior.

In visual systems, traditional image processing techniques are used to detect eye blink. The proposed interface in [51] only uses image processing techniques, including Haar-like features for face detection and template matching for eye tracking and eye blink detection, to detect and interpret eye blinks. The system requires template images of the user’s eyes for initialization unlike deep learning models. The author of [56] utilized deep Convolutional Neural Network (CNN) to detect eye blinks. Apart from a deep learning approach, the authors of [57] used Active Shape Model (ASM) to extract the facial features for eye blink detection. The usage of facial landmarks for eye blink detection can be found in [58,59,60].

Among sensor-based approaches, a study [61] developed a system called BlinkListener for detecting eye blinks using acoustic signals (chirps signals from a speaker) in a contact-free manner. The authors of [62,63] discuss the impact of eye blink artifacts on EEG data and the development of algorithms for detecting and correcting these artifacts. Eye blinking EOG signals were incorporated to improve existing EEG-based biometric authentication [64]. The authors of [65,66,67], used EOG sensors to track eye movement and detect eye blinks. These methods sometimes require dedicated hardware in order to detect eye blink and require EOG goggles, which may not suitable for the general public.

In this paper, the authors chose the facial landmarks approach for detecting eye blink as it saves computation time since face orientation detection already relies on them. Regarding the eye blink detection using facial landmarks, the authors of [58] proposed a well-known metric, eye aspect ratio (EAR, Equation (2)) computed from six salient landmarks around the eye (as shown in Figure 1) to detect whether a person’s eyes are open or closed.

$$EA{R}_{left}=\frac{\left|\right|L^{160}-L^{144}{\left|\right|}_2^2+\left|\right|L^{158}-L^{153}{\left|\right|}_2^2}{2\left|\right|L^{33}-L^{133}{\left|\right|}_2^2}$$

(2)

$$EA{R}_{right}=\frac{\left|\right|L^{385}-L^{380}{\left|\right|}_2^2+\left|\right|L^{387}-L^{373}{\left|\right|}_2^2}{2\left|\right|L^{362}-L^{263}{\left|\right|}_2^2}$$

(3)

An alternative approach, as demonstrated in [57], is that eye blinks can be detected by simply calculating the average of three vertical eyelid distances. The authors of [59] improved the robustness of a single eyelid distance in eye blink detection by smoothing the distance signal with a Savitzky-Golay (SG) filter and taking the duration of the eye blink into consideration. Additionally, according to the experimental results presented in this paper, a single vertical eyelid distance can perform comparably and better in terms of F1 score to the Eye Aspect Ratio (EAR) method (see the results in Section 4.2) in detecting eye blinks if the adaptive threshold of eye blink is calculated from face orientation. As suggested in [68], the head orientation indeed correlates to perceived eye openness and the correlation is statistically proven and explained by a non-linear regression (see Section 3.3).

Nonetheless, EAR is a commonly used method when it comes to detecting eye blinks using facial landmarks. A study [60], for instance, used eye aspect ratio to detect eye blinks with high accuracy in real-time on Raspberry Pi. The authors of [69] explored a mental fatigue assessment model for drivers using the eye aspect ratio and mouth aspect ratio.

3. Materials and Methods

The proposed algorithms are based on Facemesh solution [31] from Mediapipe framework and supports head orientation, eye blink and additionally, facial activity and movement detection. The Figure 2 depicts the algorithm (for the example application and source code, refer to Appendix C). First, facial landmarks from Facemesh are standardized before performing face orientation and eye blink detection. The facial activity is detected by calculating the difference between the standardized landmarks of the current image frame and the previous one. Likewise, face movement is calculated as the difference of the unstandardized nose landmark.

3.1. Landmark Standardization

A one-way ANOVA test (see Appendix A and the results in [32]) demonstrates that the distribution of facial landmarks is normal across different people given the same context, looking straight with neutral faces. This normality and the clinical results of [34] strongly suggest that z-score standardization can be more optimal than its counterpart, scaling normalization (Equation (1)). Additionally, standardized nose landmarks can differentiate face orientation in practice.

In conclusion, the authors utilized z-score normalization for robust head orientation and eye blink detection. Mediapipe’s facemesh solution offers 468 3D landmarks, $L:=\{x,y,z\}$ with $\{x,y,z\}$ where z is the distance from the camera to the face. In this case, the standardization becomes

$$L_{std}:=\{x_{std},y_{std},z_{std}\}$$

(4)

where $L_{std}$ is standardized landmark and is composed of $\{x_{std},y_{std},z_{std}\}$ being standardized principal axes. And $x_{std}$, for instance, is defined as

$$x_{std}=\frac{X-{\mu }_x}{{\sigma }_x}$$

(5)

where X is the x axis component of a landmark, ${\mu }_x$ the mean of all the x components and ${\sigma }_x$ the standard deviation.

3.2. Face Orientation

Since the authors are not modeling in a 3D context or analyzing behaviour, yaw and pitch can be sufficient for proctoring purposes. This is also pointed out by [44]. Therefore, the author proposes a method to estimate the face orientation with just two degrees of freedom (horizontal and vertical) directly from the nose; the significance of the nose in face orientation is also mentioned in [50,68].

The algorithm takes advantage of the simple pattern that the nose always points in the direction the head is facing. For example, if the head is facing left, the nose will be on the left side of the face. To detect this, the nose landmark must be compared to other facial landmarks, and standardization enables this comparison. The standardized nose landmark coordinate can be used directly as an indicator of face orientation without the need for further calculations and the value of zero means the nose at the center of the face, looking straight (see Figure 3 for reference). In that case, x of the standardized nose landmark represents the horizontal face orientation and y, vertical face orientation. Equations (6) and (7) presents a simple ad hoc threshold on standardized nose landmark to detect the head orientation.

$$A_{horizontal}=\left\{\begin{array}{cc}left,\hfill & \mathrm{if}{x}_{std}\le -0.3\hfill \\ right,\hfill & \mathrm{if}{x}_{std}\ge 0.3\hfill \\ straight,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(6)

$$A_{vertical}=\left\{\begin{array}{cc}up,\hfill & \mathrm{if}{y}_{std}\le 0.05\hfill \\ down,\hfill & \mathrm{if}{y}_{std}\ge 0.6\hfill \\ straight,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(7)

3.3. Blink Detection—Adaptive Threshold

The authors compare three methods for detecting eye blinks, EAR with constant threshold, eyelid distance with adaptive threshold and EAR with adaptive threshold. Both adaptive thresholds incorporate face orientation, which is the standardized landmark of the nose as mentioned in the previous section.

3.3.1. EAR with Constant Threshold

For constant EAR threshold, the author used 0.2 as in [58]. Given that $EAR$ is calculated using Equation (2) with unstandardized landmarks, an eye is considered blinking if

$$EAR<0.2.$$

(8)

3.3.2. Eyelid Distance with Adaptive Threshold

The vertical face orientation plays a statistically significant role in eye blink detection from eyelid distance as it alone accounted for 87.5% of the variation in the perceived eye openness threshold (see regression analysis in Figure 4). The author presents an adaptive threshold, Equation (9) that is derived from the regression in Figure 4.

$$T_y=-0.0228x+0.0162y+0.0792e^{y^2}$$

(9)

where $T_y$ denotes threshold function and y, the vertical face orientation. Then, the authors calculate the eye lid distances as in Equation (10) and (11) (for reference of the left eye landmarks, check Figure 5.

$$EL{D}_{left}=\left|\right|L_{std}^{159}-L_{std}^{145}{\left|\right|}_2^2$$

(10)

$$EL{D}_{right}=\left|\right|L_{std}^{386}-L_{std}^{374}{\left|\right|}_2^2$$

(11)

Given the corresponding $ELD$ and $T_y$, an eye is considered blinking if

$$ELD<T_y.$$

(12)

3.3.3. EAR with Adaptive Threshold

Unlike eyelid distance, both the horizontal and vertical face orientation play a statistically significant role in eye blink detection from EAR as 85.4% of the variation in the perceived eye openness threshold is explained by them (see regression analysis in Figure 6).

In this case, the adaptive threshold function is defined as

$$T_{x,y}^{EAR}=-0.0135y+0.1202y^2-0.0487x+0.0821e^{x^2}$$

(13)

where $T_y$ denotes threshold function. x and y denote the horizontal and vertical face orientation.

Given that $EAR$ is calculated using Equation (2) with standardized landmarks, an eye is considered blinking if

$$EAR<T_{x,y}^{EAR}.$$

(14)

3.3.4. Datasets

The authors run the algorithms against three different datasets, RT-BENE, Eyeblink8 and Talking Face, to test the performance and the robustness.

The RT-BENE (Real-Time Blink Extraction in Natural Environments) dataset [70] is a collection of videos captured in natural environments, aimed at providing a challenging and realistic testbed for blink detection algorithms. It includes individuals performing various activities, a wide range of lighting conditions and head poses, captured at 30 frames per second with a resolution of $720\times 576$ pixels, annotated with blink events, and providing information about head pose and lighting conditions. The dataset has a total of approximately 20,000 blink events and is widely used as benchmark dataset in blink detection research.

The Eyeblink8 dataset is considered to be more difficult to work with due to the presence of facial expressions, head movements, and instances of individuals looking down at a keyboard. It consists of 408 instances of both eyes blinking (2571 when counting each eye individually as detected by MediaPipe Facemesh) captured across 70,992 video frames, and the data was annotated by [71]. The videos have a resolution of $640\times 480$ and were captured at 30 frames per second, with an average length ranging from 5000 to 11,000 frames.

The Talking Face dataset features a single video recording of a subject speaking and displaying various facial expressions in front of the camera. The video was captured at 25 frames per second and has a resolution of $720\times 576$. The dataset includes 61 instances of both eyes blinking (valid 312 individual blinks) that have been annotated by [71,72].

4. Experimental Results

In this study, the performance of the face orientation estimation method is evaluated using the CMU dataset [73], which includes faces in four orientations (left, right, up and straight). For the eye blink detection, the adaptive threshold models were evaluated using three different datasets, RT-BENE [70], Eyeblink8 [71] and Talking Face [71,72]. The results of the experiments are reported using several commonly used metrics for classification models, such as F1 score, Precision, Recall, Accuracy and AUC-ROC (Area Under the Receiver Operating Characteristic Curve). Additionally, the runtime performance of the algorithms on Raspberry Pi 4 Model B is presented.

4.1. Face Orientation

The confusion matrix in Figure 7 shows the evaluation of face orientations on CMU Face dataset using the threshold function described in Equations (6) and (7) of Section 3.2. This extremely simple algorithm using a nose landmark achieved 94.78% on both Accuracy and F1-score on the CMU dataset. This is comparable to the algorithm in [50], which requires more complex feature extraction and classification.

4.2. Blink Detection

The authors evaluated eye blink detection algorithms, CEAR (EAR with Constant Threshold), AELD (Eyelid Distance with Adaptive Threshold) and AEAR (EAR with Adaptive Threshold), using three datasets: RT-BENE, Eyeblink8 and TalkingFace. Regression-based adaptive thresholds are fitted on the training part of RT-BENE dataset. The evaluations were conducted on the test part of the RT-BENE, and all videos were from Eyeblink8 and TalkingFace. The average performance of each algorithm or model can be found in

Table 1. Average Performance Metrics of Eye Blink Detection Approaches.

	Accuracy	F1	Precision	Recall	AUC-ROC
CEAR	89.59	30.49	19.34	73.27	81.59
AELD	97.59	53.14	46.43	62.53	80.44
AEAR	97.53	51.65	45.18	60.94	79.63

.

Based on the F1 score, the best-performing algorithm was AELD, with an average score of 53.14%. CEAR achieved an average score of 30.49%, while AEAR achieved an average score of 51.65%.

In terms of accuracy, AELD also achieved the highest average score of 97.59%, which was not significantly different from AEAR, which achieved an average score of 97.53%. CEAR achieved the lowest average score of 89.59%. However, CEAR achieved the highest average AUC-ROC score with 81.59% and the highest average Recall up to 73.27%.

Overall, the proposed adaptive threshold with face orientation saw a noticeable and consistent improvement in F1 score and accuracy. It is worth noting that single eyelid distance can perform on par with EAR, given the use of adaptive threshold and landmark standardization. All three models perform below average in terms of F1 score on Eyeblink8 dataset.

4.2.1. EAR with Constant Threshold

Using the constant threshold of 0.2 as in [58], on the RT-BENE dataset, CEAR has an accuracy of 89.23%, an F1 of 24.64%, and an AUC-ROC of 80.84%. On the Eyeblink8 dataset, the model has an accuracy of 84.80%, an F1 of 10.58%, and an AUC-ROC of 66.65%. On the TalkingFace dataset, the model has an accuracy of 94.74%, an F1 of 54.26%, and an AUC-ROC of 97.28%.

The model detected the most eye blinks out of all three datasets and has the highest recall and AUC-ROC scores. However, it performed poorly and was below average in terms of precision and F1 score. This indicates that the model has an imbalance between False Positives and False Negatives. The constant threshold of 0.2 has the highest trade-off between true positive rate and false positive rate, greedily detecting eye blinks. The detailed results on each dataset can be found in

Table 2. Results for EAR with constant threshold.

	Accuracy	F1	Precision	Recall	AUC-ROC
RT-BENE	89.23	24.64	14.86	72.02	80.84
Eyeblink8	84.80	10.58	5.94	47.8	66.65
TalkingFace	94.74	54.26	37.23	100	97.28

.

4.2.2. Eyelid Distance with Adaptive Threshold

Eyelid distance with adaptive threshold (Equation (9)) achieved better results in general according to the results in

Table 3. Results for eyelid distance with adaptive threshold.

	Accuracy	F1	Precision	Recall	AUC-ROC
RT-BENE	97.63	56.78	51.26	63.63	81.06
Eyeblink8	97	26.34	24.5	28.47	63.39
TalkingFace	98.15	76.31	63.53	95.51	96.87

. This simple eyelid distance threshold model performs the best in terms of accuracy and particulary, F1 score. Thus, it offers better balance between False Negatives and False Positives. Additionally, it performed well in terms of AUC-ROC score with an average of 80.44%, offering a good trade-off.

4.2.3. EAR with Adaptive Threshold

EAR with adaptive threshold (Equation (13)) also shows improvement in accuracy and F1 score, compared to CEAR. It achieved relatively similar accuracy and F1 score compared to the best, AELD. The experimental results can be found in

Table 4. Results for EAR with adaptive threshold.

	Accuracy	F1	Precision	Recall	AUC-ROC
RT-BENE	97.11	51.85	43.75	63.63	80.79
Eyeblink8	97.22	26.03	26.13	25.94	62.26
TalkingFace	98.27	77.08	65.68	93.26	95.85

.

4.3. Runtime Performance

The authors evaluated the performance of the example solution shown in Figure 2 using a sample video from [74] at two different resolutions, $1280\times 720$ (HD) and $1920\times 1080$ (FHD). The experiments were conducted on a Raspberry Pi 4 Model B with 1 GB RAM. The solution uses Mediapipe’s “FlowLimiterCalculator” component to achieve real-time low latency performance in practice, but the author disabled it during experimentation. The results, which can be found in Figure 8, show that the solution runs at an average of 30 frames per second on HD video and approximately 13 frames per second on FHD video.

5. Conclusions

To conclude, the authors have

• Proposed an initiative open-source proctoring toolkit [32] for online engagement analytics with face orientation and eye blink detection algorithms that can be used on consumer devices or edge devices
• Demonstrated the effectiveness of z-score standardization for facial landmarks
• Statistically proven the impact of face orientation on eye blink
• Improved F1 score and Accuracy for Eye Aspect Ratio (EAR) using adaptive threshold
• Proposed a simple and efficient face orientation detection.

Given the z-score standardization and adaptive threshold based on face orientation, eye blink detection with one vertical eyelid distance can achieve a better F1 score and accuracy than Eye Aspect Ratio (EAR) thresholding methods. The authors detect the individual eye blink based solely on facial landmarks. As demonstrated in [59], other factors such as timing and smoothing of eyelid distance signal can drastically improve the F1 score and the accuracy.

In the paper, the MediaPipe Facemesh [31] is utilized, which generates 468 facial landmarks. However, only the landmarks related to the eyes and nose were used in this research. Future work should include examination of z-score standardization’s impact when applied to fewer facial landmarks as in dlib [75].

It is important to note that in real-world proctoring and online engagement analytics, authentication and security measures are also crucial to ensure the integrity. Future work should include the integration of authentication mechanisms in the proposed toolkit to make it more secure and practical for real-world scenarios.