Augmented Reality-Based Framework Supporting Visual Inspection for Automotive Industry

Abstract

Visual inspection, inside an industrial environment, has attracted considerable research attention, in terms of its relation to improving productivity and its impact on building the Industry 4.0. One of the pillars of Industry 4.0 is the Augmented Reality (AR) technology, given its beneficiary for several tasks such as maintenance, assembly, and inspection. Nevertheless, such a data presenter tool essentially relies on the data collected from other modules. A keynote technology for data collecting, storing, and exploitation is the Industrial Internet of Things (IIoT) platform. In this context, this paper proposes an innovative framework in a real case-study industry. The proposed solution supports visual inspection, relying on AR to present the data and IIoT to collect it from the production line. User acceptance tests and feedback reflect the accuracy and effectiveness of the proposed system, especially when using Hand-Held Devices (HHD).

1. Introduction

Adopting the fourth industrial revolution (Industry 4.0) necessitates the use of a global and decentralized production technique [1]. This technique has to be mainly relied on fully-integrated collaborative manufacturing systems [2] presenting the concept of the Internet of Things (IoT) and real-time data transmission. The presence of such technology in the industrial network introduced the prominent concept of the Industrial Internet of Things (IIoT) which is one of the main pillars of Industry 4.0. IIoT refers to the collaboration of several components such as sensors, instruments, devices, computers, and even humans [3] allowing a better global visualization aiming at increased production [4] ensuring at the same time data security [5]. For the industry under study, which we keep anonymous for confidentiality reasons, emerging IIoT architecture should be considered for more standardized communication to implant Industry 4.0. Besides, the industry’s main purpose is the design and specification of a cognitive assistant Augmented Reality (AR)-based, which should be defined to ensure the presentation of information with a less cognitive load to the operator. To minimize the processing time during the production process, the automotive industry under study decided to implement the inspection system just on the final station of the assembly line. The reason behind the choice of that station precisely, is that it is the only station where the vehicle defects are handled and corrected. Traditionally, the inspection and correction operation is carried out by a human operator, who, by reading an identification document that follows the vehicle along the production process, identifies the vehicle present at the station, and shows it on a screen. The screen is located at the edge of the production line next to this station in order to display all defects to be corrected in the vehicle. The operator memorizes this information and then goes to the vehicle to rectify the non-conformities and returns thereafter to the computer to confirm, in the system, the correction, or non-correction, of the defects. Thus, this task remains arduous and is presenting a high probability of errors. A similar process (wiring) in [6] confirms as well that these types of tasks are time-consuming regardless of the level of proficiency of technicians. As a solution, the authors in [7] prove the ability of AR systems in solving efficiency problems such as insufficient training workers or complex tasks.

Therefore, for better organization and efficiency, a cognitive assistant system is proposed to quickly show the tasks to be performed after the vehicle has been recognized through the system. In this work, we have adopted Unity 3D for data visualization of the cognitive system given its flexibility and ability to build onto multiple platforms [8]. For data exploitation, we have acquired to Message Queuing Telemetry Transport (MQTT) [9] protocol via the IIoT platform given its messaging system simplicity, rapidity, and guaranteeing security [10]. The implementation process consists of the following steps: Firstly, a marker for vehicle identification is scanned and recognized. Secondly, the system brings the information related to the vehicle’s non-conformity via the IIoT platform in front of the operator in a User Interface (UI). Then the operator would be able to interact with the UI to validate/invalidate the correction of the non-conformity.

The rest of the paper is organized as follows. The existing literature on AR uses cases inside automotive industries is briefly discussed in Section 2. Section 3 outlines the proposed method for inspection to improve the productivity in the production line of the industry under study. The outcome of the proposed system is exhibited in Section 4, and Section 5 concludes the paper by exhibiting as well future perspectives.

2. AR in the Industry

In this work, AR is the adopted technology for data visualization. AR Attempts to integrate information/ virtual models, such as computer graphics, texts, sounds, and other modalities, into the physical environment so that users understand this information as existing in real-time. Real-time communication represents a key role and remains a challenge for AR applications implementation [11]. According to [7], a standard AR system is composed of four essential components. First, the visualization device could be Head-Mounted Displays (HMDs), Hand-Held Devices (HHDs), statistic devices, and projectors. Second, the camera, is a fundamental component in the AR system to display the environment. Third, a tracking system serves to bring the virtual objects precisely into the real environment. The tracking system can be categorized into two main classes regarding AR purpose and applications: marker-based tracking and markerless tracking [12]. However, marker-based AR is the most common AR technology used [7] since it provides better accuracy with an easy implementation [12]. Marker-based systems rely essentially on the use of a visual marker [13]. Marker recognition by AR applications brings virtual information into the real world. In contrast, markerless systems [14] provide visual data by calculating, for example, the user location.

In a production environment, considering the effects that a too high cognitive load has on human performance, implementing such technology can potentially have a significant effect on the result of the same quality and productivity [15]. Moreover, in a recent paper by Á. Segura et al. [16], authors have highlighted the staple need for visual computing technologies taking into account AR to support the operator inside the 4.0 industries. Over the past years, the fact of productivity and quality growth when using AR, encourages researchers to work on the integration of such technology in the manufacturing environment [11]. The focus of recent research on the industrial field shows mainly the following use cases of AR: assembly, maintenance, and training activities. However, as best as we know, very few publications are available in the literature that addresses the issue of inspection based on AR, which denotes the huge gap in the literature related to real use cases of AR implementation for inspection activities. U. Urbas et al. [17] have also found that most of the works are concentrated on maintenance and assembly presenting a research gap in the AR-assisted inspection. Moreover, authors in [17] have affirmed that the majority of existing works are not only limited to maintenance and assembly tasks but also represent principally conceptual frameworks for AR use. The reason behind this is the lake of specific standards and guidelines in general during the implementation of AR and particularly for inspection [17]. Besides that, bringing out AR from laboratory areas to real use-cases inside industries uncovers several challenges that may explain the main ground behind the huge gap of works related to AR for inspection. Most of the existing issues are related to security problems, hardware compatibility and weight, and the proper functioning of the software in a real industry environment which is exposed to a lot of factors such as dust, ergonomic reasons, and especially human acceptance of the technology [7,18,19].

Focusing on the automotive field, operators nowadays must deal with a huge amount of data during vehicle production or even during the diagnostic phase. In most automotive industries, the vehicle information is accessible through a station next to each production phase or even in a document on the vehicle itself. In these cases, for inspection tasks, the operator faces the timing issue to assist and repair the vehicle defects. The time needed to accomplish the inspection activities is more and more demanded to be decreased in the automotive industries, notably for those producing the high-tech vehicles where the finished car inspection requires a long time [20]. However, AR applications are usually customized for specific industry needs, so it is impossible to commercialize them, especially considering the high cost of commercial implementation. For that, very few works of AR-assisted inspection can be found. One of the examples of inspection-based AR is presented by Bosh in [21], where the company implements its own marker-less connected repair software for the vehicle equipment’s diagnostic (see Figure 1a). The implemented system does not only identify fault sources in the vehicle but also guides the operator for efficient diagnosis and repair by exhibiting the tasks that need to be performed. Volkswagen has also built a marker-less augmented reality system named MARTA (Mobile Augmented Reality Technical Assistance) for the XL1 model (see Figure 1b). MARTA shows the operator the list of tasks to be realized during the inspection process while giving the possibility of assisting the correction phase by adding to the real scene instructions in front of the operator [22]. Another AR solution in the automotive industry is described in [23], where authors have implemented an AR system for underhood inspection and maintenance (see Figure 1c). Similarly, in [24], the authors have designed and implemented a Marker-based Spatial AR system for Spot welding inspections. The proposed AR system relies essentially on the data projection technique, which serves in this use case to display the information related to weld defects (see Figure 1d).

In all the above-mentioned works, authors have proved the effectiveness and efficiency when adopting AR to carry out inspections. The results of these works show that disposing of manual-based or even computer-based inspection techniques leads to a significant gain of time with higher accuracy of tasks accomplished. Despite that few works exist for AR-assisted inspection, the existing results when adopting AR and the advancement we face in the technology should encourage researchers to work more on this technology, especially in the industry field when its adoption may serve productivity increase.

3. The Proposed Cognitive System for Inspection

3.1. Background

In this section, we detail the former way of inspection at the target station, which is located at the end of the production line in the automotive industry. The information is displayed through a screen, where the list of defects that the vehicle in the station at that moment has and which should be inspected. Two lists are displayed on this screen, the top one showing the vehicle’s defect list, such that defects that have not yet been touched up are marked in red and the ones that are already corrected appear in green color. At the bottom of the screen are the controls made to the vehicle along the line, in green those that have already been carried out and in yellow those that have yet to be carried out. Taking into account the workstation where the AR system has to be implemented, identified as being the most critical post, it is intended to develop an AR application with a tangible user interface that through the display demonstrates what currently appears on the screen (see Figure 2) interactively and dynamically, while reducing the cognitive load necessary for the execution of the operation, so that the operator can have access in the same station to non-conformities, proceed with the correction of anomalies that will be automatically validated as the touch-up is performed, preventing the operation from being completed until all anomalies are corrected. In this way, the time losses identified in the current system will be significantly reduced, while preventing the car from being considered fit for sale. According to the data provided by the automotive industry, at the factory, all workstations have defined ranges, and are mostly characterized by cyclical and repetitive operations. During the cycle time, about 228 s, the operator will have to perform all the tasks necessary for the position he occupies in the assembly of the vehicle. In terms of cognitive load, except for learning in training, the posts that make up the assembly line have a reduced level of demand, as most parts have already been separated previously, with only the control at the post. Therefore, operators are only responsible for checking and assembling parts. This station has been identified as the station for the implementation of AR technology.

3.2. Cognitive Assistant System Conception and Design

We present herein the proposed cognitive system for the industry under study. Our work provides both data processing from the IIoT platform as well as a UI to inspect, correct conformities, and send back data to the main server. Figure 3 shows the proposed architecture for the automotive production line to support the Industry 4.0 concept. Indeed, the proposed cognitive assistant is mainly composed of two parts detailed below. The first part concerns the way of data transmission via the IIoT platform whereas the second part presents how the data is visualized to the operators.

3.2.1. MQTT-Based Data Transmission

Throughout the line production, the vehicle is carried on and investigated by a set of sensors indicating its state of conformity. To avoid direct access to the main Industry DataBase (DB), ensuring data security, the IIoT platform has been implemented for data flow. In the final line production, the vehicle must be inspected by the operator based on the list of active errors which are provided by the automotive Industry DB, and which need to be fixed. The next step consists of displaying this list of errors in a UI. To do that, the MQTT protocol is adopted for the data transmission between the IIoT platform and all the other components. MQTT protocol is considered one of the principal messaging protocols of IIoT thanks to its rapidity during the machine-to-machine data transmission, which is a much-needed requirement in the manufacturing environment. Contrary to the web-based solution, which is a client-server-based communication, MQTT relies on the publish-subscribe concept allowing several clients to connect to a unique server that is called a broker [26]. These clients would be able to publish and/or subscribe to a specific topic via connection to an MQTT broker. Besides, the broker is considered the administrator for the publishing/subscribing technique. Its key role consists of receiving, filtering, and emitting messages to the correspondent subscribing clients [27]. The clients in our scenario are both the cognitive assistant system and the IIoT platform which can connect, subscribe, and publish in the broker. Having received the list of errors, collected from the sensors, and stored in the main Industry DB, the IIoT platform would be able to connect to the MQTT broker as well as publish this list into it. On the other part, the cognitive assistant system connects to the same broker and subscribes to the topic of active errors. These latter are handled by the operator and at the final process, the proposed system would be able to send the list of solved errors to the same broker. Thereafter, the IIoT platform subscribes to the topic related to the list of controls solved by the operator and sends the final list to be stored in the Manufacturing Execution Systems (MES). Indeed, MES is a real-time-based system supporting the concept of Industry 4.0, location sensing, mobile, and advanced analytics to identify most likely probable problems as soon as possible [28]. The main goal behind the adoption of such a system in Industry 4.0 is ensuring high performance and warranting the rapidity facing the quick change of costumer’s demands [29]. In our context, the used MES is designed by critical manufacturing [28] based in Portugal having the principle features of decentralization and vertically and horizontally way of integration.

The following sub-section details the way of data visualization and management from the operator side.

3.2.2. Data Visualization through the Cognitive Assistant System

Cognitive Assistants support humans and enhance their capabilities in solving a wide variety of complex tasks. In this section, we introduce a cognitive assistant that is developed using AR technology aiming at improving the operator’s performance and quick decision-making abilities. In other words, assisting the operator by displaying the defect and locating it to assist the decision of fixing the defects of the scanned vehicle on the final station.

Hence, the developed AR application displays a UI including the list of tasks to be inspected. The operator, performing the check, reads the controls that appear on the display without taking his eyes off the object of the inspection while reducing the inspection time and avoiding errors that can occur if some elements are overlooked. Based on the given scenario detailed in Section 3.1 and based on the current inspection interface (see Figure 2), we have proposed a UI where the operator would be able to visualize all the defects in one table. For the ergonomic reason, we have chosen to display the list of controls and errors, which need to be corrected by the operator, in one table differentiating them with different colors as follows: red for errors and blue for controls. As mentioned before, for UI conception and implementation we have selected Unity 3D as a development platform. However, by integrating this main platform with an augmented reality software development kit, the proposed system becomes able to detect an AR marker (the vehicle identifier), appearing in the field of view of the camera, and project a virtual table with the corresponding tasks into the real-world. In our case, adopting the marker-based AR is crucial and more accurate since the vehicle has already an identifier in a QR or bar code format. Another advantage of marker-based AR use is the reduced computation cost compared to the other type of AR when the marker is defined correctly [14]. The idea behind the use of Vuforia is that such technology can detect accurately these formats of marker supporting at the same time any device with a camera [30]. In the literature, research has found that Vuforia is the best to use for AR applications since 2018 [31]. First, the time needed to run an AR application using Vuforia is usually less than using other SDKs such as AR toolkit. Second, in the image tracking studies, the tracking time remains constant with increasing distance, so the distance between the recognition image and the camera does not affect the tracking. Finally, the racking times obtained with three different devices at different distances are often the lowest [14].

Hence, through the proposed system, the operator can validate the state of checked and corrected errors using a touch input method. In front of each error in the list, buttons are designed to be clicked, when the task is performed by the operator. The button click generates the removal of the task from the UI, but it remains saved in the system for final transmission and save to the MES. Figure 4 details the way of implementation of the above-described scenario using the 4C (Computation, Communication, Coordination, and Configuration) reference model [32].

4. Result and Validation

As mentioned previously, during the vehicle inspection, the operator must check and correct the defects that occur along the production line. For that, a UI was designed, using Unity 3D, to display dynamically the list of errors and controls, which is shown in Figure 5.

Additionally, markers stored in a cloud database using Vuforia SDK, identifying the vehicle, that need to be tracked are included in the AR application. In our scenario, the marker is presented as a QR code format given its sharpness and precision. As the main output of this study, the proposed system was built successfully into windows (see Figure 6) and Android (see Figure 7 and Figure 8) platforms. For Android, which is our main target platform for the automotive industry requirement, the proposed system is saved in an application format of type *.APK. We have also built the AR application onto an AR head-mounted display (HMD), an Android-based platform, which is Vuzix M300 smart glass [33]. Considering its lightweight (140 g), the industry under study decided to execute the proposed system on the Vuzix M300 smart glasses.

To verify the validity of the proposed AR system, we carried out several experiments. In order to do that, we referred to the base of criteria defined in [11] for AR system evaluation which are: reliability, responsiveness, and agility. We started by investigating the reliability by checking the conformity of data received from the IIoT platform. Global validation was performed within other entities that were responsible for IIoT platform conception. Experimental tests using the proposed AR system and the IIoT platform were concluded successfully. Indeed, the validation was realized in different phases. Firstly, the system specification and design were validated with the main customer, which is the automotive industry under study. Secondly, integration of the system within the rest of implemented modules for the automotive industry was successfully concluded regarding the correct data transmission and visualization. For responsiveness criteria, which denotes the ability to provide the augmented content the quickly as possible, we measured the time needed for our AR system to start. To display the tasks related to a vehicle, the AR system needed an average of around 1 s when the marker is placed in the field of view of the camera. This value may be considered low ensuring the responsiveness feature. Then, regarding agility, the software topology is developed in a way to support any device. In addition, the proposed AR system responds and provides accurately the 2D information even when changing the angle of view or the positioning of the device (vertical/ horizontal/ inclined).

To end up, it is essential to evaluate user acceptance considering the ergonomic aspect and work conditions. Ten people of two separate profiles (five researchers and five operators) were interrogated to evaluate the effectiveness of the AR system in such a manufacturing process. The reason for the low volume of users is highly linked to confidentially, as required by the industry under study. Tests are performed using five researchers within the same laboratory, and who are affected by the same project but with different research backgrounds to avoid the maximum risk of bias in the results. The selected researchers are two professors and three Ph.D. students associated with different tasks in the industry under study. As for the number of operators, we selected five random operators working on distinct shifts at the last inspection station where we implemented the system. Based on the questionnaire posed in [34] for an industrial Virtual Reality (VR) application evaluation, we proposed the following questionnaire for user experience analysis:

• Have you felt dizzy during or after the AR experience?
• Have you found the list of tasks legible?
• Are the chosen colors clear and readable?
• Is the designed interface appropriate for the lighting conditions?
• Is the interaction way within the system usable and easy?
• Do you think the system is time-saving?
• Is it easier to handle vehicle defects using the proposed AR system?
• Do you think the device is adequate for such a task?

Five possible answers can be specified: Very dissatisfied (2 points), Dissatisfied (4 points), neither agree nor disagree (6 points), Satisfied (8 points) and very satisfied (10 points).

Table 1. The score (%) of testers’ feedback regarding the questionnaire.

Questions	Using an HHD		Using an HMD (Vuzix M300)
	Researchers	Operators	Researchers	Operators
(1)	92	80	56	32
(2)	100	84	68	60
(3)	84	80	60	48
(4)	88	80	84	80
(5)	100	84	44	28
(6)	80	80	48	28
(7)	80	80	40	40
(8)	96	92	40	32
Mean	90	82.5	55	43.5

exhibits the results of the testers’ feedback when testing the AR system on both HHD and HMD.

From the mean value of the answers, we notice that all the testers from both profiles have mostly agreed that the best option was the handled device. Besides, to decrease the time needed for inspection and eventual corrections, it is proposed to use a gear to set the operator’s hands-free. However, all the testers have affirmed that the Vuzix M300 glasses could not be a good option for the current work conditions in the automotive industry. From the experiments using Vuzix M300 and regarding testers’ feedback, we had concluded that its small screen could contribute to the sight fatigue, colors, and tasks illegibility which were confirmed by testers after around 8 min of utilization. Moreover, for users suffering from a sight problem, Vuzix M300, makes the process even harder. This major disadvantage of the small display may generate the opposite effect of the desired goal which is a production increase. Therefore, it is highly recommended to use an HHD or an HMD with a larger display. In the end, regarding the system validation and users’ feedback, we can affirm that this cognitive assistant, based on the AR tool, may serve the concept of Industry 4.0 ensuring lower and reliable response time, with reduced operational costs.

5. Conclusions and Future Perspective

In this paper, we have presented an innovative view of visual inspection for an automotive industry based on AR technology. AR is a technology that can support a human being’s sense of vision. Particular attention is paid not only to the way of data visualization using AR technology but also to the data transmission via the IIoT platform adopting the MQTT protocol. Besides, the proposed system is of direct practical relevance serving to deepen the concept of Industry 4.0 in the automotive industry.

Based on the results presented in Section 4, it can be concluded that the proposed cognitive system has been very successful using a handled device. Also, from the outcome of the users’ feedback, it is possible to conclude that this novel system inside the automotive industry could ensure lower operational costs which directly affect productivity.

Out of our specific use case scenario, compatibility of the proposed solution with over 25 frameworks [35] presents a huge benefit for the proposed system given the unlimited possibility of testing on different platforms in real use cases. In other words, our proposed system can be used in different industrial contexts by changing only the data model and table components regarding industry needs/ data. Moreover, the abstraction of the implemented MQTT protocol presents a reusability feature that allows it to be re-used to retrieve any data from brokers in any other industrial context.

However, we can’t deny the presence of some limitations when using AR in the industrial environment. These limitations can be related to human technology acceptance or the lack of AR interaction techniques. In our case, occasionally, the vehicle can present some defects that are not detected through the production line. As a result, those defects are not displayed to the operator. However, besides the correction of the defects, the operator checks the overall finished vehicle. If an additional abnormality is noticed, the system should allow the operator to add his remarks. An initial idea to treat this is as follows: adding a new record to the AR table could be as:

• a voice note form if an HMD is used;
• or a “plus” touch button if an HHD is used.

Clearly, future work on the issues would be of interest.

Alternatively, another future direction for the proposed system is the use of the “IATF 16949” standard, given its ability for early defect prevention. Moreover, the IATF 16949 standard serves the reduction of variation and waste in the assembly chain, which may overcome the occasional issues mentioned above.