Pilots’ Reactions to Different Types of Alerts When Using Head-Up Displays
INTRODUCTION: The benefits of using head-up displays (HUDs) include reducing head-down time during critical flight phases, enhancing awareness of the external environment, and improving in-flight crew performance. However, the monochromatic nature of HUDs, increased head rotation, and longer gaze movement paths might affect pilots’ reactions to different types of alerts. METHODS: Pilot workload and behavior differences were examined between HUD and head-down display (HDD) configurations in three alert scenarios. The study was carried out in an A320 flight simulator and 12 pilots participated. RESULTS: Except for one engine-on-fire scenario, pilot workload when using a HUD was significantly lower than using an HDD. In one engine-on-fire (3.98 s vs. 3.57 s) and one gear-disagree (5.42 s vs. 4.69 s) scenario, pilot response time to alerts using HUD was significantly longer than using an HDD. The angle deviations were significantly smaller when using HUDs in both go-around-under-crosswind (2.67° vs. 3.37°) and one engine-on-fire scenario (1.22° vs. 1.89°). DISCUSSION: The HUD is suitable for a lengthy process of manual flight control inputs, which not only reduces workload but also promotes control accuracy. For tasks that rely on automation, the benefits for workload become less obvious. In addition, head rotation and reorientation of attention adversely affected the response time to non-time-critical warnings and cautions. For instantaneous control with high precision requirements, HUDs did not demonstrate a significant advantage. Zheng Y, Lu Y, Jie Y, Fu S. Pilots’ reactions to different types of alerts when using head-up displays. Aerosp Med Hum Perform. 2024; 95(9):688–694.
In aircraft cockpits, over 90% of flight-related information is obtained visually and visual perception is the primary source of information for pilots contributing to safe flight.4 Therefore, most modern flight decks incorporate advanced digital display technologies. The traditional analog gauge-based systems have already evolved into large-screen displays that integrate various types of flight information. In addition, a number of visual assistance technologies are being studied or have been implemented. For instance, Williams identified that when compared with conventional instruments, the amount of pilot head-up visual scan time was significantly reduced while using highway-in-the-sky display.32 Kramer et al. found that the use of the enhanced vision system helped pilots gain better situation awareness and prevented flying below-minimum approaches.14 The electronic flight bag has been approved as an alternative to traditional printed flight documents (e.g., flight manuals, checklists, aeronautical charts including approach plates, etc.) the flight crew typically carries onboard in the crew flight bag.23 Another device that is available on almost all modern aircraft models is the head-up display (HUD), which projects essential flight information onto transparent glass directly in line with the pilot’s field of view, allowing the pilot to monitor critical flight parameters while viewing the external world.18
The main objective of HUD use in civil aviation is to provide airlines with cost-effective solutions to achieve a high proportion of takeoffs, approaches, and landings in bad weather conditions.9 The benefits of using HUD have been widely demonstrated, including reducing head-down time during critical phases of flight, an enhancement in awareness of external environment, and an improvement in flight crew performance. Stanton et al. argued that for civil rotorcraft pilots, HUDs reduced their workload under adverse visual conditions and significantly improved their situation awareness.27 Fadden et al. concluded that using HUDs can lead to faster detection of traffic cues and symbol changes, and improve flight path tracking accuracy.7 Proctor even found that HUDs can help pilots maintain longer visibility in a smoke-filled cockpit.20
Although the benefits of using HUD are obvious to airlines and pilots, drawbacks have also emerged; for example, the eye mis-accommodation issue, which could impair pilots’ capability to perceive the distance and size of a target.30 This inability to effectively visually accommodate may have two origins.
Firstly, Hofer et al. reported that as pilots could not view the HUD and the external scene simultaneously, they had to switch their attentions back and forth. According to their investigation of attention switching between HUD and the far domain, 9 out of 36 alerts were missed during the approach and landing phases across 12 pilots.11 Secondly, the lack of relative motion of the fixed HUD combiner glass and its frame compared with the external scene is also a possible source of problems. Karar and Ghosh supposed that the combiner frame forced pilots to adjust their head position to observe the obscured part of the outside world, resulting in some events being missed on the HUD.13 Another adverse effect, commonly referred to as cognitive tunneling, occurs when the user’s attention is diverted from important but unexpected events in the field of vision due to the characteristics of the information displayed on the HUD.10 Wickens et al. hypothesized that if an event or visual item is unexpected and not salient, imposing it within the foveal vision does not guarantee its detection.31 Otherwise, cockpit alerts are important and effective sources for pilots to perceive abnormal or emergency states and unexpected events.
Typically, flight crew alerts are classified into three categories based on the degree of urgency. Warning alerts require immediate crew recognition and corrective action. Caution alerts need immediate crew awareness and follow-up action. Advisory alerts result in a crew awareness and subsequent crew action.6 For warnings and cautions, it is necessary to implement both visual and auditory information channels to achieve rapid attention capture. Visual alerts facilitate attention capture and flight crew response via dramatic color changes and motion, which are significantly salient as they provide strong cues for object segmentation.21 Furthermore, some warnings are extremely urgent in terms of time for the safe operation of aircraft, such as Stall and Wind Shear. These time-critical warnings usually have specific voice or tone and are indicated by a distinctive red color on the primary flight display (PFD) to ensure a pilot’s immediate compensatory action without the need of other instructions or references. However, due to the monochromatic nature of the HUD, the time-critical warnings are not able to capture a pilot’s attention through color changes as originally displayed on the PFD. For alerts that are only displayed on the Crew Alert System, pilots may require more head rotation and longer gaze movement paths when using HUDs. To our knowledge, nevertheless, few studies have focused on reactions of pilots to different types of alerts when using HUDs. Thus, we examined the pilots’ workload and behavior differences between HUDs and head-down display (HDD) configurations employing three alert scenarios.
METHODS
Subjects
A total of 12 Chinese male pilots, ranging in age from 36 to 48 yr old (Mean = 42.1 ± 4.29), participated in this study. Among them, two test pilots were from the Civil Aviation Administration of China, five pilots were from China Commercial Aircraft Corporation of China, and the remaining five pilots were from China Eastern Airlines.
The mean total flight hours of the pilots were 9542 ± 3475 (ranging from 4500 to 16,000 h), and the mean HUD flight hours were 1375 ± 726 (ranging from 500 to 2500 h). In addition, each pilot had been designated as captain of an Airbus320 and, simultaneously, some of them had been recruited as captains for some other aircraft types (two for A330, three for A350, and one for A380). Before the experiment, all subjects signed the consent form and the entire study received approval from the Institutional Review Board of Shanghai Jiao Tong University (No. E20230206P).
Equipment
The experiment was carried out on an Airbus320 level-D full-motion flight simulator which belonged to the Civil Aviation Administration of China in Shanghai, China. The flight simulator conformed to the guidance published in Federal Aviation Administration Advisory Circular AC 120-40B (Airplane Simulator Qualification) and China Civil Aviation Regulations-60 (Identification and Usage Rules of Flight Simulation Equipment).1,8 The flight simulator had also been used for pilot training and some airworthiness compliance activities and technology research. The checklist, quick reference handbook, and configuration documents were provided to the pilots during the experiment.
Dual HUDs, manufactured by Thales Group and certified by Airbus in early 2015, were equipped in both pilots’ positions in the flight simulator with the display resolution of 1280 dpi × 1024 dpi, as in Fig. 1. The eye position of the HUDs coincided with cockpit eye position. The size of the eye box, the area in which the pilots were able to view the entire display, was 2″ × 3″ × 4″. After installation, both HUDs were calibrated to align the attitude, heading, and flight path vector with the outside world. Through eye box observation, the display accuracy met the requirement of 5 mrad at the center of the HUDs.25
Citation: Aerospace Medicine and Human Performance 95, 9; 10.3357/AMHP.6392.2024
Procedure
In order to assess the reaction of pilots to different types of alerts when using HUDs, three scenarios for approach and landing phases were elaborately designed, including go around under crosswind conditions (GA), one engine on fire (OE), and gear disagree (GD), containing the following different alert types: A) go around under crosswind condition (time-critical warning), B) one engine on fire (non-time-critical warning), and C) gear disagree (caution). In each scenario, NASA Task Load Index (NASA-TLX) data was collected to assess pilots’ workload. Simultaneously, their response time and task-related flight performance were also analyzed. The corresponding scenario configurations and the flight crew operating procedures are described as follows.
The GA flight scenario was initiated at the point of JTN (N31°07′24″, E121°20′30″) and the aircraft had an airspeed of 230 kn, a heading of 310°, and an altitude of 7500 ft (2286 m). The flight crew received air traffic controller (ATC) approval to conduct an instrument landing system approach and descend to an altitude of 3000 ft (914.4 m). Then ATC announced the ground wind was 290° at a speed of 9 m/s, gusts were 10 m/s, and the visibility was 1 km. When the aircraft descended to 300 ft (91.4 m), one moderate predicted wind shear was encountered. After identifying the wind shear warning, the pilots were required to conduct a go-around procedure, pressing the Take Off and Go Around button, pushing the throttle to the maximum position, increasing the pitch angle, and maintaining the aircraft at 18° until reaching a height of 3000 ft manually. In this task, the response time to the time-critical warning and pitch angle deviations during departure from the wind shear condition were measured.
The OE flight scenario was started at the point of PDL (N31°07′48″, E121°40′18″), and the aircraft maintained a steady 210 kn airspeed, 168° heading, and 7800 ft (2377.4 m) altitude. After descending for 30 s according to the standard approach procedure, a left engine fire occurred. A fire bell and continuous aural alert ‘LEFT ENGINE FIRE’ activated instantly, with a red ‘L ENG FIRE’ message display on the crew alert system page. The flight crew needed to contact ATC to select a suitable airport to land as soon as possible. Simultaneously, they had to disconnect auto throttle, adjust the thrust level of the affected side to the idle position, cut off the fuel supply on the affected side, and press the left engine fire switch on the overhead panel. Afterwards, the pilots were asked to wait for 10 s for N1 and nacelle ventilation to decrease, and then discharge the corresponding engine extinguishing agent. If the left engine fire still existed after 30 s, the second engine extinguishing agent should also be released. Subsequently, the pilots complied with the single engine procedure, completing the approach with a 3° glide path manually, and landed at a reference speed of Vref Full + 15 kn with a three-detent flaps configuration. In this task, the response times to the warning, glide angle deviations, and the deviations between actual landing speed and reference speed were measured.
The GD flight scenario began at the point of XSY (N30°55′54″, E121°52′24″), and the aircraft maintained a steady 180 kn airspeed, 348° heading, and 3000 ft (914.4 m) altitude, with autopilot and auto throttle connected. The simulation started once the initial flight plan was entered. The flight crew performed an instrument landing system approach and established the glide slope. Before releasing the landing gear, a fault occurred indicating that the front landing gear could not be extended. When pilots lowered the landing gear lever, a caution of ‘Gear Disagree’ appeared. Then they executed a landing gear gravity extension by toggling the switch of the alternate extension system but were still unable to successfully deploy it. Therefore, a go around program was conducted and the flight crew resumed an approach with an abnormal landing gear configuration. In this task, the response times to the caution and the deviations between actual landing speed and reference speed were measured.
The research subjects were in the pilot flying role from the left seat. The experiments were carried out from 09:00 to 16:00 local time, and all the participants reported being well rested. Each pilot conducted 2 h of HDD flying in the morning and 2 h of HUD flying in the afternoon. The three scenarios appeared in a random order and the flight simulator was used in motion mode. Before the experiment, each subject was trained in the same flight simulator for half an hour to be familiar with the simulator configurations and procedures. An experienced A320 type rated flight instructor acted as the support pilot, who was not flying. After each task, the participants were required to fill out a NASA-TLX scale.
Statistical Analysis
SPSS 17.0 (IBM, United States) for Windows was used to process the experiment data. ANOVA analysis was implemented in this study. When P < 0.05, the results were considered statistically significant.
RESULTS
The experiment results are described in three dimensions, including workload and response time analyses within and among scenarios, as well as flight performance comparisons.
In three scenarios using HUD, the pilots’ workloads were almost at the same level with no significant difference [F(2, 33) = 2.695, P = 0.080], as shown in Fig. 2. The average score was maximum in GD (mean = 42.7 ± 8.78), followed by OE (mean = 36.5 ± 7.69), and the least was in GA (mean = 35.6 ± 7.87). In contrast, the use of HDD [F(2, 33) = 4.110, P = 0.025] showed a significant difference in workload. The post hoc tests showed a significant difference between GA and OE (P = 0.046). The maximum workload occurred in OE (mean = 44.8 ± 7.18), followed by GD (mean = 44.3 ± 9.01), and GA was the smallest (mean = 36.3 ± 8.40). Further comparison within the same scenarios showed that there was a significant difference only in OE [F(1, 22) = 7.522, P = 0.012] in pilots’ workload between using HDD and HUD.
Citation: Aerospace Medicine and Human Performance 95, 9; 10.3357/AMHP.6392.2024
Normally, response time is determined as the time interval between the occurrence of an alert and the first action of the pilot to execute the corresponding procedure. Respectively, the first actions of the pilots in the three scenarios were pressing the Take Off and Go Around button in GA, disconnecting the auto throttle in OE, and toggling the alternate extension system switch in GD. When using HUD, the difference of response time to the three types of alerts was significant [F(2, 33) = 25.468, P < 0.01]. The shortest response time occurred when facing a time-critical warning in GA (mean = 3.56 s ± 0.67), then when encountering a non-time-critical warning in OE (Mean = 3.98 s ± 0.43), and the longest response time appeared in the caution condition in GD (Mean = 5.42 s ± 0.84). The post hoc tests showed a significant difference between GA and GD (P < 0.01) and between OE and GD (P < 0.01). However, although the difference was also significant when using HDD [F(2, 33) = 10.913, P < 0.01], the ranking of response time lengths was inconsistent with the use of HUD. The shortest one was 3.57 s (SD = 0.48) in OE, the medium one was 3.61 s (SD = 0.69) in GA, and the longest one was 4.69 s (SD = 0.80) in GD. The post hoc analysis indicated a significant difference between GA and GD (P < 0.01), and between OE and GD (P < 0.01). Furthermore, comparing within same scenarios, both in OE [F(1, 22) = 5.032, P = 0.035] and in GD [F(1, 22) = 4.673, P = 0.042], the response time differences were significant between using HUD and HDD (see Fig. 3).
Citation: Aerospace Medicine and Human Performance 95, 9; 10.3357/AMHP.6392.2024
Two main categories of pilots’ flight performance, angle manipulation deviation, and landing speed deviation were given special attention. As shown in Fig. 4, the pitch angle deviation was smaller when using HUD (mean = 2.67°± 0.79) than HDD (mean = 3.37°± 0.77), and the difference was significant in GA [F(1, 22) = 4.81, P = 0.039]. In OE, the glide path angle deviation was smaller while using HUD (mean = 1.22°± 0.34) than HDD (mean = 1.89°± 0.39), and also showed a significant difference [F(1, 22) = 20.532, P < 0.01]. Besides, both in OE (HUD: 2.75 kn ± 0.78, HDD: 2.86 kn ± 1.05) and in GD (HUD: 2.84 kn ± 0.98, HDD: 2.95 kn ± 0.96), the landing speed deviations with the HUD were slightly smaller than with the HDD, but the differences were insignificant.
Citation: Aerospace Medicine and Human Performance 95, 9; 10.3357/AMHP.6392.2024
DISCUSSION
According to the NASA-TLX scale results, there was no apparent correlation between the alert levels and pilots’ workload. Only in OE was the difference significant between using HUD and HDD. In this scenario, pilots had to complete the approach and landing manually as the autopilot and auto throttle were inoperative due to one engine failure. Since the HUD is usually designed to provide information to pilots during the flight phases of the transition between instrument and visual flight with variable external visibility conditions, it can increase head-up time, optimize situational awareness, and reduce the number of head movements, accommodations, and adaptations as factors for increasing workload.5 Similar conclusions have been confirmed in helicopter flight and car driving.26,29
Moreover, being able to maintain the real-time perception of the runway references while making control inputs could not only reduce the physical workload but also the mental demand.15 However, the workload benefits of the HUD seem to become less apparent for tasks that rely on automation to accomplish most of the work, such as in the two scenarios where the autopilot and auto throttle were on most of the time. In such cases, the pilot’s primary responsibility is simply to monitor the status of the aircraft, unless an unexpected situation arises that requires an immediate takeover.
Regardless of the types of display devices, the pilots’ responses to different categories of alerts varied significantly, which was consistent with the definition of alerts severity. For warning, immediate awareness and immediate response were required, and for caution, only immediate awareness and subsequent response were required.17 However, the difference between time-critical warning and non-time-critical warning was insignificant. Another unexpected result was that with using HDD, the average response time of the latter was slightly faster. Because of the urgency of the warning, the request for an immediate response often serves as a memory item. For example, when hearing an engine fire alarm, pilots naturally disengage the auto-throttle promptly by pressing the switch on the outside of the throttle level. After sufficient training, such action has become a skill-based sensory motor behavior, which is highly automatic.3
Additionally, the use of HUD did not improve pilots’ response time to alerts compared to the HDD. Only with time-critical warning did the response time decrease, but the difference was insignificant. Instead of the prominent red indications originally displayed on the PFD, supplemental coding methods of special characteristics, such as flashing icons, highlighting, and/or special locations, were used on the HUD to realize attention-grabbing properties. The result suggested that these methods of compensating for the lack of color coding coupled with auditory information could achieve the same level of situational awareness as HDD.
Nevertheless, in the absence of obvious visual prompts and speech cues, head rotation and reorientation of attention adversely affected response time, as observed with pilots dealing with non-time-critical warning and caution in OE and GD. This deficiency might be related to attention tunneling issues.28 When attention is allocated to an information source for a long period, it may lead to a delayed response or neglect of other information that is crucial for completing the task. In conventional HDD configurations, the Master Warning and Master Caution lights on the glareshield in the primary visual area could provide indications to pilots flying to redirect attention to another display when an immediate maneuvering is not required. Previous studies have also demonstrated that HUD might be harmful to pilots’ awareness, especially when irrelevant information unexpectedly appears.22 Critically, Crawford and Neal stated that many pilots might have no idea what they missed during flight, especially when using a HUD.2
As Snow and French assumed that HUD helped to improve flight path tracking accuracy, the benefits were fully reflected in the control of angles, including pitch angle and glide path angle manipulations in this study.24 Although pilots both operated according to the flight guidance indicator on the HUD, and achieved better performance in the above two scenarios, the reasons for the advantages might be different. In GA, the uncertainty did not come from the aircraft itself, but from the ambient environment. Wind shear exerted an additional uncertain force on the aircraft, resulting in sudden changes in velocity and direction. The HUD helped pilots overcome the aircraft disturbances caused by external adverse weather conditions to a certain extent. With eyes focused outside, viewing the presentation of flight path vector, acceleration, etc., pilots can cognitively combine the critical flight data with external references to obtain more sensitive subtle changes and accurate situation awareness when getting rid of wind shear. Similarly, Hoh and Arencibia also found in the gusty wind, the HUD landings were well within the acceptable range.12 On the other hand, the approach and landing after a unilateral engine failure is a lengthy manually controlled process in OE. Typically, after establishing visual references, pilots need to focus on flight parameters and external objectives, resulting in a large amount of head-up and head-down transitions. The use of HUD not only effectively reduces head-down time during critical flight phases, but also decreases the need for the eyes to refocus from the near to the far domain.16 This is also in line with the workload assessments found in our study. However, for instantaneous controls with high precision requirements (for instance, the landing speed), the use of HUD did not demonstrate a significant advantage. During an aircraft landing, the grounding point is located within 200 ft of the runway aiming points generally. Therefore, the pilot has enough time and position margin to adjust the aircraft to an appropriate speed. Since the takeoff and landing phases are the most common scenarios in flight training, qualified pilots have a deep understanding of the parameters that may affect safety at critical flight moments and can implement them at the optimal time.19
In this study, we compared the effects of HUD and HDD configurations on pilots’ workload and behavior in three scenarios containing different alerts types. In further research, we will concentrate on the impact of HUD use experience on the pilot’s workload and flight performance, and attempt to develop design and training strategies to improve pilots’ alerting awareness and response when using HUDs, especially in non-time-critical warning and caution conditions.
The configuration of the HUD used.
The results of the NASA-TLX of pilots using HUD and HDD in three flight tasks, which were go around under crosswind condition (GA), one engine on fire (OE), and gear disagree (GD) (*P < 0.05). The error bars stand for the standard deviations of the NASA-TLX of the subjects.
The results of response time of pilots using HUD and HDD in three flight tasks, which were go around under crosswind condition (GA), gear disagree (GD), and one engine on fire (OE) (*P < 0.05, **P < 0.01). The error bars stand for the standard deviations of response time of the subjects.
Angle manipulation deviation of pilots using HUD and HDD in two flight tasks, which were go around under crosswind condition (GA) and one engine on fire (OE) (*P < 0.05, **P < 0.01). The error bars stand for the standard deviations of angle manipulation of the subjects.
Contributor Notes
