This document reports the results fo the SJSU analysis of the application of the INTEGRA tools in the United States.

This document is an initial draft of the user guide for the INTEGRA Real-Time Monitors application. The guide includes a description of the data collection interface between a simulation platform and the Real-Time Monitors application. It describes the functionality that has to be implemented as part of the simulation platform in order to provide the application with the required data.

This initial version of the INTEGRA Real-Time Monitors User Guide focuses mainly on the display of the metrics data.

The fundamental psychometric criteria for measures are discussed. Also considered are the types of factor
employed in experimental design, and important issues such as confounding. Specific topics addressed include the relative merits of within-subjects and between-subjects designs; calculation of statistical power to determine the correct number of participants; dealing with common data problems such as missing values and outliers (deviant scores); use of parametric versus non-parametric statistical tests; and minimising the effects of prior learning when comparing an existing system to a new concept. A flow chart is presented to assist in the selection of the most appropriate test for particular applications. Aspects of the implementation, analysis and interpretation of candidate workload measures are also described, and the possible role of modelling is discussed.

Here we apply these principles to a worked example, in the form of the specification of a trial to compare the workload experienced using conventional and PHARE Demonstration 1 (PD/1) ATM systems. The issue to be
investigated is whether the PD/1 system will allow controllers to cope with more traffic for the same level of workload. To demonstrate this increase in capacity, it is necessary to obtain evidence that the PD/1 system produces lower workload than the conventional system. The experimental design is defined in detail, and guidance for analysis, interpretation and reporting of the results is provided.

This document identifies and reviews current human-in-the-loop simulations from fields other than air traffic management (ATM). In the systems selected, the human plays an active role in the simulated system or process, i.e. the system or process is dependent on the human performing an activity, or series of activities, for its correct action. Case studies are drawn from military and non-military human in the loop (HITL) research.

This report was prepared as part of a project being conducted under the EUROCONTROL INTEGRA programme. The aim of the project is to derive principles of workload measurement in man-in-the-loop simulations from experience in non-air traffic management (ATM) domains. This report describes the outcome of Work Package 2. Types of workload measure — performance based, subjective, and physiological/biochemical — are critically reviewed, and advice is given on methods of selecting the best set of measures for ATM simulations. In the next Work Package, the development of sound experimental designs incorporating these measures will be considered.

Getting Results from Real-Time Simulations and Live Environments