Validation assessment
 Validation assessment (Click for a larger view)
The PHARE Validation Project was established

To provide a common assessment methodology across all the large scale, real-time simulations conducted by the members of PHARE.

In the execution of this activity, the PHARE validation project:
  • Developed a common assessment methodology;
  • Produced recommended Measures of Merit for use in evaluating the results from the PHARE Demonstrations;
  • Produced recommended analytical methods to be employed in the assessment of the measurements taken.
The experience from the work conducted by the PHARE Validation Project and by the PHARE partners in their application of the methodology, led to the following recommendations:
  • The development of a core validation team is central to the successful execution of validation exercises. The team should be involved throughout the complete project lifecycle and its members must have a thorough understanding of the operational concepts under investigation.
  • The development of a common Assessment/Validation Methodology is crucial. Traceability of all input and output data is essential to the analysis process. The metrics employed must be clearly defined and relevant to real-life ATM systems.
  • The Assessment/Validation Methodology should be seen as “living” and capable of adaptation to the specific requirements of the system under evaluation. Continued development of the methodology is necessary to improve the quality of the output.
  • A true validation methodology should not concentrate on one specific technique alone – as within PHARE where real-time simulations only were conducted. The use of other techniques – for example analytical models, fast-time simulators, etc. – is likely to yield more cost-effective results in the early stages of a project’s development cycle.
  • For future work similar in nature to the PHARE Programme, a Validation Methodology should be applied as opposed to an Assessment Methodology. Such a methodology should take note of work being conducted within the validation field by both Eurocontrol and the European Commission.
The problems encountered and the recommendations made by the group surprisingly remain relevant. Much work has been performed such as the European Commission project, MAEVA (Master ATM European Validation Plan) and latterly the Operational Concept Validation Methodology (OCVM) initiative by Eurocontrol to identify and attempt to standardise the methodology to be adopted in a validation activity. However, these initiatives currently stop short of recommending the methods and metrics to be used. These last steps are crucial to enable meaningful comparisons to be made between validation activities performed by different research organisations. The EUROCONTROL CARE Initiative, INTEGRA, has undertaken extensive R&D on a range of quantitative metrics but these remain a long way from gaining universal acceptance.
  MAEVA Link to the original Master ATM European Validation Plan (MAEVA) documents.
  OCVM Link to the EUROCONTROL Operational Concept Validation Methodology (OCVM) website.

For further information

The Validation activities are documented in the following:
  Acrobat 99-70-05 Validation Project Final Report114 Kb
Also in:
  Acrobat 95-70-01 PD/3 Demonstration Project Plan124 Kb
  Acrobat 96-70-24 PD1 Final Report Volume 2 Annex F Lessons Learnt from PD/1152 Kb
  Acrobat 97-70-13 PD/2 Final Report Volume 1912 Kb
  Acrobat 99-70-01 Volume 1 of PD/3 Final Report - PHARE Demonstration 3 (PD/3) Synthesis Report204 Kb
  Acrobat 99-70-01 Volume 2 of PD/3 Final Report - CENA PD/3 Final Report1,189 Kb
  Acrobat 99-70-01 Volume 4 of PD/3 Final Report - NLR PHARE Demonstration 3 Final Report528 Kb
  Acrobat 99-70-02 PHARE Ground Human Machine Interface (GHMI) project: summary report2,196 Kb
  Acrobat 99-70-09 PHARE Final Report1,482 Kb