SMS Performance Evaluation

 SMS Performance Evaluation

By OffRoadPilots

When competing in the Daytona 500, the key performance indicators are not how many errors each driver made during the race to lose, but how many laps they did better than the others to win. Each driver’s performance was compared to the other drivers in the race and the winner is the driver who performs better than everyone else. A safety management system (SMS) is in concept not any different than a Daytona 500. It is a race to perform at the highest level for operators to win their internal race to operate without regulatory findings. A regulatory compliant safety management system does not guarantee an accident-free environment, but both airport and airline operator must first adhere regulatory compliance to achieve success. There are no sound reasons to operate with regulatory non- conformances.

SMS performance assessment is a regulatory requirement, but it is not as obvious to pick a winner in an SMS as it is in the Daytona 500. After all drivers have crossed the finish line is when the winner is awarded a gold medal. It is an incomprehensible task to establish the finish line in an SMS when there are expectations to operate in a hazard-free environment without occurrences. A high performing SMS does not ensure that accidents never happen again, but it is to be prepared when they happen. Just as a Daytona 500 winner needs to be prepared when things go wrong, maintain control, and stay ahead of in the game, an SMS enterprise must have control measures in place as needed for their own operations.

The first part of a goalsetting process is for the improvement of aviation safety. Improving aviation safety is a regulatory requirement, but the regulation does not define what it means to improve safety. Since the regulations are performance based, and when the regulation does not state requirements, or definitions, it is up to each airport and airline operator to define what improving aviation safety is. If aviation safety is new and improved today implies that it was old and inferior yesterday, which is not necessarily true. Aviation may have been safe yesterday, but without process patterns to follow, it was undocumented why it was safe until SMS regulations came along. The old quote that “if it ain’t broke, don’t fix it” is not what improving safety is. Improving safety are minor, almost unnoticeable changes, within the patterns, in human factors, organizational factors, supervision factors or environmental factors.

An SMS enterprise is required to have a process for setting goals for the improvement of aviation safety and for measuring the attainment of those goals. After they have crossed the finish line is when it is possible to evaluate and analyze their performance of how they reached that goal. One trap to watch out for when setting goals is to make an attainable goal timeline so far into the future that it becomes irrelevant to the goal what an airport or airline do today. An attainable goal for an airport operator for the improvement of aviation safety could be to conduct an obstacle survey every five years of obstacles in any of the approaches. This goal is correctly assigned a five-year irrevocable timeline which is closing the timeframe gap for the goal. With this five-year timeline goal, an airport operator, which is a task normally assigned to an airport manager (APM), may set a target date for the next survey in five years and file the goal on the shelf.

When a goal is out of reach, it is also out of mind until it is triggered by a notification. This is absolutely the correct method to use for an obstacle survey, and this was how airports operated prior to SMS. However, what is forgotten in the equation is that the person managing the safety management system (SMS manager) is required to implement a reporting system to ensure the timely collection of information related to hazards, incidents and accidents that may adversely affect safety. It is crucial for the performance of a safety management system that there is an open communication line between the APM and SMS manager. An SMS manager must also define what information may adversely affects safety. Unknown obstacles in the approach may adversely affect safety, since the obstacle may cause an avoidance action by an airliner on final approach. An SMS manager may define in their safety management system manual that the definition of “adversely affect safety” is when a pilot is required to make an immediate avoidance actions. When an SMS manager has assigned definitions, and these definitions are accepted by the Accountable Executive (AE), an APM has a tool to assign daily quality control tasks for meeting the requirement of a five-year survey timeline. The first goal a daily communication line between APM and SMS manager, and the second part of the goal is for airside personnel to daily observe for new obstacles in the approach. A five-year survey goal timeline becomes manageable when an airport operator applies their daily quality control system.

Airport standards accuracy requirements for aeronautical data are based upon a 95% confidence level with three types of positional data identified. Position data are identified as surveyed points (e.g. runway threshold), calculated points (e.g. mathematical calculations from the known surveyed points of thresholds for determination of the aerodrome reference point), and declared points (e.g. flight information region boundary points). Compliance with airport standards is a condition for the issuance and maintaining an airport certificate.

Linked to the accuracy requirements are the Integrity classification of aeronautical data. Classification based upon the potential risk resulting from the use of

 

95% confidence level that all data points are acceptable.

corrupted, or incorrect data. Aeronautical data is classified as routine data where there is a very low probability when using corrupted routine data that the continued safe flight and landing of an aircraft would be severely at risk with the potential for catastrophe.

Essential data is when there is a low probability when using corrupted essential data that the continued safe flight and landing of an aircraft would be severely at risk with the potential for catastrophe.

Critical data is when there is a high probability when using corrupted critical data that the continued safe flight and landing of an aircraft would be severely at risk with the potential for catastrophe.

Another task for an airport manager is to assign classification in accordance with airport standards to aeronautical data. When all data points have been classified, an airport operator may set attainable goals for compliance. Most of these points are assumed to be permanent unless there is a major airport project planned. When surveyed points are assumed to a permanent location, a long-term goal is beyond the timespan in a position as the APM or SMS manager. An attainable goal for a permanent surveyed, measured or calculated aeronautical point comes with challenges. An attainable goal is therefore a reversal process and it is to comply with a requirement to review each issue of each aeronautical information publication and notify the regulatory of any inaccurate information. An attainable goal is then established on a 56-day cycle for review of published information, and compare this information to surveyed data points.

An SMS manager is required to implement a safety data system, by either electronic or other means, to monitor and analyze trends in hazards, incidents and accidents, monitor and evaluate the results of corrective actions, and monitor the concerns of the civil aviation industry in respect of safety and their perceived effect on their operations. All these requirements are goal oriented, and it is up to an airport operator to decide on what monitoring tool to use, and what monitoring processes to use.

A system is a set of components that works together in harmony. A system is reliable and performing its assigned purpose within a pattern. A requirement of a safety data system is not necessary that it is automated, but that it is linked to a performance expectation flow chart. Processes are impossible to design without expectations. A safety data system must show performance in data collection, data storage, data retrieval and data analysis. Without operating with a data storage system, each new data collected become its own new system. If an airport has operated with an acceptable quality control system over the past few years, and one day decide to change over to a different system, all previous collected data becomes invalid unless data, including data from inspections, audits and corrective actions are transferred into the new safety data system. A performance goal for a safety data system is not how may reports it received, but how adaptable it is to changes and to carry forward previously collected and analyzed data.

Performance evaluation of a safety management system is an evaluation of how components work together in harmony to produce a conclusion as an output. A system concludes by its deviation in time (hours- minutes-seconds), space (geographical location), and compass (direction) from an established goal. A Daytona 500 driver enters the race to win, and to be on target, but at the end of the race there is only one winner, and all other drives crossed the finish line successfully, but deviated from their goal. Deviation from goals is not a failure of a safety management system, but a success of a system without corrupt processes. A process in a race, being the Daytona 500, a 100-meter track and field, or a speedskating race could be overcontrolled, manipulated and corrupted by an expectation that everyone should cross the finish line at the same time. A corrupt SMS system is identified in statistical process control (SPC) control charts.

Performance evaluation of an SMS is to evaluate deviations from goals in time, space, and compass, and apply adjustment to human factors, organizational factors, supervision factors, or environmental factors. A Daytona 500 driver who finished second in the race today, may have to make an adjustment to the engine oil to perform with a winning team tomorrow.

OffRoadPilots