The 95% Confidence Level

The 95% Confidence Level

By OffRoadPilots 

A confidence level is the percentage of times you expect to get close to the same estimate if you run your experiment again or resample the population in the same way. A confidence interval consists of the upper and lower bounds of the estimate you expect to find at a given level of confidence. If an airport is estimating a 95% confidence interval around the mean proportion of daily tasks, based on a random sampling of reports, you might find an upper bound of 0.56 and a lower bound of 0.48. These are the upper and lower bounds of the confidence interval. The confidence level is 95%.

With the introduction of a safety management system (SMS) to the aviation, two new terminologies were introduced to the global aviation industry, which were the confidence level, commonly known as a 95% confidence level, and confidence intervals. Prior to the introduction of a confidence level, trends were assessed by the criteria that fewer events are good, and more events are bad. In this system, a trend was established when there are two events, or datapoints, in a row moving in the same direction. When an airport or airline accept two datapoints as a trend they are building their own overcontrolling trap. Determining their level of safety and based on two data points, or events, is overcontrolling of processes. Overcontrolling happens when management do not comprehend information contained in variations with the result that overcontrolling will actually increase variations and causing more unexpected consequences. The old carpenter law when using one stick as a measuring tool for where to cut, is to use the same stick each time or the last stick will be at an incorrect length. Overcontrolling after two data points requires a newtarget to measure from and the last output will be incorrect. For airports and airlines to change over from a reactive SMS culture to a safety culture where they work within an SMS and its confidence level, their first task is to conduct a system analysis of the confidence level system.

A confidence level system is a forward-looking system for strategic planning and designing processes that conform to expected, or desired outcome. A confidence level system within an SMS provides for goal setting, planning, and measuring performance. It concerns itself with organizational safety rather than conventional health and safety at work concerns. An organization's SMS defines how it intends the management of airport and airline safety to be conducted as an integral part of their business management activities. An SMS is woven into the fabric of an organization and becomes part of the culture, or the way people do their jobs. Operating within a confidence level system is a comprehensive and systematic approach to the management of aviation safety, including the interfaces between the airports and airlines, and its suppliers, sub-contractors and business partners. A confidence level approach is also a regulatory requirement for airports to maintain procedures for the exchange of information in respect of hazards, incidents and accidents among all operators of aircraft and the airport operator.

When operating within a forward-looking system, or confidence level system, an SMS becomes a predictive SMS. A predictive SMS is when statistical analyses projects datapoints into the future, by applying process reliability. A process without special cause variations is an in-control process when outputs are as expected or are within the upper and lower confidence interval limits.

The confidence level is in the method, or process itself, and is not in a particular confidence interval. If the sampling method was repeated many times, 95% of the intervals constructed would capture the true population mean. As the sample size increases, the range of interval values will narrow, meaning that a larger sample size, or an increased number of data collected, the mean of the sample will generate a much more accurate result if compared with a smaller sample, or fewer tasks completed.

A confidence level interval are the upper control limits, and the lower control limits. These limits are based on statistical principles for assessing process performances. An SMS is not the old-fashion occupational health and safety reactive system, but is about the health of organizational safety performance, and measured by process reliability performance. A reliable, or stable process, and an in-control process may produce unacceptable results based on the average and calculated upper and lower control limits. An in-control process may be changed as desired to conform to expectations. As an example, if a call center has a policy to answer any calls before the fourth ring, but most calls are answered on the fifth or sixth ring, the process is stable, it falls within the upper and lower control limits, and in-control. However, it is outside of customer service expectation to answer by the third ring. A change in process is then required to meet that goal. When applying the analysis to a predictive SMS, the expectation is that the majority of calls during the next 12-month period will not meet the third-ring goal unless it is changed. When a process is changed, make one change at a time and monitor results.

It is a misconception that a 95% confidence level is unacceptable safety goals for airports and airlines. A wise person once said that you will capture a more correct number of hazards by applying a 95% confidence level to your operations than you will by capturing all hazards. An unknown hazard is also a hazard with an opportunity to affect an outcome in operations. Working with anything else but a confidence level is an unmanageable task.

An airport operating outside a predictive SMS system, assigns airport operations responsibilities to the captain of an intended arriving or departing flight. Conventional wisdom is that publishing NOTAMs releases an airport from all airport operations responsibilities and any incidents are causedby an aircraft captain’s own faulty judgement. A commonly applied airport operations manual (AOM) policy is that an airport is operational 24 hours per day, 7 days per week, supports both day and night VFR and IFR operations to non-precision approach limits, and departures visibility limits to 1⁄2 statute mile (SM) or greater. When a 24/7 policy is published in the aeronautical publications, the airport operator is required to have someone onsite 24/7, but they don’t staff at night, weekends or holidays. 

Airports may be operating with a paper documented SMS, but their operations are still proactive without acceptable processes. Operating without incidents or accidents at an airport, does not equal an operation with a healthy or successful SMS. A process may be in-control, but by the same token it is performing in non-compliance with regulatory requirements, standard requirements and an airport’s SMS safety policy.

When operating a safety management system and applying the confidence level system, both airports and airlines have a golden opportunity to go above and beyond regulatory requirements in both safety in operations and customer service. An airline providing scheduled air service is required by the regulations to operate out of certified airports, but they are not required to use certified airports as their alternate airports. A certified airport must comply with airport standards, while a non-certified airport, or registered aerodrome, is not required to comply with any airport standards at all. When an airline is using an aerodrome as their alternate destination, this alternate aerodrome may not be suitable for operations. All that the airline knows, is what was published in the aero publications and NOTAM, but there they are unable to verify current suitability upon arrival. A registered (non- certified) aerodrome operates in a reactive culture without responsibility to ensure compliance. The only requirements for an aerodrome to be registered and publish their airfield in the aeronautical publications is that warning notices are published for low-flying aircraft, that they have a wind direction indicator installed, if operating at night they need lights to be installed, they need no entry signs installed and they need no smoking or open flame signs installed. Everything else required for the safe operation of an airport or aircraft are voluntary tasks. This includes NOTAM, snow clearing, obstacle limitations on approach, runway, taxiway and apron aircraft size support, fuel availability and more. There are also certified airport operating under the same reactive principle and believe that by publishing NOTAMs they transfer all responsibility to an aircraft operator. When working within a confidence level system with confidence limits established, both airports and airlines have an opportunity to analyze data to conduct an accept or reject risk assessment.

An airport operator is also required to conduct an airport inspection daily, or more often, depending on type of operations and cause of runway contamination. An airport inspection include runways, taxiways, aprons, lights, signage, markings, markers, approaches and items such as new obstacles outside of airport property. Let’s assume that they are required to produce one report daily. Over a year 365 reports are generated, or 1,095 reports over a 36-month period. The first question to answer is if the tasks were completed daily, with a yes or no answer. There is an expectation that over 36 months, 1,095 reports were generated. Let’s assume that 1,095 reports were submitted for an inspection. In a predictive SMS culture, or when working within a confidence level culture, the next step is to learn if the process generated expected results, or output 1,095 times. What makes airports feel secure or safe, is not so much objective security or safety in operations, as a sense of confidence in their own ability to take care of themselves as they did in the past.

Airport and airline operators need to learn what to measure. SMS is to analyze processes and the health of organizational operations, which can only be discovered by applying a confidence level system with confidence limits.

This is the second reason why the global aviation industry, being airlines or airports, need a safety management system today, when they were safe yesterday without an SMS.

OffRoadPilots