The 95% Confidence Level

The 95% Confidence Level

By OffRoadPilots 

A confidence level is the percentage of times you expect to get close to the same estimate if you run your experiment again or resample the population in the same way. A confidence interval consists of the upper and lower bounds of the estimate you expect to find at a given level of confidence. If an airport is estimating a 95% confidence interval around the mean proportion of daily tasks, based on a random sampling of reports, you might find an upper bound of 0.56 and a lower bound of 0.48. These are the upper and lower bounds of the confidence interval. The confidence level is 95%.

With the introduction of a safety management system (SMS) to the aviation, two new terminologies were introduced to the global aviation industry, which were the confidence level, commonly known as a 95% confidence level, and confidence intervals. Prior to the introduction of a confidence level, trends were assessed by the criteria that fewer events are good, and more events are bad. In this system, a trend was established when there are two events, or datapoints, in a row moving in the same direction. When an airport or airline accept two datapoints as a trend they are building their own overcontrolling trap. Determining their level of safety and based on two data points, or events, is overcontrolling of processes. Overcontrolling happens when management do not comprehend information contained in variations with the result that overcontrolling will actually increase variations and causing more unexpected consequences. The old carpenter law when using one stick as a measuring tool for where to cut, is to use the same stick each time or the last stick will be at an incorrect length. Overcontrolling after two data points requires a newtarget to measure from and the last output will be incorrect. For airports and airlines to change over from a reactive SMS culture to a safety culture where they work within an SMS and its confidence level, their first task is to conduct a system analysis of the confidence level system.

A confidence level system is a forward-looking system for strategic planning and designing processes that conform to expected, or desired outcome. A confidence level system within an SMS provides for goal setting, planning, and measuring performance. It concerns itself with organizational safety rather than conventional health and safety at work concerns. An organization's SMS defines how it intends the management of airport and airline safety to be conducted as an integral part of their business management activities. An SMS is woven into the fabric of an organization and becomes part of the culture, or the way people do their jobs. Operating within a confidence level system is a comprehensive and systematic approach to the management of aviation safety, including the interfaces between the airports and airlines, and its suppliers, sub-contractors and business partners. A confidence level approach is also a regulatory requirement for airports to maintain procedures for the exchange of information in respect of hazards, incidents and accidents among all operators of aircraft and the airport operator.

When operating within a forward-looking system, or confidence level system, an SMS becomes a predictive SMS. A predictive SMS is when statistical analyses projects datapoints into the future, by applying process reliability. A process without special cause variations is an in-control process when outputs are as expected or are within the upper and lower confidence interval limits.

The confidence level is in the method, or process itself, and is not in a particular confidence interval. If the sampling method was repeated many times, 95% of the intervals constructed would capture the true population mean. As the sample size increases, the range of interval values will narrow, meaning that a larger sample size, or an increased number of data collected, the mean of the sample will generate a much more accurate result if compared with a smaller sample, or fewer tasks completed.

A confidence level interval are the upper control limits, and the lower control limits. These limits are based on statistical principles for assessing process performances. An SMS is not the old-fashion occupational health and safety reactive system, but is about the health of organizational safety performance, and measured by process reliability performance. A reliable, or stable process, and an in-control process may produce unacceptable results based on the average and calculated upper and lower control limits. An in-control process may be changed as desired to conform to expectations. As an example, if a call center has a policy to answer any calls before the fourth ring, but most calls are answered on the fifth or sixth ring, the process is stable, it falls within the upper and lower control limits, and in-control. However, it is outside of customer service expectation to answer by the third ring. A change in process is then required to meet that goal. When applying the analysis to a predictive SMS, the expectation is that the majority of calls during the next 12-month period will not meet the third-ring goal unless it is changed. When a process is changed, make one change at a time and monitor results.

It is a misconception that a 95% confidence level is unacceptable safety goals for airports and airlines. A wise person once said that you will capture a more correct number of hazards by applying a 95% confidence level to your operations than you will by capturing all hazards. An unknown hazard is also a hazard with an opportunity to affect an outcome in operations. Working with anything else but a confidence level is an unmanageable task.

An airport operating outside a predictive SMS system, assigns airport operations responsibilities to the captain of an intended arriving or departing flight. Conventional wisdom is that publishing NOTAMs releases an airport from all airport operations responsibilities and any incidents are causedby an aircraft captain’s own faulty judgement. A commonly applied airport operations manual (AOM) policy is that an airport is operational 24 hours per day, 7 days per week, supports both day and night VFR and IFR operations to non-precision approach limits, and departures visibility limits to 1⁄2 statute mile (SM) or greater. When a 24/7 policy is published in the aeronautical publications, the airport operator is required to have someone onsite 24/7, but they don’t staff at night, weekends or holidays. 

Airports may be operating with a paper documented SMS, but their operations are still proactive without acceptable processes. Operating without incidents or accidents at an airport, does not equal an operation with a healthy or successful SMS. A process may be in-control, but by the same token it is performing in non-compliance with regulatory requirements, standard requirements and an airport’s SMS safety policy.

When operating a safety management system and applying the confidence level system, both airports and airlines have a golden opportunity to go above and beyond regulatory requirements in both safety in operations and customer service. An airline providing scheduled air service is required by the regulations to operate out of certified airports, but they are not required to use certified airports as their alternate airports. A certified airport must comply with airport standards, while a non-certified airport, or registered aerodrome, is not required to comply with any airport standards at all. When an airline is using an aerodrome as their alternate destination, this alternate aerodrome may not be suitable for operations. All that the airline knows, is what was published in the aero publications and NOTAM, but there they are unable to verify current suitability upon arrival. A registered (non- certified) aerodrome operates in a reactive culture without responsibility to ensure compliance. The only requirements for an aerodrome to be registered and publish their airfield in the aeronautical publications is that warning notices are published for low-flying aircraft, that they have a wind direction indicator installed, if operating at night they need lights to be installed, they need no entry signs installed and they need no smoking or open flame signs installed. Everything else required for the safe operation of an airport or aircraft are voluntary tasks. This includes NOTAM, snow clearing, obstacle limitations on approach, runway, taxiway and apron aircraft size support, fuel availability and more. There are also certified airport operating under the same reactive principle and believe that by publishing NOTAMs they transfer all responsibility to an aircraft operator. When working within a confidence level system with confidence limits established, both airports and airlines have an opportunity to analyze data to conduct an accept or reject risk assessment.

An airport operator is also required to conduct an airport inspection daily, or more often, depending on type of operations and cause of runway contamination. An airport inspection include runways, taxiways, aprons, lights, signage, markings, markers, approaches and items such as new obstacles outside of airport property. Let’s assume that they are required to produce one report daily. Over a year 365 reports are generated, or 1,095 reports over a 36-month period. The first question to answer is if the tasks were completed daily, with a yes or no answer. There is an expectation that over 36 months, 1,095 reports were generated. Let’s assume that 1,095 reports were submitted for an inspection. In a predictive SMS culture, or when working within a confidence level culture, the next step is to learn if the process generated expected results, or output 1,095 times. What makes airports feel secure or safe, is not so much objective security or safety in operations, as a sense of confidence in their own ability to take care of themselves as they did in the past.

Airport and airline operators need to learn what to measure. SMS is to analyze processes and the health of organizational operations, which can only be discovered by applying a confidence level system with confidence limits.

This is the second reason why the global aviation industry, being airlines or airports, need a safety management system today, when they were safe yesterday without an SMS.

OffRoadPilots

How to Run SMS

 How to Run SMS

By OffRoadPilots

The biggest challenge to run a successful safety management system (SMS) is to operate with a system where regulations are performance based, as opposed to prescriptive. Over the years, since SMS was fist implemented, both airports and airlines had, and still have, difficulty to change over to a system where “the regulations does not say that”. Changing from prescriptive regulations to performance-based regulations did not make sense to airport and airline operators, or the regulatory oversight inspectors themselves. When SMS was implemented, it was assumed that a primary challenge would be to change to a just culture and a non-punitive reporting culture. This assumption was wrong, since the most difficult obstacle to overcome was the change from a prescriptive regulatory culture to a performance-based regulatory culture.

A performance-based culture assesses processes and acceptable work practices for compliance with the regulation, while a prescriptive culture assesses compliance with the text of the regulations. A prescriptive culture is a culture where the tasks are to comply with the letter of the text, while a performance-based culture is to apply processes that produces an outcome that conforms to regulatory requirements. Complying with the text does not necessarily prevent occurrences. A prime example is the Grand Canyon midair in 1956, when two airlines collided midair while complying with prescriptive regulations.

An airport or airline working within a prescriptive regulatory environment is required to maintain 100% compliance with 100% of the prescriptive regulations at 100% of the times. On the other hand, an organization working within a performance compliance environment works within a 95% confidence level. A confidence interval (CI) is a range of values that is likely to include a population value with a certain degree of confidence. It is often expressed as a % whereby a population mean lies between an upper and lower interval. A confidence level is the percentage of times you expect to get close to the same estimate if you run your experiment again or resample the population in the same way. The confidence interval consists of the upper and lower bounds of the estimate you expect to find at a given level of confidence.

For airport and airline operators it is a leap of faith into the unknown to jump from a prescriptive compliance culture over to a performance-based compliance culture. When emotions are involved, it is difficult to accept and change from a 100% prescriptive compliance level to a 95% performance confidence level. Emotions do not comprehend that it is impossible to comply and maintain requirements of a prescriptive compliance level. Conventional wisdom is when there are no occurrences or events reported, an operator maintains 100% compliance with the prescriptive regulations.

A prescriptive regulatory requirement sets out a specific standard, requirement or process to follow or actions that a regulated party must take in order to achieve compliance. There is no leeway for errors, or to start over again when tasks don’t go according to plan. The only way to maintain compliance is to avoid occurrences by not reporting any. Since a task must be completed when operating in a prescriptive environment, it becomes impossible to go back and correct a non- compliance since the non-compliance item already has happened. A prescriptive culture is a system where the last link in the chain of events becomes the focus item since all prior tasks were completed to 100% satisfaction. In a prescriptive culture an occurrence may be justified by management statements only. A general statement after an occurrence would be that “Our training and policies were not followed and failure to comply with our policies is not consistent with our culture of safety. We have been in contact with the regulator about this isolated event. No accident took place and there has been no impact on operations.” In a prescriptive operational culture, someone in the organization must be assigned a role and responsibility to wash their hands after occurrences for operations to continue. Within prescriptive compliance system occurrences are justified with little or no improvements made to safety.

For a prescriptive compliance culture to be effective, monitoring of operations must be ongoing, and from a regulatory oversight perspective view all prescriptive regulations or standards must be applied equally to all operators. Since it is impossible for the regulator to maintain 100% oversight of 100% of the certificate

holder at 100% of the time, a prescriptive system will always be reactive. In addition, before a violation can be assessed, it must have occurred, and a reactive approach required.

When analyzing two separate events within a prescriptive compliance system, no changes were made to operations and the violation repeated itself. It was reported that the aircraft landed on runway 24 when the runway was snow covered. During the landing roll the left main gear contacted deeper snow and the aircraft veered to the left. The left main gear caught a 14-inch windrow along the south edge of the runway and the pilot lost directional control. The aircraft departed the runway surface to the left and the nose gear collapsed. The aircraft sustained substantial damage to the nose and propellers. A prescriptive non-compliance was discovered after two events took place. The first event was a snow-covered runway, and the second event when an aircraft hit the windrow. Fast forward 11 years and the same non-compliance occurred, but this time without affecting aircraft operations. An airport operated with 1/8-inch snow on top of ice covering 100% of the runway. In addition, the airport operated with 4 feet tall windrows extending 10 feet onto the runway. Another example is when an aircraft departed with ice and crashed just after takeoff. Fast forward another 5 years and an aircraft was reported to take off with ice adhering to the critical surfaces, but this time also without affecting aircraft operations. Both were non-conforming in a prescriptive environment, but without any changes made to the processes. Runways remain snow covered, and aircraft keep on departing with ice. After severe accidents the prescriptive compliance environment approach comes in handy to pass judgement, point fingers and arresting flight crew after runway incursions. With this approach there is no need for an investigation since judgement has already been passed.

For an enterprise to run their SMS successfully, a daily quality control system must in place as a system where there is monitoring of drift and deviations. Drift and deviations are not necessarily hazardous to aviation safety, but they are undocumented and therefore the output of the process does not conform to an expected outcome. Drift and deviations often occur due to ineffective processes, processes that are cumbersome to work with, management may expect unnecessary tasks to be completed, or a process compliance factor within a performance-based compliance system is not comprehended.

When a successful SMS is run with a daily quality control system, each process is linked to a regulatory requirement. The outcome of the process must meet its expected outcome and conform to regulatory requirements, standard requirements, and the SMS policy. Every process, task and action must conform to the SMS policy for an enterprise to run a successful SMS.

The very first item of a daily quality control system is to establish the roles and responsibility of the accountable executive (AE). Their role, for both airports and airlines, is to be accountable on behalf of the certificate holder (CH) for meeting the requirements of the regulations. This can be achieved by combining multiple systems, or it can be achieved by applying the quality assurance system as the primary compliance requirement. A quality assurance system is the single most important system within the safety management system to maintain compliance in a performance based regulatory environment. A prerequisite for the quality assurance system is to operate with a daily quality control system. When operational management, or operational control, for each operational task is conducted at regular intervals, their data points become immediately available for analysis, monitoring and oversight management. Just as cash in the bank is counted daily (or more often), process conformance must be counted daily. Running an SMS is hard work. There are no shortcuts, or simple way to run an SMS since individual SMS enterprises operates with their own currency, or conforming processes tailored to their operations.

Depending on size and complexity of an airport or airline, daily quality control includes several tasks to be completed hourly, daily, weekly, monthly, annually, or triennially. Triennially is the outer limit since a triennial audit process is required to conform to regulatory requirements. When running an SMS, both airlines and airports need to conduct system analyses and convert these analyses into operations plans. A daily quality control system includes operations plans for each

safety critical areas and safety critical functions in their operations. Safety critical areas and safety critical functions are designated hazard classifications in a hazard register and analyzed for trends and compliance with SPC control charts.

SMS is reliability between operators. It is an airport operator’s role to ensure aircraft ground deicing systems are available and it is the captain’s responsibility to apply it. It is an airport operator’s role to ensure that a runway is clear of contamination, and it is the captain’s responsibility to assess runway suitability. It is an airport operator’s role to ensure they have personnel available during their published hours of operations, and it is the captain’s responsibility to learn what those hours are. It is an airport operator’s role to publish in the aeronautical publications, and it is the captain’s responsibility to review publications. It is an airport operator’s role to conduct runway friction testing for turbojet aircraft, and it is the captain’s responsivity to verity NOTAM. The list could co on and on forever. Times are long gone when airport operators could wash their hands by NOTAM non-conformances to comply with standards or regulations. However, airport operators are still publishing NOTAM for their lack of compliance and are unreliable service providers when operating an airport that is incompatible with the safe operation of an airport or aircraft.

A prime example is a true story about an SMS airport that went back to their prescriptive compliance operations since their performance-based operations were extremely successful. In 2016 this airport totally failed a regulatory inspection. The regulator demanded they return their certificate but the airport operator negotiated an agreement with the regulator to invalidate the 2016 inspection and for the airport to start over again with a “clean slate”. The regulator agreed and over the years their new systems and processes became so successful that they forgot the past and reverted back to their pre-2016 processes. Their justification to revert to pre-2016 SMS was that they are at a stage which have ultimately made the airport a safer and more resilient airport, and able to maintain its federal airport certification. Not only did the airport revert, but they also abandoned their post-2016 compliance system.

As in the true example above, and until airports and airlines accept their different roles and responsibilities and comprehend their performance-oriented compliance processes, their safety management system is its own worst enemy.

OffRoadPilots