The 95% Confidence Level

The 95% Confidence Level

By OffRoadPilots 

A confidence level is the percentage of times you expect to get close to the same estimate if you run your experiment again or resample the population in the same way. A confidence interval consists of the upper and lower bounds of the estimate you expect to find at a given level of confidence. If an airport is estimating a 95% confidence interval around the mean proportion of daily tasks, based on a random sampling of reports, you might find an upper bound of 0.56 and a lower bound of 0.48. These are the upper and lower bounds of the confidence interval. The confidence level is 95%.

With the introduction of a safety management system (SMS) to the aviation, two new terminologies were introduced to the global aviation industry, which were the confidence level, commonly known as a 95% confidence level, and confidence intervals. Prior to the introduction of a confidence level, trends were assessed by the criteria that fewer events are good, and more events are bad. In this system, a trend was established when there are two events, or datapoints, in a row moving in the same direction. When an airport or airline accept two datapoints as a trend they are building their own overcontrolling trap. Determining their level of safety and based on two data points, or events, is overcontrolling of processes. Overcontrolling happens when management do not comprehend information contained in variations with the result that overcontrolling will actually increase variations and causing more unexpected consequences. The old carpenter law when using one stick as a measuring tool for where to cut, is to use the same stick each time or the last stick will be at an incorrect length. Overcontrolling after two data points requires a newtarget to measure from and the last output will be incorrect. For airports and airlines to change over from a reactive SMS culture to a safety culture where they work within an SMS and its confidence level, their first task is to conduct a system analysis of the confidence level system.

A confidence level system is a forward-looking system for strategic planning and designing processes that conform to expected, or desired outcome. A confidence level system within an SMS provides for goal setting, planning, and measuring performance. It concerns itself with organizational safety rather than conventional health and safety at work concerns. An organization's SMS defines how it intends the management of airport and airline safety to be conducted as an integral part of their business management activities. An SMS is woven into the fabric of an organization and becomes part of the culture, or the way people do their jobs. Operating within a confidence level system is a comprehensive and systematic approach to the management of aviation safety, including the interfaces between the airports and airlines, and its suppliers, sub-contractors and business partners. A confidence level approach is also a regulatory requirement for airports to maintain procedures for the exchange of information in respect of hazards, incidents and accidents among all operators of aircraft and the airport operator.

When operating within a forward-looking system, or confidence level system, an SMS becomes a predictive SMS. A predictive SMS is when statistical analyses projects datapoints into the future, by applying process reliability. A process without special cause variations is an in-control process when outputs are as expected or are within the upper and lower confidence interval limits.

The confidence level is in the method, or process itself, and is not in a particular confidence interval. If the sampling method was repeated many times, 95% of the intervals constructed would capture the true population mean. As the sample size increases, the range of interval values will narrow, meaning that a larger sample size, or an increased number of data collected, the mean of the sample will generate a much more accurate result if compared with a smaller sample, or fewer tasks completed.

A confidence level interval are the upper control limits, and the lower control limits. These limits are based on statistical principles for assessing process performances. An SMS is not the old-fashion occupational health and safety reactive system, but is about the health of organizational safety performance, and measured by process reliability performance. A reliable, or stable process, and an in-control process may produce unacceptable results based on the average and calculated upper and lower control limits. An in-control process may be changed as desired to conform to expectations. As an example, if a call center has a policy to answer any calls before the fourth ring, but most calls are answered on the fifth or sixth ring, the process is stable, it falls within the upper and lower control limits, and in-control. However, it is outside of customer service expectation to answer by the third ring. A change in process is then required to meet that goal. When applying the analysis to a predictive SMS, the expectation is that the majority of calls during the next 12-month period will not meet the third-ring goal unless it is changed. When a process is changed, make one change at a time and monitor results.

It is a misconception that a 95% confidence level is unacceptable safety goals for airports and airlines. A wise person once said that you will capture a more correct number of hazards by applying a 95% confidence level to your operations than you will by capturing all hazards. An unknown hazard is also a hazard with an opportunity to affect an outcome in operations. Working with anything else but a confidence level is an unmanageable task.

An airport operating outside a predictive SMS system, assigns airport operations responsibilities to the captain of an intended arriving or departing flight. Conventional wisdom is that publishing NOTAMs releases an airport from all airport operations responsibilities and any incidents are causedby an aircraft captain’s own faulty judgement. A commonly applied airport operations manual (AOM) policy is that an airport is operational 24 hours per day, 7 days per week, supports both day and night VFR and IFR operations to non-precision approach limits, and departures visibility limits to 1⁄2 statute mile (SM) or greater. When a 24/7 policy is published in the aeronautical publications, the airport operator is required to have someone onsite 24/7, but they don’t staff at night, weekends or holidays. 

Airports may be operating with a paper documented SMS, but their operations are still proactive without acceptable processes. Operating without incidents or accidents at an airport, does not equal an operation with a healthy or successful SMS. A process may be in-control, but by the same token it is performing in non-compliance with regulatory requirements, standard requirements and an airport’s SMS safety policy.

When operating a safety management system and applying the confidence level system, both airports and airlines have a golden opportunity to go above and beyond regulatory requirements in both safety in operations and customer service. An airline providing scheduled air service is required by the regulations to operate out of certified airports, but they are not required to use certified airports as their alternate airports. A certified airport must comply with airport standards, while a non-certified airport, or registered aerodrome, is not required to comply with any airport standards at all. When an airline is using an aerodrome as their alternate destination, this alternate aerodrome may not be suitable for operations. All that the airline knows, is what was published in the aero publications and NOTAM, but there they are unable to verify current suitability upon arrival. A registered (non- certified) aerodrome operates in a reactive culture without responsibility to ensure compliance. The only requirements for an aerodrome to be registered and publish their airfield in the aeronautical publications is that warning notices are published for low-flying aircraft, that they have a wind direction indicator installed, if operating at night they need lights to be installed, they need no entry signs installed and they need no smoking or open flame signs installed. Everything else required for the safe operation of an airport or aircraft are voluntary tasks. This includes NOTAM, snow clearing, obstacle limitations on approach, runway, taxiway and apron aircraft size support, fuel availability and more. There are also certified airport operating under the same reactive principle and believe that by publishing NOTAMs they transfer all responsibility to an aircraft operator. When working within a confidence level system with confidence limits established, both airports and airlines have an opportunity to analyze data to conduct an accept or reject risk assessment.

An airport operator is also required to conduct an airport inspection daily, or more often, depending on type of operations and cause of runway contamination. An airport inspection include runways, taxiways, aprons, lights, signage, markings, markers, approaches and items such as new obstacles outside of airport property. Let’s assume that they are required to produce one report daily. Over a year 365 reports are generated, or 1,095 reports over a 36-month period. The first question to answer is if the tasks were completed daily, with a yes or no answer. There is an expectation that over 36 months, 1,095 reports were generated. Let’s assume that 1,095 reports were submitted for an inspection. In a predictive SMS culture, or when working within a confidence level culture, the next step is to learn if the process generated expected results, or output 1,095 times. What makes airports feel secure or safe, is not so much objective security or safety in operations, as a sense of confidence in their own ability to take care of themselves as they did in the past.

Airport and airline operators need to learn what to measure. SMS is to analyze processes and the health of organizational operations, which can only be discovered by applying a confidence level system with confidence limits.

This is the second reason why the global aviation industry, being airlines or airports, need a safety management system today, when they were safe yesterday without an SMS.

OffRoadPilots

How to Run SMS

 How to Run SMS

By OffRoadPilots

The biggest challenge to run a successful safety management system (SMS) is to operate with a system where regulations are performance based, as opposed to prescriptive. Over the years, since SMS was fist implemented, both airports and airlines had, and still have, difficulty to change over to a system where “the regulations does not say that”. Changing from prescriptive regulations to performance-based regulations did not make sense to airport and airline operators, or the regulatory oversight inspectors themselves. When SMS was implemented, it was assumed that a primary challenge would be to change to a just culture and a non-punitive reporting culture. This assumption was wrong, since the most difficult obstacle to overcome was the change from a prescriptive regulatory culture to a performance-based regulatory culture.

A performance-based culture assesses processes and acceptable work practices for compliance with the regulation, while a prescriptive culture assesses compliance with the text of the regulations. A prescriptive culture is a culture where the tasks are to comply with the letter of the text, while a performance-based culture is to apply processes that produces an outcome that conforms to regulatory requirements. Complying with the text does not necessarily prevent occurrences. A prime example is the Grand Canyon midair in 1956, when two airlines collided midair while complying with prescriptive regulations.

An airport or airline working within a prescriptive regulatory environment is required to maintain 100% compliance with 100% of the prescriptive regulations at 100% of the times. On the other hand, an organization working within a performance compliance environment works within a 95% confidence level. A confidence interval (CI) is a range of values that is likely to include a population value with a certain degree of confidence. It is often expressed as a % whereby a population mean lies between an upper and lower interval. A confidence level is the percentage of times you expect to get close to the same estimate if you run your experiment again or resample the population in the same way. The confidence interval consists of the upper and lower bounds of the estimate you expect to find at a given level of confidence.

For airport and airline operators it is a leap of faith into the unknown to jump from a prescriptive compliance culture over to a performance-based compliance culture. When emotions are involved, it is difficult to accept and change from a 100% prescriptive compliance level to a 95% performance confidence level. Emotions do not comprehend that it is impossible to comply and maintain requirements of a prescriptive compliance level. Conventional wisdom is when there are no occurrences or events reported, an operator maintains 100% compliance with the prescriptive regulations.

A prescriptive regulatory requirement sets out a specific standard, requirement or process to follow or actions that a regulated party must take in order to achieve compliance. There is no leeway for errors, or to start over again when tasks don’t go according to plan. The only way to maintain compliance is to avoid occurrences by not reporting any. Since a task must be completed when operating in a prescriptive environment, it becomes impossible to go back and correct a non- compliance since the non-compliance item already has happened. A prescriptive culture is a system where the last link in the chain of events becomes the focus item since all prior tasks were completed to 100% satisfaction. In a prescriptive culture an occurrence may be justified by management statements only. A general statement after an occurrence would be that “Our training and policies were not followed and failure to comply with our policies is not consistent with our culture of safety. We have been in contact with the regulator about this isolated event. No accident took place and there has been no impact on operations.” In a prescriptive operational culture, someone in the organization must be assigned a role and responsibility to wash their hands after occurrences for operations to continue. Within prescriptive compliance system occurrences are justified with little or no improvements made to safety.

For a prescriptive compliance culture to be effective, monitoring of operations must be ongoing, and from a regulatory oversight perspective view all prescriptive regulations or standards must be applied equally to all operators. Since it is impossible for the regulator to maintain 100% oversight of 100% of the certificate

holder at 100% of the time, a prescriptive system will always be reactive. In addition, before a violation can be assessed, it must have occurred, and a reactive approach required.

When analyzing two separate events within a prescriptive compliance system, no changes were made to operations and the violation repeated itself. It was reported that the aircraft landed on runway 24 when the runway was snow covered. During the landing roll the left main gear contacted deeper snow and the aircraft veered to the left. The left main gear caught a 14-inch windrow along the south edge of the runway and the pilot lost directional control. The aircraft departed the runway surface to the left and the nose gear collapsed. The aircraft sustained substantial damage to the nose and propellers. A prescriptive non-compliance was discovered after two events took place. The first event was a snow-covered runway, and the second event when an aircraft hit the windrow. Fast forward 11 years and the same non-compliance occurred, but this time without affecting aircraft operations. An airport operated with 1/8-inch snow on top of ice covering 100% of the runway. In addition, the airport operated with 4 feet tall windrows extending 10 feet onto the runway. Another example is when an aircraft departed with ice and crashed just after takeoff. Fast forward another 5 years and an aircraft was reported to take off with ice adhering to the critical surfaces, but this time also without affecting aircraft operations. Both were non-conforming in a prescriptive environment, but without any changes made to the processes. Runways remain snow covered, and aircraft keep on departing with ice. After severe accidents the prescriptive compliance environment approach comes in handy to pass judgement, point fingers and arresting flight crew after runway incursions. With this approach there is no need for an investigation since judgement has already been passed.

For an enterprise to run their SMS successfully, a daily quality control system must in place as a system where there is monitoring of drift and deviations. Drift and deviations are not necessarily hazardous to aviation safety, but they are undocumented and therefore the output of the process does not conform to an expected outcome. Drift and deviations often occur due to ineffective processes, processes that are cumbersome to work with, management may expect unnecessary tasks to be completed, or a process compliance factor within a performance-based compliance system is not comprehended.

When a successful SMS is run with a daily quality control system, each process is linked to a regulatory requirement. The outcome of the process must meet its expected outcome and conform to regulatory requirements, standard requirements, and the SMS policy. Every process, task and action must conform to the SMS policy for an enterprise to run a successful SMS.

The very first item of a daily quality control system is to establish the roles and responsibility of the accountable executive (AE). Their role, for both airports and airlines, is to be accountable on behalf of the certificate holder (CH) for meeting the requirements of the regulations. This can be achieved by combining multiple systems, or it can be achieved by applying the quality assurance system as the primary compliance requirement. A quality assurance system is the single most important system within the safety management system to maintain compliance in a performance based regulatory environment. A prerequisite for the quality assurance system is to operate with a daily quality control system. When operational management, or operational control, for each operational task is conducted at regular intervals, their data points become immediately available for analysis, monitoring and oversight management. Just as cash in the bank is counted daily (or more often), process conformance must be counted daily. Running an SMS is hard work. There are no shortcuts, or simple way to run an SMS since individual SMS enterprises operates with their own currency, or conforming processes tailored to their operations.

Depending on size and complexity of an airport or airline, daily quality control includes several tasks to be completed hourly, daily, weekly, monthly, annually, or triennially. Triennially is the outer limit since a triennial audit process is required to conform to regulatory requirements. When running an SMS, both airlines and airports need to conduct system analyses and convert these analyses into operations plans. A daily quality control system includes operations plans for each

safety critical areas and safety critical functions in their operations. Safety critical areas and safety critical functions are designated hazard classifications in a hazard register and analyzed for trends and compliance with SPC control charts.

SMS is reliability between operators. It is an airport operator’s role to ensure aircraft ground deicing systems are available and it is the captain’s responsibility to apply it. It is an airport operator’s role to ensure that a runway is clear of contamination, and it is the captain’s responsibility to assess runway suitability. It is an airport operator’s role to ensure they have personnel available during their published hours of operations, and it is the captain’s responsibility to learn what those hours are. It is an airport operator’s role to publish in the aeronautical publications, and it is the captain’s responsibility to review publications. It is an airport operator’s role to conduct runway friction testing for turbojet aircraft, and it is the captain’s responsivity to verity NOTAM. The list could co on and on forever. Times are long gone when airport operators could wash their hands by NOTAM non-conformances to comply with standards or regulations. However, airport operators are still publishing NOTAM for their lack of compliance and are unreliable service providers when operating an airport that is incompatible with the safe operation of an airport or aircraft.

A prime example is a true story about an SMS airport that went back to their prescriptive compliance operations since their performance-based operations were extremely successful. In 2016 this airport totally failed a regulatory inspection. The regulator demanded they return their certificate but the airport operator negotiated an agreement with the regulator to invalidate the 2016 inspection and for the airport to start over again with a “clean slate”. The regulator agreed and over the years their new systems and processes became so successful that they forgot the past and reverted back to their pre-2016 processes. Their justification to revert to pre-2016 SMS was that they are at a stage which have ultimately made the airport a safer and more resilient airport, and able to maintain its federal airport certification. Not only did the airport revert, but they also abandoned their post-2016 compliance system.

As in the true example above, and until airports and airlines accept their different roles and responsibilities and comprehend their performance-oriented compliance processes, their safety management system is its own worst enemy.

OffRoadPilots

SMS Performance Evaluation

 SMS Performance Evaluation

By OffRoadPilots

Every operator with a safety management system (SMS) conducts regular SMS performance assessments of their systems. In a healthy performance environment, assessments are conducted daily within a quality control system. As a businesslike approach to safety, an SMS enterprise has an obligation to learn how their systems are performing, in the same manner as a business assess their cashflow daily. In a business the cashflow is the leftover after cash is received and cash paid. If a business's cash acquired exceeds its cash spent, it has a positive cash flow. A positive cash flow means more cash is coming in than going out, which is essential for a business to sustain long-term growth. This same principle goes for a healthy and sustainable long-term growth of a safety management system.

Conventional wisdom is that a healthy safety management system for airports or airlines, is a system without incidents, or a system with reduction of incidents over time. The question to answer with this approach is what is the next approach, or goal after the zero incidents goal is achieved. A goal must be attainable to be a valid goal, and an attainable goal is a goal with a proven tracking record. A zero incidents goal does not come with a proven tracking record. A major global air carrier states that they are committed to the highest standard of safety, and they expect the same standard of all their suppliers. Since there are no industry standards for the aviation industry of what the highest standard is, an airline or airport has an obligation to develop their own proprietary testing methods and stringent internal guidelines above and beyond regulatory requirements to ensure their services are held to the highest standard in safety.

Without a definition, or expectations of what the highest standard is, the standard could be anything an opinion desire and contracts terminated for any suppliers who are unable to comply with these opinions. When applying this approach, a safety management system may conform to regulatory compliance, but a contract agreement may be terminated if unable to comply with expectations. This was also the first approach taken by the regulator to make findings against expectations with the justification that an expectation was linked to a regulatory requirement.

When the safety management system was first implemented, a list of six components, seventeen elements and about 95 expectations were developed to assist operators to design processes that conformed to regulatory requirements. All elements of the SMS were linked to at least one regulation and divided into the following 17 elements.

Elements:

• a safety policy,

• non-punitive reporting policy,

• roles and responsibilities & employee involvement,

• communication,

• safety planning,

• performance measurement,

• management review,

• identification and maintenance of applicable regulations,

• SMS documentation,

• records management,

• reactive processes, 

• proactive processes,

• investigation and analysis,

• risk management,

• training, awareness & competence,

• quality assurance, and

• emergency preparedness & response.

  Within these 17 elements are about 95 expectations as guidance tools to maintain a successful SMS. Depending on size and complexity, a total of 154 expectations were later designed for airport operators to maintain a sustainable and long-term growth SMS.

  Expectations, policies and goals are non-action items, but are tools available in the toolbox for airport or airline operators to run a successful SMS. Without placing expectations as the first item of tools available for purchase, policies, goals, objectives and processes becomes unattainable. Since running an SMS is a businesslike approach to safety, expectations become units available for purchase (example: an expectation unit available is the purchase of a safety policy). Expectation purchased is the cost of the unit (example: the cost of a safety policy is to design and built goals). Developing, maintaining, maintenance and quality control of expectations purchased are the cost of associated actions (example: the cost of a goal is strategic planning of objectives to reach the goal). Conversion of objectives into processes is the value, or return on investment by an operator (example: the conversion of a goal to operate with a bare and dry runway during winterseasons is the value of an objective). Conversion of an objective into processes are goods and services provided to customers (example: a process to maintain a bare and dry runway is to design the 5-Ws + How; What, When, Where, Why, Who and How). As an advertising and marketing tool, processes are converted into procedures and acceptable work practices for targeting leads (example: Assign roles and responsibilities, and action items to operational personnel). When applying a businesslike approach to the SMS and working with a group of leads, the safety management system becomes a blueprint to design and develop sales presentations to convert leads into buyers.

A reduction in incidents or accidents is not a performance improvement or measurement of a safety management system since the outcome of an occurrence cannot be changed. That the number of operational incidents were decreasing annually is only a report stating that there was fewer incident during the last 12- month period than the previous period. Interpreting this information is not an interpretation of an SMS performance but is merely an emotional assessment of events.

The challenge when operating with a safety management system is to convert abstract information into tangible facts, or expectations into cash value. The only known performance measurement tool to measure businesses performance is to measure in cash value. The goal for a business is that there is more incoming cash than money going out, with an objective to operate with processes to make this goal attainable.

An SMS enterprise is obligated to perform SMS performance evaluation daily of their safety policy, their goalsetting process and attainment of these goals, a hazard identification process and evaluation of associated hazard, their process for training of personnel and evaluation of their competency, their internal reporting process, their hazard analysis process, their corrective action plan process, their SMS manual and communication processes to all personnel, including the accountable executive, a process for making personnel and associated contractors aware of their roles and responsibilities, a quality assurance program, an audit program of their safety management system with defined schedules, and any additional requirements for the safety management system that are prescribed under the regulations. One reason for daily observation of performance evaluations is to detect drift, or non-conforming process at an early stage.

When an SMS enterprise has established this platform they have established their foundation to convert SMS performance into cash-performance. More money spent on safety does not necessarily equate to a safer operational environment. It is how cash is allocated and distributed that makes the difference. The first step when evaluating their SMS performance is to operate with a daily quality control system. This system is the cash-register of a safety management system. Items sold and items purchased are registered in this system and assigned to an account with a regulatory requirement.

An SMS enterprise assessing their SMS system within a cashflow system, the complete purchase and cost of the entire SMS system is the annual cost of the SMS manager. The assumption is that the blueprints are completed, SMS designed and built to its completion. Incoming cash, such as landing fees, ticket purchases or freight revenue, can then be allocated to the SMS system and distributed to one of the expectation accounts. Over time a cashflow statement will generate an income or loss cashflow statement allocated to an expectation account. Operating with an SMS businesslike approach is no different than operating a general store where cashflow is allocated to separate accounts.

Establishing an SMS accounting system is the key to a successful SMS and opens the doors to evaluate SMS performance cashflow in SPC control chart environment. SPC control charts is a tool to make changes as needed to out of control processes, to make changes to the average in processes that are in control, or to allocate or remove expectations to the vital few processes that require attention.

OffRoadPilots

The Inverted Iceberg

 The Inverted Iceberg

By IceRoadPilots

The iceberg effect is a variant of the Heinrich Pyramid developed in the 1930’s. Herbert W Heinrich put forward the following concept that, in a workplace, for every accident that causes a major injury, there are 29 accidents that cause minor injuries and 300 accidents that cause no injuries. The Heinrich Law was widely accepted by the global aviation industry as a risk analysis tool and incorporated into the safety management system.

The Heinrich Pyramid is commonly known as the safety pyramid, or the safety triangle, and indicates a relationship between major injuries, minor injuries, and near-misses. The Heinrich Pyramid concludes that injuries and incidents are caused by a human decision to perform an unsafe action, and that by lowering the number of minor injuries, businesses could reduce the total number of major injuries and incidents. While the most often cited figure would suggest an emphasis on human errors, Heinrich also suggested that workplaces focus on hazards, not just worker behavior.

Commonly known as the iceberg ignorance, a variant of the Heinrich Pyramid principle, was later designed as the Iceberg Model. In the iceberg model, only four out of every hundred operational problems are known to the accountable executive (AE), and 96% of issues, or hazards, are hidden from the AE. The foundation of the iceberg theory is that 100%, or all problems, are known to the frontline workers, being flight crew, mechanics, or airport personnel, while only 4% are known to the AE.

The iceberg concept principle is that all serious problems are the result of several smaller problems that went unnoticed or unmanaged. For every serious incident in the iceberg model there were 59 smaller incidents, and 600 minor conditions. Conventional wisdom is that an accountable executive, as the decision authority, needs to be more aware of minor issues and conditions and initiate actions to stop these issues or conditions before they lead to a serious incident. Preventing minor event to escalate into a full-scale disaster is one reason why the iceberg of ignorance matters to the AE, directors, supervisors, and frontline workers. An accountable executive need to make a concerted efforts to be aware of minor issues and conditions, and overcoming this issue happens primarily through changing the hazard reporting system requirements. There are several valid theories and conditions to the iceberg principles, but there is a major flaw, or finding, in the system when the system is relaying on that knowledge of every minor event or preventing minor events will stop future disasters from occurring.

When applying the near- miss principles from the Heinrich Pyramid, the Safety Pyramid, or the Iceberg Model, an SMS enterprise builds their SMS platform on a misconception that major accidents only occurs after several minor incidents or near-misses are identified. When operating within a human factors system, organizational system, supervision system and environmental system, each individual person within each system performs independent of the other persons. Tasks are performed individually in a 3D environment with tasks measured in time (speed), space (location), and compass (direction). A robot would complete the task within the same timeframe each time, it would initiate the task at the exact same location, and it would follow the exact same process every time. Within a mechanical production system, the outcome of a process produces the same outcome without learning from past errors. When relying on these two principles to establish SMS reliability, an SMS enterprise is placing themselves in a box that is very difficult to crawl out of. Several years ago, a flaw was discovered in compressor turbine disk by applying non-destructive testing of the CT disk. This test was the first test after the final production stage and after it had left the production line. The test discovered a flaw in the material and was reported. There was no safety management system reporting avenue at that time, just an inspection report. The time was also before the iceberg model was widely known and understood. This material flaw was the very first flaw in a CT disk that management new of. Applying the safety pyramid as the accepted standard, 300,000 unsafe acts would be required, followed by 600 reports, or near misses before 30 incidents would occur, then 10 serious incidents, and finally one major accident. There have been several CT disk failures in turbine engines, with Sioux City IA as a high-profile accident. This does not imply that the same or similar flaw was the cause, but that applying the safety pyramid principles keeps an SMS enterprise inside the box. Another question to answer is how does the production of a CT disk crate an unsafe act. Since the safety triangle is based on 300,000 unsafe acts, there must be an astronomical number of unsafe acts daily. 300,000 hours equals 12,500 days, or just over 34 years of systematic undetected unsafe acts. For unsafe acts to go undetected, they must be few and hidden within a system that only the front-line workers knows about. An example could be the speed at which large trucks travel. The opinion of an unsafe speed varies from person to person, and it vary between jurisdictions. Each jurisdiction has their own speed limit set for large trucks, with a justification for safety. Applying the logic in the safety pyramid, the lower the speed limit is, the faster number 300,000 is reached and time between major trucing accidents shortens.

Another example is the publishing of airport NOTAM (notice to airmen). An airport operator has a tool in their toolbox to publish NOTAM when there are issues, construction, or events at an airport. When a NOTAM is published, the airport must implement a counteraction to remain within the airport standards. I have seen this time after time, that airports publish NOTAM, but does not action their operations to remain in compliance with the regulations, airport standards, operational processes, or their SMS safety policy. Just recently, this winter, an airport operator was operating with a NOTAM for four days that the runway was 100% ice-covered, with 1/8 inch of dry snow on top of the ice. Their next NOTAM was 3 inches of snow on top of compacted snow. Since the temperature was well below freezing for several days, and the ice was not removed, ice was still the base surface condition, but went unreported. In addition to the ice and snow, the runway was not cleared to full width, but left 10 feet wide windrows, and 4 feet tall on each side of the runway. The runway remained open day and night with these conditions present. Again, applying the safety pyramid principle, an unsafe act four days out of the 180 days of winterseason, gives an airport operator unlimited opportunities to operate with unsafe conditions. An airport general operating limitation is one-half statute mile visibility. An airport operations manual is a legal reference document between the airport operator and the regulator with respect to level of service. When the visibility is below one-half statute mile, the airport must close to remain compliant with their legal document. Another example of unsafe airport operations condition is that this same airport remained open, day and night, with visibility below legal limit. When operating within a safety management system, an airport operator is required to ensure that their airport is suitable for the operations of an aircraft, and sometimes this may include closing of the airport. Eventually the evidence goes away, and the airport operator continues as nothing ever happened. Both examples are current and true stories. True stories are good examples to learn from but applying safety triangle principles keeps an airport operator together with the regulator inside a box that suits their comfort level and they have no reasons to crawl out of.

When applied by an SMS enterprise operating with processes to action unsafe conditions, non-regulatory compliance processes, and hazards identified as an immediate threat to aviation safety, the Heinrich Pyramid, the Frank Bird Safety Pyramid, or the Edward T. Hall Iceberg Principle are tools to maintain conformance with safety management system principles. When actions are applied to unsafe conditions, or non-conforming processes, these three triangles are turned 

inverted, since each unsafe, or non-conforming conditions become learning experiences to be addressed. When the triangles are inverted, other unsafe conditions and non-conformances, or conditions below the waterline also become visible. The reason why the global aviation industry needs their SMS, is to start chipping away of safety concerns that became visible when the triangles turned.

A compliance guidance document states it beautifully that an SMS enterprise needs to complete a review of the finding, or observation, and clearly identify what happened, how widespread it is within its own organization, where it occurred in the system and if it was a policy, process, procedure, or culture issue. It is not the intent for an SMS

enterprise to reiterate the finding or observation, but rather that they do a factual review of the observations as it applies to their own organization. Their review includes a description of relevant factual information related to the non- compliance, identification of the enterprise system that led to the non-compliance, and they identify policy, processes, procedures, practices or organizational culture involved.

Other operators may learn from observations or findings that are shared with them, but corrective action of events is an internal performance task applicable to a single, and specific organization only. In other words, sharing information is sharing information only, and it is not sharing of corrective action plans.

OffRoadPilots

Line-Item Audits

 Line-Item Audits

By OffRoadPilots

Airports and airlines are required to conduct a triennial audit of the entire quality assurance program, calculated from the initial audit, or a series of audits conducted at intervals set out in their manual. There are two phases to a line-item audit. The first phase is a static regulatory compliance, which includes manuals, documents and records. A regulatory compliance audit is in a static environment, without movements or operational tasks. An example could be the runway surface inspection required by an airport stating that this will be done whenever there are changes to the runway conditions, such as snow, slush or standing water etc. This statement conforms to regulatory requirements. However, the important part is that processes in an operational environment also conforms to regulatory compliance. The second phase is process compliance.

A process defined in the safety management system (SMS) manual, are processes that are consistent with regulatory compliance. An example could be the runway surface condition, where the manual conforms, the process described in the operations plan conforms, and records verify that the process was completed, and result submitted. Records requirement for an airline or airport is that their recording systems which do not comprise entries on paper, including computer records, may be used to comply with the record-keeping requirements if measures are taken to ensure that the records contained in the recording systems are protected, by electronic or other means, against inadvertent loss or destruction and against tampering, and a copy of the records contained in the recording systems can be printed on paper.

Conducting a line-item audit begins with an audit of how an SMS enterprise recording systems are protected against inadvertent loss or destruction and against tampering, and if a copy of the records contained in the recording systems can be printed on paper. Paperformat systems written in ink maintain compliance to what level they conform to pre-established process for production sequence verification and the legibility, and includes production date and time. An electronic spreadsheet may be tampered with and may not be compliant unless there is an ongoing hourly or daily, depending on size and complexity, quality control of the system. Electronic systems stored and managed by a general internet technology manager, may have issues interfacing with operations in a secure environment and open the doors for tampering. Several SMS Enterprises make statement to the effect that nobody in their organization will tamper with SMS documentation. While this is true, two reasons to include tampering in the regulation is to preserve the integrity of SMS and in a representative sample of the population tampering happens, and titles or positions are not excluded from the population sample. A cloudbased third-party managing the SMS is the most reliable document and records storage and retrieval process, as long as the third-party is authorized by the accountable executive (AE) and included in their operations manual. Otherwise, it is possible for an operator to lose all data in the blink of an eye and operate with a non-conforming SMS. Non-conforming documents and records processes does not affect the rest of the audit, since an internal audit, or third- party independent audit are not regulatory findings, but are observations or opinions of non-conformances. The regulator is the only body with the authority to issues regulatory findings. An accountable executive may elect to temporarily pause the audit until a satisfactory result of the process integrity has been established.

An accountable executive is a person appointed to be responsible to the regulator for meeting the requirements of the regulations on behalf of the certificate holder. It is not an SMS manager, QA manager or airport manager who is responsible, it is the accountable executive. There is no personal liability associated with the position of an accountable executive as this individual represents the certificate holder. The certificate holder retains all liability for non-compliance with the regulations. At airports where the airport manager is the certificate holder, the airport manager may have accepted this liability. The appointment of an AE does not create an additional burden for operators, as the certificate holder has always been responsible for compliance. The appointment of an AE is primarily a matter of identifying the senior individual who will discharge the certificate holder’s responsibilities, and particular, lead the necessary cultural change.

With the appointment of an AE, a line-item audit becomes an audit of the accountable executive as opposed to an audit of the certificate holder. The outcome remains the same, but with an AE there is one person who is required to answer to internal audits, or regulatory findings. A third- party internal audit are observations and opinions only, while upon sharing an internal audit with the regulatory authority they become findings since the regulator is required to address any safety concerns reported to them.

When the SMS was first implemented by a regulatory requirement, an SMS enterprise was informed that the only responsibility for an accountable executive was to have control of the financial and human resources that are necessary for the activities and operations authorized under the certificate. However, control of financial and human resources are available resources to an accountable executive and are conditions a certificate holder must include in the AE job description to meet the requirements of the regulations.

The line-item audit tool is a tool available to verify that an accountable executive has their systems in place to ensure regulatory compliance. The first part of a line- item audit is to audit manuals for compliance. A manual makes references to standards, policies, processes, procedures, or acceptable practices, which also are audited by a line-item audit. A line-item audit is the most comprehensive and detailed audit available. Manuals and related references are audited in a static environment where an audit match manual text to regulatory references. The next step is the process audit, or to establish what level of regulatory compliance an SMS enterprise operates at. These levels are not scaled levels, but are levels to what deviation from expectations their processes produces. Visual levels of conformance may be published by SPC control charts. A successful SMS includes expectations of outputs defined by the operator. A process without an expectation is only a wish for anything to come true.

A daily quality control system is a requirement for the accountable executive to meet the requirements of the regulation. Without daily knowledge of processes and systems performance, the AE does not have a tool to verify compliance. A simple way to look a this, is to compare SMS performance to cashflow performance, which is closed out daily. A daily quality control system include links in processes to conform to regulatory requirements. With this link, and when a process performs as expected, the regulatory requirement is met. This does not imply that the process cannot be changed but is to monitor if current processes are conforming.

A line-item audit of documents and processes is a supreme tool to ensure that the AE maintain compliance. The beauty of a line-item audit system is that future audits focus on changes in documents and processes. System compliance and monitoring are key factors to maintain a healthy safety management system.

OffRoadPilots

When an Enterprise Quits SMS

 When an Enterprise Quits SMS

By OffRoadPilots

There are several ways to quit a safety management system (SMS) and an SMS enterprise may unintentionally or unknowingly have quit their SMS. A safety management system is an expensive system, requires hard work and the benefits are unknown, assumed, or abstract benefits. Benefits, if any, remain unknown since an SMS cannot tell the future, or make predications to what, where, where, why, who and how an incident will occur.

When a justification is presented to an SMS enterprise, a CAO, a CEO, or municipality, that SMS is expensive and without tangible results there is a strong temptation to accept these facts. There is no evidence that the SMS will cause a reduction of future accidents, incidents or hazards, there is no evidence of higher return on investment, and there is no evidence that an SMS has

produced better qualified flight crew, maintenance crew or airport personnel. That an enterprise quits SMS does not imply that they abolish their SMS program, but that it is possible to operate an ineffective SMS by while producing desired results.

A crucial question to answer for an airline or airport to operate with a successful SMS is “Why does the Global Aviation Industry, being Airlines or Airports, need a Safety Management System (SMS) today, when they were safe yesterday without an SMS?” The simple answer is that an SMS is needed to generate system analyses specific applicable to an airline or airport, and to have a road map when arriving at the fork in the road. An SMS enterprise is operating with defined processes to conform to regulatory requirements and each task within a system analysis is applied to a regulatory requirement and followed up with a quality control system.

An airline or airport operating with an SMS has at the least an SMS Manual in place that includes multiple processes conforming to regulatory requirements. An SMS manual contains at a minimum a safety policy, a process for setting goals and for measuring the attainment of those goals, a process for identifying hazards to aviation, a process for ensuring

that personnel are trained and competent to perform their duties, a process for analyzing of hazards, incidents and accidents and for taking corrective actions to prevent their recurrence, a document containing all safety management system processes and a process for making personnel aware of their responsibilities with respect to them, a quality assurance program, a process for periodic reviews or audits, and any additional requirements for the safety management system.

In addition, the SMS manual contains specific roles and responsibilities for the person managing the safety management system. These responsibilities are to maintain a reporting system for collecting information related to hazards, incidents and accidents, identify hazards and carry out risk management analyses of those hazards, investigate, analyze and identify the cause or probable cause of all hazards, incidents and accidents, maintain a safety data system by electronic means to monitor and analyze trends in hazards, incidents and accidents, and monitors, at defined intervals, and evaluate the results of corrective actions. An SMS manager also monitor the concerns of the civil aviation industry in respect of safety and their perceived effect on the Certificate Holder (CH), and determine the adequacy of the training for the person managing the safety management system and for personnel assigned duties under the safety management system. 

An SMS manager determines what, if any, corrective actions are required and carry out those actions, keeps records of any determination made, and the reason for it. The responsibility of an SMS manager closes the Plan-Do-Check-Act cycle by notifying the CH of any systemic deficiency and of the corrective action taken. A systemic deficiency includes the implementation of a new system to manage the safety management system.

A Certificate Holder lay their foundation from a blueprint of regulatory requirements and builds their SMS system on top of their foundation. The system must be a stable system, where minor deviations are detected as drift, and special cause variations are analyzed within the context of the SMS system with corrective action plans. If a change leading to an identified special cause variations were intended by the airline or airport, the corrective action also includes a safety case for change. A change could be a policy change, process change, or a change in acceptable practices. When drift, or deviations goes unattended, or unmonitored, an AE would have a difficult time to capture that their SMS had fallen into noncompliance. Conventional wisdom is that a previous accepted SMS manual conforms to regulatory requirements, and that new changes to the system does not affect SMS compliance.

An accountable executive is responsible for operations or activities authorized under the certificate and accountable on behalf of the certificate holder for meeting the requirements of the regulations. Without in-depth knowledge of applicable regulations and how operational processes affects these requirements, an AE may take a non-conforming turn at the fork in the road. Generally speaking, a sole proprietor business owner, or a CEO of a corporation review their financial statements regularly. At some point a demand is placed on personnel to assess expenses and find methods and areas to reduce expenses. When analyzing the safety management system, there were zero hazard reports, zero incident reports, zero accident reports and zero concerns raised by personnel about safety in operations. For an untrained eye, when analyzing the SMS with zero results, the cost of operating with an SMS system that does not produce results should be reduced, or eliminated. Since the elimination mitigation of an SMS is unavailable due to regulatory requirements, the prior step backwards is to mitigate the SMS. There are several ways to mitigate an SMS, but a common business solutions are to eliminate non-essential tasks expenses and eliminate tasks that are producing zero results. By eliminating task, such as the task conformance matrix, a new SMS system is linked to the SMS manual. This system is a conforming system, but since it is a totally new system put in place, all prior data, corrective actions and system analyses are invalidated. When a new system is put in place of how to operate an SMS, an airline or airport starts their SMS process all over again.

Simply said, when processes are removed, when a cloudbased SMS service provider is changed, the data collation system is changed, the analysis system is changed or when operational processes are changed for other than identified improvement changes, another first-time gap analysis is needed. Since results are abstract, it is an ongoing uphill battle to raise support for an effective safety management system that does not identify occurrences. An SMS requires hazards, incidents, and accidents to earn this support.

OffRoadPilots

Santa’s Just Culture

Santa’s Just Culture

By OffRoadPilots

On 15 March, 1960, Santa Claus was on a reconnaissance trip to review his last delivery trip, and verification that he had not forgotten or left someone out this time. Since Santa implemented the SMS (Streamlined Mission Service), he would do a verification trip, or a quality assurance of his deliveries to learn from the past and improve for next deliveries. Since the SMS was implemented, Santa also does triennial travel audits of the operations during the month of March. On this day in March, the reindeers suddenly lost all their flying powers. Santa had been so busy with the quality assurance program and audit preparations that he had forgotten to tell the elves to feed the reindeers that morning, and they ran out of power. This was an embarrassing moment for Santa. He was known all over the world for timely deliveries, safe transportation, except for a few roof-top crashes, and with a unique quality to know what presents people in different areas of the world wants. When the power ran out, all electrical systems also failed, including Rudolph’s red nose landing light. All Santa know at that time was that he was somewhere where it was very cold, very flat and many lakes, so he established a nice three degrees straight in approach into the unknown. 

It was a dark night, but Santa had faith in the reindeer autonomous landing system, which was powered by an onboard emergency-elf who generated power by running on a treadmill. Without lights, without a glideslope and without any visible clues on the ground, Santa prepared for the worst. One hazard Santa had identified in his SMS, was that a total power failure was a real probability and gliding to roof-tops without power was implemented in his new training program. It was the SMS Director, Mrs. Santa who discovered in her accident reviews that many of the roof tops incidents were due to lack of power for the hoof-reversals to assist braking a higher speeds. After many glide-approaches, the reindeers became very proficient in hitting their landing spots. But this time it was dak, which they had not experienced before. Santa and Mrs. Santa conducted their no-red-nose power approach scenarios by fist establishing s risk classification number, a risk analysis for a safety risk level, a root cause analysis and at the conclusion, a system analysis of the approach. 

 

Santa could now see the ground, but he had no manual control of the autonomous landing system, so he buckled up the best he could and prepared for a crash landing. The system worked well and commanded the reindeers to flare at exactly the right time and it was a smooth landing. After all the snow was cleared and Santa was looking around, he could see nothing else by snow covered land. He knew he had landed on a lake, and with his extensive knowledge of world geography, he knew exactly what lake it was. It was actually his favourite lakes for summer fishing, and he had been there several times. He had landed in his favorite narrows fishing spot at 466212.95E, 6089107.56N, 48U. Some years earlier a bush pilot flew him in a float plane to this spot, and the pilot did the worst water landing ever. Santa’s friend who came on the fishing trip was a senator and named the landing the Norwegian landing, since he had spent time in the Norwegian Sea with high waves. Santa jumped out of his sleigh and thanked Rudolph and the other reindeers for their smooth landing. After he had looked over the crash site, he got angry and wanted to punish the responsible elf. Luckily, Santa had a direct elf-to-elf telephone and could instantly communicate with Mrs. Santa. When Santa asked her for the name of the responsible elf, she answered that this is not how we run our Streamlined Mission Service (SMS) in a just culture. She also informed Santa that he was the Accountable Elf, and needed to follow the same process as everyone else when conducting root cause analyses. Santa then understood that he could not change a risk level just by the stroke of a pen, but it required hard work. 

 

Santa has struggled with the just culture principles since they implemented SMS. Just culture is a different behavioral concept and must become lasting habits to achieve positive, sustainable change. Generally speaking, there are two types of organizational cultures. The old way is the blame culture, and the new way is the just culture. The old way blame culture is simply to blame the last link in the chain for the occurrence, lack of competency, incompetent to follow procedures and the root cause for the catastrophic evens. A simple old-way example is when a Santa in training was blamed for a crash when both Santa and the Santa trainee were focused on a hoof-down and locked light malfunction, and failed to stay in the air. The blame culture is simple and easy, but human errors or other negatives are not useful for intervention to improve safety. 

In Santa’s SMS there is a just culture and a place where there is trust, learning, accountability, and information sharing. Santa comprehends the principles that elves nature is to resist changes and that Santa and Mr. Santa must take the very fist step, which starts with an action and not words, text messages or social media shows. A just culture change is to move from known into unknown. Some of the senior elves are in opposition to Santa’s SMS because they do not see their own benefits by changing. There is also uncertainty and insecurity when moving into unknown territories and there is opposition to the way changes to a just culture was presented. Santa’s objective is to instill trust since trust is a key ingredient for a successful change. Trust must be earned, and Santa realized that it cannot be implemented organizational wide supported by any other platform. 

 

Santa implemented four just culture platforms in his streamlined missions service (SMS) system. 

 

Trust

·      Believe in reliability

o   Without trust there cannot be expectations to perform

Learning

·      Self improvements

o   Organizations conduct training, but an individual can only improve by learning

§  Without trust, learning becomes difficult

Accountability

·      Forwardlooking accountability

o   With trust and learning, accountability to tasks becomes possible

Information sharing

·      Learn from others and the past

o   Without trust, learning and accountability there is no valuable information to be shared

 

The sun was rising in the East and finally help arrived for Santa. The rescue crew brought food, dry clothes (a used Santa suit) and tools to repair the broken systems. Santa enjoyed the company talked about the old days and all the landing he did over many centuries on the roof of their homes. This was the first time anyone had actually seen Santa and his reindeers. When the repair was done, the snow was cleared off the ice, Santa wanted to do a taxi-run to feel the condition of the ice surface. However, the repair was so well done, that when Santa reach 70 elves-steps per hour, the reindeers lifted off and he was on his way back to the new secret location since the pandemic. 

Santa arrived at home and had a long conversation with Mrs. Clause, who is the SMS Manager and elves, including their process coordinator. Santa did a root cause analysis, since the incident was a special cause variation, and his opinion was to site electrical failure as the root cause, since that caused the reindeer autonomous landing system. Mrs. Santa opposed strongly, but Santa demanded he had the right as the Accountable Elf to decide the root cause by the stroke of a pen. When the elves heard about the root cause, one of the elves came forward and admitted that he had forgotten to feed the reindeers and that is the reason they ran out of power. With Mrs. Santa’s support to the elves, they proposed the root cause to be the reindeer feeding system processes itself and 40% contributed by organizational factors, when compared to elves-factors (15%), supervision factors (25%) and environmental factors (20%). 

 

Santa reviewed his observations and was glad that he knew the area without relying on the GPS (Genuine Path for Santa) for travel routes. Over several centuries Santa had travelled the globe and visited every home, child and adult in the world and provided them with gift. There were no such thing as good kids or bad kids when Santa delivered. This year Santa had heard rumors that the GTS folks are changing the route to only include the good kids. However, within a Santa SMS system there is a just culture and he plan to turn off the GPS route, use his personal rout knowledge and visit all the kids in the world.  

 

Several years later Santa’s emergency landing on a remote and cold lake was published in the newspapers. The newspaper story was very different from the actual events and blamed the reindeers for the emergency. When Santa read the story, he smiled and felt good about living in an SMS just culture where issues can be resolved and improved.  

 

 

OffRoadPilots