Line-Item Audits

 Line-Item Audits

By OffRoadPilots

Airports and airlines are required to conduct a triennial audit of the entire quality assurance program, calculated from the initial audit, or a series of audits conducted at intervals set out in their manual. There are two phases to a line-item audit. The first phase is a static regulatory compliance, which includes manuals, documents and records. A regulatory compliance audit is in a static environment, without movements or operational tasks. An example could be the runway surface inspection required by an airport stating that this will be done whenever there are changes to the runway conditions, such as snow, slush or standing water etc. This statement conforms to regulatory requirements. However, the important part is that processes in an operational environment also conforms to regulatory compliance. The second phase is process compliance.

A process defined in the safety management system (SMS) manual, are processes that are consistent with regulatory compliance. An example could be the runway surface condition, where the manual conforms, the process described in the operations plan conforms, and records verify that the process was completed, and result submitted. Records requirement for an airline or airport is that their recording systems which do not comprise entries on paper, including computer records, may be used to comply with the record-keeping requirements if measures are taken to ensure that the records contained in the recording systems are protected, by electronic or other means, against inadvertent loss or destruction and against tampering, and a copy of the records contained in the recording systems can be printed on paper.

Conducting a line-item audit begins with an audit of how an SMS enterprise recording systems are protected against inadvertent loss or destruction and against tampering, and if a copy of the records contained in the recording systems can be printed on paper. Paperformat systems written in ink maintain compliance to what level they conform to pre-established process for production sequence verification and the legibility, and includes production date and time. An electronic spreadsheet may be tampered with and may not be compliant unless there is an ongoing hourly or daily, depending on size and complexity, quality control of the system. Electronic systems stored and managed by a general internet technology manager, may have issues interfacing with operations in a secure environment and open the doors for tampering. Several SMS Enterprises make statement to the effect that nobody in their organization will tamper with SMS documentation. While this is true, two reasons to include tampering in the regulation is to preserve the integrity of SMS and in a representative sample of the population tampering happens, and titles or positions are not excluded from the population sample. A cloudbased third-party managing the SMS is the most reliable document and records storage and retrieval process, as long as the third-party is authorized by the accountable executive (AE) and included in their operations manual. Otherwise, it is possible for an operator to lose all data in the blink of an eye and operate with a non-conforming SMS. Non-conforming documents and records processes does not affect the rest of the audit, since an internal audit, or third- party independent audit are not regulatory findings, but are observations or opinions of non-conformances. The regulator is the only body with the authority to issues regulatory findings. An accountable executive may elect to temporarily pause the audit until a satisfactory result of the process integrity has been established.

An accountable executive is a person appointed to be responsible to the regulator for meeting the requirements of the regulations on behalf of the certificate holder. It is not an SMS manager, QA manager or airport manager who is responsible, it is the accountable executive. There is no personal liability associated with the position of an accountable executive as this individual represents the certificate holder. The certificate holder retains all liability for non-compliance with the regulations. At airports where the airport manager is the certificate holder, the airport manager may have accepted this liability. The appointment of an AE does not create an additional burden for operators, as the certificate holder has always been responsible for compliance. The appointment of an AE is primarily a matter of identifying the senior individual who will discharge the certificate holder’s responsibilities, and particular, lead the necessary cultural change.

With the appointment of an AE, a line-item audit becomes an audit of the accountable executive as opposed to an audit of the certificate holder. The outcome remains the same, but with an AE there is one person who is required to answer to internal audits, or regulatory findings. A third- party internal audit are observations and opinions only, while upon sharing an internal audit with the regulatory authority they become findings since the regulator is required to address any safety concerns reported to them.

When the SMS was first implemented by a regulatory requirement, an SMS enterprise was informed that the only responsibility for an accountable executive was to have control of the financial and human resources that are necessary for the activities and operations authorized under the certificate. However, control of financial and human resources are available resources to an accountable executive and are conditions a certificate holder must include in the AE job description to meet the requirements of the regulations.

The line-item audit tool is a tool available to verify that an accountable executive has their systems in place to ensure regulatory compliance. The first part of a line- item audit is to audit manuals for compliance. A manual makes references to standards, policies, processes, procedures, or acceptable practices, which also are audited by a line-item audit. A line-item audit is the most comprehensive and detailed audit available. Manuals and related references are audited in a static environment where an audit match manual text to regulatory references. The next step is the process audit, or to establish what level of regulatory compliance an SMS enterprise operates at. These levels are not scaled levels, but are levels to what deviation from expectations their processes produces. Visual levels of conformance may be published by SPC control charts. A successful SMS includes expectations of outputs defined by the operator. A process without an expectation is only a wish for anything to come true.

A daily quality control system is a requirement for the accountable executive to meet the requirements of the regulation. Without daily knowledge of processes and systems performance, the AE does not have a tool to verify compliance. A simple way to look a this, is to compare SMS performance to cashflow performance, which is closed out daily. A daily quality control system include links in processes to conform to regulatory requirements. With this link, and when a process performs as expected, the regulatory requirement is met. This does not imply that the process cannot be changed but is to monitor if current processes are conforming.

A line-item audit of documents and processes is a supreme tool to ensure that the AE maintain compliance. The beauty of a line-item audit system is that future audits focus on changes in documents and processes. System compliance and monitoring are key factors to maintain a healthy safety management system.

OffRoadPilots

When an Enterprise Quits SMS

 When an Enterprise Quits SMS

By OffRoadPilots

There are several ways to quit a safety management system (SMS) and an SMS enterprise may unintentionally or unknowingly have quit their SMS. A safety management system is an expensive system, requires hard work and the benefits are unknown, assumed, or abstract benefits. Benefits, if any, remain unknown since an SMS cannot tell the future, or make predications to what, where, where, why, who and how an incident will occur.

When a justification is presented to an SMS enterprise, a CAO, a CEO, or municipality, that SMS is expensive and without tangible results there is a strong temptation to accept these facts. There is no evidence that the SMS will cause a reduction of future accidents, incidents or hazards, there is no evidence of higher return on investment, and there is no evidence that an SMS has

produced better qualified flight crew, maintenance crew or airport personnel. That an enterprise quits SMS does not imply that they abolish their SMS program, but that it is possible to operate an ineffective SMS by while producing desired results.

A crucial question to answer for an airline or airport to operate with a successful SMS is “Why does the Global Aviation Industry, being Airlines or Airports, need a Safety Management System (SMS) today, when they were safe yesterday without an SMS?” The simple answer is that an SMS is needed to generate system analyses specific applicable to an airline or airport, and to have a road map when arriving at the fork in the road. An SMS enterprise is operating with defined processes to conform to regulatory requirements and each task within a system analysis is applied to a regulatory requirement and followed up with a quality control system.

An airline or airport operating with an SMS has at the least an SMS Manual in place that includes multiple processes conforming to regulatory requirements. An SMS manual contains at a minimum a safety policy, a process for setting goals and for measuring the attainment of those goals, a process for identifying hazards to aviation, a process for ensuring

that personnel are trained and competent to perform their duties, a process for analyzing of hazards, incidents and accidents and for taking corrective actions to prevent their recurrence, a document containing all safety management system processes and a process for making personnel aware of their responsibilities with respect to them, a quality assurance program, a process for periodic reviews or audits, and any additional requirements for the safety management system.

In addition, the SMS manual contains specific roles and responsibilities for the person managing the safety management system. These responsibilities are to maintain a reporting system for collecting information related to hazards, incidents and accidents, identify hazards and carry out risk management analyses of those hazards, investigate, analyze and identify the cause or probable cause of all hazards, incidents and accidents, maintain a safety data system by electronic means to monitor and analyze trends in hazards, incidents and accidents, and monitors, at defined intervals, and evaluate the results of corrective actions. An SMS manager also monitor the concerns of the civil aviation industry in respect of safety and their perceived effect on the Certificate Holder (CH), and determine the adequacy of the training for the person managing the safety management system and for personnel assigned duties under the safety management system. 

An SMS manager determines what, if any, corrective actions are required and carry out those actions, keeps records of any determination made, and the reason for it. The responsibility of an SMS manager closes the Plan-Do-Check-Act cycle by notifying the CH of any systemic deficiency and of the corrective action taken. A systemic deficiency includes the implementation of a new system to manage the safety management system.

A Certificate Holder lay their foundation from a blueprint of regulatory requirements and builds their SMS system on top of their foundation. The system must be a stable system, where minor deviations are detected as drift, and special cause variations are analyzed within the context of the SMS system with corrective action plans. If a change leading to an identified special cause variations were intended by the airline or airport, the corrective action also includes a safety case for change. A change could be a policy change, process change, or a change in acceptable practices. When drift, or deviations goes unattended, or unmonitored, an AE would have a difficult time to capture that their SMS had fallen into noncompliance. Conventional wisdom is that a previous accepted SMS manual conforms to regulatory requirements, and that new changes to the system does not affect SMS compliance.

An accountable executive is responsible for operations or activities authorized under the certificate and accountable on behalf of the certificate holder for meeting the requirements of the regulations. Without in-depth knowledge of applicable regulations and how operational processes affects these requirements, an AE may take a non-conforming turn at the fork in the road. Generally speaking, a sole proprietor business owner, or a CEO of a corporation review their financial statements regularly. At some point a demand is placed on personnel to assess expenses and find methods and areas to reduce expenses. When analyzing the safety management system, there were zero hazard reports, zero incident reports, zero accident reports and zero concerns raised by personnel about safety in operations. For an untrained eye, when analyzing the SMS with zero results, the cost of operating with an SMS system that does not produce results should be reduced, or eliminated. Since the elimination mitigation of an SMS is unavailable due to regulatory requirements, the prior step backwards is to mitigate the SMS. There are several ways to mitigate an SMS, but a common business solutions are to eliminate non-essential tasks expenses and eliminate tasks that are producing zero results. By eliminating task, such as the task conformance matrix, a new SMS system is linked to the SMS manual. This system is a conforming system, but since it is a totally new system put in place, all prior data, corrective actions and system analyses are invalidated. When a new system is put in place of how to operate an SMS, an airline or airport starts their SMS process all over again.

Simply said, when processes are removed, when a cloudbased SMS service provider is changed, the data collation system is changed, the analysis system is changed or when operational processes are changed for other than identified improvement changes, another first-time gap analysis is needed. Since results are abstract, it is an ongoing uphill battle to raise support for an effective safety management system that does not identify occurrences. An SMS requires hazards, incidents, and accidents to earn this support.

OffRoadPilots