SMS Audit Key Pointers
By OffRoadPilots
Audit pointers for the safety management system (SMS) are guidelines and
considerations that auditors use to conduct a thorough and effective audit of
regulatory compliance statements and safety in operations. The specific pointers
may vary depending on the nature and scope of the audit, but here are some
general audit pointers that are applicable in various audit scenarios.
Understand the Business,
Safety Management System,
and Aviation Industry: An
auditor begin their audit by
gaining a deep
understanding of the
audited entity's business
operations, SMS and
industry as airline or airport.
This knowledge is crucial to
identifying key risks and
areas to focus on during the
audit. There are qualifications requirements for an auditor to do financial and tax
audits, and the requirement is more than the ability to use a pen and pencil and
being able to read and write. However, in the aviation industry, both airports and
airlines, these are the only requirements. There are no requirements for an SMS
auditor to be approved by the regulator or have superior knowledge and
experience of the SMS regulations, standards, and compliance requirements. SMS
audits are to be taken with a grain of salt. SMS audits is about processes
conformant to regulatory requirements and is not about opinions of operations
future impact.
Becoming a financial auditor typically requires a combination of education,
professional certification, and relevant skills and experience. An SMS auditor is
required to be accepted by the accountable executive as a third-party auditor, oran inhouse auditor. Auditing a safety management system is a new type of audit
and different than financial or technical audits. An SMS audit is of human factors,
organizational factors, supervision factors and environmental factors in addition to
regulatory compliance. The audit of an SMS is a performance audit and an audit of
its systems and processes, and how processes conform to regulatory
requirements, standards, and the SMS policy. The regulatory portion of an audit is
an audit without consideration of processes if an SMS enterprise is in compliance
with regulations and standards.
Requirements for financial auditors may be used as guidance for SMS auditors of
airports and airlines.
Bachelor's Degree: Most financial auditors have at least a bachelor's degree in
accounting, finance, or a related field. Some auditors may pursue degrees in
business administration or economics as well. Following this principle is an asset
for SMS auditors.
CPA (Certified Public Accountant): Many auditors pursue the CPA credential, which
is highly respected in the field of accounting and auditing. Becoming a CPA typically
requires passing the CPA exam and meeting specific education and experience
requirements, which vary by state or jurisdiction. There are third-party certification
credentials available for SMS auditors which may be helpful tools. Experience as
SMS manager and accountable executive are useful prerequisites for SMS auditors.
Analytical Skills: Auditors need strong analytical skills to review financial
statements, identify discrepancies, and assess the accuracy of financial records.
This also true for SMS auditors.
Attention to Detail: Auditors must have a keen eye for detail to spot errors or
irregularities in SMS performance data. Communication Skills: Effective communication is essential for auditors to explain their findings to clients or management and to document their work in reports.
Knowledge of Accounting Standards: Auditors must be familiar with accounting
principles, auditing standards, and financial reporting regulations (e.g., GenerallyAccepted Accounting Principles or International Financial Reporting Standards).
This principle is crucial for SMS auditor, that they have in-depth knowledge of
airport and airline regulations and standards, and how processes helps SMS
enterprises to achieve regulatory compliance and safety in operations.
Computer Skills: Proficiency in using auditing software and spreadsheet programs
like is important for data analysis and reporting. SPCforexcel.com is an invaluable
tool for SMS auditors. When audits are based on other than data and statistical
principles special cause variations remain hidden and incorrect corrective action
plans are implemented.
Entry-Level Positions: Many
auditors start their careers in
entry-level positions, such as
staff accountant or junior
auditor, to gain practical
experience in auditing. A
competent SMS auditor
must have the following
practical experience:
Airside maintainer: principles
and systems – airport
standards, technical and
processes – build an airport, and airside applications – audit an airport.
Airport manager: manual management, daily quality control and project planning. Accountable executive: an accountable executive is responsible for operations or activities authorized under the certificate and accountable on behalf of the
certificate holder for meeting the requirements of the Canadian Aviation
Regulations.
At the time of writing NLC at CYDQ offers these courses.
Progression: As auditors gain experience, they can move on to more senior roles,
such as senior auditor, audit manager, or even internal auditor positions within
organizations.Staying up to date with changes in auditing standards and regulations is crucial.
Many auditors participate in continuing education and professional development
programs to maintain their skills and knowledge. This is crucial for an SMS auditor
and requires individuals to monitor daily changes affecting their audit processes.
Auditors are expected to adhere to high ethical standards to maintain the integrity
and credibility of the audit process. Independence and objectivity are especially
critical. For smaller SMS operators this becomes an issue of personal
accountability, when internal workers, or the SMS manager is performing the
audit. The regulatory requirement is that SMS audits shall be fulfilled by persons
who are not responsible for carrying out those tasks or activities unless the size,
nature and complexity of the operations justify the fulfilling of those duties by the
person responsible for carrying out those tasks or activities, and a risk analysis, that
the fulfilling of those duties by the person responsible for carrying out those tasks
or activities will not result in an unacceptable risk to aviation safety.
Auditing an SMS enterprise is a highly specialized field. The audit of an SMS
enterprise is not just to audit the outcome, but to audit the processes that
produced the outcome. Just as a financial audit does not accept business expenses
at face value but audit the processes generating a profit or loss. An SMS auditor
must audit the data and processes applied to justify their result.
Audit Planning: Develop a comprehensive audit plan that outlines the scope,
objectives, and audit procedures. Consider the materiality threshold and risk
assessment to determine the level of audit effort needed for different areas. The
regulatory requirements are for SMS enterprises to perform an audit of the entire
quality assurance program carried out every three years, calculated from the initial
audit. The quality assurance program is a quality assurance audit of the entire
airport certificate.
Risk Assessment: Identify and assess operational risks that could impact the
accuracy of SMS enterprises statements. Focus risk assessment on safety criticalareas and safety critical functions and allocate audit resources accordingly. This
does not impact the audit itself, since the audit is of the entire certificate, but is to
prioritize risk assessments in the audit report.
Internal Controls: Evaluate the effectiveness of the organization's internal controls,
including the design and implementation of controls. Test key controls relevant to
the audit. SMS performance assessment is a regulatory requirement for both
airports and airlines. In addition, they are required to monitor the concerns of the
civil aviation industry in respect of safety and their perceived effect on their
operating certificate. Unknown bad news or reputation is a failed audit item.
Sampling: The only
acceptable method to sample
is to use random sampling
and statistical sampling
techniques to select samples
for testing. Sampling based
on gut-feelings corrupts the
audit process. Ensure the
sample is representative of
the population being tested.
When applying statistical
principles and statistical
process control (SPC), any out-of-control tests requires the SMS enterprise to
identify the special cause variation that caused the out-of-control process.
Document Everything: Maintain detailed audit documentation, including the audit
plan, procedures performed, evidence obtained, and conclusions reached. This
documentation is crucial for audit quality and compliance.
Independence and Objectivity: Maintain independence and objectivity throughout
the audit process to ensure that the audit is conducted without bias or conflicts of
interest. Learning how to keep emotions out of an audit comes with correcttraining and experience. An auditor who is trained to use “gut-feelings” is trained
incorrectly and will continue on the path to fail audits. An SMS enterprise may pass
the audit, but the audit itself failed since emotions were the foundation of the
audit process. A failed audit is a hazard to aviation safety. An audit failed by an SMS
enterprise is the correct path for aviation safety. When an audit is based on
emotions, and the auditor is required for the portfolio to fail a certain percentage
of their audits, the probability is that they will fail an SMS enterprise who under
other circumstances would pass the audit. When emotions are applied, the
corrective action plans become hazardous to their operations.
Material Misstatement Detection: Perform substantive testing to detect material
misstatements in their performance statements. This includes testing account
balances, e.g. special cause variations, transactions, e.g. data collection, and
disclosures, e.g. reports to the accountable executive.
Analytical Procedures: Use analytical procedures (SPC) to identify unusual or
unexpected trends, ratios, or fluctuations in raw data collected that may indicate
potential issues.
Audit Evidence: Gather sufficient and appropriate audit evidence such as data to
support audit conclusions. This includes examining documents, conducting
interviews, and performing physical inspections.
Irregularities Detection: Be vigilant for signs of irregularities. Consider factors that
could indicate irregularities and conduct additional testing if necessary.
Irregularities, or manipulation of an SMS is a simplest task, but is most often done
unintentionally by the operators. Within an SMS, an irregularities are often used to
embellish hazards and using emotions when identifying a special cause variation.
Communication: Maintain open and clear communication with management and
the audit committee throughout the audit process, especially regarding significant
audit findings and issues.
Documentation of Findings:
Document any significant
findings, including any
identified control deficiencies
or material misstatements,
and communicate them to
management in a timely
manner.
Audit Reporting: Prepare an
audit report that includes the
auditor's opinion on the
fairness of their SMS and any
other required disclosures or
findings. Fairness of an SMS is gauged by how SMS principles are applied to regulatory, standards, or SMS policy requirements. E.g. An SMS may apply a stronger leverage to third-party contractors than to their own workers.
Gauging a system involves assessing its performance, effectiveness, and various aspects to determine its current state and potential for changes. The specific method and metrics you use to gauge a system will depend on the nature of the system and your goals.
When gauging a system, start by clearly defining what you want to achieve by
gauging the system. What are the objectives, goals, expectations, and desired
outcomes? Understanding the objectives is essential for selecting appropriate
gauging methods and metrics.
Determine the key performance indicators (KPIs) or metrics that are most relevant
to the objectives. These metrics should be measurable, quantifiable, and directly
related to the system's performance. Examples of common metrics include
efficiency, accuracy, productivity, cost-effectiveness, and customer satisfaction.Gather data related to the chosen metrics. Depending on the system, you may
collect data through observations, surveys, interviews, experiments, or by
analyzing existing records and reports. Ensure that the data is accurate and up-to-
date.
Use data analysis techniques such as SPC to evaluate the system's performance
based on the selected metrics. This may involve calculating averages, trends,
variances, or other relevant statistics. Visualization tools such as charts and graphs
can help to present and interpret the data effectively.
Compare your system's performance to established benchmarks or industry
standards. Benchmarking can provide valuable insights into how systems perform
relative to others in the same domain.
Obtain feedback from the accountable executive, stakeholders, users, or others
who have experience with the system. They can provide valuable insights into the
system's strengths, weaknesses, and areas for changes.
Based on the data analysis and feedback, identify the strengths and weaknesses of
the SMS system. Determine what aspects are performing well and where there is
room for changes.
Define specific, measurable, and achievable goals for improving the system. These
goals should align with the SMS policy and objectives and focus on strength
identified during the gauging process. Weaknesses identified may be used for
goalsetting, but focusing on weaknesses does not necessarily, or automatically
strengthen a system. A weakness in the SMS is not necessarily a hazard to aviation
safety and may be required for the system to function. Overcontrolling by
adjusting weaknesses to a strength may cause additional hazards to operations
than working with an imperfect system. Conventional wisdom that a weakness of
an SMS is shown by quantity of hazard data produced. However, adding irrelevant
hazards is the same as overcontrolling the hazard identification process. Theregulatory requirement is than an SMS enterprise operates with a process for
identifying hazards to aviation safety and for evaluating and managing the
associated risks. A hazard which did not affect aviation safety is a non-reportable
hazard. E.g. Birds are hazardous to aviation safety, but when the birds are a mile or
two away, and they did not an unplanned action by the flight crew, such as
reporting to ATC or evasive action, they did not affect safety and therefore not a
reportable hazard.
Continuously monitor the
SMS system's performance
and progress on their path
toward the goals. Update
metrics and data collection
daily to track changes over
time.
The gauging process is not a
one-time event. It should be
an ongoing and iterative
process. Periodically revisit
objectives, metrics, and goals to adapt to changing circumstances and ensure the
system remains effective.
Share the results of the gauging efforts with the accountable executive. Effective communication can foster buy-in and support from workers, customers, users and tenants for change initiatives.
Remember that the specific steps and methods for gauging a system can vary
widely depending on the system's complexity and the context in which it operates.
Customizing the approach to fit the size and complexity of the SMS system is
essential for accurate assessment and meaningful changes.Follow Ethical Standards: Adhere to ethical standards and professional auditing guidelines, such as those established by relevant auditing standards boards.
Continuous or Continual Learning: Stay updated on changes in auditing standards,
regulations, and industry trends to enhance audit quality and effectiveness.
Continuous learning is to refresh current knowledge, while continual learning is to
add new knowledge to current knowledge. Quality Control: Ensure that the audit process follows the SMS enterprise’s quality control procedures and standards.
Timeliness: Complete the audit within the established timeline to meet reporting
deadlines. An audit should be initiated no later than six months prior to the
regulatory audit completion date. The completion date is every three years,
counted from the first audit which was due by March 31.
Feedback and Continuous Improvement: After completing the audit, gather
feedback from the audit team to identify areas for improvement in future audits.
Remember that audit procedures may vary depending on the specific audit
engagement, so it's essential to tailor these pointers to the size and complexity of
the SMS enterprise. Compliance with relevant auditing standards and audit
regulations are critical throughout the audit process. Size and complexity of an
SMS enterprise is not to ignore, or eliminate regulatory requirements for smaller
airport or airlines, but it is to audit to their own established size and complexity
daily quality control program.
When performing an SMS audit there are three key audit pointers, or takeaways
that are crucial for the integrity of the audit.
1) The purpose of an audit is not to fail or pass an SMS enterprise, but to
analyze data collected and recorded by an SMS enterprise.
2) Items subject to analytical testing by statistical process control, perform one
test only and accept the result.3) Recommendations by the auditor are not corrective action plan solutions but are recommendations for the SMS enterprise to identify the special
cause variations which lead to a regulatory non-compliance, and the special
cause variation which lead to a non-conforming operational process.
OffRoadPilots
