Practical SMS Evaluation

Practical SMS Evaluation

By Off Road Pilots 

An SMS enterprise conducts has established regular self

evaluation processes of its safety management system

(SMS). The most effective self evaluation method is to

link daily operational processes to regulatory

requirements, standards, and their SMS policy in a

daily quality control system. A governing principle of

a successful SMS is to collect operational data, e.g.

daily, hourly or by tasks, depending on size and

complexity, for process analysis. Evaluating a safety

management system is ongoing self-evaluation of the

entire system. Airline operators are using this process

in their Flight Data Monitoring system (FDM), where

flight operational parameters are monitored, and data

collected for selected parameters.

Airport operators

are lacking in this

process since there

are no regulatory

requirements for a

monitoring system.

Airports rejected

the voluntary Daily

Rundown system,

which is an

operational Airside

Data Monitoring System and is similar to airlines FDM

system, due to lack of regulatory requirements.

However, since regulations are performance based, and airports are required by regulations to operate with an SMS policy, the Daily Rundown monitoring system is a

required operational data collection and analysis

system for airport operators.

A flight data monitoring system is different that the

aircraft “black box”, since it is a system for airlines

to review collected aircraft parameter data in real

time, or daily. Generally speaking, the two reasons to

collect FDM data points frequently are to know what is

happening in real time, or close to real time, and to

manage manpower workload. It is more effective to work

with 25 datapoints daily for an aircraft, than it is to

catch up with 25,000 data points annually. A self

evaluation process of the safety management system work

by these same principles, which is the collection of

daily data points, or more often if required by size

and complexity of operations.

A self evaluation process is built to include the

required items and tasks of a safety management system.

Items and tasks to include are their safety policy,

process for setting goals and for measuring the

attainment of those goals, process for identifying

hazards and for evaluating and managing the associated

risks, a process for ensuring that personnel are

trained and competent to perform their duties, a

process for the internal reporting and analyzing of

hazards, incidents and accidents and for taking

corrective actions, a document containing (SMS Manual)

all safety management system processes and a process

for making personnel aware of their responsibilities

with respect to them, a quality assurance program, a

process for conducting periodic reviews or audits of

the safety management system and reviews or audits, for

cause, of the safety management system, and any

additional requirements for the safety management

system that are required by the regulations.

A self evaluation of the safety management system includes an evaluation of the quality assurance program. 

An evaluation of a quality assurance program

includes an evaluation of regular scheduled audits or

audits for cause of airport activities. An evaluation

also evaluates how findings resulting from the quality

assurance program are distributed to the appropriate

manager for corrective action and follow-up. There are

two options for the design of an audit system. One

complete audit may be carried out every three years,

calculated from the initial audit, or a series of

audits conducted at intervals set out in the airport

operations manual. When using the Daily Rundown FDM

process at airports, data points are collected daily

for ongoing self-evaluation of the safety management

system.

An evaluation of a safety management system must

include the position managing the SMS. This position

may be named differently, such as the SMS manager

position, safety manager position, or safety officer

position. The aviation industry is moving in the

direction of autonomous aircraft and vehicles. At the

time of writing there are no autonomous safety

management system, and every SMS enterprise has a

person as their SMS manager. Without an evaluation of

this position, the SMS manager position, it is unknown

how successful an SMS enterprise is operating.

An evaluation of the SMS manager position includes the

reporting system, hazard identification, risk

management, investigation, analyzes and root cause of

all hazards, incidents, and accidents. The evaluation

includes an evaluation of their systems to monitor and

analyze trends in hazards, incidents, and accidents,monitor and evaluate the results of corrective actions,

monitor the concerns of the civil aviation industry in

respect of safety and their perceived effect on their

operations and the adequacy of the training required.

An evaluation of the

SMS is different

than an SMS

assessment. The

terms evaluation and

assessment are often

used interchangeably, but

they can have distinct meanings in different contexts.

The exact distinction can vary depending on the field or discipline. In general, here are some common differences.

Purpose:

Assessment typically refers to the process of gathering

information and evidence to make informed decisions

about an individual, a program, or a system. Assessment

can be formative (ongoing, to inform instruction) or

summative (final, to make judgments about

effectiveness).

Evaluation involves making judgments about the quality,

value, or effectiveness of something. Evaluation often

comes at the end of a process and involves assessing

the overall impact or success of an activity, program,

or project.

Scope:

Assessment often focuses on specific skills, knowledge,

or outcomes. It may involve the measurement of

individual performance or the effectiveness of a

specific aspect of a program.

Evaluation examines the broader picture, considering

the overall effectiveness, efficiency, and relevance of

a program or project. It may involve multiple

components and assess how well the entire initiative

met its goals.

Timing:

Assessment is ongoing and occur at various stages of a

process. It may be used to provide feedback and guide

improvement.

Evaluation takes place after the completion of a

program or project. It involves looking back at the

entire process to determine its success and identify

areas for improvement. Using the Daily Rundown system

as their FDM, an SMS enterprise is evaluating processes

upon their completion.

Decisionmaking:

Assessment informs decisions about individual

performance, instructional strategies, or the need for

adjustments during an ongoing process.

Evaluation informs decisions about the overall success

or failure of a program, project, or initiative. It may

influence future planning, funding decisions, or

resource allocation.It is important to note that these 

distinctions are not always rigid, and the terms may be used differently in various fields or contexts. In some cases,

people use assessment and evaluation interchangeably, and the

specific meanings can depend on the goals and

perspectives of organizations involved.

A practical SMS evaluation involves a goalsetting plan.

Without a plan, goals are only wishes and dreams. The

14-step goalsetting plan is a practical plan to

implement into any SMS.

Define Your Vision:

Clearly articulate your long-term vision and ultimate

objectives.

Set Specific Goals:

Break down your

vision into

specific,

measurable,

achievable,

relevant, and time-

bound (SMART) goals.

Prioritize Your

Goals:

Determine the order of importance for your goals and focus on the most critical ones.

Break Goals into Milestones:

Divide each goal into smaller, manageable milestones to

track progress more effectively.

Identify Obstacles:Anticipate potential challenges and 

obstacles that may hinder your progress.

Develop Action Plans:

Create detailed plans outlining the specific actions

needed to achieve each milestone.

Allocate Resources:

Identify the resources (financial, time, personnel)

required to execute your action plans.

Establish a Timeline:

Set deadlines for each milestone to create a sense of

urgency and accountability.

Monitor and Adjust:

Regularly review your progress, identify what's working

and what isn't, and make adjustments as needed.

Celebrate Achievements:

Acknowledge and celebrate small victories along the way

to stay motivated.

Stay Flexible:

Be open to adjusting your goals and strategies based on

changing circumstances or new information.

Seek Feedback:

Solicit input from mentors, peers, or experts to gain

valuable perspectives and insights.

Reflect and Learn:

Reflect on your experiences, learn from both successes

and failures, and apply those lessons to future

endeavors.Reassess and Set New Goals:

Periodically reassess your long-term vision, adjust

goals as needed, and set new objectives based on your

evolving priorities.

An SMS enterprise should customize a 14-step

goalsetting plan within each subheading to align with

specific circumstances and preferences as an airport or

airline operator.

When conducting an evaluation of the SMS, it is just as

important to evaluate non-written processes as it is to

evaluate written processes. Remember, not everything

can be written down to explain a person how to do their

job. There will be times when non-written, but still

acceptable work practices will be applied. A trap for

accountable executives to avoid is if something should

go wrong when unwritten processes are used, is to

restrain themselves from assigning an unwritten task as

the root cause. In the absence of the written process

text, a root cause decision can only be based on

opinions and emotions.

Believing that nobody engages in non-written processes

can pose several dangers for a Chief Executive Officer

(CEO) and the organization as a whole.

Non-written processes often involve informal

communication, ad-hoc decision-making, and undocumented

workflows. If the CEO dismisses the existence of such

processes, they may have an incomplete understanding of

how work is actually done within the organization.Non-written processes can sometimes highlight areas

where personnel have found more efficient or effective

ways of doing their work. Ignoring these processes may

mean missed opportunities to identify and implement

improvements.

Written processes

may not capture all

aspects of

communication and

collaboration. Non-

written processes,

such as informal

discussions or quick

meetings, play a

crucial role in

information

exchange.

Disregarding these

processes may lead

to communication breakdowns and misunderstandings.

Innovation often 

stems from informal discussions, brainstorming sessions, and creative collaboration. 

If the CEO dismisses the importance of non-written

processes, it may hinder the organization's ability to

foster a culture of innovation.

Personnel may feel undervalued or ignored if their

informal contributions are not recognized. This can

lead to decreased morale and engagement, potentially

impacting productivity and overall organizational

performance.Non-written processes sometimes involve compliance-

related activities that may not be documented in formal

procedures. Ignoring these processes can lead to

compliance issues and legal risks for the organization.

Non-written processes often provide the flexibility, or

resilience needed to promptly adapt to changing, or

unforeseen circumstances. Rigid adherence to written

processes alone may hinder an SMS enterprise to respond

effectively to unforeseen challenges. Unwritten

processes or tasks are special cause variations and the

result of lack of organizational preparation of common

cause variations.

Succession planning often relies on a deep

understanding of how work is done within an

organization. Disregarding non-written processes may

make it challenging to identify and groom potential

successors who understand the nuances of the business.

To address these risks, It is crucial for CEOs to

recognize the value of both written and non-written

processes. A balanced approach that acknowledges the

importance of informal communication, creativity, and

adaptability can contribute to a more resilient and

dynamic organizational culture. Both written and

unwritten processes must be accepted by an SMS

enterprise’s SMS policy and their non-punitive

reporting policy.

An SMS evaluation needs to come with a practical

approach. Before assigning someone a practical

approach, it must also be known and understood what an

impractical approach is. Maintaining an impractical

approach to an evaluation means adhering to methods,criteria, or perspectives that are unrealistic or not

feasible in the context of the evaluation process. This

could involve using criteria that are overly

idealistic, difficult to implement, or not relevant to

the goals and constraints of the situation.

For example, if an SMS enterprise is evaluating a

project's success and insist on criteria that are

unattainable or do not align with the project's

objectives, would be maintaining an impractical

approach. This could lead to biased or inaccurate

assessments, as the criteria are not grounded in the

practical realities of the situation.

It is essential in evaluations to strike a balance

between setting high standards and being realistic

about what can be achieved given the resources,

constraints, and objectives of the project or process

being evaluated. An impractical approach can hinder

effective decision-making and problem-solving, as it

may lead to conclusions that are disconnected from the

practical realities of the situation.

Maintaining a practical approach to an evaluation means

adopting a realistic and pragmatic mindset when

assessing a situation, project, or problem. It involves

focusing on tangible and achievable goals, considering

real-world constraints, and utilizing practical

solutions. Here are some key aspects of maintaining a

practical approach to evaluation:

A practical approach is to establish realistic goals

and define achievable and measurable goals that align

with the overall objectives, and considering resources,time, and constraints involved in achieving those

goals.

Data-Driven Decisionmaking is to base evaluations on

reliable and relevant data rather than solely on

assumptions or personal opinions. Use quantitative and

qualitative data to inform decisions and draw

meaningful conclusions.

Cost-Benefit Analysis is to evaluate the costs and

benefits associated with different courses of action.

Consider the return on investment and assess whether

the benefits justify the resources expended. This is

crucial for a successful SMS. Without a cost benefit

analysis, the path forward is unplanned.

A cost-benefit analysis (CBA) is a systematic process

for assessing and evaluating the potential advantages

(benefits) and disadvantages (costs) of a decision,

project, or policy. It is commonly used in business,

economics, and public policy to make informed choices

by comparing the expected benefits with the expected

costs. The goal of a cost-benefit analysis is to

determine whether the benefits outweigh the costs and

whether the investment or decision is economically

justified.

Clearly define the decision, project, or policy under

consideration and establish the boundaries of the

analysis. Identify and list all the potential costs and

benefits associated with the decision or project. These

can include direct costs, indirect costs, tangible

benefits, and intangible benefits. Assign a monetary

value to each cost and benefit. This step can be

challenging for intangible factors, such asenvironmental impact or improved quality of life, but it is essential for comparing all elements on a common scale. 

Account for the time value of money by discounting future costs and benefits to their present value. This involves adjusting future values to reflect their current worth. Determine the net present value by subtracting the total discounted costs from the total

discounted benefits. 

A positive NPV indicates that the

benefits exceed the costs. Assess the impact of

uncertainties or changes in key variables on the

results. This helps to understand how sensitive the

analysis is to variations in assumptions. Establish

decision criteria, such as a minimum acceptable rate of

return or a specific threshold for the benefit-cost

ratio, to guide decision-making, which must be linked

to the SMS policy. Based on the results of the analysis

and the decision criteria, make an informed decision

about whether to proceed with the project, policy, or

investment.

Cost-benefit analysis is an invaluable tool for the

accountable executive because it provides a structured

framework for evaluating options and making choices

that maximize overall welfare or value.

Context Awareness is to understand the specific context

in which the evaluation is taking place. Consider

external factors, cultural nuances, and the environment

that may impact the outcomes.

Practical Recommendations is to provide actionable and

implementable recommendations. Consider the feasibility

and practicality of suggested solutions within the

given context.Stakeholder Involvement is to engage relevant

stakeholders in the evaluation process to gather

diverse perspectives. Take into account the needs and

concerns of those who are directly affected by the

evaluation outcomes.

Flexibility is to

acknowledge that

circumstances may

change, and plans

may need adjustment.

Be open to adapting

the evaluation

approach based on

emerging information

or shifting

priorities.

Communication is to

clearly communicate evaluation findings and

recommendations in a way that is easily understood by stakeholders. Highlight practical implications and

potential actions that can be taken.

Learn from Experience is to reflect on past experiences

and learn from both successes and failures. Use

feedback and lessons learned to continuously improve

the evaluation process.

Incorporating these elements into the SMS evaluation

process, personnel and the accountable executive can

ensure that their assessments are grounded in

practicality, facilitating effective decision-making

and implementation of recommendations.A practical SMS evaluation is to evaluate SMS by monitoring processes, procedures, and acceptable work practices. 

When unwritten processes are identified as

special cause variations, conduct a root cause

analysis, and implement practical tasks into the

written process text. It is crucial for success to

accept that it is impossible for all personnel,

including the AE, do not recall the text, or steps in a

process. The last task completed must therefore act as

a trigger to the next step of the process.

Practical may not always be simple, or easy, but it is

practical. Ultimately, the level of simplicity in

practical solutions depends on factors such as the

nature of the problem, the available resources, the

target audience, and the goals of the solution. It is

essential to strike a balance between simplicity and

complexity, choosing an approach that effectively

addresses the problem at hand while being practical and

manageable.

OffRoadPilots