Annex A (informative) — Metric Examples

A.1 Important Note

The metrics and their targets in table below are ONLY EXAMPLES.

Each organization or plant will determine:

Its own metrics
Its own targets

Actual targets could be different depending upon factors such as:

Process type:

Continuous vs. batch
Chemical vs. refining
Simple vs. complex

Operator skill:

Experienced vs. new
Training level
Team capability

HMI:

System capabilities
Display quality
Usability

Operating environment:

24/7 vs. dayshift only
Staffing levels
Support available

Types of automated procedures:

Startup/shutdown
Normal operation
Emergency response

Significance:

Safety-critical
Production-critical
Non-critical

Customize metrics to YOUR situation!

A.2 Metric Table

Below is comprehensive table with 7 example metrics:

No.	Description	Target	Reason for Metric	Potential Corrective Action
1	Percentage of times operators manually perform a procedure instead of using an available automated procedure	0%	Ensure the business justification for the procedure is met	Analyze why the procedure is not being used as expected
2	The number of operator control interactions performed by the operator in a fixed period (See Note below table)	No more than an average of x interactions in a y period No more than a peak of z interactions in a y period	Manage the operator workload	Increase automation or staffing
3	Number of times an operator had to intervene (deviate, override, abort, change a step…) when an automated procedure was run or per a period	0	Identifies problems with an automated procedure Identifies improper operator training Identifies instrumentation or process equipment problems	Redesign the procedure Improve operator training Correct or replace instrumentation or process equipment
4	The number of times support from engineering is required during execution per period	0	Identifies problems with an automated procedure Identifies improper operator training Identifies instrumentation or process equipment problems	Redesign the procedure Improve operator training Correct or replace instrumentation or process equipment
5	The number of times an automated procedure experienced an abnormal end of execution per period	0	Identifies problems with an automated procedure	Redesign the procedure
6	Average, maximum, minimum, and standard deviation of the duration of an entire automated procedure, tasks, or steps in an automated procedure	Procedure dependent	Identifies opportunities for improving plant operation	Investigate procedure, process and equipment improvement opportunities
7	Product quality and yield measures	Product and procedure dependent	Identifies opportunities for improving plant operation Identify shift differentials if possible Identifies improper operator training Identifies instrumentation or process equipment problems	Investigate procedure, process and equipment improvement opportunities

A.3 Detailed Metric Explanations

A.3.1 Metric 1: Manual Operation Instead of Automation

What it measures:

Percentage of times operators manually perform procedure instead of using available automated procedure.

Example:

Automated startup procedure available
Should be used 100% of time
Actual: Used 85% of time
Metric: 15% manual operation (problem!)

Target: 0%

Why 0% target?

Procedure was justified based on benefits
Benefits only realized if used
Manual operation = wasted investment

Reason for metric:

Ensure business justification met:

ROI requires utilization
Benefits require usage
Investment wasted if not used

Potential corrective action:

Analyze why procedure not being used:

Possible reasons:

Procedure doesn’t work well
Operators don’t trust it
Manual method faster
Training inadequate
Inconvenient to use
Poor usability

Actions:

Interview operators
Observe usage
Identify barriers
Fix problems
Improve usability
Better training

Goal: 100% automated procedure usage!

A.3.2 Metric 2: Operator Control Interactions

What it measures:

Number of operator control interactions performed by operator in fixed period.

See Note below table for definition of “control interactions.”

Target:

Two-part target:

Average:

No more than average of x interactions in y period

Peak:

No more than peak of z interactions in y period

Example targets:

Average: ≤ 30 interactions per hour
Peak: ≤ 50 interactions per hour

Why two targets?

Average alone misleading:

Could be 10/hour for 3 hours, then 100/hour for 1 hour
Average: 32.5/hour (seems OK)
But peak 100/hour = overload!

Peak alone misleading:

Could be 5/hour for 3 hours, then 50/hour for 1 hour
Peak: 50/hour (at target)
But mostly underutilized

Both needed for complete picture!

Reason for metric:

Manage operator workload:

Too many interactions = overload
Operator error increases
Safety risk
Fatigue

Potential corrective action:

Increase automation:

Automate more tasks
Reduce manual interactions
Optimize procedures

OR increase staffing:

More operators
Distribute workload
Reduce individual loading

High interaction rate = automation opportunity!

A.3.3 Metric 3: Operator Interventions

What it measures:

Number of times operator had to intervene:

Deviate from procedure
Override automation
Abort procedure
Change a step
Manual action during automation

When:

Automated procedure was run
OR per period (e.g., per month)

Target: 0

Why 0% target?

Intervention means automation not working
Should run without intervention
Interventions indicate problems

Reason for metric (3 possible causes):

Cause 1: Problems with automated procedure

Logic errors
Design flaws
Not handling conditions
Inadequate exception handling

Cause 2: Improper operator training

Don’t understand procedure
Intervene unnecessarily
Lack confidence
Poor training

Cause 3: Instrumentation or equipment problems

Sensors failing
Valves sticking
Equipment degraded
Maintenance needed

Potential corrective action:

Redesign procedure:

Fix logic errors
Improve exception handling
Better design

Improve operator training:

Better understanding
When to intervene
When not to intervene
Confidence building

Correct or replace instrumentation/equipment:

Fix sensors
Repair valves
Replace degraded equipment
Preventive maintenance

Root cause analysis determines which action!

A.3.4 Metric 4: Engineering Support Required

What it measures:

Number of times support from engineering required during execution per period.

Example:

Startup procedure executing
Problem occurs
Operator calls engineer
Engineer must help

Target: 0

Why 0% target?

Operators should be able to execute without engineering
Engineering support during execution = problem
Not scalable or sustainable

Reason for metric (same 3 causes as Metric 3):

Cause 1: Problems with procedure

Cause 2: Improper training

Cause 3: Equipment problems

Potential corrective action (same as Metric 3):

Redesign procedure
Improve training
Fix equipment

Well-designed procedure + trained operators = no engineering support needed!

A.3.5 Metric 5: Abnormal End of Execution

What it measures:

Number of times automated procedure experienced abnormal end of execution per period.

What is “abnormal end”?

Normal end:

Procedure completes successfully
All steps executed
Objectives achieved
Clean completion

Abnormal end:

Procedure aborts
Error termination
Timeout failure
Exception not handled
Crash

Target: 0

Why 0% target?

Every execution should complete normally
Abnormal ends = bugs or design flaws
Not acceptable

Reason for metric:

Identifies problems with automated procedure:

Software bugs
Logic errors
Unhandled exceptions
Design flaws
Inadequate error handling

Potential corrective action:

Redesign procedure:

Fix bugs
Improve logic
Better exception handling
More robust design
Comprehensive testing

This metric = quality indicator!

A.3.6 Metric 6: Duration Statistics

What it measures:

Duration statistics of automated execution:

Average:

Mean time to complete

Maximum:

Longest execution time

Minimum:

Shortest execution time

Standard deviation:

Variation in execution time

For:

Entire automated procedure
Individual tasks
Individual steps

Target: Procedure dependent

Why procedure dependent?

Each procedure different
Different processes
Different equipment
Different objectives

Must establish baseline:

Measure initial performance
Set targets based on baseline
Track changes over time

Example baseline:

Distillation startup procedure:

Average: 4.0 hours
Maximum: 4.5 hours
Minimum: 3.7 hours
Standard deviation: 0.15 hours

Targets could be:

Average: 3.5 hours (improvement goal)
Maximum: 4.0 hours (control variability)
Standard deviation: 0.10 hours (consistency goal)

Reason for metric:

Identifies opportunities for improving plant operation:

Long average:

Process slow
Automation inefficient
Equipment problems
Optimization opportunity

High maximum:

Inconsistent performance
Occasional problems
Investigation needed

High standard deviation:

High variability
Inconsistent execution
Different operators
Different conditions
Need standardization

Potential corrective action:

Investigate improvement opportunities:

Procedure optimization
Process improvements
Equipment upgrades
Better control strategies

This metric = optimization guide!

A.3.7 Metric 7: Product Quality and Yield

What it measures:

Product quality measures:

Specification compliance
Purity
Grade
Customer satisfaction
Rework rate

Yield measures:

Production quantity
Raw material utilization
Waste/byproducts
Efficiency

Target: Product and procedure dependent

Why product/procedure dependent?

Different products
Different specifications
Different targets
Different markets

Must establish baseline and targets.

Example targets:

Quality: 99% within specification
First-pass yield: >98%
Rework rate: <1%
Production rate: 100 ± 5 tons/day

Reason for metric (4 possible insights):

Insight 1: Opportunities for improving operation

Quality can be improved
Yield can be increased
More value available

Insight 2: Identify shift differentials

Day shift: 99.5% quality
Night shift: 98.0% quality
Different performance!

Why shift differentials?

Training differences
Experience levels
Operator skill
Procedure usage

Insight 3: Improper operator training

Quality varies by operator
Some better than others
Training needed

Insight 4: Instrumentation or equipment problems

Quality degrading over time
Equipment wearing
Sensors drifting
Maintenance needed

Potential corrective action:

Investigate improvement opportunities:

Procedure optimization
Process improvements
Equipment upgrades
Better control

This metric = business value indicator!

A.4 Note on Operator Control Actions

Operator control actions are one component of control room operator loading.

What are operator control actions?

Actions taken by control room operator:

Mode changes
Setpoint changes
Opening/closing a valve
Starting/stopping a pump
Parameter adjustments

Other loading components may include:

Display interactions
Alarm acknowledgment
Radio communications
Field actions (in some cases)

Why measure control actions?

Analysis of operator control action rates is good indicator of:

Overall health of automated procedures:

High rate = procedure requires too much interaction
Low rate = procedure highly automated
Trending = performance over time

Risk of operator error:

Any procedure requiring high level of operator interaction in fixed period:

Increases potential of operator error.

Why?

Workload overload
Attention diluted
Fatigue
Mistakes increase

Important analysis practices:

Interview operators:

Understand their experience
Get qualitative feedback
Context for metrics

Use interview information:

Understand effects of metrics
Identify causes
Guide corrective actions

Peak vs. Average:

Both should be taken into account.

Why both?

Either measurement individually could be misleading:

Example misleading average:

Hour 1: 10 actions
Hour 2: 10 actions
Hour 3: 10 actions
Hour 4: 100 actions
Average: 32.5/hour (seems OK)
But Hour 4 = overload!

Example misleading peak:

Hour 1: 5 actions
Hour 2: 5 actions
Hour 3: 5 actions
Hour 4: 50 actions
Peak: 50/hour (at limit)
But mostly underutilized

The danger:

Any period producing more operator control interactions than can be handled:

Increases likelihood of human error:

Even if average for that interval seems acceptable!

Short overload periods dangerous!

Monitor both: Average AND peak!