Diagnosing Redundant OS Server Performance Degradation Caused by Alarm Floods in PCS 7

In SIMATIC PCS 7 V7.0 SP2, severe OS Client response sluggishness occurring exclusively when both OS Servers run concurrently in redundant mode—while operating normally when either server is stopped—is primarily caused by an alarm flood saturating the WinCC Redundancy synchronization pipeline. In single-server mode, process alarms are committed locally; however, in redundant mode, every incoming and outgoing Alarm_8P message triggers real-time cross-server synchronization telegrams and SQL lock contention between the servers. Generating an Alarm Hit List to identify and eliminate chattering alarms ("bad actors") in the automation program restores redundant server responsiveness immediately.

WinCC Redundancy Synchronization Mechanics and Overhead

In a high-availability SIMATIC PCS 7 installation, redundant OS Servers (Server A and Server B) maintain continuous database parity:
1. Real-Time Process Image: Both servers receive live process data independently from the AS controllers over the Plant Bus.
2. Internal Message Synchronization: When an alarm occurs, the server receiving the event commits it to its local Microsoft SQL Server database and transmits a redundancy synchronization frame across the redundancy network link to its partner.
3. State and Acknowledgment Replication: Acknowledging an alarm on an OS Client requires cross-server verification to ensure both servers update their alarm journal identically.

Under normal operational conditions (typically fewer than 10 alarms per minute), redundancy synchronization consumes negligible network bandwidth and negligible SQL Server processing overhead.

Why Alarm Floods Cripple Redundant Mode Specifically

A perplexing phenomenon occurs when an automation system runs smoothly with a single active OS Server, but exhibits severe client freezes (5 to 15-second delays when opening pictures or faceplates) as soon as the redundant partner is activated:

Operational StateAlarm Processing PipelineSQL Lock ContentionClient Interface ResponsivenessSingle Server ModeAlarms committed to local SQL instance onlyLow (Single writer; no cross-server sync)Fast, normal screen switching (< 1s)Redundant Mode (Healthy)Synchronized over redundancy linkModerate (Standard transaction batches)Normal client responseRedundant Mode (Alarm Flood)Millions of sync telegrams queued; continuous replicationExtreme (Deadlocks between alarm insert and sync threads)Severely degraded (5–15s delays, client freezes)

When a chattering digital sensor or poorly tuned analog transmitter triggers hundreds of alarms per second (an alarm flood), each individual event generates:
* An incoming message (+)
* An outgoing message (-)
* A cross-server redundancy sync request
* A SQL transaction commit across both servers.

This saturates the redundancy network buffer and forces Microsoft SQL Server into perpetual row-level and table-level locks, starving client graphic query threads.

Identifying Bad Actors Using the WinCC Alarm Hit List

To identify the specific plant tags generating the alarm flood:

In WinCC Explorer, open Alarm Logging.
Open the runtime Alarm Control in WinCC Runtime.
Switch to the Hit List view (or configure a Hit List control in Graphics Designer).
Sort the list by Frequency of Occurrence:
- The Hit List immediately ranks the top 10 most frequent alarms in the plant.
- Typically, over 80% of total alarm volume is generated by fewer than 5 'bad actor' tags (e.g., a chattering level switch or fluttering pressure interlock).
Note the block instance paths for these offending tags.

Engineering Solutions for Chattering Alarms in CFC

To eliminate alarm floods and restore redundant performance:

Implement Alarm Hysteresis: On analog monitoring blocks (MonAnL / PIDConL), parameterize a non-zero hysteresis (Hysteresis e.g., 2% of range) to prevent signals hovering around a threshold from continuously triggering alarms.
Apply On-Delay and Off-Delay Filtering:
- In CFC, configure delay timers (TimeOn, TimeOff on MonDiL or channel drivers) to filter out mechanical contact bounce or hydraulic transients.
Alarm Suppression via Feature Bits: Utilize APL alarm suppression features (e.g., MsgLock / MsgSuppression) during equipment startup to inhibit predictable transient alarms.
Alarm Rationalization (ISA-18.2 / EEMUA 191): Decommission redundant, obsolete, or uninformative alarm points in the automation program.

Verification

To verify that redundant server performance is restored:
1. In the WinCC Alarm Hit List, verify that total alarm frequency drops to normal baseline levels (< 1 alarm per minute under steady state).
2. With both OS Server A and Server B running in redundant mode, navigate across process pictures on all 7 OS Clients; confirm that picture selection and faceplate opening occur within 1 second.
3. In Windows Task Manager on both servers, confirm that CPU utilization for sqlservr.exe drops below 15% and network utilization on the Terminal Bus stabilizes.

Limitations and Operational Precautions

Never attempt to resolve redundant server performance issues by permanently disabling the redundancy network link. Disabling redundancy synchronization creates an archive split, resulting in missing historical data and unrecoverable alarm journal divergence. Always resolve the root operational cause by eliminating alarm floods in the controller logic.

Engineer FAQ

What constitutes an 'alarm flood' according to industrial standards?

According to ANSI/ISA-18.2 and EEMUA 191, an alarm flood is defined as an operational state where an operator station receives more than 10 alarms per 10 minutes.

Why didn't the server show high CPU usage in single mode?

In single mode, SQL Server does not have to coordinate distributed transactions or synchronize tables across network sockets, allowing it to absorb alarm bursts without deadlock contention.

How can I measure current alarm frequency in WinCC?

WinCC Alarm Control features a built-in statistical toolbar button that graphs the message rate per hour, allowing engineers to track alarm distribution across plant shifts.