Diagnosing Severe OS Client Lag and Runtime Freezes Caused by Redundant Server Synchronization Stalls

Severe OS Client response lag (10-15 second delays to open pictures or faceplates) that occurs only when the standby OS Server is activated is caused by WinCC Redundancy Archive Synchronization stalls and Terminal Bus network saturation. When the standby server boots into runtime, the WinCC Redundancy Manager initiates massive SQL archive replication (Tag Logging and Alarm Logging) across the Terminal Bus to backfill missed data. If network bandwidth is constrained, if redundant servers lack a dedicated cross-cable redundancy link, or if clients lack assigned 'Preferred Servers', real-time client tag queries are starved, triggering 'TAG TIMEOUT' errors and frozen screens. To resolve this: assign Preferred Servers to load-balance clients, verify dedicated redundancy communication paths, and monitor CCRedMon.exe.

Executive Summary

In fault-tolerant SIMATIC PCS 7 architectures utilizing redundant WinCC OS Servers (OS Server Master and Standby), plants are designed to maintain full visualization availability even during maintenance outages. A severe operational crisis occurs when an engineering team downloads changes to a standby server and activates runtime: suddenly, all connected OS Clients experience crippling response delays (taking 10 to 15 seconds to switch screens or open faceplates) and eventually freeze completely with blank picture windows and 'TAG TIMEOUT' diagnostic errors. Crucially, deactivating either server instantly restores instantaneous client performance. This article diagnoses the root causes within WinCC redundancy archive synchronization and network topology, and outlines the definitive engineering remedies.

Architecture and Fundamental Mechanics

WinCC Redundancy Synchronization Mechanics, Terminal Bus Contention, and Client Starvation:

How WinCC Redundancy Synchronization Works:
When redundant OS Servers operate normally, each server logs process tags and alarms independently into its local Microsoft SQL Server database segments.
If Server B is deactivated for maintenance or project download, Server A records all runtime data and logs missing gap intervals in internal replication tables (CC_SysMessage).
Upon reactivation, the WinCC Redundancy Manager (CCRedMgr.exe) initiates dynamic archive synchronization: Server A queries its local SQL database, exports missing archive records, and replicates them to Server B across the network to achieve identical database states.
The Mechanism of Client Starvation and Lag:
During synchronization, SQL Server processes gigabytes of historical data. If the archive gap spans days or if database segments are large, synchronization consumes massive disk I/O, CPU cycles, and network bandwidth.
If the plant lacks a dedicated, physical Redundancy Link (a separate network connection between Server A and Server B), archive synchronization traffic floods the Terminal Bus (the network shared with OS Clients).
OS Clients requesting dynamic tag values (CCDataServer.exe) encounter bandwidth exhaustion and SQL query queuing. When the client request timer exceeds the WinCC DCOM timeout limit (default ~10-15s), the client logs TAG TIMEOUT in its diagnostic file and freezes the operator screen.
The Role of Preferred Server Assignment:
If OS Clients are not configured with Preferred Servers, all client stations default to querying the same master server, exacerbating server processing bottlenecks.
Distributing clients (e.g. Clients 1-4 preferred to Server A, Clients 5-8 preferred to Server B) divides client request loads evenly across both machines.

Step-by-Step Implementation Procedure

Monitor Redundancy Synchronization Status via CCRedMon:
* On both OS Servers, open Windows File Explorer and navigate to <WinCC_Installation_Path>in.
* Launch CCRedMon.exe (WinCC Redundancy Monitor).
* Inspect the synchronization status indicators: Check whether 'Archive Synchronization' is actively running, stalled on a specific segment, or completed.
* If synchronization is actively running, observe the progress percentage; allow the backfill to complete before initiating further downloads.
Configure Dedicated Redundancy Network Interface Connection:
* Verify physical network topology: Redundant OS Servers must have a dedicated, direct Ethernet cable (or isolated redundancy switch) connecting Server A and Server B, completely independent of the Terminal Bus and Plant Bus.
* In WinCC Explorer on the OS Server, open the Redundancy editor.
* Under 'Connection to redundant partner', ensure that the dedicated redundancy adapter IP address is selected.
* Check the option 'Serial/Network connection to redundant server' to ensure heartbeat and synchronization traffic does not consume Terminal Bus bandwidth.
Assign Preferred Servers to Load-Balance Clients:
* In SIMATIC Manager on the Engineering Station, navigate to the OS Client projects in Component View.
* Right-click the first group of OS Clients (e.g. Clients 1-4) -> Open Object.
* In WinCC Explorer, right-click Server Data -> Configure.
* In the Server Data dialog, select the server package and assign Server A as the Preferred Server.
* For the second group of OS Clients (e.g. Clients 5-8), assign Server B as the Preferred Server.
* Download the updated server data to all clients.
Verify Terminal Bus and Simatic Shell Health:
* Open Windows Network Connections (ncpa.cpl) on all servers and clients; confirm that link speeds report 1 Gbps full duplex without packet drop errors.
* In Simatic Shell -> Settings, confirm that all machines bind exclusively to the Terminal Bus NIC and share identical encrypted communication settings.
Inspect and Quarantine Damaged Archive Segments (If Sync Stalls):
* If CCRedMon.exe indicates that synchronization is permanently stuck on a specific date, inspect the WinCC diagnostics log WinCC_Sys_01.log.
* If corrupted .mdf archive segments are detected in ArchiveManager, quarantine the damaged segment to allow the redundancy manager to complete synchronization of subsequent records.

Verification

Verify Redundant Server Synchronization and Client Performance:
1. In CCRedMon.exe on both servers, confirm that status displays 'Redundancy Normal' with both servers active and synchronized.
2. On all OS Clients, test screen navigation and faceplate openings; confirm that screens switch instantaneously (< 1 second) without freezing.
3. Inspect client diagnostic logs (<WinCC_Dir>\Diagnostics\WinCC_Sys_01.log); confirm that zero 'TAG TIMEOUT' errors occur.

Limitations and Operational Precautions

Operational Precautions and Maintenance Rules:
- Never download project changes to both redundant servers simultaneously; always update Server B first, verify synchronization, failover clients, and then update Server A.
- Ensure that OS Clients never have physical connections to the Plant Bus; clients must communicate exclusively across the Terminal Bus.

Engineer FAQ

Why do OS Clients lag only when the standby server is activated?

Activating the standby server triggers WinCC Redundancy Archive Synchronization. If synchronization traffic floods the Terminal Bus or saturates SQL Server, client tag queries time out (10-15s delays).

What tool monitors the progress of WinCC redundancy synchronization?

The CCRedMon.exe utility located in the WinCC bin directory displays real-time archive synchronization progress.

What is the purpose of assigning a 'Preferred Server' to an OS Client?

Assigning preferred servers divides operator stations evenly between Server A and Server B, preventing all clients from overwhelming a single server.

Why is a dedicated redundancy link necessary between servers?

A dedicated redundancy link isolates heavy archive replication and heartbeat traffic from the client Terminal Bus, preventing network saturation.

What causes 'TAG TIMEOUT' errors in WinCC client diagnostic logs?

A TAG TIMEOUT error occurs when the OS Client requests process values from the server, but the server fails to reply within the DCOM timeout window due to high CPU, disk, or network load.