Diagnosing Simultaneous OS Client Freezes and Transient Hangs in SIMATIC PCS 7

In SIMATIC PCS 7, when all OS Clients freeze simultaneously for approximately one minute before resuming normal operation, the root cause is located at the central OS Server or plant network infrastructure, rather than individual client hardware. The most common triggers are redundant OS Server failover negotiations resulting from transient network jitter, long-running synchronous C-scripts blocking the WinCC script thread, heavy SQL lock escalations during Process Historian store-and-forward flushing, or network switch spanning-tree (STP) recalculations. Diagnose the root cause using 'apdiag.exe' on the OS Server, reviewing WinCC_Sys_01.log, and auditing network switch diagnostics.

Executive Summary

In multi-station SIMATIC PCS 7 architectures (e.g., redundant OS Servers serving 8 or more OS Clients), an alarming failure scenario occurs when every operator workstation in the plant freezes at the exact same instant for 30 to 60 seconds. During this window, faceplates refuse to open, mouse clicks on process valves produce no response, trend curves flatline, and alarm banners cease updating. After roughly a minute, all clients unfreeze simultaneously and catch up on pending telemetry.

Because OS Clients are lightweight presentation nodes that do not run autonomous control logic, simultaneous client-wide freezes point conclusively to a shared bottleneck. This guide analyzes the core architectural causes—spanning WinCC redundancy arbitration timeouts, script thread starvation, SQL storage latency, and industrial Ethernet spanning-tree reconvergence—and provides an exact diagnostic methodology to isolate and resolve the issue.

Architecture and Fundamental Mechanics

The PCS 7 client-server communication model relies on continuous cyclic polling and event callbacks between WinCC OS Clients and redundant OS Servers over the Industrial Ethernet Terminal Bus:

Redundancy Heartbeat and Failover Timeouts: Redundant OS Servers monitor each other across two redundant communication paths: the dedicated redundancy serial/Ethernet link and the terminal bus. If transient network packet loss occurs on the terminal bus, the secondary server detects a missed heartbeat. It enters redundancy arbitration to determine whether the primary server has failed. WinCC redundancy failover protocols enforce a wait-and-arbitrate window (typically 30 to 60 seconds). During this negotiation, the internal WinCC data manager pauses answering client request packets, causing all connected OS Clients to freeze until the active server state is re-established.
Script Execution Thread Starvation: WinCC processes user and global scripts via SCRIPT.EXE. If an engineer implements a synchronous script on the OS Server (for example, executing a blocking ADODB SQL query to an external enterprise database, an unbuffered file write to a slow network share, or an unconstrained loop), the entire WinCC C-script or VBScript thread freezes. While the script waits for an external network response (defaulting to a 30-60 second timeout), cyclic tag updates and client RPC calls back up in memory queues, locking all client user interfaces.
SQL Server Storage Bottlenecks: When WinCC Tag Logging or Alarm Logging compresses and archives segment files or flushes massive store-and-forward backlogs to Process Historian, the Microsoft SQL Server engine can experience severe I/O disk contention. If disk queues saturate, SQL lock escalations occur, temporarily stalling read/write transactions across the server data manager.
Spanning Tree Recalculations: If the plant terminal bus switches are configured with classic Spanning Tree Protocol (STP) instead of Rapid Spanning Tree (RSTP), any temporary link bounce triggers a network topology recalculation, blocking all network traffic across the switches for 30 to 50 seconds.

Step-by-Step Implementation Procedure

Follow these steps to diagnose and resolve simultaneous OS Client freezes in SIMATIC PCS 7:

Inspect WinCC System Logs on Redundant OS Servers:
- Log on to both OS Server 1 and OS Server 2.
- Navigate to the WinCC diagnostic directory: C:\Program Files (x86)\Siemens\WinCC\Diagnostics.
- Open WinCC_Sys_01.log (and WinCC_Sys_02.log) in a text editor.
- Search for entries matching the exact timestamp of the freeze.
- Look for redundancy messages such as:
- Redundancy: Partner state changed to FAULT / STANDBY
- CCRedEngine: Ping timeout on partner station
- Cannot connect to server... timeout 60000ms
- If redundancy state transitions appear, the freeze was caused by a redundancy failover timeout triggered by terminal bus packet loss.
Analyze Script Execution Performance with APDIAG:
- On the active OS Server, navigate to C:\Program Files (x86)\Siemens\WinCC\uTools (or bin).
- Launch apdiag.exe as Administrator.
- In the APDIAG menu, select 'Output Window' to capture runtime script errors.
- Select 'Queue' > 'Check' to inspect the execution queues for C-scripts and VBScripts.
- Observe the number of queued scripts during peak plant operation. If the queue count spikes into dozens or hundreds, an unoptimized script is blocking the thread.
- Check for scripts performing synchronous network queries, unbuffered file I/O, or infinite while loops.
Audit Physical Network and Switch Topology:
- Log on to the Web Based Management (WBM) of your SCALANCE terminal bus switches.
- Check the switch event logs for 'RSTP Topology Change' or 'Port Link Down/Up' events matching the freeze time.
- Ensure all switches utilize RSTP (Rapid Spanning Tree) with correctly assigned Root Bridges, rather than legacy STP (which incurs 30-50 second reconvergence delays).
- Verify that network cables and SFP fiber transceivers report zero CRC errors or packet drops.
Inspect SQL Server Disk Performance:
- On the OS Servers, open Windows Performance Monitor (perfmon.msc).
- Monitor the counter PhysicalDisk > Avg. Disk Queue Length.
- If the queue length sustained during archive segmentation exceeds 2 per physical disk, the storage subsystem is bottlenecking SQL transactions.
- Ensure solid-state drives (SSDs) or high-performance RAID arrays are deployed for WinCC project and archive directories.
Verify Antivirus Exclusions:
- Confirm that Microsoft Defender or third-party antivirus suites on the OS Servers have the mandatory PCS 7 exclusions applied to prevent on-access scanning of .mdf and .ldf database files.

Verification

In APDIAG on the OS Server, verify that script queue depths remain consistently near zero during active plant control.
In WinCC Channel Diagnosis on the OS Server, verify that all S7 connections report stable 'State: OK' without cyclic error counts.
Monitor the OS Clients continuously over multiple production shifts to confirm zero recurring freeze episodes.
Intentionally disconnect one terminal bus cable on a redundant server in a planned maintenance window: verify that RSTP converges within sub-second timescales without causing client UI freezing.

Limitations and Operational Precautions

APDIAG must be used carefully in live production; avoid setting excessive trace buffers that could place additional load on the WinCC graphics engine. Never implement blocking network communication inside cyclic WinCC C-scripts; use asynchronous threads or standard OPC interfaces. Hardware switches, servers, and network cards must be validated against Siemens networking specifications through an authorized Siemens distributor or the Siemens Industry Mall.

Engineer FAQ

Why do all clients freeze together instead of just one?

Because all clients depend on the central OS Server for real-time tag updates and screen generation. When the server data manager stalls or renegotiates redundancy, all client request queues halt simultaneously.

What tool identifies long-running WinCC scripts?

The Siemens diagnostic utility apdiag.exe (located in the WinCC uTools or bin directory) displays script execution queues and runtime error outputs.

Can an Ethernet switch cause a 1-minute client freeze?

Yes. If an industrial switch triggers Spanning Tree Protocol (STP) topology recalculation due to a bouncing link, the network completely halts data forwarding for 30 to 50 seconds until convergence completes.

How does WinCC redundancy cause a freeze during failover?

During a network partition, the servers perform heartbeat arbitration. If timeout thresholds are exceeded, client communication is briefly paused while the new master server claims active control.

Does increasing OS Client RAM solve the freezing issue?

No. Because the root cause is central server starvation or network interruption, adding RAM to client IPCs will have zero impact on the freezing symptom.