Diagnosing Unplanned Redundant OS Server Failover and Switchover in SIMATIC PCS 7 V8.0 SP2

Unplanned master-to-standby switchovers in a redundant SIMATIC PCS 7 OS Server pair occur when the WinCC Redundancy Manager detects a disruption in process communication, terminal network heartbeat loss, or internal software execution stalls. To identify the root cause in PCS 7 V8.0 SP2, engineers must inspect key diagnostic logs located in C:\Program Files (x86)\Siemens\WinCC\Diagnose\: specifically WinCC_Sys_01.log (system-level component errors), SIMATIC_S7_Protocol_Suite_01.log (AS plant bus connection breaks), and Redundancy.log (precise failover timestamps and trigger causes). Common underlying reasons include transient plant bus disconnections, network interface card (NIC) power-saving disconnects, excessive Global Script delays blocking the Data Manager, or corrupted SQL Server transactions.

WinCC Redundancy State Machine and Failover Criteria

In a SIMATIC PCS 7 redundant OS Server architecture, both servers run parallel runtime processes while continuously monitoring each other's health via two distinct channels: the terminal bus network connection and a dedicated redundancy interconnection (serial RS-232 cable or secondary Ethernet cross-cable). The server designated as Master actively serves connected OS Clients, while the Standby server logs process data and synchronizes archives.

According to the system manual 'PCS 7 - Fault-tolerant Process Control Systems', a server failover is triggered exclusively under specific fault conditions:

Loss of Connection to Automation Systems: If Server A loses communication with the AS controllers while Server B maintains healthy AS connections, mastership immediately transitions to Server B.
Terminal Bus Heartbeat Timeout: If the master server stops responding across the terminal bus for longer than the configured redundancy timeout period.
Internal Process Failure: If a core WinCC runtime executable (such as CCDataMgr.exe, CCAlgV6.exe, or pdlrt.exe) crashes, hangs, or experiences an internal deadlock.
Manual Switchover: Operator command or engineering maintenance request.

Diagnostic Log Architecture and Investigation Checklist

When an unexplained switchover occurs, engineers should not guess or restart systems prematurely. WinCC maintains detailed event traces in C:\Program Files (x86)\Siemens\WinCC\Diagnose\:

Diagnostic Log FilePurposeSpecific Patterns to SearchRedundancy.logRecords state transitions and redundancy triggersSwitch to Master, Switch to Standby, Connection to partner lostWinCC_Sys_01.logCore WinCC system and database transactionsError in Database, Process terminated unexpectedly, Out of MemorySIMATIC_S7_Protocol_Suite_01.logIndustrial Ethernet / Named Connection diagnosticsConnection broken, Error Code 0x8000, PDU timeoutCCProjectMgr.logProject activation, deactivation, and runtime healthHeartbeat failed, Server state changedWindows Event ViewerOperating system kernel and driver eventse1express (Intel NIC drop), TCP/IP link down, Disk IO error

Cross-referencing the exact timestamp of the switchover in Redundancy.log with entries in SIMATIC_S7_Protocol_Suite_01.log almost invariably reveals whether network drops or software stalls precipitated the event.

Common Root Causes for Unexplained Server Switchovers

Field experience across PCS 7 installations highlights four frequent culprits behind mysterious server switchovers:

Network Interface Card Power Management: Windows OS default settings often enable 'Allow the computer to turn off this device to save power' on Intel or Broadcom server NICs. A momentary power-saving state drops Ethernet link for 50ms, tripping WinCC redundancy.
Asymmetrical AS Connection Drops: A loose patch cord, faulty switch port, or duplicate IP address on the plant bus causes Server A to lose communication with even a single AS. Because Server B still communicates with that AS, Server A relinquishes mastership.
Blocking ANSI-C Global Scripts: A user-written Global Script executing synchronous API calls (e.g., waiting for a network file share or executing a slow database query) can block the WinCC Data Manager thread, causing internal health heartbeats to miss their deadline.
Antivirus / Backup Interference: Heavy antivirus scanning of the WinCC project directory or uncoordinated disk backups lock Microsoft SQL Server database files, triggering database timeout exceptions.

Verification

Verify redundancy health and switchover stability using the following diagnostic steps:

Redundancy Internal Tag Check: Monitor internal tags @RM_MASTER, @RM_SERVER_NAME, and @RedundancyServerState on both servers. Confirm stable values (1 on Master, 0 on Standby).
NIC Power Management Audit: In Windows Device Manager, open properties for all network cards. Under Power Management, verify that 'Allow the computer to turn off this device' is unchecked.
Controlled Failover Test: Simulate an AS disconnection on the Master server by disconnecting its plant bus cable. Verify in Redundancy.log that switchover occurs smoothly within 3 to 5 seconds and OS clients transfer without alarm loss.

Limitations and Operational Precautions

When maintaining redundant OS systems, observe these guidelines:

Redundancy Cable Integrity: Ensure the secondary redundancy link (serial or network) is fully functional. If the terminal bus fails while the secondary link is broken, a 'split-brain' condition occurs where both servers declare themselves Master.
Database Archive Synchronization: After a switchover, verify that WinCC Redundancy completes archive synchronization (@RM_SYNCHRONIZING returns to 0) before initiating any subsequent maintenance shutdowns.
Consistent Software Hotfixes: Both redundant servers must have identical PCS 7 service packs, updates, and Windows security rollups installed. Mismatched software versions lead to erratic redundancy behavior.

Engineer FAQ

Where is the Redundancy.log file stored in PCS 7?

It is located in C:\Program Files (x86)\Siemens\WinCC\Diagnose\Redundancy.log. On older 32-bit systems (PCS 7 V7), it resides in C:\Program Files\Siemens\WinCC\Diagnose\.

Can a server switchover cause data loss in historical archives?

No. WinCC Redundancy automatically executes delta archive synchronization via internal SQL replication tasks once both servers are communicating, backfilling missing alarms and process tags seamlessly.

Why does WinCC switch over even though the network ping is fine?

ICMP ping tests only low-level network connectivity. WinCC Redundancy monitors S7 Named Connections to the AS and internal application threads; an application deadlock or S7 PDU timeout triggers a switchover even if ping succeeds.