Investigating Spontaneous Reboots of Redundant WinCC OS Servers via Self-Diagnosis and Event Logs

In SIMATIC PCS 7, spontaneous and seemingly random reboots of a redundant WinCC OS Server are triggered by the WinCC Self-Diagnosis watchdog mechanism (CCProjectMgr.exe / AutoReboot). When WinCC detects an internal process crash, thread deadlock, severe memory leak, or tag management database corruption—such as asymmetric OPC tag configurations between servers—the operating system intentionally initiates an emergency reboot to prevent a corrupted server from acting as a zombie master. To identify the failing component, engineers must inspect Windows Event Viewer (Application and System logs), collect diagnostic bundles using the SIMATIC Assessment Suite - Data Collector (SAS-DC / SDT), and audit WinCC Diagnose log files (WinCC_Sys_*.log) for access violations and tag connection faults.

Architecture of the WinCC Redundancy Self-Diagnosis Watchdog

In high-availability SIMATIC PCS 7 process control systems, a redundant OS Server pair (Master and Standby) continuously monitors hardware and software health to guarantee uninterrupted operator visualization. Central to this architecture is the WinCC Self-Diagnosis Watchdog engine (CCProjectMgr.exe, CCDmRuntime.exe, pdlrt.exe).

The watchdog continuously monitors critical subsystem metrics:

Thread Heartbeats: Core processes must respond to internal keep-alive queries within microsecond windows.
Memory Growth Limits: WinCC tracks private byte allocations to detect run-away script memory leaks.
Database Connectivity: Continuous health checks monitor Microsoft SQL Server transactions.

If a core process deadlocks or crashes, leaving the server in an unresponsive state where it cannot serve clients but still holds network locks, the watchdog executes an automated emergency restart (AutoReboot). This forces a clean server switchover to the healthy partner and attempts automatic recovery.

Primary Root Causes of Spontaneous OS Server Reboots

While hardware faults (RAM defects, power supply dips) can cause restarts, spontaneous reboots in redundant PCS 7 servers most frequently stem from software and configuration discrepancies:

Root Cause ClassificationSpecific Technical FaultObservable Failure MechanismDiagnostic IndicatorTag Household AsymmetryOPC / Third-Party tags modified on one server onlyTag Manager encounters unresolvable pointers; thread abortsWinCC_Sys_01.log access violation in CCDataServerRunaway C/VBScriptsInfinite loop or memory leak in global actionDesktop Heap / RAM exhausted; watchdog triggers AutoRebootSCRIPT.EXE application crash in Windows Event ViewerCorrupted SQL Transaction LogPartition D ran low on space; .ldf write stallsSQL Server database transitions to SUSPECT modeEvent ID 1053 / MSSQL$WINCC write timeoutsRedundancy Link FailureNetwork packet drops on serial/Ethernet sync cableServers enter split-brain; watchdog reboots secondary@RM_MASTER collision in Redundancy.log

A particularly common operational error is adding or renaming OPC tags on one server without performing a synchronized full download from the Engineering Station.

Step-by-Step Diagnostic Investigation Workflow (SAS-DC and Event Logs)

To determine the exact process and root cause triggering spontaneous reboots, follow this technical investigation workflow:

Check Windows Event Viewer:
- On the rebooted server, open Event Viewer -> Windows Logs -> Application.
- Filter by Event Sources: Application Error, WinCC, MSSQL$WINCC.
- Look for crashes occurring within 60 seconds prior to the reboot timestamp. Note the faulting application name (e.g., CCProjectMgr.exe, SCRIPT.EXE, pdlrt.exe) and faulting module (.dll).
- Inspect System log around the reboot timestamp. Note Event ID 1074 (indicates who or what initiated shutdown) or Event ID 6008 (unexpected shutdown).
Inspect WinCC Diagnostic Directory:
- Navigate to C:\Program Files (x86)\Siemens\WinCC\Diagnose\.
- Open WinCC_Sys_*.log and search for keywords: FATAL, EXCEPTION, FAIL, or TERMINATE.
- Open SIMATIC_S7_Protocol_Suite_*.log and check for channel disconnections or buffer overruns.
- Open Redundancy.log to audit partner server state transitions.
Collect Diagnostic Data Using SAS-DC (SDT):
- Download and run the SIMATIC Assessment Suite - Data Collector (SAS-DC) (formerly SIMATIC Diagnostic Tool - SDT).
- Select Standard Collection -> WinCC / PCS 7 OS.
- SAS-DC compiles event logs, WinCC traces, SQL error logs, and system crash dumps into a single encrypted ZIP file for Siemens Technical Support analysis.
Reconcile Variable Households: Open SIMATIC Manager on the ES. Execute an OS compilation (Compile OS -> Entire OS) with 'Tags and messages' enabled. Download the compiled project symmetrically to BOTH redundant servers (Entire Project).

Verification

Verify redundant server stability and watchdog health through these diagnostic steps:

Redundancy State Verification: In WinCC Runtime on both servers, inspect internal tags @RM_MASTER and @RM_SERVER_STATE. Confirm that one server is Master (1) and the partner is Standby (2), with zero flapping.
WinCC Diagnostic Log Audit: Monitor WinCC_Sys_*.log in real time. Confirm that zero critical errors or repeated access violations are logged.
Extended Uptime Check: Observe the redundant server pair over a 72-hour operating window. Confirm that zero unplanned reboots occur and that OS Clients fail over seamlessly during manual maintenance testing.

Limitations and Operational Precautions

When managing redundant OS Servers, observe these critical high-availability guardrails:

Never Modify Databases Online on Single Servers: Never use WinCC Explorer to manually edit tags, graphics, or scripts directly on one server in a redundant pair. All modifications must originate on the Engineering Station and be deployed via standard OS download.
AutoReboot Configuration: While AutoReboot can be disabled in the registry for troubleshooting, doing so in production leaves frozen servers running as zombie nodes, preventing OS Clients from executing failovers.
Hardware Diagnostics: If software logs report zero faults prior to shutdown, suspect physical server hardware issues (overheating CPU, failing power supply unit, ECC memory double-bit errors) and inspect server IPMI / iLO logs.

Engineer FAQ

What is the difference between SAS-DC and SDT?

SIMATIC Assessment Suite - Data Collector (SAS-DC) is the modern, expanded evolution of the legacy SIMATIC Diagnostic Tool (SDT), providing automated diagnostic gathering across PCS 7, WinCC, and TIA Portal.

Can a corrupted graphic screen (.pdl) cause an OS Server to reboot?

If the graphic screen contains a C-script with an unhandled pointer exception and the screen is cached on the OS Server (or Web Server), pdlrt.exe can crash, prompting the watchdog to trigger AutoReboot.

Why does Windows System Log report Event ID 1074 before reboot?

Event ID 1074 indicates a graceful software-initiated restart. If initiated by CCProjectMgr.exe, the event details confirm that WinCC Self-Diagnosis requested the reboot.