Resolving WinCC Redundancy Failures and Red Tray Status on Secondary OS Server in PCS 7 V9.0

In SIMATIC PCS 7 V9.0 (WinCC V7.4 SP1) on Windows Server 2016, when the second started redundant OS server turns red in the system tray after 30 seconds and alarms 'redundancy error', the failure is caused by mismatched WinCC Redundancy settings, missing or blocked Simatic Shell terminal bus communication, or orphaned SQL database attachments following manual file deletion. WinCC Redundancy must be configured exclusively on the Engineering Station inside the master OS project—never edited locally on individual servers. To resolve, verify in Simatic Shell that both servers select the identical terminal bus network adapter with matching multicast settings, remove invalid serial port configurations in the WinCC Redundancy editor on the ES, generate server data, and perform a clean sequential download from the Engineering Station.

Executive Summary

During the commissioning of redundant WinCC OS servers (OSS1 and OSS2) on PCS 7 V9.0 Update 3, engineers encountering file write lockouts often resort to stopping Microsoft SQL Server services and manually deleting project directories. Following a fresh download, an intractable redundancy failure occurs: whichever server is started second runs normally for approximately 30 seconds, after which its tray icon turns red, the OS Client marks it unavailable, and an alarm reports 'redundancy error'. Interestingly, tag control and alarm acknowledgments continue to sync. This technical article diagnoses the root causes—including Simatic Shell multicast binding failures, SQL Server instance corruption, and ES-level configuration discrepancies—and provides the definitive resolution procedure.

Architecture and Fundamental Mechanics

The WinCC Redundancy system in PCS 7 V9.0 relies on coordinated multi-layer heartbeat and database synchronization mechanisms:
1. Tray Icon State Mechanics (CCApp.exe / CCRtsLoader.exe):
- When WinCC Runtime starts, the tray icon displays yellow (starting), transitioning to green (active runtime).
- The secondary server initiates a 30-second handshake with the primary server over both the redundant network connection and terminal bus.
- If the secondary server fails to establish redundant state validation within this window, the tray icon turns red (Fault / Redundancy Error), and internal tag @RedundancyState shifts to an error code (value 3 or 4).
2. Simatic Shell Terminal Bus and Multicast Encryption:
- In Windows Server 2016, Simatic Shell governs inter-station discovery.
- If Simatic Shell on OSS1 is bound to Network Adapter A (Terminal Bus), but OSS2 is bound to Network Adapter B (Plant Bus), or if 'Encrypted Communication' is enabled on one server with a mismatched migration password, heartbeat UDP multicasts are dropped.
3. The Perils of Manual Project Folder Deletion:
- Manually stopping MSSQL and deleting project folders leaves orphaned database metadata in the SQL master database and Windows registry.
- When a new project is downloaded, SQL Server cannot register the duplicate database names correctly, causing CCAlgRtServer.exe and CCTlgServer.exe archive synchronization threads to fail and declare redundancy fault.
4. Local vs ES Configuration Drift:
- Modifying Redundancy properties directly on the runtime servers breaks synchronization with the Engineering Station master project database, causing subsequent downloads to overwrite or corrupt configuration tables.

Step-by-Step Implementation Procedure

Clean Up Corrupted SQL Server Database Attachments:
a. On both OS servers, deactivate WinCC Runtime and close WinCC Explorer.
b. Open SQL Server Management Studio (SSMS) connecting to .\WINCC.
c. Expand 'Databases': inspect for any disconnected, offline, or orphaned databases associated with prior deleted projects.
d. Detach or delete orphaned test databases cleanly through SSMS.
e. Restart the server PC to ensure clean operating system handles.
Configure Simatic Shell Terminal Bus Settings on Both Servers:
a. On OSS1 and OSS2, open Windows Explorer, navigate to 'This PC', and right-click Simatic Shell -> Settings....
b. Confirm that the selected network adapter is strictly the Terminal Bus network adapter on both servers.
c. Under 'Multicast Time to Live (TTL)', set the value to 1 (or 2 for routed subnets).
d. Under 'Encrypted Communication', verify both servers share identical settings (either both Disabled, or both Enabled with identical password keys).
e. Click 'OK' and verify that in Simatic Shell, OSS1 can see OSS2 and vice versa.
Configure Redundancy Parameters Exclusively on the ES:
a. On the Engineering Station, open the master OS project in SIMATIC Manager.
b. Open the WinCC project for the OS Server pair.
c. Double-click the Redundancy editor in the WinCC tree.
d. In 'General', verify: 'Default Master' is assigned to OSS1, and 'Redundant Partner Server' is assigned to OSS2.
e. Under 'Connection to redundant partner': select Network (LAN). Uncheck Serial Connection unless an actual physical null-modem RS-232 cable is connected.
f. Under 'Check of the partner server', select the dedicated redundant network connection.
g. Save and close WinCC Explorer on the ES.
Execute Clean Redundant Download from SIMATIC Manager:
a. In SIMATIC Manager on the ES, right-click the OS project and select Compile (Scope: Entire project or Changes).
b. Right-click OSS1 and select PLC -> Download (Scope: Entire project).
c. Right-click OSS2 and select PLC -> Download (Scope: Entire project).
Sequential Runtime Startup and Diagnostic Validation:
a. Start WinCC Runtime on OSS1 first. Wait until the tray icon turns green and @RedundancyState = 1 (Master).
b. Start WinCC Runtime on OSS2. Monitor the tray icon: confirm it transitions from yellow to solid green.
c. Open RedundancyControlDiagnostics.exe (located in C:\Program Files (x86)\Siemens\WinCC\bin) on both servers: confirm 'Redundancy State' displays healthy without timeout errors.

Verification

In WinCC Explorer on OSS1: verify internal tag @RedundancyState is 1 (Master).
In WinCC Explorer on OSS2: verify internal tag @RedundancyState is 2 (Standby / Redundant).
On the OS Client station, verify the redundancy status banner shows both OSS1 and OSS2 with green healthy status bars.

Limitations and Operational Precautions

Never configure or alter WinCC Redundancy properties locally on runtime servers; all changes must originate from the Engineering Station to prevent database schema desynchronization.

Engineer FAQ

Why does the second started server turn red after 30 seconds?

It indicates that the secondary server timed out while attempting to establish redundant heartbeat and database synchronization with the primary server.

What causes redundancy errors when network pings work fine?

Ping only verifies ICMP; WinCC redundancy requires Simatic Shell UDP multicast, SQL Server database communication, and matching Redundancy editor configurations.