Troubleshooting Lost OS Server Client Connection on SIMATIC Process Historian

When a PCS 7 redundant OS Server drops connection to SIMATIC Process Historian (PH) shortly after activation while the standby server maintains a continuous heartbeat, the failure is typically caused by corrupted Microsoft Message Queuing (MSMQ) outgoing queues or desynchronized PH-Ready metadata resulting from an interrupted OS download. While the standby server continues updating tag and alarm configurations, the master server's CCProcessHistorian.exe service stalls when attempting to transfer uncommitted schema deltas. To resolve this, stop WinCC Runtime, clear the stuck wincc_ph outgoing message queues in Computer Management, verify PH-Ready client registration certificates, and execute a full OS compilation and download to realign the server database.

Process Historian Client Communication Architecture

SIMATIC Process Historian (PH) serves as the centralized, long-term archiving system for SIMATIC PCS 7 and WinCC. The communication bridge between an OS Server and the central Process Historian is managed by the PH-Ready service suite (CCProcessHistorian.exe and Simatic.ProcessHistorian.Client.exe). This communication relies on Microsoft Message Queuing (MSMQ) over TCP port 1801 and encrypted Windows Communication Foundation (WCF) channels.

When tags or alarms are configured for long-term archiving in WinCC, the OS Server packages metadata configuration frames and cyclical process value packets into MSMQ queues. These queues are forwarded asynchronously to the Process Historian. During runtime startup, a mutual heartbeat handshake verifies the client registration. If the OS Server database schema and the PH configuration tables disagree on tag definitions, or if an MSMQ queue contains malformed data packets from an incomplete project download, the PH client connection is terminated by the historian.

Root Causes of Asymmetric Client Disconnections

In redundant OS server configurations, both servers independently archive data to the Process Historian. When one OS Server functions normally while its partner disconnects after a few seconds of runtime, the discrepancy typically stems from:

Corrupted MSMQ Outgoing Queues: Following an OS project download containing newly archived tags, the master server may generate malformed configuration packets. If MSMQ cannot deliver these packets to PH, they accumulate in the outgoing queue (FormatName:DIRECT=OS:<PH_Server>\private$\wincc_ph_*), blocking all subsequent heartbeat messages.
Interrupted Changes-Only Download: Downloading changes to an active OS server while ArchiveManager is busy creating archive segments can corrupt the internal CC_ExternalApplications or PH_Config SQL tables. Consequently, 'Last alarm configuration' and 'Last tag configuration' timestamps stop updating.
Desynchronized Security Certificates: PH uses Windows certificates to authenticate client nodes. If the master OS Server's certificate is revoked, expired, or mismatched in the PH Management Console, the PH accepts the initial TCP handshake but severs the authenticated session immediately thereafter.

Diagnostic Logs and MSMQ State Inspection Matrix

The following table identifies key diagnostic logs and message queuing indicators used to diagnose PH client dropouts:

Diagnostic LocationFile / Object NameExpected Healthy StateFailure IndicatorWinCC DiagnosticsProcessHistorian.logHeartbeat acknowledged, Status: OKError sending message: Timeout (0x80004005)WinCC System LogSIMATIC_SFS_*.logClean service startup logsCCProcessHistorian.exe terminated unexpectedlyMSMQ Outgoing QueuesDIRECT=OS:<PH_Host>\*Message count 0 or fluctuating lowThousands of unacknowledged messages stuck in queuePH Management ConsoleDiagnostics -> ClientsState: Connected; Heartbeat: ActiveState: Disconnected; Heartbeat: ExpiredSQL Server OS RTdbo.PH_Config tableMatches Standby OS Server GUIDNULL values or invalid timestamps in LastConfigChange

Step-by-Step Recovery and Queue Cleansing Procedure

To restore the failed OS Server client connection to the Process Historian:

Deactivate WinCC Runtime on the affected OS Server. Allow the redundant partner to maintain plant control.
Stop Process Historian Client Services:
- In Windows Services (services.msc), stop CCProcessHistorian and Simatic SFS Service.
Purge Corrupted MSMQ Outgoing Queues:
- Open Computer Management -> Services and Applications -> Message Queuing.
- Expand Outgoing Queues.
- Locate queues directed toward the Process Historian hostname. If thousands of messages are stuck, right-click each queue and select Purge.
- Restart the Message Queuing Windows service.
Re-Register PH-Ready Client:
- Launch the PH-Ready Configuration Wizard from the Siemens SIMATIC program group.
- Confirm the Process Historian server IP address or NetBIOS name.
- Re-establish security certificates and click Finish.
Execute Full OS Compile and Clean Download:
- On the Engineering Station, right-click the OS project and compile with scope Entire OS.
- Download the OS project to the target server while runtime remains inactive.
Activate Runtime and Monitor Heartbeat:
- Start WinCC Runtime on the OS Server.
- Open the Process Historian Management Console on the PH server and observe the client status dashboard.

Verification

Verify persistent communication restoration:
1. Open the Process Historian Management Console on the PH server.
2. Navigate to Diagnostics -> Clients and confirm that both OS Server and Standby OS Server indicate Connected.
3. Verify that the heartbeat timestamp updates continuously every few seconds.
4. Check that Last alarm configuration and Last tag configuration reflect the timestamp of the recent OS download.
5. In WinCC Explorer on the OS Server, open ProcessHistorian.log and verify continuous status code 200 acknowledgments.

Limitations and Operational Precautions

Purging MSMQ Data Loss: Purging MSMQ queues removes uncommitted historical records that were buffered during the failure. These specific historical samples cannot be backfilled.
OS Server Redundancy State: Always ensure the redundant partner server is fully healthy and synchronized before deactivating WinCC Runtime on the failing node.
Full Download Requirement: If SQL metadata is severely corrupted, a 'Changes-Only' download will repeatedly fail; a full OS download with runtime deactivation is required.

Engineer FAQ

Why does the Standby OS Server stay connected while the Master drops?

Each OS Server maintains an independent MSMQ queue and WCF connection to the Process Historian. A corrupt queue or interrupted download on one server does not impact the network pipeline of the redundant partner.

Can Process Historian run without MSMQ?

No. Microsoft Message Queuing is a mandatory architectural dependency of Siemens PH-Ready for reliable, asynchronous packet delivery across Industrial Ethernet.

Where are Process Historian diagnostic logs stored on the OS Server?

Logs are located in C:\Program Files (x86)\Siemens\WinCC\Diagnostics\ProcessHistorian.log and the Windows Event Viewer under Application logs.

What firewall ports must be open between OS Server and Process Historian?

TCP port 1801 (MSMQ), TCP port 135 (RPC/DCOM), TCP port 1433 (SQL Server), and the dynamic WCF communication ports specified in the PCS 7 Security Concept manual.