C135 PCS 7 (PCS7)
6C
6 Cielo Ciencia y Tecnología
·Publicada en C135 PCS 7 (PCS7)

How to Resolve WAN Watchdog Storms and Client Disconnects on Routed PCS 7 Terminal Buses

How to Resolve WAN Watchdog Storms and Client Disconnects on Routed PCS 7 Terminal Buses

In SIMATIC PCS 7 distributed architectures spanning wide area networks (WANs) or routed VLANs, thick OS Clients sporadically disconnect from OS Servers, causing WinCC graphics to turn gray and redraw continuously. This behavior is driven by Simatic Shell's internal watchdog mechanisms: when clients are separated from servers across routed subnets with firewalls, each client issues rapid ICMP ping and multicast/unicast heartbeats every 4 to 5 seconds to every server in the multiproject. If latency exceeds 20 to 50 ms or firewall state tables drop idle RPC/DCOM sockets, WinCC declares a server failure. To resolve these stability failures, deploy dedicated Web clients (WinCC WebNavigator / WebUX) for long-distance remote sites, use Remote Desktop Services, or implement Layer 2 VPN tunneling with tuned firewall TCP timeouts.

Table of Contents

Root Cause: Simatic Shell Watchdog Mechanisms and WAN Latency

SIMATIC PCS 7 Terminal Bus architecture is engineered primarily for local campus Industrial Ethernet local area networks (LANs) where round-trip latency is under 1 ms and packet loss is zero.

To monitor server health and coordinate redundancy failover, WinCC uses Simatic Shell and the CCServerWatchdog service:
1. Every station broadcasts UDP heartbeats on multicast address 224.0.2.0 (Port 5227).
2. On routed multi-subnet systems, stations fall back to unicast ICMP ping sweeps and TCP handshake queries every 4 to 5 seconds to every known server in the server list.
3. Because all clients in a central dispatching center monitor all remote servers (even those belonging to separate plants), dozens of clients concurrently ping remote servers across WAN links, generating hundreds of ICMP requests per minute.
4. If a single watchdog ping packet is dropped by an intermediate router or delayed past the internal timeout threshold, WinCC assumes the primary OS server is dead.

Why Unmanaged WAN Routing Causes UI Freezes and Screen Redraws

When the watchdog timer expires, the OS Client enters a fault recovery routine:
* The Gray Screen Phenomenon: The client initiates an emergency disconnect, turning all dynamic process elements gray and displaying Server Disconnected.
* Cyclic Reconnection Loops: The client immediately attempts to bind to the partner redundant server. It executes Microsoft Remote Procedure Calls (RPC), opens TCP socket connections to Microsoft SQL Server (Port 1433), and re-reads the WinCC Picture Tree.
* UI Distortion: Because this disconnect/reconnect cycle occurs repeatedly (often several times per minute during transient WAN congestion), the client UI continuously resets its graphical cache, shifting menu toolbars and locking operator input.

Firewall and Multicast Traversal Analysis

Deploying Palo Alto, Fortinet, or Cisco enterprise firewalls between remote plant sites and a central control room introduces specific protocol failure points:

Protocol / PortFunction in PCS 7Common Firewall / WAN Failure PointICMP (Echo Request)Simatic Shell station liveness pingFlooding protection or rate-limiting filters block rapid pings, triggering false disconnects.UDP 5227Simatic Shell discovery and coordinationEnterprise WAN routers block IP multicast (224.0.2.0), breaking station discovery.TCP 1433SQL Server runtime database replicationState-inspection firewalls drop idle database sessions without sending TCP RST packets.TCP 135 & High Dynamic PortsMicrosoft DCOM / RPC communicationsDynamic RPC port negotiation fails across strict NAT or unrouted packet inspection.

Tuning the ping rate inside WinCC is not supported; the polling frequency is hardcoded into the compiled runtime binaries.

Method: Selecting and Implementing an Approved Remote Architecture

To eliminate WAN communication crashes and watchdog storms, choose one of the three Siemens-approved architectures:

Option 1: SIMATIC WinCC WebNavigator / WebUX (Recommended for Long Distance)

Replace thick OS Clients at remote dispatching centers with Web clients:
1. Install WinCC WebNavigator or WinCC WebUX on the on-site OS Server.
2. Publish process graphics using the Web View Publisher wizard.
3. Connect remote dispatch operators via standard HTTPS (Port 443) or an encrypted corporate intranet tunnel.
4. Advantage: HTTP/HTTPS communication tolerates high latency (up to 500 ms), eliminates ICMP watchdog storms, and operates cleanly through enterprise firewalls without opening DCOM ports.

Option 2: Microsoft Remote Desktop Services (PCS 7 Terminal Services)

Deploy a local OS Client configured as a Remote Desktop Session Host (RDSH) at the plant site:
1. Install an OS Client on the local plant terminal bus.
2. Enable Remote Desktop Services with the PCS 7 Terminal Server license.
3. Operators at the central dispatching room connect to the local client via Microsoft RDP (Port 3389).
4. Advantage: Process data and watchdog queries remain on the local gigabit LAN; only keyboard, video, and mouse data traverse the WAN.

Option 3: Layer 2 VPN Tunneling (VXLAN / MPLS VPLS) for Thick Clients

If thick OS Clients must be maintained across sites:
1. Implement a Layer 2 bridged VPN tunnel (such as VXLAN or MPLS) between sites to present a single broadcast domain.
2. In Simatic Shell, configure all stations with matching Multicast Proxy settings and identical TTL values (TTL >= 4).
3. On enterprise firewalls, disable ICMP flood rate-limiting for PCS 7 terminal bus IP ranges.
4. Set firewall TCP idle session timeouts for Port 135 and Port 1433 to at least 120 minutes to prevent socket termination.

Verification

To verify that terminal bus stability is restored:

  1. On the central dispatching station, open Simatic Shell in Windows Explorer. Verify that all remote OS servers appear in the navigation tree with active blue icons.
  2. In the Palo Alto / corporate firewall traffic log, verify that TCP connections on Port 135, 1433, and 5227 remain established without DROP or RESET flags.
  3. In WinCC Runtime, observe operator screens continuously for 60 minutes:
    - Confirm zero gray-screen disconnect events.
    - Confirm that alarm banners update within 1 second of occurrence.
  4. Review the WinCC diagnostic file (WinCC_Sys_01.log) located in <Project_Dir>\Diagnostics. Confirm the absence of CCServerWatchdog: Server failure detected warnings.

Limitations and Operational Precautions

  • Latency Ceilings for Thick Clients: Thick PCS 7 OS Clients require round-trip network latency strictly under 20 ms. If your WAN latency exceeds 20 ms, thick clients will experience intermittent synchronization faults regardless of firewall tuning.
  • Bandwidth Demands: Each thick OS client requires a minimum sustained bandwidth of 100 Mbps with low jitter. Connecting multiple thick clients over a shared low-speed WAN quickly saturates available throughput.
  • Separation of Plant Buses: Never bridge the plant/system bus (AS-OS communication) across WAN links; WAN bridging must be restricted entirely to the Terminal Bus.

Engineer FAQ

Can I modify the Simatic Shell ping interval in the Windows Registry?
No. The 4 to 5 second polling frequency is embedded within compiled C++ system components and cannot be altered via registry modifications.

Why do clients at Site X ping servers at Site A that they do not even display?
Because all OS clients belong to the same multiproject configuration. Simatic Shell queries all servers defined within the distributed server data packages (*.pck) to verify global project topology.

Does increasing the Time-to-Live (TTL) in Simatic Shell resolve WAN disconnects?
Increasing the TTL allows multicast packets to traverse more router hops, enabling station discovery across subnets. However, it does not mitigate network latency or packet loss.

Technical References

  • Siemens Industry Online Support Entry ID 109742323: SIMATIC Process Control System PCS 7 PC Configuration and Communication Manual
  • SIMATIC PCS 7 PC Configuration Manual (Document No. A5E00278782)
  • Siemens Industry Online Support Entry ID 109809264: Which ports does PCS 7 use and what firewall rules are required?

Related Articles

Related Tags

  • https://industrialautomation.net/search?query=simatic+pcs7
  • https://industrialautomation.net/search?query=simatic-shell
  • https://industrialautomation.net/search?query=industrial-networks
  • https://industrialautomation.net/search?query=firewall-configuration
20 respuestas