Diagnosing CPU 410-5H Redundancy Loss Triggered by MBT and BZS Events

In a redundant SIMATIC PCS 7 CPU 410-5H system, a sudden transition to solo mode initiated by diagnostic buffer 'Event ID 16# 0391 MBT Event' followed by OB72 failover (Event ID 16# 7331) and 'Event ID 16# 0184 BZS Event' indicates a hardware synchronization failure across the fiber-optic sync submodules (IF 964-DP) or the backplane Module Bus Transceiver (MBT). The MBT event signifies a frame dropout or optical jitter on the redundancy link, forcing the standby CPU to take over while the failed CPU resets memory (BZS event) and attempts resynchronization. Resolve this condition by inspecting optical cables for minimum bend radius (30mm), cleaning fiber ferrules, replacing degraded sync submodules, and upgrading CPU firmware.

Anatomy of CPU 410-5H Redundancy Loss: MBT and BZS Event Signatures

In a fault-tolerant SIMATIC PCS 7 CPU 410-5H redundant pair (6ES7410-5HX08-0AB0), the two controllers maintain microsecond-level event-synchronized execution via two dedicated fiber-optic synchronization submodules (IF 964-DP) installed in the CPU faceplates. When a sudden glitch occurs on this optical link, the diagnostic buffer records a distinct cascading sequence:

Event ID 16# 0391 (MBT Event): Represents a Modul-Bus-Transceiver error. The internal communication ASIC detects a hardware frame checksum fault, optical signal dropout, or bus transceiver timeout on the sync path.
Event ID 16# 43F1 / 7331 (Switchover to Standby): The standby CPU detects loss of master synchronization, invokes OB72 (CPU redundancy error), and assumes master status to maintain uninterrupted plant control.
Event ID 16# 43E3 (Transition to Solo Mode): The surviving master transitions from redundant RUN to solo RUN mode.
Event ID 16# 0184 (BZS Event): Represents a Betriebszustandswechsel (Operating State Transition). The failed CPU initiates a self-diagnostic routine, executes an internal memory reset, and enters link-up and update mode to re-synchronize.

Physical Layer Failure Modes: Fiber Optic Synchronization Submodules

Field investigation into recurring MBT events reveals three predominant physical failure modes:

Fiber Optic Bending Radius and Stress: The standard synchronization cables require a minimum bending radius of 30 mm. Pinched cables, tight zip-ties inside cabinet raceways, or cabinet door pinching induce optical attenuation, causing sporadic packet drops during peak sync transmission.
Contaminated Optical Ferrules: Airborne industrial dust, oil vapors, or touching the LC/SC optical connector tips creates optical insertion loss. Microscopic debris interferes with the high-frequency optical transceiver beam.
Thermal Degradation of Sync Submodules: The synchronization submodules (6ES7960-1AA06-0XA0 for short range up to 10m; 6ES7960-1AB06-0XA0 for long range up to 10km) dissipate significant heat. In cabinets lacking adequate forced-air cooling, submodule thermal aging increases bit error rates.

Step-by-Step Diagnostic and Hardware Remediation Workflow

Follow this sequential troubleshooting procedure to stabilize redundant CPU 410-5H operation:

Analyze Diagnostic Buffer Timestamps:
- Save the diagnostic buffer of both Rack 0 and Rack 1 CPUs to text files.
- Verify which rack initiated the MBT event (16# 0391) and match additional information words (000E 0270 026C) to identify whether Sync Line 1 or Sync Line 2 failed.
Physical Inspection and Cable Replacement:
- Inspect the orange/black fiber optic cables for sharp bends or mechanical tension.
- Clean all optical connector ferrules using a specialized optical fiber cleaning click-pen or lint-free alcohol wipes.
- If degradation is suspected, replace the fiber optic patch cables.
Inspect Cabinet Environment and Submodule Seating:
- Power off the standby CPU, unseat the sync submodules, inspect pins for corrosion, and reseat firmly, tightening the retaining screw.
- Verify cabinet temperature remains within the specified range (below 60°C).
Firmware Modernization:
- Review the installed CPU 410-5H firmware. Early firmware versions (V8.0 to V8.2.0) had known sensitivity to sync jitter under heavy communication bursts.
- Upgrade firmware to the latest manufacturer release (e.g. V8.2.2 or V8.2.3) using bumpless redundant firmware updates.

Diagnostic Buffer Event Sequence and Failure Progression Matrix

The following table details the event chain during an MBT-induced redundancy loss on CPU 410-5H:

Event IDEvent NameResponsible Task / OBPhysical System ActivityOperational Impact16# 0391MBT EventInternal ASIC FirmwareOptical frame dropout or sync bus transceiver timeoutPrimary redundancy trigger16# 43F1Master / Standby SwitchoverRedundancy ManagerStandby initiates master takeover protocolSeamless plant control maintained16# 7331CPU Redundancy ErrorOB72 (Priority 28)Controller logs partner failure and evaluates rack healthOperator alarm generated in WinCC16# 43E3Change to Solo ModeOperating SystemMaster operates without partner redundancyRedundancy lost; solo RUN mode16# 0184BZS EventInternal OS RecoveryFailed CPU performs self-test, memory reset, and link-upAutomatic recovery sequence

[!NOTE]
Commercial Accuracy Notice: SIMATIC S7-400H synchronization submodules (6ES7960-1AA06-0XA0 / 6ES7960-1AB06-0XA0) and > optical patch cables are standard hardware accessories. Replacing physical cables or updating CPU 410-5H firmware does > not consume Process Objects or require new System Expansion Card (SEC) capacity authorizations.

Verification

Confirm that CPU 410-5H redundancy is fully restored and stable:

LED Status Inspection: Verify that both redundant CPUs display solid green RUN LEDs and that the REDF (Redundancy Fault) LED is completely extinguished on both controllers.
Diagnostic Buffer Clearance: Monitor the diagnostic buffers over 48 hours. Confirm that no further 16# 0391 MBT events or unexpected OB72 calls appear.
Controlled Redundancy Test: In SIMATIC Manager, perform a manual master-standby switchover via PLC -> Diagnostics/Setting -> Operating Mode -> Switch over. Verify bumpless transition without I/O glitching.

Limitations and Operational Precautions

Adhere to these operational rules when maintaining fault-tolerant systems:

Hot-Swapping Sync Cables: Never disconnect both synchronization cables simultaneously while the system is running; disconnecting both lines creates a split-brain condition where both CPUs may attempt solo operation or halt.
Pair Replacement of Submodules: When replacing sync submodules due to age, replace both upper and lower submodules on both CPUs to maintain identical optical transmitter power.
Firmware Parity: Both CPUs in a redundant pair must run 100% identical firmware releases and hardware revision numbers.

Engineer FAQ

What does 'MBT Event' stand for in the diagnostic buffer?

MBT stands for Modul-Bus-Transceiver. It indicates an internal communication or optical transceiver frame error on the redundancy synchronization interface.

Why did the failed CPU recover automatically after a few minutes?

The BZS event indicates that the failed CPU executed an automated self-diagnostic, cleared its work memory, linked up over the surviving sync line, and performed an online memory copy to restore redundant mode.

Does a single optical cable fault cause a total redundancy loss?

S7-400H controllers utilize two redundant sync submodules. However, if one line has high optical attenuation and a communication burst occurs on the remaining line, an MBT timeout can trigger a momentary redundancy trip.