What to check on a ControlLogix redundancy system before FAT: a checklist for 5580 and 5570

ControlLogix redundancy increases the availability of the control system: the primary controller runs the program and synchronises data to the secondary so that the secondary can take over control when a fault occurs (1756-UM015O p. 13, p. 33). The 5570 redundancy content formerly in 1756-UM535 is now in 1756-UM015, so this manual applies to both 5580 and 5570 (p. 242). Page numbers refer to UM015O unless stated otherwise.

1. Hardware: matched pairs, correct slots

• Both chassis are the same size; partner modules must be the same type, in the same slot, at the same firmware revision. No I/O is placed in a redundant chassis (p. 19, p. 25).
• 5580: at most one controller per chassis, for example 1756-L83E. 5570: at most two controllers of the same family, catalog numbers 1756-L71…L75, for example 1756-L73 (p. 19–21). Partner controllers must have the same series, catalog number and redundancy firmware revision (p. 20).
• Minimum controller firmware: 5580 from revision 33; 5570 with 1756-L71 from 20.054, other catalog numbers from 19.053 (p. 21).
• Redundancy modules: one per chassis, in the same slot, same catalog number, connected end-to-end by fibre directly, not through a switch. The 1756-RM2 and 1756-RM3 are used on both platforms; the 1756-RM is for 5570 only (p. 22).
• Communication modules must be enhanced types (with a 2 or 4 in the catalog number, such as 1756-EN2TR), not the "B" types such as 1756-ENBT; same firmware within a pair (p. 23). The 1756-EN4TR can be used in a 5580 redundant chassis from Logix Designer v34 (p. 23); for 5570, release notes 36.052_kit1 state that the EN4TR is not supported in the local rack (RN p. 5). ControlNet, for example 1756-CN2R, is available on 5570 only (p. 23, p. 25).

2. Firmware: one bundle for both chassis

Firmware packages: "1756-L8x Enhanced Redundancy Bundle" for 5580, "1756-Lxx Enhanced Redundancy Bundle" for 5570 (p. 40). Every module in both chassis must use firmware from the same bundle, with revisions checked after flashing (p. 44). The bundle release notes list the revision for each catalog number, the module series (RN p. 4) and the compatible software versions (RN p. 2); check against exactly that table. RMCT must be compatible with the redundancy module firmware, otherwise only the Module Info tab is shown (p. 43). With 1756-Lxx bundles from 34.051_kit1, RSLinx RMCT is no longer supported and FactoryTalk Linx RMCT must be used (1756-RM010Q p. 9).

3. Qualification and synchronisation

At start-up, the redundancy modules check that both chassis have the correct modules and firmware (qualification) before synchronising. Crossloaded data includes changed tag values, force values, online edits, project information and major/minor faults (p. 33). Power up the chassis intended to be primary first, wait for PRIM, then power up the other chassis (p. 45). The display moves from DISQ → QFNG → SYNC in about 1–3 minutes; the target state is PRIM/PwQS on the primary and SYNC/QS on the secondary. Download to the primary only; the system crossloads to the secondary automatically (p. 46, p. 99).

In the workshop, an I/O tree with many remote chassis that are not yet installed increases communication load and lengthens qualification; inhibiting the absent I/O only helps partly (p. 96). Enabling time sync on the controller when no other module in the primary chassis has time sync enabled leaves the system qualifying for up to 10 minutes, after which qualification fails (p. 67).

4. Switchover tests

Rockwell recommends testing switchover before operation, once the system is fully qualified (p. 98). Switchover occurs on loss of power, a controller major fault, removal/insertion or failure of any module, loss of EtherNet/IP connections to the point of being lonely (the module sees no other device), or a command from the program or RMCT (p. 34). Run the following tests in turn:

1. Initiate Switchover in the Synchronization tab of RMCT (p. 98, p. 108).
2. Power off the primary chassis, the second test named in the manual (p. 98).
3. Remove a module from the primary chassis, for example the controller (p. 34).
4. Disconnect all network cables from the primary EtherNet/IP module: this only causes a switchover when the module becomes lonely (p. 34).
5. Disconnect one fibre between the two redundancy modules: with the 1756-RM2, the backup channel takes over from the active channel and the chassis pair stays synchronised; the 1756-RM3 runs both channels in parallel, so there is no channel switchover; synchronisation is lost only when both channels fail (p. 36–37). During a channel switchover, scan time may be delayed by about 10 ms (p. 35, p. 37).

Before each test, check that the keyswitches of both controllers are in the same position: with the primary in PROG and the secondary in RUN, the primary then reports major fault type 12, code 34 (p. 36). After each test, with Auto-synchronization = Always (recommended for 5580 and 5570), the system resynchronises automatically; watch the Secondary Readiness column in RMCT (p. 51, p. 98).

5. Numbers to measure

• Switchover can be as fast as 20 ms (p. 34); when best practices are followed it can take up to 50 ms, which is the figure to use when calculating the RPI timeout for I/O (p. 106).
• Applicable when the project has no continuous task (p. 82): maximum scan time of each task with the system synchronised and with HMIs and external systems connected; watchdog no less than 3 times the maximum scan or 50 ms, whichever is greater; maximum scan below 80% of the task period, controller CPU not above 80% (p. 82).
• Crossload: read LastDataTransferSize and MaxDataTransferSize with GSV (in DINTs) and apply the formula for the firmware revision and redundancy module type (p. 80–81). Tags that are written are crossloaded even if their value does not change (p. 79).
• Alarm burst: each 1–25 tag-based alarm conditions in one scope add about 0.4 ms to the scan; measure during commissioning (p. 97, p. 209).
• EtherNet/IP module CPU no higher than 80%; if exceeded, the secondary may not resynchronise after a switchover (p. 144).

6. EtherNet/IP network

• IP address swapping: the two partner modules have the same IP address, subnet mask and gateway; the secondary automatically takes the next address, and the two swap addresses on switchover (p. 64–65; 1756-RM010Q p. 40). Do not assign that next address to any other device (p. 65).
• Use static IP: switchover is significantly faster; dynamic IP cannot be used with swapping (p. 64). The two partner modules must have the same duplex configuration (p. 66).
• Remote controllers consuming tags: set the Connection of the redundant chassis module to None instead of Rack Optimized (p. 63). A redundant project must not contain unicast consumed tags (p. 17).

7. HMI and FactoryTalk

• Do not direct active communication to the secondary chassis (p. 26, p. 35).
• FactoryTalk Linx can lose communication for up to 30 seconds during a switchover and then recover by itself, affecting HMI, FactoryTalk Batch and FactoryTalk Alarms and Events (p. 35). Measure this interval (data server recovery time, p. 208) and record it in the test report.
• FactoryTalk View SE: FactoryTalk Linx from 5.0, with the HMI and both chassis on the same subnet (p. 26). A redundant shortcut path (Linx from 6.00) shortens recovery, but the module used for it must not use IP swapping (p. 35).
• Redundancy status is not reported to the HMI automatically; it must be programmed with GSV (p. 140).
• Release notes 36.052_kit1 (5570): an alarm acknowledged just before or during a switchover may reappear as unacknowledged and need to be acknowledged again (RN p. 8).

Summary checklist

Common mistakes

• Enabling Match Project to Controller: after a switchover you cannot go online or download to the new primary (p. 17; RN p. 6).
• Leaving event tasks, inhibited tasks, or using IOT (p. 17).
• Critical outputs not placed in the highest-priority task (p. 78, p. 209).
• Leaving Auto-synchronization at Always and then using Disqualify Secondary: the system immediately resynchronises (p. 108).
• Display cycling QFNG/DISQ: set synchronisation to Never until the configuration error is fixed (RN p. 6).