AVEVA PI System troubleshooting: data gaps, pibufss buffering, the message log and pidiag

A gap in a PI trend usually means data stopped somewhere between the interface that reads the control system and the PI Data Archive that stores it. AVEVA's PI Server documentation gives a clear path for finding where: the interface and its buffering service, the buffer sessions to each Data Archive, the PI message log and the error codes it contains. This article follows that path from AVEVA's PI Server buffering and Data Archive administration pages and the UniInt interface framework 4.7.3. What a historian does and how PI compares with other historians: What is a process historian.

1. The interface and its buffering

UniInt-based interfaces can buffer values while the connection to Data Archive is down, and AVEVA states that without buffering, data collected during the outage is lost for good. Buffering is set up in PI Interface Configuration Utility (PI ICU), and PI Buffer Subsystem is the recommended service, particularly version 4.3 or later, which can buffer to several servers and collectives (Data Archive 3.4.375 or later for collectives).

When buffering is configured, the interface service cannot run unless the buffering service runs too. AVEVA names wrong service dependencies between the interface and the buffering service as a common troubleshooting point.

2. Check PI Buffer Subsystem

How to check depends on the version:

• PI Buffer Subsystem 4.3 or later: open Buffering Manager (from PI ICU, PI SDK Utility or PI System Explorer) and read the dashboard. In a command window in \PIPC\bin, pibufss -health lists each issue with the time it started.
• PI Buffer Subsystem 3.4.380 or earlier: run pibufss -cfg in \PIPC\bin to see critical errors, the type of data being buffered and the state of each buffer session.

AVEVA's health alerts table covers the common messages. Two examples: "Failed to create API shared memory buffer" usually means an interface still held the shared memory from an earlier session, or the service account lacks permissions; AVEVA's fix is to stop the interfaces, wait about a minute, check the dashboard, then restart them. A user-issued pibufss -bc stop queues data without sending it until pibufss -bc start is run, while pibufss -bc dismount stops queuing as well, so incoming data for that server is lost.

3. Read the buffer session state

PI Buffer Subsystem keeps one buffer session per Data Archive server. In normal operation a session is either Registered (connected, nothing to send) or SendingData. The other states point to the fault:

After an outage, pibufss -cfg shows whether everything has been sent: the line with total events sent should end with queued events: 0, and the session state should be SendingData. AVEVA notes that the time to empty the queue depends on the number of events, the network and the load on Data Archive.

4. Configuration files worth checking

AVEVA's buffering page lists three settings that often explain a session problem:

• In piclient.ini (in \pipc\dat), AUTHENTICATIONOPTIONS must match the security method, for example SSPI for Windows security.
• In the same file, BUFFEREDSERVERLIST must contain the correct server names; a ping to each name confirms they resolve.
• For more detail, a [DEBUG] section with LEVEL=5 in pibufss.ini turns on detailed logging to \pipc\log after the service restarts.

5. Read the PI message log

Data Archive writes messages with a severity letter: C for critical, E for error, W for warning, I for information and D for debug, followed by the time, the source component, the message ID and the text. The pigetmsg utility searches and filters them. AVEVA's examples include today's errors one screen at a time (pigetmsg -st t -se -i), the last ten messages from one process followed live (-t 10 -f -pr), and the last hour without debug messages (-si -st *-1h). A saved message log file is read with pigetmsg -if.

6. Translate the error code

Messages often carry an error number. The pidiag utility translates it: pidiag -e followed by the number. AVEVA's example is -11137, which means the primary archive is read-only and archiving is disabled. Negative numbers are Data Archive errors and positive numbers are operating system errors, which pidiag also translates; the AboutPI-SDK utility has an Error Lookup tool as well.

Checks we make first

These are TDL's working habits, not AVEVA requirements:

• The exact start and end of the gap in the trend, and whether all points of one interface or only some are affected.
• The interface and buffering service states and their dependencies on the interface node.
• The buffer session state and queued events before restarting anything, so that buffered data is not discarded.

How TDL approaches it

TDL Industrial Automation is listed in AVEVA's Partner Finder as a System Integrator in the AVEVA Partner Ecosystem. Our historian work so far has been on FactoryTalk Historian at the Vinh Hoan FeedOne aquafeed plant and on AVEVA System Platform with historian and reports at Samsung SEHC. On a PI System we start from the interface nodes and their buffering, because that is where data is kept or lost during an outage. More on historian services: SCADA, HMI and plant data integration.