A year has gone by since Abel Rochwarger, chief engineer at Gas Turbine Controls (GTC), shared with CCJ ONsite’s editorial team his report on an incident in which the Mark V control system on a GE F-class gas turbine inexplicably shut down all the unit’s lube-oil pumps causing extensive damage. Rochwarger said the customer’s team investigating the incident found out that the malfunction was “logic forcing without operator intervention”—hence the term “self-inflicted logic forcing.” CCJ’s editorial staff followed up with Abel for an update, just ahead of the 7F Users Group’s annual meeting, where GTC will be exhibiting Wednesday, May 9.
The OEM obviously took notice of the Mark V malfunction given that sixteen days after the CCJ ONsite’s publication, it released Product Service Information Bulletin (PSIB) 20170519A, “Mark V Communication Interface Overload—Loss of Lube Oil.” According to the PSIB, the OEM’s team simulated the site conditions in a laboratory environment and were able to confirm the self-inflicted logic forcing.
In Rochwarger’s opinion, the tests made a positive contribution to the collective knowledge—for example, by dispelling the suspicion of a possible cyber attack. But they also proved, he said, “there is a bug buried deep—and latent—in the core of the control system, which under some conditions, will manifest itself as it did in the incident described last year.” An analogy, in PC-user’s terms, the chief engineer continued, “If the Mark V configuration and sequence is the software, the bug lurks in the operating system.”
Rochwarger believes the bug is likely to remain, because the OEM probably cannot allocate any valuable engineering resources to eliminate a bug in a mature and discontinued product like the Mark V. Rochwarger recommends that users familiarize themselves with the PSIB, which provides a list of guidelines to prevent, in Rochwarger’s words, the “haywire scenario.”
So, what should a prudent operator realistically consider to “quarantine” the bug? First, suggests Rochwarger: Assess the risk. The PSIB provides excellent guidelines for doing this, he says. Second: Evaluate the situation with the incumbent parties, and define the appropriate protective measures for the site. It should be noted that, for many Mark V operators, a very reasonable conclusion for their situation will be that no action is required. After all, the Mark V has been running in hundreds of plants, with hundreds of thousands of successful operating hours since the early 1990s, and last year’s incident was the first self-inflicted-logic-forcing event ever registered. Third (if required): Implement PSIB recommendations that apply, and consider eventually adding some “foolproof,” hardwired protective measures.
The primary concern is, at a minimum to keep, the DC emergency pumps (Lube and Seal Oil) running even if the Mark V goes “haywire.” This can be accomplished by implementing certain hardwired logic modifications of the Motor Control Center (MCC). And, in the case of gas turbines, an additional modification can be implemented in order to ensure that the control sequence that cycles the DC emergency pumps in case of a complete AC failure stays intact.
Rochwarger told the editors that GTC’s service team can provide an assessment, and develop, implement, commission, and test these modifications to ensure the foregoing protection measures are satisfied. He estimated it would take about four days onsite during a shutdown to implement the hardwired mods.
Finally, the editors asked GTC’s chief engineer if the Mark V bug might have carried over to subsequent versions of the system; Abel wouldn’t hazard a guess, and commented, “This event took us all, literally all, by surprise. And, although this is a question for GE, since it happened over a year ago, knowing first-hand GE’s prudent and conservative approach, combined with their commitment to the highest quality standards, we would expect that they carried out their due diligence to ensure the integrity of their newer products. So, without any further communications, the logical conclusion is: no news is good news.”
