Power Users Annual Conference dives deep into emerging industry trends, concerns

Quality of work, changing missions challenge aging plants

Related Links

• What you shouldn’t miss at the upcoming Power Users Annual Conference featuring five user group tracks
• Power Users Annual Conference dives deep into emerging industry trends, concerns
• Combined Cycle Users Group: 2023 vendor presentation recaps
• Steam Turbine Users Group: 2023 vendor presentation recaps
• Generator Users Group: 2023 vendor presentation recaps
• Power Plant Controls Users Group: 2023 vendor presentation recaps

CCJ’s highlight reel from last summer’s meeting, which follows, offers the opportunity to evaluate the depth of content you can expect at this year’s event (August 26-29 in Phoenix) and to reflect on the positive impacts these user groups have had on plant operations and maintenance industry-wide. The knowledge, experience, and expertise shared on how to keep the nation’s aging gas-turbine-powered generating facilities operating reliably and efficiently in the face of shoestring budgets, supply-chain disruptions, and labor shortages, is perhaps unprecedented.

Two broad themes stood out at the 2023 Combined Conference. The first: How aging plants adapt to changing, sometimes radically different, operating missions. The second: Continuing issues with supply chains, both labor and materiel, especially in terms of quality of work.

For more than a decade, combined-cycle plants have been following wind and solar with greater load cycling and more starts and stops. Today, many of those plants are cycling more frequently for even shorter periods with as much capacity as they can muster. One reason: The well-worn “duck curve” has become a “canyon,” referring to the changes in dispatch curves where more gas capacity is “in the money” during daytime hours (the canyon floor) compared to five years ago (Fig 1).

But if three of the facilities highlighted in the user-driven presentations last year are representative, some plants are having to adapt to radically changing missions, like being scheduled for shutdown in a few years, but then suddenly being called upon to run for years more—if not one or two decades. This presents obvious challenges in many areas, including safety, obsolescence, staff knowledge, and others.

Even the newest, most efficient H-class machines installed over the last five years or so are not running baseload and are “cycling all over the place,” said an OEM rep, who also noted that earlier model gas turbines expected to be retired in 2045 are now having to plan on operating another 20 years.

Supply chain issues take many forms, but quality of work continues to dominate discussions in many industry forums. There was a time, when competitive power supply emerged as a counter to regulated-rate-of-return utilities, industry soothsayers anticipated that OEMs and/or contract service providers would be responsible for the major plant subsystems—like the gas turbine, steam turbine, HRSG, condensers, etc—while onsite staff was reduced. While this indeed had been a major trend over the last two decades, it may have run its course.

Several presentations made last year suggest that owner/operators are re-asserting responsibilities for onsite work, or at least acknowledging that OEMs and contractors require much greater oversight. Other speakers noted that equipment commissioned recently is failing prematurely, or exhibiting unexpected downtime and repair needs.

A seasoned expert from a major owner/operator reported on implementing a broad fleet-centric borescope inspection strategy for its turbines to respond to “narrow grid margins and cost pressure on lower-priority units.”

A steam-turbine expert for a large fleet concluded that two D11s of similar vintage can exhibit vastly different first- and second-major outage characteristics. Somewhat surprisingly, the one with 1000 starts and 20+ years of operation required few repairs, while the one with 70+ starts and eight years of operation required significant repairs during its first major. The two units were procured around the same time but one remained in long-term storage for seven years.

In one session on NERC compliance, there appeared to be audience consensus around the statement, “Don’t trust the equipment OEM or the control system OEM to do any of this [verify models, capabilities, data, etc] for you.” An expert in a session on severe-service valves noted “make sure your vendor has QA/QC processes you can refer to after the fact.” A presenter on a BOP subsystem replacement noted “major communication issues,” referring to a non-responsive vendor.

Some of the comments in a “generator quality roundtable” echo these sentiments:

• Experience has dropped over the last decade across the industry.
• One-third of events in 2022 are accounted for by quality issues.
• Cannot rely on contractors: Owners must force accountability.
• Be wary of vendors “preferred by management,” which may not be best for the job at hand.

Finally, the CCUG conference presented its traditional share of deep discussions in broad areas affecting all plants all the time, like safety and lock-out tag-out (LOTO), hot and cold weather ops and associated regulatory compliance, as well as unique equipment events and failures which may seem like “one-offs” but offer warnings, guidance, and lessons learned which can often be broadly applied in similar or other situations.

New lease on life

Consider this situation: Your cogen plant is facing the end of its steam contract, and you experience a brushless exciter loss-of-field trip. During the repair period, the steam contract is extended several years. Soon after, the contract is extended another five years. Now you need a different strategy, one that ensures exciter reliability at least through the end of the contract. On top of that, the outage dates have been moved up by two years.

The source of the failure, and subsequent damage described as “carnage,” was excessive moisture in the exciter enclosure. Source of the water was steam leaks from the main stop valve and extraction flange channeling from the steam-turbine enclosure along the generator to the exciter. The full story, and some of the breathtaking details, is captured in the slides but the lessons learned are generically applicable:

• Steam can flow through flashing and skirting and get into very unwanted places.
• Keep a stock of brushless exciter parts.
• Double check planned power ratings versus existing equipment.
• Variable-frequency drives may not be suitable to replace the permanent magnet generator.
• Consider a subject matter expert (SME) to oversee such a project.
• Take advantage of redundancy if necessary (this unit has “double double redundancy,” parallel and series circuits).

Where do you start?

A presentation on a DCS replacement, prompted by lack of spare parts for the original, began with the subhead question. Commercial operating date for the plant was 1997, and it still runs baseload. A new Mark VIewas incompatible with the original RS3 technology, given its old networking protocol, lack of OEM support and cybersecurity protection, and reliability/spare-parts issues. Plant selected Ovation to be installed at the same time as an ST/G major, a 30-day outage. Note that the equipment has to be operating and LOTOs removed to perform the DCS functional checks.

During process startup, 124 variances were identified, and all but 11 were completed with the plant on-line. The remainder were completed during an outage three months later. Generic lessons learned include:

• Onsite training needs to be site-specific.
• Pay attention to establishing links with existing software, including PI links.
• Data links between Mark VIe and Ovation presented problems for the ST/G.
• Make sure there is access to the portal for transferring data to/from the plant.
• Budget contingency funds for additional IC&E technicians.

Although not listed in the lessons learned, perhaps the most important one is that the plant withheld 10% of the contract funds until all variances were addressed.

From “hardly on” to baseload

Imagine how many challenges a more than 40-year-old simple-cycle plant might face going from peaking, which is “hardly ever run,” to baseload ops. This presentation focused on the H₂ purge system for the 501D5 generators, proving that even the replacement of a relatively minor subsystem can lead to major headaches (Fig 2).

When your H₂ purge dryer breaks as a peaker, maybe you can get away with leaving it and locking it out and continuing to suffer from the control cabinet hung onto the unit which had presented many problems over the years (for example, vibration). Also, the skid was located outside. Solution for baseload service was to replace it with a new purge control system, dryer included, and change all the lines to stainless steel.

Project challenges included failure modes resulting from an OEM design disagreement between a fail-open or fail-safe valve; integrating the new equipment with an old, not optimized, CO₂ system; lots of wires to be pulled and P&IDs to update; islands of automation and “heartburn” over PLCs and HMI; infant mortality; cabinet purge; regulator issues; and components not designed for outdoor service and requiring heat tracing. Many others are discussed in the slides. Thankfully, the plant arranged payments based on engineering deliverables, rather than lump sum.

Generic lessons learned include:

• Involve the controls staff early in the process and make sure to check design against engineering standards.
• Identify equipment supply from the vendor before finalizing the contract.
• Demand (or ensure) that operating manuals are adapted to your site.
• Decide whether failure modes for valves are specced for safety or reliability.
• Make sure your skid is designed for the right service—for example, simple cycle, combined cycle, etc.

Another 25 years? Aye aye, sir! Limping along with a 1967-vintage excitation system (controls were upgraded in 2012) is fine if you’re planning to be shut down in a few years. Things change when management says it’ll be running for another 25 years. Check out the slides if you want to know why the outage for the excitation system upgrade (it’s an old coal unit) had to be extended from 27 days to 49 (then extended again because of issues with the ST valves). Three of the lessons learned are: Collect all drawings and files well ahead of project start, get the structural study done early, and use a variation of the old adage, “measure twice, cut once” when it comes to clearances.

Newer units, unexpected events

The slides for the presentation referenced earlier, comparing two similar D11s with very different operating histories, have an incredible amount of detail on design, predicted versus actual diaphragm dishing, casing disassembly, opening clearances and alignment, packing, N1 and N3 offsets performed, and inlet-bowl cracks and weld repairs. Diaphragm dishing is described as “a base scope item on these units” and is the focus of the material.

Regarding new diaphragms ordered for Stages 12-14 for this heavily cycled unit, they were specified with 0.125 in. extra stock and the OEM was required (1) “to provide a matrix of the material/weld/profile changes for each possible diaphragm ordered,” and (2) that the component adhere to TIL385A1598.

Generator maladies. After less than four years operation, a generator at a 2 × 1 1100-MW site tripped on phase-phase and phase-ground targets. Initial inspection revealed that the “inner nozzle shield directing cooling gas toward the stator end caps had liberated, fallen across the endwindings, and eventually cut through the insulation, creating the fault.”

The failure led the owner/operator to shut down and inspect the other GT and ST generators at this site and at a sister site (six total) of similar design and start date. During “extent of condition” inspections, cracks and epoxy voids with dowel pins were discovered. The OEM issued TIL-2323 after inner-nozzle shields were removed on all the units. Further inspection results and shop repairs are described and illustrated in the slides. One end result was that the OEM issued TIL-2337 for potential field coil migration on H65 and H84 units.

The owner/operator decided to send another generator from the sister site to the OEM shop to address blocking replacement and sliding-surface length changes. There was significant evidence of coil movement, a number of blocks on the underside of the winding fell out, the winding had shifted from the turbine end toward the exciter end (unlike the first unit), and significant distortion was present on the exciter end of the winding. Full-scope repairs were not possible within the outage window.

Temporary repairs were made, and the plant is monitoring vibration and flux-probe readings in anticipation of a partial rewind in 2025.

Cracked and broken through-bolts (TB) on the MBH model generator are believed to be a fleetwide issue (fleet here referring to the owner/operator’s units). Presenter reported on units placed in service between 2009 and 2011. One broken TB was discovered during a minor inspection in 2018. Crack-like indications discovered through NDE were found on bolts in two other units. No conclusion was reached on root cause (OEM elected not to perform a root-cause analysis). Short-term solution was to encapsulate both ends with epoxy resin, but the user is seeking a solution that can get them to a rewind outage several years from now.

An arcing incident in a water-cooled generator (different OEM and model) outlet manifold pipe is also reviewed in these slides.

Solid particle erosion (SPE) is one of the issues plaguing the steam valves for D11 units (and its variants from other OEMs), according to several presentations which took much of the STUG conference’s first day.

One user said, “We are struggling to control SPE.” One piece of evidence is that a main steam valve incurred 30% loss of material after less than 12,000 operating hours. A “Next Gen” valve installed in spring 2020 showed dramatic improvement after around 20,000 hours. However, pins had migrated out of the stem and stem nuts, causing damage downstream. A user from the audience noted that their valves have had so much erosion, they question the material choice.

Another user presenter reported that they cut maintenance intervals on D11 main steam valves from three to four years to one to two years because of SPE and noted that this was “true for most users.” Replacing with a “Next Gen” valve also posed a new set of issues with large coupling actuator bolts, pressure seal head jacking-plate bolts, broken balance-chamber bolting, and out-of-spec valve-disc seal rings. A non-OEM “solution” was able to reduce the erosion rate by around 35%.

Needless to say, if you are experiencing SPE or other problems, or suspect such, this group of slide decks is a must-view.

Generator borescope inspections “can provide more and higher-quality information than a traditional minor,” said an expert from one of the largest GT/CC-fleet owner/operators in a GUG session, and require a minimal level of disassembly, and help identify issues especially after protective-relay operating events. His company has expanded a formalized borescope inspection program initially applied to gas turbines as a way to reduce costs for low-priority units. The inspections are generally performed every 25,000 EOH or “for cause”; planning documents are delivered to the plants so equipment can be properly prepped.

The list of generator components subject to the inspections, descriptions of guidance documents, examples of findings, and program statistics are all available in the slides. Since 2021, five major issues were identified through 59 inspections, 45 planned and 14 emergent. One finding was loose stator wedging on three units, which prevented a failure and potential stator rewind, valued at $6-million per unit.

Limitations to the program include the following: Some generators cannot be inspected, borescopes can’t reach all target areas, and inspection capabilities and experience have to be nurtured internally.

Old standbys: LOTO and NERC

For the best of reasons, safety is always a popular topic at the CCUG, and this year was no exception. The LOTO Roundtable Discussion on best practices included a detailed discussion of what should be subjected to LOTO (or enhanced safety procedures) during outages. An example included how a seasoned maintenance worker caught a gloved finger in a fan that typically only runs during high humidity. Digital solutions for LOTO and safety were discussed; some plants have completely automated these processes using hand-helds and commercially available software, others have hardly automated at all, based on the ensuing discussion.

LOTO “traps” listed include electronic inlet-guide-vane (IGV) actuators, electronic fuel-gas valve actuators, and peripheral equipment like ventilation vans, blowers/heaters, 480-V element space heaters, injectors, and fire protection equipment. Users noted that they “struggle with electrical” and that “electrical is not as easy [in terms of LOTO] as mechanical equipment.”

Some of the best practices include:

• Fostering a learning culture where blame is absent and communication is high.
• Daily face-to-face meetings before work begins.
• Mandatory plant walkdowns before starting up or outages begin.
• Having a trailer dedicated to LOTO outside of the control room.
• Having systems experts at the plant during major work and updating P&IDs.
• Having one person dedicated to the LOTO program at the plant.
• Making sure LOTO tags are secure against wind and weather.

NERC and ISO regulations have plants laser-focused on hot and cold weather prep, a topic of another CCUG roundtable. One plant manager illustrated how much has changed in this area by stating that his plant (central Midwest facility) was originally designed as a plus-8F facility (above zero) and now it is a minus-23F facility. Some of the best practices for hot-weather work which arose from this session include these:

• Use the 40/20 rule—40 minutes work, 20 minutes rest—and force contractors to take breaks.
• Provide cooling vests (and make sure employees wear them), hydration stations, tents, and even salty beverages (for example, pickle juice) or just salt.
• Make sure air conditioners (AC) are covered under annual PM and inventory replacement parts. Also, cross train a tech to work on ACs so you aren’t waiting for a repair when everyone’s ACs in the area are hiccuping.
• Wire cabinet temperatures into the control room.

For cold weather:

• Verify that your HRSGs are, in fact, drained and drain valves are not plugged (such as with magnetite).
• Verify electrical draw on heaters.
• Cycle valves to make sure they start in cold weather.
• Be prepared to drain an HRSG quickly, such as when you only have fuel gas for one unit and the other won’t be running.
• Give staff guidance on making sure the cooling tower doesn’t turn into an “ice palace.”

Other user presentations with valuable content covered a range of important topics, that didn’t “fit” conveniently into the foregoing narrative, which was based mostly on content from the CCUG meeting. That additional material is presented below by user group.

Steam Turbine Users Group

“IP to LP Turbine Crossover Failure” for a 1974-vintage coal/gas-fired unit makes the point that units with a welded tie-rod configuration on the crossovers (as opposed to a solid bar) are susceptible to the same failure mechanism described here, where the tee balance end plate liberated from the piping (60 in.). Importantly, the crossover bellows was replaced in 2018. One key point in the post-mortem evaluation is that the vendor for the bellows replacement could not verify that Loctite was applied to the fasteners during reassembly. Outage incurred was more than 60 days.

“ST Casing and Valve Body Indications Strategy” addresses the questions of how long and at what capacities can a unit run with multiple indications in the main-steam valve chest, HP and IP inlet leads, and HP-to-IP packing head fits. A 48-in.-long crack in a steam chest wall is particularly notable (Fig 3).

“Combined Cycle A10 HP Turbine Shell Issues” traces inspection, analysis, repair, and monitoring of a crack discovered in a HP bowl crack (caused by low-cycle thermal fatigue) during the first major outage for a 2003-vintage unit, along with operating strategies (for example, potentially altering the cold-start ramp rate) to avoid a casing replacement.

“Know Your Fleet” reviews a large owner/operator’s steam-turbine units from the perspective of informing a fleetwide O&M strategy.

“Major Outage Findings on BB-43 Uprated Turbine” takes you through major outage activities for a 324-MW combined-cycle steam turbine—including replacement of all HP/IP rotating blade rows and sealing components, high-speed balancing, replacement of LP rotor seals, bore inspection, and low-speed balancing. Cracked L-2 wheel steeple and blade dovetail (leading blade in a group of six) was discovered and addressed with titanium blade replacements.

“LP Bucket/Blade Removal—The Good, The Bad, and the Ugly” is a pictorial review of findings (the majority ugly) on various L-0 blades (mostly titanium) in the owner/operator’s fleet discovered between 2011 and 2023.

“HP16 and KA24 Issues and OOEM Partnerships” reviews inspections focused on TILs 2052 (cracking in IP/LP rotor) and 2010 (radax cracking) for the KA24. Plus, barrel extraction and refurbishment, upgrades, and process improvements for the HP16.

“D11 Lift Oil Leak” investigates an oil leak discovered at the left-side housing connecting joint (threaded) at bearing 5 and posits risk scenarios for the technique implemented to arrest the leak.

“Replacing HP First Stage on D11 Due to FOD” covers damage and repair to diaphragms and blades from parts missing from the main stop and control valve seats, caused by lack of bonding between the Stellite® and base material, as per TIL1629-R1. As a result, all control-valve seats are now UT-inspected during minor and major outages.

“LP Gland Replacement on a Legacy Fossil Unit” covers the 1965-vintage coal unit configuration and history, air-inleakage and previous inspections, and shop surveillance over repairs. Among the last with respect to lessons learned: Owner/operator will insist on ultrasonic testing of the rough machined castings before they ship from the foundry, along with renewed emphasis on vendor surveillance—including technical and schedule considerations.

Generator Users Group

“Generator SFRA Testing” answers these questions: “What is sweep frequency response analysis” and “What is it good for?” Details of this owner/operator’s test standard, numerous examples of insight offered through the test, and tips for successful SFRA testing are included.

“Modern Tool for Managing a Large Fleet’s Electrical Issues” (listed on the Power Users website as “Generator Inspection/Data App”) covers the use of a mobile-device software called Power DB to organize data sheets for hundreds of generators, transformers, and exciters for maintenance, outage scoping, and most importantly, NERC compliance. Screens essentially replicate the look and feel of paper technical data sheets and checklists, but data across units is searchable, queryable, and navigable.

“Aeropac I—Main Lead Failures” reviews a three-phase-fault trip event occurring during heavy rainfall, which according to initial engineering reports “didn’t look good.” Water ingress and flashover were the culprits. Photos of before and after, damage evidence, disassembly, and repair are included. Second similar event is also described.

“Rotor Slot Filler Migration” (listed on the Power Users website as “Blocked Rotor Cooling Holes”) covers a generator failure from a stator ground fault in 2009, a subsequent stator ground-fault failure with the replacement used generator in 2011, modifications, and findings during a 2019 outage. Core hot spots, rotor core filler migration, and blocked cooling slots are some of the maladies. Slides conclude with OEM recommendations and owner/operator actions and follow-up.

“Turbine Generator Shaft Grounding Case History” begins with the sources, detrimental effects, and ways intended to control shaft voltage and resulting damage, then goes into the case history of ineffective shaft grounding on an old generator for which the solution was a simple braid assembly retrofitted around the machine.

“Collector Brush Flashover Case History” pertains to a trip of a 148-MVA unit (critical to the grid) attached to a steam turbine after which inspection of the collector house indicated brush rigging failure (Fig 4) was caused by inadequate maintenance. Inspection, repairs, recovery, and subsequent shop repair of the rotor main leads during a scheduled outage are covered.

Power Plant Controls Users Group

“Battery Storage Energy System (BESS) Black Start” describes in remarkable detail a BESS added to a site with a 2 × 1 combined cycle and one simple-cycle unit but one section notes that the BESS harmonic filter design, which was based on the Load Commutated Inverter (LCI) load for the simple-cycle GT (BESS is used to start the GT), had to be doubled in size based on actual startup data. If a BESS for similar duty is under consideration for your site, this is a must-read (Fig 5).

“Controls Discussion” reviews a variety of issues with GT controls, some associated with TILs and other OEM documents, but several include this large fleet owner/operator’s own technical recommendations. Topics include exhaust-spread monitoring, turbine compartment temperatures, non-optical flame detectors, TTUR card failure, and CDM (combustion dynamics monitor) probe failure mitigation.