Cloud-Hosted SCADA with MQTT Gives Water Agency Faster Access to Data

Waterford Township Department of Public Works (DPW) in Oakland County, Michigan, manages 715 miles of water and wastewater sewer, including 19 production wells, three storage tanks, 11 treatment plants, and 63 sewer lift stations. In 2015, Perceptive Controls undertook a project to upgrade the department’s core SCADA infrastructure, which monitored RTUs over licensed RF using a poll/response protocol. Now, the department’s system runs on a cloud-hosted Ignition MQTT infrastructure, which has improved the speed, scalability, accuracy, and fault-tolerance of the system with lots of potential for future integration.

PROBLEM:

Originally, Waterford DPW’s SCADA upgrade was driven by the simple need to replace aging components and avoid obsolescence. But it quickly grew into a vision for fast, scalable, precise monitoring of critical infrastructure.

In 2015, DPW had two GE iFIX workstations that used a custom poll/response protocol to monitor dozens of RTUs in a round-robin fashion. This system had aged well, providing internal redundancy and deep integration with other department systems like GIS, AMS, and ECM. However, there was a growing risk of irreplaceable hardware failures.

DPW began upgrading field devices and SCADA system hardware, but in 2018, leadership at DPW learned about MQTT and decided to address other big problems the department had been dealing with. For instance, the serial polling scheme, combined with the limited bandwidth of the radio network, meant that data from each site could only be updated every 3-4 minutes.

Sometimes a lift station would run in between polling cycles, creating gaps in reporting, and inhibiting operators’ ability to accurately detect issues in the field. And with each I/O point added to the system, latency grew worse.

DPW wanted to re-architect its network around MQTT’s report-by-exception behavior. However, that introduced a new problem because the aging SCADA software had no way to work with MQTT.

SOLUTION:

Waterford considered other industrial MQTT solutions, like VTScada. However, it lacked support for Sparkplug, which Waterford knew it wanted. In the end, Perceptive persuaded the DPW to consider Ignition because of how it allowed the DPW to manage all its MQTT infrastructure in one place and integrate seamlessly with field devices.

Now, the DPW runs MQTT Distributor and Engine on an Ignition server that is hosted on Amazon Web Services (AWS), and it runs MQTT Transmission in each of the groov EPICs installed in the field as replacements for the older RTUs.

The DPW’s radio network has been upgraded to 4G LTE cellular modems, which are firewalled to only permit specific IP addresses to connect. The solution also uses Ignition’s support for TLS encryption and certificates of trust to establish secure connections with each EPIC.

The iFIX HMIs in the office have been replaced by Ignition Vision clients, which are being migrated to Ignition Perspective, which the DPW technicians have been using in the field.

We also make extensive use of the Alarming and Tag Historian modules in Ignition.

AWS sends weekly snapshots of Waterford’s Ignition server and database server to a separate data center so the system can be quickly resurrected in the unlikely event that there is a lost connection with the local data center. This is exciting, not just because of how fault-tolerant the Perceptive Controls system is, but also because the design goes far beyond established best practices. To Perceptive’s knowledge, Waterford is one of the first municipal departments to set up a cloud-based control system.

RESULT:

This design is unconventional for the industry, but it avoids the kinds of problems with local utilities that have made the news lately and is more resilient than it used to be. In fact, Waterford DPW recently experienced a loss of internet connectivity in their office building, and of course, the system was totally unaffected.

With MQTT, the system is also much more responsive and stable. The DPW system has gone from multi-minute cycles to sub-second latency and a significant reduction in actual data transmission because of report-by-exception. This allows the DPW to send out more data than before and never miss a system action or an alarm notification, which enables operators to accurately diagnose issues in the field.

The new HMI screens complement the level of data integrity with a friendlier user interface. Critical factors like wet well level and pump runtime in each lift station are highlighted so an operator can spot problematic behaviors early on.

The DPW also has lots of communication diagnostic information, which simply wasn’t possible before. Scalability is no longer a concern, and the DPW is looking forward to future projects built on this foundation. In the future, the DPW sees huge potential for integrating with local and regional systems to build predictive systems that help avert disaster and operate more efficiently.

Support costs went down by around $10,000 from what they were when paying for iFIX, and the DPW saved a lot of money in travel time because Perceptive can do all the programming through a secure remote connection. Tag counts and related metrics will increase by the time the project is completed in late 2021.