Water Board Discusses What Lead To Boil Advisory

Oskaloosa Water Board Special Meeting – April 25, 2016

Missing Dialer Couldn’t Alert Staff To Situation

Oskaloosa, Iowa – Learning from and preventing another boil advisory was the thought behind a special meeting of the Oskaloosa Water Board this Monday.

Mike Vore, Chairman of the Oskaloosa Water Board, said that documenting the situation to make sure that water is always available to Oskaloosa Water customers was the purpose of the meeting.

Oskaloosa Water Department Manager Chad Coon took the board through the chain of events that lead to the boil advisory.

Coon was notified of pumps being locked out shortly after 7am by the plant operator. “He was asking what he needed to do to get them up and running again, because the towers were showing zero.”

The plant operator hours are from 7 am to 3:30 pm. The operator’s job is to run tests on the water. The operator also verifies pump readings.

The SCADA system had locked the pumps out when the reading of the chlorine level had fallen below acceptable levels. “I instructed him to take a chlorine test and confirm that there was a safe level of chlorine in the water, and once that was confirmed, then to go ahead and go back to the control panel and turn that pump on to PAN. “

The SCADA system runs the water plants at all times, explained Coon.

The plant operator was able to verify that there was adequate chlorine in the water supply, so the pumps were activated at approximately 7:08 am and water began flowing back into the Oskaloosa water supply.

When Coon arrived at the water treatment plant at 7:30 am, he and the plant operator began to look at the history “to see what had happened”.

Coon said that flushing fire hydrants had began on Monday, “and we was able to maintain levels in the towers, that was about mid-way on the north (tower). The south was more than half full.” That hydrant flushing program had left the towers lower than normal, and ultimately played a factor in seeing water levels in the towers extremely low the next morning.

The water plant operator left for the day at 3:30pm. At approximately 5pm Monday night the plant alarmed out “that we had no chlorine in the water,” said Coon. “In going back and looking, what it was is the reagent in the chlorine analyzer was empty and it was missed.”

“It (SCADA) was trying to dial out, but as we later confirmed, the hardware that actually runs that dialer was not there,” said Coon. The dialer is the equipment that physically calls out to the designated individuals when trouble happens.

Oskaloosa Mayor Dave Krutzfelt questioned Coon, “What do you mean by that?”

“I mean that the dialer was gone,” said Coon. “In talking with staff, we don’t know where it went. And so it’s a little bit troubling on this end from a security aspect to try and understand what happened with that device.”

According to Coon, that dialer is a USB based dialer that plugs into the SCADA computer system. The dialer made its last known call to staff 2 weeks prior to the discovery of it missing, and none of the staff knows where the dialer is now.

Coon said that when a plant is running as it should be, it’s not unusual not to receive an alarm call from the dialer in 2 weeks. Going forward, Coon sees daily checks for the system becoming the norm.

Coon went back to the timeline of events. The pumps were locked out at 5pm, because of the SCADA system doing its job in questioning the chlorine levels in the water.

The towers continue to drain from demand on the system.

“It goes through the night continuinng to drain. It hit low level alarm. Those also tried to call out, but it was unable to call out,” said Coon. “The north tower sits a little bit higher than the south tower and so that one will empty first. That one [north tower] showed being empty sometime in the middle of the night. The south tower continued to have water in it.”

“About 6:50am Tuesday morning is when that south tower finally had no water left in the bowl. There was still water left in the standpipe, which elevation on those pipes is what creates pressure.”

“We had an 18 minute lapse from the time when that south tower showed zero, until we had the pumps up and running.”

Coon said they evaluated what the water level may have been in the standpipe of the south tower, but regulations require that there be a minimum of 20 psi. “If we fall below 20 psi, we have to go through the act of issuing a boil water advisory.”

After the pumps were reactivated and the towers began to fill, crews looked to see if a potential leak had helped to drain the towers during the overnight.

Coon then reached out to the company that services the SCADA system to get the sytem serviced immediately. He then reached out to the Iowa Department of Natural Resources, which is the regulatory body for Oskaloosa Water Department.

Coon reached out to the DNR field agent a little after 9am, and was unsuccessful. “At 11 o’clock, I called again and was unable to get a hold of her. I called a different person at the DNR to discuss the situation. He gave me another contact to go ahead and call.”

Coon was then able to discuss the situation with a DNR field officer, and was working with them to help determine how much pressure may have been in the system. “And through that discussion, we were looking at the need to issue an advisory.”

The DNR then sent the template for the advisory, and Oskaloosa Water then worked through the language. Coon sent the initial copy back to the DNR at approximately 11:30 am, in which they required some changes.

At approximately 12:30 pm, the DNR approved the boil adviosry.

Staff was then directed to begin sampling the water supply to determine if any contamination had occurred. Tuesday is their normal testing day, but intstead of collecting a normal round for testing, staff collected a month’s worth of bacteriological samples.

Coon checked with staff doing samples to see what levels of chlorine they were finding. Coon said he, “was told everything they were seeing was close to typical for any other time they were sampling.”

For Coon and the staff, the chlorine samples initially indicated that, “if we had seen lower chlorine levels, that would be more of an indicator that we had suffered a little bit of contamination somewhere.”

“A little after 1 pm, I talked to Mike [Vore] about what was going on and the need to issue the advisory,” explained Coon. At nearly 5 minutes til 2 pm, the email with the advisory language, about the water situation and the boil water advisory, was sent to the rest of the water board, the city council, mayor and city manager.

At 2:30 pm the press release draft was sent to Mike Vore for review, and then was distributed via email to the local media outlets at 2:42 pm.

The staff then began the process of calling their emergency call list to alert them to the boil water advisory.

By 3pm, staff was enroute to Newton with the bacteriological samples for the first round of testing.

The results from that first round of testing came back nearly 24 hours later absent of any contamination. Coon said they did not release those results because they wanted users to continue following the boil advisory.

On Wednesday, staff once again collected rounds of samples for testing, and took them to Newton for analysis. When those sample results came in on Thursday absent of contamination, the boil adviosry was then lifted, after consulting with the DNR.

Water Board member Joe Ryan questioned Coon about the timeline of letting the community know about the potentially contaminated water.

“In hindsight, it should have been done quicker than it was, and I bear that. I think, looking at the process, and should anything of this nature happen again, I will try to get the notification through the DNR a little sooner, and not the lag in time. I know that was some different comments we’ve heard as well.” says Coon. “It seems like the notification should have gone out sooner.”

Mike Vore said that converstations he had with individuals indicated that the city did fall below the 20 psi requirement, because water customers indicated to him that they experienced extremely low pressure or no pressure “in their homes that morning when they got up”.

On that Tuesday morning, the Oskaloosa Fire Department did answer a fire call. The fire department was able to extringuish the flames with the water on the truck, but had the need arisen, they would have hooked up to fire hydrants.

Under the circumstances, the potential of sucking the system dry was a real possibilty in fighting such a fire.

Fire Chief Mark Neff said that if they were aware of the situation, they could have made the appropiate plans to help provide water, and in turn adequate fire protection for the citizens of Oskaloosa.

The Oskaloosa Water Board has instructed Coon to develop further plans to help remedy future situations, and to document the process the water staff needs to take in the event of a similar situation. They are expecting those plans and further discussion at their next board meeting in April.