Posts Tagged ‘human error’
Wednesday, April 24, 2013 @ 07:04 AM gHale
By Gregory Hale
It is time to focus on human reliability.
“We have done a fantastic job in improving productivity, let’s try to prevent human error,” said Eddie Habibi, chief executive and founder of PAS, during his keynote address Tuesday at the PAS Technical Conference in Houston. “Human error is to human reliability as pump failure is to asset reliability.”
In these days of faster, better, more, the reliance on technology is becoming greater. The catch is technology is solid and it keeps getting better, but what about the people running the technology? Are they getting better?
RELATED STORIES
Firms Don’t Budget to Protect IP
Manufacturing Most Attacked Industry
Simulated Attacks Hike Security Awareness
Malware Attacks Hit Constantly
There is a tremendous value automation systems bring as asset reliability helps bring in a solid return on investment, but now manufacturers need to also focus on human reliability and the human automation relationship. “We can’t take the human out,” Habibi said. “The operator is the most critical element in production.”
“Technology is available to improve human reliability,” said Larry Evans, founder and former chief executive at AspenTech and current advisor to PAS, during his keynote address Tuesday.
“These solutions allow engineers and operators to make good decisions and fewer bad ones,” Evans said.
We all know about what happens when there is a big disaster, but what about the small upsets that occur. They end up costing companies quite a bit and they don’t have to happen, Habibi said. In addition, the small upsets can lead to bigger problems. “The little things can add up to make huge things.”
To Habibi, the operator is the vital link in the automaton chain. That person needs to always have a good day because “when he has a bad day, everyone has a bad day.”
“The human is the weakest link. The role of the operator is to supervise the automation at the plant, making sure the automation system is doing the right things,” he said.
There are two types of human error, Habibi said: Unintentional and intentional.
Unintentional errors are the things we don’t mean to do. He gave the example of when he was in a plant one time and an operator changed a setting from 4.7 percent to 4.9 percent, but instead changed it to 49 percent. He said he never saw anyone run so fast as soon as he realized what he did. He ran out of the room to fix something and that averted a potential catastrophe.
Intentional errors, he said, come when the operator thinks he or she is smarter than the procedure. “They happen because we think we are doing the right thing,” he said. “The goal is to prevent human error, similar to physical asset reliability.”
To help alleviate human error, operators need to have situation awareness, which is having a collective understanding and cognizance of the environment around you. It is ongoing, constantly looking to do the right thing.
Habibi said there are three types of situation awareness:
• Physical environment
• Organizational culture
• Human automation relationship.
Physical factors include control room ergonomics, lighting, temperature comfort, along with traffic and noise issues.
Organizational culture deals with policies and procedures, shift schedule and reporting, work ethic and motivation, and training, knowledge and skills. “You have got to take care of people; give people the right tools,” he said.
The human automation relationship has different tiers. Tier one is direct operator interaction. Tier two is decision support systems. Tier three is automation asset management.
When it all comes down to it, as Evans said technology is going to continue to grow, but humans need to be able to keep up with it and make sure everything is running safely.
“Faster, better, more is great,” Habibi said. “But faster, better, more and safe is a hell of a lot better.”
Friday, November 2, 2012 @ 01:11 PM gHale
The shutdown of the No. 1 nuclear reactor at Korea’s Wolseong power station Monday was the result of human error, Korea Hydro and Nuclear Power (KHNP) officials said.
The plant operator made a mistake while manipulating a circuit breaker, causing a power shortage to some equipment which led to a malfunction in the cooling system, officials said.
RELATED STORIES
White Finding for WA Nuke
Bad Circuit Card Forced Nuke Down
Self-Powered Sensors Watch Fuel Rods
Cosmic Rays Could Aid Fukushima
“An internal probe revealed that a senior employee handed over his responsibilities to a junior with only two years of experience, who misunderstood his orders and operated the wrong switch,” a KHNP staffer said.
This is similar to the mistake that caused a malfunction at reactor No. 1 at the Gori nuclear power station that led to a blackout in February. At that time, a subcontracted worker with little experience failed to follow procedure and pushed the circuit breaker, causing the entire nuclear plant to lose power for 12 minutes.
“Human error is a more serious problem than faulty components and is the result of a lack of safety awareness,” said Prof. Hwang Il-soon at Seoul National University.
The latest incident appears to spell the death warrant for the Wolseong No. 1 reactor, which reaches the end of its 30-year life cycle Nov. 20.
A KHNP official said an inspection revealed serious damage to the inside of the reactor and a detailed investigation will take place after they take apart the reactor. The inspection will take more than a month.
Tuesday, August 28, 2012 @ 12:08 PM gHale
Application Whitelisting can Toughen Up Weakest Link
By Gregory Hale
IT folks were happy at one major U.S. manufacturer a few years ago as they were installing state of the art security technology. “This is the best move for the organization to keep free and clear from any miscreant bug or viruses launched into the network,” they were saying at the time. Just as they neared the end, the crew worked over the weekend to iron out all kinks so they could have it ready first thing Monday morning.
When Monday came, the long-time process control engineering team came in and promptly turned off all the new security measures because it was too different and not the way they always did things.
Human error.
It seems a hacker element left a bunch of malware-riddled USB sticks in parking lot at Dutch chemical giant DSM. Instead of plugging the discarded drives into a workstation, which would have infected the machine, a DSM worker who found one of the devices handed it in to the IT department.
The IT workers did a quick check and found an unspecified password-stealing keylogger.
System saved.
Technologies like antivirus, firewalls and whitelisting, are vital to helping secure any manufacturing automation system, but the human factor is the key ingredient to shepherd any process to ensure continued uptime that will hike productivity and profitability. The catch is though, everyone needs to be on the same security page.
“The gray beards are saying ‘unplug it, we don’t’ need it, who cares. I have been running this plant for 30 years,’ ” said Rick Kaun, global business manager Industrial IT Solutions at Honeywell Process Solutions. “That just isn’t realistic given the business needs for data.”
Arms Around Information Flow
Information, and information flow, is more valuable than ever to organizations. Despite its importance, companies don’t really understand how to effectively manage this valuable resource. An estimated 49 percent of the worth of organizations derives from the information they own, according to the “State of Information Survey” from Symantec Corp.
When asked what would happen if their organization’s information were irrevocably lost with no chance of recovery, survey respondents said they would lose customers (49 percent), damage the brand (47 percent), decrease revenue (41 percent), increase expenses (39 percent) and suffer a tumbling stock price (20 percent).
Protecting against stolen data, information, intellectual property, business market plans, and even money is becoming more complicated and sophisticated. That is why a solid defense in depth strategy for manufacturers, including application whitelisting is more important than ever.
Complexity, or perceived complexity, of technology is an automatic turn off for users. That has been the problem in the past with whitelisting, but this technology is much too valuable to dismiss with a mere perception. Whitelisting, unlike other security programs, can actually be an application where you put it on the system and forget about it. Just maintain it when you do a security assessment.
“Whitelisting can reduce the need to patch, but it will not eliminate the need to patch. It is protecting you from certain vulnerabilities until the opportunity comes to apply the patches.”
– Mike Baldi, Honeywell Process Solutions
The goal of application whitelisting for an industrial control system is to prevent unauthorized applications from running, enforce a list of approved applications, include an administration tool that allows for adjustment to the whitelist, and monitor and report attempts to violate the policy.
“I think whitelisting further enhances the value of a skill set that has the knowledge of process control and IT,” Kaun said.
The initial pushback against whitelisting always seems to fall along the lines of complexity and restrictiveness. But in reality, a manufacturer can make the program as restrictive as it wants and building it can be as easy as following directions.
“You have to build a list, said Shawn Gold, global solutions leader, industrial IT solutions at Honeywell Process Solutions. “There are tools that come with the whitelisting that has some installation scripts, but you have to build a list of things that are allowed.”
“Basic whitelisting provides protection by creating a list of known good executables that can run on your systems,” said Mike Baldi, chief cyber security architect for Honeywell Process Solutions. “All the application whitelisting systems available provide that functionality, but there are additional features. For example you can choose to protect areas of your registry if you want. You can choose to lock down your USB devices. You can enter rules for the whitelisting to protect against certain memory type attacks. These are above and beyond the basic white listing protection. Everything you configure in the system has a risk that you may lock down some normal operation that is needed to run the system. So you have basic whitelisting that can be restrictive to a certain point or you can continue to lock down the system extremely tight with whitelisting, but you have to be very careful to understand the consequences of locking it down.”
Constant Vigilance
With a slowly recovering economy, the need to keep producing more product these days at a lower cost point is at a premium. That means any unplanned downtime could be devastating to any manufacturer’s bottom line. That is why companies need to avoid dreaded downtime and work with multiple layers of defense and constant user education.
The problem is end users tend to be the most common and hard-to-remediate weak point, and even security researchers struggle to address the problem. “You can’t patch users,” said Greg Conti, associate professor of computer science at West Point in the Georgia Tech Information Security Center and the Georgia Tech Research Institute, “Georgia Tech Emerging Cyber Threats Report for 2012.” “And there’s always a human being somewhere behind the security technology.”
One source in that study agreed with Conti, “People are always the most vulnerable part of the IT infrastructure,” he said. “We have so many security layers and defenses, from separating physical control systems from the standard business network, to DMZs, to limiting network protocols that communicate with physical systems, and securing all the primary UIs to the Internet. At the end of the day, there’s a person on the end of all that security that can make decisions that will have an impact.”
Installing application whitelisting presents an upfront learning curve for users, but it is one that can be worth the time and effort.
“Our customers are learning really quickly,” Gold said. “I think the majority expect whitelisting to be more all encompassing and reduce the level of management significantly. It will help, but you have to really be careful about it. The maturity of our customers is increasing, but I do think there are a lot of misconceptions still.”
“The hype about whitelisting is high,” Baldi said. “There has been a lot of publicity. The understanding at the technical level at how involved it is and how tightly it has be interlaced with your system isn’t there. They hear words that this wonderful technology is available and it is going to increase your security protection, but there hasn’t been a lot of activity so far in applying whitelisting so there is not a lot of practical knowledge with that.”
Human Issue
Fear of complexity is one issue, but there is another Kaun feels has a strong human factor involved.
“It is apathy,” Kaun said. “I think the big vendors last year came up with 4,000 viruses or threats, but internally we came up with about 15,000 threats out there. Your least informed employees are your single biggest threat, so you can have all the technologies in the world, but if someone is holding the door open or handing out passwords then you have a problem.”
“I read one study that said 50 or 60 percent of people on the street said they would give over their favorite password for a free chocolate bar. It’s not whether we have application whitelisting or not; whether we have intrusion detection or not; whether we have a full robust program; whether we have point solutions, it is apathy.’’
Users need a solid technology base and a good plan that everyone knows, Kaun said.
“The source of the threat is not as important as when it gets here, and some day it will in some shape or form. How equipped are we to weather that storm, that is the real risk. If you see it as you want to spend how many dollars to make sure al-Qaeda doesn’t hack us, your problem there isn’t budget, it is education and awareness.”
Who takes responsibility and what should a user do often becomes an issue at a plant. Should it be IT, or should the process engineering team take control? At the end of the day, it often becomes an all hands on deck effort.
“Manufacturers are using every tool available to them,” Gold said. “Every combination exists; from the IT group being responsible, to the IT group embedding a person within the process control group, to the process control group being totally responsible and not having anything to do with the IT group.”
“The worst situation is where no one does anything, which is more common than one would expect. Then there is the thinking that we don’t have to do much because we are locking things down with an air gap. Even when air gaps are used in combination with locking down USB ports and not allowing vendors with their laptops to connect to their system, they are missing the critical point on how to mitigate or manage a virus when a path in is eventually compromised There are various levels of preparedness.”
Patch Threat
Patch management is one more way whitelisting can help users overcome some threat issues.
“Whitelisting can reduce the need to patch, but it will not eliminate the need to patch. It is protecting you from certain vulnerabilities until the opportunity comes to apply the patches,” Baldi said. “It is a tremendous potential benefit as long as the limits of that benefit are realized. There are combinations of technologies and benefits that we call defense in depth that together can significantly reduce the need to patch. What I mean by that is they can allow you to run with known vulnerabilities in your system longer until you can schedule maintenance time to do your patches. That would be your antivirus software, your whitelisting software and a third technology called virtual patching, which is basically intrusion protection from the network out. Those three technologies together can significantly reduce your need to patch and allow you to better manage your patch cycles.”
“A lot of people in the plant environment think along the lines of you set it and forget it,” Kaun said. “Part of the value of whitelisting is it works on that premise. The flip side is when you go to change something, how do we manage that so we don’t turn the whitelisting off? The problem is if anything changes it becomes completely useless. That is the challenge when we apply patches we have to make sure the scrutiny has to be much greater so we don’t break our application whitelisting. So, a very detailed technical analysis and a more thorough change management needs to take place.”
Application whitelisting all comes down to helping eliminate human error so manufacturers can keep their system up and running during a time when sophisticated attacks are on the rise.
“It is about safe reliable expected operation,” Kaun said. “Am I concerned? I am concerned because there is more risk and the clients we serve are increasingly under pressure and hitting downtime.”
“We need to not worry about the noise and just get down to work.”
Gregory Hale is the Editor and Founder of Industrial Safety and Security Source (ISSSource.com).
Thursday, July 5, 2012 @ 03:07 PM gHale
Digital information costs businesses worldwide $1.1 trillion annually, a new study said.
From confidential customer information, to intellectual property, to financial transactions, organizations possess massive amounts of information that not only enable them to be competitive and efficient – but also stay in business, said the first ever “State of Information Survey” from Symantec Corp. In fact, the survey revealed digital information makes up 49 percent of an organization’s total value.
RELATED STORIES
Security Discord between CEO, CISO
Smart Grid Needs More Security
RPI’s Disaster Management Simulator
Security a Weak Link for States
“The vast amount of information that organizations produce today can help them better serve their customers and increase productivity. However, the same information can also become a major liability if it is not properly protected. Companies that effectively use their information will have a major competitive advantage over those who cannot, and in some cases it can be the difference between success and failure,” said Francis deSouza, group president, Enterprise Products and Services, Symantec. “With its increasing value and rising cost, successful companies will find ways to more effectively protect their information and unleash the productivity it can bring.”
Businesses of all sizes are dealing with enormous amounts of data. The total size of information stored today by all businesses is 2.2 zettabytes. Small to medium sized businesses (SMBs) on average have 563 terabytes of data, compared with the average enterprise that has 100,000 terabytes. The survey also reveals information should grow 67 percent over the next year for enterprises and 178 percent for SMBs.
On average, enterprises spend $38 million annually on information, while SMBs spend $332,000. However, the yearly cost per employee for SMBs is a lot higher at $3,670, versus $3,297 for enterprise. For example, a typical 50-employee small business spends $183,500 on information management, whereas a typical large enterprise with 2,500 employees would spend $8.2 million.
The consequences of losing business information would be disastrous.
“We would have to fold our operations for at least a couple of years before we’d come back again,” noted an IT manager at a large engineering firm when asked about the consequences of losing the enterprise’s information. Respondents highlighted the impact of data loss to their business, including lost customers (49 percent), damage to reputation and brand (47 percent), decreased revenue (41 percent), increased expenses (39 percent) and a tumbling stock price (20 percent).
With so much at stake, protecting information should be a top priority, yet businesses are still struggling. In the last year, 69 percent of businesses experienced some form of information loss for reasons, such as human error, hardware failure, security breach, or lost and stolen devices. In addition, 69 percent have had confidential information exposed outside of the company, and 31 percent have experienced compliance failures related to information. Another challenge is the amount of duplicate information businesses are storing – an average of 42 percent of data duplicated. Storage utilization is also low, at only 31 percent within the firewall and 18 percent outside.
All these risks and inefficiencies result in businesses spending more than necessary on storing and protecting their information. A key issue identified by 30 percent of businesses is information sprawl – the overwhelming growth of unorganized information that is difficult to access and often duplicated elsewhere.
To help businesses more effectively protect their information, Symantec has the following recommendations:
• Focus on the information, not the device or data center: With BYOD and cloud, information is no longer within the four walls of a company. Protection must focus on the information, not the device or data center.
• Not all information is equal: Business must be able to separate useless data from valuable business information and protect it accordingly.
• Be efficient: Deduplication and archiving help companies protect more, but store less to keep pace with exponential data growth.
• Consistency is key: It is important to set consistent policies for information that can be enforced wherever it’s located … physical, virtual and cloud environments.
• Stay agile: Plan for your future information needs by implementing a flexible infrastructure to support continued growth.
Tuesday, September 6, 2011 @ 01:09 PM gHale
Unplanned shutdowns are the scourge of the industry. Lost uptime means lost revenues. However, when you throw a nuclear plant into the mix, regulators start to get involved. That is exactly what happened at the Pilgrim nuclear plant.
The Nuclear Regulatory Commission (NRC) issued a critical report following the unplanned shutdown of the Plymouth, MA, nuclear plant in May.
RELATED STORIES
Quake Brings Questions on Nuke Design
Nuke Fuel Casks Shift during Quake in VA
NRC Testing VA Nuke after Quake
Nuke Plant Safety Rules Inadequate
The NRC in a report criticized Pilgrim’s owners for inadequate training and enforcement of its standards following the automatic shutdown, which occurred as the reactor was returning to service after a refueling and maintenance outage. Human error was the cause of the shutdown.
Although there was no danger to the public, the incident was serious enough for the NRC to send an inspection team to the Plymouth plant, which is a rare occurrence.
The NRC said the incident will bring increased scrutiny to Pilgrim.
Plant owner Entergy Corp. said it is reviewing the report and added it has evaluated procedures and retrained staff.
Wednesday, February 2, 2011 @ 06:02 PM gHale
For an organization to ensure safety and security, the concepts have to be part of the everyday way of thinking and working.
That is why the federal Nuclear Regulatory Commission is considering adopting a policy on safety culture designed to minimize human error and managerial problems at reactors across the country.
NRC Chairman Gregory B. Jaczko said the federal agency’s draft, which they have been working on for three years, irons out key issues. It is not, however, an enforceable regulation, but rather a guide to “expectations” about how reactor employees should conduct themselves to enhance safety and security, he said.
Click here to view the draft.
The proposed policy defines safety culture as “the core values and behaviors resulting from a collective commitment by leaders and individuals to emphasize safety over competing goals to ensure protection of people and the environment.”
It also identifies nine “traits” that exemplify adherence to the policy, ranging from leadership safety values and personal accountability to effective communication and avoiding complacency buy maintaining a questioning attitude.
The commission could act on the policy within the next few weeks, said spokesman Scott Burnell.