Researchers at Kraft Foods, Glenview, IL, have developed Remote Observer of Manufacturing Equipment (ROME), a concept inspired by the black-box flight recorder used to analyze airplane disasters.
ROME provides analysis of high-speed equipment stoppages, slowdowns, or inefficiencies. It combines data acquisition and video imaging to ensure peak performance and faster, predictable new line start-ups, according to Kraft. ROME is an outgrowth of Kraft’s Advanced Manufacturing Systems (AMS) approach under the direction of Dragan Filipovic, scientist at Kraft’s Global Technology & Quality Department. The objective of the AMS platform is to develop advanced diagnostic and servicing tools to maximize line performance and reduce equipment downtime.
Kraft licenses the technology through PercepTek, Inc., which will consult, deploy, and service ROME systems. The highlights to the basic ROME system include the following:
• Two color video cameras using 640x480 pixel resolution operate from 30-60 frames/sec; as many as four cameras can be used.
• It accepts up to 265 digital inputs (optically insulated, input voltage from 5 VDC/VAC to 250 VDC/VAC).
• It accepts 16 analog signals of 4-20mA (12-bit resolution).
• ROME has a real-time clock and is designed with no outputs. That’s intentional so that there is no possibility it will influence the operation of a machine’s control system, according to Dr. Filipovic.
• Sampling time for all digital and analog signals is 10 msec.
• The system can be set to capture events 10 seconds before the trigger occurs.
• Up to 80 independent 10-sec “events” can be automatically captured.
• ROME can monitor machinery equipped with any type or brand of PLC control system.
ROME’s “off-the-shelf” hardware components are from National Instruments. The hardware for a basic system as described above is available for under $20ꯠ, according to Alok Sarwal, program manager and senior robotics scientist of PercepTek, which has expertise in real-time programming of vision-equipped robotics.
When asked about PercepTek’s involvement, Dr. Sarwal says, “A common thread from robotics to ROME is in programming and analyzing real-time behavior and sensor monitoring, such as 10 to 15 updates per second.” Sarwal says that most of ROME’s elements are common to automated robotic systems that PercepTek has developed.
Like a black-box recorder, ROME stores data in a “cyclical” buffer that holds the last 10 seconds of video and data information from all 256 digital channels, 16 analog channels and two video cameras (First-In-First-Out memory concept). When an event occurs, the ROME system is triggered and the last 10 seconds of information stored in the ROME’s buffer is dumped to its internal hard drive where it is stored for later analysis.
Filipovic says that the root cause of a machine jam or an unexpected stoppage is often misinterpreted by line operators or even masked by another event. Sometimes one event tends to trigger others; in other words, a problem situation often leads to another in cascading fashion so the root-cause of the problem is very difficult to catch. Furthermore, problems could be sporadic and could happen over longer periods of time—even experienced line operators assigned to watch for a root-cause may return with no conclusive answer. For periodic problems even weeks apart, ROME remains watchful and ready. “It’s automatic and hands-free,” says Sarwal.
Because of the direct connections required, the data collection hardware is located in close proximity to the machine it is monitoring. A laptop computer can be connected to the ROME system at any point to upload data from ROME for a detailed analysis.
Software crucial
Although the components are standard, Sarwal says it took PercepTek six months to develop the programming that makes everything work together. “We have to instruct the system how to acquire signals, how to time-stamp them, and how to fuse the data and video together.”
Meticulous postprocessing of the events captured by ROME system is handled by a PC-based program called the Data-Viewer; the user-friendly Graphic User Interface (GUI) permits step-by-step analysis of video frames and PLC data in 10 msec increments on the same screen.
Dr. Filipovic says the collected information can be handled in three ways:
1. Off-line analysis aimed to pinpoint the root cause of an unexpected event. This is typically a jam or other error condition, he says.
2. Off-line analysis to understand why a machine is not operating at full capability under certain conditions or operates differently in different time periods such as shift changes.
3. Online to make “near real-time” comparisons between two machine signatures, a reference machine that runs at peak performance and a lower performer.
“The first step is to understand the problem,” Filipovic says.
That’s exactly how Kraft has been testing the ROME prototype on its own packaging machinery.
ROME on track at Kraft
A single ROME prototype system was developed and tested at three Kraft locations: its R&D center in Glenview, IL; a cartoner at another facility in the Midwest; and its present installation on the West Coast, where it was installed earlier this year.
The concept was tested in the lab using a train set two years ago, according to Filipovic. He was amused to see that setup drew curious interest from associates. “The train setup helped us to create a powerful, yet straight-forward analogy with a machine operation concept that gets down to the root causes of problems.” It works whether the event is a derailed toy train or a vertical form/fill/seal machine—Kraft researchers applied what they’d learn about the technology in analyzing simulated toy train wrecks to packaging machinery problems.
Kraft moved ROME from the lab to a plant where it was installed and closely watched during an extended period of time. The main goal of this stage was to verify that ROME hardware would survive unforgiving industrial environments.
Kraft has also used the ROME diagnostic system on another packaging machine to determine why the machine occasionally loses 25% efficiency in an unpredictable manner.
Kraft uses a lot of highly customized machinery, Filipovic says, which results in start-ups that are often a lot longer than with standard equipment.
Some of the lessons and restrictions Kraft has learned from its use of ROME:
• Affixing cameras to the machine frame is not a good idea due to vibration, so a floor tripod works better.
• Additional illumination provides better video quality
• Preparation of a machine for ROME deployment requires “double wiring” of many input/output channels.
• ROME must be located close to a machine due to signal cables, a maximum of about 15 ft.
Sarwal says that one of the benefits of the system is that data can be uploaded to a laptop computer and from there can be electronically transmitted via the Internet. That means the analyzing technician can be anywhere in the world.
“It’s a generic platform—ROME can be hooked up to any machine,” says Sarwal, noting that older machinery may require custom rewiring for output to the ROME system.
“This is new, it’s licensed, and it’s available,” says Filipovic. The one catch is that ROME is not available to Kraft competitors, which covers a lot of territory.
