MAINTLET

MAINTLET: Advanced Sensory Network Cyber-Infrastructure for Smart Maintenance in Campus Scientific Laboratories

PIs:

Klara Nahrstedt – Coordinated Science Laboratory (CSL )

John Dallesasse – Holonyak Micro-and-Nanotechnology Laboratory (HMNTL)

Mauro Sardela – Materials Research Laboratory (MRL)

Gianni Pezzarossi – Engineering IT Shared Services (Engrit)

Students:

Beitong Tian
Cody Wang
Robert Bruce Kaufman
Janam Bipinbhai Bagdai
Leah Espenhahn
Logan Keating
Ahmadreza Eslaminia
Ragini Gupta
Zhe Yang (Alumni)
Patrick Su (Alumni)
Hessam Moeini (Alumni)
Mark David Kraman (Alumni)

Project Description:

Studies show that in some industry and scientific environments, between 15 and 60 percent of the total costs originate in maintenance activities, and about 33 cents of every dollar spent on maintenance in the US is wasted because of unnecessary and preventable maintenance activities. The cost of scientific instruments’ maintenance is even greater in universities because the scientific instruments, and associated support equipment, such as vacuum pumps, serve diverse students, staff, and faculty populations for educational and research purposes over much longer periods with smaller budgets than in industry. Instruments’ down-time greatly limits research productivity and programs. Hence, MAINTLET investigates an advanced sensory network cyber-infrastructure with modern AI-guided big data methods that helps the campus scientific laboratories to see patterns that indicate the right time to purchase kits, parts, and services, and minimize opportunity cost due to down-time and all repairs and maintenance.

MAINTLET enables cost-effective, scalable, and sustainable reactive, preventive and predictive maintenance solutions for scientific instruments. MAINTLET provides two important indicators. For preventive and predictive maintenance, simulations identify potential instrument failures, using data from instruments’ surrounding sensors such as acoustic sensors, water flow sensors, and contact water temperature sensors. These data help predict in real-time, using AI techniques, when a pump may need condition-based preventive maintenance. For reactive maintenance, trained failure detectors detect failures in real-time. MAINTLET includes sensors; edge devices such as Raspberry Pis executing reactive maintenance services; WiFi and Zigbee access points and networks interconnecting sensors, edge and cloud devices; and a private cloud with predictive and preventive maintenance services.

The impact of MAINTLET is in terms of decreased instrument failures and down-time and hence speed-up and accuracy of scientific discoveries, and in terms of security (as uncertainty about failed scientific lab equipment can cause both cyber and physical harm). MAINTLET’s various insights are taught in undergraduate and graduate courses to students from Materials Science & Engineering, Computer Science, and other departments. MAINTLET is presented at the Advanced Materials Characterization Workshop with instrument vendors’ exhibit, “Nano at Illinois” event, and other scientific venues. During the summers, the Worldwide Youth in Science and Engineering program for high school students, and other outreach programs, organized within the Grainger College of Engineering, receive a series of MAINTLET lectures.

As part of the MAINTLET’s objective, fault diagnosis and health monitoring of scientific cleanroom equipment such as vacuum pumps are performed. These vacuum pumps play an instrumental role in semiconductor manufacturing industries and are highly prone to failures due to overloading, overheating, or mechanical wear/tear over time. In order to ensure reliability and reduced maintenance costs, it is important to detect faults early and accurately. Subsequently, a network of sensors including surface temperature, current, vibration, and microphone (sound sensor) is implemented for continuous status monitoring of the vacuum pumps. By studying frequency change of vibration signals and ML-based acoustic data analysis, the collected sensory data is correlated with the operational performance of the pump. This helps in proactively identifying and alerting any type of faults/failures within the vacuum pumps in real-time, thus avoiding any kind of unplanned downtime or outages within scientific laboratories. In order to provide a more comprehensive performance monitoring of cleanroom equipment, digital twins of vacuum pumps are designed which are real-time replicas of the physical assets deployed within cleanrooms. These digital twins can empower the users to simulate different “What-if” failure scenarios for the actual pump which can be analyzed to understand how the pump will perform in different environments, situations, and stressors, and subsequently, predictions can be made for the corrective actions or adjustments needed to achieve the desired performance in real-time.

MAINTLET’s website https://t2c2.csl.illinois.edu/projects/maintlet includes links to data, code, results, and simulations as they are developed. The project-related information will be accessible for at least five years after the project ends.

MAINTLET Core System:

MAINTLET V1 Prototype:

Customized Data Acquisition System Used by MAINTLET

Customized Vibration Sensor

To reduce the cost of the vibration sensor, we customized the vibration for the academic cleanroom environment. We use an accelerometer to acquire the vibration acceleration and use a metal plate, and adhesives to attach the vibration sensor to the pump. The experiment result shows this low-cost vibration sensor can capture vibration with frequency from 100 to 20KHz. At the same time, it has 10X less cost than industrial ones on the market.

Sensing Terminal

The Sensing Terminal consists of a Raspberry Pi Zero with a customized sensing array hat and a SENSELET interface. It is used to record environmental temperature, humidity, surface temperature (via SENSELET interface), vibration, magnetic field, and audio signals (via customized Sensing Array).

Microphone Array

The Microphone Array consists of a Raspberry Pi 4 with a microphone array hat. It is used to record multi-channel audio signals with which we can detect pump failures.

MAINTLET V2 Production Version:

Building on the success of the MAINTLET prototype, we have developed the MAINTLET production version hardware, including an edge device and compatible vibration, audio, and magnetic field sensor. The edge device can sample up to six sensory streams at 48 kHz and run lightweight deep-learning models simultaneously with minimal latency. All sensors have been carefully selected, and their enclosures are specifically designed to ensure reliable attachment and superior sensing quality. We compared our hardware system with a data acquisition system commonly used in academia, using a welding machine wear detection task as a benchmark. Our system delivers comparable performance at just one-twentieth of the cost. In summary, the production version hardware is more cost-effective, compact, and reliable than the prototype, and is ready for large-scale deployment.

We have deployed the production version hardware to HMNTL and MRL on various types of pumps, as shown in the following image. Image (a) shows the prototype. Images (b) and (c) show the new hardware deployed on a test pump and an in-use pump, respectively.

Vacuum Pressure Tracker:

When using vibration, audio, or current data emitted from a pump to indirectly detect a pump anomaly, it’s important to note that these anomalies might not always be a result of pump damage; human factors can also play a role. To ensure the sensory data collected by the MAINTLET system accurately reflects the level of pump damage, it’s essential to collect data that truly represents pump performance, such as pressure data for vacuum pumps.

To collect this pressure data, we constructed a customized vacuum system. The tracker edge device comprises an esp32-c3 MCU, an analog/digital converter, a voltage regulator, and a commercial digital vacuum gauge. This gauge is configured to read the current pressure inside the pump. We collect this data through the ADC, transmit it to a Raspberry Pi via Wi-Fi, and make the vacuum data streams available on a server database. These data streams are stored alongside anomaly scores and raw vibration data for correlation analysis.

After conducting several vacuum pump leakage simulation experiments, we observed a strong correlation between our anomaly prediction scores and the vacuum readings. As a future endeavor, we plan to train a deep-learning model to predict the vacuum gauge readings based on the multimodal sensory data collected by the MAINTLET system.

MAINTVision System:

Vision data, being the most informative data modality, is essential for maintenance. A significant portion of daily maintenance tasks involves reading and logging important gauges and level indicators, which can be time-consuming and prone to errors when done manually. To address this, we have designed a system to automate this process, as shown in the following image. The system consists of a 3D-printed camera mount and an ESP32-S3 microcontroller with a camera. The camera takes a picture of the gauge, processes the image, and sends the gauge reading to the Maintenance Data Management System. With the entire system costing less than $20, it can be easily deployed on a large scale. By modifying the 3D-printed mount, we can easily adapt this system to monitor other important status indicators, such as a pump’s oil level. We have finished the hardware design and plan to develop a deep-learning model to read the gauge in the coming year.

MAINTTrack System:

We have designed a data management system to efficiently handle pump information, equipment details, and maintenance history, and scheduling. The system is built with scalability in mind to accommodate use by multiple labs on our campus. Our design process involves close collaboration with experts from MRL and HMNTL to incorporate features that meet the needs of our end users. Furthermore, the system integrates data from the MAINTLET smart systems, including anomaly events detected by the MAINTLET system and gauge readings from MAINTVision. Thus, it serves as a comprehensive, one-stop interface for users to manage and view maintenance data from multiple sources.

MAINTStorage System:

The MAINTStorage project aims to simplify the complex process of chemical storage in laboratories by tagging chemicals from their introduction into the lab until their disposal. Chemicals, classified as campus purchases, lab purchases, or personal purchases, will have smart tags providing necessary information like which content is inside, who it belongs to, for which purpose, etc. These tags will be tracked by edge devices in both storage and cleanroom areas, enabling automatic usage monitoring. The project faces challenges including choosing the right sensing technique due to distance constraints and varying transmission properties of different containers and chemicals. Current efforts are focused on overcoming these challenges.

Publications:

Klara Nahrstedt, Ragini Gupta, Beitong Tian, Zhe Yang, Patrick Su, Robert Kaufmann, Xiaoyang Wang, Cody Wang, Leah Espenbahn, Ahmadreza Eslaminia, John Dallesasse, Gianni Pezzarossi, Mauro Sardela, “Sensing and Computing Challenges in Academic Ultra-Clean Environments for Enhanced Data Integrity”, Open Access article, University of Illinois Urbana-Champaign, 2022.

T2C2: Timely and Trusted Curation and Coordination