Introduction

 

A detailed summary and analysis have been conducted on the application of computer vision in industrial inspection Join the Polar City CV Technology Exchange Group and stay at the forefront of computer vision

 

[Too long to read the read-only summary, please refer to the following two paragraphs.]

 

Computer vision plays an important role in detection, recognition, and localization analysis in intelligent manufacturing industry inspection, making significant contributions to improving the detection speed and accuracy as well as the level of intelligent automation in industrial inspection. However, computer vision has always faced technical application difficulties in the application process, among which the three major bottleneck problems are:

 

Computer vision applications are susceptible to lighting effects

 

The sample data is difficult to support deep learning

 

Difficulty in incorporating prior knowledge into evolutionary algorithms

 

This article summarizes and analyzes the current situation and existing solutions of the three major bottleneck problems in computer vision applications. After in-depth analysis, it was found that:

 

For the problem of being greatly affected by lighting, it can be solved through two stages: algorithm and image acquisition;

 

To address the issue of difficulty in supporting deep learning with sample data, small sample data processing algorithms and balanced sample size distribution methods can be used to solve the problem;

 

The problem of difficulty in incorporating prior knowledge into evolutionary algorithms can be solved through machine learning and reinforcement learning.

 

[Main text section]

 

The Development Status and Needs of Computer Vision in Intelligent Manufacturing

 

1.1 Current Development Status of Computer Vision in Intelligent Manufacturing

 

The development of computer vision has gone through four stages. The first stage is called computational vision, the second stage is active and target vision, the third stage is hierarchical 3D reconstruction theory, and the fourth stage is learning based vision, as shown in Figure 1.

 

The Four Stages of Computer Vision Development

 

Specifically, in the application of intelligent manufacturing, industrial inspection is the main direction of computer vision applications. Due to inevitable defects and errors in production, defective parts or products may appear. Therefore, detection is required at the backend of the assembly line. At present, most of them use manual detection methods or machine methods with low automation, which leads to the originally improved efficiency of the assembly line slowing down again due to the detection process.

 

 

Therefore, the application of computer vision inspection technology in the field of intelligent manufacturing industrial inspection is crucial. At present, various related studies are very extensive, ranging from visual inspection of car bodies in the automotive manufacturing industry to surface defect detection of bearings. Table 1 presents a computer vision method for image-based bearing fault diagnosis.

 

Table 1 Comparison of Computer Vision Methods for Bearing Fault Diagnosis

 

 

 

1.2 Development Requirements of Computer Vision Technology for Intelligent Manufacturing

 

The intelligent manufacturing industry involves a large number of detection processes, such as defect detection, deformation detection, texture detection, size detection, etc.

 

As one of the most effective methods in the field of inspection, computer vision technology will inevitably trigger a revolutionary transformation in the manufacturing mode in the application of industrial inspection. It can once again liberate labor, significantly improve the production efficiency of the manufacturing industry, reduce production costs, reduce production links, and promote the formation of fully automated production lines.

 

However, there are still many bottleneck problems in the practical application of computer vision in the field of intelligent manufacturing industry inspection that have not been solved, and three key bottleneck problems are worth studying and discussing.

 

1) The actual intelligent manufacturing industry has a complex environment and simple light sources, which can easily cause uneven lighting and make it difficult to solve the problem of image quality being greatly affected by lighting. In practical applications in the field of detection, the uncertainty of industrial site environmental changes can affect the image acquisition process of computer vision. In industrial testing, products with high consistency on the assembly line are usually detected, and the defects that need to be detected are usually relatively small, so there is a high demand for images. In addition to ensuring consistency in camera parameters, it is also necessary to control the influence of environmental factors, which is one of the unique control factors in industrial testing. Due to the large randomness of environmental changes, controlling lighting has become a key bottleneck problem in computer vision in the field of intelligent manufacturing inspection.

 

2) In the actual intelligent manufacturing industry, obtaining balanced sample data with a scale of over ten thousand is costly and difficult to solve the problem of sample data not supporting deep learning based computer vision detection tasks. Among all learning methods, sample data is one of the most important factors. Especially in deep learning, it often requires a very large number of samples to achieve excellent detection results. Within a certain order of magnitude (under learning), the sample and detection performance are even directly proportional. In the intelligent manufacturing industry, the collection of sample data is a major issue. Due to the pursuit of profit by enterprises, sample data collection cannot be carried out smoothly like research, and even the total production of some products cannot reach the sample data size required for deep learning.

 

3) In the intelligent manufacturing industry, computer judgment is difficult to reach the level of professional judgment, and how to incorporate prior knowledge into algorithms to improve the effectiveness of evolutionary algorithms is a major challenge. How to effectively utilize prior knowledge and reduce the dependence of deep learning on large-scale annotated data has become one of the main directions of research in the industry. Due to the diverse forms of prior knowledge, how to effectively integrate it with deep learning is a major challenge. In the field of industrial testing, the problem is even more severe. While solving the above problems, it is also necessary to consider the following difficulties: how to transform industrial testing professional knowledge that is more complex than ordinary prior knowledge into knowledge graphs and other forms to integrate into algorithms; How to establish standardization, standardization, and unification of prior knowledge in industrial testing; How to infer the prior knowledge of other similar products that are not included in the knowledge base based on the prior knowledge of existing products.

 

The problem of computer vision applications being easily affected by lighting in intelligent manufacturing

 

2.1 Overview of issues that are greatly affected by lighting

 

Industrial testing is different from other testing fields, and testing of different industrial products is usually carried out in different environments. Generally speaking, the production process of industrial products is carried out in an open workshop or warehouse environment. Poor natural lighting, complex light sources, and unprofessional lighting equipment are common problems. In addition, the intelligent manufacturing field has more stringent requirements for detection accuracy and speed. Lighting control, as an important way to improve recognition rate, requires better technical support.

 

 

For large-scale intelligent manufacturing industrial sites, the open and complex working environment can easily cause large-scale changes in lighting intensity during the process of capturing images. In contrast, small industrial sites have less interference from natural lighting and other factors, but the detection precision of small products is higher, and the requirements for lighting stability are also raised, making it more difficult to control lighting. The model trained on the sample dataset obtained under stable lighting conditions in the laboratory cannot achieve good detection results in industrial sites. Not only does the harsh lighting conditions increase the difficulty of computer vision applications in intelligent manufacturing, but the accuracy requirements in the field of intelligent manufacturing are also more stringent. Therefore, how to control the uniformity of lighting is currently a major bottleneck issue.

2.2 Solutions for high exposure to light

At present, research on lighting problems in computer vision in the field of intelligent manufacturing industry inspection mostly starts from algorithms. A machine vision based image correction and fault detection algorithm for non-uniform illumination conveyor belts is proposed, based on Retinex's X-ray non-uniform steel wire rope core conveyor belt image correction and enhancement algorithm. The lighting normalization method based on statistical characteristics fully considers the lighting locality of the image. By adjusting the mean and variance of the local image, linear interpolation method is introduced, and the logarithmic transformation is combined with the method proposed in this paper to adjust the lighting of the image, which can effectively normalize the lighting. The lighting characteristics of industrial testing vary randomly, and this method can be dynamically adjusted based on the characteristics of the image, making it applicable.

In the field of intelligent manufacturing industrial inspection, cameras are needed to capture images for further processing and analysis through computer vision technology. Therefore, controlling lighting in the image acquisition process is another commonly used method. Its characteristic is that it can avoid complex algorithm implementations, but it will increase hardware costs and complexity. In the process of image acquisition for a certain object, the configuration of the camera and lens directly affects the imaging effect. The optimal camera and lens configuration can be determined through debugging. At the same time, external light sources can effectively reduce the interference of ambient light on image acquisition, ensure the stability of a series of images, and also adjust to specific lighting suitable for industrial testing.

The optimal situation for visual acquisition and the lighting coordination technology mainly based on the light source and camera coordinate the camera and light source as a whole, using the alternating method to optimize the configuration of the optimal lighting and camera position, so that the image in the current environment has a high average brightness and contrast. This method can improve the quality of sample image data, increase the convergence index of lighting by about 0.15%, but also increase the complexity of image acquisition. The adaptive adjustment algorithm for digital camera parameters is used to improve the robustness of machine vision systems to lighting changes. It can adjust the parameters and settings of digital cameras online according to the lighting conditions of the external environment, in order to capture images with pixel grayscale within the preset value range, effectively reducing the impact of lighting changes on image grayscale. The disadvantage is that real-time adjustment requires high requirements for the camera and adjustment algorithm, making it difficult to ensure long-term accurate implementation and easy to lose real-time performance.

The problem of difficulty in supporting deep learning with sample data from computer vision applications in intelligent manufacturing

3.1 Overview of Difficulty in Supporting Deep Learning with Sample Data

In intelligent manufacturing industry inspection, in addition to traditional template matching methods, the more mainstream method currently is based on deep learning. Due to the high accuracy requirements of industrial testing, supervised deep learning can meet the requirements. A major characteristic of supervised deep learning is the need for a training image dataset with known class labels, which theoretically contains a larger amount of data, resulting in better detection performance. However, considering the actual scenarios of industrial testing, it is difficult to collect such a large amount of sample data. In addition, the number of defective products in the manufacturing industry is far less than that of qualified products, and the randomly obtained training set samples will have the problem of imbalanced sample distribution. Even in the binary classification problem of large sample data, the sample distribution should strive to account for 50% of each class.

 

 

In the field of intelligent manufacturing, there are two methods to obtain sample image data: online collection during assembly line operations, and offline collection of sample images.

The advantage of online collection during assembly line operations is that it does not affect industrial production and can complete the collection of sample image data. The image data comes from industrial sites and can be collected in a sample image dataset that is equivalent to the maximum production quantity. But the problem is that the sample categories are distributed according to the pass rate, and the sample images of defective products will be much smaller than those of qualified products, resulting in imbalanced sample distribution. Secondly, the shooting environment on industrial sites cannot be guaranteed, which can easily lead to inconsistent quality of sample images and affect training effectiveness.

On the other hand, the advantage of the offline collection mode of sample swing is the controllable sample distribution, which manually controls the consistent distribution of sample quantities for each category; A controllable laboratory shooting environment can ensure obtaining high-quality sample images. The disadvantages are also very obvious. Collecting samples requires a lot of time and shooting is time-consuming and labor-intensive. The training model may not be suitable for industrial field detection, and large sample image datasets cannot be obtained in a short period of time. Therefore, due to the small size and imbalance of sample data, it is difficult to support deep learning, which has become a major bottleneck problem at present.

3.2 Solutions for Difficulty in Supporting Deep Learning with Sample Data

There are difficulties in obtaining a large amount of effective sample data in the field of intelligent manufacturing detection, as well as difficulties in obtaining sample datasets with balanced distribution of various sample sizes. The problem of difficulty in obtaining a large amount of valid sample data can be transformed into the problem of how to achieve good detection and classification performance for small sample data. The main methods are shown in Table 2.

Table 2 Methods for detecting classification using small sample data

 

There are solutions as shown in Table 3 for sample datasets that are difficult to obtain balanced distribution of sample sizes across different categories.

Table 3 Solutions to imbalanced sample distribution

 

The research results indicate that the current difficulties in supporting deep learning with sample data mainly include small sample size and imbalanced sample data. There are already research systems and methods for these two problems as described above. This type of sample data processing algorithm is based on the number of samples rather than the content of the samples. Therefore, in the field of industrial detection in intelligent manufacturing, using the above method to adjust sample data is completely feasible.

Difficulty in incorporating prior knowledge of computer vision applications into evolutionary algorithms in intelligent manufacturing

Overview of the difficulty of incorporating prior knowledge into evolutionary algorithms

The application of computer vision in intelligent manufacturing is essentially a data-based method, but it is difficult to obtain a large amount of uniform sample data in the field of industrial inspection. Therefore, researchers propose to incorporate prior knowledge into computer vision algorithms to achieve better detection results. By applying methods based on prior knowledge, samples can be used for training during the training phase to improve the accuracy of model parameters, reduce learning difficulty, facilitate the convergence of the training process, and thus improve the accuracy of prediction. In the prediction stage, prior knowledge can be used to correct the judgment results, improve accuracy, and also improve detection speed, avoiding the occurrence of accidental errors.

 

 

At present, prior knowledge is difficult to incorporate into evolutionary algorithms, and it is even more difficult to guide algorithms such as machine learning and deep learning. There are also many bottleneck problems that need to be solved. For example, how to use knowledge graphs as the main form of knowledge representation to guide deep neural networks; How to use natural language to guide intelligent agents in reinforcement learning to quickly and accurately understand learning; How to integrate transfer learning as knowledge into reinforcement learning; How to apply reinforcement learning methods to industrial testing through domain knowledge.

 

4.2 Solutions for Evolutionary Algorithms Unable to Support Prior Knowledge

 

There are currently the following research and methods on how to apply prior knowledge to learning and in what form. One approach is to apply the compactness information of samples as prior knowledge to the construction of support vector machines. By modeling the confidence level of tight density, fuzzy connectivity can distinguish support vectors from noisy samples. This method can obtain support vector machines with better noise resistance and classification ability. By using the compactness information of samples as prior knowledge, it not only considers the center distance between sample classes, but also considers the relationship between samples and other samples within the class, and distinguishes them through fuzzy support vector machines. In the field of industrial testing, there is not much difference between defect samples and normal samples themselves. If the tightness information of the samples can be added to the training, it will help improve the training effect and more accurately separate small and weak defect samples.

 

Xuan Dongmei et al. proposed two methods to integrate prior knowledge with deep learning models. The first approach essentially uses the output obtained from deep learning as the conditional probability of a given sample, which is a random deep learning classifier with prior knowledge (RPK). The second method involves adding a parameter to adjust the sparsity of prior knowledge, which is a deterministic deep learning classifier with prior knowledge (DPK). The classifiers obtained by these two methods can better predict the results. However, these two methods are not high enough in terms of recognition rate for multi classification tasks, and can only accurately perform binary classification tasks. Both incorporate prior knowledge in the form of matrices within the framework of deep learning models, which can be any matrix form of content. In the field of intelligent manufacturing industry inspection, neural network parameters such as those used for detecting bearing defects can serve as a prior knowledge matrix for detecting circular injection molded parts.

 

In addition, knowledge-based reinforcement learning has significant advantages in the application of prior knowledge. This type of method has good performance in analyzing and predicting directions, therefore it has huge development prospects in the field of industrial testing. Table 4 presents four typical methods.

 

Table 4 Knowledge based Reinforcement Learning Methods

 

The above research results indicate that in the process of combining prior knowledge and deep learning, a knowledge-based reinforcement learning theory has been formed, and its birth further verifies the effectiveness of applying prior knowledge in evolutionary algorithms. This provides an important direction for improving the detection effect of intelligent manufacturing industry through prior knowledge.

5 Conclusion

With the deepening of computer vision in the field of intelligent manufacturing, it will involve more and wider applications, and its role will also be increasingly significant. In the process from theory to application, many bottleneck problems have been encountered. How to overcome these difficulties and explore more practical solutions will be the focus of research work in the next stage.

At present, there are numerous research methods for the three bottleneck issues, and further research can be conducted from the following perspectives:

1) To address the issue of computer vision applications being susceptible to light interference, a black box packaged image acquisition device can be designed to eliminate external light interference and installed on industrial production lines, achieving a laboratory level detection environment and fundamentally solving the problem of light interference.

2) In response to the problem of difficulty in supporting deep learning with sample data in computer vision applications, small sample and imbalanced sample processing methods can be used to increase the sample data volume without reducing the quality of the sample data, and traditional methods such as template matching and similarity detection can be combined to assist in increasing detection accuracy.

3) In response to the difficulty of incorporating prior knowledge into evolutionary algorithms in computer vision applications, in addition to starting from the training and prediction stages, decision-making can also be made based on prior knowledge. In the inspection of parts with high qualification rates, some uncommon erroneous judgments can be modified to correct judgments based on prior knowledge, thereby improving accuracy.

Note: * This article (including images) is a reprint. If there is any infringement, please contact us to delete it.