Created by: José Miguel Sanchiz - May 6th 2022
José Miguel Sanchiz is Full Professor at the Jaume I University of Castellón. His background is a Telecommunications Engineer, a Bachelor of Physical Sciences and a Computer Science Engineering Doctor. He is the author of 10 articles published in peer-reviewed international articles, and of a number of papers at international conferences that exceeds 50ies
1. Projects to develop machines with artificial vision
KEELTEK ENGINEERING SOLUTIONS S.L. is one of the companies with which it collaborated as a partner and expert in design and solution projects that our clients require for their industrial and production plants when we find improvement opportunities in which introducing artificial vision is the key factor to increase the “capacity” of the equipment and/or “increase” the quality of the products manufactured
The 3 most interesting applications of artificial vision that I have developed in projects for the design and commissioning of equipment and machinery are:
Online decorated ceramic tile inspection, model learning and defect detection
Detection of ceramic bases, their holes to insert decorations, and of decorative tesserae, all online for manufacturing by a robot arm
Boxes and packages detection and measurement online using 3D vision
This POST exposes some basic concepts about the evolution of artificial vision and its applications in the development of equipment and machinery for industrial and production processes in the sectors of the world’s economy that need to incorporate innovation to increase the productive capacity of their plants and take advantage of the opportunities for improvement within the production lines in which artificial vision can be incorporated. I am writing this article because many people responsible of industrial processes ask me: What is artificial vision applied to Industry 4.0 about?
Let's start by asking ourselves: What do we understand by Artificial Vision? They are systems that basically detect two types of images:
Light Reflectance: Those in which the sensor or sensors measure the intensity of reflected light
Distance: Those in which the distance to the surface of the object is measured
Although both types of systems have the mission of detecting data to organize them in a matrix of rows and columns, that is, an image, their content is very different in nature. In the case of data obtained by Light Reflectance we have what is commonly known as an image, which can be in color or black and white (gray levels). In the case of Distance sensors we have a range image, that is, an image of depths from the sensor to the surface of the object, in each measurement direction. If the geometry of the sensor is also known, which is always known, the depths can be converted into 3D coordinates (x, y, z) of each point of the surface taken, in the sensor coordinate system, that is, we obtain an image known as 3D.
Images are said to be 3D because the data is arranged in rows and columns. There are sensors that generate unorganized 3D points; this is known as a point cloud. Its usefulness is quite different from the one we are dealing with here, being 3D modeling the main use.
In projects about development of machines and equipment for industrial plants and processes, various technologies are used to obtain 2D (of reflected light) and 3D (of the object's surface) images. Let us detail these systems:
2. Imagin capture systems
They are the usual images of reflected light. They can be classified according to the spectrum of the light that is captured and according to the geometry of the sensor.
Looking at the spectrum of the captured reflected light, we have:
Images of gray or black and white levels: A single spectral band is captured, which can be in the visible, infrared or ultraviolet. The range of the captured spectrum is normally fixed by means of a filter on the lens, or between the lens and the sensor (CCD or CMOS). An industrial application can be the detection and measurement of shapes, that is, the contour of objects, for example for fruit classification.
Colour images: Three tuned bands in three ranges of the spectrum around the colors red, green and blue. These bands are normalized. By Manufacturing, tri-CCD sensors are usually used; in which each pixel has three small sensors with three color filters, in red, green and blue. An industrial application may be the color measurement of mosaic tiles for quality control.
Hyper-spectral images: These are images in which the sensor is tuned to a specific spectrum band. The tuning is set electronically and is quickly reconfigurable. Normally many bands are taken at different times, one after the other, which is applicable if the object is stationary. Main applications are in capturing satellite’s images and inspecting products looking for a specific chemical element, molecule, etc., which is known to reflect more light in a certain band of the spectrum.
Regarding the geometry of the sensor, there are matrix sensors or cameras and linear sensors or cameras. The sensors have a different construction as well as different uses.
Matrix cameras: are those in which the sensor is a matrix, that is, it is two-dimensional. Examples are all types of cameras and conventional industrial cameras. An image of an area, of an entire object, is taken at once. This technique is useful when the objects or working areas are small. The drawbacks regarding the industrial application are that it is difficult to achieve a constant lighting throughout the area. Also, objects can cause shadows. Its use is usually restricted to inspection areas no more than 500 x 500 mm and with flat or nearly flat objects.
Linear cameras: The sensor is one-dimensional, a line. To obtain an image of an object, it must be moved at a constant speed, or the camera must be moved (which would have many drawbacks, such as: generating vibrations and obtaining blurred or blurred images). While the object is moving at a constant speed, lines are taken at such a rate that the line spacing is equal to the pixel resolution. Square images are thus obtained. An alternative, if the speed of the objects is not constant, is to use an encoder and trigger the line grab every time the object moves at fixed distance, regardless of its speed. This technique should be used if the line can stop and start again, causing accelerations. An industrial application can be the inspection of large format ceramic tiles.
The main advantages of linear cameras are that the lines are always taken in the same one-dimensional area, which makes it much easier to produce constant illumination. There are also no shadows. On the other hand, these cameras provide 2D images as if they were matrix, and the objects can be of considerable size, as long as they pass under the camera. The image of the entire object is not formed until everything has passed through, but in many inspection applications partial images of the object can be analyzed as it passes. Linear cameras have also a much higher resolution than matrix cameras, with models ranging from 4K, 8K or 16K pixels per line. The number of lines that an image taken with a linear camera can have is indeterminate.
3. 3D imagin systems
3D images are points in space arranged in a matrix. Normally the sensors that take 3D images also provide a 2D image of the reflected light, but they are used in applications in which theinterest is to analyze the shape of the surface of the objects, and not the light that they reflect in each part or point of the object’s surface. So the 2D image they provide (of gray levels) is not often used, but is available if needed, such as to locate a printed logo on the object.
There are basically two types of 3D sensors:
TOF cameras or time of flight of light (TOF: Time of Flight): They are matrix cameras. They incorporate a spotlight or illuminator that illuminates the object with light modulated with a high-frequency signal. Normally infrared light so that it is not visible. The light reflected and received at each sensor element, on each pixel, is also modulated and compared to the emitted light. The comparison is made by electronic means. It consists of measuring the phase difference between the wave that modulates the emitted light, available in the sensor, and the wave that modulates the received light. Obviously, the lag time is proportional to the distance traveled by light, since the speed of light is a universal constant.
Today these cameras do not have much resolution or precision to be used in industrial applications. Nor do they have the robustness that industrial elements require. They are basically oriented to the entertainment industry, video games, etc., to capture the movements of the players so that they can interact.
Profilometers: These devices are the 3D counterpart to linear cameras. Its operation is very similar in the way images are formed (3D in this case). The object must pass at constant speed under the profilometer, and it takes lines. Each pixel contains the 3D coordinates (x, y, z) of a point on the surface. If the speed of the object is not constant, the line capture can be triggered by means of an encoder each time it advances a fixed distance. The operating principle is that a laser plane is emitted and there is a high-speed matrix camera with a filter tuned to the spectrum of the emitted laser. The laser intersects the object producing a line on its surface. The line is easily identified automatically by signal analysis methods. Software included in the profilometer analyzes each image taken. From the calibration of the laser plane with the camera, it obtains the 3D coordinates of each point.
Profilometers have a basic industrial use. They are designed for industry and, depending on the model; security measures due to the power of the laser will be required. The main manufacturers of industrial sensors include them in their catalogues. Its precision depends on the size of the area to be captured, and can reach hundredths of a millimeter.
4. Advantages of using 3D images versus 2D
3D images taken by scanning parts as they pass along a line are different in nature from light reflectance images, whether conventional color images, gray level images in the visible or infrared bands, or multi-band images in the case of infrared images.
Multispectral Establishing strategic alliances with professionals and companies to create and lead high-performance teams in order to deliver to our clients the industrial equipment required by the evolution of industry 4.0 and the potential capacity that our clients want to achieve with excellent levels of productivity because operators and maintenance personnel are formed to keep OEE efficiency levels > 95%, which are world-class values in industrial, service and production companies in any sector of the world that are committed to operational and operational excellence and have an industrial asset plan to migrate to Industry 4.0 productivity levels.If you have identified an opportunity for improvement or a “bottleneck” in your industrial plant, contact us and KEELTEK will give you ideas and suggestions without any commitment. We like technology and our vision is to help industrial companies by contributing ideas and innovation
Notice that the measurements contained in a 3D image are distances to the surface of the parts. The sensor converts these distances into (x, y, z) coordinates of the points from their geometry, given in the sensor's coordinate system. These are real coordinates from which real measurements can be extracted without calibration needs, as would be necessary with 2D images.
The 3D images are organized points in a matrix, unlike other 3D acquisition systems that provide disorganized point clouds. This is very important because the surrounding area of each point is known, so you can think on the surface. This surrounding area would have to be reconstructed in the case of a point cloud. The measurements of a 3D image are independent of light reflectance, which represents a great advantage over reflectance images, since it allows reasoning and detecting facets of interest on the surface regardless of its colour.
Having the geometry of a surface and not its color or gray level in a certain band allows detection to be based on the shape of the surface, looking for facets such as extreme areas of curvature, peaks, valleys, heights, corners, directions of the principal curvatures, etc. However, 3D imaging equipment often also provides a 2D image of reflected light, which correspond to pixel-by-pixel to the 3D image.
The most suitable sensor for obtaining 3D images in the industry is a profilometer. This equipment scans objects as they pass under it by projecting a laser plane, which draws a line or profile on the object. Therefore, external lighting is not necessary. The distortion of the line following the shape of the surface allows the coordinates to be detected at each point on it, which is handled by the sensor's built-in software. The number of points in each line depends on the sensor’s resolution; typical values are 1,280 or 2,048 points.
5. I need a project company for a project that requires machine vision
En KEELTEK ENGINEERING SOLUTIONS S.L is an expertise integrating the people who are going to make up the project teams for the solution development for the industry.
Establishing strategic alliances with professionals and companies to create and lead high-performance teams in order to deliver to our clients the industrial equipment required by the evolution of industry 4.0 and the potential capacity that our clients want to achieve with excellent levels of productivity because operators and maintenance personnel are formed to keep OEE efficiency levels > 95%, which are world-class values in industrial, service and production companies in any sector of the world that are committed to operational and operational excellence and have an industrial asset plan to migrate to Industry 4.0 productivity levels.
If you have identified an opportunity for improvement or a “bottleneck” in your industrial plant contact us and we will give you ideas and suggestions without any commitment.
We like technology and our vision is to help industrial companies by contributing ideas and innovation.