WO2023219514A1

WO2023219514A1 - System and method for estimating weight of biomass

Info

Publication number: WO2023219514A1
Application number: PCT/NO2023/050104
Authority: WO
Inventors: Emil BJØRKLYHAUG; Daniel KVAM; Marius NEDRELID; Eivind STORØY; Johan ESPELUND
Original assignee: Optimar As
Priority date: 2022-05-09
Filing date: 2023-05-05
Publication date: 2023-11-16
Also published as: NO20220528A1

Abstract

A system for estimating weight of biomass (1), comprising: a conveyor (2a), (2b) with the biomass (1), one or more laser units (3) and/or one or more cameras (4) scanning the biomass, one or more light sources to illuminate the biomass, a computer unit (5), receiving input data from the one or more laser units (3) and/or one or more cameras (4), the computer unit (5) includes software that provides height map and outer measurements of the biomass from the input data and estimates the weight of the biomass (1) based on product data, biological ratios and adjustment factors, and where the system include artificial intelligence and machine learning to continuously improved the system and the weight estimation.

Description

System and method for estimating weight of biomass

INTRODUCTION

The present invention relates to a system for estimating weight of biomass and a method for estimating weight of biomass transported on a conveyor.

BACKGROUND

Several devices for weighing of biomass such as fish, based on mechanical or electronic techniques are known. Many processing plants primary sort products, for example fish, by that an operator observes the sizes and lifts the product for primary sorting in different weight classes. A grading weight is often located adjacent, which measures weight of the product.

It is desirable for most food producers to have continuously control of the biomass in production in different processes. For example, sorting fish into different sizes and/or weight classes, or counting or measuring total biomass that passes through a process plant. The biomass from the net-pen is an estimate from growth factors, but actual biomass to slaughter is unknown until the fish passes graders. Knowing the actual amount of biomass as early as possible in the process is desirable to plan post-processing and sales to customers. Therefore, the present innovation which can be on any conveyor at any point in the factory, is a useful tool for fish processing plants.

Counting biomass of cleaner fish is also desirable for the fish processors and will become a demand. On processing vessels, catch needs to be logged. When combining the present method with species recognition in the vision module, each species can have a logged total biomass.

NO 336911 B1 discloses a method for estimating the weight of fish, comprising transporting the fish on a conveyor belt with associated sensor for measuring the speed of the conveyor belt. The fish passing one or more recording units which recording data on the part of the fish passing under a light source. The height profile of the fish is divided according to variation in density of different parts of the fish for the current species. The volume of the fish is estimated for each part of the height profile. US2008137104 discloses a method for recording and estimation of the weight of fish. A number of cameras, especially CCD-cameras, record pictures of fish moving by the cameras in a transfer conduit. The fish is illuminated from different sides in the transfer conduit and pictures of different parts of the fish are recorded by a sequence control, in such a way that a compound image recording is made, which is used as a base for an estimation of the weight of fish. The present invention relies only on capturing images from one side, and combining the heightmap with images to estimate a volume. The “empty” space below the midpoint of the fish’s height is removed from the volume estimation through relationships between height (thickness) and volume.

There is a need of a method for estimating weight of biomass transported on a conveyor in an effective way and that provides accurate results in a user-friendly way. It is further desirable that a system could handle and collect a very large amount of data and where the data could be stored, reported and possibly used quickly and effectively. It is an aim of the present invention to provide a system and method for estimating weight of biomass transported on a conveyor, that addresses at least one of these problems while retaining the benefits of prior art. Current weight estimation systems relies on “one and one” fish entering the system, whilst the present innovation calculates the entire biomass on a conveyor and only requires the fish to not overlap significantly.

SUMMARY OF THE INVENTION

This is achieved according to the present invention by providing a system for estimating weight of biomass, comprising: a conveyor with the biomass, one or more laser units and/or one or more cameras, one or more light sources, a computer unit, receiving input data from the one or more laser units and/or one or more cameras, the computer unit includes software that provides height map and outline I outer measurements of the biomass from the input data and estimates the weight of the biomass based on biomass data, biological ratios and adjustment factors, and where the system further comprises artificial intelligence and machine learning algorithms, analyzing large amounts of data, to continuously improve the system by utilizing data from downstream processes and improving the weight estimation.

In another aspect, the present invention provides a method for estimating weight of biomass transported on a conveyor, where the biomass enters the conveyor and passes one or more light sources illuminating the biomass, and one or more laser units and/or one or more cameras scanning the biomass, which provides input data to a computer unit, and processing the input data in the computer unit which gives height map and outline I outer measurements of the biomass, and estimating the weight of the biomass by use of biomass data, biological ratios and adjustment factors, and continuously improving the method by utilizing data from downstream processes and by using artificial intelligence and machine learning analyzing of large amounts of data, to improve the weight estimation.

In yet another aspect, the present invention concerns use of the system 10 to provide, collect and report data of biomass weight or number of biomass such as marine products e.g. fish and crabs etc., passing on a conveyor, and optionally to sort different species and/or to sort the biomass t based on weight or volume of the biomass.

Further advantageous features of the present invention are defined in the dependent claims.

BRIEF DESCRIPTION OF DRAWINGS

Example embodiments will now be described with reference to the following drawings, where:

Figure 1 is a schematic representation of a conveyor-belt where the biomass is e.g. fish.

Figure 2 shows a schematic representation of a channel including product transport equipment with fish as the biomass.

Figure 3 shows a schematic block diagram of a computer unit receiving input data from the laser units and/or cameras, including artificial intelligence. Figure 4 shows a simplified program sequence.

DETAILED DESCRIPTION

In the following, general embodiments as well as particular exemplary embodiments of the invention will be described. References will be made to the accompanying drawings. It shall be noted, however, that the drawings are exemplary embodiments only, and that other features and embodiments may well be within the scope of the invention as claimed.

In one aspect, the present invention relates to a system 10 as shown in figure 1 and figure 2, for estimating weight of biomass 1 . The system comprises a conveyor 2a, 2b with the biomass 1 , one or more laser units 3 and/or one or more cameras 4 scanning the biomass. The system can optionally comprise one or more light sources to illuminate the biomass and reflecting colours of the biomass. As shown in shown in figure 1 and figure 2 the conveyor 2a, 2b could be a conveyor-belt, moving gutter or a moving channel including product transport equipment.

The laser units 3 and/or cameras 4 and light sources are positioned in a suitable distance from the conveyor 2a, 2b, and the biomass 1 which enters the conveyor 2a, 2b passes the laser 3 and/or camera 4 and light sources if required. The laser unit 3 points directly down and camera unit 4 should be angled such that the center of the cone "hits" the laser line. The cameras used could be black and white or colour cameras. A stereo camera, with the ability to capture three-dimensional images, could also be used. The laser units 3 could be of any known type, a.o. laser triangulation units, a time-of-flight camera and structured light.

The system also includes a computer unit 5, as shown in figure 3, receiving input data from the one or more laser units 3 and/or one or more cameras 4, where the computer unit 5 includes software that provides height map and outline I outer measurements of the biomass from the input data and estimates the weight of the biomass 1 based on product data, biological ratios and adjustment factors. The product data are e.g. density and biological ratios are outer dimensions as length, width and height of the biomass. It is different adjustment factors for different species. Examples of adjustment factors can also be seasonal variations, variations in fishing grounds (location), ecological or conventional farmed fish and other factors. It is possible to change the adjustment factors all the time, to take account of variations that may occur, and these will affect the calculated weight based on volume since the biomass parameters will change on the basis of these. The ecological salmon can have a higher fat content than normal farmed fish, for example, which causes the density of the fish to decrease since fat takes up more space than muscle. In the same way, it is differences in the fat content of cod from different fishing grounds and in different seasons, which also affects the factors that determine the ratio between volume and weight.

The blocks illustrated in figure 3 are explained in more detail below;

Laser beam: line laser emitting a line across the conveyor.

Laser triangulation: camera is angled such that it’s focus point is on the laser line. Camera capture: 1 mm resolution, e.g. one image is stripped to 1mm in height across the conveyor width. Laser line “hits” the 1 mm strip, which gives an intensity value of the laser beam across the 1mm strip.

Images: A builds up of 1 mm strips continuously. The different 1 mm strips are accumulated into whole images up to a certain value, at which point a new image is started and the following 1 mm strips are accumulated into an image.

Defining height map and outline I outer measurements: The camera is angled such that the focus point hits the laser beam. The beam will have a light intensity depending on how high it is, and the different intensities build up a height map of the fish. The height map consequently also produces the outer measurements, due to threshold values for the height map. Calculate biomass weight: Volume x weight ratio.

Biomass ratios: Found through feed backing actual weight of total biomass and number of fish to the weight estimation system.

Weight of biomass: From downstream weighing operations I the predicted biomass from volume x weight ratio.

Artificial intelligence improvements: Continuously improving the factors by utilizing data from downstream processes to increase prediction accuracy.

Biomass data: Data from weighing system downstream. The system 10 according to the invention will be continuously improved by using machine learning and artificial intelligence to improve the weight estimation. By use of artificial intelligence and machine learning algorithms the system can analyze large amounts of data with accurate results. Adjustment factors are provided by machine learning and the adjustment factors used can be improved through a control loop, as shown in figure 3, that returns e.g. the weight of biomass to the system if the biomass is weighed later downstream. Information is retrieved from the grading system (package system) about total biomass and number of fish, and this information fed to the algorithm. In this way, the algorithm is continuously improved. The system 10 further comprises a human-machine interface (HMI) for collecting and reporting data.

In an embodiment the system comprises a biomass 1 weight-sorting unit downstream the conveyor 2a, 2b. Different density factors for different species can be used, where different density factor for different species is provided from a weight reference. Based on the weight estimation from the computer unit 5, the system is used for sorting of different species. The biomass product 1 is marine products such as fish and crabs etc., and the system 10 is located on a fishing vessel or onshore.

In another aspect the invention concerns method for estimating weight of biomass 1 transported on a conveyor 2a, 2b, where the biomass 1 enters the conveyor 2a, 2b and passes one or more laser units 3 and/or one or more cameras 4 scanning the biomass 1 , and providing input data to a computer unit 5. The input data are processed in the computer unit 5 which gives height map and outer measurements of the biomass, and by use of product data, biological ratios and adjustment factors estimating the weight of the biomass 1 . The method for estimating weight of biomass is continuously improved by using machine learning and artificial intelligence.

The reference used in the machine learning is a weight unit downstream or from the slaughter report, i.e. total biomass for each species slaughtered divided on the number of fish counted. During the development, reports were received after the slaughtering, i.e. of the total biomass and the number of fish. When the fish have been graded, it has also been weighed, but this happened much later in the process than desired by the customers. This information is now, according to the invention, included in the machine learning algorithm, and it is now possible to estimate total biomass and count the number of fish as soon as it enters the fish slaughterhouse. This is very beneficial to the customers compared to the previous solution, when they do not have to wait until all the fish have been packed to know the exact number fish and biomass. This allows the customers to plan/staff postprocessing (packaging, etc.) earlier, as well as allowing adjustment of the sales predictions and then sold more accurate fish/biomass. Previously, the fish was sold based on estimates, and the error range was often large. This error range is now significantly reduced because the adjustment can be done earlier.

The adjustment factors are provided by machine learning and the adjustment factors are improved by a control loop that returns e.g. the weight of biomass to the system when the biomass is weighed later downstream. The method according to the invention, further comprises collecting and reporting data of the biomass 1 by a human-machine interface.

In an embodiment the method of the invention comprises weight sorting of the biomass 1 by a weight-sorting unit downstream the conveyor 2a, 2b. The biomass product is marine products such as fish and crabs etc. entering the conveyor 2a, 2b, and the fish could be gutted or ungutted.

The present invention also concerns use of the system to provide, collect and report data of biomass weight or number of biomass such as marine products e.g. fish and crabs etc., passing on a conveyor, and optionally to sort different species and/or to sort the biomass product based on weight or volume of the biomass. The present invention uses the system 10 to calculate and show/report the biomass weight distributed for each individual of the different species of biomass. The system recognizes the species and calculates the weight. For each individual, the system will classify the individual (recognition of species) and give the individual a weight. Such that each individual gets an ID with the associated species and weight. Figure 4 shows how the machine vision system according to the invention works to estimate volume and weight of biomass, as well as recognize species automatically. Data collection is done using two cameras. One to find a height profile, and one to take a colour image. The height profile is used to estimate volume and to count objects. Colour image is used to identify species. To calibrate the system, various target objects are sent through to adjust lighting and colours, distances and volume estimates.

The sequence management as described below and shown in figure 4, is used in the machine vision and species recognition. The steps A-R shown in figure 4, are explained below:

A: "Timed image producer"

Each camera has a picture frequency that is adjusted according to the speed of the conveyor belt. An image from colour camera and one image from black and white camera are collected every second for processing.

B: "Heightmap and colour image"

An image is extracted from each of the cameras covering a desired time frame.

C: "Interpolation of image"

The images are interpolated such that each pixel in the image represents 1x1 mm in reality.

D: "Image to machine learning"

The images are sent to the machine learning model module and further to the graphics processing unit (GPU).

E: "Machine Learning"

F: "Composite heightmap and monochrome representation" A combined height profile and grayscale image will be created. G: "Clean detection area of passed fish"

Fish that have a timestamp outside the image are removed.

H: "Find blobs in current image"

If conventional imaging methods are used, fish of a given size in the image will be found here.

I: "Watershed the blobs"

Watershed algorithm is used to find volume.

J: "Add new fish" If new fish are found in steps H-J and these meet given criteria, these will be registered as fish and stored with time frame and outline.

K: "Create label image of current fish"

An image marking new fish is created to step P.

L: "Collect machine learning predictions"

Results from step D are taken from the GPU.

M: "Determinate species"

For all new fish, use the information in the image to determine species.

N: "Make monochrome and RGB machine learning predictions and watershed label image"

An image showing the results is created in step P.

0: "Upload image to cloud"

Images produced in step C are sent to the module for upload to Azure (cloud computing platform).

P: "Store image for video"

Images produced in steps C, F, K and N are sent to the module for video streaming. Video can be viewed in the machine panel or in SCADA solution. -Stacked image

-ML predicted image

-Colour image

-High map image

Q: "Update fish counters"

New subjects with weight are registered in a database.

R: "Print roundtrip time for all actions performed"

Prints a message in the log showing time spent for all steps, collectively and individually.

This can optionally be stored in a database or made available elsewhere if necessary.

The software architecture is designed such that it is easy to replace individual modules to improve the results or extract more information. This also applies to the models, such that the model or combination of models that will give the best result on a given set of data are selected. Registration of training data:

Which images that are uploaded is at present dependent on the facility's internet connection, the customer's wishes and what remains to train the models. The most common is that maximum one image is uploaded per minute, as long as something worth saving is found. The machine has an overview of how many images of each species have been uploaded and makes the assessment accordingly. When the images have been uploaded, a cloud service is used to draw an outline around the fish in the images.

Raw data for inspection during inspection or failure:

All settings have a time stamp for the last change, and it is possible to upload the entire setup as a backup. Process values, such as accumulated volume and counting are stored when changing, and gradually aggregated in larger and larger time intervals to save space. Process values that have been aggregated such that they apply to entire hours are never deleted. Subject size is aggregated as frequency distributions with 'buckets' of 100g/mm3. Images are uploaded according to item 0. It will be possible to temporarily store locally if the network connection is interrupted.

The system for weight estimation provided by the invention is based on outer measurements, it gives accurate results and provides continuously control of the biomass. It is also advantageously by the present invention that it is continuously improved by using machine learning and artificial intelligence to improve the weight estimation. The system could easily store, report and use a very large amount of data i.a. by that the system include a human-machine interface for collecting and reporting data. By the present invention a large amount of data can be handled, stored and reported quickly and efficiently. By estimating biomass early in the slaughter process, it is easier to plan the sale of fish more accurately and also know the needed amount of operators for the machines doing post processing. This reduces waste and increases the price for the fish, by not having the price dumped if a rest of fish is present in the factory when everything is sold based on early estimates. Having described example embodiments of the invention it will be apparent to those skilled in the art that other embodiments incorporating the concepts may be used. These and other examples illustrated above are intended by way of example only and the actual scope of the invention is to be determined from the following claims.

Claims

1 . A system (10) for estimating weight of biomass (1 ), comprising:

- a conveyor (2a, 2b) with the biomass (1 ),

- one or more laser units (3) and/or one or more cameras (4),

- one or more light sources,

- a computer unit (5), receiving input data from the one or more laser units (3) and/or one or more cameras (4), the computer unit (5) includes software that provides height map and outline I outer measurements of the biomass from the input data and estimates the weight of the biomass (1 ) based on biomass data, biological ratios and adjustment factors, and where the system (10) further comprises artificial intelligence and machine learning algorithms, analyzing large amounts of data, to continuously improve the system by utilizing data from downstream processes and improving the weight estimation.

2. The system (10) according to claim 1 , where the one or more laser units (3) and/or one or more cameras (4) are positioned in a suitable distance from the conveyor (2a, 2b), and where the biomass (1 ) enters the conveyor (2a, 2b) and passes the laser (3) and/or camera (4).

3. The system (10) according to any of the preceding claim, wherein the biomass data are e.g. density, and biological ratios are outer dimensions as length, width and height of the biomass.

4. The system (10) according to any of the preceding claim, wherein the adjustment factors are specific and vary;

-for different species,

-at different seasons,

-at variations in fishing grounds,

-for ecological or conventional farmed fish, and because of other affecting factors, and where the adjustment factors can be changed to take account of variations that may occur.

5. The system (10) according to any of the preceding claim, wherein adjustment factors are provided by machine learning and the adjustment factors used can be improved through a control loop that returns e.g. the weight of biomass to the system when the biomass is weighed later downstream. The system (10) according to any of the preceding claim, where the one or more laser units (3) are laser triangulation unit(s), a time-of-flight camera(s) or structured light. The system (10) according to any of the preceding claim, where the system (10) comprises a human-machine interface for collecting and reporting data, and optionally a biomass (1 ) weight-sorting unit downstream the conveyor (2a, 2b). The system (10) according to any of the preceding claim, where different density factors for different species are used, and based on the weight estimation from the computer unit (5), the system is optionally used for sorting of different species. The system (10) according to any of the preceding claim, where the biomass (1 ) is marine products such as fish and crabs etc., and the system (10) is located on a fishing vessel or onshore. The system (10) according to any of the preceding claim, where the conveyor (2a, 2b) is a conveyor-belt, gutter or channel including product transport equipment. A method for estimating weight of biomass (1 ) transported on a conveyor (2a, 2b), where the biomass (1 ) enters the conveyor (2a, 2b) and passes one or more light sources illuminating the biomass, and one or more laser units (3) and/or one or more cameras (4) scanning the biomass (1 ), which provides input data to a computer unit (5), and processing the input data in the computer unit (5) which gives height map and outline I outer measurements of the biomass, and estimating the weight of the biomass (1 ) by use of biomass data, biological ratios and adjustment factors, and continuously improving the method by utilizing data from downstream processes and by using artificial intelligence and machine learning analyzing of large amounts of data, to improve the weight estimation. The method according to claim 12, wherein the adjustment factors are provided by machine learning and the adjustment factors are improved by a control loop that returns e.g. the weight of biomass to the system when the biomass is weighed later downstream. The method according to any of the preceding claim 11-12, further comprising collecting and reporting data of the biomass (1 ) by a human-machine interface, and optionally weight sorting the biomass (1 ) by a weight-sorting unit downstream the conveyor (2a, 2b). The method according to any of the preceding claim 11-13, where the biomass product is marine products such as fish and crabs etc. entering the conveyor (2a, 2b), and where the conveyor (2a, 2b), the one or more laser units (3) and/or one or more cameras (4) and the computer unit (5) are located on board a vessel or onshore. The method according to claim 14, where the fish is gutted or ungutted. Use of the system (10) according to any of claims 1 -10 to provide, collect and report data of biomass weight or number of biomass such as marine products e.g. fish and crabs etc., passing on a conveyor, and optionally to sort different species and/or to sort the biomass based on weight or volume of the biomass.