Institute of Integrated Sensor Systems...©2019 Andreas König, Institute of Integrated Sensor Systems Andreas König Institute of Integrated Sensor Systems, TU Kaiserslautern, 67663

© 2019 Andreas König, Institute of Integrated Sensor Systems

Institute of Integrated Sensor SystemsDepartment of Electrical and Computer Engineering


Andreas König

Institute of Integrated Sensor Systems, TU Kaiserslautern,67663 Kaiserslautern, Germany

E-Mail: [email protected]

IndusBee4.0 – Sensory Systems and Machine Learning for Optimized Bee-Tending and Hive-Keepers’ Assistance

Outline: Motivation Survey of Bee Hive Instrumentation Activities Condition Monitoring and Industry 4.0 Integrated Sensory Systems for Hive State Monitoring Intelligent Systems for Hive-Keepers’ Assistance Conclusions & Outlook


The role of insects in general, and honey bees in particular, undis-putably is of high relevance for the up-keeping of our ecosystem

Dramatic environmental changes, e.g., from pesticides to parasites threaten the existence of insects, wild bees, and honey bees alike

One consequence: Manual pollination !

Motivation Relevance of Insects and Bees for Ecosystem, Agriculture, and Nutrition

Source: http://www.3sat.de/mediathek/

The movie of M. Imhoof and the book of J. Tautz [1] moved the urgent issue to public awareness


Traditional bee keeping works on scheduled and postmortem basis

It bases on frequent invasive hive inspection (costly, risky, futile …)

Digitized Bee Keeping in the wake of omnipresent automating and networking (Internet_of_Everything !) minimizes hive invasion

Sensors and Electronics analogous to maintenance I4.0 allow to move from scheduled/postmortem to predictive/event-driven activity

Two major lines of activities:– Bee Hive Monitoring by sensors, embedded systems + SW– Hive Keeper Assistance Systems by discrete appliances for a

group of hives

Motivation Traditional and ‘Digitized’ Hive Keeping


Market for Digital Bee Hives [26, 25]:

Motivation Market and Cost Considerations & Constraints for Solutions

Source :https://deutscherimkerbund.de

Total of 1.2 (14) million hives (Germany, EU) → use off-the-shelf components Professional hive keepers have about 100 - 500 hives Price sensitive yet suitable/reliable solutions required


The vivid development in the sensors field, driven by automotive, mobile devices, and automation offer increasing possibilities [4]:

Motivation Sensor Technology, Internet of Everything, Machine Learning, and Hive Keeping

MEMS + IC + RF for cheap & disappearing application systems (Berkeley’s ‘Smart Dust’ vision getting reality)

Source: Yole Developpement

Microphones, Vibration, Pressure, as employed in I4.0 ...


Sensory principles constrained to the use in major research institutes become available for every day/place application:




In particular, infrared sensors and cameras become affordable and small, e.g., Apple I-Phone IR extension module:



Get in reach for hive keeper assistance (external hive monitoring) and soon for at/in hive monitoring use

Source: Indusbee4.0, small bee colony in winter position by low-cost IR-camSource: Yole Developpement


Sensors + IC + RF + Energy Harvesting + Machine Learning/AI

→ Technical Cognition Systems:


AmBEEant Intelligence



The main focus of an increasing Maker community is on bee hive instrumentation by off-the-shelf sensors & embedded systems [27]

Predominantly, hive weight, temperature, moisture, as well as general weather station info is collected by autonomous system

Example of work of Ken Meyer (2015 or earlier):

Survey of Bee Hive Instrumentation Activities Monitoring Weight, Temperature, and Moisture

Source: https://hackaday.io/project/1741/gallery#95fd822b27c667a6daa97ddda3ea5bcf

Arduino/XBEE-based


With regard to cost of commercial hive scales (~ $300-$1400), custom made scales are regarded by [5, 6, 9, 10, 14, 15, 16, 17]

Most approaches differ in detail, i.e., Arduino or Pi, issues of creep and hystesis in scales rarely discussed

IoBEE project [10] also deals with feeder level and activity monitoring (flight hole, see also BeeCounter [23])

OSBEE project [11] deals with acoustical monitoring by AI, data base from discrete monitoring available [11], Buzzbox product

Survey of Bee Hive Instrumentation Activities Adding Sound and Machine Learning

Source: https://www.osbeehives.com/

Housekeeping tool by BeeManager [7] On agenda: IR-Cam, flight hole monitoring [23, 10, 15], etc.

Visible since 06/05/2019


The ‘low hanging fruits’ collected, Makers and Spin-Offs active !

Instrumentation improvements by price & size, power consumption reduction and reliability increase !

Enhancing sensory palette enjoying the progress in other application fields driving technology, e.g., Mini-Visual, IR, Hyperspectral cameras, multiple MEMS Mics. etc.

Applications’ analysis and robust integrated realization required

Sensor fusion and advanced information processing (AI) required

Parallel development to, e.g., Industry 4.0 (Condition Monitoring, Self-X-Systems ….)

Survey of Bee Hive Instrumentation Activities Summary of SoA and Outlook

https://hackaday.io/project/1741/gallery#95fd822b27c667a6daa97ddda3ea5bcf


Condition Monitoring and Industry 4.0Intelligent Condition Monitoring and Self-X Properties

Vibration monitoring of bearings etc. → predictive maintenance, established topic in MEMS/microsystems activity field ...

Sensors, electronics, integration technology, and intelligent systems provide the leverage for next industrial revolution: Industry 4.0

Activities: Smart Factory (DFKI), ICM and Self-X (Mechatronics)

Source: Case Western Reserve University (CWRU) Bearing Data Center [2]

Source: www.moses-pro.de (BMBF-Funded)

https://www.osbeehives.com/


Condition Monitoring and Industry 4.0Advanced IT-Architecture with Self-X-Properties for Polymer Film Industry

PhD work of M. Kohlert, ISE, Mondi Gronau

Currently scaled to 8-16 units On ‘Landkarte Industrie 4.0’

of BMWi

Interactive Multivariate (Big) Data Analysis & Visualization

Anomaly Detection/Prediction


Feature-computation

y = AsxClassifier

Training/TestClassification results

• Saccades (Occular motorics)• Stereo vision• Color vision• Motion detection/estimation• Invariant feature extraction

(Hyper columns)• Figure/ground separation• Gestalt theory

Example by functions of human visual system:

Recognition, Understanding, Acting

Technical cognition systems still mostly nutshell models of bio-evidence:

Adapted from Zell 94

Condition Monitoring and Industry 4.0 Nature makes wide-spread use of diverse sensors, fusion & dynamic processing

http://www.moses-pro.de/


Condition Monitoring and Industry 4.0Intelligent Condition Monitoring by Deep-Learning Approach [2]

Several way to design intelligent system:

Deep neural networks have become popular AI manifestation (Issues !)

90 300

3000 19800

Source: SENSORS, MPDI [2]

Source: SENSORS, MPDI [2]Source: SENSORS, MPDI [2]


Condition Monitoring and Industry 4.0 Automated Intelligent System Design and Self-x Sensor Systems

QuickCogGenesisDAICOX


As in numerous activities, T, H, and weight are monitored Concept of SmartComb introduced for IndusBee4.0:

Integrated Sensory Systems for Hive State Monitoring Standard Parameters Temperature, Moisture, and Weight

DHT22

DHT11

Pi Zero WMEMS I2S Mic. SPH0645LM4H (Cheap, ~ $7)

Pi Zero W with Jessie as core element Half-assembled unit


Hive scale pursued as in numerous SoA activities Modular scale, e.g., for honey combs in honey storage:

Integrated Sensory Systems for Hive State Monitoring Standard Parameters Temperature, Moisture, and Weight

DHT11

DHT22

MEMS I2S Mic. SPH0645LM4H

HX711

Pi Zero W

Pi Zero W SC extension to HX711 Honey storage separate scale as gadget


More scarcely visited is the acoustical monitoring of bee colonies Employing low-cost MEMS microphones as in vibrational/acousti-

cal monitoring in Industry4.0 By Machine Learning approach in IndusBee4.0 integrated sensor system continously monitors and reports hive state:

Integrated Sensory Systems for Hive State Monitoring Acoustical Monitoring of Hive by Integrated Sensory System

Monitoring Result of a Mini Colony

Calm/Ok State

Agitated/Disturbed State Ok/Calm Agitated/Disturbed Knocking/Pecking Scratching Swarm Mood Missing Queen Looting ….

SmartComb



Feature-computation

DimensionReduction

ClassifierTraining/Test

Classification results

Autonomous acoustical bee hive monitoring requires the design of an intelligent decision (Technical cognition) system

Currently, two major ways to go: Deep-Learning (black-box, data greedy) or ‘classical way’ (grey/white box, moderate data requirements)

The latter is chosen with regard to problem complexity, labeled data acquisition, and moderate run-time platform:

Mono Mic.700ms, 16k Sa/s

MFCC(Python speech features)

Selection(QuickCog)

13

KNN (SVM,...)(Python sklearn

Resubstitution~ 97 %

SmartComb in hive excitation, windowing, and labeling:



Monitoring Result of a Mini Colony

Agitated/Disturbed State

Calm/Ok State

Knocking/Pecking

Scratching

Scratching (Soft)

25

20

2514

18



Live classification performed, occasional confusion of class 4 & 2 (see similarity in feature space on prev. slide !)

Improved training/performance by a) richer data collection, dimensionality reduction (AFS: 2,4,5,9,10 SBS qsi),

more/better microphones

b) anomaly detection detecting everything off the ok state in the first step ….

Infineon Donated Sample

1 1 1 1 1 1 11 3 3 1 14 4 2 2 2 2 2 2 2 2 2 2

…. 470 Samples ….1 11 1 1 1 1 1 1 1 1 1 1


Integrated Sensory Systems for Hive State Monitoring Acoustical Monitoring of Hive Employing Anomaly Detection

A common issue known well from, e.g., Industrial Quality Control: It is fairly easy to describe the ‘Good’ or ‘Ok’ state and lots of examples

are commonly available ! But it is extremely hard to a priori know all possible deviations from ‘Ok’

and acquisition of examples might be hard/infeasible !

Learning the Classifier from only ‘Ok’patterns

Classify new patterns by a similarity threshold extracted in training as normal or abnormal


Intelligent Systems for Hive-Keepers’ Assistance Hand-Held Analysis Device

The refractometer is an established tool for meas- urement of humidity degree in honey

Extension to impedance and optical spectros-copic analysis, e.g., as in food inspection (LoS) Source: https://arcarda.com


Intelligent Systems for Hive-Keepers’ Assistance Automating Varoa Counting in Debris

The control and treatment of the varoa parasite requires the repeated tedious checking of hive debris

Assistance system to count from photo in two-stage AI approach

Image sent to host, automated counting, archiving of image/count



The idea is, if feasible, to use a simple commercial camera/mobile Issues: Invariant registration (lighting, camera parameters), effort ! First assumption: The slider edges are grabbed as limits of picture Focus is appropriately set to get unblurred mite images:



To generate train and test data with the required ‘Ground Truth’ for classification, a set of sliders with debris have been acquired:

Confirmed Varoa

Location



To obtain a valid ‘Ground Truth’ number and location of Mites must be known by manual inspection or placement of dead mites

A subsection of a cleaned slider with placed mites is regarded first:

Simple cues can be found for a first computationally efficient attention stage, that produces mite candidate regions



Based on color information on the mites, blob-like regions of minimum size are

detected and counted :

Post-Classification



Histogram of red channel as first simple feature for post classification

Mite/Background data

Real debris will require more examples/feature

True mite detections

Background


Intelligent Systems for Hive-Keepers’ Assistance Automating Varoa Counting in Debris – Real Data Tests

Confirmed Varoa

Locations

Stage 1 filters isolated mites

Meets difficulties in clustered/stapled scenarios



The visual inspection of hive debris on the slider and counting of the varoa parasite is tedious and surprisingly complex

Large inspection area with regard to small object (varoa mite) size ! Actually, a 3D problem, as debris/mites potentially stapled Human inspection: magnifying lens (multi-scale) & hand-eye

coordination for manipulation in the inspection process

Fluorescence photography could be one approach Currently, only a tentative prototype of VaroaCounter available

https://adaptalux.com/wp-content/uploads/2017/12/UV-Spider-2.jpg

Source:


Intelligent Systems for Hive-Keepers’ Assistance WiP:Automated Evaluation of ‘Flight Deck’ Activity

Counting ingoing vs. outgoing bees and check their payload !

Reckoning on the daily pollen input and cues on breeding activity

Video processing and content classification Classical or Deep-NN approach common Computationally hard (for Pi) Requires acceleration, e.g., by Intel NCSx

in emerging NeuroBee-Computer ….

Mockup, work in progress !


Intelligent Systems for Hive-Keepers’ Assistance WiP: Meta-Level AI for Advanced Hive Monitoring

Aggregating information of monitoring and asssistance systems Meta-level AI can give more sophisticated alerting/scheduling

Hives-DB MetAI Unit

Context from DB: Knowledge on Hive Status and Maintenance Activities Varoa Status from Assistance System

Hive Sensors(T, H, Weight, Sound, ...)

Wheather station/forecast information

Flight Hole Observation ... (Bee Counting, Bee Balance, Payload)

Alert and/or Task List to Hive Keeper

Honey

weight



Conclusions and Outlook

Ecological and economical needs & challenges impose increasing load on hive keepers

Approaches from Automation/Instrumentation, Machine-Learning, and Industry 4.0 can provide suitable tools and methods

Two lines of activity: Bee hive monitoring and Hive keeper assistance systems

IndusBee 4.0 long term activity adding hive sound monitoring and automated varao counting system under cost constraints

Flight hole (camera-based) observation (Passenger counting !) and analysis (payload) one promising topic pursued in IndusBee 4.0


Thank you for your Attention !

To bee or not to be !

https://adaptalux.com/wp-content/


References

[1] Jürgen Tautz, Phänomen Honigbiene, Spektrum Akademischer Verlag (Elsevier), 2007[2] Wei Zhang, Gaoliang Peng *, Chuanhao Li, Yuanhang Chen and Zhujun Zhang, A New Deep Learning Model for Fault Diagnosis with Good Anti-Noise and Domain Adapt- ation Ability on Raw Vibration Signals, Sensors 2017, 17, 425; doi:10.3390/s17020425[3] Modular Sensor Systems for Real-Time Process Control and Smart Condition Monitoring https://www.moses-pro.de, 2019[4] Yole Developpement, Technology Reports and Roadmaps, www.yole.fr, 2019[5] Ken Meyer, https://hackaday.io/project/beemon , 2015 or earlier[6] http://beeandmegmbh.com/[7] https://www.beemanager.de/, 2017 [8] https://duino4projects.com/honey-bee-counter-using-arduino/, 2012 or earlier[9] https://www.hackster.io/pvalyk/beekeeping-with-arduino-4216bb[10] IoBEE, http://users.telenet.be/iobee/index_en.html, 2017 or earlier[11] OSBEE, https://www.osbeehives.com/pages/about-us[12] Acoustical Archive OSBEE, https://www.dropbox.com/sh/us1633xi4cmtecl/AAA6hplscuDR7aS_f73oRNyha?dl=0[13] http://hivetool.net/index.shtml,[14] R. Esser, http://www.randolphesser.de/imkerei/, 2017 or earlier[15] R. Esser, http://claudiaesser.de/imkerei/projekte/Bienenstockwaage_Esser.pdf


References

[12] Acoustical Archive OSBEE, https://www.dropbox.com/sh/us1633xi4cmtecl/AAA6hplscuDR7aS_f73oRNyha?dl=0[13] http://hivetool.net/index.shtml,[14] R. Esser, http://www.randolphesser.de/imkerei/, 2017 or earlier[15] R. Esser, http://claudiaesser.de/imkerei/projekte/Bienenstockwaage_Esser.pdf [16] https://www.imker-nettetal.de/tag/stockwaage/ , 2016 or earlier[17] https://www.honey-pi.de/, 2018[18] [19] https://www.magnetkontor.de/, 2019[20] https://www.supermagnete.de/scheibenmagnete-neodym/scheibenmagnet , 2019[21] https://www.magnet-shop.net/, 2019[22] Web-Discussion on Localization, https://www.mikrocontroller.net/topic/156639 , 2009[23] BeeCounter, http://duino4projects.com/honey-bee-counter-using-arduino/ , 2012[24] BeeMonitor (from 2012), https://beemonitor.org/, 2019[25] Marie-Pierre Chauzat, Laura Cauquil, Lise Roy, Stephanie Franco, Pascal Hendrikx, Magali Ribiere-Chabert, ‘Demographics of the European Apicultural Industry’ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3827320/pdf/pone.0079018.pdf[26] Deutscher Imkerbund, https://deutscherimkerbund.de


References

[27] Paul Wallich, ‘Beehackers – Cheap widgets are like honey to hive keepers’, IEEE Spectrum, pp. 20-21, May, 2011

Documents

Institute of Integrated Sensor Systems...©2019 Andreas König, Institute of Integrated Sensor Systems Andreas König Institute of Integrated Sensor Systems, TU Kaiserslautern, 67663