Development of a Bridge Inspection Support System Using

Development of a Bridge Inspection Support SystemUsing Two-Wheeled Multicopter and 3D Modeling Technology

Paper:

Development of a Bridge Inspection Support System UsingTwo-Wheeled Multicopter and 3D Modeling Technology

Yoshiro Hada∗1,†, Manabu Nakao∗1, Moyuru Yamada∗1, Hiroki Kobayashi∗1,Naoyuki Sawasaki∗1, Katsunori Yokoji∗1, Satoshi Kanai∗2, Fumiki Tanaka∗2, Hiroaki Date∗2,

Sarthak Pathak∗3, Atsushi Yamashita∗3, Manabu Yamada∗4, and Toshiya Sugawara∗5

∗1Safety Solution Business Unit, Fujitsu Limited10-1 Morinosato-Wakamiya, Atsugi, Kanagawa 243-0197, Japan

†Corresponding author, E-mail: [email protected]∗2Hokkaido University, Sapporo, Japan∗3The University of Tokyo, Tokyo, Japan

∗4Nagoya Institute of Technology, Nagoya, Japan∗5Docon Co. Limited, Sapporo, Japan

[Received October 24, 2016; accepted May 19, 2017]

Recently, many countries have faced serious prob-lems associated with aging civil infrastructures suchas bridges, tunnels, dams, highways and so on. Ag-ing infrastructures are increasing year by year andsuitable maintenance actions are necessary to main-tain their safety and serviceability. In fact, infrastruc-ture deterioration has caused serious problems in thepast. In order to prevent accidents with civil infras-tructures, supervisors must spend a lot of money tomaintain the safe conditions of infrastructures. There-fore, new technologies are required to reduce mainte-nance costs. In 2014 the Japanese government startedthe Cross-Ministerial Strategic Innovation PromotionProgram (SIP), and technologies for infrastructuremaintenance have been studied in the SIP project [1].Fujitsu Limited, Hokkaido University, The Universityof Tokyo, Nagoya Institute of Technology and DoconCo. Limited have been engaged in the SIP projectto develop a bridge inspection support system usinginformation technology and robotic technology. Oursystem is divided into the following two main parts:bridge inspection support robots using a two-wheeledmulticopter, and an inspection data management sys-tem utilizing 3D modeling technology. In this paper,we report the bridge inspection support system devel-oped in our SIP project.

Keywords: bridge inspection, UAV, 3D point cloud data,3D CAD model, geotagging

1. Introduction

Recently, many countries have faced a serious problemassociated with aging civil infrastructures such as bridges,tunnels, dams, highways and so on. Aging infrastructuresare increasing year by year and suitable maintenance isrequired to maintain their safety and serviceability. InJapan, it became mandatory in 2014 to perform close vi-

sual inspections of all bridges once every five years in or-der to ensure the continuous long-term use of the agingbridges. However, Japan has 700,000 bridges spanning atleast 2 meters, and the visual inspection of bridges is amonumental and costly task. To date, bridge inspectionshave been carried out by human inspectors, but there arenot enough human resources with sufficient skills to per-form inspections. Because of these issues, new technolo-gies are required to make the bridge inspection operationefficient and low-cost.

We consider that conventional inspection methods havethe following problems:

• High cost to access inspection locations: Whenbridge inspectors go to high positions of a bridge,they often use a special bridge inspection vehicle ora scaffold or rope access. The use of the bridge in-spection vehicle can result in long periods of trafficinterruptions. The formation of the scaffold requiresa lot of time and money. Rope access requires in-spectors to learn special skills to move at high alti-tude.

• High cost of organizing inspection data: Bridge in-spectors organize inspection data such as photos ornotes to create an inspection report by hand. Creat-ing the inspection report is time-consuming and suchmanual work is likely to cause human error.

• Difficulty to understand aging of a structure: InJapan, most bridge inspection results are recordedand managed in electronic formats such as PDF orMicrosoft Excel. However, they are not managed in adatabase system, making it difficult to compare pastinspection results with current data. This preventsraising the efficiency of the bridge inspection.

To solve the above problems, Fujitsu Limited,Hokkaido University, The University of Tokyo, NagoyaInstitute of Technology and Docon Co. Limited have beenengaged in the national project of the Cross-Ministerial

Journal of Disaster Research Vol.12 No.3, 2017 593

https://doi.org/10.20965/jdr.2017.p0593

© Fuji Technology Press Ltd. Creative Commons CC BY-ND: This is an Open Access article distributed under the terms of the Creative Commons Attribution-NoDerivatives 4.0 International License (http://creativecommons.org/licenses/by-nd/4.0/).

http://creativecommons.org/licenses/by-nd/4.0/

Hada, Y. et al.

Fig. 1. The outline of the proposed bridge inspection support system.

Strategic Innovation Promotion Program (SIP), and wehave been developing a bridge inspection support systemsince 2014. In this paper, we introduce the bridge in-spection support system that is being developed in the SIPproject.

This paper is organized as follows: Section 2 showsan overview of the bridge inspection support system thatwe have developed in the SIP project. Section 3 intro-duces two types of our newly developed bridge inspec-tion robots. Section 4 describes the bridge inspectiondata management system using 3D modeling technology.In Section 5, we propose three bridge inspection scenariosutilizing the technologies that we have developed, and theresults of substantive experiments demonstrated at bridgesites are presented. Section 6 discusses related works andsection 7 provides a conclusion.

2. Overview of Bridge Inspection Support Sys-tem

This section outlines the bridge inspection support sys-tem that we have developed in the SIP project. Thebridge inspection is divided into two main works: fieldwork, such as performing visual inspections of bridges,and office work, such as creating reports of inspection re-sults. Fig. 1 depicts the outline of the proposed bridge in-spection support system. The system consists of the fol-lowing five key technologies: (a) bridge inspection sup-port robots, (b) automatic geotagging technique, (c) 3Dmodel-based bridge maintenance database, (d) bridge in-

spection support software and (e) a semiautomatic 3DCAD model generation method from point cloud data.

(a) Inspection support robots: In a conventional bridgeinspection, an inspector approaches the inspectionpoint using a special bridge inspection vehicle toconduct a visual examination and/or a hammeringtest. It is labor-intensive and time-consuming work.In recent years, an unmanned aerial vehicle with amulti-rotor, called a multicopter, has rapidly becomethe standard tool to capture visual information inplaces that humans have not been able to access. Inthe SIP project, we developed a two-wheeled multi-copter to support the bridge inspection. We describethe details of our newly developed inspection robotin Section 3.

(b) Automatic geotagging technology: To date, bridgeinspection reports have been created to organize in-spection data such as photos and notes by hand.Therefore, there have been problems due to it takingtoo much time to create reports and because humanerror occurs. To solve these problems, we have de-veloped automatic geotagging technology based onthe structure from motion (SFM) technique using a360-degree spherical camera in the SIP project. Weprovide the details of our newly developed geotag-ging technology in Section 4.

(c) 3D model-based bridge maintenance database: Todate, traditional bridge inspection methods havedistilled three-dimensional information into two-dimensional planes. That is to say, inspection re-

594 Journal of Disaster Research Vol.12 No.3, 2017


Fig. 2. Deteriorations in 2D sketch.

sults are currently expressed as twodimensional in-formation, though bridge deteriorations appear in itsthree-dimensional structure. Up to now, cracks onfloor slabs have been depicted in two-dimensionalsketches, for example, as shown in Fig. 2. We con-sider that human beings can intuitively and easily un-derstand the state of bridge deteriorations if the in-spection results are visualized using 3D expression.In addition to this, if inspection data are stored in adatabase system, it is possible to easily track the ag-ing of bridge damage. If a combination of 3D tech-nologies and database technologies was available inbridge management, we would be able to comparethe current damage at a position on a bridge with thedamage record at the same point which was regis-tered in the database five years ago on the 3D mod-els of the bridge. This is helpful for both bridge su-pervisors and inspectors to understand the progressof damage. In the SIP project, we have developedthe 3D model-based bridge maintenance database, inwhich the Industry Foundation Classes (IFC)-baseddata model we proposed is used. We describe thedetails in Section 4.

(d) Bridge Inspection Support Software: We have devel-oped software to support bridge inspection, whichcan access to the 3D model-based bridge mainte-nance database to read/write inspection data. Usingthis software, the inspector can check the progress ofbridge deteriorations by comparing past inspectionresults on the 3D bridge model displayed on his orher tablet PC. If the inspector finds new damage on

the bridge, he or she can register them on site usingthis software.

(e) Semiautomatic 3D CAD model generation method:In the 3D model-based bridge maintenance database,the inspection data are managed using a 3D CADmodel of the bridge. But there are few 3D CAD mod-els, or even 2D CAD models, of existing bridges.Notably, 3D measurement technologies such asSFM, which is a standard range imaging technique,and terrestrial laser scanning (TLS) are becomingrapidly available. Therefore, we developed a semi-automatic 3D CAD model generation method from3D point cloud data in the SIP project. With thistechnique, we will create a 3D CAD model of thebridge, for which there has been no design drawingso far.

3. Bridge Inspection Support Robot

In this section we describe the bridge inspection robotsthat we have developed in the SIP project. As discussedabove, bridge inspection is currently labor-intensive andtime-consuming work. To reduce the maintenance cost ofcivil infrastructures such as bridges, tunnels and so on, itis necessary to improve the efficiency of inspection meth-ods by utilizing new technologies. These days, unmannedaerial vehicles (UAVs) are often used to examine bridges,especially at high positions, instead of a human inspec-tor. Use of the multicopter, which is a kind of multi-rotorUAV, has been spreading around the world since around2005. The multicopter has the features that its mechanismis very simple and easy to manufacture at a low cost, andanyone can fly it easily with little practice. Because of itsavailability and simple operability, many bridge inspec-tion robots using multicopters have been studied.

The basic method of bridge inspection by human in-spector is visual inspection, where the inspector ap-proaches to within around 1 m of an inspection point toconfirm the presence or progress of deterioration such ascracks on concrete surfaces, rusting of steel and so on.Hence, a main task of a bridge inspection multicopter isto capture images of the structure surface. In fact, manyresearchers have been tried to capture images of bridgesfor inspection purposes, but multicopters have not entirelyreplaced human inspector yet. The reason is that it is diffi-cult for the multicopter to fly under windy conditions andcapture images that have the necessary quality for inspec-tion. In addition to this, because of the complex airflowaround a bridge, it is difficult to fly the multicopter whileavoiding collisions with the bridge. To overcome theseproblems, we developed the two-wheeled multicopter [2,3]. The two-wheeled multicopter has the following advan-tages compared to conventional multicopters for bridgeinspection purposes. The two-wheeled multicopter hasa wind-resistant performance against crosswind by us-ing contact friction between the wheels and the structuresurface. It can also move along the surface to maintaina constant distance between an onboard camera and the


Hada, Y. et al.

Fig. 3. The appearance of the large sized two-wheeled mul-ticopter.

structure surface. It allows images to be captured at closerange. In fact, we have successfully captured a deterio-ration image of a crack 0.1 mm in width on a concretesurface.

We consider it is impossible to survey all necessary por-tions of a bridge with a single robot type, and have devel-oped two types of inspection robots that have differentinspection targets. One has a large size for high positions,and the other a small size for narrow spaces.

3.1. Large-Sized Two-Wheeled MulticopterUntil now, bridge inspectors have usually examined

high piers over 30 m in height by the rope access, which islabor-intensive and time-consuming work. As an alterna-tive method, we have developed a large two-wheeled mul-ticopter for high pier inspection. Because of restrictionsplaced on UAV flight by Japan’s Ministry of Land, In-frastructure, Transport and Tourism, the multicopter mayonly be used in conditions where the wind speed is nogreater than 5.0 m/s [15]. It is necessary for the high pierinspection UAVs to work for a long time. However, cur-rent battery technology does not allow UAVs to fly forover 20 minutes continuously and so does not provideenough time to perform inspection of high piers. In or-der to achieve a long flight time we adopted a method ofwired power feeding.

Figure 3 shows the appearance of the large-sized two-wheeled multicopter. Its specification is listed in Table 1.As depicted in Fig. 4, a bundle of power supply cables,optical fiber cables for video transmission and Kevlarfiber wire are connected to the large-sized two-wheeledmulticopter through the cable guide tube. The cable guidetube can rotate around the wheel axis. By this mechanism,we can connect the cable to the multicopter both from theupper and lower side of the bridge. It also prevents the

Table 1. Specification of large sized two-wheeled multicopter.

Width 1100mmWheel diameter 800mm

Weight 3.5kg (without camerasand wire guide tube)

Fig. 4. The layout of cables.

Fig. 5. The two-wheeled multicopter is suspended by the cable.

bundle of cables from being entangled in the UAV’s ro-tors. The length of Kevlar wire is a little shorter than thepower feeder cable and the optical fiber cable, and is con-nected to the tip of the cable guide tube so that the weightof the two-wheeled multicopter is not applied to the powerfeeder cable and the optical fiber cable. It is expected thatsuspending the two-wheeled multicopter from the bridgeas shown in Fig. 5 prevents the multicopter from crashing.

The two-wheeled multicopter is equipped with a close-



Fig. 6. The developed wired power supply system.

up camera to capture image of a bridge surface. We useda small action camera with 2K (1080 p) resolution so thatwe could recognize cracks of 0.1 mm width on the con-crete surface. 2K resolution images are transmitted tobridge inspectors on the ground via an optical fiber ca-ble in real time. We can select different angles of viewand resolution for the close-up camera depending on theinspection purpose. In order for the multicopter to flycontinuously for a long time, we have developed a wiredpower supply system, shown in Fig. 6.

3.2. Small-Sized Two-Wheeled Multicopter forNarrow Space Inspection

The shoe is one of the crucial parts of a bridge. Damageis likely to occur around shoes, and a bridge inspector ex-amines them thoroughly during inspections. Notably, theconventional multicopter cannot capture close-up imagesof the shoe, especially its side view, because it is difficultfor the multicopter to fly toward a bridge shoe. Therefore,we developed a small-sized two-wheeled multicopter inthe SIP project. In general, wind-resistant stability andsize of body are in the relationship of trade-off. We con-sider that accessibility to the narrow space is more impor-tant than wind-resistant stability for practical use. There-fore, we designed the small multicopter to use under weakwind condition (2.0 m/s) in our operation scenario.

In order for the small-sized two-wheeled multicopterto enter the narrow space, a protection mechanism is re-quired. The mechanism prevents bridge structures frombeing damaged by collision with the multicopter. We de-signed the small-size two-wheeled multicopter as shownin Fig. 7. The quad copter is within the cylindrical shapeof the cage, which plays the role of protector. The speci-fications are listed in Table 2. The two rings attached onboth the right and left sides are passible wheels and theycan rotate around the pitch axis independently. This pas-sive wheel mechanism enables the multicopter to rotatearound the yaw axis being in contact with the horizontalplane. A small action camera, the GoPro HERO4 Session,is mounted to capture images. The camera transmits cap-tured images to the inspector via Wi-Fi and he or she cansee close-up images of the bridge on his/her smartphonein real time. The small multicopter is powered by an on-

Fig. 7. The small sized two-wheeled multicopter.

Table 2. Specification of small sized two-wheeled multicopter.

Width 450mmWheel diameter 400mm

Weight 1.2kg (without camera)

board Li-Po battery and is capable of continuous flight forup to 7 minutes.

4. Inspection Data Management System

In this section, we describe the inspection data man-agement system for bridge maintenance. The system con-sists of the following components: (1) automatic geotag-ging technology using a 360-degree spherical camera, (2)3D model-based bridge maintenance database, (3) bridgeinspection support software, and (4) semiautomatic 3DCAD model generation method.

4.1. Geotagging Technology Using 360-DegreeSpherical Camera

In general, the region under a bridge is a GPS-deniedenvironment, where GPS is not available or does not workwell. Simultaneous localization and mapping (SLAM) isa well-known technique to estimate position in a GPS-denied environment. For example, a laser scanner-basedSLAM is proposed in [4]. However, the weight of thelaser scanner is too large to mount on a UAV. In addition,its power consumption is also large. To realize a posi-ton estimation of a UAV in a GPS-denied environment,The University of Tokyo, a member of the SIP project,has developed a method of position estimation using asequence of images from a 360-degree spherical camera.The method is based on an SFM technique. We adopt theRICOH THETA, which is lightweight (about 130 g) andcan work continuously for about one hour using a built-inbattery.

We use the position estimation method for geotagging.Geotagging is the process of adding position informationto photos or video. In the SIP project, we use the geotag-


Hada, Y. et al.

Fig. 8. The geotagging flow.

Fig. 9. A screenshot of the geotagging.

ging technique to determine where close-up images werecaptured. The geotagging flow is shown in Fig. 8. TheRICOH THETA and a close-up camera are mounted onthe two-wheeled multicopter as shown in Fig. 3 to captureimages synchronously during the bridge inspection. Afterthe bridge inspection, the inspector feeds the movie filecaptured by RICOH THETA into the position estimationsoftware developed by the University of Tokyo. The soft-ware estimates position using dense optical flow-basedmotion estimation to determine the point where a framecaptured by the spherical camera was taken. The detailalgorithm of position estimation using RICOH THETAis described in [5]. The position estimation result andthe close-up photography video are fed to the geotaggingsoftware developed by Fujitsu Ltd. The geotagging soft-ware shows both the movement locus of the two-wheeledmulticopter and close-up images of deterioration on the3D CAD model of the bridge inspected. The inspectorcan check the close-up deterioration image on the com-puter display as shown in Fig. 9. Geotagged deteriorationimages are registered into the database system mentionedabove.

In order to evaluate the measurement accuracy of theproposed position estimation algorithm, we conductedan experiment on a real bridge pillar. Because camera-based SFM does not discern the real-world scale of move-ment, we used markers attached at known locations on thebridge pillar to provide the multicopter’s initial position aswell as ground truth for localization accuracy, as shown inFig. 10. Two trajectories of 6.6 m and 11 m were evalu-ated. The errors were 0.27 m and 0.69 m, respectively, in-dicating errors of around 4–6 %. Because the presence oftextures and feature points is necessary for camera-basedSFM methods, the textures present on the bridge surface

Fig. 10. Experimental setup to evaluate proposed.

play a large role in determining accuracy, as do the light-ing conditions. In conditions of bright light or darkness,textures can become invisible to the camera and SFM ac-curacy may be reduced. At such times, it may be neces-sary to adjust the camera’s exposure in order to properlyview the texture patterns on the concrete surface.

4.2. 3D Model-Based Bridge MaintenanceDatabase

The inspection information needs to be stored and man-aged for a long period because bridges are utilized overa long period of more than 50 years. In Japan, bridgeinspection is conducted once every five years. Hence,new inspection data are added every 5 years. In the de-velopment of the 3D CAD model-based bridge mainte-nance database, we consider that it would be pointless toadopt a 3D CAD model format that was dependent on aparticular CAD vendor, because it is possible that the 3DCAD format will change by upgrading to new version orthe CAD vendor’s product will disappear due to changesin the business environment. Therefore, a new design of3D CAD model that can withstand long-term use is re-quired. In the SIP project, we have proposed the datamodel which is an extension of the IFC-Bridge to managethe bridge inspection data. IFC-Bridge is an ISO stan-dard of the 3D CAD model of bridge structure developedby buildingSMART R©. We consider that the data modelbased on the extension of IFC-Bridge is suitable for long-term use. Once the specification is determined, it is notsubject to change. See [6] for details of the IFC-Bridge-based data model.

4.3. Bridge Inspection Support SoftwareWe developed Web browser-based software that can ac-

cess the 3D mode-based bridge maintenance database to



Fig. 11. A screenshot of Web-based bridge inspection.

read or write inspection data. A screenshot of the de-veloped software is shown in Fig. 11. By utilizing thisweb-based software, we can resister geotagged deterio-ration images, which were developed using the geotag-ging software described in section 4.1, to the maintenancedatabase. Geotagged images are automatically linked tothe 3D CAD model of the inspected bridge. The inspec-tor can view inspection results on the 3D CAD model dis-played on his or her computer screen. See [6] for details.

4.4. Semiautomatic 3D CAD Model GenerationMethod from 3D Point Cloud Data

We have developed a method to generate a 3D CADmodel of a bridge from three-dimensional point clouddata in the SIP project. One key component is the reg-istration algorithm [7], which performs alignment of 3Dpoint cloud data acquired at different measured points byTLS to generate an as-built model of the bridge automati-cally. The other is a technique to extract a principal planefrom the 3D point cloud data automatically. The CADoperator feeds the extracted plane information into CADsoftware and then follows the ridge line on the extractedplane by hand to generate a 3D CAD model of the bridge.Fig. 12 depicts the flow of the proposed method.

5. Inspection Scenarios and Substantiative Ex-periments

In this section, we propose some bridge inspectionscenarios to test the two-wheeled multicopters in actualbridge inspections. We conducted substantiative experi-ments based on the proposed scenarios.

5.1. Bridge Inspection Scenarios Utilizing Devel-oped Technologies

In Japan, an bridge administrator orders a collectiveinspection of bridges in their jurisdictional area from abridge inspection company every year. After accepting

Fig. 12. The flow of semiautomatic 3D CAD model generation.

Fig. 13. Work flow of bridge inspection.

the order, the inspection company performs bridge inspec-tion by following the work flow shown in Fig. 13. We de-scribe the main steps of the inspection work flow below:

• Preliminary survey: The bridge inspector visits all


Hada, Y. et al.

bridges to survey them from a far distance. The in-spector spends about 2 or 3 hours per bridge on thesurvey.

• Planning and scheduling: The inspector plans the in-spection method of each bridge based on the infor-mation gathered in the preliminary survey. Suspi-cious bridges having critical deterioration are sched-uled to be inspected on a priority basis based on theresults of the preliminary survey.

• Bridge inspection: The bridge inspector approachesthe inspection points using a special bridge inspec-tion vehicle or by rope access, and performs a thor-ough visual inspection. The time needed to inspectone bridge is dependent on its size. The inspectorspends half a day on a small-scale bridge, and a fewdays on a large-scale bridge.

• Report: The inspection company submits the inspec-tion reports to the administrator.

In this paper, we propose three bridge inspection sce-narios to test the two-wheeled multicopter.

The first scenario is to confirm the state of shoes onhigh piers using the large-sized two-wheeled multicopterin the preliminary survey. Since deterioration of a shoeis likely to occur, the bridge inspector wants to investi-gate the status of it as soon as possible via a preliminarysurvey. However, the inspector has not been able to as-sess shoes on high pier bridges in preliminary surveys upto now because there have been no methods to do that.If the inspector can find serious damage to shoes on thehigh pier bridge using the large-sized two-wheeled multi-copter in a short time in the preliminary survay, it wouldbe a great advantage as the bridge administrator can takemeasurements as soon as posible.

The second scenario is a visual inspection of the en-tier concrete surface of a high pier bridge over 30 m inheight using the large-sized two-wheeled multicopter inthe bridge inspection phase. Bridge inspectors have con-ducted the visual inspection of high piers by rope accessuntil now because they have not been able to use spe-cial bridge inspection vehicles to perform visual inspec-tion of high piers. Rope access is labor-intensive, time-consuming and high-cost work, and only a small num-ber of bridge inspectors are able to conduct inspection byrope access. If bridge inspectors can perform a visual in-spection of an entire concrete surface of high piers usingthe large-sized two-wheeled multicopter, it is possible thatdeterioration points of the high pier can be picked up in ashorter time than by using rope access. The multicoptercan also capture images of damage to the high pier withgeotags that are added to the images automatically usingthe geotagging technique.

The third scenario is to confirm shoes using the small-sized two-wheeled multicopter in the preliminary survey.Until now, bridge inspectors have not been able to checkshoes on bridges at least 10 m in height from all anglesin the preliminary survey. In this scenario, the bridge in-spector utilizes the small-sized two-wheeled multicopter

Fig. 14. Experimental setup.

to enter the gap between the pier and the bridge girder tocheck shoes from various angles in the preliminary sur-vey.

5.2. Substantiative Experiments at Bridge SitesIn order to confirm the effectiveness of the proposed

inspection scenarios using two-wheeled multicopters, weconducted substantiative experiments at two bridge sites:one is a high pier bridge about 40 m in height and theother is a box-girder bridge whose height is about 10 m.

5.2.1. Confirmation of State of Shoe on a High Pier Us-ing Large-Sized Two-Wheeled Multicopter inPreliminary Survey

We conducted an experiment assuming that the bridgeinspector confirms the state of shoes at the top of the highpier at a height of 40 m using the large-sized two-wheeledmulticopter in the preliminary survey. The experimentalsetup is shown in Fig. 14. We used the action cameraFDR-X3000 manufactured by Sony. This camera has anangle of view of about 100 degrees in the horizontal di-rection and it can capture a wide range of images aroundthe shoe. Furthermore, since the camera can output a 2Kvideo stream from an HDMI port, we can connect thecamera to the mobile display through the optical fiber ca-ble with HDMI optical transmission converters so that thebridge inspector on the ground can see the captured im-ages in real time. In the experiment, the multicopter startsfrom the ground and moves toward the shoe while in con-tact with the pier. When the multicopter arrives at theshoe, the inspector confirms the state of shoe by lookingat the real-time images captured by the on-board camera.

Figure 15(a) shows the details of the experiment andFig. 15(b) is a shoe image captured by the on-board cam-era. Although the wind was blowing at speeds of about



(a)

(b)

Fig. 15. Rresult of experiment to confirm state of shoe.

4 meters per second from the left side during the experi-ment, the two-wheeled multicopter could reach the shoeto capture images. In this experiment, since a worker onthe bridge adjusted the length of the cable in accordancewith the altitude of the multicopter by hand, the multi-copter could move at a speed of less than about 0.5 metersper second. Therefore, the travel time for the round tripwas about 160 seconds. At the shoe, the multicopter wasmoved laterally to capture images from different points.Therefore, assessment of the shoe took about 300 sec-onds in total. In the current experimental setting, we spentabout 30 minutes on preparation, and 5 minutes to assessthe shoe. It took about 20 minutes to clean up the setup.So we needed about 1 hour to assess the high pier’s shoein the preliminary survey. In future work, we should im-prove the operability of the two-wheeled multicopter toshorten the total time needed for the preliminary survey.

5.2.2. Visual Inspection of Entire Concrete Surfaceof High Pier Using Large-Sized Two-WheeledMulticopter

We conducted an experiment assuming that the bridgeinspector confirms the presence or absence of new deteri-oration of the entire concrete surface of the bridge pier ata height of 40 m using the large-sized two-wheeled multi-copter in the bridge inspection. The experimental setup is

(a)

(b)

(c)

Fig. 16. Result of experiment to find cracks on concrete surface.

as mentioned above. In this experiment, the multicoptermoves toward the top of the pier starting from the ground.After reaching the top, the multicopter moves laterallyand then lowers to the ground. In this experiment, themulticopter moves at the maximum speed of 0.2 metersper second so that the bridge inspector, who is looking atthe bridge surface image captured by the on-board cam-era, can determine whether there is damage on the con-crete surface of the pier.

Figure 16(a) shows the swept trajectory of the largetwo-wheeled multicopter. If the bridge inspector findsbridge deterioration, he or she instructs the multicopter’s


Hada, Y. et al.

operator to maintain the multicopter in a hovering stateto capture photos of the deterioration. Deterioration im-ages taken in this manner are shown in Figs. 16(b) and(c). A 0.2-mm-wide crack taken at location A is depictedin Fig. 16(b), and the bridge inspector can visually iden-tify it on his or her mobile display. In this experiment, wecould determine crack width using a chalk line remainingfrom a past inspection. Another crack image taken at lo-cation B is shown in Fig. 16(c). Locations A and B wereat heights of 30 m and 15 m, respectively.

In this experiment, the camera’s range of width is about1 m. The height of the pier is 40 m and the multicoptersweeps the 40 square meters within about 200 seconds.The pier, whose front width is 6 m and side width is 4 m,has a total surface area of 800 square meters. Therefore,it takes about 70 minutes for the multicopter to sweep theentire surface of the pier. It is expected that the inspectiontime will actually be over two hours because the multi-copter hovers at every deterioration point for a while tocapture images. If the bridge inspector conducts a visualinspection of the pier by rope access, we expect that itwill take over 7 hours per pier. Thus, the multicopter isexpected to perform a visual inspection of the high pier inabout 30% of the time taken using rope access.

5.2.3. State Confirmation of Shoe at a Box-GirderBridge in Preliminary Survey

We conducted an experiment assuming that the bridgeinspector confirms the state of shoes using the small-sizedtwo-wheeled multicopter in the preliminary survey. Theexperiment is performed at a box-girder bridge whoseheight is about 10 m. The small multicopter starts fromthe ground and flies toward the target shoe. The shoe islocated between the pier and the box girder and its heightis about 50 cm. We tried to capture a side view image ofthe shoe using the small multicopter.

The experimental result is shown in Fig. 17(a). Thesmall multicopter could enter the narrow space betweenthe pier and the box girder and turned so as to face theshoe direction. Fig. 17(b) is a side view image capturedin this experiment. In this experiment, we used a Go-Pro HERO4 Session as the on-board camera. The cam-era transmits the captured image to the bridge inspector’ssmart phone via Wi-Fi and he or she can check the imagesin real time.

In this experiment, the operator maneuvers the smallmulticopter from the ground while directly viewing it.When the multicopter enters the narrow space, the opera-tor is unable to verify the state of the multicopter. To solvethis problem, we have to improve the remote operationmethod to enable the operator to control the multicopterin situations where the multicopter is not directly visible.

6. Related Work

Many countries have faced the serious problem of ag-ing civil infrastructures, and many technologies have been

(a)

(b)

Fig. 17. Result of experiment to confirm shoe using smallsized two-wheeled multicopter.

developed to try to solve the problem. In this section,we introduce information technologies and robotics tech-nologies for aging infrastructure and discuss the similar-ities and differences between them and our technologiesdeveloped in the SIP project.

The Aerial Robotic Infrastructure Analyst (ARIA)project [4] at Carnegie Mellon University studies methodsof modeling and analyzing civil infrastructure, especiallybridges, using UAVs. In this project, the aerial vehiclegathers 3D point cloud data of the target bridge. The pointcloud data is transformed into a semantic, component-based mode. The semantic model is converted into afinite element model, and structural assessment is con-ducted by simulating the resulting model. Semantic En-richment Engine for Bridges (SeeBridge) [8, 9] is an in-ternational project in action since 2015. Seven organiza-tions in the US, UK, Germany and Israel are participat-ing in the project. In SeeBridge, various advanced re-mote sensing technologies are used to rapidly and accu-rately capture the state of a bridge from point cloud datato build an as-is model. The concept of managing the in-spection data on the IFC-Bridge extension data model issame as in our approach. These three projects are com-



mon in terms of their development of an inspection datamanagement technique utilizing a 3D model, but differ-ent in terms of the data model. Hereafter the data modelfor managing inspection data will be an international stan-dardization. Another difference between our project andthe aforementioned projects is that we developed originalbridge inspection robots to be suitable for capturing close-up images. In the ARIA and SeeBridge projects, an orig-inal hardware mechanism for the UAV was not developedand conventional UAVs were used instead. Additionally,our project proposes both bridge inspection robots and abridge inspection data management system. Focusing onbridge inspection robots, there have been many studiesand much development so far. Tohoku University has per-formed studies on a quadcopter having a spherical shellfor bridge inspection in the SIP project [10]. We con-sider that the two-wheeled multicopter is superior to thespherical shell multicopter in terms of movement stabil-ity on a plane, because the spherical shell multicopter of-ten bounces on the plane. On the other hand, the spheri-cal shell multicopter demonstrates an excellent movementperformance in a complex environment, such as within abridge girder composed of steel beams. PRODRONE Ltd.developed a unique L-shaped UAV with wheels [11]. Thisbridge inspection UAV has a similar concept to the two-wheeled multicopter because the UAV contacts the sur-face of the bridge using its wheels. On the other hand, theweight balance of the UAV is not effective because of itsshape, and the UAV is also difficult to fly in air, especiallyunder windy conditions. Bridge inspection robots using amulticopter were proposed in [12–14]. In many studies,the UAVs took images of bridges from a distance. How-ever, our two-wheeled multicopter can capture images ofa bridge at close range.

The authors do not consider it possible for a singlerobot type to cover bridge inspections of all types. In thefuture, various types of bridge inspection robots will bedeveloped.

7. Conclusion

In this paper, we provided an overview of the bridge in-spection support system that Fujitsu Limited, HokkaidoUniversity, The University of Tokyo, Nagoya Instituteof Technology and Docon Co. Limited developed inthe SIP project from 2014 to 2016. Details of the two-wheeled multicopter and inspection scenarios using themulticopter were discussed, and the experimental resultsshow the effectiveness of our approach. The research out-come introduced in this paper is in the middle of develop-ment and the technologies are not completed yet. Here-after we will put our developing technologies into practi-cal use.

AcknowledgementsThis work was supported by the Council for Science, Technol-ogy and Innovation, “Cross-ministerial Strategic Innovation Pro-

motion Program (SIP), Infrastructure Maintenance, Renovation,and Management.” (Funding agency: NEDO)

References:[1] Y. Fujino, “The Cross-ministerial Strategic Innovation Pro-

motion Program, Infrastructure Maintenance, Renovation,and Management,” Cabinet office, Government of Japan,http://www8.cao.go.jp/cstp/panhu/sip english/34-37.pdf [accessedMay 8, 2017]

[2] M. Nakao, Y. Hada et al., “Development of a bridge inspection sup-port robot system that uses a two-wheeled quiad-rotor helicopter,”Proc. of The Fourteenth East Asia-Pacific Conf. on Structural Engi-neering and Construction (EASEC-14), S3.460, 2016.

[3] N. Takahashi, M. Yamada et al., “All-round two-wheeled quadrotorhelicopters with protect-frames for air-land-sea vehicle,” AdvancedRobotics, Vol.29, No.1, pp. 69-87, 2015.

[4] “The Aerial Robotic Infrastructure Analyst (ARIA) Project,”Carnegie Mellon University, http://aria.ri.cmu.edu/ [accessed May8, 2017]

[5] S. Pathak, A. Moro, A. Yamashita and H. Asama, “3D Recon-struction of Structures using Spherical Camera with Small Motion,”Proc. of 16th Int. Conf. on Control, Automation and Systems (IC-CAS 2016), 2016.

[6] F. Tanaka, M. Hori, M. Onosato, H. Date, and S. Kanai, “BridgeInformation Model Based on IFC Standards and Web Content Pro-viding System for Supporting an Inspection Process,” Proc. of 16thInt. Conf. on Computing in Civil and Building Engineering (ICC-CBE 2016), 2016.

[7] H. Date, T. Yokoyama, S. Kanai, Y. Hada, M. Nakao, and T. Sug-awara, “Automatic Registration of Laser-Scanned Point Clounds ofBridges Using Linear Features,” Proc. of the Asian Conf. on Designand Digital Engineering 2016 (ACDDE 2016), 2016.

[8] “SeeBridge – Automated Compilation of Semantically Rich BIMModels of Bridges,” An Infractructure Innovation Programme,Infravation, http://www.infravation.net/projects/SEEBRIDGE [ac-cessed May 8, 2017]

[9] R. Sacks, A. Kedar et al., “SeeBridge Information Delivery Manual(IDM) for Next Generation Bridge Inspection,” Proc. of 33rd Int.Symposium on Automation and Robotics in Costruction (ISARC2016), 2016.

[10] S. Mizutani, Y. Okada, C. Salaan, T. Ishii, K. Ohno and S. Ta-dokoro, “Proposal and experimental validation of a design strategyfor a UAV with a passive rotating spherical shell,” Proc. of 2015IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS),2015.

[11] “Self-Propelling Surface-Clinging Drone PD6-CI-L,” Pro-drone Co., Ltd., 2016, https://www.youtube.com/watch?v=FjoPsWYtfxo&feature=youtu.be [accessed May 8, 2017]

[12] A. Bulgakow and S. Emeliznov, “Inspection of Flyover Bridges Us-ing Quadrotor,” Proc. of 32th Int. Symposium on Automation andRobotics in Costruction (ISARC 2015), 2015.

[13] C. Yang, M. Wen, Y. Chen, and S. Kang, “An Optimized UnmannedAerial System for Bridge Inspection,” Proc. of 32th Int. Symposiumon Automation and Robotics in Costruction (ISARC 2015), 2015.

[14] M. Gillins, D. Gillins, and C. Parrish, “Cost-Effective Bridge SafetyInspection using Unmanned Aircraft Systems (UAV),” Proc. ofGeotechnical and Structural Engineering Congress 2016, 2016.

[15] Ministry of Land, Infrastructure, Transport and Tourism ofJapan, “The standard manual of unmanned aerial vehicleflight” (in Japanese), Aug. 2016, https://www.mlit.go.jp/common/001140548.pdf [accessed May 8, 2017]


Hada, Y. et al.

Name:Yoshiro Hada

Affiliation:Safety Solution Business Unit, Fujitsu Limited

Address:10-1 Morinosato-Wakamiya, Atsugi, Kanagawa 243-0197, JapanBrief Career:1999- Assistant Professor, Univ. of Electro-Communications2006- Joined Fujitsu Laboratories Limited2014- Safety Solution Business Unit, Fujitsu Limited.Selected Publications:• “Development of Delivery Robot System Utilizing StructuredEnvironmental Information,” J. of the robotics society of Japan, Vol.27,No.10, p. 1105, 2009.Academic Societies & Scientific Organizations:• Robotics Society of Japan (RSJ)• Information Processing Society of Japan (IPSJ)• Society of Instrument and Control Engineers (SICE)

Name:Manabu Nakao


Address:4-1-1, Kamikodanaka, Nakahara-Ku, Kawasaki-Shi, Kanagawa 211-8588,JapanBrief Career:1997- Fujitsu Limited2003- Fujitsu Laboratories Limited2015- Safety Solution Business Unit, Fujitsu LimitedSelected Publications:• “Development of a bridge inspection support robot system that uses atwo-wheeled quad-rotor helicopter,” Proc. of The 14th East Asia-PacificConf. on Structural Engineering and Construction, S3.460, 2016.Academic Societies & Scientific Organizations:• Robotics Society of Japan (RSJ)

Name:Moyuru Yamada


Address:10-1 Morinosato-Wakamiya, Atsugi, Kanagawa 243-0197, JapanBrief Career:2009- Ph.D. in Engineering, Toyohashi University of Technology2012- Fujitsu Laboratories Limited2014- Safety Solution Business Unit, Fujitsu LimitedSelected Publications:• “Immediately-available Input Method Using One-handed Motion inArbitrary Postures,” Procedia Computer Science, Vol.39, pp. 51-58, 2014.Academic Societies & Scientific Organizations:• Robotics Society of Japan (RSJ)

Name:Hiroki Kobayashi


Address:4-1-1 Kamikodanaka, Nakahara-Ku, Kawasaki-Shi, Kanagawa 211-8588,JapanBrief Career:1993- Fujitsu Laboratories Limited2015- Safety Solution Business Unit, Fujitsu LimitedSelected Publications:• “Noise Prediction System for Low Acoustic-Noise Server Design,”ASME IDETC/CIE2011, 2011Academic Societies & Scientific Organizations:• Japan Society of Mechanical Engineers (JSME)• Japan Society for Simulation Technology (JSST)

Name:Naoyuki Sawasaki


Address:10-1 Morinosato-Wakamiya, Atsugi, Kanagawa 243-0197, JapanBrief Career:1992- Fujitsu Laboratories Limited2014- Safety Solution Business Unit, Fujitsu LimitedSelected Publications:• “Development of Personal Robot,” The 13th Int. Symposium onRobotics Research, pp. 363-374, 2007.Academic Societies & Scientific Organizations:• Robotics Society of Japan (RSJ)



Name:Katsunori Yokoji

Affiliation:Manager, Safety Solution Business Unit, FujitsuLimited

Address:Fujitsu Kosugi Building, 1812-10 Shimonumabe, Nakahara-ku, Kawasaki,Kanagawa 211-8588, JapanBrief Career:1998- Joined Fujitsu Limited2011-2012 Worked for National Institute for Land and InfrastructureManagement2013- Safety Solution Business Unit, Fujitsu Limited2017- Manager, Safety Solution Business Unit, Fujitsu Limited

Name:Satoshi Kanai

Affiliation:Professor, Division of Systems Science and In-formatics, Graduate School of Information Sci-ence and Technology, Hokkaido University

Address:Kita-14, Nishi-9, Kita-ku, Sapporo 060-0814, JapanBrief Career:1987-1989 Research Associate, Hokkaido University1989-1996 Associate Professor, Tokyo Institute of TechnologySelected Publications:• “Motion-capture-based walking simulation of digital human adapted tolaser-scanned 3D as-is environments for accessibility evaluation,” J. ofComputational Design and Engineering, Vol.3, No.3, pp. 250-265, 2016.• “Optimum laser scan planning for as-built 3D modeling of HVAC systemwith difficult-to measure regions,” 3rd Int. Conf. on Civil and BuildingEngineering Informatics, 2017.Academic Societies & Scientific Organizations:• Institute of Electrical and Electronics Engineers (IEEE)• Japan Society for Precision Engineering (JSPE)

Name:Fumiki Tanaka

Affiliation:Associate Professor, Graduate School of Infor-mation Science and Technology, Hokkaido Uni-versity

Address:Kita-14, Nishi-9, Kita-ku, Sapporo 060-0814, JapanBrief Career:1988-2004 Assistant Professor, Hokkaido University2003 Received Ph.D. degree from Hokkaido University2004- Associate Professor ,Hokkaido UniversitySelected Publications:• F. Tanaka et. al., “Bridge Information Model Based on IFC Standardsand Web Content Providing System for Supporting an Inspection Process,”Proc. of 16th Int. Conf. on Computing in Civil and Building Engineering,pp. 1140-1147, 2016.Academic Societies & Scientific Organizations:• Japan Society for Precision Engineering (JSPE)• Japan Society of Mechanical Engineers (JSME)• American Society of Mechanical Engineers (ASME)

Name:Hiroaki Date

Affiliation:Associate Professor, Graduate School of Infor-mation Science and Technology, Hokkaido Uni-versity

Address:Kita-14, Nishi-9, Kita-ku, Sapporo 060-0814, JapanBrief Career:2005- Assistant Professor, Graduate School of Information Science andTechnology, Hokkaido University2010- Associate Professor, Graduate School of Information Science andTechnology, Hokkaido UniversitySelected Publications:• “Automatic registration of MLS point clouds and SfM meshes of urbanarea,” Geo-spatial Information Science, Vol.19, No.3. pp. 171-181, Oct.,2016.Academic Societies & Scientific Organizations:• Japan Society of Precision Engineering (JSPE)• Information Processing Society of Japan (IPSJ)


Hada, Y. et al.

Name:Sarthak Pathak

Affiliation:Doctoral Student, Department of Precision Engi-neering, The University of Tokyo

Address:7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, JapanBrief Career:2014 Received the Bachelor of Technology (BTech) and Master ofTechnology (MTech) degrees in Engineering Design with a specializationin Automotive Engineering from the Indian Institute of Technology,MadrasSelected Publications:• S. Pathak, A. Moro, A. Yamashita, and H. Asama, “A Decoupled VirtualCamera Using Spherical Optical Flow,” Proc. of the 2016 IEEE Int. Conf.on Image Processing (ICIP2016), pp. 4488-4492, Phoenix (USA), Sep.2016.Academic Societies & Scientific Organizations:• IEEE Robotics and Automation Society, Student Member

Name:Atsushi Yamashita

Affiliation:Associate Professor, Department of PrecisionEngineering, The University of Tokyo

Address:7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, JapanBrief Career:2001 Received Ph.D. degree from the University of Tokyo2001-2008 Assistant Professor, Department of Mechanical Engineering,Shizuoka University2006-2007 Visiting Associate, California Institute of Technology2008-2011 Associate Professor, Department of Mechanical Engineering,Shizuoka University2011- Associate Professor, Department of Precision Engineering, TheUniversity of TokyoSelected Publications:• A. Yamashita, T. Arai, J. Ota, and H. Asama, “Motion Planning ofMultiple Mobile Robots for Cooperative Manipulation andTransportation,” IEEE Trans. on Robotics and Automation, Vol.19, No.2,pp. 223-237, 2003.Academic Societies & Scientific Organizations:• IEEE, ACM, RSJ, JSME, JSPE, IEICE, IEEJ, IPSJ, ITE, SICE, Societyfor Serviceology

Name:Manabu Yamada

Affiliation:Professor, Department of Electrical and Mechan-ical Engineering, Nagoya Institute of Technol-ogy

Address:Gokiso-cho, Showa, Nagoya, Aichi 466-8555, JapanBrief Career:1992 Received Ph.D. degree from Nagoya Institute of Technology1992- Assistant Professor, Department of Mechanical Engineering,Nagoya Institute of Technology2011- Professor, Department of Mechanical Engineering, Nagoya Instituteof Technology2016- Professor, Department of Electrical and Mechanical Engineering,Nagoya Institute of TechnologySelected Publications:• “All-Round Two-Wheeled Quadrotor Helicopters with Protect-Framesfor Air-Land-Sea Vehicle – Controller Design and Automatic ChargingDevice –,” Advanced Robotics, Vol.29, No.1, pp. 69-87, 2015.Academic Societies & Scientific Organizations:• Institute of Electrical and Electronics Engineers (IEEE)• Japan Society of Mechanical Engineers (JSME)• Society of Instrument and Control Engineers (SICE)

Name:Toshiya Sugawara

Affiliation:Senior Engineering Manager, Structure Depart-ment, Docon Co. Limited

Address:5-4-1 Atsubetsucho1, Atsubetsu-ku, Sapporo 004-8585, JapanBrief Career:1985- Joined Sakurada Co. Limited1991- Joined Docon Co. Limited2001- Registration of a Professional EngineerSelected Publications:• “Identification of Dynamic Behavior of Hakucho-ohashi by AmbientVibration Measurement,” J. of Structural Engineering, Vol.47A, Mar.,2001.Academic Societies & Scientific Organizations:• Japan Society of Civil Engineers (JSCE)


Powered by TCPDF (www.tcpdf.org)

http://www.tcpdf.org

Documents

Development of a Bridge Inspection Support System Using