Embed Size (px)
Citation preview
Results of benchmarking traffic services provided
by TomTom and Garmin
22.01.2015 © NavCert 1
Content
• Executive Summary
• Who is NavCert
• Used Methodology
• Data Analysis
• Conclusions
• Annex
22.01.2015 © NavCert 2
Content
• Executive Summary
• Who is NavCert
• Used Methodology
• Data Analysis
• Conclusions
• Annex
22.01.2015 © NavCert 3
Executive Summary
On behalf of TomTom a benchmark of the traffic services offered by Garmin and TomTom was
conducted in and around the region Berlin. The following Personal Navigation Devices (PNDs) were
used:
• Garmin 2699 LMT-D with Garmin Live Traffic via Smartphone Link on a Bluetooth connected
Samsung S5 using T-Mobile
• Garmin 3598 LMT-D with Garmin HD Digital Traffic® based on DAB+
• TomTom Go 500 with TomTom Traffic via a Bluetooth connected Samsung S5 using T-Mobile
The objective was the independent evaluation of the performance of the traffic services by Garmin and
TomTom for typical commuter routes in Berlin. The tests have been conducted during the typical
commuter times with respective traffic jams:
• Morning session: 6:30 to 9:30
• Afternoon session: 16:00 to 19:00
To provide similar conditions the DUTs (Devices under Test) were installed in identical or comparable
vehicles.
Trips with the same origin – destination pairs (“OD pairs”) and the same start times were driven with
each DUT in parallel as shown in Annex 1. Each device was used in average 107 trips to get statistical
significant data.
22.01.2015 © NavCert 4
Executive Summary ctd.
22.01.2015 © NavCert 5
TomTom
GO 500
Garmin
2699
Garmin
3598
Minimum 0.0% 0.0% 0.0%
Maximum 37.5 % 42.5% 55.4%
Median 0.0% 7.1% 8.1%
Mean 3.0% 9.5% 11.0%
Std. Deviation of
Sample 6.0 % 10.9% 12.5%
Std. Deviation of
Mean 0.6% 1.1% 1.3%
Lower 95% CI of
mean perc. delay 1.8% 7.4% 8.4%
Upper 95% CI of
mean perc. delay 4.2% 11.6% 13.4%
The below table provides the results for the measured delay per trip per DUT. The provided values
in table1 refer to the percentage of delay in relation to shortest duration for that trip.
TomTom outperforms Garmin: The average delay of TomTom
with 3.0% is significantly better than the mean of the two
Garmin devices with 7.1% and 8.1% as visualized in figure 1.
Table 1
Figure 1
0%
2%
4%
6%
8%
10%
12%
14%
16%
TomTom GO 500 Garmin 2699 Garmin 3598
Mean
Confidence Interval
Content
• Executive Summary
• Who is NavCert
• Used Methodology
• Data Analysis
• Conclusions
• Annex
22.01.2015 © NavCert 6
Who is NavCert?
NavCert is a technical service provider
offering testing, verification, validation and certification in the fields of positioning,
navigation and communication worldwide. The enthusiastic NavCert team offers a
dependable and high quality service behind a strong logo. Its services are complemented
with detailed knowledge of navigation and localization applications and experience in the
analysis of both commodity and safety critical systems and services. NavCert provides
certification for a variety of GNSS based products, systems and services.
NavCert is an accredited laboratory by the German national accreditation body DAkkS in
the scope of GNSS, eCall and EETS.
NavCert has agreed to cooperate with TÜV SÜD product services, which allows NavCert
to market certifications in an area not regulated by law and to mark them with the TÜV
SÜD quality logo.
22.01.2015 © NavCert 7
Signals conform to
Galileo specification
Functionality
checkedwww.navcert.de/
PPP80020
Test laboratory conform
to ISO 17025
Receiver integrity
testing (RAIM) for SoL
applications
Content
• Executive Summary
• Who is NavCert
• Used Methodology
• Data Analysis
• Conclusions
• Annex
22.01.2015 © NavCert 8
Methodology
The test methodology is divided in three parts:
• Preparation
– Devices and supporting equipment
– Definition of start and end points per trip
– Test team requirements
• Execution
– Per day
– Per trip
– Data logged
• Results analysis & documentation
– Preparation of data
– Processing of data
– results
22.01.2015 © NavCert 9
Preparation: Devices and supporting equipment
The following devices were used to validate the performance of the traffic service
provider:
• Garmin 2699 LMT-D with Garmin Live Traffic connected via Bluetooth to a
Samsung S5 using T-Mobile
• Garmin 3598 LMT-D with Garmin HD Digital Traffic® based on DAB+
• TomTom Go 500 with TomTom Traffic connected via Bluetooth to a Samsung S5
using T-Mobile
As mobile service provider T-Mobile was selected with a data only service for the
Samsung S5.
The PNDs were bought a few days prior to the execution of the tests and updated to the
latest map and software one working day prior to the execution of the tests. The
smartphones were updated to the latest version both for the operating system Android
and the used apps (Garmin Smartphone Link, MyDrive mobile app). PNDs were set to
“fastest route” and “automatically reroute,” or equivalent.
To ensure equal test conditions for all traffic service providers, passenger cars with
similar dimensions and engine specifications were used as detailed in Annex 2.
22.01.2015 © NavCert 10
Preparation: Definition of trips
The total selection of addresses was representative for the geographical area covered by
Berlin. In order to represent typical commutes, origin/destination pairs representing
centre-to-periphery, periphery-to-centre, within-centre, and within-periphery routes were
identified.
The definition of trips was planned to cover typical commutes (including delay times), with
various times ranging from 30 minutes to 1 hour, with the average trip times being around
40 minutes. During the planning an average delay of 7.5 minutes was assumed.
In order to allow a combined start of all vehicles for the next trip, gathering points were
defined as new starting points. Three trips were grouped into one session and in total 34
sessions were defined. The average, minimum and maximum length of the trips in km
together with the same values for the duration in minutes are depicted below:
22.01.2015 © NavCert 11
length
km
time
minutes
average 32 48
minimum 11 43
maximum 100 53
The distribution of the start and end points is provided in Annex 1.
Preparation: Test team
The test team consisted of a project manager, team leaders and drivers with following
profiles:
• project manager: senior expert and technical certifier in the scope of GNSS
• team leader: technical expert experienced in laboratory work and team leading
• Driver: driving license > 3 years, driving regularly and using PNDs
A half day training session consisted of one theoretical part, the introduction to the
benchmark drive test with the defined processes and a hands-on session, in which all
drivers got used to interact with the DUTs. A hand out was provided describing all
processes and the driver interaction with the DUTs.
A trip logging template was prepared, to log all relevant data during a trip and descriptions
of unexpected events (e.g. engine failure)
After installation into the vehicles, the team leader checked the proper installation and
followed the procedure defined for the start point. The team leader joined the drivers of his
team for the first trip. He validated proper driving and monitored the processes on arrival,
proper documenting and departing from the destinations of the first session.
22.01.2015 © NavCert 12
Execution: Per day
Trips were executed during typically commuter times from Monday till Friday in the
morning from 06.30 to 09.30 and in the afternoon from 16.00 to 19.00 hours. In every
session three trips were driven. All vehicles were parked in the same location, named the
base, from where they started to their first team specific starting point. Drivers received
instructions which sessions they had to drive on that day. Within a team the driver rotated
the used DUTs from one to the other before each session. If necessary additional trip logs
were provided. All vehicles were equipped with a video camera logging date, time and GPS
data.
The team leader validated the proper installation and configuration of all DUTs and required
supporting equipment and provided a new SD card for the video camera. Smartphones and
tethered devices were mounted in close proximity to each other.
In a team leader log the number plate of the vehicle, driver name and used DUT were
documented together with the inventory id of the SD card and the session id. Then the
team leader started his team to go to the first starting point.
All teams drove back after each session to the same parking space (the base). The team
leader interviewed the driver, collected the SD-card of the video camera, the trip logs, and
checked the proper storage of the equipment.
22.01.2015 © NavCert 13
Execution: Per trip
Before the start of each trip, every driver verified where possible that traffic information
services are available. Route planning was finished on each device, and all cars were ready
to go facing the same direction before departure. Cars within a team departed in as short
as feasible intervals from each other. The duration from the departure of the first to the last
car was as short as possible and the drivers strived to keep the duration less than two
minutes. The route was followed without deviation; in case automatic rerouting was not
available, any reroute prompts from the device was accepted and followed. Drivers obeyed
the driving regulations at all times, never exceed speed limits, or break traffic regulations /
take forbidden turns etc. Drivers matched the traffic flow and they didn’t take any kind of
priority lanes like bus or taxi lanes.
22.01.2015 © NavCert 14
Execution: data logged
The following data was recorded per trip, using the trip log (on paper):
• Trip number
• Start and arrival times for each DUT, marked at the exact times in HH:MM format
• Estimated time of arrival (ETA, e.g. 17:45 with a start time of 17:00) at the start of the
trip for each DUT
• Estimated kilometres and actual kilometres driven
• Periods during which a device loses GPS reception, from start to end time, in
HH:MM:SS, however not in locations where loss of GPS connectivity is expected like in
tunnels
• Periods during which a device loses network connectivity, from start to end time, in
HH:MM:SS
• Periods during which a device loses connectivity to the traffic service, from start to end
time, in HH:MM:SS
In parallel a video camera documented the behaviour of the DUT complemented by GPS
location and date and time.
All times documented do not include seconds to avoid wrong impression of high precision
which cannot be measured. This does not influence overall results as this is the same and
consistent methodology applied to all devices
22.01.2015 © NavCert 15
Content
• Executive Summary
• Who is NavCert
• Used Methodology
• Data Analysis
• Conclusions
• Annex
22.01.2015 © NavCert 16
Result analysis: preparation of data
In the first step an elimination of invalid trips took place. A trip was regarded as invalid if
due to system extrinsic factors a deviation from the route given by the DUT took place or an
external event influenced the driving time:
• Driver error
• Devices crashed or stopped working
• Devices loose Bluetooth tethering, necessary for traffic information
• Map error (guidance in conflict with applicable laws)
• Car break down
• Accident
• Police control
• etc.
However if a route was temporarily blocked, the respective trip was not excluded from
processing as this information was in part communicated by the traffic service providers.
22.01.2015 © NavCert 17
Result analysis: delay per trip
The delay per trip is determined as delay in respect to the minimum duration for this trip.
𝑲𝑷𝑰𝟏𝒂 𝑫𝑼𝑻𝒊 ∶= duration of trip DUT𝒊 − duration of trip DUT𝒔𝒉𝒐𝒓𝒕𝒆𝒔𝒕
In order to reflect the duration of the single trips in relation to the respective delay, the
percentage of the delay is evaluated.
𝑲𝑷𝑰𝟏𝒃 𝑫𝑼𝑻𝒊 ∶=duration of trip DUT𝒊 − duration of trip DUT𝒔𝒉𝒐𝒓𝒕𝒆𝒔𝒕
duration of trip DUT𝒔𝒉𝒐𝒓𝒕𝒆𝒔𝒕
22.01.2015 © NavCert 18
Result analysis: processing of data
As a first step in the analysis the outlier detection according to GRUBBS with alpha of 0.01
was performed. Then the statistic values were calculated. In the end the statistical
significance was determined with the Anova in the first step for all DUTs. Then the student
test was used to compare TomTom against the other two DUTs.
Below table depicts the performance of the DUTs with respect to the selection of fastest
route based on KPI 1b, the percentage of delay in relation to shortest duration for that trip.
22.01.2015 © NavCert 19
TomTom GO 500 Garmin 2699 Garmin 3598
Minimum 0.0% 0.0% 0.0%
Maximum 37.5 % 42.5% 55.4%
Median 0.0% 7.1% 8.1%
Mean 3.0% 9.5% 11.0%
Std. Deviation of Sample 6.0 % 10.9% 12.5%
Std. Deviation of Mean 0.6% 1.1% 1.3%
Lower 95% CI of mean 1.8% 7.4% 8.4%
Upper 95% CI of mean 4.2% 11.6% 13.4%
Result analysis: statistical evaluation
Trip data was analyzed in an unpaired t test comparing TomTom against the two Garmin
devices. Then a paired comparison between TomTom and one DUT was done. Here only
those trips were analyzed which both TomTom and the other DUT provided evaluable
results. The results are depicted below, where the first three lines are the results from the
unpaired and the lower 6 representing the paired tests. The P value doesn’t differ
between the two approaches. Therefore the significance in the performance between
TomTom and the two Garmin devices is confirmed.
22.01.2015 © NavCert 20
TomTom -
Garmin 2699
TomTom - Garmin 3598
Unpaired
evaluation of all
results
Dif. btw. means -9.1% ± 1.4% -9.2% ± 1.5%
95% conf. Int. -9.4% to -8.8% -9.5% to -8.9%
P value < 0.0001 < 0.0001
Paired evaluation
only for paired
evaluable for the
paired results
P value < 0.0001 < 0.0001
Number of pairs 93 84
Mean of dif. -7.4% -8.3%
SD of dif. 14% 16%
SEM of dif. 1.4% 1.7%
95% conf. int. -10% to -4.5% -12% to -4.9%
Content
• Executive Summary
• Who is NavCert
• Used Methodology
• Data Analysis
• Conclusions
• Annex
22.01.2015 © NavCert 21
Conclusions
An independent benchmark of the performance of the traffic services provided by Garmin
and TomTom was done in Berlin for typical commuter trips in beginning of December 2014.
The test was set up to eliminate all external influences from vehicle, driver, etc.
The delay per trip has been measured and statistically evaluated.
The mean value of the percentaged delay is the smallest with 3% for the TomTom GO 500,
compared to the 6.1% and 7.1% for the two Garmin devices.
TomTom GO 500 outperforms the Garmin 2699 and the Garmin 3598 significantly with
respect to the percentaged delay per trip.
22.01.2015 © NavCert 22
Content
• Executive Summary
• Who is NavCert
• Used Methodology
• Data Analysis
• Conclusions
• Annex
22.01.2015 © NavCert 23
Annex 1: Start / End points of trips
Annex II: Used Vehicles
22.01.2015 © NavCert 25
Vehicle type Quantity of
vehicles
Number of trips
per vehicle
VW Touran
105hp, diesel, manual
gearbox
3 54
VW Golf Sportsvan
125hp, petrol, manual
gearbox
12 15
A6, Mercedes E class,
BMW 5
Automatic gearbox
3 3
NavCert GmbH
● NavCert GmbH
Hermann-Blenk-Straße 22
D-38108 Braunschweig
Germany
● Phone: +49 531 35479-0
Fax: +49 531 35479-491
Mobil: 0151 11315420
www.navcert.de
22.01.2015 © NavCert 26
Registered
Amtsgericht Braunschweig
HRB 200 364
VAT ID: DE 249 45 61 89
● Managing Director
Martin Grzebellus
