Cross-border Interoperability TESTFEST · KPI IVS & Cellular IVS & Cellular Use-case count Dynamic Speed Limit Information Southbound (Showed 50 Speed limit and Speed Limit End) Dynamic

Cross-border Interoperability TESTFEST

Initial evaluation resultsMarch 28th, 2019

Content

• Logistics

• Technical evaluation• Introduction• TESTFEST UK• TESTFEST FR• TESTFEST BE/NL:

• Results from the app logging• Results from OBU logging• Results from the teams

• Final remarks and closing

Results from the UK TESTFESTMarch 18-19, 2019

Rochester/London, United Kingdom

Overview

Sessions 18th March 19th March

Country In UK with EU partners

(FR, BE, NL)

In UK with EU partners (BE

and NL)

Location Urban (London) Motorway (A2M2) and Rural

(Kent)

Participants 8 vehicles (BeMobile x 1,

BE x 1, RWS x 2, V-

TRON x 1, HE x 1, FR x

1, TfL x1)

6 vehicles (BeMobile x 1, BE

x 1, RWS x 2, V-TRON x 1,

HE x 1)

Providers TfL HE

Protocol and

Services

Live Cellular IVS and Live

Cellular GLOSA

Hybrid IVS, Hybrid RWW,

ITS-G5 GLOSA

Architecture

18/03 London (Urban)19/03 A2M2 (Motorway) + Kent (Rural)

Architecture

Services and Uses cases

Services ProtocolUse cases in London

(Urban)

Use cases in

A2M2 (Motorway) and Kent

(Rural)

Security ITS-G5

IVS CellularDynamic Speed Limit Information

and Dynamic Lane Management

IVS HybridDynamic Speed Limit Information and

Dynamic Lane Management

RWW Hybrid Lane Closure

GLOSA CellularTime-to-green (speed advice)

Time-to-red (speed advice)

GLOSA ITS-G5Time-to-green (speed advice)

Time-to-red (speed advice)

Test statistics

Security tests: “certified/uncertified RSU sent message”

KPI A2M2 (Motorway)

Use-case count AA certificates

Participation 6

Success use-cases 4

Unsuccessful use-cases 2*

*Note: AA certificates were loaded manually by 4 participants. 2 Were unable to load

AA certificates

Test statistics

In-Vehicle Signage – London (Urban)KPI IVS & Cellular IVS & Cellular

Use-case

count

Dynamic Speed Limit Information

Southbound (Showed 50 Speed limit

and Speed Limit End)

Dynamic Lane Management Northbound

(VMS 1 & 2 showed Red “X” lane message

and VMS3 showed “Sign under Test” free

text)

Participation 8 8

Success

use-cases4* 2*

Unsuccesful

use-cases

• RWS x 2 – Unable to receive IVI

messages via cellular.

• FR x 1 – Received IVS, but not

displayed. HMI issue.

• V-TRON ?

TOTAL = 3 +?

• RWS x 2 – Unable to receive IVI

messages via cellular.

• BE x 2 (incl. BeMobile)– No messages

VMS. HMI is not configured to see text.

• FR x 1 – Received IVS, but not

displayed. HMI issue.

• V-TRON ?

TOTAL = 5 +?

*Note: 1 Southbound gantry (G2) was faulty, so HMI replicated only 2 lanes, not 3.

Test statistics

In-Vehicle Signage – A2M2 (Motorway)

KPI IVS & Hybrid IVS & Hybrid

Use-case count

Dynamic Speed Limit Information

Northbound (Display 60 speed

limit until National Speed Limit

just before exit).

Dynamic Lane Management Southbound

(Display 50 speed limit southbound to

gantry 8487A until RWW. Display of

national speed limit for final gantry section).

Participation 6 6

Success

use-cases6 6

Unsuccesful

use-casesTOTAL = 0 TOTAL = 0

Test statistics

Road Works Warnings (RWW)

KPIRWW & Hybrid –

A2M2 (Motorway)

Use-case count1 (Lane closure or other restrictions - Displayed with

50 speed limit prior to national speed limit)

Participation 6

Success use-cases 5

Unsuccesful use-cases

• BE x 1 – Lanes not inverted. OBU/HMI issue.

TOTAL = 1

*Note: BeMobile suggested RWW trace was short.

Test statistics

GLOSA – London (Urban)

KPI GLOSA & Cellular

Use-case countTime-to-green

Time to red

Participation 7 (note: FR since doesn’t handle GLOSA Service)

Success

use-cases1


• RWS x 2 – Unable to receive cellular service.

• 4 (see notes*)

TOTAL = 6

*Notes:

- GLOSA Service was inaccurate.

- Speed advice on HMI display may not have reflected the actual units km/h.

Test statistics

GLOSA – Kent (Rural)

KPI GLOSA & G5 (petrol station an main carriageway)

Use-case countTime-to-green

Time-to-red

Participation 6

Success

use-cases

3 (displayed accurate time)

V-TRON ?


• RWS x 2 – Some issues for receiving GLOSA on

main carriageway.

TOTAL = 2

*Notes:

- Speed advice on HMI display may not have reflected the actual units km/h.

- Time to green occurred only sometimes. This is likely due to petrol station access road only

triggering a phase change occasionally.

Results from FrenchTestfest

March 21-22, 2019

Reims, France

Participation

• 4 countries: FR, NL, UK, BE

• 9 institutions (UK, BE, NL, V-Tron, Renault, PSA, Université Valenciennes, UniversitéReims, SANEF, DIT)

• 14OBUs, 4 RUSs

• PKIs: Common CTL, eCTL (including AA of the four countries)

• 38 registered participants

Scenarios

• 11 events sent for each scenario on a trajectory of 23 kms

• I2V, V-road operator-2-V, V-user-2-V events

• DENM on RWW and OHLN and IVI messages

• PVD has not been run

• Communications on G5 and/or cellular networks

• 4 tours for each OBU

Coverage

Events

Senders and receivers

• For I2V communications:• For V2V communications: SANEF

• From road operators: SANEF, NeoGLS

• From user vehicles: PSA, Renault, NéoGLS

• All OBUS have displayed events at least once (some after correcting configurations on their software)

• 10 vehicles receiving events

• The first day we had a platoon of 10 cars

Some results

•TOUR 1 (10 OBUs)

•Event Result

•1 90%

•2 90%

•3 10% (IVI over G5)

•4 70%

•5 80%

•6 50%

•7 60%

•8 60%

•9 20% (IVI over cellular)

•10 NA

•11 60%

•TOUR 2 (13 OBUs)

•Event Result

•1 85%

•2 77%


•4 54%

•5 54%

•6 18%

•7 17%

•8 25%


•10 55%

•11 33%

•TOUR 3 (10 OBUs)

•Event Result

•1 70%

•2 80%


•4 60%

•5 60%

•6 63%

•7 30%

•8 70%


•10 30% (G5 coverage )

•11 70%

•TOUR 4 (8 OBUs)

•Event Result

•1 63%

•2 75%


•4 38%

•5 75%

•6 43%

•7 38%

•8 63%


•10 NA

•11 25% (G5 coverage)

Lessons learned

• Still some troubles to handle security• ITS-AID on used certificates

• SSP with wrong values (rights)

• Certificate request on the air not implemented by some partners

• Time window to consider any received message (too strict for some partners messages are dropped)

Lessons learned

• Message structure• IVI message not well implemented by somepartners

• Some use cases on IVI not implemented as multi-lane case, …

• IF2 for cellular communications• Communications between National Nodes seemto work well

Next steps

• Analysis of log files

• Analysis of pcaps

• Target Measure IF2 performances

Results from the BE/NL TESTFEST

Overview

Overview

Countries: 6 (FR, UK, BE, NL, DE, CZ)

Locations: Highways + GLOSA site

Participants/providers: 15 teams (4G/G5/IF2)

Test site (BE)

Test site (NL)

RW trailers

RW trailers

Tunnel closure

Lane closure

GLOSASecurity, RWW-DENM, IVS

HLN

vehicle

Use cases

Example of GLOSA

Example of speed advice

Overview of the participating teams

• Alps• Agentschap Wegen

en Verkeer • Rijkswaterstaat 1&2• Be-Mobile• Cohda Wireless• Dynniq NL• Dynniq BE• TNO• Transport for London

& HE• Université Reims

• UniversitéValenciennes UPHF

• V-Tron• Viveris• YoGoKo

Data analysis setup

• Analyses based on 3 main sources:

• Logging data from the app

• Logging data from the NL OBU

• Service-specific information from the teams• RWW• IVI• GLOSA• Other

• Geographical analysis, app usage, messages transmissions/receptions, …


Analysis of app logging

Logging app

App log data(time of day)

• Number of registered records (every 2 seconds):• Monday 25 / Tuesday 26 / Wednesday 27 March• 309 740 in total• 219 959 between 9h-20h (71%)

-

5 000

10 000

15 000

20 000

25 000

30 000

35 000

40 000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Nu

mb

er

of

regi

ste

red

re

cord

s

App log data (time between message start stop)

0:00:35

App log data(day of week)

939

485

764

-

100

200

300

400

500

600

700

800

900

1 000

Monday Tuesday Wednesday

App log data(per team)

- 50 100 150 200 250 300 350 400

Be-Mobile

V-Tron

Flanders G5

Alps

Rijkswaterstaat 2

Dynniq NL

TfL and HE

TNO

Dynniq IF2 (by RWS)

YoGoKo

Evaluation team

Rijkswaterstaat 1

UnivValenciennes UPHF

Flanders (Dynniq BE)

Université Reims

App logs (per message type)

0

200

400

600

800

1000

1200

1400

1600

IVI DENM THUMB SPAT

App log data(per message detail)

0 50 100 150 200 250 300 350

RWW

lane closed

lane free

speedLimitMax 70

clear lane to right

speedLimitMax 100

speedLimitMax 90

speedLimitMax 50

thumbs-up

accident

speedLimitEnd

clear lane to left

SPAT straight

any other warning

thumbs-down

SPAT general

speedLimitMax 110

SPAT left

Free Text

speedLimitMax 30

SPAT right

speedLimitMax 60

speedLimitMax 80

App log data(spatial scope, all teams)

App log data (spatial scope,(de)activation of messages)

• Example: RWW messages:

• Example: Lane closed/free:

App log data (spatial scope,(de)activation of messages)

• Example: GLOSA

App log data(thumbs up and down)

App log data QGis Cases

• RWW (case with dynamic maintenance vehicles)

• RWW (Classic info)

25/03 26/03 27/03

#Passes 6 5 3

Detected 6 3* 2*

Succesrate 100% 60% 66%

25/03 26/03 27/03

#Passes 6 5 3

Detected 6 3* 2*

Succesrate 100% 60% 66%

App log data QGis Cases

• IVS (lane closed)

• Warnings (accident)

25/03 26/03 27/03 Total

#Passes 7 8* 9 24

Detected 4 4 6 14

Succesrate 57% 50% 67% 58%

25/03 26/03 27/03 Total

#Passes 7 7 7 21

Detected 6 5 4 15

Succesrate 85% 71% 57% 71%


Analysis of NL OBU logging

Logging from OBU (& RSU)

Interoperability in communication, application and driver support

Intermediate results – still collecting and analysing data!

GLOSA event

Test statistics

GLOSA

Observations:

• Absolute waypoints in MAP (compliant but not used before)

• SPAT not sent (temporary technical issue)

• Static / adaptive TLC (realistic, - affects advices)

KPI G5 Hybrid

Use-case count 2 intersections N12

Participation (#OBU) 2 2

Success use-cases 2 2

# OBU events 103 133

Test statistics

In-Vehicle Signage

Observations:

• > 1700 events broadcasted (including real matrix sign changes)

• Same IVI sent by multiple stations (not uncompliant)

• Non-standard traffic codes used (configuration issue solved)

KPI G5 Hybrid

Use-case count E34 – E313 - A16




Test statistics

Road Works Warnings - DENM

KPI G5 Hybrid

Use-case count E34 – E313 - A16




Outlook

• Calculate communication delays • Alternative (member state) data provisioning services (IF2)

• ITS-G5 versus 4G

• Security• Certificate validation (authorities)

• Driver advice quality• Timing and location accuracy of warnings

• Updating of matrix signs

Team-specificresults and feedback

Team

Team

• RWW• Received warnings from moving ‘botsers’

• IVI• Signs on the highway received via ITS-G5 and

cellular• GLOSA

• Fixes made on OBU to suppress bicycle paths• Worked both via ITS-G5 and cellular

• Other• Cross border testing in Helmond (NL)• Cellular feed via Talking Traffic• ITS-G5 in Helmond with complex intersections

Team V-Tron & Rijkswaterstaat

ZOOF CITS app

•Right hybrid

•Left Wifi-P only

• RWW • IVI• Other


ZOOF GLOSA app

•Hybrid


• RWWWorked well on both channels (Cellular & WIFI-P)

• IVIWorked well on both channels (Cellular & WIFI-P)

• GLOSAWe made changes during the test days to fix things that doesn’t worked at the start of TestFest BE. We left the TestFest with a matching ZOOF GLOSA app!

• OtherNo further notes

Team TNO/NL PKI

• BE (25/03-26/03) – Working properly• Received unknown AA certificates from RSU’s via

CAM Certificate Request procedures

• NL (27/03) – Working mostly properly• Appeared to be an issue with the RSU’s, where

we did not receive a reply in all cases, but once the AA was cached, the requests where no longer necessary

Team TNO/NL GLOSA

• BE • Only one prediction -> end time of current

phase, no future phases are present in the SPAT

• Before 16:30• Perfectly stable countdown on time to phase

change• Perfectly stable speed advices can be calculated

• After 16:30• Unknown timing value from VRi was 36000, should

have been 36001 (was later fixed)• Countdown timers are unpredictable causing

calculation of speed advices to also be unpredictable

• NL• Very unpredictable phase timing, much

more than BE after 16:30• VRi is sending more predictions, although

because of the unpredictability of the phase timing this cannot be benefitted to it’s full potential

Team TNO/NL DENM&IVI

• BE• Latency for real-time IVI update ~5

seconds (rough guess)• Second day had some overlapping

traces(?) for IVI, causing our HMI to display ~8 lanes when there were 4

• 2x DENM for same location/event was sent out (was later fixed)

• NL• Messages for lanes from Rotterdam ->

Dordrecht were only sent out over G5 (was later fixed)

Team Alps

Team Alps

• RWW• No particular problems detected on the Antwerp/E313 site, but our UI is very simple.• Tested both ITS-G5 and Hybrid.

• IVI• Initially some problems with timing on Hybrid, but it was caused by our (test-only) IF2

client. No problems on ITS-G5.

• GLOSA• Our UI only displays TTG. In general we had very good match with real signals. • We had problems with MAPEMs over ITS-G5 on Helmond 502, 505 intersections,

reasons are still to be analyzed.• On system level, we had problems with ITS-G5 and cellular messaging overlap.

• Other:• Very good test event for us, we’ve found many problems with our implementation

(mostly GUI side)• The Helmond site with high traffic of complex MAPEMs was causing system overload in one of our low-power embedded targets, This problem would probably go unnoticed for some time if it wasn’t for this event.

Team France (Viveris)Pilot vehicle / French Ministry delegation

• Vehicle : Renault Megane• Validation of the PKI ; 2 runs on 27th March :

E313 + E34

• Interface overview: On-board HMI


• Interface overview: DENM for RWW, received on E313, East direction


• RWW• DENM received and displayed

• IVI• Events received but not displayed. Multilane not

supported yet• GLOSA

• Out of scope on this vehicle, for Renault current development

• Other: PKI• Interoperability works well with either DENM and

IVI messages• Use of FL RCA for signature verification• Use of FR AA1 for message signature


Team Be-Mobile

Team YoGoKo

• Objective for this TESTFEST: Interoperability of Hybrid architecture (C-ITS-S + OBU)

IF2 IF2

Quad-tree tiling over MQTT

Direct reception

over ITS-G5

YoGoKo Central ITS-Station

Built for scalability and interoperability

Team YoGoKo

• Messages tested• Spat/Map (GLOSA)• DENM (RWW, RHS)

• Modes tested• Hybrid mode: both ITS-G5 and cellular connected• Cellular only mode: ITS-G5 antenna disconnected

• Results• Good behavior experienced• Supports a high load of messages• Further analysis to be done

Architecture already deployed in Norway(E8Borealis project) with Aventi

Team UK

Team UK

• IVS• All IVS events displayed correctly on the HMI on

the E313 & E34• All IVS events displayed correctly on HMI in

Dordrecht

• RWW• RWW events received and displayed on the HMI

on test runs through Dordrecht• Received DENM messages whilst on the E313 &

E34. Further investigation regarding their display

Team UK

• GLOSA• All GLOSA events displayed correctly and

accurately on the HMI in Schilde• Received GLOSA events in Helmond were

correct and accurately displayed on the HMI• GLOSA in Helmond was intermittent. Possible

OBU cellular issue

• Other• IF2 provided services in all member states• HMI unable to display ITS-G5 messages in

Flanders. Possible issue during certificate preloading

Conclusions

TESTFEST goals

• TESTFEST 1 (NL): G5

• TESTFEST 2 (FR): G5 + security

• TESTFEST 3 (UK): G4/4G hybrid

• TESTFEST 4 (all): cross-border interoperability

Conclusions

• Overall• The events that were transmitted by the Road Side Units were received:

speed limits, road works, lane closures/opening, …

• GLOSA worked• Participants received the status of ahead traffic lights (red/green)

including speed advice and timing

• ITS-G5 and cellular data are consistent• The information received trough both channels is consistent and

generally received synchronously

• PKI• Generally worked well

Conclusions

Cross-border interoperability achieved!

• Problems that were registrated were mainly programming issues and not directly related to interoperability

• More detailed results on interoperability will be processed

• Thanks to all the teams for this successful TESTFEST!

Thank you!

Documents

Cross-border Interoperability TESTFEST · KPI IVS & Cellular IVS & Cellular Use-case count Dynamic Speed Limit Information Southbound (Showed 50 Speed limit and Speed Limit End) Dynamic