Human-Behavior Sensing andVisualization for Service Quality Control

Takeshi Kurata, Masakatsu Kourogi, Takashi OkumaTomoya Ishikawa, Ryoko Ueoka, Ryuhei Tenmoku, Koji Makita

Center for Service Research, AIST, Japan

CSCW 2012 Workshop: Exploring collaboration in challenging environments: From the car to the factory and beyond

Evidence-based Service (EBS) Through Real-World Virtualization

• Real-Virtual correspondences of products with POS (Point-Of-Sales) systems– Facilitate modeling and designing the flow of the

products by not strongly relying on tacit knowledge.– Brought about drastic changes in retail, chain

restaurant, logistics, etc.– Realized EBS to some extend for Service Quality

Control (QC).

On the analogy…• One of the next key issues for service innovation How to Make better correspondence betweencustomers/employees/environmentsand the computerized ones.

14

Virtualizing Human Behaviors and Environments in Service Fields

• Current Progress in virtualizing human activities– ID-POS (POS in cooperation with Customer

Relationship Management (CRM))– Driving recorders in logistics and transportation

services– Participatory sensing such as Foursquare

• Not yet realized to sufficiently grasp– Where customers are in a shopping mall– What employees are doing in a large facility

• Needs for virtualizing environments– In a laboratory it should be known or controllable.

2

MRIS in Service Engineering• If human behavior and environments are clarified…

– More comprehensive and synthetic visualization (Mieruka: 見える化) for Evidence-based Service (EBS ) by the combination of

• “Results” such as POS accounting data• “Behavior” that brought the results• “Environmental stimuli and Constraints” that influenced the

behavior

• In this talk,– Human behavior measurement of employees and the

visualization by using “Mixed-Reality Information Sharing” (MRIS) technologies

– Applications in actual service fields• Japanese restaurant, Nursing home, Japanese-style hotel

3

HBIplus for Service Quality Control4

* HBI: Human-Behavior Indicator

Human Behavior Human Behavior Indicators and

Responses

Labor quantity, Trajectory, Service operation log

Resulting Indicators

Environment Stimuli and Constraints

Diverse and Normalized Indicators

Accounting history, Nurse call log, Energy consumption indicators, Plant operation log, Subjective evaluation results

Alleviation of issues and biases due to Intangibility and Heterogeneity

Layout, Appearance, Perception, Skill, Economy, Brand image

MRIS Technologies for Multi-Stakeholders5

PDR（Pedestrian Dead-Reckoning）

Estimates velocity vector, relative altitude, and actionsby measurements from waist-mounted sensor module.Wearing sensor module on waist Easy to wear and maintain Easy to measure data for action

recognition Relatively easily to apply for handheld

setting compared to she-mounted PDR based on Zero Velocity Updates (ZUPTs)

Recognition of walking locomotion Low-cost sensors

M. Kourogi and T. Kurata, “Personal Positioning Based on Waling Locomotion Analysis with Self-Contained Sensors and a Wearable Camera”, ISMAR2003, pp. 103-112, 2003.

15

Sensor module・Accelerometers・Gyro-sensors・Magnetometers・Barometer

Indoor Pedestrian Tracking Using SDF

Pedestrian Dead-Reckoning (PDR)

IDreader

IDRSSI

Acceleration /angular velocity

Building Structure/Layout

Magneticvector

Magnetometer

Output of position/orientation

Pair-wise-sensorbased positioning

Atmospheric pressure

Barometer

TrajectorySensor/Data Fusion (SDF)

(Particle filter)

Accelerometers/ gyro-sensors

Walking velocity

Position / Orientation

Trajectory matching/Velocity estimation

Absolute position

3D environment model

Velocity vector /Relative altitude /Action type

Sensor module

ActiveRFID tagID

Surveillance camera/

RGB-D sensor

ID-LEDID

Video/Depth

T. Ishikawa, M. Kourogi and T. Kurata: "Economic and Synergistic Pedestrian Tracking System with Service Cooperation for Indoor Environments", Int’l Journal of Organizational and Collective Intelligence, Vol.2, No.1 pp.1-20 (2011)

Application 1: QCC in Japanese Cuisine Chain Restaurant

7

Staying-time rate at each premise area per personSales at each premise area per employee

Visualization tool combining human-behavioral and accounting history

Employee taking order while cleaning up the

guest room

Icons showing the number of customers at each table

POS data log

QCC in manufacturing industryPurpose: Productivity improvement

Conventional QCC in service industryPurpose: Productivity improvement

Subjective QCC in service industryPurpose: Improvement of CS/ES

Computer-supported QCC (CSQCC)Purpose: Productivity improvement

w/ reasonable ways to gather objective data in plants

In 1980s, applying QCC for service industry

w/o reasonable ways to gather objective data in service fields

In 1990s, Service industry lost interest in QCC

In 2000

In 2010

CSQCC in the futureProductivity improvementImprovement of CS/ES

QCC in the Service Industry in Japan8

w/ reasonable ways to gather subjective data continuouslyw/ reasonable ways to

gather objective data in service fields

During Discussion in CSQCC9

Trajectory of a waitress (hostess) in lunch time: 12:00-14:00

R. Ueoka, T. Shinmura, R. Tenmoku, T. Okuma, T Kurata: “Introduction of Computer Supported Quality Control Circle in Japanese Cuisine Restaurant”, In Proc. Int’l Conf. Human Side of Service Engineering (HSSE2012) jointly with AHFE2012, July, 2012.

Fact: Going in and out of the kitchen/office to no small extent.Possible result: Difficulty in concentrating on guest service.Cause: Cell phone everywhere, but reservation book only in the office room.Possible improvement: e-reservation book

Front of the house

Kitchen

Office room

Application2: Job Analysis in Nursing Home

• How long it took to respond to the nurse call (NC)• Which care worker responded to the NC• Where and how far the care worker and other workers were when the NC was switched on• How each care worker actually took action until the NC was turned off

10

Indicator for on-state of NC by the resident of room #201

Care worker responding to

nurse call (NC)

Timeline showing on/off-state of NC with room #

NC data log

40

60

80

100

40

60

80

100

40

60

80

100

• Nurse R: Role as a leader. Mainly desk work and sometimes vital check of residents.• Nurse S: Taking care of each resident while relatively flexibly circulating.

Care worker E, I, K Care worker D, H, MCare worker A, G

• Flexibly changing the role?• Or low skill?

• High skill?• Or assigned at specific floor?

• Mainly desk work?

# of steps

# of utterance(VAD)

# of floor change

Time spent inresidents’ rooms

Nurse R

Nurse S

# of steps

# of utterance

# of floor change

Time spent inresidents’ rooms

# of steps

# of utterance

# of floor change

Time spent inresidents’ rooms

Voice Activity Detection (VAD) FrequencyLow High

RestroomBath/Dressing roomResidents’ rooms Corridor Nurse Station Stairs/EV Dining room

Application2: Job Analysis in Nursing Home

Validation of the hypotheses on what is related to high skills:e.g. ‘Workers who are skillful at comprehensive awareness is to talk to residents frequently everywhere, but each conversation is basically short.’

Interview with FPV

Passage of Time

+ Over 50% cost reduction on labor cost and preparation time compared with existing time studies+ Consideration of customer privacy by not using cameras+ FPV with less motion sickness+ Effective in episodic memory retrieval for retrospective interviews based on definitive rationality

Worker’s trajectory

3D model built from a set of photos

First-person view (FPV)

CCE (Cognitive Chrono-Ethnography) Lite

Results, Behavior, Stimuli, and Constraints(Transforming Service Fields into Laboratories)

Human Behavior

Results

Environment Stimuli and Constraints

InfluenceInfluence

Decision-making process• Definitive rationality• Satisficing principle

Long-term Trend

Accounting, Energy consumption, Plant operation, Others’ condition (CS/ES)

Layout, Appearance, Weather, Economy, Brand image

InfluenceInfluence

13

Virtualization in indoor environments

AR/DR Presentation

AR/DR Presentation

Skill, Perception

Influence