Color-plus-Depth Level-of-Detail in 3D Tele-immersive Video: A
Psychophysical Approach
Color-plus-Depth Level-of-Detail in 3D Tele-immersive Video: A
Psychophysical ApproachWanwin Wu et la., MONET Group UIUCACM MM
2011 Best PaperTele-immersion
Psychophysics (PP)
Psychophysical Thresholds
Problems
Real-time 3D model constructionMulti-party
communicationInter/Intra-node synchronizationComputational
resource-hungryNetworking resource-hungry
Computational resource BalancingConstruction of 3D Models
Parameters of Color-plus-Depth Level-of-Detail (CZLoD)Detailing
parameter of CZLoD: THvar (Threshold of Variance)- Recursively
refining bisection until the variance within every polygon
(triangle) is less than THvar - THvar decides the size of
triangles- THvar decides both texture (color) and spatial
resolutionDegradation of CZLoD: DR (Degradation Ratio)- Fi: the ith
3D frame- Nx(Fi): number of vertices in Fi if THvar = x
GoalsFinding the PP thresholds in tele-immersive videos- JNDG
(Just Noticeable DeGradation)- JUADG (Just UnAcceptable
DeGradation)Utilize the thresholds on constructing a real-time
adaptation scheme for resource balancing- QoS Monitor- Decision
Engine- Variance CalculationGoal 1: Finding PP ThresholdsPhysical
stimuli: DRPerceptual quantity: JNDG and JUADG(participant was
asked if he/she could tell any difference between the two clips,
and whether he/she thought any video had unacceptable
quality)Experiment method: Ascending Method of Limit
Video
Finding PP Thresholds (conti.)Participants: - 16 graduate
students of CS UIUC- all had normal or corrected vision- 4 Indian,
3 American, 2 Chinese, 2 German, 2 Bangladeshi, 1 Mexican, 1 South
African- 6 women (37.5%) and 10 men (62.5%)- 5 experts of
tele-immersion (31.25%) and 11 novices (68.75%)Mapping From Thvar
To DRVery much content-dependent
The input parameter (THvar) is manually adjusted to form a
series of gradually degrading videos
ResultsThe average thresholds among participants and among the
four videos areJNDG: 61.5%; JUADG: 81.25%Thresholds of low
resolution videos are lower (easier for users to notice
degradations)Influence of content is less with higher resolutionThe
size of gray zone is 10~20%
Goal 2: Adaptation Scheme
QoS MonitorInput: the last 3D frame displayedAnalyze: DR and
construction time (frame rate)
Output- DR value - abnormal frame rate (FR) event: if (FR >
THh || FR < THl)
More Details On QoS MonitorAccording to [16], (THh, THl) is set
as (8, 12)In calculation of DR, N0(Fi) is periodically computed to
reduce complexityDR here is hence defined as min{1, DR(Fi)}The
frame rates are measured within a sliding window of 5 frames
Decision EngineInput: DR value and abnormal FR reportOutput:
target DR and DR error (explained later)The purpose is fairly
simple, actually- If FR is too low, increase it (by d)- If FR is
too high, decrease it (by u)The nontrivial are the amounts of
changes (d and u)The changes can follow several protocols-
constant- AIMD- ...
What about the PP Thresholds?Three decision zones
The adaptation zone = [JNDG-Bn, JUADG+Ba]d and u are set as 0
outside the adaptation zone
Variance CalculatorInput: target DR and DR errorOutput: THvarThe
same problem again: a mapping F from DR to THvar is neededSolution:
least-square regression-based learning
When DR error is larger then some threshold (THerr) then the
learning process is triggeredWith ten training points (s0~s9), the
median residual is 0.022%
Performance EvaluationFrame rate improvement
(a) w/o extra CPU stress; (b) w/ 16% CPU stress
Performance Evaluation (Conti.)DR THvar mapping
Crowd-sourcing (Youtube) scoring
Summary / Future WorkPP thresholds do exist in tele-immersion PP
thresholds can help improving resource balancing between spatial
and temporal resolutionsAlthough mapping between parameter setting
and actual CZLoD is content dependent and nontrivial, a simple
regression-based learning provides sufficient predictionExpect
higher computing power to support real-time N0 computation in the
futureMore complex protocols to deal with the adjustment (d and u)
of FR
RemarksNo variance shown in the experiment results?No
statistical support of significance?Sampling of stimuli needs to be
non-uniform
