MIMiC  –  Mobile  Immersive  Media  in  the  Cloud    

 

   

Immersive   media   uses   novel   digital   media   and   media   processing   technologies   to   dramatically   enhance   user  interaction,   collaboration   and   sharing   of   digital   media   information   over   a   wide   variety   of   social   and   business  contexts   including   the   enterprise   environment,   education,   gaming,   health,   telecoms,   tourism,   retail   and   many  more.    

Underlying  technologies   that  enhance  user  experience   include  mixed  reality   (augmented  reality   (AR)  and  virtual  reality(VR)),   natural   user   interfaces   (gesture  based   input,   touch   screen),   enhanced   media  formats   (3D   video,   computer   graphics),   new  forms  of  sensory  media  (scent,  touch)  as  well  as  image   processing.   Improved   telecom   network  support   will   also   be   needed   for   immersive  media   delivery   to   the  many   forms   of   end-­‐user  device   (smart   phone,   tablet,   PC,   TV).   Enabling  immersive   communication   on   resource  constrained  mobile   devices  will   be   a   particular  challenge   and   will   require   the   use   of   cloud  based  solutions.  

Mobile  Augmented  Reality  allows   the  overlay  of  virtual   objects   or   information   in   mobile   device  real-­‐world   views   to   give   context   aware  information.      Mobile  AR  is  used  in  a  wide  variety  of   fields   from   instruction   and   maintenance,  education,   health,   retail   and   many   more.  

Immersive   Collaboration   will   greatly   advance   the   state   of   the   art  videoconferencing   solutions   to   enable   geographically   dispersed  teams  to  engage   in   improved   interworking  and   information  sharing  in   the   enterprise   and   in   social   contexts.   Image   recognition   and  gesture   based   input   will   add   greater   context   awareness   to   the  conversation  while  mixed  reality  technologies  will  be  used  to  aid   in  information   understanding.   3D   Virtual   worlds   will   increase   the  immersive   experience   even   further   through   shared   virtual  environments.     Serious   Gaming   will   extend   this   collaboration  platform  with  the  application  of    gaming  engines  to  create  scripted  applications   in   non   gaming   domains   such   as   health,   education,  industrial   process   control,     and   sport.  Multi   sensory  media   such   as  touch   (haptic)   and   scent   (olfaction)   will   increase   the   users  

experience  in  television  (“smellovision”)  and  gaming  .  

The  purpose  of  this  document  is  to  seek  input  from  industry  to  define  a  research  agenda  for  a  research  funding  proposal  (under  EI  Applied  Research  Enhancement  program)  with  a  central  theme  of  

Mobile  Immersive  Media  un  

underu  

   

 Immersive  Collaboration  -­‐  click  to  see  example  from  Alcaltel-­‐Lucent  

   Click  to  see  Serious  Gaming  examples  

 

 

 

 

                     

Mobile  Immersive  Research  Research  in  mobile  immersive  media  spans  many  fields.  A  high  level  architecture  of  such  a  mobile  immersive  system  is  shown  in  the  diagram  below  and  helps  organize  the  discussion  on  the  research  topics  i.e.  the  main  components  of  the  architecture  are  considered  as  separate  research  strands.  Each  of  these  is  discussed  further.  

Mobile  Cloud  Media  Platform  Powered  by  the  growth  of  smart  mobile  devices  mobile-­‐cloud   interaction   is  a  sub-­‐field  that   is  growing  rapidly  and  within   this   sub-­‐field  mobile   media   cloud   computing   is   emerging   as   a   very   active   research   area   with   some   initial  deployments  being  made  commercially.  General  research  challenges  for  mobile  cloud  include  

Latency,  bandwidth  and  QoE  –  For   immersive  media  applications   latency   issues  can   influence  QoS  and  QoE   .While  new   radio   networks  will   help     on   novel   cloud   architectures   based  on   edge   “micro-­‐clouds”   (“cloudlets”)   are   being  researched  a  potential  solution.  

Resource  constrained  devices  –  cloud  processing   is  a  natural  fit  to  complement  mobile  device  computing  capacity.  Topics  for   research   include   tools,   environments   and   protocols   to  partition   the   application   processing   between  mobile   device  and  cloud,   computing  platforms   for   cloud  media  processing  (graphics   rendering,   image   processing     etc),   media  adaptation  to  heterogeneous  devices  etc.  

Immersive  Collaboration  Platform  The  goal  of   immersive  collaboration   is   to  enable   interaction  between   geographically   dispersed   people   so   that   a  perception  of  “actually  being  together”  is  created.  The  main  research   challenge   is   how   to   enhance   the   conferencing  platform   with   media   middleware   enablers   in   order   to  provide   immersive   collaborative   experiences.   The   conferencing   platform   includes   basic   capability   for   multiparty  video   and   audio   communication.   The   choice   of   which   technologies   to   include   to   enhance   the   platform   is   largely  determined  by  which  application  the  platform  is  to  be  used  for  –  see  discussion  below.  

Enabling  Technologies  This   includes  both  media  and  non  media  enablers.   In  the  scope  of  MIMiC  research  challenges  are  how  to  combine  these   technologies   into   the   immersive   collaboration  platform   to   solve  application   specific  problems.   Technologies  include  

• Analytics:  can  be  used  to  determine  usage  patterns  to  improve  performance  and  application  adaptation  e.g.  to  adapt  to  user  learning  preferences.  

• Natural  user  interaction:  includes  the  use  of  gesture,  tangible  user  interfaces  etc.    • Mobile  Payment  System  • Mobile  Advertising  (mobile  gaming  advertising  projected  to  increase  10  fold  between  2010  and  2015),    • Security  –  cross  cutting  theme,    • Image  processing/Computer  vision  –  needed  for  user   interaction,  object  recognition  or  tracking  as   in   facial  

recognition  or  mobile  AR  point-­‐of-­‐interest  recognition.  

Application  Scenarios  Immersive  communication  can  apply  in  many  different  daily  scenarios.  Immersive  Media  research  in  this  context  can  be   considered   a  mix   of  application  domain   research   as  well   as   immersive   platform   research.   Application   domain  research   explores   domain   user   interaction   and   is   carried   out   by   domain   experts.   This   will   identify   models   and  modalities  of  interaction  that  can  then  be  mapped  to  immersive  platform  technologies  and  mechanisms.  Application  collaboration  scenarios  include  

 MIMiC  Ecosystem  

 

 

                     

Enterprise   Collaboration-­‐   Research   challenges   here   include   how   to   enhance   today’s   video   conferencing   to   create  more   immersive   interaction  through  the  use  of  media  enablers  such  as  mixed  reality,  computer  vision  and  natural  users   interfacing   (e.g.   gesture   control).  Other   challenges  are  how   to   incorporate   social  media   to   create  enterprise  social  networking   to  further  enhance  collaboration  while  other  forms  of  collaboration  such  as  whiteboard  etc  may  also  be  included.  

Immersive   Education   includes   the   use   of  mixed   reality   techniques   to   create  more   appealing   learning   and   training  experiences   all   the   way   from   primary   schools   to   corporate   education   and   across   all   types   of   industry.   Research  challenges   include   selection   of   best   modalities   (AR/VR)   or   different   learning   situation,   interworking   between   e-­‐learning  ecosystems  based  on  different  virtual  worlds  etc.  

Immersive   Health   –   can   include   interaction   between   patient   and   doctor   or   between   medical   teams.   Health  applications   may   require   HD   video,   high   resolution   graphics   and   large   network   bandwidth.   Cloud   based   media  platforms  may  be  a  very  suitable  candidate  to  provide  immersive  health.  

Gaming    -­‐      gaming  challenges  include  exploration  of  new  ways  to  deliver  gaming  using  cloud  platforms  including  to  mobile  devices  and  enhanced    MMOG  performance    and  use  of  mobile  augmented  reality  to  enhance  games.    

Serious  Gaming  -­‐Serious  gaming  is  seen  as  an  important  research  topic  at  both  a  national  and  international  level  with  application  in  many  fields.  Research  challenges  are  how  to  evolve  the  immersive  collaboration  platform  to  meet  the  needs  of  application  domain  researchers.    

Mobile  Mixed  Reality  –  Mobile  Augmented  Reality  is  becoming  a  mainstream  technology  with  technology  improving  al  the  time.  Research  challenges  include  exploring  the  use  of  AR  techniques  in  different  applications  fields  as  well  as  exploring  the  user  of  natural  user  interaction  techniques  and  new  platforms  such  as  head  mounted  devices.      

Television   will   change   greatly   during   the   coming   years.   Research   challenges   include   the   use   of   natural   user  interfacing,   using   automated   content   recognition   technique   to   enable  dual   screen   (TV,   tablet)   synchronization,   or  multisensory  stimulation  e.g.  see  the  pizza,  smell  the  pizza.    

About  SRI  and  MIMiC  The  Software  Research   Institute   in  Athlone   Institute  of   technology   conducts   research   in   multimedia  networking   and  network  monitoring   in   a  number  of  domains     –   see   www.ait.ie/sri   .   The   group   also  researches   cloud   monitoring   and   security   a   part   of  the  Enterprise   Ireland  Cloud  Technology  Centre.  SRI  operates   the   SUNAT   (Seamless   Use   of   Network  Abstraction   Technologies)   ARE   which   researches  ways   to   intelligently   adapt   media   applications   to  changing  network  and  device  conditions.  SRI  has  an  extensive   track   record  of  engagement  with   industry  and   has   participated   in   many   Innovation  Partnerships   and   Innovation   vouchers   as   well   as  contract   research  with   a   number   of   companies.   SRI  has   also   had   a   number   of   academic   collaborations  including  European  research  partnerships.    

With  MIMiC   SRI   builds   on   its   SUNAT   experience.   In  particular   SRI   will   extend   video   conferencing   technology   currently   being   developed   within   SUNAT   to   form   a  conferring  platform  for  MIMiC.  SRI  will  also  leverage  its  other  research  activities  including  cloud  security  research  to  enhance  the  output  from  the  MIMiC  program.  

   

MIMiC  ARE  Way  of  Working