Copyright  ©  2013  Splunk  Inc.  

Ant  Lefebvre  Senior  Systems  Engineer,  Middlesex  Hospital  #splunkconf  

HIPAA  and  Meaningful  User  Audit  Reports  Using  Splunk  

About  Middlesex  Hospital  !   We  offer  a  complete  range  of    

medical  services  !   Some  of  ConnecMcut’s  highest  quality    

and  paMent  saMsfacMon  raMngs  –  25  Networked  Offsite  LocaMons  –  9  Primary  Care  Offices  –  3  Emergency  Departments  

!   Named  to  100  Top  Hospitals  list  two    years  running  

!   Named  to  HealthCare’s  Most  Wired  List  2012  &  2013  

2  

Who  Am  I?  

!   Systems  Engineer  !   Network  Engineer  !   Security  /  Compliance  !   Wireless  !   IT  Director  !   IT  Consultant  ! Splunker  

3  

Splunk  for  Hospital  Network  OperaMons  

Challenges  in  Healthcare  

   

Event  Log  CorrelaMon  

VirtualizaMon  Management  

Global  View    of  Environment    

ApplicaMon  Performance  

5  

Hospital’s  Visibility  Gap  

!   Windows  event  viewer  is  not  easy  to  navigate  !   TroubleshooMng  mulMple  hosts  means  opening  each  log  individually  

!   CorrelaMng  event  Mmes  in  mulMple  systems  a  manual  process  !   Host  down  or  off  network  made  it  impossible  to  access  logs  !   Took  hours  or  days  to  find  root  cause(s)  for  end  user  device  issues  

Wasted  (me  and  effort  to  track  down  issues  

6  

Splunk  Solves  Visibility  Gap  Steps  to  success:  1.  Downloaded  free  demo  2.  Globally  installed  Splunk  Universal  Forwarders  on  Windows  server  

and  client  operaMng  systems  3.  Indexed  Windows  event  log  data  4.  Instantly  gained  visibility  into  windows  environment  like  never  

before  

Troubleshoo(ng  (me  now  a  frac(on  of  what  it  used  to  be  

Splunk  Enterprise  in  ProducMon  

!   Finding  new  use  cases  every  day  !   Audit  consolidaMon  –  One  tool  to  monitor  all  systems  !   Event  correlaMon  –  Is  the  issue  happening  everywhere?  When?  !   Recognize  anomalous  acMviMes  –  Something  strange  going  on?  !   Add  new  log  sources  –  See  what  shakes  out…  

No  need  to  purchase  addi(onal  products.  Index  the  data  in  Splunk  Enterprise!  

8  

Middlesex  Splunk  Enterprise  Success  Stories    •  Mystery  “wireless  disconnects”  persisted  for  years;  using  Splunk  Enterprise  searched  on  User  ID  /  tablet  name  at  drop  Mmes;  discovered  crashing  process  on  Citrix  server  at  dropping  event  Mme!  

•  Mystery  name  resoluMon  issues;  connecMng  to  wrong  workstaMons  when  using  hostname;  *error*  search  found  DNS  record  scavenging  was  accidentally  off  amer  AD/DNS  server  migraMon  

•  Started  to  index  firewall  traffic  logs;  using  Splunk  Enterprise  and  Google  Maps  app  discovered  a  Health  library  machine  connected  to  an  internaMonal  bot  net;  no  business  need  to  communicate  with  Peru  

•  Used  Splunk  Enterprise  to  discover  slowest  booMng  computers  to  prioriMze  new  PC  rollout;  transacMons  from  first  boot  service  start  to  last  boot  service  to  start  

•  User  files  “vanish.”  File  audit  tool  gave  no  insight;  Splunk  search  for  user  id  AND  delete  finds  over  300  events  in  an  hour  over  the  weekend;  user  accidentally  deleted  one  too  many  folders  

 The  list  goes  on  and  on…  

9  

Botnet  Computer  

10  

Blocking  Streaming  HDTV  Through  Firewall  

11  

Boot  Times  Table  

12  

Found  File  DeleMon  Incident  

13  

Table  of  Files  Deleted  Report  

14  

Program  Intelligence  into  Apps/Dashboards  !   Created  useful  dashboards  for  operaMons/helpdesk  team  

!   Don’t  need  to  know  Splunk  search  commands  to  use  

!   Help  less  knowledgeable  staff  troubleshoot  environment  issues  

!   Each  new  dashboard  is  created  in-­‐house;  no  need  for  addiMonal  purchase;  no  need  to  ask  for  product  enhancement  or  feature  from  vendors  

!   Single  point  of  reference  for  mulMple  uses  

The  Splunk  Admin  can  create  point  and  click  knowledge  

15  

Citrix  Disconnect  Dashboard  

16  

Power  Dashboard  

17  

Windows  NPS  RADIUS  Dashboard  

18  

Print  Server  Log  Dashboard  

19  

Print  User  to  IP  CorrelaMon  

Print  logs  do  not  contain  where  user  prints  from  Windows  Event  logs  show  where  user  last  logged  in  

20  

Viral  Spread  of  Splunk  Enterprise  

Word  of  Splunk  Enterprise  capability  to  audit  systems  and  solve  mysteries  tricked  through  other  IT  staffers    AddiMonal  systems  I  didn’t  even  know  we  had  were  added  to  Splunk  Enterprise  

21  

IT  Director’s  Challenge  !   A  system  to  audit  our  Electronic  Health  Record  access  !   A  single  soluMon  to  audit  mulMple  systems  !   Easy  to  manage  !   Cost  is  always  a  factor  !   We  have  two  opMons.  Which  one  is  beqer?  

!   The  answer:  OpMon  3  –  Splunk!  

22  

HIPAA  and  Meaningful  Use  

Healthcare  Jargon  !   EMR/EHR  –  Electronic  PaMent  Records  !   HIPAA  –  The  Health  Insurance  Portability  and  Accountability  Act  of  1996  !   HITECH  Act  –  Health  InformaMon  Technology  for  Economic  and  Clinical  Health  Act  

!   Meaningful  Use  –  Goal  is  to  not  just  adopt  an  EHR,  but  to  leverage  it  to  achieve  significant  improvements  in  care  

!   Cerner  –  Middlesex  Hospital’s  Primary  EHR    !   Results  –  Middlesex  Hospital’s  home  grown  EHR  lookup  applicaMon  ! eClinicalWorks  –  Middlesex  Hospital’s  Primary  Care  /  Family  PracMce  /  MulMspecialty  EHR  

24  

Sweetening  the  Deal  –  Managing  EHRs  !   Federal  reimbursement  for  having  cerMfied  technologies  to  audit  Electronic  Health  Record  (EHR)  access,  enforce  Meaningful  Use  

!   EHR  provider  offers  specialized  (and  expensive)  point  soluMon  –  Only  shows  who’s  logged  in  to  app  

!   Experiment:  EHRs  into  Splunk  –  no  problem  

“Splunk  provides  audi(ng  capabili(es  &  delivers  opera(onal  intelligence.”  

25  

Raw  EHR  Audit  Data  <audit_list><audit_version>1</audit_version><event_dt_tm>2013-08-23 08:30:06.00</event_dt_tm><outcome_ind>0</outcome_ind><user_name>SYSTEM</user_name><prsnl_id> 1.000000</prsnl_id><prsnl_name>SYSTEM</prsnl_name><role>DBA</role><role_cd>24209801.000000</role_cd><enterprise_site>HNAM</enterprise_site><audit_source>TEST/Default Logical Domain</audit_source><audit_source_type>274986</audit_source_type><network_acc_type>1</network_acc_type><network_acc_id>mhscnpap</network_acc_id><context><![CDATA[MzQ2NzgyOTc3fDI3NDk4NnwyNzMwMjJ8MjY1MTE4fDI0fA==]]></context><application>SCS Netting Server</application><task>UPDATE SCS Netting Task</task><request>scs_get_proc_server_netting</request><appl_ctx>346782977</appl_ctx><perform_cnt>24</perform_cnt><event_list><event_name>Maintain Order</event_name><event_type>Tasks</event_type><participants><participant_type>System Object</participant_type><participant_role_cd>Order</participant_role_cd><participant_id_type>Order</participant_id_type><participant_id>419526210.000000</participant_id><participant_name>Blood, Timed Study collect, 08/23/13 5:00:00, Lab Collect</participant_name><data_life_cycle>Origination/Amendment</data_life_cycle><person_id>4480371.000000</person_id><person_name>BCMA, Dana</person_name><vip_display></vip_display><encounter_id>15571493.000000</encounter_id><encounter_org>MIDDLESEX HOSPITAL</encounter_org><medical_service>Medical Services</medical_service><location>CCU</location><encounter_confid_level></encounter_confid_level><admit_dt_tm>2012-11-21 13:53:23.00</admit_dt_tm><discharge_dt_tm>0000-00-00 00:00:00.00</discharge_dt_tm><encounter_type>Inpatient</encounter_type><encounter_status>Active</encounter_status><encounter_mrn>9913</encounter_mrn><encounter_fin>11452</encounter_fin><relationship_creation_reason></relationship_creation_reason><relationship_creation_dt_tm>0000-00-00 00:00:00.00</relationship_creation_dt_tm><relationship_created_by></relationship_created_by><relationship_creation_type></relationship_creation_type><relationship_type></relationship_type><participant_query></participant_query><facility>MIDDLESEX HOSPI</facility><building>Middlesex Bld</building><nurse_unit>CCU</nurse_unit><room></room><bed></bed><external_source></external_source><person_alias></person_alias><sensitivity_codes></sensitivity_codes></participants></event_list><alt_user_name></alt_user_name><user_organization_name></user_organization_name><user_organization_cd> 0.000000</user_organization_cd><personnel_role></personnel_role><application_number>274986</application_number><task_number>273022</task_number><request_number>265118</request_number><prsnl_alias></prsnl_alias><user_organization_alias></user_organization_alias></audit_list>!

Splunk  to  the  rescue…  26  

Under  the  Hood  

Cerner  Listener  /  Splunk  Universal  

Forwarder  

Cerner  Audit  Outbound  Server  

Splunk  Indexer  

IngesMng  Cerner  EHR  audit  data  into  Splunk  

27  

Under  the  Hood  Part  2  IngesMng  Results  EHR  audit  data  into  Splunk  

FTP  server  /  Splunk  Universal  

Forwarder  

Results  Backend  Server  

Splunk  Indexer  

28  

Not  Sure  What  Hood  to  Look  Under  IngesMng  eClinicalWorks  EHR  audit  data  into  Splunk  

?  /  Splunk  Universal  

Forwarder  ?   Splunk  

Indexer  

?  

Engage  your  EHR  vendor  EARLY!    

29  

Vision  Into  Our  Future  

Splunk  search  heads  with  TAs  (Technology  Add  -­‐ons)    and  a  Common  Healthcare  App  

Splunk  indexing  mulMple  diverse,  but  related  systems  

EHR,  Finance,  Infrastructure,  Clients,  Servers,  the  list  goes  

on….  

Compliance  Officers,  Auditors,  ApplicaMon  Staff,  OperaMons  Team,  Infrastructure  

Team  

30  

Middlesex  Hospital’s  Cerner  EHR  App  !   ApplicaMon  Report  Categories    –  AcMvity  Audit  –  Admin  Audit  –  Disclosure  Report  –  Login  Report  –  PaMent  Record  Access  –  Suspicious  AcMvity  –  User  Account  Sharing  –  VIP  PaMent  Access  

!   New  reports  are  only  limited  by    the  logs  and  the  imaginaMon  

31  

Cerner  EHR  App  Demo  

!   Get  right  to  the  facts  !   Compliance  isn’t  preqy  !   Auditors  are  going  to  love  it!  !   Meaningful  Use  of  EHR  logs  !   HIPPA  violaMon  invesMgaMon  made  easy  

 

Before we begin

Application demo with test environment data.

Application written speci!cally for

Cerner EHR for MU2

Common Information Model in development

Universal Healthcare App in

development

32  

What’s  Next?  !   Common  InformaMon  Model  for  Healthcare  !   Universal  Meaningful  Use  and  HIPAA  App  across  mulMple  systems  !   Onboard  more  systems.  Greater  visibility!  !   VMware  and  Citrix  Apps  on  hqp://apps.splunk.com/    

When  we  need  to  know  what  happened  in  and  on  our  systems,  We  turn  to  splunk>  

33  

THANK  YOU