0    

 Program  Review  Self-­‐Study  Pardee  RAND  Graduate  School      Pardee  RAND  Graduate  School  August  2014  

   

1    

 

   

1    

Overview    Forty-­‐four  years  after  its  founding,  the  Pardee  RAND  Graduate  School  remains  true  to  its  original  mission  of  providing  the  next  generation  of  policy  leaders  an  unsurpassed  education  in,  and  application  of,  the  fundamental  tools  of  policy  analysis  and  proud  of  its  unique  status  as  the  only  policy  PhD  program  located  within  a  major  independent  public  policy  research  institution.    Pardee  RAND  is  poised  to  achieve  its  vision  of  becoming  the  premier  policy  PhD  program  in  the  nation.    To  accomplish  this  goal,  our  priorities  are  as  follows:    (1)  continue  to  improve  the  academic  quality  and  breadth  of  our  PhD  program  with  a  particular  emphasis  on  addressing  gaps  in  an  overall  rigorous  set  of  requirements  and  offerings;  (2)  enhance  financial  support  to  our  students  in  order  to  enable  them  to  focus  on  their  educational  experience  and  training;  and  (3)  build  upon  and  strengthen  the  intellectual  linkages  between  RAND  and  its  graduate  school.    These  priorities  address  all  the  elements  essential  to  a  premier  policy  graduate  program—our  institutional  foundation  including  faculty,  students,  and  finances;  our  curriculum,  including  coursework,  project  work  (on-­‐the-­‐job-­‐training  or  OJT),  and  the  dissertation;  and  support  to  students,  including  admissions,  orientation,  scholarships,  dissertation  support,  enhanced  library  services  and  career  development.      

From  its  founding,  Pardee  RAND  has  had  a  distinctive  and  well-­‐defined  purpose  that  grew  out  of  and  is  shaped  by  its  location  within  RAND,  one  of  the  highest  quality  and  effective  policy  research  institutions  in  the  United  States,  which  now  has  a  global  reach  and  presence  as  well.    The  distinctive  elements  of  the  Pardee  RAND  experience  are  all  determined  by  this  enduring  relationship  with  RAND.    Because  of  the  nature  of  the  rigorous  analytic  work  done  at  RAND  along  with  RAND’s  commitment  to  interdisciplinary  approaches,  our  curriculum  is  designed  to  teach  the  multi-­‐disciplinary  tools  necessary  to  do  high-­‐quality  and  objective  policy  analysis.    The  faculty  at  Pardee  RAND  is  drawn  primarily  from  the  RAND  research  staff  –  a  staff  of  over  800  research  professionals  including  PhDs,  MDs  and  JDs  in  a  wide  range  of  fields  and  disciplines  -­‐-­‐  with  the  result  that  our  professors,  in  addition  to  having  outstanding  academic  credentials,  are  also  policy  research  practitioners.    A  defining  feature  of  the  program  is  that  to  augment  what  they  learn,  all  of  our  students  are  expected  to  work  as  team  members  on  RAND  research  projects  as  part  of  their  academic  training.    This  on-­‐the-­‐job-­‐training  (OJT)  enables  our  students  to  obtain  project  and  professional  skills  and  to  graduate  with  both  a  diploma  and  a  resume.      

In  additional  to  a  solid  intellectual  foundation,  RAND  also  provides  the  graduate  school  with  a  firm  financial  foundation.    However,  beginning  in  the  early  1990s,  Pardee  RAND  has  sought  to  shift  its  financial  position  from  a  one-­‐way  dependence  on  RAND  to  a  more  inter-­‐dependent  and  mutually  supportive  model.    

Over  the  past  four  years  under  the  leadership  of  Dean  Susan  L.  Marquis,  the  Pardee  RAND  Graduate  School  has  undertaken  a  major  fundraising  campaign  to  enable  us  to  achieve  our  three  priorities  listed  above.    Thus  far,  we  have  increased  endowed  support  of  the  school  by  

2    

$10.6  million  and  secured  an  additional  $8.6  million  to  fund  current  operational  needs.    To  augment  our  role  as  the  “engine  of  innovation”  for  RAND,  Pardee  RAND  has  raised  funds  for  and  launched  a  number  of  initiatives  designed  to  stimulate  new  thinking  and  new  ideas  in  a  number  of  promising  areas  for  increased  RAND  research.    Several  of  these  new  initiatives,  namely  the  Frederick  S.  Pardee  Initiative  for  Human  Global  Progress  and  the  John  and  Carol  Cazier  Environmental  and  Sustainability  Initiative,  will  enable  students  and  faculty  to  work  together  on  some  of  the  world’s  most  pressing  problems  in  the  field  of  international  development  and  economic  and  environmental  sustainability  and  allow  the  graduate  school  to  bring  to  campus  some  of  the  best  minds  and  practitioners  from  the  field  to  challenge,  inspire,  inform,  and  educate  Ph.D.  candidates,  faculty,  RAND  researchers  and  the  RAND  community  .      

Moreover,  enabled  by  the  growth  in  our  endowment  and  increased  philanthropic  giving  for  scholarships,  we  have  successfully  made  the  transition  to  a  funding  model  that  combines  tuition  scholarships  in  the  first  two  years  with  stipends  based  on  escalating  amounts  of  RAND  project  work  as  students  move  through  the  academic  program  and  into  their  dissertations.      The  goals  of  this  move  are  to  provide  more  money  directly  to  students,  decrease  their  tax  burden,  reduce  the  complexity  of  the  program,  and  further  reduce  educational  debt  for  our  students.    If  students  meet  their  project  work/OJT  targets,  they  can  graduate  without  debt  to  the  program.  We  also  believe  this  step  is  essential  for  us  to  continue  to  attract  the  best  and  the  brightest  as  higher  education  recruitment  becomes  more  and  more  competitive.    

Finally,  we  are  laying  the  groundwork  for  the  longer-­‐term  future  of  the  school  (beyond  the  period  of  the  campaign),  a  future  that  emphasizes  significant  support  from  the  graduate  school  to  RAND  research  outside  of  the  constraints  of  client-­‐sponsored  work.    Research  support  includes  funding  for  RAND  researchers  as  faculty  and  bringing  innovative,  policy-­‐oriented  new  faculty  and  visiting  scholars  and  practitioners  to  the  school  and  to  RAND.  

Program  Objectives  The  focus  for  this  Program  Review  is  our  academic  program  which  has  three  components:    coursework,  on-­‐the-­‐job  training  and  the  dissertation.    Through  the  combination  of  these  three  components  students  are  expected  to  achieve  five  educational  objectives:  

• Understand  the  purpose  of  policy  analysis  and  its  place  within  the  political  process;  

• Master  the  basic  methodologies  used  in  policy  research  including:    economic  analysis,  quantitative  methods,  and  social  and  behavioral  science  methods;  

• Acquire  an  in-­‐depth  knowledge  in  one  of  these  three  methodological  fields;  

• Obtain  a  deep  understanding  of  a  specialized  substantive  field  of  public  policy;  and  

• Develop  project  and  professional  skills  relevant  to  the  selected  field  of  policy  analysis.  

3    

For  each  of  these  objectives,  our  aim  is  to  educate  our  students  so  that  they  are  able  to  both  produce  and  utilize  knowledge  across  two  dimensions.  First,  we  want  them  to  be  able  to  master  the  ability  to  produce  policy  relevant  knowledge,  and  second  we  want  them  to  be  able  to  master  the  ability  to  use  evidence  to  address  policy  relevant  questions.    We  believe  our  core  functions  are  well-­‐aligned  to  enable  our  students  to  attain  these  goals  through  coursework,  on-­‐the-­‐job  training  (OJT)  and  the  dissertation  process.    Moreover,  we  believe  we  are  well  on  our  way  to  creating  an  institution  that  is  geared  towards  regular  assessment  and  evaluation  of  significant  aspects  of  our  program,  enabling  us  to  routinely  gauge  how  effective  we  are  at  ensuring  that  all  students  are  achieving  proficiency  in  these  areas,  and  to  make  changes  and  improvements  as  needed  to  keep  us  always  moving  closer  to  our  goals.  

In  this  report,  we  describe  in  detail  how  we  have  revised  our  academic  program  in  recent  years  and  what  further  changes  we  are  contemplating  in  the  core  components  of  our  academic  program:    1)  classroom  learning  and  coursework;  2)  on-­‐the-­‐job  training  (OJT)  and  3)  dissertation  quality,  paying  particular  attention  to  changes  that  we  have  made  that  have  contributed  to  improving  student  achievement  and  learning  outcomes.    But  before  doing  so,  we  provide  a  brief  overview  of  the  support  framework  that  undergirds  the  academic  program.      

Admissions  Policy  Analysis  as  an  academic  discipline  is  multi-­‐disciplinary  by  nature,  demanding  students  possess  competence  across  a  large  number  of  disciplines  including  economics,  statistics,  econometrics,  operations  research,  social  and  behavioral  science,  and  cost  benefit  analysis  in  order  to  be  successful.    Students  also  need  to  obtain  a  certain  depth  of  knowledge  in  a  particular  field  of  policy.    Not  every  student  has  the  drive,  dexterity,  and  breadth  of  interest  to  succeed  at  Pardee  RAND.    Therefore,  our  academic  standards  necessarily  must  start  with  the  admissions  process.    We  set  the  bar  high  for  GRE  scores,  especially  the  quantitative  score  but  also  the  verbal  score,  because  the  nature  of  the  core  curriculum  demands  that  our  students  have  a  facility  with  calculus,  linear  and  matrix  algebra  and  other  advanced  mathematical  concepts  and  the  ability  to  communicate  effectively.    We  are  not  looking  solely  for  mathematical  genius.    We  also  select  students  who  have  demonstrated  a  commitment  to  making  a  difference  in  a  particular  policy  field  and  who  have  already  displayed  a  talent  for  leadership  in  their  educational  experience  and  careers.    Accordingly,  students  with  either  an  advanced  degree  or  work  experience  are  valued  above  those  coming  directly  out  of  an  undergraduate  program.    Finally,  we  believe  that  the  quality  of  the  educational  experience  will  be  enhanced  for  everyone  if  our  student  body  more  closely  reflects  the  populations  that  we  anticipate  our  graduates  will  be  serving,  especially  if  they  pursue  leadership  positions  in  public  service.    For  this  reason,  we  emphasize  to  our  Admissions  Committee  that  diversity  of  background,  experience,  and  policy  interests  are  key  characteristics  of  a  top  graduate  program  in  policy  analysis.      

4    

Over  the  past  several  years,  we  have  worked  to  strengthen  our  recruiting,  admissions,  and  selection  processes.  Our  objective  has  been  to  broaden  our  outreach  in  order  to  identify  students  with  the  greatest  talent  and  potential  throughout  the  United  States  and  internationally.    We  have  worked  to  refine  the  questions  included  in  our  application  to  gain  better  insight  into  our  applicants,  to  move  into  an  entirely  online  application  process,  and  to  improve  the  efficiency  of  our  admissions  and  selection  process.  We  have  maintained  a  “pre-­‐application”  process  that  enables  potential  applicants  to  express  interest  in  the  school  and  provides  us  basic  information  about  their  academic  credentials,  work  experience,  and  policy  interests.  We  have  automated  the  “pre-­‐application”  so  that  we  now  easily  collect  data  on  these  500+  individuals  and  can  track  how  applicants  learned  about  us,  where  they  come  from,  and  their  academic  information.    Each  of  these  is  reviewed  by  our  director  of  admissions,  who  then  provides  specific  feedback  to  each  potential  applicant  about  their  likely  competitiveness  for  the  program,  and,  when  appropriate,  steps  they  might  take  to  improve  their  qualifications.  This  process  improves  the  quality  of  the  full  applications  we  receive  and  serves  to  eliminate  those  potential  applicants  who  are  not  credible  candidates.  The  number  of  full  applications  we  review  as  a  result  has  averaged  about  140  per  year  over  the  past  few  years,  the  majority  of  which  are  from  highly  competitive  applicants.    We  offer  admission  to  35-­‐40  applicants  each  year  with  the  aim  of  creating  a  class  of  21-­‐25  students.      

The  chart  below  provides  a  snapshot  of  the  entering  cohorts  for  the  past  three  years.  

 

2012 2013 2014

22 students 21 students 21 students

GRE: 160 (760) Quant; 160 (610)Verbal

GRE: 160 (760) Quant; 162 (640)Verbal

GRE: 162 (770) Quant; 164 (670)Verbal

10 female, 12 male 11 female, 10 male 8 female, 13 male

6 international students 10 international students 7 international students

16 domestic students 11 domestic students 14 domestic students

60% have advanced degrees 78% have advanced degrees 73% have advanced degrees

 

5    

Orientation  We  focus  on  the  screening  process  at  admission  because  we  believe  if  we  have  done  our  job  well,  each  admitted  student  will  arrive  at  Pardee  RAND  prepared  and  able  to  learn  the  material  we  expect  them  to  learn.    Once  here,  our  goal  is  to  support  each  admitted  student  through  the  core  coursework,  the  qualifying  exams,  the  dissertation  process,  OJT,  the  completion  of  the  dissertation  and  finally  advancing  in  their  careers  beyond  Pardee  RAND.    To  that  end,  we  invest    significant  effort  at  the  beginning  of  the  program  to  prepare  students  for  the  rigor  ahead.    We  offer  a  week-­‐long  pre-­‐term  language  and  cultural  “bootcamp”  for  all  international  students;  assign  each  entering  student  a  student  mentor  and  a  faculty  advisor;  and  offer  refresher  courses  in  mathematics  for  policy  analysis  and  microeconomics.    We  also  provide  each  first  year  student  with  a  faculty  mentor  for  their  on-­‐the-­‐job-­‐training  and  3  days  of  time  subsidized  by  the  School  they  can  use  on  the  project  of  their  choice.      

To  acquaint  students  with  the  broad  range  of  research  and  researchers  associated  with  the  various  policy  areas  at  RAND,  all  entering  students  are  required  to  participate  in  weeklong  policy  seminars.  These  seminars  are  designed  as  an  introduction  to  each  research  unit  at  RAND—Labor  and  Population,  RAND  Health,  RAND  Education,  Justice,  Infrastructure  and  Environment,  and  the  three  national  defense  and  international  security  units:  the  National  Security  Research  Division,  Project  Air  Force,  and  the  Arroyo  Center.  The  seminars  cover  the  key  research  currently  underway  in  these  units  on  topics  of  interest  to  our  students,  and  serve  as  an  entry  point  for  project  work  introducing  students  to  streams  of  research  as  well  as  the  researchers  leading  project  teams.      

Faculty  Drawn  principally  from  the  doctoral  staff  of  RAND's  research  units,  Pardee  RAND  faculty  members  possess  extensive  experience  both  in  conducting  research  and  in  teaching.  A  few  distinguished  scholars  from  outside  RAND  are  also  members  of  the  faculty.  Pardee  RAND  offers  a  faculty-­‐student  ratio  that  is  probably  among  the  best  in  the  country.  Our  faculty  of  193  members  serves  a  student  body  of  just  over  100.    Most  of  our  faculty  are  fulltime  researchers  at  RAND.    Unlike  research  universities,  however,  there  is  no  requirement  that  faculty  also  teach.    This  means  that  our  teaching  faculty  are  in  the  classroom  because  they  enjoy  the  give  and  take  of  passing  on  knowledge  to  the  next  generation,  and  not  because  it  is  a  job  requirement.  

RAND  researchers  can  become  Pardee  RAND  faculty  members  by  teaching  courses,  serving  on  dissertation  committees,  and/or  supervising  a  significant  amount  of  on-­‐the-­‐job  training  (OJT).      Faculty  members  are  generically  called  professors.  No  rank  distinctions  are  made.    There  are  two  categories  of  faculty  membership:  Core  and  Affiliate.    

Core  faculty  are  individuals  who  take  primary  responsibility  for  teaching  a  course,  chair  a  student’s  dissertation  committee,  participate  in  two  or  more  dissertation  committees,  supervise  OJT  for  at  least  two  students  on  projects  lasting  at  least  20  days,  supervise  one  student  in  OJT  for  a  project  lasting  at  least  40  days,  serve  on  the  admissions,  faculty  appointments  or  curriculum  and  qualifying  exams  committee  or  make  other  important  contributions.    

6    

Affiliated  faculty  are  those  who  assist  in  the  teaching  of  a  course,  serve  as  a  student's  advisor  in  the  pre-­‐dissertation  phase,  supervise  OJT  for  one  student  on  a  project  lasting  at  least  20  days,  serve  as  a  member  of  at  least  one  dissertation  committee,  or  serve  Pardee  RAND  in  other  capacities.    

Teaching  and  Learning:    A  Rigorous  and  Relevant  Curriculum  Maintaining  a  rigorous  and  relevant  curriculum  is  central  to  our  continuing  objective  of  being  the  premier  public  policy  Ph.D.  program.    The  curriculum  must  offer  students  both  an  excellent  classroom  experience  and  an  opportunity  to  extend  their  policy  analysis  training  outside  of  the  classroom.      

Pardee  RAND  has  continued  to  evaluate  and  refine  our  curriculum  to  ensure  we  are  providing  the  academic  foundation  essential  to  policy  analysis  and  are  staying  current  with  respect  to  the  needs  of  our  students  and  graduates.    As  we  do  so,  we  have  drawn  on  ideas  and  feedback  we  have  received  from  our  Faculty  Committee  on  Curriculum  and  Appointments  (FCCA),  our  students,  and  our  alumni,  as  well  as  through  the  reaccreditation  process  that  we  completed  in  2011.    All  of  these  changes  and  refinements  rest  on  the  premise  that  the  core  curriculum  should  comprise  four  critical  components  –  or  what  we  often  refer  to  as  “legs  of  a  stool.”    The  components  include:  economics,  quantitative  methods,  social  and  behavioral  sciences,  and  policy  analysis.    Below  we  provide  some  details  on  recent  refinements  and  plans  for  future  changes.  

Core  Curriculum  Over  the  past  10  years,  faculty  members  have  led  the  revision  of  the  core  curriculum.    Both  the  Economics  sequence  and  the  Empirical  Analysis  sequence  have  been  revised  with  an  eye  towards  making  each  more  coherent  as  a  sequence  and  more  appropriate  for  a  school  of  policy  analysis.    Courses  in  Macroeconomics  and  Cost  Benefit  and  Cost  Effectiveness  Analysis,  the  Policy  Analysis  sequence,  and  a  stand-­‐alone  Social  and  Behavioral  Science  sequence,  were  added  as  a  result.      

Currently  our  core  requirements  consist  of  12  units  with  one  unit  equal  to  a  single  10-­‐week  course.  Students  complete  the  majority  of  the  core  curriculum  during  their  first  year  of  study.    The  core  courses  are:  

Empirical  Analysis  (5  courses)    • Decision  Analysis  (half-­‐course)    • Empirical  Analysis  I,  II  and  III  (three  courses)  

o Probability  and  Statistics    o Regression  Analysis    o Econometrics    

• Operations  Research  (one  course)    

7    

• Cost-­‐Benefit  Analysis  (half-­‐course)    Economics  (2.5  courses)    

• Microeconomics  I  and  II  (two  courses)    • Macroeconomics  (half-­‐course-­‐2nd  year)    

Policy  Analysis  (2.5  courses)    • Policy  Analysis  I:  Perspectives  on  Policy  Analysis  (one  course)    • Policy  Analysis  II:  Case  Studies  in  Policy  Analysis  (half  course)    • Policy  Analysis  III:  Organizational  Culture  of  Government  Institutions  (one  course—2nd  

year)    Social  and  Behavioral  Science  (SBS)  (2  courses)    

• Social  and  Behavioral  Science  I:  Social  and  Behavioral  Science  Perspectives  (one  course)    • Social  and  Behavioral  Science  II:  Methods  of  Social  Science  Research  (one  course)    

 

Policy  Analysis  In  2009,  Pardee  RAND  introduced  a  sequence  of  Policy  Analysis  courses  to  provide  a  framework  that  integrates  each  of  the  analytic  disciplines.    As  the  foundation  for  the  degree,  this  three-­‐course  policy  analysis  sequence  starts  with  an  introductory  course  entitled  Policy  Analysis  I:    Perspectives  on  Public  Policy  Analysis  co-­‐taught  fall  semester  of  the  first  year  by  Jeffrey  Wasserman,  a  Pardee  RAND  alumnus  and  current  Director  of  RAND  Health,  and  Gery  Ryan,  Assistant  Dean  for  Academic  Affairs  and  a  senior  behavioral  science  researcher.    Winter  quarter  of  their  first  year  students  take  Policy  Analysis  II:    Case  Studies  in  Policy  Analysis,  that  challenges  them  to  take  a  critical  look  at  a  series  of  case  studies  drawn  from  multiple  policy  domains.    Dean  Susan  Marquis’  course,  Policy  Analysis  III:    The  Organizational  Culture  of  Government  Institutions:  Why  Government  Agencies  Behave  the  Way  They  Do,  taught  during  the  second  year,  provides  students  with  an  understanding  of  the  context  and  reality  of  public  policy  making  and  implementation  from  the  perspective  of  government  agencies  and  other  institutions.    We  now  believe  that  we  have  a  strong  set  of  courses  in  this  area  that  will  provide  students  with  a  solid  understanding  of  the  field  of  policy  analysis,  as  well  as  a  framework  for  the  analytic  tool  kit  they  will  acquire.  

Social  and  Behavioral  Sciences  The  development  of  a  separate  Policy  Analysis  sequence  created  a  need  to  also  re-­‐examine  our  offerings  in  Social  and  Behavioral  Sciences.    In  2010,  at  the  suggestion  of  the  core  faculty  in  Social  and  Behavioral  Science  (SBS)  and  with  the  support  of  the  Dean,  we  conducted  a  comprehensive  review  of  the  SBS  curriculum,  spanning  both  the  core  and  the  elective  offerings.    Sandy  Berry,  professor  of  the  long-­‐standing  core  course,  Social  Science  Research  Methods,  took  the  lead  on  this  initiative.      

The  goals  of  the  SBS  curriculum  revision  were  to  design  a  curriculum  that:  

8    

• Provides  all  students  with  a  core  set  of  basic  concepts  and  tools  that  come  from  social  and  behavioral  sciences  and  are  needed  for  policy  analysis    

• Provides  students  who  wish  to  focus  on  social  and  behavioral  sciences  with  a  menu  of  elective  courses  that  are  relevant  across  the  substantive  areas  RAND  covers    

• Can  be  staffed  and  taught  at  a  high  level  of  quality  

To  meet  the  first  objective  above  in  academic  year  2012-­‐13,  we  introduced  a  new,  faculty-­‐developed  course  required  of  all  first  year  students  called  Social  and  Behavioral  Science  Perspectives  (SBS  I).    This  course  is  designed  to  expose  students  to  the  major  theoretical  frameworks  used  in  sociology,  political  science,  psychology,  and  anthropology  which  are  used  most  frequently  in  policy  research  and  analysis.    The  course,  led  by  Chloe  Bird  (a  sociologist)  and  co-­‐taught  with  a  political  scientist,  psychologists  and  anthropologists,  focuses  on  how  these  various  frameworks  can  be  applied  to  tough  policy  problems  in  ways  that  will  yield  important  insights  for  policy  makers.    The  second  course  in  the  SBS  sequence  is  Sandy  Berry’s  foundational  course  on  social  science  research  methods.    We  will  discuss  the  development  of  a  menu  of  corresponding  elective  courses  below.  

Electives  Beginning  in  academic  year  2012-­‐13,  we  have  conducted  a  review  of  our  elective  curriculum  with  the  goal  of  determining  which  electives  will  be  offered  on  a  regular  basis  (either  every  year  or  every  other  year)  and  which  will  be  offered  only  when  student  demand  and  faculty  availability  converge.    We  have  completed  the  review  for  the  electives  in  economics  which  was  led  by  distinguished  RAND  researcher,  and  long-­‐time  faculty  member,  Sue  Hosek  with  the  result  that  we  will  be  offering  the  following  six  courses  on  a  regular  basis:  

• Labor  and  Population  Economics  • Health  Economics    • Public  Economics    • Development  Economics    • Industrial  Organization/Regulation  • Research  Methods  in  Empirical  Economics      

We  are  in  the  final  stages  of  determining  the  appropriate  list  of  courses  for  Empirical  Analysis,  Policy  Analysis  and  Social  and  Behavioral  Sciences.  We  have  currently  identified  the  following  elective  courses  to  be  offered  on  a  regular  basis:  

In  Empirical  Analysis:    

• Advanced  Econometrics  I  &  II  • Operations  Research  II  • Robust  Decision  Making  • Game  Theory  • Survey  Sampling  and  Analysis  I  &  II  

9    

 

In  Social  and  Behavioral  Sciences:  

• Social  Network  Analysis  • Qualitative  Research    • Program  Evaluation  

 

In  Policy  Analysis:  

• Principles  of  Client  Oriented  Policy  Analysis    • Food  Policy  • Healthcare  and  Healthcare  Reform  

 We  are  now  exploring  with  the  faculty  and  FCCA  which  additional  courses  should  be  taught  on  a  regular  basis.  We  anticipate  adding  1  to  2  more  courses  to  each  of  the  non-­‐economic  areas  over  the  next  two  to  three  years.  For  example,  we  believe  there  is  a  need  for  a  course  on  survey  development  and  administration  methodology  in  the  social  sciences  area,  and  a  national  security-­‐related  course  in  the  policy  analysis  area.    Setting  a  multi-­‐year  schedule  for  these  key  electives  will  have  the  benefit  of  providing  students  with  information  they  need  to  plan  their  coursework  for  their  degree.    By  committing  to  offer  a  pre-­‐determined  number  of  courses  each  year,  we  hope  to  free  up  time  and  money  to  offer  more  innovative  courses  as  well.    This  will  benefit  students  by  enabling  us  to  be  more  nimble  in  responding  quickly  to  changes  in  the  field  of  policy  analysis  and  it  will  also  allow  us  to  reach  out  across  RAND  to  bring  new  faculty  into  the  Pardee  RAND  fold.  This  past  year,  we  asked  researchers  at  RAND  to  propose  new  courses  that  they  would  be  interested  in  teaching.  We  received  over  20  nominations  on  which  we  then  surveyed  students  to  gauge  their  interests.  We  anticipate  offering  5-­‐8  of  them  this  coming  academic  year  depending  on  budgetary  constraints.      

RAND-­‐wide  Courses  Unlike  a  traditional  research  university,  the  Pardee  RAND  Graduate  School  is  a  central  part  of  a  global  research  organization—the  RAND  Corporation,  established  to  inform  and  shape  domestic  and  international  policy  decision  making  through  objective  research  and  analysis.  And  unlike  RAND  as  an  educational  institution,  Pardee  RAND  is  not  limited  by  client-­‐driven  issue  interests.    As  a  result,  Pardee  RAND  combines  the  intellectual  freedom  of  cutting-­‐edge  academic  research  within  the  immediacy  and  relevance  of  applied  policy  analysis.  

In  addition  to  offering  traditional  lecture  and  seminar  courses,  we  have  also  invested  in  experimenting  with  developing  and  offering  cross-­‐site  and  multi-­‐disciplinary  courses  that  draw  on  intellectual  resources  from  across  RANDs  multiple  offices.  For  example,  last  academic  year,  we  commissioned  a  RAND-­‐wide  course  on  US  healthcare  reform.  The  course  was  taught  by  6  professors  from  different  RAND  offices  and  was  opened  to  all  RAND  researchers  and  staff.  It  was  the  largest  course  ever  offered  by  Pardee  RAND  and  had  over  40  participants  at  each  of  the  presentations.    

10    

Building  on  the  success  of  the  Healthcare  Reform  course,  we  will  be  offering  two  or  three  RAND-­‐wide  courses  each  year  that  will  be  open  to  RAND  staff  as  well  as  students.    These  courses  will  be  offered  from  various  RAND  locations  and  will  be  accessible  via  video  teleconference  or  other  technology  enabling  as  many  RAND  staff  as  possible  to  audit  or  otherwise  participate  in  this  learning  experience.    Topics  we  are  exploring  for  2014-­‐15  include  Food  Policy,  Security  in  East  Asia,  Big  Data  Policy  Analysis,  and  Nuclear  Security.      

Faculty  Research  and  Scholarship  Critical  to  maximizing  the  unique  nature  of  Pardee  RAND  is  faculty  support.    It  is  essential  that  our  faculty  have  the  support  needed  to  develop  expertise  and  provide  mentorship  in  a  range  of  policy  areas,  particularly  those  that  will  be  critical  in  the  future,  for  which  there  may  not  yet  be  clients  or  project  funding.    Equally  important  is  research  support  for  faculty  who,  often  working  with  our  graduate  students,  are  developing  groundbreaking  analytic  tools  and  methodologies  that  will  help  us  untangle  and  explain  complex  problems.        

Because  of  they  are  nearly  all  members  of  RAND’s  research  staff,  the  Pardee  RAND  faculty  are  remarkably  prolific  in  their  publications.  While  most  of  the  research  and  publications  our  faculty  produce  are  the  result  of  RAND  research  funded  through  traditional  sources  (client  funding  or  RAND  investments),  in  recent  years  the  School  has  raised  funds  to  support  faculty  fellowships  in  a  number  of  issue  areas.    In  academic  year  2013-­‐14,  we  offered  two  such  fellowships:    the  Rosenfeld  Program  in  Asian  Economic  Development  and  the  Brown  Fellowship.    Through  these  faculty  fellowships,  we  are  able  to  support  selected  faculty  member’s  ability  to  do  innovative  research  in  areas  where  there  is  strong  student  interest,  but  RAND  clients  are  not  yet  willing  or  able  to  fund.    For  example,  Professor  Krishna  Kumar  continued  to  receive  support  to  work  with  students  to  build  a  portfolio  of  innovative  work  on  issues  related  to  economic  development  in  Asia.    Professor  Nicholas  Burger  used  the  second  year  of  his  two-­‐year  Brown  Fellowship  to  work  with  students  on  the  climate  implications  of  the  natural  gas  boom  in  the  United  States.  These  opportunities  for  faculty  and  student  support  will  continue  to  increase  over  the  next  several  years  through  the  Pardee  Initiative  for  Global  Human  Progress  (international  development)  and  the  Cazier  Initiative  for  Environmental  and  Energy  Sustainability.    

Opportunities  Beyond  the  Pardee  RAND  Classroom  At  Pardee  RAND,  we  believe  a  premier  public  policy  program  needs  to  reach  beyond  its  own  walls  and  offer  opportunities  to  our  students  to  engage  with  the  outside  world.      

Certificate  in  Legal  Studies  Pardee  RAND  students  have  the  opportunity  to  complement  their  PhD  degree  program  with  a  unique  understanding  of  the  interplay  between  the  law  and  public  policy.  Pardee  RAND  Gradate  School,  in  conjunction  with  Southwestern  Law  School,  offers  a  "Certificate  in  Legal  

11    

Studies."  Commencing  in  2011,  qualified  students  at  Pardee  RAND  can  take  courses  at  Southwestern  covering  many  areas  of  law.  This  is  not  a  joint  degree  program,  but  is  instead  intended  to  provide  participating  students  with  a  basic  knowledge  of  law  and  an  understanding  of  relevant  tools  and  frameworks.    

Currently,  few  public  policy  programs  offer  courses  in  legal  studies.  While  this  gap  is  addressed  to  some  extent  in  MPA/MPP-­‐JD  programs,  they  do  not  meet  the  needs  of  policy  Ph.D.  students  who  wish  to  gain  substantial  exposure  to  law  school  course  work  without  the  requirement  of  an  additional  degree.  No  other  policy  PhD  program  offers  students  this  opportunity  to  address  a  major  gap  in  policy  and  legal  education.    

Over  its  100-­‐year  history,  Southwestern  has  earned  a  reputation  for  the  continuing  involvement  of  its  graduates  in  leadership  positions  in  law  and  policy,  particularly  within  California.  At  present,  for  instance,  Southwestern  alumni  account  for  more  than  75  sitting  Superior  Court  judges,  175  district  attorneys,  100  public  defenders,  and  2  federal  district  court  judges  in  Los  Angeles.  The  partnership  between  the  two  schools  provides  greater  access  to  the  LA  community  and  expands  the  curriculum  at  Pardee  RAND.  The  certificate  program  is  small  and  highly  competitive.  Tuition  is  covered  through  a  reciprocity  agreement.  

Funding  for  Externships  and  Fieldwork  As  described  in  detail  below,  one  of  the  most  unique  aspects  of  this  program  is  the  opportunity  to  work  on  RAND  public  policy  research  projects  as  part  of  the  academic  requirement  of  the  program.    But  we  recognize  that,  in  addition,  our  students  may  benefit  from  the  chance  to  work  at  a  local  health  clinic  or  homeless  shelter,  or  to  do  field  work  for  an  international  organization  or  NGO.    Frequently,  however,  these  organizations  can  provide  them  with  valuable  experiences,  but  cannot  compensate  them  at  the  level  they  require.    Therefore,  to  enable  students  to  pursue  these  opportunities,  we  have  raised  some  funds  which  are  available  to  be  distributed  on  a  competitive  basis.    Funds  from  the  Pardee  Initiative  in  Global  Human  Development  may  also  be  available  to  support  field  work  related  to  improving  the  lot  of  the  poorest  of  the  poor  in  Africa  and  Asia.  

The  Concept  of  On-­‐the-­‐Job-­‐Training  (OJT)  One  of  the  most  distinctive  aspects  of  the  Ph.D.  program  in  policy  analysis  at  Pardee  RAND  is  the  opportunity  for  students  to  work  at  RAND  on  client-­‐oriented  RAND  research  projects.  Through  this  On-­‐the-­‐Job  Training  (OJT),  students  enter  a  community  of  practice,  gaining  professional  skills  and  knowledge  that  courses  alone  cannot  convey.  Our  students  sharpen  their  analytic  skills,  confront  messy  data  sets,  learn  about  team  effort,  and  experience  the  thrill  and  agony  of  searching  for  rewarding  work.  Through  project  work,  they  learn  how  to  work  for  supervisors  with  different  professional  styles,  participate  in  the  writing  and  editing  of  RAND  reports,  gain  an  understanding  of  the  interests  and  constraints  facing  government  

12    

policymakers,  and  garner  invaluable  insight  into  the  challenges  faced  by  senior  RAND  researchers  and  the  clients  and  policymakers  whom  RAND  serves.  No  university  environment  offers  students  this  exposure  to  real-­‐world  policy  research  at  the  same  time  they  are  learning  new  skills  in  the  classroom.     Pardee  RAND  is  not  responsible  for  providing  OJT  for  the  student  but  we  do  provide  support  for  students  in  their  search  for  work  through  guidance,  advice,  and  the  OJT  Brokers,  a  group  of  students  each  assigned  to  a  research  area  who  assist  with  connecting  projects  to  students  and  vice  versa.  Students  search  for  potential  OJT  in  a  variety  of  ways,  including  direct  outreach  to  RAND  researchers,  seminars,  faculty  and  coursework,  and  research  on  the  RAND  intranet.  The  School  sponsors  the  student-­‐run  OJT  Brokerage  that  exists  to  help  students  navigate  the  internal  market  and  provide  advice  on  “best  practices”  for  the  job  search  and  for  interacting  with  researchers.  The  Deans  will  also  help  students  match  their  interests  with  opportunities.  Students  are  encouraged  to  use  all  available  resources  and  to  search  for  OJT  immediately  upon  arriving  at  Pardee  RAND  and  throughout  their  time  as  enrolled  students.      Most  students  work  on  a  variety  of  projects  during  their  time  at  RAND,  giving  them  exposure  to  a  range  of  policy  areas,  research  methods,  colleagues,  and  clients.  By  the  time  they  graduate,  students  accumulate  the  equivalent  of  at  least  two  years  of  job  experience  in  policy  analysis  and  policy  consulting-­‐in  addition,  of  course,  to  their  Ph.D.  degrees.  Ideally,  OJT  also  provides  an  important  part  of  the  foundation  for  the  dissertation  required  of  all  graduating  students.      During  their  tenure  at  Pardee  RAND,  students  find  that  their  value  to  RAND  through  OJT  increases  steadily  with  time.  In  their  first  year,  and  even  early  in  their  second  year,  the  search  for  OJT  may  be  arduous.  Some  students  experience  stress  about  whether  they  will  meet  their  OJT  requirements.  However,  as  the  skills  of  students  increase  and  their  visibility  and  credibility  throughout  RAND  improves,  they  find  more  OJT  opportunities  come  their  way.  This  may  create  a  new  kind  of  stress:  the  need  to  weigh  multiple  offers  or  accept  some  offers  and  decline  others  due  to  the  fact  that  even  fellows  have  limited  hours  in  the  day!  The  capacity  to  graciously  say  "no"  to  less-­‐valued  OJT  opportunities  is  just  as  important  as  the  enterprising  skill  of  generating  OJT.      The  biggest  challenge  that  students  often  face  is  finding  a  way  to  link  OJT  with  fulfillment  of  the  dissertation  requirement.  Although  there  are  many  factors  that  influence  the  quality  of  dissertations,  a  student's  success  at  forging  a  compelling  dissertation  topic  in  the  context  of  OJT  is  certainly  an  important  one.  Even  if  dissertation  research  cannot  be  fully  funded  through  RAND  OJT,  Pardee  RAND  encourages  students  to  look  for  opportunities  to  tie  portions  of  their  dissertations  to  RAND  research  projects.    

13    

 RAND  operates  according  to  an  internal  market  economy  and  OJT  is  no  exception.  Like  other  RAND  researchers,  students  will  find  that  their  interests,  skills,  and  enthusiasm  can  lead  them  to  rewarding  and  diverse  opportunities.  But  as  in  all  markets  of  this  type,  success  depends  on  the  active  participation  and  entrepreneurial  skills  of  all  parties,  including  the  students.  RAND’s  units  vary  in  the  ways  they  manage  their  demand  for  labor.  Some  look  ahead  to  each  fiscal  year  and  make  fairly  firm  plans.  Others  depend  on  a  project-­‐award  process  that  often  permits  only  short-­‐term  planning.      

Learning  Through  On-­‐the-­‐Job  Training  As  described  above,  On-­‐the-­‐job-­‐training  (OJT)  is  an  integral  part  of  the  Pardee  RAND  education  and  a  defining  characteristic  of  the  School,  setting  it  apart  from  other  schools  of  public  policy.    As  they  work  as  team  members  on  RAND  research  projects,  students  gain  invaluable  professional  experience,  apply  research  techniques  learned  in  the  classroom,  and  obtain  in-­‐depth  substantive  knowledge  about  their  chosen  policy  field.    Their  project  work  is  also  the  mechanism  by  which  they  fund  their  graduate  studies.    Given  its  significant  role  within  the  doctoral  program,  Pardee  RAND  recognized  the  need  to  evaluate  whether  and  to  what  extent  OJT  is  serving  as  a  conduit  of  learning.    To  enable  this  evaluation  we  developed  a  formal  system  for  tracking,  understanding,  and  improving  OJT-­‐related  learning.    The  goals  of  the  accountability  system  we  created  as  a  result  are  to:    

(a)  make  explicit  the  expectations  and  responsibilities  for  OJT  learning;    (b)  identify  ways  to  measure  and  monitor  each  student’s  OJT  experience;    (c)  incorporate  the  monitoring  process  into  ongoing  student  assessments  and  annual  

program  assessments;    (d)  establish  mechanisms  for  motivating  students,  OJT  project  leaders  and  others  to  

meet  these  expectations;  and    (e)  ensure  that  students,  project  leaders  and  others  have  the  capacity  to  carry  out  their  

OJT  responsibilities.        We  believe  this  will  improve  the  effectiveness  of  the  OJT  experience  and  enable  us  to  more  directly  shape  this  unique  learning  environment.  

There  are  four  interrelated  requirements  to  ensure  the  functioning  of  this  accountability  system:  

§ To  develop  learning  standards  that  can  evolve  as  the  program  evolves;    § To  continuously  measure  and  monitor  learning;  § To  establish  mechanisms  to  provide  feedback  to  and  receive  feedback  from  students;  

and  § To  use  the  information  gathered  through  these  processes  to  strengthen  the  program.  

14    

 

Evolving  learning  standards      Learning  from  OJT  is  such  a  central  part  of  the  program,  we  decided  we  needed  to  specifically  identify  the  skills  we  expected  students  to  acquire  through  their  project  work  and  then  measure  progress  in  the  acquisition  of  those  specific  skills  for  each  student  throughout  their  time  at  Pardee  RAND.  By  doing  so,  we  hope  to  gain  some  assurance  that  students  are  exposed  to  and  have  the  opportunity  to  improve  the  methodological  and  professional  skills  they  need  to  succeed  in  their  careers.  Insofar  as  there  is  a  gap  between  the  skills  students  desire  and  the  skills  they  develop  through  RAND  project  work,  we  would  like  to  address  this  gap  and  increase  enriching  OJT  opportunities.  Our  ideas  for  how  best  to  address  this  measurement  have  evolved  over  the  past  five  years.  We  will  discuss  the  original  plan,  exploration  of  learning  standards,  and  the  current  thinking.  

Development  of  Core  Learning  Objectives  At  the  time  of  the  last  re-­‐accreditation  report,  we  developed  a  core  list  of  methodological  and  professional  skills  we  expected  students  to  obtain  primarily  through  their  work  on  RAND  research  project  teams.    Through  consultation  with  students  and  faculty,  we  agreed  upon  an  initial  list  of  14  skills  later  expanded  to  19  which  fall  into  five  broad  categories.  (See  the  list  in  the  table  below).      

15    

 

Continuing  to  measure  and  monitor  learning  To  support  the  standards  of  learning  through  OJT,  we  developed  a  system  to  explicitly  measure  and  monitor  OJT  learning.  Gathering  this  information  enables  us  to  both  consider  what  learning  should  be  expected  and  to  gauge  attainment  of  that  learning.        

Initially,  the  only  formal  mechanism  for  monitoring  the  student  OJT  experience  was  through  RAND’s  financial  system,  OASIS,  which  tracks  the  hours  that  students  bill  to  each  project.  While  this  provides  information  on  the  amount  of  work  a  student  is  doing  with  whom  and  on  what  topic,  it  yields  little  information  about  the  quality  of  the  experience  from  either  the  student’s  or  supervisor’s  perspective.  Moreover,  students  are  not  evaluated  in  the  same  manner  as  fulltime  RAND  research  staff.    The  lack  of  a  more  formal  review  process  made  it  difficult  to:  (a)  track  progress  in  learning  over  time  –  either  at  the  individual  or  aggregate  level;  (b)  make  systematic  comparisons  across  students  and  their  cohorts;  (c)  assess  to  what  degree  students  are  fulfilling  the  expectations  of  their  OJT  supervisors  and  to  what  degree  OJT  supervisors  are  meeting  the  learning  needs  of  the  students;  and  (d)  identify  areas  for  quality  improvement  in  the  OJT  experience.  

Category Skills

Proposal  writing

Literature  review

Human  subjects  protection  process

Primary  data  collection  

Secondary  data  collection

Qualitative  data  management

Qualitative  data  analysis

Quantitative  data  management

Quantitative  data  analysis

Modeling

Collaborating

Critiquing  research

Client  relations

Writing

Self-­‐management

Interpersonal  management

Project  and  team  management

Presenting

Publication

Project  Initiation  and  Preparation

Data  collection

Data  management  and  analysis

Project  activities  and  management

Professional  activities

16    

As  part  of  our  last  re-­‐accreditation  review,  a  task  force  composed  of  students,  faculty  and  deans  created  a  web-­‐based  self-­‐assessment  tool  through  which  Pardee  RAND  students  report  annually  what  and  how  much  they  have  learned  through  their  OJT  experiences.    A  preliminary  version  of  the  survey  was  developed  and  pilot  tested  on  a  select  group  of  students  in  May  2009.    The  assessment  tool  was  subsequently  revised  based  on  user  feedback  and  administered  to  the  entire  student  body  in  December  2009,  with  questions  pertaining  to  OJT  completed  in  the  prior  fiscal/academic  year  (October  2008-­‐September  2009).  The  second  annual  self-­‐assessment  was  administered  in  October  2010,  which  gathered  data  on  OJT  completed  in  FY2010.  Surveys  have  been  fielded  annually  since  that  time.    We  now  have  data  for  five  years  of  students  that  allows  us  to  begin  to  see  patterns  and  make  assessments.      

We  have  compiled  and  begun  to  analyze  the  data  at  both  the  program  and  individual  levels.  At  the  program  level,  for  example,  we  are  now  able  to  begin  to  address  the  following  issues:    

• What  kinds  of  learning  opportunities  (in  terms  of  core  skills  and  substantive  areas)  do  students  report  being  consistently  exposed  to  during  their  OJT  experiences?    

• What  types  of  learning  are  missing  or  at  lower  levels  than  expected?  

• How  much  do  students  report  having  learned  in  each  of  the  core  skill  sets  and  substantive  areas?  

• How  are  the  kinds  and  amounts  of  learning  opportunities  distributed  across  OJT  projects  and  across  students?  For  instance,  to  what  degree  are  high-­‐learning  projects  concentrated  among  few  students  or  a  few  projects  or  does  everyone  have  at  least  a  few  very  good  learning  opportunities?  

The  chart  below  provides  an  illustration  of  how  the  students  rated  their  OJT  learning  experience  in  academic  years  2011-­‐12  (FY12)  and  2012-­‐13  (FY13).    In  FY12  over  60%  of  the  97  students  who  filled  out  the  survey  reported  significant  learning  in  self-­‐management,  writing,  collaborating  and  quantitative  data  analysis.    At  the  other  end  of  the  scale,  fewer  than  30%  reported  learning  in  the  area  of  human  subjects  protection  or  proposal  writing.  In  FY13  over  60%  of  the  79  students  who  filled  out  the  survey  reported  learning  of  “a  fair  bit”  or  “a  lot”  in  collaborating,  writing,  quantitative  data  analysis,  and  quantitative  data  management.  Fewer  than  30%  reported  significant  learning  in  client  relations,  proposal  writing,  and  human  subjects  protection.  

   

17    

 

2012  

 

2013  

 

 

Collecting  the  data  each  year  allows  us  to  make  comparisons  and  begin  to  see  trends.    For  instance,  we  are  able  to  see  whether  the  list  of  skills  that  students  rated  as  providing  “a  lot”  or  “a  fair  bit”  of  learning  is  consistent,  or  not,  over  time.    Comparing  fiscal  years  2009  through  2013  we  see  that  some  skills  provided  high  quality  learning  experiences  across  most  years,  

18    

while  others  changed  in  rank.    The  chart  below  shows  the  “top  five”  skills  students  rated  each  of  the  past  five  years  as  providing  the  most  learning.  

Top  Five  Tasks  by  Learning  FY09   FY10   FY11   FY12   FY13  

Quant.  Data  Analysis   Collaboration   Inter-­‐personal   Self-­‐Management   Collaborating  

Quant.  Data  Mgmt   Lit  Review   Collaboration   Collaboration   Writing  

Writing   Writing   Writing   Writing  Quant.  Data  Analysis  

Collaboration  Quant.  Data  Analysis   Self-­‐Management  

Quant.  Data  Analysis  

Quant.  Data  Management  

Secondary  Data  Coll.   Self-­‐Management  

Lit  Review/Quant.  Data  Mgmt  

Lit  Review/Quant.  Data  Mgmt   Self-­‐management  

 

From  this  list  we  can  ascertain  that  in  aggregate  students  are  consistently  learning  on-­‐the-­‐job  how  to  write,  collaborate  and  manage  their  workloads.    They  are  also  gaining  experience  in  data  management  and  analysis,  although  they  report  much  more  exposure  to  quantitative  data  than  to  qualitative  data.    Literature  reviews  are  consistently  listed  as  one  of  the  primary  vehicles  for  learning,  particularly  early  in  the  program.  

Equally  important  from  the  School’s  perspective  is  to  understand  what  skills  students  are  consistently  ranking  in  the  “bottom  five”  in  terms  of  providing  students  with  opportunities  to  learn.      

Bottom  Five  Tasks  by  Learning  FY09   FY10   FY11   FY12   FY13  

Proposal   Human  Subjects   Human  Subjects   Human  Subjects   Modeling    

Client  Relations   Client  Relations   Proposal   Proposal    Primary  Data  Coll.  

Qual.  Data  Mgmt.   Proposal   Client  Relations   Client  Relations   Client  Relations  

Critiquing  Research   Qual.  Data  Analysis  Qual.  Data  Mgmt.   Primary  Data  Coll.   Proposal    

Presenting   Qual.  Data  Mgmt.   Modeling   Critiquing  Research   Human  Subjects  

 

From  this  list  we  can  clearly  see  that  in  aggregate  over  the  past  five  years  students  have  not  been  learning  much  about  the  human  subjects  protection  process,  they  tend  not  to  be  given  much  responsibility  on  proposals,  and  they  have  fewer  opportunities  to  engage  with  clients.    Moreover,  while  they  have  ample  opportunities  to  learn  quantitative  data  analysis  and  management,  they  have  fewer  opportunities  to  learn  by  gathering  primary  data  and  managing  and  analyzing  qualitative  data  on  RAND  projects.      

These  results  are  not  surprising.    It  is  perhaps  to  be  expected  that  project  leaders  do  not  delegate  critical  tasks  at  the  beginning  (proposal  writing,  human  subjects  protection,  client  relations)  and  end  of  projects  (client  relations,  presenting,  critiquing  and  publication)  to  the  most  junior  members  of  the  project  team.    We  hope  that  at  some  point  during  their  tenure  at  

19    

Pardee  RAND  each  student  will  have  the  opportunity  to  have  exposure  to  these  aspects  of  a  project,  but  we  recognize  that  in  some  cases,  the  only  instance  they  will  have  to  complete  some  of  these  tasks  on  their  own  will  be  in  work  on  their  dissertation.  

Providing  Feedback  to  Students  The  data  we  have  collected  provides  us  with  information  which  allows  us  to  address  the  following  issues  at  the  level  of  the  individual  student  at  each  year  in  the  student’s  Pardee  RAND  career:  

• What  are  each  student’s  strengths  and  weaknesses  in  terms  of  their  core  learning  skills  (as  described  by  their  own  reports  of  proficiency)?      

• To  what  degree  is  the  student  making  progress  toward  augmenting  their  skills  sets  (breadth)?  Or  are  they  building  on  their  existing  strengths  (depth)?    

• To  what  degree  has  a  student’s  OJT  experience  allowed  them  to  learn  more  about  their  area  of  policy  specialization?  

 One  visualization  of  individual  student  learning  is  the  radar  graph.  Here  we  graphed  each  student’s  reported  learning  in  particular  skill  areas.  A  zero  indicates  no  learning,  a  1  indicates  a  little  bit  of  learning,  a  2  indicates  a  fair  bit  of  learning,  and  a  3  indicates  a  lot  of  learning.  A  negative  1  indicates  the  student  was  not  exposed  to  that  particular  skill.  These  learning  responses  are  in  the  outer  line,  coded  with  color  by  skill  area.  The  purple  line  indicates  with  a  1  or  a  0  whether  or  not  the  student  indicated  a  desire  for  more  learning  in  a  particular  skill  area  in  the  survey.  Two  examples  of  anonymized  radar  graphs  from  actual  students  are  below.  

 

 

The  student  in  example  A  has  had  a  fair  bit  of  learning  or  better  in  every  area  except  human  subjects  protection.  However,  this  student  also  wants  to  be  exposed  to  more  learning  in  the  skills  on  the  left  half  of  the  graph,  many  of  which  correspond  to  later  stages  of  research.  The  

20    

student  in  example  B,  on  the  other  hand,  has  had  few  opportunities  of  exposure  and  also  has  a  desire  for  exposure  in  several  areas  in  which  he  or  she  reported  no  exposure  previously.    

Review  of  Performance  Pardee  RAND  conducts  First  Year  Reviews  for  every  student  at  the  end  of  their  first  year.    Each  student  is  evaluated  based  on  their  performance  in  the  classroom,  on  qualifying  exams  and  on  OJT.    They  provide  a  self-­‐assessment  of  their  performance  and  we  receive  feedback  from  their  OJT  supervisors  who  provide  information  on  the  quality  of  their  work,  whether  they  showed  improvement  over  time,  if  they  would  hire  them  again  and  how  they  would  assess  the  chances  that  they  will  successfully  complete  the  PhD.    There  is  also  a  space  for  comments  which  most  OJT  supervisors  use  to  more  fully  describe  the  student’s  strengths  and  weaknesses  on  project  work.    At  the  end  of  the  review,  each  student  receives  a  memo  from  the  First  Year  Review  Committee  which  includes  an  evaluation  of  their  first  year  OJT  performance.  

Students  beyond  the  first  year  have  been  encouraged  to  seek  direct  feedback  from  their  OJT  supervisors  at  regular  intervals,  and  project  leaders  will  from  time  to  time  reach  out  to  discuss  situations  where  students  are  performing  consistently  above  or  below  expectations.    But  the  School  has  not  conducted  formal  reviews  of  OJT  performance  beyond  the  first  year  until  now.    Beginning  in  September  2014,  we  will  be  conducting  a  Second  Year  Review  which  will  include  an  evaluation  of  second  year  performance  in  the  classroom,  on  OJT,  and  assess  progress  towards  the  dissertation.    In  addition  to  seeking  input  from  OJT  supervisors,  we  will  also  be  using  the  graphs  above  for  each  individual  student  (generated  from  their  first  year  data)  as  starting  point  for  a  discussion  about  what  the  student  would  like  his  or  her  graph  to  look  like  when  they  graduate  and,  if  needed,  how  to  address  deficiencies  as  in  example  B.      

In  January  or  February  2015,  we  will  be  conducting  a  Third  Year  Review  looking  at  progress  to-­‐date  by  students  who  will  just  be  beginning  their  fourth  year  in  the  program.    By  the  time  of  this  Third  Year  Review,  we  will  have  accumulated  three  years  of  information  and  have  a  good  understanding  of  the  student’s  academic  performance  and  progress  toward  a  dissertation,  as  well  as  the  quality  of  their  OJT  work.  These  three  pieces  together  can  give  the  deans  a  sense  of  whether  the  student’s  Pardee  RAND  experience  as  a  whole  is  meeting  his  or  her  needs  and  meeting  performance  and  timeline  expectations.  

Using  Data  to  Strengthen  the  Program  One  of  our  goals  in  gathering  these  data  was  to  use  it  to  make  improvements  in  the  program.    As  noted  above,  we  recognized  that  students  were  not  getting  as  much  exposure  to  clients  as  we  want  them  to  have.    In  part,  this  is  because  the  majority  of  RAND  clients  are  in  the  Washington,  D.C.  area  and  most  of  our  students  are  located  in  Santa  Monica.      

21    

To  help  students  bridge  the  gap,  in  FY2014  with  RAND  investment  funds,  we  piloted  a  program  offering  short-­‐term  travel  grants  to  students  to  travel  for  up  to  a  week  to  RAND  offices  in  either  DC  or  Pittsburgh  to  build  connections  to  RAND  researchers  for  OJT  and/or  dissertations.    We  encouraged  students  to  use  this  funding  to  seek  out  opportunities  to  interact  with  clients  wherever  possible.    Thus  far  we  have  supported  trips  by  16  students  with  4  going  to  Pittsburgh  and  12  going  to  DC.    Students  used  these  funds  for  a  variety  of  activities,  including  client  meetings,  joining  project  team  members  for  intensive  working  sessions,  networking  with  researchers  in  person  to  obtain  new  OJT  opportunities,  and  discussing  dissertation  plans  with  existing  or  potential  committee  members.    We  are  building  these  funds  into  our  core  budget  for  next  year  and  will  continue  to  provide  support  for  travel  for  these  purposes.      

We  also  believed  we  could  incentivize  faculty  to  provide  more  mentoring  to  students  on  their  OJT  projects.    Our  reliance  on  RAND  for  Pardee  RAND  faculty  is  one  of  our  greatest  strengths.    It  allows  us  to  recruit  teaching  faculty  that  are  truly  committed  to  teaching.    And  it  enables  us  to  include  On-­‐the-­‐Job-­‐Training  as  an  integral  component  of  our  academic  program.    The  custom  and  mechanism  for  paying  teaching  faculty  is  well-­‐established.    But  until  FY2013  we  had  not  compensated  mentoring  faculty  (those  who  take  on  the  responsibility  of  turning  inexperienced  students  into  peers  by  working  with  them  for  a  significant  period  of  time  on  RAND  research  projects).    In  FY2013  we  used  RAND  investment  funds  for  a  pilot  program  for  faculty  mentors.    This  program  provides  all  first  year  students  with  3  days  to  work  with  a  researcher  of  their  choice.    The  researchers  receive  up  to  3  days  of  coverage  for  mentoring  students.    In  FY2014  we  absorbed  this  program  into  our  Student  Support  budget.    Overall,  this  program  appears  to  have  been  successful  at  providing  first  year  students  more  substantial  project  work.    We  will  be  surveying  the  students  to  better  understand  whether  they  are  also  receiving  the  mentorship  we  anticipate.  

Creating  Universal  Norms  for  OJT  Learning?  Since  2010  we  have  continued  to  monitor  and  analyze  exposure  of  students  to  the  19  core  skills.  The  self-­‐assessment  surveys  have  given  us  information  about  the  learning  most  students  receive  in  these  areas.    We  have  discussed  whether  or  not  we  want  to  create  a  universal  norm  or  benchmark  for  each  of  the  skills  that  we  expect  every  student  in  the  program  to  obtain.    We  took  this  idea  to  the  Faculty  Committee  on  Curriculum  and  Appointments  (FCCA)  for  discussion.    The  outcome  was  an  emphatic  rejection  by  the  FCCA  of  the  idea  of  having  universal  norms,  even  in  broad  categories.    The  FCCA  pointed  out  several  challenges  to  imposing  universal  norms.  First,  Pardee  RAND  students  have  diverse  backgrounds  and  varied  skills  before  beginning  the  program.  Researchers  sometimes  seek  out  particular  students  with  pre-­‐existing  skills  for  work  on  OJT.  The  faculty  did  not  wish  to  either  penalize  students  who  come  in  with  a  high  level  of  understanding  or  give  them  an  unfair  advantage.  Handling  this  kind  of  discrepancy  is  doubly  hard,  since  it  would  probably  entail  some  sort  of  evaluation  other  than  the  student’s  own  assessment  of  his  or  her  mastery.  Second,  although  a  core  set  of  skills  used  in  OJT  does  exist,  each  student  may  have  a  different  level  of  need  or  desire  for  certain  specific  skills.  There  was  little  consensus  on  what  would  be  a  minimum  level  of  learning  in  particular  areas.  Finally,  if  

22    

norms  were  set  for  OJT,  then  the  responsibility  for  assigning  projects  would  need  to  be  shared  between  the  administration  and  students  to  ensure  fair  access  to  opportunities  for  skill  improvement.  Such  a  system  would  no  doubt  be  complicated  and  also  quite  different  from  the  way  the  rest  of  the  RAND  labor  management  operates.  It  would  also,  potentially,  detract  from  one  of  the  advantages  of  OJT:  encouraging  students  to  network  and  develop  opportunities  on  their  own.  

Due  to  the  incredibly  varied  backgrounds,  interests,  and  future  goals  of  the  Pardee  RAND  student  body,  the  FCCA  and  the  administration  have  chosen  to  pursue  an  individualized  strategy  for  applying  learning  standards  to  OJT  work.  We  will  continue  to  review  responses  to  the  self-­‐assessment  on  an  individual  level  and  at  the  program  level.  In  the  case  of  the  individual  level,  we  will  use  these  data  to  initiate  a  conversation  with  each  student.  These  conversations  about  OJT  learning  will  occur  primarily  in  the  second-­‐year  and  third/fourth  year  review  meetings  which  we  discussed  earlier  in  this  document.  

OJT  as  Preparation  for  the  Dissertation  In  addition  to  providing  exposure  to  these  aforementioned  19  skills,  OJT  can  open  up  avenues  for  students’  dissertations.    It  can  enable  them  to  learn  about  new  datasets  and  analytic  techniques  not  covered  in  the  coursework  and  to  meet  and  work  with  researchers  in  the  broader  RAND  community  who  may  join  their  dissertation  committees.    In  the  OJT  survey,  we  ask  students  about  ten  such  opportunities.    Students  reported  accessing  additional  opportunities  from  a  higher  proportion  of  OJT  projects  in  FY2012  than  in  FY2013.  More  than  a  third  of  students  reported  that  over  50%  of  their  projects  aided  with  dissertation  data,  provided  a  dissertation  topic,  dissertation  research  questions,  methods  or  subject  knowledge  in  FY2012.  In  contrast,  in  FY2013  the  same  proportion  of  students  reported  that  more  than  half  of  their  projects  helped  in  only  two  areas:  developing  professional  skills  and  developing  career  qualifications.    Prior  to  FY2013,  the  survey  responses  looked  very  similar  to  FY2012.    For  FY2013  we  saw  a  dramatic  increase  in  students  reporting  no  learning  and  a  dramatic  decrease  in  students  reporting  significant  learning.    We  do  not  yet  believe  this  change  indicates  a  trend,  but  we  are  looking  at  the  data  at  a  more  granular  level  and  talking  with  students  to  try  to  better  understand  why  we  see  such  a  change.    We  will  also  monitor  this  in  our  FY2014  survey.  

 

23    

 

2012  

 

 

2013  

 

 

24    

Learning  Objectives  and  the  Dissertation  

Monitoring  and  Measuring  Dissertation  Quality  The  dissertation  is  the  culmination  of  the  educational  process.    As  such,  we  expect  each  dissertation  to  demonstrate  the  author’s  mastery  of  both  a  policy  issue  area  and  the  analytic  techniques  appropriate  to  conduct  a  policy  analysis  in  this  issue  area  given  the  state  of  development  of  the  data  and  what  is  known  about  the  issue.    We  also  expect  all  dissertations  to  make  a  contribution  to  the  policy  literature.    As  part  of  this  Program  Review  we  are  investigating  to  what  extent  students  are  demonstrating  mastery  in  their  dissertations.      

But  before  we  addressed  the  outcomes  of  the  dissertation  process,  we  first  needed  to  refine  and  clarify  dissertation  requirements  and  quality  standards,  and  to  devise  better  procedures  for  tracking  student  progress  through  the  program.    In  response  to  repeated  requests  from  students  and  faculty  for  clearer  guidance  on  what  was  expected  at  each  stage  of  the  dissertation  process,  we  have  broken  down  the  process  into  ten  discrete  steps,  provided  more  detailed  guidance  in  our  Student  Handbook  on  what  is  expected  at  each  stage,  and  devised  a  “stop  light”  chart  to  help  students  navigate  their  way  through  the  process.  

Our  goal  is  to  enable  students  to  complete  all  degree  requirements  including  the  dissertation  in  no  longer  than  5  years.    In  thinking  through  how  to  improve  the  quality  of  Pardee  RAND  dissertations  and  to  facilitate  timely  completion,  the  faculty  and  administrative  staff  have  decided  the  primary  levers  we  have  are  the  following:  

• Provide  more  time:    get  students  started  earlier  and  keep  them  on  track  • Provide  more  mentoring:  create  structure  so  students  don’t  get  lost  and  get  committees  

on  board  with  these  structures  • Provide  more  funding:  to  enable  more  creative  dissertations,  beyond  the  constraints  of  

client-­‐sponsored  work.    

To  provide  more  time  in  the  program  for  students  to  focus  on  their  dissertations,  in  academic  year  2006-­‐7  the  core  curriculum  was  consolidated  into  the  first  year.      Comprehensive  exams  are  now  held  in  July  after  the  first  year.    This  enables  students  to  focus  their  academic  preparation  on  those  subjects  and  methodologies  that  they  will  need  to  master  in  order  to  employ  in  their  dissertation  research  beginning  in  their  second  year.  

To  create  structure  and  increase  mentoring  along  the  way,  we  have  devised  a  dissertation  process  [laid  out  in  the  flow  chart  below]  with  various  points  of  check-­‐in  and  intervention  either  by  the  administration,  by  the  dissertation  committee,  by  other  faculty  or  some  combination.    Beginning  in  the  second  year,  students  are  required  to  participate  in  one  of  the  three  monthly  dissertation  workshops.    These  non-­‐credit  workshops  are  constituted  along  analytic  method  

25    

lines  (economics,  empirical  analysis  or  social  and  behavioral  science)  and  students  elect  to  attend  one  depending  upon  the  primary  methodology  they  will  use  in  their  dissertation.    The  purpose  of  the  dissertation  workshop  is  to  provide  students  with  a  more  concrete  understanding  on  how  to:  consider  topics,  develop  and  focus  a  researchable  question,  form  a  dissertation  committee,  identify  data  sources,  identify  funding  possibilities,  write  a  proposal,  and  choose  to  write  a  classic  monograph  or  follow  a  three-­‐essay  format.    Moreover,  we  believe  that  having  students  go  through  the  process  as  a  group  can  provide  positive  peer  effects  and  build  camaraderie.  

 

Figure  1.  Dissertation  Process  

 

 

 

Students  must  be  enrolled  in  a  dissertation  workshop  even  beyond  the  second  year  until  the  instructor  informs  the  Registrar  that  the  student  has  met  the  following  minimum  requirements:    

• submited    a  six-­‐slide  briefing  articulating  the  policy  issue,  research  questions,  motivation,  method  and  approach,  feasibility,  and  a  personalized  work  plan  for  the  dissertation;    

• submited  a  two-­‐  to  three-­‐page  paper  covering  the  items  on  the  slides  in  slightly  more  detail;  and    

• identifed  a  chair  for  their  dissertation  committee.      

Second  Year  Review

First  Year  Course  Work

OJT

6  Slide  Briefing  &  Committee  

Chair

Dissertation  Research

OJT Proposal  Defense

Sign-­‐off  by  Dissertation  Workshop  Leader

Dissertation  Draft  to  

Committee

Dissertation  Research

OJT

Dissertation  Research

OJT Dissertation  Defense Dissertation  

Submission

Cmte  Sign-­‐off  &  Proposal  Eval.

Cmte  Sign-­‐off  &  

Dissertation  Eval.

         Dissertation  Awards

2nd  Year  Electives  

Dissertation  Workshop

OJT

First  Year  Review      

26    

We  find  that  these  three  requirements  help  motivate  students  to  start  considering  (and  ultimately  making)  key  decisions  regarding  what  questions  their  dissertation  will  address  and  how  they  will  go  about  answering  such  questions.    

The  program  has  also  introduced  revised  eligibility  requirements  for  proposing  a  dissertation  to  ensure  that  each  student  is  well-­‐grounded  in  the  tools  of  policy  analysis  and  has  a  solid  understanding  of  their  chosen  policy  field.    To  schedule  a  proposal  defense,  students  must  show  that  they  have:  

• Satisfactorily  completed  all  core  courses  

• Satisfactorily  completed  first  year  review  • Declared,  but  not  necessarily  completed,  an  analytic  concentration  (optional)  • Completed  a  policy  area  specialization  which  consists  of:  

§ At  least  three  substantive  policy  seminars  § At  least  50  days  of  OJT  in  their  chosen  policy  area  § At  least  one  independent  study  in  their  policy  area  

• Successfully  completed  the  dissertation  workshop  • Obtained  tentative  approval  from  their  committee  of  the  proposal  draft  

 We  believe  if  we  can  get  students  to  successfully  defend  their  dissertation  proposals  within  their  third  year,  they  are  more  likely  to  complete  the  program  within  five  years.    We  also  believe  that  if  students  successfully  defend  their  dissertation  proposals,  those  that  take  a  leave  of  absence  are  far  more  likely  to  complete  their  degrees.    We  will  be  looking  at  our  data  over  the  coming  years  to  see  if  this  is,  in  fact,  true.    Another  tool  we  have  devised  to  assist  students  and  faculty  to  understand  the  expectations  of  the  program  is  the  Dissertation  Timeline  (aka  “stoplight  chart”).    As  you’ll  see  below,  the  chart  conveys  visually  at  which  point  in  the  program  students  should  be  reaching  each  milestone  from  completing  their  dissertation  workshop  to  notification  of  their  full  committee  to  submitting  their  final  dissertation.    At  any  given  academic  quarter,  students  can  clearly  see  what  we  expect  them  to  have  accomplished  on  their  dissertations  and  can  ascertain  whether  they  are  making  satisfactory  progress  (green),  are  falling  behind  (yellow)  or  may  be  subject  to  an  intervention  (red).    This  chart  also  provides  the  Pardee  RAND  administration  a  mechanism  to  easily  track  each  student’s  progress  and  intervene  early  if  a  student  begins  to  go  off-­‐track.      

27    

 

To  provide  additional  incentives  towards  timely  completion  of  each  milestone,  we  have  linked  funding  for  conference  travel  and  dissertation  awards  to  progress  towards  completion.    Students  who  are  on  track  are  given  priority  for  this  funding.    Conversely,  students  who  are  deemed  not  to  be  making  satisfactory  progress  may  be  denied  the  opportunity  to  apply  for  funding  or  may  have  access  to  funding  temporarily  suspended  until  they  get  back  on  track.  

As  noted  above,  students  are  not  guaranteed  enrollment  beyond  the  fifth  year  in  the  program.    They  must  petition  the  Dean  and  develop  a  concrete  work  plan  and  timeline  for  completion.    There  is  also  a  small  financial  disincentive  built  into  the  program  as  even  if  they  are  approved  for  their  sixth  year  or  beyond,  tuition  increases  by  $1000  per  quarter  above  the  cost  for  students  in  Years  3-­‐5.  

The  third  lever  we  have  developed  to  promote  quality  dissertations  that  can  be  completed  in  a  timely  manner  is  by  increasing  the  amount  of  funding  we  have  available.    As  stated  earlier  in  this  document,  the  Graduate  School  has  made  great  strides  in  securing  funds  to  support  students’  work  on  dissertations.    While  many  Pardee  RAND  dissertations  continue  to  be  funded  by  and  closely  linked  to  RAND  project  work,  beginning  in  2002  with  a  multi-­‐year  $350,000  pledge  from  a  member  of  the  Pardee  RAND  Board  of  Governors,  Pardee  RAND  began  to  provide  dissertation  awards  on  a  competitive  basis  with  the  goal  of  catalyzing  superb  policy  research  on  some  of  the  most  intractable  problems  facing  our  nation  and  the  world.    In  2004,  Pardee  RAND  awarded  three  dissertation  fellowships  of  $42,500  each,  for  a  total  of  $127,500.    In  2013-­‐14,  a  total  of  $368,000  was  awarded  to  twenty-­‐three  students  in  grants  ranging  from  $5000-­‐$42,000.    We  expect  to  award  a  comparable  amount  in  2014-­‐15.  

Dissertation  Quality  Assessment  We  believe  that  the  dissertation  and  the  process  by  which  students  produce  it  are  critical  components  of  Pardee  RAND’s  learning  objectives.    We  chose  to  focus  our  assessment  of  dissertation  quality  on  the  explicit  milestones  directly  connected  to  the  dissertation  process  including  dissertation  workshops,  mentoring,  and  clearer  communication  of  standards.    Working  directly  with  faculty  and  students,  our  initial  efforts  were  relatively  simple  and  straightforward  changes  to  existing  processes  and,  as  described  above,  involved  establishing  multiple  means  of  communicating  information  about  these  processes  to  students,  faculty  and  

28    

other  stakeholders.    This  included  establishing  more  explicit  objectives  and  requirements  for  the  dissertation  workshops  and  clarifying  the  dissertation  requirements  for  all  cohorts  of  students  as  described  above.    The  final  and  largest  task  was  to  establish  explicit  dissertation  standards  for  both  the  process  of  completing  the  dissertation  and  the  final  product.  

Creation  of  Explicit  Pardee  RAND  Quality  Standards  As  part  of  the  Accreditation  Process,  we  worked  with  Pardee  RAND  faculty  and  students  to  refine  our  existing  standards  and  to  create  two  new  and  related  tools  for  the  dissertation  committees  to  use  to  assess  dissertation  quality.  The  result  was  the  Dissertation  Proposal  Quality  Assessment  Tool  and  the  Dissertation  Quality  Assessment  Tool.    [see  appendix].    Both  were  fully  deployed  in  academic  year  2011-­‐12.    The  quality  assessment  tools  are  sent  to  dissertation  committees  in  advance  of  the  proposal  and  dissertation  defenses  and  then  at  each  proposal  defense  and  dissertation  defense  the  committees  are  required  to  fill  out  the  one-­‐page  evaluation  sheet  indicating  whether  the  student  passed,  passed  with  conditions,  or  failed  the  defense.    They  are  also  asked  to  use  this  tool  to  provide  the  committee’s  evaluation  of  the  student’s  progress  on  each  included  standard.      

The  Proposal  Defense  Evaluation  Tool  asks  the  committee  members  to  rate  the  student  across  nine  standards  in  three  categories:    research  goals;  research  methods  and  background  and  preparation.    The  Dissertation  Quality  Assessment  Tool  is  more  comprehensive  consisting  of  twenty-­‐two  questions  relating  to  eighteen  standards.    In  both  cases,  members  of  the  dissertation  committee  are  asked  to  arrive  at  a  consensus  assessment  at  the  time  the  student  offers  his  or  her  dissertation  defense.  

Both  tools  have  been  in  place  for  nearly  three  years.    While  originally  conceived  as  a  method  to  uncover  weaknesses  in  the  curriculum  or  in  individual  student’s  preparations,  we  have  found  that  instead  the  tools  are  most  useful  in  conveying  to  committee  members  what  Pardee  RAND  wants  them  to  focus  on  at  each  stage  of  the  process.    These  two  assessment  tools  have  created  a  constructive  dialogue  between  the  student  and  his  or  her  committee.    Furthermore,  and  perhaps  most  importantly,  these  assessments  have  been  used  to  inform  students  and  faculty  of  what  Pardee  RAND  desires  all  students  to  accomplish  through  the  dissertation  process.  

Looking  Ahead  Now  that  we  have  systems  in  place  to  monitor  and  measure  progress  towards  the  dissertation  and  the  quality  of  the  products,  we  are  beginning  to  gather  data  which  we  will  use  in  evaluating  how  well  individual  students  and  the  program  as  a  whole  are  doing.    In  particular,  we  are  asking  ourselves:  

1. Are  the  evaluation  tools  (for  proposal  and  final  dissertation)  giving  us  meaningful  data?  2. Is  the  average  time  to  degree  completion  improving?  3. Are  we  seeing  any  increase  in  students  publishing  their  dissertations?  

29    

4. Are  there  program  changes  we  should  be  making  as  a  result  of  what  we’ve  learned?    

Below  is  a  list  of  our  most  recently  completed  dissertations.  

 

Student  Achievement  As  benchmarks  of  quality,  we  also  track  student  achievement  in  terms  of  time  to  completion,  initial  job  placements  and  career  trajectories  and  publications.    We  include  some  recent  data  in  each  category  below.  

Time  to  Completion  As  of  2013,  the  average  time  for  completion  was  just  under  5  years  (59  months).  

Throughout  its  history,  Pardee  RAND  has  maintained  a  higher  5-­‐year  completion  average  than  other  Ph.D.  programs.  Data  from  a  recent  study  by  the  Ph.D.  Completion  Project  of  the  Council  of  Graduate  Schools  showed  an  average  5-­‐year  completion  rate  by  discipline  between  a  low  of  11.8%  in  the  humanities  to  a  high  of  34.5%  in  engineering.  Pardee  RAND  5-­‐year  completion  rates  are  over  40%.  The  average  time  to  completion,  for  students  who  graduate,  is  about  5  years.  And  over  the  past  10  years,  our  overall  completion  rate  is  74%.  Our  withdrawal  rate  averages  about  12%  of  all  matriculated  students  over  that  same  time  period.  

First Name Last Name PhD Year Diss. Title

Lisa Klautzer 2013Can economic openness inspire better corporate governance? - An exploration of the link between openness and corporate governance based on the Asian experience

Ruopeng An 2013 Eating Better for Less--Effectiveness of Financial Incentives in Modifying Dietary and Grocery Shopping Behavior

Benjamin Bryant 2013Monopoly and Micro-irrigation in Smallholder Water Markets: Using exploratory modeling to consider interactions between market structure and agricultural technology subsidies

James Burgdorf 2013 Labor Market Outcomes of Health Shocks and Dependent Coverage ExpansionStephanie Chan 2013 Fighting Obesity in the United States with State Legislation

Amber Didier 2013The Influence of Contact with Children, Contact with Healthcare Professionals, and Age on Influenza Vaccine Uptake

Kay Faith 2013 Patterns of Creation and Discovery: An analysis of defense laboratory patenting and innovationEileen Hlavka 2013 Policy Impacts on Wind and Solar Innovation: New Results Based on Article CountsSeo Yeon Hong 2013 Three Essays on Child Labor and Education in Developing CountriesDavid Johnson 2013 Improving Flood Risk Estimates and Mitigation Policies in Coastal Louisiana Under Deep UncertaintyRussell Lundberg 2013 Comparing Homeland Security Risks Using a Deliberative Risk Ranking MethodologyTewodaj Mengistu 2013 Emerging Infrastructure Financing Mechanisms in Sub-Saharan AfricaHaralambos Theologis 2013 Capacity Management and Changing Requirements: Cost Effective Decision Making in an Uncertain WorldXiao Wang 2013 The Role of Economic Development Zones in National Development Strategies: The Case of China

Alessandro Malchiodi 2013Three Essays on Education Policy - Empiricial Analyses of the Challenges and Opoortunities with For-Profit Colleges, Military Enlistment and Immigration

Daniel Waxman 2013The Impact of Tort Reform, Medicare Plan Choice, and Geography on Health Care Processes, Outcomes, and Expenditures

Matthew Hoover 2014 It Takes a Village: Network Effects on Rural Education in AfghanistanSarah Kups 2014 Topics in Migration ResearchChristopher Lau 2014 New Medical Technology Development and Diffusion: Policy Challenges and ConsiderationsJesse Sussell 2014 Changing Constituencies and Rising Polarization in the Congress: Three EssaysSteven Isley 2014 Evaluating the Political Sustainability of Emission Control Policies in an Evolutionary Economics SettingChristopher McLaren 2014 Reducing the Economic Burden of Work-Related Injuries Caroline Tassot 2014 Three Essays on Subjective Well-beingMichael Scarpati 2014 Designing Efficient Systematic Reviews for Improved Synthesis of Medical EvidenceEthan Scherer 2014 Three Essays on Education Reform in the United States

30    

 

 

Career  Development    Another  check  on  the  quality  of  the  program  is  the  types  of  jobs  students  are  able  to  obtain  directly  after  completing  the  program.    To  assist  students  in  their  job  search  we  established  in  2009  the  Office  of  Career  Development  which  provides  comprehensive  support  to  doctoral  fellows  throughout  their  time  at  the  institution.  The  office  exists  to  help  fellows  explore  the  many  paths  open  to  them  and  refine  their  career  goals  proactively,  so  that  they  are  ready  to  enter  the  job  market  with  a  focused  approach  to  the  career  search.  Career  Development  provides  counseling,  interview  coaching  and  resume  preparation  as  well  as  a  variety  of  events  to  help  familiarize  fellows  with  the  various  options  available  to  them  upon  graduation.  Career  talks  from  visiting  alumni  and  recruiting  events  happen  at  least  monthly  during  the  academic  year;  other  skills  workshops  are  held  regularly.    

The  office  is  staffed  by  an  experienced  senior  consultant  active  in  the  field  of  executive  recruiting,  and  a  partner  at  one  of  the  top  executive  search  firms.  He  is  supported  by  the  program  coordinator,  an  administrative  assistant,  and  the  student-­‐led  Career  Services  Advisory  Council.  

An  indication  of  the  success  is  that  almost  every  entity  which  has  hired  one  of  our  graduates  comes  back  to  hire  another.    Employment  in  the  field  within  six  months  of  graduation  is  over  95%.    Below  is  a  list  of  our  placements  for  those  students  who  graduated  in  2013  and  2014.  

 

31    

Graduate Placements, 2013-2014 (as of July 2014)

2013 Graduates

Cohort Position

2008-2009 Tenure track Assistant Professor, College of Applied Health Sciences, University of Illinois Urbana-Champaign

2007-2008 Assistant Professor, Sam Houston State University

2008-2009 Operations Research Analyst, Office of the Secretary of Defense CAPE

2010-2011 Combat Rescue Officer Trainee, United States Air Force

2010-2011 Visiting Associate Professor, University of California, Los Angeles

2005-2006 Postdoctoral Scholar, Stanford University

2007-2008 Prevention Effectiveness Fellow, Centers for Disease Control and Prevention

2009-2010 Fiscal and Policy Analyst, California Legislative Analyst's Office

2007-2008 Air Pollution Specialist, State of California Air Resources Board

2005-2006 Economist Consultant, The World Bank Group

2008-2009 Policy Officer, European Commission, Directorate - General for Internal Market and Services

2007-2008 Teutsch Fellow, Centers for Disease Control

2008-2009 Senior Economic Analyst, Port Authority of New York and New Jersey

2014 Graduates

Cohort Position

2010-2011 Post-doctoral Fellow, National Renewable Energy Laboratory

2008-2009 Junior Economist - Migration, Organization for Economic Co-operation and Development (OECD)

32    

2005-2006 Policy Researcher, Abu Dhabi Police

2009-2010 Manager, Analytics and Strategy, One Acre Fund

2010-2011 Manager, Genentech

2007-2008 Quantitative Consultant, Corporate Executive Board Company

2006-2007 Staff Researcher, University of California, San Diego

2008-2009 Mathematician, RAND Corporation

2010-2011 Senior Quantitative Associate, Social Policy Research Associates

2010-2011 Research Associate, University of Southern California

 

Student  Publications  We  track  student  publications  primarily  through  the  RAND  publications  database.  This  records  all  RAND  publications  that  students  authored  or  co-­‐authored  as  well  as  publications  in  non-­‐RAND  peer-­‐reviewed  journals  and  other  publications  that  they  subsequently  recorded  in  RAND’s  database.  A  list  of  recent  student-­‐authored  publications  is  maintained  online.    

The  tables  below  provide  data  on  the  number  of  registered  publications  by  students  who  have  graduated  from  the  program.  The  numbers  reflect  publications  that  students  worked  on  while  at  Pardee  RAND,  but  in  some  cases  publication  occurred  after  their  exit  date  from  the  program.  

Pardee  RAND  Air  Force  fellows  are  included  in  this  data;  note  that  these  students  are  required  to  complete  their  Ph.D.  within  three  years  and  often  have  lower  rates  of  publication  than  other  students  not  on  this  limited  timeline.  

Pardee  RAND  does  not  currently  have  a  systematic  way  of  collecting  data  on  the  publication  records  of  our  alumni,  but  we  are  exploring  the  possibility  of  conducting  a  survey  of  our  graduates  at  the  5-­‐year  and  the  10-­‐year  mark  to  collect  data  on  their  career  paths  and  contributions  to  the  field  including  publications.  

Average Number of Registered Publications at Graduation by Cohort

Entrance Year Average Number of Publications per Student

2003 8.50

2004 5.15

2005 7.39

33    

2006 7.38

2007 6.67

2008 6.50

2009 2.00

2010 4.00

Overall 6.44

Number of Registered Publications by Cohort, as of May 2014

Entrance Year Publications Graduates from Cohort

2003-2004 68 8

2004-2005 67 13

2005-2006 133 18

2006-2007 96 13

2007-2008 80 12

2008-2009 52 8

2009-2010 10 5

2010-2011 16 4

Grand Total 544 81

34    

Publications by Cohort

Final  Thoughts  As  is  evident  from  the  discussions  above,  we  have  yet  achieved  a  state  of  perfection.    Instead,  we  are  continuously  evaluating  what  we  are  doing  with  a  view  towards  doing  better.    We  are  designing  on-­‐going  processes  that  will  enable  us  to  collect  better  and  more  complete  data  on  student  learning  outcomes  in  the  three  areas  of  our  program:    classroom  learning,  OJT  and  the  dissertation.    As  we  gather  this  data,  we  will  be  in  constant  dialogue  with  the  faculty  to  make  sure  that  the  data  we  are  gathering  are  useful  and  relevant  for  them  as  well  as  the  Pardee  RAND  administration.