Evaluating Data Purpose of Experiments Evaluating Experiments Is to test theory? Let’s do it!!!

Evaluating DataPurpose of Experiments

Evaluating ExperimentsIs to test theory?

Let’s do it!!!

Evaluating DataPurpose of Experiments

Evaluating Experiments

U s e e x p e r i m e n t a l u n c e r t a i n t i e s a n d t h e p r o p e r t i e s o f p r o b a b i l i t y d i s t r i b u t i o n s t o e v a l u a t e t h e e x t e n t t o w h i c h d a t a s u p p o r t s o r r e f u t e s a p h y s ic a l m o d e l .

2006 LEP Constraints

Or is to win Nobel Prizes forTheorists?

July 4 2012 discovery announced

Evaluating Data

Purpose of Experiments

Evaluating Data

Purpose of ExperimentsPeter Higgs2013 Nobel Prize Winner

Austin Ball –Technical Director of CMS (He was in charge of building the CMS experiment)

Pu r p o s e o f Ex p e r im e n ts● Pur p o s e o f e x p e r im e n t :

t e s t e x is t in g m o d e l

“ M o d e l ” ( W e b s te r ) – a s y s t e m o f p o s tu la t e s , d a t a , a n d in f e r e n c e s p re se n t e d a s a m a t h e m a t ic a l d e sc r ip t io n o f a n e n t i t y o r s ta t e o f a f f a i r s

i.e. a f u n c t io n a l r e la t io n s h ip b e t w e e n v a r ia b le s

Standard Model of Particle Physics

https://en.wikipedia.org/wiki/Standard_Model_%28mathematical_formulation%29

CERN Experiments

http://home.web.cern.ch/about/experiments

U.S. HEP Experiments

http://www-sld.slac.stanford.edu/sldwww/sld.html

http://www-cdf.fnal.gov/about/index.html

Standard Model of Particle Physics

Electron-positron cross-section and Z-boson Lineshape

http://rsta.royalsocietypublishing.org/content/370/1961/805

W h a t c a n e x p e r im e n ts a c h i e v e?

● A n e x p e r im e n t c a n n o t p r o v e a m o d e l t o b e t r u e .

A n e x p e r im e n t c a n :– D e m o n s t r a te th a t a

m o d e l is f a ls e w i t h s o m e p r o b a b i l i t y

– Sh o w th a t a m o d e l is v a l id w ith in so m e p re c is ion

W h a t c a n e x p e r im e n ts a c h i e v e?

● A n e x p e r im e n t im p r o v e s th e s t r e n g th o f i t s s t a t e m e n ts b y d e c re a s in g u n c e r t a i n t ie s .

H o w d o I q u a n t i f y t h e e x t e n t to w h i c h th e d a t a to t h e r i g h t is in c o n s is t e n t w i t h O h m ' s l a w ?

E xp e r i m e n ta l E va lu a t i o n

● D e t e r m i n e s ig n i f i c a n c e o f y o u r m e a su r e m e n t .

W h a t d o e s t h e e x p e r im e n t sa y a b o u t t h e m o d e l y o u ' r e t e s t i n g ?

– D o e s i t su p p o r t o r r e f u te th e m o d e l?– Ca n w e q u a n t i f y o u r c o n f id e n c e in t h e

m o d e l?

G a u s s ia n D i s t r i b u t i o n

A “ 1 s i g m a ” e r r o r in te r v a l is n o t a l l - in c lu s i v e !

U s e “ 2 /3 R u le ” a s a R o u g h G u id e

● Is d a t a c o n s is t e n t w i t h a Y a n d X h a v i n g a l in e a r r e l a t io n s h ip ?

● Sp r e a d o f p o i n t s , e r r o r b a rs c o n s is t e n t w i t h n o r m a l e r r o r d is t r ib u t io n .

● W h a t is t h e p r o b l e m h e re ?

U s e “ 2 /3 R u l e ” a s a R o u g h G u i d e

Ev a lu a t in g M o d e ls : 2

● D e f in i t i o n o f 2

● 2 h a s m e a n in g r e la t i v e t o t h e n u m b e r o f d e g r e e s o f f r e e d o m :

# D O F = ( # d a ta p o in t s ) -( # p a ra m e t e rs d e te r m in e d f ro m d a ta )

Ro u g h ly , 2 / # D O F ~ 1 f o r g o o d d a ta /m o d e l m a t c h

Ev a lu a t in g M o d e ls : 2

● D e f in i t i o n o f 2

● 2 h a s m e a n in g r e la t i v e t o t h e n u m b e r o f d e g r e e so f f r e e d o m :

# D O F = ( # d a ta p o in t s ) -( # p a ra m e t e rs d e te r m in e d f ro m d a ta )

Ro u g h ly , 2 / # D O F ~ 1 f o r g o o d d a ta /m o d e l m a t c h

“CONSTRAINTS”

2 is a m o r e q u a n t i t a t i v e m e t h o d :

What is #DOF?

Co u n t i n g D e g r e e s o f F r e e d o m● # D e g r e e s o f f r e e d o m i n

th is s t r a ig h t l in e f i t s h o u l d b e e a s y → w h a t is i t? ?

W h a t i f in te r c e p t w e r e a ls o to b e d e t e r m in e d f r o m th e d a t a ?

H o w m a n y c o n s t r a in ts i f I ' m f i t t in g a G a u ss ia n in a c o u n t in g e x p e r im e n t?

. . . i f I ' m f i t t i n g a G a u ss ia n in a n o n -c o u n t in g e x p e r im e n t?

p -v a l u e o r “ c h a n c e p r o b a b i l i t y ” : Th e p r o b a b i l i t y o f o b t a i n i n g a r e s u l t a t l e a s t a s e x t r e m e a s a g i v e n d a ta p o in t , a s su m i n g t h e d a t a p o i n t w a s t h e r e s u l t o f c h a n c e a lo n e

● A s t a t e m e n t a b o u t t h e p r o b a b i l i t y t h a t a n “ e f f e c t ” is r e a l ly a s t a t is t i c a l f lu c t u a t io n .

Re p l a c e s d is c re t e T r u e /Fa ls e (o r 1 /0 ) c h o ice w i t h a c o n t i n u u m r a n g i n g f r o m “ l ik e l y t r u e ” t o “ l ik e l y f a l s e ” .

T r u t h → “ Tr u t h in e s s ”

Ex a m p le : Su p p o s e t h e re a re 3 0 , 0 0 0 U n iv e r s it y o f U ta h s tu d e n t s , o f w h ic h 4 0 0 a re p e r m i t t e d t o c a r r y g u n s . If I 'm t e a c h in g a n a s t r o n o m y c la ss o f 1 2 0 s tu d e n t s , w h a t is t h e p ro b a b i l i t y t h a t o n e o r m o re is c a r r y in g a g u n ?

A n s w e r : T h e p r o b a b i l i t y d is t r i b u t i o n f o r t h e n u m b e r o f s tu d e n t s w it h w e a p o n s is

a p p r o x im a t e ly P o iss o n ia n , w it h a m e a n v a lu e o f ( 4 0 0 /3 0 , 0 0 0 ) x 1 2 0 = 1 .6

Th e p ro b a b i l i t y o f o b se r v in g o n e o r m o re in a P o iss o n d is t r ib u t i o n w it h m e a n 1 .6 is :

P ( > = 1 , 1 .6 ) = 1 – P ( 0 ,1 .6 ) = 1 – 0 .2 0 1 = 0 .7 9 9

Fol lo w u p : Su p p o s e a m e t a l d e t e c t o r r e v e a ls t h a t th e r e a re 6 p e o p le in John B e l z ' s c la ss c a r r y in g g u n s. W h a t is th e p -v a lu e o f t h is o b se rv a t io n ? Sh o u ld JB b e w o r r ie d t h a t th e y ' re o u t to g e t h im ?

Fo l lo w u p : Su p p o s e a m e t a l d e t e c t o r r e v e a ls t h a t t h e r e a re 6 p e o p le in JB ' s c la ss c a r r y in g g u n s. W h a t is t h e p -v a lu e o f t h is o b se r v a t i o n ? Sh o u ld JB b e w o r r ie d t h a t t h e y ' re o u t to g e t h im ?

● A n s w e r : Th e p -v a l u e i s t h e s u m o f p r o b a b i l i t i e s f o r n = 6 , 7 , 8 , 9 . . .

● P-v a lu e P ( > = 6 , 1 . 6 )P

= 0 .0 0 6● Th is is a v e r y s m a l l

c h a n c e p ro b a b i l i t y . Per h a p s JB s h o u ld w e a r a b u l le t p r o o f v e s t !

Example: I f l ip a c o in 2 0 t im e s , 1 4 t im e s i t c o m e s u p h e a d s . U s e t h is d a ta a s a t e s t o f t h e h y p o th e s is t h a t t h e c o in is f a i r.

p = ( B in o m ia l P r ob a b i l i t y o f > = 1 4 h e a d s ) + ( B i n o m ia l P r ob a b i l i t y o f < = 6 h e a d s )

= 0 .1 1 5

Example: I f l ip a c o in 2 0 t im e s , 1 4 t im e s i t c o m e s u p h e a d s . U s e t h is d a ta a s a t e s t o f t h e h y p o th e s is t h a t t h e c o in is f a i r.

p = ( B in o m ia l P r ob a b i l i t y o f > = 1 4 h e a d s ) + ( B i n o m ia l P r ob a b i l i t y o f < = 6 h e a d s )

= 0 .1 1 5

W h ile t h e “ f a ir c o in ” h y p o t h e s is d o e s n o t h a v e a h ig h le v e l o f t r u t h in e ss ( p -v a lu e ) , i t s n o t so lo w t h a t w e c a n sa y f o r su r e t h a t t h e c o in isn ' t f a ir.

Th e 2 a n d n u m b e r o f d e g r e e s o f f r e e d o m

c a n b e d i r e c t ly t r a n s la t e d in t o a p -v a l u e :

So u r c e : p d g . lb l . g o v

http://pdg.lbl.gov/2014/reviews/rpp2014-rev-statistics.pdf

http://pdg.lbl.gov/2014/reviews/contents_sports.html

Ta b le f r o m p a g e 2 9 3 o f Ta y lo r

Ex a m p le : Taylor 1 2 . 0 4I t h ro w t h r e e d ic e t o g e t h e r a to t a l o f 4 0 0 t im e s , re c o r d t h e n u m b e r o f s ix e s in e a c h t h r o w , a n d o b ta in t h e r e su l t s sh o w n b e lo w . U se t h e b in o m ia l d is t r i b u t io n to f in d t h e e x p e c t e dn u m b e r E f o r e a c h o f t h e t h r e e b in s a n d t h e n c a lc u la te 2 .

D o I h a v e re a so n t o s u s p e c t t h e d ic e a r e lo a d e d ?

Taylor 1 2 . 0 4

● N = 3 t r ia ls

= 0 , 1 , 2 , 3

su c c e sse s p = 1 /6

Taylor 1 2 . 0 4

W h a t is t h e n u m b e r o f d e g r e e s o f f r e e d o m ?39

Taylor 1 2 . 0 4

2 / d o f =

4 . 1 /2 p -v a lu e

~ 0 .1 4

“Consistent with a statistical fluctuation at the 10% level”

A d d i t i o n a l R e a d i n g a n d Pr o b l e m s

● Re a d in Ta y l o r :

– Ch 5 : T h e N o r m a l D i s t r i b u t i o n

– Ch a p t e r 1 2 : Th e Ch i - Sq u a r e d Te s t f o r a D i s t r i b u t i o n

● Tr y t h e p r o b l e m s :– 5 .1 1 , 5 .1 2 , 5 .2 0 ,

5 .2 1– 1 2 . 4 , 1 2 . 6 , 1 2 .1 1 ,

1 2 .1 4 , 1 2 .1 6

CERN Root Analysis Software Tutorials

• Start RDP session on orion, draco or Cygnus• Open an x-terminal window and issue the following commands

• The following should spew forth….

• Then run the tutorials benchmark by typing the following command

• And now your odyssey begins

CERN Root Tutorials

CERN Root Tutorials• Now let’s play….• From the root command line issue the following command

• The following should spew forth….

• Open another xterm window• Copy and edit the “program” (Macro) tutorials/fit/fit1.C to

~/phys6719/root/sandbox/myfit1.C by typing the following commands.

• Modify the program …

Exercises

1. Use the examples from the tutorials to write a CERN root macro to generate a Histogram containing events sampled from the sum of an exponential probability distribution.

2. Fit this histogram with an exponential function. Verify that the fit returns parameters consistent with the probability distribution that you used to generate your histogram. Report the c2 /D.O.F. goodness of fit parameter.

3. Repeat exercises 1 and 2 for a Gaussian and a Poisson Distribution.4. Document your results including graphs and written descriptions in

a libreoffice document.

Evaluating Data Purpose of Experiments Evaluating Experiments Is to test theory? Let’s do it!!!

Documents

Let’s learn together! Let’s sing Story time? Let’s learn

Evaluating State Programmes: “Natural Experiments” and ... · “NATURAL EXPERIMENTS” AND PROPENSITY SCORES 285 affect response. Sometimes quite simple methods are used to analyse

Field Experiments for Evaluating Wind Erosion Models 384 J Field experiments for...Field Experiments Bagnold (1941) was probably the first to simultaneously measure wind speed and

Audio Myths And Critical Gear Evaluation. Agenda Evaluating Gear – Design of experiments to properly evaluate gear Goal: Let you invest your money in

Experiments · List of Experiments 1. List of Experiments

Methods for Evaluating Natural Experiments in Obesity ...• Natural experiments most commonly used regression models comparing exposed and unexposed groups at one time. • Natural

Research Reproducibility - Nanyang Technological University€¦ · Camerer et al. (2018) Evaluating the replicability of social science experiments in Nature and Science between

Evaluating the Functional Role of Enhancing Progenitor Cell Survival Following Stroke ... · 2017. 7. 13. · spatial learning on the Barnes maze. Together, these experiments suggest

DISCUSSION PAPER SERIES - GOV UK...DISCUSSION PAPER 10: A CONCEPTUAL FRAMEWORK FOR ANALYSING AND EVALUATING DFID‐RIU EXPERIMENTS IN BROKERING AGRICULTURAL INNOVATION AND DEVELOPMENT

Evaluating Experiments D. Crowley, 2007. Evaluating Experiments To be able to evaluate experiments, and know the difference between mass and weight Monday,

Summative Evaluation - inf.ed.ac.uk · Summative Evaluation. Mar-23-12! Adaptive Learning Environments! 2! Contents 1. Using Experiments for System Design and Evaluation 2. Evaluating

Chapter 3personal.kent.edu/~asamba/cs23021/CS-23021Chap03.pdfIntroduction Computer Science 1 CS 23021 Let’s review evaluating Arithmetic Expressions Rule: PEMDAS (x + 1) * ( x +

British HCI 2017: Evaluating the social impact of youth digital culture co-creation- let’s participate and play

Supported Employment:Evaluating Your Program · Evaluating Your Program 5 Conduct a Readiness AssessmentAppendix A: Cover Sheet Let’s assume that administrators and SE ... not revert

Experiments and Quasi-Experiments

Experiments and Quasi-Experiments

Unit5 B Let’s learn Let’s drink. Let’s go to the KFC

Studies in isothermal heat conduction calorimetrysites.chem.colostate.edu/diverdi/C431/experiments/isothermal... · Isothermal Heat Conduction Calorimetry ... evaluating the radiant

LET’S YARN ABOUT GAMBLING LET’S SHARE OUR ......TOGETHER LET’S PREVENT THIS 8 – 14 OCTOBER GAMBLING HARM. LET’S SHARE OUR STORIES LET’S YARN ABOUT GAMBLING LET’S LOOK

Journal article mining - STM...Institute for Neuroscience, Amsterdam Let’s start with a potential user (2) Use-case-2: Information needed as result of laboratory experiments •Which