Oct. 13, 2006Patterns in Education, AECT 20061 Patterns in Education: Linking Theory to Practice Theodore Frick Department of Instructional Systems Technology

Oct. 13, 2006 Patterns in Education, AECT 2006

1

Patterns in Education: Linking Theory to Practice

Theodore Frick

Department of Instructional Systems TechnologySchool of Education

Indiana University Bloomington


2

Overview of APT&C Analysis of Patterns in Time and

Configuration: APT&C Fundamental change in perspective for

measurement and analysis Bridges quantitative and qualitative

paradigms APT for temporal patterns (both joint

and sequential occurrences of events) APC for structural patterns

(configurations)


3

Overview cont’d

APT&C based on mathematical theories and general systems theory

Value of APT&C is that results can be directly related to practice

Through APT&C we have new ways of conducting educational research


4

Outline of this presentation The dilemma: qualitative vs. quantitative

methodologies Three examples of empirical studies that

used APT&C: Academic learning time (APT joint occurrences) Patterns of mode errors in human-computer

interfaces (APT sequential occurrences) Student autonomy structures in a Montessori

classroom (APC patterns of student choice of work and guidance of learning)


5

Quantitative vs. Qualitative Paradigms Research methods in 20th century were largely

quantitative. Qualitative and mixed methods are gaining

more use in research during past two decades. Main problems:

Quantitative methods seldom yield significant results that can be directly linked to educational practice (due to large within-group variances in experiments or treatments)

Qualitative methods can provide good insights into practice, but conclusions are often restricted (low generalizability due to sampling strategy, and may or may not transfer to similar situations)


6

Three Empirical Studies to Illustrate Value of APT&C

Academic learning time of mildly handicapped children (Frick, 1990)

Patterns of mode errors in human-computer interfaces (An, 2003)

Student autonomy structures in a Montessori classroom (Koh, 2006)


7

Study # 1:Academic Learning Time Study 25 systems observed in central and southern

Indiana Tracked 25 target students in academic activities

over several months for 8 -10 hours each Trained observers coded types of academic

learning contexts, task difficulty and task success Observers also coded student and instructor

behaviors in math and reading (about 500 time samples at one-minute intervals for each target student)

Nearly 15,000 time moments sampled overall.


8

What observers coded in math and reading activities each minute

Types of student engagement: written, oral, and covert on-task; off-task behaviors (later recoded as engagement, EN, and non-engagement, NE)

Types of instructor behaviors: structuring, explaining, demonstrating, questioning, feedback (later recoded as direct instruction, DI), and monitoring academic seatwork (non-direct instruction, ND).

Observer comments to elaborate what was happening


9

Observer coding form


10

Codes for target student moves


11

Codes for instructor moves and focus


12

Standard analysis: columns 1 and 2: independent measures of DI and of EN were correlated (n = 25)

p(DI)

p(EN)

p(DI & EN)

p(DI & NE)

p(ND & EN)

p(ND & NE)

p(EN|DI)

p(EN|ND)

0.50 0.80 0.46 0.04 0.34 0.16 0.92 0.67 0.39 0.49 0.37 0.02 0.12 0.49 0.95 0.20 0.27 0.56 0.26 0.01 0.30 0.43 0.97 0.41 0.34 0.69 0.34 0.00 0.35 0.31 1.00 0.53 0.48 0.73 0.47 0.01 0.25 0.26 0.98 0.49 0.40 0.75 0.39 0.01 0.35 0.25 0.98 0.59 0.44 0.84 0.40 0.04 0.44 0.11 0.91 0.80 0.36 0.75 0.33 0.03 0.42 0.22 0.92 0.65 Etc. Etc.

Mean (SD)

Mean (SD)

Mean (SD)

Mean (SD)

Mean (SD)

Mean (SD)

Mean (SD)

Mean (SD)

0.432 (0.144)

0.741 (0.101)

0.416 (0.139)

0.015 (0.010)

0.324 (0.114)

0.243 (0.104)

0.967 (0.029)

0.573 (0.142)


13

Linear Models Approach

Linear models approach (quantitative method): Relates independent measures

through a mathematical function Treats deviation from model as error

variance


14

Linear Models Approach cont’d


15

Linear models results: Means and standard deviations

Mean p(DI) = 0.432 s.d. = 0.144 Mean p(EN) = 0.741 s.d. = 0.101

Regression equation EN = 0.57 + 0.40DI R2 = 0.33 DI “explains” 33 percent of the variance in

student engagement; 67 percent unexplained


16

Analysis of Patterns in Time

APT measures a relation directly by counting occurrences of when a temporal pattern is true or false in observational data

Probability of joint or sequential occurrence can be estimated for a pattern from the counts


17

APT Results for same 25 systems: includes measures of joint and conditional occurrences

p(DI)

p(EN)

p(DI & EN)

p(DI & NE)

p(ND & EN)

p(ND & NE)

p(EN|DI)

p(EN|ND)

0.50 0.80 0.46 0.04 0.34 0.16 0.92 0.67 0.39 0.49 0.37 0.02 0.12 0.49 0.95 0.20 0.27 0.56 0.26 0.01 0.30 0.43 0.97 0.41 0.34 0.69 0.34 0.00 0.35 0.31 1.00 0.53 0.48 0.73 0.47 0.01 0.25 0.26 0.98 0.49 0.40 0.75 0.39 0.01 0.35 0.25 0.98 0.59 0.44 0.84 0.40 0.04 0.44 0.11 0.91 0.80 0.36 0.75 0.33 0.03 0.42 0.22 0.92 0.65 Etc. Etc. Etc. Etc. Etc. Etc. Etc. Etc.

Mean (SD)

Mean (SD)

Mean (SD)

Mean (SD)

Mean (SD)

Mean (SD)

Mean (SD)

Mean (SD)

0.432 (0.144)

0.741 (0.101)

0.416 (0.139)

0.015 (0.010)

0.324 (0.114)

0.243 (0.104)

0.967 (0.029)

0.573 (0.142)


18

APT Results Means and standard deviations for the relations

Mean p(EN | DI) = 0.967 s.d. = 0.029 Mean p(EN | ND) = 0.573 s.d. = 0.142

When direct instruction is occurring, students are highly engaged.

When non-direct instruction is occurring they are less engaged.

Students were 13 times more likely to be off-task during non-direct instruction compared with direct instruction: (1 - 0.573) / (1 – 0.967) = 12.94.


19

APT: joint occurrence calculation example

Time Instr. Eng.

1:00 DI EN

1:01 DI NE

1:02 DI EN

1:03 ND NE

p(DI) = ¾ = 0.75p(ND) = ¼ = 0.25p(EN) = ½ = 0.50p(NE) = ½ = 0.50p(DI & EN) = 2/4 = 0.50p(DI & NE) = ¼ = 0.25p(ND & EN) = 0/4 = 0.0p(ND & NE) = ¼ = 0.25p(EN|DI) = 2/3 = 0.67p(EN|ND) = 0/1 = 0.00


20

LMA vs. APT Linear models relate the independent

measures by a function for a line: e.g., EN = 0.57 + 0.40DI

APT measures the relation in terms of joint, conditional, or sequential occurrence: e.g., p (EN|DI) = 0.967 e.g., p (EN|ND) = 0.573

DI = direct instruction, EN = student engagement, ND = non-direct instruction


21

Study #2:Patterns of Mode Errors in HCI Software mode: when the same action results in

two or more outcomes (Raskin, 2000). E.g., In one context, pressing the ‘d’ key results in

the letter ‘d’ echoed on the screen In another context, pressing the ‘d’ key results in

deleting a file. Mode errors by humans can cause serious problems:

Destruction of important work Decreased productivity Not able to complete tasks

Modes occur in almost all modern human-computer interfaces (e.g., OS 10, Windows XP, Word, Photoshop, etc.)


22

An (2003) study of mode errors Mixed methods approach (usability

evaluation, qualitative and quantitative) 16 college students performed eight

computer tasks with three modern GUI interfaces (word processor, address book, image editor).

Participants were videotaped, and stimulated- recall interviews were conducted immediately afterwards to clarify why certain actions were taken, when viewing their videos.


23

An (2003) study of mode errors (cont’d) Over 280 problematic actions were

observed, and 52 were problems due to mode errors

52/280 = .19, or roughly 1 out of 5 problems were due to software modes

Three general patterns (conditions) of mode errors emerged from qualitative analyses: Type A: Right action, wrong result Type B: It isn’t there where I need it Type C: It isn’t there at all


24

An (2003) study of mode errors (cont’d) Source of error analysis revealed that

mode errors appeared to result from 8 types of design incongruity: Unaffordance Invisibility Misled expectation Unmet expectation Mismatched expectation Inconsistency Unmemorability Over-automation


25

An (2003) study of mode errors (cont’d)

Consequences of mode errors: Can’t find hidden function Can’t find unavailable function False success Stuck performance Inhibited performance Inefficient performance


26

APT: analysis of sequential patterns of mode errors, sources and consequences

Query

Relative Frequency

Likelihood (p)

Type A

1

IF type of mode error IS right action, wrong result,

34 out of 52

0.65

a) AND IF source of mode error IS unaffordance, 15 out of 34 0.44 THEN consequence IS can’t find hidden function OR false

success?

10 out of 15

0.67 b) AND IF source of mode error IS invisibility, 6 out of 34 0.18 THEN consequence IS stuck performance? 5 out of 6 0.83 c) AND IF source of mode error IS misled expectation, 7 out of 34 0.21 THEN consequence IS false success? 6 out of 7 0.86


27


Query

Relative Frequency

Likelihood (p)

Type B

2

IF type of mode error IS it isn’t there where I need it,

8 out of 52

0.15

a) AND IF source of mode error IS mismatched expectation, 8 out of 8 1.00 THEN consequence IS can’t find hidden function? 8 out of 8 1.00

Query

Relative Frequency

Likelihood (p)

Type C

3

IF type of mode error IS it isn’t there at all,

10 out of 52

0.19

a) AND IF source of mode error IS unmet expectation, 10 out of 10 1.00 THEN consequence IS can’t find unavailable function? 10 out of 10 1.00 b) AND IF source of mode error IS unaffordance, 3 out of 10 0.30 THEN IF source of mode error IS unmet expectation, 3 out of 3 1.00 THEN consequence IS can’t find unavailable

function? 3 out of 3 1.00


28


APT results have practical implications E.g., if the mode error is ‘right action,

wrong result’ and if the source of the error is unaffordance (function not obvious), then 67 percent of the time users could not find a hidden function or thought they did the task correctly when in fact they had not (false success).


29

APT Methodology: sequential occurrence When one event precedes another, and

when observers code the order in which events occur: APT can estimate the probability of the

consequent following the antecedent event. APT can estimate likelihoods of sequences

longer than two (unlike Markov chains). APT can estimate both joint and sequential

event occurrences in complex combinations.


30

APT Coding (temporal configuration)

Clock Time

Target Student Instruction

Student Engagement

9:01 Mona Direct Off-task 9:02 9:03 On-task 9:04 9:05 9:06 Off-task 9:07 On-task 9:08 Non-Direct 9:09 9:10 9:11 Off-task 9:12 9:13 Null Null Null


31

APT Classifications and Categories Each column is a classification Classifications co-exist in time Categories of events within a classification

cannot co-exist in time (since they are mutually exclusive, by definition)

An observer codes event changes within each classification in the order that they occur.

Date/time is always a classification and is recorded whenever there is an event change.


32

Example of sequential coding with three classifications

Clock Time


Student Engagement



33

Clock Time


Student Engagement


APT Query: IF target student IS Mona?


34

APT Query and Results

QueryIF target student IS Mona?

ResultsCumulative duration = (9:13 – 9:01) = 12

minutesCumulative frequency = 1 eventLikelihood = 1 out of 1 relevant event changes

= 1.00Proportion time = 12 minutes out of 12 = 1.00


35

APT Query: IF target student is Mona AND instruction is direct?

Clock Time


Student Engagement



36

APT Query Results

QueryIF target student IS Mona

AND instruction IS direct?

ResultsCumulative duration = (9:08 – 9:01) = 7

minutesCumulative frequency = 1 eventLikelihood = 1 out of 2 relevant event changes

= 0.50Proportion time = 7 minutes out of 12 = 0.583


37

APT Query: IF target student IS Mona AND instruction IS direct, THEN student engagement IS on-task?

Clock Time


Student Engagement



38

APT Query Results

QueryIF target student IS Mona

AND instruction IS direct, THEN student engagement IS on-task?

ResultsCumulative duration = (9:06 – 9:03) + (9:08 –

9:07) = 4 minutesCumulative frequency = 2Likelihood = 2 out of 4 = 0.50Proportion time = 4 minutes out of 6 = 0.667


39

APT Query Syntax


40

APT Syntax (cont’d)


41

APT Syntax (cont’d)


42

APT Query Syntax Thus, simple to very complex

temporal patterns can be specified within APT queries.

Joint and/or sequential occurrences of events can be specified.

Results include frequency counts, likelihood estimates, durations and proportions of total time.


43

Theoretical Foundationsof APT Mathematical theory

Set theory Probability theory

Information theory Classifications (more than one, non-exclusive) Categories within each classification must be

mutually exclusive and exhaustive General systems theory

SIGGS Theory Model


44

Advantages of APT APT brings theoretical rigor to pattern

identification in qualitative research. APT measures relations not possible in

quantitative methods such as the linear models approach.

APT requires a different kind of conceptual framework for measurement and analysis than those for qualitative and quantitative approaches.


45

APC: Analysis of Patterns in Configuration

Thompson (2005) realized that APT could be extended to measure and analyze structure of systems.

Structure pertains to relationships among parts.


46

Familiar Patterns: Structural Geographical relation:

Bloomington is located in southern Indiana on the North American continent.

Bloomington is south of Indianapolis. Organizational relation:

Gerardo Gonzalez is University Dean of the School of Education who directs and supervises:

Peter Kloosterman, Executive Associate Dean, SoE, IUB campus

Khaula Murtahda, Executive Associate Dean, SoE, IUPUI campus


47

Familiar Patterns: Structural Familial relation:

Philip and Irma Frick are the parents of Theodore Frick

William and Helen Brophy are the parents of Kathleen Brophy

Instructional relation: During fall semester, 2005,T. Frick was the

R690 instructor of: Andrew, Omer, Shyamasri, Nichole, Jamison,

Sunnie, Emmanuel, Uvsh, Chris, Theano


48

A pattern is a relation

General form of a relation:


49

Temporal & Structural Patterns & Logical Relations

Temporal Patterns A precedes B A co-occurs with B

Structural Patterns or Configurations A affect relation B

Logical Relations A implies B A is equivalent to B


50

Affect relation: guides research of

Faculty Person 1

Faculty Person 2

Student 1Student 2

Student 3

Student 4 Student 5

Old IST Ph.D. structure


51

Affect relation: guides research of

Faculty Person 1

Faculty Person 2

Student 1Student 2

Student 3

Student 4 Student 5

New IST Ph.D. structure


52

APC allows us to measure structural properties of di-graphs

Property Count Value

Active Dependence 1.00 paths 5.97

Centrality 4.00 paths 23.89

Compactness 9.00 paths 53.76

Complete Connectedness 0.00 paths 0.00

Complexness 5.00 paths 5.00

Etc. Etc. Etc.


53

Study #3: Autonomy structures in a Montessori classroom (Koh, 2006)

Case study to explore Montessori classroom structures that support student autonomy

Observed on 10 occasions for about an hour at different times of morning session (1 head teacher, 2 assistant teachers, 28 students ages 10-12)

Ethnographic approach initially


54

Koh (2006) study cont’d Class activities were built around

two different activity structures: Head problems Morning work period

Koh was interested in two kinds of affect relations: s chooses work y y guides learning of s


55

Koh (2006) study cont’d

Digraphs were drawn for affect relation structures during Head Problems and during Morning Work Period

APC software was used to calculate structure measures of these digraphs (Frick & Thompson, 2006)


56

Koh (2006) study cont’d

Structures measured: Active dependence Centrality Complexity Independence Interdependence Complete connectivity


57

Active Dependence: definition and measure

Active dependent-component partition, AD

S, =df

a partition, Y = (VGO,R GA), characterized by

initiating component affect-relations.

ADS =df Y | vi,vjY(V )r d(I)(e)Y(R )[e = (vi,vj) r d(I)(e) = 1]

M: Active dependent-component partition measure, M(AD

S), =df

a measure of initiating

affect-relations.

M(AD

S) =df [(i=1,…,n[j=1,…,mdI(j)(v) log2|A i|]) n] 100


58

APC Results fromKoh (2006) study

0

10

20

30

40

50

60

Active dependence

Centrality Complexity Independence Interdependence Complete connectivity

PropertyValue

Structural Property

Morning Work Period Head Problems


59

APC Results fromKoh (2006) study (cont’d) Active dependence higher in Head

Problems vs. Morning Work Period Centrality higher in Head Problems

vs. Morning Work Period Interdependence lower in Head

Problems vs. Morning Work Period Complexity lower in Head

Problems vs. Morning Work Period


60

APC Results fromKoh (2006) study (cont’d) The structure of the Morning Work

Period supported student autonomy

During the Morning Work period there was: Less active dependence No centrality Greater complexity Greater interdependence


61

APC Results fromKoh (2006) study (cont’d) The 3 teachers’ responses to the Problems

in Schools Questionnaire (SDT, 2006) showed them to be “highly autonomy supportive”.

Student responses to the Academic Self-Regulation Questionnaire (SDT, 2006) indicated a greater tendency to undertake learning activities because they perceived some personal value and identification with the learning goals, rather than because they felt compelled by external factors.


62

APC Results fromKoh (2006) study (cont’d) The structural configuration of the Morning

Work Period, where students chose learning activities and worked at their own pace is characteristic of Montessori classrooms.

The structural configuration of the Head Problems activity chosen by the head teacher with all students working on the same problems, is more typical of traditional K-12 classrooms in the U.S.

APC allowed analysis and comparison of structural properties of those two configurations of affect relations.


63

Summary

APT allows measurement and analysis of temporal properties Joint occurrences Sequential occurrences Combinations of joint and sequential

occurrences


64

APT: joint occurrence example

Time Instr. Eng.

1:00 DI EN

1:01 DI NE

1:02 DI EN

1:03 ND NE


65

APT joint and sequential occurrence example

Clock Time


Student Engagement



66

Summary

APC allows measurement and analysis of structural properties


67

APC allows measures of structural properties of an affect relation (e.g., guides research of)

Faculty Person 1

Faculty Person 2

Student 1Student 2

Student 3

Student 4 Student 5

New IST Ph.D. structure


68

APC property measures and values

Property Count Value

Active Dependence 1.00 paths 5.97

Centrality 4.00 paths 23.89

Compactness 9.00 paths 53.76

Complete Connectedness 0.00 paths 0.00

Complexness 5.00 paths 5.00

Etc. Etc. Etc.


69

Summary: APT&C Analysis of Patterns in Time and

Configuration permits measurement and analysis of human learning and work environments.

The value of APT&C methodology was illustrated by clear results from three empirical studies.

These results have direct implications for practice. APT&C is a way to link theory to practice.

Software is under development to do APT&C.


70

Questions

For more information on APT&C:

http://www.indiana.edu/~aptfrick

Documents

Oct. 13, 2006Patterns in Education, AECT 20061 Patterns in Education: Linking Theory to Practice Theodore Frick Department of Instructional Systems Technology