WEBB’S DEPTH OF KNOWLEDGE

a scale of cognitive demand

DOK Introduction? + !

Code with a ? to indicate that you have never heard of this,

Code with a + to indicate that you know something about this,

Code with an ! to indicate that you know this so well that you could teach this!

Origin In 1997, Norman Webb,

of the Wisconsin Center of Educational Research ; University of Wisconsin-Madison, developed a process and criteria for systematically analyzing the alignment between standards and standardized assessments.

No Child Left Behind (NCLB) requires assessments to “measure the depth and breadth of the state academic content standards for a given grade level”

(U.S. Department of Education, 2003, p. 12)

Mechanism to ensure that the intent of the standard and the level of student demonstration required by that standard matches the assessment items.

Since then, the process and criteria have demonstrated application to reviewing curricular alignment as well. Webb’s Depth of Knowledge (DOK) model may be employed to analyze the cognitive expectation demanded by standards, curricular activities and assessment tasks (Webb, 1997).

DOK Level 1Recall

Basic recall of concepts, definitions, facts, and processes

Answering a Level 1 item can involve following a simple, well-known procedure or formula

Simple skills and abilities or recall characterize DOK 1

Examples of DOK Level 1 Locate or recall facts found in text Apply a well known formula Use a dictionary to find the meaning of

words Represent math relationships in words,

pictures, or symbols Perform a simple science process or a set

of procedures

DOK Level 2Skill/Concept

DOK 2 includes the engagement of some mental processing beyond recalling or reproducing a response.

Items require students to make some decisions as to how to approach the question or problem.

These actions imply more than one mental or cognitive process/step.

Examples of DOK Level 2 Identify and summarize the major events,

problem, solution, conflicts in literary text

Explain the cause-effect of historical events

Predict a logical outcome based on information in a reading selection

Retrieve information from a table, graph, or figure and use it to solve a problem requiring multiple steps.

DOK Level 3Strategic Thinking

DOK 3 requires deep understanding as exhibited through planning, using evidence, and more demanding cognitive reasoning.

The cognitive demands at Level 3 are complex and abstract.

An assessment item that has more than one possible answer and requires students to justify the response they give would most likely be a Level 3.

Examples of DOK Level 3 Justify a response when more than one

answer is possible.

Analyze or evaluate the effectiveness of literary elements (e.g. characterization, setting, point of view, conflict and resolution, plot structures).

Solve a multiple-step problem and provide support with a mathematical explanation that justifies the answer.

DOK Level 4Extended Thinking

DOK 4 requires high cognitive demand and is very complex.

An investigation or application that requires time to research, think or process multiple conditions of the problem.

Students are expected to make non-routine manipulations or connections across disciplines/content areas/multiple sources.

Due to the complexity of cognitive demand, DOK 4 requires an extended period of time.

Examples of DOK Level 4

Analyze and synthesize information from multiple sources.

Design and conduct experiments and projects.

Conduct a study that requires taking into consideration a number of variables.

DOK TipsIt’s not just the verbs: DOK 1- Describe three characteristics of

metamorphic rocks. (Requires simple recall)

DOK 2- Describe the difference between metamorphic and igneous rocks. (Requires cognitive

processing to determine the differences in the two rock types)

DOK 3- Describe a model that you might use to represent the relationships that exist within the rock cycle. (Requires deep understanding of the rock cycle and

a determination of how best to represent it)

DOK TipsIt’s not the difficulty of the task, it’s the complexity. What is the definition of exaggerate? DOK 1— recall (NOT difficult!)

What is the definition of glabella? DOK 1— recall

(DIFFICULT!)

DOK Tips

If there is only one correct answer, it is probably level DOK 1 or DOK 2  

DOK 1: you either know it (can recall it, locate it, do it) or you don’t

DOK 2 (conceptual): apply one concept, then make a decision before going on applying a second concept

DOK Tips

 If there is more than one solution/approach, requiring evidence, it is DOK 3 or 4

DOK 3: Must provide supporting evidence and reasoning (not just HOW solved, but WHY – explain reasoning)

DOK 4: all of “3” + use of multiple sources or texts

Let’s review Turn and talk to your neighbor. Discuss

how looking at DOK levels of activities in your lessons might increase rigor in your classroom.

In 3 minutes we will share.

New York City Department of Education DOK Video

OK, what do you think?

Level 3.  “An activity that has more than one possible answer and requires students to justify the response they give would most likely be a Level 3.” Since there are multiple possible approaches to this problem, the student must make strategic decisions about how to proceed, which is more cognitively complex than simply applying a set procedure or skill.

Grade 4 Assessment :

Think carefully about the following question. Write a complete answer. You may use drawings, words, and numbers to explain your answer. Be sure to show all of your work.Laura wanted to enter the number 8375 into her calculator. By mistake, she entered the number 8275. without clearing the calculator, how could she correct her mistake?Without clearing the calculator, how could she correct her mistake another way?

Here’s another:

Level 1. Students only need to be able to recognize even numbers.

Look at the drawing.

1. 23 2. 25

1. 3. 2 23. 3. 28 4. 30

5. 32

Now try this one.

Level 3. This item can be approached through a number of viable strategies: pattern recognition, guess-and-check, algebra, etc. This freedom means that the student must make choices and assumptions. Furthermore, no matter what strategy she employs, she must keep track of a complex logical chain. The multiple choices provided do not make this task any less complex.

Grade 8 AssessmentLook at the drawing. The numbers alongside each column and row are the total of the values of the symbols within each column and row. What should replace the question mark?

How about this one?

Level 3. This item gives the student a new definition and asks her to use it as a basis for reasoning. In order to ascertain whether the student really understands the asymptotic behavior that makes a 6% effective rate impossible, this item must be open-ended. This is why most Level 3 items are open-response items, because the complexity of thinking they require the students to display could not be displayed using the multiple choice option.

One plan for a state income tax requires those persons with income of $10,000 or less to pay no tax and those persons with income greater than $10,000 to pay a tax of 6 percent only on the part of their income that exceeds $10,000.

A person's effective tax rate is defined as the percent of total income that is paid in tax.

Based on this definition, could any person's effective tax rate be 5 percent? Could it be 6 percent? Explain your answer. Include examples if necessary to justify your conclusions.

Grade 11 Assessment

And one more?

Level 2. This item is not Level 1 because it is not routine, nor does it focus on a memorized definition or procedure. In fact, it involves numerous steps, because it requires students to compare several different pairs of shapes before arriving at the correct answer. For these reasons, many spatial reasoning items are Level 2. Note that this may be a coded as a source-of-challenge item, because choice C seems to be drawn in a misleading way.

More to think about . . . Karin Hess, from the The National Center for the Improvement of Educational Assessment, Inc. has done extensive research regarding rigor in the classroom and she applies Webb’s DOK to Bloom’s Cognitive Process Dimensions.

More to think about . . . Hess’ Cognitive Rigor Matrix and Curricular Examples Applying Webb’s Depth-of-Knowledge Levels to Bloom’s Cognitive Process Dimensions – Math/Science

Hess’ Cognitive Rigor Matrix & Curricular Examples: Applying Webb’s Depth-of-Knowledge Levels to Bloom’s Cognitive Process Dimensions – Math/Science

Revised Bloom’s Taxonomy

Webb’s DOK Level 1 Recall & Reproduction

Webb’s DOK Level 2 Skills & Concepts

Webb’s DOK Level 3 Strategic Thinking/ Reasoning

Webb’s DOK Level 4 Extended Thinking

Remember Retrieve knowledge from long-term memory, recognize, recall, locate, identify

o Recall, observe, & recognize facts, principles, properties

o Recall/ identify conversions among representations or numbers (e.g., customary and metric measures)

Understand Construct meaning, clarify, paraphrase, represent, translate, illustrate, give examples, classify, categorize, summarize, generalize, infer a logical conclusion (such as from examples given), predict, compare/contrast, match like ideas, explain, construct models

o Evaluate an expression o Locate points on a grid or

number on number line o Solve a one-step problem o Represent math relationships in

words, pictures, or symbols o Read, write, compare decimals

in scientific notation

o Specify and explain relationships (e.g., non-examples/examples; cause-effect)

o Make and record observations o Explain steps followed o Summarize results or concepts o Make basic inferences or logical

predictions from data/observations o Use models /diagrams to represent

or explain mathematical concepts o Make and explain estimates

o Use concepts to solve non-routine problems

o Explain, generalize, or connect ideas using supporting evidence

o Make and justify conjectures o Explain thinking when more than

one response is possible o Explain phenomena in terms of

concepts

o Relate mathematical or scientific concepts to other content areas, other domains, or other concepts

o Develop generalizations of the results obtained and the strategies used (from investigation or readings) and apply them to new problem situations

Apply Carry out or use a procedure in a given situation; carry out (apply to a familiar task), or use (apply) to an unfamiliar task

o Follow simple procedures (recipe-type directions)

o Calculate, measure, apply a rule (e.g., rounding)

o Apply algorithm or formula (e.g., area, perimeter)

o Solve linear equations o Make conversions among

representations or numbers, or within and between customary and metric measures

o Select a procedure according to criteria and perform it

o Solve routine problem applying multiple concepts or decision points

o Retrieve information from a table, graph, or figure and use it solve a problem requiring multiple steps

o Translate between tables, graphs, words, and symbolic notations (e.g., graph data from a table)

o Construct models given criteria

o Design investigation for a specific purpose or research question

o Conduct a designed investigation o Use concepts to solve non-routine

problems o Use & show reasoning, planning,

and evidence o Translate between problem &

symbolic notation when not a direct translation

o Select or devise approach among many alternatives to solve a problem

o Conduct a project that specifies a problem, identifies solution paths, solves the problem, and reports results

Analyze Break into constituent parts, determine how parts relate, differentiate between relevant-irrelevant, distinguish, focus, select, organize, outline, find coherence, deconstruct

o Retrieve information from a table or graph to answer a question

o Identify whether specific information is contained in graphic representations (e.g., table, graph, T-chart, diagram)

o Identify a pattern/trend

o Categorize, classify materials, data, figures based on characteristics

o Organize or order data o Compare/ contrast figures or data o Select appropriate graph and

organize & display data o Interpret data from a simple graph o Extend a pattern

o Compare information within or across data sets or texts

o Analyze and draw conclusions from data, citing evidence

o Generalize a pattern o Interpret data from complex graph o Analyze similarities/differences

between procedures or solutions

o Analyze multiple sources of evidence

o analyze complex/abstract themes

o Gather, analyze, and evaluate information

Evaluate Make judgments based on criteria, check, detect inconsistencies or fallacies, judge, critique

o Cite evidence and develop a logical argument for concepts or solutions

o Describe, compare, and contrast solution methods

o Verify reasonableness of results

o Gather, analyze, & evaluate information to draw conclusions

o Apply understanding in a novel way, provide argument or justification for the application

Create Reorganize elements into new patterns/structures, generate, hypothesize, design, plan, construct, produce

o Brainstorm ideas, concepts, or perspectives related to a topic

o Generate conjectures or hypotheses based on observations or prior knowledge and experience

o Synthesize information within one data set, source, or text

o Formulate an original problem given a situation

o Develop a scientific/mathematical model for a complex situation

o Synthesize information across multiple sources or texts

o Design a mathematical model to inform and solve a practical or abstract situation

Today’s Task Look at your current lessons and code

activities. Enhance lessons to increase opportunities

for rigor. Collaborate! DOK objectives in specific subject areas are

available, just ask: Math Language Arts Science Social Studies

So many choices: Differentiation I/E times Opportunities for:

Research Deeper understanding Self-selection Long-term projects More complex problems Integrating subjects

We all know. . . This work WON’T be finished today. You have choices! You know your students better than anyone. Collaboration among colleagues will greatly

help your efforts. There are resources for level 3 and 4

activities! This work will change the way your

classroom looks. Your students’ expectations and outcomes

will increase.

Thank you!