1 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 What Cisco’s ADI Group is Doing in...

What Cisco’s ADI Group is Doing in Performance Testing

PTC: June 18, 2004

Agenda

• Organizational Context

• Background of our Organization

• Assessment Strategy and Implementation

• Four Process Delivery Model

• Illustration: NetPASS

• Evaluation

• Moving Forward

Organizational Context

Areas we bring to bear:

• Statistical science

• Computing science

• Assessment science

• Psychological science

• Networking domain knowledge

Some background on our work in the Networking Academies

Global E-Learning Laboratory

Over 450,000 StudentsEnrolled

30,000 to 40,000 Online Tests DailyOver 30 million Total Tests Taken

10,000 + Academies,150 Countries

Over 24,000 Instructors

And there is probably one in your neighborhood….

New Courses Sponsored by IT Industry Leaders:

So we have some decent delivery volume…….

100,000

200,000

300,000

400,000

500,000

600,000

700,000

800,000

900,000

1,000,000

1,100,000

1,200,000

January

August

Assessment strategy and implementation in ADI

Some words about assessment implementation

• Assessment is about making inferences about knowledge, skills and abilities from limited data.

• Assessment development is not only about content, but about the models that allow us to make inferences from the observations.

Some over arching thoughts

• We conceptualize and work in assessment from end to end

• We have tremendous pressure to reduce time to market

• We will continually innovate in presentation, scoring and statistical technologies.

• All form assembly approaches require lots of items

• Items need to be complex to fight rote memorization

This suggests assessment and task design and construction are strongest leverage points

Performance Tasks

• Need similarity to real-world tasks

Cognitive

Visual

Physical

• Need to represent the relevant cognitive representations and features

Our primary framework for thinking about Assessment is Evidence Centered Design

• Provides a language for modern assessment

• Suggests structure for accomplishing the assessment

• Supports transfer and sharing

The world is complex ……..

From Mislevy, Steinberg & Almond (in press):

So is Evidence Centered Design

How do we describe the world and the important claims and tasks in the world

What components do the assessment tasks need?

How do we score the responses from tasks?

How do we pass that information to statistical analysis?

How do we understand and report these inferences?

Performance Task Design

• The need and design of performance tasks is determined during the claims and evidence process

What claims are we making about a student at the end of this course? (claims)

What evidence will we look for that determines the degree to which a student has met that claim? (evidence)

What situation can we set up for an examinee to demonstrate this evidence? (task specification)

Our goal is

• Not just to make a lot of content

• But to specify the underlying logic and design so many people can build off that specification

• Knowledge engineering of the domain

• This means we may need to track data at a more detailed level (e.g. field test items).

Claims database for CCNA

Contextualize the claim with scope specification and instructional guidance

Types of evidence you would look for

Representations to the student, and sources of variation

Representations from the student (work products), and features to detect

Add some additional information about relevant cognitive processes

Templates

Task Design

• Domain

A link to the claim or objective to which the task is seeking to provide evidence (e.g. Trouble shoot a 5 router topology running RIP with faults of type …..).

Enumeration of the relevant representations and artifacts (typology, routers, diagram, scenario)

Features to vary in setting (# of routers, # of hosts, type of fault)

Relevant requirement dimensions (collect information, determine problem, apply fix, check, repeat) versus (apply fix to problem that is described).

Task Design Template – Census of Relevant Task Characteristics

• Setting Corporation

Conference Center

University

• Building Length Less than 100m

More than 100m

• Ethernet Standard 10BaseT

100BaseT

• Subgroup Name Teacher

Student

Customer

• Bandwidth for a Subgroup Drop 10Mbps

100Mbps

• Growth Requirements Given

Task Specification Template -Choosing Key Features

• Setting Corporation

Conference Center

University

• Building Length Less than 100m

More than 100m

• Ethernet Standard 10BaseT

100BaseT

• Subgroup Name Teacher

Student

Customer

• Bandwidth for a Subgroup Drop 10Mbps

100Mbps

• Growth Requirements Given

Use Selected Key Task Features to Make Display Model for the Scenario

Problem Statement:

1. Setting sentence: A(n) setting is [create something that is a typical activity for this setting].

2. Building size sentence: The setting is buildingLength long.

3. Network type sentence: The setting has been asked to install a Ethernet Standard network for this [the typical activity for this setting created above].

4. subgroup 1 sentence: The subgroup 1 connections require a bandwidth of bandwidthForASubgroup 1.

7. Force closets sentence: No networking equipment can be stored in the subgroups 1, 2, & 3 area.

8. Location of POP sentence: The link to the internet is located locationOfExternalConnection(POP).

3. Instantiate individual values into the template:

• Simple (LAN Design)

• A small university needs to set up a network for the English department. The English Building is 50 m long. The building needs a 100BaseT network for the English department. Each faculty member’s drop requires a bandwidth of 100Mbps while the student drops require a bandwidth of 10Mbps. The faculty and student areas are housed at opposite corners of the building. The LAN needs to have following additional characteristics:

• 1. The faculty area needs 8 network drops.

• 2. The student area needs 1 network drop.

• 3. One networking device should be placed in the student area. Use the device that would meet the minimum standards in the student area.

• 4. Students need access to 10 network connections to connect laptop computers.

• 5. No networking equipment can be stored in the faculty or student areas.

• 6. The link to the internet is located in the faculty area.

Advantages: A model that…..

• Guides task and test construction

• Provides extremely high efficiency and scalability

• Forms the basis of defensible evidence for high stakes tests

• By relating task characteristics to difficulty, we can create tasks with known properties in advance!

And…

• These processes are being applied to simulation features as well

Mini-lecture:The Four Process Model for Assessment Delivery

A language for modern assessment implementation

The Four Process Delivery Model

Source: Almond, R., Steinberg, L. & Mislevy, R. 2000

ActivitySelection Process Presentation Process

Task/Evidence

CompositeLibrary

EvidenceIdentification

Process

EvidenceAccumulation

Process

AdministratorExaminee

Activity selection determines the activities to be invoked in the presentation process

Task/Evidence

CompositeLibrary

Process

Messages

Evidence identification looks for important features of the work product (can be used for feedback).

Task/Evidence

CompositeLibrary

Process

Work Product

Messages

Evidence identification looks for important features of the work product (can be used for feedback).

Task/Evidence

CompositeLibrary

Process

Work Product

Messages

Task Level Feedback

The result of evidence identification is values of observations

Task/Evidence

CompositeLibrary

Process

Observations

Work Product

Messages

Task Level Feedback

Evidence accumulation synthesizes the information for summary feedback

Task/Evidence

CompositeLibrary

Process

Observations

Work Product

Messages

Summary FeedbackTask Level Feedback

In adaptive situations, evidence accumulation informs activity selection

Task/Evidence

CompositeLibrary

Process

ExamineeRecord

Observations

Work Product

Messages

Some other processes go on in the background

Task/Evidence

CompositeLibrary

Process

ExamineeRecord

Observations

Work Product

Messages

PresentationMaterial

Evidence RuleData

Weights ofEvidence

Description DataAdministrator

Examinee

Find Task Fetch Task

Fetch Scoring DataFetch Weights

Four Process Model and Performance Tasks

• Performance tasks are really just a different means of presenting tasks to examinees

Evidence Identification of Performance Tasks

• Performance Tasks can be scored just like other more traditional tasks

0/1 (all or nothing)

Partial credit (degrees of correctness)

Different observables providing evidence for different student model variables

Evidence Accumulation and Performance Tasks

• Assessments containing performance tasks can still be summarized the same as more traditional assessments

Add number of correct

Ability estimates based on IRT

Probability estimates based on Bayesian Inference Nets

Activity Selection and Performance Tasks

• Assessments containing performance tasks can still implement the same activity selection procedures as do the more traditional assessments

Go to the next task in a specified sequence

Randomly select a task from a specified bucket

Task selected by examinee

Task chosen through adaptive algorithm

Illustration: NetPass Research Prototype

• Create on-line performance assessment of networking skills

• Focus on learner feedback rather than high-stakes testing

• Learn something about assessment in general

The task starts with a scenario and description of goals

To capture their mental model of the network, we ask them to draw the network with a diagramming tool

The diagram is created by dragging and dropping icons

When more detailed information is required, it is collected as well

When students are done, they press “Submit”…

And the graphical representation is converted into a text representation

The text file is scored following detailed rules, which result in characterizations of the work

And feedback is created for the student…

New Computational Tools Evolving all the time: A Bayesian Inference Network

Design

Network modeling

Troubleshoot Implement (Configure)

Domain disciplinary knowledge

Network Proficiency

Addressing

Hardware

Security

Protocols

Correlation

Part-of

Part-ofPart-of

Part-of

Part-ofPart-of

Prequisite

Correlation

OperationPlanning

Part-of

Summary Feedback – Probability of Mastery

Performance Tasks can also be Multiple-Choice

Assessment Evaluation

Exam Analyses

Examinee Comments

Item Performance

Moving Forward

Lessons Learned: Scoring Performance Tasks

• How can we get the most information from these performance tasks?

• Scoring the Product

–What if there is more than one accepted outcome?

• Scoring the Process

–More than one accepted process?

–Is there really a ‘right’ way?

• Be Wary of Expert Judgment

–Need to gather data to verify

Future Directions

• Figuring out how to present the performance task is only one aspect to delivering a performance assessment

• The issues is really applying existing technologies and processes to more complex performance-based items and assessments

1 © 2003 Cisco Systems, Inc. All rights reserved. PTC 6/18/04 What Cisco’s ADI Group is Doing in...

Documents

Cisco’s Agile Journey

Module Support on Cisco’s Integrated Services Routers ... · Module Support on Cisco’s Integrated Services Routers ... HWIC-1T IP Base 1941 ... Module Support on Cisco’s Integrated

Cisco’s acquisition strategy

Using Cisco’s Vmdc to Facilitate FISMA Compliance

CISCO’s Cloud Journey (Keynote at Cloud Symposium)

PTC University eLearning Simplifies PTC Creo Upgrade

Knowledge Map. Work Management Change Management + - > > > > Choose… PTC Windchill PTC Mathcad PTC Arbortext PTC Creo PTC Integrity PTC Servigistics All

Module Support on Cisco’s

Cisco’s IOS

Cisco’s Presence in the Datacenter

PTC Creo Update · PTC Creo Update PTC Creo 2.0 Flexible Modeling PTC Creo 3.0 Outlook Christoph von Andrian-Werburg Director PTC Creo Europe PTC User Sweden Gothenburg , May 8th

Cisco’s Public Sector Agile Architecture

PTC Tech Day - The PTC Value Roadmap - Shoemaker

PTC Creo 3.0 Assembly Best Practices and Design Exploration Best Practices... · November 17-18, 2015 Stuttgart, Germany #PTCLiveWorx PTC Creo 3.0 – Assembly Best Practices and

Pipeline Inline Inspection Seminar at Pipeline ... - ptc · PROGRAM OVERVIEW MONDAY, 18 MARCH 2019 PTC SIDE CONFERENCE Qualiﬁ cation & Recruitment PTC SIDE CONFERENCE Public Perception

Cisco’s Borderless Network Architecture Vision and Strategy

Understanding Cisco’s Security Focus And Its Integrated Architectural Approach · 2020-03-22 · Understanding Cisco’s Security Focus And Its Integrated Architectural Approach

Life Cycle Services for Cisco’s SAP HANA

PTC Mathcad Prime 5.0.0.0 Installation and Administration Guide · 2020. 8. 7. · PTC Mathcad Prime 2.0, PTC Mathcad Prime 3.0 or PTC Mathcad Prime 3.1, PTC Mathcad Prime 4.0, PTC

Cisco’s Made-for- Midmarket Portfolio