Software Evaluation Criteria

Automated Assignment Applications

RSCoyner

10/8/04

Software Evaluation Criteria

• Are all assignment materials from each chapter of the text included?

• Are additional assignment materials beyond those appearing in the text or supplements available?

• Is it easy to create new materials for the application?

• Can other course content be included?– Syllabus?– Homepage?

Software Evaluation Criteria

• How easy is it to setup the application?

• Is the documentation provided on the application– Understandable?– Organized so that answers to questions can

be found easily?– Extensive?

• Does it cover all the user questions that might arise?

Software Evaluation Criteria

• What other support is provided for instructors and students working with the system?– Printed training materials?– On-line help?– Help desk telephone number?

• How much does the application cost the student?

Software Evaluation Criteria

• What question types are available to format new materials?– True-false?– Multiple-choice?– Computational problems?– Short answer?– Completion?– Essays?

Software Evaluation Criteria

• Does the application use algorithms to generate new questions or numbers for each student’s attempt?– Is there an option to present the student with

the same question material on repeated tries?– Are the generated questions or numbers

“relatively” error free?

Software Evaluation Criteria

• Are error corrections to the question materials made by the publisher on a timely basis?– If not, does the user have the capability to

modify the questions?• Does this capability include the algorithms?

Software Evaluation Criteria

• How does the system handle student formatting differences?– Can it recognize alternative student formats

as acceptable?– Does it provide guidance to students as to

acceptable formatting?

Software Evaluation Criteria

• What feedback is provided to students?– Grade information?– Hints from instructor?– Hints linked to text?– Info as to where student made errors?

• Providing the correct answers?• Without providing the correct answers?

• Does the application control for successful completion before student can progress to another segment?

Software Evaluation Criteria

• Is the application integrated with– Blackboard?– Web CT?– Other applications?

• What integration features are included?– Login?– Grade book?– Other?

Software Evaluation Criteria

• What grade book features are available?– Different views?

• Individual student record?• All student records?• By item?• By type of item?• By section?

– Can the items be given different weights?• Point-based?• Percentage based?

Software Evaluation Criteria

• Can the grade information be presented by– Most recent attempt?– Average score received on all attempts?– The best score received in multiple attempts?

• Can the number of allowed attempts be specified?

• Can the time allowed for each attempt be specified?

• Can start and end times be specified?

Software Evaluation Criteria

• What statistical information can be presented for each item and overall?– Count?– Mean?– Mode?– Quartile?– Standard deviation?– Z-score?

Software Evaluation Criteria

• Can the student practice on a question after the grade has been recorded?

• Can a late penalty be assessed for completion of an assignment after it is due?

• Is the security adequate?– For student access?– For instructor access?

Software Evaluation Criteria

• How reliable is the application?– Is the application always available to

students?– Is the response time acceptable?– Are there frequent software bugs with the

system?– What avenues are available to communicate

problems to the provider?• Does the student have the ability to communicate

problems to the provider?