ETS 2016 Research Publications — Year in Review | 1

ETS 2016 Research Publications — Year in Review

Since its founding in 1947, ETS has conducted and disseminated scientific research to support its products and services and to advance the measurement and education fields.

In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public.

This is a list of 2016 ETS research publications – those that ETS published as well as those that appeared in independent journals and books. In most cases, ETS staff members or affiliates were the lead authors of the independent publications listed. Publication titles link to abstracts, citation records, or full text where available. The full text for most ETS-published works may be accessed online at http://search.ets.org/researcher.

Published accounts of ETS research, including papers in the ETS Research Report series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards.

All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes.

Due to production deadlines, a small number of 2016 research publications may not be listed in this document.

ETS 2016 Research Publications — Year in Review | 2

Contents

This guide to 2016 ETS research publications lists works in these areas:

K–12 Teaching, Learning, and Education Policy ......................................... 3

Cognitive Science, Accessibility, and Technology ....................................... 8

English Language Learning and Assessment ............................................ 11

Automated Scoring and Natural Language Processing ............................... 14

Higher Education and Workforce Readiness ............................................. 17

Validity Research ................................................................................. 20

Statistics and Psychometrics .................................................................. 21

ETS 2016 Research Publications — Year in Review | 3

K–12 Teaching, Learning, and Education Policy

Using State Early Care and Education Workforce Registry Data to Inform Training-Related Questions: Issues to Consider

ACKERMAN, D. J. ETS Research Report No. RR-16-31, ETS Policy Information Report

Educating Young Children: Some Questions to be Answered ACKERMAN, D. J. Wiley Blog Post

Better Feedback for Better Teaching: A Practical Guide to Improving Classroom Observations

ARCHER, J., CANTRELL, S., HOLTZMAN, S. L., JOE, J. N., TOCCI, C. M., & WOOD, J. Book; Jossey-Bass A Wiley Brand

Mind the Gap: 20 Years of Progress and Retrenchment in School Funding and Achievement Gaps

BAKER, B. D., FARRIE, D., & SCIARRA, D. G. ETS Research Report No. RR-16-15

Understanding Consequential Assessment Systems for Teachers: Year 1 Report to the Los Angeles Unified School District

BELL, C. A., JONES, N., LEWIS, J., QI, Y., STICKLER, L., LIU, S., & MCLEOD, M. ETS Research Memorandum No. RM-16-12

Opt Out: An Examination of Issues

BENNETT, R. E. ETS Research Report No. RR-16-13

Meeting the Challenges to Measurement in an Era of Accountability

In Braun, H. (ed.) Routledge

ETS staff members were coauthors of two chapters in this volume:

BROWN, N. J. S., MADERER, C. S., & WOOD, J. Assessing Physical and Earth and Space Science in the Context of the NRC Framework for K-12 Science Education and the Next Generation Science Standards, Chapter 12, p349-386

KLAG, P. A., & KLUEMPEN, F. L. Transforming Assessment in Mathematics: Introduction, Chapter 13, p387-410

ETS 2016 Research Publications — Year in Review | 4

Teaching High School Students to Manage Time: The Development of an Intervention

BURRUS, J., JACKSON, T., HOLTZMAN, S., & ROBERTS, R. D. Improving Schools, Online First

Assessing Academic Language in an Elementary Mathematics Teacher Licensure Exam

CASTELLANO, K. E., DUCKOR, B., WIHARDINI, D., TELLEZ, K., & WILSON, M. Teacher Education Quarterly, v43 n1, p3-27

Assessing Mathematical Argumentation Through Automated Conversation

CAYTON-HODGES, G. A. Paper in Proceedings of the 38th Annual Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education, p1471-1478

Exploring Pre-K Age 4 Learning Standards and Their Role in Early Childhood Education: Research and Policy Implications

DEBRUIN-PARECKI, A., & SLUTZKY, C. Policy Information Report/ETS Research Report No. RR-16-14

Handbook of Research on Teaching, Fifth Edition

GITOMER, D. H., & BELL, C. A. (eds.) American Educational Research Association (AERA)

Two ETS staff members were coauthors of an introduction and a chapter in this 23-chapter volume:

GITOMER, D. H., & BELL, C. A. Introduction to Handbook of Research on Teaching, p1-6

SYKES, G., & WILSON, S. M. Can Policy (Re)form Instruction?, Chapter 13, p851-916

Exploring the Relationships Among Teacher Confidence, Learning, and Test Performance Within the English-for-Teaching Course

GU, L., & PAPAGEORGIOU, S. ETS Research Report No. RR-16-24

Promoting Cognitive and Social Aspects of Inquiry Through Classroom Discourse

JIN, H., WEI, X., DUAN, P., GUO, Y., & WANG, W. International Journal of Science Education, v38 n2, p319-343

ETS 2016 Research Publications — Year in Review | 5

Vertical Articulation of Cut Scores Across the Grades: Current Practices and Methodological Implications in the Light of the Next Generation of K-12 Assessments

KANNAN, P. ETS Research Report No. RR-16-29

The Dynamics of Opportunity in America: Evidence and Perspectives

KIRSCH, I., & BRAUN, H. (eds.)

Springer International Publishing

Choosing Our Future: A Story of Opportunity in America

KIRSCH, I. S., BRAUN, H., LENNON, M. L., & SANDS, A. ETS Center for Research on Human Capital and Education

Substantiating Educational Testing Service’s Products and Services With Research Support: A Formalized Methodology

LEUSNER, D. M. ETS Research Memorandum No. RM-16-13

When Lightning Strikes Twice: Profoundly Gifted, Profoundly Accomplished

MAKEL, M. C., KELL, H. J., LUBINSKI, D., PUTALLAZ, M., & BENBOW, C. P. Psychological Science: A Journal of the Association for Psychological Science, v27 n7, p1004-1018

Investigating the Relevance and Importance of Mathematical Content Knowledge Areas for Beginning Elementary School Teachers

MARTIN-RAUGH, M. P., REESE, C. M., HOWELL, H., TANNENBAUM, R. J., STEINBERG, J., & XU, J. ETS Research Memorandum No. RM-16-10

Investigating the Relevance and Importance of English Language Arts Content Knowledge Areas for Beginning Elementary School Teachers

MARTIN-RAUGH, M. P., REESE, C. M., PHELPS, G. C., TANNENBAUM, R. J., STEINBERG, J., & XU, J. ETS Research Memorandum No. RM-16-08

Investigating the Relevance and Importance of High-Leverage Practices for Beginning Elementary School Teachers

MARTIN-RAUGH, M. P., REESE, C. M., TANNENBAUM, R. J., STEINBERG, J., & XU, J. ETS Research Memorandum No. RM-16-11

ETS 2016 Research Publications — Year in Review | 6

Behaviorally Anchored Rating Scales: An Application for Evaluating Teaching Practice

MARTIN-RAUGH, M. P., TANNENBAUM, R. J., TOCCI, C. M., & REESE, C. M. Teaching and Teacher Education, v59, p414-419

Transition and Protective Agency of Early Childhood Learning Behaviors as Portents of Later School Attendance and Adjustment

MCDERMOTT, P. A., RIKOON, S. H., & FANTUZZO, J. W. Journal of School Psychology, v54, p59-75

Editorial: Noncognitive Assessment in K-12 Education: New Constructs and Approaches for the Twenty-First Century

PETWAY II, K. T., ALLEN, V., & ROBERTS, R. D. European Journal of Psychological Assessment, v32 n1, p1-4

Assessing Mathematical Knowledge for Teaching: The Role of Teaching Context

PHELPS, G., & HOWELL, H. The Mathematics Enthusiast, v13 n1-2, p52-70

Educational Measurement: From Foundations to Future

In Wells, C. S., & Faulkner-Bond, M. (eds.) Guilford Press

Six ETS staff members were coauthors of five chapters in this 25-chapter volume:

PITONIAK, M. J., & CIZEK, G. J. Standard Setting, Part II, Chapter, 3, p38-61

WELLS, C. S., RIOS, J., & FAULKNER-BOND, M. Testing Assumptions of Item Response Theory Models, Part III, Chapter 10, p162-181

WAY, D., & ROBIN, F. The History of Computer Based Testing, Part IV, Chapter 11, p185-207

VON DAVIER, A. A., & MISLEVY, R. J. Design and Modeling Frameworks for 21st Century Simulation and Game Based Assessments, Part IV, Chapter 14, p230-256

ZENISKY, A., MAZZEO, J., & PITONIAK, M. J. Toward Improving the Reporting of Test Score Results: Lessons Learned from the Evolution of NAEP Reports, Part VI, Chapter 20, p335-355

ETS 2016 Research Publications — Year in Review | 7

Eliciting Student Thinking: Definition, Research Support, and Measurement of the ETS® National Observational Teaching Examination (NOTE) Assessment Series

QI, Y., & SYKES, G. ETS Research Memorandum No. RM-16-06

Assessing Social-Emotional Learning

RIKOON, S. H., BRENNEMAN, M. W., & PETWAY II, K. T. National Association of State Boards of Education, p20-23

Research to Support Reading for Understanding

SANDBERG, H. Focus on ETS R&D, n2

Teaching and Assessing Mathematics Using Learning Progressions

SANDBERG, H. Focus on ETS R&D, n3

Automated Writing Evaluation: An Expanding Body of Knowledge

SHERMIS, M. D., BURSTEIN, J., ELLIOT, N., MIEL, S., & FOLTZ, P. W. In MacArthur, C. A., Graham, S., & Fitzgerald, J. (eds.)

Handbook of Writing Research, Second Edition. Guilford Press, Part V, Chapter 26, p395-409

A Case Study of Formative Reading Assessment to Support Teaching of EL Reading Comprehension

SHORE, J. R., WOLF, M. K., & HERITAGE, M. Journal of Educational Research and Innovation, v5 n2

Modeling and Explaining Content: Definition, Research Support, and Measurement of the ETS® National Observational Teaching Examination (NOTE) Assessment Series

STICKLER, L., & SYKES, G. ETS Research Memorandum No. RM-16-07

Underpinnings of the Praxis® Performance Assessment for Teachers (PPAT®)

SYKES, G. ETS Research Memorandum No. RM-16-14

ETS 2016 Research Publications — Year in Review | 8

In-Service Teachers’ Reasoning about Scenarios of Teaching Mathematics to English Language Learners

TURKAN, S. The Mathematics Enthusiast, v13 n1-2, p130-148

Statistical Results From the 2013 CBAL® English Language Arts Multistate Study: Parallel Forms for Policy Recommendation Writing

VAN RIJN, P. W., CHEN, J., & YAN-KOO, Y. ETS Research Memorandum No. RM-16-01

Statistical Results From the 2013 CBAL® English Language Arts Multistate Study: Parallel Forms for Argumentative Writing

VAN RIJN, P. W., & YAN-KOO, Y. ETS Research Memorandum No. RM-16-15

Leading a Classroom Discussion: Definition, Supporting Evidence, and Measurement of the ETS® National Observational Teaching Examination (NOTE) Assessment Series

WITHERSPOON, M., SYKES, G., & BELL, C. A. ETS Research Memorandum No. RM-16-09

Success Starts Young: Closing Achievement Gaps Where They Begin: A National Symposium on Early Learning Standards, Kindergarten Readiness, and Technology and Early Learning

YAFFE, D. ETS Policy Notes v23 n1

Cognitive Science, Accessibility, and Technology

From Cognitive-Domain Theory to Assessment Practice

BENNETT, R. E., DEANE, P., & VAN RIJN, P. W. Educational Psychologist, v51 n1, p82-107

Comparing Human and Automated Essay Scoring for Prospective Graduate Students With Learning Disabilities and/or ADHD

BUZICK, H. M., OLIVERI, M. E., ATTALI, Y., & FLOR, M. Applied Measurement in Education, v29 n3, p161-172

Assessing Mathematical Argumentation Through Automated Conversation

CAYTON-HODGES, G. A. Paper in Proceedings of the 38th Annual Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education, p1471-1478

ETS 2016 Research Publications — Year in Review | 9

Automated Classification of Collaborative Problem Solving Interactions in Simulated Science Tasks

FLOR, M., YOON, S.-Y., HAO, J., LIU, L., & VON DAVIER, A. A. Paper in Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, p31-41

An Evaluation of the Usefulness of Prosodic and Lexical Cues for Understanding Synthesized Speech of Mathematics

FRANKEL, L., & BROWNSTEIN, B. ETS Research Report No. RR-16-33

Development and Initial Evaluation of the ClearSpeak Style for Automated Speaking of Algebra

FRANKEL, L., BROWNSTEIN, B., SOIFFER, N., & HANSEN, E. G. ETS Research Report No. RR-16-23

Towards Accessible Innovative Assessment Items

HANSEN, E. G., CAVALIE, C., KING, T., HAKKINEN, M. T., WHITE, J. J., & GRANT, J. In Miesenberger, K., Bühler, C., & Panaz, P. (eds.)

Lecture Notes in Computer Science: Computers Helping People with Special Needs. Springer International Publishing, v9758, p251-258

Designing Innovative Science Assessments That Are Accessible for Students Who Are Blind

HANSEN, E. G., LIU, L., ROGAT, A., HAKKINEN, M. T., & DARRAH, M. Journal of Blindness Innovation and Research, v6 n1

Common Core TERA: Text Ease and Readability Assessor

JACKSON, G. T., ALLEN, L. K., & MCNAMARA, D. S. In Crossley, S. A., & McNamara, D. S. (eds.)

Adaptive Educational Technologies for Literacy Instruction. Routledge, Part I, Chapter 4, p49-68

Timing Game-based Practice in a Reading Comprehension Strategy Tutor

JACOVINA, M. E., JACKSON, G. T., SNOW, E. L., & MCNAMARA, D. S. In Micarelli, A., Stamper, J., & Panourgia, K. (eds.)

Lecture Notes in Computer Science: Intelligent Tutoring Systems. Springer International Publishing, v9684, p59-68

ETS 2016 Research Publications — Year in Review | 10

Lost in Springdale: An Interactive Narrative for Adult Literacy Learners

JOHNSON, A. M., JACOVINA, M. E., JACKSON, G. T., TIGHE, E. L., & MCNAMARA, D. S. In Micarell, A., Stamper, J., & Panourgia, L. (eds.)

Lecture Notes in Computer Science: Intelligent Tutoring Systems. Springer International Publishing, v9684, p476-477

Computerising Assessment: Impacts on Education Stakeholders

KATZ, I. R., & GORIN, J. S. In Brown, G. T. L, & Harris, L. R. (eds.)

Handbook of Human and Social Conditions in Assessment. Routledge, Chapter 27, p472-489

Visualizing Changes in Strategy Use across Attempts via State Diagrams: A Case Study

KERR, D. International Journal of Computer Games Technology, v2016, p1-9

Does It Matter Whether One Takes a Test on an iPad or a Desktop Computer?

LING, G. International Journal of Testing, v16 n4, p352-377

The Handbook of Cognition and Assessment: Frameworks, Methodologies, and Applications

RUPP, A. A., & LEIGHTON, J. P. (eds.) Wiley Blackwell

The following nine chapters in this 24-chapter book were either written or cowritten by current or former ETS staff:

RUPP, A. A., & LEIGHTON, J. P. Introduction to Handbook, Chapter 1, p1-11

KEEHNER, M., GORIN, J. S., FENG, G., & KATZ, I. R. Developing and Validating Cognitive Models in Assessment, Part I, Chapter 4, p75-101

KYLLONEN, P. C. Socio-emotional and Self-management Variables in Learning and Assessment, Part I, Chapter 8, p174-197

BEJAR, I. I., MISLEVY, R. J., & ZHANG, M. Automated Scoring with Validity in Mind, Part I, Chapter 10, p226-246

GONZÁLEZ-BRENES, J. P., BEHRENS, J. T., MISLEVY, R. J., LEVY, R., & DICERBO, K. E. Bayesian Networks, Part II, Chapter 14, p328-353

ETS 2016 Research Publications — Year in Review | 11

ORANJE, A., KEEHNER, M., PERSKY, H., CAYTON-HODGES, G. A., & FENG, G. Educational Survey Assessments, Part III, Chapter 18, p427-445

KERR, D., ANDREWS, J. J., & MISLEVY, R. J. The In-Task Assessment Framework for Behavioral Data, Part III, Chapter 20, p472-507

JACKSON, G. T., & ZAPATA-RIVERA, D. Conversation-Based Assessment, Part III, Chapter 23, p563-579

LEIGHTON, J. P., & RUPP, A. A. Conclusion to Handbook, Part III, Chapter 24, p580-587

The Good and the Bad of Game-Based Assessment

SANDBERG, H. Focus on ETS R&D, n1

iSTART-2: A Reading Comprehension and Strategy Instruction Tutor

SNOW, E. L., JACOVINA, M. E., JACKSON, G. T., & MCNAMARA, D. S. In Crossley, S. A., & McNamara, D. S. (eds.)

Adaptive Educational Technologies for Literacy Instruction. Routledge, Part I, Chapter 7, p104-121

Meeting Accessibility Challenges with Web Components

WHITE, J. J., HAKKINEN, M. T., & GRANT, J. Journal on Technology & Persons with Disabilities, v4, p. 29-41

Assessing Science Inquiry Skills in an Immersive, Conversation-Based Scenario

ZAPATA-RIVERA, D., LIU, L., CHEN, L., HAO, J., & VON DAVIER, A. A. In Daniel, B. K. (ed.)

Big Data and Learning Analytics in Higher Education. Springer International Publishing, Part II, p237-252

English Language Learning and Assessment

Setting Language Proficiency Score Requirements for English-as-a-Second-Language Placement Decisions in Secondary Education

BARON, P. A., & PAPAGEORGIOU, S. ETS Research Report No. RR-16-17

ETS 2016 Research Publications — Year in Review | 12

The Language Muse Activity Palette: Technology for Promoting Improved Content Comprehension for English Language Learners

BURSTEIN, J., & SABATINI, J. In Crossley, S. A., & McNamara, D. S. (eds.)

Adaptive Educational Technologies for Literacy Instruction. Routledge, Part III, Chapter 17, p275-281

Designing the TOEFL® Primary™ Tests

CHO, Y., GINSBURGH, M., MORGAN, R., MOULDER, B., XI, X., & HAUCK, M. C. ETS Research Memorandum No. RM-16-02

Effects of Printed Option Sets on Listening Item Performance Among Young English-as-a-Foreign-Language Learners

GETMAN, E., CHO, Y., & LUCE, C. ETS Research Memorandum No. RM-16-16

Modelling Communicative Language Ability: Skills and Contexts of Language Use

GU, L. In Tsagari, D., & Banerjee, J. (eds.)

Contemporary Second Language Assessment: Contemporary Applied Linguistics. Bloomsbury Academic. v4, Part One, Chapter 1, p17-35

Conceptualizing Accessibility for English Language Proficiency Assessments

GUZMAN-ORTH, D., LAITUSIS, C., THURLOW, M. & CHRISTENSEN, L. ETS Research Report No. RR-16-07

Creating a Next-Generation System of K-12 English Learner Language Proficiency Assessments

HAUCK, M. C., WOLF, M. K., & MISLEVY, R. J. ETS Research Report No. RR-16-06

Examining Content Representatives of a Young Learner Language Assessment: EFL Teachers’ Perspectives

HSIEH, C.-N. In Nikolov, M. (ed.)

Assessing Young Learners of English: Global and Local Perspectives. Springer International Publishing, v25, p93-107

ETS 2016 Research Publications — Year in Review | 13

Key Issues and Opportunities in the Initial Identification and Classification of English Learners

LOPEZ, A. A., POOLER, E., & LINQUANTI, R. ETS Research Report No. RR-16-09

Textual Complexity as a Predictor of Difficulty of Listening Items in Language Proficiency Tests

LOUKINA, A., YOON, S.-Y., SAKANO, J., WEI, Y., & SHEEHAN, K. M. Paper in Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers (COLING-16), p3245-3253

Language Muse: Automated Linguistic Activity Generation for English Language Learners

MADNANI, N., BURSTEIN, J., SABATINI, J. P., BIGGERS, K., & ANDREYEV, S. Paper in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics – System Demonstrations, p79-84

Effects of Strength of Accent on an L2 Interactive Lecture Listening Comprehension Test

OCKEY, G. J., PAPAGEORGIOU, S., & FRENCH, R. International Journal of Listening, v30 n1-2, p84-98

Situating Standard Setting within Argument-Based Validity

PAPAGEORGIOU, S., & TANNENBAUM, R. J. Language Assessment Quarterly, v13 n2, p109-123

Cognition and Literacy in English Language Learners at Risk for Reading Disabilities: A Latent Transition Analysis

SWANSON, H. L., KUDO, M., & GUZMAN-ORTH, D. Journal of Educational Psychology, v108 n6, p830-856

Learning and Development of Second and Foreign Language Pragmatics as a Higher-Order Language Skill: A Brief Overview of Relevant Theories

TIMPE-LAUGHLIN, V. ETS Research Report No. RR-16-35

Considerations for Evaluating Instruction for English Language Learners

TURKAN, S. In Gabriel, R. E., & Allington, R. L. (eds.) Evaluating Literacy Instruction: Principles and Promising Practices, Routledge, Chapter 4, p44-57

ETS 2016 Research Publications — Year in Review | 14

Complexities and Issues to Consider in the Evaluation of Content Teachers of English Language Learners

TURKAN, S., & BUZICK, H. M. Urban Education, v51 n2, p221-248

A Study of the Use of the TOEFL iBT® Test Speaking and Listening Scores for International Teaching Assistant Screening

WAGNER, E. ETS Research Report No. RR-16-18, TOEFL iBT-27

Validating English Language Proficiency Assessment Uses for English Learners: Academic Language Proficiency and Content Assessment Performance

WOLF, M. K., & FAULKNER-BOND, M. Educational Measurement: Issues and Practice, v35 n2, p6-18

Next-Generation Summative English Language Proficiency Assessments for English Learners: Priorities for Policy and Research

WOLF, M. K., GUZMAN-ORTH, D., & HAUCK, M. C. ETS Research Report No. RR-16-08

Integrating Scaffolding Strategies Into Technology-Enhanced Assessments of English Learners: Task Types and Measurement Models

WOLF, M. K., GUZMAN-ORTH, D., LOPEZ, A. A., CASTELLANO, K. E., HIMELFARB, I., &

TSUTAGAWA, F. S. Educational Assessment, v21 n3, p157-175

Automated Scoring and Natural Language Processing

Differences in Reaction to Immediate Feedback and Opportunity to Revise Answers for Multiple-Choice and Open-Ended Questions

ATTALI, Y., LAITUSIS, C., & STONE, E. A. Educational and Psychological Measurement, v76 n5, p787-802

Enhancing STEM Motivation through Personal and Communal Values: NLP for Assessment of Utility Value in Student Writing

BEIGMAN KLEBANOV, B., BURSTEIN, J., HARACKIEWICZ, J. M., PRINISKI, S. J., &

MULHOLLAND, M. Paper in Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, p199-205

ETS 2016 Research Publications — Year in Review | 15

Topicality-Based Indices for Essay Scoring

BEIGMAN KLEBANOV, B., FLOR, M., & GYAWALI, B. Paper in Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, p63-72

Semantic Classifications for Detection of Verb Metaphors

BEIGMAN KLEBANOV, B., LEONG, C.-W., GUTIERREZ, E. D., SHUTOVA, E., & FLOR, M. Paper in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, p101-106

Argumentation: Content, Structure, and Relationship with Essay Quality

BEIGMAN KLEBANOV, B., STAB, C., BURSTEIN, J., SONG, Y., GYAWALI, B., & GUREVYCH, I. Paper in Proceedings of the 3rd Workshop on Argument Mining, p70-75

A Protocol for Annotating Parser Differences

BRUNO, J. V., CAHILL, A., & GYAWALI, B. ETS Research Report No. RR-16-02

Building e-rater® Scoring Models Using Machine Learning Methods

CHEN, J., FIFE, J. H., BEJAR, I. I., & RUPP, A. A. ETS Research Report No. RR-16-04

Identifying Useful Features to Detect Off-Topic Essays in Automated Scoring Without Using Topic-Specific Training Essays

CHEN, J., & ZHANG, M. Paper in Springer Proceedings in Mathematics & Statistics, Quantitative Psychology Research. Springer International Publishing, v167, p315-326

Automated Guidance for Student Inquiry

GERARD, L. F., RYOO, K., MCELHANEY, K. W., LIU, O. L., RAFFERTY, A. N., & LINN, M. C. Journal of Educational Psychology, v108 n1, p60-81

String Kernels for Native Language Identification: Insights from Behind the Curtains

IONESCU, R. T., POPESCU, M., & CAHILL, A. Computational Linguistics, v42 n3, p491-525

Validation of Automated Scoring of Science Assessments

LIU, O. L., RIOS, J. A., HEILMAN, M., GERARD, L., & LINN, M. C. Journal of Research in Science Teaching, v53 n2, p215-233

ETS 2016 Research Publications — Year in Review | 16

Automatically Scoring Tests of Proficiency in Music Instruction

MADNANI, N., CAHILL, A., & RIORDAN, B. Paper in Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, p217-222

Model Combination for Correcting Preposition Selection Errors

MADNANI, N., HEILMAN, M., & CAHILL, A. Paper in Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, p136-141

The Effect of Multiple Grammatical Errors on Processing Non-Native Writing

NAPOLES, C., CAHILL, A., & MADNANI, N. Paper in Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, p1-11

Bootstrapping Development of a Cloud-Based Spoken Dialog System in the Educational Domain From Scratch Using Crowdsourced Data

RAMANARAYANAN, V., SUENDERMANN-OEFT, D., LANGE, P., IVANOV, A. V., EVANINI, K., YU, Z., TSUPRUN, E., & QIAN, Y. ETS Research Report No. RR-16-16

The Criterion® Online Writing Evaluation Service

RAMINENI, C., & DEANE, P. In Crossley, S. A., & McNamara, D. S. (eds.)

Adaptive Educational Technologies for Literacy Instruction. Routledge, Part II, Chapter 10, p163-183

A Review of Evidence Presented in Support of Three Key Claims in the Validity Argument for the TextEvaluator® Text Analysis Tool

SHEEHAN, K. M. ETS Research Report No. RR-16-12

Monitoring the Performance of Human and Automated Scores for Spoken Responses

WANG, Z., ZECHNER, K., & SUN, Y. Language Testing, Online First

Evaluating the Advisory Flags and Machine Scoring Difficulty in the e-rater® Automated Scoring Engine

ZHANG, M., CHEN, J., & RUAN, C. ETS Research Report No. RR-16-30

ETS 2016 Research Publications — Year in Review | 17

Use of Automated Scoring and Feedback in Online Interactive Earth Science Tasks

ZHU, M., LIU, O. L., MAO, L., & PALLANT, A. Paper in Proceedings of the 2016 IEEE Integrated STEM Education Conference (ISEC), p224-230

Higher Education and Workforce Readiness

Argumentative Writing in Assessment and Instruction: A Comparative Perspective

BEIGMAN KLEBANOV, B., KAUFER, D., YEOH, P., ISHIZAKI, S., & HOLTZMAN, S. In Stukker, N., Spooren, W., & Steen, G. (eds.)

Genre in Language, Discourse and Cognition. De Gruyter, p167-192

Agent-Based Modeling of Collaborative Problem Solving

BERGNER, Y., ANDREWS, J., ZHU, M., & GONZALES, J. E. ETS Research Report No. RR-16-27

Who’s on First? Gender Differences in Performance on the SAT® Test on Critical Reading Items With Sports and Science Content

CHUBBUCK, K., CURLEY, W. E., & KING, T. C. ETS Research Report No. RR-16-26

In Search of Validity Evidence in Support of the Interpretation and Use of Assessments of Complex Constructs: Discussion of Research on Assessing 21st Century Skills

ERCIKAN, K., & OLIVERI, M. E. Applied Measurement in Education, v29 n4, p310-318

The Assessment of Biases in Cognition: Development and Evaluation of an Assessment Instrument for the Measurement of Cognitive Bias

GERTNER, A., ZAROMB, F. M., SCHNEIDER, R., ROBERTS, R. D., & MATTHEWS, G. MITRE Technical Report, Case No. 16-0956, p1-245

Assessing Perceptions of Interpersonal Behavior with a Video-Based Situational Judgment Test

GOLUBOVICH, J., SEYBERT, J., MARTIN-RAUGH, M. P., NAEMI, B. D., VEGA, R. P., & ROBERTS, R. D. International Journal of Testing, Online First

ETS 2016 Research Publications — Year in Review | 18

Contemporary Assessment Challenges: The Measurement of 21st Century Skills

GREIFF, S., & KYLLONEN, P. C. Applied Measurement in Education, v29 n4, p243-244

Assessing Intercultural Competence in Higher Education: Existing Research and Future Directions

GRIFFITH, R. L., WOLFELD, L., ARMON, B. K., RIOS, J. A., & LIU, O. L. ETS Research Report No. RR-16-25

Collaborative Problem Solving Skills versus Collaboration Outcomes: Findings from Statistical Analysis and Data Mining

HAO, J., LIU, L., VON DAVIER, A. A., KYLLONEN, P. C., & KITCHEN, C. Paper in Proceedings of the 9th International Conference on Educational Data Mining, p382-387

Perspectives on the Development and Future of Advanced Placement® Human Geography

HILDEBRANT, B. Journal of Geography, v115 n3, p137-138

Information Literacy in Digital Environments: Construct Mediation, Construct Modeling, and Validation Processes

KATZ, I. R., & ELLIOT, N. In D’Angelo, B. J., Jamieson, S., Maid, B., & Walker, J. R. (eds.)

Information Literacy: Research and Collaboration Across Disciplines. The WAC Clearinghouse & University Press of Colorado, Chapter 5, Part I, p97-116

We Are What We Repeatedly Do – But in What Context? The Role of Situational Factors in Assessing Personality via Nonverbal Behavior

KLAFEHN, J., KELL, H. J., ANDREWS, J., BARNWELL, P., & KHAN, S. ETS Research Report No. RR-16-28

The Prediction of Labor Force Status: Implications From International Adult Skill Assessments

LI, T., VON DAVIER, M., HANCOCK, G. R., & KIRSCH, I. S. ETS Research Report No. RR-16-11

Handbook of Research on Technology Tools for Real-World Skill Development

In Rosen, Y., Ferrara, S., & Mosharraf, M. (eds.) Information Science Reference

ETS 2016 Research Publications — Year in Review | 19

ETS staff members were coauthors of three chapters in this two-volume book:

LIU, L., HAO, J., VON DAVIER, A. A., KYLLONEN, P. C., & ZAPATA-RIVERA, D. A Tough Nut to Crack: Measuring Collaborative Problem Solving, Chapter 13, p344-359

ROUET, J.-F., VÖRÖS, Z. & VON DAVIER, M. Assessing Problem Solving in Technology-Rich Environments: What Can We Learn from Online Strategy Indicators?, Chapter 27, p706-724

HE, Q., & VON DAVIER, M. Analyzing Process Data from Problem-Solving Items with N-Grams: Insights from a Computer-Based Large-Scale Assessment, Chapter 29, p750-777

An Investigation of the Use and Predictive Validity of Scores From the GRE® revised General Test in a Singaporean University

LIU, O. L., KLIEGER, D. M., BOCHENEK, J. L., HOLTZMAN, S., & XU, J. ETS Research Report No. RR-16-05, ETS GRE®-16-01

Investigating College Learning Gain: Exploring a Propensity Score Weighting Approach

LIU, O. L., LIU, H., ROOHR, K. C., & MCCAFFREY, D. F. Journal of Educational Measurement, v53 n3, p352-367

Assessing Critical Thinking in Higher Education: The HEIghten™ Approach and Preliminary Validity Evidence

LIU, O. L., MAO, L., FRANKEL, L., & XU, J. Assessment & Evaluation in Higher Education, v41 n5, p677-694

Pilot Testing the Chinese Version of the ETS® Proficiency Profile Critical Thinking Test

LIU, O. L., MAO, L., ZHAO, T., YANG, Y., XU, J., & WANG, Z. ETS Research Report No. RR-16-37

Using Indirect vs. Direct Measures in the Summative Assessment of Student Learning in Higher Education

LUCE, C., & KIRNAN, J. P. Journal of the Scholarship of Teaching and Learning, v16 n4, p75-91

Prosocial Knowledge Mediates Effects of Agreeableness and Emotional Intelligence on Prosocial Behavior

MARTIN-RAUGH, M. P., KELL, H. J., & MOTOWIDLO, S. J. Personality and Individual Differences, v90, p41-49

ETS 2016 Research Publications — Year in Review | 20

Implicit Trait Policies About Prosocial Professionalism

MOTOWIDLO, S. J., KELL, H. J., GHOSH, K., & MARTIN-RAUGH, M. P. In Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (eds.) The Oxford Handbook of Organizational Citizenship Behavior, Online First. Oxford University Press

SJTs as Measures of General Domain Knowledge for Multimedia Formats: Do Actions Speaker Louder Than Words?

NAEMI, B., MARTIN-RAUGH, M. P., & KELL, H. J. Industrial and Organizational Psychology: Perspectives on Science and Practice, v9 n1, p77-83

Dimensionality Analyses of the GRE® revised General Test Verbal and Quantitative Measures

ROBIN, F., BEJAR, I. I., LIANG, L., & RIJMEN, F. ETS Research Report No. RR-16-20, ETS GRE®-16-02

Investigating Student Learning Gains in College: A Longitudinal Study

ROOHR, K. C., LIU, H., & LIU, O. L. Studies in Higher Education, Online First

Assessing Digital Information Literacy in Higher Education: A Review of Existing Frameworks and Assessments With Recommendations for Next-Generation Assessment

SPARKS, J. R., KATZ, I. R., & BEILE, P. M. ETS Research Report No. RR-16-32

Exploring Validity and Fairness of the Text-to-Speech Accommodation for College and Career Readiness Assessments

STONE, E. A., LAITUSIS, C., & CAVALIE, C. PARCC White Paper

Examining the Patterns of Communication and Connections Among Engineering Professional Skills in Group Discussion: A Network Analysis Approach

ZHU, M., & ZHANG, M. Paper in Proceedings of 2016 IEEE Integrated STEM Education Conference (ISEC), p181-188

Validity Research

Effort in Low-Stakes Assessments: What Does It Take to Perform as Well as in a High-Stakes Setting?

ATTALI, Y. Educational and Psychological Measurement, v76 n 6, p1045-1058

ETS 2016 Research Publications — Year in Review | 21

What’s in a Topic? Exploring the Interaction Between Test-taker Age and Item Content in High-States Testing

BANERJEE, J., & PAPAGEORGIOU, S. International Journal of Listening, v30 n1-2, p8-24

Validity as the Evaluation of the Claims Based on Test Scores

KANE, M. T. Assessment in Education: Principles, Policy & Practice, v23 n2, p309-311

How Developments in Psychology and Technology Challenge Validity Argumentation

MISLEVY, R. J. Journal of Educational Measurement, v53 n3, p265-292

Development of the Mission Skills Assessment and Evidence of Its Reliability and Internal Structure

PETWAY II, K. T., RIKOON, S. H., BRENNEMAN, M. W., BURRUS, J., & ROBERTS, R. D. ETS Research Report No. RR-16-19

Facets of Conscientiousness and Their Differential Relationships with Cognitive Ability Factors

RIKOON, S. H., BRENNEMAN, M. W., KIM, L. E., KHORRAMDEL, L., MACCANN, C., BURRUS, J., & ROBERTS, R. D. Journal of Research in Personality, v61, p22-34

Statistics and Psychometrics

An Evaluation of Different Statistical Targets for Assembling Parallel Forms in Item Response Theory

ALI, U. S., & VAN RIJN, P. W. Applied Psychological Measurement, v40 n3, p163-179

Beyond Correctness: Development and Validation of Concept-based Categorical Scoring Rubrics for Diagnostic Purposes

ARIELI-ATTALI, M., & LIU, Y. Educational Psychology, v36 n6, p1083-1101

Semantic Processes and Individual Differences Detected Through Error-Related Negativities

BALASS, M., HALDERMAN, L. K., BENAU, E. M., & PERFETTI, C. A. Journal of Neurolinguistics, v37, p82-97

ETS 2016 Research Publications — Year in Review | 22

Fairness in Educational Assessment and Measurement

DORANS, N. J., & COOK, L. L. (eds.) National Council on Measurement in Education/Routledge

The following nine chapters in this 16-chapter volume were either written by or cowritten by current or former ETS staff:

DORANS, N. J., & COOK, L. L. Introduction, Chapter 1, p1-6

ZIEKY, M. J. Fairness in Test Design and Development, Part I, Chapter 2, p9-32

LIU, J. & DORANS, N. J. Fairness in Score Interpretation, Part I, Chapter 5, p77-96

SINHARAY, S. Commentary on Ensuring Fairness in Test Design, Construction, Administration, and Scoring, Part I, Chapter 6, p97-108

CASTELLANO, K. E., & KOLEN, M. J. Comparing Test Scores across Grade Levels, Part II, Chapter 8, p135-156

STONE, E. A., & COOK, L. L. Testing Individuals in Special Populations, Part II, Chapter 9, p157-180

SIRECI, S. G., RIOS, J. A., & POWERS, S. Comparing Scores from Tests Administered in Different Languages, Part II, Chapter 10, p181-202

ZWICK, R., & DORANS, N. J. Philosophical Perspectives on Fairness in Educational Assessment, Part III, Chapter 14, p267-282

DORANS, N. J., & COOK, L. L. The Implications of Societal Changes for Fairness Assessment, Part III, Chapter 16, p295-297

Applications of Multidimensional Item Response Theory Models With Covariates to Longitudinal Test Data

FU, J. ETS Research Report No. RR-16-21

LRDIF ES: A SAS Macro for Logistic Regression Tests for Differential Item Functioning of Dichotomous and Polytomous Items

FU, J., & MONFILS, L. ETS Research Memorandum No. RM-16-17

ETS 2016 Research Publications — Year in Review | 23

A Note on the Poisson’s Binomial Distribution in Item Response Theory

GONZÁLEZ, J., WIBERG, M., & VON DAVIER, A. A. Applied Psychological Measurement, v40 n4, p302-310

Predicting Rights-Only Score Distributions from Data Collected Under Formula Score Instructions

GUO, H. Psychometrika, Online First

A New Procedure for Detection of Students’ Rapid Guessing Responses Using Response Time

GUO, H., RIOS, J. A., HABERMAN, S. J., LIU, O. L., WANG, J., & PAEK, I. Applied Measurement in Education, v29 n3, 173-183

Evaluation of Different Scoring Rules for a Noncognitive Test in Development

GUO, H., ZU, J., KYLLONEN, P. C., & SCHMITT, N. ETS Research Report No. RR-16-03

A Program for Nonparametric Raw-to-Scale Conversion

HABERMAN, S. J. ETS Research Memorandum No. RM-16-03

A Program for Iterative Proportional Fitting of Two-Way Arrays

HABERMAN, S. J. ETS Research Memorandum No. RM-16-04

Handbook of Item Response Theory, Volume Two: Statistical Tools

In van der Linden, W. J. (ed.) Chapman and Hall/CRC

Two ETS staff members are authors of three chapters in this volume:

HABERMAN, S. J. Exponential Family Distributions Relevant to IRT, Chapter 4, p47-69

HABERMAN, S. J. Models with Nuisance and Incidental Parameters, Chapter 9, p151-170

MISLEVY, R. J. Missing Responses in Item Response Modeling, Chapter 10, p171-194

ETS 2016 Research Publications — Year in Review | 24

Taming Log Files from Game/Simulation-Based Assessments: Data Models and Data Analysis Tools

HAO, J., SMITH, L., MISLEVY, R. J., VON DAVIER, A. A., BAUER, M. I. ETS Research Report No. RR-16-10

Investigating Robustness of Item Response Theory Proficiency Estimators to Atypical Response Behaviors Under Two-Stage Multistage Testing

KIM, S., & MOSES, T. ETS Research Report No. RR-16-22, ETS GRE®-16-03

Bayesian Psychometric Modeling

LEVY, R., & MISLEVY, R. J. Book; Chapman & Hall/CRC

Linking Composite Scores: Effects of Anchor Test Length and Content Representativeness

LIN, P., DORANS, N. J., & WEEKS, J. P. ETS Research Report No. RR-16-36

Estimating True Student Growth Percentile Distributions Using Latent Regression Multidimensional IRT Models

LOCKWOOD, J. R., & CASTELLANO, K. E. Educational and Psychological Measurement, Online First

Analyzing Fairness Among Linguistic Minority Populations Using a Latent Class Differential Item Functioning Approach

OLIVERI, M. E., ERCIKAN, K., LYONS-THOMAS, J., & HOLTZMAN, S. Applied Measurement in Education, v29 p17-29

Psychometrics in Support of a Valid Assessment of Linguistic Minorities: Implications for the Test and Sampling Designs

OLIVERI, M. E., & VON DAVIER, A. A. International Journal of Testing, v16 n3, p 220-239

The Impact of Population Shift on Equating and Differential Item Functioning of Anchor Items: An Empirical Study

QU, Y., WEI, Y, & MORGAN, R. ETS Research Memorandum No. RM-16-05

Handbook of Test Development, 2nd Edition

In Lane, S., Raymond, M. R., & Haladyna, T. M. (eds.) Routledge

ETS 2016 Research Publications — Year in Review | 25

Thirteen ETS staff members were coauthors of nine chapters in this 32-chapter handbook:

RICONSCENTE, M. M., MISLEVY, R. J., & CORRIGAN, S. Evidence-Centered Design, Chapter 3

KANE, M. T. Validation Strategies: Delineating and Validating Proposed Interpretations and Uses of Test Scores, Chapter 4

ZIEKY, M. J. Developing Fair Tests, Chapter 5

GRAF, E. A., & VAN RIJN, P. W. Learning Progressions as a Guide for Design: Recommendations Based on Observations from a Mathematical Assessment, Chapter 9

KYLLONEN, P. C. Designing Tests to Measure Personal Attributes and Noncognitive Skills, Chapter 10

SWYGERT, K. A., & WILLIAMSON, D. M. Using Performance Tasks in Credentialing Tests, Chapter 15

SHERMIS, M. D., BURSTEIN, J., BREW, C., HIGGINS, D., & ZECHNER, K. Recent Innovations in Machine Scoring of Student and Test Taker Written and Spoken Responses, Chapter 17

WENDLER, C., & WALKER, M. E. Practical Issues in Designing and Maintaining Multiple Test Forms, Chapter 22

DAVEY, T., PITONIAK, M. J., & SLATER, S. C. Designing Computerized Adaptive Tests, Chapter 24

An Extension of IRT-Based Equating to the Dichotomous Testlet Response Theory Model

TAO, W., & CAO, Y. Applied Measurement in Education, v29 n2, p108-121

Assessment of Fit of Item Response Theory Models Used in Large-Scale Educational Survey Assessments

VAN RIJN, P. W., SINHARAY, S., HABERMAN, S. J., & JOHNSON, M. S. Large-scale Assessments in Education, v4 n10

ETS 2016 Research Publications — Year in Review | 26

High-Performance Psychometrics: The Parallel-E Parallel-M Algorithm for Generalized Latent Variable Models

VON DAVIER, M. ETS Research Report No. RR-16-34

An Evaluation of the Single-Group Growth Model as an Alternative to Common-Item Equating

WEI, Y., & MORGAN, R. ETS Research Report No. RR-16-01

A Scaled F Distribution as an Approximation to the Distribution of Test Statistics in Covariance Structure Analysis

WU, H., & LIN, J. Structural Equation Modeling: A Multidisciplinary Journal, v23 n3, p409-421

Exploring the Effectiveness of a Measurement Error Tutorial in Helping Teachers Understand Score Report Results

ZAPATA-RIVERA, D., ZWICK, R., & VEZZU, M. Educational Assessment, v21 n3, p215-229

Classification of Writing Patterns Using Keystroke Logs

ZHANG, M., HAO, J., LI, C., & DEANE, P. Paper in Springer Proceedings in Mathematics & Statistics, Quantitative Psychology Research. Springer International Publishing, v167, p299-314

Correspondence Analysis of Multirelational Multilevel Networks

ZHU, M., KUSKOVA, V., WASSERMAN, S., & CONTRACTOR, N. In Lazega, E., & Snijders, T. A. B. (eds.)

Multilevel Network Analysis for the Social Sciences: Theory, Methods and Applications. Springer International Publishing, Part II, v12, p145-172

Using Networks to Visualize and Analyze Process Data for Educational Assessment

ZHU, M., SHU, Z., & VON DAVIER, A. A. Journal of Educational Measurement, v53 n2, p190-211