CoderRank: Creating Gold Standards

Dr. Stuart W. ShulmanFounder & CEO, Texifter

@stuartwshulman

“…a wealth of information creates a poverty of attention.” - Herbert Simon, 1971

Text Classification

A 2500 year-old problem Plato argued it would be frustrating. It still is.

Grimmer & Stewart “Text as Data” Political Analysis (2013)

Volume is a problem for scholarsCoders are expensive

Groups struggle to accurately label text at scaleValidation of both humans and machines is “essential”

Some models are easier to validate than othersAll models are wrong

Automated models enhance/amplify, but don’t replace humansThere is no one right way to do this

“Validate, validate, validate”“What should be avoided then, is the blind use

of any method without a validation step.”

Free, Open-Source, Web-based Text Analytics Toolkit

Original Software Kernel: Tools for Measurement

Avoid Tennis Elbow

Items load to the screen and the coder hits the keystroke

Keystroke Human Coding

Human coding can be distributed to individuals, groups & crowds

Computer Science & National Science Foundation: Measure Everything

How fast?How reliable?

How accurate?

Annotator Speed

Redacted

Interrater Reliability: A Critical Measurement

Adjudication

CoderRank for enhanced machine-learning is our key innovation

Patent issued March 1, 2016

CoderRank for Enhanced Machine-learning

CoderRank is to text analytics what PageRank was to search. Just as Google said not all web pages are created equal, Texifter argues that not all humans are created equal. When training machines, it is best to rely most on the humans most likely to create a valid observation. We proposed a unique way to rank humans on trust and knowledge vectors.

ActiveLearning engines and human coding tools combine…

what humans do best… with what computers do best.

Humans and machines learning together

It is always good to keep humans “in-the-loop”

Word sense disambiguation (relevance)

Human coding converts into machine classifiers

Accumulated human coding becomes training data via machine-learning

Crowdsourcing accelerates the insight generation

Distribute coding for synchronous & asynchronous collaboration

Thank-you for listening!

@stuartwshulmanstu@texifter.com

CoderRank: Creating Gold Standards

Software

Creating Instrument Calibrations - Iowa State University · Creating Instrument Calibrations ... gold Au 1064.2 25 1150 950 1155 950 1160 1250 ... Most of the standards are in sheet

Morpheme Segmentation Gold Standards for Finnish …nlp.cs.swarthmore.edu/.../papers/creutz2004-morpheme.pdfMorpheme Segmentation Gold Standards for Finnish and English Mathias Creutz

Gold Standards: True and False

CREATING A PREMIER NORTH AMERICAN GOLD MINING …

CREATING A LEADING UNDERGROUND AFRICAN GOLD PRODUCER

Creating New MechWarrior Pilots for MechCommander Gold · Creating New MechWarrior Pilots for MechCommander Gold By CMunsta Once again, I present a preliminary document from my research

Creating enterprise standards 09302010

Creating a “Gold Standard” Physician Recruitment Strategy · CREATING A “GOLD STANDARD” PHYSICIAN RECRUITMENT STRATEGY PAGE 2 I. INTRODUCTION Physician recruitment is a multi-faceted

Creating a Prominent North American Gold Exploration Company · . Creating a Prominent North American Gold Exploration Company. Strong Portfolio of Highly Prospective District Scale

Creating a “Gold Standard” Physician Recruitment c.ymcdn.com/sites/ · PDF filerecruiter ratios, low costs, and high ... Creating a “Gold Standard” Physician Recruitment Strategy

Cross tab webinar gold standards in outsourcing

Creating a Prominent North American Gold Exploration Company · Corporate Presentation August 2018 . Creating a Prominent North American Gold Exploration Company. ... GFG Resources

Creating a modern, standards-compliant website

The WDC Gold Standards for Product Feature …webdatacommons.org/productcorpus/paper/WDC-EC_GS.pdfThe WDC Gold Standards for Product Feature Extraction and Product Matching ... TVs,

Searching for Gold Standards

Gold Standards

Gold standards for frog

Creating standards-compliant web pages in XHTML

New Standards and Format Training Creating Coupons

Developing MDS Standards Bronze, Silver and Gold Standard