Reviewing and evaluating archived collections...Workshop 1 | Boulder, CO | September 18-20 2015 Mini-presentation Session 1: Evaluation Anthony Woodbury Reviewing and Evaluating Archived

Developing Standards for Data Citation & Attribution in Linguistics Workshop 1 | Boulder, CO | September 18-20 2015

Mini-presentation Session 1: Evaluation Anthony Woodbury

Reviewing and Evaluating Archived Collections

Reviewing and evaluating archived

collections Anthony C. Woodbury, University of Texas at Austin

[email protected]




1. Archived collections as intellectual communication




Archived collections !  Based on LANGUAGE DOCUMENTATION, the creation,

annotation, preservation, and dissemination of transparent records of a language.

!  To be TRANSPARENT, these records must have an APPARATUS, including metadata, translation, and sometimes transcription and annotation (including formative glossing, categorial parsing, immediate constituent analysis/tagging)




Archived collections Am ARCHIVED COLLECTION (OR CORPUS) is a body of interepreted records, coherent or not, for example:

•  Collections of records taken from various sources that are related to a given theme

•  ARCHIVAL FONDS, i.e., records emanating from a single project or group or individual

•  Increasingly, archives are explicitly AUTHORED: Digital archiving is seen as a primary medium for publication of corpora, and are assembled “like a book”, with coherence and purpose




Archived collections CORPUS THEORIZATION (‘how it hangs together’)

!  A “Noah’s Archive”: a one-time sampling of the uses of a language for a grammar, dictionary, or thumbnail linguistic ethnography

!  A collection of narratives (e.g., classic Boasian ‘texts’)

!  Additions to an extant corpus (e.g., Alice Taff’s Aleut conversation)

!  An ethnobotany (e.g., Jonathan Amith’s work on insects in Nahuat)

!  A song collection (e.g., Barwick, Birch & Evans on Iwaija Jurtbirrk love songs)

!  Experimental records, or recordings of grammatical elicitations along with summary paradigms or word- or sentence lists

!  Language acquisition corpus for a given child or community




Archived collections FURTHER WAYS COLLECTIONS ARE (OR COULD OR SHOULD BE) ‘AUTHORED’

!  Making a guide and finding aids

!  Including meta-documentation for the whole collection: !  describing the design and conduct of activities or projects from which

the corpus arose

!  articulating the theorization of the corpus

!  describing the appraisal process in assembling the corpus

!  Assigning the corpus to a genre (or inventing one)

!  Extending the ‘apparatus’ beyond simple translation and annotation (to philological techniques such as retranslation, commentary, version comparison, literary/poetic analysis)

!  Including narratives, logs, journals

!  Addressing a broad range of relevant audiences




Archived collections AUDIENCES

!  Community members interested in family, neighbors, social identity, verbal art, education, reclamation, or nostalgia

!  Scientists interested in philology, ethnohistory, human ecology, language typology, or linguistic theory

!  Humanists interested in linguistic and cultural expression and its products

!  General publics with any of these interests, and more

!  Holton (2012): archives made by linguists ‘aren’t just for linguists’

!  Conathan (2011:238) ‘Over time, the importance of records may change and records may be put to unanticipated uses’




Archived collections A MORE LIMITED THEORIZATION (contrasting with the broader view taken here):

!  Publish ‘enough texts to permit a verification of the analysis’. Samarin (1967:46)

(On this very narrow view, citation is only a way to source and verify the data, not a way to acknowledge and build on the work of the archive author)




2. Validating archived collections through review, evaluation, and

citation




Review, evaluation, citation !  Review, evaluation, and a ‘culture’ of citation

validate archived collections and reward authors and archivists

!  Without such rewards, there is a disincentive to the creation of archival collections due to:

!  Lack of academic ‘credit’ for a time-consuming activity

!  Fears about credit going to users higher on the ‘food chain’ (data ‘compilers’ < descriptivists < typologists < framework testers)




Review, evaluation, citation Journal editors can accept REVIEWS of archived collections, encouraging the reviews to offer: !  A BASIC DESCRIPTION of the archival collection !  A statement of ITS SCOPE, PURPOSE, AND THEORIZATION !  A consideration of the RELATIONSHIP of the collection to

other collections and scholarship !  An evaluation of how effectively and richly the CONTEXT

for the material is established !  An assessment of TECHNICAL ISSUES (systematicity;

design; appropriateness; clarity; and adequacy of transcription, translation, annotation)

!  An assessment of likely AUDIENCES (and of the author’s own position as speaker/non-speaker, archive user, etc.)




Review, evaluation, citation JOURNALS can encourage articles by author/archivists describing their archival collections

!  For example, Sophie Salfner has written an extensive article describing her Ikaan corpus at ELAR

THOSE CITING ARCHIVAL COLLECTIONS can describe the collection in more detail to provide context, and can draw on the author/archivist’s apparatus and interpretative material

!  As a group we might explore types of citation and how to accomplish it practically




Review, evaluation, citation WRITERS OF ACADEMIC PROMOTION LETTERS can point out that among the key achievements for a documentary linguist is a well-reviewed documentary corpus in a highly-regarded archive, and likewise can evaluate how well (if at all) linguists who rely on data make use of archived collections and their associated apparatus

In turn, UNIVERSITIES can establish criteria for reviewing and evaluating archived collections for promotion.




Review, evaluation, citation Ironically and somewhat circularly, what is fundamental to the emergence of an evaluative infrastructure such as I have described is the recognition that archived collections are a form of intellectual communication.




Thanks! Thanks for listening and thanks especially to my many, many interlocutors on this and related matters, including especially my AILLA colleagues Joel Sherzer, Pattie Epps, Susan Kung, and Heidi Johnson.




Linguistic data sets and tenure/promotion

evaluation Richard P. Meier (UT Austin)

& Sarah G. Thomason (U of Michigan)




Why Consider Data Sets in Tenure & Promotion?

!  Data sets are valuable. For so many of the world’s languages, the available data are so limited.

!  Developing data sets requires extensive linguistic skills—in fact, most of the skills required for a first-rate scholar in our field.




“Training” Our Colleagues

!  Departments, deans, promotion committees—all must be “trained” to understand what goes into the construction of a substantial dataset and how it should be evaluated.

!  Training can be successful! In comp ling, there are different publication standards than in much of linguistics. These publication standards have been accepted by promotion committees.

!  Some university officials are already receptive: UT’s recent president indicated that data sets can be one factor in a promotion case, “if published.”

But he wasn’t thinking of linguistics. And he didn’t define “published.”

!  We as linguists want a crucial role in determining how data sets are evaluated in our own field.




Our Role as Linguists !  We need specific criteria for evaluating data sets in

promotion decisions.

!  Linguists will need to come up with criteria and explain them to colleagues and deans.

!  Nonlinguists won't be likely to see why data sets are valuable research products unless linguists explain themselves well.




What Criteria?

To be weighed as a significant part of a tenure case, a dataset must be:

a.  extensive, and it must be organized and annotated, not

just a dump of raw data.

Sally: I type up my fieldnotes after each day's session, and the entire set of typed-up notes is a dataset; but nobody could use it easily but me or another Salishanist, and it couldn't possibly be a useful part of a promotion case.

The dataset must include detailed description of how it was constructed.




What Criteria?


b. publicly available, or at least (in the case of fieldnotes where privacy issues may prevent wider distribution) available to the speech community. The data set must also be available to whichever colleagues & committees are evaluating the promotion case.




What Criteria? To be weighed as a significant part of a tenure case, a

dataset must be:

c. deposited in an archiving system where the dataset will be maintained permanently in a form that permits distribution as widely as possible (within community constraints).

Even a dataset residing entirely on the compiler's computer could be relevant and important for a promotion case, IFF it is made available to everyone participating in evaluating the case, including, crucially, the external reviewers whose assessment is solicited.




What Criteria?


d. subject to evaluation by external reviewers as well as by colleagues in the candidate's department. If the dataset has not previously undergone peer review (for instance if it is unpublished), the external reviewers' opinions will be especially important in the promotion process.




… we need to know more about standards in the archiving community.

To develop criteria for evaluating datasets in a

promotion case…




Standards in the Archiving Community

!  a. What are the standards that govern the acceptance of deposits?

!  b. Are there ways in which those standards could be made more rigorous and/or more consistent across archives so that archived material could be given more weight in tenure and promotion processes?




Standards in the Archiving Community

!  c. Could archived material be subjected to a peer review process prior to "publication" on the web?

But what about cases where a speech community’s restrictions prevent "publication"?

!  d. Still more rigorously, could we imagine an e-journal that published peer-reviewed descriptions of archived materials?




How Much of a Promotion is a Dataset Worth?

!  We do not foresee the development of explicit guidelines for deciding how much a dataset is worth in a promotion process: too much depends on the nature, quality, and size of the dataset.

!  But note: no one can say with precision how much a particular type of article should be worth in promotion either. Quality of an article matters more than the publication venue.




Coda

!  We have more questions than answers.

!  But we hope that these thoughts will at least serve to start a useful discussion.

Thank You!




Evaluating dissertations based on primary data

collections

Meagan Dailey Ryan Henke University of Hawai‘i at Mānoa University of Hawai‘i at Mānoa

With contributions from

Nick Williams Jaime Perez Gonzalez University of Colorado at Boulder University of Texas at Austin




Our premise !  What does the usual dissertation look like?

!  We’re here re-thinking our approach to data management and its value

!  Why not re-think dissertations too?




Some possibilities 1.  Collection(s) of data

2.  Dictionaries

3.  Getting further outside the box




Collection(s) of data !  Annotated !  Archive-ready !  Accessible

!  Some considerations … !  What counts as a collection? !  Level of transcription !  Depth of linguistic analysis !  Quantity !  Scope !  Organization !  Access




Dictionaries !  Valuable to linguists and speech communities

!  Created from data collection !  Demonstrate scientific ability

!  Some considerations … !  Type !  Multimedia? !  Levels of linguistic analysis !  Internal organization !  Languages included !  Linked directly to collection of data?




Moving further out !  Other products linked to, built from a data

collection

!  Some ideas: !  Collection of articles !  Templates for language documentation !  Revitalization materials




Combining ideas !  Different combinations of these possibilities

!  Collection + grammar sketch

!  Collection + companion linguistic analysis !  Dictionary + collection

!  Dictionary + companion linguistic analysis !  Collection + documentation template




Questions, comments? !  Thanks!

!  Meagan Dailey [email protected]

!  Ryan Henke [email protected] !  Nick Williams

[email protected] !  Jamie Perez Gonzalez [email protected]




Altmetrics & Archives Susan Smythe Kung

Archive of the Indigenous Languages of Latin America The University of Texas at Austin

[email protected]




Traditional Assessment Metrics !  Article-level citation metrics: Measure how often

an article is cited in other articles, books, dissertations, etc.

!  Journal-level citation metrics: Measure the average citation count for the articles in a journal.

!  Author-level citation metrics: Measure total citations, or average citation count per article, of an individual author.




Altmetrics Altmetrics: Alternatives to traditional assessment metrics that focus on the broader impact of traditional academic output (articles, books), as well as other academic products, like data sets, software, blog posts, etc.

Altmetrics track

!  Page views and downloads (of articles, data sets, etc.)

!  Discussions, mentions, or non-academic citations (on social media, in blogs, on Wikipedia)

!  Saves (to academic or social bookmark sites)

!  Recommendations by academic literature services




Altmetrics Pros & Cons Altmetrics Pros:

!  They tell a fuller story about the impact of research beyond traditional academic citations.

!  They demonstrate value of all research products, not just traditional academic publications.

Altmetrics Cons:

!  They do not distinguish between positive and negative impact.

!  It is easy to cheat or game the system.




Altmetrics Tracking

Fairly easy to track:

!  Page views, media views

!  File downloads

!  Click throughs

Not as easy to track (but getting easier):

!  Citations of data, no matter where they are found (traditional academic citations, social media or blog mentions, tweets, Facebook likes, etc.)




Altmetrics in Archives 1 What Altmetrics should language archives track?

!  Page Views for the !  Collection level

!  Resource/bundle level !  File/item level

!  Media views

!  File downloads

!  Search queries?




Altmetrics in Archives 2 Who should get to see these metrics?

!  Any site user

!  Only Registered users

!  Only the depositors (the researchers who deposited the data)

!  Only the archives’ staff, but make these metrics available to depositors if they request it




Promoting archival data !  Share buttons that link directly to social media

sites (but which ones?)

!  Citation buttons that send the citation for the data set directly to reference management software (e.g. EndNote, Zotero), web-based reference services (RefWorks), and social sites (Mendeley).




Bibliography Altmetrics. (2015, September 4). In Wikipedia, The Free Encyclopedia. Retrieved 17:39, September 15, 2015, from https://en.wikipedia.org/w/index.php?title=Altmetrics&oldid=679451532

Article-level metrics. (2015, August 14). In Wikipedia, The Free Encyclopedia. Retrieved 17:37, September 15, 2015, from https://en.wikipedia.org/w/index.php?title=Article-level_metrics&oldid=676012719

Dinsmore A, Allen L, Dolby K (2014) Alternative Perspectives on Impact: The Potential of ALMs and Altmetrics to Inform Funders about Research Impact. PLoS Biol 12(11): e1002003. doi:10.1371/journal.pbio.1002003

Kwok, Roberta. (2013, August 22). Altmetrics make their mark. Nature, vol. 500: 491-3.

Journal ranking. (2015, August 21). In Wikipedia, The Free Encyclopedia. Retrieved 17:39, September 15, 2015, from https://en.wikipedia.org/w/index.php?title=Journal_ranking&oldid=677171635

National Information Standards Organization (2014, June 6). NISO Altmetrics Standards Project White Paper, draft 4. Retrieved from http://www.niso.org/apps/group_public/download.php/13295/niso_altmetrics_white_paper_draft_v4.pdf

Piwowar, Heather. (2013, January 10). Value all research products. Nature, vol. 493: 159.

Priem, Jason and Heather Piwowar. (2012, September 25). The launch of ImpactStory: using altmetrics to tell data-driven stories. Retrieved from http://blogs.lse.ac.uk/impactofsocialsciences/2012/09/25/the-launch-of-impactstor/




Thank you!

Documents

Reviewing and evaluating archived collections...Workshop 1 | Boulder, CO | September 18-20 2015 Mini-presentation Session 1: Evaluation Anthony Woodbury Reviewing and Evaluating Archived