Review of online argument representation platforms€¦ · theory (Manosevitch 2014). Consequently, how developers talk about the objectives of the platform may differ from concepts

AREVIEWOFONLINEARGUMENTREPRESENTATIONPLATFORMS

AworkingpaperfortheSchoiloproject

https://www.scholio.net

December2018

MartinKing

CentrefortheStudyofDemocracy

UniversityofWestminster,UK

Introduction

Adistinction can be drawnbetween attempts to promotemore deliberative and humble

engagement on common forms of online communication and efforts to design novel

platforms(Shane2004,Davies2009).Researchintocommonformsofonlinecommunication

focusesonlightlystructuredforumsandsocialmediaplatforms,oftennotdesignedexplicitly

tosupportdeliberation(LoukisandWimmer2010).DaviesandChandler(2011)observehow

thisresearchhasrevealedthelimitationsoftheopeninternetandlightlystructuredforums

in supporting online deliberation, highlighting challenges including noise, information

overload,negativebehaviourandpolarisation(Sunstein2001).Researchershavepointedto

designandstructureasameansofaddressingsomeoftheseissuesandchallenges(Pingree

2009,ColemanandMoss2012,Manosevitch2014).

Thesecondstrandofworkononlinecommunicationfocusesonthedevelopmentofnovel

platformsthataimtosupportdeliberation,often,butnotexclusively,atscale(e.g.thereis

workontheeffectivesupportofsmallgroupdeliberationonline).Aparticularlyvibrantsub-

fieldthatwillbethefocusofthispaperistheuseofdifferentformsofargumentmapping

andvisualisationtechniques.Thedevelopersoftheseplatformsareofteninfluencedbyfields

outsideofdeliberativedemocratictheory,notablyinformallogicandcollectiveintelligence.

Althoughtheliteraturehighlightsthepromiseofdesign(Pingree2009,ColemanandMoss

2012), research into design has often been limited (Towne and Herbslep 2012), typically

neglectingmoreexperimentalplatformsandthemorenuancedandnovelformsofdesign

they utilise (Pingree 2009, Delborne et al 2011, Davies and Chandler 2011, Towne and

Herbslep2012).

This paper provides an analysis of platforms and tools that utilise novel design choices

associatedwithargumentmappingandvisualisationtosupportlargescaledeliberation.In

the first section, the paper discussesmethodological challenges in studying this field and

outlines theapproach to identifying, collectingand reviewing relevantonlinedeliberation

platforms.Thepaperidentifies12exemplarycasesandexplorestheseexamplesingreater

detail.Themainbodyofthepaperanalyseseachoftheplatformsindividually,explainingthe

background,objectives,design,existingresearchandapplicationsoftheplatform.

Methodology

TheMethodologicalChallengesofStudyingOnlineDeliberationPlatformsA number of methodological challenges need to be navigated in analysing novel online

deliberationplatforms.Thefirstissueweencounterconcernsdefinition.Inthispaper,wewill

usetheterm“experimentalonlinedeliberationplatform”.Thepaucityofsystematicanalysis

of suchexperimental platforms (TowneandHerbslep2012)means that there is a lackof

agreeddefinitionandclassificationofthisrangeofplatforms.Thischallengeisexacerbated

bythesensethattheterm“deliberation”iscontestedwithnoclearconsensusonitsbasic

characteristics (ColemanandMoss2012).Furthermore,developersof themostsuccessful

andpromisingplatformsareofteninfluencedbyfieldsdistinctfromdeliberativedemocratic

theory(Manosevitch2014).Consequently,howdeveloperstalkabouttheobjectivesofthe

platformmaydiffer fromconcepts familiar todeliberative theory. In this sense, the term

“online deliberation”, if it is used, may be used and understood differently by different

developers,producingverydifferentkindsofplatforms.Finally,manyplatformsandtoolsdo

not support all elements wemight typically associate with a process of deliberation, for

examplenotablyfewplatformsexplicitlysupportdecisionmaking.Insomecases,platforms

havebeenusedincollaborationwithofflineprocesses,ortosupportaparticulartasksuchas

ideageneration.

Sinceaprecisedefinitionforidentifyingexperimentalonlinedeliberationplatformsdoesnot

currentlyexistintheliterature,thisstudyadoptsabroadandflexibleunderstandingofthe

term “experimental online deliberation platform”. This allows the inclusion of platforms

influencedbyotherfieldsthatremainrelevanttothestudybyvirtueofthecontextoftheir

useanddesignfeatures.Thisalsoallowstheinclusionofinnovativeplatformsthatsupport

limitedelementsofthedeliberativeprocess.Asaminimaldefinition,toqualifyforinclusion

in this study a platform should allow more than one participant to express ideas and

argumentsinthecontextofpublicdebate.Thedesignandstructureoftheplatformmustalso

beabletodemonstratesomefeatureaimedatsupportingdeliberationthatdistinguishesit

fromthelightlystructuredformatoftypicalformsofonlinecommunication.

The second challenge concerns our methodological approach to studying the design of

platformsanddesign’spotentialinfluenceondeliberation.Currentresearchintodesignhas

beenlimitedandthemethodologicalapproachesadoptedencounterchallengesandareill

suited to the study of platforms utilising novel or experimental design choices. An initial

challengeisthecontesteddefinitionandconceptualcriteriaofdeliberation(Naurin2007).

Themovefromconceptualcriteriatoevaluativestandardspresentsfurtherchallengesinhow

deliberation,andthequalitiesassociatedwith it,canbeoperationalisedandmeaningfully

evaluated(seeNeblo2007).Inareviewofonlinedeliberationliterature,ColemanandMoss

(2012) raise the concern that studies focus on different deliberative criteria and

operationalise them differently, generating problems for judging the success of different

platforms and comparing performance across platforms. These concerns are particularly

relevant to the studyof design and experimental platforms. Studies haveoperationalised

deliberativecriteriausingmeasuresthatassumespecificdesignchoices.Forexample,Friess

andEilders (2015)discuss researchmeasuringhowoftenparticipants ina forumquoteor

refertoeachotherasameasureofreciprocity.Applyingthismeasuretoaplatformutilising

anargumentmapcannotproduceresultsforameaningfulcomparisonForexample,ifspecific

authors are not identifiable for citation it may measure no reciprocity, or, since every

argumentisconnectedtoanotherpointinthemap,itmaymeasureabsolutereciprocity.In

somecases, researchershave countedwordspermessage to judge,quite indirectly, how

oftenparticipantsjustifytheirbeliefs(JanssenandKies2005).Thisisadeeplyproblematic

measure of justification, that would produce very different results in platforms that

deliberately limit the word count of contributions or emphasise voting over writing. The

designchoicesofexperimentalonlinedeliberationplatformsvarysignificantly,thusexisting

approachestooperationalisingdeliberationthatassumespecificdesignchoiceswouldnotbe

helpfulforstudyingthisgroupofplatforms.

Studiesexploringdesign,suchasDaviesandChandler(2011)andBlack(2011),havediscussed

broad design variables such as differences between synchronous and asynchronous

communication,oranonymousandidentifiableparticipants.Thismaybecontrastedwiththe

morenuancedrangeofdesignchoicesdisplayedbyexperimentalplatforms.Thesevariables

includevariousapproachestointerfacedesign,aesthetics,andchoicesconcerningargument

visualisationandrepresentation.Thesevariablesmaysignificantlyimpacttheexperienceand

behaviour of participants, the quality of deliberation and the success of the platform. A

further element is the sense in which current approaches attempt to establish a causal

relationshipbetweenaspecificdesignchoiceandanoutcomeintermsofdeliberativequality.

Thisapproachrequiresfocusingononedesignchoice,controllingforconfoundingvariables

arisingfromotherdifferencesindesignaswellasexternalfactorssuchasthecontextofits

use. This is a challenging task: Karlsson’s (2010) study of deliberation in 28 identically

designedforumshighlightsthesignificanceofcontextualfactorsraisingconcernsregarding

thecapacityofcurrentapproachestoisolatetheinfluenceofdesign.It isalsoarguablyan

unhelpfulapproachwhenstudyingexperimentalonlinedeliberationplatforms,whichvary

greatlywith respect to theirdesignchoicesandthecontextof theirapplication.Although

Scheuer et al (2010) are discussing computer supported argumentation systems, their

comments on the challenges facing empirical study are pertinent to online deliberation

platforms,giventhesimilarconditionsofvarietybetweensystemsandverydifferentcontexts

ofapplication.Theywrite:

A simple explanation for the lack of studies that systematically compare different

argumentationsystemdesignsisthat it isquitedifficulttopracticallydosuchstudies;

varyingfactorsinacontrolledmannerwouldrequireeliminatingconfound[variables],

which isquitedifficultwhentwoexistingsoftwaresystemsarecomparedasawhole.

(Scheueretal2010:49)

Inviewofthemethodologicalchallengesdiscussedabovethispaperadoptsanexploratory

casestudyapproach.Anexploratoryapproachismostappropriategiventhelackofexisting

researchinthisareaandtheflexibledefinitionofthepopulationofcasesthestudyisadopting

(ShieldsandRangarajan2013,Reiter2013,Schutt2015).Thisallowsthestudytoexplorethe

widerandmorenuancedvarietyofdesignchoicesdisplayedbyexperimentalplatformsunder

thecircumstancesinwhichtheyarefound(Yin1994),allowingconsiderationoftheimpactof

thedesignchoicesofthesystemasawhole.Thestudywillfocusonasmallnumberofonline

deliberationplatformsandtoolsthatrepresentexemplarycases.Althoughthisfocuslimits

thecapacityofthestudytoexplorethefullrangeofpotentialinnovations,itallowsforin-

depth exploration of the experiences of the platforms and the potential of design. The

following section will review experimental online deliberation platforms and discuss the

criteriaforselectingexemplarycasestudies.

SelectingPlatforms

Technologydemonstrationshavedocumentedanumberofexamplesofexperimentalonline

deliberationplatforms(forexampleConklin(2008),Tauroetal(2008)Fishkin(2008),DeLiddo

andBuckingham-Schum(2010b)),andfurtherexampleshavebeenidentifiedandcollected

onthesitesParticipateDBandParticipedia(TowneandHerbsleb2012).ParticipateDBlists

around350toolsandservicesforwebbasedparticipation(ParticipateDB2017).Scheueret

al (2010)provideareviewofcomputersupportedargumentationsystems,someofwhich

relatetothecriteriaforonlinedeliberationplatformsoutlinedabove.MarkKlein(2017)has

producedanoverviewofcollectiveintelligencetools,creatingandbrieflyanalysingalistof

around90platforms, aswell as apaper seeking to categorise available large scaleonline

deliberationplatforms(Klein2015).

ParticipateDBprovidesataxonomyof24classesofplatform,thetaxonomydefinesplatforms

intermsoftheorganisationofinformationaswellastheirpurpose:

Argumentmapping

Audienceresponsesystem

Budgetsimulator

Budgetvisualization

Collaborativedocumentwriting

Contentmanagementsystem(CMS)

Crowdfunding

Discussionforum

E-learningplatform

E-voting

Electronicmailinglist

Groupdecisionsupportsystem(GDSS)

Ideation&brainstorming

Livevideostreaming

Onlineconsultationsuite

Onlinedialogueanddeliberation

Onlinesurvey

Socialmedia

Virtualtownhall

Virtualwhiteboard

Virtualworld

Webconferencing

Weblog

Wiki

(ParticipateDB2017)

Inreviewingpotentialplatforms,thisstudyhasusedParticipateDB,MarkKlein’sanalysisof

collectiveintelligencetools,aswellasasearchforonlinedeliberationplatforms.Thestudy

identifiedminimal criteria to qualify for inclusion in the study, thesewere applied to the

platformsidentifiedfromthesearch.ParticipateDBforexampleincludesmanyplatformsthat

utilise lightly structured forumsor support activitiesdistinct fromdeliberation, suchase-

votingande-learning,andwerethereforenotconsideredrelevanttothecurrentstudy.The

reviewofpotentialplatformsrevealedafurtherneedtoconsiderpracticalissuesinselecting

platforms.Klein(2017)observesinhisdescriptionsthatmanyoftheplatformsandtoolslisted

arenolongeractive(forexampleAhoona),maynothavebeenused,orhavenoevidenceof

largescaleuse(Cluxton).Insuchcases,thereislimitedmaterialavailableforin-depthstudy

andopportunityforassessingthepotentialofdesigninaddressingthechallengesofonline

deliberation.Inviewoftheseissues,theselectionofcasesconsiderspracticalissuessuchas

the availability of material and indications of substantial use of the platform. These

considerationsincludeeaseofaccess,evidencethattheplatformhassupportedlargescale

useorsustainedparticipationoveraperiodoftime,theavailabilityofmaterialandliterature

ontheplatformthatprovideevidenceoftheplatformsapplicationandinfluence.

In addition to basic criteria and practical considerations, the reviewbelow also considers

trendsindesignapproachesandnotableapplicationsoftheoreticalapproachesinfluentialin

the field (such as informal logic and collective intelligence), for example, the use of

argumentationschemestoorganiseandsupportrealworlddebate(e.gParmenides),andthe

use of gamification to support participant engagement (e.g @stake). The selection of

exemplarycasesaimstocapturearangeofapproachesandparticularlysuccessfulexamples

ofthesetrends.

ReviewingandCategorisingPlatforms

Thereislittleexistingresearchtoguideourreviewandcategorisationofexperimentalonline

deliberationplatforms.Klein’s(2015)reviewof“CrowdScaleOnlineDeliberation”isanotable

exampleofanoverviewofthisarea.Klein(2015)identifiesfivedifferentcategoriesofonline

deliberationsystems.Thesearetime-centric,question-centric,topic-centric,debate-centric

andargumentcentric.Thissystemofcategorisationisbasedonhowinformationisstructured

and visually represented. Scheuer et al (2010) provide a review of computer supported

argumentation systems. While this review is not specific to deliberation systems, the

categoriesandpatternsidentifysimilarfeatures;forexample,time-centriccorrespondswith

chatandforums,debate-centriccorrespondswithcontainers,argumentcentriccorresponds

withgraphsinScheueretal’sanalysis.Timecentricsystems,includingblogs,chatroomsand

webforums,maybeunderstoodasrepresentingthemoretypical,lightlystructuredforums

discussed in other literature (Towne and Herbsleb 2012). In question-centric platforms

(examplesprovidedinclude,stockoverflow.com,IdeaScaleandMindjet),acentralquestion

organisesinformation,butthereisnofurtherorganisingorcuratingprinciplethatensures

informationisnotrepeated,noristhereaprocesstohelpidentifypreferableideasoridentify

critiques and critical discussion of ideas. Topic-Centric systems, such as wiki’s, collect

informationarounda topic, andaredescribedas capturing consensuswhile controversial

aspectsofdebatearemovedtotalkpagesthatareorganisedaccordingtothetime-centric

principle described earlier. Debate-centric systems (examples include, whysaurus.com,

Debatepedia.com,debatewise.org(Klein2015))presentinformationintheformofprosand

cons and are often curated to ensure an effective overview of a topic with minimum

redundancyofinformation.Finally,Klein(2015)outlinesargument-centricsystems.Theseare

systemsthattypicallypresentinformationintheformofargumentmapsortrees,identifying

centralquestionsandideas,followedbyargumentsforandagainstandreasonsandevidence

supportingthosearguments.

Klein’s (2015) discussion can be understood as an argument for a particular approach to

online deliberation systems, specifically argument-centric systems ofwhich he is a noted

developer.Thediscussionprovidesausefulapproach tonavigating the rangeofpotential

online deliberation systems: time-centric and question-centric platforms describe lightly

structured approaches that are not relevant to this study. Debate-centric and argument

centricconstitutetwocommonapproachestoorganisinginformationamongstplatformsthat

areclearlyrelevant. Inadditiontotheuseofargumentmapsandtheproandcon listsof

debate-centricvisualisations,thereviewidentifiedtwofurthergeneraltrendsinapproaches

to design and visualisation techniques: annotation and group clustering. Some of the

platforms and tools reviewed use annotation of existing web pages as an approach to

deliberation (for example, Hypothesis and Rbutr). Annotation involves providing an extra

layer of meaning to a given web page by allowing users to annotate it and see other

annotationsincludinglinkstootherpages.Otherplatformsandtoolsrepresentinformation

intermsofgroupclustering,inwhichparticipantsarerepresentedinspacesandgroupedin

suchawayas to reflect their support forparticularpositions (forexampleConsiderItand

Pol.is).Groupclusteringisanotherformofvisualrepresentationwhichallowsuserstoview

the positions of other individuals in a community, for example reflecting how strongly

different users agreed or disagreed with a statement. Group clustering can give a

representation of participants relative to positions without attempting to place those

positions in the context of an argument structure relative to other positions. Such an

approach,canbeunderstoodasavisualisationtechniquethatprovidessocial information

concerningthedebateincontrasttoinformationrelatingtothecontentoftheargument.

Within these general trends, the platforms and tools reviewed display a range of more

nuancedchoicesinrelationtointerfacedesign,aesthetics,andthecombinationofothertools

andtechniquestosupportdifferentaspectsofdeliberation.Forexample,whilewecantalk

generally of the approach of argumentmapping, the platforms and tools will differ in a

numberofrespects:theontologyoftheargumentmapmayvary,withdifferentchoiceson

theelementsthatcanbeusedtocomposeamap;choicesoverwordlimitsorthevisibilityof

repliesandcounterarguments;andthegeneral lookandaestheticsoftheargumentmap.

These factorsmay greatly influence the success of theplatformand its ability to address

certain challenges; for example, aesthetic considerationsmay impact the capacity of the

platformtoattractandmaintainparticipantengagement.Ourreviewofonlinedeliberation

platforms also found that platforms may combine different approaches to argument

visualisation, and they may utilise other tools or design approaches that help address

challengesorissuesassociatedwithagivenapproachtovisualisation.Forexample,ConsiderIt

combines both debate centric visualisations in the form of pro and con tables, but also

displaysgroupclusteringandallowsforfurtheranalysisofthisdata(forexamplethelevelsof

supportforpositionsbasedonthedemographicsofthegroup).TheDeliberatoriumfeatures

bothargumentcentricandtimecentricsystemsofcommunicationintheformofanargument

mapandachatfunction.Finally,someapproachesfoundinpracticedonotfiteasilyintoany

category of visualisation or design approach already identified. For example, Parmenides

drawsonargumentationschemesbutorganisesinformationthroughthepresentationofa

dialoguewiththeparticipant.

Thestudyseekstoselectplatformsthatrepresentnotableorsuccessfulexamplesofsolutions

to the challenge of online deliberation; platforms that combine these approaches and

techniques in interestingways,aswellasnovelanduniqueapproaches.Thepurposeisto

captureasenseofgeneralandnotabletrendsindesignapproachesandexplorehowthey

may address challenges of online deliberation. The study does not seek to make strong

generalisationsfromtheexperiencesofselectedplatforms,whichgiventhediversityofthe

populationofcasesandmethodologicalapproachwouldbeproblematic. Insummary, the

selectioncriteriaareoutlinedbelow.

• Theplatformortoolmeetstheminimalcriteriaofsupportingoneormoreaspectof

online deliberation and demonstrating novel and relevant design features that

distinguishitfromlightlystructuredforums.

• Practicalconsiderations:easeofaccesstotheplatform,evidencethattheplatformis

successfulandhasreceivedlargescaleorsustainedparticipation,theavailabilityof

materialandliteratureonapplicationsoftheplatform.

• Instanceswherethedesignchoicesrepresentnotableapplicationsofcommontrends

oruniqueapproachestovisualisationandstructurethatcanbe linkedtoengaging

withthechallengesofonlinedeliberation.

Wherepossibletheplatformsandtoolsavailableweretrialledandreviewedaccordingtothe

criteria above. From this process, a selection of platforms and tools were identified as

exemplarycasestudiesforfurtherexploration.

ExemplaryCasesofOnlineDeliberationPlatforms

Twelveplatformsandtoolswereselectedasexemplarycasesforfurtherstudy,theseare:

@stake

ArvinaandOva

BCisive/Rationale

ClimateCoLab

Cohere/EvidenceHub

ConsiderIt

Debategraph

Deliberatorium

Parmenides

Pol.is

Rbutr

Truthmapping

Thissectionprovidesadescriptionofeachoftheseplatforms;providinginformationonthe

backgroundof theplatform,design features, illustrative screenshotsof theplatform, and

reviewsexistingstudiesandapplicationsoftheplatform.

@Stake

Background,ObjectivesandDesignofthePlatform

@Stakeisa“roleplayingonlinecardgamedevelopedtofosterempathyandcollaboration”

(eLab2017a:1)and“enhancedeliberationinreal-worldprocesses”(eLab2017b:3).@Stake

wasdevelopedby the Engagement Lab, anorganisation that seeks toproduce games for

socialchange,participationandengagement(eLab2017a:1).Thedevelopersdescribethe

game as relying on “rapid fire ideation, discussions facilitated through role playing,

experimentation with ideas, and collaboration among a diversity of stakeholders” (eLab

2017b:3).

@Stake isacardgamethat takesaboutonehour toplay. It requires thedivisionof large

groupsintosmallergroupsof4-5players.Participantsareassignedcharactersthroughcards.

Thecardcontainsbiographicalinformationandanagenda,visibleonlytotheparticipant.The

agendaprovidesdetailsofthecharacter’sobjectives,withpointsattributedtoeachobjective.

One person of the group is elected as Decider for the round, with which comes various

responsibilities.Tokensformacurrencyofthegame,eachplayerisgiventhreetokens,with

anadditionalthreetokensinapotandfivetokensfortheDecider.

Distributionoftokens(eLab2017b:4)

Charactercard(eLab2017b:11)

Theroundsconsistofthefollowingstages:

Introduction:Participantsintroducethemselvesincharacter

Brainstorm: TheDeciderannounces the issue,participantshaveoneminute todevelopa

proposal

Pitch:MovingclockwisefromtheDecidereachplayerhasaminutetopitchtheirproposal.

Theymayusetokenstoallowthemselvesextratime.

Deliberate:TheDeciderleadsafollowupdiscussion,participantsmayaskeachotherabout

theirproposals,offercounterargumentsandsuggestamendmentstooneanother’splans.

Tokensmaybeusedtoextendtime.

TheDecision:TheDeciderannouncesawinningproposal.Theplayerwhoproposedthiswins

allthetokensinthepot,plusbonustokensbasedonpointsontheiragenda.Allotherplayers

score points for their agenda items, if thewinning proposal satisfied these requirements

(determinedbytheDecider).ThewinnerthenbecomestheDeciderforthenextround(or

alternativelypassesthisontosomeoneelse)(eLab2017b:5-6)

@stakewasinitiallydevelopedasafacetofacecardgamein2014,thedevelopershavesince

producedadigitalmobileversion(EngagementLab2017).Thisversionsimplifiestherules,

reducestheneedforfacilitationandallowsforeasiertrackingofideasgeneratedduringthe

gameandtheexperiencesof theplayers (Gordonetal2016).Thegamewouldappear to

requirebespokecardsforeacheventitisappliedto,withcharactersandagendasrelevantto

theevent.Incontrasttomanyoftheotherplatformsandtoolsexploredinthisthesis,the

developers make explicit reference to deliberative democratic theory and the desire to

supportthegoalsandoutcomesofdeliberativedemocracy(Gordonetal2016).@Stakeand

theotherprojects developedby Engagement Lab canbeunderstoodas themost explicit

example of “gamification”, as discussed in the collective intelligence literature, available

amongst the platforms and tools reviewed. Although it aims to support deliberation and

decision making in general, the developers also stress the role it can play in supporting

empathybuildingbetweenparticipants(Gordonetal2016).

ImagesandScreenshotsofthePlatform

Anexampleofthecardsisreproducedbelow1.

LiteratureandApplicationsofthePlatform

Inadditiontoinformationontheengagementlabwebsite,thedevelopersproducedapaper

forthe2016CSCW(ComputerSupportedCooperativeWork)conferenceabouttheplatform.

Thispaperidentifiesalackofresearchinto“civicgames”,itoutlinestherulesandmechanics

1https://medium.com/engagement-lab-emerson-college/announcing-a-printable-stake-game-211f76804086#.uy36q159k

of the game and describes a pilot study comparing the game to a traditional ice breaker

(Gordonetal2016).Thepaperreportsthefindingsofthepilotstudyaspositive,findingthat

roleplayingmayencourageparticipantstobemorecomfortablewithpublicspeakingand

engagement, aswell as greater capacity toempathisewithothers and retain information

abouttheirideasandpersonalities(Gordonetal2016).Michelson(2015)alsoreportsonthe

resultsofplaytestsofthegames;theplatformwasusedasawarmupactivitybyarchitecture

andplanningdesignfirmUtileintheImagineBoston2030initiative.@Stakewasalsopiloted

inthreeParticipatoryBudgetingmeetingsinNewYorkCityinAutumn2014(eLab2017a).A

studyof thiseventusedparticipantobservation, surveydata,and followup interviews to

evaluateanumberofissuesincludingwhethergameplayincreasedempathy,efficacy,affinity

towardscivicengagement,andfutureparticipation.Oneoftheobservationsofthisstudywas

thesenseinwhichsomeparticipantsobjectedtospendingtimeonagame(eLab2017a).The

gamehasbeenusedinavarietyofsettingsincludingorganisationalplanning,UNDPpolicy

meetings on youth unemployment inMoldova, Egypt and Bhutan, educators’ curriculum

designworkshops,andseveralacademicconferences(Gordonetal2016:271,eLab2017a).

ArvinaandOva


Arvinaiswebbaseddiscussionsoftware,that“allowparticipantstodebatearangeoftopics

inreal-timeinawaythat isstructuredbutatthesametimeunobtrusive”(Lawrenceetal

2012:1).ArvinaandOVAweredevelopedbytheARG-tech,CentreforArgumentTechnology

attheUniversityofDundee.ARG-techdevelopstoolsaimedatargumentmining,argument

visualisationandanalysisandtheuseofartificialintelligenceindialogue.Thecentrehasbeen

influential in this field and has worked in collaboration with the BBC and IBM debating

technologies(Reed2017).ArvinaandOVAaretoolsdevelopedtosupportargumentation.

ArvinaandOVAareanapplicationofArgument InterchangeFormat (AIF) theory thathas

beenusedtosupportpoliticaldebate.OVA(OnlineVisualisationofArgument)isatoolfor

analysingandmappingargumentsonline.The interfaceallowsuserstohighlighttextona

webpageandextract this to apremisewhich canbeused to supportor challengeother

premises.Missingpremises(orenthymemes)canalsobeaddedbyusers.Arvinaisadialogue

tool thatusesgooglewave,anonline toolwhichallows for real timecommunicationand

collaboration(Google2010 inSnaithetal2010:7).ArvinaisaWaveapplicationwhichbuilds

on the Google API, allowing a user to choose a topic from any previously analysed AIF

resources. The AIF resource is examined to determine the participants involved in the

dialogueandanewrobotisaddedtothewaverepresentingeachoftheseparticipants.The

participantsmaybehumanorartificial,withartificialparticipantsusingknowledgeassigned

fromtheAIFresource.

ArvianandOVAarerelevanttothecurrentstudyforanumberofreasons.ArvinaandOVA

representanexampleoftheuseofartificialintelligenceinonlinedeliberationplatforms,and

anattempttoprovidesomeformalstructuretonaturallanguageargumentthatcanbeused

acrossdifferentplatforms.ArtificialintelligenceandparticularlytheworkofChrisReedand

ARG-techweredescribedbyanumberofotherdevelopersinthisareaasbeingparticularly

influential and promising as a direction for online deliberation. This is an area that is

particularlyrelevanttoissuesoffeasibilityandscaleaswellasinformationmanagement.


ArvinaInterface(Lawrenceetal2012:2)

PremisepropertiesinOVAShowingParticipants(Snaithetal2010:5)


ArvinaandOVAwereappliedinthecontextofaproposaltobuildatransmissionlineforthe

BeaulytoDennypowerlinethroughareasofoutstandingnaturalbeauty(Snaithetal2010).

Thepurposeofthestudywastotestthecapacityforinterchangebetweendifferentformats,

argumentvisualisationanddialogueformatusingAFI theory.Thestudyobservedthatthe

toolsdemonstratethatformallydescribableprocessesofdeliberationcanbelinkedtothe

formallydescribablestructuresofknowledge.Itclaimedthatmuchremainedtobedoneto

expand and refine the tools, and integrate the toolswith other parts of the deliberative

process,suchasinquiryanddecisionmaking.Lawrenceetal(2012)testedArvina’scapacity

to support human and artificial intelligence groups in mixed initiative argumentation.

Lawrence et al (2015) discuss the challenges of current argument mining techniques to

identify complex structural relationships between concepts, a lack of consistency in

formatting,andalackoflargequantitiesofappropriatelyannotatedargumentstoserveto

trainandtesttools.Ineffortstoaddressthis,researchershaveturnedtootheronlinetools

(including several discussed in this study,Debategraph, Truthmapping andRationale) and

soughttoconvertthemtoAIFformats(Lawrence2015).

BCisive/Rationale


BCisiveallowsthecreationofdecisionmapsto“capturediscussion,organiseideas,explore

options, testhypothesisandanalyse reasoning” (BCisiveOnline2017).Rationaleclaims to

allowusers tomake argumentmaps to “structure arguments, analyse reasoning, identify

assumptionsandevaluateevidence”(RationaleOnline2017).BCisiveandRationalearetools

developedbyAusthinkandrepresentcommercialsuccessorstoReason!Able(Scheueretal

2010).ThetoolsarecurrentlyrunbytheReasoningLab;bothtoolsallowforthecollaborative

constructionofargumentmaps.Thesoftwarecodeforthetoolsareidentical,howeverthe

interfaceallowsfordifferentoptionsinmapconstruction(Kunschetal2014).BCsisiveaims

atsupportingdecisionmakingwhileRationaleisusedasaneducationaltoolforsupporting

criticalthinkinganddevelopingargumentinessays.

Thedescriptionsfortheargumentvisualisationontologyaredifferentforeachtool.Thebasic

ontology of BCisive allows for the construction of maps from the following: Situations,

Options,Pros,Cons,Reason,Objection,Evidence,CounterEvidence,Questions,Challenges

andaFix.InthecaseofRationale,mapsarebasedonthefollowingontology:contention,

reason,objection,note,exampleandco-premise.Therearefurtheroptionstoidentifythe

natureofdifferent typesofevidenceorbasis including:Commonbelief,data, case study,

assertion,law,quote,statistic,personalexperience,andpublication.

BcisiveandRationalearecharacteristicof theargumentmapping toolsavailable inonline

deliberation. They allow relatively extensive options in constructing argument maps,

including task specific items. Participants are free to construct maps and use categories

however theychoose,however thereareprompts.Forexample, the“rabbit rule”prompt

alertsparticipantsifsomethingismentionedintheconclusionthatisnotmentionedinthe

reasosn(Twadry2004).ThedevelopersinvolvedinRationalearecurrentlydevelopingtheuse

of probabilistic judgements in argument maps, which would affect the way the system

organisesthemap.Thisisparticularlyrelevanttothechallengeofinformationmanagement

(howwell platforms canmanage different data types). In addition to these features the

developersarealsodevelopingwaysofallowingrealtimecollaborationonargumentmapsin

whichauthorscanseethechangestheirpartnersintendtomakeinrelationtoanargument

map.InthissenseBcisiveandRationalearegoodexamplesofargumentmappingapproaches

withmorenovelfeaturestosupportdecisionmakingandcriticalthinking.


BCisive

ArgumentmapinterfaceforBCisive2

2https://www.bcisiveonline.com/editor/

Rationale

Rationale argument map editor interface3

Rationaleargumentmapexample4

3https://www.rationaleonline.com/editor/4https://www.rationaleonline.com/editor/#?id=8ek8jh

LiteratureandApplicationsofthePlatforms

A literature review reveals seven papers on Rationale, Bcisive and an earlier version of

rationalecalledReason!Able.Scheueretal(2010)includeRationaleandReason!Ableintheir

reviewofCSAsystems.Kunschetal(2014)comparestheuseofBcisiveandRationaleinthe

contextofeducationforbusinessstudents.TheysuggestthatRationalewaspreferablefor

demonstratingthebasicsofargumentmapping,whileBCisivewasbettertoanalysebusiness

cases and present findings due to its capacity to conduct more complex analysis and

compatibilitywithformatssuchasPowerPoint.Lengbyer(2014)usesrationaletoexplorethe

useofargumentmappingtosupportdecisionmakinginspecificcasesratherthanforgeneral

educational purposes. There is discussion of the use of Rationale to support lawyers in

Australia (Drummond2006 inVanGelder2007)andhelping judgeswithexpert testimony

(vanDrielandPrakken2010),thelaterconcludingthatitwasmorelikelytobehelpfulfor

educationalandtrainingpurposes.Rationalehasbeentestedinothereducationalsettings

finding positive results when testing students’ critical thinking skills (Tawdry 2004) and

understandingofthematerial(Davies2009b).TheliteraturehastendedtofocusonRationale

oritsearlierversionsinaneducationalcontext.BCisiveappearstobealaterdevelopment

intendedforuseintheprivatesectorasatoolforsupportingdecisionmakinginorganisations

(ReasoningLab2017).TherehasbeenmoretakeupfortheRationaletool,whichhasbeen

usedinvariouseducationsettingsbystudentsandaspartofcourses,notablyinAmsterdam

andAustralia(ReasoningLab2017).Therearefreeonlineversionsavailableforbothtools.

ClimateCoLab


ClimateCoLabisdevelopedbytheMITCenterforCollectiveIntelligence.Itdescribestheaims

oftheprojectasattemptingtoaddresswickedproblems,specificallyclimatechange,through

collective intelligence.Theprojectappeals to the idea that it iscreatinganopenproblem

solvingplatformdrawingonthesuccessofprojectssuchasWikipedia(Maloneetal2009).

Climate CoLab involves contestswhere participants can put forward and discuss ideas to

addresstheproblemofclimatechange.Acontestconsistsofdifferentstagesandelements.

People can propose solutions to specific problems identified on the site, such as land

managementandenergysupply.Thereisafurtherstagewherebyproposalsareintegrated

towardthedevelopmentofacomprehensiveplanthatcouldbefeasiblyadoptedasnational

policy.A final stage considerswhether the comprehensiveplanmeetsestablished targets

(Maloneetal2009).Theproposalsareassessedbyexpertjudges,thoughthereareplansto

replacethisprocesswithaformofcrowd-basedassessment.ForthefirstthreeyearsClimate

CoLabinvolvedoneortwocontestsperyear,from2013theyintroducedcontestfamiliesof

17ormoreconteststhatseektobreakdowntheissuesofclimatechangewhicharethen

integrated(Maloneetal2017).

ClimateCoLabinvolvesthreedesignelementssupportingcollectivedecisionmaking:model

basedplanning,onlinedebatesandvoting.Modelbasedplanningallowsparticipantstouse

simulation models to provide information about the impact of different proposals. The

literaturedescribeshowthesystemusestheC-LEARNmodel,anonlineversionofC-ROADS,

aclimatechangepolicysimulator(Maloneetal2009).C-LEARNtakesasinputasetofregional

commitments to emission reductions and produces as outputs projections of carbon

concentration,temperaturechangeandsealevelrise.Theseoutputsarethenusedtodrive

elevenadditionalmodelsusedbyClimateCoLab,whichpredictanticipatedeconomiccosts,

qualitativeimpacttohumanandphysicalsystems(suchasagriculture,waterandhealth).

Online debates utilise a system similar to Deliberatorium and Compendium (see later),

providinggreaterstructurethantraditionalforumsandclassifyingeachcontributionas(1)a

question,(2)aposition(proposedsolutiontothequestion),(3)anargumentfor,or(4)an

argumentagainst.Somedebatescaptureargumentsandinformationthatcutacrossissues

thatunderlieanumberofdifferentplans,thusplancreatorsareencouragedtospecifywhat

positionstheirplanstakeoncrosscuttingissues(Iandolietal2008).Finallyparticipantscan

vote on debate positions and plans that they prefer, allowing users to identify promising

proposalsforthecontests.Cashprizesareawardedtotheproposalsthatarejudgedtobe

thebestoverallinthecontest.

ClimateCoLabisrelevanttothecurrentstudyforthefollowingreasons:ClimateCoLabisan

ambitiousandwellestablishedplatformthathasalargenumberofparticipants.Theplatform

providesadecisionmakingmechanisminadditiontoideagenerationanddebate.Through

theclimatechangepolicysimulatorandtheuseofjudgesandmoderators,theplatformalso

seeks to verify information provided in debate and ground claims made into a shared

understanding of the facts. These features are of particular relevance to the theme

information management, and issues concerning how platforms deal with contested

knowledge,moderationanddifferentdatatypes.Theprojectisalsowellresourcedasit is

able to award $10 000 to the best overall proposal per contest. This is significant when

thinkingaboutfeasibilityandscaleandthesustainabilityofonlinedeliberationplatforms.The

platform uses a number of different techniques to address different challenges in online

deliberation and represents an interesting approach drawing from the field of collective

intelligence.


Introneetal(2011:5)

SimulationModel

ClimateCoLabDebateInterface(Introneetal2011:6)


TheClimateCoLabhasbeenconductingcontestssince2009andtheseapplicationshavebeen

documentedindifferentresearchpapers(seeforexample,Maloneetal2009,Introneetal

2011).Theprojecthasover85000participantsincluding200expertsonclimatechangeand

relatedfields(Maloneetal2017).Duhaimeetal(2015)usedonlinesurveysandananalysis

ofwebactivitytodevelopapictureofthecharacteristicsandbehaviouroftheClimateCoLab

security. It foundthatthecommunitywasgeographicallydiverse,andtendedtobehighly

educatedandexperiencedwithclimatechangeissues.Italsofoundthatthoseoutsidethe

usual conversations about climate change are influenced by and contribute effectively to

collectiveproblemsolving.Memberswhodidnothavegraduateeducationpreviousclimate

changeexperienceordidnotliveintheUnitedStatesreportedsignificantlyhigherlevelsof

learning,beliefchangeandincreaseinclimaterelatedactivityasaresultofparticipation,and

thesemembers,andwomen,wereatleastaslikelytosubmithighqualityproposals(Duhaime

etal2015).Throughthelaterimplementationofcontestwebs,Maloneetal(2017)explored

whether participants would reuse their own and other’s work effectively, and whether

participantswould be able to exploremultiple combinations of interchangeable parts (of

solutions).Theresultsoftheirexperimentwerefoundtobepositive,observingtheirsystem

facilitated widespread knowledge sharing and reuse, and the combining of solutions at

multiplepointsofaggregation(Maloneetal2017).

Cohere/EvidenceHub


CohereisaprojectdevelopedbytheKnowledgeMediaInstituteattheOpenUniversity.Itis

a visual tool to create, connectand share ideas.DeLiddoandBuckinghamShum (2010b)

identifiescontestedcollective intelligenceasadistinctareaofcollective intelligence,with

Coheredevelopedasaprototypefortestingtheirdesignrationalefortheseideas.Evidence

Hubispartofthisprojectandaimstoprovideaplatformforcollaborativeknowledgebuilding

basedon theconceptof contestedcollective intelligence (DeLiddoandBukinghamShum

2013),allowinguserstopoolandmapknowledgearoundaspecificissueortheme.

The evidence hub organises information according to the following categories: key

challenges, potential solutions, research claims, evidence and counter evidence, and

resourcesontheweb.Inadditiontothisattempttopoolrelevantknowledgearounddifferent

individuals and researchers working on projects, this is mapped geographically and also

accordingtothemesandquestions.

EvidenceHubisanambitiousprojectthatutilisesanumberofdifferentapproaches,including

annotation,argumentvisualisation,chatfunctionsfordebatestosupportcollaborationand

deliberationaswellasprovidingaresourcetocollectevidencearoundagivendebate.Itsuse

ofvariousapproaches toargumentvisualisationand informationmanagementmake itan

interesting case for further exploration. The developers draw explicitly on collective

intelligenceliteratureaswellasinformallogictheoristssuchasWalton(EvidenceHub2017)

andthetoolscanbeunderstoodasamatureand influentialexampleofanapplicationof

thesetheoriesinpractice.


MappingArgumentationChainsintheEvidenceHub5

EvidenceMap6

LiteratureandApplicationofthePlatform

The E-Hub website describes examples of hubs running in collaboration with external

partners, includingtheCommunityofPractice for the InstituteofHealthVisiting (aclosed

Hub) and the Systems Learning& LeadershipHub (University of Bristol). OpenUniversity

CommunityHubs includetheOpenUniversity inScotland’sWork&LearningHub,andthe

OpenUniversity’sFacultyofEducationandLanguageStudiesdepartmentthroughtheirHubs

forReadingforPleasure,andResearchbyChildrenandYoungPeople(EvidenceHub2017).

De Liddo and Buckingham Shum (2013) describe the concept of Evidence Hub and its

developmentinresponsetoexperiencesfollowingitsuseinthecontextofhealthcareand

education.Thisworkhighlightedapervasivechallengeofatrade-offbetweentheneedfor

structuretomaximisethesignal-to-noise-ratioandpermittingpeopletomakecontributions

5http://rcyp.evidence-hub.net/?max=20&orderby=date&sort=DESC&filternodetypes=Challenge#home-list6http://rcyp.evidence-hub.net/explore.php?id=861572362500105612001342686197

withvery little indexingorstructurethatrequires less learning.Theysuggestthatthis isa

problemthatisfarfromsolved.

ConsiderIt


ConsiderIt “helps individuals make sense of complex issues through familiar deliberative

activities” (ConsiderIt 2017). It is described as a novel platform for supporting public

deliberation on difficult decisions (Kriplean et al 2012, Kriplean et al 2011). The platform

allows users to create forums and introduce questions or proposals for a community to

address.Thecommunitycanthencontributetothisforumbyidentifyingtheirpositionona

scaleofagree/disagreeorhighpriority/lowpriority,andselectingthemost importantpro

andconpointsforagivenposition.ConsiderItthenpresentsavisualrepresentationofthe

communityasawhole,itshowsopinionsalongascaleofagreetodisagree,alongwithalist

ofrankedprosandcons.

ConsiderItcombinesanumberofdifferentapproachestovisualisingargumentanddebate.It

representsprosand cons toagivenproposal,but it alsomapsgroups to show the social

contextofanindividual’sposition.Userscaninteractwiththisvisualisationtoidentifygroups

withsharedopinionsandpointsofconsensusamongstotherwisedisparateparties. Inthis

sense,thesystemallowspeopletovoicetheiropinionswhilealsogivingthemtheopportunity

to recognise areas of agreement with political opponents. ConsiderIt also allows for

potentiallymorenuancedunderstandingofdifferencesbyallowingparticipantstoarticulate

differentprosandconsandalsoplace levelsofpriorityonargumentsandproposals.The

developersarguethissupportsempathy,mutualunderstandingandareasofconsensus.For

example, if80%ofpeoplewhoopposean ideashare thesametwoconcerns thatcanbe

resolved,thissuggestsanopportunityforaddressingtheconflict (Freelonetal2012).The

developersoftheplatformhavealsoattemptedtoaddresstheproblemofverificationoffacts

inonlinedeliberation,albeitexternallyinapplicationsoftheplatform.

The featuresdescribedabovedistinguishConsiderIt from theother available examplesof

platformvisualisationsusingprosandconstables.ConsiderIthasalsobeendescribedasusing

gamificationinitsapproachtotutorialsandaesthetics.Giventhesophisticationofitsdesign

choices and thewell documented applicationsof its use, itmakes an interesting case for

furtherexploration.


ThescreenshotsbelowillustratethedesignofConsiderIt,andtheprocessofgeneratingand

contributingtoaforumdiscussion.

Displayingausercreatingaforumandpoliciesforacommunitytodiscuss7

7https://consider.it

Displaystherepresentationofacommunityinrelationtotheprioritythatisplacedongiven

topicsofdiscussionorpolicies8

Displayingthevisualisationofaspecifictopicorpolicy,allowinguserstoseetopprosand

consandthegeneralconsensusofthegroup9

Displayingauserintroducinganargument10

8https://consider.it9https://consider.it10https://consider.it


TheliteratureonConsiderItshowsithasbeenappliedandtestedinanumberofsituations.

Itbeganaspartofaprojectcalledthe“LivingVotersGuide”,thatincludedanexperimentin

aU.Sstateelectionallowingresidentstodebatenineballotmeasures(Kripleanetal2012).

AlthoughtheLivingVotersGuideisnolongeractivetheplatformcontinuestobeappliedin

othercitizenengagementprojectsanditisalsoavailableforfreeforpublicuseandthrough

a paid planwith additional features (ConsiderIt 2017). Research has tested howwell the

platform encourages engagement with different views (Freelon et al 2012), participants’

perceptionsofdifferentstandpointsandtheirownknowledgeofthesubject(Stieglerandde

Jong2015).Thesestudieshavegenerallyfoundpositiveresultsfortheplatform,additionally

differentvariationsoftheplatformhavebeentestedinthecontextofadebateonGreece

andtheEuropeanUnion(StiegleranddeJong2015).Thedevelopersoftheplatformhave

alsoengagedwiththeproblemofverificationandthetrustworthinessofsourcesandclaims

appealedtoinonlinedeliberation(Freelonetal2012andKripleanetal(2014)).Kripleanet

al (2014)trialledtheuseof librariansasfactcheckers inoneexampleoftheLivingVoters

Guide.Thetrialusedaquantitativeanalysisoftheuseofthefactcheckingservices,finding

14.2%ofclaimsweresubjecttofactcheckingrequests,andhalfoftheseconcernedclaimsof

factwhile theothers involved claimsof principleor other claims thatwerenot verifiable

(Kripleanetal2014).Thetrialalsoevaluatedtheexperienceofparticipantsandlibrarians.It

foundthattwothirdsofthosewhohadhadtheirsubmissionsfactcheckedfelttheprocess

hadbeenfair(noneclaimedthatithadbeenunfair),whilemanyusersexpresseddesirefor

bettercommunicationwiththefactcheckers(forexampletheabilitytorespondtotheresults

ofthefactcheck).Thelibrariansreportedpositiveexperiencesoftheprocess,theyfeltthey

wereabletoconductfactcheckinginaneutralmanner,althoughsomehighlightedthefact

thattheyfelttheylackedthelegalexpertisetocorrectlyrespondtosomeissues.Whilethe

studyfoundtheprocesswasbroadlysuccessful,theyacknowledgedproblemswithapplying

thisapproachtolargescaledeliberation(Kripleanetal2014).TravisKriplean(thedeveloper

oftheplatform)alsodescribeshowtheplatformhasbeenusedwiththebitcoincommunity

and other open source communities, in large organisations to support strategic planning

effortsand inschoolsasaneducationaltooltosupportcritical thinking.Healsodescribes

future plans to use the tool in join initiatives project in Hawaii, involving the Hawaiian

languageinanumberofschools.

Debategraph


Debategraphisdescribedasaservicewhichallowsindividualsand“communitiesofanysize

toexternalize, visualize,questionandevaluateallof theconsiderations thatanymember

thinksmayberelevanttothetopicathand”(Debategraph2017).Thevisualisationspresent

colourcodedmapsbasedonthefollowingcriteria:issues(orquestions),positions,arguments

fororagainst.Themapsareopentoeditingbythegeneralpublic,andtheideassubmittedin

themapcanberatedbyothers.Thestrongestargumentsareindicatedbythewidthofthe

arrowsconnectingtheideasinthemap.Theontologyoftheargumentmapisnotexplicitly

groundedinaparticulartheoryofargumentation;thebasicbuildingblocksofthemapshare

similaritieswithotherargumentmaps,howeveritdevelopsamuchmorecomplexrangeof

connections between ideas and relationships betweenmaps. In addition to an extensive

rangeofconnections,themapsalsoflowintooneanother,allowingparticipantstonavigate

fromoneissuetoanother.TheontologyoftheDebategraphisoutlinedinthevisualisations

below:

Illustrationofthebasicontologyoftheargumentmap11

Following this basic ontology the system develops a more elaborate set of connections

betweenideas,identifiedwithdifferentcolouredarrows.Thetablebelowdetailstherange

ofconnectionsavailableforideas.

11http://debategraph.org/Stream.aspx?nid=400384&vt=bubble&dc=focus

Tabledetailingrelationshipsbetweenitemsonamap12

Inadditiontothehierarchicalstructureformingthebasisofthemaps,therearealsoaseries

ofconnectionscalled“crosslinks”whichprovidelinksbetweenmapsorinformationabout

themapitself(forexampleinstancesofinconsistenciesorequivalence).Thesearedetailedin

thetablebelow.


Tableshowing“crosslinks”,relationshipsbetweenitemsonthemapandothermaps13

Debategraph provides a side menu offering further help in construction of maps and

additional details; there are alternative ways of viewing the maps which allow for the

inclusionof imagesandvideos. Themaps canbeembeddedonother sites,with changes

madetoamaponagivensitesharedacrosstheothermaps.Themapscanbenavigatedsuch

that when a user clicks on a particular element of themap, the perspective of themap

changes revealing further connections. In this sense Debategraph hopes to capture the

interconnectednatureofmanyissuesthataresubjecttoargumentmapping.

Incomparisontoplatformstakingasimilarapproachtoargumentmappingandvisualisation,

Debategraphcouldbesaidtoofferaricherexperience inrelationtothekindsofmedia it

supports, the options it allows for visualisation and navigation, and the aesthetic of the

platform.Theplatformincludesanumberofnovelfeaturesnotfoundinmorebasicargument

mappingplatforms,notably themaps’capacity to rearrangethemselvesaroundparticular

pointsasusersnavigateandtheextensiveontologyofthemaps.Thisprovidesaninteresting

approach to issues of framing and structuring debate. Of the platforms reviewed,

Debategraphisoneofthemostsuccessfulinrelationtoapplicationsbyotherorganisations


andexistingresearchandliterature.Therelativesuccessoftheplatformandnoveldesign

choicesmakeitveryrelevantforthepurposesofthecurrentstudy.

ScreenshotsandImagesofthePlatform

Thescreenshotsbelowprovideillustrationsofthedesignofdebategraph.

OpeningpageofDebategraph14

ExampleofDebategraphmap15

14http://debategraph.org/home


(Tambourisetal2011:4)


TheDebategraphsiteclaimsthattheplatformisbeingusedinover100countriesandlists

applications inareas including “education,health, governance,media, conferences, group

facilitation,conflictresolutionandpublicconsultationandplanning”(Debategraph2017).It

hasbeenusedbyorganisationssuchasCNN,theWhiteHouse(onopengovernment),theUK

Prime Minister’s Office (on media policy), The Independent, Bill and Melinda Gates

Foundation(onglobalhealth),andtheForeignOffice(BullenandPrice2015,Debategraph

2017).Tambourisetal(2011)observethatDebategraphisoneofthemostmatureandstable

examplesofargumentvisualisation tools,and therehavebeenstudiesusing the tool ina

number of different contexts. Bullen and Price (2015) explore the use of Debategraph in

supporting analysis of complex policy problems, specifically obesity. One policy maker

involved in the studyemphasised thepotentialofDebategraphasamethodof collecting

variousdifferenttypesofdataonanissueandpresentingitclearly,withlessinterestinits

capacity to support debate (Capehorn in Bullen and Price 2015). Crossley-Frolick (2017)

explores theuse of debategraph in educating undergraduate political science students. A

classuseddebategraphtoengage indebatesconcerningcomplex issuessuchaspolicyon

climatechangeandsextrafficking.Crossley-Frolickfoundthatthetooldidimprovestudents’

understandingofthetopic,yetthestudentsreportedissueswiththenavigationsystemand

easeofuseoftheplatform.Afurtherissuehighlightedwasprivacy,asuserswhowerenot

partoftheclassbeganeditingdebates.Tambourisetal (2011)studiedtheexperiencesof

policymakersandexpertsusingDebategraphinthecontextofEuropeanlegislation.Inthis

studyDebategraphwas used in conjunctionwith other software calledWAVE. The study

foundmixedresultsagainwiththeusabilityoftheplatform,thoughparticipantsnotedthat

easeofuseimprovedafterashortlearningperiod.Participantsnotedthattheplatformwas

attractiveandwouldbemostappropriateforanalysisanddraftingandevaluationofpolicy,

as well as consultation on policy, while it would be less useful for formulation or

implementationofpolicy(Tambourisetal2011).Scheueretal (2010)noteDebategraph’s

support for large scale argumentation and large community use. They highlight potential

problemswithgraphicalrepresentationbeingusedfordebates,notablythattheymightfeel

unnaturalandunintuitiveanddependingonthetopicandnumberofparticipants,theboxes

andarrowsmaybesubstantialleadingtothemapsbecomingclutteredandhardtoread.

Deliberatorium


TheDeliberatorium(formerlyknownastheCollaboratorium)isdescribedasan“innovative

internet tool whose goal is to enable better collaborative deliberation” (Deliberatorium

2017). The project is developed and led byMark Klein atMIT and draws on the field of

collectiveintelligenceandspecificallytheIBIStomaparguments.Discussionisorganisedby

topicandbrokendownintothefollowingcomponents:

Issue:Aproblemthatneedstobesolved

Idea:Anapproachforaddressingthatissue

Argument:Apointfor(pro)oragainst(con)anidea(Deliberatorium2017)

The literatureon thedeliberatoriumdescribes the followingexpectationsofauthors, that

authorssubmitasingleissue,ideaorargument,thatitnotreplicateapointthathasalready

beenmade,andshouldbeattachedtotheappropriatepartofthemap.Postsshouldonlybe

editedtostrengthenthem,ifonedisagreesoneshouldcreateanewpostthatcountersthe

idea(the liveand let liverule) (Klein2011).Toguideargumentation,moderatorsevaluate

postsforcorrectstructureandvalidity(Scheueretal2010:8)

(Klein2011:5)

It is estimated that 1 moderator for every 20 contributors is required for sufficient

maintenanceoftheDeliberatorium(Klein2011).Inadditiontothemapthereisachatroom

areaforlessformalisedconversation.

Theplatformprovidesagoodexampleoftheargumentmappingapproach,basedontheIBIS

approach common in collective intelligence literature. It is supported by applications in

practicethatdemonstrateitsuseinlargescalediscussionsleadingtoadecision(discussedin

greaterdetailbelow).Thedevelopersdirectlydescribetheintentiontoaddresschallenges

relating to feasibility and scale, participant behaviour and citizen capacity as well as

informationmanagement. The platform is also a notable example of the combination of

synchronousandasynchronouscommunicationtosupportdeliberation(Delborneetal2011).


Thescreenshotsbelowprovideanillustrationofthevisualrepresentationandaestheticsof

the delibratorium. The Scholio project is currently working on the aesthetics of

Deliberatoirum.

Displayinghowuserscanintroduceanideaandvoteonit16

Displayingthegeneralargumentmap17

16http://franc2.mit.edu:8000/ci/show-top

17http://deliberatorium.mit.edu/

Displayingauserclickingonanindividualideainanargumentmaptogetfurtherdetail18


TheDeliberatoriumhasbeentestedandappliedinanumberofsettings.Initially“TheCarbon

Offsetting Thought Experiment” attempted to translate a 13pagediscussion into amuch

moresuccinct8itemdeliberationmap(Kleinetal2012).Thefirstmajorevaluationofthe

Deliberatoiruminvolved220mastersstudentsattheUniversityofNaplesinadebateabout

bio fuels. The experiment focused on whether the demands of structure would put off

participants, andwhethermoderatorswere able todealwith thedemandsof large scale

participation.Theresearchreported thatneitherof theseconcernswereproblematicand

theyobservedveryhighlevelsofuserparticipation(Klein2011).Thestudentscreatedamap

that was judged by content experts to represent a remarkably comprehensive and well

organisedreviewofthekeyissuesandoptionsaroundbiofueladoption(Klein2011).Further

evaluationsoftheDeliberatoriumhavetakenplacewithIntelCorporation,USBureauofLand

Management,theUniversityofZurichandHMCInc(Klein2011).TheDeliberatoriumhasbeen

usedbytheItalianDemocraticPartyinaninternalpartydebateoverelectoralreform.This

experimentinvolved400people,withtwogroupsof160participantsassignedtodiscussthe

topic through either the Deliberatorium or through a standard forum. It found that the

restrictedstructureofdiscussiondidnotaffectusers’retentionratenortheiraveragedaily

18http://deliberatorium.mit.edu/

activity,whiletheargumentmapreducesthequantityof ideaspostedbyusers, the ideas

postedtendedtobemoredevelopedintermsofsupportingarguments(Kleinetal2012).This

suggestscertainadvantagesanddisadvantagestothetoolthatmaydeterminewhere it is

best applied. Scheuer et al (2010) discuss Collaboratorium, an earlier version of

Deliberatorium,describinghowtheplatformsupportsthenotionofcollectiveintelligenceby

allowing participants to rate contributions, the highest rated being considered the

community’sdecisions.

Parmenides

BackgroundandObjectivesofthePlatform

Parmenidesisdescribedasan“e-participationforum...asystemfordeliberativedemocracy

that allows the government to present policy proposals to the public and lets the public

submittheiropiniononthepolicyandits justification”(Parmenides2017,Cartwrightetal

2009). In other literature Parmenides has been described as a platform that collects

argumentsforandagainstagivenproposal(Atkinsonetal2004).

Parmenides is informedby informal logic,specificallyamodificationof theargumentation

schemesofWaltonandtheBeliefDesireIntentionarchitecture(Atkinson2006).Thisguides

the platforms heavily structured interface design. The platform webpage explains that

“Parmenidesexploitstwomethodsofargumentrepresentation:Argumentationschemesto

structurepolicyproposalsandargumentationframeworkstodiagrammaticallyanalysethe

opinions submitted by users” (Parmenides 2017). It consists of fourmain components: a

debate creator (administrators can create a debate by instantiating elements of the

argumentationscheme);theParmenidesinterface(allowingpeopletoparticipateandsubmit

their opinions); administration tools (allowing argumentation schemes to be added); and

analysis tools (allowing information submitted to be analysed using argumentation

frameworksandvalue-basedargumentationframeworks)(Cartwrightetal2009).

Parmenidesdialoguestructureseesthejustificationforanactionasinvolvingthefollowing

argumentationscheme:anunderstandingof thecurrent situation;aviewof thesituation

whichwillresultfromtheperformanceoftheaction;featuresofthenewsituationwhichare

considereddesirable(theaspectswhichtheactionwasperformedinordertorealise);the

social goals which are promoted by these features (the reasons why they are desirable)

(Atkinsonetal2004).Fromthisscheme,aseriesofpotentialwaysof‘attacking’aproposal

areidentified.

(Atkinsonetal2004:314)

Attacks12,13,14and3(detailedinthetableabove)areneglectedonthegroundsthatthe

developersfeltthatanargumentproposedthroughtheirsystemcouldbepresumedtobe

sound and describe actions that were possible. The remaining attacks are used for the

structuringofaninterfacethatguidestheuserthroughajustificationforaproposal,giving

userstheopportunitytodisagreeatselectedpointsandcollectinginformationonwheremost

usersdisagreewithaproposal.Attacks7,8,9and11areusedinthestructureasthebasisto

allowuserstosubmitalternativeproposalsforaction(Atkinsonetal2004).Inadditiontothe

platform itself there are further tools to support information gathering with Parmenides

includingadebatecreator,profilinginformationofthosewhoparticipateandanalysistools

(Parmenides2017).

Parmenidesisuniqueamongstonlinedeliberationplatformsinitsapproachoftakingusers

through a heavily structured dialogue process. It represents an interesting application of

informal logic theory to the problem of online deliberation. Applications of the platform

howeverarelimitedandthecurrentpubliclyavailableversionoftheplatformisrestrictedto

fourspecificdebates.


ThedesignofParmenidesisillustratedbelowwiththeexampleofafoxhuntingdebate.

DisplayinghomepageoftheParmenidesplatform19

Displayingintroductiontospecifictopic,inwhichausergivestheirinitialresponse20

19http://cgi.csc.liv.ac.uk/~parmenides/foxhunting/20http://cgi.csc.liv.ac.uk/~parmenides/

Displayinganexampleofthestructuredinterface,inwhichauserindicatestheircommitment

toparticularvalues,andisgiventheoptionofincludinganyadditionalvaluesrelevanttothe

topic21

21http://cgi.csc.liv.ac.uk/~parmenides/foxhunting/

Parmenides(2017)Displaysasummaryofauser’sresponsestoaparticularargument22


AversionofParmenidesispubliclyavailableandallowsfortheexplorationoffourcurrent

debates, including fox hunting and speed cameras, and a record of one previous debate.

Although the literature discusses Parmenides use in the context of e-participation and e-

democracy(Atkinsonetal2004,Cartwrightetal2009),theapplicationofthetoolinthisarea

hasbeenlimitedandthefocusofitscurrentusehasmovedtowardsprivateapplicationin

the field of law. Atkinson et al (2004) argued that Parmenides was usable by its target

audienceandcanbeusedtoidentifypointsofdisagreement,andrecordthemsothatthe

weightofopiniononvariousissuescanbegauged.CartwrightandAtkinson(2008)document

22http://cgi.csc.liv.ac.uk/~parmenides/foxhunting/

thedevelopmentoftoolsfore-democracy,includingallowingthesystemtocollectopinions

ondifferenttopics,analysedata,anddemographicprofilingofusers.Thepaperdescribesthe

intentiontoconductlargescalefieldteststovalidatetheeffectivenessofthesystem.

Pol.is

Background,ObjectivesandDesign

Pol.is“helpsorganisationsunderstandthemselvesbyvisualizingwhattheirmembersthink

(Pol.is2017). Itaims toallowthegatheringofopenended feedback from largegroupsof

people. Users click “agree”, “disagree” or “pass” in response to statements others have

contributed.Usersareabletosubmittheirowncomments;however,theyarenotallowedto

replydirectlytoacomment.Pol.isrunsstatisticalanalysisonthesevotingpatterns,surfacing

opiniongroups,comments thatbroughtgroupstogetherandcomments that foundbroad

consensus(Pol.is2017).Itthenprovidesavisualrepresentationofthesegroupsandclusters

ofopinion.Thedecisiontorepresentclustersofopinionratherthanplacingthoseopinionsin

thecontextofanargumentmapandnotallowingdirectresponsestomessagesdistinguishes

Pol.is from many other approaches to online deliberation and online communication in

general.

Pol.iswasconceivedduringthetimeoftheArabSpringandOccupyWallStreetmovement,

andsoughttodevelopacommentsystemthatcouldscaleupandretaincoherencewithlarge

groupsofpeople.ThedevelopersofPol.isclaimthatthedesignchoiceofnotallowingreplies

iskeytomakingitpossibletomakesenseoflargegroups;itisclaimedthatargumentsdonot

scale,andthemomentonebeginstotrackaconversationbetween individuals,andother

people’sresponsestospecificcomments,thenthatsenseofscalebreaksdown(Megill2016).

Thedevelopersfurtherstatetheaimsofensuringpeoplefeelsafe,listenedto,thatpeople

mayparticipateatanytimeinthelifecycleoftheconversation,thattheyhaveasenseof

what others felt and minority opinions are preserved and represented (Pol.is 2017). In

allowingstructuredresponses(agree,disagree,pass)andnodirectrepliesitisclaimedthat

problemsoftrollingandothernegativebehaviourareaddressed.Thevisualrepresentation

aims to ensure that participants can see all voices represented as well as any points of

consensus,particularlyamongstotherwisedisparategroups.Inthissense,itisclaimedthe

problemsofechochambersandfilterbubblesareaddressedthroughthesechoicesaround

visualrepresentation(Pol.is2017).InorderforPol.istobemoreeffectiveinrepresentingthe

different views of a collective group it encourages participants to respond rather than

submittingoriginalcomments.Oneofthewaysitdoesthisisbyusingafuzzysearchtoshow

usersothercommentsthatmaybesimilartotheonetheyaretyping.

Pol.isrepresentsaninterestingexampleoftheuseofAIinonlinedeliberation.Ithasbeen

successfullyappliedinanumberofcontexts,notablyindecisionmakinginTaiwan.Anumber

ofdesignchoicesmake itunusualamongstonlinedeliberationplatforms;notably the fact

thattheplatformdoesnotattempttodirectlysupportthecriticalworkofplacingopinions

withinthecontextofanargumentstructure,andthedecisionnottoallowdirectrepliesto

messages.Theplatformisnotableforitsuniquedesignanditssuccessfulapplications.


Thescreenshotbelowillustratesthevisualisationprovidedbypolisofcommentsandgroups.

Conversationinterface(Megill2017)

Representationofopiniongroups,highlightingthelargestgroup,groupB(Megill2017)

Metadataonthegroup,includinginformationonareasofuncertainty(Megil2017)


Pol.ishasbeenusedinanumberofsettings,notablyinTaiwan,wheretheprojectvTaiwan

usedPol.istoaddressparticularissuessuchasliquorsales,crowdfunding,Uber,andAirbnb

(Megill2016,Berman2017,Barry2016,Tang2016).Inthiscase,decisionmakingtookthe

formoffourstages.First,Pol.iswasdistributedthroughFacebookadsandnetworkstargeting

participants.Publicmeetingswerethenbroadcastwherescholarsandofficialsrespondedto

the comments emerging through Pol.is. This was followed by face to face stakeholder

meetingsbroadcasttootherparticipants.Inthefinalstage,Barry(2016)describeshowPol.is

wasusedinabindingway,withthegovernmentcommittingtoeithertransformconsensus

intonationallegislatureorprovideapointbypointexplanationofwhythisisnotpossible.

Theseapplicationshavebeenreportedtobesuccessful, forexample it isclaimedthatthe

issueofonlineliquorsaleshadbeenindeadlockforfiveorsixyears,butthroughvTaiwanand

theuseofPolisadecisionhadbeenreachedinthreetofivemonths.AsofFebruary2018,26

cases have been discussed through vTaiwan, 80% have led to decisive action from the

government(vTaiwan2018).Tang(2016)describeshowthecaseconcerningUberandTaxi

servicesresultedintheadministrationpledgingtoratifyallPol.isconsensusitemsintonew

regulation.Asanindicationofthesuccessoftheprocess,Taiwan’spremierisquotedassaying

“allsubstantialnationalissuesshouldgothroughavTaiwan-likeprocess”(Barry2016).

Rbutr

Background,ObjectivesandDesign

Rbutr“isacommunity-drivenappwhichconnectswebpagestogetheronthebasisthatone

pagearguesagainsttheother”(Rbutr2017). Itutilisescrowdsourcingto identifyrebuttals

andcriticalresponsestoagivenwebpageorarticle.Itwasinitiallydevelopedasaplug-in

alertinguserstorebuttalsofarguments.Thereisanadditionalframeoptiontoviewrebuttals,

andanaccompanyingwebsiteofferingfurtherfunctions.ThedevelopersofRbutrdescribeits

aimsasbeingtopromotecriticalthinkingforfuturegenerations,tacklingtheproblemoffilter

bubblesandensuringthat“misinformationiscorrected,scamsareexposed...andcontextis

providedtoaclaimthatallowsreadersaccesstothefullstory”(Rbutr2017).

UsersofRbutrcanuse theplug-in tosubmit rebuttals.Usersareable to linkonepageto

anotherpagethatcontainsarebuttalofthefirst.Usersarealsoabletoaddrelevanttags.

Rbutralsoutilisessocialmedia,forexampleitidentifieswhereanarticlehasbeenlinkedto

byTwitterusersandallowsforanautomaticresponsealertingthatusertorebuttalsofthat

article.Agivenarticle canhaveanumberof rebuttals linked to it, these rebuttals canbe

rankedbyusersinthehopethatpeoplewillbedirectedtowhatisconsideredtobethebest

exampleofarebuttalofagivenarticle.

TheRbutrwebsitestatesthattheprojectisrunbyasmallteamwithverylittlefunding.It

receivedseedfundingandsupportfromtheStartUpChileProgramme,thedevelopersmade

Rbutropensourceandnon-profitandarecurrentlyengagedinfundraisingwitheducational

organisations, fact checking organisations and publishing platforms aswell as looking for

volunteersanddonors(Rbutr2017).Rbutrisagoodexampleofanannotationapproachto

argument representation and online deliberation. It aims to address a specific set of

challenges inonlinedeliberation,thefilterbubbleandpolarisationofdebate,andaimsto

fostercriticalthinkingandengagementwithalternativeviewpoints.


ThescreenshotsbelowillustrateRbutr’sdesign.

DisplayinghowRbutrappearswhenvisitingawebsite23

23https://www.youtube.com/watch?v=V9qi3vaLKyU

Interface

foraddingrebuttals24

DisplayingRbutr’swebsite,displayingtherebuttedarticle,2rebuttals,anddetailsoftweets

sharingtherebuttedarticle25


There is currently no literature on the use of Rbutr, however a study into its use in an

educational context is being undertaken currently (Rbutr 2017b). The developer, Shane

24http://www.rbutr.com25http://www.rbutr.com

Greenup,describeshowRbutrhasreachedapeakof20,000usersandhasbeensurprisingly

robust. It has not experienced any problems with spam and every rebuttal has been

appropriate.

Truthmapping


TruthMappingisawebsitethatallowspeopletoconstructargumentmaps.Thedevelopers

describethedesignofTruthmappingasanattempttoresolvetheproblemof“noise”,which

isassociatedwithforumsorganisedbytime.Theproblemof“noise”isattributedtoanumber

offactors:Digressionfromthetopic,the“soapboxproblem”andparticipantsspeakingpast

oneanother.The“soapboxproblem”,itisargued,isaresultofparticipantsbeingincentivised

tobethelastpersontotalkorthemostvocal.Itisarguedthatthisisaprobleminforums

organisedbytimewhereapersonismostlikelytobeheardiftheircommentislistedfirstor

iftheymakethecommentrepeatedly.Similarly,whenparticipantsdigressorspeakpastone

anothertheyproduceinformationthatisredundantanddetractsfromtheinformationmore

relevant to the discussion (Truthmapping 2017a). Truthmapping is an attempt to resolve

theseissuesthroughtheuseofargumentvisualisation.Thisapproachtoargumentmapping

wascreatedbythedevelopersofTruthmappingandisnotgroundedinaspecifictheoretical

accountofargument.Theontologyoftheargumentmapsconsistsofconclusionssupported

by premises; these premises can then be critiqued and those critiques rebutted. This is

illustratedbelow.

Truthmapping2017

Onlyonerebuttalcanbeaddedtoacritique.Theseelementscanbeedited;thefinaldraft

being visible to users,while the previous drafts are archived. This process is intended to

ensurethatthebestexamplesofthecritiquesandrebuttalsarepreserved.Itisclaimedthis

mechanism removes the incentive of the “soapbox problem” and discourages digression

(Truthmapping2017a).Truthmapping isagoodexampleofargumentmappingtechniques

beingusedforgeneralpublicdiscussionandpoliticaldebate,theattempttolimitrebuttals

throughaneditingprocessalsorepresentsaninterestingapproachtotheissueofinformation

managementthatisnotfoundinmostotherargumentmappingtools.


ThescreenshotsbelowillustratethedesignandaestheticsofTruthmapping.

Displayinggeneralviewofargumentmap26

26https://www.truthmapping.com/map/806/#s5361

Displayingauserclickingonindividualargumentswithinthemaptoviewfurtherdetails27


AliteraturesearchfoundnodetailsonapplicationsorstudiesofTruthmapping,beyondthe

material available on the Truthmapping site. There is a publicly available version of

Truthmapping which limits the number of maps one can publish, in addition there are

subscriptionplans forsmallgroupsand largergroupsaimedatnon-profitandeducational

organisationswithadditionalfeatures(Truthmapping2017b).Thepubliclyavailableversion

ofTruthmappinghasbeenrunningforover10yearsandcontinuestohaveregularvisitors

andpublishedmaps.TodatethemostpopularcategoriesoftopicconcernPhilosophy(89

maps),Politics(73maps)andSocialIssues(59maps).

Conclusion

What is clear from this review is that there is a diversity of novel designs for the

representationofideasandargumentsandthepromotionofelementsofdeliberationonline.

Thecloseanalysisoftwelveofthemostwellrespecteddesignscurrentlyinoperationgives

usasenseofthebreadthofthefieldandthediversityofapproachestosupportingaspects

of deliberation. However, it is hard to capture the full range of activity because of

methodological challenges in defining the population and its characteristics.We are also

victims of the speed of change in the field: new designs emerge rapidly to improve the

27https://www.truthmapping.com/map/806/#t5363-c5484

interface,aestheticsandargumentrepresentation.Practiceismovingfasterthanourcapacity

toconceptualiseandcompare.

BibliographyAtkinson,K,BenchCapon,T,McBurney,P(2004)“Parmenides:FacilitatingDemocraticDebate”inInternationalconferenceonElectronicGovernment313-316Atkinson,K(2006)Whatshouldwedo?:Computationalrepresentationofpersuasiveargumentinpracticalreasoning,Ph.D.thesis,Dept.ofComp.Sci.,Univ.ofLiverpoolBarry,L(2016)VTaiwan:PublicParticipationMethodsOntheCyberpunkFrontierofDemocracy[ONLINE]Availableat;https://civichall.org/civicist/vtaiwan-democracy-frontier/[Accessed9/7/2017)

BCisiveOnline(2017)BCisiveOnline[ONLINE]Availableat:https://www.bcisiveonline.com[Accessed9/7/2017]BermanP(2017)“Hackingideology:pol.isandvTaiwan”inDemocracyEarth[ONLINE]Availableat:https://words.democracy.earth/hacking-ideology-pol-is-and-vtaiwan-570d36442ee5[Accessed:2.12.17]Black,LW.(2011)“Thepromiseandproblemsofonlinedeliberation”inCharlesF.KetteringFoundation,IncBullen,V,Price,D(2015)“Debategraph:ANewwaytoaddressthecomplexityofobesity”inBritishJournalofObseity1:54-59Cartwright,DandAtkinson,K(2008)“PoliticalEngagementthroughToolsforArgumentation”inCOMMAconferenceproceedingsCartwright,D,Atkinson,K,Bench-Capon,T(2009)“SupportingArgumentinE-Democracy”inElectronicGovernment:thirdinternationalelectronicdemocracy(EDEM2009)(Vienna,Austria)151-160Coleman,SandGoetze,J(2001)BowlingTogether:Onlinepublicengagementinpolicydeliberation,report,London:HansardSocietyColeman,S.andMoss,G.(2012)‘Underconstruction:Thefieldofonlinedeliberationresearch’inJournalofInformationTechnology&Politics,9(1),1–15Conklin,J(2008)“GrowingaGlobalIssueBase:AnIssue-BasedApproachtoPolicyDeliberation”inToolsforParticipation:Collaboration,DeliberationandDecisionSupport,Proceedings,26–29June2008

ConsiderIt(2017)ConsiderIt[ONLINE]Availableat:https://consider.it[Accessed9/7/2017]

Crossley-Frolick,K(2017)“Visualising“wickedproblems”byusingdebategraphandDialogueMapping”[ONLINE]Availableat:http://glcateachlearn.org/2017/04/10/visualizing-wicked-problems-by-using-debategraph-and-dialogue-mapping-katy-crossley-frolick-denison-university/[Accessed09/7/2017]

Davies,T.(2009)“TheblossomingfieldofOnlineDeliberation”inTDaviesandSPGangadharan(eds)OnlineDeliberation:Design,ResearchandPractice,SanRancisco,CA:CSLIPublication

Davies,W.M.,(2009b)“Computer-AssistedArgumentMapping:ARationaleApproach”2009.HigherEducationDavies,TandChandlerR(2011)“OnlineDeliberationdesign:choices,criteriaandevidence”inTNabatachi,MWeiksner,JGastilandMLeighninger(eds)DemocracyinMotion:EvaluatingthePracticeandImpactofDeliberativeCivicEngagement,Oxford:OxfordUniversityPressDebategraph(2017)Debategraph[ONLINE]Availableat:http://debategraph.org/Stream.aspx?nid=65027&vt=bubble&dc=focus[Accessed9/7/2017]Delborne,JA,Anderson,AA,Kleinman,DL,Colin,M,Powell,M(2011)“VirtualDeliberaton?ProspectsandChallengesforintegratingtheInternetinConsensusConferences”inPublicUnderstandingofScience20(3)367-384Deliberatorium(2017)Deliberatorium[ONLINE]Availableat:http://cci.mit.edu/klein/deliberatorium.html[Accessedon9/7/2017]DeLiddo,AandBuckinghamShum,S(2010a)“CapturingandRepresentingDeliberationinParticipatoryPlanningPractices”inFDeCincio,AMachintoshandCPeraboni(eds)Onlinedeliberation,proceedings,FourthInternationalConference,Leeds,UKDeLiddo,A.,BuckinghamShum,S.(2010b)“Cohere:Aprototypeforcontestedcollectiveintelligence”InACMComputerSupportedCooperativeWorkConference(CSCW2010):WorkshoponCollectiveIntelligenceinOrganizations,Feb6--10,2010DeLiddo,AandBuckinghamShum,S(2013).“TheEvidenceHub:harnessingthecollectiveintelligenceofcommunitiestobuildevidence-basedknowledge”inLargeScaleIdeationandDeliberationWorkshop,29June–02July2013,Munich,GermanyDriel,S.van,&Prakken,H,(2010)“VisualisingtheargumentationstructureofanexpertwitnessreportwithRationale”InA.Z.Wyner(ed.),ProceedingsoftheWorkshoponModellingLegalCasesandLegalRules,inconjunctionwithJURIX-10,Liverpool2010,pp.1-8

Duhaime,E,Olson,GThomasW.Malone,T(2015)“BroadParticipationinOnlineProblemSolvingCanInfluenceParticipantsandLeadtoBetterSolutions:EvidencefromtheMITClimateCoLab”inMITCCIWorkingPaperNo.2015-001,June2015.eLab(2017a)“ParticipatoryBudgetingWhitePaper”inEngagementLabatEmersonCollege[ONLINE]Availableat:elabhome.blob.core.windows.net/resources/stake-pb-white-paper.docx[Accessed09/07/2017]eLab(2017b)“@Stake:GameMaterialsandDirections”inEngagementLabatEmersonCollege[ONLINE]Availableat:http://elabhome.blob.core.windows.net/resources/stake-rules-and-role-cards.pdf[Accessed30/07/17]EngagementLab(2017)EngagementLab[ONLINE]Availableat:https://elab.emerson.edu/projects/participation-and-engagement/atstake[Accessed9/7/2017]EvidenceHub(2017)EvidenceHub[ONLINE]Availableat:http://evidence-hub.net/about/[Accessed9/7/2017]

Fishkin,J(1995)“TheVoiceofthePeople:PublicOpinionandDemocracy”NewHaven:YaleUniversityPressFreelon,DG,Kriplean,T,Morgan,J,Bennett,W,Borning,A(2011)“Facilitatingencounterswithpoliticaldifference:Engagingvoterswiththelivingvotersguide”inProceedingsoftheJournalofInformationTechnologyandPoliticsAnnualConference,2011Freelon,DG,Kriplean,T,Morgan,J,Bennett,WL,Borning,A(2012)“Facilitatingdiversepoliticalengagementwiththelivingvotersguide”inJournalofinformationTechnologyandPolitics9(3)279-297Friess,DandEilders,C(2014)“ASystematicReviewofOnlineDeliberationResearch”inPolicyandInternet7(3)Gelder,T.van,(2007)“TherationaleforRationale”inLaw,ProbabilityandRisk,(2007)6,23−42

Gordon,E.,Michelson,B.,&Haas,J.(2016).“@Stake:AGametoFacilitatetheProcessofDeliberativeDemocracy”InProceedingsofthe19thACMConferenceonComputerSupportedCooperativeWorkandSocialComputingCompanion(pp.269-272).ACM

Iandoli,L,Klein,M,Zollo,G(2008)“Canweexploitcollectiveintelligenceforcollaborativedeliberation?Thecaseoftheclimatechangecollaboratoirum”inMITSloanResearchPaperNo.4675-08Introne,J,Laubacher,R,Olson,G,andMalone,T,(2011)“TheClimateCoLab:LargeScaleModel-BasedCollaborativePlanning”InInternationalConferenceonCollaborationTechnologiesandSystems2011

JanssenD,KiesR.(2005).“Onlineforumsanddeliberativedemocracy”inActaPolit.40:317-35Karlsson,M(2010)“Whatdoesittaketomakeonlinedeliberationhappen?Acomparativeanalysisof28onlinediscussionforums”inFDeCincio,AMachintoshandCPeraboni(eds)Onlinedeliberation,proceedings,FourthInternationalConference,Leeds,UKKlein,MandIandoli,L(2008)“SupportingCollaborativeDeliberationUsingaLargeScaleArgumentationSystem:TheMITCollaboratorium”inToolsforParticipation:Collaboration,DeliberationandDecisionSupport,Proceedings,26–29June2008Klein,M(2011)“HowtoHarvestCollectiveWisdomonComplexProblems:AnintroductiontotheMITDeliberatorium”inCCIworkingpaper2011Klein,M,Spada,P,Calabretta,R(2012)“Enablingdeliberationsinapoliticalpartyusinglargescaleargumentation:Apreliminaryreport”in10thInternationalConferenceontheDesignofCooperativeSystemsfromresearchtopractice:Resultsandopenchallenges,atMarseille,May29,2012Klein,M(2015)“ACriticalReviewofCrowd-ScaleOnlineDeliberationTechnologies”inMITSloanResearchPaperNo.5143-15Klein,M(2017)“CollectiveIntelligenceWorkingNotes”,unpublishedKriplean,T,Morgan,JT,Freelon,D,Borning,A,Bennett,L(2011)“ConsiderIt:Improvingstructureddeliberation”inCH’11ExtendedabstractsonHumanFactorsinComputingSystems,1831-1836Kriplean,T,Morgan,J,Freelon,D,Borning,A,Bennett,L(2012)“SupportingReflectivePublicThoughtwithConsiderIt”inCSCW12ProceedingsoftheACM2012ConferenceonComputerSupportedCooperativeWork265-274Kriplean,T,Bonnar,C,Borning,A,Kinney,B,Gill,B(2014)“Integratingon-demandfact-checkwithpublicdialogue”inProceedingsofthe17thACMconferenceonComputerKunsch,D,W,Schnarr,K,vanTyle,R(2014)TheUseofArgumentMappingtoEnhanceCriticalThinkingSkillsinBusinessEducationinJournalofEducationforBusiness89403-410Lawrence,JBex,FandReed.,C(2012)“DialoguesontheArgumentWeb:MixedinitiativeargumentationwithArvina”InB.Verheij,S.Szeider,andS.Woltran,editors,ProceedingsoftheFourthInternationalConferenceonComputationalModelsofArgument(COMMA2012),pages513–514,Vienna,Austria,2012.IOSPress.Lawrence,J,Janier,MandReed.C,(2015)“WorkingwithOpenArgumentCorpora”InEuropeanConferenceonArgumentation(ECA),2015

Lengbyer,L.,(2014)“CriticalThinkingintheIntelligenceCommunity:ThePromiseofArgumentMapping”inInquiry:CriticalThinkingacrossthedisciplines.Summer2014,Vol.29,No.2Loukis,EandWimmer,M(2010)“AnalysingDifferentModelsofStructuredElectronicConsultationonLegislationunderFormation”inFDeCincio,AMachintoshandCPeraboni(eds)Onlinedeliberation,proceedings,FourthInternationalConference,Leeds,UKMalone,T,Laubacher,R,Introne,J,Klein,M,Ableson,H,Sterman,J,Olson,G(2009)”TheClimateCollaboratoirum:ProjectOverview”inMITCentreforCollectiveIntelligenceWorkingPaperNo.2009-03Malone,T,Nickerson,J,Laubacher,R,HesseFisher,L,DeBoer,P,Han,Y,Towne,WB(2017)“PuttingthePiecesBackTogetherAgain:ContestWebsforLargeScaleProblemSolving”inProceedingsoftheACMConferenceonComputerSupportedCoopertativeWorkandSocialComputing,PortlandORFebruary25March1,2017Manosevitch,I(2014)“TheDesignofOnlineDeliberation:ImplicationsforPracticeTheoryandDemocraticCitizenship”inJournalofPublicDeliberation10(1)

MegillC(2016)Pol.isinTaiwan[Online]Availableat:ttps://blog.pol.is/pol-is-in-taiwan-da7570d372b5#.fq6w6zf4p[Accessed9/7/2017]

MegillC(2017)Pol.isCaseStudy[Online]Availableat:https://blog.pol.is/pol-is-case-study-temperature-check-a02dff7cc838[Accessed9/8/2017]

Michelson,B(2015)“@Stake:ASeriesofFieldNotesinGameDevelopment”inEngagementLabatEmersonCollege[ONLINE]Availableat:https://medium.com/engagement-lab-emerson-college/stake-a-series-of-field-notes-in-game-development-570078fe2e74[Accessed09/07/2017]Naurin,D(2007)“WhyGiveReason?MeasuringArguingandBargaininginSurveyResearch”inSwissPoliticalScienceReview13(4):559–75

Neblo,MA.(2007).Familydisputes:diversityindefiningandmeasuringdeliberation.SwissPoliticalScienceReview13:527–57

Parmenides(2017)Parmenides[ONLINE]Availableat:http://cgi.csc.liv.ac.uk/~parmenides/foxhunting/[Accessedon9/7/2017]ParticipateDB(2017)PartitipateDB[ONLINE]Availableat:http://www.participatedb.com/categorizations[Accessed9/7/2017]Pingree,RJ(2009)“DecisionStructure:ANewApproachtoThreeProblemsinDeliberation”inT.DaviesandS.P.Gangadharan(Eds.),Onlinedeliberation:Design,research,andpractice(pp.309-316).Stanford,CA:CSLIPublications

Pol.is(2017)Pol.isOverview[ONLINE]Availableat:https://docs.pol.is/welcome/Overview.htmlRationaleOnline(2017)RationaleOnline[ONLINE]Availableat:https://www.rationaleonline.com[Accessed9/7/2017]Rbutr(2017)AboutUs[ONLINE]Availableat:http://blog.rbutr.com/about-us/[Accessed9/7/2017]Rbutr(2017b)AcademicResearchandrbutr–AnnouncingacollaborationwiththeDelftUniversityofTechnology[Online]Availableat:http://blog.rbutr.com/2017/05/academic-research-and-rbutr-announcing-a-collaboration-with-the-delft-university-of-technology/[Accessed9/7/2017]ReasoningLab(2017)ReasoningLab[ONLINE]Availableat:http://www.reasoninglab.com[Accessed9/7/2017]

Reed,C(2017)“AgreementbetweenBBCandARG-Tech”inARG-tech:centreforArgumentTechnology[ONLINE]Availableat:http://www.arg-tech.org/index.php/agreement-between-bbc-and-arg-tech/[Accessed09/10/17]

Reiter,B.(2013).“Theepistemologyandmethodologyofexploratorysocialscienceresearch:CrossingPopperwithMarcuse”inThedialecticsofcitizenship:exploringprivilege,exclusionandracialization(pp.1-22).EastLansing,MI:MichiganStateUniversityPress

Scheuer,O.,Loll,F.,Pinkwart,N.,&McLaren,B.M.(2010).Computer-supportedargumentation:Areviewofthestateoftheart.InternationalJournalofComputer-SupportedCollaborativeArgumentation,5(1),43-102.

Schutt,R.K(2015)InvesitgatingtheSocialWorld:TheProcessandPracticeofResearch8thEditionSingapore:SagePublications

Shane,P(2004)“IntroductiontheProspectsforElectronicDemocracy”inDemocracyOnline:ProspectsforPoliticalRenewalThroughtheInternet(ed)Shane,PNewYork:Routledge

Shields,PandRangarajan,N(2013)APlaybookforResearchMethods:IntegratingConceptualFrameworksandProjectManagementStillwater,OK:NewForumsPress

Snaith,M.;Devereux,J.;Lawrence,J.;andReed,C.(2010)“Pipeliningargumentationtechnologies”inCOMMA2010,447–453

Steigler,H,DeJong,M(2015)“Facilitatingpersonaldeliberationonline:Effectsoftwoconsider.itvariations”inComputersinHumanBehaviour51(A)461-469

Sunstein,C(2001)Republic.com,Princeton,NJPrincetonUniversityPress

TambourisE.,DalakiouridouE.,PanopoulouE.,andTarabanisK.(2011)EvaluationofanArgumentVisualisationPlatformbyExpertsandPolicyMakers.InE.Tambouris,A.MacintoshandH.deBruijn(Eds.):ePart2011,LNCS6847,pp.73–84

Tang,A(2016)“DigitalToolsOpenUpTaiwan’sDemocraticImaginations”inpol.isblog[ONLINE]Availableat:https://blog.pol.is/digital-tools-open-up-taiwans-democratic-imaginations-d8f80432305c[Accessed9/7/2017]Truthmapping(2017a)Truthmapping[ONLINE]https://www.truthmapping.com/#[Accessed9/7/2017]Truthmapping(2017b)TruthmappingPlans[ONLINE]https://www.truthmapping.com/plans/[Accessed9/7/2017]Twardy,C(2004)“ArgumentMapsImproveCriticalThinking”,2004.TeachingPhilosophy,27:2.Tauro,C,Ahuja,M,Perez-Quinones,A,Kavanaugh,AandIsenhour,P(2008)“VizBlog:AVisualizationToolforBlogDiscover”inToolsforParticipation:Collaboration,DeliberationandDecisionSupport,Proceedings,26–29June2008Towne,WBandHerbslebJD(2012)“Designconsiderationsforonlinedeliberationsystems”inJournalofInformationTechnologyandPolitics9:97-115vTaiwan(2017)“Afewaccomplishments”invTaiwan[ONLINE]Availableat:http://info.vtaiwan.tw/#three[Accessed09/07/2017]

Yin,RK(1994)CaseStudyResearch:DesignandmethodsSecondEditionThousandOaks:SagePublications

Documents

Review of online argument representation platforms€¦ · theory (Manosevitch 2014). Consequently, how developers talk about the objectives of the platform may differ from concepts