Learning as a Machine: Crossovers between Humans and Machines · Learning as a machine: Crossovers between humans and machines. ... , data protection by design, privacy, machine learning

(2017).Learningasamachine:Crossoversbetweenhumansandmachines.JournalofLearningAnalytics,4(1),6–23.http://dx.doi.org/10.18608/jla.2017.41.3

ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 6

Learning as a Machine: Crossovers between Humans and Machines

MireilleHildebrandtLaw,Science,Technology,&SocietyStudies

FacultyofLawandCriminologyVrijeUniversiteitBrussel

[email protected]

ABSTRACT:ThisarticleisarevisedversionofthekeynotepresentedatLAK’16inEdinburgh.Thearticle investigates some of the assumptions of learning analytics, notably those related tobehaviourism.BuildingontheworkofIvanPavlov,HerbertSimon,andJamesGibsonaswaysof“learning as a machine,” the article then develops two levels of investigation (processing ofpersonal data and profiling based on machine learning) to assess how data driven educationaffects privacy, non-discrimination, and the presumption of innocence. Finally, the articlediscusseshowdataminimizationandprofiletransparencywillcontributetothemethodologicalintegrityof learninganalytics,whileprotectingthe fundamental rightsand freedomsofhumanlearnersthussafeguardingthecreativity,humour,andcontestabilityofhumanlearning.

Keywords:Dataprotection,dataprotectionbydesign,privacy,machinelearning,behaviourism,nudging,Pavlov,Simon,Gibson,capture,optimization,affordance

1 INTRODUCTION

Humanbeingslearnfromthemomenttheyareborn(Jarvis,2005;Campbell&Dawson,1995),andevenbeforethat.Welearnbecausewearevulnerable,becausewecansuffer,becausewehavesomethingtolose.Welearnbecausewecandie—becausewewilldie,likealllivingorganisms(ornearlyall),andwewanttodelaythatmomentandflourish.Welearnforthejoyoflearning,weareawarethatwelearn,and we help others to learn (our children, friends, colleagues, pupils). Machines learn because weconfigurethemassuch,toperformspecifictasksforus,orevenbecausewewilllearnfromthemaboutnewsolutionstoknownproblems,abouttheproblemsthatwemissed,andweevenconfiguremachinestolearnforthepurejoyofit—aspartofourcreativity.1Machinesalsolearntocontributetoourownlearningprocesses.

TheLAK’16conferencewas(also)aboutprivacyandlearninganalytics(LA),2makingapivotalpointbyinvestigatingthepitfallsofdata-drivenlearningasanissuethatshouldinformthedevelopmentofLAasa discipline, instead of framing such issues as side-effects to be dealt with at a later stage. Havingstudied the new Journal of LearningAnalytics, Imust admit to being deeply impressedwith the high

1Onthechallengesofcomputationalcreativity,seeLeijnen(2014).2Special2016issueon“EthicsandPrivacyinLearningAnalytics,”JournalofLearningAnalytics,3(1).



standardsofresearchandtheintegrationofmethodological,ethical,andlegalreflection.3Inmykeynote—andinthispaper—Icontributetosuchreflectionbyraisingthequestionofhowmachinelearningwill affect human learning, highlighting the need for effective redress of learners subjected toautomatedscoringpracticesbasedonLA. Thisistheintroduction.

LAcanbedefinedasdata-drivenapplicationsthathelpstudentsto improvetheir learningbehaviours,and/orapplicationsthathelpschools,colleges,anduniversitiestomanage,coordinate,andadministrateinformation and communication regarding the progress of their students’ performances (Baker &Inventado,2014).Suchprogressismeasuredinmachine-readableformat,whichimpliesthatwhicheverpartofthelearningprocesscannotbemademachine-readablewillnotbepartoflearninganalyticsandwillnotfeedintotheperformancemetrics.Totheextentthattheseperformancemetricswillcometodetermineour learningprocesses,weneedtoacknowledgethatwhateverremainsofftheradarofLAwillnotvisiblycontributetotheflourishingofthestudentsorthelearninginstitutionswedevelopandcreate. Thismayhave consequences for individual liberty and for thedignity of individuals insofar astheir learning capabilities cannot be translated into machine-readable formats.4 The famous legalphilosopherDworkinsummedupdemocracyandtheruleoflawintermsofequalrespectandconcernforeach individualperson.Thisgroundsdemocracy—oneperson,onevote—andtheruleof law—humanrightsoverrulemajoritariandisrespect(Dworkin,1991).Totheextentthatapersonisreducedtoinferencesabouthermachine-readablebehaviours,andtargetedbasedonsuchinferences,herfreedomfromunreasonable constraints (liberty) to constructher identity (dignity)maybe violated. This is notmerely her private interest, but regards the capability of each and every citizen to develop anindependentmind,which istheunspokenassumptionofaviabledemocracy.Beingtestedby learningmachineswilllurelearnersinto“thinking”likethem,becausetheyneedtoanticipatehowthesesystemsanticipatethem.Learnersmayevenstartthinkingintermsoftheirbehavioursinsteadoftheiractions,5or be nudged into a preconceived learning process instead of being challenged to develop a criticaldistance.6The latterhasconsequences forourcapability toengage indemocraticparticipation,whichdependsonindependentmindswillingtochallengeconvention,power,andauthority.

Inthispaper,Iwill investigatesuchpotentialimplicationsinthelightofdataprotectionlaw.Iwillnot,however,merelyprovideananalysisofcurrentandupcominglegalrules.Rather,IhopetoexplainwhatisatstakewhenemployingLAandhowEUdataprotectionlawaddressespotentialthreats.IwillarguethattodevelopandintegratefairandsustainableLAintofutureprooflearninginstitutions,weneedtobuild effective protection and feedback into the relevant architectures,where feedback concerns theinner workings of the analytics and the foreseeable consequences of their scoring operations. Thisshouldenablethecontestationoftheanalytics,wheneveritsignificantlyaffectsanindividuallearner.Insectiontwo,IwilldistinguishtwolevelsofLA:1)interventionsatthelevelofidentifiablestudents,and

3Forexcellentmethodologicalreflection,seeBerg,Mol,Kismihók,andSclater(2016).4OnecouldrephrasebyinquiringhowLAscoringpracticesaffecttheagencyandvulnerabilityofindividualhumanlearners,seesalientlyPrinslooandSlade(2016).5Forasalientinvestigationofthedifferencebetweenbehaviourandaction,seeVender(2011).6On“thepre-emptionofintent,”seeMcStay(2011).



2)analysisandpredictionbasedonaggregatedata.Thoseconcernedaboutdataprotectionoftengetstuckatlevel1,becausethisregardstheprocessingofpersonaldata.Mymainconcernhereiswithlevel2, because it enables personalized targeting based on machine learning. In section three, I willinvestigatehowweshouldunderstandtheideaof“learningasamachine,”distinguishing1)thePavlovapproach,2)theSimonapproach,and3)theGibsonapproach.ThiswillenableustobetterframethethreatsthatLAposestoprivacy,non-discrimination,dueprocess,andthepresumptionofinnocence,tobediscussed in section fouratboth levels. Finally, in section five, Iwilldiscuss the legalobligation toprovideprofile transparency,andhowdataprotectionbydesigncouldserveasaneffectivemeanstoachieve“technologicaldueprocess”withinthesettingsofLA.

2 TWO LEVELS OF LA

2.1 First Level of LA: Processing Personal Data

First, LA concerns interventionsat the levelof an identifiable student,bothwhencollectingherdata,andwhenapplyingtheresultsofLAtoher,whetherornotsheisawareofthis.NotethatintermsofEUdata protection law, identifiability includes both direct and indirect identification,while identificationincludes “being singled out.”7 The data collected may be data about previous grades, previouslyattendedschoolsoruniversities, recordsof teachers’evaluations,grading,all typesof tests,andevenalumnidatacapableoflinkingschoolorcollegebehaviourswithemploymentachievements.Thistypeofdataprocessingconcernstheautomationofpreviouslymanualadministration.LA,however,alsorefersto the collection of behavioural data, traced and tracked from new types of software that enableeLearning (Clark & Mayer, 2011). Such behavioural data may include keystroke and clickstreambehavioursthattracereadinghabitsandmeasuretimespentonvarioustasks,or,forinstancebiometricbehaviourssuchaseyemovementsthatindicateboredomorlossoffocus.Itispivotaltonotethatsuchbehaviouraldatamayalsobederivedfromsourcesoutsidetheeducationalsettingofthestudentandinstitution.Theymayconcernsocialmediadata,quantified-selfmetrics,orothertypesofdatathatmay— for instance — be bought from data brokers. Such out-of-context data may be correlated witheducational data to figure out how a student’s background, lifestyle, or context correlates with herlearningachievements.Thismayactuallybe legalor illegal,ethicalorunethical. Thepoint is that it ispossible.

Next, LA concerns capturing student data: interventions at this level include feedback given to thestudent,suchasreal-timefeedbackduringeLearningexercisesor“early-warning”systemsthatsupporttimely reconsideration of a student’s capacities or learning strategies. The first level, however, alsoincludes interventionsregardingan identifiablestudentthatshemayormaynotbeawareof,suchas

7Art.4.1GeneralDataProtectionRegulationR2016/697EU(GDPR)definespersonaldataasfollows:“‘personaldata’meansanyinformationrelatingtoanidentifiedoridentifiablenaturalperson(“datasubject”);anidentifiablenaturalpersonisonewhocanbeidentified,directlyorindirectly,inparticularbyreferencetoanidentifiersuchasaname,anidentificationnumber,locationdata,anonlineidentifierortooneormorefactorsspecifictothephysical,physiological,genetic,mental,economic,culturalorsocialidentityofthatnaturalperson.”Recital26determinesthat“singlingout”rendersapersonidentifiable.Thedefinitionisnearlyidenticaltothecurrentlyapplicableart.2(a)DataProtectionDirectiveD95/46/EC(DPD).TheGDPRwillcomeintoforceinMay2018andhavedirecteffectinalltheMemberStatesoftheEU,replacingrelevantnationallaw.



real-time reconfiguration of her adaptive eLearning software, placement in a specific class or group,automatedevaluationoftests,referraltoacounsellor,selectionforawards,grants,orbeingcategorizedasapotentialdrop-out,orfailuretomeetthestandardsofthecourseoreducationalinstitution(Fiaidhi,2014; Kalampokis, Tambouris, & Tarabanis, 2013). It could eventually include decisions based oninferencesconcerninghealthrisks,drugabuse,earningcapacity, financial resources,orpublicsecurityrisks.

2.2 Second Level of LA: Analytics, Machine Learning, KDD

The second level of intervention is not about interactionwith identifiable students, but concerns theanalyses of the data that prepare potential interventions regarding the student. This data can beanonymousdataorpersonaldata;ifitispersonaldata,itcanbepseudonymous.8Anonymizationwouldrule out the applicability of data protection law, as this implies that the data no longer qualifies aspersonal data. Pseudonymization implies that the “anonymization” is not irreversible, and thereforedataprotectionlawapplies.9Pseudonymization,however,mayqualifyasproperprotectionofthedata,meaningthatitconstitutessecuritybydesignordataprotectionbydesign.Thisdependsonwhethertheprocessingofthedataisfairandlawfulandonthekindsofrisksitentailsfordatasubjects(thelearner).The processing is unlawful if there is no legal ground or because the purpose of processing is notlegitimate or has been exhausted. In that case, pseudonymization will not make the processingoperationslawful.10

Analyzing largeamountsofdata canbedonebymeansof, for instance, knowledgediscovery indatabasis(KDD)ormachinelearning(ML).KDDhasbeendefinedasa“nontrivialprocessofidentifyingvalid,novel,potentiallyuseful,andultimatelyunderstandablepatternsindata”(Piatetsky-Shapiro,1996).MLhasbeendefinedasmachinesthat“learnfromexperienceEwithrespecttosomeclassoftasksTandperformancemeasureP, if itsperformanceattasksT,asmeasuredbyP, improveswithexperienceE”(Mitchell, 2006). The second level of intervention is about the detection of relevant correlations,association rules, and so forth. Basically, this level is all about pinpointing the key classifiers thatwillallowteachersandtheirinstitutionstogaininsightsintothedatapointsdeemedkeytotheirstudents’learning capabilities and to their organizations’ achievements. This can be done bymeans of varioustypesofmachinelearning(Greller&Drachsler,2012).Itisthesecondlevelthatforagespatternsinbig

8Art.4(5)GDPRdefinespseudonymousdataaspersonaldata:“‘pseudonymization’meanstheprocessingofpersonaldatainsuchamannerthatthepersonaldatacannolongerbeattributedtoaspecificdatasubjectwithouttheuseofadditionalinformation,providedthatsuchadditionalinformationiskeptseparatelyandissubjecttotechnicalandorganizationalmeasurestoensurethatthepersonaldataarenotattributedtoanidentifiedoridentifiablenaturalperson.”9Recital26GDPRspecifies:“Todeterminewhetheranaturalpersonisidentifiable,accountshouldbetakenofallthemeansreasonablylikelytobeused,suchassinglingout,eitherbythecontrollerorbyanotherpersontoidentifythenaturalpersondirectlyorindirectly[myemphasis].”Thismeansthatevenencryptiondoesnotanonymize,exceptifthedecryptionkeycannotbeaccessedbyanyonebutthedatasubject.SeealsoOpinion5/2014onAnonymizationTechniquesoftheArt.29WorkingParty,availableathttp://ec.europa.eu/justice/data-protection/article-29/documentation/opinion-recommendation/files/2014/wp216_en.pdf10Lawfulprocessingrequiresoneofsixlegalgrounds,specifiedinart.6GDPR(art.7DPD),andcompliancewiththeconditionsofart.5GDPR(art.6DPD),notablythatofpurposespecificationanduselimitation.Recital28GDPRconfirmsthatpseudonymizationcancontributetocompliancewiththeGDPRbyreducingtherisksfordatasubjects.



datatodevelopmoreadequateinterventionsatthefirstlevel,inordertoimprovetheperformanceofthestudentsand/ortoadvisethem—oreventoobligethem—“tochangecourse.”Suchpatternsmaybeabout“whenarestudentsreadytomoveonthenexttopic?Whenarestudentsfallingbehind inacourse?When is a student at risk for not completing a course?What grade is a student likely to getwithoutintervention?Whatisthebestnextcourseforagivenstudent?Shouldastudentbereferredtoacounsellor forhelp?” (Fiaidhi,2014).Thisquotation is fromofaneditorialon learninganalytics thatnicely sums up some of the key objectives of LA. In the editorial, Fiaidhi (2014) actually proposes toinvest in the analysis of so-called unstructured data to further the objectives of monitoring andimproving the cognitive and intellectual development of “learners.” He ends by commending acomprehensive learning architecture that integrates the analytics for structured and for unstructureddatabymeansofapredictiveengine,acontentengine,aninterventionengine,afeedbackengine,anda measurement engine. The objectives and the various “engines” help to remind us of the closerelationshipofbothlevelsofintervention,sincethesecondlevelisinstrumentalfortargetingstudentsatthefirstlevel,eitherbyprovidingthemwithfeedbackbasedontheknowledgemined,orbyprovidingtheirteachers,institutionalmanagement,employers,orothersthatpayfortheireducationwithadviceandrelevantpredictions.Suchpredictionsenablecoverttargeting,excludingstudentsfromenteringthenextlevelofanapplicationorfromthenextgrade,fromaschoolorcollege,orfromremedialprograms.Thisshouldbekeptinmindwhendiscussingprivacyandotherfundamentalrightsthatmaybeinfringedwith large-scale implementation of LA. Though from a security and confidentiality perspectiveanonymizationorpseudonymizationshouldbeaprimaryconcern,therearemajorconcernsthatcannotbe addressedby de-identifying the data used for LA. This concerns notmerely privacy, but also non-discrimination,dueprocess,andthepresumptionofinnocence.Moreover,itconcernstheaffordancesoflearninginstitutions,andthekindofcapabilitiestheyenableforhumanlearners.11

3 THREE WAYS OF “LEARNING AS A MACHINE”

Beforemoving into fundamental rights infringements,weneed tounderstandwhatmachine learningdoes(asatooltocontributetohumanlearning)andhowitmaytransformthepracticeandconceptofhuman learning. Thismeans investigating the affordances ofmachine learning applications in LA andhowtheyinteractwiththecapabilitiesofhumanlearnerstargetedbyLA.Theinvestigationwillinquireinto three ways of “learning as a machine,” which I will coin “the Pavlov approach,” “the Simonapproach,”and“theGibsonapproach.”Afterthat,Iwillinvestigatehowsecond-levelinterventions(KDDandMLoperations)may impact fundamentalrights,andfinallyexplainhowdataprotectionbydesign(DPbD)mayhelptopreventdataobesitas,lossofreputation,anduntrustworthyLAfromdestroyingitspotentialasatoolforbettereducation.

3.1 The Pavlov Approach: Manipulating Behaviour

At the beginning of the 20th century, psychologist Ivan Pavlov developed the so-called stimulus–

11Ontheconceptofanaffordance,seechapterfiveinGibson(1986).Ontheconceptofacapability,seeSen(1999)andNussbaum(2011).Bothconceptsemphasizetherelationalnatureofanindividualpersonandherenvironment.



responsetheoryonlearningmechanisms(Pavlov,1927).Hebasedhistheoryonaseriesofexperimentswithadog,allegedlydemonstratingthatbehaviouristheresultofarecurrentchainofstimuli(sensedobjectsand/orbehaviours, suchas foodor signalsassociatedwithprovisionof food).Thesechainsofstimuli supposedly trigger subsequent responses (actuated by means of specific behaviours, such asmanipulatinga lever,orproducing saliva inanticipationof food). Interestingly,Pavlovwasoneof thefirst behaviourists, claiming to base his theoretical insights on observable behaviours without anyreference to the human or animalmind,which he treated as a black box. This supposedly deliveredobjective, scientific fact-finding instead of the assumedly subjective theories of psychologists such asWilhelm Wundt (Kim, 2014), who based their inquiries into the human mind on a methodology ofintrospection— aiming to observe what goes on in the mind of an individual person and trying togeneralize fromthere.Oneof theassumptionsofPavlov’s theorywas the idea thatbehaviourcanbeexplained bymeans of physiological reflexes that can be tested in an experimental setting. Becausequiteafewbehaviouralresponsesseemtodependonadaptationtotheenvironmentinsteadofinnatereflexes,Pavlovdevelopedexperimentsto“prove”thatanimalresponsescanbetheresultofacquiredreflexes,basedonaphysiologicalassociationofspecificevents(theringingofabell)withotherspecificevents (the provision of food). Pavlov’s theorywas further developed byWatson (1930) and Skinner(1976),andisactuallycloselyconnectedwithTuring’s(1950)thoughtexperimentandDennett’s(1989)“intentional stance.” Both Turing andDennett avoid the issue of consciousness, focusing onwhetherobservable behaviour can help us to anticipate an organism’s — or even a person’s — next move.Watson(1930),forinstance,suggestedthat:

Theinterestofthebehavioristinman’sdoingsismorethantheinterestofthespectator—hewantstocontrolman’sreactionsasphysicalscientistswanttocontrolandmanipulateothernaturalphenomena.Itisthebusinessofbehavioristicpsychologytobeabletopredictandcontrolhumanactivity.Todothisitmustgatherscientificdatabyscientificmethods.Onlythencanthetrainedbehaviouristpredict,giventhestimulus,whatreactionwilltakeplace;or,giventhereaction,statewhatthesituationorstimulusisthathascausedthereaction.

Clearly, behaviourism has strong links with determinism and physicalism, assuming that mind isultimately amatter ofmatter, while in fact themindmay not reallymatter oncewe can detect themechanisms that rule behaviour, which is assumed to be subject to and defined by the laws ofcausality.12

ContemporaryheirsofthebehaviouristapproachcanbefoundinPentland’s(2014)“socialphysics,”andHelbing’s (2012) “computational social science.” Their approaches to social science are akin to socialengineering,aimingtounlockthetroveofknowledgethatcanbeminedfromdata-drivenapplications(Pentland) and/or computational simulations of human society by means of multi-agent systems(Helbing).Bothseek thekey tounderstand individualsasnodes inanetwork,hoping tocontribute tothe engineering of a fair and sustainable society. The idea is that if we know— statistically— howpeoplelearn(howtheydevelopwhichconditionalreflexes),wecanreconfiguretheirenvironment(thesetofrelevantstimuli)tomakesuretheyindeedlearnwhat“we”thinktheyshould learn(intermsof

12Ontheideathatthemindisanaccidentalside-effectofhumanintelligence,seeRobinson(2015).



behaviour).Thequestioniswhois“we”andwhetherwereallywanttomanipulatepeopleintolearningwhat “we” think is best (for whom?). Though there may be noble intent behind such societalengineering,itentailstheassumptionthatwe“learnasamachine,”andthiseasilyleadstoattemptstomanipulateusintosupposedly“good”behavioursasifweweremindlesspawnsinagameofchess(orevenGo)(Thaler&Sunstein,2008;Hausman&Welch,2010).Suchaviewmissesoutonthecomplexityofsocietalnetworks,theplasticityofthehumanbrain,andthevulnerabilityaswellasthecreativityofbothindividualhumansandtheirsociety.Thisreferstotheintractabilityofhumanlearning,andsomemight suggest that increasing computational power will push the threshold of this intractability. Butthere is also the question of computability.13 Our life world is built on the ambiguity of “natural”language that is generativeofahighlydynamicwebofmeaning.Thecuriousamalgamofuncertaintyand creativity that springs from our language-saturated environment iswhat holds us together (as apeople)andwhat setsusapart (fromotheranimals).Amechanisticunderstandingofhuman learningthat ignores our capabilities for generating meaning, and for giving reasons for our actions, has anumberofhighlyproblematicethicalandepistemological implications.Thismayberelatedtoamovefromteaching students, that isputtingnewknowledge in frontof them, sharingacquiredknowledge,explainingthereasonsandcausalconnectionsforsuchknowledge,toinducinglearningprocesseswithstudents, that is manipulating their environment in a way that it induces them to acquire specificinsights. One need not be radical to guess that this will corrupt their autonomy, their capability toreason and tomakeup their ownmind aboutwhat to learnhow (Slade&Prinsloo, 2013;Dalsgaard,2006).

3.2 The Simon Approach: Optimal Optimization

Inthesecondhalfofthe20thcentury,HerbertSimon(1983)wroteanintriguingarticleunderthetitle“Why should machines learn?” This was the time when machine learning was hardly a success andnotably “nervenet learning,” ashe calls it, achievednext tonothing. Simon (oneof thepatriarchsofartificial intelligence),however,makesanumberofhighlyrelevantobservationsaboutthedifferencesbetweenhumanandmachinelearning.Oneofhismainpointsreferstowhathecalls“thetediousnessof human learning,” as this is a slow process that takes a lot of time. To achieve a certain level ofknowledge,peoplespendmanyyearsinschool,whereasonceacomputerhasbeenprogrammedwithspecificknowledgethiscaneasilybetransferredfromonecomputertothenext—nointerantlearningprocessisneeded:

[W]eshouldaskwhetherwereallywanttomakethecomputergothroughthattediousprocess,orwhethermachinesshouldbeprogrammeddirectlytoperformtasksavoidinghumanoidlearningentirely!

Onlyonecomputerwouldhavetolearn;noteveryonewouldhavetogotoschool.

Wemay smile about thenaïvetéof theobservation Simon seems to endorse, but I guess Simonwas

13Oncomputability,seeBrey&Søraker(2009).Onintractability,seeDean(2015).



using the rhetorical deviceof a tongue-in-cheek,making thepoint that such an approachwouldonlywork if the goal of learning were a matter of “doing the same task over and over again, but moreefficiently.”If,however,thegoaloflearningisto“acquiretheabilitytoperformawiderangeoftasks,”or if learning refers to discovering new knowledge and finding away to retain it, the tediousness ofhuman learningmaybe thebestway tooptimize theprocess.According to Simon, this is indeed thecase, because programs facing real-life problems are highly complex and suffer numerous bugs andcomputationalartefacts.Thisresultsinthemendingupasbeinghighlyinefficientandineffectiveinthecourse of time. So, he says, instead of putting all hope in one program, human society consists ofnumerousindependentprogramsthatallowfornewbeginnings:

Oldprogramsdonotlearn,theysimplyfadeaway.Sodohumanbeings,theirundebuggableprogramsreplacedbyyounger,possiblylesstangledonesinotherhumanheads.

So, thedeath and thebirthof individual humans serves to resolve the legacyproblem for optimizinglearningprograms.Simon’sironicmessageisthatthefactthatmyknowledgeandinsightscanonlybetransferred to others by means of great effort on both sides is not a disadvantage but rather anincrediblysmartwaytoensurecreativeandrobustadaptation.Anyonewhoclaimstohaveinventedorengineered the optimal learningmachine for a particular set of problemsmust be put to the test bymeansofretrialsandalternativemachines.WecouldframethisintermsofWolpert’sfamous“nofreelunch theorem” that provesmathematically that specific optimizationmechanisms that “work” for aspecific type of problem do not necessarilywork for other types of problems (Wolpert &Macready,1997).

3.3 The Gibson Approach: The Ecological Niche

James Gibson has developed one of the most crucial concepts for our current age, that of “anaffordance.” It is firmlyembedded inevolutionarytheory,physicalreality,psychological inquiry,andadeep understanding of the relationship between a living entity and its environment. It skips naïveattemptstodividetheworldintomindandmatter,withoutdiscardingvaluesormentalcapabilities.Letmequotehisowndefinitionofanaffordance:

Theaffordancesoftheenvironmentarewhatitofferstheanimal,whatitprovidesorfurnishes,eitherforgoodorill.Theverbtoaffordisfoundinthedictionary,butthenounaffordanceisnot.Ihavemadeitup.Imeanbyitsomethingthatreferstoboththeenvironmentandtheanimalinawaythatnoexistingtermdoes.Itimpliesthecomplementarityoftheanimalandtheenvironment.(Gibson,1986)

When speaking of an affordance, Gibson is not interested in separating objective physical propertiesfromthesubjectiveexperienceoftheperceivingorganism.Thoughthis isofcoursepossible,Gibsonisinterestedintheactionable“properties”ofanenvironment,whichdependontheagentthatdependsontheenvironment.Eachtypeofagentthushasitsownecologicalniche:thesetofaffordancestunedto a specific type of agent that is in turn tuned to its ecological niche. For a human being, this forinstance impliesbreathableair,walkablesurfaces,graspablethings,butalsootherhumanscapableofspeaking or learning her language. For humans, affordances are— in terms of Pavlov— the set of



stimulithatenableandconstrainthemtobehaveonewayoranother,includingenablingthemtolearnnewbehavioursoreventounlearninnatebehaviours.OtherthanPavlov’sdeterminism,Gibson’skeenattentiontothemutualshapingofanagentanditsecologicalnichecelebratescreativityandrenouncestheideathateithertheagentorthenicheisdefinedbytheircurrentproperties.IntermsofSimon,wecanadd,machines similarlydependonaparticularenvironment thataffords themto learnone thingand not another. As we know, many machines require an artificial environment geared to theirbehavioural potential, while also preventing them from causing harm to their human masters. Inrobotics this is called theenvelop (Floridi,2014),acontrolledenvironment thatmakessure the robotfinds its way and is constraint in ways that enable its productivity. Since environments are sets ofaffordances, theyarecontingentuponthetypeoforganismable toperceiveandactuponthem.Fishdon’tdowellonland;humansdon’tdowellunderwaterorhighupintheair;surfacesthatcansustainants may not sustain elephants. An ecological niche, therefore, is not the habitat of a particularorganism,buttheactionabilityoftheenvironmentofthatorganism.

Humans have managed to reconfigure their ecological niche in myriad ways — with far-reachingconsequencesforotherorganisms,asweallknow.Majortransitionsoftheaffordancesofourecologicalnichehavebeengeneratedbytheintroductionoftoolstosupportcalculationandwriting,includingtheprinting press. Each of them afforded entirely novel learningmechanisms, including the retention ofwhat has been learnt outside the body of an individual human (e.g., on a piece of paper). This hastriggeredtheestablishmentofformallearningandteachinginstitutionssuchasschoolsanduniversities,aswellasarchivesand librariesandotherrepositoriesof retainedknowledge,suchascomputersandservers.

What fascinatesme ishowGibson’sapproachenablesus to raise thequestionofhoweLearningandother types of LAwill transform the affordances of our learning institutions, changing our ecologicalnicheandthusalsoourselves.Whereas,sofar,theotheragentsinournichewerelivingorganismssuchasanimals, fromcattle topets,wearenow increasingly confrontedwithmindless agentsofourownmaking,capableofobservingusandadaptingtheirbehaviourtothefeedbacktheygainfromours.Itisnotmerelythatweusethesemachines,forinstancetohelpustolearnfasterormoreefficiently,theyalsouseustoimprovetheirperformance.And,finally,wemaycometothepointofacknowledgingthatweareactuallyinteractingwiththem—e.g.,teachingthemhowtomakeuslearn.Trainingalgorithmsona training setandchecking theoutcomeon the test set isalreadya typeof interaction,especiallywhereknowingthepreciseoperationsofthesoftwareisnotfeasible(aswithneuralnets)orirrelevantfor the outcome. When I teach a human person, I cannot access her brain and follow the neuraloperations— they would certainly be incomprehensible to me. But this has never stopped us fromteaching or otherwise interacting with another — instead, it keeps us on our toes because of theuncertainty it implies and the creativity it requires. The same seems to go for teaching a neural net.However,otherthanlivingbeings,cyber-physicalsystemsthatintegrateMLhavenothingtolose;theycannotsufferorfeartheirowndeath.Theycanprobablysimulateallthatandmore,basedonsyntheticemotionsandaffectivecomputing;butsimulation isnotthesameaswhat issimulated.Totheextentthatourlearningenvironmentsbecomedependentonthesesystems,thismaybecauseforconcern.



What should concern us here is the fact that in order to interact with machines, people need toanticipate how thesemachines learn (Christian, 2011). This goes for those that develop the LA, e.g.,whentheytrainthealgorithmsforeLearning,orwritethecodeforadministrativedecision-making.Butitalsogoesforthestudents,e.g.,whentheyinteractwitheLearningsystemsthatcanonlyrespondtotheir machine-readable behaviours, or when they figure out which of their behaviours is taken intoaccount when decisions are made on their eligibility for grants, admission to selected courses, oreducation institutions. As a result, our ecological niche may increasingly be shaped in a way thataccommodates“learningasamachine”byhumans.Thisispartlyduetotheunfathomableplasticityofthehumanbrain.Thoughwearenotbornwithareadingbrainwesomehowmanagetoreconfiguretheorganization,themorphology,andthebehaviourofourbrainstoaffordreading,asthecombinationofexperimentalpsychologyandbrainsciencehasdemonstrated(Wolf&Stoodley,2008).Oncewelearnedtoreadandwrite,ourecologicalnichewashugelyextendedwiththeaffordancesofwrittenandprintedtext. There is no reason to think that “ordinary people” cannot develop a brain more geared tomathematics, computation, and calculation if—asa society—wedecide tomake theeffort (Sousa,2014). This might indeed fast-track their interactions with LA systems, as they gain a more intuitiveunderstanding of how thesemachines operate or even “think.” But it may also have transformativeeffectsonwhatitmeanstobehuman.

4 PRIVACY, NON-DISCRIMINATION, DUE PROCESS, AND THE PRESUMPTION OF INNOCENCE

4.1 First Level Issues: Identification and Targeting

Asdiscussedabove,thefirstlevelconcernsinterventionswithidentifiableindividuallearners.Itisaboutcollectionoftheirpersonaldata,whetherfromtheiradmissionformsorBlackboard,fromteacherinputsuchasgrades,orfromeLearningapplicationsthatcapturebehaviouraldata(orevenfromdatatrovesgathered outside the educational setting, such as social networks and data brokers). It is also aboutapplyinginferencesbasedonLAtoanindividualstudent,whethersheisawareofthisornot.Thismayinvolveautomatedorsemi-automateddecision-systemsthatcategorizestudentsandconditionaccesstoextrafacilities,counselling,rewards,orgrants.ResultsofLAmayevenconditionsharingdatawiththepoliceorintelligenceagencies.Iamspeakingofaffordances,not—yet—aboutwhetherthisisethicalorlegal.

Atthislevel,privacymaybeconcernedwheneverdataisprocessed(forinstance,shared)inawaythatviolateslegitimateexpectations,notablywhendataissharedoutofcontext(Nissenbaum,2010).Thisisatstakewhenaccesstothedatawithinthelearninginstitutionisnotappropriatelysecured(unreliableauthentication),whichresultsinteachingandotherstaffbeingabletolookuphowaparticularstudentisdoing—withoutanynecessity.ThisisalsoatstakeifpublishersofelectronictextbooksoreLearningsoftwaregainaccesstothedataofindividualstudentstoimprovethefunctionalityoftheirsoftware,orifgrantprovidersdemandanabundanceofpersonaldatatoconductfrauddetection.AsKhaliahBarnes,director of the student privacy project at the Electronic Privacy Information Center notes, “Rampantdatacollection isnotonlydestroyingstudentprivacy, italsothreatensstudents’ intellectual freedom”



(Barnes, 2014; Drachsler et al., 2016). The tax authority, the police, or the intelligence servicesmaydemandaccess—probablyaccompaniedbyaprohibitiontocommunicatethisaccesswiththerelevantpersonoranyoneelse.Thisimpliesthatundetectedprivacyinfringementscouldeasilyoccur,dependingonthewhimsofwhoeverisincharge.Unlessasystemofsafeguardsisbuiltintothearchitectureofthetechnicalsystemsandtheorganizationaldesign,privacywillbeupforgrabs.Thepointhereisnotbadintentions,butafargreaterthreat:thetransformationoftheaffordancesoflearninginstitutions.

When LA inferences are applied to an individual based on knowledge that is not sharedwith her, orwhen such applications confront her with things about herself that she was not aware of, this mayamounttoanothertypeofprivacyviolation(Hudson,2005). Inferencesmayindicatehealthproblems,notably concerningmental health, or theymay show that studentswith a certain religious or ethnicbackgroundlackspecificcopingstrategies,resultinginfailuretofinishthecourseorobtaintherelevantdegree.Thisisnotonlyaboutprivacy,butalsoaboutnon-discrimination.Itmaybegreattoknowthatpeople from a certain background (ethnic, religious, gender, economic, geographic) have specificlearningdisabilitiesthatcanberemediedbymeansofspecificinterventions,basedonextensiveABtestdesigns.14Thatimplies,however,thatlotsofstudentswillbeusedasguineapigs,whiletheirsensitivedataareemployedforscientificresearch,tosecurethepolicygoalsoftheeducationalinstitution,or—moremundane—toincreasetheprofitsofaserviceprovider.

Totheextentthatstudentsarenotawareofallthis,theirdueprocessrightsmaybeatstake(Citron&Pasquale,2014).Theycouldbeplacedincertaincategoriesbasedonclusteringtechniquesthatresultinclassifiers that discriminate (which is not necessarily prohibited) based on certain features that their“nearestneighbours”have,thoughtheymaynothavethesefeatures(theprofilesinferredwilloftenbenon-distributive).15 To the extent that they arenot awareof this, they cannot object. Indeed, even iftheyknewaboutthis,theymighthaveaproblemarguingtheircase,becausethecategorizationmaybepresentedasobjectiveandapplicableandcouldonlybecounteredbymeansofsoftwareverificationortrainingalternativealgorithmsonthesamedataset.Anotherproblemisthattheargumentsavailabletoobject to theirbeingprofiledmaybe restrictedbecauseobjectionsmustbesubmittedonlineandareformatted in a specificway. This probably entails that arguments that don’t fit the format cannot beregistered. Finally, those involved in dealing with the appeals may have formatted responses, e.g.,statingthatthesystemdoesnotallowtheargumentation.Thisisalreadyoftenthecasewhenobjectingto fines from the taxation office or complaining about telecom providers. “Computer says no” is apopularphraseforgoodreason(Dean,2016).This,finally,touchesuponthepresumptionofinnocence.In administrative law, unlike in criminal law, it is assumed that the government got it right and thecitizenneedstoproveherinnocence.So,thestudentmayhavetoprovethatthesystemgotitwrong,intermsdictatedby thesystem.Kafka isaround thecorner—oramongstus (Solove,2004).This isnotsciencefictionanditisnotevenaboutbadintentions;itisabouttheconsequencesofenteringanotherecologicalnichewithadifferentsetofaffordances.

14ForanearlyprecursorofABresearchdesigninaneducationalsetting,seeFoster,Watson,Meeks,andYoung(2002).15Onnon-distributiveprofiles,seeVedder(1999).



4.2 Second Level Issues: Analytics and Prediction

Let’snowchecksecondlevelissues.Theseareentirelydifferent,becausetheinferencesdrawnneednotbe personal data. Instead they will be patterns in the data set, linking specific attributes (features,behaviours, background, context) to specific performance metrics to create hindsight (what kind ofattributes correspond with what level of performance?), foresight (what can be expected ceterisparibus?) and insight (which interventions could improve future student performance, depending onwhatcircumstances?).Theissueusuallyhighlightedatthislevelisprivacyinthenarrowsenseofhidingpersonaldataand/ormakingsurethatthedataisprotectedagainstunauthorizeddisclosure.Therearenowanumberoftechniquesmeanttoenablebigdataanalyticsincompliancewithdataprotectionlaw,basedonpseudonymization(asirreversibleanonymizationisusuallynotanoption).FrameworkssuchasMITOpenPDSwithitsSafeAnswersmodule,16MaxPlanck’sAIRCLOAKwithitsInsightsAQR,17andtheso-called PEP framework (polymorphous encryption & pseudonymization) developed at RadboudUniversity (Verheul,2015;Verheul, Jacobs,Meijer,Hildebrandt,&Ruiter,2016),providesophisticatedways of combining encryption with pseudonymous sharing or enable running code on the raw datawithout ever sharing the data (zero knowledge protocols). These are highly relevant and provideimportantmeansofengagingindataprotectionbydefault(dataminimization).Theydonot,however,go to the heart of thematter.Once a service provider, grant provider, insurance company, potentialemployer,oreducationalinstitutiongetsaholdoftheresultsoftheanalytics,anewsetofaffordancescomesintoplay.

Asdiscussedunder first level issues, as soonas the inferences are applied to individual students, theinfringementsofprivacy,non-discrimination,dueprocess,andthepresumptionofinnocencereturnatfullspeed.ThatiswhyIdevotethefinalpartofthisarticletotheimplicationsofLAforthefundamentalcore of human learning, i.e., our ability to reflect on our own behaviours. This “ability” may be anaffordanceofwrittentextthatisnotnecessarilyanaffordanceofLA.18

5 PROFILING AND DATA PROTECTION BY DESIGN

5.1 The Relevance of the GDPR for First Level LA: Data Minimization

TheupcomingGeneralDataProtectionRegulationconsolidatesandreinforcesthecoreprinciplesofthecurrent Data Protection Directive. Nevertheless, it will be a game-changer for those who processpersonaldata—notablyforthosewhoconductadditionalprocessing,engaginginsecondaryusage,andre-purposingofpersonaldata.ComparedtotheDPD,theGDPRwill“do”threethings (amongstmanyothers). First, itwill create a level playing field for stakeholders that base their business case on theprocessingofpersonaldata.This levelplaying field is theresultofdeterrence:high fines,databreachnotification, transparency requirements with high impact on reputation, and tort liability.19 This will

16http://openpds.media.mit.edu/#architecture.17https://www.aircloak.com/downloads/Aircloak-One-Pager.pdf.18ThishasbeenextensivelyarguedbyHildebrandt(2015).19Art.83GDPRstipulatesthatsupervisoryauthoritiescanpunishviolationsoftheGDPRwithfinesofupto4%ofglobal



institute countervailing powers capable of mitigating some of the problematic affordances of LA —speakinglawtopower.Intheend,thislevelplayingfieldwillenablecompaniestoactethicallywithoutbeingpushedoutofthemarket.Second, itwillrequireadataprotectionimpactassessmentwheneverhigh risk is expected for the rights and freedoms of natural persons (DPIA, art. 35 GDPR), which isrelatedtotheobligationtoimplementdataprotectionbydefaultandbydesign(DPbD,art.25GDPR).The latter requires that data minimization and other legal obligations be embedded into thearchitecture of data processing systems. These legal obligationswill reconstruct the ecological niche;theywillreconfigurethesetofaffordancesthatemergeinthewakeoflarge-scaleapplicationofLA.Thisis notmerely speaking law topower,but architecting trustworthy infrastructure that reduces risks toprivacy andother fundamental rights by default, basedon the technical specifications of LA systems.Third, it will privilege the processing of pseudonymized data— subject to stringent conditions— toenable big data analysis.20 In combinationwith the previous points (creating a level playing field andintegrating data protection into the hardwired and softwared LA systems), this will provide for anincentive structure that favours systems thatenable secondaryuse foranumberof specificpurposeswhilereducingtheriskofidentification.Again,thiswillhaveatransformativeimpactontheaffordancesofLA,asitwillsimultaneouslyconstraintheuseofdatathatenablesdirectidentification,whileenablingbig data analytics on pseudonymized data. An excellent example of this type of Data Protection byDesignandDefaultcanbefoundintheDELICATEchecklist,developedbyDrachslerandGreller(2016),highlightingtheneedtodevelopasustainableframeworkforsafeandagileLAthatsafeguardsthedataprotectionrightsoflearners.

5.2 The Relevance of the GDPR for Second Level LA: Profile Transparency

DPbD, however, also applies to the obligation of profile transparency, which is already part of thecurrentlegalframeworkbutfurtherclarifiedintheGDPR.Idaresaythatprofiletransparencyistheonlylegalconstraintthatdirectlytargetstheissueofdueprocessandthepresumptionof innocenceinthecaseofdata-drivenapplicationsthatinvolvepersonaldata.OnecouldsaythatprofiletransparencyisaformofwhatCitronandPasquale (2014) termed“technologicaldueprocess” in theirgroundbreakingarticleonthe“scoredsociety.”21

Profiling refers to thepatterns that result fromMLoperations,notably theanalysisorpredictionofaperson’s“performanceatwork,economicsituation,health,personalpreferences, interests, reliability,behaviour, location or movements” (art. 4.4 GDPR).22 Profile transparency requires providinginformationabout three typesof information:1) theexistenceofprofiling,2)meaningful information

turnover.Art.33GDPRimposesadutytonotifythesupervisoryauthoritywithin72hoursofadatabreach,andart.34GDPRimposesasimilardutytonotifythedatasubjectwhosedatahavebeenleaked,“[w]henthepersonaldatabreachislikelytoresultinahighrisktotherightsandfreedomsofnaturalpersons.”20Art.6.4.eGDPRmitigatesthedutytocomplywithpurposelimitationinthecaseofre-useandre-purposing,qualifyingpseudonymizationasasafeguardthatmaycontributetolegitimizeadditionalprocessing.Art.25(DPbD)qualifiespseudonymizationasanappropriateorganizationalandtechnicalmeasuretocomplywithdataminimization.Art.32(securityofpersonaldata)qualifiespseudonymizationasaformofsecuritybydesign.21Seealso“proceduraldatadueprocess”inCrawfordandSchultz(2014). 22Stillhighlyrelevant:HildebrandtandGutwirth(2008),co-authoredbycomputerscientists,sociologists,andlawyers.



aboutthe logic involved,and3) informationabouttheenvisagedconsequences (art.13.2.fand14.2.gGDPR,12.aDPD).Enablingindividualstoexercisethisrightwillbetherealchallenge,asitimpliestakinglearnersseriously,engagingthemontheirownterms,andhelpingthemtocriticallyassessthelearningalgorithms that define their progress (Hildebrandt, 2012). This is crucial in the case of automateddecision-making,especiallywhensuchdecisionshavelegaleffect(failorpassanexam,beadmittedtoalearning institutionor a jobposition) or similarly significantly affect the concernedperson. TheGDPRprovidesdatasubjectswitharightnottobesubjecttosuchdecisions(art.22.1),andprohibitsthemiftheyarebasedonso-calledsensitivedata(ethnicity,religion,etc.;art.22.4).Hereweseehowandwhythe fundamental right to data protection is not restricted to privacy but explicitly addresses non-discrimination,dueprocess,andeventhepresumptionofinnocenceinitsbroadestsense.Eventhoughexceptions apply, it should be clear that the GDPR is ready to address the harms that result fromunwarranted automated application of LA. Learning institutions and App developers that work tointegrate LA into thewaywe learn should take to heart that human learning is not in the first placeabout the management of learning behaviours, but about one of the most salient characteristics ofbeinghuman.Thechallengewillbetomakesurethatlearnersaretreatedasindividualpersonsworthyof equal respect and concern, by creating systems that afford capabilities such as intellectualindependence, critical distance, and human flourishing. Having the right not to be subject to theoutcomeoffullyautomatedLAdecisionmachinesmay,therefore,bethemostimportantfundamentalrightintheeraofdata-driveneducation.

6 CONCLUSION

HumanlearningcannotbereducedtoPavlov’sstimulus–responsehypothesis.Evenanimalsaredrivencrazy by his experiments that disrespect their animal dignity,while the limited set of responses thatdefines their laboratory situation says little about the repertoire they develop in their “normal”ecologicalniche(Buytendijk&Plessner,1936;deWaal,2016).Simonnicelydemonstratedthebenefitofindividuallearningprocesses,clarifyinghowthetediousnessofhumanlearningmaybetheoutcomeofan optimization strategy in view of the unfathomable complexity of human society. Taking Gibsonseriously,weneedtoacknowledgethatthiscomplexityisconnectedwithourlanguagecapabilitiesandthe institutionaldynamicstheyrequireandproduce.Ourecologicalnichehasmyriadaffordancesthatare continuously renegotiated and attuned to the capabilities they afford while they — in turn —reshapetheniche.

Core to human learning are creativity, humour, and reflection, corresponding with art and ethics,judgement,politics,andlaw.Creativityandhumourcombine,detectingtheunexpectedwiththeneedto faceboth life anddeath.Reflectionentails anexternalizedawarenessofwhatwe thinkwe learnt,instead of succumbing entirely to unconscious learning processes. That is why teaching will remaincrucialtodemocracy.Teachingreferstoexplicitpresentationandexplanationofknowledge,suchthatitcanbetheobjectofdebateanddiscussion.LAmaybeagreatwayto induce learningprocesses,as itwere, behind a person’s back and, although this is not necessarily wrong or bad, LA must be (re-)configured in a way that allows for critical reflection on what and on how we learn. Therefore,



pseudonymization, encryption, and discrimination-aware data mining must be complemented withprofiletransparencyandtheeffectivemeanstoobjecttobeingprofiledasaspecifictypeoflearner.ThisiswhatUS legal scholarscall “technologicaldueprocess”;asEuropeans,weshouldstart theconcreteimplementationofprofiletransparencyandtherighttoobjecttoautomateddecision-makingascutting-edgeandpertinentelementsoftheRuleofLaw.

REFERENCES

Baker,R.S.,&Inventado,P.S.(2014).Educationaldataminingandlearninganalytics.InJ.A.Larusson&B.White,(Eds.),Learninganalytics:Fromresearchtopractice(pp.61–75).SpringerNewYork.http://dx.doi.org/10.1007/978-1-4614-3305-7_4

Barnes,K.(2014).Bigdataintheclassroomisoutofcontrol.TheNewYorkTimes,19December2014.Retrievedfromhttp://www.nytimes.com/roomfordebate/2014/09/24/protecting-student-privacy-in-online-learning/student-data-collection-is-out-of-control

Berg,A.M.,Mol,S.T.,Kismihók,G.,&Sclater,N.(2016).Theroleofareferencesyntheticdatageneratorwithinthefieldoflearninganalytics.JournalofLearningAnalytics,3(1),107–128.http://dx.doi.org/10.18608/jla.2016.31.7

Brey,P.,&Søraker,J.H.(2009).Philosophyofcomputingandinformationtechnology.InA.Meijers(Ed.)PhilosophyofTechnologyandEngineeringSciences,Vol.14.D.Gabbay,P.Thagard,&J.Woods(Gen.Eds.)TheHandbookforPhilosophyofScience.Elsevier.https://dx.doi.org/10.1016/B978-0-444-51667-1.50051-3

Buytendijk,F.J.,&Plessner,H.(1936).DiephysiologischeErklärungdesVerhaltens.ActaBiotheoretica,1(3),151–172.http://dx.doi.org/10.1007/BF02147637

Campbell,S.,&Dawson,A.J.(1995).Learningasembodiedaction.InR.Sutherland&J.Mason(Eds.),Exploitingmentalimagerywithcomputersinmathematicseducation(pp.233–249).SpringerBerlinHeidelberg.http://dx.doi.org/10.1007/978-3-642-57771-0_16

Christian,B.(2011).Themosthumanhuman:Whattalkingwithcomputersteachesusaboutwhatitmeanstobealive.NewYork:Doubleday.

Citron,D.K.,&Pasquale,F.A.(2014).Thescoredsociety:Dueprocessforautomatedpredictions.WashingtonLawReview,89(1),1–33.

Clark,R.C.,&Mayer,R.E.(2011).E-Learningandthescienceofinstruction:Provenguidelinesforconsumersanddesignersofmultimedialearning.SanFrancisco,CA:Pffeifer.

Crawford,K.,&Schultz,J.(2014).Bigdataanddueprocess:Towardaframeworktoredresspredictiveprivacyharms.BostonCollegeLawReview,55,93–128.

Dalsgaard,C.(2006).Socialsoftware:E-learningbeyondlearningmanagementsystems.EuropeanJournalofOpen,DistanceandE-Learning,9(2).Retrievedfromhttp://www.eurodl.org/materials/contrib/2006/Christian_Dalsgaard.htm

deWaal,F.(2016).Arewesmartenoughtoknowhowsmartanimalsare?NewYork:W.W.Norton&Co.

Dean,W.(2015).Computationalcomplexitytheory.InE.N.Zalta(Ed.),TheStanfordEncyclopediaofPhilosophy.http://plato.stanford.edu/archives/fall2015/entries/computational-complexity/



Dean,J.(2016).Computersaysno:Gochampionlosesinmanvmachine.TheTimes(London),16March2016.Retrievedfromhttp://www.thetimes.co.uk/tto/technology/article4709094.ece

Dennett,D.C.(1989).Theintentionalstance.Cambridge,UK:BradfordBook.Drachsler,H.,Hoel,T.,Cooper,A.,Kismihók,G.,Berg,A.,Scheffel,M.,Chen,W.,&Ferguson,R.(2016).

Ethicalandprivacyissuesinthedesignoflearninganalyticsapplications.Proceedingsofthe6thInternationalConferenceonLearningAnalyticsandKnowledge(LAKʼ16),25–29April2016,Edinburgh,UK(pp.492–493).NewYork:ACM.https://dx.doi.org/10.1145/2883851.2883933

Drachsler,H.,&Greller,W.(2016).Privacyandanalytics:It’saDELICATEissueachecklistfortrustedlearninganalytics.Proceedingsofthe6thInternationalConferenceonLearningAnalyticsandKnowledge(LAKʼ16),25–29April2016,Edinburgh,UK(pp.89–98).NewYork:ACM.https://dx.doi.org/10.1145/2883851.2883893

Dworkin,R.(1991).Law’sEmpire.Glasgow,UK:Fontana.Fiaidhi,J.(2014).Thenextstepforlearninganalytics.ITProfessional,16(5),4–8.

https://dx.doi.org/10.1109/MITP.2014.78Floridi,L.(2014).Future/thegreengambit.Thefourthrevolution:Howtheinfosphereisreshaping

humanreality,Chapter7.Oxford,UK:OxfordUniversityPress.Foster,L.H.,Watson,T.S.,Meeks,C.,&Young,J.S.(2002).Single-subjectresearchdesignforschool

counselors:Becominganappliedresearcher.ProfessionalSchoolCounseling,6(2),146–154.Gibson,J.J.(1986).Theecologicalapproachtovisualperception.Mahwah,NJ:LawrenceErlbaum.Greller,W.,&Drachsler,H.(2012).Translatinglearningintonumbers:Agenericframeworkforlearning

analytics.EducationalTechnology&Society,15(3),42–57.Hausman,D.M.,&Welch,B.(2010).Debate:Tonudgeornottonudge.JournalofPolitical

Philosophy,18(1),123–136.http://dx.doi.org/10.1111/j.1467-9760.2009.00351.xHelbing,D.(2012).Socialself-organization:Agent-basedsimulationsandexperimentstostudyemergent

socialbehavior.SpringerBerlinHeidelberg.Hildebrandt,M.(2012).Thedawnofacriticaltransparencyrightfortheprofilingera.Digital

EnlightenmentYearbook2012,41–56.Hildebrandt,M.(2015).Smarttechnologiesandtheend(s)oflaw:Novelentanglementsoflawand

technology.Cheltenham,UK:EdwardElgarPublishing.Hildebrandt,M.,&Gutwirth,S.(2008).ProfilingtheEuropeancitizen:Cross-disciplinaryperspectives.

Dordrecht:Springer.Hudson,B.(2005).Secretsofself:Punishmentandtherighttoprivacy.InE.Claes&A.Duff(Eds.),

Privacyandthecriminallaw(pp.137–162).Antwerp/Oxford:Intersentia.Jarvis,P.(2005).Towardsacomprehensivetheoryofhumanlearning.London:Routledge.Kalampokis,E.,Tambouris,E.,&Tarabanis,K.(2013).Understandingthepredictivepowerofsocial

media.InternetResearch,23(5),544–559.http://dx.doi.org/10.1108/IntR-06-2012-0114Kim,A.(2014).WilhelmMaximilianWundt.InEN.Zalta(Ed.),TheStanfordEncyclopediaofPhilosophy.

http://plato.stanford.edu/archives/win2014/entries/wilhelm-wundt/Leijnen,S.(2014).Creativityandconstraintinartificialsystems.http://www.leijnen.org/McStay,A.(2011).Themoodofinformation:Acritiqueofonlinebehaviouraladvertising.NewYork:



Continuum.Mitchell,T.M.(2006).Thedisciplineofmachinelearning(Vol.9).CarnegieMellonUniversity,Schoolof

ComputerScience,MachineLearningDepartment.http://www-cgi.cs.cmu.edu/~tom/pubs/MachineLearningTR.pdf

Nissenbaum,H.(2010).Privacyincontext:Technology,policy,andtheintegrityofsociallife.StanfordLawBooks.

Nussbaum,M.C.(2011).Creatingcapabilities.HarvardUniversityPress.Pavlov,I.P.(1927).Conditionedreflexes:Aninvestigationofthephysiologicalactivityofthecerebral

cortex.TranslatedandeditedbyG.V.Anrep.Oxford,UK:OxfordUniversityPress.Pentland,A.(2014).Socialphysics:Howgoodideasspread:Thelessonsfromanewscience.NewYork:

PenguinPress.Piatetsky-Shapiro,G.(1996).Advancesinknowledgediscoveryanddatamining(Vol.21).U.M.Fayyad,

P.Smyth,&R.Uthurusamy(Eds.).MenloPark,CA:AAAIPress.Prinsloo,P.,&Slade,S.(2016).Studentvulnerability,agencyandlearninganalytics:Anexploration.

JournalofLearningAnalytics,3(1),159–182.http://dx.doi.org/10.18608/jla.2016.31.10Robinson,W.(2015).Epiphenomenalism.InE.N.Zalta(Ed.),TheStanfordEncyclopediaofPhilosophy.

http://plato.stanford.edu/archives/fall2015/entries/epiphenomenalism/Sen,A.(1999).Commoditiesandcapabilities.NewYork:OxfordUniversityPress.Simon,H.A.(1983).Whyshouldmachineslearn?InR.S.Michalski,J.G.Carbonell,&T.M.Mitchell

(Eds.),Machinelearning(pp.25–37).SpringerBerlinHeidelberg.http://dx.doi.org/10.1007/978-3-662-12405-5_2

Skinner,B.F.(1976).Aboutbehaviorism.NewYork:Vintage.Slade,S.,&Prinsloo,P.(2013).Learninganalytics:Ethicalissuesanddilemmas.AmericanBehavioral

Scientist,57(10),1510–1529.https://dx.doi.org/10.1177/0002764213479366Solove,D.J.(2004).Thedigitalperson:Technologyandprivacyintheinformationage.NewYork

UniversityPress.Sousa,D.A.(Ed.).(2014).Mind,brain,&education:Neuroscienceimplicationsfortheclassroom.

Bloomington,IN:SolutionTreePress.Thaler,R.H.,&Sunstein,C.R.(2008).Nudge:Improvingdecisionsabouthealth,wealth,and

happiness.YaleUniversityPress.Turing,A.M.(1950).Computingmachineryandintelligence.Mind,59(236),433–460.Vedder,A.(1999).KDD:Thechallengetoindividualism.EthicsandInformationTechnology,1(4),275–

281.http://dx.doi.org/10.1023/A:1010016102284Vender,D.(2011).Isbalancingemblematicofaction?TwoorthreepointersfromReidand

Pierce.Humana:MenteJournalofPhilosophicalStudies,1(15),251–270.Verheul,E.R.(2015).Privacyprotectioninelectroniceducationbasedonpolymorphic

pseudonymization.IACRCryptologyePrintArchive,2015,1228.Verheul,E.R.,Jacobs,B.,Meijer,C.,Hildebrandt,M.,&deRuiter,J.(2016).Polymorphicencryptionand

pseudonymisationforpersonalisedhealthcare.IACRCryptologyePrintArchive,2016,411.Watson,J.B.(1930).Behaviorism.UniversityofChicagoPress.



Wolf,M.,&Stoodley,C.J.(2008).Proustandthesquid:Thestoryandscienceofthereadingbrain.Cambridge,UK:Icon.

Wolpert,D.H.,&Macready,W.G.(1997).Nofreelunchtheoremsforoptimization.IEEEtransactionsonevolutionarycomputation,1(1),67–82.https://dx.doi.org/10.1109/4235.585893

Documents

Learning as a Machine: Crossovers between Humans and Machines · Learning as a machine: Crossovers between humans and machines. ... , data protection by design, privacy, machine learning