Technology for automated assessment: The World-Wide-Mind

Technology for automated assessment:

The World-Wide-Mind

Ciarán O’Leary

Dublin Institute of Technology

22nd May 2003

Quick Overview

• Requirements– Large amounts of software online– Large number of participants in project– Diversity is key!

• Need to– Provide architecture and protocol that will

make this software available– Have a simple entry level – like the WWW

e-Learning Overview

• Reusability of components

• Potential for automated assessment

• Potential for comparisons with other students

Agenda

• World-Wide-Mind– Introduction to goal of project– Definition of a service– Available software

• Blocks World example• Usage in learning environment• Comparison with existing models • Possibilities for future development

Part I

World-Wide-Mind

• Dublin City University– Dr. Mark Humphrys

• Artificial Intelligence Project– Distributed– Artificial Minds

The Jolly Little Creature

Agent

The Jolly Little Creature’s Mind

• Multiple (massively diverse) modules• Since, according to faculty psychologists, the

mental causation of behaviour typically involves the simultaneous activity of a variety of distinct psychological mechanisms, the best research strategy would seem to be divide and conquer: first study the intrinsic characteristics of each of the presumed faculties, then study the ways in which they interact– Jerry Fodor – The Modularity of Mind, p1.

Building JLC’s Mind

• Divide and conquer• Required diversity through multiple

authors– The more authors the more diversity

• Make modules available online– Researchers– Amateurs– Students!

Building JLC’s Mind

• Requirements– Architecture – Standard/protocol to use to integrate

components

• Requirement for protocol– No barriers to entry– Simple– Not tied to any platform

Summary Part I

• World-Wide-Mind• Distributed Artificial Intelligent Project• Success dependent on participation of large

number of very different researchers– Require diversity

• Need an architecture and protocol for components to be plugged together

• Major goal is simplicity

Part II

JLC’s Mind

Mind

Perceive state

Select action

Body

State

Take action

World

State

New state

JLC’s Mind Online

Mind

Perceive state

Select action

Body

State

Take action

World

State

New state

Internet

World-Wide-Mind Architecture

World Server

World Server

Mind ServerMind Server

ClientClient

World-Wide-Mind Architecture

World Server

World Server

World Server

World Server

Mind ServerMind Server

Mind ServerMind Server

Mind ServerMind Server

Mind ServerMind Server

Mind ServerMind Server

Mind ServerMind Server

Mind ServerMind Server

ClientClient

Service Methods

World Service

• newrun• getstate• takeaction• endrun

Mind Service

• newrun• getaction• endrun

World-Wide-Mind Protocol

• “The term Web services describes a standardized way of integrating Web-based applications using the XML, SOAP, WSDL and UDDI open standards over an Internet protocol backbone. XML is used to tag the data, SOAP is used to transfer the data, WSDL is used for describing the services available and UDDI is used for listing what services are available”– http://www.webopaedia.com

• Society of Mind Markup Language (SOML)– Lightweight Web Services

SOML Message

1. <soml version="0.9">2. <request type="getaction" runid="123456">3. <param name="state">4. (0, 0, 0, 9, 7, 8, 2, 0, 4)5. </param>6. <argument name="usemem" value=“true”/>7. </request>8. </soml>

Web Based Client – Java Applet

Web Based Client – Java Applet

Constructing complex minds, online

World Server

World Server

MindM Server

MindM Server

Mind ServerMind Server

Mind ServerMind Server

ClientClient

Who builds the sub-minds?

• Everyone– Professional researchers– Students– Amateurs– Anyone interested in AI!

Who builds the parent minds

• Everyone– Anyone can put software online– Anyone can put a world online– Anyone can use my mind as part of their mind– Best minds are the ones that will be re-used

How does this make the AI problem easier?

• Society A solves problem A

• Society B solves problem B

• Problem A + B is solved by writing parent mind to arbitrate between A and B’s parent minds

Society of Mind

World AWorld A

MindM AMindM A

MindMind MindMind MindMind

World BWorld B

MindM BMindM B

MindMind MindMind MindMind

MindM A+BMindM A+B

World A+BWorld A+BClientClient

Summary Part II

• Distribute the mind on the Internet

• Use server-centric architecture

• Use existing HTTP protocol

• Use simple XML-like messages

• Simple entry level

• Potential for provision of large numbers of components required for project

Part III

World-Wide-Mind in learning environment

• 10 Students presented with an online problem

• Have to interact with the online problem using client software

• Need to create “mind” or solution

First Implementation

StartD

CF

EBA

GoalF

E

D

C

B

A

• Blocks World

Blocks World Service

• Action

empty(1)&ontable(A,2)&clear(A)&ontable(B,3)&on(F,B)&on(D,F)&clear(D)&ontable(E,4)&on(C,E)&clear(C)&empty(5)&gripper()

• State

grip(A)

ungrip(3)

Blocks World Run

Blocks World Server

Blocks World Server

Mind ServerMind Server

ClientClient

getstatetakeactiontakeactiontakeactiontakeaction

getactiongetactiongetactiongetactiongetaction

endrunnewrun

endrunnewrun

Blocks World Service

• Supports– newrun

• Provides runid– getstate

• Provides state in form shown

– takeaction• Receives action sent by client• Returns updated state following execution of action

– endrun• Ends run, clears up server

• All messages carry error code information

Building a blocks world “mind”

• Create service that supports– newrun

• Provides runid• Can also provide startup-parameters

– getaction• Receives state from client (which had got it from world

service)• Provides state in form shown

– endrun• Ends run, clears up server

• All messages should carry error code information

Scoreboard

• Scores of best performing minds recorded in a list on the “world” site– Score is the number of moves needed to stack blocks

• Any mind that interacts with world will have its score recorded and displayed

• Others can then run that mind in the world• Performance based assessment

– Performance of algorithm can be judged from score board

Technical Requirements

• Web Server space• One of…

– Java Servlets– ASP– JSP– PHP– CGI – C++/Perl/any language

• To write own client (or to integrate another mind), need to know how to connect to URL– Simple code in most languages

Connect to URL (Java)

String getResponseFromService(String serviceURL, String request) {

URLConnection connection = new URL(serviceURL).openConnection();

PrintWriter outs = new PrintWriter(connection.getOutputStream());

outs.println(request); // SOML message

outs.close();

StringBuffer returnedText = new StringBuffer();

BufferedReader ins = new BufferedReader(new InputStreamReader(connection.getInputStream()));

String line = "";

while((line = ins.readLine()) != null) {

returnedText.append(line);

}

return returnedText.toString();

}

Connect to URL (Perl)

sub getResponseFromService() {

$serviceURL = @_[0];

$request = @_[0];

new LWP::UserAgent->request(

POST “$serviceURL”,

Content_Type => 'form-data',

Content => [ q(q) => "$request"])->content;

}

Summary Part III

• Simple blocks world problem online using protocol/architecture developed for World-Wide-Mind

• Software still available online for others to interact with

• Client software also available

• Students put minds online– Minimal knowledge of network programming required

– Only really need to read from stdin and write to stdout.

• Scoreboard records best performing algorithms/minds

Part IV

Comments on Architecture

• Lightweight web services– Low entry level

• Protocol simple

• Technical requirements minimal

– Software continually available– Problem can be re-used remotely by others

• At the very least it can be tested remotely

Comments on Architecture

• Comparison with World-Wide-Web– Students put project work (HTML pages/PDF)

online (DCU/DIT)– Others can review their project work– Very simple entry level

• Before there were tools (MS Front Page) there was simple (terrible) HTML

Reusability

• Why don’t we use other peoples software?– Platform issues– Programming language isues

• Its in our favour for others to make software available• What is their incentive?

– If you’re putting it online anyway…

• Learning objects– SCORM

• PEARL• TOIA (IMS QTI compatible tools)

Summary Part IV

• We need to re-use other peoples software

• Local installation has many issues

• Complex standards require tools

• Low entry level

Part V

Future Development

• Existing standard– Portal site will provide

• Sample code

• Instructions on creating services

• Services

• Client software

Future Development

• Possible– Integration at higher levels with other standards

• SOAP

• XML

• WSDL

• DAML

– e-Learning standards• PEARL uses HTTP

Questions?