47
QuickTime™ and a None decompressor are needed to see this picture. Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

Embed Size (px)

Citation preview

Page 1: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

Using Little Rails Apps to Solve a Hefty Problem

pairing durable content with lightweight apps

Matt ZumwaltHydraCamp

05 October 2010

Page 2: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

“the web” vs. deep web

Think about all of the digital stuff that humanity is creating right now...

Page 3: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

“the web” vs. deep web

Page 4: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

“the web” vs. deep web

How deep does it run?

How deep could it run?

Page 5: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

“the web” vs. deep web

Page 6: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

“the web” vs. deep web

Page 7: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

what are we doing?

Imagine capturing all of humanity’s digital output from a single day.

How long would it take to go through it all?

How much could we learn about 2010 from that corpus?

Is any of it worthless? What’s worth preserving?

Page 8: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

what could we do?What deep information would be

useful if it was online in a searchable, machine-parseable structure?

census data? crime data?

climate data? genomics?

government documents?

medical research?

Page 9: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

curation

creating, using and preserving

embrace connections

embrace re-use

many perspectives

Page 10: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

curate what?how much deep information is

currently online in useful formats?

how much of the knowledge we produce is being preserved?

what will your answer be in 10 years?

Page 11: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

golden age of digital information

wide open fields of possibilities

ownership, copyright & channels of distribution have not been frozen yet

Page 12: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

long lived, complex contentthe data we create extend beyond and outlast the software used to

create them“the open data movement is more

important than the open source movement”

- Ian Davis Code4Lib 2009

Page 13: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

the funny problemOur vision is to create complex, evolving, long-lived content.  Meanwhile, each set of users wants to see & interact with that content in different ways relative to temporal and intellectual context.

content in open, complex, long-lived context

users in task-specific, time-

specific context

Page 14: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

content in open context, users in specific context

Page 15: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

Page 16: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

let’s take a look at some of the apps we’ve written

Page 17: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

key points• allow applications to be small and specialized

• build iteratively, allowing app to evolve as you learn & have ideas

• reuse your good code, find patterns

• use your model as skeleton then customize views

• make testing & prototyping easy

• think carefully about generators & pre-fab plugins

Page 18: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

standing astride two different

worlds

SOA

Content Modeling

Page 19: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

conflicting sensibilities & requirements

• flash-in-the-pan applications vs. long lived content

• agile development vs. big design up front

• the importance of developer happiness

• convention over configuration vs. standards & interoperability

• "XML is a quagmire" vs. XML assumed as preservation format, validation point, a basis of interoperability, and (sometimes) an integral part of users' vernacular

• "Enterprise is a dirty word" vs. "Enterprise is our reality"

Page 20: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

flash-in-the-pan applications vs. long lived content

allow apps to be short lived while content is long lived

agile development vs. big design up front

keep your eyes on the horizon (big goals) and your feet on the ground (iterative development)

the importance of developer happiness

start talking about developer happiness & get developers talking to each other

convention over configuration vs. standards & interoperability

put standards in their place. when they serve you, make them the assumed (thus hidden) convention

"XML is a quagmire" vs. XML assumed as preservation format, validation point, a basis of interoperability, and (sometimes) an integral part of users' vernacular

XML is a serialization, NOT a programming language

"Enterprise is a dirty word" vs. "Enterprise is our reality"

use the strengths of enterprise to neutralize its pitfalls

Page 21: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

many lightweight views onto

complex content

Page 22: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

flash-in-the-pan applications vs. long lived content

allow apps to be short lived while content is long lived

agile development vs. big design up front

keep your eyes on the horizon (big goals) and your feet on the ground (iterative development)

the importance of developer happiness

start talking about developer happiness & get developers talking to each other

convention over configuration vs. standards & interoperability

put standards in their place. when they serve you, make them the assumed (thus hidden) convention

"XML is a quagmire" vs. XML assumed as preservation format, validation point, a basis of interoperability, and (sometimes) an integral part of users' vernacular

XML is a serialization, NOT a programming language

"Enterprise is a dirty word" vs. "Enterprise is our reality"

use the strengths of enterprise to neutralize its pitfalls

Page 23: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

eyes on the horizon,

feet on the groundThe iterative approach of agile development, combined with RAD, transforms the dynamic

between an application’s developers, its users, and the users’ tasks.

Detachment from reality is the real

danger.

Big designs, lofty goals, and detailed descriptions of needs are all great.

(Don’t build the death star.)

Page 24: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

flash-in-the-pan applications vs. long lived content

allow apps to be short lived while content is long lived

agile development vs. big design up front

keep your eyes on the horizon (big goals) and your feet on the ground (iterative development)

the importance of developer happiness

start talking about developer happiness & get developers talking to each other

convention over configuration vs. standards & interoperability

put standards in their place. when they serve you, make them the assumed (thus hidden) convention

"XML is a quagmire" vs. XML assumed as preservation format, validation point, a basis of interoperability, and (sometimes) an integral part of users' vernacular

XML is a serialization, NOT a programming language

"Enterprise is a dirty word" vs. "Enterprise is our reality"

use the strengths of enterprise to neutralize its pitfalls

Page 25: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

developer happinessHappy developers write

better, more innovative software.

Page 26: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

flash-in-the-pan applications vs. long lived content

allow apps to be short lived while content is long lived

agile development vs. big design up front

keep your eyes on the horizon (big goals) and your feet on the ground (iterative development)

the importance of developer happiness

start talking about developer happiness & get developers talking to each other

convention over configuration vs. standards & interoperability

put standards in their place. when they serve you, make them the assumed (thus hidden) convention

"XML is a quagmire" vs. XML assumed as preservation format, validation point, a basis of interoperability, and (sometimes) an integral part of users' vernacular

XML is a serialization, NOT a programming language

"Enterprise is a dirty word" vs. "Enterprise is our reality"

use the strengths of enterprise to neutralize its pitfalls

Page 27: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

put standards in their place

When they serve you, make standards the assumed, thus

hidden convention. When they’re harmful, say so.

Convention over configuration is not at odds with standards, it merely gets to the heart of

the matter.

Page 28: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

flash-in-the-pan applications vs. long lived content

allow apps to be short lived while content is long lived

agile development vs. big design up front

keep your eyes on the horizon (big goals) and your feet on the ground (iterative development)

the importance of developer happiness

start talking about developer happiness & get developers talking to each other

convention over configuration vs. standards & interoperability

put standards in their place. when they serve you, make them the assumed (thus hidden) convention

"XML is a quagmire" vs. XML assumed as preservation format, validation point, a basis of interoperability, and (sometimes) an integral part of users' vernacular

XML is a serialization, NOT a programming language

"Enterprise is a dirty word" vs. "Enterprise is our reality"

use the strengths of enterprise to neutralize its pitfalls

Page 29: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

XML is a serialization

%table#search_results

%th Title

%th Narrator

%th Location

%th Subject Headings

%tr.result

%td My Title

%td Nancy Narrator

%td Boston, Ma

%td Subject Heading -- Detail, Subject Heading -- Detail II

Page 30: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

XML is a serialization<table id='search_results'>

<th>Title</th>

<th>Narrator</th>

<th>Location</th>

<th>Subject Headings</th>

<tr class=”result”>

<td>My Title</td>

<td>Nancy Narrator</td>

<td>Boston, Ma</td>

<td>Subject Heading -- Detail, Subject Heading -- Detail II</td>

</tr>

</table>

Page 31: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

XML is a serialization

• HTML model has become more important than the serialization

• JSON is very useful

• XSLT is declarative (good for speed, bad for dynamic code) and XML-centric (ivory tower)

• Example: ORE & RDF are both models first while serializations are interchangeable

Page 32: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

flash-in-the-pan applications vs. long lived content

allow apps to be short lived while content is long lived

agile development vs. big design up front

keep your eyes on the horizon (big goals) and your feet on the ground (iterative development)

the importance of developer happiness

start talking about developer happiness & get developers talking to each other

convention over configuration vs. standards & interoperability

put standards in their place. when they serve you, make them the assumed (thus hidden) convention

"XML is a quagmire" vs. XML assumed as preservation format, validation point, a basis of interoperability, and (sometimes) an integral part of users' vernacular

XML is a serialization, NOT a programming language

"Enterprise is a dirty word" vs. "Enterprise is our reality"

use the strengths of enterprise to neutralize its pitfalls

Page 33: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

teach the beast to dance

Use the strengths of enterprise to your benefit while side-stepping its pitfalls.

Page 34: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

flash-in-the-pan applications vs. long lived content

allow apps to be short lived while content is long lived

agile development vs. big design up front

keep your eyes on the horizon (big goals) and your feet on the ground (iterative development)

the importance of developer happiness

start talking about developer happiness & get developers talking to each other

convention over configuration vs. standards & interoperability

put standards in their place. when they serve you, make them the assumed (thus hidden) convention

"XML is a quagmire" vs. XML assumed as preservation format, validation point, a basis of interoperability, and (sometimes) an integral part of users' vernacular

XML is a serialization, NOT a programming language

"Enterprise is a dirty word" vs. "Enterprise is our reality"

use the strengths of enterprise to neutralize its pitfalls

Page 35: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

storing files & metadata

Database-Driven Fedora Repositoryie. Paperclip

demo:5demo:5xyz.mp3xyz.mp3

DC metaDC metatech metatech metarelationshirelationshi

psps......

/uploads/xyz.mp3/uploads/tuv.doc/uploads/blf.jpg

.../xyz.mp3

.../demo-5.foxml

tuv.doctuv.doc.../tuv.doc

Page 36: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

CRUD

CreateRetrieveUpdateDelete

Page 37: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

CRUD

Create/SubmitModifyDeleteIndex

Page 38: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

CRUD

SearchBrowse

View DetailsRender Edit View

Page 39: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

CRUD in Rails

Rails MVCRails MVC

RDBMSRDBMS

Page 40: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

CRUD in repositories

Create/SubmitCreate/Submit(CUD)(CUD)

Search/ViewSearch/View(R)(R)

IndexingIndexing

Page 41: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

the Hydra framework

Hydra Rails Hydra Rails EngineEngine(CUD)(CUD)

BlacklightBlacklight(R)(R)

SolrizerSolrizerFedoraFedora SolrSolr

http://github.com/projecthydra/hydrangeahttps://wiki.duraspace.org/display/hydra/

Page 42: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

rich, faceted search

BlacklightBlacklight(Read only)(Read only)

SolrSolr

• faceted search & discovery• metadata agnostic• content-aware• supports specialized users• Rails plugin• solr-driven • strong dev community

http://projectblacklight.org

Page 43: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

models that work

ActiveFedoraActiveFedora

FedoraFedora

• models inspired by datamapper• xml tools -- OM• convenience methods• update_attributes

OMOM

http://github.com/mediashelf/active_fedorahttp://github.com/mediashelf/om

Page 44: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

rails engine

FedoraFedora

ActiveFedoraActiveFedora

CustomCustomControlleControlle

rsrs

Custom Custom Views & Views & HelpersHelpers

OMOM

• ActiveFedora & OM• controllers integrate everything• views & helpers to simplify dev• custom JQuery plugin

Page 45: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

the final piece: indexing

Hydra Rails Hydra Rails EngineEngine(CUD)(CUD)

BlacklightBlacklight(R)(R)

??????FedoraFedora SolrSolr

Page 46: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

the Hydra framework

Hydra Rails Hydra Rails EngineEngine(CUD)(CUD)

BlacklightBlacklight(R)(R)

SolrizerSolrizerFedoraFedora SolrSolr

http://github.com/projecthydra/solrizer

Page 47: Using Little Rails Apps to Solve a Hefty Problem pairing durable content with lightweight apps Matt Zumwalt HydraCamp 05 October 2010

QuickTime™ and aNone decompressor

are needed to see this picture.

now the fun begins...