java . blogs

java.blogs“The movement, the site, the technology”

Prepared by Mike Cannon-Brookes - June, [email protected] - http://www.atlassian.com

2

Agenda

1. Blogging & Java2. What is java.blogs?3. Technology behind javablogs.com

3

What is a weblog?

• “News site” written by one person• Online, chronological journal• Personal list of thoughts, links etc• Newest entries at the top of the page• Usually has an RSS version

• Hard to categorize, easier to show you an example

4

Posts

Typical Blog Pieces

Silly Title

Ugly Author Pic

5

Posts

Nav

Search

Blog Roll

About

6

Why read blogs?

• Get a personal view point• Bloggers usually discuss latest content /

ideas• In the Java world, often a good source of

solutions and personal experience– Example: commons-logging & log4j

• Comparison: best feature on Amazon?– book reviews!

7

Why write a blog?

• Everyone has their own reasons• A place for personal expression • Share your thoughts, wisdom • Feel part of a community• Rant / rave / vent / discuss

– BileBlog perfect example

8

RSS

• Really Simple Syndication? • Rich Site Summary?

• RSS is the “protocol” of the syndication world • An XML format representing a list of links• Used to syndicate weblogs in a machine readable

format

• It is a very simple format…

9

RSS Example

<rss><channel>

<title>rebelutionary</title><link>http://blogs.atlassian.com/rebelutionary/</link><description>Mike Cannon-Brookes on J2EE, Java, software development, bug tracking, Atlassian, JIRA and whatever comes to mind.</description>

<item><title>The BileBlog is painfully accurate</title><link>http://blogs.atlassian.com/rebelutionary/archives/000166.html</link><description>Let me start by saying …</description>

</item><item>

<title>Two Weeks Left to TSS </title><link>http://blogs.atlassian.com/rebelutionary/archives/000165.html</link><description>Less than two weeks left until TSS Symposium... </description>

</item>. . .

</channel></rss>

10

Blog Syndication

Blog 1 Blog 2Blog 3

News Aggregator

RSS(XML)

RSS(XML)HTML HTML

Reader

11

News Aggregators

• Software to aggregate news from your personal list of favourite sites.

• Popular:– AmphetaDesk (Perl)– NetNewsWire (OSX)– NewsCrawler (Win)– NewsMonster (Moz)

• java.blogs is a ‘web based’ news aggregator for the Java community.

12

Blogging using Java

• Lots of blog software written in Java• 3 biggest projects:

– Roller, SnipSnap and Blojsom• Other projects include:

– PersonalBlog, MiniBlog, CocoBlog etc.

13

Tools: Roller

• www.rollerweblogger.org• The most advanced Java-based blogging tool• Supports comments, templating, RSS, XML-RPC,

multi-user.• Deploys on Servlet container & JDBC database• Technologies used:

– Persistence: Hibernate or Castor– MVC: Struts– Views: Velocity

14

Tools: SnipSnap

• www.snipsnap.org• Innovative Java-based ‘Bliki’ (Blog + Wiki)• Features include a complex Wiki rendering

engine, comments, search, RSS, XML-RPC• Standalone Jetty instance with McKoi DB• Technologies:

– Views: JSP– Everything else hand rolled!

15

Tools: Blojsom

• blojsom.sf.net• Java port of Bloxsom (Perl-based tool)• Uses the file system for persistence• Supports comments, RSS, XML-RPC,

referrer tracking, searching.• Runs on JDK 1.4 w/ Servlet 2.3 container.• Technologies:

– Flexible view dispatcher: JSP, Velocity or Freemarker

16

Tools: Wrap Up

• If you’re looking to start a new blog, without software - try freeroller

http://www.freeroller.net

• No installation required• Completely web based• Good way to get blogging!

17

Agenda

• Blogging & Java• What is java.blogs?• Technology behind javablogs.com

18

java.blogs Overview

• www.javablogs.com• A web aggregator for Java focused weblogs• Currently aggregates:

– Over 1.5 million words– Written in 18,000+ entries– Posted by 360+ bloggers

19

20

Information Flows

Blogs(Internet)

Readers

Browse Website

Email

RSS Feeds

java.blogs

Bloggers

Thoughts

Aggregated RSS

21

java.blogs Features

• Single ‘feed’ of all Java blog entries• Decentralized, uncontrolled, focused community

– Each blog owner controls their own blog• Tracks “popular” entries

– Popular entries show what the community reads• Text & date searchable content

– eg find all entries about “AOP” this week• Daily email notifications

– Keep up to date with most popular entries from within your email client

22

java.blogs daily update

23

How is it different?

• It is a true community of equals• Find a wide range of views on any topic

– eg Over 30 entries discussing the impact of java.net within 24 hours of launch.

• There is no ‘agenda’ - just personal views• “Good entries” rise to the top• More news than you can ever want

– Now averaging over 200 entries / day

24

java.blogs

• 350 authors• 1000 users• Users in total control

– This is good and bad!• Content is personal

– 100% opinions• Analogy: having a

conversation with 350 developers

• Few authors • 290,000 users• Centralised editorial

– Quality assurance• Content is controlled

– Opinions in comments• Analogy: reading a

newspaper or magazine

TSSvs

25

java.net?

• java.net contains Sun ‘by-invitation only’ weblogs

• This is good for Javablogs.com!• Many new blogs to aggregate (java.net has RSS):

– James Gosling,– Sam Ruby,– Mike Clark, etc

• See http://weblogs.java.net/ for more information.

26

Agenda

• Blogging & Java• What is java.blogs?• Technology behind javablogs.com

27

javablogs.com Components

DB(Postgres)

Persistence(OFBiz)

MVC(WebWork)

Presentation(SiteMesh,WebWork

& JSP)

Scheduling(Quartz &Atlassian

Scheduler)

java.blogscode

(Java!)

Mail(JavaMail &

Velocity)

Search &Indexing(Lucene)

XML / SOAP(Electric XML & Glue)

UserManagement

(OSUser)

Web

Non-Web

28

Component Tour

Persistence MVC

Presentation

Scheduling

MailIndexing

Security

29

Component Tour: Scheduling

Persistence MVC

Presentation

Scheduling

MailIndexing

Security

Components: Quartz & atlassian-scheduler

30

Scheduling: Quartz

• www.part.net/quartz.html• J2EE job scheduling system• Alternatives?

– Threads, util.Timer, external cron• Operates around a job/trigger model• Built in period and cron triggers• Extremely flexible, friendly component

• USE: scheduling RSS retrieval, sending daily email updates, any periodic task

• atlassian-scheduler is a Quartz XML config…

31

Scheduling: Atlassian Scheduler

• Example: scheduler-config.xml <scheduler>

<jobs><job name="Updater" class="com...UpdateJob" /> <job name="DailyEmail" class="com...EmailJob" />

</jobs><triggers>

<trigger name="UpdaterTrigger" job="Updater"><period>3m</period><startDelay>3m</startDelay>

</trigger><trigger name=”MailTrigger" job="DailyEmail" type="cron">

<startDelay>1m</startDelay><expression>00***?</expression>

</trigger></triggers>

</scheduler>

32

Component Tour: Persistence

Persistence MVC

Presentation

Scheduling

MailIndexing

Security

Component: OFBiz Entity Engine

33

Persistence: OFBiz EE

• www.ofbiz.org - Open For Business project• “Different” persistence engine• Light wrapper around JDBC datasources• Proud to be relational!• Data entities are generic value objects• Entity, field, view and relation definitions

are in XML• Use: all data persistence in Javablogs.

34


• Retrieving all blogs:Collection blogs = ee.findAll(“Blog”);

• Retrieving a specific blog entity:Map idField = UtilMisc.toMap(“id”, id);GenericValue blog = ee.findByPrimaryKey("Blog", idField);

• Getting and setting a field value:GenericValue blog = retrieveBlog();String url = blog.getString(“url”);blog.set(“url”, newUrl);blog.store();

35


• Retrieving blog entries:Collection entries = blog.getRelated(“ChildEntry”);

• Complex find operation:Map fields = UtilMisc.toMap(“author”, “bob”);List sortorders = UtilMisc.toList(“dateadded”);entries = ee.findByAnd(“Blog”, fields, sortorders);

36

• Automatically creates and updates tables• Code talks to the logical model• Logical model + field mappings = physical db


Logical Modelentitymodel.xml

Field Mappingsfieldtypes-oracle.xml

javablogs code

PhysicalDatabase

OFBiz EE

37


• Sample entity from entitymodel.xml<entity entity-name= "BlogEntry" package-name= "" >

<field name= "id" type= "numeric" col-name= ”entry_id" /> <field name= "blog" type= "numeric" /> <field name= "title" type= "long-varchar" /> <field name= "created" type= "date-time" /> ...<prim-key field= "id" /> <relation type= "one" title= "Parent" rel-entity-name= "Blog" >

<key-map field-name= "blog" rel-field-name= "id" /> </ relation >

</ entity >

38


• Sample of fieldtypes-postgres.xml<fieldtypemodel>

<field-type-def type= "date-time" sql-type= "TIMESTAMP" java-type= "java.sql.Timestamp" />

<field-type-def type= "short-varchar" sql-type= "VARCHAR(60)" java-type= "String" />

<field-type-def type= "very-long" sql-type= "TEXT" java-type= "String" />

<field-type-def type= "credit-card-number" sql-type= "VARCHAR(40)" java-type= "String">

<validate name= "isAnyCard"/></ field-type-def>

39

Component Tour: Indexing

Persistence MVC

Presentation

Scheduling

MailIndexing

Security

Component: Jakarta Lucene

40

Indexing: Lucene

• jakarta.apache.org/lucene• The full-text index and search component• Highly scalable and efficient architecture

• Why use full text searching?– No more SQL: LIKE ‘%foo%’

• Used for full text search of blog entries

41

Component Tour: Security

Persistence MVC

Presentation

Scheduling

MailIndexing

Security

Components: OSUser & atlassian-security

42

User Management: OSUser

• www.opensymphony.com/osuser• Server-agnostic user management API• Handles user, group and profile data

• Why needed?– User data different to other data– Pluggable storage providers (ie LDAP)– Writing portable J2EE auth code very hard

43


• Providers for storage:– EJB, JDBC, OFBiz, Hibernate, LDAP, XML files etc

• Adapters for server integration– Weblogic, Orion, JBoss, Resin etc

OSUser

javablogs code

Storage-users

-groups-profiles

ApplicationServer

adapters providers

44


• Retrieving a specific user and their groups:User user = userManager.getUser(“fred”);Collection groups = user.getGroups();

• Retrieving a user’s profile:PropertySet profile = user.getPropertySet();String country = profile.getString(“user.address.country”);Date signedUp = profile.getDate(“user.signup”);Long logins = profile.getLong(“user.numlogins”);

• Creating a new user in the administrators group:User newUser = userManager.createUser(“mike”);newUser.setEmail(“[email protected]”);newUser.addToGroup(userManager.getGroup(“administrators”));

45

Security: atlassian-security

• Security is worst part of Servlet spec– Not portable between servers– Very limited URL patterns

• To perform effective security checks on a web application, need to roll your own framework.

• atlassian-security is our solution to this– in process of being open sourced

46

atlassian-security architecture

Security Services Calculate the authorisation needed for a particular request

Interceptors Interject code to run before / after security events

Authenticator Authenticate a user (login, logout, retrieve) against a user system

Controller Governs whether security checks are enabled or disabled

SecurityFilter Calculates roles required for this request against all configured security services

LoginFilter Looks for credentials (eg username / password) in the request and tries to log user in if found

Integration Points

Concepts

47

atlassian-security Services

• Services allow you to check the roles required for any request

• Two bundled services:– Path Service: looks at the request URL. Allows

extremely flexible path lookups• /admin/*, **/admin/*, /admin/Setup* etc

– WebWork Service: looks at the action being executed

• Other examples could be IPService or KeyService.

48

atlassian-security PathService

• PathService is configured in a security-paths.xml file:

<security-paths><path name= "admin">

<url-pattern>/secure/admin/*</url-pattern><role-name>administrators</role-name>

</path><path name= "user">

<url-pattern>/secure/*</url-pattern><role-name>users, developers</role-name>

</path></security-paths>

49

atlassian-security Interceptors

• Problem: run code before / after security events– Events? login, logout, auth attempts etc.– Under the Servlet spec you can’t do this.

• Solution: AOP-like interceptors

• Example: used in java.blogs to store the date of last login, and number of logins

50

Component Tour: MVC

Persistence MVC

Presentation

Scheduling

MailIndexing

Security

Component: OpenSymphony WebWork

51

MVC: WebWork

• www.opensymphony.com/webwork• MVC framework• Implementation of GoF command pattern

– WW actions are ‘method objects’

• Used for all actions in javablogs.com, with JSP views

• “WebWork 2” presentation later today for detailed information.

52

MVC: WebWork

• Simple, clean APIs– No more formbeans

• Field or model driven actions• Actions are easily testable• View agnostic

– Supports JSP, Velocity, XML, JasperReports

• Let’s look at a brief example…

53

MVC: WebWork - Result

54

MVC: WebWork - Actionpublic class AddBlog extends WebsiteActionSupport { // fields

// getters & setters …

protected void doValidation() { if (url != null && !url.startsWith("http://")) url = "http://" + url;

if (url == null) addError("url", ”Please fill in a blog URL"); }

protected String doExecute() { try { BlogUtils.addBlog(url, getRemoteUser()); return SUCCESS; } catch (BlogException e) { addErrorMessage(e.getMessage()); return ERROR; } }}

55

MVC: WebWork - Config

• actions.xml is where actions are configured:

<action name="user.AddBlog" alias="AddBlog"> <view name=”success"> /secure/views/user/addblog2.jsp </view> <view name=”error"> /secure/views/user/addblog.jsp </view></action>

56

MVC: WebWork - View<html><head><title>Add Blog</title></head>

<body> <page:applyDecorator name="jbform"> <page:param name="description"> <b>Step 1 of 2</b>: Specify the url of the blog to add. </page:param> <page:param name="action">AddBlog.jspa</page:param> <page:param name="submitName">Add</page:param> <page:param name="cancelURI">ViewProfile.jspa</page:param>

<page:param name="width">100%</page:param>

<ui:textfield label="'RSS Feed URL'" name="'url'" size="'70'"/> </page:applyDecorator></body></html>

57

MVC: WebWork - Error

58

Component Tour: Presentation

Persistence MVC

Presentation

Scheduling

MailIndexing

Security

Component: OpenSymphony SiteMesh

59

Presentation: SiteMesh

• www.opensymphony.com/sitemesh• Page layout and decoration engine• Clean separation of content vs presentation

– Designers design presentation– Developers develop content

• Applies GoF decorator pattern to HTML– Metaphor: Swing look and feel changer

60Decorated page: Green sections are put there by the decorator.

61

Un-decorated page: Notice that there is no header / footer, and no style sheet.

62

Another page: using a TSS decorator - no GIF trickery!http://www.javablogs.com/Welcome.jspa?decorator=tss

63


Result(HTML)

1. render content

4. Mergecontent &

presentation

Content(HTML

fragment)

ContentSource

(JSP, Perl, PHP, HTML etc)Field Map

2. parse HTML

SiteMesh

Presentation(Decorator -

JSP)

DecoratorMappers

3. select decoratorfor request

64

SiteMesh: Raw HTML

<html><head>

<title>About JavaBlogs</title><meta name=“section” content=“About”>

</head>

<body bgcolor=“#ffffff”>JavaBlogs <b>aggregates</b> the blogs of Java bloggers.

</body></html>

• Benefit: keeps your HTML simple

65

SiteMesh: Parsed Field Map

• Turns your HTML into a map of fields

Key Valuetitle About JavaBlogs

meta.section About

body JavaBlogs <b>aggregates</b> the blogs of Java blogges

body.bgcolor #ffffff

66

SiteMesh: Decorator JSP

<%@ taglib uri= "sitemesh-decorator" prefix= ”dec" %>

<html><head><title>java.blogs - <dec:title /></title>

</head><body bgcolor="<dec:getProperty property= ”body.bgcolor" />">

<h2 class= "pagetitle"><dec:title /></h2>

<dec:isPropertySet name=“meta.section”> You are in the <dec:getProperty property=“meta.section”> section.<br></dec:isPropertySet>

<p><dec:body /></p></body></html>

• Decorators are generally JSP pages, and use a single sitemesh-decorator taglib

• Benefit: simple for designers to understand

67

SiteMesh: Result Code

<html><head><title>java.blogs - About JavaBlogs</title>

</head><body bgcolor=”#ffffff">

<h2 class= "pagetitle">About JavaBlogs</h2>

You are in the About section.<br>

<p>JavaBlogs <b>aggregates</b> the blogs of Java bloggers.</p>

</body></html>

• Resulting code is plain HTML again

68


• How are decorators chosen?– Chosen on meta data in the fields map and request

objects• Mapping is decoupled from pages themselves

– No more fragile <jsp:include .. /> statements

PageDM Uses <meta> propertiesFrameSetDM Handles framed sitesPrintableDM For making printable versionsRobotsDM Serves robots special decoratorsParameterDM Uses request parametersConfigDM Uses decorators.xml and URL paths

69


• Most often use ConfigDecoratorMapper - decorators.xml:

<decorators><decorator-mapping decorator="none">

<url-pattern>/styles/*</url-pattern></decorator-mapping>

<decorator name="admin” page="/decorators/admin.jsp"><url-pattern>/secure/admin/*</url-pattern>

</decorator></decorators>

70

Component Tour: Mail

Persistence MVC

Presentation

Scheduling

MailIndexing

Security

Component: Jakarta Velocity

71

Mail: Velocity

• Problem: generating email content is not as easy as HTML content

• We use Velocity to generate text and HTML email

• Alternatives?– JSP pages with wget, String replacement, Java code!

72

Mail: Velocity

• http://jakarta.apache.org/velocity• Text templating engine from Jakarta• Advantages

– Extremely fast to generate lots of text– Simple syntax to learn– Plugs into your existing model easily

• Merges in a similar way to SiteMesh– Separates content & presentation

MergedResult

VelocityTemplate

VelocityContext

(Map of fields)

73

Mail: Velocity - Template• Here is text/dailyemail.vm as a sample:

java.blogs daily update-----------------------Here is a list of the most popular blog entries from the last 24 hours on ${baseurl}, brought to you by the world's best issue tracker:

JIRA 2 - http://www.atlassian.com/software/jira - enjoy! :)

Entries-------#foreach ($entry in $entries)

#if (${entry.getString("link")})${entry.getString("title")}http://www.javablogs.com/Jump.jspa?id=${entry.getLong("id")}${entry.getRelatedOne("ParentBlog").getString("title")}

#end#end

74

Mail: Velocity - Result

•Resulting email looks like:

java.blogs daily update-----------------------Here is a list of the most popular blog entries from the last 24 hourson http://www.javablogs.com/Welcome.jspa, brought to you by the world's best issue tracker:

JIRA 2 - http://www.atlassian.com/software/jira - enjoy! :)

Entries-------OSCache: You too can pretend you're competent!http://www.javablogs.com/Jump.jspa?id=38371The BileBlog

BlueOxygen Java Open Source CollectionWow,http://www.javablogs.com/Jump.jspa?id=38374Frans Thamura's Blogger

. . .

75

Component Tour: Build System

Persistence MVC

Presentation

Scheduling

MailIndexing

Security

Component: Apache Maven

Build System

76

Build System: Maven

• Most popular Java build tool - Ant

• Problems with Ant?– lots of duplicate code

• we don’t tolerate duplication in our Java code, why tolerate it in our build.xml?!

– no dependency management

• Maven is one attempt at an Ant alternative

77

Build System: Maven

• maven.apache.org• Build system evolving from Ant

– can still call any Ant task from Maven• Aim: to separate meta data from function• Still very ‘beta’• Plugins to do many common tasks

– ie compile, JAR, clean syntax, upload files

78

Build System: Maven

• Manages and downloads dependencies <dependency> <id>velocity</id> <version>1.3</version> <properties> <war.bundle.jar>true</war.bundle.jar> </properties> </dependency>

• Each developer maintains a local repository– One copy of each JAR on your system

• Dependencies not found locally are downloaded– example above will download velocity-1.3.jar

79

Important Lessons

• Performance vs caching– Cache 20% of pages that generate 80% of load– OSCache used heavily to cache web tier– Gzip filter to reduce bandwidth

• javablogs.com churns > 1GB traffic/day

• Working in a fragile environment– Users complained that javablogs wasn’t working– Problem: RSS feeds can go down / be malformed– Solution: Provide feedback on errors to users directly

80

Future Challenges

• Challenges: – How do we decide what a ‘good’ post is?– Should we display non-Java posts?

• Solution: Algorithmically calculate ‘best’ posts– Needs to be simple and as automated as possible– Distributed moderation framework?– What metrics to use?

• Views vs user votes vs java-ness

• Solution: Bayesian filtering of posts– Solve the java vs non-java problem– Classifier4J - http://classifier4j.sourceforge.net/

81

Want More?

• The site - http://www.javablogs.com• My blog - http://blogs.atlassian.com/rebelutionary • Email - [email protected]

Thank you for listening - questions?

Mike Cannon-BrookesATLASSIAN - www.atlassian.com

Documents

java . blogs