Writing Pluggable Software Tatsuhiko Miyagawa [email protected] Six Apart, Ltd. / Shibuya Perl Mongers YAPC::Asia 2007 Tokyo

Writing Pluggable Software

Tatsuhiko Miyagawa [email protected]

Six Apart, Ltd. / Shibuya Perl Mongers YAPC::Asia 2007 Tokyo

Tatsuhiko MiyagawaTatsuhiko Miyagawa

For non-JP attendees …

If you find \ in the code,Replace that with backslash.

(This is MS' fault)


PlaggableSoftware


PlaggableSoftware


PluggableSoftware


Agenda


#1How to make

your app pluggable


#2TMTOWTDP

There's More Than One Way To Deploy Plugins

Pros/Cons by examples


First-of-all:Why pluggable?


Benefits


#1Keep the app design

and code simple


#2Let the app users

customize the behavior

(without hacking the internals)


#3It's fun to write

pluginsfor most hackers

(see: Plagger and Kwiki)


"Can your app do XXX?"

"Yes, by plugins."


"Your app has a bug in YYY""No, it's the bug in plugin

YYY,Not my fault."

(Chain Of Responsibilities)


Good EnoughReasons, huh?


#1Make your app

pluggable


Example


ack (App::Ack)


grep –r for programmers


Ack is a "full-stack"software now.


By "full-stack" I mean:

Easy installNo configurationNo way to extend


Specifically:These are hardcoded

Ignored directoriesFilenames and types


Ignored Directories

@ignore_dirs = qw( blib CVS RCS SCCS .svn

_darcs .git );


Filenames and languages mapping

%mappings = ( asm => [qw( s S )], binary => …, cc => [qw( c h xs )], cpp => [qw( cpp m h C H )], csharp => [qw( cs )],… perl => [qw( pl pm pod tt ttml t )],

…);


What if makingthese pluggable?


DISCLAIMER


Don't get me wrong Andy,I love ack the way it is…

Just thought it can bea very good example for the

tutorial.


Quickstart:Class::Trigger

Module::Pluggable

© Six Apart Ltd. Employees


Class::Trigger SYNOPSIS

package Foo;use Class::Trigger;

sub foo { my $self = shift; $self->call_trigger('before_foo'); # some code ... $self->call_trigger('after_foo');}

package main;Foo->add_trigger(before_foo => \&sub1);Foo->add_trigger(after_foo => \&sub2);


Class::TriggerHelps you to

implementObserver Pattern.

(Rails calls this Observer)


Module::Pluggable SYNOPSIS

package MyClass;use Module::Pluggable;

use MyClass;my $mc = MyClass->new();# returns the names of all plugins installed

under MyClass::Plugin::*my @plugins = $mc->plugins();

package MyClass::Plugin::Foo;sub new { … }1;


Setup plugins in App::Ack

package App::Ack;

use Class::Trigger;use Module::Pluggable require => 1;

__PACKAGE__->plugins;


Setup plugins in App::Ack

package App::Ack;

use Class::Trigger;use Module::Pluggable require => 1;

__PACKAGE__->plugins; # "requires" modules


Ignored Directories (Before)

@ignore_dirs = qw( blib CVS RCS SCCS .svn

_darcs .git );


Ignored Directories (After)

# lib/App/Ack.pm__PACKAGE__->call_trigger('ignore_dirs.add', \@ignore_dirs);


Ignored Directories (plugins)

# lib/App/Ack/Plugin/IgnorePerlBuildDir.pmpackage App::Ack::Plugin::IgnorePerlBuildDir;

App::Ack->add_trigger( "ignore_dirs.add" => sub { my($class, $ignore_dirs) = @_; push @$ignore_dirs, qw( blib ); },);

1;


Ignored Directories (plugins)

# lib/App/Ack/Plugin/IgnoreSourceControlDir.pmpackage

App::Ack::Plugin::IgnoreSourcdeControlDir;

App::Ack->add_trigger( "ignore_dirs.add" => sub { my($class, $ignore_dirs) = @_; push @$ignore_dirs, qw( CVS RCS .svn _darcs .git ); },);

1;


Filenames and languages (before)

%mappings = ( asm => [qw( s S )], binary => …, cc => [qw( c h xs )], cpp => [qw( cpp m h C H )], csharp => [qw( cs )],… perl => [qw( pl pm pod tt ttml t )],

…);


Filenames and languages (after)

# lib/App/Ack.pm__PACKAGE__->call_trigger('mappings.add', \%mappings);


Filenames and languages (plugins)

package App::Ack::Plugin::MappingCFamily;use strict;

App::Ack->add_trigger( "mappings.add" => sub { my($class, $mappings) = @_; $mappings->{asm} = [qw( s S )]; $mappings->{cc} = [qw( c h xs )]; $mappings->{cpp} = [qw( cpp m h C H )]; $mappings->{csharp} = [qw( cs )]; $mappings->{css} = [qw( css )]; },);

1;


Works great with few lines of

code!


Now it's time to add Some useful stuff.


Example Plugin:Content Filter


sub _search { my $fh = shift; my $is_binary = shift; my $filename = shift; my $regex = shift; my %opt = @_;

if ($is_binary) { my $new_fh; App::Ack->call_trigger('filter.binary', $filename, \

$new_fh); if ($new_fh) { return _search($new_fh, 0, $filename, $regex, @_); } }


Example:Search PDF content

with ack


PDF filter plugin

package App::Ack::Plugin::ExtractContentPDF;use strict;use CAM::PDF;use File::Temp;

App::Ack->add_trigger( 'mappings.add' => sub { my($class, $mappings) = @_; $mappings->{pdf} = [qw(pdf)]; },);


PDF filter plugin (cont.)

App::Ack->add_trigger( 'filter.binary' => sub { my($class, $filename, $fh_ref) = @_; if ($filename =~ /\.pdf$/) { my $fh = File::Temp::tempfile; my $doc = CAM::PDF->new($file); my $text; for my $page (1..$doc->numPages){ $text .= $doc->getPageText($page); } print $fh $text; seek $$fh, 0, 0; $$fh_ref = $fh; } },);


PDF search

> ack --type=pdf Audreyyapcasia2007-pugs.pdf:3:Audrey Tang


Homework

Use File::ExtractTo handle arbitrary media files


Homework 2:

Search non UTF-8 files(hint: use Encode::Guess)You'll need another hook.


Summary

Class::Trigger+ Module::Pluggable= Pluggable app easy


#2TMTOWTDP

There's More Than One Way To Deploy Plugins


Module::Pluggable+ Class::Trigger

= Simple and Nicebut has limitations


In Reality, we need more control

over how plugins behave


1)The order of

plugin executions


2)Per user

configurationsfor plugins


3)Temporarily

Disable pluginsShould be easy


4)How to install

& upgrade plugins


5)Let plugins

have storage area


Etc, etc.


Examples:Kwiki

Plaggerqpsmtpd

Movable Type


I won't talk aboutCatalyst plugins

(and other framework thingy)


Because they're NOT

"plug-ins"


Install plugins And now you write

MORE CODE


95% of Catalyst plugins

Are NOT "plugins"But "components"

95% of these statistics is made up by the speakers.


Kwiki 1.0


Kwiki Plugin code

package Kwiki::URLBL;use Kwiki::Plugin -Base;use Kwiki::Installer -base;

const class_id => 'urlbl';const class_title => 'URL Blacklist DNS';const config_file => 'urlbl.yaml';

sub register { require URI::Find; my $registry = shift; $registry->add(hook => 'edit:save', pre =>

'urlbl_hook'); $registry->add(action => 'blacklisted_url');}


Kwiki Plugin (cont.)

sub urlbl_hook { my $hook = pop; my $old_page = $self->hub->pages->new_page($self-

>pages->current->id); my $this = $self->hub->urlbl; my @old_urls = $this->get_urls($old_page->content); my @urls = $this->get_urls($self->cgi-

>page_content); my @new_urls = $this->get_new_urls(\@old_urls, \

@urls); if (@new_urls && $this->is_blocked(\@new_urls)) { $hook->cancel(); return $self->redirect("action=blacklisted_url"); }}


Magic implementedin Spoon(::Hooks)


"Install" Kwiki Plugins

# order doesn't matter here (according to Ingy)Kwiki::DisplayKwiki::EditKwiki::Theme::BasicKwiki::ToolbarKwiki::StatusKwiki::Widgets# Comment out (or entirely remove) to disable# Kwiki::UnnecessaryStuff


Kwiki plugin config

# in Kwiki::URLBL plugin__config/urlbl.yaml__urlbl_dns: sc.surbl.org, bsb.spamlookup.net,

rbl.bulkfeeds.jp

# config.yamlurlbl_dns: myowndns.example.org


Kwiki plugins are CPAN modules


Install and Upgrade plugins

cpan> install Kwiki::SomeStuff


Using CPAN as a repository

Pros #1:reuse most of current CPAN infrastructure.


Using CPAN as a repository

Pros #2:Increasing # of

modules= good motivation

for Perl hackers


Cons #1:Installing CPAN deps

could be a mess(especially for Win32)


Cons #2:Whenever Ingy

releasesnew Kwiki, lots of plugins just break.


Kwiki plugin storage

return if grep {$page->id} @{$self->config->cached_display_ignore};

my $html = io->catfile( $self->plugin_directory,$page->id)->utf8;


Kwiki 2.0


Same as Kwiki 1.0


Except:plugins are now

in SVN repository



Plagger plugin

package Plagger::Plugin::Publish::iCal;use strict;use base qw( Plagger::Plugin );

use Data::ICal;use Data::ICal::Entry::Event;use DateTime::Duration;use DateTime::Format::ICal;

sub register { my($self, $context) = @_; $context->register_hook( $self, 'publish.feed' => \&publish_feed, 'plugin.init ' => \&plugin_init, );}


Plagger plugin (cont)

sub plugin_init { my($self, $context) = @_;

my $dir = $self->conf->{dir}; unless (-e $dir && -d _) { mkdir $dir, 0755 or $context->error("Failed to mkdir $dir: $!"); }}


Plagger plugin storage

$self->conf->{invindex} ||= $self->cache->path_to('invindex');


Plagger plugin config

# The order matters in config.yaml# if they're in the same hooksplugins: - module: Subscription::Config config: feed: - http://www.example.com/ - module: Filter::DegradeYouTube config: dev_id: XYZXYZ - module: Publish::Gmail disable: 1


Plugins Install & Upgrade

> notest cpan Plagger# or …> svn co http://…/plagger/trunk plagger> svn update


Plagger impl.ripped off by

many apps now


qpsmtpd


mod_perl for SMTP

Runs with tcpserver, forkserver or Danga::Socket standalone


Plugins: Flat files

rock:/home/miyagawa/svn/qpsmtpd> ls -F pluginsasync/ greylistingauth/ hosts_allowcheck_badmailfrom http_configcheck_badmailfromto ident/check_badrcptto logging/check_badrcptto_patterns miltercheck_basicheaders parse_addr_withhelocheck_earlytalker queue/check_loop quit_fortunecheck_norelay rcpt_okcheck_relay relay_onlycheck_spamhelo require_resolvable_fromhostcontent_log rhsblcount_unrecognized_commands sender_permitted_fromdns_whitelist_soft spamassassindnsbl tlsdomainkeys tls_cert*dont_require_anglebrackets virus/


qpsmtpd plugin

sub hook_mail { my ($self, $transaction, $sender, %param) = @_;

my @badmailfrom = $self->qp->config("badmailfrom") or return (DECLINED);

for my $bad (@badmailfrom) { my $reason = $bad; $bad =~ s/^\s*(\S+).*/$1/; next unless $bad; $transaction->notes('badmailfrom', $reason) … } return (DECLINED);}


Actually qpsmtpdPlugins are "compiled"to modules


my $eval = join("\n", "package $package;", 'use Qpsmtpd::Constants;', "require Qpsmtpd::Plugin;", 'use vars qw(@ISA);', 'use strict;', '@ISA = qw(Qpsmtpd::Plugin);', ($test_mode ? 'use Test::More;' : ''), "sub plugin_name { qq[$plugin] }", $line, $sub, "\n", # last line comment without newline? );

$eval =~ m/(.*)/s; $eval = $1;

eval $eval; die "eval $@" if $@;


qpsmtpd plugin config

rock:/home/miyagawa/svn/qpsmtpd> ls config.sample/config.sample:IP logging

require_resolvable_fromhostbadhelo loglevel rhsbl_zonesbadrcptto_patterns plugins

size_thresholddnsbl_zones rcpthosts

tls_before_authinvalid_resolvable_fromhost relayclients tls_ciphers


config/plugins

# content filtersvirus/klez_filter

# rejects mails with a SA score higher than 2spamassassin reject_threshold 20


config/badhelo

# these domains never uses their domain when greeting us, so reject transactions

aol.comyahoo.com


Install & Upgrade plugins

Just use subversion



MT plugins are flat-files

(or scripts that call modules)


MT plugin code

package MT::Plugin::BanASCII; our $Method = "deny";use MT; use MT::Plugin;

my $plugin = MT::Plugin->new({ name => "BanASCII v$VERSION", description => "Deny or moderate ASCII or Latin-1

comment",}); MT->add_plugin($plugin);MT->add_callback('CommentFilter', 2, $plugin, \

&handler);


MT plugin code (cont)

sub init_app { my $plugin = shift; $plugin->SUPER::init_app(@_); my($app) = @_; return unless $app->isa('MT::App::CMS'); $app->add_itemset_action({ type => 'comment', key => 'spam_submission_comment', label => 'Report SPAM Comment(s)', code => sub { $plugin->submit_spams_action('MT::Comment',

@_) }, } );




MT plugin storage

require MT::PluginData;my $data = MT::PluginData->load({ plugin => 'sidebar-manager', key => $blog_id },);unless ($data) { $data = MT::PluginData->new; $data->plugin('sidebar-manager'); $data->key($blog_id);}$data->data( \$modulesets );$data->save or die $data->errstr;


Order control

MT->add_callback('CMSPostEntrySave', 9, $rightfields, \&CMSPostEntrySave);

MT->add_callback('CMSPreSave_entry', 9, $rightfields, \&CMSPreSave_entry);

MT::Entry->add_callback('pre_remove', 9, $rightfields, \&entry_pre_remove);

Defined in plugins. No Control on users end


Conclusion

Flat-filesvs.

Modules


Flat-files:☺ Easy to install (Just grab it)

☻ Hard to upgradeOK for simple plugins


Modules:☺ Full-access to Perl OO

goodness☺ Avoid duplicate efforts of CPAN ☻ Might be hard to resolve deps.

Subversion to the rescue(could be a barrier for newbies)


Nice-to-haves:Order control

Temporarily disable pluginsPer plugin config

Per plugin storage


Resources

Class::Triggerhttp://search.cpan.org/dist/Class-Trigger/Module::Pluggablehttp://search.cpan.org/dist/Module-Pluggable/Ask Bjorn Hansen: Build Easily Extensible Perl

Programshttp://conferences.oreillynet.com/cs/os2005/view/e_sess/6806qpsmtpdhttp://smtpd.develooper.com/MT pluginshttp://www.sixapart.com/pronet/plugins/Kwikihttp://www.kwiki.org/Plaggerhttp://plagger.org/

Documents

Writing Pluggable Software Tatsuhiko Miyagawa [email protected] Six Apart, Ltd. / Shibuya Perl Mongers YAPC::Asia 2007 Tokyo