Embed Size (px)
DESCRIPTION
Unix has always had a philosophy of composable tools, where one tool outputs to the next in a pipeline. But the technique of piping a *textual* stream of data, and having to extract data out of it is looking a bit long in the tooth. Microsoft (not historically an innovator in its shell environment :-) has stolen a march with its Powershell. Can we do better in Perl? With composable streams of objects? Written in a modern OO framework (Moose)? You bet we can! This talk was given at the London Perl Workshop 2009, http://conferences.yapceurope.org/lpw2009/talk/2456
Citation preview
Semantic Pipes
osfameron
London Perl Workshop 5 Dec 2009
Everyone likes pipes
Unix philosophy
“This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams,
because that is a universal interface.”– Doug McIlroy
http://www.faqs.org/docs/artu/ch01s06.html
Pipes
Foo foo fooBar bar barBaz baz baz
foo bar baz
generate
map
Pipes
Foo foo fooBar bar barBaz baz baz
Bar bar bar
generate
grep
Pipes
Foo foo fooBar bar barBaz baz baz
Bar bar barBaz baz bazFoo foo foo
generate
sort
ls
SYNOPSIS ls [OPTION]... [FILE]...
OPTIONS
--format=WORD across -x, commas -m, horizontal -x, long -l, ...
-h, --human-readable human readable sizes (e.g., 1K 234M 2G)
--hide=PATTERN do not list implied entries matching shell PATTERN
-r, --reverse reverse order while sorting
--sort=WORD none -U, extension -X, size -S, time -t, version -v
ls
SYNOPSIS ls [OPTION]... [FILE]...
OPTIONS
--format=WORD across -x, commas -m, horizontal -x, long -l, ...
-h, --human-readable human readable sizes (e.g., 1K 234M 2G)
--hide=PATTERN do not list implied entries matching shell PATTERN
-r, --reverse reverse order while sorting
--sort=WORD none -U, extension -X, size -S, time -t, version -v
grep
ls
SYNOPSIS ls [OPTION]... [FILE]...
OPTIONS
--format=WORD across -x, commas -m, horizontal -x, long -l, ...
-h, --human-readable human readable sizes (e.g., 1K 234M 2G)
--hide=PATTERN do not list implied entries matching shell PATTERN
-r, --reverse reverse order while sorting
--sort=WORD none -U, extension -X, size -S, time -t, version -v
sort
ls
SYNOPSIS ls [OPTION]... [FILE]...
OPTIONS
--format=WORD across -x, commas -m, horizontal -x, long -l, ...
-h, --human-readable human readable sizes (e.g., 1K 234M 2G)
--hide=PATTERN do not list implied entries matching shell PATTERN
-r, --reverse reverse order while sorting
--sort=WORD none -U, extension -X, size -S, time -t, version -v
map/format
ls | … ?
ls | grep
ls | wc -l
ls -ltr | … ?
total 3272-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 READMEdrwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp
ls -ltr | … ?
total 3272-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 READMEdrwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp
using cut
ls -ltr | … ?
total 3272-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 READMEdrwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp
cut … --bytes? or --delimiter?
ls -ltr | … ?
total 3272-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe˽˽˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README˽˽˽˽˽˽˽˽˽drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp˽˽
ls -ltr | … ?
total 3272-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe˽˽˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README˽˽˽˽˽˽˽˽˽drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp˽˽
OK, find...
● rather baroque● separate command● “consistent interface” ?
So who does do piping properly?
So who does do piping properly?
Windows Powershell
● originally called Monad Shell● (this is important)
● streams of .Net objects● flexible and consistent filter/map/sort
Disclaimer
● I've never used Powershell...
Disclaimer
● I've never used Powershell...● but the idea is genuinely exciting
Pipe.pm
package Pipe;use Moose;extends 'MooseX::App::Cmd'; 1;
pipe
#!/usr/bin/perluse lib 'lib';use Pipe;Pipe->run;
Pipe.pm
Pipe::Command::find
use Moose;extends 'Pipe::Command';use MooseX::Types::Path::Class qw(File to_File);use File::Next; sub go { my ($self, $session, $args) = @_; my @files = @$args; push @files, '.' unless @files; my $it = File::Next::files( @files ); $session->iterator( sub { if (my $file = $it->()) { return to_File($file); } else { return; } });}
Pipe::Command::grephas where => ( isa => 'Str', is => 'rw', documentation => 'string to match', );has where_sub => ( … );
sub filter { my ($self, $session, $iterator) = @_; my $where_sub = $self->where_sub; my @queue; return sub { # yucky code, go read HOP instead return pop @queue if @queue; { my @values = $iterator->() or return; @queue = grep { my $result = eval { $where_sub->($_) }; $result; } @values or redo; } return pop @queue; };}
Pipe in action
$ ./pipe find lib
List of files found!
lib/Pipe/Command.pm lib/Pipe/Command/find.pm lib/Pipe/Command/grep.pm lib/Pipe/Session.pm lib/Pipe.pm
Pipe in action
$ ./pipe find lib | ./pipe grep '$_->stat->size > 1000'
List of files found!
lib/Pipe/Command.pm
Pipe::Command::find
sub pretty_print { my ($self, $session) = @_; say "List of files found!\n\n"; my $it = $session->iterator; while (my @files = $it->()) { say join "\n", @files; }}
Pipe to an external tool
$ ./pipe find lib | cat --- command_stack: !!perl/hash:Pipe::Command::find app: !!perl/hash:Pipe arg0: pipe . . <snip> . --- !!perl/hash:Path::Class::File dir: !!perl/hash:Path::Class::Dir dirs: - lib file_spec_class: ~ volume: '' file: Pipe.pm file_spec_class: ~
Pipe to an external tool
$ ./pipe find lib –-pretty | cat
lib/Pipe/Command.pm lib/Pipe/Command/find.pm lib/Pipe/Command/grep.pm lib/Pipe/Session.pm lib/Pipe.pm
Pipesapp: !!Pipestack: generate blah blah blah
generate
Foo foo fooBar bar barBaz baz baz
Pipesapp: !!Pipestack: generate blah blah blah
generate
Foo foo fooBar bar barBaz baz bazI'm using YAML,
but could just as well be XML etc.
Pipesapp: !!Pipestack: generate blah blah blah
app: !!Pipestack: map generate
blah blah blah
generate
map
foo bar baz
Pipesapp: !!Pipestack: generate blah blah blah
app: !!Pipestack: map generate
blah blah blah
generate
map
app: !!Pipestack: sort map generate
blah blah blah
sort
bar baz foo
Pipesapp: !!Pipestack: generate blah blah blah
app: !!Pipestack: map generate
blah blah blah
generate
map
app: !!Pipestack: sort map generate
blah blah blah
sort
bar baz foo
●nothing left to pipe to (! -p)●or --pretty
Pipesapp: !!Pipestack: generate blah blah blah
app: !!Pipestack: map generate
blah blah blah
generate
map
app: !!Pipestack: sort map generate
blah blah blah
sort
bar baz foo
●nothing left to pipe to (! -p)●or --pretty
Pipesapp: !!Pipestack: generate blah blah blah
app: !!Pipestack: map generate
blah blah blah
generate
map
app: !!Pipestack: sort map generate
blah blah blah
sort
pretty_print
Database
● from mydb.foo | grep '$_->name =~ /Bob/' | select foo bar baz
Database
● from mydb.foo | # 1 billion rows? grep '$_->name =~ /Bob/' | select foo bar baz
“You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles. Do you understand this? And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat.”
“You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles. Do you understand this? And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat.”
Pipesapp: !!Pipestack: generate blah blah blah
app: !!Pipestack: map generate
blah blah blah
generate
map
app: !!Pipestack: sort map generate
blah blah blah
sort
Pipesapp: !!Pipestack: generate blah blah blah
app: !!Pipestack: map generate
blah blah blah
generate
map
app: !!Pipestack: sort map generate
blah blah blah
sort
Pipes Monadsapp: !!Pipestack: generate
app: !!Pipestack: map generate
generate
map
app: !!Pipestack: sort map generate
sort
generate>> map>> sort>> pretty_print (first [sort, map, generate])
Thank you● Details/links
● Early version of this talk given at NorthWestEngland.pm, 1st July 2009● email: [email protected]● http://github.com/osfameron/pipe/● http://github.com/rhaen/Bicycle-Workshop (nice MooseX::App::Cmd example)
● Images● CC-licensed, via Flickr (thanks for making your photos free to use!!)
– http://www.flickr.com/photos/rxmflickr/4102530508/ Pipe Dream (title picture), by Rishi Menon– http://www.flickr.com/photos/technodad/3827297755/ The Unix® License Plate takes a beach break, by technodad– http://www.flickr.com/photos/darwinbell/2422933100/ Pipe Dreams, by Darwin Bell– http://www.flickr.com/photos/heritagefutures/2106174676/ Urban Moose, by ausphoto!– http://www.flickr.com/photos/james_michael_hill/88311128/ Pipes, by james_michael_hill– http://www.flickr.com/photos/intherough/3244476512/ The Chain, by ...-Wink-...– http://www.flickr.com/photos/carlos/3224149/ Pipe, by Nuevo Anden
● Classic images, via Google– Einstein with pipe– I'm a PC, from the Mac ads– Ceci n'est pas une pipe– Einstein
