Semantic Pipes (London Perl Workshop 2009)

Semantic Pipes

osfameron

London Perl Workshop 5 Dec 2009

Everyone likes pipes

Unix philosophy

“This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams,

because that is a universal interface.”– Doug McIlroy

http://www.faqs.org/docs/artu/ch01s06.html

Foo foo fooBar bar barBaz baz baz

foo bar baz

generate

Bar bar bar

generate

Bar bar barBaz baz bazFoo foo foo

generate

SYNOPSIS ls [OPTION]... [FILE]...

OPTIONS

--format=WORD across -x, commas -m, horizontal -x, long -l, ...

-h, --human-readable human readable sizes (e.g., 1K 234M 2G)

--hide=PATTERN do not list implied entries matching shell PATTERN

-r, --reverse reverse order while sorting

--sort=WORD none -U, extension -X, size -S, time -t, version -v

OPTIONS

map/format

ls | … ?

ls | grep

ls | wc -l

ls -ltr | … ?

total 3272-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 READMEdrwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp

ls -ltr | … ?

using cut

ls -ltr | … ?

cut … --bytes? or --delimiter?

ls -ltr | … ?

total 3272-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe˽˽˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README˽˽˽˽˽˽˽˽˽drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp˽˽

ls -ltr | … ?

total 3272-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe˽˽˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README˽˽˽˽˽˽˽˽˽drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp˽˽

OK, find...

● rather baroque● separate command● “consistent interface” ?

So who does do piping properly?

Windows Powershell

● originally called Monad Shell● (this is important)

● streams of .Net objects● flexible and consistent filter/map/sort

Disclaimer

● I've never used Powershell...

Disclaimer

● I've never used Powershell...● but the idea is genuinely exciting

Pipe.pm

package Pipe;use Moose;extends 'MooseX::App::Cmd'; 1;

#!/usr/bin/perluse lib 'lib';use Pipe;Pipe->run;

Pipe.pm

Pipe::Command::find

use Moose;extends 'Pipe::Command';use MooseX::Types::Path::Class qw(File to_File);use File::Next; sub go { my ($self, $session, $args) = @_; my @files = @$args; push @files, '.' unless @files; my $it = File::Next::files( @files ); $session->iterator( sub { if (my $file = $it->()) { return to_File($file); } else { return; } });}

Pipe::Command::grephas where => ( isa => 'Str', is => 'rw', documentation => 'string to match', );has where_sub => ( … );

sub filter { my ($self, $session, $iterator) = @_; my $where_sub = $self->where_sub; my @queue; return sub { # yucky code, go read HOP instead return pop @queue if @queue; { my @values = $iterator->() or return; @queue = grep { my $result = eval { $where_sub->($_) }; $result; } @values or redo; } return pop @queue; };}

Pipe in action

$ ./pipe find lib

List of files found!

lib/Pipe/Command.pm lib/Pipe/Command/find.pm lib/Pipe/Command/grep.pm lib/Pipe/Session.pm lib/Pipe.pm

Pipe in action

$ ./pipe find lib | ./pipe grep '$_->stat->size > 1000'

List of files found!

lib/Pipe/Command.pm

Pipe::Command::find

sub pretty_print { my ($self, $session) = @_; say "List of files found!\n\n"; my $it = $session->iterator; while (my @files = $it->()) { say join "\n", @files; }}

Pipe to an external tool

$ ./pipe find lib | cat --- command_stack: !!perl/hash:Pipe::Command::find app: !!perl/hash:Pipe arg0: pipe . . <snip> . --- !!perl/hash:Path::Class::File dir: !!perl/hash:Path::Class::Dir dirs: - lib file_spec_class: ~ volume: '' file: Pipe.pm file_spec_class: ~

Pipe to an external tool

$ ./pipe find lib –-pretty | cat

lib/Pipe/Command.pm lib/Pipe/Command/find.pm lib/Pipe/Command/grep.pm lib/Pipe/Session.pm lib/Pipe.pm

Pipesapp: !!Pipestack: generate blah blah blah

generate

Foo foo fooBar bar barBaz baz bazI'm using YAML,

but could just as well be XML etc.

app: !!Pipestack: map generate

blah blah blah

generate

foo bar baz

blah blah blah

generate

app: !!Pipestack: sort map generate

blah blah blah

bar baz foo

blah blah blah

generate

blah blah blah

bar baz foo

●nothing left to pipe to (! -p)●or --pretty

blah blah blah

generate

blah blah blah

bar baz foo

●nothing left to pipe to (! -p)●or --pretty

blah blah blah

generate

blah blah blah

pretty_print

Database

● from mydb.foo | grep '$_->name =~ /Bob/' | select foo bar baz

Database

● from mydb.foo | # 1 billion rows? grep '$_->name =~ /Bob/' | select foo bar baz

“You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles. Do you understand this? And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat.”

blah blah blah

generate

blah blah blah

generate

blah blah blah

Pipes Monadsapp: !!Pipestack: generate

generate

generate>> map>> sort>> pretty_print (first [sort, map, generate])

Thank you● Details/links

● Early version of this talk given at NorthWestEngland.pm, 1st July 2009● email: osfameron@cpan.org● http://github.com/osfameron/pipe/● http://github.com/rhaen/Bicycle-Workshop (nice MooseX::App::Cmd example)

● Images● CC-licensed, via Flickr (thanks for making your photos free to use!!)

– http://www.flickr.com/photos/rxmflickr/4102530508/ Pipe Dream (title picture), by Rishi Menon– http://www.flickr.com/photos/technodad/3827297755/ The Unix® License Plate takes a beach break, by technodad– http://www.flickr.com/photos/darwinbell/2422933100/ Pipe Dreams, by Darwin Bell– http://www.flickr.com/photos/heritagefutures/2106174676/ Urban Moose, by ausphoto!– http://www.flickr.com/photos/james_michael_hill/88311128/ Pipes, by james_michael_hill– http://www.flickr.com/photos/intherough/3244476512/ The Chain, by ...-Wink-...– http://www.flickr.com/photos/carlos/3224149/ Pipe, by Nuevo Anden

● Classic images, via Google– Einstein with pipe– I'm a PC, from the Mac ads– Ceci n'est pas une pipe– Einstein

Semantic Pipes (London Perl Workshop 2009)

Technology

Modern Perl for the Unfrozen Paleolithic Perl Programmer

Perl/DBI - accessing databases from Perl

Database Programming with Perl - Perl Training Australia

Perl 101 - The Basics of Perl Programming

Perl 1997 Perl As A System Glue

CGI & Perl Programming Language What is CGI? Introduction of Perl Programming Language –Simple Peal program –Perl variables –Perl Operators –Flow Controls

Perl family: 15 years of Perl 6 and Perl 5

Unix, Perl and BioPerl - barc.wi.mit.edubarc.wi.mit.edu/education/bioinfo2005/unix-perl/slides/Unix_Perl_3... · Unix, Perl and BioPerl III: Sequence Analysis with Perl - Modules

Intermediate Perl · Related titles Learning Perl Programming Perl Advanced Perl Programming ... that spark revolutionaryindustries. We specialize in docu-menting the latest tools

Perl Perl 24 Hours

Advanced Perl WUC 2006. Perl Poetry Variable Scope

Programming Perl - Perl Training · This is version 1.39 of Perl Training Australia’s "Programming Perl" training manual. Table of Contents 1. About Perl Training Australia

Learning To Program With Perl - Babraham Bioinf Introduction.pdf · Learning to program with Perl 4 Section 1: Getting Started with Perl What is Perl / perl? Perl is a high-level

Learning Perl - Amazon S3 · 2019-02-26 · Perl Cookbook Perl Debugger Pocket Reference Perl in a Nutshell Perl Testing: A Developer’s Notebook Practical mod-perl Perl Books Resource

1 The Refined Semantic Network James Geller Yehoshua Perl New Jersey Institute of Technology

Perl Power Tools - Saint Perl 6

Brno Perl Mongers 28.5.2015 - Perl family by mj41

Use Perl like Perl

Semantic Pipes- Web 3.0 Information Systems · Columbia University bookinfo

1 Introduction to Perl and Perl Syntax Learning Perl, Schwartz