Chalkboard: Mapping Functions to Polygons - ITTCandygill/talks/20090923-ifl-cb.pdf · Chalkboard: Mapping Functions to Polygons Andy Gill and Kevin Matlage The University of Kansas

  • Upload
    vuhanh

  • View
    221

  • Download
    1

Embed Size (px)

Citation preview

  • Chalkboard: Mapping Functions to Polygons

    Andy Gill and Kevin Matlage

    The University of Kansas

    September 23, 2009

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 1 / 19

  • ChalkBoard

    What is ChalkBoard?

    An Uncompromisingly Declarative description language,

    for two dimensional images,

    for the sake of visualizing information and ideas.

    ChalkBoard Images are Immutable Values.

    ... $ fmap (\ x -> if xthen greenelse white)

    $ rotate 0.05$ scale 50$ checker

    An efficient image generator will allow an efficient animation language.

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 2 / 19

  • Chalkboard Motivation

    Why another graphics language?

    DSLCapture

    FunctionalReactive

    ProgrammingGPUs

    ChalkBoard

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 3 / 19

  • Pan Images

    ChalkBoards main image structure is inspired by Conal Elliots Pan.

    type Image a = (R,R) -> a

    This function captures a 2-D plane of values.

    Examples from Pan.

    ... :: Image Bool -- region

    ... :: Image (R,R) -- location

    ... :: Image RGB -- Picture

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 4 / 19

  • Chalkboard Abstract Pan-style Images

    Gameplan for Chalkboard

    Board, an abstract version of Pans Image.

    Build a deep DSL around Board.

    Compile Board to an efficient graphics engine.

    ChalkBoard IR (CBIR) is a polygon pushing language.

    ChalkBoardImage

    Specification

    DeepDSL

    ChalkBoardIR

    OpenGL

    GPU

    DSL Capture

    & Compile// ChalkBoard

    Back End//

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 5 / 19

  • Primitives in Chalkboard

    Generatorspure :: a -> Board acoord :: Board (R,R)

    Moversscale :: Float -> Board a -> Board amove :: (Float,Float) -> Board a -> Board arotate :: Float -> Board a -> Board a

    Pointwise

    :: (a -> b) -> Board a -> Board b :: Board (a -> b) -> Board a -> Board bover :: (Over a) => Board a -> Board a -> Board a

    run

    lookup :: Board a -> (R,R) -> a

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 6 / 19

  • ChalkBoard Example

    ... = (\ (x,y) -> RGB (x + 0.5) (y + 0.5) 0) coord

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 7 / 19

  • ChalkBoard Overlay Example

    board :: Board RGB

    board = unAlpha (sq1 over cir1 over pure (alpha white))

    where

    cir1 :: Board (Alpha RGB)

    cir1 = move (0.2,0.2) $ choose (withAlpha 0.5 green) (transparent white) circle

    sq1 :: Board (Alpha RGB)

    sq1 = move (-0.2,-0.2) $ choose (withAlpha 0.5 red) (transparent white) square

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 8 / 19

  • Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 9 / 19

  • ChalkBoard: Conceptual Challenges

    Two parts to the creation of a complex board.

    Outer (Applicative) Functor Shape of the computation

    Small set of primitive combinators (, fmap, scale, etc.)Finite tree (perhaps a DAG) of computation.Capturing this part of the DSL is possible using IO-based observablesharing.

    Inner individual points or values.

    Infinite number of computations possible over points and values.Capturing these functions requires some new techniques.Still an open problem inside ChalkBoard.

    fmap and

    fmap :: forall a, b . (a -> b) -> Board a -> Board b

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 10 / 19

  • Big Idea in ChalkBoard

    Optimize what we can, generate what we can not!

    Many operations can be given algebraic style interpretations

    Draw a unit circle, in red, on white background

    fmap (\ x -> if x then red else white) circle

    Draw a unit circle, in red, on a green square, over white

    fmap (withDefault white) (cir over sqr)wherecir, sqr :: Board (Maybe RGB)cir = fmap (withMask red) circlesqr = fmap (withMask green) square

    What we can not optimize, generate an appropriately sized bufferdirectly, and use this in future computations.

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 11 / 19

  • ChalkBoard IR - An Imperative Polygon Pushing Language

    Main operations are

    Allocate Buffer

    Size, depth, background are specified.

    Splat Polygon

    Takes list of point maps.Splats a polygon from one buffer to another.

    Others include deallocation, import/export of images.

    Buffers are true first class objects in CBIR.

    Buffers can be used as both textures andtargets.

    Alpha blending is associative in CBIR.

    Source Buffer

    Target Buffer

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 12 / 19

  • Compiling to ChalkBoard IR

    Current implementation of our ChalkBoard DSL uses type functions.

    data Board a

    = PrimFun ((R,R) -> O a)

    | Trans Trans (Board a)

    | Join (Board a) (Joiner a) (Board a)

    | Extras (BoardExtra a)

    data Trans = Move (R,R) | Scale (R,R) | ...

    data family BoardExtra a

    data instance BoardExtra Bool = Polygon_Bool [(R,R)]

    data instance BoardExtra RGB = Fmap_Bool_RGB (O Bool -> O RGB) (Board Bool)

    | ...

    ...

    We explicitly call out all supported fmaps conversions in the typefunctions (for now).

    This is a significant limitation to DSL(requires transliteration from original ChalkBoard code)

    We can observe the functions inside these fmaps because they havethe type O a -> O b, where O is an observable value.

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 13 / 19

  • Observable Values

    An observable value is a value that can be both interpreted and observed.

    data O a = O a E

    newtype E = E (Entity E)

    data Entity s = Entity String [Expr s]data Expr s = Expr s

    | Lit R

    We use IO-based observable sharing tricks from Haskell09 here.

    Overloading arithmetic.

    Use combinators for pattern matching O values.

    lifted versions of red, green, etc.

    Small first order language can be extracted.

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 14 / 19

  • Compiling to CBIR (continued)

    In the compiler

    We think of Board Bool as a sequence of imperative True setters,over a False Buffer.

    We think of Board RGB as a sequence of (color drawing) imperativesetters, over a specific colored Buffer.

    . . .

    over becomes sequencing (with conditions).

    Transformations like rotate and scale are independent of theseinterpretations, and flow through to the primitives.

    We lose the ability to compile efficiently specifications like

    circle :: Board Boolcircle = (\ (x,y) -> x*x + y*y

  • ChalkBoard IR Virtual Machine

    We have written two engines for interpreting CBIR.

    Reference implementation

    Really slow.Uses ChalkBoard itself.

    OpenGL based optimized implementation.

    Much faster animation within reach.CBIR instructions are sent to an MVar.Instructions are then translated on the fly into a series of efficientOpenGL commands.The engine make appropriate choices about buffer implementation.Various hooks for debugging and monitoring.Basic animation support (frequent redraw).

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 16 / 19

  • OpenGL pipeline

    CommandsVertex

    OperationPrimitiveAssembly Rasterization

    FragmentOperation

    FrameBuffer

    TextureMemory

    // // // // //OO

    We use (the widely supported) Frame Buffer Objects.

    This allows the frame buffer to also be a texture map.

    This makes it much easier to implement implement splat.

    We implement a backwards compatible mode, where a singleframe buffer is re-used over and over, and pixels are copied intotextures automatically.

    We also compensate for the built-in alpha blending, making theact of splating associative.

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 17 / 19

  • Status

    We have basic graphicals on the screen, via our back end.

    We can import images as a Board.

    We can also write out popular file formats.

    We can write to the screen in real time.

    The generation is fast enough for small animations.

    Ken Burns style image panning and zooming works.Sprites (.png images)Wiggling lines

    Have a small number of libraries on top of ChalkBoard.

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 18 / 19

  • Future Work and Open Programs

    Improve compiler, via algebraic transformations

    Improve (or understand) fmap function capture.

    Use OpenGL texture functions to implement applicative functors.

    OpenCL and GPUs.

    Explore timezones inside FRP.

    type UI = Doublelifetime :: Behavior UI -- 0 to 1lifetime = Behaviour (\ t -> t)

    Fonts, and other libraries.

    Movie import (and export).

    Release ChalkBoard on hackage.haskell.org

    Questions?

    Andy Gill and Kevin Matlage Chalkboard: Mapping Functions to Polygons September 23, 2009 19 / 19