FBO – Image Processing GPGPU – Expressive Rendering

Christian Jacquemin, Master in Computer Science, OpenGL & GLSL, University Paris-Sud

Master – Computer ScienceUniversity Paris-Sud

Open GL & GLSL Course

FBO – Image Processing GPGPU – Expressive Rendering

Christian Jacquemin


GLSL Tutorial & Relief Techniques

(I)

Frame Buffer ObjectsMulti-pass Rendering

GPGPU Cellular Automata


Frame Buffer Object● 2 types of frame buffers

– default frame buffer: on-screen rendering– Frame Buffer Objects (FBOs): off-screen storage for

possible later reuse● FBOs can be bound as input (reading) or output

(rendering) of a rendering pass (an OpenGL context)● images (arrays of pixels) can be attached to FBOs

– FBO have several possible attachment point, we will only consider color attachments

– texture objects can hold multiple images need to →specify which image to attach to each attachment...but we only consider single image textures (2D or rectangle textures without mipmapping)


Frame Buffer Object● 2 main uses of frame buffer objects

– multipass rendering such as deferred shading:FBOs store intermediate geometry in so-called Geometry Buffers: normals, position, depthThe last pass is the frame buffer rendering for visual output

– GPGPU (General-purpose computing on graphics processing units) through multipass processing:FBOs store intermediate computationsSuch applications do not necessary have a visual output, instead the result can be read back in the CPU and used for other purposes than graphics.1 pass = a kernel applied in // on a data matrix


Multi-pass Rendering● FBOs can be used

for multi-pass rendering: the output of some passes can be used as input for later ones.

● One FBO cannot be both read and drawn


Case study: 2-pass Rendering

Example of a simple 2-pass rendering with 3 FBO attachment draw & read


Case study: 2-pass Rendering

Example of a simple 2-pass rendering with 3 FBO attachment draw & read


Frame Buffer Object programming● 3 FBO color attachments as output (1st pass) and as input (2nd pass)

FBO initialization (ID allocation and image attachments)// FBO ID + FBO texture ID allocation for the nb of attachmentsglGenFramebuffers( 1, &frameBufferId );glBindFramebuffer( GL_FRAMEBUFFER, frameBufferId );glGenTextures(NB_ATT, frameBufferTextureId);

// attaching images to FBOs (for each attachment)for( int ind = 0 ; ind < NB_ATT ; ind++ ) { glBindTexture(GL_TEXTURE_RECTANGLE, frameBufferTextureId[ind]);

// specify texture storage requirements glTexStorage2D(GL_TEXTURE_RECTANGLE,1,GL_RGBA32F,width, height);

// specify the texture attached to the FBO current attachment glFramebufferTexture2D( GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0 + ind , GL_TEXTURE_RECTANGLE, frameBufferTextureId[ind] , 0 );}

// specify the color attachment points of the draw bufferglDrawBuffers(NB_ATT, drawBuffers);glBindFramebuffer( GL_FRAMEBUFFER, 0 );


Frame Buffer Object programming● 3 FBO color attachments as output (1st pass)

FBO bindings // output FBO binding and input FBO unbindingglBindFramebuffer(GL_READ_FRAMEBUFFER, 0);glBindFramebuffer(GL_DRAW_FRAMEBUFFER, frameBufferId);

● Fragment shader outputs// output FBO binding and input FBO unbindinglayout (location = 0) out vec4 outColor0;layout (location = 1) out vec4 outColor1;layout (location = 2) out vec4 outColor2;


Frame Buffer Object programming● 3 FBO color attachments as input (2nd pass)

FBO bindings // texture unit bindingglUniform1i(uniform_texture_FBO1, 0); //Texture unit 0

// first input texture unit (FBO attachment #1)glActiveTexture(GL_TEXTURE0 + 0);glBindTexture(GL_TEXTURE_RECTANGLE, frameBufferTextureId[0]);

● Fragment shader uniform inputs// desaturated texturelayout (binding = 0) uniform sampler2DRect FBO1;// inverse colorlayout (binding = 1) uniform sampler2DRect FBO2;// gray level colorlayout (binding = 2) uniform sampler2DRect FBO3;


Case study: Ping-pong GPGPU

Example of a simple 2-pass GPGPU computation and result display:Cellular Automata computation and display (the case of Game of Life)


“Game of Life” Cellular Automaton

John Conway's Game of life is a Cellular Automaton based on Moore neighborhood and 2-state cells (dead or alive):● live cell && neighbors < 2 dies (under-population)→● live cell && neighbors == 2 || 3 survives→● live cell && neighbors > 3 dies (overcrowding)→● dead cell && neighbors == 3 alive (reproduction)→● dead cell && neighbors != 3 stays dead→

update is made in parallel from one generation to the nextevolution is based on the rules and a seed pattern


“Game of Life” Cellular Automaton

Extensively studied even though it is not necessarily the most interesting CASeveral configurations of cells have been highlighted:● Still patterns (pseudo static)● Oscillators (with various periods)● Gliders (also called spaceships) that reproduce by

translation over a period

http://en.wikipedia.org/wiki/Conway%27s_Game_of_Life

http://en.wikipedia.org/wiki/Conway's_Game_of_Life


“Game of Life” Computation Fragment Shader

Fragment shader implementation of “Game of Life”// Moore neighborhood alive cells countingint nb_neighbors = (texture(FBO,texCoordOut + vec2(1,0) ).r > 0? 1 : 0 ) + (texture(FBO,texCoordOut + vec2(1,0) ).r > 0? 1 : 0 ) + … + + (texture(FBO,texCoordOut + vec2(0,1) ).r > 0? 1 : 0 );

bool alive = (texture(FBO,texCoordOut).r > 0);

// dies or stays deadoutColor0 = vec4(0.0,0.0,0.0,1.0);

// live cell with two or three live neighbors survivesif( alive && (nb_neighbors == 2 || nb_neighbors == 3 ) ) { outColor0 = vec4(1.0,1.0,1.0,1.0);}

// dead cell with exactly three live neighbors becomes aliveif( !alive && nb_neighbors == 3 ) { outColor0 = vec4(1.0,1.0,1.0,1.0);}


“Game of Life” ScreenshotsScreen after seeding: red = new-born, white = surviver


“Game of Life” Screenshots

Close lookup after vertical seed

→possibility of interactive“re-seeding”



(II)

Real-time (Video) Image Processing


Real-time Image processing● General field: computer vision● Convolution for local image processing (filtering):

Output(i,j) = ΣmΣn Input(m,n) * Kernel(i-m,j-n)


Real-time Image processing● Example of convolution Kernel: Sobel filter● High pass filter (regions with high gradient)


Real-time Image processing● Example of convolution Kernel: blur filter● Low pass filter (regions with low gradient)


Convolution Fragment Shader

Convolution Kernel implementation in a Fragment Shader// predefined matrices for offsets (Moore neighborhood) // and kernels (Sobel)vec2 offsets[9] = { { 1, 1 }, { 0, 1 }, { 1, 1 }, { 0, 1 }, { 0, 0 }, { 0, 1 }, { 1, 1 }, { 0, 1 }, { 1, 1 } };

float sobelMatrixX[9] = { 1, 0, 1, 2, 0, 2, 1, 0, 1 };

// samples the center pixel and its Moore neighborhoodvec3 sampler[9];for( int i = 0 ; i < 9 ; i++ ) { sampler[i] = texture(video,texCoordOut + offsets[i]).xyz;}

// sobelvec3 sobelX = vec3(0.0);for( int i = 0 ; i < 9 ; i++ ) { sobelX += sobelMatrixX[i] * sampler[i];}


Real-time Video Processing

The video frames are sent to the GPU and processed by the shader before being displayed



(III)

Toon Shading


Toon shading● Toon shading belongs to “non-photorealistic” or

“expressive” rendering:rendering modes in which the colors are not made to resemble reality but for other purposes such as– imitate hand drawing or painting (e.g. toon shading)– visualize information (information or scientific

visualization)– assist artists or designers in their work...

● Toon shading tries to imitate cartoons, comics, mangas...but however uses underlying 3D geometry for shape, lighting...


Toon Shading Implementation● Toon shading can rely on

– shading quantization (also called posterization)– external contour highlighting– internal crest or crease such as wrinkles highlighting

● Techniques:– shading with a toon texture accessed through diffuse light

value

float diffuse = max(0.3,dot(normalOut,normalize(lightOut)));

frag_colour.rgb = texture(toonTexture,vec2(diffuse*1024,0)).rgb;


Toon Shading Implementation● Techniques:

– crest and crease highlighting by detecting the pixels at which the surface is almost orthogonal to the viewif( dot(normalOut , viewOut) < 0.2 ) { frag_colour.rgb = vec3(0.0);}

– contour highlighting through 2-pass rendering:● 1st pass: surface expansion along normals in vertex shadervec3 new_vp = vp + 0.02 * normalize( norm ); black rendering (without depth drawing)

● 2nd pass: normal rendering


Toon Shading Implementation● Mesh export from

Blender in obj format● Parsing and indexed

rendering + toon shading + external contour + internal relief

Mesh Copyright: Alien Head by Jonathan Williamson http://jw3d.com/

Documents

FBO – Image Processing GPGPU – Expressive Rendering