Removing False Dependencies to Speedup Software Build Processes

Yu, Dayani-Fard, Mylopoulos CASCON 2003

1

Removing False Dependencies to Speedup Software Build

Processes

Yijun YuHomy Dayani-FardJohn Mylopoulos

IBM Toronto LabUniversity of Toronto


2

Background Phenomenon/Problem: large C/C++ software takes

much time to be built and rebuilt; in the development processes, debugging, testing, integration, releasing all have to wait for the binary image…

Observation from IBM Toronto lab: it takes > 7 hours to fully compile a system of > 10 million LOC.

Time is moneyCost = Single build time x Number of People x Build freq. … and more Why? Excessive compile-time dependency among

headers and implementations. Well-known. Light weight solution? makedepend cleanup target

dependencies (~0%), removing unused header inclusions and shorten the inclusion chain (estimated by KLOCwork, 37%)


3

Our approach Finding false dependencies among

finest grain program units Removing them Recent case studies (3 months after

reported in the CASCON paper)• VIM 6.2, 269 KLOC in C (complete result)

• A large commercial software package, 1580 KLOC in C++ (new result)

Future work and collaborations


4

1. False dependencies A PU X depends on a PU Y if X is using Y and Y occurs before X

We consider a program unit (PU) as a declaration or a definition at program or file scope. • Examples: function, variable, typedef, union,

struct, class, template, enum, macro• Anti examples: data field, member function, auto

variables, blocks, etc. False dependency: dependency at inclusion

level, no dependency at program unit level


5

An analogy Chemical properties are kept in molecule

structures, physical properties are kept in atom structures

Moving molecules doesn’t change the smell of the code, but replacing atoms in the molecule does!

Risk of change •2H2 + O2 = 2H2O + heat

•E = m c 2 leads to a nuclear bomb!


6

main

foo.h main.c

foo

bar

true

false

Fileboundar

y

DefineUnit

DeclareUnit

remove

#include

#include “foo.h”int main() { foo();}

void foo();void bar();


7

Negative impacts

•Larger preprocessed image

•Longer compilation time

•False couplings among program units, components and packages


8

2. Extracting and removing false dependencies asg2dep: Extracting PU dependencies from

abstract syntax graph • based on parser like Datrix, KLOCwork• dependency graph + locations

header: Partitioning the dependency graph (algorithm in the paper) and generating code

• Removing unused declarations• Grouping declarations used by the same definitions

into the same header files• Location is output as a #line directive

Regression test and measurements


9

Example (original)

MAIN

A

B

C

D

F

G

H

I

1.c 2.c 3.c main.c

c

header.h

E

a b d e f g h i


10

Example (restructured)

MAIN

A

B

C

D

F

G

H

I

1.c 2.c 3.c main.c

a

b

c

d

f

e

gi

h

header.hE


11

3. Case study: VIM 6.2include relations


12

Program units dependencies


13

Preprocessed image size

0

100000

200000

300000

400000

500000

600000

700000

ori g-si zenew-si ze

Buffer419 dcl.

charset

154 dcl.

gui_gtk

if_cscope


14

Build time

0

0. 5

1

1. 5

2

2. 5

3

3. 5

4

4. 5

ori g- t i menew- ti me


15

Synch-and-Stabilize Continually synchronize parallel

teams Periodically stabilize the product in

increments versus once at the end Also known as:

•Milestone, daily build, nightly build, zero-defect process


16

0

500000

1000000

1500000

2000000

2500000

3000000

3500000

4000000

ckdev

apiRollforward

denv

dump2

gart

gchd

gicl

giue

glsn

golf

grlg

gtsp

init

ldwi

lgpp

lnfd

luhd

rchk

rfrt

rpar

rupe

sttr

tlog

trmv

txid

xcm2

xrec

or i g_si zenew_si ze

Case study 2. A C++ package in IBM preprocessed image size


17

0

2

4

6

8

10

12

14

16

18

ckdev

apiReadLogNoConn

chkd

draw

gadf

gcal

gcht

gicl

gint

gLogFileMgmt

gmrp

grbk

grpd

gulf

init

ldmp

lgdd

lkss

lrl

PatchLog

recm

rfrt

RollforwardSE

rudm

sque

tfrg

tmadr

trsp

txid

xcm1

xprp

or i g_t i menew_t i me

Build time


18

Cumulative build time of directly changed .C files

0. 00

20. 00

40. 00

60. 00

80. 00

100. 00

120. 00

140. 00

160. 00

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Savi ng bui l d t i me for the changed C fi l es

Ol d t i meNew ti me


19

Cumulative build time of recompiled .C files

0. 00

200. 00

400. 00

600. 00

800. 00

1000. 00

1200. 00

1400. 00

1600. 00

1800. 00

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

Savi ng bui l d t i me for C fi l es that depends on the changed headers

Ol d t i meNew ti me


20

4. Summary False dependencies can lower

productivity, thus are harmful to both managers and developers

Removing false dependencies in C++ requires supporting tools

Case studies have shown significant improvements (demo: VIM results can be downloaded from http://www.cs.toronto.edu/~yijun)


21

5. Future work and possible collaborations Integrate with IBM development on demand Leverage KLOCwork’s analysis tools Potential collaborations with CSER and CASCON

members (give the grass, milk the cow)• UVictoria and UAlberta: visualize dependencies to assure

programmers that there is no MAGIC behind the scene• UWaterloo: speedup the parser to meet the everyday need

in the restructuring: Datrix + ALV? Bring 45 minutes per file down to 45 /184 minute

• UWaterloo and UQueens: Quantitative goal modelling• UWindsor: web-services compositions and repackaging• UGhent: Link-time optimizations: Diablo: shrink VIM

memory footprint to 1/3

Documents

Removing False Dependencies to Speedup Software Build Processes