COMP20005

COMP20005 Engineering Computation

Additional Notes

Numeric Computation, Part A

c© The University of Melbourne, 2020

Lecture slides prepared by Alistair Moffat

COMP20005Algorithmic objectives

For symbolic processing (for example, sorting strings), desirealgorithms that are:

I Above all else, correct

I Straightforward to implement

I Efficient in terms of memory and time

I (For massive data) Scalable and/or parallelizable

I (For simulations) Statistical confidence in answers andin the assumptions made.

COMP20005Algorithmic objectives

For numeric processing, desire algorithms that are:

I Above all else, correct

I Straightforward to implement

I Effective, in that yield correct answers and have broadapplicability and/or limited restrictions on use

I Efficient in terms of memory and time

I (For approximations) Stable and reliable in terms of theunderlying arithmetic being performed.

The last one can be critically important.

COMP20005Example 1

Wish to compute

f (x) = x ·(√

x + 1−√x)

andg(x) =

x√x + 1 +

√x.

So write the obvious functions, and print some values...

I sqdiff.c

Hmmmm, why did that happen?

COMP20005Example 2

Wish to compute

h(n) =n∑

i=1

1

i

So write the obvious function, and print some values...

I logsum.c

Hmmmm, why did that happen?

COMP20005Pitfalls

In all numeric computations need to watch out for:

I subtracting numbers that are (or may be) closetogether, because absolute errors are additive, andrelative errors are magnified

I adding large sets of small numbers to large numbersone by one, because precision is likely to be lost

I comparing values which are the result of floating pointarithmetic, zero may not be zero.

And even when these dangers are avoided, numerical analysismay be required to demonstrate the convergence and/orstability of any algorithmic method.

COMP20005Numbers – bits, bytes, and words

Inside the computer, everything is stored as a sequence ofbinary digits, or bits.

Each bit can take one of two values – “0”, or “1”.

A byte is a unit of eight bits, and most computers a word isa unit of either four or eight bytes.

A word typically stores a set of 32 or 64 bits. Theinterpretation of that bit sequence depends on the type ofthe variable involved, and the representation used for thedifferent data types.

COMP20005Variables and sizes

The preprocessor “function” sizeof() can be supplied witheither a type or a variable, and at compilation time isreplaced by the number of bytes occupied by that type:.

I sizeof.c

COMP20005Binary numbers

In char, short, int, and long variables, the bits are used tocreate a binary number.

In decimal, the number 345 describes the calculation3× 102 + 4× 101 + 5× 100.

Similarly, in binary, the number 1101 describes thecomputation 1× 23 + 1× 22 + 0× 21 + 1× 20, or thirteen indecimal.

COMP20005Binary numbers

Binary counting: 1, 10, 11, 100, 101, 110, 111, 1000, 1001,1010, 1011, 1100, 1101, 1110, 1111, 10000, and so on.

With a little bit of practice, you can count to 1,023 on yourfingers; and with a big bit of practice, to 1,048,575 if youuse your toes as well.

There are two further issues to be considered:

I negative numbers, and

I the fixed number of bits w in each word.

COMP20005Integer representations

In an unsigned w = 4 bit system, the biggest value than canbe stored is 1111, or 15 in decimal.

Adding one then causes an integer overflow, and the result0000.

Integer values are stored in fixed words, determined by thearchitecture of the hardware and design decisions embeddedin the compiler.

COMP20005Integer representations

The second column of Table 13.3 (page 232) shows thecomplete set of values associated with a w = 4 bit unsignedbinary representation.

When w = 32, the largest value is 232 − 1 = 4,294,967,295.

COMP20005Integer representations

Bit patternInteger representation

unsigned sign-magn. twos-comp.0000 0 0 00001 1 1 10010 2 2 20011 3 3 30100 4 4 40101 5 5 50110 6 6 60111 7 7 71000 8 −0 −81001 9 −1 −71010 10 −2 −61011 11 −3 −51100 12 −4 −41101 13 −5 −31110 14 −6 −21111 15 −7 −1

COMP20005Sign-magnitude representation

To handle negative numbers, one bit could be reserved for asign, and w − 1 bits used for the magnitude of the number.

The third column of Table 13.3 shows this sign-magnitudeinterpretation of the 16 possible w = 4-bit combinations.

There are two representations of the number zero.

Adding one to INT MAX gives −0.

COMP20005Twos-complement representation

The final column of Table 13.3 shows twos-complementrepresentation. In it, the leading bit has a weight of−(2w−1), rather than 2w−1.

If that bit is on, and w = 4, then subtract 23 = 8 from theunsigned value of the final three bits.

So 1101 is expanded as1×−(23) + 1× 22 + 0× 21 + 1× 20, which is minus three.

COMP20005Twos-complement representation

The advantages of twos-complement representation are that

I there is only one representation for zero, and

I integer arithmetic is easy to perform.

For example, the difference 4− 7, or 4 + (−7), is worked outas 0100 + 1001 = 1101, which is the correct answer of minusthree.

Most computers use twos-complement representation forstoring integer values.

COMP20005Twos-complement representation

Revisiting an old program, it should now make more sense.

I overflow.c

Adding one the the biggest number in twos-complementgives the smallest number.

COMP20005Twos-complement representation

On a w = 32-bit computer the range is from−(231) = −2,147,483,648 to 231 − 1 = 2,147,483,647.Beyond these extremes, int arithmetic wraps around andgives erroneous results.

If w = 64-bit arithmetic is used (type long long), the rangeis −(263) to 263 − 1 = 9,223,372,036,854,775,807,approximately plus and minus nine billion billion, or 9× 1018

The type char is also an integer type, and using 8 bits canstore values from −(27) = −128 to 27 − 1 = 127

COMP20005Unsigned types

C offers a set of alternative integer representations, unsignedchar, unsigned short, unsigned int (or just unsigned),unsigned long, and unsigned long long.

Negative numbers cannot be stored.

But will get printed out if you still use "%d" formatdescriptors. Use "%u" instead, or "%lu", or "%llu".

COMP20005Unsigned types

C also provides low-level operations for isolating and settingindividual bits in int and unsigned variables.

These operations include left-shift (<<), right-shift (>>),bitwise and (&), bitwise or (|), bitwise exclusive or (^), andcomplement (~).

There are some subtle differences between int and unsigned

when bit shifting operations are carried out.

I intbits.c

COMP20005Unsigned types

Table 13.5 (page 236) gives a final precedence table thatincludes all of the bit operations. If in doubt, overparenthesize.

Variables that are declared as unsigned types store positivevalues only, and can be used to manipulate raw bit strings.

COMP20005Octal and hexadecimal

C also supports constants that are declared as octal (base 8)and hexadecimal (base 16) values. Beware! Any integerconstant that starts with 0 is taken to be octal:

int o = 020;int h = 0x20;printf("o = %oo, %dd, %xx\n", o, o, o);printf("h = %oo, %dd, %xx\n", h, h, h);

gives

o = 20o, 16d, 10xh = 40o, 32d, 20x

COMP20005Octal and hexadecimal

The standard Unix tool bc can be used to do radix conversions:

mac: bcibase=10obase=22511001obase=82531obase=162519

COMP20005Floating point representations

The floating point types float and double are stored as:

I a one bit sign, then

I a we-bit integer exponent of 2 or 16, then

I a wm-bit mantissa, normalized so that the leadingbinary (or sometimes hexadecimal) digit is non-zero.

COMP20005Floating point representations

When w = 32, a float variable has around wm = 24 bits ofprecision in the mantissa part. This corresponds to about 7or 8 digits of decimal precision.

In a double, around wm = 48 bits of precision aremaintained in the mantissa part.

COMP20005Floating point representations

For example, when w = 16, ws = 1, we = 3, wm = 12, theexponent is a binary numbers stored using we-bittwos-complement representation, and the mantissa is awm-bit binary fraction:

Number Number Exponent Mantissa Representation(decimal) (binary) (decimal) (binary) (bits)

0.5 0.1 0 .100000000000 0 000 1000 0000 00000.375 0.011 −1 .110000000000 0 111 1100 0000 0000

3.1415 11.001001000011· · · 2 .110010010000 0 010 1100 1001 0000−0.1 −0.0001100110011 · · · −3 .110011001100 1 101 1100 1100 1100

The exact decimal equivalent of the last value is−0.0999755859375. Not even 0.1 can be represented exactlyusing fixed-precision binary fractional numbers.

COMP20005Floating point representations

Floating point representations can also be investigated:

I floatbits.c

0.0 is 00000000 00000000 00000000 000000001.0 is 00111111 10000000 00000000 00000000

–1.0 is 10111111 10000000 00000000 000000002.0 is 01000000 00000000 00000000 00000000

10.5 is 01000001 00101000 00000000 0000000020.1 is 01000001 10100000 11001100 11001101

COMP20005More types

The (non-ANSI) extended types long long (64-bit integer)and long double (128-bit floating point value) might alsocome in useful at some stage.

And, to end with two programming jokes:

Why do programmers always get Christmas and Halloweenconfused? Because 31Oct = 25Dec !.

There are 10 types of people in the world: those who knowbinary, and those who don’t. (Boom boom).

COMP20005Root finding

Given a continuous function f (), determine the set of valuesx such that f (x) = 0.

Why? Function often represents a point of balance betweento opposing constraints (for example, gravity and frictionalresistance on falling object at terminal velocity).

Also required to find maxima and minima. Turning pointshave zero derivative.

Three main methods:

I graphical methods;

I bracketing methods; and

I open methods.

COMP20005Graphical methods

Plot the graph, and look for the roots.

Can use a graphing calculator, even if it can’t identify theexact roots.

Knowledge of general shape of function, and impreciseknowledge of roots is of benefit to more principled methods.

COMP20005Bisection method

A better algorithm uses bisection to rapidly locate anapproximation of the root.

Start with a pair of values x1 and x2 for whichf (x1)× f (x2) < 0.

At each step the mid-point xm = (x1 + x2)/2 of the currentrange is tested. Keep root sandwiched between two x valuesat which f (x) has opposite signs, by moving either x1 or x2

to xm.

I bisection.c

COMP20005Bisection method

f(x)

x1 mid

(x1,fx1)

(x2,fx2)

x2

COMP20005Bisection method

The function judges termination by two different conditions:

I Whether the root has been found to the requiredprecision (and not by a test of the form f(x)==0.0), and

I Whether some preset number of iterations has beenexceeded.

The latter is necessary to prevent endless non-convergingcomputation caused by a too-tight tolerance on theapproximation.

In floating point arithmetic, there are no continuousfunctions, and every function is discrete.

COMP20005Bisection example

If f (x) = x2 − 10, and x0 = 0 and x1 = 10, the sequence ofmidpoint estimates xm is

5, 2.5, 3.75, 3.125, 3.4375, 3.28125, 3.203125, 3.1640625

The correct value is x =√

10 = 3.16227766016.

This form of calculation can also be easily done with aspreadsheet.

COMP20005Bisection convergence

Each iteration adds one bit of accuracy, or a little under onethird of a decimal digit. This observation allows the numberof loop iterations to be bounded.

So 32 iterations will give an accurate float, and 64iterations an accurate double.

Bisection is robust and reliable, provided an initial interval isavailable. Can do this via iterative search through through xvalues, stepping with some suitable interval.

COMP20005False position

Rather than use midpoint, could also use proportions andtriangles to get a “better” next estimate,

xm =f (x2)x1 − f (x1)x2

f (x2)− f (x1).

This is the method of false position.

Need to be a bit more careful with termination test, best betis to use fabs(f(xm))<EPSILON, because x2-x1 might notconverge.

May require fewer loop iterations than bisection to reachsimilar level of precision.

COMP20005Convergence

In practice convergence speed is not a problem, 50 or 100loop iterations is nothing.

But both bisection and false position are unreliable (or mayfail completely) when multiple roots fall within preliminaryinspection interval.

COMP20005Open methods

A range of other options are also available that do notrequire an initial bracketing interval.

In the one-point iterative approach, the defining equation isrewritten to isolate (one instance of) x .

For example, if f (x) = ex − 10x + 1, solve for x via therewrite x = g(x) = (ex + 1)/10.

Then use this function to generate a sequence:

xi+1 = g(xi ) =exi + 1

10

COMP20005Fixed point iteration

The fixed-point iteration method converges when themagnitude of the derivative of the new function g() in thevicinity of the root (including the initial estimate) is lessthan one.

Starting at x0 = 1 gives 0.37, 0.15, 0.227, 0.2256, 0.22530,0.22527, 0.225266, 0.2252656, and so on.

But doesn’t converge for x0 = 3.54, which is just a littlebigger than another root.

COMP20005Newton-Raphson

If the slope of the function at the estimated root is known, itcan be used to compute a tangent to the curve.

Starting with x0 is an estimate of the true value x , computea sequence of better estimates via:

xi+1 = NR(xi ) = xi −f (xi )

f ′(xi ).

This approach requires that f is differentiable, and that thederivative is well-behaved over the interval in question. Butdoes not require that the absolute gradient be less than one.

COMP20005Newton-Raphson

The number of correct digits approximately doubles at eachiteration, provided certain conditions are met.

Gives faster convergence than bisection method, but also hasgreater risks – when f ′(x) ≈ 0, or when f ′(xi ) = 0 for anyxi , may have serious problems.

COMP20005Newton-Raphson

Suppose that f (x) = x2 − 10, and√

10 is to be computed.

In this case, f ′(x) = 2x , and hence

NR(x) = x − x2 − 10

2x=

1

2

(x +

10

x

).

This yields the sequence (assuming x0 = 1)

1.0, 5.5, 3.66, 3.196, 3.16246, 3.1622777, 3.16227766 .

(And the correct value is x =√

10 = 3.162277660168379.)

COMP20005Secant method

Combining the ideas of false position and Newton-Raphson,generate a sequence from two starting points (x0, y0) and(x1, y1):

xk+1 =ykxk−1 − yk−1xk

yk − yk−1

The tangent f ′(x) is approximated by computing thegradient between two points lying on the curve.

COMP20005Multiple simultaneous equations

Systems of equations also sometimes arise:

f1(x1, x2, x3, . . . , xn) = 0

f2(x1, x2, x3, . . . , xn) = 0

f3(x1, x2, x3, . . . , xn) = 0

· · ·fn(x1, x2, x3, . . . , xn) = 0

where the “roots” are vectors of n values.

COMP20005Multiple simultaneous equations

Example:

sin x1 − cos x2 = 0

x1 + x2 − 1 = 0

has solution (x1, x2) = (−1.85619, 2.85619).

A multidimensional version of NR can be used in a similariterative approach, making use of partial derivatives andhyperplanes.

COMP20005Root finding – summary

For simple well-behaved situations, use NR or bisection, bothare fine. NR is faster, but requires that the derivative beavailable. Difference in speed is likely to be inconsequential.

Where there may be multiple roots: beware, both methodsmay have issues.

Turning points near the root being sought can give divergentoutcomes with NR; and multiple roots can affect bothapproaches. So make sure you know the nature of thefunction first.

COMP20005Numerical integration

Given some continuous function f (x) and two limits, x1 andx2, compute ∫ x2

x1

f (x)dx .

Useful in a range of situations where volumes of objects arerequired, total energy in dynamic systems, and etc.

COMP20005One trapezoid

As a crude approximation, assume that f (x) is linearbetween the points (x1, f (x1)) and (x2, f (x2)), and computethe area of the trapezoid that is defined:

I = (x2 − x1)f (x1) + f (x2)

2.

But the error can be arbitrarily large. Considerf (x) = −x2 + k , with x1 = −

√k and x2 =

√k.

COMP20005Trapezoidal method

To get a better estimate, break the interval up into n stepseach of size h = (x2 − x1)/n. Then the area of each smalltrapezoid is worked out, and added to a total.

I =∑

trapezoids

=n−1∑i=0

h · f (x1 + h · i) + f (x1 + h · (i + 1))

2

=h

2

(f (x1) + 2

n−1∑i=1

f (x1 + h · i) + f (x1 + h · n)

).

COMP20005Trapezoidal method

The greater the value of n, the smaller the value of h, andthe better the approximation.

In theory, that is.

In practice, rounding errors arising from summation of manyvalues of comparable magnitude needs to be monitored.Techniques for progressive aggregation might be helpful.

And note that multiplication by h is factored out and onlyperformed once, avoiding another possible source ofrounding error.

COMP20005Simpson’s method

The trapezoidal approach fits a linear polynomial to pairs ofpoints. If n is even (or is doubled to make it even), can fit aquadratic to consecutive triples of points.

It turns out that if x1, x2, and x3 are evenly spaced, and ifP(x) = px2 + qx + r is fitted through (x1, f (x1)),(x2, f (x2)), and (x3, f (x3)), then∫ x3

x1

P(x)dx =x3 − x1

6(f (x1) + 4f (x2) + f (x3)) .

COMP20005Simpson’s method

Hence, Simpson’s method for numerical integration:

I =h

3

(f (x0) + f (x0 + h · n) +

4n−1∑

i=1,3,5,...

f (x0 + h · i) + 2n−2∑

i=2,4,6,...

f (x0 + h · i)

.

The improved approach has significantly better convergenceproperties. For a given level of mathematical error it allowsuse of a smaller n, meaning that rounding errors are lesslikely to intrude.

COMP20005Numeric interpolation

Suppose that instead of being given a function f (x), getgiven some pairs of points that lie on f : (x1, f (x1)),(x2, f (x2)), (x3, f (x3)), · · · (xn, f (xn)).

And f is not known.

Then how can the roots of f , or the integral of f over arange, be estimated?

How even can single values f (x) be estimated for x 6= xi?

COMP20005Linear interpolation

If n = 2, the only plausible estimate for f is as a linearcombination:

f̂1(x) = f (x1) + (x − x1)f (x2)− f (x1)

x2 − x1.

The function is fitted as a straight line that passes throughthe two specified points.

COMP20005Linear interpolation

When more that two points, an option is to construct apiecewise linear interpolation – for xi ≤ x ≤ xi+1, take

f̂1(x) = f (xi ) + (x − xi )f (xi+1)− f (xi )

xi+1 − xi.

That is, between each pair of consecutive (in terms ofposition along the x axis), assume a straight line.

Problem now is that f̂ is continuous, but f̂ ′ is not – at eachpoint xi , the derivative f̂ ′(xi ) may be undefined, as theremay be two limits.

COMP20005Newton quadratic interpolation

If wish f̂ to be differentiable, fit a polynomial of degree n− 1through the n points.

Suppose three points supplied, (x1, y1), (x2, y2), (x3, y3).

Three points define a unique quadratic. Just need to find it!

COMP20005Newton quadratic interpolation

Define:

f̂2(x) = b1 + b2(x − x1) + b3(x − x1)(x − x2)

Considering x = x1 implies b1 = y1, since f̂2(x1) = y1.

Substituting b1, and considering x = x2 implies

b2 =y2 − y1

x2 − x1,

since f̂2(x2) = y2.

COMP20005Newton quadratic interpolation

Finally, substituting b1 and b2, and considering x = x3

implies

b3 =

y3−y2x3−x2

− y2−y1x2−x1

x3 − x1

since f̂2(x3) = y3.

COMP20005Newton quadratic interpolation

To get a quadratic in the usual form f̂2(x) = px2 + qx + r ,expand the original:

f̂2(x) = b1 + b2(x − x1) + b3(x − x1)(x − x2)

to determine that

p = b3

q = b2 − b3x1 − b3x2

r = b1 − b2x1 + b3x1x2

COMP20005Example 1

Given these points:

xi yi1.00 -2.501.50 -1.502.50 3.50

Calculate a quadratic f̂2(x) = px2 + qx + r that passesthrough the three points.

COMP20005Example 2

Consider the function

g(x) = ln1

1 + x.

Find a quadratic approximation to g that matches g at thepoints x ∈ {1, 2, 4}.

COMP20005Newton polynomial interpolation

Over n points (xi , yi ), for 1 ≤ i ≤ n, define

F [i ] = yi .

Then from that basis recursively define

F [j , j − 1, . . . , i + 1, i ] =

F [j , j − 1, . . . , i + 1]− F [j − 1, . . . , i + 1, i ]

xj − xi.

Finally, define bj = F [j , . . . , 1] for j ∈ {1 . . . n}.

COMP20005Newton polynomial interpolation

Then, by construction, taking

f̂n(x) = b1 +

b2(x − x1) +

b3(x − x2)(x − x1) +

b4(x − x3)(x − x2)(x − x1) +

· · ·bn(x − xn−1)(x − xn−2) · · · (x − x2)(x − x1)

yields the required polynomial.

COMP20005Splines

The Newton polynomial is continuous and differentiable, butthe function is likely to diverge rapidly outside the zone forwhich points are supplied, and may also overshoot/return ifthere are step changes within the spanned zone.

In a spline, component polynomials are pieced together, witha section over the interval [xi , xi+1] connecting with similarsections left and right.

The simplest spline is the piecewise linear form.

COMP20005Splines

If the degree of the component polynomials is increased(from first degree linear), added freedom is added to thesolution space (compared to linear).

Additional requirements can then be added withoutover-constraining the system of equations:

I f̂ ′(xi ) can be required to be continuous; and

I f̂ ′′(xi ) can be required to be continuous.

The first is achieved in a quadratic spline, the second addedin a cubic spline.

COMP20005Splines

A cubic spline is continuous at each knot point, and its firstand second derivatives are also continuous. The two “spare”coefficients are usually set by taking f̂ ′′(x1) = f̂ ′′(xn) = 0.

The maths gets a (little) bit harder, and there are morecoefficients to be stored, plus (slightly) more complexevaluation.

But a cubic spline is normally visually attractive, givessmooth transitions of gradient, and has smaller errors onwell-behaved functions.