Embed Size (px)
Citation preview
Computation: Cleaner, Faster, Seamless, and Higher Quality
Alan EdelmanMassachusetts Institute of Technology
Unique Opportunity
• Raise the bar for all functions:– Linear Algebra!– Elementary Functions– Special Functions– Hard Functions: Optimization/Diff Eqs– Combinatorial and Set Functions
Unique Opportunity• Raise the bar for all functions:
– Linear Algebra!– Elementary Functions– Special Functions– Hard Functions: Optimization/Diff Eqs– Combinatorial and Set Functions– 0-d, 1-d, n-d arrays, empty arrays– Inf, Not Available (NaN)– Types (integer, double, complex) and embeddings
• (real inside complex, matrices insidse n-d arrays)• Methodology
– Work in Progress: Create a web experience that guides and educates users in the amount of time that it takes to say Wikipedia
– Futuristic?: The Semantic Web and Other Automations?
What the world understands about floating point:
• Typical user: In grade school, math was either right or wrong. Getting it “wrong” meant: bad boy/bad girl. (Very shameful!)
• By extension, if a computer gets it wrong “Shame, Bad computer! Bad programmer!”
What the world understands about floating point:
• Typical user: In grade school, math was either right or wrong. Getting it “wrong” meant: bad boy/bad girl. (Very shameful!)
• By extension, if a computer gets it wrong “Shame, Bad computer! Bad programmer!”
• People can understand a bad digit or two in the insignficant part at the end. We might say “small forward error,” but they think something more like fuzzy measurement error.
The Linear Algebra World
• Some of the world’s best practices.
• Still Inconsistent, still complicated.
• Never go wrong reading a book authored by someone named Nick, a paper or key example by Velvel, all the good stuff in terms of error bounds of LAPACK and papers from our community
Landscape of Environments
• Traditional Languages plus libraries– (Fortran, C, C++, Java?)
• Proprietary Environments– MATLAB®, Mathematica, Maple
• Freely Available Open Source– Scilab, Python(SciPy)(Enthought!), R, Octave
• Other Major Players– Excel, Labview, PVWave
Computing Environments: what’s appropriate?
• “Good” numerics?– Who defines good?– How do we evaluate?
• Seamless portability?– Rosetta stone?– “Consumer Reports?”
• Easy to learn?• Good documentation?• Proprietary or Free?
– “Download, next, next, next, enter, and compute?”
Numerical Errors: What’s appropriate?
• The exact answer?
Numerical Errors: What’s appropriate?
• The exact answer?
• What MATLAB® gives!
Numerical Errors: What’s appropriate?
• The exact answer?• What MATLAB® gives!• float(f(x))??? (e.g. some trig libraries)• Small Forward Error• Small Backward Error• Nice Mathematical Properties• What people expect, right or wrong• Consistent?• Nice mathematics?
Simple Ill Conditioning
• Matrix: >> a=magic(4);inv(a) Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND = 1.306145e-17.
• Statistics Function: “erfinv”erfinv(1-eps)Exact: 5.8050186831934533001812 MATLAB: 5.805365688510648 erfinv(1-eps-eps/2) is exactly
5.77049185355333287054782 anderfinv(1-eps+eps/2) is exactly 5.86358474875516792720766
condition number is O(1e14)
Some Conclusions?• MATLAB® should have had a small forward error and it’s been bad?• Nobody would ever ask for erfinv at such a strange argument?• MATLAB® should have been consistent and warned us, like it has
so nicely done with “inv” which raises the bar high, but has let us down on all the other functions?
• We have a good backward error so we need not worry?• Users either are in extremely bad shape or extremely good shape
anyway:– Have such huge problems due to ill-conditioning they should have
formulated the problem completely differently (as in they should have asked for the pseudospectra not the spectra!)
– The ill-conditioning is not a problem because later they apply the inverse function or project onto a subspace where it doesn’t matter
Discontinuity
• octave:> atan(1+i*2^64*(1+[0 eps]))ans =
• 1.5708 -1.5708
Discontinuity
• octave:> atan(1+i*2^64*(1+[0 eps]))ans =
• 1.5708 -1.5708
• Exact? tan(x+pi) is tan(x) after all so is it okay to return atan(x) + pi * any random integer?
• There is an official atan with definition on the branch cuts
±i(1,∞)Continuous everywhere else!
MATLAB: atan(i*2^509) is pi/2 and atan(i*2^510) is –pi/2 atan(1+i*2^509) is pi/2 as is atan(1+i*2^510)
QR?
• No different from atan• Good enough to take atan+pi*(random integer?)• Is it okay to take Q*diag(random(signs?))?
– There is an argument to avoid gratuitous discontinuity! Just like atan
– Many people take [Q,R]=qr(randn(n)) to get uniformly distributed orthogonal matrices! But broken because Q is only piecewise continuous.
– Backward error analysis has a subtlety. We guarantee that [Q,R] is “a” QR decomposition of a nearby matrix, not “the” QR decomposition of a nearby matrix. Even without roundoff, there may be no algorithm that gives “THIS” QR for the rounded output.
• Is it okay to have any QR for rank deficiency?
Snap to grid?
• Sin( float(pi) * n) = 0??? Vs. float(sin(float(pi)*n))
• Determinant of a matrix of integers?– Should it be an integer?– It is in MATLAB! Cleve said so on line
But is it right?• a=sign(randn(27)),det(a)• a = 1 1 1 1 -1 -1 -1 1 1 -1 1 -1 -1 1 -1 1 -1 1 -1 -1 -1 -1 -1 1 -1 -1 -1 1 -1 -1 1 -1 1 -1 -1 -1 -1 -1 1 -1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 1 1 1 -1 1 -1 -1 1 1 1 1 1 -1 1 -1 1 1 1 1 -1 -1 1 -1 -1 1 1 -1 1 1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 1 -1 -1 1 1 -1 -1 1 -1 -1 -1 -1 -1 -1 1 1 -1 -1 -1 -1 -1 1 -1 -1 1 1 1 -1 -1 -1 -1 -1 1 1 1 -1 1 1 1 -1 1 1 1 1 1 1 -1 1 -1 -1 1 1 -1 1 -1 -1 1 1 -1 -1 1 1 1 1 -1 1 -1 -1 -1 -1 -1 -1 -1 1 -1 1 -1 1 -1 1 -1 1 1 1 1 1 -1 1 -1 1 -1 -1 1 -1 1 1 1 -1 1 1 -1 -1 1 1 -1 -1 1 1 -1 1 -1 1 1 1 1 1 -1 -1 1 1 -1 -1 1 -1 1 1 1 -1 -1 1 -1 1 1 1 1 -1 1 1 1 -1 1 1 -1 1 1 1 -1 -1 -1 1 -1 1 1 -1 -1 -1 -1 1 1 1 1 1 -1 1 -1 1 1 -1 -1 1 1 1 1 -1 -1 -1 -1 -1 -1 1 1 -1 -1 -1 1 1 -1 -1 1 -1 -1 1 1 1 1 -1 1 1 1 -1 -1 1 1 1 -1 -1 1 -1 -1 -1 1 -1 -1 -1 -1 -1 -1 1 -1 1 -1 1 1 -1 1 1 1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 1 -1 1 -1 -1 1 1 -1 -1 1 -1 -1 1 1 -1 1 1 -1 1 1 -1 -1 -1 1 1 1 1 -1 -1 1 -1 1 1 -1 1 -1 -1 1 -1 -1 1 -1 1 -1 1 1 1 -1 -1 1 1 -1 -1 -1 -1 -1 1 1 1 1 1 -1 1 1 -1 -1 -1 -1 1 -1 1 -1 1 -1 -1 1 -1 -1 -1 -1 -1 -1 1 -1 1 -1 1 1 -1 1 -1 -1 1 1 1 -1 1 1 -1 1 -1 -1 1 1 -1 -1 1 -1 -1 1 1 -1 1 -1 1 1 -1 1 1 -1 1 1 -1 1 1 -1 1 1 1 -1 1 -1 1 -1 1 1 1 1 -1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 1 -1 1 -1 -1 -1 -1 1 1 -1 -1 1 1 -1 -1 -1 1 -1 1 1 1 1 -1 -1 -1 -1 1 1 -1 1 -1 1 -1 1 1 1 1 1 -1 -1 -1 -1 -1 -1 1 1 1 -1 -1 -1 1 1 -1 -1 -1 -1 -1 -1 -1 -1 1 1 1 -1 -1 1 -1 -1 1 -1 -1 1 1 -1 -1 -1 1 -1 -1 1 1 -1 -1 1 -1 -1 -1 1 -1 1 -1 -1 1 1 -1 1 -1 1 1 -1 1 -1 1 -1 -1 1 -1 1 1 1 1 1 -1 1 -1 1 -1 1 -1 1 1 1 1 1 1 1 1 -1 -1 -1 1 -1 -1 1 1 1 -1 -1 -1 1 -1 1 -1 -1 1 -1 -1 -1 -1 1 -1 -1 1 1 1 1 1 -1 1 1 1 1 1 -1 1 -1 1 -1 -1 1 1 -1 -1 1 1 1 -1 1 -1 -1 1 1 -1 1 1 1 -1 1 -1 1 -1 1 1 -1 1 -1 1 1 -1 1 -1 -1 1 -1 -1 1 1 1 -1 1 -1 -1 1 -1 1 1 -1 -1 1 1 1 -1 1 1 -1 1 1 1 1 1 -1 1 -1 1 -1 1 1 -1 1
• ans = 839466457497601
MPI Based Libraries
Typical sentence: … we enjoy using parallel computing libraries such as Scalapack
• What else? … you know, such as scalapack
• And …? Well, there is scalapack
• (petsc, superlu, mumps, trilinos, …)
• Very few users, still many bugs, immature
• Highly Optimized Libraries? Yes and No
Star-P
Some of the hardest parallel computing problems(not what you think)
• A: row distributed array (or worse)
• B: column distributed array(or worse)
• C=A+B
• Indexing:
• C=A(I,J)
LU Example
Scientific Software• Proprietary:
– MATLAB, Mathematica, Maple• Open Source
– Octave and Scilab– Python and R
• Also:– IDL, PVWave, Labview
• What should I use and what’s the difference anyway?• Seamless: how to avoid the tendency to get stuck with
one package and believe it’s the best• Seamless: portability,• Higher Quality: standards, accuracy
Documentation I would like
• Tell me on page 1 how to type in a 2x2 matrix such as
1 2
1+i nan
function [funct]=obj_func(v,T)sigalpha=v(1);T0=v(2);beta=v(3);f=v(4);R=0.008314;aN=sigalpha*f/2;aP=sigalpha*(2-f)./2;H=[-1000:2:1000]';[Hr Hc]=size(H);G0=zeros(Hr,1);
for i=1:Hr if H(i)<0 G0(i)=-2*beta*(H(i)./aN).^2+abs(beta)*(H(i)./aN).^4; else G0(i)=-2*beta*(H(i)./aP).^2+abs(beta)*(H(i)./aP).^4; endend
sth=exp(-G0./(R*T0));PT0=sth./(trapz(sth*(H(2)-H(1))));[Tr Tc]=size(T);for i=1:Tr sth2=PT0.*exp(-(1/R)*((1/T(i))-(1/T0)).*H); PT(:,i)=sth2./(trapz(sth2*(H(2)-H(1)))); Hi(i)=(1./(trapz(sth2*(H(2)-H(1)))))*trapz(H.*sth2*(H(2)-H(1))); H2(i)=(1./(trapz(sth2*(H(2)-H(1)))))*trapz((H.^2).*sth2*(H(2)-H(1))); Cpex(i)=(H2(i)-Hi(i).^2)./(R.*(T(i)^2));endfunct=Cpex';
function [funct]=obj_func(v,T)beta=v(3);f=v(4);R=0.008314;H=[-1000:2:1000]';HH =2*[ [-1000:2:-2]'/f;[0:2:1000]'/(2-f)]/v(1);G0=-2*beta*HH.^2 + abs(beta)*HH.^4;sth=exp(-G0/(R*v(2)));PT0=sth/sum(sth); EE=exp((1/v(2)-1./T)*H'/R);tt= EE*PT0;Hi=(EE*(H.*PT0))./tt;H2=(EE*(H.^2.*PT0))./tt;funct=(H2-Hi.^2)./(R*T.^2);
A Factor of 10 or so in MATLAB
• Life is too short to write ”for loops” • Trapz is just a “sum” if the boundaries are 0
• H(2)-H(1) never needed (cancels out)
• Sums are sometimes dot products
• Dot products are sometimes matvecs
Higher Quality
New Standards for Quality of Computation
• Associative Law:(a+b)+c=a+(b+c)
• Not true in roundoff
• Mostly didn’t matter in serial
• Parallel computation reorganizes computation
• Lawyers get very upset!
Accuracy: Sine Function Extreme Argument sin(2^64)=0.0235985099044395586343659…One IEEE double ulp higher: sin(2^64+4096)=0.6134493700154282634382759…over 651 wavelengths higher. 2^64 is a good test caseMaple 10 evalhf(sin(2^64)); 0.0235985099044395581
note evalf(sin(2^64)) does not accurately use 2^64 but rather computes evalf(sin(evalf(2^64,10))) Mathematica 6.0 Print[N[Sin[2^64]]]; Print[N[Sin[2^64],20]];
0.312821*
0.023598509904439558634MATLAB sin(2^64) 0.023598509904440
sin(single(2^64)) 0.0235985Octave 2.1.72: sin(2^64) 0.312821314503348*
python 2.5 numpy sin(2**64) 0.24726064630941769**R 2.4.0 format(sin(2^64),digits=17) 0.24726064630941769**
Scilab format(25);sin(2^64) 0.0235985099044395581214
Notes: Maple and Mathematica use both hardware floating point and custom vpaMATLAB discloses use of FDLIBM. MATLAB and extra precision MAPLE and Mathematica can claim small forward errorAll others can claim small backward error Excel doesn’t compute 2^64 accurately and sin is an error. Google gives 0.312821315
Ruby on http://tryruby.hobix.com/ gives -0.35464… for Math.sin(2**64). On a sun4 the correct was given with Ruby.Relative Backward Error <= (2pi/2^64) = 3e-19 Relative Condition Number = abs(cos(2^64)dx/sin(2^64))/(dx/2^64) = 8e20
One can see accuracy drainage going back to around 2^21 or so, by powers of 2
*Mathematica and python have been observed to give the correct answer on certain 64 bit linux machines. **The python and R above numbers listed here were taken with Windows XP. The .3128 number was seen with python on a 32 bit linux machine. The correct answer was seen with R on a sun4 machine.It’s very likely that default N in MMA, Octave, Python, and R pass thru to various libraries, such as LIBM, and MSVCRT.DLL.
Coverage: Sine Function Complex Support
• All packages seem good» Maple: evalf(sin(I))» Mathematica: N[Sin[I]]» MATLAB: sin(i) » Octave: “ “» Numpy: sin(1j)» R: sin(1i)» Ruby: require 'complex'; Complex::I; Math::sin(Complex::I)
» Scilab: sin(%i)» EXCEL: =IMSIN(COMPLEX(0,1))
Note special command in excel
Coverage: Erf Function Complex Support
python/scipy is the only numerical package good• Maple: evalf(erf(I)); 1.650425759 I• Mathematica: N[Erf[I]] 0. + 1.65043 I• MATLAB: erf(i) ??? Input must be real.
• Octave: “ “ error: erf: unable to handle complex arguments
• Python/Scipy: erf(1j) 1.6504257588j• R: pnorm(1i) Error in pnorm … Non-numeric argument …
•
• Scilab: erf(%i) !--error 52 argument must be a real …
Acceptable or not?
• tan(z+kπ)=tan(z)• atan officially discontinuous on branch cut• atan discontinuous off branch cut?• MATLAB does it• MATLAB highly inaccurate in some zones• Python, octave, R worse in some ways• Scilab very Good• QR does it? (unacceptable in my opinion)
Clones
• Just because Octave is a MATLAB “wannabe” (want to be) doesn’t mean they are the same
• Better some ways, worse in other ways
Type Casting
• Mathematica and Maple, type cast as the symbolic languages have excellent numerics
• MATLAB “the matrix laboratory” uses the same lapack and math libraries as everyone else
• MATLAB has 1500 functions some terrific some trivial
Optimization
• Would love to see the “Consumer Reports Style” Article
