Email: lijian.pku@gmail - 统计之都 · 1 ó 2 ˜:‘z R˙S +n ... ØubigmemoryØŒ§¢yapply!kmeans!lm!colmean˜’ {bigtabulate ¢y„Ý ˙table!tapply˜’{bigalgebra ¢y„Ý

�ó Ä:`z â»S�� \�BLAS ¿1O� �ïÆ

R�p5U$�{0Ú¢~

oé

Email: [email protected]

Homepage: http://www.jliblog.com/

1o3¥IR�ó¬Æ£�®¬|¤

2011 c 5 �

the 4th Chinese R Conference R and HPC


8 ¹

1 �ó

2 Ä:`zR�S�+ncompiler

3 â»S��bigmemory[xÄuêâ¥�`z

4 \�BLASBLAS0�Benchmarking

5 ¿1O�Ä:�£wª¿1Ûª¿1MapReduce

6 �ïÆ



8 ¹

1 �ó





6 �ïÆ



8 ¹

1 �ó





6 �ïÆ



8 ¹

1 �ó





6 �ïÆ



8 ¹

1 �ó





6 �ïÆ



8 ¹

1 �ó





6 �ïÆ



Some truths

S�ó��Ong

Computer time is inexpensive in comparison with personneltime

<��m[�'Åì��m�B

R��¯¢

)º.�ó

êâ�ÜÖ\S�

ü�§



J,5U��{

XÚ,?

,?M�

¦^64 ö�XÚ

|^GPU

u^�O�

mu�¡�`z

�{

N^C½öFortran

¦^�¡�`z

¿©|^R�SÜÅ�))R�Ä:`z

�êâ�?n))â»S��

OrR�Ý$�))\�BLAS

¿1O�


�ó Ä:`z â»S�� \�BLAS ¿1O� �ïÆ R�S�+n compiler

S��

32 XÚ

ö�XÚU|±��S�� 232

10243 = 4G

R332 WindowsXÚ¥��¦^S��3G

Ï~R3S�¦^�L 2G£Windows¤½ 3G£Unix¤�Ò¬��

64 XÚ

ö�XÚU|±��S�� 264

10243 = 172× 108G

32 �R3�64 �WindowsXÚe��S��k4G

RU¦^�S�É�uM�Úö�XÚ

duvk64 ��.êâ(�§Ã{½ÂL��Ý



cannot allocate vector of size

memory.size(max = FALSE)

memory.size(NA)§�wXÚ©��R��S�

memory.size(F)§�w�c®²¦^�S�

memory.size(T)§�w®©��S�

memory.limit(size = NA)

memory.limit()§�wXÚ©��R��S�

memory.limit(2000)§©��S��2G

Rgui –max-mem-size 2Gb



)R�S�

S�y©

æS�£Heap¤§Ä�ü�´/Vcells0§z��8i!

/�é£cons cells¤§��ü��332 XÚ¥´28i!!64 XÚ¥´56i!

R��;�ª

storage.mode£x¤

éu�.Ý§storage.mode(x¤< − ”integer”

Ré�S�

ls()§�w�cé�

object.size(x)§�wé�¤Ó^�S�



R�-Ã�n

rm()Úgc()

rm()�Øé��Ú^

gc()�×S��m§?1-Ã�n

§S¥�-Ã�n

éu�ÝO\�Ý§¦þk½Â��Ý§,�ÅÚO\

5¿�Ø¥mé�



compiler�

R 2.13.0�#A5§compiler�compiler�´R 2.13.0g��IO�§§�±r�ãR�è?È¤i!è§l 3�1�J,�Ç"

«~

> source("rhpcbj4.R")

> require(compiler)

> la1c <- cmpfun(la1)

> y <- 1:100

> system.time(for (i in 1:1000) la1(y, is.null))

user system elapsed

0.32 0.00 0.33

> system.time(for (i in 1:1000) la1c(y, is.null))

user system elapsed

0.12 0.00 0.13


�ó Ä:`z â»S�� \�BLAS ¿1O� �ïÆ bigmemory[x Äuêâ¥�`z

Äu©��bigmemory[x

bigmemory

ïáÄu©��Ý§¢ynwitch!order��{

biganalytics

éubigmemoryé�§¢yapply!kmeans!lm!colmean��{

bigtabulate

¢y�Ý�table!tapply��{

bigalgebra

¢y�Ý�BLASÚLAPACK8cLinuxe®U¦^

synchronicity

¢yBoost mutex�õU



1Z�^P¹60�Cþ�È�Double.êâ�«~



R�êâ¥��p

~�êâ¥��

RODBC

DBI

ROracle, RMySQL, RPostgreSQL, RSQLite

¦^RSQLite?1êâ?n

�|�þ�'X.S�êâ¥

ÏL¢Ú`z?n�êâ

RSQLite.extfuns�Jø*Ð¼ê



êâ¥�û�|±

RODM

Oracle Data Mining (ODM) Jø�R��§�±3R¥Ûª/N^ODMS��êâ�÷�{"

http://cran.r-project.org/web/packages/RODM

/index.html

TeradataR

Teradata Warehouse Miner ÚR��§�RJø20�Úêâó¥�p�Ä:¼ê±944�©Û¼ê"

http://downloads.teradata.com/download/applications

/teradata-r/1.0



non-sql.êâ¥£SN�Ö¿¤

DÚ�'X.êâ¥�¯Ö.�êâI¦ )

�I¦�é{¤êâ?1©Û��ÿ§/'X0�k^íº


�ó Ä:`z â»S�� \�BLAS ¿1O� �ïÆ BLAS0� Benchmarking

BLASÚLAPACK

BLAS

Basic Linear Algebra Subprograms§Ä:�5�ê§S8

http://www.netlib.org/blas/

Ä:��5�êö�§�þÚÝ¦{�

C. L. Lawson, R. J. Hanson, D. Kincaid, and F. T. Krogh,Basic Linear Algebra Subprograms for FORTRAN usage, ACMTrans. Math. Soft., 5 (1979)

LAPACK

Linear Algebra PACKage

�6BLAS§)õ��5�§ª!�5XÚ�§|��²�)!O�A��þ!^uO�ÝQR©)�Householder=�!±9ÛÉ�©)�



Atlas

Automatically Tuned Linear Algebra System

�éA½²�?1`z

netlibþ`zBLAS�ó§§X8®²£�http://math-atlas.sourceforge.net/

R��¢yµhttp://mirrors.geoexpat.com/cran/bin/windows

/contrib/ATLAS/

Linux�¸e£Ubuntu 10.04 64 §eÓ¤

Atlas 3.6.0§ü�§

Atlas 3.9.25 õ�§

sudo apt-get install libatlas3gf-base

sudo wajig install libatlas39 3.9.25-1 amd64.deb



MKL

IntelJø�BLAS�LAPACK��

Linuxek�û^��¤�

http://software.intel.com/en-us/articles/intel-

mkl/

Linux�¸e?ÈSC

XiaoNan�Æ�k�[�SC0�µhttp://www.road2stat.com/cn/r_language

/optimize.html

�{ü�SC�ªµrevolution-mkl

sudo apt-get install revolution-mkl r-revolution-revobaserevolution-r



Goto BLAS

Kazushige Goto £�BÚj¤mu�BLAS

|±õ�§

â`´8c�¯�BLAS

WindowseSC�{

http://prs.ism.ac.jp/nakama/kR��BLASJøe1

LinuxeSC�{

�http://www.tacc.utexas.edu5þáÒapt-get install gotoblas2-helperLogin(register) http://www.tacc.utexas.edu/?id=402vi /etc/gotoblas2-helper/gotoblas2-site.conf/etc/init.d/gotoblas2-helper start



�Ö¿


�ó Ä:`z â»S�� \�BLAS ¿1O� �ïÆ Ä:�£ wª¿1 Ûª¿1 MapReduce

�o´¿1

¿1O�

Ó�?1ØÓ�O�

üØõ�§

õØ½õCPU�¿1

wª¿1

d^r��¿1§I�3�{þ�;��?n

Ûª¿1

XÚgÄ?1�¿1?n



MPI

�o´MPIMessage Passing Interface§�«�ED4��

§SÏLÂuè�p��E?1�p

MPI¢yÅ�



snow[x

snow

�±¦^MPI!NWS!PVM!Socketso«D4�ª?1¿1

3õØ½öO�Å8+þ¢y¿1O�

snowfall

ésnow?1{z�C��

http://www.wentrue.net/blog/?p=878



Ûª¿1

õ�§BLAS

gÄõØ?1�ê$�

multicore

ÏLaqlapply��ª ©?Ö

gÄ©��õ�ØO�

doMCÚforeach

doMC5þõØ

foreach��Ì�§©��õØ$�



«~



MapReduce

Google��;|��

2010c1�¼1§?Ò�7 650 331§¶�System and methodfor efficient large-scale data processing£p��5�êâ?n¤"´Google�Ú�gÍ�¤J��§�´�O��Ø%Eâ��"

MapReduce�A^

GoogleÄ:A^

äm|¢

Amazon�Elastic MapReduceÑÖ

m �8Apache Hadoop



RHIPE{0

m �MapReduceµHadoop

Hadoop ´Google MapReduce ��Java¢y

½ÂMapper§?nÑ\�Key-Valueé§ÑÑ¥m(J"½ÂReducer§�À§é¥m(J?15�§ÑÑ�ª(J"½Âmain¼ê"J�JOB§XÚgÄ�¤

RÚHadoop��ÜµRHIPE

m �8§òRÚHadoop8¤3�å

8c�kLinuxÚMac OS��



¢Sö��:ïÆ

XJÝ$�'�õ

��BLAS

10Gêâ±e¦^bigmemory

°þêâ}ÁHadoop

XJêâ?n'�õ

¦^êâ¥§`z�Î

}Á¿1

`z�{´��



Thank you!Email: [email protected]

Homepage: http://www.jliblog.com/


Documents

Email: lijian.pku@gmail - 统计之都 · 1 ó 2 ˜:‘z R˙S +n ... ØubigmemoryØŒ§¢yapply!kmeans!lm!colmean˜’ {bigtabulate ¢y„Ý ˙table!tapply˜’{bigalgebra ¢y„Ý