BIO5312 Biostatistics Lecture 14: Creating R Packages · res = list(x=x,y=y,coefficients = coef) res} Load the function Yujin Chung Lec14: R Packages Fall 2016 22/33. R function LSE

BIO5312 BiostatisticsLecture 14: Creating R Packages

Yujin Chung

December 6th, 2016

Fall 2016

Yujin Chung Lec14: R Packages Fall 2016 1/33

Previous

Maximum likelihood approach

MLEs

Analytic solution for MLE

Model comparisonsI LRTI AIC


Today’s lecture

Numerical OptimizationI Derivative-based approach

F The Newton-Raphson methodF The Quasi-Newton methodF Constraints

I Derivative-free approachF Nelder-Mead

Creating R packages


Derivative-Based Optimization

Newton-Raphson method (or Newton method, Gradient method)

a powerful technique for solving equations numericallybased on a quadratic Taylor expansionwant to solve f(x) = 0 for x (e.g., f(x) is a derivative oflog-likelihood function and x is a parameter)

The Newton-Raphson Iteration

1 x1 is an initial value

2 The tangent line to y = f(x)at the point (x1, f(x1)) hasequationy = f(x1) + f ′(x1)(x− x1).

3 x2 is the x-intercept (sety = 0) of the tangent line.Then x2 = x1 − f(x1)/f

′(x1)

4 Repeat (1) and (2) untilconverge. x

r x2 x1

y=f(x)

y=f(x1)+f'(x)(x−x1)

y=f(x2)+f'(x)(x−x2)

y


The Newton-Raphson method: example

Consider X1, . . . , X50 ∼ N(µ, 1) and find the MLE of µ.

Log-likelihood: l(µ) =n

2log(2π)−

∑50i=1(xi − µ)2

2

The first derivative: l′(µ) =∑50

i=1(xi − µ)

Let l′(µ) = 0 and solve it for µ. Then the MLE is X̄.

l′′(µ) = −nApply the Newton-Raphson method to find the MLE:

# simulate a sample

> x = rnorm(50,5,1)

> mean(x) # MLE

[1] 4.962583


The Newton-Raphson method: example

# the first derivative. want to solve l’(mu)=0 for mu

> der.loglik = function(dat,mu){sum(dat-mu)}

# the newton-raphson algorithm

> NR = function(dat,val){

+ conv=0; para = val; new.para =0

+ while(conv==0){

+ new.para = para - der.loglik(dat,para)/(-length(dat))

+ if(abs(new.para-para)<10^(-5)){ conv=1}

+ para=new.para

+ }

+ return(new.para)

+ }

> NR(x,10)

[1] 4.962583

> NR(x,-10)

[1] 4.962583


Derivative-Based Optimization

We want to find the minimum or the maximum of f(x).The newton-Raphson alogrithm: xt+1 = x1 − f ′(xt)/f ′′(xt)Quasi-Newton methods

to approximate f ′(xt)/f′′(xt)

useful when f ′′(xt) is not available or too expensive to compute atevery iteration

Several approximations to f ′(xt)/f′′(xt)

The steepest descent method

Conjugate gradient methods

The Broyden-Fletcher-Goldfarb-Shanno (BFGS)algorithm


BFGS: optim()

optim(par,fn,gr,...,method,...)

# want to minimize -loglikelihood

> neg.loglik=function(par,dat){-sum(dnorm(dat,par,1,log=T))}

> optim(par=10,fn=neg.loglik, dat=x

gr=function(par){-1*der.loglik(x,par)},

method ="BFGS")

$par

[1] 4.962583

$value

[1] 69.51976

# if ’gr’ is not given, an approximation will be used

> optim(10,neg.loglik,dat=x,method ="BFGS")

$par

[1] 4.962583

$value

[1] 69.51976


Variants of BFGS

L-BFGS (Limited-memory BFGS)I using a limited amount of computer memoryI popular for parameter estimation in machine learning

L-BFGS-BI extends L-BFGS to handle box constraints (or bound constraints)

on parametersI e.g. maxL(µ) subject to −10 ≤ µ ≤ 10


L-BFGS-B: optim()

> optim(10,neg.loglik,dat=x,method ="L-BFGS-B"

+ ,lower=-10,upper=10)

$par

[1] 4.962583

$value

[1] 69.51976

# To find the MLEs of mu and sigma(>0)

> neg.loglik2=function(par,dat){

+ -sum(dnorm(dat,par[1],par[2],log=T))}

> optim(c(10,10),neg.loglik2,dat=x,method ="L-BFGS-B",

+ lower=c(-Inf,1e-4))

$par

[1] 4.9625843 0.9710372

$value

[1] 69.47741


Optimizations subject to (in)equality constraints

Example: Consider X1, . . . , Xn ∼Multinomial(1, (p1, p2, p3)), wherep1 + p2 + p3 = 1. We want to find the MLEs of p1, p2, p3.

The MLEs are

arg maxp1,p2,p3

l(p1, p2, p3) subj. to p1 + p2 + p3 = 1 and 0 ≤ pi ≤ 1

The method of Lagrange multipliersConsider the two-dimensional problem:

maximize f(x, y) subject to g(x, y) = c.

If (x0, y0) maximizes (or minimizes) f(x, y) subject to g(x, y) = c,then there exists λ such that ∇f = λ∇g. λ is a Lagrangemultiplier.

Lagrange function: L(x, y, λ) = f(x, y)− λ(g(x, y)− c)Want to maximize L(x, y, λ)

I ∇x,y,λL = 0 implies g(x, y) = c and ∇f = λ∇g


Lagrange multipliers

If (x0, y0) maximizes (or minimizes) f(x, y) subject to g(x, y) = c, thenthere exists λ such that ∇f = λ∇g.


Optimization subj. to constraints: solnp()

> x = rmultinom(50,1,c(.1,.3,.6))

> dim(x)

[1] 3 50

> x[,1:5]

[,1] [,2] [,3] [,4] [,5]

[1,] 0 0 0 0 0

[2,] 1 0 0 1 1

[3,] 0 1 1 0 0

# MLEs of p1, p2, and p3: sample proportions

> rowMeans(x)

[1] 0.10 0.34 0.56


Optimization subj. to constraints: solnp()

> library(Rsolnp)

> negloglik = function(p,dat){

+ -sum(apply(dat,2,function(a)dmultinom(a,prob=p,log=T)))}

> constr=function(p){sum(p)}

> solnp(pars=rep(1/3,3),fun=function(p){negloglik(p,x)},

+ eqfun=constr,eqB=c(1),LB=rep(0,3),UB=rep(1,3))

Iter: 1 fn: 46.0876 Pars: 0.10000 0.34000 0.56000

Iter: 2 fn: 46.0876 Pars: 0.10000 0.34000 0.56000

solnp--> Completed in 2 iterations

$pars

[1] 0.09999997 0.34000000 0.56000003

$values

[1] 54.93061 46.08761 46.08761

$lagrange

[,1]

[1,] 4.971365e-06


Derivative-free approach: Nelder-Mead method

To find the minimum or maximum of a function f

Robust but relatively slow

Algorithm (2-dimensional case, but can be extended easily for a generalcase)

1 Start with three points x0, x1, x2 (a triangle) such thatf(x0) ≥ f(x1) ≥ f(x2)

2 Compute centroid xm = (x1 + x2)/2

3 reflection operator: xr = x2 + F1(xm − x2) x2 (worst point) isreflected through the opposite face of the triangle. If xr is not theworst, go to (1).

4 expansion operator. If xr is not the worst, go to (1).

5 contraction operator. If xr is not the worst, go to (1).

6 shrinks the entire triangle. Go to (1)


Nelder-Mead


Nelder-Mead: optim()

> y=rnorm(50,5,1)

> mean(y)

[1] 5.015369

> neg.loglik=function(par,dat){

+ -sum(dnorm(dat,par[1],par[2],log=T))}

> optim(par=c(10,10),fn=neg.loglik,dat=y,

+ method="Nelder-Mead")

$par

[1] 5.0152367 0.9663514

$value

[1] 69.2309


Optimizations in R

optim()I Nelder-Mead (default): derivative-freeI BFGS: a quasi-Newton methodI CG: a conjugate gradients methodI L-BFGS-B: allows box constraintsI SANN, Brent

Also see optimize() and nlm()

solnp(): (in)equality constraintsAlso see constrOptim() (inequality constraints)

DEoptim()I Differential evolutionI in R package DEoptimI derivative-free, more reliable than Nelder-Mead method


R Packages

a collection of R functions and data sets

To distribute statistical methodology to others


Creating R Packages

As example we implement a function for linear regression models.

1 Put all R functions to bundle in a file

2 We then structure the pieces into the form of an R package

3 Build an R package and packs everything into an archive file

4 Install the zipped archive file to use the functions


Example

As running example in this tutorial we will develop R code for thestandard linear regression model

> sbp =read.table("Table11.9.DAT",header=T)

> formula = SBP~Birthweight+Age

> x = model.matrix(formula,data=sbp)

> mf = model.frame(formula=formula, data=sbp)

> y = model.response(mf)

> solve(t(x)%*% x)%*%t(x)%*%y

(Intercept) 53.4501940

Birthweight 0.1255833

Age 5.8877191

>

> lm(formula,dat=sbp)

Coefficients:

(Intercept) Birthweight Age

53.4502 0.1256 5.8877


R function LSE

LSE = function(formula, data){

x = model.matrix(formula,data)

mf = model.frame(formula, data)

y = model.response(mf)

coef = solve(t(x)%*% x)%*%t(x)%*%y

res = list(x=x,y=y,coefficients = coef)

res

}

Load the function


R function LSE

> LSE(SBP~Birthweight+Age,data=sbp)

$x


1 1 135 3

2 1 120 4

3 1 100 3

...

$y

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

89 90 83 77 92 98 82 85 96 95 80 79 86 97 92 88

$coefficients



Age 5.8877191


R function LSE

Note that we have used the standard names “coefficients”. As a bonuson the side we get methods for several standard generic functions forfree, because their default methods work for our object:

> lse1=LSE(SBP~Birthweight+Age,data=sbp)

> coef(lse1)

[,1]



Age 5.8877191

Similary, if we have output elements “fitted.values” and “residuals”, wecan apply fitted() and resid() to the output object of LSE().


Creating R packages

Create an R code file LSE.R containing function LSE().

1 Clean your workspace: Session → Clear Workspace...

2 Load all functions and data sets you want in the package into aclean R session: click “Source”.

3 Run package.skeleton(...).1 The objects are sorted into data and functions, skeleton help files

are created and a DESCRIPTION file is created.2 The function then prints out a list of things for you to do next.


Creating the structure of a package

> package.skeleton(name="LSE", code_files="LSE.R")

Creating directories ...

Creating DESCRIPTION ...

Creating NAMESPACE ...

Creating Read-and-delete-me ...

Copying code files ...

Making help files ...

Done.

Further steps are described in ’./LSE/Read-and-delete-me’.

It created a directory called ”LSE” in the current working directory.


The structure of a package

File Read-and-delete-me contains further instructions:

* Edit the help file skeletons in ’man’, possibly combining

help files for multiple functions.

* Edit the exports in ’NAMESPACE’, and add necessary imports.

* Put any C/C++/Fortran code in ’src’.

* If you have compiled code, add a useDynLib() directive

to ’NAMESPACE’.

* Run R CMD build to build the package tarball.

* Run R CMD check to check the package tarball.

Read "Writing R Extensions" for more information.


The package DESCRIPTION file

Package: LSE

Type: Package

Title: What the package does (short line)

Version: 1.0

Date: 2016-12-06

Author: Who wrote it

Maintainer: Who to complain to <[email protected]>

Description: More about what it does (maybe more than one line)

License: What license is it under?


The structure of R package

R code file Under the folder R: copied the R code

documentation files (manuals) under folder man

you may have more sub-folders for datasets, C/C++ source etc


Building and installing a package using devtools

We can then build and install the package from within R.

1 Start R within your package directory (so that your packagedirectory is Rs working directory).

2 Install devtools by typing (within R)

> install.packages(devtools)3 Load the devtools package with

> library(devtools)4 to build the package

> build()

It removes temporary files from the source tree of the package andpacks everything into an archive file. This will createLSE 1.0.tar.gz file.

5 To install the R package, youd type

> install()

> library(LSE)

> ?LSE


Installation of R packages from the archive files

Click Packages tab → Install

Install from: Package Archive File (.zip; tar.gz)

Package archive: select LSE 1.0.tar.gz

Click Install.

> install.packages("C:/Users/user/Desktop/Biostat_2016/Lec14/LSE_1.0.tar.gz", repos = NULL, type = "source")

> library(LSE)

> LSE(SBP~Birthweight+Age,data=sbp)

$x


1 1 135 3

2 1 120 4

3 1 100 3


Creating R packages: more readings

Creating R Packages: A Tutorialftp://cran.r-project.org/pub/R/doc/contrib/Leisch-CreatingPackages.pdf

Google examples using devtools (easy building) and roxygen2

(easy documentation)


Summary

OptimizationI Derivative-based approach: optim(), solnp()I Derivative-free approach:optim(), DEoptim()

Creating R packagesI package.skeleton()I R packages devtools and roxygen2


Documents

BIO5312 Biostatistics Lecture 14: Creating R Packages · res = list(x=x,y=y,coefficients = coef) res} Load the function Yujin Chung Lec14: R Packages Fall 2016 22/33. R function LSE