BeSS: An R Package for Best Subset Selection in GLM and ... · PDF fileThe canonical approach to subset selection problem is to ... arising in modern statistics. Recently,Bertsimas,

JSS Journal of Statistical SoftwareMMMMMM YYYY, Volume VV, Issue II. doi: 10.18637/jss.v000.i00

BeSS: An R Package for Best Subset Selection in

Linear, Logistic and CoxPH Models

Canhong WenUniversity of Science and

Technology of ChinaSun Yat-sen University

Aijun ZhangThe University of

Hong Kong

Shijie QuanSun Yat-sen University

Xueqin WangSun Yat-sen University

Abstract

We introduce a new R package, BeSS, for solving the best subset selection problem inlinear, logistic and Cox’s proportional hazard (CoxPH) models. It utilizes a highly efficientactive set algorithm based on primal and dual variables, and supports sequential andgolden search strategies for best subset selection. We provide a C++ implementation of thealgorithm using Rcpp interface. We demonstrate through numerical experiments basedon enormous simulation and real datasets that the new BeSS package has competitiveperformance compared to other R packages for best subset selection purpose.

Keywords: best subset selection, primal dual active set, model selection.

1. Introduction

One of the main tasks of statistical modeling is to exploit the association between a responsevariable and multiple predictors. Linear model (LM), as a simple parametric regression model,is often used to capture linear dependence between response and predictors. The other twocommon models: generalized linear model (GLM) and Cox’s proportional hazards (CoxPH)model, can be considered as the extensions of linear model, depending on the types of re-sponses. Parameter estimation in these models can be computationally intensive when thenumber of predictors is large. Meanwhile, Occam’s razor is widely accepted as a heuristicrule for statistical modeling, which balances goodness of fit and model complexity. This ruleleads to a relative small subset of important predictors.

The canonical approach to subset selection problem is to choose k out of p predictors for eachk ∈ {0, 1, 2, . . . , p}. This involves exhaustive search over all possible 2p subsets of predictors,which is an NP-hard combinatorial optimization problem. To speed up, Furnival and Wil-

arX

iv:1

709.

0625

4v1

[st

at.C

O]

19

Sep

2017

http://dx.doi.org/10.18637/jss.v000.i00

2 BeSS: Best Subset Selection

son (1974) introduced a well-known branch-and-bound algorithm with an efficient updatingstrategy for LMs, which was later implemented by R packages such as the leaps (Lumley andMiller 2017) and the bestglm (McLeod and Xu 2010). Yet for GLMs, a simple exhaustivescreen is undertaken in bestglm. When the exhaustive screening is not feasible for GLMs, fastapproximating approaches have been proposed based on a genetic algorithm. For instance,kofnGA(Wolters 2015) implemented a genetic algorithm to search for a best subset of a pre-specified model size k, while glmuti (Calcagno, de Mazancourt et al. 2010) implemented agenetic algorithm to automated select the best model for GLMs with no more than 32 covari-ates. These packages can only deal with dozens of predictors but not high-dimensional dataarising in modern statistics. Recently, Bertsimas, King, Mazumder et al. (2016) proposed amixed integer optimization approach to find feasible best subset solutions for LMs with rela-tively larger p, which relies on certain third-party integer optimization solvers. Alternatively,regularization strategy is widely used to transform the subset selection problem into computa-tional feasible problem. For example, glmnet (Friedman, Hastie, and Tibshirani 2010; Simon,Friedman, Hastie, and Tibshirani 2011) implemented a coordinate descent algorithm to solvethe lasso problem, which is a convex relaxation by replacing the cardinality constraint in bestsubset selection problem by the L1 norm.

In this paper, we consider a primal-dual active set (PDAS) approach to exactly solve thebest subset selection problem for sparse LM, GLM and CoxPH models. The PDAS algorithmfor linear least squares problems was first introduced by Ito and Kunisch (2013) and laterdiscussed by Jiao, Jin, and Lu (2015) and Huang, Jiao, Liu, and Lu (2017). It utilizes anactive set updating strategy and fits the sub-models through use of complementary primaland dual variables. We generalize the PDAS algorithm for general convex loss functions withthe best subset constraint, and further extend it to support both sequential and golden sectionsearch strategies for optimal k determination. We develop a new package BeSS (BEst Sub-set Selection, Wen, Zhang, Quan, and Wang (2017)) in the R programming system (R CoreTeam 2016) with C++ implementation of PDAS algorithms and memory optimized for sparsematrix output. This package is publicly available from the Comprehensive R Archive Net-work (CRAN) at https://cran.r-project.org/package=BeSS. We demonstrate throughenormous datasets that BeSS is efficient and stable for high dimensional data, and may solvebest subset problems with n in 1000s and p in 10000s in just seconds on a single personalcomputer.

The article is organized as follows. In Section 2, we provide a general primal-dual formulationfor the best subset problem that includes linear, logistic and CoxPH models as special cases.Section 3 presents the PDAS algorithms and related technical details. Numerical experimentsbased on enormous simulation and real datasets are conducted in Section 4. We concludewith a short discussion in Section 5.

2. Primal-dual formulation

The best subset selection problem with the subset size k is given by the following optimizationproblem:

minβ∈Rp

l(β) s.t. ‖β‖0 = k, (1)

where l(β) is a convex loss function of the model parameters β ∈ Rp and k is a positiveinteger. The L0 norm ‖β‖0 =

∑pj=1 |βj |0 =

∑pj=1 1βj 6=0 counts the number of nonzeros in β.

https://cran.r-project.org/package=BeSS

Journal of Statistical Software 3

It is known the solution to (1) is necessarily a coordinate-wise minimizer, which we denoteby β�. For each coordinate j = 1, . . . , p, write lj(t) = l(β�1 , . . . , β

�j−1, t, β

�j+1, . . . , β

�p) while

fixing other coordinates. Let gj(t) = ∂lj(t)/∂t and hj(t) = ∂2lj(t)/∂2t be the first and second

derivatives of lj(t) with respect to t, respectively. Then, the local quadratic approximation oflj(t) around β�j is given by

lQj (t) = lj(β�j ) + gj(β

�j )(t− β�j ) + 1

2hj(β�j )(t− β�j )2

=1

2hj(β

�j )

(t− β�j +

gj(β�j )

hj(β�j )

)2

+ lj(β�j )−

(gj(β�j ))2

2hj(β�j )

=1

2hj(β

�j )(t− (β�j + γ�j )

)2+ lj(β

�j )−

(gj(β�j ))2

2hj(β�j ),

(2)

where γ�j = −gj(β�j )/hj(β�j ) denotes the standardized gradient at β�j . Minimizing the objective

function lQj (t) yields t∗j = β�j + γ�j .

The constraint in (1) says that there are (p− k) components of {t∗j , j = 1, . . . , p} that would

be enforced to be zero. To determine them, we consider the sacrifice of lQj (t) if we switch t∗jfrom β�j + γ�j to 0, as given by

∆j =1

2hj(β

�j )(β�j + γ�j )2. (3)

Among all the candidates, we may enforce those t∗j ’s to zero if they contribute the leasttotal sacrifice to the overall loss. To realize this, let ∆[1] ≥ · · · ≥ ∆[p] denote the decreasingrearrangement of ∆j for j = 1, . . . , p, then truncate the ordered sacrifice vector at position k.Therefore, upon the quadratic approximation (2), the coordinate-wise minimizer β� is shownto satisfy the following primal-dual condition:

β�j =

{β�j + γ�j , if ∆j ≥ ∆[k]

0, otherwise,for j = 1, . . . , p. (4)

In (4), we treat β = (β1, · · · , βp) as primal variables, γ = (γ1, · · · , γp) as dual variables, and∆ = (∆1, . . . ,∆p) as reference sacrifices. These quantities are key to the algorithm we willdevelop in the next section for finding the coordinate-wise minimizer β�. In what follows weprovide three special cases of the general problem (1).

Case 1: Linear regression. Consider the linear model y = Xβ + ε with design matrixX ∈ Rn×p and i.i.d. errors Here X and y are standardized such that the intercept term isremoved from the model and each column of X has

√n norm.

Take the loss function l(β) = 12n‖y −Xβ‖2. It is easy to verify that for a given β,

gj(βj) = − 1

ne>X(j), hj(βj) = 1, for j = 1, . . . , p (5)

where e = y −Xβ denotes the residual and X(j) denotes the jth column of X. Thus,

γj =1

ne>X(j), ∆j =

1

2(β�j + γ�j )2, for j = 1, . . . , p. (6)


Case 2: Logistic regression. Consider the logistic model log(p(x)/(1− p(x))) = β0 + x>βwith p(x) = Pr(Y = 1|x),x ∈ Rp. Given the data (X,y) = {(xi, yi)}ni=1 with yi ∈ {0, 1}, thenegative log-likelihood function is given by

l(β0,β) = −n∑i=1

{yi(β0 + x>i β)− log(1 + exp(β0 + x>i β))

}. (7)

We give only the primal-dual quantities for β according to the L0 constraint in (1), whileleaving β0 to be estimated by unconstrained maximum likelihood method. For given (β0,β),write pi = exp(β0 + x>i β)/(1 + exp(β0 + x>i β)) the i-th predicted probability. Then,

gj(βj) = −n∑i=1

Xij(yi − pi), hj(βj) =n∑i=1

X2ijpi(1− pi), for j = 1, . . . , p (8)

Thus,

γj =

∑ni=1Xij(yi − pi)∑ni=1X

2ijpi(1− pi)

, ∆j =1

2

n∑i=1

X2ijpi(1− pi)(βj + γj)

2, for j = 1, . . . , p. (9)

Case 3: CoxPH regression. Consider the CoxPH model λ(t|x) = λ0(t) exp(x>β) withan unspecified baseline hazard λ0(t) and x ∈ Rp. Given the survival data {(Ti, δi,xi) : i =1, . . . , n} with observations of survival time Ti and censoring indicator δi, By the method ofpartial likelihood (Cox 1972), the model parameters β can be obtained by minimizing thefollowing convex loss,

l(β) = −∑i:δi=1

(x>i β − log

( ∑i′:Ti′≥Ti

exp(x>i′ β)

)). (10)

For a given β, write ωi,i′ = exp(x>i′β)/∑

i′:Ti′≥Tiexp(x>i′β), then it can be verified that

gj(βj) = −∑i:δi=1

(Xij −

∑i′:Ti′≥Ti

ωi,i′Xi′j

)(11)

hj(βj) =∑i:δi=1

∑i′:Ti′≥Ti

ωi,i′

(Xi′j −

∑i′:Ti′≥Ti

ωi,i′Xi′j

)2

(12)

so that γj = −gj(βj)/hj(βj) and ∆j = 12hj(βj)(βj + γj)

2 for j = 1, . . . , p.

3. Active set algorithm

For the best subset problem (1), define the active set A = {j : βj 6= 0} with cardinality k andthe inactive set I = {j : βj = 0} with cardinality p − k. For the coordinate-wise minimizerβ� satisfying the primal-dual condition (4), we have that

• When j ∈ A, β�j 6= 0, γ�j = 0 and ∆j = 12hj(β

�j )[β�j]2

;

• When j ∈ I, β�j = 0, γ�j = −gj(0)/hj(0) and ∆j = 12hj(0)γ2j ;


• ∆j ≥ ∆j′ whenever j ∈ A and j′ ∈ I.

Clearly, the primal variables βj ’s and the dual variables γj ’s have complementary supports.The active set A plays a crucial role in the best subset problem; indeed if A is known a priori,we may estimate the k-nonzero primal variables by standard convex optimization:

min l(βA) = minβI=0

l(β), where I = Ac. (13)

We may use an iterative procedure to determine the active set A. Suppose at the m-thiteration with the current estimate Am, we may estimate βm by (13) and derive (γm,∆m)as discussed above, then update the active set by

Am+1 ={j : ∆m

j ≥ ∆m[k]

}, Im+1 =

{j : ∆m

j < ∆m[k]

}. (14)

This corresponds to the following algorithm:

Algorithm 1 Primal-dual active set (PDAS) algorithm

1. Specify the cardinality k of the active set and the maximum number of iterations mmax.Initialize A0 to be a random k-subset of {1, . . . , p} and I0 = (A0)c.

2. For m = 0, 1, 2, . . . ,mmax, do

(2.a) Determine βm by βmIm = 0 and βmAm = arg min{βIm=0} l(β);

(2.b) For each j ∈ A, γmj = 0 and ∆mj = 1

2hj(βmj )[βmj]2

;

(2.c) For each j ∈ I, γmj = −gj(0)/hj(0) and ∆mj = 1

2hj(0)[γmj]2

;

(2.d) Update the active and inactive sets by

Am+1 ={j : ∆m

j ≥ ∆m[k]

}, Im+1 =

{j : ∆m

j < ∆m[k]

}.

(2.e) If Am+1 = Am, then stop; else m = m+ 1 and return to steps (2.a)-(2.d).

3. Output {Am,βm,∆m}.

3.1. Determination of optimal k

The subset size k is usually unknown in practice, thus one has to determine it in a data-drivenway. A heuristic way is using the cross-validation technique to achieve the best predictionperformance. Yet it is time consuming to conduct the cross-validation method especially forhigh-dimensional data. An alternative way is to run the PDAS algorithm from small to largek values, then identify an optimal choice according to some criteria, e.g., Akaike informationcriterion (Akaike (1974), AIC) and Bayesian information criterion (Schwarz et al. (1978),BIC) and extended BIC (Chen and Chen (2008, 2012), EBIC) for small-n-large-p scenarios.This leads to the sequential PDAS algorithm.

Algorithm 2 Sequential primal-dual active set (SPDAS) algorithm


1. Specify the maximum size kmax of active set, and initialize A0 = ∅.

2. For k = 1, 2, . . . , kmax, do

Run PDAS with initial value Ak−1 ∩ {j ∈ Ik−1 : j ∈ arg max ∆k−1j }. Denote the

output by {Ak,βk,∆k}.

3. Output the optimal choice {A∗,β∗,∆∗} that attains the minimum AIC, BIC or EBIC.

2

4

6

8

L(β)

2

4

6

8

Loss function and Solution Path

0 5 10 15 20

−2

−1

0

1

2

3

β

Model size

1

2

34

5

678

9

1011121314151617181920

−2

−1

0

1

2

3

Figure 1: Plot of the loss function against the model complexity k and solution path for eachcoefficients. The orange vertical dash line indicates number of true nonzero coefficients.

To alleviate the computational burden of determining k as in SPDAS, here we provide analternative method based on the golden section search algorithm. We begin by plotting theloss function l(β) as a function of k for a simulated data from linear model with standardGaussian error. The true coefficient β = (3, 1.5, 0, 0,−2, 0, 0, 0,−1, 0, . . . , 0) and the designmatrix X is generated as in Section 4.1 with ρ = 0.2. From Figure 1, it can be seen thatthe slope of the loss plot goes from steep to flat and there is an ‘elbow’ exists near the truenumber of active set, i.e., k = 4. The solution path for the same data is presented at thebottom of Figure 1 for a better visualization on the relationship between loss function andcoefficient estimation. When a true active predictor is included in the model, the loss functiondrops dramatically and the predictors already in the model adjust their estimates to be closeto the true values. When all the active predictors are included in the model, their estimateswould not change much as k value becomes larger.

Motivated by this interesting phenomenon, we develop a search algorithm based on the goldensection method to determine the location of such an ‘elbow’ in the loss function. In this way,


we can avoid to run the PDAS algorithm extensively for a whole sequential list. The goldensection primal-dual active set (GPDAS) algorithm is summarized as follows.

Algorithm 3 Golden section primal-dual active set (GPDAS) algorithm

1. Specify the number of maximum iterations mmax, the maximum size kmax of active setand the tolerance η ∈ (0, 1). Initialize kL = 1, and kR = kmax.

2. For m = 1, 2, . . . ,mmax, do

(2.a) Run PDAS with k = kL and initial valueAm−1L ∩{j ∈ Im−1L : j ∈ arg max(∆m−1L )j}.

Denote the output by {AmL ,βmL ,∆

mL }.

(2.b) Run PDAS with k = kR and initial valueAm−1R ∩{j ∈ Im−1R : j ∈ arg max(∆m−1R )j}.

Denote the output by {AmR ,βmR ,∆

mR}.

(2.c) Calculate kM = kL+ 0.618× (kR−kL). Run PDAS with k = kM and initial valueAm−1M ∩{j ∈ Im−1M : j ∈ arg max(∆m−1

M )j}. Denote the output by {AmM ,βmM ,∆

mM}.

(2.d) Determine whether kM is an ‘elbow’ point:

• Run PDAS with k = kM − 1 and initial value AmM . Denote the output by{AmM−,β

mM−,∆

mM−}.

• Run PDAS with k = kM + 1 and initial value AmM . Denote the output by{AmM+,β

mM+,∆

mM+}.

• If |l(βmM )− l(βmM−)| > η|l(βmM )| and |l(βmM )− l(βmM+)| < η|l(βmM )|/2, then

stop and denote kM as an ‘elbow’ point, otherwise go ahead.

(2.e) Update kL, kR and AmL ,AmR :

• If |l(βmM )− l(βmL )| > η|l(βmM )| > |l(βmR )− l(βmL )|, then kR = kM , AmR = AmM ;

• If min {|l(βmM )− l(βmL )|, |l(βmR )− l(βmL )|} > η|l(βmM )|, then kL = kM , AmL =AmM ;

• Otherwise, kR = kM ,AmR = AmM and kL = 1,AmL = ∅.(2.f) If kL = kR − 1, then stop, otherwise m = m+ 1.

3. Output {AmM ,βmM ,∆

mM}.

3.2. Computational details

We study the computational complexity of the PDAS algorithms with a pre-specified k. Con-sider one iteration in step (2) of the PDAS algorithm. Let Ng and Nh denote the computa-tional complexity of calculation of gj(βj) and hj(βj) for a given β. Then according to thedefinition, the calculation of γ in steps (2.b)-(2.c) costs O((p − k) max(Nh, Ng)), and thecalculation of ∆ in steps (2.b)-(2.c) costs O(pNh). Assume the solver on active set requiresNl flops, then the overall cost of one iteration is O(max(Nl, pNh, (p− k)Ng)).

The number of iterations in step (2) could depend on the the signal-to-noise ratio, the di-mensionality p of parameter β, and the selected sparsity level k. For linear regression model,


it was shown that PDAS algorithm stops at most O(log(R)) iterations, where R is the rel-ative magnitude of the nonzero coefficients (Ito and Kunisch 2013; Huang et al. 2017). Al-though there is no theoretical guarantee on the number of iterations for other models, wehave not encountered cases with many iterations. Denote NP to be the complexity of PDAS,then the computational complexity of the SPDAS and the GPDAS are O(kmax × NP) andO(log(kmax)×NP), respectively.

Case 1: Linear regression. The computation of hj(βj) = 1 is negligible, i.e., Nh = O(1).The matrix vector product in the computation of hj(βj) takes O(n) flops. For the leastsquares problem on the active set, we use Cholesky factorization to obtain the estimate,which leads to Nl = O(max(nk2, k3)). Thus the total cost of one iteration in step (2) isO(max(nk2, k3, n(p− k))).

In particular, if the true coefficient is sparse with the underlying k � p and n = O(log(p)),then we can choose an appropriate kmax value, e.g., kmax = n/ log(n), to speed up the algo-rithm. In this way, the cost of the PDAS algorithm is O(np). This rate is comparable with thesure independence screening procedure (Fan and Lv 2008) in handling ultrahigh-dimensionaldata. In fact, even if the true coefficient is not sparse, we could use a conjugate gradient(Golub and Van Loan (2012), CG) algorithm with a preconditioning matrix to achieve asimilar computational rate.

Case 2: Logistic regression. The key computation for logistic regression is the predictedprobabilities pi’s, which costs O(p) flops. Thus Ng = O(np) and Nh = O(np). We usethe iteratively reweighted least squares (Friedman, Hastie, and Tibshirani (2001), IRLS)for parameter estimation on the active set. At each iteration in the IRLS algorithm, thecomputational complexity of reweighed least squares is the same as that of least squares.Assume there are NI iterations needed in the IRLS, then NI = O(Nl max(nk2, k3)). Thus,the total cost of one iteration in step (2) is O(max(nk2NI , k

3NI , np2)).

Case 3: CoxPH regression. The key computation for CoxPH regression is ωi,i′ , which costsO(np) flops. Assume the censoring rate is c, then Ng = O(n3p(1−c)) and Nh = O(n3p(1−c)).Like the coxph command from the survival package, we adopt the standard Newton-Raphsonalgorithm for the maximum partial likelihood estimation on the active set. Its difficulty arisesin the computation of the inverse of the hessian matrix, which is full and dense. The hessianmatrix has k2 entries and it requires O(n3k(1−c)) flops for the computation of each entry. Thematrix inversion costs O(k3) via Gauss-Jordan elimination or Cholesky decomposition. Hence,for each Newton-Raphson iteration, the updating equation requires O(max(n3k3(1− c), k3))flops. We may speed up the algorithm by replacing the hessian matrix with its diagonal, whichreduces the computational complexity per updating to O(max(n3k2(1 − c), k3)). Denote byNnr the number of Newton-Raphson iterations, then Nl = O(Nnr max(n3k2(1 − c), k3)) andthe total cost of one iteration in step (2) is O(max(n3p2(1− c), n3k2(1− c)Nnr, k

3Nnr)).

3.3. R package

We have implemented the active set algorithms described above into a R package called BeSS(BEst Subset Selection), which is publicly available from the CRAN at https://cran.r-

project.org/package=BeSS. The package is implemented in C++ with memory optimizedusing sparse matrix output and it can be called from R by a user-friendly interface.

The package contains two main functions, i.e., bess.one and bess, for solving the best subsetselection problem with or without specification of k. In bess, two options are provided to de-




termine the optimal k: one is based on the SPDAS algorithm with criteria including AIC, BICand EBIC; the other is based on the GPDAS algorithm. The function plot.bess generatesplots of loss functions for the best sub-models for each candidate k, together with solutionpaths for each predictor. We also include functions predict.bess and predict.bess.one tomake prediction on the new data.

4. Numerical examples

In this section we compare the performance of our new BeSS package to other well-knownpackages for best subset selection: leaps, bestglm and glmulti. We also include glmnet as anapproximate subset selection method and use the default cross-validation method to deter-mine an optimal tuning parameter. All parameters use the default values of the correspondingmain functions in those packages unless otherwise stated. In presenting the results of BeSS,bess.seq represents bess with argument method = "sequential" and bess.gs representsbess with argument method = "gsection", two different way to determine the optimal pa-rameter k. In bess.seq, we use AIC for examples with n ≥ p and EBIC for examples withn < p. We chose kmax = min(n/2, p) for linear models and kmax = min(n/ log(n), p) forlogistic and CoxPH models.

All the R codes are demonstrated in Section 4.3. All computations were carried out on a64-bit Intel machine with a single 3.30 GHz CPU and 4 GB of RAM.

4.1. Simulation data

We compare the performances of different methods in three aspects. The first aspect is therun time in seconds (Time). The second aspect is the selection performance in terms of truepositive (TP) and false positive (FP) numbers, which are defined by the numbers of truerelevant and true irrelevant variables among the selective predictors. The third aspect is thepredictive performance on a held out validation data of size 1000. For linear regression, weuse the relative mean squares error (MSE) as defined by ‖Xβ̂−Xβ∗‖2/‖Xβ∗‖2). For logisticregression, we calculate the classification accuracy by the average number of observationsbeing correctly classified. For CoxPH regression, we compute the median time on the testdata, then derive the area under the receiver operator characteristic curve (i.e., AUC) usingnearest neighbor estimation method as in Heagerty, Lumley, and Pepe (2000).

We generated the design matrix X and the underlying coefficients β as follows. The designmatrix X is generated with X(j) = Zj + 0.5 × (Zj−1 + Zj+1), j = 1, . . . , p, where Z0 =0,Zp+1 = 0 and {Zj , j = 1, . . . , p} were i.i.d. random samples drawn from standard Gaussiandistribution and subsequently normalized to have

√n norm. The true coefficient β∗ is a vector

with q nonzero entries uniformly distributed in [b, B], where b and B will be specified. In thesimulation study, the sample size is fixed to be n = 1000. For each scenario, 100 replicationswere conducted .

Case 1: Linear regression. For each X and β∗, we generated the response vector y =Xβ∗ + σε, with ε ∼ N (0, 1). We set b = 5σ

√2 log(p)/n, B = 100m and σ = 3. Different

choices of (p, q) were taken to cover both the overdetermined cases (p = 20, 30, or 40, q = 4)and the high-dimensional cases (p = 100, 1000, or 10000, q = 40). For glmulti, we onlypresent the result for p = 20 and p = 30 since it can only deal with at most 32 predictors.Since leaps and bestglm cannot deal with high-dimensional case, we only report the results


of glmnet, bess.seq and bess.gs. The results are summarized in Table 4.1.

In the overdetermined cases, the performances of all best subset selection methods are com-parable in terms of prediction accuracy and selection consistency. However, the regularizationmethod glmnet has much higher MSE and lower FP, which suggests LASSO might leads biasin the coefficient estimation. In terms of computation times, both bess.seq and bess.gs

has comparable performance with glmnet and cost much less run times than the branch-and-bound algorithms. Unlike leaps, bestglm and glmulti, the run times of bess.seq andbess.gs remain fairly stable across different dimensionality.

In the high-dimensional cases, both of bess.seq and bess.gs work quite well and havesimilar performance in prediction and variable selection. Furthermore, with increasing p andincreasing sparsity, sparse data (from left to right in Table 4.1), their performances becomebetter. On the other hand, glmnet has higher FP as p increases. In particular, when p = 10000and only 40 nonzero coefficients are involved, the average TP equals to 40 and the average FPis less than 3.06, while the average FP of glmnet increases to 30. While the computationalcomplexity of both algorithms seems to grow at a linear rate of p, the bess.gs offers speedupsby factors of 2 up to 10 or more.

Case 2: Logistic regression. For each x and β∗, the binary response is generated byy = Bernoulli(Pr(Y = 1)), where Pr(Y = 1) = exp(x>β∗)/(1 + exp(xβ∗)). The range ofnonzero coefficients were set as b = 10

√2 log(p)/n, B = 5b. Different choices of p were

taken to cover both the low-dimensional cases (p = 8, 10, or 12) and the high-dimensionalcases (p = 100, 1000, or 10000). The number of true nonzero coefficients was chosen to beq = 4 for low-dimensional cases and q = 20 for high-dimensional cases. Since bestglm isbased on complete enumeration, it may be used for low-dimensional cases yet it becomescomputationally infeasible for high dimensional cases.

The simulation results are summarized in Table 4.1. When p is small, both bess.seq andbess.gs have comparable performance with bestglm, glmulti and glmnet, but have con-siderably faster speed in computation than bestglm and glmulti. In the high-dimensionalcases, we see that all methods perform very well in terms of accuracy and TP. Yet both ofbess.seq and bess.gs have much smaller FP than glmnet. Among them, the run time forbess.gs is around a quarter of that for bess.seq and similar with that for glmnet.

Case 3: CoxPH regression. For each x and β∗, we generate data from the CoxPH modelwith hazard rate λ(t|x) = exp(x>β∗). The ranges of nonzero coefficients were set as thosein logistic regression, i.e., b = 10

√2 log(p)/n, B = 5b. Different choices of p were taken

to cover both the low-dimensional cases (p = 8, 10, or 12) and the high-dimensional cases(p = 100, 1000, or 10000). The number of true nonzero coefficients was chosen to be q = 4 forlow-dimensional cases and q = 20 for high-dimensional cases. Since glmulti only can handleno more than 32 predictors and is computationally infeasible for high dimensional cases, weonly report the low dimensional result for glmulti.

The simulation results are summarized in Table 4.1. Our findings about bess.seq andbess.gs are similar to those for the logistic regression.

4.2. Real data

We also evaluate the performance of the BeSS package in modeling several real data sets.Table 4.2 lists these instances and their descriptions. All datasets are saved as R data objectsand available online with this publication.


We randomly split the data into a training set with two-thirds observations and a test setwith remaining observations. Different best subset selection methods are used to identifythe best sub-model. For each method, run time in seconds (Time) and size of selected model(MS) are recorded. We also included measurements of the predictive performance on test dataaccording to the types of models as in Section 4.1. For reliable evaluation, the aforementionedprocedure is replicated for 100 times.

The modeling results are displayed in Table 4.2. Again in low-dimensional cases, bess hascomparable performance with the state-of-art algorithms (branch-and-bound algorithm forlinear models and complete enumeration algorithm and genetic algorithm for GLMs). Besides,bess.gs has comparable run times with glmnet and is considerably faster than bess.seq

especially in high-dimensional cases.

4.3. Code demonstration

We demonstrate how to use the package BeSS on a synthesis data as discussed in Section 3.1and a real data in Section 4.2. Firstly, load BeSS and generate data with the gen.data

function.

R> require("BeSS")

R> set.seed(123)

R> Tbeta <- rep(0, 20)

R> Tbeta[c(1, 2, 5, 9)] <- c(3, 1.5, -2, -1)

R> data <- gen.data(n = 200, p = 20, family = "gaussian", beta = Tbeta,

+ rho = 0.2, sigma = 1)

We may call the bess.one function to solve the best subset selection problem a specified bestsubset size. Then we can print or summary the bess.one object. While the print methodallows users to obtain a brief summary of the fitted model, the summary method presents amuch more detailed description.

R> fit.one <- bess.one(data$x, data$y, s = 4, family = "gaussian")

R> print(fit.one)

Df MSE AIC BIC EBIC

4.0000000 0.8501053 -24.4790117 -11.2857422 12.6801159

R> summary(fit.one)

----------------------------------------------------------------------

Primal-dual active algorithm with maximum iteration being 15

Best model with k = 4 includes predictors:

X1 X2 X5 X9

3.019296 1.679419 -2.021521 -1.038276

log-likelihood: 16.23951


deviance: -32.47901

AIC: -24.47901

BIC: -11.28574

EBIC: 12.68012

----------------------------------------------------------------------

The estimated coefficients of the fitted model can be extracted by using the coef function,which provides a sparse output with the control of argument sparse = TRUE. It is recom-mended to output a non-sparse vector when bess.one is used, and to output a sparse matrixwhen bess is used.

R> coef(fit.one, sparse = FALSE)

(intercept) X1 X2 X3 X4 X5

-0.07506287 3.01929556 1.67941924 0.00000000 0.00000000 -2.02152109

X6 X7 X8 X9 X10 X11

0.00000000 0.00000000 0.00000000 -1.03827568 0.00000000 0.00000000

X12 X13 X14 X15 X16 X17

0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000

X18 X19 X20

0.00000000 0.00000000 0.00000000

To make a prediction on the new data, a predict function can be used as follows.

R> pred.one <- predict(fit.one, newdata = data$x)

To extract the selected best model, we provide a lm, glm, or coxph object named bestmodel

in the fitted bess.one object depending on the type of model. Users could print, summaryor predict this bestmodel object just like working with classical regression modeling. Thiswould be helpful for those who are familiar with lm, glm, or coxph already.

R> bm.one <- fit.one$bestmodel

R> summary(bm.one)

Call:

lm(formula = ys ~ xbest)

Residuals:

Min 1Q Median 3Q Max

-2.54220 -0.63600 -0.04702 0.64100 3.11518

Coefficients:

Estimate Std. Error t value Pr(>|t|)

(Intercept) -0.07506 0.06603 -1.137 0.257

xbestX1 3.01930 0.06715 44.962 <2e-16 ***

xbestX2 1.67942 0.06577 25.535 <2e-16 ***


xbestX5 -2.02152 0.06577 -30.735 <2e-16 ***

xbestX9 -1.03828 0.06313 -16.446 <2e-16 ***

---

Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.9338 on 195 degrees of freedom

Multiple R-squared: 0.9566, Adjusted R-squared: 0.9557

F-statistic: 1075 on 4 and 195 DF, p-value: < 2.2e-16

In practice when the best subset size is unknown, we have to determine the optimal choiceof such sub-model size. The function bess provides two options: method = "sequential"

corresponds to the SPDAS algorithm, and method = "gsection" corresponds to the GPDASalgorithm. Next we illustrate the usage of bess in the trim32 data. We first load the datainto the environment and show that it has 18975 variables, a much larger number comparedwith the sample size 120.

R> load("trim32.RData")

R> dim(X)

[1] 120 18975

Below is an example of running bess with argument method = "sequential", epsilon =

0 and other argument being default values. We use the summary function to give a summaryof the fitted bess object.

R> fit.seq <- bess(X, Y, method="sequential", epsilon = 0)

R> summary(fit.seq)

----------------------------------------------------------------------------

Primal-dual active algorithm with tuning parameter determined by

sequential method

Best model determined by AIC includes 25 predictors with AIC = -890.9282

Best model determined by BIC includes 25 predictors with BIC = -821.2409

Best model determined by EBIC includes 2 predictors with EBIC = -561.2689

-----------------------------------------------------------------------------

As in the bess.one, the bess function outputs a “lm” object bestmodel associated with theselected best model. Here the bestmodel component outputs the last fitted model since wedidn’t use any early stopping rule as shown in the argument epsilon = 0.

R> bm.seq <- fit.seq$bestmodel

R> summary(bm.seq)


Call:


Residuals:


-0.039952 -0.012366 -0.001078 0.011401 0.075677

Coefficients:


(Intercept) 5.618703 0.407769 13.779 < 2e-16 ***

xbest1368348_at -0.089394 0.014563 -6.139 1.97e-08 ***

xbest1370558_a_at -0.122228 0.010712 -11.410 < 2e-16 ***

xbest1372548_at -0.179410 0.012085 -14.846 < 2e-16 ***

xbest1377032_at -0.062936 0.016733 -3.761 0.000294 ***

xbest1382223_at 0.497858 0.023655 21.047 < 2e-16 ***

xbest1388491_at 0.266606 0.021538 12.378 < 2e-16 ***

xbest1388657_at -0.085292 0.015030 -5.675 1.53e-07 ***

xbest1389122_at -0.101926 0.015317 -6.655 1.88e-09 ***

xbest1390269_at 0.106434 0.012130 8.774 7.40e-14 ***

xbest1378024_at -0.123666 0.017614 -7.021 3.40e-10 ***

xbest1378552_at -0.049578 0.010397 -4.768 6.77e-06 ***

xbest1379586_at -0.066086 0.013526 -4.886 4.22e-06 ***

xbest1379772_at -0.096651 0.010166 -9.507 2.05e-15 ***

xbest1379933_at 0.186271 0.015806 11.785 < 2e-16 ***

xbest1380696_at 0.028347 0.006882 4.119 8.19e-05 ***

xbest1380977_at 0.104704 0.018148 5.769 1.01e-07 ***

xbest1382392_at -0.033764 0.005830 -5.791 9.21e-08 ***

xbest1384690_at -0.083789 0.013985 -5.991 3.80e-08 ***

xbest1385015_at 0.131036 0.011803 11.102 < 2e-16 ***

xbest1385032_at 0.100631 0.012171 8.268 8.73e-13 ***

xbest1385395_at -0.139164 0.010919 -12.745 < 2e-16 ***

xbest1385673_at 0.071119 0.011828 6.013 3.46e-08 ***

xbest1392605_at -0.051400 0.008229 -6.246 1.21e-08 ***

xbest1394502_at 0.020363 0.006134 3.320 0.001283 **

xbest1398128_at -0.084070 0.012728 -6.605 2.36e-09 ***

---

Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1



F-statistic: 194.5 on 25 and 94 DF, p-value: < 2.2e-16

Alternatively, we might use criteria like AIC to select the best model among a sequential listof model size. As shown above, the output of the bess function includes criteria, i.e., AIC,BIC and EBIC, for best subset selection. Since the “trim32” data is high dimensional, we optto use the EBIC criterion to determine the optimal model size here. We then run the coef

function to extract the coefficients in the bess object and output the nonzero coefficients in


the selected model.

R> K.opt.ebic <- which.min(fit.seq$EBIC)

R> coef(fit.seq)[, K.opt.ebic][which(coef(fit.seq)[, K.opt.ebic]!=0)]

(intercept) 1382223_at 1388491_at

0.8054785 0.5715478 0.3555834

We can also run the predict function to output prediction value of a given newdata. Theargument type specifies which criteria is used to select the best fitted model.

R> pred.seq <- predict(fit.seq, newdata = data$x, type="EBIC")

The plot routine provides the plots of loss functions in the best sub-models for different kvalues, as well as solution paths for each predictor. It also adds a vertical dashed line toindicate the optimal k value as determined by EBIC. Figure 2 shows the result from thefollowing R code.

R> plot(fit.seq, type = "both", breaks = TRUE, K = K.opt.ebic)

0.002

0.004

0.006

0.008

L(β)

0.002

0.004

0.006

0.008

0 5 10 15 20 25

−0.2

0.0

0.2

0.4

0.6

0.8

β

Model size

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457

458

459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936937938939940941942943944945946947948949950951952953954955956957958959960961962963964965966967968969970971972973974975976977978979980981982983984985986987988989990991992993994995996997998999100010011002100310041005100610071008100910101011101210131014101510161017101810191020102110221023102410251026102710281029103010311032103310341035103610371038103910401041104210431044104510461047104810491050105110521053105410551056105710581059106010611062106310641065106610671068106910701071107210731074107510761077107810791080108110821083108410851086108710881089109010911092109310941095109610971098109911001101110211031104110511061107110811091110111111121113111411151116111711181119112011211122112311241125112611271128112911301131113211331134113511361137113811391140114111421143114411451146114711481149115011511152115311541155115611571158115911601161116211631164116511661167116811691170117111721173117411751176117711781179118011811182118311841185118611871188118911901191119211931194119511961197119811991200120112021203120412051206120712081209121012111212121312141215121612171218121912201221122212231224122512261227122812291230123112321233123412351236123712381239124012411242124312441245124612471248124912501251125212531254125512561257125812591260126112621263126412651266126712681269127012711272127312741275127612771278127912801281128212831284128512861287128812891290129112921293129412951296129712981299130013011302130313041305130613071308130913101311131213131314131513161317131813191320132113221323132413251326132713281329133013311332133313341335133613371338133913401341134213431344134513461347134813491350135113521353135413551356135713581359136013611362136313641365136613671368136913701371137213731374137513761377137813791380138113821383138413851386138713881389139013911392139313941395139613971398139914001401140214031404140514061407140814091410141114121413141414151416141714181419142014211422142314241425142614271428142914301431143214331434143514361437143814391440144114421443144414451446144714481449145014511452145314541455145614571458145914601461146214631464146514661467146814691470147114721473147414751476147714781479148014811482148314841485148614871488148914901491149214931494149514961497149814991500150115021503150415051506150715081509151015111512151315141515151615171518151915201521152215231524152515261527152815291530153115321533153415351536153715381539154015411542154315441545154615471548154915501551155215531554155515561557155815591560156115621563156415651566156715681569157015711572157315741575157615771578157915801581158215831584158515861587158815891590159115921593159415951596159715981599160016011602160316041605160616071608160916101611161216131614161516161617161816191620162116221623162416251626162716281629163016311632163316341635163616371638163916401641164216431644164516461647164816491650165116521653165416551656165716581659166016611662166316641665166616671668166916701671167216731674167516761677167816791680168116821683168416851686168716881689169016911692169316941695169616971698169917001701170217031704170517061707170817091710171117121713171417151716171717181719172017211722172317241725172617271728172917301731173217331734173517361737173817391740174117421743174417451746174717481749175017511752175317541755175617571758175917601761176217631764176517661767176817691770177117721773177417751776177717781779178017811782178317841785178617871788178917901791179217931794179517961797179817991800180118021803180418051806180718081809181018111812181318141815181618171818181918201821182218231824182518261827182818291830183118321833183418351836183718381839184018411842184318441845184618471848184918501851185218531854185518561857185818591860186118621863186418651866186718681869187018711872187318741875

1876

18771878187918801881188218831884188518861887188818891890189118921893189418951896189718981899190019011902190319041905190619071908190919101911191219131914191519161917191819191920192119221923192419251926192719281929193019311932193319341935193619371938193919401941194219431944194519461947194819491950195119521953195419551956195719581959196019611962196319641965196619671968196919701971197219731974197519761977197819791980198119821983198419851986198719881989199019911992199319941995199619971998199920002001200220032004200520062007200820092010201120122013201420152016201720182019202020212022202320242025202620272028202920302031203220332034203520362037203820392040204120422043204420452046204720482049205020512052205320542055205620572058205920602061206220632064206520662067206820692070207120722073207420752076207720782079208020812082208320842085208620872088208920902091209220932094209520962097209820992100210121022103210421052106210721082109211021112112211321142115211621172118211921202121212221232124212521262127212821292130213121322133213421352136213721382139214021412142214321442145214621472148214921502151215221532154215521562157215821592160216121622163216421652166216721682169217021712172217321742175217621772178217921802181218221832184218521862187218821892190219121922193219421952196219721982199220022012202220322042205220622072208220922102211221222132214221522162217221822192220222122222223222422252226222722282229223022312232223322342235223622372238223922402241224222432244224522462247224822492250225122522253225422552256225722582259226022612262226322642265226622672268226922702271227222732274227522762277227822792280228122822283228422852286228722882289229022912292229322942295229622972298229923002301230223032304230523062307230823092310231123122313231423152316231723182319232023212322232323242325232623272328232923302331233223332334233523362337233823392340234123422343234423452346234723482349235023512352235323542355235623572358235923602361236223632364236523662367236823692370237123722373237423752376237723782379238023812382238323842385238623872388238923902391239223932394239523962397239823992400240124022403240424052406240724082409241024112412241324142415241624172418241924202421242224232424242524262427242824292430243124322433243424352436243724382439244024412442244324442445244624472448244924502451245224532454245524562457245824592460246124622463246424652466246724682469247024712472247324742475247624772478247924802481248224832484248524862487248824892490249124922493249424952496249724982499250025012502250325042505250625072508250925102511251225132514251525162517251825192520252125222523252425252526252725282529253025312532253325342535253625372538253925402541254225432544254525462547254825492550255125522553255425552556255725582559256025612562256325642565256625672568256925702571257225732574257525762577257825792580258125822583258425852586258725882589259025912592259325942595259625972598259926002601260226032604260526062607260826092610261126122613261426152616261726182619262026212622262326242625262626272628262926302631263226332634263526362637263826392640264126422643264426452646264726482649265026512652265326542655265626572658265926602661266226632664266526662667266826692670267126722673267426752676267726782679268026812682268326842685268626872688268926902691269226932694269526962697269826992700270127022703270427052706270727082709271027112712271327142715271627172718271927202721272227232724272527262727272827292730273127322733273427352736273727382739274027412742274327442745274627472748274927502751275227532754275527562757275827592760276127622763276427652766276727682769277027712772277327742775277627772778277927802781278227832784278527862787278827892790279127922793279427952796279727982799280028012802280328042805280628072808280928102811281228132814281528162817281828192820282128222823282428252826282728282829283028312832283328342835283628372838283928402841284228432844284528462847284828492850285128522853285428552856285728582859286028612862286328642865286628672868286928702871287228732874287528762877287828792880

2881

28822883288428852886288728882889289028912892289328942895289628972898289929002901290229032904290529062907290829092910291129122913291429152916291729182919292029212922292329242925292629272928292929302931293229332934293529362937293829392940294129422943294429452946294729482949295029512952295329542955295629572958295929602961296229632964296529662967296829692970297129722973297429752976297729782979298029812982298329842985298629872988298929902991299229932994299529962997299829993000300130023003300430053006300730083009301030113012301330143015301630173018301930203021302230233024302530263027302830293030303130323033303430353036303730383039304030413042304330443045304630473048304930503051305230533054305530563057305830593060306130623063306430653066306730683069307030713072307330743075307630773078307930803081308230833084308530863087308830893090309130923093309430953096309730983099310031013102310331043105310631073108310931103111311231133114311531163117311831193120312131223123312431253126312731283129313031313132313331343135313631373138313931403141314231433144314531463147314831493150315131523153315431553156315731583159316031613162316331643165316631673168316931703171317231733174317531763177317831793180318131823183318431853186318731883189319031913192319331943195319631973198319932003201320232033204320532063207320832093210321132123213321432153216321732183219322032213222322332243225322632273228322932303231323232333234323532363237323832393240324132423243324432453246324732483249325032513252325332543255325632573258325932603261326232633264326532663267326832693270327132723273327432753276327732783279328032813282328332843285328632873288328932903291329232933294329532963297329832993300330133023303330433053306330733083309331033113312331333143315331633173318331933203321332233233324332533263327332833293330333133323333333433353336333733383339334033413342334333443345334633473348334933503351335233533354335533563357335833593360336133623363336433653366336733683369337033713372337333743375337633773378337933803381338233833384338533863387338833893390339133923393339433953396339733983399340034013402340334043405340634073408340934103411341234133414341534163417341834193420342134223423342434253426342734283429343034313432343334343435343634373438343934403441344234433444344534463447344834493450345134523453345434553456345734583459346034613462346334643465346634673468346934703471347234733474347534763477347834793480348134823483348434853486348734883489349034913492349334943495349634973498349935003501350235033504350535063507350835093510351135123513351435153516351735183519352035213522352335243525352635273528352935303531353235333534353535363537353835393540354135423543354435453546354735483549355035513552355335543555355635573558355935603561356235633564356535663567356835693570357135723573357435753576357735783579358035813582358335843585358635873588358935903591359235933594359535963597359835993600360136023603360436053606360736083609361036113612361336143615361636173618361936203621362236233624362536263627362836293630363136323633363436353636363736383639364036413642364336443645364636473648364936503651365236533654365536563657365836593660366136623663366436653666366736683669367036713672367336743675367636773678367936803681368236833684368536863687368836893690369136923693369436953696369736983699370037013702370337043705370637073708370937103711371237133714371537163717371837193720372137223723372437253726372737283729373037313732373337343735373637373738373937403741374237433744374537463747374837493750375137523753375437553756375737583759376037613762376337643765376637673768376937703771377237733774377537763777377837793780378137823783378437853786378737883789379037913792379337943795379637973798379938003801380238033804380538063807380838093810381138123813381438153816381738183819382038213822382338243825382638273828382938303831383238333834383538363837383838393840384138423843384438453846384738483849385038513852385338543855385638573858385938603861386238633864386538663867386838693870387138723873387438753876387738783879388038813882388338843885388638873888388938903891389238933894389538963897389838993900390139023903390439053906390739083909391039113912391339143915391639173918391939203921392239233924392539263927392839293930393139323933393439353936393739383939394039413942394339443945394639473948394939503951395239533954395539563957395839593960396139623963396439653966396739683969397039713972397339743975397639773978397939803981398239833984398539863987398839893990399139923993399439953996399739983999400040014002400340044005400640074008400940104011401240134014401540164017401840194020402140224023402440254026402740284029403040314032403340344035403640374038403940404041404240434044404540464047404840494050405140524053405440554056405740584059406040614062406340644065406640674068406940704071407240734074407540764077407840794080408140824083408440854086408740884089409040914092409340944095409640974098409941004101410241034104410541064107410841094110411141124113411441154116411741184119412041214122412341244125412641274128412941304131413241334134413541364137413841394140414141424143414441454146414741484149415041514152415341544155415641574158415941604161416241634164416541664167416841694170417141724173417441754176417741784179418041814182418341844185418641874188418941904191419241934194419541964197419841994200420142024203420442054206420742084209421042114212421342144215421642174218421942204221422242234224422542264227422842294230423142324233423442354236423742384239424042414242424342444245424642474248424942504251425242534254425542564257425842594260426142624263426442654266426742684269427042714272427342744275427642774278427942804281428242834284428542864287428842894290429142924293429442954296429742984299430043014302430343044305430643074308430943104311431243134314431543164317431843194320432143224323432443254326432743284329433043314332433343344335433643374338433943404341434243434344434543464347434843494350435143524353435443554356435743584359436043614362436343644365436643674368436943704371437243734374437543764377437843794380438143824383438443854386438743884389439043914392439343944395439643974398439944004401440244034404440544064407440844094410441144124413441444154416441744184419442044214422442344244425442644274428442944304431443244334434443544364437443844394440444144424443444444454446444744484449445044514452445344544455445644574458445944604461446244634464446544664467446844694470447144724473447444754476447744784479448044814482448344844485448644874488448944904491449244934494449544964497449844994500450145024503450445054506450745084509451045114512451345144515451645174518451945204521452245234524452545264527452845294530453145324533453445354536453745384539454045414542454345444545454645474548454945504551455245534554455545564557455845594560456145624563456445654566456745684569457045714572457345744575457645774578457945804581458245834584458545864587458845894590459145924593459445954596459745984599460046014602460346044605460646074608460946104611461246134614461546164617461846194620462146224623462446254626462746284629463046314632463346344635463646374638463946404641464246434644464546464647464846494650465146524653465446554656465746584659466046614662466346644665466646674668466946704671467246734674467546764677467846794680468146824683468446854686468746884689469046914692469346944695469646974698469947004701470247034704470547064707470847094710471147124713471447154716471747184719472047214722472347244725472647274728472947304731473247334734473547364737473847394740474147424743474447454746474747484749475047514752475347544755475647574758475947604761476247634764476547664767476847694770477147724773477447754776477747784779478047814782478347844785478647874788478947904791479247934794479547964797479847994800480148024803480448054806480748084809481048114812481348144815481648174818481948204821482248234824482548264827482848294830483148324833483448354836483748384839484048414842484348444845484648474848484948504851485248534854485548564857485848594860486148624863486448654866486748684869487048714872487348744875487648774878487948804881488248834884488548864887488848894890489148924893489448954896489748984899490049014902490349044905490649074908490949104911491249134914491549164917491849194920492149224923492449254926492749284929493049314932493349344935493649374938493949404941494249434944494549464947494849494950495149524953495449554956495749584959496049614962496349644965496649674968496949704971497249734974497549764977497849794980498149824983498449854986498749884989499049914992499349944995499649974998499950005001500250035004500550065007500850095010501150125013501450155016501750185019502050215022502350245025502650275028502950305031503250335034503550365037503850395040504150425043504450455046504750485049505050515052505350545055505650575058505950605061506250635064506550665067506850695070507150725073507450755076507750785079508050815082508350845085508650875088508950905091509250935094509550965097509850995100510151025103510451055106510751085109511051115112511351145115511651175118511951205121512251235124512551265127512851295130513151325133513451355136513751385139514051415142514351445145514651475148514951505151515251535154515551565157515851595160516151625163516451655166516751685169517051715172517351745175517651775178517951805181518251835184518551865187518851895190519151925193519451955196519751985199520052015202520352045205520652075208520952105211521252135214521552165217521852195220522152225223522452255226522752285229523052315232523352345235523652375238523952405241524252435244524552465247524852495250525152525253525452555256525752585259526052615262526352645265526652675268526952705271527252735274527552765277527852795280528152825283528452855286528752885289529052915292529352945295529652975298529953005301530253035304530553065307530853095310531153125313531453155316531753185319532053215322532353245325532653275328532953305331533253335334533553365337533853395340534153425343534453455346534753485349535053515352535353545355535653575358535953605361536253635364536553665367536853695370537153725373537453755376537753785379538053815382538353845385538653875388538953905391539253935394539553965397539853995400540154025403540454055406540754085409541054115412541354145415541654175418541954205421542254235424542554265427542854295430543154325433543454355436543754385439544054415442544354445445544654475448544954505451545254535454545554565457545854595460546154625463546454655466546754685469547054715472547354745475547654775478547954805481548254835484548554865487548854895490549154925493549454955496549754985499550055015502550355045505550655075508550955105511551255135514551555165517551855195520552155225523552455255526552755285529553055315532553355345535553655375538553955405541554255435544554555465547554855495550555155525553555455555556555755585559556055615562556355645565556655675568556955705571557255735574557555765577557855795580558155825583558455855586558755885589559055915592559355945595559655975598559956005601560256035604560556065607560856095610561156125613561456155616561756185619562056215622562356245625562656275628562956305631563256335634563556365637563856395640564156425643564456455646564756485649565056515652565356545655565656575658565956605661566256635664566556665667566856695670567156725673567456755676567756785679568056815682568356845685568656875688568956905691569256935694569556965697569856995700570157025703570457055706570757085709571057115712571357145715571657175718571957205721572257235724572557265727572857295730573157325733573457355736573757385739574057415742574357445745574657475748574957505751575257535754575557565757575857595760576157625763576457655766576757685769577057715772577357745775577657775778577957805781578257835784578557865787578857895790579157925793579457955796579757985799580058015802580358045805580658075808580958105811581258135814581558165817581858195820582158225823582458255826582758285829583058315832583358345835583658375838583958405841584258435844584558465847584858495850585158525853585458555856585758585859586058615862586358645865586658675868586958705871587258735874587558765877587858795880588158825883588458855886588758885889589058915892589358945895589658975898589959005901590259035904590559065907590859095910591159125913591459155916591759185919592059215922592359245925592659275928592959305931593259335934593559365937593859395940594159425943594459455946594759485949595059515952595359545955595659575958595959605961596259635964596559665967596859695970597159725973597459755976597759785979598059815982598359845985598659875988598959905991599259935994599559965997599859996000600160026003600460056006600760086009601060116012601360146015601660176018601960206021602260236024602560266027602860296030603160326033603460356036603760386039604060416042604360446045604660476048604960506051605260536054605560566057605860596060606160626063606460656066606760686069607060716072607360746075607660776078607960806081608260836084608560866087608860896090609160926093609460956096609760986099610061016102610361046105610661076108610961106111611261136114611561166117611861196120612161226123612461256126612761286129613061316132613361346135613661376138613961406141614261436144614561466147614861496150615161526153615461556156615761586159616061616162616361646165616661676168616961706171617261736174617561766177617861796180618161826183618461856186618761886189619061916192619361946195619661976198619962006201620262036204620562066207620862096210621162126213621462156216

6217

62186219622062216222622362246225622662276228622962306231623262336234623562366237623862396240624162426243624462456246624762486249625062516252625362546255625662576258625962606261626262636264626562666267626862696270627162726273627462756276627762786279628062816282628362846285628662876288628962906291629262936294629562966297629862996300630163026303630463056306630763086309631063116312631363146315631663176318631963206321632263236324632563266327632863296330633163326333633463356336633763386339634063416342634363446345634663476348634963506351635263536354635563566357635863596360636163626363636463656366636763686369637063716372637363746375637663776378637963806381638263836384638563866387638863896390639163926393639463956396639763986399640064016402640364046405640664076408640964106411641264136414641564166417641864196420642164226423642464256426642764286429643064316432643364346435643664376438643964406441644264436444644564466447644864496450645164526453645464556456645764586459646064616462646364646465646664676468646964706471647264736474647564766477647864796480648164826483648464856486648764886489649064916492649364946495649664976498649965006501650265036504650565066507650865096510651165126513651465156516651765186519652065216522652365246525652665276528652965306531653265336534653565366537653865396540654165426543654465456546654765486549655065516552655365546555655665576558655965606561656265636564656565666567656865696570657165726573657465756576657765786579658065816582658365846585658665876588658965906591659265936594659565966597659865996600660166026603660466056606660766086609661066116612661366146615661666176618661966206621662266236624662566266627662866296630663166326633663466356636663766386639664066416642664366446645664666476648664966506651665266536654665566566657665866596660666166626663666466656666666766686669667066716672667366746675667666776678667966806681668266836684668566866687668866896690669166926693669466956696669766986699670067016702670367046705670667076708670967106711671267136714671567166717671867196720672167226723672467256726672767286729673067316732673367346735673667376738673967406741674267436744674567466747674867496750675167526753675467556756675767586759676067616762676367646765676667676768676967706771677267736774677567766777677867796780678167826783678467856786678767886789679067916792679367946795679667976798679968006801680268036804680568066807680868096810681168126813681468156816681768186819682068216822682368246825682668276828682968306831683268336834683568366837683868396840684168426843684468456846684768486849685068516852685368546855685668576858685968606861686268636864686568666867686868696870687168726873687468756876687768786879688068816882688368846885688668876888688968906891689268936894689568966897689868996900690169026903690469056906690769086909691069116912691369146915691669176918691969206921692269236924692569266927692869296930693169326933693469356936693769386939694069416942694369446945694669476948694969506951695269536954695569566957695869596960696169626963696469656966696769686969697069716972697369746975697669776978697969806981698269836984698569866987698869896990699169926993699469956996699769986999700070017002700370047005700670077008700970107011701270137014701570167017701870197020702170227023702470257026702770287029703070317032703370347035703670377038703970407041704270437044704570467047704870497050705170527053705470557056705770587059706070617062706370647065706670677068706970707071707270737074707570767077707870797080708170827083708470857086708770887089709070917092709370947095709670977098709971007101710271037104710571067107710871097110711171127113711471157116711771187119712071217122712371247125712671277128712971307131713271337134713571367137713871397140714171427143714471457146714771487149715071517152715371547155715671577158715971607161716271637164716571667167716871697170717171727173717471757176717771787179718071817182718371847185718671877188718971907191719271937194719571967197719871997200720172027203720472057206720772087209721072117212721372147215721672177218721972207221722272237224722572267227722872297230723172327233723472357236723772387239724072417242724372447245724672477248724972507251725272537254725572567257725872597260726172627263726472657266726772687269727072717272727372747275727672777278727972807281728272837284728572867287728872897290729172927293729472957296729772987299730073017302730373047305730673077308730973107311731273137314731573167317731873197320732173227323732473257326732773287329733073317332733373347335733673377338733973407341734273437344734573467347734873497350735173527353735473557356735773587359736073617362736373647365736673677368736973707371737273737374737573767377737873797380738173827383738473857386738773887389739073917392739373947395739673977398739974007401740274037404740574067407740874097410741174127413741474157416741774187419742074217422742374247425742674277428742974307431743274337434743574367437743874397440744174427443744474457446744774487449745074517452745374547455745674577458745974607461746274637464746574667467746874697470747174727473747474757476747774787479748074817482748374847485748674877488748974907491749274937494749574967497749874997500750175027503

7504

7505750675077508750975107511751275137514751575167517751875197520752175227523752475257526752775287529753075317532753375347535753675377538753975407541754275437544754575467547754875497550755175527553755475557556755775587559756075617562756375647565756675677568756975707571757275737574757575767577757875797580

7581

75827583758475857586758775887589759075917592759375947595759675977598759976007601760276037604760576067607760876097610761176127613761476157616761776187619762076217622762376247625762676277628762976307631763276337634763576367637763876397640764176427643764476457646764776487649765076517652765376547655765676577658765976607661766276637664766576667667766876697670767176727673767476757676767776787679768076817682768376847685768676877688768976907691769276937694769576967697769876997700770177027703770477057706770777087709771077117712771377147715771677177718771977207721772277237724772577267727772877297730773177327733773477357736773777387739774077417742774377447745774677477748774977507751775277537754775577567757775877597760776177627763776477657766776777687769777077717772777377747775777677777778777977807781778277837784778577867787778877897790779177927793779477957796779777987799780078017802780378047805780678077808780978107811781278137814781578167817781878197820782178227823782478257826782778287829

7830

783178327833783478357836783778387839784078417842784378447845784678477848784978507851785278537854785578567857785878597860786178627863786478657866786778687869787078717872787378747875787678777878787978807881788278837884788578867887788878897890789178927893789478957896789778987899790079017902790379047905790679077908790979107911791279137914791579167917791879197920792179227923792479257926792779287929793079317932793379347935793679377938793979407941794279437944794579467947794879497950795179527953795479557956795779587959796079617962796379647965796679677968796979707971797279737974797579767977797879797980798179827983798479857986798779887989799079917992799379947995799679977998799980008001800280038004800580068007800880098010801180128013801480158016801780188019802080218022802380248025802680278028802980308031803280338034803580368037803880398040804180428043804480458046804780488049805080518052805380548055805680578058805980608061806280638064806580668067806880698070807180728073807480758076807780788079808080818082808380848085808680878088808980908091809280938094809580968097809880998100810181028103810481058106810781088109811081118112811381148115811681178118811981208121812281238124812581268127812881298130813181328133813481358136813781388139814081418142814381448145814681478148814981508151815281538154815581568157815881598160816181628163816481658166816781688169817081718172817381748175817681778178817981808181818281838184818581868187818881898190819181928193819481958196819781988199820082018202820382048205820682078208820982108211821282138214821582168217821882198220822182228223822482258226822782288229823082318232823382348235823682378238823982408241824282438244824582468247824882498250825182528253825482558256825782588259826082618262826382648265826682678268826982708271827282738274827582768277827882798280828182828283828482858286828782888289829082918292829382948295829682978298829983008301830283038304830583068307830883098310831183128313831483158316831783188319832083218322832383248325832683278328832983308331833283338334833583368337833883398340834183428343834483458346834783488349835083518352835383548355835683578358835983608361836283638364836583668367836883698370837183728373837483758376837783788379838083818382838383848385838683878388838983908391839283938394839583968397839883998400840184028403840484058406840784088409841084118412841384148415841684178418841984208421842284238424842584268427842884298430843184328433843484358436843784388439844084418442844384448445844684478448844984508451845284538454845584568457845884598460846184628463846484658466846784688469847084718472847384748475847684778478847984808481848284838484848584868487848884898490849184928493849484958496849784988499850085018502850385048505850685078508850985108511851285138514851585168517851885198520852185228523852485258526852785288529853085318532853385348535853685378538853985408541854285438544854585468547854885498550855185528553855485558556

8557

8558855985608561856285638564856585668567856885698570857185728573857485758576857785788579858085818582858385848585858685878588858985908591859285938594859585968597859885998600860186028603860486058606860786088609861086118612861386148615861686178618861986208621862286238624862586268627862886298630863186328633863486358636863786388639864086418642864386448645864686478648864986508651865286538654865586568657865886598660866186628663866486658666866786688669867086718672867386748675867686778678867986808681868286838684868586868687868886898690869186928693869486958696869786988699870087018702870387048705870687078708870987108711871287138714871587168717871887198720872187228723872487258726872787288729873087318732873387348735873687378738873987408741874287438744874587468747874887498750875187528753875487558756875787588759876087618762876387648765876687678768876987708771877287738774877587768777877887798780878187828783878487858786878787888789879087918792879387948795879687978798879988008801880288038804880588068807880888098810881188128813881488158816881788188819882088218822882388248825882688278828882988308831883288338834883588368837883888398840884188428843884488458846884788488849885088518852885388548855885688578858885988608861886288638864886588668867886888698870887188728873887488758876887788788879888088818882888388848885888688878888888988908891889288938894889588968897889888998900890189028903890489058906890789088909891089118912891389148915891689178918891989208921892289238924892589268927892889298930893189328933893489358936893789388939894089418942894389448945894689478948894989508951895289538954895589568957895889598960896189628963896489658966896789688969897089718972897389748975897689778978897989808981898289838984898589868987898889898990899189928993899489958996899789988999900090019002900390049005900690079008900990109011901290139014901590169017901890199020902190229023902490259026902790289029903090319032903390349035903690379038903990409041904290439044904590469047904890499050905190529053905490559056905790589059906090619062906390649065906690679068906990709071907290739074907590769077907890799080908190829083908490859086908790889089909090919092909390949095909690979098909991009101910291039104910591069107910891099110911191129113911491159116911791189119912091219122912391249125912691279128912991309131913291339134913591369137913891399140914191429143914491459146914791489149915091519152915391549155915691579158915991609161916291639164916591669167916891699170917191729173917491759176917791789179918091819182918391849185918691879188918991909191919291939194919591969197919891999200920192029203920492059206920792089209921092119212921392149215921692179218921992209221922292239224922592269227922892299230923192329233923492359236923792389239924092419242924392449245924692479248924992509251925292539254925592569257925892599260926192629263926492659266926792689269927092719272927392749275927692779278927992809281928292839284928592869287928892899290929192929293929492959296929792989299930093019302930393049305930693079308930993109311931293139314931593169317931893199320932193229323932493259326932793289329933093319332933393349335933693379338933993409341934293439344934593469347934893499350935193529353935493559356935793589359936093619362936393649365936693679368936993709371937293739374937593769377937893799380938193829383938493859386938793889389939093919392939393949395939693979398939994009401940294039404940594069407940894099410941194129413941494159416941794189419942094219422942394249425942694279428942994309431943294339434943594369437943894399440944194429443944494459446944794489449945094519452945394549455945694579458945994609461946294639464946594669467946894699470947194729473947494759476947794789479948094819482948394849485948694879488948994909491949294939494949594969497949894999500950195029503950495059506950795089509951095119512951395149515951695179518951995209521952295239524952595269527952895299530953195329533953495359536953795389539954095419542954395449545954695479548954995509551955295539554955595569557955895599560956195629563956495659566956795689569957095719572957395749575957695779578957995809581958295839584958595869587958895899590959195929593959495959596959795989599960096019602960396049605960696079608960996109611961296139614961596169617961896199620962196229623962496259626962796289629963096319632963396349635963696379638963996409641964296439644964596469647964896499650965196529653965496559656965796589659966096619662966396649665966696679668966996709671967296739674967596769677967896799680968196829683968496859686968796889689969096919692969396949695969696979698969997009701970297039704970597069707970897099710

9711

9712971397149715971697179718971997209721972297239724972597269727972897299730973197329733973497359736973797389739974097419742974397449745974697479748974997509751975297539754975597569757975897599760976197629763976497659766976797689769977097719772977397749775977697779778977997809781978297839784978597869787978897899790979197929793979497959796979797989799980098019802980398049805980698079808980998109811981298139814981598169817981898199820982198229823982498259826982798289829983098319832983398349835983698379838983998409841984298439844984598469847984898499850985198529853985498559856985798589859986098619862986398649865986698679868986998709871987298739874987598769877987898799880988198829883988498859886988798889889989098919892989398949895989698979898989999009901990299039904990599069907990899099910991199129913991499159916991799189919992099219922992399249925992699279928992999309931993299339934993599369937993899399940994199429943994499459946994799489949995099519952995399549955995699579958995999609961996299639964996599669967996899699970997199729973997499759976997799789979998099819982998399849985998699879988998999909991999299939994999599969997999899991000010001100021000310004100051000610007100081000910010100111001210013100141001510016100171001810019100201002110022100231002410025100261002710028100291003010031100321003310034100351003610037100381003910040100411004210043100441004510046100471004810049100501005110052100531005410055100561005710058100591006010061100621006310064100651006610067100681006910070100711007210073100741007510076100771007810079100801008110082100831008410085100861008710088100891009010091100921009310094100951009610097100981009910100101011010210103101041010510106101071010810109101101011110112101131011410115101161011710118101191012010121101221012310124101251012610127101281012910130101311013210133101341013510136101371013810139101401014110142101431014410145101461014710148101491015010151101521015310154101551015610157101581015910160101611016210163101641016510166101671016810169101701017110172101731017410175101761017710178101791018010181101821018310184101851018610187101881018910190101911019210193101941019510196101971019810199102001020110202102031020410205102061020710208102091021010211102121021310214102151021610217102181021910220102211022210223102241022510226102271022810229102301023110232102331023410235102361023710238102391024010241102421024310244102451024610247102481024910250102511025210253102541025510256102571025810259102601026110262102631026410265102661026710268102691027010271102721027310274102751027610277102781027910280102811028210283102841028510286102871028810289102901029110292102931029410295102961029710298102991030010301103021030310304103051030610307103081030910310103111031210313103141031510316103171031810319103201032110322103231032410325103261032710328103291033010331103321033310334103351033610337103381033910340103411034210343103441034510346103471034810349103501035110352103531035410355103561035710358103591036010361103621036310364103651036610367103681036910370103711037210373103741037510376103771037810379103801038110382103831038410385103861038710388103891039010391103921039310394103951039610397103981039910400104011040210403104041040510406104071040810409104101041110412104131041410415104161041710418104191042010421104221042310424104251042610427104281042910430104311043210433104341043510436104371043810439104401044110442104431044410445104461044710448104491045010451104521045310454104551045610457104581045910460104611046210463104641046510466104671046810469104701047110472104731047410475104761047710478104791048010481104821048310484104851048610487104881048910490104911049210493104941049510496104971049810499105001050110502105031050410505105061050710508105091051010511105121051310514105151051610517105181051910520105211052210523105241052510526105271052810529105301053110532105331053410535105361053710538105391054010541105421054310544105451054610547105481054910550105511055210553105541055510556105571055810559105601056110562105631056410565105661056710568105691057010571105721057310574105751057610577105781057910580105811058210583105841058510586105871058810589105901059110592105931059410595105961059710598105991060010601106021060310604106051060610607106081060910610106111061210613106141061510616106171061810619106201062110622106231062410625106261062710628106291063010631106321063310634106351063610637106381063910640106411064210643106441064510646106471064810649106501065110652106531065410655106561065710658106591066010661106621066310664106651066610667106681066910670106711067210673106741067510676106771067810679106801068110682106831068410685106861068710688106891069010691106921069310694106951069610697106981069910700107011070210703107041070510706107071070810709107101071110712107131071410715107161071710718107191072010721107221072310724107251072610727107281072910730107311073210733107341073510736107371073810739107401074110742107431074410745107461074710748107491075010751107521075310754107551075610757107581075910760107611076210763107641076510766107671076810769107701077110772107731077410775107761077710778107791078010781107821078310784107851078610787107881078910790

10791

10792107931079410795107961079710798107991080010801108021080310804108051080610807108081080910810108111081210813108141081510816108171081810819108201082110822108231082410825108261082710828108291083010831108321083310834108351083610837108381083910840108411084210843108441084510846108471084810849108501085110852108531085410855108561085710858108591086010861108621086310864108651086610867108681086910870108711087210873108741087510876108771087810879108801088110882

10883

1088410885108861088710888108891089010891108921089310894108951089610897108981089910900109011090210903109041090510906109071090810909109101091110912109131091410915109161091710918109191092010921109221092310924109251092610927109281092910930109311093210933109341093510936109371093810939109401094110942109431094410945109461094710948109491095010951109521095310954109551095610957109581095910960109611096210963109641096510966109671096810969109701097110972109731097410975109761097710978109791098010981109821098310984109851098610987109881098910990109911099210993109941099510996109971099810999110001100111002110031100411005110061100711008110091101011011110121101311014110151101611017110181101911020110211102211023110241102511026110271102811029110301103111032110331103411035110361103711038110391104011041110421104311044110451104611047110481104911050110511105211053110541105511056110571105811059110601106111062110631106411065110661106711068110691107011071110721107311074110751107611077110781107911080110811108211083110841108511086110871108811089110901109111092110931109411095110961109711098110991110011101111021110311104111051110611107111081110911110111111111211113111141111511116111171111811119111201112111122111231112411125111261112711128111291113011131111321113311134111351113611137111381113911140111411114211143111441114511146111471114811149111501115111152111531115411155111561115711158111591116011161111621116311164111651116611167111681116911170111711117211173111741117511176111771117811179111801118111182111831118411185111861118711188111891119011191111921119311194111951119611197111981119911200112011120211203112041120511206112071120811209112101121111212112131121411215112161121711218112191122011221112221122311224112251122611227112281122911230112311123211233112341123511236112371123811239112401124111242112431124411245112461124711248112491125011251112521125311254112551125611257112581125911260112611126211263112641126511266112671126811269112701127111272112731127411275112761127711278112791128011281112821128311284112851128611287112881128911290112911129211293112941129511296112971129811299113001130111302113031130411305113061130711308113091131011311113121131311314113151131611317113181131911320113211132211323113241132511326113271132811329113301133111332113331133411335113361133711338113391134011341113421134311344113451134611347113481134911350113511135211353113541135511356113571135811359113601136111362113631136411365113661136711368113691137011371113721137311374113751137611377113781137911380113811138211383113841138511386113871138811389113901139111392113931139411395113961139711398113991140011401114021140311404114051140611407114081140911410114111141211413114141141511416114171141811419114201142111422114231142411425114261142711428114291143011431114321143311434114351143611437114381143911440114411144211443114441144511446114471144811449114501145111452114531145411455114561145711458114591146011461114621146311464114651146611467114681146911470114711147211473114741147511476114771147811479114801148111482114831148411485114861148711488114891149011491114921149311494114951149611497114981149911500

11501

1150211503115041150511506115071150811509115101151111512115131151411515115161151711518115191152011521115221152311524115251152611527115281152911530115311153211533115341153511536115371153811539115401154111542115431154411545115461154711548115491155011551115521155311554115551155611557115581155911560115611156211563115641156511566115671156811569115701157111572115731157411575115761157711578115791158011581115821158311584115851158611587115881158911590115911159211593115941159511596115971159811599116001160111602116031160411605116061160711608116091161011611116121161311614116151161611617116181161911620116211162211623116241162511626116271162811629116301163111632116331163411635116361163711638116391164011641116421164311644116451164611647116481164911650116511165211653116541165511656116571165811659116601166111662116631166411665116661166711668116691167011671116721167311674116751167611677116781167911680116811168211683116841168511686116871168811689116901169111692116931169411695116961169711698116991170011701117021170311704117051170611707117081170911710117111171211713117141171511716117171171811719117201172111722117231172411725117261172711728117291173011731117321173311734117351173611737117381173911740117411174211743117441174511746117471174811749117501175111752117531175411755117561175711758117591176011761117621176311764117651176611767117681176911770117711177211773117741177511776117771177811779117801178111782117831178411785117861178711788117891179011791117921179311794117951179611797117981179911800118011180211803118041180511806118071180811809118101181111812118131181411815118161181711818118191182011821118221182311824118251182611827118281182911830118311183211833118341183511836118371183811839118401184111842118431184411845118461184711848118491185011851118521185311854118551185611857118581185911860118611186211863118641186511866118671186811869118701187111872118731187411875118761187711878118791188011881118821188311884118851188611887118881188911890118911189211893118941189511896118971189811899119001190111902119031190411905119061190711908119091191011911119121191311914119151191611917119181191911920119211192211923119241192511926119271192811929119301193111932119331193411935119361193711938119391194011941119421194311944119451194611947119481194911950119511195211953119541195511956119571195811959119601196111962119631196411965119661196711968119691197011971119721197311974119751197611977119781197911980119811198211983119841198511986119871198811989119901199111992119931199411995119961199711998119991200012001120021200312004120051200612007120081200912010120111201212013120141201512016120171201812019120201202112022120231202412025120261202712028120291203012031120321203312034120351203612037120381203912040120411204212043120441204512046120471204812049120501205112052120531205412055120561205712058120591206012061120621206312064120651206612067120681206912070120711207212073120741207512076120771207812079120801208112082120831208412085120861208712088120891209012091120921209312094120951209612097120981209912100121011210212103121041210512106121071210812109121101211112112121131211412115121161211712118121191212012121121221212312124121251212612127121281212912130121311213212133121341213512136121371213812139121401214112142121431214412145121461214712148121491215012151121521215312154121551215612157121581215912160121611216212163121641216512166121671216812169121701217112172121731217412175121761217712178121791218012181121821218312184121851218612187121881218912190121911219212193121941219512196121971219812199122001220112202122031220412205122061220712208122091221012211122121221312214122151221612217122181221912220122211222212223122241222512226122271222812229122301223112232122331223412235122361223712238122391224012241122421224312244122451224612247122481224912250122511225212253122541225512256122571225812259122601226112262122631226412265122661226712268122691227012271122721227312274122751227612277122781227912280122811228212283122841228512286122871228812289122901229112292122931229412295122961229712298122991230012301123021230312304123051230612307123081230912310123111231212313123141231512316123171231812319123201232112322123231232412325123261232712328123291233012331123321233312334123351233612337123381233912340123411234212343123441234512346123471234812349123501235112352123531235412355123561235712358123591236012361123621236312364123651236612367123681236912370123711237212373123741237512376123771237812379123801238112382123831238412385123861238712388123891239012391123921239312394123951239612397123981239912400124011240212403124041240512406124071240812409124101241112412124131241412415124161241712418124191242012421124221242312424124251242612427124281242912430124311243212433124341243512436124371243812439124401244112442124431244412445124461244712448124491245012451124521245312454124551245612457124581245912460124611246212463124641246512466124671246812469124701247112472124731247412475124761247712478124791248012481124821248312484124851248612487124881248912490124911249212493124941249512496124971249812499125001250112502125031250412505125061250712508125091251012511125121251312514125151251612517125181251912520125211252212523125241252512526125271252812529125301253112532125331253412535125361253712538125391254012541125421254312544125451254612547125481254912550125511255212553125541255512556125571255812559125601256112562125631256412565125661256712568125691257012571125721257312574125751257612577125781257912580125811258212583125841258512586125871258812589125901259112592125931259412595125961259712598125991260012601126021260312604126051260612607126081260912610126111261212613126141261512616126171261812619126201262112622126231262412625126261262712628126291263012631126321263312634126351263612637126381263912640126411264212643126441264512646126471264812649126501265112652126531265412655126561265712658126591266012661126621266312664126651266612667126681266912670126711267212673126741267512676126771267812679126801268112682126831268412685126861268712688126891269012691126921269312694126951269612697126981269912700127011270212703127041270512706127071270812709127101271112712127131271412715127161271712718127191272012721127221272312724127251272612727127281272912730127311273212733127341273512736127371273812739127401274112742127431274412745127461274712748127491275012751127521275312754127551275612757127581275912760127611276212763127641276512766127671276812769127701277112772127731277412775127761277712778127791278012781127821278312784127851278612787127881278912790127911279212793127941279512796127971279812799128001280112802128031280412805128061280712808128091281012811128121281312814128151281612817128181281912820128211282212823128241282512826128271282812829128301283112832128331283412835128361283712838128391284012841128421284312844128451284612847128481284912850128511285212853128541285512856128571285812859128601286112862128631286412865128661286712868128691287012871128721287312874128751287612877128781287912880128811288212883128841288512886128871288812889128901289112892128931289412895128961289712898128991290012901129021290312904129051290612907129081290912910129111291212913129141291512916129171291812919129201292112922129231292412925129261292712928129291293012931129321293312934129351293612937129381293912940129411294212943129441294512946129471294812949129501295112952129531295412955129561295712958129591296012961129621296312964129651296612967129681296912970129711297212973129741297512976129771297812979129801298112982129831298412985129861298712988129891299012991129921299312994129951299612997129981299913000130011300213003130041300513006130071300813009130101301113012130131301413015130161301713018130191302013021130221302313024130251302613027130281302913030130311303213033130341303513036130371303813039130401304113042130431304413045130461304713048130491305013051130521305313054130551305613057130581305913060130611306213063130641306513066130671306813069130701307113072130731307413075130761307713078130791308013081130821308313084130851308613087130881308913090130911309213093130941309513096130971309813099131001310113102131031310413105131061310713108131091311013111131121311313114131151311613117131181311913120131211312213123131241312513126131271312813129131301313113132131331313413135131361313713138131391314013141131421314313144131451314613147131481314913150131511315213153131541315513156131571315813159131601316113162131631316413165131661316713168131691317013171131721317313174131751317613177131781317913180131811318213183131841318513186131871318813189131901319113192131931319413195131961319713198131991320013201132021320313204132051320613207132081320913210132111321213213132141321513216132171321813219132201322113222132231322413225132261322713228132291323013231132321323313234132351323613237132381323913240132411324213243132441324513246132471324813249132501325113252132531325413255132561325713258132591326013261132621326313264132651326613267132681326913270132711327213273132741327513276132771327813279132801328113282132831328413285132861328713288132891329013291132921329313294132951329613297132981329913300133011330213303133041330513306133071330813309133101331113312133131331413315133161331713318133191332013321133221332313324133251332613327133281332913330133311333213333133341333513336133371333813339133401334113342133431334413345133461334713348133491335013351133521335313354133551335613357133581335913360133611336213363133641336513366133671336813369133701337113372133731337413375133761337713378133791338013381133821338313384133851338613387133881338913390133911339213393133941339513396133971339813399134001340113402134031340413405134061340713408134091341013411134121341313414134151341613417134181341913420134211342213423134241342513426134271342813429134301343113432134331343413435134361343713438134391344013441134421344313444134451344613447134481344913450134511345213453134541345513456134571345813459134601346113462134631346413465134661346713468134691347013471134721347313474134751347613477134781347913480134811348213483134841348513486134871348813489134901349113492134931349413495134961349713498134991350013501135021350313504135051350613507135081350913510135111351213513135141351513516135171351813519135201352113522135231352413525135261352713528135291353013531135321353313534135351353613537135381353913540135411354213543135441354513546135471354813549135501355113552135531355413555135561355713558135591356013561135621356313564135651356613567135681356913570135711357213573135741357513576135771357813579135801358113582135831358413585135861358713588135891359013591135921359313594135951359613597135981359913600136011360213603136041360513606136071360813609136101361113612136131361413615136161361713618136191362013621136221362313624136251362613627136281362913630136311363213633136341363513636136371363813639136401364113642136431364413645136461364713648136491365013651136521365313654136551365613657136581365913660136611366213663136641366513666136671366813669136701367113672136731367413675136761367713678136791368013681136821368313684136851368613687136881368913690136911369213693136941369513696136971369813699137001370113702137031370413705137061370713708137091371013711137121371313714137151371613717137181371913720137211372213723

13724

137251372613727137281372913730137311373213733137341373513736137371373813739137401374113742137431374413745137461374713748137491375013751137521375313754137551375613757137581375913760137611376213763137641376513766137671376813769137701377113772137731377413775137761377713778137791378013781137821378313784137851378613787137881378913790137911379213793137941379513796137971379813799138001380113802138031380413805138061380713808138091381013811138121381313814138151381613817138181381913820138211382213823138241382513826138271382813829138301383113832138331383413835138361383713838138391384013841138421384313844138451384613847138481384913850138511385213853138541385513856138571385813859138601386113862138631386413865138661386713868138691387013871138721387313874138751387613877138781387913880138811388213883138841388513886138871388813889138901389113892138931389413895138961389713898138991390013901139021390313904139051390613907139081390913910139111391213913139141391513916

13917

13918139191392013921139221392313924139251392613927

13928

139291393013931139321393313934139351393613937139381393913940139411394213943139441394513946139471394813949139501395113952139531395413955139561395713958139591396013961139621396313964139651396613967139681396913970139711397213973139741397513976139771397813979139801398113982139831398413985139861398713988139891399013991139921399313994139951399613997139981399914000140011400214003140041400514006140071400814009140101401114012140131401414015140161401714018140191402014021140221402314024140251402614027140281402914030140311403214033140341403514036140371403814039140401404114042140431404414045140461404714048140491405014051140521405314054140551405614057140581405914060140611406214063140641406514066140671406814069140701407114072140731407414075140761407714078140791408014081140821408314084140851408614087140881408914090140911409214093140941409514096140971409814099141001410114102141031410414105141061410714108141091411014111141121411314114141151411614117141181411914120141211412214123141241412514126141271412814129141301413114132

14133

1413414135141361413714138141391414014141141421414314144141451414614147141481414914150141511415214153141541415514156141571415814159141601416114162141631416414165141661416714168141691417014171141721417314174141751417614177141781417914180141811418214183141841418514186141871418814189141901419114192141931419414195141961419714198141991420014201142021420314204142051420614207142081420914210142111421214213142141421514216142171421814219142201422114222142231422414225142261422714228142291423014231142321423314234142351423614237142381423914240142411424214243142441424514246142471424814249142501425114252142531425414255142561425714258142591426014261142621426314264142651426614267142681426914270142711427214273142741427514276142771427814279

14280

142811428214283142841428514286142871428814289142901429114292142931429414295142961429714298142991430014301143021430314304143051430614307143081430914310143111431214313143141431514316143171431814319143201432114322143231432414325143261432714328143291433014331143321433314334143351433614337143381433914340143411434214343143441434514346143471434814349143501435114352143531435414355143561435714358143591436014361143621436314364143651436614367143681436914370143711437214373143741437514376143771437814379143801438114382143831438414385143861438714388143891439014391143921439314394143951439614397143981439914400144011440214403144041440514406144071440814409144101441114412144131441414415144161441714418144191442014421144221442314424144251442614427144281442914430144311443214433144341443514436144371443814439144401444114442144431444414445144461444714448144491445014451144521445314454144551445614457144581445914460144611446214463144641446514466144671446814469144701447114472144731447414475144761447714478144791448014481144821448314484144851448614487144881448914490144911449214493144941449514496144971449814499145001450114502145031450414505145061450714508145091451014511145121451314514145151451614517145181451914520145211452214523145241452514526145271452814529145301453114532145331453414535145361453714538145391454014541145421454314544145451454614547145481454914550145511455214553145541455514556145571455814559145601456114562145631456414565145661456714568145691457014571145721457314574145751457614577145781457914580145811458214583145841458514586145871458814589145901459114592145931459414595145961459714598145991460014601146021460314604146051460614607146081460914610146111461214613146141461514616146171461814619146201462114622146231462414625146261462714628146291463014631146321463314634146351463614637146381463914640146411464214643146441464514646146471464814649146501465114652146531465414655146561465714658146591466014661146621466314664146651466614667146681466914670146711467214673146741467514676146771467814679146801468114682146831468414685146861468714688146891469014691146921469314694146951469614697146981469914700147011470214703147041470514706147071470814709147101471114712147131471414715147161471714718147191472014721147221472314724147251472614727147281472914730147311473214733147341473514736147371473814739147401474114742147431474414745147461474714748147491475014751147521475314754147551475614757147581475914760147611476214763147641476514766147671476814769147701477114772147731477414775147761477714778147791478014781147821478314784147851478614787147881478914790147911479214793147941479514796147971479814799148001480114802148031480414805148061480714808148091481014811148121481314814148151481614817148181481914820148211482214823148241482514826148271482814829148301483114832148331483414835148361483714838148391484014841148421484314844148451484614847148481484914850148511485214853148541485514856148571485814859148601486114862148631486414865148661486714868148691487014871148721487314874148751487614877148781487914880148811488214883148841488514886148871488814889148901489114892148931489414895148961489714898148991490014901149021490314904149051490614907149081490914910149111491214913149141491514916149171491814919149201492114922149231492414925149261492714928149291493014931149321493314934149351493614937149381493914940149411494214943149441494514946149471494814949149501495114952149531495414955149561495714958149591496014961149621496314964149651496614967149681496914970149711497214973149741497514976149771497814979149801498114982149831498414985149861498714988149891499014991149921499314994149951499614997149981499915000150011500215003150041500515006150071500815009150101501115012150131501415015150161501715018150191502015021150221502315024150251502615027150281502915030150311503215033150341503515036150371503815039150401504115042150431504415045150461504715048150491505015051150521505315054150551505615057150581505915060150611506215063150641506515066150671506815069150701507115072150731507415075150761507715078150791508015081150821508315084150851508615087150881508915090150911509215093150941509515096150971509815099151001510115102151031510415105151061510715108151091511015111151121511315114151151511615117151181511915120151211512215123151241512515126151271512815129151301513115132151331513415135151361513715138151391514015141151421514315144151451514615147151481514915150151511515215153151541515515156151571515815159151601516115162151631516415165151661516715168151691517015171151721517315174151751517615177151781517915180151811518215183151841518515186151871518815189151901519115192151931519415195151961519715198151991520015201152021520315204152051520615207152081520915210152111521215213152141521515216152171521815219152201522115222152231522415225152261522715228152291523015231152321523315234152351523615237152381523915240152411524215243152441524515246152471524815249152501525115252152531525415255152561525715258152591526015261152621526315264152651526615267152681526915270152711527215273152741527515276152771527815279152801528115282152831528415285152861528715288152891529015291152921529315294152951529615297152981529915300153011530215303153041530515306153071530815309153101531115312153131531415315153161531715318153191532015321153221532315324153251532615327153281532915330153311533215333153341533515336153371533815339153401534115342153431534415345153461534715348153491535015351153521535315354153551535615357153581535915360153611536215363153641536515366153671536815369153701537115372153731537415375153761537715378153791538015381153821538315384153851538615387153881538915390153911539215393153941539515396153971539815399154001540115402154031540415405154061540715408154091541015411154121541315414154151541615417154181541915420154211542215423154241542515426154271542815429154301543115432154331543415435154361543715438154391544015441154421544315444154451544615447154481544915450154511545215453154541545515456154571545815459154601546115462154631546415465154661546715468154691547015471154721547315474154751547615477154781547915480154811548215483154841548515486154871548815489154901549115492154931549415495154961549715498154991550015501155021550315504155051550615507155081550915510155111551215513155141551515516155171551815519155201552115522155231552415525155261552715528155291553015531155321553315534155351553615537155381553915540155411554215543155441554515546155471554815549155501555115552155531555415555155561555715558155591556015561155621556315564155651556615567155681556915570155711557215573155741557515576155771557815579155801558115582155831558415585155861558715588155891559015591155921559315594155951559615597155981559915600156011560215603156041560515606156071560815609156101561115612156131561415615156161561715618156191562015621156221562315624156251562615627156281562915630156311563215633156341563515636156371563815639156401564115642156431564415645156461564715648156491565015651156521565315654156551565615657156581565915660156611566215663156641566515666156671566815669156701567115672156731567415675156761567715678156791568015681156821568315684156851568615687156881568915690156911569215693156941569515696156971569815699157001570115702157031570415705157061570715708157091571015711157121571315714157151571615717157181571915720157211572215723157241572515726157271572815729157301573115732157331573415735157361573715738157391574015741157421574315744157451574615747157481574915750157511575215753157541575515756157571575815759157601576115762157631576415765157661576715768157691577015771157721577315774157751577615777157781577915780157811578215783157841578515786157871578815789157901579115792157931579415795157961579715798157991580015801158021580315804158051580615807158081580915810158111581215813158141581515816158171581815819158201582115822158231582415825158261582715828158291583015831158321583315834158351583615837158381583915840158411584215843158441584515846158471584815849158501585115852158531585415855158561585715858158591586015861158621586315864158651586615867158681586915870158711587215873158741587515876158771587815879158801588115882158831588415885158861588715888158891589015891158921589315894158951589615897158981589915900159011590215903159041590515906159071590815909159101591115912159131591415915159161591715918159191592015921159221592315924159251592615927159281592915930159311593215933159341593515936159371593815939159401594115942159431594415945159461594715948159491595015951159521595315954159551595615957159581595915960159611596215963159641596515966159671596815969159701597115972159731597415975159761597715978159791598015981159821598315984159851598615987159881598915990159911599215993159941599515996159971599815999160001600116002160031600416005160061600716008160091601016011160121601316014160151601616017160181601916020160211602216023160241602516026160271602816029160301603116032160331603416035160361603716038160391604016041160421604316044160451604616047160481604916050160511605216053160541605516056160571605816059160601606116062160631606416065160661606716068160691607016071160721607316074160751607616077160781607916080160811608216083160841608516086160871608816089160901609116092160931609416095160961609716098160991610016101161021610316104161051610616107161081610916110161111611216113161141611516116161171611816119161201612116122161231612416125161261612716128161291613016131161321613316134161351613616137161381613916140161411614216143161441614516146161471614816149161501615116152161531615416155161561615716158161591616016161161621616316164161651616616167161681616916170161711617216173161741617516176161771617816179161801618116182161831618416185161861618716188161891619016191161921619316194161951619616197161981619916200162011620216203162041620516206162071620816209162101621116212162131621416215162161621716218162191622016221162221622316224162251622616227162281622916230162311623216233162341623516236162371623816239162401624116242162431624416245162461624716248162491625016251162521625316254162551625616257162581625916260162611626216263162641626516266162671626816269162701627116272162731627416275162761627716278162791628016281162821628316284162851628616287162881628916290162911629216293162941629516296162971629816299163001630116302163031630416305163061630716308163091631016311163121631316314163151631616317163181631916320163211632216323163241632516326163271632816329163301633116332163331633416335163361633716338163391634016341163421634316344163451634616347163481634916350163511635216353163541635516356163571635816359163601636116362163631636416365163661636716368163691637016371163721637316374163751637616377163781637916380163811638216383163841638516386163871638816389163901639116392163931639416395163961639716398163991640016401164021640316404164051640616407164081640916410164111641216413164141641516416164171641816419164201642116422164231642416425164261642716428164291643016431164321643316434164351643616437164381643916440164411644216443164441644516446164471644816449164501645116452164531645416455164561645716458164591646016461164621646316464164651646616467164681646916470164711647216473164741647516476164771647816479164801648116482164831648416485164861648716488164891649016491164921649316494164951649616497164981649916500165011650216503165041650516506165071650816509165101651116512165131651416515165161651716518165191652016521165221652316524165251652616527165281652916530165311653216533165341653516536165371653816539165401654116542165431654416545165461654716548165491655016551165521655316554165551655616557165581655916560165611656216563165641656516566165671656816569165701657116572165731657416575165761657716578165791658016581165821658316584165851658616587165881658916590165911659216593165941659516596165971659816599166001660116602166031660416605166061660716608166091661016611166121661316614166151661616617166181661916620166211662216623166241662516626166271662816629166301663116632166331663416635166361663716638166391664016641166421664316644166451664616647166481664916650166511665216653166541665516656166571665816659166601666116662166631666416665166661666716668166691667016671166721667316674166751667616677166781667916680166811668216683166841668516686166871668816689166901669116692166931669416695166961669716698166991670016701167021670316704167051670616707167081670916710167111671216713167141671516716167171671816719167201672116722167231672416725167261672716728167291673016731167321673316734167351673616737167381673916740167411674216743167441674516746167471674816749167501675116752167531675416755167561675716758167591676016761167621676316764167651676616767167681676916770167711677216773167741677516776167771677816779167801678116782167831678416785167861678716788167891679016791167921679316794167951679616797167981679916800168011680216803168041680516806168071680816809168101681116812168131681416815168161681716818168191682016821168221682316824168251682616827168281682916830168311683216833168341683516836168371683816839168401684116842168431684416845168461684716848168491685016851168521685316854168551685616857168581685916860168611686216863168641686516866168671686816869168701687116872168731687416875168761687716878168791688016881168821688316884168851688616887168881688916890168911689216893168941689516896168971689816899169001690116902169031690416905169061690716908169091691016911169121691316914169151691616917169181691916920169211692216923169241692516926169271692816929169301693116932169331693416935169361693716938169391694016941169421694316944169451694616947169481694916950169511695216953169541695516956169571695816959169601696116962169631696416965169661696716968169691697016971169721697316974169751697616977169781697916980169811698216983169841698516986169871698816989169901699116992169931699416995169961699716998169991700017001170021700317004170051700617007170081700917010170111701217013170141701517016170171701817019170201702117022170231702417025170261702717028170291703017031170321703317034170351703617037170381703917040170411704217043170441704517046170471704817049170501705117052170531705417055170561705717058170591706017061170621706317064170651706617067170681706917070170711707217073170741707517076170771707817079170801708117082170831708417085170861708717088170891709017091170921709317094170951709617097170981709917100171011710217103171041710517106171071710817109171101711117112171131711417115171161711717118171191712017121171221712317124171251712617127171281712917130171311713217133171341713517136171371713817139171401714117142171431714417145171461714717148171491715017151171521715317154171551715617157171581715917160171611716217163171641716517166171671716817169171701717117172171731717417175171761717717178171791718017181171821718317184171851718617187171881718917190171911719217193171941719517196171971719817199172001720117202172031720417205172061720717208172091721017211172121721317214172151721617217172181721917220172211722217223172241722517226172271722817229172301723117232172331723417235172361723717238172391724017241172421724317244172451724617247172481724917250172511725217253172541725517256172571725817259172601726117262172631726417265172661726717268172691727017271172721727317274172751727617277172781727917280172811728217283172841728517286172871728817289172901729117292172931729417295172961729717298172991730017301173021730317304173051730617307173081730917310173111731217313173141731517316173171731817319173201732117322173231732417325173261732717328173291733017331173321733317334173351733617337173381733917340173411734217343173441734517346173471734817349173501735117352173531735417355173561735717358173591736017361173621736317364173651736617367173681736917370173711737217373173741737517376173771737817379173801738117382173831738417385173861738717388173891739017391173921739317394173951739617397173981739917400174011740217403174041740517406174071740817409174101741117412174131741417415174161741717418174191742017421174221742317424174251742617427174281742917430174311743217433174341743517436174371743817439174401744117442174431744417445174461744717448174491745017451174521745317454174551745617457174581745917460174611746217463174641746517466174671746817469174701747117472174731747417475174761747717478174791748017481174821748317484174851748617487174881748917490174911749217493174941749517496174971749817499175001750117502175031750417505175061750717508175091751017511175121751317514175151751617517175181751917520175211752217523175241752517526175271752817529175301753117532175331753417535175361753717538175391754017541175421754317544175451754617547175481754917550175511755217553175541755517556175571755817559175601756117562175631756417565175661756717568175691757017571175721757317574175751757617577175781757917580175811758217583175841758517586175871758817589175901759117592175931759417595175961759717598175991760017601176021760317604176051760617607176081760917610176111761217613176141761517616176171761817619176201762117622176231762417625176261762717628176291763017631176321763317634176351763617637176381763917640176411764217643176441764517646176471764817649176501765117652176531765417655176561765717658176591766017661176621766317664176651766617667176681766917670176711767217673176741767517676176771767817679176801768117682176831768417685176861768717688176891769017691176921769317694176951769617697176981769917700177011770217703177041770517706177071770817709177101771117712177131771417715177161771717718177191772017721177221772317724177251772617727177281772917730177311773217733177341773517736177371773817739177401774117742177431774417745177461774717748177491775017751177521775317754177551775617757177581775917760177611776217763177641776517766177671776817769177701777117772177731777417775177761777717778177791778017781177821778317784177851778617787177881778917790177911779217793177941779517796177971779817799178001780117802178031780417805178061780717808178091781017811178121781317814178151781617817178181781917820178211782217823178241782517826178271782817829178301783117832178331783417835178361783717838178391784017841178421784317844178451784617847178481784917850178511785217853178541785517856178571785817859178601786117862178631786417865178661786717868178691787017871178721787317874178751787617877178781787917880178811788217883178841788517886178871788817889178901789117892178931789417895178961789717898178991790017901179021790317904179051790617907179081790917910179111791217913179141791517916179171791817919179201792117922179231792417925179261792717928179291793017931179321793317934179351793617937179381793917940179411794217943179441794517946179471794817949179501795117952179531795417955179561795717958179591796017961179621796317964179651796617967179681796917970179711797217973179741797517976179771797817979179801798117982179831798417985179861798717988179891799017991179921799317994179951799617997179981799918000180011800218003180041800518006180071800818009180101801118012180131801418015180161801718018180191802018021180221802318024180251802618027180281802918030180311803218033180341803518036180371803818039180401804118042180431804418045180461804718048180491805018051180521805318054180551805618057180581805918060180611806218063180641806518066180671806818069180701807118072180731807418075180761807718078180791808018081180821808318084180851808618087180881808918090180911809218093180941809518096180971809818099181001810118102181031810418105181061810718108181091811018111181121811318114181151811618117181181811918120181211812218123181241812518126181271812818129181301813118132181331813418135181361813718138181391814018141181421814318144181451814618147181481814918150181511815218153181541815518156181571815818159181601816118162181631816418165181661816718168181691817018171181721817318174181751817618177181781817918180181811818218183181841818518186181871818818189181901819118192181931819418195181961819718198181991820018201182021820318204182051820618207182081820918210182111821218213182141821518216182171821818219182201822118222182231822418225182261822718228182291823018231182321823318234182351823618237182381823918240182411824218243182441824518246182471824818249182501825118252182531825418255182561825718258182591826018261182621826318264182651826618267182681826918270182711827218273182741827518276182771827818279182801828118282182831828418285182861828718288182891829018291182921829318294182951829618297182981829918300183011830218303183041830518306183071830818309183101831118312183131831418315183161831718318183191832018321183221832318324183251832618327183281832918330183311833218333183341833518336183371833818339183401834118342183431834418345183461834718348183491835018351183521835318354183551835618357183581835918360183611836218363183641836518366183671836818369183701837118372183731837418375183761837718378183791838018381183821838318384183851838618387183881838918390183911839218393183941839518396183971839818399184001840118402184031840418405184061840718408184091841018411184121841318414184151841618417184181841918420184211842218423184241842518426184271842818429184301843118432184331843418435184361843718438184391844018441184421844318444184451844618447184481844918450184511845218453184541845518456184571845818459184601846118462184631846418465184661846718468184691847018471184721847318474184751847618477184781847918480184811848218483184841848518486184871848818489184901849118492184931849418495184961849718498184991850018501185021850318504185051850618507185081850918510185111851218513185141851518516185171851818519185201852118522185231852418525185261852718528185291853018531185321853318534185351853618537185381853918540185411854218543185441854518546185471854818549185501855118552185531855418555185561855718558185591856018561185621856318564185651856618567185681856918570185711857218573185741857518576185771857818579185801858118582185831858418585185861858718588185891859018591185921859318594185951859618597185981859918600186011860218603186041860518606186071860818609186101861118612186131861418615186161861718618186191862018621186221862318624186251862618627186281862918630186311863218633186341863518636186371863818639186401864118642186431864418645186461864718648186491865018651186521865318654186551865618657186581865918660186611866218663186641866518666186671866818669186701867118672186731867418675186761867718678186791868018681186821868318684186851868618687186881868918690186911869218693186941869518696186971869818699187001870118702187031870418705187061870718708187091871018711187121871318714187151871618717187181871918720187211872218723187241872518726187271872818729187301873118732187331873418735187361873718738187391874018741187421874318744187451874618747187481874918750187511875218753187541875518756187571875818759187601876118762187631876418765

18766

1876718768187691877018771187721877318774187751877618777187781877918780187811878218783187841878518786187871878818789187901879118792187931879418795187961879718798187991880018801188021880318804188051880618807188081880918810188111881218813188141881518816188171881818819188201882118822188231882418825188261882718828188291883018831188321883318834188351883618837188381883918840188411884218843188441884518846188471884818849188501885118852188531885418855188561885718858188591886018861188621886318864188651886618867188681886918870188711887218873188741887518876188771887818879188801888118882188831888418885188861888718888188891889018891188921889318894188951889618897188981889918900189011890218903189041890518906189071890818909189101891118912189131891418915189161891718918189191892018921189221892318924189251892618927189281892918930189311893218933189341893518936189371893818939189401894118942189431894418945189461894718948189491895018951189521895318954189551895618957189581895918960189611896218963189641896518966189671896818969189701897118972189731897418975

−0.2

0.0

0.2

0.4

0.6

0.8

Figure 2: Best subset selection results for the trim32 data with bess.seq. The optimal kvalue is determined by EBIC, which is indicated by a orange vertical dashed line.

Next we call the function bess with argument method = "gsection" to perform the GPDASalgorithm. At each iteration, it outputs the split information used in the GPDAS.

R> fit.gs <- bess(X, Y, family = "gaussian", method = "gsection",

R+ epsilon = 1e-2)


1-th iteration s.left:1 s.split:16 s.right:25





From the above code, we know that the best selected model has 6 predictors and the algorithmends at the 5-th iteration. To show more information about the best selected model, we extractit via fit.gs$bestmodel and present its summary information via the S3 method summary.

R> bm.gs <- fit.gs$bestmodel

R> summary(bm.gs)

Call:


Residuals:


-0.114598 -0.036829 -0.007365 0.041804 0.161688

Coefficients:


(Intercept) 1.41219 0.50792 2.780 0.006363 **

xbest1368316_at -0.16680 0.03479 -4.794 5.02e-06 ***

xbest1372248_at 0.22120 0.05579 3.965 0.000129 ***

xbest1373887_at 0.27947 0.05707 4.897 3.27e-06 ***

xbest1387160_at -0.12456 0.02989 -4.168 6.05e-05 ***

xbest1389910_at 0.54459 0.07220 7.543 1.25e-11 ***

xbest1381978_a_at -0.16091 0.03454 -4.658 8.77e-06 ***

---

Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1



F-statistic: 99.28 on 6 and 113 DF, p-value: < 2.2e-16

The corresponding coefficients can be extracted by running the coef function on the bestmodelsubject or the fitted bess model.

R> coef(bm.gs)

(Intercept) xbest1368316_at xbest1372248_at xbest1373887_at

1.4121869 -0.1668030 0.2211982 0.2794672

xbest1387160_at xbest1389910_at xbest1381978_a_at

-0.1245576 0.5445936 -0.1609133

The output of running the coef function directly on the bess output can be sparse matrixas shown below, with the last column corresponding to the best fitted coefficients.


R> beta <- coef(fit.gs, sparse = TRUE)

R> class(beta)

[1] "dgCMatrix"

attr(,"package")

[1] "Matrix"

R> beta[, ncol(beta)][which(beta[, ncol(beta)]!=0)]

(intercept) 1368316_at 1372248_at 1373887_at 1387160_at

1.4121869 -0.1668030 0.2211982 0.2794672 -0.1245576

1389910_at 1381978_a_at

0.5445936 -0.1609133

We can also output the predicted response via the predict function as follows.

R> pred.gs <- predict(fit.gs, newdata = X)

5. Discussion

In this paper, we introduce the primal dual active set (PDAS) algorithm for solving thebest subset selection problem and implement it as the BeSS package in the R statisticalenvironment. The PDAS algorithm allows identification of the best sub-model with a pre-specified model size via a primal-dual formulation of feasible solutions. To determine thebest sub-model over all possible model sizes, both a sequential search algorithm (SPDAS)and a golden section search algorithm (GPDAS) are proposed. We find that the GPDAS isespecially efficient and accurate in selecting variables for high-dimensional and sparse data.

Package BeSS provides R users with a new and flexible way to carry out best subset selectionfor sparse LM, GLM and CoxPH models. It allows us to identify the best sub-model over allcandidates efficiently (usually in seconds or a few minutes) even when the number of predictorsis extremely large, say p ≈ 10000, based on a standard personal computer. In both simulationand real data examples, it was shown that the BeSS package is highly efficient compared toother state-of-the-art methods. Future versions of the package will focus on scenarios includingthe “large-p-big-n” problem and more general types of data (e.g., Multinormial).

Acknowledgments

We are grateful to all those who have provided comments and encouragements while thepackage was being developed.

References

Akaike H (1974). “A New Look At The Statistical Model Identification.” IEEE transactionson automatic control, 19(6), 716–723.


Bertsimas D, King A, Mazumder R, et al. (2016). “Best Subset Selection Via A ModernOptimization Lens.” The Annals of Statistics, 44(2), 813–852.

Calcagno V, de Mazancourt C, et al. (2010). “glmulti: An R Package for Easy AutomatedModel Selection with (Generalized) Linear Models.” Journal of statistical software, 34(12),1–29.

Chen J, Chen Z (2008). “Extended Bayesian Information Criteria for Model Selection withLarge Model Spaces.” Biometrika, pp. 759–771.

Chen J, Chen Z (2012). “Extended BIC for small-n-large-P Sparse GLM.” Statistica Sinica,pp. 555–574.

Cox D (1972). “Regression Models and Life-Tables.” Journal of the Royal Statistical SocietyB, pp. 187–220.

Fan J, Lv J (2008). “Sure Independence Screening for Ultrahigh Dimensional Feature Space.”Journal of the Royal Statistical Society B, 70(5), 849–911.

Friedman J, Hastie T, Tibshirani R (2001). The Elements of Statistical Learning, volume 1.Springer series in statistics, Springer-Verlag, Berlin.

Friedman J, Hastie T, Tibshirani R (2010). “Regularization Paths for Generalized LinearModels via Coordinate Descent.” Journal of statistical software, 33(1), 1–22. URL http:

//www.jstatsoft.org/v33/i01/.

Golub GH, Van Loan CF (2012). Matrix Computations, volume 3. JHU Press.

Heagerty PJ, Lumley T, Pepe MS (2000). “Time-dependent ROC Curves for Censored Sur-vival Data and a Diagnostic Marker.” Biometrics, 56(2), 337–344.

Huang J, Jiao Y, Liu Y, Lu X (2017). “A Constructive Approach to High-dimensional Re-gression.” 1701.05128, URL http://arxiv.org/abs/1701.05128.

Ito K, Kunisch K (2013). “A Variational Approach to Sparsity Optimization Based on La-grange Multiplier Theory.” Inverse problems, 30(1), 015001.

Jiao Y, Jin B, Lu X (2015). “A Primal Dual Active Set with Continuation Algorithm for thel0-Regularized Optimization Problem.” Applied and Computational Harmonic Analysis,39(3), 400–426.

Lumley T, Miller A (2017). Leaps: Regression Subset Selection. URL https://CRAN.R-

project.org/package=leaps.

McLeod A, Xu C (2010). bestglm: Best Subset GLM. URL https://CRAN.R-project.org/

package=bestglm.

R Core Team (2016). R: A Language and Environment for Statistical Computing. R Founda-tion for Statistical Computing, Vienna, Austria. URL https://www.R-project.org.

Scheetz TE, Kim KYA, et al. (2006). “Regulation of Gene Expression in the MammalianEye and Its Relevance to Eye Disease.” Proceedings of the National Academy of Sciences,103(39), 14429–14434.

http://www.jstatsoft.org/v33/i01/


1701.05128

http://arxiv.org/abs/1701.05128

https://CRAN.R-project.org/package=leaps

https://CRAN.R-project.org/package=leaps

https://CRAN.R-project.org/package=bestglm

https://CRAN.R-project.org/package=bestglm

https://www.R-project.org


Schwarz G, et al. (1978). “Estimating the Dimension of a Model.” The annals of statistics,6(2), 461–464.

Simon N, Friedman J, Hastie T, Tibshirani R (2011). “Regularization Paths for Cox’s Pro-portional Hazards Model via Coordinate Descent.” Journal of statistical software, 39(5),1–13. URL http://www.jstatsoft.org/v39/i05/.

Wen C, Zhang A, Quan S, Wang X (2017). BeSS: Best Subset Selection for Sparse GeneralizedLinear Model and Cox Model. R package version 1.0.1, URL https://CRAN.R-project.

org/package=BeSS.

Wolters MA (2015). “A Genetic Algorithm for Selection of Fixed-Size Subsets with Applicationto Design Problems.” Journal of Statistical Software, Code Snippets, 68(1), 1–18. doi:

10.18637/jss.v068.c01.

Affiliation:

Canhong WenDepartment of Statistics and Finance, School of ManagementUniversity of Science and Technology of China230026 Hefei, AH, ChinaE-mail: [email protected] of Statistics, School of MathematicsSouthern China Center for Statistical ScienceSun Yat-sen University510275 Guangzhou, GD, China

Aijun ZhangDepartment of Statistics and Actuarial ScienceThe University of Hong KongHong Kong, ChinaE-mail: [email protected]

Shijie QuanDepartment of Statistics, School of MathematicsSouthern China Center for Statistical ScienceSun Yat-sen University510275 Guangzhou, GD, China


https://CRAN.R-project.org/package=BeSS

https://CRAN.R-project.org/package=BeSS

http://dx.doi.org/10.18637/jss.v068.c01

http://dx.doi.org/10.18637/jss.v068.c01

mailto:[email protected]



Xueqin WangDepartment of Statistics, School of MathematicsSouthern China Center for Statistical ScienceSun Yat-sen University510275 Guangzhou, GD, ChinaE-mail: [email protected]

Journal of Statistical Software http://www.jstatsoft.org/

published by the Foundation for Open Access Statistics http://www.foastat.org/

MMMMMM YYYY, Volume VV, Issue II Submitted: yyyy-mm-dddoi:10.18637/jss.v000.i00 Accepted: yyyy-mm-dd


http://www.jstatsoft.org/

http://www.foastat.org/

http://dx.doi.org/10.18637/jss.v000.i00


Low-dimensional Method p = 20 p = 30 p = 40

Time leaps 0.00(0.01) 0.39(0.13) 58.79(28.78)bestglm 0.02(0.01) 0.51(0.15) 69.39(32.27)glmulti 11.91(2.60) 18.41(4.13) —glmnet 0.08(0.02) 0.09(0.02) 0.08(0.01)bess.seq 0.18(0.01) 0.23(0.02) 0.25(0.03)bess.gs 0.16(0.01) 0.18(0.02) 0.17(0.02)

MSE leaps 1.91(0.83) 2.18(0.81) 2.44(1.15)(×10−2) bestglm 1.91(0.83) 2.18(0.81) 2.44(1.15)

glmulti 1.87(0.72) 2.16(0.79) —glmnet 3.90(1.30) 3.51(1.23) 3.51(1.37)bess.seq 1.93(0.82) 2.12(0.76) 2.43(1.21)bess.gs 2.14(2.45) 2.06(1.78) 2.80(3.37)

TP leaps 3.97(0.17) 3.99(0.10) 3.97(0.17)bestglm 3.97(0.17) 3.99(0.10) 3.97(0.17)glmulti 3.99(0.10) 4.00(0.00) —glmnet 3.96(0.20) 3.97(0.17) 3.95(0.22)bess.seq 3.96(0.20) 3.91(0.35) 3.84(0.44)bess.gs 3.78(0.42) 3.73(0.51) 3.63(0.61)

FP leaps 2.37(1.83) 3.92(2.39) 5.53(2.66)bestglm 2.37(1.83) 3.92(2.39) 5.53(2.66)glmulti 2.29(1.63) 4.15(2.29) —glmnet 0.73(0.80) 0.82(0.83) 0.78(1.10)bess.seq 3.75(4.25) 4.98(5.80) 7.59(8.64)bess.gs 1.35(2.94) 4.31(6.93) 5.42(8.74)

High-dimensional Method p = 100 p = 1000 p = 10000

Time glmnet 0.16(0.03) 1.77(0.09) 14.82(1.73)bess.seq 1.29(0.09) 74.54(1.33) 137.04(13.80)bess.gs 0.53(0.12) 3.72(0.41) 12.87(2.89)

MSE glmnet 1.42(0.18) 2.51(0.28) 2.47(0.22)(×10−2) bess.seq 1.65(0.41) 1.20(0.62) 0.70(0.23)

bess.gs 1.33(0.29) 0.98(0.37) 1.00(0.35)

TP glmnet 39.74(0.54) 39.80(0.45) 39.75(0.46)bess.seq 35.30(2.17) 38.72(1.29) 39.53(0.70)bess.gs 35.78(2.12) 39.43(0.88) 39.58(0.71)

FP glmnet 15.45(3.65) 12.73(5.50) 29.82(11.91)bess.seq 27.15(10.66) 4.92(6.99) 0.32(1.92)bess.gs 28.86(8.90) 1.51(2.53) 3.06(3.84)

Table 1: Simulation results for linear regression. Time stands for run times (CPU seconds),MSE stands for Mean Squared Error, TP stands for true positive number and FP stands forfalse positive number. The number of true nonzero coefficients is q = 4 for low-dimensionalcases and q = 40 for high-dimensional cases.



Time bestglm 1.83(0.15) 7.55(0.26) 28.35(1.93)glmulti 2.08(0.11) 13.91(2.43) 21.61(4.54)glmnet 0.49(0.07) 0.56(0.09) 0.63(0.17)bess.seq 0.70(0.33) 0.79(0.35) 0.78(0.52)bess.gs 0.52(0.20) 0.78(1.14) 0.65(0.23)

Acc bestglm 0.949(0.012) 0.950(0.013) 0.950(0.011)glmulti 0.949(0.012) 0.950(0.013) 0.950(0.011)glmnet 0.949(0.013) 0.951(0.013) 0.950(0.011)bess.seq 0.949(0.012) 0.950(0.013) 0.950(0.011)bess.gs 0.948(0.013) 0.951(0.012) 0.949(0.013)

TP bestglm 3.99(0.10) 4.00(0.00) 3.99(0.10)glmulti 3.99(0.10) 4.00(0.00) 4.00(0.00)glmnet 4.00(0.00) 4.00(0.00) 4.00(0.00)bess.seq 3.96(0.20) 3.95(0.30) 3.91(0.32)bess.gs 3.87(0.37) 3.87(0.42) 3.89(0.40)

FP bestglm 0.73(0.85) 1.02(1.05) 1.41(1.44)glmulti 0.73(0.85) 1.02(1.05) 1.37(1.20)glmnet 1.62(0.96) 2.07(1.16) 2.83(1.44)bess.seq 1.77(1.59) 2.19(2.20) 2.39(2.40)bess.gs 0.15(0.41) 0.31(0.93) 0.64(1.57)



Acc glmnet 0.969(0.006) 0.945(0.009) 0.922(0.011)bess.seq 0.963(0.012) 0.972(0.011) 0.979(0.006)bess.gs 0.970(0.010) 0.976(0.008) 0.978(0.009)



Table 2: Simulation results for logistic regression. Time stands for run times (CPU seconds),Acc stands for classification accuracy, TP stands for true positive number and FP stands forfalse positive number. The number of true nonzero coefficients is q = 4 for low-dimensionalcases and q = 20 for high-dimensional cases.



Time glmulti 1.53(0.06) 10.11(1.75) 15.20(2.86)glmnet 1.07(0.20) 1.09(0.20) 1.16(0.23)bess.seq 0.42(0.20) 0.49(0.23) 0.52(0.22)bess.gs 0.35(0.15) 0.46(0.19) 0.51(0.18)

AUC glmulti 0.973(0.012) 0.972(0.010) 0.974(0.010)glmnet 0.973(0.012) 0.972(0.010) 0.974(0.010)bess.seq 0.973(0.012) 0.972(0.010) 0.974(0.010)bess.gs 0.972(0.012) 0.972(0.010) 0.974(0.011)

TP glmulti 4.00(0.00) 3.99(0.10) 4.00(0.00)glmnet 4.00(0.00) 4.00(0.00) 4.00(0.00)bess.seq 4.00(0.00) 4.00(0.00) 4.00(0.00)bess.gs 3.89(0.35) 3.96(0.20) 3.99(0.10)

FP glmulti 0.60(0.77) 1.06(1.17) 1.14(1.21)glmnet 1.17(1.01) 1.56(1.04) 1.82(1.14)bess.seq 1.62(1.69) 1.98(2.25) 2.38(2.69)bess.gs 0.11(0.35) 0.04(0.20) 0.06(0.37)



AUC glmnet 0.993(0.005) 0.992(0.006) 0.991(0.007)bess.seq 0.993(0.005) 0.992(0.006) 0.991(0.007)bess.gs 0.990(0.008) 0.992(0.006) 0.991(0.007)



Table 3: Simulation results for CoxPH regression. Time stands for run times (CPU seconds),AUC stands for the integrated time-dependent area under the curve, TP stands for true posi-tive number and FP stands for false positive number. The number of true nonzero coefficientsis q = 4 for low-dimensional cases and q = 20 for high-dimensional cases.


Dataset n p Type Data source

prostate 97 9 Continuous R package ElemStatLearn

SAheart 462 8 Binary R package ElemStatLearn

trim32 120 18975 Continuous Scheetz, Kim et al. (2006)

leukemia 72 3571 Binary R package spikeslab

gravier 168 2905 Binary https://github.com/ramhiser/

er0 609 22285 Survival https://www.ncbi.nlm.nih.gov/geo/

Table 4: Description for the real data sets. Here n denotes the number of observations, pdenotes the number of predictors, and ‘Type’ denotes the type of response.

Data Method leaps bestglm glmulti glmnet bess.seq bess.gs

prostate Time 0.00(0.01) 0.01(0.01) 0.61(0.05) 0.07(0.01) 0.22(0.01) 0.22(0.01)PE 0.61(0.14) 0.61(0.14) 0.61(0.14) 0.65(0.19) 0.60(0.13) 0.60(0.14)MS 4.27(1.11) 4.25(1.12) 4.25(1.12) 3.58(0.87) 4.29(1.17) 6.11(0.87)

SAheart Time — 1.58(0.07) 4.03(0.53) 0.13(0.01) 0.27(0.04) 0.26(0.04)Acc — 0.72(0.03) 0.72(0.03) 0.70(0.04) 0.72(0.03) 0.72(0.03)MS — 5.68(0.98) 5.68(0.98) 4.61(0.84) 5.68(0.99) 6.29(1.09)

trim32 Time — — — 3.23(0.15) 1.95(0.53) 1.08(0.19)PE — — — 0.01(0.01) 0.01(0.01) 0.01(0.00)MS — — — 24.89(11.79) 1.60(0.62) 7.82(2.26)

leukemia Time — — — 0.38(0.01) 1.74(0.77) 1.14(0.53)Acc — — — 0.93(0.05) 0.90(0.06) 0.91(0.06)MS — — — 11.76(4.40) 1.54(0.77) 2.00(0.00)

gravier Time — — — 0.68(0.03) 6.64(4.09) 2.93(2.50)Acc — — — 0.71(0.07) 0.72(0.06) 0.72(0.06)MS — — — 10.83(7.39) 9.23(1.05) 10.80(2.47)

er0 Time — — — 154.97(15.75) 184.51(86.15) 55.20(22.07)AUC — — — 0.52(0.04) 0.53(0.05) 0.60(0.05)MS — — — 3.06(7.35) 1.02(0.14) 56.85(6.90)

Table 5: Results for the real data sets. Time stands for run times (CPU seconds), MS standsfor the size of selected model. PE stands for mean prediction error in linear model; Accstands for classification accuracy in logistic regression model; AUC stands for the integratedtime-dependent area under the curve in CoxPH regression model.

https://github.com/ramhiser/datamicroarray/wiki/Gravier-(2010)

https://www.ncbi.nlm.nih.gov/geo/

Documents

BeSS: An R Package for Best Subset Selection in GLM and ... · PDF fileThe canonical approach to subset selection problem is to ... arising in modern statistics. Recently,Bertsimas,