Upload
shathis007
View
50
Download
5
Tags:
Embed Size (px)
DESCRIPTION
Beowulf Cluster Server
Citation preview
CHAPTER 1
INTRODUCTION
1.1 Background
Figure 1.1 – Depiction of Beowulf in old English poems
Literally, the term Beowulf is deriverd from an old English poem , which tells the story of
how a single warrior hero has the capability and strength of "thirty men's heft of grasp in the
gripe of his hand". The same concept can be seen in the application of the Beowulf Cluster
Server, where multiple separate computers function together to form a powerful, single
1
computer unit. The term Beowulf in computing is actually first referred to a single computer
developed by Thomas Sterling and Donald Becker for NASA in 1994. However, today the
term Beowulf is mostly recognized as a reference to the Parallel Computing architecture
which comprises of various components functioning as one.
1.2 Problem Statement
Figure 1.2 – The cost of setting up a server can sometimes be astronomical
The main issue regarding setting up a conventional server is the fact that a substantial
amount of cost is required in the building of a server. This single factor is considered to be
one of the major deterrence for new users to be able to set up a server of their own.
However, should there be an alternative method of setting up a server system which could
provide the same level of performance as that of a conventional server; moreover at only a
fraction of the cost, more and more users would be inclined to choose this alternative method
in creating a server of their own. The solution to this is to introduce the Beowulf Clustered
Server system.
1.3 Objectives
1. To provide an alternative method of creating a server.
2. Create a Beowulf Cluster Server using used components.
3. To verify connectivity between nodes using Ping and SSH tests.
2
1.4 Scope
This project will be developed according to the scope of personal users, small business and
organizations. The Beowulf Cluster Server is developed to cater their needs of conventional
server system, minus the cost.
1.4.1 System Scope
The system is intended to perform basic server tasks such as web hosting, FTP
hosting, DNS service, or even DHCP service, just as any server would do. In terms
of the usage scope, the Beowulf Cluster Server is capable of operating similarly to a
regular server. In addition, it may also be used for processor-intensive applications
such as graphic renderings and calculations which involve larger processing loads.
1.4.2 User Scope
The system is targeted for use by personal users, small business and organizations
which require the service of a conventional server computer. This system will prove
to be particularly useful for users who are in possession of outdated desktop PC
hardware, such as those which are abundantly found in schools, institutions and
businesses, where they can easily be transformed into a Beowulf Cluster Server.
1.5 Project Significance
Our priority lies first and foremost in the aspect of saving cost, in which case the aspect of
keeping the total project cost to a minimum. This is to comply with our second objective,
which states that this project is intended to be efficient in terms of costs. Apart from the
factors of cost, our second priority regarding this project is to make the method appeal to the
masses. By promoting the use of Server Clustering technology, we are able to diversify the
user market by giving them another alternative. Doing so will undoubtedly help promote
Open Source technology to the masses, asides from allowing more freedom to users in
setting up and experimenting with new server technologies.
3
1.6 Conclusion
Conclusively, we hope to achieve the goal of introducing the Beowulf Cluster Server to the
masses in attempt to create an awareness of Server Clustering methods and it’s potential. In
the local Malaysian market, not many businesses are able to either support the cost of
owning a server or maintain it. However, if the masses were informed of this alternative,
there would no doubt be an increase in the use of Cluster Server technology in the local IT
industry.
4
CHAPTER 2
LITERATURE REVIEW
2.1 Introduction
In the aspects of research carried out for this project, it involves the study of Parallel
Computing as a whole, which ranges from the basics of Cluster Computing to comparisons
between other methods of Computer Clustering. In order to provide a more detailed
perspective of Cluster Computing, various researches has been carried out including the
study of similar existing projects, comparisons of locally available projects, and the research
of current and past market prices of computer hardware.
2.2 History of the Beowulf Cluster Server
The concept of Beowulf Server Clusters originated at the Center of Excellence in Space Data
and Information Sciences (CESDIS), located at the NASA Goddard Space Flight Center in
Maryland, United States of America in 1994. Donald Becker and Thomas Sterling were the
two prominent figures responsible for developing the Beowulf Clusters, where they
envisioned the goal of building a Beowulf cluster to create a cost-effective parallel
computing system from mass-market commodity, off-the-shelf components to satisfy
specific computational requirements in the earth and space sciences. The usage of the name
5
Beowulf stems from an old English poem, which tells the tale of the main character, a
warrior hero who has "thirty men's heft of grasp in the gripe of his hand".
2.2.1 Definition of the Beowulf Cluster Server
A Beowulf cluster is a group of usually identical servers running Unix-like
Operating Systems, such as Linux. They are networked into a small TCP/IP
LAN, and have libraries and programs installed which allow processing to be
shared among them. There is no particular piece of software that defines a
cluster as a Beowulf. Commonly used parallel processing libraries include
MPI (Message Passing Interface) and PVM (Parallel Virtual Machine).
Both of these permit the user to divide tasks among a group of networked
computers, and recollect the results of processing.
2.2.2 Usage of the Beowulf Cluster Server
Figure 2.1 – The Beowulf Cluster can even be utilized for personal use
Beowulf Cluster Servers have a wide array of uses; they can be used for virtually
everything, ranging from simple data mining, file serving, database serving, or web
serving, to flight simulation, computer graphics rendering, weather modelling, or
even simple number crunching applications. Today, the Beowulf Cluster Server
6
method has been adopted and used worldwide, mainly in support of scientific
computing.
2.3 Comparable Projects
Figure 2.2 – The Borg, a Beowulf Cluster setup
Most projects involving the Beowulf Cluster Server are normally done on a large scale. For
example, the McGill University Pulsar Group located in Quebec, Canada, has developed a
working Beowulf Cluster called The Borg which consists of 52 nodes all working together to
search for pulsations from binary pulsars.
2.3.1 The IBM Sequoia
The IBM Sequoia is one of the most prominent examples of a cluster built on a large
scale. The Sequoia is a cluster supercomputer developed by IBM for use by the
National Nuclear Security Administration of America in June of 2012 where it was
fully operational. The supercomputer was programmed to handle various tasks, most
of which concerned scientific research, such as astronomy, study of the human
genome, energy and climate change. However, its primary use was to provide
nuclear weapons simulation. The IBM Sequoia is a massive cluster of nodes which
consists of 98,304 computer nodes, each equipped with a 16 core CPU and 16
Gigabytes of DDR3 RAM. The Sequoia is too large in terms of scale to be compared
7
with a project of our stature. However, this provides us with evidence of how a
Cluster Computer can function as a single, large, powerful system unit.
2.3.2 The Stone Soupercomputer
Built in the late 1990’s, the Stone Soupercomputer was a computer cluster built
using the Beowulf Cluster method. Developed by William Hargrove and Forrest
Hoffman, the two lab employees at the Oak Ridge National Laboratory designed the
cluster using freely available software such as the Linux OS, Private Virtual
Machine (PVM) and Message Passing Interfaces (MPI) and discarded Desktop PCs
which were deemed as too slow. As of May 2001, the Stone Soupercomputer
consisted of 133 nodes which consisted of Intel 80486, Intel Pentium and DEC
Alpha workstations. The clustered computer was built only to serve the purpose of
aiding in the studies of cluster computing. The Stone cluster was no longer in use as
of August 2003.
2.3.3 The DEGIMA
The Destination for GPU Intensive Machine, or shortened as NEGIMA, is a high
performance computer cluster used for the purpose of studying N-body simulations
in the fields of astrophysics. The cluster computer is situated in the Nagasaki
Advanced Computing Center in Nagasaki, Japan. The DEGIMA consists of roughly
144 computer nodes each equipped with an Intel i7 CPU, two GeForce GTX295
Graphic Cards 12 Gigabyte DDR3 RAMs and interconnected via an InfiniBand
interconnection. The DEGIMA system was proven to be one of the most highly
energy and cost efficient methods of clustering.
8
2.4 Conclusion
Figure 2.3 – The scalability factor of Cluster Servers is endless
Conclusively, the research and literature studies are all important aspects in developing a
quality system. This chapter covers the comparisons between this project and other projects
similar and relatable to this. Comparisons are also made in order to identify each of their
strengths and weaknesses. However, due to the fact that Cluster Computing has yet to
become popular in Malaysia, difficulties in terms of obtaining information regarding
prototype clusters from local Private Institutions has denied us the chance to research more
on the Cluster technologies used in Malaysia.
9
CHAPTER 3
METHODOLOGY
3.1 Introduction
The planning report is one of the methods available for use which can be adopted to ensure
the smoothness in executing our project. This detailed report would aid us in planning the
project in greater detail, based on the planned date and activities. Activities planning are
made are implemented during the course of carrying out the project. This is to ensure that
whatever that has been planned, such as project schedules (Gantt charts), budget cost
estimations and determining the project scope will complete the objectives.
3.2 Project Development
Before carrying out the development or implementation of the project, it is crucial to first
determine the methods to be used in advance. In this project, the Waterfall method aids in
helping ease the process of planning the development of the Beowulf Cluster Server. The
phases of a method is also important, primarily because it helps keep the the project’s
development to be more systematic and organized, precise time organization, allowing for
modifications, correcting errors and eventually granting the success of the project.
10
Figure 3.1 – The Waterfall method flowchart
3.3 Planning Phase
This project is developed for the sole purpose of providing a means of hosting web services
without purchasing a complete server unit. Within the development phase, several important
aspects of the development process have gone into careful consideration. These include the
project’s scope of use and estimated project costs.
3.4 Estimated Costs
The estimated costs for the development of this project are mostly reserved for the
procurement of the hardware required for setting up a Beowulf Cluster Server, which mostly
11
1st Phase
involves the use of common Desktop PCs, Switches and UTP cabling. A more detailed list is
provided as follows :
SOFTWARE QUANTITYCOST PER
UNIT
TOTAL
COST
Intel i3 Desktop Computer
- Intel i3 3.30 GHz (Dual core)
- 4GB DDR3 RAM
- 500GB SATA HDD
1 unit RM 1000.00 RM 1000.00
Intel Pentium 4 Desktop Computer
- Intel Pentium 4 1.80 GHz (Single core)
- 1GB DDR1 RAM
- 20GB IDE HDD
2 units RM 300.00 RM 600.00
Linux Ubuntu Operating System 3 copies Free of charge Free of charge
6-Port Network Hub 1 unit RM 50.00 RM 50.00
Category 5E Straight-through UTP Cable 3 pieces RM 5.00 RM 15.00
TOTAL COST RM 1665.00
Table 1 – Project cost estimation
3.5 Conclusion
In conclusion, the stated planning report has helped us in carrying out this project according
to the schedule. We had provided both the project planning and cost estimations to aid us
developing this project with ease and efficiently.
12
CHAPTER 4
ANALYSIS
4.1 Introduction
Analysis is the method of gathering data to be compiled into a useful form of information in
a way of various measures. Analysis should be based on subjects in a project scope such as
computer users, society, organizations and IT experts. Gathered data should then be
processed into an understandable form to avoid making mistakes in other phases of carrying
out the project. In this chapter, we will go through on the methods of analysis and how the
data are gathered and processed. Additionally, we will pursue project guidelines both on the
network and technical aspects.
4.2 Project Requirements
Beowulf Clustered Servers are an alternative method to setup a fully functional server for
those who are lacking the resources or funding to purchase a custom server. The requirement
of this project is to make sure all objectives can be achieved. Clustered servers should
perform well in processing data equivalent or more powerful than any conventional server.
Configuration on software and hardware should be done carefully as it will affect the
smoothness of the overall project development.
13
4.3 Analysis Techniques
The method of analysis adopted by us for this project is the use of the multiple choice
questionnaire method.
4.3.1 Questionnaire
Please refer to Appendix A attached at the back of this report.
4.3.2 Research
Research is the method of gathering and processing raw data before it is used to
develop a project. It can be done by reading, observation, interviewing a person or
any necessary subject and by conducting testing. In our case, we carry out research
by studying tutorials on the internet, and by making an appointment with other
lecturers. Other than that, we also use try and error methods, most of which are
based on configuration of nodes and installed packages. Also, it helps the progress
of our project.
4.4 Network Project Environment Guidelines
4.4.1 Number of users.
Only 1 user is required, either a network administrator or a single end user.
4.4.2 Locations of data collected.
Testing in the Project Laboratory of JTMK in front of our supervisor
without connecting to the internet.
4.4.3 Current internet connectivity
No, we only use local LAN network.
4.4.4 Existing network infrastructure.
Not available because this project platform is created in a new network.
14
4.4.5 Reliability and uptime expectations.
Reliability is high because we use 3 nodes. If one fails, others can still run.
Uptime is 24/7 depending on user requirements.
4.4.6 Security and privacy.
We implement SSH between nodes to ensure security and privacy.
4.4.7 Budget Constraint
Budget only allows purchasing of used equipment.
4.5 Technical Requirement Guidelines
4.5.1 Hardware requirements
1 Intel i3 Desktop PC
2 Intel Pentium 4 Desktop PCs
3 Linux Ubuntu Ver. 12.0.4
1 Desktop Switch
3 UTP Straight-through cables
4.5.2 Operating system requirement
Linux Ubuntu Version 12.0.4
4.5.3 Network requirements
IP configurations and local LAN network.
15
4.5.4 Physical topology
Figure 4.1 – The Beowulf Cluster Server Topology
4.5.5 Networking software
OpensshServer
Mpich2-1.5
OpenMPI
NFS
4.5.6 Internetworking devices
1 Desktop switch
16
CHAPTER 5
DEVELOPMENT AND TESTING
5.1 Introduction
The implementation and testing phases are detrimental to a project’s development. How the
method is carried out and applied would have a great impact on how the project as a whole
would be formed. In the implementation of the Beowulf Cluster Server, it involves the
configuration of the Linux OS config files, installing MPI file packages, enabling SSH
protocols, and applying Firewall rules. On the other hand, testing of this project would mean
the operation of the Beowulf Cluster Server, and ensuring that the output results which it
produces are in line with the expected results given. By ensuring both of these criteria, the
project will be deemed as a success and complete.
5.2 Development
Please refer to Appendix B attached at the back of this report.
5.3 Testing
Testing is carried out in order to identify possible issues regarding the project. The project
will simulate a trial run, and its outcome will be compared with a hypothesis, or a test
expectation. If the results come out differently, then proper addressing and correction of the
problem will need to be carried out. The purpose of carrying out testing is to ensure that this
17
project is capable of operating within the desired scope of use. In the Beowulf Cluster Server
project, testing is performed by the following method :
Change directory to our mirror folder and write this MPI helloworld program in a file mpi_hello.c .
#include <stdio.h>
#include <mpi.h>
int main(int argc, char** argv) {
int myrank, nprocs;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
printf("Hello from processor %d of %d\n", myrank, nprocs);
MPI_Finalize();
return 0;
}
Compile the following :
mpiuser@node0:~$ mpicc mpi_hello.c -o mpi_hello
18
and run it (the parameter next to -n specifies the number of processes to spawn and distribute among nodes):
mpiuser@node0:~$ mpiexec -n 8 -f machinefile ./mpi_hello
We should now see the output similar to the following :
Hello from processor 0 of 1
5.4 Conclusion
As a whole, this section of the documentation explains in detail the techniques and method
used for testing the project, and what to be expected from the actual results. The importance
of carrying out testing is not only that it would allow the project to be developed without
problems and occurring errors, but also ensure that all the objectives are met and satisfies the
fulfil requirements.
19
CHAPTER 6
CONCLUSION AND SUGGESTIONS
6.1 Introduction
This section of the Project Report discusses the conclusion of the Beowulf Cluster Server
project as a whole, the overall achievements of its objectives, improvements, its advantages
and disadvantages and also constructive comments for future considerations. This section is
important in that it ensures the project is capable of operating and benefitting its users alike.
6.2 Overall Objective Achievement
Conclusively, after completing this project within the allocated time, the end result is a
system which meets the objective criteria which is the Beowulf Cluster Server system. This
system was developed to be used as an alternative method of providing a computer server at
a fraction of the cost. The usage and implementation of this method will serve as an
indication that the objective of this project has been achieved.
6.3 Contributions
The development of this project is aimed for users to be provided with an alternative in
setting a host server. The biggest dilemma regarding server set ups and small businesses,
20
organizations and personal users is that Cluster Servers allow for the utilization of a normal
host server, but with a cheaper price tag. Users will be given the choice of cost efficiency
and complete freedom over their desired server setup.
6.4 System Advantages
The main selling point of this project are listed as the following :
i. The Beowulf Cluster Server method allows users to choose another method of
setting up a server without the need to purchase a ready built unit. Users need
only gather any usable Desktop PC component to form a Beowulf Cluster.
ii. Used Desktop and computer hardware are easy to come across. They can also be
bought for relatively cheap prices, due to the fact that they are obsolete.
iii. The Beowulf Cluster Server is freely scalable, meaning that it can be designed
with as little as two nodes to hundreds of nodes.
iv. Beowulf Cluster Servers are also reliable in terms of operational uptime. If one
node malfunctions, other remaining nodes will be able to continue operating
without being interrupted.
v. By adopting Cluster Server technology and reusing obsolete computer
components, more and more used hardware components can be recycled and
reused without being discarded.
21
6.5 System Disadvantages
Although the Beowulf Cluster Server is capable of attaining the set objectives, there are also
disadvantages to the project which need to be addressed. These include :
i. The hardware used in Cluster Server may not be as durable as custom server-use
hardware. Server hardware is designed to be used non-stop, being able to
tolerate harsh operating conditions which Desktop equipment are not meant for.
ii. Obtaining current equipment may prove to be expensive in the future. Due to the
introduction of new technology, equipment, including hardware will become
obsolete much faster, and more expensive.
iii. Setting up Beowulf Cluster Servers mean users would need to master the basics
of Linux operation. For users who are accustomed to operate Microsoft Windows
and Apple Mac Operating Systems, there will be a steep learning curve for them.
iv. Cluster Server setups would require a large amount of space to be allocated for
the placement of the nodes. Also increased cooling will be necessary for those
nodes.
6.6 Future Considerations
In order to further improvise the current Beowulf Cluster Server project, a number of
suggestions and recommendations has been listed for future considerations :
i. The use of better performing equipment to improve the overall specifications of
the completed Beowulf Cluster Server.
ii. Adding more Slave nodes to the Cluster setup to test scalability factors and to
also improve performance.
22
iii. Connect the Beowulf Cluster Server to an outside network to test internetwork
operability and access.
6.7 Conclusion
Overall, this chapter discusses in detail regarding the advantages and disadvantages of the
project, its importance, its contributions and also recommendations for future improvements.
It is hoped that the success of this project would help encourage users to adopt this method
in addition to the current existing methods of setting up servers.
23
REFERENCES
For the duration of carrying out this project, we have consulted various people and sources
of references, amongst which includes :
WEBSITES
“HPC Overview”, http://www.digitalwaves.in/hpc-overview.html
“Parallel Computing”, http://www.dmoz.org/computers/parallel_computing/
“MPICH Configuration”, https://help.ubuntu.com/community/MpichCluster
BOOKS
“Introduction to Parallel Computing” , Ananth Grama, Anshul Gupta, George
Karypis, Vipin Kumar (Addison-Wesley).
PEOPLE
Encik Mohd. Assiddiq Bin Che Ahmad, JTMK, Politeknik Ungku Omar.
Encik Norshahril Bin Abu Talib, JTMK, Politeknik Ungku Omar.
24