DVL Final Report

8/8/2019 DVL Final Report

1/64

Disk Virtualization onLinux

Chapter 1

Introduction

Chapter 1

1


2/64


3/64

Disk Virtualization onLinuxThose of you who need to keep large quantities of data on hand (such as an average

administrator) would benefit by using this technology. Primary reasons to use the

system include:

Enhanced speed

Increased storage capacity

Reducing complexity for managing disks

1.1.3 System overview

Fig1.1: Disk virtualization: Overview Client:

1: client application will request for storing the data.

2: client application will request for accessing the data.

3: client application will delete stored data.

Central server:

1: keep the track of request from client.

2: Managing the cluster node information.

3: Authenticate the users.

4: central server responsible for storing and retrieving data.

Cluster nodes:

1: Cluster nodes together forms single virtual disk(logical abstraction of

physical disks).

3


4/64


2: Every member of a cluster is a cluster itself.

3. Every member will cluster the disk of its own LAN network and will

provide a virtual space and acts as a member of the cluster.

1.1.4 Working of Proposed System:

Fig1.2 (a): Working of proposed system (Storing data)

Fig1.2 (b): Working of proposed system (Retrieving data)

The end user will have the functionality of File Management. The user need to

connect to the centralized server, uploads a file. The centralized server provides

Creatingaccount for

New UserLogin

Virtual FileServer

Splitting

Storing blockon Cluster

Node

4


5/64

Disk Virtualization onLinuxInformation to splits the file into blocks and keeps each block on cluster nodes. When

the user retrieves the file the client application uses centralized server caches all the

blocks merges them and send the copy to requesting client. This module also provides

administrative functions such as monitoring/management of the space (memory) used

by the clients.

1.2 Scope

Provision for storage platforms and reducing the need to maintain duplicate

data for sharing.

Improve application and data availability, and lower failover costs by reducing

or eliminating application downtime for storage and data management tasks.

Simplify and lower the cost of data backups through application server.

Provide an on demand storage environment for fast and simple deployment

through file level virtualization, policy-based automation, and support of open

standards.

1.3 Objectives

1. Design of Virtual File Server

2. Managing Cluster.

3. Managing Space and User Accounts.

4. Security to username and password by Encryption.

5. File Splitting.

6. File Merging

7. File Deleting.

8. Adding and Deleting Nodes from the cluster.

5


6/64


Chapter 2

Requirement analysis &System Design

Chapter 2

Requirement Analysis & System Design

6


7/64

Disk Virtualization onLinuxRequirement analysis help the software engineer to better understand the problem they

will work to solve. It includes the set of tasks that lead to an understanding:-

1. What customer wants exactly?

2. What is the information proposed by the system?

3. What function the systems perform?

4. What is the behavior the system?

2.1 General Aspect

2.1.1 Initiating the Process

The most commonly used analysis technique to bridge communication gap

between customer and the developer and to get the communication process started.

The idea of project disk virtualization in Linux came from the demand of customer for

using this system as a mode for having store file as a number of blocks on any clusternodes.

2.1.2 Information Domain

The information domain consists of three different views of the data and control

and each is processed by a computer program.

Information content and relationships for this we collected specific

need about the requirements of the customer ,jotting down the field names and

functionalities of the system

Information flow: It represents the manner in which data and control

each move through the system .This is required for the analysis of the flow of

control in the system giving the idea about the different modules and parts of the

software system.

Information structure: It represents the internal organization of various

data and control items. It provides us the information about the data structure, files

and tables to be used with functionality in the system.

2.2 Requirements Specification 2.2.1Normal Requirements

N1. Virtual management applied to systems that make many small resources look

like one system.

7


8/64


N2. Providing virtual space without interaction with system and no need to worry

about storage space available on server.

N3. Adding and Deleting Nodes from the cluster.

N4. Reducing cost for storing the data.

N5: Graphical user interface provides user for easy interaction with application

software.

2.2.2 Expected Requirements

Exp1. Client application that allows the authorized user to interact with the

system.

Exp2. File to be splits and merge

Exp3. Security to user accounts.

2.2.3 Exciting Requirements

Ex1. System must splits and merges any type and size of file or data.

2.3 Validation of Requirements N1: a system developed whereby a large number of low cost hard drives could be

linked together to form a single large capacity storage device that offered superior

performance, storage capacity and reliability over older storage solutions.

N2: e.g., 3 machines of 50 GB each, or 5 machines of 30 GB. The clients would

request the central server for storage space giving him. The centralized server

should provide information to splits the file into blocks and keeps track of each

block.

Exp1: Every user interacting with system must have account.

Exp2: The user need to connect to the centralized server, uploads & download a

file.

2.4 Software Requirements

Linux ,Windows

Jdk1.5,MySQL

2.5 Hardware Requirements

Processor P3 or above( for minimum three systems)

RAM 256 MB

XLANBIT Hub with minimum 8 ports

8


9/64


2.6 Process Model

The goal of system design is to produce a model or representation that exhibit,

commodity and delight. It provides information about the application domain for the

software to be built. it fully describes the internal details of each software.

Fig 2.1: Linear sequential model

2.7 Breakdown Structure (Modules)-

Fig 2.2 DVL: Breakdown structure (modules)

CommunicationProject initiationRequirement gathering

PlanningEstimatingSchedulingTracking

ModelingAnalysis

Design

ConstructionCodingTesting

DeploymentDeliverySupportFeedback

Virtual fileserver

Clientapplication

Subnetclustering

AuthenticationSplitting &Merging

Analysis

DVSystem

ClusterEncryption

9


10/64

Disk Virtualization onLinux2.7.1 Virtual file Server

The Server programs links all the clients connected to the System. This module

keeps track of the client file, the cluster machines. The Server Program will be

installed on one machine and will process the request being given by the client

program.

Analysis

The Administrator can make an analysis cluster nodes & Client Application.

2.7.2 Client Application

This module is a user interface system through which the users can interact

with the system. Its a user side File Explorer that allows the authorized user to

interact with the system. Client software will be installed on every machine from

where you want to access the Virtual Disk functionality.

2.7.3 Authentication and Encryption

Every user of the System should be an Authorized user. The User before starts

working should register itself. The Username and password at the time of registration

will be used to login to the system. Username and Password will be encrypted.

2.7.4 Subnet Clustering

This module will link the entire Cluster and will form another cluster. Means every

member of a cluster is a cluster itself. Every member will cluster the disk of its own

LAN network and will provide a virtual space and acts as a member of the cluster.

Here the admin can add disk or can remove a disk. He can decide which Subnet can

be used as member. This module will be used by the administrator to allot Cluster

member. Here the admin can add disk or can remove a disk. He can decide which

Subnet can be used as member.

Cluster

10


11/64

Disk Virtualization onLinux This application accepts the data from client and gives the track for storing file.

Also sends the data to the user according to request provided by client.

2.7.5 Splitting and Merging

This module splits the user file into number of blocks and keeps each block on

each member of the cluster. This technique is based on reading file in the form of bytes

& then splitting file as per no of nodes. Whenever client wants to access particular file

then this request passes to central server. At the time of retrieval of the same file Client

application requests central server and access the file blocks, save into temporary file

merge them and send to the requested client.

2.8 Project Estimation

2.8.1 Estimation in KLOC

Function Estimated KLOC

Virtual file server 1-1.20

Client application 0.5-1

Splitter & merger 1.0- 1.25

Subnet Clustering 0.5 0.75

Graphical user interface 0.25-0.30

Authentication 0.25-0.40

Total: 3.50-4.90

2.8.2 Efforts in person in months

E= 3.2 * (KLOC) ^ 1.05

E=3.2*3.50^1.05 to 3.2*4.90^1.05

E11.92 to 16.17 months

2.8.3 Development time in months

D=E/N, where E=Effort in person month, N=number of persons required.

D=11.92/3 to 16.17/3

D=4 to 4.5 months

2.8.4 No of persons required

3 persons are required to complete the project with given time span

successfully.

11


12/64


Chapter 3

System Analysis & RiskManagement

12


13/64


13


14/64


Chapter 3System Analysis & Risk Management

Analysis part represents the customer requirements e.g. Analysis step includes

stepwise specifications of the program or process to represent requirements, analysisincludes three domains

Information domain

Functional domain

Behavioral domain

3.1 Project Scheduling and Tracking

3.1.1 Project Work Breakdown structure (Analysis)

Fig 3.1: Task Network for Analysis and Design

T1: Communication:

Software development process starts with the communication between customer and

developer. According to need of project, we gathered the requirements related to

project.

T2: Planning:

It includes complete estimation and scheduling (complete timeline chart for project

development shown in 3.3) and tracking (shown in 3.3).

T3: Modeling:

It includes detailed requirement analysis and project design as shown in chapter 2.

14

T1: Communication T2: Planning T3: Modeling

DVL

T11:Projectinitiation

T12:

RequirementGathering

T21:

Estimation

T22:Project

schedulingand tracking

T31: Analysis T32: Design


15/64


3.1.2 Project Work Breakdown structure (Implementation)

Fig 3.2 Task Network for Implementation

T1: Construction

It includes coding and testing steps. Design details are implemented using java

programming language.

T11:-Virtual file Server


keeps track of the client file, the cluster machines. The database is processed from the

server side system. The database maintains all records related to file stored on cluster

nodes. It contain where actually file stored, time of storing file, all related information

of storing space. The Server Program will be installed on one machine and will

process the request being given by the client program. In the design, it also provides

facilities of adding and deleting cluster nodes as per the requirements.

T111:Analysis

T11: Virtualfile server

T12: Clientapplication

T14: Subnet

clustering

T13:Authentication

T1: Construction

T131:

Encryption

T15: Splitting& Merging

T122: Uploadapplications

n

T111: AnalysisT121:Graphicaluser interface T123: Down

loadapplication T141:Cluster

15

DVL

T2: Delivery

T16:IntegratingS/W & H/W

T17:Testing& Debugging


16/64


The analysis of data is maintained in the server in the form of mysql database

and it keep track of all information related to storing data on cluster nodes.

T12: Client Application


with the system. Client software will be installed on every machine from where you

want to access the Virtual Disk functionality.

T121: Graphical user interface

This module is provides user for easy interaction with system. All the file related

operations that user want to do is provided in the form of graphical way. Also on

virtual file server GUI provides easy interaction to the administrator.

T122: Upload applications

When user wants to upload their data, this application transfer data to splitter.

T123: Download applications

When user want to retrieve their data, this application requests cluster nodes using

virtual file server and these spitted blocks are given to the merging application. And

finally data is stored in local directory.

T13: Authentication

Every user of the System should be an Authorized user. The User before starts

working should register itself. The Username and password at the time of registration

will be used to login to the system.

T131: Encryption

Username and Password will be encrypted. We are using Message-Digest algorithm

for encryption and provide better security to the user.

T14: Subnet Clustering

This module will link the entire Cluster and will form another cluster. Means

every member of a cluster is a cluster itself. Every member will cluster the disk of its

own LAN network and will provide a virtual space and acts as a member of the

16


17/64

Disk Virtualization onLinuxcluster. Here the admin can add disk or can remove a disk. He can decide which

Subnet can be used as member. As soon as cluster application started IP address of

that node automatically gets added into mysql database on virtual file server side. As

well if Cluster sends exit message to server that cluster is automatically deleted from

database. For this Socket programming is used.

T141: Cluster

This application includes FTP file server application which is used for storing

and retrieving data.

T15: Splitting and Merging

This module splits the user file into number of blocks according to number of clusternodes and keeps each block on each member of the cluster by splitting algorithm.In

this technique first the file is read in the form of bytes.

Steps for splitting any type of file:-

1. First read the file in terms of bytes.

2. Then read no of cluster nodes connected to central server.

3. Then divide file size by no of nodes which is split size.

4. Then read file up to split size & and write another file with same

name and format only one more extension is given that is 0,1,2

etc..

5. Store each part on separate cluster node by using FTP file server.

Merging of File:-

1. For merging execute one batch file.

2 In batch file write command copy/b.

3. After that read the parts of file to be merged.

4. Next block (part) separated by + operator.

17


18/64

Disk Virtualization onLinux5. Finaly write destination.

T16: Integrating S/W & H/W

In this phase we will collect all the s/w modules & integrate them with h/w devices.

T17: Testing and debugging

Testing is carried out to check whether flow of coding is correct, to check out the

errors of required module.

T2: Delivery

Final delivery of project will be given along with the necessary documentation.

Each task is assigned to one or more team members, where

D1: Gunjal S. N.

D2: Dumbare S. M.

D3: Gaikwad T. W.

3.1.3 Preparing project table

Table 3.1: Project scheduling table (Analysis and Design)

Task Plan start Actualstart

Plancomplete

Actualcomplete

Personassign

T11 1 July 06 3 July 06 15 July 06 31 July 06 D1,D2,D3

T12 1 Aug 06 1Aug 06 30 Aug 06 7 Sept 06 D1,D2,D3

T21 8 Sept 06 9 Sept 06 15 Sept 06 20 Sept 06 D1

T22 8 Sept 06 8 Sept 06 20 Sept 06 21 Sept 06 D2,D3

T31 22 Sept 06 22 Sept 06 30 Sept 06 7 Oct 06 D1,D3

T32 22 Sept 06 22 Sept 06 2 Oct 06 8 Oct 06 D2

Table 3.2: Project scheduling table (Implementation)

Task Plan start Actual

start

Plan

complete

Actual

complete

Person

assigned

T11 20Dec 06 11March07 15 Jan 07 18March 07 D1,D2

18


19/64


T11

1

16 Jan07 11March07 25 Jan07 18 March07 D1,D2

T12 20 Dec 06 29 Jan 07 10 Jan 07 3 March 07 D3,D1

T121

11 Jan 07 29 Jan 07 15 Jan 07 10 Feb 07 D3,D2

T12

2

16 Jan 07 19 Feb 07 25 Jan 07 24 Feb 07 D3

T12

3

16 Jan 07 26 Feb 07 25 Jan 07 3 March 07 D3

T13 26 Jan 07 15 Jan 07 10 Feb 07 26 Jan 07 D1

T13

1

26 Jan 07 15 Jan 07 10 Feb 07 26 Jan 07 D2

T14 26 Jan 07 18March07 15 Feb 07 2 April 07 D1,D2,D3

T14

1

16 Feb07 18March07 21 Feb 07 2 April 07 D1,D2,D3

T15 11 Feb 07 5 March 07 21 Feb 07 11 March07 D1,D2,D3

T16 22 Feb 07 2 April 07 5 April 07 11 March07 D1,D2,D3

T17 22 Feb 07 6 April 07 28 Feb07 13 April07 D1,D2,D3

T2 11 March

07

- 27 April07 D1,D2,D3

3.1.4 Time Line Chart

19


20/64


Fig 3.3 Time line chart (analysis and design)

M1: Requirement gathering and analysis must be achieve.M2: All necessary design must be complete.

M1

M2

20

3/07 17/07 31/07 28/08 11/09 25/09 02/1014/08

T11

T12

T31

T22T21

T32

16/10


21/64


Fig 3.4 Time line chart (Implementation)

M1: The data should be stored efficiently.

M2: The splitting & merging of file should be achieved in case of insufficient space.

M3:All the s/w and h/w modules should be easily integrated.

3.2 Analysis Model

Analysis includes three domains

Behavior modeling

Functional modeling

Architectural modeling

M1

M2

M3

21

20/12 03/01 17/01 18/02 04/03 18/0304/02

T11

T111

T122

T121

T12

T123

T2

T17

T16

T15

T141

T14

T131

T13


22/64

Disk Virtualization onLinuxUML Diagrams:

Design modeling uses a combination of text and diagrammatic forms to depict the

requirements for data, function and behavior in a way that is relatively easy to

understand and more important, straightforward to review for correctness,

completeness and consistency.

A diagram is the graphical presentation of a set of elements most often rendered as a

connected graph of vertices (things) and arcs (relationship). These diagrams are drawn

to visualize a system from different perspectives so a diagram into a system.

3.2.1 Behavioral Modeling

3.2.1 1 Use case Diagram: It shows a set of use cases and actors ( a special kind of

class and their relationships). Use case diagrams address the static use case view of

system. These diagrams are especially important in organizing and modeling thebehavior of a system.

Delete IP address

Create User Account

Login

File Information

Block information

Upload files

Download FilesClient Application

Delete Files

store IP Address

Cluster

Virtual File Server

Fig 3.5 Use Case diagram for overview of DVL

22


23/64


New user

Exiting user

Registration form

Client Aplication

Login

invalid

Authentication Operation

Virtual file Server

invalid

For new user

For exiting

user

Fig 3.6 Use Case diagram for login

3.2.2 Functional Modeling

In the functional modeling it describes the function of the propose system. it consist of

two diagram. They are as follows:- Dataflow diagram

-Control flow diagram

3.2.2.1 Data Flow Diagram

Data flow diagram (DFD) is also called as Bubble chart is a graphical technique,

which is used to represent information flow, and transformers those are applied when

data moves from input to output.

Control flow diagram (CFD) is representing the control flow of system from one state

to another state.

23


24/64


Fig. 3.7: Level 0 DFD

Fig.3.8: Level 1 DFD

3.2.2.2 Control Flow Diagram

The large class of application having following characteristics requires control flow

modeling

The application those are driven by events rather than data

The application those produce control information rather than reports or

display.

The application those process information in specific times.

Cluster nodesclient Virtual

file server

Store file blocksProcess requestInput files& data

Client

Database Server

CheckPassword

ProcessInformation

Server

File Request

Logininfo

Valid

File Data

Switching Server

If Failure

24


25/64


Fig 3.9 Control Flow Diagram for DVL system

3.3 Risk Identification

Risk management involves Risk Identification, Risk Analysis and Risk

Prioritization; while Risk protection involves Risk Management Planning, Risk

Resolution and Risk Monitoring

R1: Illiterate to project management skills

R2: Unfamiliar to the programming language, e.g. Advanced Java

R3: Unfamiliar to concept of project.

R4:.Unfamiliar to software tools.

R5:Given that the technical experience among project members is limited,

there is concern that the size estimate may drift considerably resulting in schedule

overruns.

R6: N/W Hardware cannot integrated easily, requiring redesign and rework.

3. 4 Risk projection

Risk Protection involves Risk Management Planning, Risk Resolution and

Risk Monitoring.

Requestfrom clientfor storingdata

Split file

Store Blockon clusternode(s)

Record onVirtualFile Server

Merge andprovidedata toclient

StreamFileData

25


26/64

Disk Virtualization onLinux3.4.1 Preparing risk table

ST: Staff related size and experience

TE: Technology to be built related

DE: Development Environment

Table 3.3: Risk Table

Risk Probability Type Impact

R1 40% ST Marginal

R2 25% ST Marginal

R3 20% ST Marginal

R4 20% DE Marginal

R5 20% ST Marginal

R6 20% TE Critical

3.4.2 Risk table along with RMMM plan

This Risk Mitigation Monitoring and Management Plan identifies and

documents the risks associated with Project. In addition to project risks and technical

risks, business risks are also identified, analyzed and documented. Both generic and

product-specific risks have been considered. In addition to identification, thisdocument outlines the proactive strategy that we have adopted to avoid these risks. A

contingency plan is also prepared for each risk, in case it becomes a reality. Only those

risks have been treated whose probability and impact are relatively high, i.e. above a

referent level.

Table 3.4:Risk table along with RMMM plan

Risk Mitigation Monitoring Management

R1 Refer the Software Review with Searching over Internet

26


27/64


Engineering books guides

R2 Schedule studying

Languages from

Standard Books

Coding in

programming

language

Searching for the

language specification

for faced problems

R3 1) Schedule studying

concepts from

books

2)Discussion with

Guide

By doing

Analysis

1)Requirement

gathering should be

done

2)Search the web

for the concept

R4 Schedule studying of

specification of

Software tools in

development phase

Setting up the

Software tools

during

implementation

1)Breaking down the

study in members

2)Checking the

alternative for softwaretools

R5 Study the related past

projects and try to

learn as much from

their experience

Project guide

monitors our

project and

provide

guidance

whenever

required.

Interact with internal

and external guide in

case of difficulties and

decision making.

R6 Study N/W related

Books

Setting up

required H/w

for final

implementation.

.

Searching for new H/W

If current is not

working.

27


28/64


Chapter 4

Implementation Details

Chapter 4

Implementation Details

28


29/64


We use a Disk Virtualization technique in which we combine disk space of

Pcs over a LAN and form a virtually continuous memory area called as the Virtual

Disk(LAN drive).This Virtual Disk is actually formed by the combination of some

dedicated disk space from each hard disk of Cluster computer in the LAN. The break

down structure of system is given below and also detailed implementation of each

module is given.

Breakdown Structure (modules)

Fig 4.1: Breakdown Structure (modules)

4.1 Designing of Virtual File Server:

4.1.1 Sequence diagram for Virtual file Server

Virtual fileserver

Clientapplication

Subnetclustering

AuthenticationSplitting &Merging

Analysis

DVSystem

ClusterEncryption

29


30/64


: VFServer : Client :Cluster

Start Server()

Establish conn. toClient

Send info about File Blk to Client

establish Connectionto mysql server

Store Account info()

Store info.about Splitted file Block

StoreIPof Cluster node()

Exit( )

StopServer( )

Delete IPClusternode()

Store info about uploaded files

Check user login( )

Fig 4.2:Sequece diagram for Virtual File Server

4.1.2 Implementation Details for Designing of Virtual File Server

The Virtual File Server programs links all the clients connected to the System.

This module keeps track of the client file, cluster machines, user account, file blocks

information. The Server Program will be installed on one machine and will process the

request being given by the client program.

Implementation Detail:

1. Configuring Mysql Server in Linux.

Steps a) Execute at root /etc/init.d/ mysql password.

b) Execute at root /etc/init.d/mysql start.

c) Enter the Password of Mysql.

2. Create Database in Mysql Server.

30


31/64


3. Create Socket and writing Data to the socket.

Writing to the Socket

ServerSocket ss = new ServerSocket(2000);

Socket connection = ss.accept();

BufferedOutputStream bos = new BufferedOutputStream(connection.

getOutputStream());

/** Instantiate an OutputStreamWriter object with the optional

character * encoding. */

OutputStreamWriter osw = new OutputStreamWriter(bos, "US-ASCII");

We could use the BufferedOutputStream.write() method to write bytes across the

socket. We prefer to use OutputStreamWriter objects to write on because we usually

dealing in multiple platforms and like to control the character encoding. Also, with

OutputStreamWriter you can pass objects such as Strings without converting to byte,

byte arrays, or int values. We create an OutputStreamWriter osw by instantiating it

with ourBufferedOutputStreambos and optionally the character encoding US-ASCII.

TimeStamp = new java.util.Date().toString();

String process ="Calling the Socket Server on "+ host + "

port " + port +

" at " + TimeStamp + (char) 13;

/** Write across the socket connection and flush the

buffer */

osw.write(process);

osw.flush();

As shown above, were creating two Strings TimeStamp and process to be

written to the server. We call the osw.write () method from our OutputStreamWriter

object, passing the String process to it. Please note that we placed a char (13) at the

end of process...well use this to let the server know were at the end of the data were

sending. The last item we need to take care of is flushing the buffer. If we dont do

this, then we cant guarantee that the data will be written across the socket in a timely

manner.

31


32/64


Reading from the Socket

/** Instantiate a BufferedInputStream object for reading

/** Instantiate a BufferedInputStream object for reading

* incoming socket streams.

*/

BufferedInputStream bis = new

BufferedInputStream(connection.

getInputStream());

/**Instantiate an InputStreamReader with the optional

* character encoding.

*/

nputStreamReader isr = new InputStreamReader(bis, "US-

ASCII");

/**Read the socket's InputStream and append to a

StringBuffer */

int c;

while ( (c = isr.read()) != 13)

instr.append( (char) c);

/** Close the socket connection. */

connection.close();

System.out.println(instr);

}

catch (IOException f) {

System.out.println("IOException: " + f);

}

catch (Exception g) {

System.out.println("Exception: " + g);

}

}

}

We start by instantiating aBufferInputStream object bis, calling thegetInputStream()

method of our Socket object connection. We instantiate anInputStreamReaderobject

isr, passing ourBufferedInputStream object bis.

32


33/64

Disk Virtualization onLinuxSingleSocketServer: A Server That Process One Socket at a Time

package FileServer;

import java.net.*;

import java.io.*;

import java.util.*;

public class SingleSocketServer {

static ServerSocket socket1;

protected final static int port = 19999;

static Socket connection;

static boolean first;

static StringBuffer process;

static String TimeStamp;

We start by importing the same packages we did with our SocketClient class. We set

up a few variables. Of note, this time were setting up a ServerSocket object called

socket1. As its name implies, we use the ServerSocket class to set up a new server. As

well see later, ServerSockets can be created to listen on a particular port and accept

and deal with incoming sockets. Depending on the type of server we build, we can

also process InputStreams and OutputStreams.

public static void main(String[] args) {

try{

socket1 = new ServerSocket(port);

System.out.println("SingleSocketServer Initialized");

int character;

In the main() method, we start with a trycatch block. Next we instantiate a new

ServerSocket object socket1 using the port value of 19999the same port that we

were looking to connect to with our SocketClient class.

33


34/64


while (true) {

connection = socket1.accept();

BufferedInputStream is = new

BufferedInputStream(connection.getInputStream());

InputStreamReader isr = new InputStreamReader(is);

process = new StringBuffer();

while((character = isr.read()) != 13) {

process.append((char)character);

}

System.out.println(process);

//need to wait 10 seconds for the app to update database

try {

Thread.sleep(10000);

}

catch (Exception e){}


String returnCode ="SingleSocketServer repsonded at "+

TimeStamp + (char) 13;

BufferedOutputStream os = new

BufferedOutputStream(connection.getOutputStream());

OutputStreamWriter osw = new OutputStreamWriter(os, "US-

ASCII");

osw.write(returnCode);

osw.flush();

}

}

catch (IOException e) {}

try {

connection.close();

}

catch (IOException e) {}

}

}

Within this block of code, we start with the Server Socket socket1s accept () method.

The accept() method basically stops the flow of the program and waits for an incoming

34


35/64

Disk Virtualization onLinuxsocket connection. When a client connects, our Socket object connection is instantiated.

Once this is done, our program continues through the code MultipleSocketServer: A

Server That Handles Multiple Client Connections

package bdn;

import java.net.*;

import java.io.*;

import java.util.*;

public class MultipleSocketServer implements Runnable {

private Socket connection;

private String TimeStamp;

private int ID;

public static void main(String[] args) {

int port = 19999;

int count = 0;

try{

ServerSocket socket1 = new ServerSocket(port);

System.out.println("MultipleSocketServer Initialized");

while (true) {Socket connection = socket1.accept();

Runnable runnable = new MultipleSocketServer(connection,

++count);

Thread thread = new Thread(runnable);

thread.start();

}

}

catch (Exception e) {}

}

MultipleSocketServer(Socket s, int i) {

this.connection = s;

this.ID = i;

}

We instantiate a new Thread threadby runnable to the Thread class. We call the start

() method of thread and were ready go. Invoking the start () method spawns a new

35


36/64

Disk Virtualization onLinuxthread and invokes the objects run() method. The actual work within the thread

happens within the run () method.

The run() Method

public void run() {

try {

BufferedInputStream is = new

BufferedInputStream(connection.getInputStream());

InputStreamReader isr = new InputStreamReader(is);

int character;

StringBuffer process = new StringBuffer();

while((character = isr.read()) != 13) {

process.append((char)character);}

System.out.println(process);

//need to wait 10 seconds to pretend that we're processing

something

try {

Thread.sleep(10000);

}

catch (Exception e){}


String returnCode ="MultipleSocketServer responded at "+

Timestamp + (char) 13;

BufferedOutputStream os = new

BufferedOutputStream(connection.getOutputStream());

OutputStreamWriter osw = new OutputStreamWriter(os, "US-

ASCII");

osw.write(returnCode);

osw.flush();

}

catch (Exception e) {

System.out.println(e);

}

finally {

try {

connection.close();

36


37/64


}

catch (IOException e){}

}

}

}

The run() methods responsibility is to run the block of code that we want

threaded. In our example the run() method executes the same block of code we built in

the SingleSocketServer class. Once again, we read the BufferedInputStream, pretend

to process the information, and write a BufferedOutputStream. At the end of the run ()

method, the thread dies.

4. Making Database connectivity between Java and Mysql in Linux:

SETPATH= /path to mysql-connector-java-[version]-bin.jar:$CLASSPATH

4.2 Client Application

4.2.1 Sequence diagram for Client Application

37


38/64


: User : Client. : VF Server. : Cluster

Check Connection( )

Create Account( )

send Account info( )

login( )

Valid( )

invalid( )

Upload Files( )

send File info( )

send block info( )

request for Download files( )

download files( )

Delete Files( )

update Database( )

logout( )

: User : Client. : VF Server. : Cluster

Check Connection( )

Create Account( )

send Account info( )

login( )

Valid( )

invalid( )

Upload Files( )

send File info( )

send block info( )

request for Download files( )

download files( )

Delete Files( )

update Database( )

logout( )

Request to create userdir

Fig 4.3:Sequece diagram for Client Application

4.2.2 Implementation Details for Client Application

38


39/64



with the system. Only authorized user can interact with the system. Client software

will be installed on every machine from where you want to access the Virtual Disk

functionality.

Implementation Details:-

1. Design of Graphical user interface with the help of Swing.

2. Creation of account.

After creating the user account the information about user gets stored in

database on virtual file server with the help of socket programming. At the time of

creation of account user directory gets created over the each cluster node with the help

of running the cluster application (ftp fileserver) parallel over each cluster node and

executing mkdir command over each cluster node to create user directory.The next

piece of information we need to know is the TCP/IP port that the program is going to

be communicating on. TCP/IP uses ports because it is assumed that servers will be

doing more than one network function at a time and ports provide a way to manage

this. For example: a server may be serving up web pages (port 80), it may have a FTP

(port 21) server, and it may be handling a SMTP mail server (port 25). Ports are

assigned by the server. The client needs to know what port to use for a particular

service. In the TCP/IP world, each computer with a TCP/IP address has access to

65,535 ports. Keep in mind that ports are not physical devices like a serial, parallel, or

USB port. They are an abstraction in the computers memory running under the

TCP/IP transport layer protocol.

3. Login: after creation of account user can interact with system. Many users can login

at a time.

4. Logout: User logging out.

if(server!=null){

server.delete();

server.quit();

System.exit(0);}

5.Upload:

When user wants to upload their data, this application transfer data to splitter.

6.Download:

39


40/64


When user want to retrieve their data, this application requests cluster nodes

using virtual file server and these spitted blocks are given to the merging application.

And finally data is stored in local directory.

7.Delete:

Similarly user can delete their data, this application requests cluster nodes

using virtual file server and these spitted blocks are deleted and database is updated on

central server side.

4.2 Authentication and Encryption

4.3.1 Sequence diagram for Authentication and Encryption

: User :Authentication

: MysqlDatabase

Enter user name and Pwd( )

Encryption( )

Save( )

login( )

Req to Username Adn pwd

accept( )

Decryption( )

Fig 4.4: Sequence diagram for Authentication and Encryption

40


41/64

Disk Virtualization onLinux4.3.2 Implementation Details for Authentication and Encryption

Username and Password will be encrypted. The Username and password at the

time of registration will be used to login to the system. The Encryption and

Decryption of the Username and password is done with the use of MD5 algorithm.

The MD5 is quite fast and produces 128 bit message digests. Detail implementation of

MD5 is given as follow:

Padding:

Add Padding bits to original Message. Aim is to make length of original

message equal to a value which is 64 bit less than an exact multiple of 512.

Append Length:

Calculate the original length of message and add it to end of message .

Fig 4.5: Append Length

Divide Input Into 512 bit blocks:

Initializing Chaining Variables:

These are 32 bit .initial hexadecimal values as bellow:

A Hex 01 23 45 67

B Hex 89 AB CD EF

C Hex FE DC BA 98

D Hex 16 54 32 10

Process Blocks:

41


42/64


o Copy 4 Chaining variables into 4 corresponding variables a,b,c,d.

o Divide current 512 bit blocks into 16 sub blocks.

o There are Four rounds in each round we process all 16 sub blocks

belonging to the block.

Fig 4.6: Conceptual process in round.

There are 16-iteration in each rounds

1. A process p is first performed on b,c,&d.This process p is different in

all four rounds.

2. The variable a is added to a process p.

3. The message sub block m[i] is added to output of step 2.

4. The constant t[k] is added to output of step 3.

5. The output of step2 is circular test shifted by 5 bits.

6. The variable b is added to a output of step 5.

7. The output of step 6 becomes the new ahead of next step

4.3Subnet Clustering:

4.4.1 Sequence diagram for Subnet Clustering

42


43/64


: Client: Cluster : VF Server.

Make Connection( )

Send IP Address( )

request for creating user Dir( )

create UserDir( )

Request to store FileBlk

SaveFileBlk( )

Request to Retrive File Block

sendFileBlk( )

Exit and Delete IP( )

Fig 4.7: Sequence diagram for Subnet Clustering

4.4.2 Implementation Details for Subnet Clustering

This module will link the entire Cluster and will form another cluster. Means

every member of a cluster is a cluster itself. Every member will cluster the disk of its

own LAN network and will provide a virtual space and acts as a member of the

cluster. Here the admin. Can add disk or can remove a disk.

1. As soon as cluster application started IP address of that node automatically gets

added into mysql database on virtual file server side. As well if Cluster sends exit

43


44/64

Disk Virtualization onLinuxmessage to server that cluster is automatically deleted from database. For this Socket

programming is used.

2. we can design the FTPFileserver that perform the function of creating User

directory by executing the MKDIR command on each cluster node and also Perform

different type of command related to the File.

3. When the Client upload the File Over the cluster node the file get stored on

specified directory by providing ServerConfig.cfg as a command line argument.The

detail of the file is

#SRES

#FTP server section ------------------------------

#port to use for ftp connections ( listenport) default = 21

ftp.Port=21

#ftp.user(nn) username;password;restrict to loginpath; login

directory

ftp.user0=Admin;Admin;false;c:\\User

4.4Splitting and Merging:

4.5.1 Sequence diagram for Splitting and Merging

44


45/64


: Splitting and

Merging

: Cluster: User

UploadFiles( )

Split Files( )

Save FileBlock( )

Download files( )

Req. for File Block

Collect file Blocks

Merge FileBlocks( )

Save in local Dir( )

Fig 4.8 Sequence diagram for Splitting and Merging

. 4.5.2 Implementation Details for Splitting and Merging


each member of the cluster by splitting algorithm. This technique is based on reading

file in the form of bytes & then splitting file as per no of nodes. .

Whenever client wants to access particular file then this request passes to central

server. At the time of retrieval of the same file client uses central server to access the

file block and then merge them and saved on local directory.

Splitting any type of File

The technique is based on reading file in the form of bytes & then splitting file

according to number of Cluster nodes.

45


46/64


1. First read the file in terms of bytes.

File nextFile = new File(Filename);

long fileSize = nextFile.length();

2. Then read no of cluster nodes connected to central server.

3. Then divide file size by no of nodes which is split size.

long splitSize = fileSize / NoOfNodes;

4.Check for Equal Distribution

if (fileSize%splitSize > 0) parts++;

5. Then read file up to split size & and write another file with same

name and format only one more extension is given that is 0,1,2

etc..

for (long k = 0; k < parts; k++)

{

String filePartName = nextFile.getName();

filePartName=filePartName+"."+String.valueOf(k);

File outputFile = new

File(destinationDirectory, filePartName);

FileOutputStream fos = new FileOutputStream(outputFile);

DataOutputStream dos = new DataOutputStream(fos);

long bytesWritten = 0;

while ((bytesWritten < splitSize)&&(bytesRead


47/64

Disk Virtualization onLinuxclient want to download the file following sample code get executed to get required

file.

Merging of File:-

1. For merging we have to execute one batch file.

2 In batch file write command copy/b.

String batch="copy /b ";

int blkno=0;

3.Create one String array, for temporary storage.

String[] FileArray = new String[10];

4.After that read the parts of file(block) one by one.

while(st.hasMoreTokens())

{

dstfile=st.nextToken();

host=st.nextToken();

blkno = Integer.parseInt(st.nextToken());

batch = batch + dstfile + " + ";

FileArray[j]=dstfile;

j=j+1;

5.. Connect to the ftp file server. FtpConnect cn = FtpConnect.newConnect("ftp://" + host);

cn.setUserName("Admin");

cn.setPassWord("Admin");

Ftp cl = new Ftp();

// source FtpFile remote file

file = new FtpFile(name.trim()+"\\"+dstfile, cl);

6.If connection to the ftp server fails then trying to get block from

backup server.FtpConnect cn1 = FtpConnect.newConnect("ftp://" +

server.BackUpIp);

7. Finaly write destination.

CoFile to =new LocalFile(System.getProperty("user.dir"), dstfile);

8. Disconnect from server this must be always run

47


48/64


finally

{

cl.disconnect();

}

9.End of While

10. Execute batch file by executing process p

p = r.exec("cp.bat");

11.Delete temporary files.

for(j=j-1;j>=0;j--)

{

try

{

File f = new File(FileArray[j]);

f.delete();

}

catch(Exception e){}

}

.

48


49/64


Chapter 5

Testing and Results

49


50/64


Chapter 5

Testing and Results

Tests are the individual tests specified in a test plan document. Each test is

typically described by

An initial system state.

A set of actions to be performed.

The expected results of the test.

Testing As a Continuous Process:

All testing follows a pre-planned process, which is agreed to. All tests consider

not only a nominal system condition but also address anomalous and recovery aspects

of the system. The system is tested in a stressed environment, nominally in excess of

150 percent of its rated capacities. All test products (test cases, data, tools,

configuration, and criteria) are documented in a software description document. Every

test shall be described in traceable procedures and have pass-fail criteria included.

Implementation:

Test cases are planned in accordance to the test process and documented with

detailed test descriptions. These test cases use cases based on projected operational

mission scenarios. The testing process also includes stress / load testing for stability

purpose (i.e., at 95% CPU use, system stability is still guaranteed). The test process

thoroughly tests the interfaces and modules. Software testing includes a traceable

white box testing, black box testing and other test processes verifying implemented

software against design documentation and requirements specified. Types of Testing:

o White Box Testing:

A level of white box test coverage is specified that is appropriate for the software

being tested. The white box and other testing uses automated tools to instrument the

software to measure test coverage.

Black Box Testing:

50


51/64


A black box test of integration builds includes functional, interface, error

recovery, stress and out-of-bounds input testing. All black box software tests are

traced to control requirements. In addition to static requirements, a black box of a

fully integrated system against scenario sequences of events is designed to model field

operation. Performance testing for systems is integrated as an integral part of the black

box test process.

5.1 Unit Testing

Validations were made for each single module and each module was tested

independently. Maximum error detection was made at this stage. Proper flow of in

bound and out bound information was made.

5.1.1 Virtual File Server


keeps track of the client file, the cluster machines. The database is processed from the

server side system. The database maintains all records related to file stored on cluster

nodes. Following test cases.

Input: Start Server

Expected Output-: Server listening on port no. 6564 and ready for communication

Validation-: Client side socket will communicate with server side socket that is server

accepts incoming string from client socket.

Input -: Stop Server

Expected Output: Communication between client and Server stop and socket

connection will close.

Validation-: Server never responding for client request.

Input-: Cluster Node IP for storing in mysql database.

Expected Output:-Cluster node IP successfully stored in cluster table of mysql

database.

Validation-: Manage cluster node window displays Cluster node IPs successfully.

5.1.2 Client Application

This module is a user interface system through which the users can interact with the

system. Client software will be installed on every machine from where you want to

access the Virtual Disk functionality. Following are test cases.

51


52/64

Disk Virtualization onLinuxInput -: IP address of Virtual file Server.

Expected Output: Client application window will display successfully.

Validation-: Client can do all functionality. E.g. Create account, login etc.

5.1.3 Authentication and Encryption

Input-:user name and Password in plain text.

Expected Output-:Encrypted text.

Validation-: After decryption it will give same plain text as entered for input. e.g. For

input of sachin the encrypted text will be

15285722f9def45c091725aee9c387cb.After decryption it will gives original text

sachin.

5.1.4 Subnet Clustering

Input ServerConfig.cfg and IP address of the VF Server.

Expected output -: Displaying message Cluster Node Started continuously.

Validation-: Input data from port 21 will store successfully on local machine.

Input-Data on port 21.

Expected output-: Successfully sending or storing data.

Validation-: Incoming data from port 21 will successfully stored on specified path

given by ServerConfig.cfg or outgoing data will successfully transferred to remote

location.5.1.5. Splitting and Merging

Splitting


each member of the cluster.

Input -: Any type of file.

Output -: Equal size of file blocks equal to number of cluster nodes .

Validation-:output files equals to number of cluster nodes, size of each file equal to

split size and files are with same name and format only one more

extension is given that is .0,.1,.2etc..

Example 1 -Suppose a file of name Jay. jpg of size 10000 bytes and cluster nodes

equal to 4.Then following output we get.

Jay.jpg.0 having size 2500 bytes.

52


53/64

Disk Virtualization onLinuxJay.jpg.1 having size 2500 bytes.



Example 2 -Suppose a file of name Jay. jpg of size 1000 bytes and cluster nodes equal

to 3.Then following output we get.





Merging

At the time of merging of the same files merging application merge them.

Input -: Splitted Block of File.

Output -: Whole file without loss of any kind of data.

Validation-: output file size will exactly equal to addition of all input splitted file sizes

and it will work properly.

Example 1 Suppose following are splitted input files.





Then output after merging will be Jay. jpg of size 10000 bytes.

5.2 Integration Testing

Integration testing was carried out to check whether each module works properly

with the integrated module. Different inputs were given to the secondary module from

the primary, to check whether it works properly for each input which comes from the

higher level. Both Bottom-up and Top-down integration tests were carried out.

53


54/64


Virtual file Server, Client Application

After selecting create account or login operation , separate windows for both gets

displayed on screen. It will show that the connection between client and server gets

successfully created. Otherwise connection gets failed.

Virtual file Server, Client Application and Authentication

User can create account, login and logout after integrating above modules.

Information regarding new user stored into the database on VFS.

Input-: Creation of user account.

Output-: information regarding user get stored on virtual file Server and username,

password is in Encrypted form

Validation-: user get successfully login.

Input-:User loginOutput-:login information regarding user get displayed on VF server side window.

Validation-:User can successfully interact with system.

Virtual file Server, Client Application , Authentication

and Cluster Application

User can create account, login and logout after integrating above modules.

information regarding new user stored into the database on VFS.Addition to that

directory get created on each cluster node with same name as username on specified

path.

Input-: Creation of user account.

Output-:account creates successfully.

Validation-: Directory get created on each cluster node with same name as username

on specified path in ServerConfig.cfg.and information regarding user stored in

UserInfo table of VFS database.

Virtual file Server, Client Application, Authentication

Splitting and Merging, Cluster Application

Every user interacting with our system should authorized user. If user has already

created his account then he can login directly. Information regarding user maintained

by database on VFS side. User can upload, download or delete his file as per

requirement after login. Many users can communicate with system at same time.

54


55/64

Disk Virtualization onLinux.Input -: upload file.

Output -: File gets splitted in equal size depending upon the number of cluster nodes.

Validation-: File blocks get stored on cluster nodes in Specified directory with same

name as uploaded file with extension 0,1,2.etc depending on sequence of IPs in

database on VFS side.Information about file blocks get stored on virtual file server.

Input -: Download file.

Output-: previously uploaded file.

Validation-: The retrieved file will store on local directory and its contents must

exactly equal to uploaded file

Input -: delete files.

Output -: Files blocks get deleted on every Cluster node

Validation-: Information about File block will delete from VF server database.5.3 Results

Following results are obtained using test cases:

Snapshot For Virtual Fileserver showing login users with managing

cluster manually:

55


56/64


Fig 5.1: Snapshot For Virtual Fileserver showing login users with managing

cluster manually

Snapshot for starting Cluster Application:

Fig 5.2:Snapshot for starting Cluster Application

Snapshot for new user Registration:

56


57/64


Fig 5.3:Snapshot for new user Registration

Snapshot for Uploading operation in Client Application:

Fig 5.4: Snapshot for Uploading operation in Client Application

57


58/64


Chapter 6

Application & Future Scope

Chapter 6

Application & Future Scope

58


59/64


6.1 Applications

LAN Based Systems:-

Disk Virtualization is mainly used in companies where data storage need is larger.

So it mainly used in LAN Based Systems.

Bank Transaction servers:-

We can also use it in banking application where large storage of data needed.

6.2 Future Scope

Crash Recovery:-

When any disk drive fails we can use backup functionality of system for crash

recovery.

Dynamic cluster nodes:-

As per data storage requirement get higher we can add as many cluster nodes as

per need using this system.

Auto Failure Detection:-

Using this system we can detect failure of any of cluster node in network.

6.3 Limitations

All cluster nodes should be in active state while downloading the application.

Currently It is limited for only LAN based systems.

59


60/64


Chapter 7

Conclusion

60


61/64


Chapter 7

Conclusion

We have implemented Disk Virtualization on Linux which is a key enabling

technology that can be leveraged to achieve business benefits. Disk Virtualization

technology enables clustering of Heterogeneous Disk of remote systems to be

appearing on a single physical disk, where each of the disk systems runs as a self-

contained computer.

This project has clustered Heterogeneous nodes for data Storage Management.

And allowing enterprise and research environments that utilize clustered file systems

for data storage to integrate storage management with their overall cluster

management. Using and managing clustered file systems is particularly important in

industries where the volume of data being analyzed is very large. With this new

option, the system provides a complete cluster management solution that now includes

the installation and configuration of a clustered file system within a cluster. This

project provides facility to detect cluster node automatically and also provides backup

facility in case of failure.

61


62/64


References

62


63/64


References

Book References:

1.Herbert Schildt, The Complete reference of Java 2,Tata McGraw Hill, FifthEdition,2002,ISBN 0-07-049543-2.

2. Christopher Negus, Red Hat Linux ver.(10) Bible, WILEY, First

Edition,2004,ISBN 81-265-0478-1.

3. R. S. Pressman, Software Engineering, Tata -McGraw Hill Publications,

Enterprise Edition, 2001.

4.Michael-Jon Ainsley Hore, Main Stream parallel programming, Linux journal,

September 2006 pp.72-74.

Web References:

1IBM storage solutions: Reducing cost and complexity, July 2003.

www.03.ibm.com/servers/storage/software/virtualization/wpapers/ibm_storage_reduci

ng_cost.pdf

2.Frank Bunn, Storage Virtualization-Part III,A SNIA TECHNICAL TUTORIAL,

November 11,2005.

www.exitcertified.com/newsletter/2006Q3/resources/veritas_storage_virtualization_3.

pdf

3.Prentice Hall, Internetworking with TCP/IP Volume III: Client Server

Programming and Applications,1996.

www.pdlit.com/downloads/7042Q3/socket.htm

4.EMC.Enginuity(TM), The Storage Platform Operating Environment(Whitepaper),

august 2002.

http://www.emc.com/pdf/techlib/c1033.pdf

5. E.Eide, Encryption for security, Jan 2000.

www.isstm.com/encrypt/MessageDigest.htm

6.Kevin Yank, Database Driven Website.

http://www.mysql.com

63
http://www.pdlit.com/downloads/7042Q3/socket.htmhttp://www.emc.com/pdf/techlib/c1033.pdfhttp://www.isstm.com/encrypt/MessageDigest.htmhttp://www.mysql.com/http://www.pdlit.com/downloads/7042Q3/socket.htmhttp://www.emc.com/pdf/techlib/c1033.pdfhttp://www.isstm.com/encrypt/MessageDigest.htmhttp://www.mysql.com/


64/64


Documents

DVL Final Report