Embed Size (px)
Citation preview
TORQUEKerry Chang
CCLS
December 13, 2010
OUTLINE
Torque How does it work? Architecture
MADA Demo Results Problems Future Improvements
TORQUE – WHAT IS IT?
Open source project by Cluster Resources Inc.
Cluster resource manager Manages batch jobs
A series of programs to be executed without manual intervention
Manages distributed compute nodes Distributed servers on which to execute batch
jobs
TORQUE ARCHITECTURE
TORQUE SCHEDULER
Currently using standard built-in schedule (FIFO)
MOAB – more advanced scheduler
WHAT HAVE I DONE?
Used MADA as an application of TORQUE Treated the application as a blackbox Text parallelization on input
Created a series of scripts for text manipulation and job submission to Torque queue
Linear improvement in processing time by using Torque
MADA
System for Morphological Analysis and Disambiguation for Arabic
Input file is separated line by line
MADA ARCHITECTURE
HOW DO THE SCRIPTS WORK?
1) First split the text file evenly across the number of specified jobs to be
submitted 2) Create a script for each newly split text file
e.g. If you wanted to run 5 jobs, split the text into 5 files and create a script to run each of the 5 files.
3) Submit each script to Torque 4) Concatenate the output of each script
DEMO
Demonstration of Torque and MADA 3 Output Files
file.bw file.bw.mada file.bw.mada.tok
RESULTS
30 lines
0 2 4 6 8 10 12 14 16 18 200
50
100
150
200
250
300
350
400
With TorqueWithout Torque
RESULTS
300 Lines
0 1 2 3 4 5 6 7 8 9 100
200
400
600
800
1000
1200
1400
1600
With TorqueWithout Torque
RESULTS
3,000 Lines
0 5 10 15 20 25 30 35 400
200
400
600
800
1000
1200
1400
1600
1800
With TorqueWithout Torque
RESULTS
30,000 Lines
0 1 2 3 4 5 6 7 80
2000
4000
6000
8000
10000
12000
14000
16000
With TorqueWithout Torque
RESULTS
Network – Local Temp comparison (seconds)
Network Local Temp Improvement
114 108 6
201 193 8
1,477 1,400 77
14,544 13,973 571
145,105 131,649 13,456
PROBLEMS
How do we know when MADA has finished and we can concatenate the results?
Where do we run MADA and have the results output to?
Submission to compute node hangs Use smarter scheduler Supply machines dedicated to running Torque
jobs
FUTURE IMPROVEMENTS
Pipeline many jobs to Torque Work from local temp folders instead of on
the network Split and rebuild certain output files by
looking at provided testing.madaconfig file MADA TOKAN Preprocessor
QUESTIONS
