View
32
Download
1
Category
Tags:
Preview:
DESCRIPTION
Chapter 10 Cloud-Enabling Dust Storm Forecasting. Qunying Huang, Jizhe Xia, Manzhu Yu, Karl Benedict and Myra Bambacus. Learning Objectives. General computing challenges for computing intensive applications How cloud computing can help address those issues - PowerPoint PPT Presentation
Citation preview
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Chapter 10 Cloud-Enabling Dust Storm Forecasting
Qunying Huang, Jizhe Xia, Manzhu Yu, Karl Benedict and Myra Bambacus
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Learning Objectives
1. General computing challenges for computing intensive applications
2. How cloud computing can help address those issues
3. Configure HPC cluster on the cloud4. Deploy dust storm model onto the cloud5. Run the dust storm on the cloud6. Performance and cost analysis
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Learning Materials
•Videos: o Chapter_10_Video.mp4
•Scripts, Files and others:o mirror.tar.gz
3
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Learning Modules1. Dust storm modeling and challenges
2. Dust storm model cloud deployment• General steps• Special considerations
3. Use case: Arizona Phoenix 2011 July 05• Nested modeling
• Cloud performance and efficiency analysis
4. Conclusion and discussions
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Dust Storm Hazards
Desertification
Illness & Diseases
Traffic & Car accidences
Air Pollution
Ecological System
Global/regional Climate
Phoenix Dust Storm a "100-Year Event“, 2011, July 5th
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Dust storm models
• Eta-4bin (Kallos et al., 1997; Nickovic et al., 1997)• Low resolution (30 KM)• Large area
• Eta-8bin (Nickovic et al., 1997; Nickovic et al., 2011)• 8 categories of dust particles• Low resolution (30 KM)• Large area
• NMM-dust (Janjic et al., 2001; Janjic, 2003)• High resolution (3 KM)• Small area
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Dust storm modeling • Dust storm modeling (Purohit et al., 1999)
• Divide the domain into three-dimensional grid cells• Solve a series of numerical equations on each cell• Numerical calculations are repeated on each cell
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Dust Storm Forecasting Challenges
Computing Intensity
Big Data
Large Geographic Scope
Time Critical Task
Dust Storm Forecasting
O(n^4)
Input/output
Finish One-day forecasting in 2 hours
Southwest U.S.
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Learning Modules
1. Dust storm modeling and challenges2. Dust storm model cloud deployment
• General steps• Special considerations
3. Use case: Arizona Phoenix 2011 July 05• Nested modeling• Cloud performance and efficiency analysis
4. Conclusion and discussions
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Dust Storm Model Deployment onto the Cloud
2. Launch one cluster instance as the head node
3. SSH to the instance
1. Authorize network access
10. Export the NFS directory to the computing node
9. Deploy the model on the NFS exporting directory
8. Mount the volume to the NFS exporting directory
4. Install the software dependency andmiddleware, e.g., NFS, and MPICH2
5. Create a new AMI from the head nodeand start an instance from the new AMI
11. Configure and test the model
12. Create two new AMIs from the running instances
6. Configure the middleware on both nodes to enable communication 7. Create an EBS volume
Video: Chapter_10_Video.mp4
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Step 1. Authorize network Access
Dust Storm Model Deployment onto the Cloud
• Configure firewall rule for the firewall group “hpc” • Open port 22 for SSH• Open port 9000 -10000 for MPICH2
Video: Chapter_10_Video.mp40:00 – 1:35
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Step 2. Lunch one cluster instance as the head node
Dust Storm Model Deployment onto the Cloud
• Use Amazon cluster compute AMI• Create a ssh key pair “hpc”, and save the public key pair file to local
storage as “hpc.pem”• Use the security group “hpc”
Step 3. SSH to the instance
• Use the public ssh key file “hpc.pem”• Change the user permission for “hpc.pem” as 600 (user root read permission
only) Video: Chapter_10_Video.mp4
1:35 – 4:10
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Step 4. Install software dependencies
[root@domU-head~] yum install gcc gcc-c++ autoconf automake[root@domU-head~] yum –y install nfs-utilsnfs-utils-lib sytem-config-nfs #install NFS[root@domU-head~] wget http://www.mcs.anl.gov/research/projects/mpich2staging/goodell/downloads/tarballs/1.5/mpich2-1.5.tar.gz # download MPICH2 package[root@domU-head~] tar –zvxf mpich2-1.5.tar.gz # Unzip[root@domU-head~] mkdir /home/clouduser/mpich2-install # create an installation directory[root@domU-head~] cd mpich2-1.5 [root@domU-head~] ./configure -prefix=/home/clouduser/mpich2-install --enable-g=all --enable-fc --enable-shared --enable-sharedlibs=gcc --enable-debuginfo [root@domU-head~] make # Build MPICH2[root@domU-head~] make install # Install MPICH2
Install NFS and MPICH2:
Dust Storm Model Deployment onto the Cloud
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Step 5. Create a new AMI from the head node
Dust Storm Model Deployment onto the Cloud
• Create a computing node from the new AMI
[root@domU-head ~] mkidr /headMnt # create a NFS export directory[root@domU-head ~] echo "/headMnt *rw " >> /etc/exports [root@domU-head ~] exportfs -ra[root@domU-head ~] service nfs start #start up NFS
Configure and start NFS:
Step 4. Install software dependencies
Video: Chapter_10_Video.mp44:10 – 13:02
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Step 6. Configure the head node and computing node Keyless access from the head node to the computing nodes
[root@domU-head ~] vi /etc/hosts #access to the hosts list of the head node[root@domU-head ~] ssh-keygen -t rsa #create a public key at the head node
[root@domU-computing~] mkdir –p /root/.ssh/[root@domU-computing ~] scp root@domU-head: /root/.ssh/id_ras.pub /root/.ssh/ #copy the public key from the head node to the computing node[root@domU-computing ~] cat /root/.ssh/id_ras.pub >> /root/.ssh/authorized_keys
Dust Storm Model Deployment onto the Cloud
Video: Chapter_10_Video.mp413:02- 17:55
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Step 7. Create an EBS volume
Dust Storm Model Deployment onto the Cloud
• Attach to the head node
Step 8. Mount the volume to the NFS exporting directory
• Make a file system for the EBS volume• Mount the volume to head node directory /headMnt
Video: Chapter_10_Video.mp413:02 – 20:40
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Step 9. Deploy the model
Dust Storm Model Deployment onto the Cloud
• Download the model under NFS directory (/headMnt)• Export the NFS directory (/headMnt) to the computing node
[root@domU-computing ~] mkdir /computingMnt # create the directory [root@domU-computing ~] mount -t nfs -o rw domU-head:/headMnt /headMnt #Mount the volume to the NFS export directory
Step 10. Export the NFS directory to the computing node
Video: Chapter_10_Video.mp420:40 – 24:19
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Step 11. Configure and Test the model
Dust Storm Model Deployment onto the Cloud
• cd /headMnt/mirror/performancetest/scripts• ./run_test.sh ec2 >& ec2.log &
Step 12. Create two new AMIs from the running instances
Video: Chapter_10_Video.mp424:19 – 28:18
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Learning Modules1. Dust storm modeling and challenges
2. Dust storm model cloud deployment
• General steps
• Special considerations
3. Use case: Arizona Phoenix 2011 July 05• Nested modeling
• Cloud performance and efficiency analysis
4. Conclusion and discussions
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Special considerations
1. Configuring a virtual cluster environment
• Create placement group
2. Loosely coupled nested modeling and cloud
computing
3. Auto-scaling
• write scripts with the EC2 APIs
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Create placement group
12
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Learning Modules1. Dust storm modeling and challenges
2. Dust storm model cloud deployment
• General steps
• Special considerations
3. Use case: Arizona Phoenix 2011 July 05• Nested modeling
• Cloud performance and efficiency analysis
4. Conclusion and discussions
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Nested model
Tightly Nesting
AOI: NMM-dust(3KM)
AOI: NMM-dust(3KM)
ETA-8bin (30KM)
Loosely Nesting
Nested Subdomain #2(3KM)
Nested Subdomain #3(3KM)
Domain #1(30KM)
A model run with multiple resolutionsModifications of models (Michalakes et al.,
1998)Knowledge of placement for high-resolution
nested subdomains
ETA-8bin identifies AOIs(Area of Interesting)<Low resolution (30 KM) , Large area>NMM-dust performs forecasting over AOIs<High resolution (3 KM ), Small area>
Nested model: provide high resolution results for one or several area of interests over a large area.
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Loosely Nested Model
Low-resolution model domain area and sub-regions (Area Of Interests, AOIs) identified for high-resolution model execution
Low-resolution model Results
Figure a. 18 AOIs Distribution
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Learning Modules
1. Dust storm modeling and challenges
2. Dust storm model cloud deployment
• General steps
• Special considerations
3. Use case: Arizona Phoenix 2011 July 05• Nested modeling
• Cloud performance and efficiency analysis
4. Conclusion and discussions
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Use 2 hours for one-day forecasting over Southwest of U.S. (37 X 20 degree)
18 Subregions run on 18 Amazon EC2 virtual cluster
Run Under Cloud >> Performance analysis
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Run Under Cloud >> Cost analysis cont
The yearly cost of a local cluster is around 12.7 times higher than that of the EC2 cloud service if 28 EC2 instances (with 400 CPU cores) are leveraged to handle the high resolution and concurrent computing requirements for a duration of 48 hours.
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Run Under Cloud >> Cost analysisItems Local cluster Amazon EC2 Options for cloud environment
Procure cluster ~4 weeks None N/A
Configure cluster operating
system (OS)~1 weeks
None
Use a public AMI with OS installed
~1 week Harden image from scratch
Configure dust storm model ~1 days~2 hours
Use a public AMI with most
required software dependencies
installed
~1 days Harden image from scratch
Start cluster 120s 45s
N/AStop cluster 60s 57s
Resume cluster N/A 45s
Total time needed for the first
time deployment~ 5 weeks
~2 Hours Use a public AMI
~ 1 week Harden image from scratch
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Large-scale forecasting is Computable (2 hours) Loosely coupled nested model Cloud computing
Being capable of provisioning a large amount of
computing power in a few minutes Economically sustaining low access rates and low
resolution models
Conclusion
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
Discussion questions1. What are computing challenges for dust storm forecasting?2. What are the general steps to deploy dust storm model on the cloud?3. Which instance type is better for dust storm forecasting, regular instance or HPC
instance? Why?4. How to configure a virtual high performance computing (HPC) cluster to support
computing-intensive applications?5. How Elastic Block Storage (ebs) service is used in supporting the dust storm model
deployment to the cloud?6. How to create a placement group for HPC instances using both Amazon web console
management and command line tools? Why we need this step?7. Compared to Chapter 5 deployment for general applications onto the cloud, what are
the special considerations for dust storm model?8. Why can cloud computing achieve cost-efficiency?9. Why cloud computing provides a good solution to support disruptive event (e.g., dust
storm) simulation?10. What are the remaining issues while using cloud computing to support dust storm
simulation?
Q. Huang, J. Xia, M. Yu, K. Benedict, M. Bambacus, 2013. Chapter 10 Cloud-enabling dust storm forecasting, In Spatial Cloud Computing: a practical approach, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 163-178.
1. Huang Q., Yang C., Benedict K., Chen, S., Rezgui A., Xie J., 2013. Enabling Dust storm Forecasting Using Cloud Computing, International Journal of Digital Earth. DOI:10.1080/17538947.2012.749949.
2. Huang Q., Yang C., Benedict K., Rezgui A., Xie J., Xia J., Chen, S., 2012. Using Adaptively Coupled Models and High-performance Computing for Enabling the Computability of Dust Storm Forecasting, International Journal of Geographic Information Science. DOI:10.1080/13658816.2012.715650.
3. Xie J., Yang C., Zhou B., Huang Q., 2010. High Performance Computing for the Simulation of Dust Storms. Computers, Environment, and Urban Systems, 34(4): 278-290
Reference
Recommended