Distributed Video Rendering using Blender, VirtualBox,and BOINC. Christopher J. Reynolds.Centre for Parallel Computing,University of Westminster.

Material:

• The general concept and motivations.

• Choosing a system virtualization technology.

• Interactions with the hypervisor.

• Final comments.

Background: University of Westminster

• Approx 1600 node Windows & BOINC based DG.

• Runs applications from Enabling Desktop Grids for E-Science (EDGeS).

• 20 node GPU cluster under construction to run further EDGeS/DEGISCO applications.

• For most of 2009 we ran a public video/animation rendering service on a dedicated cluster (~256 Nodes).

Blender: http://www.blender.org

• Rastorization based rendering of frames; a Bag of Tasks (BoT).

• Fits the DG model so move from cluster to DG.

• Uses Blender, a well known animation/image rendering suite.

• Q: What are Blender’s requirements? • A: Python.

• Q: What other considerations are required for a public service?• A: Python scripts can be embedded within Blender input files (Security)

• Q: How to address these issues?• A: Virtualization

Existing Virtualization Solutions

• Existing solutions use either process virtualization or system virtualization.

• Process virtualization (using Entropia VM) proposed by Calder et al [1] is tied to Windows as the host OS.

• VM Wrapper: Generic but more complex and incomplete (Windows Host Support).

• Marosi et al described proof of concept using QEMU based on features [2] .

• Q: QEMU/KQEMU now obsolete on Windows, so what next? • A: Look at latest feature set and choose something else for a

Lightweight solution.

QEMU/KQEMU vs VirtualBox

Feature VirtualBox QEMU KQEMU

X86 guest only. Yes No Yes

Requires admin privileges to run.

No – Add user to allow execution of

VBox DCOM service.

No No

Requires admin to install.

Yes No Yes

CPU performance overhead (x86) [3]

20%(SP)10%(FP)

Huge (emulation) 100%(SP)30%(FP)

Supports disk overlay/cow images.

Yes Yes Yes

Guest to Host File I/O (Without Network)

Yes (Guest Additions)

No No

Monitor/Hypervisor interface.

COM Sockets Sockets

Windows support Yes Yes (Partial) Obsolete

Solution: A Lightweight VM Controller.

Host/DG Node

DG (BOINC) Server

BOINC Core Client

VM (Guest)

Disk image (guest OS), VM config file, inputs.

Instantiate VMand transfer

inputs

React to messages: suspend, resume,

abort

Transfer outputs

VM Controller (standard BOINC job)

Start

1

2

3

45

6

VBox Hypervisor

Transfer outputs

VirtualBox Hypervisor/Monitor ControlVM Controller (C++ Boinc application)

• Win32 CreateProcess() or fork() to VboxManage cmdln tool.

• COM preferable.

(COM)

VM Controller

• Two Threads.• Worker launches unzipper (lzma), prepares VM config files, creates I/O

shared directories, polls for Job completion.• Monitor thread deals with messages and VM control.

VM Controller (C++ Boinc application)

Worker Thread

Monitor Thread

VM Worker Script: Blender

echo 15 > /proc/$$/oom_adjecho "Setting oom_adj to 15 for PID=$$"

case "$1" in start)

mkdir /mnt/sharemount -t vboxsf Shared /mnt/share

mkdir /mnt/share/Outputs

swapon /dev/sdaswapon /dev/sdb

START_FRAME=`cat /mnt/share/input.txt | head -1 `END_FRAME=`cat /mnt/share/input.txt | head -2 |tail -1 `

sudo –u blender_user sh -c "blender -b /mnt/share/input.blend -o /mnt/share/Outputs/Frame-# -F PNG -x 0 -s $START_FRAME -e $END_FRAME –a”

echo $? > /mnt/share/Outputs/blender_return.log

echo "Finished Rendering Application";;

Further Comments

• We adhere to the “one VM per Workunit” model.

• Our BOINC user account has restricted permissions.

• This means VirtualBox.xml and MachineDefinition.xml are loaded from C:\Windows\system32\config\systemprofile\.VirtualBox

• Single xml file location restricts us to one WU per node.

• Proof of concept, though proposed roll out soon (late 2010).

References

[1] Calder, B., Chien, A., & Yang, J. W. D. (2005). The entropia virtual machine for desktop grids. In VEE ’05: Proceedings of the 1st ACM/USENIX international conference on Virtual execution environments (pp. 186–196).

[2] Marosi, A., Kacsuk, P., Fedak, G., & Lodygensky, O. (2010). Sandboxing for Desktop Grids using Virtualization. In The 18th Euromicro International Conference on Parallel, Distributed and Network-Based Computing.

[3] P. Domingues and F. Araujo and L. Silva. (2009) Evaluating the performance and intrusiveness of virtual machines for desktop grid computing. In IPDPS '09:Proceedings of the 2009 IEEE International Symposium on Parallel & Distributed Processing.

Questions?