Breeze User Manual 2.11.1

Contents1. Introduction:

How can Breeze help me to solve deployment and tuning problems?

2. Getting started: Tracing applications with Breeze:An introduction to tracing and profiling.

3. Tracing applications and scripted flows:Using Breeze to find application arguments, environment and dependencies.

4. Accessing timing information:How to understand your application over time and see complex events chronologically.

5. Tracing larger applications:Understanding complex applications with thousands of processes and files.

6. Tracing applications remotely on a cluster or with customers:How to trace applications on the command line and load the data into Breeze so that you can trace jobs on a cluster or trace remote customer jobs.

7. Comparing two applications:How to trace two applications run on different machines or with different user accounts and compare them in Breeze.

8. Profiling file I/O:How to profile file I/O and record file system and network access patterns.

9. Getting application I/O dependencies:How to find out what files (local or remote) your application used and how they were used.

10. Breeze Automation Platform and exporting data:How to use the Breeze Automation Platform to automatically check and analyse many applications to find performance issues before they matter.

-1/27-

1. IntroductionBreeze is aimed at IT managers and people who deploy complex applications. It gives youthe information you need to solve software build and installation issues quickly.

Even when the software and IT infrastructure have been well designed, installation andconfiguration problems often arise, particularly on distributed systems. Files stored in thewrong place can affect the performance of your application and everything else on thecluster. Breeze profiles your applications so you can be sure that they are using yourcompute resources as they should.

Breeze works by tracing and profiling application dependencies. It is sometimes referred toas a 'system debugger', to be used when debugging scripted flows or third-partyapplications. From build and installation to customer services and tuning, Breeze forms acommon platform to enable IT managers and product engineers to work together andsolve problems more quickly.

When an application works on one system but not on another, Breeze can record what theapplications are actually doing and highlight the differences. Or you can use Breeze to seein detail what your customer has set up and quickly find out what is wrong.

In this guide you will learn how to analyse complex build processes and large applications,and you will learn how to optimise your program for your compute resources so that youcan be sure to get the most from your system.

Tracing and profilingBreeze traces application arguments, environment and dependencies. This means you cantroubleshoot build or installation issues and resolve missing files or libraries.

Breeze also records I/O patterns so that you can understand how your programs are usingthe network and file system. This means you can resolve performance problems andassess the ability of your application to scale in parallel environments.

Using Breeze remotelyBreeze can be used interactively, but you can also trace and profile programs remotely.This is perfect for tracing applications on a job scheduling computer cluster, or usingBreeze for customer services, because you can log the data and analyse it later on yourown machine.

The Breeze Automation PlatformThe Breeze Automation Platform gives you complete access to the trace and profile dataso you can write your own custom queries and automate quality checks in your system.

-2/27-

2. Getting started: Tracing applications with BreezeBreeze is designed to be up and running as quickly as possible. You don't need to changeanything about your programs or your system, just follow these simple steps to accessyour program trace in minutes.

InstallationDownload the latest version of Breeze from our website and extract it somewheresensible. You should make sure that you choose the right version of Breeze (32 or 64bit)for the machine you want to run it on.

Starting BreezeStart Breeze with the breeze.sh script available at the top level of the installation.Although Breeze has a Java GUI, it comes with its own version of Java so you don't need tohave Java installed to run it. You do need to have write permissions in the current workingdirectory from which you launch Breeze and you do need to have enough RAM to run aJava Eclipse application. Usually 1GB of RAM is sufficient.

Breeze has a GUI with a built in command line. To trace and profile an application you justtype 'trace' followed by your command and arguments.

The screen shot below shows a trace of the program 'ls'. The program runs as usual in thecommand line and we can see a small graph with a single process. When you first openBreeze, try tracing something small so that you get used to the application before you tryit on a larger program.

-3/27-

3. Tracing applications and scripted flowsThis chapter will show you how to analyse a program, find its arguments and environment,and list file and library dependencies. In the screen shot below I have traced firefox via acall to bash. We show the root node in green, scripts in blue and programs in orange. Youcan see that bash calls firefox, which is an alias for a script. This script calls 'which' andthe 'firefox' binary itself.

If you click on a program in the graph you can see what it did in the Node View. You cansee what network connections it has made as well as which files it has opened. Breezeeven tells you which files it failed to open. This is particularly useful when debugging 'filenot found' errors.

By double clicking on a program you can see its arguments, environment and libraries.This will show you which environment variables have changed since the start of the traceand which libraries have been loaded. If there are any missing libraries preventing yourprogram from starting you will see them listed here.

-4/27-

Customer case study: Resolving environment issuesOne of our customers was running a complex build and they got a GCC 'library notfound' error message. They traced it with Breeze and compared the failed open callswith the program environment settings. They found that an environment variable hadbeen misconfigured and so GCC was looking in the wrong place.

Using the graph in Breeze they found the script that had set the incorrect variable andfixed it within minutes.

How tracing worksBreeze works by wrapping up your application in a special library designed to replace theinterface between your application and the Linux kernel. This allows us to intercept librarycalls without modifying your system or your application.

This powerful tracing technology allows us to build custom tracing and profiling tools.Come and talk to us if you want to extend the functionality of Breeze or you want to see adifferent type of data.

Customer case study: Tracing scriptsOne of our customers was running a complex simulation controlled by an in-housescripted flow. They were constantly getting an error that didn't make sense given thesettings they were using. After tracing it with Breeze, they found that their scripts werepicking up the wrong simulation tool. They had five versions installed and one of theconfiguration variables was incorrect. They fixed the script and were back up andrunning in minutes.

-5/27-

LinuxKernel

GLibC ApplicationSystemcalls

Librarycalls

BreezeLD_PRELOAD

library

4. Accessing timing informationSometimes it is important to see timing information in order to debug a performanceissue, to see the order of a number of operations, or to see what happened just before acrash.

Timeline View

The Timeline View shows you a tree of processes with timing bars. This is the sameinformation as was shown in the graph in the previous chapter, but represented differently.You can move between the graph and Timeline View easily by right clicking on a programand choosing a 'Select in..' option from the pop-up menu.

The mouse wheel will change the scale of the timeline to some extent, but to scale to aparticular program, right click on the program and choose 'Select and centre in TimelineView'. The Timeline View will zoom so that bar fills the window.

-6/27-

Event View

To see the calls made by a program in chronological order, click on that process in eitherthe Graph or Timeline View and select the Event View on the right. There is a detailed listof what the program did, and below that a summary of the same information on atimeline. You can search for any term in the list to skip to a particular event, action ortime. Events in the list are initially sorted by time, but you can sort them differently byclicking on the column headers.

There will be gaps in the Event View timeline if the trace you are investigating was notgenerated with the --trace=all-io option, because Breeze records only the first readfrom (or write to) each file. If you used the --trace=all-io option, and you find that thereare too many events to display, you can use the check-box to filter some events (reads,writes and seeks). A gap in the Event View timeline next to a connect call may indicatethat your program is hanging on a particular network operation.

Duration View

If you want to find information about which of your programs are taking the most time,you can use the Duration View. This view sorts program by duration (by default, but youcan change this by clicking the columns headers), showing the longest first.

-7/27-

You'll find a right-click menu for each table item similar to that in the Timeline View, handywhen you have identified a suspicious program and want more details about it.

It also possible to filter the elements of the table by specifying a time in the filter text box. It can be a date format like in the figure or a unix time. Activating the filter will show on the table only the processes that were active at the time specified.

Customer case study: Troubleshooting performance issuesA customer had problems one day when all their applications slowed down. Their Indianoffice was down for the day, but their applications shouldn't have been connecting tothat office, so they didn't know what was wrong.

They traced their applications with Breeze and found that, after connecting to locallicense servers, their simulation software was still trying to connect to the Indian office.It hung for a few seconds before timing out, causing the slow down across their wholesystem.

-8/27-

5. Tracing larger applicationsOnce you get used to using Breeze you will want to trace your applications for real. Thiswill probably give you a lot more data than we have been looking at so far.

Removing commonly used programs from the graphWhen Breeze is given a trace which is too big to show in the graph view, it will try toremove commonly used programs. It will open the Program Summary List (from the Viewsmenu) and automatically deselect programs, such as 'grep' and 'uname', that are oftenused but are rarely interesting. You can do this manually one program at a time or removethe whole group by right clicking and choosing to remove commonly used programs.

Removing scripts and makefiles from the graphIf the graph is still too big then Breeze will remove scripts and makefiles from the graph.You can modify this behaviour by selecting 'Trace Menu > Preferences > Node options',but it is best to leave these options set to their defaults as you can accidentally removeimportant information from the graph and other views.

Showing just the root nodeFinally, if the graph is still too big then Breeze will show you just the root node. It is thenup to you to use the other views to find the information you need.

-9/27-

Using the search view to find programsEven if a program is not shown in the graph you can search for it in the Search View. Youcan then click on a search result to see more information in the Node View and EventView, or you can right click for more menus.

If you choose to show just that program in the graph then other programs will disappear,but you will see the chosen program in orange with its children in yellow. By right clickingon the nodes you can selectively add parent programs or expand children to get a moredetailed picture of this part of your application.

Using the Search View to find files, libraries and network nodesYou can also use the Search View to locate particular files and network nodes. If you selecta search result, the programs that used that resource will be listed in the Node View so itis easy to find out which files have been used by a particular program and which programshave used a particular file.

Get the chain of programs from the root node to the selected nodeIf you want to visualise the chain of programs that lead to the execution of a particularprogram, you can search for it in the Search View. Then select it in the result list and openthe Parent View. This will show you a list of programs that were called before the one you

-10/27-

selected got ultimately executed. The first item on the list will always be the root node andthe last one, the program you selected.

Listing all the files used by your applicationThe Files View (opened from the Views Menu) will show you a list of all the files used byyour run. This can help you to see which files need to be shipped with your application, orto see whether any illegal or user-specific files have been used by mistake.

Get a list of symlinks and their target(s) used by your applicationThe Symlink View (opened from the Views Menu) will show you what symlinks are openedor read by your application. You will get the paths of the links on the left column and thepaths of the targets on the right column. In some cases, Breeze might not be aware of thetarget of the link, in which case it will be clearly indicated. If your application uses thesame link for different targets over time, you will see one line per target.

-11/27-

6. Tracing applications remotely on a cluster or with customersSo far we have been tracing from the command line inside Breeze, but you will probablywant to trace applications on your computer cluster, or you may want to ask customers totrace their programs and send you the results. To do this you will use the trace-program.sh script, which is available in the full Breeze download as well as the trace-onlyversion.

You use the trace-program.sh script just like the trace command in Breeze, but you alsoneed to use the -f option to give the location of the trace output directory.

> ./trace-program.sh -f traceOutput make all

The trace-program.sh script doesn't need to access the license server so you can makeas many traces as you like anywhere in the world. This makes Breeze easy to use forcustomer services.

Importing a trace

Once the trace has finished, you can copy the traceOutput directory back to your machineand import the directory into Breeze by selecting 'File Menu > Import Saved Trace'.

Breeze will ask if you are importing a Breeze trace, or a trace recorded using strace.Breeze can import data from strace, but in general we recommend using traces recordedby Breeze, because that will give you much more information in a way that is easier tounderstand.

You can of course use 'File Menu > Export Current Trace' to save a trace that you havemade interactively.

Decoding a trace to read as plain text

You or your customers may want to read the trace data before importing it into Breeze.Your customers may want to remove confidential file names and environment variablesfrom the trace. They may also want to verify that Breeze doesn't collect the contents offiles.

> ./decode-trace.sh traceOutput

By running the script above you can convert the trace data to plain text. The plain textoutput will be stored in traceOutput/decoded where each of the files can be opened bya standard editor such as emacs.

All strings, names and variables in the trace are listed in the file called strings. Yourcustomers can change the values in that file to hide confidential names before sendingyou just the plain text trace. Both the raw trace and the plain text trace can be imported

-12/27-

into Breeze.

Advanced options for tracing using trace-program.sh

You can see all the options offered by trace-program.sh by running the script with nooptions, but a summary is included below.

The following options let you control whether you are running Bash or Tcsh (Csh)commands. It will also help you to deal with command line aliases since aliases need to bepassed in explicitly. You should save your current alias definitions to a file then use theoptions below to pass that file to Breeze. This is more of a problem for Tcsh than Bashbecause it is common to use complex Tcsh or Csh alias definitions to make up for the lackof functional support.

-ab <alias file>Supply a file of bash alias definitions. Breeze needs the definitions in order to tracealiases. A suitable alias file can be generated by running the following command inbash before running this script:

alias > alias.txt

-at <alias file>Supply a file of tcsh or csh alias definitions. Breeze needs the definitions in order totrace aliases. A suitable alias file can be generated by running the followingcommand in tsch or csh before running this script:

alias > alias.txt

-tRun the command to be traced in tcsh. By default commands are interpreted asbash commands.

-STrace calls in the stat family (stat, fstat and lstat). By default these calls are nottraced. Specifying this option can significantly slow down tracing and will increasethe size of the traced data.

-s <shell path>The path to the user's shell. This is used in tracing interactive sessions executedusing su, ssh, and similar programs.

--remote=<bsub|qsub|rsh|ssh|[bsub][,qsub][,rsh][,ssh]>--remote=<yes|no>-r

This options turns remote tracing on or off. Remote tracing is on by default. Whenthe option is turned on, Breeze traces jobs launched on remote machines usingbsub, qsub, rsh or ssh. Using the first form, any combination of bsub, qsub rshand ssh can be enabled while the second form can be used to turn them all on or

-13/27-

off.

The remote machine must have an identical Breeze installation in the samedirectory as the first machine, and the trace output directory must be located on ashared file system that is mounted in the same place on each machine. The Breezeenvironment variables don't need to be set up on the remote machine. The remotetrace outputs will have the same basic name as the local trace, but they will havethe hostname and the date appended.

The traces of the child jobs will not be loaded into Breeze at first, but you can dothis easily by double clicking on the yellow 'child job' nodes in the graph.

-14/27-

Customer case study: Using Breeze for customer servicesOne of our customers had a customer who kept seeing an error that indicated that thewrong arguments were being passed into a tool. They were sure that the argumentswere right; they even sent screen shots to prove what they were typing on thecommand line.

Our customer asked them to trace it with Breeze and send him the results. It was acomplex set of scripts and programs so he compared it with a run that he'd made in-house. He saw that there was an extra wrapper script calling the tool in question. Thiswrapper script had a bug and was messing up the arguments and causing the error.

Using Breeze he was able to see the problem and tell them how to fix it. It saved him alot of time and an on-site visit.

7. Comparing two applications:Sometimes you will want to compare two applications that have been run by differentusers or on different machines. This can help you to quickly resolve a problem if one worksand one doesn't. It can also be a good way of making sure that the environments reallyare the same if you are struggling to reproduce an issue.

Once you have loaded your two traces into Breeze you should select 'Trace menu >Compare two runs'. Use the drop down menus to select one run on the left and the otheron the right.

If your runs are very large, or you want to compare a subset of the application, you canright-click on any node and select 'Make this the root program'. You are now ready tocompare the traces.

Step 1: Pair the programs by name. Programs that are found in both traces will turngreen, while programs that only appear in one trace will turn red.

-15/27-

-16/27-

Step 2: You can now compare each pair of programs to see if they are the same. Choosethe property to be compared from the drop down and then click 'Compare PairedPrograms'. Programs that have the same property will be marked with a green tick andthose that are different with a blue cross. So, for example, if I choose to compare 'programargs' then all the programs that have different arguments will be marked with a blue cross.

You can compare a number of different program properties. Comparing file system reads isa good way of checking application dependencies. You can see, for example, if both runsof your application are using the same perl libraries.

Step 3: You can compare program differences in more detail by viewing a text comparisonof the differences, or by comparing node and event views. By comparing applications inthe event view you can sometimes spot performance differences that are causing aproblem.

You can also compare programs that are not paired by deselecting the 'pair programselection automatically' option at the bottom of the dialog.

Customer case study: Comparing programs on different machinesOne of our customers couldn't work out why a program had stopped working on one oftheir servers, but not the others. They traced the program on a working machine as wellas on the problem machine. When they compared the results in Breeze, they saw thatthey were sourcing different Perl libraries.

Some weeks previously, someone had accidentally updated the Perl installation on oneof the machines. They were able to find the problem with Breeze, even though theydidn't know their build had Perl dependencies.

-17/27-

8. Profiling file I/OWe are now going to look at profiling file system and network I/O. This will help you to findperformance bottlenecks and improve the scalability of your application. Breeze profilingdata can help you detect when your application is storing files in the wrong place oraccessing your shared file system inefficiently. Breeze will also tell you what kind ofperformance your application is seeing from your storage infrastructure.

NOTE: Profiling used to be a separate mode, but is now part of tracing soyou can make one trace and see both trace information and I/O profilingdata at the same time.

For more information about the options you can choose, look at the profiling-options.sh script, which lists all the configuration variables you can change.

To access the profiling data choose 'File Menu > View I/O Stats'

-18/27-

Breeze can be configured to record the following different types of data. By default theywill all be enabled, but once you are familiar with the possibilities we recommend that youdecide what you want to see and then record just that, to minimise the performanceoverhead. Edit the profiling-options.sh script to reflect your needs, and run:

source profiling-options.sh

It is also possible to export these data with a valid BreezeAP license. This license shouldreside in the same directory as the Breeze license.

In each view, there is a “Show in Duration View” button, which will switch to the DurationView and set the selected time as filter. This is very useful to have a better understandingof which programs were active at that specific time and contributed in the statistics.

When showing profile data for a specific PID and switching to the Duration View via theavailable button, it might happen that the table does not show this specific PID. Thishappens when the option “Fold forks to combine multi-process programs into one node” isset, so the program that triggered the profile log is folded in another node. To have a fulloverview of all the programs, the option mentioned above could be deselected, but beaware that this might lead to a graph with too many nodes which could not be rendered infull.

File-system bucketsBreeze collects statistics for specific areas of the file system, which we refer to as buckets.A bucket may be any file or directory, but we recommend choosing remote file systemmount points and logical areas of the file system, such as install directories or areas wheredata is stored. This will give you a high-level understanding of what your program is doing.The list of buckets can be configured using an environment variable in profiling-options.sh. By default, this will contains the top level directories and the mount points ifthe list of mount points does not exceed the maximal size for an environment variable.

-19/27-

File-system I/O countsThis option tells Breeze to count the number of times the file system is accessed. Breezewill measure the number of open, seek, read or write calls, as well as the number of bytesread or written and the seek distance.

Other operations recorded are stat calls which check permissions of files, and file systemchanges such as move or delete. By default the counts are reported every second, but youcan configure that as well. There will be a count for each operation type for each of the'buckets' you have chosen.

-20/27-

File-system I/O latencyBreeze can measure the time taken to access the file system. It will collect separatestatistics for each of the file-system 'buckets' that you have chosen. For each bucket andaccess type (e.g. read, open, stat etc.) you will get a latency table.

The first two columns show the minimum and maximum latency in microseconds. The nextcolumn headings are the latency ranges, which can be configured in profiling-options.sh. The table contains counts for each of these ranges. In the example below, inthe first second the longest read call took 62μs, there were 215 read calls that took 0-1μs,and 183 read calls that took 1-10μs.

Network I/O count and latencyThese options provide the same sort of information as the file-system I/O counts andlatency, but for the network. You don't need to set up network 'buckets' because Breezewill record information for every network location. This is practical because Breezeassumes that your application will be accessing a small number of network locations. Ifthat is not the case for your application then please contact us so that we can tune Breezefor your application.

-21/27-

Error code counts and zero-byte I/O countsBreeze can show you any errors returned by I/O calls and the number of zero byte readsand writes. Some errors and zero-byte operations are expected as they are used byprograms to understand permissions and to test the existence of files. Lots of errors, orunusual errors, may indicate a more serious problem. Each error value (errno) is countedseparately, so you can diagnose any network or file system problems you are having.These statistics will also be reported in terms of any 'buckets' that you have set up.

Memory consumptionIf enabled, Breeze will report the memory used by the program being profiled. This recordsthe VM (virtual memory) and RSS (resident set size) as an approximation for how muchmemory is used. The total is not available for this measure as it doesn't make sense, soinstead you must specify the process ID of the program you want to see the data for.

CPU UsageThis mode shows you various information about how CPU is stressed by your application.As measures cannot be enforced to be at regular intervals across processes, you will onlybe able to see per-PID information. Displayed information includes User and System load,and voluntary and involuntary context switches.

Symlink chain countsA long chain of symbolic links could impact on performance. If this option is enabled, eachfile opened will be checked to see if it is in a chain of symbolic links.

-22/27-

File-system trawlsA program will often look in a number of directories to find a particular file or program, andin the worst case it may search the entire file system before it finds it. For example, aprogram might attempt to access a file within each directory on the PATH until iteventually finds the file it is looking for. On distributed file systems this can affect not onlythe performance of that program, but also the performance of all the other programsrunning on the cluster. We refer to such a sequence of calls as a 'trawl'.

In more detail, if Breeze sees a specific file system call fail at least N times, without anyother intervening calls, it considers this to be a 'trawl' (the value of N is configured inprofiling-options.sh). The trawl is ended either by a successful call of the same type,or a different call. Breeze records the number of failed calls in the trawl, the name of thefile associated with the final failed call, and the total time taken by the whole sequence offailed calls.

-23/27-

Customer case study: Optimising file-system using BreezeWe developed the I/O profiling features for the IT department at ARM. They wereworried that some of their programs were storing temporary files on remote areas ofthe file system. These programs were hogging the network and harming their ownperformance, as well as making poor use of their networked storage.

We made a version of Breeze that could measure how often areas of the file systemwere accessed, instead of looking at which files were used, as it normally does. The firsttime we ran Breeze on their tools, we saw that programs were storing files in the wrongplace. They are now able to get much better use of their resources.

-24/27-

9. Getting application I/O dependencies

In order to see what files your application used (or tried to use), you can have a look at theFilesView, in the left panel. This view contains a row for each file that your application accessed (or tried to). The table is made of 8 column groups which can be expanded to get more information:

- Location: this tells you what file the row is about. Initially displays file names, but can also display full path, file type and bucket containing the file if any.

- Read, Write and Seek: gives you information about these calls. Initially displays number of calls, but you can also get the amount of Bytes that was involved, the total latency of these calls, the maximum latency and the number of failed calls.

- Small Read, Small write and Small seek: similar to the above but only take into account calls that involved less than BREEZE_PROFILE_SMALL_IO (defaults to 4096) Bytes.

- Other Operation: gives you information non I/0 operations that targeted this file. Initially shows the total number of non-I/O operations but can be expanded to break this number down between Open, Load and Stat calls, differentiating between successful ones and failed ones.

The DirectoriesView (also in the left pane) shows similar data, except that it is aggregated per directories. This can be useful to check that your application used the rightareas of the file-system, e.g. local vs. shared storage.

-25/27-

If your application uses Message Passing Interface (MPI) you might want to have a look at which files are accessed through MPI. You can do that by opening the MPIView. It will display the number of calls, the total latency, the maximum latency and the number of failed calls for each of the following calls: Open, Filesystem Change, Stat and Sync. These numbers only take into account the calls from the MPI implementation your application uses. Currently, only the following implementations are supported: MPICH (version 3.2), MVAPICH2 (version 2.2) and OpenMPI (version 2.0.2). If your application uses another MPI implementation, then the numbers might not be accurate.

Note that files that are not accessed at all through MPI will not appear in this view, but youcan still find them in the FilesView.

You can also get an overview of the I/O that was done over the network with the NetworkView. This will show you data about the following calls:

- accept, connect, bind and listen calls: number of calls, total latency, maximum latency and number of failed calls;

- read and write calls: number of calls, number of bytes read/written, total latency, maximum latency and number of failed calls.

-26/27-

10. Breeze Automation Platform and exporting data

The Breeze Automation Platform offers a full solution to integrate Breeze into yourcompute infrastructure. It enables you to convert trace data straight into XML and CSVoutput for automatic processing. This allows run checks over thousands of tracedprograms.

You can run automatic checks on your IT set-up to be assured of high-quality configurationand long-term system compliance. The Breeze Automation Platform gives you full accessto tracing and profiling data so you can write your own custom queries. By integrating theBreeze Automation Platform into your working practices, you can resolve issues beforethey become problems.

The Breeze Automation Platform is licensed as a separate product from Breeze so you willneed to contact Ellexus if you do not already have a license. For more information abouthow to run the Breeze Automation Platform, simply run breezeAP.sh with no arguments.

Customer case study: Profiling applications for ARMARM has a tiered storage architecture which makes it important to ensure that jobs andtool flows do not spend too long waiting while a file is fetched. Accessing the file systemunnecessarily can slow down the network and affect other jobs.

Before tools and scripted flows are allowed to run on the main cluster, they now arechecked on a smaller practice cluster with Breeze. The initial benefits of using Breezewere seen immediately. ARM is now able to sign off new flows more quickly and moresafely.

“We are looking forward to catching problems before they happen by profilingour flows with Breeze. Long term we expect to see significant gains frommore accurate prediction of our network and file system load and an overallreduction in traffic.”

Julien Palleau, Director of Engineering IT at ARM

-27/27-