The Promise of Enterprise Hybrid Cloud File Storage€¦ · exabytes by 2022 respectively.1 An exabyte is a million terabytes. To put that in perspective, you can store 341 billion

W H I T E P A P E R

The Promise of Enterprise Hybrid Cloud File Storage

FILE STORAGE AT THE CROSSROADS

August 12, 2019

The Promise of Enterprise Hybrid Cloud File Storage 2

W H I T E P A P E R

Table of Contents

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

A Perfect Storm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Scale-Across File Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Re-Thinking The File Storage Industry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

The Limitations Of Legacy Storage Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

The Challenges of Object Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Inadequacy of Cloud-Based File Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Qumulo Enterprise-Proven Hybrid Cloud Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

How Qumulo Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

The Qumulo File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Real-Time Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Real-Time Quotas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Audit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Continous Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Scalable Block Store (SBS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16


W H I T E P A P E R

Executive SummaryLarge-scale file storage has reached a tipping point. The amount of unstructured data has been growing steadily, and its growth has accelerated to the point that companies wonder how they will be able to manage the ever-increasing scale of their digital assets. In addition, the global reach of public clouds are creating new demand for the mobility of file data.

As a result, new requirements for file storage are emerging for enterprises, underscoring the need for a scale-across file storage system. Such a system would have no upper limit on the number of files it could manage, regardless of size, and it would be able to run anywhere, on-prem or in the cloud, or both.

Qumulo offers an enterprise-proven, highly scalable, hybrid cloud file storage system that can span the data center and the cloud. It scales to billions of files, costs less, and has a lower TCO than legacy storage solutions. Qumulo also provides the highest performant file storage system on-prem and in the cloud. With built-in real-time analytics, administrators can easily manage data no matter the file size or where it’s located globally.

Qumulo’s continuous replication enables data to move where it’s needed, when it’s needed; for example, between on-prem clusters and clusters running in the cloud, or between cloud clusters. Qumulo’s software runs on Intel® Xeon® Gold based industry-standard hardware and was designed from the ground up to meet today’s requirements for scale. Qumulo offers the world’s first scale-across file storage system, allowing modern enterprises to easily store and manage files numbering in

the billions, in any operating environment, anywhere in the world.

A Perfect StormIDC predicts that the amount of data deployed in public clouds, private clouds, and on-prem for file services is expected to reach 45.5 exabytes, 10.6 exabytes and 57.3 exabytes by 2022 respectively.1 An exabyte is a million terabytes. To put that in perspective, you can store 341 billion three-minute MP3s in an exabyte - that is a lot of music.

Machine-generated data, virtually all of which is file-based, is one of the primary factors behind this dramatic acceleration of data growth. Life sciences researchers, for example, who are developing the latest medical breakthroughs, use vast amounts of file data for genome sequences and share that data with colleagues around the world. Oil and gas companies’ greatest assets are their file-based seismic data used for natural gas and oil discovery. Every movie and television program we watch is produced on computers and stored digitally as files. Text-based log files—data about machines, created by machines—are proliferating at an ever-increasing rate. The increasing need for security and safety monitoring has caused video surveillance cameras and security devices to be pervasive across many public and private organizations, resulting in an extraordinary amount of unstructured data.

One drop of human blood creates enough data to fill an entire laptop computer, and some research projects require a million drops.

1 Worldwide File-Based Storage Forecast, 2018–2022: Storage by Deployment Location, IDC December, 2018Intel, the Intel logo, the Intel Inside logo and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries.


W H I T E P A P E R

There is also a trend toward higher-resolution digital assets. Uncompressed 4K and 8K video is the new standard in media and entertainment. The resolution for video and images created by digital sensors and scientific equipment is constantly increasing. Higher-resolution causes file sizes to grow more than linearly. By doubling the resolution of a digital photograph, it’s size increases by four times. As organizations demand more fidelity from digital assets, storage requirements continue to grow.

The continued rise in data volumes is paralleled by the advent of the public cloud. Its arrival overturned many basic assumptions about how storage should work.

The rise of the public cloud signalled that compute resources and global reach were now achievable without building data centers across the world. Consequently, new ways of working have arrived and are here to stay. All businesses realize that, in the future, they will no longer be running their workloads out of single, self-managed data centers. Instead, they will be moving to multiple data centers, with one or more in the public cloud, or completely in the cloud. This flexibility will help them adapt to a world with geographically-dispersed employees and business partners. Companies will focus their resources on their core business lines instead of on IT expenditures. Most will improve their disaster recovery and business continuity plans, and many will do this by taking advantage of the cloud.

Users of legacy scale-up and scale-out file systems, long considered workhorses of file data, find that those systems are often inadequate for a future shaped by tremendous amounts of unstructured data. A core part of this problem is that the metadata within large file systems—their directory structures and file attributes—has itself become unmanageable.

Legacy solutions often rely on brute force to provide insight into the storage system, and brute force has been defeated by scale. For example, tree walks - the sequential processes that scan nested directories as part of routine management tasks - have become computationally infeasible. Brute force methods are fundamental to the way legacy file systems are designed and cannot be fixed with patches.

Against this backdrop of profound change, file storage users still need to maintain and safely manage large-scale, complex workflows that rely on collaborations between many distinct software programs, operating systems, and individuals. Moreover, the traditional buying criteria of price, performance, ease-of-use, and reliability remain as important as ever, no matter how much the landscape has changed.

The storage industry finds itself at a crossroads, which includes both new challenges and new opportunities. Without innovation among storage providers, users of large-scale file storage will continue to struggle to understand what is going on inside their systems. They will struggle to cope with massive amounts of data. They will struggle to meet the demands for global reach, with few viable options for file data that span both the data center and the cloud.


W H I T E P A P E R

Scale-Across File Storage Traditionally, companies face two problems when deploying file-based storage systems: they need to scale both capacity and performance simultaneously. In the world where the growth of unstructured data is unrelenting, scale is no longer limited to these two axes. New criteria for scale have emerged, including the number and size of files stored, the ability to control enormous amounts data in real-time, to distribute data globally, and the flexibility to leverage on-prem, hybrid, or cloud deployments. These requirements define a new market category called scale-across file storage.

Scale-across file storage scales to billions of files. The notion that capacity is only measured in terms of bytes of raw storage is giving way to a broader understanding that capacity is just as often defined by the number of files that can be stored. Modern file-based workflows include a mix of large and small files, especially if they involve any amount of machine-generated data. As legacy file systems reach the limits in the number of files they can effectively store, buyers can no longer assume that they will have adequate file capacity.

Scale-across file storage works across operating environments, including on-prem data centers, as well as private and public clouds. Proprietary hardware is increasingly a dead end for users of large-scale file storage. Today’s businesses need flexibility and choice. They want to store files in data centers, in private clouds and/or public clouds, opting for one or the other based on business decisions rather than on the technical limitations of their storage platform.

Today, capacity means more than terabytes of raw storage.


W H I T E P A P E R

Companies want to benefit from the rapid technical and economic advantages of standard hardware, such as denser drives and lower-cost components. They want to reduce the complexity of hardware maintenance through standardization and streamlined configurations. The trend of software-defined storage on standard hardware will only continue. Users responsible for large amounts of data require their storage systems to run on a variety of operating environments and not be locked in to proprietary hardware.

Scale-across file storage scales across geographic locations with data mobility. Today’s businesses are global. Their file-based storage systems must now scale across geographic locations. This may involve multiple data centers, private clouds and almost certainly public clouds. A piece-meal approach and a label that says “cloud-ready” simply won’t work. True mobility and global reach are now required.

Scale-across file storage provides real-time visibility and control. As the number of files being managed today has grown to billion-file scale, the ability to control storage resources in real-time has become an urgent requirement. Storage administrators must be able to instantly monitor all aspects of system performance and capacity, regardless of the size of the storage system.

Scale-across file storage gives access to rapid innovation. Modern file storage needs a simple, elegant design and advanced engineering. Companies that develop scale-across file storage will leverage Agile development processes that emphasize rapid release cycles and continual access to innovation. Three-year update cycles, a result of cumbersome “waterfall” development processes, are a relic of the past that customers can no longer tolerate.

As the needs of lines-of-business surpass what central IT can provide in a reasonable time frame, accessing cloud-based resources has become a requirement. A flexible, on-demand usage model is a hallmark of the cloud. However, the shift to cloud has stranded users of large-scale file storage, who often have no effective way to harness the power that the cloud offers. A file system that is enterprise-proven and can scale-across to the cloud is required.

Re-Thinking The File Storage Industry Legacy scale-up and scale-out file systems are not capable of meeting the emerging requirements of managing storage on-prem and/or in the cloud at scale. The engineers who designed them 20 years ago never anticipated the number of files and directories, and mixed file sizes, that characterize modern workloads. They could also not foresee cloud computing.


W H I T E P A P E R

THE LIMITATIONS OF LEGACY STORAGE SYSTEMS

Qumulo often hears from organizations that they’re in a scalability crisis as the growth of their unstructured data is rapidly outpacing the design assumptions of their existing storage solutions. These legacy solutions are difficult to install, difficult to maintain, and inefficient. Putting in one of these systems is usually a service engagement that can take a week, assuming the person doing the installation is experienced. These systems often have many inherent limitations, such as volume sizes and the number of inodes (inodes store the attributes and disk block location of the object’s data), which interact and make it challenging to avoid bottlenecks.

Legacy systems are expensive, and their inefficiency adds even more to their total cost. Generally, only 70 to 80 percent of the provisioned storage capacity is actually available. System performance suffers if the disk gets any fuller. Another problem is that legacy systems were not designed for the higher drive densities that are now available. Rebuild times in the event of a failed disk can stretch into days.

Finally, traditional storage systems offer no visibility into an organization’s data. Getting information about how the system is being used is often clumsy and slow. It can take so long to get the information that it is outdated even before the administrator sees it.

THE CHALLENGES OF OBJECT STORAGE

Object storage allows for very large systems with petabytes of data and billions of objects, and works well for its intended use. In fact, it was the default that object storage technologies were the solution to the scale and geo-distribution challenges of unstructured storage. Cloud providers believed wholeheartedly in object storage.

Adopting object storage in use cases for which it was never intended is a poor technical fit. In order to achieve their scale and geo-distribution properties, some object stores have intentionally traded off features many users need and expect, including transactional consistency, modification of objects (e.g. files), fine-grained access control, and use of standard protocols such as NFS and SMB, to name a few.

Object storage also does not handle the problem of organizing data. Instead, users are encouraged to index the data themselves in some sort of external database. This may suffice for the storage needs of stand-alone applications, but it complicates collaboration between applications, and between humans and those applications. Modern workflows almost always involve applications that were developed independently but work together by exchanging file-based data, an interop scenario that is simply not possible with object storage.

A surprising amount of valuable business logic is encoded in the directory structure of enterprise file systems. The need for file storage at scale remains compelling. Qumulo’s software provides the scalability benefits of object without sacrificing features.

Object stores have intentionally sacrificed features users need and expect.


W H I T E P A P E R

INADEQUACY OF CLOUD-BASED FILE SOLUTIONS

While there is tremendous demand for running file-based workloads in the cloud, existing solutions for unstructured file management in the cloud are often inadequate. These solutions are either sold by the cloud providers themselves or by legacy storage vendors. In the first case, the solutions are immature. In the second, they apply 1990’s technology to 21st century problems.

For example, cloud-only file systems are limited by the fact that they don’t connect with a company’s on-prem data center in any way. Further, they lack important enterprise features, such as support for the Server Message Block (SMB) protocol, quotas, snapshots, replication and audit that are needed for modern file-based workflows in data-intensive industries.

The efforts of legacy storage vendors to pivot to the cloud have resulted in solutions with limited capacity and limited scalable performance. This inflexibility negates the very reason businesses are turning to the cloud - the ease of adding more compute power.

None of the legacy solutions provide real-time visibility and control of the data in the cloud, which leads to over-provisioning of capacity, performance, or both. In general, current solutions for file storage in the cloud are piecemeal approaches that address only parts of the problem. Customers are stranded in their attempts to integrate file-based workloads with the cloud.

Qumulo’s Enterprise-Proven Hybrid Cloud StorageQumulo was founded in 2012, as the crisis in file storage was beginning to reach its tipping point. A group of storage pioneers, the inventors of scale-out NAS, joined forces and formed a different kind of storage company, one that would address these new requirements head-on. The result of their work, and of the team they assembled, is Qumulo, which developed the world’s first scale-across file storage system.

Qumulo’s enterprise-proven, hybrid cloud file storage system spans the data center, the private clouds and/or public clouds. It scales to billions of files, costs less, and has a lower TCO than legacy storage solutions. It is also the highest performance file storage system on-prem and in the cloud. Real-time analytics let administrators easily access and manage data regardless of size or location. Qumulo’s continuous replication enables data to move where it’s needed, when it’s needed; for example, between on-prem clusters and clusters running in the cloud or between clusters running on different cloud instances.

There are no mature products among existing cloud-based file solutions.


W H I T E P A P E R

With Qumulo’s file storage, cloud instances or computing nodes with Intel® Xeon® Gold based standard hardware work together to form clusters that provide scalable performance a single, unified file system. Qumulo clusters work together to form a globally distributed, highly connected, storage solution tied together with continuous replication.

Customers interact with Qumulo clusters using industry-standard file protocols such as NFS and SMB, the Qumulo REST API and a web-based graphical user interface (GUI) for storage administrators. Below is an example of the GUI.

Qumulo clusters work together to form a globally distributed but highly connected storage solution tied together with continuous replication.

Intel, the Intel logo, the Intel Inside logo and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries.


W H I T E P A P E R

Qumulo’s software has a unique ability to scale. Here are some of the capabilities that set our file system apart from legacy file storage solutions.

Qumulo scales to billions of files. With Qumulo, you can use any mix of large and small files, and store as many files as you need. There is no practical limit with Qumulo’s advanced file system. Many of Qumulo customers have data in excess of a billion files. This is in stark contrast to legacy scale-out storage systems which were not designed to handle modern workflows with mixed file sizes, which become very inefficient when there are many small files. This is because these legacy systems are based on a decades-old design that forces them to mirror (or double mirror, sometimes even triple mirror) files under a 128KB threshold. Qumulo is vastly more efficient at representing and protecting small files than legacy scale-out NAS, typically requiring one-third of the storage capacity and half of the protection overhead.

We developed a fundamentally different approach to data protection, protecting at the block level versus the file level. Working at the block level rather than the file level using our custom erasure coding makes it possible to protect data effectively without having to create a one-to-one copy of the entire data volume.

Qumulo provides the highest performance. Qumulo is the highest performance file storage system whether on-prem and/or in the cloud. It provides twice the price performance compared to legacy storage systems. In the data center, Qumulo’s file system is optimized for Intel® Xeon® Gold based standard hardware with Intel® SSD Data Center Family for NVMe, SSDs and HDDs, which cost less than proprietary hardware. In the cloud, Qumulo’s software intelligently trades off between low-latency block resources and higher-latency, lower-cost block options.

Qumulo has lower cost. Qumulo’s file system costs less and has a lower TCO than legacy storage solutions on a capacity basis, as measured by cost-per-usable terabyte. Qumulo’s cost advantage comes from its efficient use of storage capacity and its use of Intel® Xeon® Gold based standard hardware.

Qumulo’s cost efficiencies also make it extremely reliable. Storage system reliability is usually measured in terms of mean time to data loss (MTTDL). MTTDL is the average number of years a given cluster will survive before there’s a hardware failure that causes a significant loss of data. At a minimum, MTTDLs should be measured in the tens of thousands of years.

While some variables that affect reliability can’t be controlled by the storage system, one that can is the reprotect time, or how long it takes to recover data if a disk fails. Reprotect times matter because the longer it takes to reprotect the cluster, the more vulnerable the cluster is to other failures and the poorer the MTTDL. As disks become denser, data volumes increase, and clusters grow, a legacy storage system’s reprotect times can turn into weeks.

“Our research organization falls between the cracks for most storage vendors, with giant imaging sets and millions of tiny genetic sequencing scraps. Finding a system that reasonably handled all our complex workflows was difficult, and in the end only Qumulo was the right fit.”

— Bill Kupiec, IT Manager, Department of Embryology, Carnegie Institution for Science

Intel, the Intel logo, the Intel Inside logo and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries.


W H I T E P A P E R

Qumulo uses sophisticated data protection techniques that enable the fastest reprotect times in the industry. They are measured in hours, not days or weeks. When reprotect times are fast, reliability increases. Better reliability means that administrators can greatly reduce the level of redundancy they need to achieve target MTTDL standards, which in turn increases storage efficiency and lowers cost.

Qumulo also takes into account the drop in performance that occurs when a disk failure happens and needs to be rebuilt. Qumulo, with I/O Assurance, automatically adjusts all users’ performance so no one person or application experiences a significant performance degradation.

Qumulo makes 100 percent of user-provisioned capacity available for user files, in contrast to legacy scale-up and scale-out NAS that only recommend using 70 to 80 percent to ensure consistent performance. In addition to this 20 to 30 percent, legacy vendors often require additional capacity reserved for data protection or for administration. Further, Qumulo’s software can safely run at 2-drive protection where others require 3-drive protection given our leading-edge restripe and rebalance performance. The difference between 2-drive and 3-drive protection can be up to 15 percent of raw capacity. Certain vendors also have a “small file tax,” where managing small files less than 128K in size adds to the problem of not being able to use all of your storage.

Qumulo has real-time analytics that tell you what’s happening in your file system instantly. Analytics is an integral part of the Qumulo file system; it is not an afterthought. Instead of running multiple commands, parsing through pages of log files, and running separate programs, an administrator can simply look at the GUI and understand what’s happening. For example, an administrator can immediately see if a process or user is hogging system resources and, in real-time, apply a capacity quota.

Qumulo gives you the freedom to store and access your data anywhere. Qumulo is hardware-independent and can run both in the data center and/or in the cloud, while still offering the same interface and capabilities to users, whether they are on-prem, off-prem, or spanning both. Administrators have the freedom to take advantage of the compute resources that the cloud offers, and then move data back to their data centers as needed.

Qumulo has industry-leading support. Many storage customers are dissatisfied with the support they receive from their vendors. They find them to be unresponsive and reactive rather than proactive. Qumulo offers responsive, personal, customer support, with one of the highest Net Promoter Scores (NPS) in the industry.

Qumulo has simple subscription pricing. Businesses often feel they are being held hostage by the high cost of their existing storage solutions. If they want to upgrade their hardware after three years, they’re forced to throw out software licenses associated with their legacy hardware. Even if they wish to run their storage systems

“With critical high-profile projects, you want to know exactly what you’re going to be leaning on for successful delivery. When the ‘La La Land’ project came around it was make or break, and we were never down for a moment. Qumulo is our rock, allowing us to focus on the visual effects with absolute confidence that the data is safe.”

— Tim LeDoux, Founder/VFX Supervisor, Crafty Apes


W H I T E P A P E R

for seven years instead of three, their vendor forces them to replace their hardware by way of exorbitant support quotes. Pricing is complicated and figuring out how much a system will cost is far from straightforward. In contrast, Qumulo’s pricing is based on a single, simple subscription service that covers everything, including software, updates and support.

Qumulo provides cloud-based monitoring and trends. A Qumulo software subscription includes cloud-based monitoring that proactively detects potential problems, such as disk failures. Administrators can also access the Qumulo trends service, which provides historical data about how the system is being used. This information can help lower costs and optimize workflows.

Qumulo provides access to innovation. Qumulo follows Agile and other modern development practices, which means it has many small releases that steadily improve the product and keep it on the leading edge of what’s possible. This is in contrast to legacy storage vendors that have infrequent releases that can keep customers waiting years for improvements.

Qumulo has no hardware lock-in. Qumulo uses Intel® Xeon® based standard hardware provided by Qumulo or by partners such as HPE and Dell. In the cloud, our file system can use a range of instances within AWS or GCP that you can pick according to your capacity and performance requirements.

Qumulo’s Intel® Xeon® based hardware platforms ensure that you can get the perfect solution for your needs. Qumulo’s NVMe-based system provides capacity and sustained performance, the hybrid SSD/Disk-based system provides the performance of flash at the price of disk, and the active archive solution provides incredible density.

Qumulo provides a fully programmable REST API. Customers get programmatic access to any feature or administrative setting in Qumulo. The Qumulo REST API is built for developers. The API is suitable for DevOps and Agile operating approaches, which are how modern application stacks are constructed and managed, particularly in the cloud. For example, you can use tools such as Terraform and CloudFormation to automatically spin-up Qumulo clusters in the cloud.

It turns out only 14 percent of B2B companies have a customer-centric culture. Qumulo is all about customer feedback and constantly evolves its offerings to match customer needs. Forbes, August 20192

“For a critical digital media archive, Qumulo is the safest place I can think to put it, short of directly in a backup vault. Soon we won’t need anything else but backup, high-speed virtual storage, and Qumulo.”

— Joel Hsia, Assistant Head for Systems Development, Marriott Library, University of Utah

“Managing data with Qumulo is so simple it’s hard to describe the impact. It has given us tremendous ROI in terms of time saved and problems eliminated, and having that reliable storage we can finally trust makes us eager to use it more broadly throughout the company.”

— John Beck, IT Manager, Hyundai MOBIS

2 “100 Of The Most Customer-Centric Companies”, Blake Morgan, Forbes, June 30, 2019Intel, the Intel logo, the Intel Inside logo and Xeon are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries.


W H I T E P A P E R

Qumulo offers out-of-the-box simplicity. It might seem obvious that storage administrators want a system that is easy to install and easy to manage. They have better ways to spend their time. Unfortunately, legacy storage systems can take days to set up and configure. For data center installations, Qumulo’s file system is extremely simple to install. Once the nodes are racked and cabled, all an administrator has to do is sign the end-user license agreement, name the cluster, set up an admin name and password, and perhaps enter some IP addresses. Installation is painless.

From the moment Qumulo’s software is unboxed to when it can start serving data is a matter of hours, not days. It is also extremely easy to create a Qumulo cluster in the public cloud.

“Qumulo customer care is absolutely phenomenal – the best support I’ve seen from any vendor. It’s been a real pleasure to deal with Qumulo.”

— Nathan Larsen, Director of IT, Sinclair Oil Corporation

“Qumulo allows us to move file-based data sets to a Qumulo cluster on AWS, complete our analysis, and move the artifact back to our on-prem Qumulo storage cluster, saving us time and money. The flexibility for us to move our file-based data where we need it to be is something that nobody else in the market can provide at scale.”

— Tyrone Grandison, CIO, Institute for Health Metrics and Evaluation (IHME)


W H I T E P A P E R

How Qumulo Works Qumulo is a new kind of storage company, based entirely on advanced software and modern development practices. Intel based industry standard hardware running advanced, distributed software is the basis of modern, low-cost, scalable computing. This is just as true for file storage at large scale as it is for search engines and social media platforms.

Qumulo’s file system is unique in how it approaches the problems of scalability. Its design implements principles similar to those used by modern, large-scale, distributed databases. The result is a file system with unmatched scale characteristics.

THE QUMULO FILE SYSTEM

For massively scalable files and directories, Qumulo’s file system makes extensive use of index data structures known as B-trees. B-trees minimize the amount of I/O required for each operation as the amount of data increases. With B-trees as a foundation, the computational cost of reading or inserting data blocks grows very slowly as the amount of data increases.

REAL-TIME ANALYTICS WITH QUMULO

When people are introduced to Qumulo’s real-time analytics and watch them perform at scale, the first question is usually, “How can it be that fast?”. The breakthrough performance of Qumulo’s analytics is that it continually maintains up-to-date metadata summaries for each directory. It uses the file system’s B-trees to collect information about the file system as changes occur. Various metadata fields are summarized inside the file system to create a virtual index. The performance analytics that you see in the GUI, and can pull out with the REST API, are based on sampling

“We use the same Agile methodology at Sinclair, and I’ve seen first-hand its ability to drive good products into production so much faster than with traditional 18-month monolithic releases. Given Qumulo’s existing lead on its competitors, I knew that fast development pace would help keep it out in front of our needs.”

— Nathan Larsen, Director of IT, Sinclair Oil Corporation


W H I T E P A P E R

mechanisms that are enabled by Qumulo’s metadata aggregation. In contrast, metadata queries in legacy storage appliances are answered outside of the core file system by an unrelated software component.

REAL-TIME QUOTAS

Just as real-time aggregation of metadata enables Qumulo’s real-time analytics, it also enables real-time capacity quotas. Quotas allow administrators to specify how much capacity a given directory is allowed to use for files.

Qumulo’s quotas are deployed immediately and do not have to be provisioned. They are enforced in real-time, and changes to their capacities are immediately implemented. Quotas can be specified at any level of the directory tree.

AUDIT

Qumulo’s auditing capability is easy to set-up and integrates with standard monitoring systems for enhanced security. Audit will track all events and actions with your data and can scale from thousands to millions of IOPS with minimal performance impact.

SNAPSHOTS

Snapshots let system administrators capture the state of a file system or directory at a given point in time. If a file or directory is modified or deleted unintentionally, users or administrators can revert it to its saved state. Snapshots in Qumulo’s file system have an extremely efficient and scalable implementation. A single Qumulo cluster can have a virtually unlimited number of concurrent snapshots without performance or capacity degradation.

CONTINUOUS REPLICATION

Qumulo provides continuous replication across storage clusters, whether on-prem or in the cloud. Once a replication relationship between a source cluster and a target cluster has been established and synchronized, Qumulo’s software automatically keeps data consistent. There’s no need to manage the complex job queues for replication associated with legacy storage appliances.

Continuous replication in Qumulo’s file system leverages advanced snapshot capabilities to ensure consistent data replicas. With Qumulo snapshots, a replica on the target cluster reproduces the state of the source directory at exact moments in time. Qumulo replication relationships can be established on a per-directory basis for maximum flexibility.

In the event of a disaster, Qumulo will get you back to a consistent known state with minimal impact to the business. Qumulo’s software is able to to failover to point in time snapshot efficiently by only considering new data that has written to the source. No tree walk is required. Qumulo also preserves the configuration after fail-back and enables the replication and fail-back of local users.

When people are introduced to Qumulo’s real-time analytics and watch them perform at scale, their first question is usually, “How can it be that fast?”


W H I T E P A P E R

SCALABLE BLOCK STORE (SBS)

The Qumulo file system sits on top of a transactional virtual layer of protected storage blocks called the Scalable Block Store (SBS). Instead of a system where every file must figure out its protection for itself, data protection exists beneath the file system, at the block level. Qumulo’s block-based protection, as implemented by SBS, provides outstanding performance in environments that have petabytes of data and workloads with mixed file sizes. SBS has many benefits, including:

• Fast rebuild times in case of a failed disk drive;

• The ability to continue normal file operations during rebuild operations;

• No performance degradation due to contention between normal file writes and rebuild writes;

• Equal storage efficiency for small files and for large files;

• Timely, accurate reporting of usable space;

• Efficient transactions that allow Qumulo clusters to scale to many hundreds of nodes; and

• The ability to balance performance during rebuilds.

The virtualized protected block functionality of SBS is a huge advantage for the Qumulo file system. In legacy storage systems that do not have SBS, protection occurs on a file-by-file basis or using fixed RAID groups, which introduces many difficult problems such as long rebuild times, inefficient storage of small files, and costly and inefficient management of disk layouts.

Conclusion At Qumulo, we believe that file data is the engine of innovation and that it fuels the growth and long-term profitability of modern enterprises. File data is more important than ever, and there are new requirements for how file storage must scale.

Qumulo opens new possibilities for its customers. With Qumulo’s file system, meeting the release date of a major animated motion picture gets easier. Qumulo’s technology makes it possible to achieve medical breakthroughs from multi-petabyte experimental datasets. With Qumulo, identifying security threats in a billion-file network log can be a daily reality. Determining when an event or intrusion happened that might involve thousands of video files is now possible.

Instead of a system where every file must figure out its protection for itself, data protection in Qumulo’s software exists beneath the file system, at the block level.

“Right now, Qumulo is the closest thing to an Apple unboxing, setup, and support experience in the storage world.”

— High-level executive, Top U.S.-Based Mobile Carrier


W H I T E P A P E R

At Qumulo, we believe that file data becomes transformative when it gives people the freedom to collaborate, to innovate, and to create. The needs of our customers, who are leaders and innovators in so many industries, are the sole drivers of our aggressive product roadmap.

In a enterprise-proven modern file storage system, unparalleled reliability, scale and performance are table stakes. A great storage system goes beyond that and gives companies the global access and data insight they need to make their own dreams of greatness come true. A great file storage system moves data where it’s needed, when it’s needed, and at massive scale, and it does these things at lower cost, with higher performance, more reliability and greater ease of use than other systems. Qumulo is a different kind of storage company. As the creators of the world’s most advanced file storage system, our own team of innovators puts what we believe into practice every day. Scale-across file storage that supports massive scale is our vision and our passion.

ABOUT QUMULO

Qumulo’s hybrid cloud file storage delivers real-time visibility, scale and control of data across on-prem and cloud. Qumulo customers understand storage at a granular level; programmatically configure and manage usage, capacity and performance; and are continuously delighted with new capabilities, 100 percent usable capacity, and direct access to experts. For more information visit www.qumulo.com

“I’ve worked with many different vendors, and while I’ve learned to expect problems I’ve also learned no one’s going to knock themselves out to help me. Qumulo is the complete opposite. I’ve never had so many smart people working so hard to curve the product toward what we’re trying to do.”

— Tim LeDoux, Founder/VFX Supervisor, Crafty Apes

Documents

The Promise of Enterprise Hybrid Cloud File Storage€¦ · exabytes by 2022 respectively.1 An exabyte is a million terabytes. To put that in perspective, you can store 341 billion