DUG Technology: Exascale Flash Storage

Making the move from HDDs to SSDs

DUG Technology switched from hard disk drives to petabytes of flash storage powered by Solidigm™ technology.

• DUG Technology is at the forefront of high-performance computing (HPC), combining innovative hardware and software solutions that enable clients to make use of large and complex datasets.
• To build a resilient and adaptive storage environment that enabled expansion into new markets, DUG switched from hard disk drives to petabytes of flash storage with VAST

Solidigm SSDs help make seismic data actionable 

Seismic analysis is a high-performance computing (HPC) discipline that creates images of what lies under the surface of the earth from nothing more than the reflection of sound. Useful 3D analysis requires petabytes (PB) of data and thousands of powerful computers with vast amounts of SSD storage.

Most oil and gas companies don’t possess the computational resources necessary to conduct all of this analysis in-house. Even the industry’s largest players, with access to powerful machines, often turn to companies like DUG Technology that have the capability to deliver results faster, more accurately and at a lower cost.

High-performance computing as a service 

DUG refers to this capability as HPC-as-a-service (HPCaaS): specialized, full-stack exascale-capable computation available on demand. Traditionally, DUG’s compute-as-a-service technology was available only to specific customers, such as major oil and gas companies. However, as the market took notice of its capabilities, DUG expanded its offering to other industry verticals that use this same service to tackle a diverse set of extreme computational needs.

DUG decided to bring the same “bring-nothing-but-your-data” ease of service to businesses outside of the energy sector. DUG knew that it could serve these new industry verticals economically because of its specialized DUG HPC Cloud service. The VAST Data Platform, which includes a storage foundation layer powered by Solidigm technologies, undergirds DUG HPC Cloud and enabled DUG to successfully break into new verticals, including academia, astrophysics, medicine and genomics, wildfire modeling, and defense. However, getting to this point required a sea change in how DUG dealt with its storage.

Solidigm SSDs accept the challenge

For its first decade of operation, DUG had been deploying and managing HDD-based storage to deliver the scale and cost economy that its seismic workloads required. During that time, DUG thoroughly optimized its applications to make use of the capabilities of its Lustre HDD-based infrastructure, while working with their limits.

DUG had to make compromises to maximize productivity while managing cost. For example, when Lustre file system clients would hit peak throughput for a given workflow, other users sharing the same file system would suffer slowdowns. Although DUG designed its software to protect against HDD failures, the need to swap out failed drives on a weekly basis was a constant thorn in DUG’s side, pushing up costs in both time and resources .

New applications for HPCaaS called for new storage solutions

While DUG’s applications were well optimized for Lustre and HDD storage, DUG was evolving new applications that handled storage input/output (I/O) differently. These applications call for storage versatility and multitenancy. DUG needed new solutions that would support a broad set of requirements at exascale.

DUG also needed storage solutions that could handle the multiplicity of throughput requirements for different applications. DUG looked to solid state drive (SSD)-based storage to provide higher performance and reliability. However, moving to SSDs on Lustre would have been prohibitively expensive, and affordability was paramount for DUG.

To build a resilient and adaptive storage environment that enabled expansion into new markets, DUG required a new approach to storage.

Solution: VAST Data universal storage using Solidigm SSDs

DUG chose the VAST Data Platform to expand its business and support the needs of a wide variety of new markets and customers. The platform offering combines the speed and scale of a parallel file system with a new level of flash affordability and multitenancy to deliver a complete technological leap forward for DUG. VAST Data’s disaggregated shared everything (DASE) architecture also provides consistent performance by isolating non-optimized I/O so as not to impact other tenants.

With the DASE approach, VAST Data eliminates the concurrency challenges of parallel storage to deliver high performance for specific workloads that does not come at the expense of other workloads.

Beyond significantly improving the customer performance experience, VAST Data provides a combination of reliability, management, and support that is not otherwise found with legacy HPC storage technologies.

Exascale scalability with VAST Data’s DASE architecture 

VAST Data’s DASE architecture supplies exascale scalability, which enabled DUG to grow to tens of petabytes of flash storage. This architecture can quickly recover from failure, too, since there are no single points of failure. The reliability of the DASE architecture comes “for free” as a direct result of VAST Data’s data-protection efficiency and the architecture's statelessness.

Beyond resilience, the VAST Data Platform also simplifies DUG’s deployment and management experience. The integrated scale-out appliance consistently pushes out new features that are automatically applied while the system is online, so there’s no downtime for DUG.

Overview of VAST Data universal storage with Solidigm storage technologies

The VAST Data Platform provides a single, global namespace so that each application has access to all of the associated data for that workload. The VAST Data solution combines:

• All-flash drive performance
• Massive scalability
• The economics of archive storage
• The simplicity of plug-and-play network-attached storage (NAS) connectivity
• Full multiprotocol support (SMB 2.1, 3.0, NFS v3, v4.1, S3 with NVMe/TCP for block access)

Solidigm’s D5-P5336 provides the hardware basis for the cost-efficiency and reliability of the VAST Data Platform. Solidigm utilizes complementary metal-oxide-semiconductor (CMOS) under-array architecture, which delivers the highest areal density (gigabytes of storage per square millimeter) in the industry for the same bits per cell.¹

This means that Solidigm QLC 3D NAND SSDs provide not only greater areal density than previous-generation triple-level cell (TLC) media, but greater areal density and higher reliability than competing quad-level cell (QLC) designs.