Storage Server

Overview

Exadata provides a storage layer that cooperates in the processing of database requests, not just by storing and shipping data, but by scaling out and parallelizing Oracle Database requests. For example, rather than send an entire database table across the network to the compute server to find a small number of records, such data filtering is done in storage so that only the resulting records need be sent across the network. These powerful servers run intelligent database software to act as a storage array, modular in design, built to easily scale in capacity and performance as the database grows. The database-aware Exadata Storage Server, invented by Oracle to replace the traditional storage array, is the foundation of Exadata.

Exadata uniquely understands database workloads and knows when to avoid caching data that will negatively affect overall performance. For example, if large write I/Os caused by backups or large table scans are likely to disrupt higher priority OLTP or scan operations, those large I/Os will bypass the flash cache and go straight to disk. Otherwise, Exadata System Software will utilize additional spare flash capacity and I/O bandwidth to optimize performance by caching these I/Os. In addition to automatic caching, administrators can optionally provide SQL directives to ensure that specific tables, indexes, or partitions are preferentially retained in the flash cache.

The automatic data tiering between RAM, Exadata RDMA memory (XRMEM), flash, and disk in Exadata provides tremendous advantages over other flash-based solutions. Many storage vendors have developed flash-only arrays to achieve higher performance than traditional arrays. These flash-only arrays deliver better performance but cannot match the cost advantages of Exadata's smart tiering of data between disk, flash, and memory, as the overall size of data that can benefit from flash is limited to the size of expensive flash. And these flash arrays are unable to benefit from Exadata's unique database-aware storage optimization technologies. Generic data deduplication provided by some flash arrays is effective for Virtual Desktop Infrastructure environments, but not for databases. Exadata not only delivers much more capacity than generic all-flash arrays, it also delivers better performance. Flash-only storage arrays cannot match the throughput of Exadata's integrated and optimized architecture with full 100 Gb RDMA over converged ethernet (RoCE) based scale-out network, fast flash drives, offload of data intensive operations to storage, and algorithms throughout that are specifically optimized for databases.

A single rack configuration of Exadata Database Machine X11M can achieve up to 25.2 Million 8K database read1 I/O operations per second (IOPS) workloads. For data warehouse environments that require the highest performance, Exadata X11M High Capacity and Extreme Flash storage servers are capable of scanning up to 100 GB/s per storage server. In addition, Smart Scan also reads columnar data in XRMEM accelerating analytics query throughput up to 500 GB/s per server for a total of 8.5 TB/s2 scan throughput from a single rack configuration. These are real-world end-to-end performance figures measured running SQL workloads with standard 8K database I/O sizes inside a single rack Exadata system, unlike storage vendor performance figures based on small I/O sizes and low-level I/O tools and are therefore many times higher than can be achieved from realistic SQL workloads. Exadata's performance on real database workloads is orders of magnitude faster than traditional storage array architectures, and is also much faster than current all-flash storage arrays, whose architecture bottlenecks on flash throughput. Exadata X11M uses a combination of scale-out storage, RDMA over Converged Ethernet networking, database offload, Exadata RDMA memory accelerator, and PCIe Flash to deliver extremely high performance from memory and flash.

1Elastic configuration with 9x Exadata X11M Database Servers and 9x Exadata X11M Extreme Flash or 9x Exadata X11M High Capacity Storage Servers
2Elastic configuration with 2x Exadata X11M Database Servers and 17x Exadata X11M High Capacity or 17x Exadata X11M Extreme Flash Storage Servers

Extreme Flash (EF) Storage Server

Exadata Extreme Flash (EF) Storage Server is the foundation of a database-optimized all-flash Exadata Database Machine. Each EF Storage Server contains four capacity-optimized 30.72 TB flash drives for an aggregate, raw storage capacity of 122.88 TB and increasing the usable storage capacity by 2.4x1.

Each EF Storage Server includes four 6.8 TB performance-optimized flash drives, offering 27.2 TB of Exadata Smart Flash Cache. The size of the Smart Flash Cache increases by 11.5x2 and is used to satisfy read and writes requests. Exadata delivers ultra-high performance by placing all the flash devices directly on the high-speed PCIe interface rather than behind slow disk controllers. Exadata EF Storage Servers include 1.25 TB of DDR5 Exadata RDMA Memory (XRMEM) as a data acceleration tier in front of Flash Cache.

For data warehouse environments that require the highest performance, Exadata X11M HC and EF storage servers are both capable of scanning data in flash at up to 100 GB/s with Smart Scan and up to 8.5 TB/s by reading columnar data cached in Exadata RDMA Memory3 from a single rack configuration.

These represent real-world, end-to-end performance metrics measured running SQL workloads with standard 8K database I/O sizes inside a single rack Exadata system. Exadata's performance on real Oracle Database workloads is orders of magnitude faster than traditional storage array architectures and much faster than current all-flash storage arrays.

Exadata X11M Extreme Flash Storage Server Specifications

CPU
  • 2 x 32-core AMD EPYC™ 9J15 processors 2.95 GHz (up to 4.4 GHz)
Memory
  • 256 GB
  • 1.25 TB Exadata RDMA Memory
Flash
  • 4 x 6.8 TB NVMe performance-optimized Flash cards
  • 4 x 30.72 TB NVMe capacity-optimized Flash cards
Network
  • 2 x 100 Gb QSFP28 RoCE Fabric ports
  • 1 x 1 Gb Ethernet port (RJ45, management)
  • 1 x 1 Gb Ethernet port (RJ45, ILOM)

1Compared to X9M Extreme Flash Storage Servers
2Exadata X7 – X9M Extreme Flash Storage Servers allocated 2.32 TB to Flash Cache of the total 51.2 TB raw flash capacity per server
3Elastic configuration with 2x Exadata X11M Database Servers and 17x Exadata X11M High Capacity or 17x Exadata X11M Extreme Flash Storage Servers

High Capacity (HC) Storage Server

This server includes twelve 22 TB disk drives with 264 TB total raw disk capacity. It also has four 6.8 TB performance-optimized flash drives for a total raw capacity of 27.2 TB of Exadata Smart Flash Cache and 1.25 TB of Exadata RDMA Memory (XRMEM) in front of flash to boost performance further.

Flash in the HC Storage Server can be used directly as flash disks but is almost always configured as a flash cache (Exadata Smart Flash Cache) in front of disk storage and behind the Exadata RDMA Memory. Exadata Smart Flash Cache is used with the XRMEM Data Accelerator to automatically cache frequently accessed data while keeping infrequently accessed data on disk, delivering the high I/O rates and fast response times of flash with the large capacity and low cost of disk. Exadata uniquely understands database workloads and knows when to avoid caching data that negatively affects overall performance. For example, if large write I/Os caused by backups or large table scans are likely to disrupt higher-priority OLTP or scan operations, those large I/Os will bypass the flash cache and go straight to disk. Otherwise, Exadata System Software will utilize additional spare flash capacity and I/O bandwidth to optimize performance by caching these I/Os. Administrators can also manually provide SQL directives to ensure that specific tables, indexes, or partitions are preferentially retained in the flash cache.

It is common for hit rates in the Exadata Smart Flash Cache to be over 95%, or even 99% in real-world database workloads, yielding an effective flash capacity many times larger than the physical flash.

Exadata Smart Flash Cache also caches database block writes using Exadata Write-Back Flash Cache technology. Write caching eliminates disk bottlenecks in large-scale OLTP and batch workloads. The flash write capacity of a single Exadata Database Machine X11M rack reaches 13 Million 8K flash write I/O operations per second (IOPS)1. The Exadata Write-Back Flash Cache is transparent, persistent, and fully redundant, with performance comparable to dozens of enterprise disk arrays with thousands of disk drives.

The automatic data tiering between memory, flash, and disk in Exadata provides tremendous advantages over other flash-based solutions. Many storage vendors have developed flash-only arrays to achieve higher performance than traditional disk-based arrays. However, they cannot match the cost advantages of Exadata’s smart data tiering between disk and flash. Traditional flash arrays lack Exadata’s unique database-aware storage optimization. In addition, generic data deduplication provided by some flash arrays may be effective for workloads such as Virtual Desktop Infrastructure environments but are not for databases. In addition to utilizing its integrated and optimized hardware architecture, Exadata delivers superior performance by offloading data-intensive processing to unique algorithms in storage which have been specifically optimized for Oracle Database.

Exadata X11M High Capacity Storage Server Specifications

CPU
  • 2 x 32-core AMD EPYC™ 9J15 processors 2.95 GHz (up to 4.4 GHz)
Memory
  • 256 GB
  • 1.25 TB Exadata RDMA Memory (XRMEM)
Flash
  • 4 x 6.8 TB NVMe PCIe4.0 performance-optimized Flash cards
Disk
  • 12 x 22 TB 7,200 RPM disks
Network
  • 2 x 100 Gb QSFP28 RoCE Fabric ports
  • 1 x 1 Gb Ethernet port (RJ45, management)
  • 1 x 1 Gb Ethernet port (RJ45, ILOM)

1Elastic configuration with 6x Exadata X11M Database Servers and 13x Exadata X11M Extreme Flash Storage Servers or 13x Exadata X11M High Capacity Storage Servers

High Capacity-Z (HC-Z) Storage Server

As an entry point into the Exadata ecosystem, or for the smaller workloads, Exadata High Capacity-Z Storage Servers contain less memory, flash, and hard drive capacity of the High Capacity Storage Server.

Exadata X11M High Capacity-Z Storage Server Specifications

CPU
  • 1 x 32-core AMD EPYC™ 9J15 processor 2.95 GHz (up to 4.4 GHz)
Memory
  • 192 GB
  • 576 GB Exadata RDMA Memory (XRMEM)
Flash
  • 2 x 6.8 TB NVMe performance-optimized Flash cards
Disk
  • 6 x 22 TB 7,200 RPM disks
Network
  • 2 x 100 Gb QSFP28 RoCE Fabric ports
  • 1 x 1 Gb Ethernet port (RJ45, management)
  • 1 x 1 Gb Ethernet port (RJ45, ILOM)