Enterprise, Hardware

Titan Comes to Life: 46 Million Nvidia CUDA Cores, 300,000 AMD x86 Cores

The world of supercomputers is moving at incredibly high pace, and systems that once represented the pinnacle of computing are being replaced by 10-20x more powerful systems. Over the next couple of days, we will see the unveil of quite a few new supercomputers on the Top500 list, given the run up to the SC 12 conference in Salt Lake City, UT.

Meet the Titan - successor to Jaguar

We received press releases from AMD, Nvidia and Oak Ridge National Laboratories, all citing details of this successor to Jaguar supercomputer. Launched in 2005, Jaguar started its life as a 25 TFLOPS system (Cray XT3, single-core AMD Opteron with IBM Cell processors). Further upgrades brought the compute power to massive 1.75 PFLOPS, thanks to combining the Cray XT4 (quad-core AMD Opteron "Budapest") with XT5 (sexa-core AMD Opteron "Istanbul").

Old slide from ORNL shows the goals for Titan supercomputer
A 2010 slide from ORNL shows the goals for Titan supercomputer

The next step was naturally, moving from quad- and sexa-core processor to a 16-core Bulldozer and Tesla K20 (GK110 GPUs), bringing the compute power from 1.75 TFLOPS to 20PFLOPS in the same space. While the previous Jaguar setup consisted out of 18,688 compute nodes equipped with a single CPU, the new Cray XK7 system is consisted out of 16-core AMD Opteron CPU and Nvidia Tesla K20 compute card.

The Jaguar is thus renamed into Titan, and the sheer numbers are quite impressive:

  • 46,645,248 CUDA Cores (yes, that’s 46 million)
  • 299,008 x86 cores
  • 91.25 TB ECC GDDR5 memory
  • 584 TB Registered ECC DDR3 memory
  • Each x86 core has 2GB of memory

This system bucks the trend of heterogeneous supercomputers which mix CPU with special accelerated hardware, that being GPGPU (Chinese used Radeon HD 4870 X2, Nvidia Tesla series) or Xeon Phi (ex-Larrabee).

The system will be used in following projects by Oak Ridge National Labs and U.S. Department of Energy (DoE):

Materials Science

The magnetic properties of materials hold the key to major advances in technology. The application WL- LSMS provides a nanoscale analysis of important materials such as steels, iron-nickel alloys and advanced permanent magnets that will help drive future electric motors and generators. Titan will allow researchers to improve the calculations of a material’s magnetic states as they vary by temperature.


The S3D application models the underlying turbulent combustion of fuels in an internal combustion engine. This line of research is critical to the American energy economy, given that three-quarters of the fossil fuel used in the United States goes to powering cars and trucks, which produce one-quarter of the country’s greenhouse gases.

Titan will allow researchers to model large-molecule hydrocarbon fuels such as the gasoline surrogate isooctane; commercially important oxygenated alcohols such as ethanol and butanol; and biofuel surrogates that blend methyl butanoate, methyl decanoate and n-heptane.

Nuclear Energy

Nuclear researchers use the Denovo application to, among other things, model the behavior of neutrons in a nuclear power reactor. America’s aging nuclear power plants provide about a fifth of the country’s electricity, and Denovo will help them extend their operating lives while ensuring safety. Titan will allow Denovo to simulate a fuel rod through one round of use in a reactor core in 13 hours; this job took

60 hours on the Jaguar system.

Climate Change

The Community Atmosphere Model?Spectral Element simulates long-term global climate. Improved atmospheric modeling under Titan will help researchers better understand future air quality as well as the effect of particles suspended in the air.

The system is currently operational. Exact performance figures, as well as its positioning on the Top500 list will be revealed at the SC’12 Conference in less than two weeks time.