Moscow University Supercomputing Center

Former Russian President Dmitry Medvedev and the President of Moscow State University Victor Sadovnichy at the Lomonosov supercomputer presentation

Moscow State University hosts a number of HPC systems. SKIF MSU «Chebyshev» cluster has been the most powerful one until recently. This 60 TFlops supercomputer was installed in 2008; and after deployment it very soon became clear that the demand for computing power far exceeded its capabilities. By 2009 a significant expansion of MSU’s supercomputing facilities became a necessity and MSU decided to acquire a new, much more powerful system enabling researches to expand computations and to perform more accurate simulations. It has become evident that by fostering discoveries and innovations in leading research centers of Russia, the new supercomputer would contribute to the growth of country’s overall competitiveness.

Robust price/performance, scalability, and fault tolerance were key requirements to the new system. “Lomonosov” supercomputer was delivered to MSU by T-Platforms company in 2009 with initial peak performance 420 TFlops. In 2011, following the trend in the supercomputer industry, “Lomonosov” was additionally supplemented with 777 compute nodes equipped with GPU based computing accelerators. A total peak performance of the computer system 1.37 Pflops, and Linpack performance – 674 Tflops. “Lomonosov” was ranked 13th in the June 2011 edition of Top500 list.

GPU part of “Lomonosov” supercomputer is based on the next generation of T-Platforms blade systems TB2-TL. The TB2-TL system is based on the newest TL-blade design. With 16 TL blades, it packs 32 Tesla X2070 GPUs and 32 Intel Xeon 5630 CPUs to deliver 17.8 TFlops of peak DP performance per single TB2 enclosure. With 6 TB2-TL systems installed into a 42U rack cabinet, total performance of 106.6 TFlops per rack is reached.

The primary compute nodes generating over 94% of the x86 part performance are based on T-Platforms T-Blade2 system. Using sixcore Intel Xeon X5670 Westmere processors, T-Blade2 brings up to 27 TFlops of computing power in a standard 42U rack. “Lomonosov” also contains a number of T-Blade 1.1 compute nodes with increased amount of RAM and local disk storage for memory-intensive applications. The 3rd type of compute nodes is based on T-Platforms PeakCell S platform using PowerXCell 8i processors.

In June 2011, “Lomonosov” was included in the Graph500 list. According to tests results, the system ranked third (positions were allocated depending on the workload), but showed the best performance among all systems in the list. During tests, a result of 43,471,500,000 TEPS (Traversed Edges Per Second) was obtained using 8192 cores/4096 nodes based on Intel Xeon 5570 processors. Lately the system was ranked 2nd in the November 2011 list edition with 103,251,000,000 TEPS using 32,768 cores / 4,096 nodes based on Intel Xeon 5570 processors.

In 2012, the supercomputer has been additionally equipped with 288 compute nodes with Intel® Xeon X5570/X5670 processors and GPU-based computing accelerators. Its total amount of memory has increased to 92 TB, and now computer consumes 2.6 MW. As a result of modernization, a peak performance of the computing system has been increased to 1.7 Pflops, and Linpack performance reached 872.5 Tflops.

“Lomonosov” supercomputer uses 40Gb/sec QDR Infiniband technology as a primary interconnect. The dedicated global barrier network of T-Blade2 allows fast synchronization of computing jobs running on separate nodes, while the global interrupt network significantly reduces the influence of OS jitter by synchronizing the process scheduling over the entire system.

“Lomonosov” exploits 3-level storage system: 500 TB of T-Platforms ReadyStorage SAN 7998 external storage with Lustre parallel file system, 300 TB of high reliable NAS storage and 1 PB tape library storage.

A very high degree of fault tolerance is a necessity for installations of such scale. To this end, redundancy of all critical subsystems and components was implemented – from cooling fans and power supplies on compute nodes to the entire engineering infrastructure. To ensure even greater reliability, primary compute nodes have neither hard discs nor cables inside the chassis, and contain a number of special hardware features such as fault-tolerant memory module slots.

«LOMONOSOV» KEY FEATURES

1.7 PFlops «Lomonosov» supercomputer