JuRoPA Supercomputer simulates galaxy formation in unprecedented detail over a huge part of the observable Universe
An international team of astrophysicists presents the world’s largest simulation of the universe and an accurate theoretical model for the growth of galaxies and supermassive black holes. This simulation, completed on the world’s most efficient Supercomputer Cluster at the Forschungszentrum Jülich, maps the development of the universe from a time shortly after the big bang, 13 billion years ago, to the present day.
This information herein is sourced from an article published in inSiDE • Vol. 8 No. 2 • Autumn 2010 entitled:
The Millennium-XXL Project: Simulating the Galaxy Population of dark Energy Universes
authored by:
• Volker Springel, Heidelberg Institute for Theoretical Studies and Heidelberg University, Germany
• Raul Angulo, Simon D.M. White, Max-Planck-Institute for Astrophysics, Garching, Germany
• Adrian Jenkins, Carlos S. Frenk, Carlton Baugh, Shaun Cole, Institute for Computational Cosmology, University of Durham, United Kingdom
Source: http://inside.hlrs.de/htm/Edition_02_10/article_06.html
Modern cosmology as encoded in the leading ΛCDM model confronts astronomers with two major puzzles. One is that the main matter component in today's Universe appears to be a yet undiscovered elementary particle whose contribution to the cosmic density is more than 5 times that of ordinary baryonic matter. This cold dark matter (CDM) interacts extremely weakly with regular atoms and photons, so that gravity alone has affected its distribution since very early times. The other is a mysterious dark energy force field, which dominates the energy content of today's Universe and has led to its accelerated expansion in recent times. In the standard model, this component is described in terms of Einstein's cosmological constant ('Λ'). Uncovering the nature of dark energy has become one of the most actively pursued goals of observational cosmology.
Simulations of the galaxy formation process are arguably the most powerful technique to accurately quantify and understand these effects. However, this is an extremely tough computational problem, because it requires ultra-large volume N-body simulations with sufficient mass resolution to identify the halos likely to host the galaxies seen in the surveys, and a realistic model to populate these halos with galaxies. Given the significant investments involved in the ongoing galaxy surveys, it is imperative to tackle these numerical challenges to ensure that accurate theoretical predictions become available both to help to quantify and understand the systematic effects, and to extract the maximum amount of information from the observational data.
The State of the Art
The N-body method for the collisionless dynamics of dark matter is a long established computational technique used to follow the growth of cosmic structure through gravitational instability. The Boltzmann-Vlasov system of equations is here discretized in terms of N fiducial simulation particles, whose motion is followed under their mutual gravitational forces in an expanding background space-time. While conceptually simple, calculating the long-range gravitational forces exactly represents an N2-problem, which quickly becomes prohibitively expensive for interesting problem sizes. However, it is fortunately sufficient to calculate the forces approximately, for which a variety of algorithms have been developed over the years.
This allowed the sizes of cosmological simulations to steadily increase since the early 1980s, roughly doubling the particle number every 17 months and hence providing progressively more faithful models for the real Universe. Such simulations have proven to be an indispensable tool to understand the low- and high-redshift Universe by comparing the predictions of CDM to observations, since these calculations are the only way to accurately calculate the outcome of non-linear cosmic structure formation.
A milestone in the development of cosmological simulations was set by the Millennium Run (MR), performed by theVirgo Consortium group in 2004. This simulation was the first, and for many years the only run with more than 1010 particles, exceeding the size of previous simulations by almost an order of magnitude. Its success was not only computational but most importantly scientific – more than 300 research articles in the fields of theoretical and observational cosmology have used the MR data-products since. The MR has an exquisite mass resolution and accuracy but, unfortunately, its volume is insufficient to get reliable statistics on large scales at the level needed for future surveys.
The team of researchers around Prof. Springel have therefore set out to perform a new ultra-large N-body simulation of the hierarchical clustering of dark matter, featuring a new strategy for dealing with the data volume, and combining it with semi-analytical modeling of galaxy formation, which allows a prediction of all the luminous properties of the galaxies that form in the simulation. This group of researchers designed the simulation project, dubbed Millennium-XXL (MXXL), to follow more than 303 billion particles (67203) in a cosmological box of size 4.2 Gpc across, resolving the cosmic web with an unprecedented combination of volume and resolution. While the particle mass of the MXXL is slightly worse than that of the MR, its resolution is sufficient to accurately measure dark matter merger histories for halos hosting luminous galaxies, within a volume more than 200 times larger than that of the MS. In this way the simulation can provide extremely accurate statistics of the large-scale structure of the Universe by resolving around 500 million galaxies at the present epoch, allowing for example highly detailed clustering studies based on galaxy or quasar samples selected in a variety of different ways. This comprehensive information is indispensable for the correct analysis of future observational datasets.
The computational Challenge
It was clear at the outset that performing a simulation with these characteristics poses significant challenges in terms of raw execution time, algorithm scalability, memory consumption and the disk space required for the output data. For example, simply storing the positions and velocities of the simulation particles in single precision, consumes of order 10 TB of RAM memory. This figure, of course, is greatly enlarged by the extra memory required by the complex data structures and algorithms employed in the simulation code for the force calculation, domain decomposition, and halo and subhalo mapping..
The JuRoPA cluster at the Jülich Supercomputing Centre (JSC) in Germany was identified as being particularly well suited to fulfilling these challenging computing requirements. Nevertheless, on the JuRoPA machine, the simulation demanded a partition of 1,536 nodes, each equipped with two quad-core X5570 processors and 24 GB of RAM, translating to 12,288 cores in total. This represents a substantial fraction (70%) of the entire machine. Single job execution on such a large machine partition is a relatively uncommon event. It therefore required substantial support on the part of the system administrators at JSC to ensure successful completion of the MXXL production calculation. In particular, severe problems with the memory usage of the simulation jobs were encountered initially, as the code required essentially all the available physical memory of the compute nodes, leaving very little room for memory buffers allocated by the MPI library or the parallel Lustre filesystem. Fortunately, with help from JSC and ParTec GmbH, the software vendor behind the ParaStation MPI software stack, these problems were successfully overcome.
The final production run of MXXL required more than 86 trillion force calculations and took in excess of 2.7 million CPU hours (˜300 years), corresponding to 9.3 days of runtime on 12,288 cores. However, a significant fraction (15%) of the overall run time was spent running the sophisticated ‘on-the-fly’ postprocessing software which included group mapping, substructure mapping and power spectrum calculations. Another 14% of runtime was consumed by I/O operations. The long-range force calculations based on 92163 sized FFTs consumed only about 3% of the overall wall time. The group of researchers note that the parallel "friends-of-friends" group mapping for the 303 billion particles at the final output time took just 470 seconds, which they think is a remarkable achievement.
First Results and Outlook
The Millennium XXL is by far the largest cosmological simulation ever performed and the first multi-hundred billion particle run. The scope of the simulation project has pushed the envelope of what is feasible on current world-class HPC resources, but the expected rich scientific return from the project makes it well worth the effort. At the same time, the successful scaling of the cosmological code to well beyond ten-thousand cores is an encouraging sign for computational research in cosmology, which will address yet more demanding problems on future HPC machines.
Prof. Springel explained that this Millennium XXL project was 30 times larger than the previous simulation with more than 30 TByte RAM and 100 TByte storage data used. The ParaStation MPI has successfully scaled to the enormous job size, with 3072 MPI tasks, employing 1536 nodes and 12288 cores, which is 3 times the size of normally allowed jobs at JSC of 512 nodes. The other crucial element for the successful run of Millennium XXL was the professional support also during off hours and on the weekends. So the results were achieved in less than 9,5 days total machine time, which would correspond to 311 years of computation on a single core.
Prof. Thomas Lippert, Director of the Jülich Supercomputing Centre, explained that "The Simulation on supercomputers has become key to resolving the true nature of dark matter, one of the most important scientific enigma of today. Virtual universes in the supercomputer will either allow us to validate the dark matter hypothesis or ultimately they might force us to accept modifications of the most basic laws of nature."
“I am very pleased that our contribution to the Millennium XXL run, the ParaStation MPI and our support quality enabled results, which could not be achieved on any other supercomputer in the world, namely to simulate the development of Galaxies over a time duration of 13 billion years, which was never done in such a quality before”, said Hugo R. Falter, Chief Operating Officer of ParTec Cluster Competence Center, GmbH.