The Mystery of Dark Energy

We are told that we live in a universe filled with a mysterious entity called dark energy. In this article we try to explain what scientists mean by dark energy, and how we infer it exists.

It was the year 1920. At a meeting of the National Academy of Sciences, USA, the leading astronomers of the day were having a debate on ‘The Scale of the Universe’. Their arguments were centred on cloudy-looking objects like the Great Andromeda Nebula (Latin nebula ~ little cloud), also known as M31 (refer Fig. 1a). According to one school of thought, nebulae were just clouds of gas and dust within the Milky Way. The other school of thought agreed with only part of this argument — there were indeed some gas clouds in the Milky Way. However, they argued that many nebulae were collections of stars that were too far away to be within our galaxy. 

Fig. 1a. The Andromeda Galaxy: a modern view. Credits: NASA/JPL-Caltech, Wikimedia Commons.

URL: License: CC-BY.

Fig. 1b. Immanuel Kant. Credits: A photograph ( of a painting by Johann Gottlieb Becker uploaded by Daube aus Böblingen, Wikimedia Commons. URL: License: CC-BY.

The latter was not a new idea. As far back as in 1775, the philosopher Immanuel Kant (refer Fig. 1b) had conjectured that nebulae, which he called ‘island universes’, were ‘distant’ objects. By 1925, detailed observations — especially those made by the American astronomer, Edwin Hubble — had settled the debate. Hubble started by successfully resolving individual stars in M31 using the Hooker Telescope at Mt. Wilson Observatory, California (refer Fig. 2). Soon, he and his colleagues were able to estimate the distance to these stars. This showed that not only was M31 too distant to be within the Milky Way; it was a galaxy in itself, containing billions of stars. This proved that the universe was much bigger than previously believed. Not surprisingly, the light emitted by M31 shows spectral lines of the various chemical elements seen in stars. 
Fig. 2a. Edwin Hubble. Credits: Johan Hagemeyer, Wikimedia Commons.
URL: https://commons. License: CC-BY.
Fig. 2b. The 100 inch Hooker Telescope at Mt. Wilson Observatory in Los Angeles County, California. Credits: Ken Spencer, Wikimedia Commons.
The universe is expanding
During 1916-1919, it was found that while some ‘nebulae’, such as Andromeda, showed a blue shift in their spectral lines, a majority exhibited a red shift. This was  particularly true of the more distant galaxies. The Doppler Effect (refer Box 1) allows us to relate the spectral shift to the motion of an object, and leads us to conclude that most galaxies are moving away from us. In the latter part of the 1920s, Hubble and his collaborators measured the red shifts of most galaxies known at the time. They found that the galaxies which were further away from us had higher red shifts, i.e. they were moving away (receding) from us at higher speeds. A plot published in  Hubble’s first scientific paper in 1929 showed that the speed of recession of a galaxy was proportional to its distance from us (refer Fig. 3). This is today known as  Hubble’s Law (refer Box 2).
Fig. 3. Hubble’s plot. Credits: Edwin Hubble, Proceedings of the National Academy of Sciences, vol. 15 no. 3, pp.168-173.
By the 1930s, as more data became available, Hubble’s Law was confirmed. Indeed, the redshift (the preferred spelling nowadays) of a galaxy is used as a measure of distance. Yet, Hubble and  his collaborators could not explain the reason for this phenomenon.
Today, we no longer believe in the ancient notion that human beings are privileged and occupy the centre of the universe. We recognise the Milky Way as being just one of the billions of galaxies in the universe. We also believe that the universe as a whole looks the same to all observers, wherever they are located — this deeply philosophical statement is called the Cosmological Principle. In other words, according to Hubble’s Law, intelligent beings in another galaxy would also observe other galaxies receding from them. This would mean that all the galaxies in the universe are receding from one another with speeds proportional to the distances between them. To give an analogy, think of a balloon with printed patterns on it. As we blow air into the balloon, it expands, and each printed bit moves away from all others (refer Fig. 6). While technically this analogy is not quite correct, it does help us visualise the physical meaning of Hubble’s Law — the universe as a whole is expanding.
Fig. 6. The printed bits on the surface of an expanding balloon move apart at a rate proportional to their distance.
Credits: Adapted from an illustration by Yuen Pui-ho (translation by Wong Ka-lei) on Hong-Kong Physics World.
Fig. 7. Albert Einstein. Credits: Photograph by Orren Jack Turner, Princeton, N.J.;
modified with Photoshop by PM_Poon and later by Dantadd; uploaded on Wikimedia Commons.
URL: https://commons. License: Public Domain.
The theoretical basis for Hubble’s Law came from the work of a German theoretical physicist, Albert Einstein (refer Fig. 7). In 1917, Einstein had solved the equations of his General Theory of Relativity to find a mathematical model for the structure of the universe (refer Box 3). The original theory had led to solutions indicating that everything in the universe changed with time. While this implied that the universe was either expanding or contracting, there was no observational evidence for either at the time. This led Einstein to add a new constant, called the cosmological constant, to his equations — which had the effect of making the universe static. This paper went on to become so influential that it is believed to mark the origins of modern cosmology as a science that studies the universe as a whole. However, years later, when Einstein heard of Hubble’s redshifts, he recognised it as evidence for the idea of an expanding universe. This led him to describe the introduction of the cosmological  constant as the ‘biggest blunder’ of his life.
The expansion is speeding up
In physical terms, the fact that the graph remains a straight line in Hubble’s plot implies that the expansion of the universe remains constant with time. However, the 1930s brought the realisation that the relationship between the distance of a galaxy and its speed of recession may be more complicated than that. The ‘actual’ shape of the graph would depend on the mathematical model of the universe. 
In most models of the universe, its rate of expansion decreases with time because the force of gravity pulls all matter together. Cosmologists even defined a deceleration parameter to measure the rate of this decrease. However, observations made over a period as long as the next fifty years were unsuccessful in establishing the value of this parameter. Was it zero — meaning that the Hubble plot would be an exact straight line? Was it positive as most theoretical arguments suggested? Or, was it perhaps negative? For years, students entering the field were told that observations were consistent with all three possibilities.
The year 1998 saw a major breakthrough. Two groups, independently working on exploding stars called supernovae (refer Box 4), arrived at a surprising conclusion: the deceleration parameter appeared to be negative. They observed that the distances of highredshift supernovae appeared to be systematically 10-15% greater than expected. This is possible if the universe had expanded more slowly in the past than it does today. Since light would then have to travel a longer distance to reach us, the supernovae would appear fainter. In other words, contrary to expectations, the expansion of the Universe appeared to be speeding up.
This was the biggest discovery to be made in cosmology in three decades. The American astronomers Saul Perlmutter, Brian Schmidt and Adam Riess were jointly awarded the Nobel Prize in Physics in 2011 for their contribution to it (refer Fig. 9). By then, several other observations, not related to supernovae, had confirmed the accelerated expansion of the Universe.
Fig. 9a. Saul Perlmutter. Credits: Berkeley Lab. URL: License: CC-BYNC-ND.
Fig. 9b. Brian Schmidt. Credits: Markus Pössel (User name: Mapos), Wikimedia Commons.
URL: https://commons. License: CC-BY-SA.
Fig. 9c. Adam Riess. Credits: Adam.riess, Wikimedia Commons.
The mystery of accelerated expansion
As mentioned before, any kind of matter is expected to slow down the rate of cosmic expansion. Thus, observations implying accelerated expansion are puzzling — how is this possible in a universe filled with ordinary matter and radiation? Even the presence of a large quantity of dark matter (refer: Throwing Light on Dark Matter, iwonder, Issue 3. URL: throwing-light-on-dark-matter.aspx) does not help explain this phenomenon.
Theorists hypothesised that the accelerated expansion was caused by the presence of a still unknown form of energy that pushes galaxies apart. In an analogy to dark matter (DM), the term ‘dark energy’ (DE) was coined for this ‘mysterious’ form of energy. This is, in fact, a misleading name. The only thing common between DM and DE is that neither can be seen with telescopes. Dark matter behaves like ordinary matter under the force of gravity — it clumps together to slow down the expansion of the universe. In contrast, dark energy is simply shorthand for ‘whatever is causing the expansion of the universe to speed up’. This, of course, doesn’t explain anything.
The nature of dark energy has been a subject of speculation for the last twenty years. One approach to this ‘problem’ assumes that Einstein’s theory of general relativity is correct, but the universe is filled with something that does not behave like matter. Among the many theoretical models that fit this category — collectively called DE Models, the most popular one at the moment is based on Einstein’s idea of a cosmological constant. According to this model, dark energy can be imagined to fill all the empty space in the universe — thus, called vacuum energy — at a density that remains constant with space and time. Using thermodynamics, it is easy to see that if an empty space or vacuum has energy, it necessarily has a negative pressure (refer Box 5). Thus, if the universe expanded slightly, the empty space would expand too. This would increase the amount of dark energy, which would in turn cause more expansion. This sounds weird, but offers the simplest explanation for the accelerated cosmic expansion (refer Fig. 10). Other DE models, in which dark energy is known as ‘quintessence’ or ‘phantom’, can be thought of as dynamic models of vacuum energy. While we will not describe these models in detail here, they hypothesize that the density of dark energy in the universe is not a constant; it varies with space and time.
Fig. 10. Energy distribution in the Universe according to Planck probe measurements, March 2013.
Credits: Adapted from an image by Szczureq, Wikimedia Commons.
Another approach is based on the possibility that Einstein’s description of gravity by general relativity may be incomplete. An alternate version may account for the accelerated cosmic expansion and do away with DE altogether. If this turns out to be true, the important question to consider is — would such a model be able to satisfy the other observational tests that Einstein’s theory does? While there are many alternate theories of gravity, none of them have seemed convincing enough yet.
A natural question that arises from these discussions is — how much energy must the empty space in the universe have in order to account for its accelerated expansion? Data from the Planck satellite shows that over 68% of the total energy of the Universe is contributed by dark energy. Ordinary matter — we ourselves, other life on earth, the Earth, the solar system, and all the stars in the visible parts of galaxies — makes up less than 5%.
In the future
Although observations of the universe seem to support the model based on the cosmological constant, this story is far from over. Our present understanding of elementary particles and the forces between them relies on a framework called quantum field theory. This framework has been used to calculate how much energy empty space should contain. The predicted value turns out to be several (~10120) times bigger than the value of the cosmological constant inferred from observations related to the accelerated cosmic expansion. Thus, the current challenge is to explain not only why vacuum energy exists, but why it has the observed value. This is an exciting field for further work, with the potential to improve observations and refine existing theories.

Amitabha Mukherjee retired as a Professor at the Department of Physics and Astrophysics, University of Delhi. Email:


19301 registered users
7712 resources