6 Visualizing data
6.1 Why visualize?
In social sciences, we are often interested in exploring or testing a relation between two variables: Does poverty lead to lower health or life expectancy? Does social media use influence mental health? Do populist politicians use more polarizing speech? Very often, the focus for answering such questions is a statistical measure or test such as correlation, t-test, or regression. Such a test would tell us whether there is a (linear) relationship between the two variables.
In 1973, Francis Anscombe published a paper called Graphs in Statistical Analysis in which he famously shows why very often this statistical test doesn’t show the whole story.
Suppose you have two variables (e.g. poverty and life expectancy), and you find a very strong and highly significant correlation (\(r=0.82, p<.01\)). What would you conclude about the relationship between those variables?
Now, run the code below and think about your conclusion. How do graphs and correlation contribute to the understanding of the relation?
(Don’t worry about the specific code in the example - the ggplot code will be explained in detail in this chapter and in Chapter 9. And if you’re brave, feel free to peek into the data preparation box!)
Don’t worry about the data preparation below – but if you’re curious, the recode_values was explained in Section 5.2, group_by |> summarize was explained in Section 5.3, and the pivot_longer code will be explained in Section 7.2
These graphs raise a number of intriguing questions. How do you interpret the relation between the variables as expressed in the graphs? All datasets have the same correlation strength and significance, why do they have such different shapes? What does that say about the difference between graphs and statistical tests of relationships?
This example powerfully showcases the need to explore data as well as computing statistical values, and this is in fact one of the two big use cases of data visualization (the other being telling the story of your analysis to readers).
In this case, (a) is probably what you expected the data to look like: a cloud of points dispersed randomly around the blue trendline; (b) shows a clearly non-linear relationship: as x increases, y first goes up, but then goes down towards the end; (c) shows how a single outlier can affect an otherwise very regular linear pattern; and (d) shows how a single outlier can create a (spurious) relationship that is not present in the other points at all. Crucially, the data behind all four plots showed the same underlying linear relationship, but the plots clearly show a completely different story in all four cases, and point at a modelling issue in all cases except the first.
Now that you are convinced of the need for visualization to explore your data (and tell your story to a reader), let’s explore how you can do this using ggplot!
This chapter will introduce most of the technical details of ggplot needed to make visualizations, but does not go as deeply into some aspects of visualization as you might want. To learn more, check these fantastic and freely available sources:
- Data Visualization by Kieran Healy. This is a freely available book which goes much more deeply into the why of visualizations. What is a good visualizations? What works and what doesn’t? What makes this such a fantastic resource is that it gives many social scientific examples and provides the full ggplot code for all examples.
- From data to viz is a website that is a great overview of the different types
of visualization, with a nice guide on when to use which chart type and what the strengths and possible drawbacks of each are, with links to examples with R code provided - The R Graph Gallery is a fantastic collection of hundreds of graphs, from simple bar charts to maps, flow charts and dendrograms. Each chart has the full ggplot code available, so it’s a great source of inspiration as well as example code.
- The ggplot2 book by tidyverse creator Hadley Wickham has a great explanation of how to use ggplot, but is especially useful as a resource for understanding the design and inner workings of ggplot better.
6.2 Scatter plots with geom_point
Let’s take a step back and work towards understanding ggplot syntax better. This can be a bit of a learning curve, as ggplot uses a layer-based syntax that’s a bit different from the data cleaning functions you’ve seen so far. However, it is extremely well designed and you will find that the better you understand this design, the easier it becomes to read and write your own visualization code.
Let’s get started with a minimal graph based on the gapminder dataset used earlier.
As you can see, this creates a scatter plot with fertility on the vertical (y) axis, and infant mortality on the horizontal axis.
Let’s take a closer look at the ggplot syntax. It consists of the three components which you will see in every ggplot call:
ggplot(data=gap_2010)starts a new plot defining thegap_2010data frame as its data sourcemapping=aes(x=infant_mortality, y=fertility)creates an aesthetic mapping, which links ‘aesthetic’ elements of the plot (i.e., how things appear) with columns in the data. In this case, theinfant_mortalitycolumn is mapped to the x aesthetic, and thefertilitycolumn is mapped to the y aesthetic.- Finally,
geom_point()defines a geometric layer to be added to the plot, which creates the actual visual representation of data rows, in this case as points.
These three elements form the core of any ggplot visualization: a data set, which is represented by a geometric layer, and a (‘aesthetic’) mapping between elements of the data source and elements of the geometry.
As you can see, the geometric layer is added to the plot using the plus (+) operator.
In fact, this is the general pattern of ggplots: a single ggplot() function which creates the empty canvas, to which any number of elements are added.
Adding more layers: geom_smooth
As an example of building a plot by adding layers, the code below adds a linear regression line to the graph by adding geom_smooth to the existing plot:
How do you interpret the blue line and grey shaded areas that were added to the plot?
What do you think lm stands for? What happens if you remove it or replace it with loess? For which cases would either smoothing method be more appropriate?
Finally, What happens if you use geom_smooth(method="lm", formula=y ~ poly(x,2))?
Exercise: Effect of GDP on Life Expectancy
The code below shows the relation between infant mortality and fertility in 2010. Change it to show the relation between GDP per capita (gdp_percap) on the x axis and life expectancy (life_expectancy) on the y axis instead, keeping the trend line.
Change both the x and y arguments inside aes(): use x = gdp_percap and y = life_expectancy.
Does the linear trend line fit the cloud of points well?
How does that change your interpretation of the relation between GDP and life expectancy?
For what type of countries does extra income have the largest effect of life expectancy?
What would a better way be to visualize this relation?
6.3 Aesthetic mappings
The section above introduced the basics of ggplot: a data source, a geometric layer, and a aesthetic mapping between columns in the data and visual elements of the geometric layer.
Before we introduce different geoms or other elements that can be added to a plot, let’s have a look at some other aesthetic elements that can be used:
What exactly does adding the color aesthetic do? What happens when you replace it by color=life_expectancy? How do the scale and legend change, and why?
As an added challenge, try adding size=population or size=gdp. Does the result make sense to you? Does it improve the story, or mostly clutter the results?
The code above shows two very important considerations. Technically, you can see that ggplot inspects your data, and gives a sensible default given the column type and values range. For color=continent, it sees that it is a nominal variable and creates a specific color for each value. With color=life_expectancy, it sees that it is a numeric variable and creates a scale running from the lowest to the highest value. Similarly, it maps the size such that the smallest point is still visible, and the largest point is not too large. GGplot generally creates a sensible default scale based on your data, but as we will see below you can customize this as much as you want.
Substantively, choosing the geom and aesthetics is the most important part of good data visualization. What columns of your data are important, and how can you represent these in an informative (and hopefully pretty) way? After adding the size mapping, we used four different variables for x, y, color and size. We could add more mappings (such as shape or alpha), but you will often find that adding more mappings also makes graphs harder to read. Visualization is often the art of choosing between showing as much of your data as needed, while keeping the figure easy to interpret. Finding the balance between these takes practice, and also very much depends on the goal and audience of your visualization: for data exploration for yourself or colleagues familiar with the data you can choose a more complex visualization than for a graph aimed at telling a story to a broader audience.
You’ve now seen x, y, color, and size. These are the aesthetics you’ll reach for most often:
| Aesthetic | Used for |
|---|---|
x, y |
position on the axes |
color |
outline/point/line color, mapped to a variable |
fill |
interior color of an area, mapped to a variable |
size |
point size or line width |
shape |
point symbol (circle, triangle, …) |
alpha |
transparency, useful for overlapping points |
linetype |
solid, dashed, dotted, … |
label |
text drawn at a position |
group |
which rows belong to the same line/shape |
Most of these aesthetics can be used in almost any plot type. linetype is the clearest exception: it only does something when a plot actually draws lines. Not every geom uses every aesthetic, and some aesthetics are required rather than optional (e.g. geom_point needs x and y, but color is optional). Each geom’s help page (e.g. ?geom_point) lists exactly which aesthetics it understands and which ones are required.
Also note the distinction between color (an outline or line) and fill (an interior): for default scatter plots, only color normally matters, while for e.g. bar plots you can change both the outline and the fill color of the bars.
Geom-specific mappings
Up till now, we’ve set the aesthetic mapping in the ggplot() function. You can also set mappings in the geom function; the following three blocks of code produce the exact same plot:
# Mapping in ggplot:
ggplot(data=gap_2010, mapping=aes(x=infant_mortality, y=fertility, color=continent)) +
geom_point()
# Mapping in geom:
ggplot(data=gap_2010) +
geom_point(mapping=aes(x=infant_mortality, y=fertility, color=continent))
# Mappings in both plot and geom:
ggplot(data=gap_2010, mapping=aes(x=infant_mortality, y=fertility)) +
geom_point(mapping=aes(color=continent))The distinction is that mappings defined in ggplot apply to the whole plot, while mappings defined in a geom only apply to that geom.
Of course, this only becomes relevant when we have more than one geom. In the example below, we add a trendline again using geom_smooth, but set color aesthetic only on the geom_point
As you can see, the points are colored by continent, but the trendline is still blue.
Let’s see how this changes when we move the color mapping back to the ggplot:
Now the trend line is also colored by continent. In fact, this creates a separate trend line per continent.
In all the examples above, we created mappings between data columns and aesthetic elements of the plot. For example, geom_point(aes(color=continent)) created a mapping between the color and the continent column.
In addition to mappings, you can also set an aesthetic to a constant: geom_point(color="blue") will color all points blue.
If you are not careful, this can lead to unexpected results. What do you think the code below will produce?
The reason for this unexpected behaviour is that ggplot treats “blue” not as a literal constant for the color, but as a nominal value to be used in a mapping – and just as with the continent colors above, red is the first color in the default palette for plotting nominal values.
6.4 Bar charts with geom_col
A second common chart type is the bar chart. Where scatter plots (geom_point) are used for exploring individual data points, bar charts are generally used for comparing values for different cases or categories.
For example, we could create a bar chart comparing life expectancy for South-East Asian countries using geom_col:
geom_col vs. geom_bar
The code above uses geom_col, but there is also a geom called geom_bar. The difference is that by default geom_bar does not use the y aesthetic, but instead plots the number of rows in each bar. So, the following code will produce the same plot twice:
Although it would seem that the second example is the simpler one, I normally prefer the first option (calculate before you plot) for two reasons. First, it makes it more explicit what is happening, and allows you to inspect the calculated countries_per_continent values more easily. Second, it is much more flexible, as the calculation can contain any of the data transformation operations explained in the previous chapter, as will be showcased by the grouped and stacked examples below.
Making it beautiful
The graph above works, but it’s not the most beautiful. Let’s make a nicer version:
Looking at the code above, three things were changed apart from adding a theme:
- Using
labs, labels were set for the main title and the axes. The y axis label was set to"", removing the (redundant) label. Good labels are one of the most important things in data visualization: a reader should be able to instantly understand the visualization, making it essential to label the variables, values, source, and anything else. - The
xandyaxes where flipped, changing it into a horizontal bar chart. This is often a good solution to make a bar chart more readable, especially if category labels are long. fct_reorderwas used to reorder the countries by life expectancy. By default, R orders alphabetically, which very often is not the most logical ordering to tell a story.
You may have wondered why the fct_reorder function started with fct. The reason for this is somewhat technical, but it will help you understand why ggplot does what it does (but feel free to skip this block for now!)
For ggplot, text columns are treated as nominal variables, i.e. groups. Such groups are best represented by factor columns, which have a fixed set of possible values called the factor levels. Factor levels have an ordering, and ggplot uses that internal ordering for plotting.
When you use a normal text column in ggplot, it silently converts it into a factor, and the default ordering of factor levels is alphabetically. So, ggplot orders groups by their alphabetical ordering by default.
To change this, you can use functions like fct_reorder either in the data preparation step, or directly in the aes function. Some other useful functions are fct_infreq to use the frequency of cases; fct_relevel to manually specify the ordering; fct_rev to reverse and ordering; and fct_lump to turn the least frequent categories into ‘other’ These functions all come from the forcats package,
which is of course purely an anagram of factors.
Grouped bar charts
It’s also possible to group bars within a category. For example, we could take the total GDP per continent per decade:
Some important things to note in the code above:
- Before the actual
ggplotcall, we first calculate the gdp per decade usingmutate(see Section 5.1) andgroup_by |> summarize(see Section 5.3). - To calculate the decade, we divide year by 10 (so 1976 becomes 197.6), then round down using
floor(so it becomes 197) and then multiply by 10 to add the zero back. - GDP has missing values (especially at the start of the series). We ignore this by adding
na.rm=TRUE(remove NAs), but of course this could skew the figure as it essentially assumes the GDP for these data points is zero. Try removing thena.rm=TRUE(or setting it toFALSE) and see what happens! - In the aesthetic mapping (
aes), we perform two light calculations:- gdp is divided by
1e12(1 with 12 zeroes) to render gdp in trillions; and - decade is converted to a factor. This latter operation matters because ggplot treats columns based on their type: if decade is a number, it would apply a gradient fill rather than a separate bar for each decade. Try removing the
as.factorand see what happens!
(note that we do these calculations in the plot since they don’t really calculate values, but mostly affect their appearance)
- gdp is divided by
- We use
position="dodge"to tell ggplot to put the bars next to each other (they ‘dodge’ each other) rather than stacking them. - We use
fill=NULLwithin thelabscall to set the label for the fill legend - in this case to remove it.
You might also have noticed that the 2010s total is actually lower than the 2000s. Of course, the world didn’t suddenly get much poorer.
- What do you think causes the 2010s values to be lower than the 2000s?
- How can you confirm that this is the case?
- What are good ways to fix this?
Finally, can you think of a better way to deal with the imbalanced NAs that we noted above?
These questions showcases an important skill in data analysis: noticing that something is odd, investigating why this is the case, and then dealing with it.
Stacked bar charts
As a final example, let’s see how the share of GDP per continent changed over time by using a stacked bar chart:
As you can see, this code is mostly the same as the plot above – we swapped the x and fill aesthetics and removed the position argument. By default, ggplot will stack bars if no position is given.
6.5 Line charts with geom_line
The chart above used bar charts to show change over time. In most cases, however, line charts are better suited to visualize temporal data.
The plot below shows fertility per continent over time:
As before, we first calculate the values to plot by grouping by continent and year and computing the mean fertility, ignoring missing values. Then, we use a ggplot as usual, with a geom_line and year and mean fertility on the x and y axes. Note that we use color for the continents rather than fill, since it’s a line rather than a bar or area.
How are data points grouped into lines?
The code above worked directly because there was only one data point per year and continent. But what if we did not aggregate before drawing the plot?
The plot above is certainly artistic, but not exactly informative.
- Looking at the graph, what do you think is happening? What do the lines represent? Why are there vertical lines for each year?
- What happens if you add
group=countryto theaesfunction above? Why?
As you have noticed, there is a vertical line for each year and continent. This is a common gotcha with line graphs, caused by how ggplot groups points into lines. The way it works is that by default it creates a group for each combination of aesthetics that are mapped to a categorical variable. So, it creates a group for each color, or each combination of color and line type, assuming they are all categorical variables (factors). Then, for each group it orders the points by x value, and draws a line connecting these points.
In the graph above, there are many points for each year on each continent, so it draws a vertical line connecting all these points, and then a diagonal line from the last country in one year to the first country in the next year.
The important lesson here is that if you see a line chart with these telltale vertical lines, you can immediately conclude that there is a problem with the groups not uniquely identifying the lines.
In most cases, the solution is to make sure that the data is grouped by x value and color, and also to make sure color is a factor and not a number (as it only creates groups for categorical variables).
However, as indicated in the challenge block above, you can also manually specify the grouping variable. When you added it to the plot, it created a correct line plot, but it was mostly an uninformative spaghetti rather than an informative chart.
The next example shows that we can use this to create an informative plot showing both the average value per continent and the trajectories of the individual countries:
This actually creates a graph that is quite rich in detail, but still quite readable and shows both the spread of individual countries within a continent. For example, you can see that most European countries started low and declined at about the same pace, and that African countries are a fairly wide band rising until the early 1970s and then slowly declining. It also shows countries going against the overall trend, such as the two Asian countries where fertility was still increasing until the 1980s. These details give a rich impression of both trendline and individual trajectories, and can show places where it might be worth a closer look at individual countries rather than just looking at the group mean.
Looking at the code, note especially the use of two geom_lines: first plotting the individual country lines with y=fertility and group=country. This explicitly creates a group per country, drawing 185 individual lines. Next, we plot the overall solid line using data=fertility_per_year (calculated earlier) and y=mean_fertility. Since we don’t add an explicit grouping here, ggplot (this time correctly) creates 5 groups from the color aesthetic.
Some things to note:
- We plot the individual country lines first, and then the continent means: ggplot draws the lines in order, and we want the continent means to be drawn over the country lines.
- The
gapdata is specified in the ggplot call, and is used by default in the first geom. However, it is overridden with the explicitdata=fertility_per_yearin the second geom. - Similarly, the
xandcoloraesthetics are given in the ggplot function, so they are shared by both line geoms, whileyandgroupare specified per geom. alpha=.1andlinewidth=.75are placed as constants outside theaesfunction, to set the transparency and line width for all lines in these respective geoms.- Aesthetics are evaluated based on the data for the specified layer. So,
color=continentrefers to the continent column of thegapdata for the first geom, and to the column with the same name in thefertility_per_yeardata frame in the second geom.
Dealing with missing rows using complete
A final gotcha for line graphs is how ggplot deals with missing rows. This is especially important for count data, where a lack of data often implies zero, rather than actual missing data.
Let’s make a graph of how many countries have a fertility below the replacement level of 2.1 children per woman:
At first sight, this graph looks correct and understandable: over time, more and more countries’ fertility drops below replacement, with European countries doing so from the start of the dataset, while countries in Africa and Oceania stay mostly above replacement.
What is strange, however, is that Africa does not seem to have any data before 1997. Looking at the below_replacement data set created above, can you see what the problem is?
Looking at the filter command, it keeps only rows with fertility below 2.1, so if a continent has zero countries that match that condition, no rows are left to summarize, leading to a missing data point that should have been a zero.
The easiest way to remedy this is to use the complete function, which makes sure every combination of the mentioned variables exist:
Can you spot the differences between the two graphs?
If you look closely, you can see the second graph correctly starts at zero for all continents except Europe and Asia; and you can also see that Asia in fact dropped to zero countries below replacement in 1970 as fertility in Japan briefly passed 2.1. So, the first graph (with the missing rows) wasn’t just incomplete, it was actually incorrect for Asia in 1970.
The reason for this is that if a row is missing, ggplot simply plots to the next point, drawing a straight line between the known points. This is often incorrect as it seemingly implies a data point where none exists. Note that if the row would exist but with an explicit NA, it would interrupt the line at that point, which would be correct if the value for that year is actually unknown.
Looking at the code, there are some things to note. First, in complete we first provide the year and continent columns for which to complete the rows, and then use the list(n=0) to indicate which value columns should be added. Second, since the below_replacement data was still grouped by year, we need to ungroup before completing. Finally, complete in this form only completes years (and continents) that are actually in the data set. If there are years that are left out for all continents, you can fix this by using complete(year=full_seq(year, 1), continent) instead, which creates the full sequence for the year variable between the observed extremes.
By default, ggplot shows the plot in the viewer pane (or on this website), which is useful for your own exploration. However, for making production-level graphics (for publications or reports), you want to save and use them in a good way.
On this interactive website, your only real option is to use the ‘Save image as’ in the context (right-click) menu. When using R on your own computer you can also directly save the image from the preview pane. Since the output then depends on the size and ratio of your pane, this is not a very good option to create production-ready plots in a reproducible way.
The easiest option is to create the whole report using Quarto. Quarto is a tool for writing data-driven books and reports and R and ggplot are integrated natively into it (in fact, this book is written completely using quarto). You can render the report as a PDF or Word document and the pictures will be included automatically. In this case, you can use execution options to control e.g. figure size in the output.
A second good option is to save the plot automatically from code using the ggsave function. This can save a specific plot, or save the last plot that was displayed, and allows you to specify output format, size, etc. This is especially useful for published scientific articles, as publishers generally want separate files for each image.
# Save the last plot
ggplot(anscombe, aes(x=x1, y=y1)) + geom_point()
ggsave("figure1.png", width = 1000, height = 1000, units = "px")
# Assign and save a specific plot
fig2 <- ggplot(anscombe, aes(x=x2, y=y2)) + geom_point()
ggsave("figure2.pdf", fig2, width = 20, height = 10, units = "cm", dpi=300)When saving plots to a file like this, it’s important to check the actual outputhe file: especially fonts can scale differently than you might expect, so the best way to create quality plots is to first settle on a format and size and then tweak font size, margins etc until you are satisfied, checking the rendered file after each change.
Exercise: Which countries go against the trend?
Part (a): which countries went against the trend?
Earlier in this section we noticed that some Asian countries went against the overall trend, with fertility still increasing well after everyone else had started to decline. The spaghetti plot was far too crowded to tell which ones, so let’s ask the data instead.
Fill in the blanks below to find, for each Asian country, the year in which its fertility was highest, and then keep only the countries that peaked after 1980.
- You want the highest fertility per country, so that is what you should group by (see Section 5.3.6).
slice_maxkeeps the rows with the highest value of the column you give it inorder_by. Here, a country’s peak is the year in which its fertility was highest.n=1keeps a single row per country, andwith_ties=FALSEmakes sure you get exactly one row even if two years are tied for the highest value.
Part (b): show them
Now that you know which countries to talk about, you can put them back in the graph. The trick is to draw all Asian countries in grey first, and then draw only the countries you are interested in on top of that in colour.
The peaks data frame from part (a) is already available: the second line uses peaks$country to select the country names, and then uses it to filter the full history of those five countries.
Fill in the blanks to:
- make sure the grey background lines are drawn per country rather than all connected;
- tell the second
geom_lineto use theoutliersdata rather than all of Asia.
- The grey lines suffer from exactly the problem described above: nothing is mapped to colour, so ggplot puts every Asian country in a single group and connects them all into one zigzag.
- The second
geom_lineshould draw theoutliersdata frame created on the second line, overriding the data given in theggplotcall for that layer only.
This exercise represented a typical data science cycle: you plot all data (the ‘spaghetti plot’ made earlier), and notice something odd. Then, you use data wrangling commands to investigate which cases displayed the odd behaviour (part A of this exercise), and finally used that to plot those specific cases to understand the story better (part B of this exercise).
Yemen and Oman are the two you could actually see in the earlier plot, and both show the same shape: fertility rises for two decades before it starts to fall. Why would fertility go up before it goes down?
Timor-Leste is stranger still: it drops until the late 1970s and only then starts climbing. Look up what happened there in 1975, and what happened in 1999.
Finally, two questions about the method rather than the substance:
- We picked “peaked after 1980” as our criterion. Try 1970 and 1985 instead: the list grows to nine countries and shrinks to one. Now try 1975 – you get exactly the same five as with 1980. What does that tell you about how much our five depend on where we drew the line?
- China’s fertility rose further than any other Asian country (from 4.0 children per woman in 1960 to 7.4 in 1963), but it does not show up in our list at all. Add
"China"to theoutliersand see if you can work out why our criterion missed it.
6.6 Other chart types
This chapter introduced the main chart types for data visualization: the scatter plot, bar plot, and line graph.
There are many more exciting ways to visualize data, including charts focused on the distribution of data such as box plots, histograms, violin plots and ridgeline graphs; heat maps as a way to visually represent cross-tabulations; choropleth maps based on geographical data; or Cleveland plots showing differences between two groups of rows.
As pointed out at the top of the chapter, two wonderful resources for learning more about these are the R Graph Gallery and Data-to-Viz.
Note that this chapter explicitly does not teach pie charts. Pie charts make it very hard to compare groups and in almost all cases a bar chart is the better choice. However, if you must make a pie chart you certainly can: in fact, it’s just a bar chart with coord_polar() added to create a circular plot.