8 Advanced Visualization

9.1 Scales: the link between values and aesthetics

Substantively, this looks about right: Democrats and Republicans both hover around 50%, with third parties playing a very marginal role (with the 1992 bid by Ross Perot as the most notable exception).

However, anyone who follows US politics will probably be bothered by the colors: Republicans are traditionally depicted with red, and democrat with blue, which is reversed in the graph.

Discrete scales

The reason for this is the way ggplot assigns colors by default: for a categorical variable (like party), it assigns colors alphabetically, starting with red for the first category, in this case Democrats.

To override the default way values are turned into colors (or other aesthetics), you use a scale function:

In this plot, we used scale_color_manual, creating a manual scale for the color aesthetic. The values= argument within this scale should then be a named vector which assigns a color name (or hex pattern such as #ff0000) to each value.

Another common option is to stick to an automatic scale, but use a different palette. Try replacing the scale function in the above example with scale_color_brewer(palette="Accent") – this uses the ‘Accent’ palette in the ColorBrewer package. Another good option can be the Viridis scale which is designed to be colorblind safe and also work when printed greyscale: scale_color_viridis_d() (d for discrete, i.e. categorical data)

Aesthetics, Geoms and Scales

In Chapter 6 we introduced the aesthetic mapping created by the aes() function, and explained how it linked columns in your data (e.g. year or party) to aesthetic elements of the graph (such as the x position or color or a point or line). In other words, it links values from your data to the geometrical elements that are drawn on the screen.

Scales can be seen as the bridge between the value and the aesthetic, and define how specific values are displayed. All scales have (mostly sensible) defaults based on your data: if you supply a color mapping, ggplot will inspect the values, determine whether it needs a qualitative palette (for categorical data) or a color gradient (for numerical data), and use a color range that matches the values in the data.

The role of the scale_* functions is to customize the way values are visualized. GGplot defines multiple functions for this, all named using the same scale_{aesthetic}_{type}. So, to set a manual color scale we used scale_color_manual. All aesthetics can be set with scales; for example, scale_linewidth_continuous would create a continuous scale for the linewidth aesthetic.

The arguments for the scale functions depend on the scale type used. For example, the manual scale had the values= argument, while a color gradient allows you to set low= and high= colors and a scale_size_continuous has a range= argument to specify the size of point to represent the smallest and lowest values.

Color gradients

Let’s make a scatter plot of 2024 state-level votes:

Again, substantively this makes some sense: Democrats seem to do better in states with higher levels of education and a more ethnically diverse population.

However, aesthetically the graph leaves a lot to be desired. A first thing to remedy would be the color scale: let’s make sure the vote share runs from red (strongly republican) to blue (strongly democrat):

As you can see, this is close to the scale_color_manual we saw earlier, but now we use a gradient running from low to high. Also note that we can set the name of the legend (or axis) here, which is equivalent to setting it in the labs() as we did earlier.

Numerical scales and transformations

A second thing that we could improve is the display of the numerical scales. The default setting for the percentages is to display it as a regular number, e.g. 40 for college education or .4 for vote share, while both are actually 40%. Also, the default for population is to use scientific notation, showing 1e+7 rather than 10,000,000.

As you might have guessed, all this can be fixed in the scales:

Some points to note:

scale_x_continuous and scale_y_continuous are used to customize the scale for the x and y axis. In particular, we use labels=label_percent() to render them as percentages, setting scale = 1 to indicate that the numbers are already coded as 0-100 (rather than the default 0-1).
We set the same labels = label_percent() in the scale_color_gradient to display the numbers in the legend as percentages. We don’t set scale = 1 here as the vote share is actually a number between 0 and 1.
label_percent is part of the scales package, so we either include it as a library, or we need to use labels = scales::label_percent() in the ggplot call.
Finally, we set transform = "sqrt" in scale_size_continuous to make the dot size proportional to the square root of the population, rather than having it scale linearly. In this graph the effect is relatively small, but very often a small number of outliers can dominate a scale. In that case, using a square root or even transform=log10 can be an effective way to also see variation in the smaller cases.

Axis scales: Breaks and limits

The axis scale above uses the ggplot default of fitting the data and creating a sensible amount of ticks. We can customize this using the breaks= and limits= arguments to the x and y scales. While we’re at it, let’s also use scale_fill_gradient2 to make a gradient with an explicit mid-point:

Please take a good look at this code as it contains quite a number of moving parts. None of it is really complex by itself, but it can take a minute to understand all the different options.

Some things worthy of note:

We’ve added limits=c(0, 100) to force the y axis to display a full percentage scale
We’ve added breaks=c(0, ..., 100) to manually specify the y axis tick marks, and use n.breaks=5 to specify the x axis should have 5 marks.
Using scale_fill_gradient2 with mid="white", and midpoint = .5 means that the scale first goes from red to white and then from white to blue, with 50% vote being displayed as white. This can be a very useful option if there is a natural mid-point which would not show as clearly with a default gradient.
Since white circles are hard to see on a grey or white background, we use shape=21 to select a shape with an outline color as well as a fill. We leave the color to the default black, and map the fill to the vote share.
In the scale_size_continuous, we now explicitly give the range of sizes to use – try setting it to range = c(0, 10) to see the difference! We also specify which values should be shown in the legend (1 to 25 million) and set labels = label_comma() to suppress scientific notation.

Guides

A final useful argument for the scale_* function is guide=.
This allows you to customize the guides, i.e. the legend or axis marks:

You can see a number of applications in this example (admittedly, not all of them equally useful in this plot):

In the fill scale, we add a guide = guide_colorbar() (the default guide for color gradients) with options to reverse it to descending order, position it on the bottom, make it horizontal rather than vertical, and order it so it would come as the last guide
In the size scale, we add a guide_legend (the default guide for most aesthetics), overriding one aesthetic to make the dots purple
In the y scale, we disable the guide meaning that the percentages are not displayed
We add a guide_axis to the x scale, setting the text angle to 45 degrees.

9.2 Theming your plot

The customizations in the previous section all changed the way data was visualized in the plot. With theming, we can control the other visual aspects of the plot such as the backgroun color, font styles, grid lines etc.

As an example, let’s spice up the first plot used above:

This accomplishes a number of things:

First, theme_classic() selects an existing theme as a starting point. If you type theme_ and then press control_space, Rstudio should show you a list of available themes (Note that this also works great with geom_ and scale_!)

Subsequently, the theme() function further customizes the theme:

All text is set to use the “monospace” font at size 12 using the element_text helper function. Note that you can also set the font for individual elements, e.g. plot.title or axis.title.
The plot background is set to a delightful pink using the element_rect (rectangle) helper
The major horizontal gridlines are set to a dashed grey line using element_line. Note that you can also set the style for more grid lines using a less specific argument, e.g. panel.grid= or panel.grid.major=.
The x axis title is removed by setting it to element_blank() (which can be used to remove almost all plot elements, e.g. legend.title=element_blank() removes the legend title)
Finally, the legend is moved to the bottom with the title displayed on the top. You can also use legend.position="none" to disable all legends for the plot.

This only shows a tiny fraction of the available theming options, but they all work similarly to these so you can just play around with it. The argument names are very systematic, so you can e.g. type theme(panel. and press control+space to get a list of available panel options. For a complete list and explanation of available arguments, see the documention for the theme function

Please note that font rendering is highly device specific. The above example used the monospace font family, which (probably) works in your browser because the browser recognizes that family, but probably won’t work in Rstudio which ships with family=mono. For a more flexible handling of fonts, please see the showtext package.

As a final note, you can see overlap between labs, scale, guide, and theme for some options: legend position can be set either in the guide or in the theme; legend titles can be set either in the labs or in the scale; and axis and other titles can be removed in the labs, the scale, or the theme. Any of those places is fine, so just put it where it’s most convenient.

library(scales) vs. scales::

In the last example, we used scales::label_percent rather than calling library(scales) at the top.

Technically, using library attaches the package, making all functions available directly in the rest of the session. Sometimes, however, it’s more convenient to use the package::function notation instead: it makes it clearer where a function comes from, and prevents having 20 library() calls at the top of a script of which some might not even be used anymore.

Either notation works fine, and you can choose which you prefer. As a rule of thumb, I would advise using library(package) for packages (such as tidyverse) that are used throughout a script, and package:: for functions that are only used once or twice.

Re-using themes

If you are writing a report with multiple figures, it makes sense to set things like font and color preferences globally rather than on each plot separately. There are two mechanisms for this.

First, you can use theme_set() to change defaults for all plots in this session:

Note that you can call theme_set() without argument to revert back to the default theme.

Alternatively, since themes are just objects, you can assign your theme to a variable and re-use it:

9.3 Comparing groups side by side: facets

When visualizing complex data, it can be tempting to put as much information in a graph as possible. In theory, every aesthetic can be used to represent a variable, so it’s possible to visualize many different variables. However, putting too much information in a plot can quickly render it very hard to read: 3 lines in a graph is fine, but 12 different line colors with 4 different line types is often very hard to read.

An alternative way to display more information is to use facets. For example, we can plot vote shares in a number of key states:

As you can see, this produces a separate graph for each of the selected states. This works by using facet_wrap, which automatically arranges the graphs in a grid which can be controlled by setting nrow and ncol. As you can see, by default the x and y axis scales are kept the same for all graphs, so you can easily compare values between graphs.

Note that for technical reasons, you need to enclose the column names in vars() similar to how you put the columns in aes for specifying the mapping.

9.4 Annotations

A central goal in making good graphs is to ensure that a graph should be self-explanatory: it should be understandable and interpretable without needing to read or understand the surrounding text. Most of the tools needed for this have been discussed already, especially good plot and axis titles and useful legends.

Sometimes, however, it really helps to add annotations to a graph to make it easier to understand. For example, let’s add some annotations to the line graph made in the first example:

This shows three different annotations:

First, the geom_hline paints a simple horizontal line at y-intercept 0.5 (50%), using color, linewidth and lty (line type) to style the line. Note that this geom is placed before the actual plot geom to ensure that the plot lines are drawn over the hline: ggplot plots the geoms in the order given, so later geoms overwrite earlier ones.
Next, we place a textual annotation at the position of the ‘other’ vote share in 1992. Using vjust=0 justifies the label so the bottom is at the given y value. This uses the “label” annotation, which draws a small box around the text.
Finally, we add a regular “text” annotation to explain the 50% line. This is positioned all the way at the right hand side, with x=2024 and hjust=0, which left aligns the text to start from 2024. Of course, the graph only runs to 2024, so it would normally be mostly outside the plotting area. This is solved by increasing the right margin in the theme, and setting clip="off" in the cartesian coordinates, meaning that the text won’t be clipped if it’s plotted outside the actual plotting limits.

Finally, note the use of breaks = scales::breaks_width(4) and guide=guide_axis(angle=90)) in the scale_x. This ensures every election receives a label on the x axis, setting them at 90 degrees to avoid overlapping.

9.5 Exercise: Recreating a published figure

Let’s bring together stacking, colors and annotations to recreate a figure from a real published article.

Let’s put stacking, colors and annotations together to recreate a real published figure: Figure 1 from the original 538 article, The Dollar-And-Cents Case Against Hollywood’s Exclusion of Women (open it in a new tab so you can compare as you go).

The starting code below already does the data preparation: it bins year into 5-year periods labeled like 1970-'74, and strips the -disagree suffix from test so we’re left with five categories: nowomen, notalk, men, dubious and ok, from clearest fail to clearest pass. We’ve also created two helper variables: the 538 color palette with red shades for the “fail” categories, and blue shades for “pass”; and a breaks variable that contains the labels that should be shown on the y axis:

Note that this code uses some interesting ‘tricks’:

We compute the lustrum (5 year period) by dividing, flooring, and multiplying again.
We create a period variable using str_glue, ‘glueing’ together the start year, a literal dash, newline and apostrophe, and the last two digits of the end year of that period (see (sec_str?) for more details on the str_* functions)
We extract the unique periods with pull(column) |> unique() |> sort(), and then select all odd labels by indexing with a sequence from 1 to the number of periods, incrementing by 2 every step.

A number of things are still missing:

The bar plot should be turned into a stacked proportion scaling every bar to 100%, with the correct ordering of the factor levels.
The colors given above should be used for the bars
You should add a title and subtitle to match the original plot
The Y axis should be in percentages with the right labels
The X axis should have the labels only for the first half of each decade
You should add two text annotations for PASS and FAIL in the top and bottom center, respectively.

Fix all the TODOs below. You can run the graph at any moment to see how you are doing, and of course you can ask for hints on which functions to use.

🎓 Solution:

Since position_stack() puts the first factor level at the top, listing test from clearest fail (nowomen) to clearest pass (ok) puts ok at the bottom of each bar, matching the original. position="fill" turns the counts into proportions. Given that stacking order, the upper half of the bars (y=.75) is the “fail” categories, and the lower half (y=.25) is “pass”.

ggplot(fig1data) +
  geom_bar(aes(x = period, fill = factor(test, levels = names(colors))), position = "fill") +
  scale_fill_manual(values = colors, guide = "none") +
  labs(title = "The Bechdel Test Over Time",
       subtitle = "How women are represented in movies") +
  scale_y_continuous(name = "", labels = scales::label_percent()) +
  scale_x_discrete(name = "", breaks = breaks) +
  annotate("text", label = "FAIL", y = .75, x = 5, size = 20, fontface = "bold") +
  annotate("text", label = "PASS", y = .25, x = 5, size = 20, fontface = "bold") +
  theme_minimal() +
  theme(panel.grid = element_blank())

Compare your result to the original figure. A few things are still different:

The original draws a thick black line separating “pass” from “fail” in each bar.
Functioning as a legend, the original has small labels on the right-hand side naming each of the five categories, connected to the chart with a short tick mark.
Colors, fonts, axis styling – how many of the remaining differences can you track down and fix?

Some of these (especially the dividing line and the side labels) go well beyond what we’ve covered in this book. The {webr} block below starts you off with the solution to the exercise above – feel free to experiment, look up ggplot functions you haven’t seen yet, and see how far you get. There’s no single right answer here, and no checker: this is purely for the fun of it.

Below is our solution – don’t peek until you’ve had a go yourself!

Our solution – click to expand

Here’s how far we took it. This uses several tricks not covered elsewhere in the book which are explained below the code, but have a look at the code first and see if you can understand what’s going on. There are many other ways to reach a similar result – this is just one of them.

Some explanation:

We first define the colors, labels and positioning for the outcome labels in a separate data frame The alternative is spreading this information over the graph in various scale, annotation etc. calls, which makes it much harder to change later. In general, it can be a good idea to place this kind of information in a separate data frame rather than inside long blocks of code since it separates substantive decisions from implementation code.
Next, we compute the figure data as before, but join it with the outcomes data frame created in (1). This makes the color and positioning of the outcomes available in the plot.
We then compute the positioning of the outcome labels on the right hand side. The goal is to compute a y position that is halfway in the bar for that outcome in the last period. This first counts the number of films in each outcome for the last period using count, and converts it to a percentage by dividing by sum(n). Next, it takes the cumulative sum of these percentages ordered by descending position, which effectively computes the upper limit of each bar since they are stacked. Finally, we subtract half of the current bar’s size to get the midpoint for the bar.
We compute the y position of the line separating pass from fail for each bar. The trick here is computing the mean of a boolean (yes/no) variable, i.e. whether it has passed. R treats TRUE as 1 and FALSE as zero, so the mean is the same as the percentage of cases that passed. A second trick is defining the x position as a number based on the factor level. For categorical axes, ggplot also accepts numerical positions from 1 to the number of bars drawn.
In the ggplot call, we set the width of the bars to 0.96 to mimic the small gaps in the original. We can now also set the fill to outcome_color and use scale_fill_identity, which tells ggplot to treat the values as actual color names rather than categorical values that need to be mapped to a palette. We can also use fct_reorder(outcome_color, position) here since we joined the position column to the data, so we can order the outcomes based on this position.
We now add the line between the pass and fail bars based on the pass_line data constructed in (4). Note that ggplot will draw the first bar with 1 as the center, so a bar of width 0.96 will be drawn from 0.52-1.48, leaving a 4 point gap between the bars. So, for the horizontal bars, we compute x as x - .52 and xend as x + .52: since the width of each bar is .96, ggplot renders it from -0.48 to 0.48 with a .04 gap between two bars, so we use .52 to cover the gap as well. This would also extend left and right of the first and last bars, so we use pmax and pmin to limit the values to 0.52 and 9.48 rather than 0.48 and 9.52 for the first and last value. For the vertical bars, we draw a line between this bar and the next, setting x=x+.5 to point to the right hand side of the bar, and draw a line from y to lead(y), which takes the next (‘leading’) value of the same column. For the last bar this will give NA (since there is no next bar), so no vertical bar is drawn at the end.
The pass/fail annotations are simple text annotations. For the outcome labels and segments, however, we can now use the labels dataset we created in the step (3), and use a geom_segment and geom_segment line based on this dataset rather than having to specify 5 separate annotations for the text and lines.
We draw the line from 9.48 to 9.6 and start the text at 9.7 (left-justified), which draws them to the right of the last bar, again using the fact that the last (9th) bar is drawn to 9.48.
The original figure had 100% for the top mark, but 0, 25 etc without percentage sign for the rest. There is no built-in option for this, but we can achieve this by using a function to compute the label, setting it to "100%" for y==1 and to a simple y*100 for the rest
The original figure had no space between the bottom of the bar and the axis line. This is copied by setting the expansion of the vertical scale to 0 for the bottom and 5% for the top using mult=c(0, 0.05). Similarly, the space to the left of the graph is set to exactly 0.48 to ensure there is no space between the bar and the tick marks, using add rather than mult since here we give the exact space to be added to the plot, rather than a factor with which to multiply it. 10.To copy the grey bar with the source attribution, we use annotations in negative vertical positions. Note that these are dependent on the exact size of the graph, so would need to be tweaked after fixing the exact output format. 11.Finally, we make space for the outcome labels on the right and the grey attribution bar by setting larger margins on the right and bottom of the plot and specifying clip="off" in the coordinate function (to prevent these elements from being clipped). We also fix the xlim and ylim to make sure the axes don’t extend beyond the actual bar chart, because otherwise the negative values used in the grey bar would be seen as part of the plotting area, and the axes would actually be drawn below the gray bar instead of the other way around.

As said, this shows a number of “tricks” that might be useful in specific circumstances. Some things, such as using a data frame for the outcome colors and labels, using a geom with separate data for plotting multiple data-driven annotations, and using numbers for horizontal positions in categorical scales are useful in many circumstances. Using the black bar between the blue-ish and red-ish colors and the big pass/fail annotations also drive home the overall message in a very clear way. Using annotations linked to the data rather than a generic legend is also quite a useful way to make a graph both more aesthetically pleasing and easier to read, since the reader doesn’t have to make a connection between the colors and labels.

Other tricks, especially the grey bar at the bottom and suppressing the percentage mark, are not as useful and done here mostly to showcase how much you can customize plots if desired. Especially the use of a custom annotation which has to be fine-tuned every time the plot size changes instead of a simple caption= is probably not really worth it in most applications. But now you know that you can if you really want to!

9 Advanced GGplot usage

Data used in this chapter

9.1 Scales: the link between values and aesthetics

Discrete scales

Color gradients

Numerical scales and transformations

Axis scales: Breaks and limits

Guides

9.2 Theming your plot

Re-using themes

9.3 Comparing groups side by side: facets

Facet on multiple variables with `facet_grid`

9.4 Annotations

9.5 Exercise: Recreating a published figure

9 Advanced GGplot usage

Data used in this chapter

9.1 Scales: the link between values and aesthetics

Discrete scales

Color gradients

Numerical scales and transformations

Axis scales: Breaks and limits

Guides

9.2 Theming your plot

Re-using themes

9.3 Comparing groups side by side: facets

Facet on multiple variables with facet_grid

9.4 Annotations

9.5 Exercise: Recreating a published figure

Facet on multiple variables with `facet_grid`