Skip to main content

10  Text as Data: tidytext

Chapter objectives

In this chapter, you will learn how to use tidytext to treat ‘text as data’:

  • Using unnest_tokens to ‘tokenize’ a text, turning textual data into a tidy data frame with one word per row (Section 10.1)
  • Analysing that data frame to show which words occur (Section 10.2)
  • Comparing two corpora (text collections) to highlight typical terms (Section 10.2.2)
  • Finding and using n-grams (word pairs or short phrases; Section 10.3)
  • Using and validating dictionaries for document filtering and sentiment analysis (Section 10.4)
  • Using LDA Topic Modeling to cluster words into topics (Section 10.5)

As always, the chapter ends with a multiple choice quiz organized per section that you can use to test your knowledge and understanding. You can also expand the table below to see which functions are covered in this chapter.

Function What it does Section
unnest_tokens() split a text column into tokens (words), with one row per token Section 10.1
stop_words, get_stopwords() stop word lists, as data frames with a word column Section 10.2
bind_log_odds() add the weighted log odds of each word per corpus (from tidylo) Section 10.2.2
unnest_tokens(token="ngrams", n=) split a text column into n-grams rather than single words Section 10.3
sentiments the Bing sentiment dictionary, as a data frame with a word column Section 10.4
cast_sparse() turn a tidy token data frame into a (sparse) document-term matrix Section 10.5
LDA() fit an LDA topic model (from topicmodels) Section 10.5
posterior() get the word-topic (terms) and document-topic (topics) weights from a topic model Section 10.5
reorder_within(), scale_y_reordered() order bars within each facet of a faceted plot Section 10.5

Data for this chapter

For this chapter we will make use of the ‘PImPo’ (Parties’ Immigration and Integration Positions) dataset (Lehmann and Zobel 2018a, 2018b), which is part of the Manifesto Project (Lehmann et al. 2015). The sample used in this chapter contains about 3,100 English and German sentences from political party manifestos that have been hand-coded for immigration and integration policy positions:

Since many analyses are language specific, we also created an English language subset pimpo_en to make it a bit easier to work only on one language.

Let’s have a look at the distribution over topics and directions per country:

As you can see, about half of the sentences are about immigration or integration, and all of these are coded for the position of that party on the topic.

Note that this dataset is a small sample of the full Manifesto and PImPo datasets, redistributed with permission from the Manifesto Project. The full dataset can be obtained using the manifestoR package, see this page for instructions on how to get an API key and download the data, and see our data folder README for the script we used to prepare it.

Variable Type Explanation
id numeric (dbl) Unique number of each sentence in the original PImPo data
country character Country of the party (Australia, Austria, Ireland, or United States)
year numeric (dbl) Year of the election for which the manifesto was written (1999–2013)
party character Name of the party, as given by the Manifesto Project
party_abbrev character Abbreviated party name
text character The text of the (quasi-)sentence
topic character Whether the sentence is about immigration, integration, or neither (other)
direction character Position towards immigration or integration: supportive, neutral, or sceptical (missing if the topic is other)
certainty_topic numeric (dbl) Percentage of coders who agreed on the topic (missing if the topic is other)
certainty_direction numeric (dbl) Percentage of coders who agreed on the direction (missing if the topic is other)

Note that in the full manifestos data set, only a couple percent of sentences are about immigration or integration. This data set consists of all those sentences (n=1,548) and an equal number of randomly drawn sentences not about immigration or integration, so exactly 50% of each country’s sentences is about immigration/integration.

TipUsing your own data with readtext

The examples in this book all use ready-made csv files available through the Internet. There is a good chance, however, that you might want to analyse text files that are on your own computer.

By far the easiest way to read in multiple text files is using the readtext package. You can point this package at a folder (or zip file) and it will automatically read all files inside it. Even better, it automatically parses pdf and docx files and turns them into plain text columns:

> readtext("your_folder_or_zipfile") |> as_tibble()
# A tibble: 3 × 2
  doc_id             text                    
  <chr>              <chr>                   
1 plaintext.txt      "A normal text file"    
3 word_file.docx     "Text from a .docx file"
4 document.pdf       "Text from a pdf file"

See Chapter 15 (coming soon!) for more details on reading data into R.

10.1 unnest_tokens: From characters to words

In the previous chapter we have used regular expressions to do text analysis, but this is sometimes awkward as we are normally interested in words rather than characters. Using the tidytext package, we can unnest a text and create a row for each word. For example, this creates a data frame with all the words used to talk about immigration by the English-language manifestos:

Since this is just a normal data frame, we can use regular tidyverse functions to do analysis. For example, let’s plot the most contrasting words used for each position:

As you can see, sentences that express skepticism towards immigration in this sample talk about aliens and use words such as enter, boats, borders, rule and enforcement. Sentences supportive of immigration instead talk about refugees using words like rights, needs, family/families, and comprehensive reform.

Looking at the code, it might look complex at first but it really only uses the tidyverse building blocks you mastered in earlier chapters, especially Section 5.3. First, we compute the count of each word per direction (stance) and in total. Next, we compute the proportion of each word in that direction, showing us whether a word is almost exclusively used in that direction (proportion near 1) or used as much regardless of the direction (proportion near the total percentage of words in that direction). We can then filter to only keep words with more than five mentions, and use slice_max to select only the ten words that have the highest proportion per direction. Finally, we make a bar chart of the top words (Chapter 6), using facet_wrap to create separate facets for sceptical and supportive words (Section 8.3).

TipTokenization in detail

Even the simple example above already shows the benefits of working with words instead of character strings. Questions such as which words are used or overused become simple tidyverse operations. However, as the example above shows, words that seem similar such as “family” and “families” are treated as distinct words, while “Australia’s” is seen as a single word.

This shows that it is important to understand a bit better what’s going on under the hood. The process of turning text (a string of characters) into words is called tokenization, with words often referred to as tokens in language processing. The unnest_tokens function above is quite powerful and can deal with different languages, including languages like Chinese or Japanese that don’t use spaces to mark word boundaries:

As the example above shows, Mao’s call to “serve the people” is correctly tokenized into 为 (“for”), 人民 (“the people”), and 服务 (“serve”).

Tokenization, however, is not a one-size-fits-all operation. Let’s look at some more examples:

As these examples show, tokenization depends strongly on the presence of spaces and punctuation, although it does not blindly split on all punctuation. As you can see, both 1:30 and #ge2024 are split into multiple tokens, which is probably not how we would normally see it. In the next text, Sinn Féin is split into two tokens, with the possessive s in Féin's seen as part of the token. In the url below it, the :// part is seen as three distinct tokens, while the domain name is kept as a single token. Finally, the beautiful German compound Kettenasylantragstellungen (“chain asylum applications”) is seen as a single word.

In each case, the tokenization is correct (or at least defensible). However, it does show that in analyses you need to pay attention to tokenization issues: for example, a naive search for asylum applications in German would miss that compound, and similarly looking for Sinn Féin or the use of hyperlinks after tokenization would be hard. All of these issues can be addressed by preprocessing and/or more careful morphological analysis, but make sure that you check the outcome of your tokenization to make sure that what you would call a word or term is also what comes out of unnest_tokens.

Note finally in the code that we first use unnest_tokens to create a long data frame, and then use summarize to collapse all the tokens of each sentence back to a single text, with a distinctive marker between the tokens.

10.2 Corpus Analysis

Let’s have a closer look at the PImPo corpus (corpus being the fancy word linguists use when they mean a collection of texts).

Which words are used to discuss immigration?

Although this is a literal answer to our question, it is not a very informative one: in almost any text words like “the” and “to” would be in the top of this list.

Stop words

Such words are called ‘stop words’, i.e. words that should be “stopped” from entering the analysis. Various stop word lists exist for most languages, and for English a selection of stop word lists is immediately available as stop_words data frame:

Since these stopword lists are data frames with a ‘word’ column, we can directly use anti_join to remove them from our analysis:

And the result is clearly much more informative, with “immigration”, “asylum” and “seekers” forming the top three.

Can you adapt the code above to give the list of most frequent words for the Austrian sentences?

Corpus comparison

It is often very informative to compare word use in two corpora. As in the opening example of the chapter, we can see which words are used more in sceptical relative to supportive statements about immigration. We could also investigate whether US party manifestos use different words to talk about immigration than Irish, or whether immigration vocabulary changed over time.

Let’s start by looking at the proportion of each word’s usage in Irish manifesto sentences about integration or immigration:

This produces some interesting findings: apparently asylum and employment are uniquely Irish concerns in the immigration manifestoes. Unsurprisingly, it also contains a lot of words referring to Ireland and the EU. If you change the last line to arrange(prop_ie) you will see that in the US discourse even more of the top-10 words refer to the nation.

However, this comparison is very naive. First, we did not do any cleaning (although unnest_tokens by default lowercases and removes punctuation). Proportion as a measure is also deeply flawed: it does not take into account that the Irish corpus is simply bigger than the American (probably because they have more parties, even though there are three US election cycles in the data versus only two Irish ones). Moreover, since proportion peaks at 1 when only one corpus uses a word, it will show mostly which words are unique, even if there are much more frequent words that are overused in either corpus. This is mitigated a bit by setting a minimum frequency (total>10), but that is an arbitrary cutoff value.

For the cleaning, this is actually a case where stopword removal probably will do more harm than good: comparison naturally filters out words that are very frequent in both corpora, and if US parties use ‘should’ more while Irish parties use ‘will’ more (which they actually do!), this might be an interesting finding rather than noise. However, two cleaning steps are probably a good idea: removing the possessive 's, and creating a manual ‘stop word list’ containing country and party names.

To create the manual list, we can actually use the party column for the party names, using unnest_tokens again to separate the parties into words:

This also shows how stop words are often not a trivial decision: as the list includes ‘united’, ‘fine’ and ‘left’, it would remove these words regardless of whether they are used as party names. This can partly be solved by postponing the lower casing until after using the stop word list, but e.g. ‘Left’ could also be used as the start of a sentence so it won’t fully solve it. Using word combinations (“n-grams”) would be a better solution here, see Section 10.3 below.

A classical measure in corpus comparison is chi-squared, which is a standard non-parametric statistic that we can use to test whether a country uses a word more (or less) frequently than ‘expected’, i.e. than if usage were unrelated to country. Although this is an oft-used measure, it overestimates the unexpectedness for rare words (and paradoxically, in language rare words are very common!)

For this reason, Monroe et al. (2008) suggest using a weighted log-odds measure which they demonstrate by comparing Republican and Democratic word use on issues such as abortion. The log-odds ratio essentially tells us how much less surprised we would be to encounter a word if we know whether it came from the Irish or American corpus. It is also the same statistic used in logistic regression to indicate how strongly the independent variable (country) is related to the dependent variable (is it this word or a different word). This measure is then smoothed to deal with low frequency words, and weighted by dividing by the standard error to produce something akin to a z-score. The good news is that this measure is implemented in the tidylo package, so we don’t need to know any mathematics to be able to use it.

Let’s put these steps together and get a more well-founded corpus comparison:

As you can see from this output, the words that are found are quite different from the words in the earlier (proportion-based) list. The weighted log odds focuses less on the words that were exclusive to either corpus (for which proportion=1) and more on more common words that are overused in either corpus. For example, take our in the American manifestos: it is also used in the Irish manifestos, so its proportion is much lower than 1 and it didn’t show up in the first results above. But because it’s a very common word, a difference this large is much less likely to be due to chance, making it the top word by weighted log odds. Arguably, overused common words are substantively more interesting than less frequent exclusive words, but of course that ultimately depends on the research question.

Besides this substantive difference, the statistical measure also avoids the need for an arbitrary frequency cutoff, gives a more informative measure of the ‘keyness’ of each word, and solves the problem of not accounting for the relative size of both corpora.

Looking at the code, most of it should start looking familiar. We start by filtering and tokenizing as above. Then we have two cleaning steps, removing all word-ending 's and removing words from our custom stop word list. Then we count the frequency of each word in each corpus, and use complete to ensure words not used in a corpus get a zero (which is necessary for computing the log odds ratio correctly). Then we add two additional columns: n_other so we can compare frequencies more easily (using .by= as a shortcut for group_by), and bind_log_odds to add the log odds column.

Finally, let’s use this to create a graph similar to those used in Monroe et al. (2008), with word frequency on the horizontal axis and keyness on the vertical:

This graph shows all words from the comparison, with words overused in American manifestos on the top (and in blue), and words used more in Ireland at the bottom in green. More common words are on the right and rare words on the left, with the horizontal ‘bands’ indicating how many words shared the same count and log odds value.

This graph nicely shows the different words used in both countries: American manifestos talk about borders, law, illegal, aliens and immigrants and use ‘our’ and ‘we’ a lot; while Irish manifestos talk about asylum (seekers), integration, permits, and applications. On the right you can see that traditional stop words such as “in” or “and” have low log odds scores, making normal stop word removal unnecessary (and even harmful as it generally also removes words like ‘our’ or ‘we’).

Looking at the code, there are a number of interesting “tricks” used to make this graph:

  • First, we select the US words as main data set, and manually calculate the x position by adding some random noise to the total word frequency. We could also have used geom_jitter, but by adding the noise before the log transformation it has less effect on words with a higher count.
  • Next, we define a separate set of ‘focus words’ to highlight, in this case taking all words with an absolute log odds of above .8 or a total frequency above 100.
  • In the ggplot, we use a regular geom_point to show all words as points, with a low alpha value to show overlapping points. A hline shows a dashed y=0 line as a visual guide.
  • Then, a geom_text with data=focus_words shows text only for the chosen words, using vjust to make sure the labels are shown above the points.

The rest is fairly straightforward: a log x scale, a minimal theme, and labels to make the graph interpretable.

TipPreprocessing choices

The steps involved in cleaning or transforming a text before the analysis are generally called preprocessing. Although there are many steps that are commonly used, such as converting to lower case and stripping punctuation, there is no agreed-on “recipe” that each analysis should use. In fact, how you should preprocess depends on your research question (cf. Grimmer et al. 2022): if you are looking at social identity you probably want to keep stop words like “us” and “them”, while a topical analysis probably wants to ignore those.

Below we list a table of the most common preprocessing steps, based loosely on the reviews by Denny and Spirling (2018) and Hickman et al. (2022):

Choice Description Method (or Code)
Lowercasing Treat Immigration and immigration as the same word On by default; turn off with unnest_tokens(..., to_lower = FALSE)
Punctuation and symbols Remove tokens such as :, # or - (and emoji) On by default; keep them with unnest_tokens(..., strip_punct = FALSE)
Numbers Remove numeric tokens such as 1,500 unnest_tokens(..., strip_numeric = TRUE)
Stop words Remove frequent words with little content (the, and, der) anti_join(stop_words); for other languages get_stopwords(language = "de")
Trimming Remove words that are too rare (or too common) to be informative add_count(word) |> filter(n >= 5)
Stemming Cut words back to a common stem (families, family → famili) Provided by e.g. the SnowballC package
Lemmatization Replace words by their dictionary form (families → family, went → go) Provided by e.g. the udpipe or spacyr packages
n-grams Use combinations of adjacent words as tokens (sinn féin, not good) unnest_tokens(..., token = "ngrams", n = 2)
Negation Keep track of negated words (not good) Use bigrams, or join the negation to the next word before tokenizing
Spelling and abbreviations Normalise variants (U.S., US, USA) and spelling errors Recode with str_replace_all() before tokenizing, or a spell checker such as hunspell

Exercise: Sceptical or supportive?

At the start of this chapter we compared the words used in sceptical and supportive statements about immigration, using a simple proportion. Now that we know about weighted log odds, we can redo that comparison properly.

Compare the words used in sceptical and supportive statements using weighted log odds. Use the same sentences as in the opening example: sentences about immigration (not integration) with a sceptical or supportive direction, from all English-language countries. It’s up to you to decide which cleaning steps make sense here. The result should be a data frame with the weighted log odds for each word and direction, shown in descending order of log odds.

TipHint:

The steps are the same as in the country comparison above, so have a look at that code.

  • First select the right sentences, then split them into words.
  • Do you want to remove the possessive 's or stop words, as discussed above?
  • Then count how often each word is used in each direction.
  • Make sure that every word has a count for both directions, even if it is zero.
  • Compute the log odds: which column defines the two groups that you are comparing?
  • Arrange by log odds (descending) to show the most typical words.
Important🎓 Solution:
  • Use filter(topic == "immigration", direction != "neutral") to select the sentences. As pimpo_en only contains English-language countries, there is no need to filter on country.
  • Tokenize with unnest_tokens. We chose to remove the possessive 's with str_remove, but to keep the stop words, as words like we or our can say a lot about how parties frame immigration. Removing (or keeping) the 's or stop words would also be fine, as long as you can justify your choice.
  • Use count(direction, word) to count each word per direction.
  • Use complete() to add a zero count for words that are only used in one direction. If you skip this step, bind_log_odds doesn’t know that these words were never used in the other direction, and their log odds will be wrong.
  • bind_log_odds(direction, word, n) compares the two directions.
  • arrange to see the most typical words.
pimpo_en |> filter(topic == "immigration", direction != "neutral") |> unnest_tokens(text, output=word) |> mutate(word = str_remove(word, "['’]s$")) |> count(direction, word) |> complete(direction, word, fill = list(n = 0)) |> bind_log_odds(direction, word, n) |> arrange(desc(log_odds_weighted)) 
pimpo_en |>
  filter(topic == "immigration", direction != "neutral") |>
  unnest_tokens(text, output=word) |>
  mutate(word = str_remove(word, "['’]s$")) |>
  count(direction, word) |>
  complete(direction, word, fill = list(n = 0)) |>
  bind_log_odds(direction, word, n) |>
  arrange(desc(log_odds_weighted))

Sceptical statements are characterized by illegal, border(s), enforcement, aliens, boats and enter, while supportive statements talk about refugees, rights, visa, family and reform. This is quite similar to the opening example, but the log odds also pick up more frequent words that are only somewhat overused, such as illegal (22 times in sceptical statements) or even a in supportive statements.

  • Which of the two comparisons (the proportions at the start of the chapter, or the log odds above) tells you more about how sceptical parties talk about immigration? Why?
  • Can you change the code below to compare Australian and American manifestos? What do you notice about the top words? How can you fix this?

10.3 Collocations and N-grams

In the examples so far, we have focused on single words. However, very often it is more informative to treat word combinations like United states or rule of law as a single token.

To do this, we can use unnest_tokens(token="ngrams") to tokenize to n-grams rather than words. In language processing, an n-gram is a sequence of n words, so United states is a 2-gram (or bigram) and rule of law is a trigram.

As you can see, most bigrams are actually not that interesting, with patterns like “of the” and “in the” dominating the list.

As before, this can be remedied relatively easily by removing stop words. Of course, we cannot simply anti_join on the stop word list as the tokens are now bigrams, but we can create a regular expression pattern for any ngrams that contain a bigram by collapsing all stop words to a single |-separated pattern and placing \\b on either side:

This is already a quite informative list, showing meaningful combinations like asylum seekers and party names like fine gael.

However, as above stop words are a very blunt solution. For example, united states is not present above because the onix stop word list contains states as a stop word. On the other extreme, the smaller ‘snowball’ stop word list does not contain a word such as ‘will’, so switching to that list will bring in patterns like “will also”.

In corpus linguistics, point-wise mutual information (PMI) is often used for detecting collocations: word combinations which are more frequent than you would expect from their underlying frequencies. There is no built-in function for computing PMI in tidytext, but fortunately it is relatively easy to compute it ourselves:

Substantively, this list looks mostly right, with Irish party names topping the list as both fianna and fáil only occur as part of the collocation. It also contains terms like ‘green card’ and ‘climate change’, titles like ‘president bush’, and constructs like ‘long term’. Most importantly, bigrams like ‘of the’ are not included: Since stop words are themselves very frequent, combinations with stop words are expected to be relatively frequent as well.

Using n-grams in frequency analysis

We can use the bigrams we found above to preprocess the texts before a regular frequency analysis. In particular, we can create a list of replacements so e.g. “green party” becomes “green_party”:

As you can see, this is a named vector of strings, where the name is the pattern to look for (\\bunited states\\b) and the value is the replacement (united_states). This is created by first selecting those bigrams with a pmi over 8 and ‘pulling’ the bigram column. We then replace the spaces by underscores, and finally set the pattern as the names.

This can be used to augment earlier analyses, e.g. the word comparison:

As you can see, united_states is now seen as a single token, and (if you scroll down) so are terms like asylum_seekers and illegal_aliens and Irish parties like fine_gael. Note that we call str_replace_all on the lowercased text, as our collocations are all lower cased.

Note that we used the value 8 based on inspection of the bigrams, as there is no sensible agreed-on cutoff value. Depending on the research question it can also be a good idea to manually curate this list to ensure that e.g. all party names are included.

10.4 Dictionary Analysis

One of the oldest tools in the computational social science toolbox is dictionary analysis: using a (generally hand crafted) set of words to detect whether a concept occurs or to measure a variable such as tone or stance.

Topic classification

As a first application of a dictionary, let’s see if we can create a dictionary to classify whether topics are relevant for a specific topic. In general, identifying relevant documents for analysis is an excellent use case for dictionaries.

In this case, we might want to know specifically whether a document is about migration. Although it is entirely possible (and sometimes the best idea) to create a dictionary yourself, it is also a good idea to check the literature for existing dictionaries. For example, Palicki et al. (2023) published an ‘expert validated search query’ for migration, which we can use in a shortened form:

As you can see, this created a new column immigration, which is TRUE if the word matched the dictionary and FALSE otherwise. To validate how well the dictionary works, we can summarize the output per sentence and compare the dictionary output to the actual topic:

The resulting table is called a confusion matrix, with the gold standard outcomes in the rows, and the classification predictions in the columns, and each cell showing the frequency of that combination. Note that we group by c(id, topic) – since every sentence has a single topic, this doesn’t actually affect the grouping compared to just grouping by id, but it means that the topic per sentence is preserved in the summarization result.

The good news is that when the dictionary found a word (the `TRUE` column) the sentence was almost always about immigration or integration. The less good news is that many of the immigration and integration sentences were missed by the dictionary.

TipPrecision and recall

The pattern found above is fairly common in dictionaries: because so many different words can be used to refer to a topic, when there is a match it’s likely to be relevant, but many relevant texts are missed.

In text analysis, we often use the terms precision and recall to indicate these metrics: Precision is the proportion of all matched items that were indeed relevant, while recall is the proportion of actually relevant items that were matched.

In the example above, if we consider immigration and integration to both be relevant, we have 217+93=310 true positives: relevant items that were correctly identified.

To calculate precision, we compare this to the false positives, items that were incorrectly classified as relevant. Here we only found 2, so precision is 310/(310+2)=99%

For recall, we instead compare to the false negatives, items that were missed (i.e. incorrectly classified as irrelevant). In this case the dictionary missed 112+128=240 sentences, giving a recall of 310/(310+240)=56%.

We can also use summarize to calculate these:

You can change topic != "other" to topic == "immigration" in the chunk above to calculate precision and recall if we would only consider immigration sentences as relevant. As you can see, this increases recall at the expense of precision – which makes sense given that the confusion matrix showed that the current dictionary also matches 93 integration sentences.

The code below combines all steps from this section: matching the dictionary, and computing precision and recall.

  • How would you extend the dictionary to get better recall, especially for integration? Does it work? What happens to the precision?
  • How could you use the corpus analysis and comparison techniques from Section 10.2 to find candidate words for the dictionary? Which sentences would you compare?
  • If you choose words that way, do you think the precision and recall computed above are still a good estimate of how well your dictionary would work on new texts? Why (not)? How can this be improved?

Sentiment analysis

Dictionaries are frequently used in sentiment analysis, even though research often finds they perform poorly in social science applications (cf Van Atteveldt et al. 2021). Let’s see whether an existing dictionary can differentiate between sceptical and supportive sentences.

Tidytext ships with the sentiment dictionary published by Hu and Liu (2004) (often called the Bing dictionary after its second author’s first name)

Since this dictionary contains full words instead of word stems, we can join it to the result of unnest_tokens:

As you can see, most words are not matched, resulting in NAs, while the word integrated is actually coded as positive. Note that we need to use left_join here, as otherwise any sentences without matches would be silently dropped from the dataset.

Let’s sum the values per sentence, and see how they relate to the coded sentiment:

So, if we assume the sign of the summed sentiment values should correspond to the direction of the stance, this dictionary is performing quite poorly. Overall accuracy is only 49%, performing even worse than just guessing ‘supportive’ for each sentence. Especially sceptical is problematic, with almost as many sceptical sentences coded as positive as negative. This is quite likely due to words that express a desire for stricter immigration and integration rules, but that are coded as positive. We saw the word integrated above, but also words like ‘secure’ and ‘effective’ are coded as positive in the dictionary.

Exercise: Precision and recall per direction

The confusion matrix above shows that the sentiment dictionary performs poorly overall. But how well does it detect the sceptical statements in particular?

Starting from the predictions data frame created above, compute the precision and recall of the sentiment dictionary for detecting sceptical statements. The result should be a data frame with a precision and a recall column.

TipHint:
  • Have a look at the precision and recall code in the callout in Section 10.4.1 above. There, relevant indicated whether a sentence was actually about immigration, and match whether the dictionary matched it.
  • What are the equivalents of these two columns if we want to detect sceptical statements? You can create them with mutate.
Important🎓 Solution:
  • A sentence is relevant if it is actually sceptical: direction == "sceptical".
  • A sentence is matched if the dictionary predicted it to be sceptical: predicted == "sceptical".
  • With these two columns, we can use exactly the same summarize as in the callout.
predictions |> mutate(relevant = direction == "sceptical", match = predicted == "sceptical") |> summarize(precision = sum(relevant & match) / sum(match), recall = sum(relevant & match) / sum(relevant)) 
predictions |>
  mutate(relevant = direction == "sceptical",
         match = predicted == "sceptical") |>
  summarize(precision = sum(relevant & match) / sum(match),
            recall = sum(relevant & match) / sum(relevant))

So, the dictionary does quite poorly for sceptical statements: less than half of the sentences classified as sceptical are actually sceptical (47% precision), and it finds only 38% of the sceptical statements (recall).

  • Can you compute the precision and recall for all three directions? You could repeat the code above for each direction and combine the results with bind_rows, but there is also a much shorter way: if you group by the actual direction, the recall for each direction is simply the proportion of correct predictions, mean(predicted == direction). What do you need to group by to compute the precision in the same way?
  • For which direction does the dictionary perform best? Can you explain why it performs so poorly for neutral statements?

10.5 Topic Modeling

This chapter so far has mostly looked at which words (and bigrams) occur in a document. It can also be very informative to see which words co-occur: can we find coherent clusters of words that often occur together?

Although we will go into this in much more detail in Chapter 12, this section will give a small demo of topic modeling, which used to be a very popular method for exploring a corpus (Blei et al. 2003; see e.g. Jacobi et al. 2016 for an application to political communication).

The document-term matrix as a bag of words

Before running the actual topic model, we convert the ‘long’ format of the tokens into a ‘wide’ format with one word in every column: a document-term matrix or DTM. Since this would produce thousands of columns with mostly zeroes in the cells, rather than using a regular pivot_wider we use cast_sparse. This creates a sparse matrix format to represent the DTM, which is an efficient way to store such a matrix as only the non-zero cells are actually stored.

Looking at the first 5 rows and a hand-picked set of columns shows the structure of the dtm: each row is a document, and every column is a word. The word ensure occurs only in document 193798, but social occurs also in document 194040. This latter document also contains anti and behaviour (twice). It is tempting to read this as 193798 talking about ensuring (something) social, and 194040 as talking about anti social behaviour, but the DTM only tells us these words occur, not in which order or context – you would have to check the actual sentences to learn the usage and context. This is why a dtm is often called a ‘bag of words’: every document is represented as a set of word frequencies, but all ordering or syntactic structure is ignored.

Estimating the topic model

We can use the topicmodels package to cluster this matrix into topics:

This shows the top-10 words for each topic. A number of topics all seem related to immigration in different ways: topics 1-3 all mention immigration prominently, but also mention specific countries (Ireland, Australia, and the US respectively). Topic 5 talks about law enforcement and language, and topic 9 talks about asylum seekers. Other topics are less directly immigration-related: topic 4 is about government services, while topic 7 talks about workers and emplotment and 10 about families and schools. Note that it is not surprising that this corpus yields a mix of immigration and non-immigration topics, with a strong focus on immigration, as it was sampled to have 50% of sentences about immigration or integration.

Looking at the code, we can see some interesting things:

  • The LDA call creates a topic model t with k=10 clusters. We set a random seed for reproducibility since topic modeling is non-deterministic.
  • To analyse the results, we take the posterior distribution from the model, from which we pluck the “terms”, which is a matrix of probabilities for each term within each topic (pluck is a helper function that takes an element from a list within a tidyverse pipeline)
  • We convert this matrix to a tibble, and pivot it to a long format of weights (“betas”) per topic per word
  • Finally, we select the top-10 words with the highest weight, and collapse them to a single value for easy inspection (using the .by shortcut to skip the group_by step)

One small note: LDA is a ‘mixture’ model, which means that each document (and word) can be assigned to multiple topics. In our case this is not very useful since our documents are short sentences, but we can still use it to showcase the general working of LDA.

You can alter the code above to play around with the topic modeling algorithm.

  • What happens if you change the seed? Do you get similar clusters, or quite different. Note that the ordering of clusters is completely random, so even if you get similar clusters they will have different topic numbers.
  • What happens if you decrease k, e.g. to 2 or 5? Or increase it to 20? Do the topics feel more coherent, or less? Does their meaning shift?

Inspecting topic loadings

Using similar code, we can also inspect the LDA topic loadings for each document, and compare it to the manually assigned topic:

The code here starts very similar to the earlier code, but plucking the topics from the posterior. It then averages the ‘gamma’ weights per manual topic and LDA topic and uses that to create a heat map. The gamma shows how each document is assigned to the various topics, so the columns add up to 100%.

This shows some interesting patterns. First, LDA topics 2 and 5 are most clearly related to immigration, but 1, 3 and 9 also each account for more than 10% of the immigration sentences.
Integration peaks in topic 5, but this topic is shared with immigration and integration sentences are mostly spread over all the topics. This is perhaps unsurprising given that our dictionary also had more difficulty finding sentences about integration. Finally, as we also saw in the top words, topic 10 is the clearest non-migration topic, with 4, 7, and 8 also having more than 10% of ‘other’ sentences.

The graph also shows that we should not read too much coherence into topics: the majority of immigration sentences ultimately ends up spread over other topics than 2 and 5. This makes sense as immigration is often also discussed in its relation to other topics. For example, looking at a number of immigration sentences from LDA topic 10, which we earlier saw contained families and education, it is clear that immigration is indeed also discussed in this context: keeping families together and the role of foreign talent. However, it also shows that topic 10 is much broader than just education, with many sentences talking about e.g. the rule of law. This highlights the importance of reading actual text to interpret topics rather than just looking at the top words, and avoiding the temptation to assume that a LDA topic maps perfectly on what we would normally call (political) topics.

The code below shows the highest loading ‘immigration’ sentences for LDA topic 10:

  • How would you evaluate and label the earlier ‘families and education’ topic (10) given these sentences? Does it change if you look at sentences in the ‘other’ topic?
  • For topic 5, have a look at both integration and immigration sentences in this topic. Does it make sense that these are combined in a single topic?
  • Do other topics such as 2 or 9 make more sense? Can you find good labels for these topics?

Plotting words per topic

Finally, let’s combine the document loadings with the word loadings to show the top words per topic, coded by how strongly the topic maps to an immigration topic:

This graph nicely showcases how hard it often is to interpret LDA topics. Topic 10 is most strongly non-immigration related, with ‘public’ as the most prominent word and many words mentioning families, children, and schools. When you checked these sentences earlier in the challenge code, however, you saw that it was a hotchpotch of various public policies, including education but also environmental concerns and public sector reforms. Topic 5 is most clearly immigration-related, and seems to focus on enforcement and language. When we inspect the sentences in this topics, there seems to be a split with integration sentences focusing on language requirements, and immigration sentences focusing on border security and enforcement. Topics 3 and 9 both mix immigration and non-immigration sentences, with topic 9 focusing on asylum seekers and refugees and topic 3 containing many US-related words.

Looking at the code, we first take the top topic for each sentence, and then compute the proportion of sentences in each LDA topic being in the handcoded ‘other’ topic (note that slice_max uses by=, while summarize uses .by=). Then we join it with the topic_terms computed earlier, select 4 interesting topics, and take the top 8 words for each topic. We then use tidytext’s reorder_within (and scale_y_reordered) to order the words by their loading within each topic, and create a faceted bar plot.

TipIs topic modeling still state of the art?

In the opening lines of this section I wrote that topic modeling “used to be a very popular method for exploring a corpus” – which raises the question of its current status in the methodological toolbox.

The short answer is that it is still somewhat useful for a quick exploration of a corpus, but has lost most of its use as an analytical tool.

The reason for this is possibly that it never quite delivered on its ‘promise’ of finding topics in documents. As the demo above showed, many LDA topics don’t neatly correspond to what we would call topics. The output is also hard to validate if you don’t know what topics you are interested in before the analysis (but see Chang et al. 2009). And if you do have hand coded topics, such as in our PImPo data set, the correlation of LDA topics with manually coded topics is often relatively weak: Even if we take topics 2, 3, 5 and 9 together, these topics still only contain about 57% of all immigration sentences, and even the most clearly immigration/integration related topic (5) still only has a precision of 72% while finding only 15% of sentences. These numbers are clearly worse even than the small dictionary used above – at 57% recall, precision drops to 62%, against 99% precision for a dictionary with similar recall. Note that this is not a fault in the method, but rather underscores that LDA topic modeling is not a very useful technique for finding specific topics known beforehand, but rather for exploring an unknown corpus.

In Chapter 12 we will explore a more modern approach to word clustering using embeddings, which have the benefit of re-using linguistic and pragmatic knowledge from existing language models, so the clustering does not have to ‘discover’ that e.g. immigration and migrant are related terms.

If you do want to use regular topic modeling, it is a good idea to check out Structural Topic Models (Roberts et al. 2014), which allow specifying covariates such as country or year to affect the topic loadings.

Quiz questions

Section 10.1 quiz: from characters to words

Q1: From sentences to words

Given this data frame d:

id text
1 Secure our borders
2 Welcome refugees

Which data frame does d |> unnest_tokens(text, output=word) return?

  • id word
    1 secure
    1 our
    1 borders
    2 welcome
    2 refugees
  • id word
    1 Secure
    1 our
    1 borders
    2 Welcome
    2 refugees
  • id word
    1 <chr [3]>
    2 <chr [2]>
  • id text word
    1 Secure our borders secure
    1 Secure our borders our
    1 Secure our borders borders
    2 Welcome refugees welcome
    2 Welcome refugees refugees
Q2: What makes a token?

These are the tokens that unnest_tokens() produces for two texts:

text tokens
1:30 to go 🙏 #ge2024 #ireland 1 · 30 · to · go · ge2024 · ireland
Sinn Féin’s vice-president sinn · féin’s · vice · president

Which statement about tokenization is true?

  • Tokenization mostly splits text at spaces and punctuation, so whether its tokens match what you consider a word (a time, a party name) depends on your analysis, and you should check the output.
  • The tokenizer recognizes meaningful units such as times and party names, and keeps them together as one token.
  • Tokenization splits text at every space and every punctuation mark.
  • There is one correct way to tokenize a text, so with a good tokenizer such as unnest_tokens() you don’t need to check the output.
Q3: Looking for asylum in German

You tokenize the Austrian sentences and run filter(word == “asyl”) to find sentences about asylum. The sentence Verhinderung von Kettenasylantragstellungen (“preventing chain asylum applications”) is not found. Why?

  • German compounds are not split, so the whole compound kettenasylantragstellungen is one token, which is not equal to asyl.
  • Because unnest_tokens() only works for English text.
  • Because the tokens are capitalized (Kettenasylantragstellungen), and the search is in lower case.
  • Because von is a stop word, so the whole sentence is removed.

Section 10.2 quiz: corpus analysis and comparison

Q1: Getting rid of ‘the’

You have a tokenized data frame words:

id word
1 the
1 asylum
1 seekers
2 to
2 immigration

Which call removes the stop words, keeping asylum, seekers and immigration?

  • words |> anti_join(stop_words)
  • words |> inner_join(stop_words)
  • words |> left_join(stop_words)
  • words |> filter(word != stop_words)
Q2: A different column name

A student tokenizes with unnest_tokens(text, output=token), giving:

id token
1 the
1 borders

Running anti_join(stop_words) on this data frame gives the error by must be supplied when x and y have no common variables. Why?

  • The stop word list has a column word, while the tokens are in a column token, so the join has no column to match on.
  • Because stop_words needs to be loaded with get_stopwords() first.
  • Because anti_join() only works on the output of count().
  • Because the id column does not exist in stop_words.
Q3: Us and them

Consider these two manifesto sentences:

party text
A We must protect our borders against them.
B Newcomers are part of our society.

For which research question would removing stop words (with anti_join(stop_words)) do the most harm?

  • How parties draw boundaries between an in-group and an out-group (we, our, them).
  • Which substantive issues parties link to immigration (borders, society).
  • Whether parties use words like protect and borders.
  • Which words are most frequent in the immigration sentences, apart from very common words like the and to.
Q4: Proportion versus log odds

These are two words from the comparison of Irish and US immigration sentences. The columns us and ie give how often each country uses the word, prop_us is the proportion of uses that are American (us / (us + ie)), and log_odds_us is the weighted log odds for the US (the higher, the more typical for the US):

word us ie prop_us log_odds_us
aliens 9 0 1.00 1.04
our 66 21 0.76 1.85

By proportion, aliens is the more American word, as Irish manifestos never use it. By log odds, however, our is much more typical for the US. Why?

  • Proportion is 1 for any word used by only one country, even a rare one; for a frequent word like our, a big difference is much less likely to be due to chance.
  • The log odds prefers words that both countries use, and gives a lower score to words used by only one country.
  • The log odds measures how many more times the US uses a word: 66 − 21 = 45 times for our, against only 9 for aliens.
  • The log odds gives extra weight to our because it is a stop word.
Q5: Missing zeros

After count(country, word), you have:

country word n
Ireland asylum 32
Ireland border 1
United States border 8

Why does the chapter add complete(country, word, fill = list(n = 0)) before calling bind_log_odds()?

  • Without it there is no row for asylum in the US, so bind_log_odds() does not know that the US used it zero times, which distorts the score for words used by only one country.
  • Because bind_log_odds() gives an error if any word is missing for a country.
  • To remove words that only occur in one country, such as asylum.
  • To make the plot later in the section look better.
Q6: Reading the sign

These rows are taken from immigration_comparison |> filter(country == “United States”):

country word n n_other log_odds_weighted
United States our 66 21 1.85
United States in 75 88 0.00
United States asylum 0 32 -1.86

What can you conclude from these values?

  • US manifestos overuse our, Irish manifestos overuse asylum, and in is used about equally in both.
  • The higher the log odds score, the more often the word occurs, so in is used less often than our.
  • The Irish manifestos use asylum 1.86 times as often as the US manifestos.
  • The word in is typical for US manifestos, as it is used 75 times.
Q7: Custom stop words

To keep party names out of the comparison, the chapter builds a custom stop word list from the party column. Among others, it contains:

word
fine
gael
left
united

What is the main downside of removing these words with anti_join()?

  • Words like left and united are also removed when they are not part of a party name, such as in United Nations.
  • The list will not match anything, because the party names are capitalized in the data.
  • All sentences that mention a party are removed from the analysis.
  • You cannot combine a custom list with the country words, as anti_join() only accepts one stop word list.

Section 10.3 quiz: collocations and n-grams

Q1: Bigrams of a short sentence

Which tokens does tibble(text = “rule of law matters”) |> unnest_tokens(text, output=token, token=“ngrams”, n=2) produce?

  • rule of, of law, law matters
  • rule of, law matters
  • rule of law, matters
  • rule, of, law, matters
Q2: Stop words and bigrams

You have these bigrams in a column called word:

word n
of the 900
in the 700
asylum seekers 60

What happens if you run anti_join(stop_words) on this data frame?

  • Nothing is removed, because no bigram is exactly equal to a single stop word.
  • The rows of the and in the are removed.
  • The stop words are removed from inside the bigrams, leaving of, in and asylum seekers.
  • It gives an error, because the columns have no common names.
Q3: Which bigram is a collocation?

Consider these bigrams and the frequencies of their words in the corpus:

bigram n (bigram) n (first word) n (second word)
of the 900 4000 8000
fianna fáil 40 40 40
will also 30 1200 300

Which bigram will have the highest pointwise mutual information (PMI)?

  • fianna fáil, because its words only ever occur together.
  • of the, because it is by far the most frequent bigram.
  • will also, because it is the least frequent bigram.
  • They all have the same PMI, as PMI only depends on the bigram count.
Q4: Joining collocations

The replacements vector contains patterns (as names) and replacements (as values), for example:

pattern (name) replacement (value)
\bunited states\b united_states
\bfine gael\b fine_gael

A student runs mutate(text = str_replace_all(text, replacements)) on the sentence The United States will act, but the text does not change. Why?

  • The patterns are lower case, but the text is not, and str_replace_all() is case sensitive. The text should be lowercased first with str_to_lower().
  • Because str_replace_all() cannot use a named vector; you need a separate call for each collocation.
  • Because the text needs to be tokenized with unnest_tokens() before you can replace collocations.
  • Because the word boundary marker (\b) in the pattern prevents matches at the start of a sentence.

Section 10.4 quiz: dictionary analysis

Q1: Matching word stems

The immigration dictionary is applied to tokens with str_starts(word, “asyl|immigra|migra”). What does it return for these four words?

word
immigrants
asylum
emigrated
transmigration
  • TRUE, TRUE, FALSE, FALSE
  • TRUE, TRUE, TRUE, TRUE
  • TRUE, FALSE, FALSE, FALSE
  • FALSE, FALSE, FALSE, FALSE — no word starts with all three stems.
Q2: Precision and recall

A dictionary for detecting sentences about the economy gives this confusion matrix on 1,000 hand-coded sentences:

topic FALSE TRUE
economy 120 80
other 780 20

What are the precision and recall of the dictionary?

  • Precision 80%, recall 40%
  • Precision 40%, recall 80%
  • Precision and recall are both 86%
  • Precision 80%, recall 80%
Q3: Joining the sentiment dictionary

Given these tokens, and an excerpt of the Bing dictionary bing:

id word
1 effective
1 enforcement
2 border
2 control
word sentiment value
effective positive 1
illegal negative -1
secure positive 1

The sentiment score of each sentence is computed with: 

tokens |>
  left_join(bing, by = "word") |>
  summarize(value = sum(value, na.rm = TRUE), .by = id)

Why does the chapter use left_join() here rather than inner_join()?

  • With inner_join(), sentence 2 would silently disappear, as none of its words are in the dictionary.
  • With inner_join(), effective would not get a sentiment value.
  • Because left_join() gives all words that are not in the dictionary a value of 0.
  • Because inner_join() gives an error when not all words are in the dictionary.
Q4: A positive sceptic

This sentence was hand-coded as sceptical about immigration:

text direction
We will secure our borders with effective enforcement. sceptical

The Bing dictionary gives it a positive score, as secure and effective are positive words. What does this illustrate?

  • A general sentiment dictionary measures the tone of the words, which is not the same as a position on immigration.
  • The left_join() did not work, so the wrong words were matched.
  • Stop words should have been removed before applying the dictionary.
  • The hand coding is probably wrong, as the sentence is clearly positive.

Section 10.5 quiz: topic modeling

Q1: Casting to a matrix

Given these tokens:

id word
1 secure
1 borders
1 borders
2 secure
2 jobs 
tokens |>
  count(id, word) |>
  cast_sparse(row=id, column=word, value=n)

What does this code produce? (In a sparse matrix, . means zero.)

  • borders secure jobs
    1 2 1 .
    2 . 1 1
  • 1 2
    borders 2 .
    secure 1 1
    jobs . 1
  • borders secure jobs
    1 1 1 .
    2 . 1 1
  • id word n
    1 borders 2
    1 secure 1
    2 jobs 1
    2 secure 1
Q2: A bag of words

Consider these two sentences:

id text
1 Unions fund parties
2 Parties fund unions

What do their rows in the document-term matrix look like?

  • They are identical, because a DTM only stores word counts and ignores word order.
  • They are different, because unions is the first word in sentence 1 and the last word in sentence 2.
  • They are different, because Unions is capitalized in one sentence and not in the other.
  • Both rows are empty, as all words are removed as stop words.
Q3: Terms and topics

From posterior(t) you can pluck() either “terms” or “topics”. After turning them into a long tibble, you get, for example:

lda_topic term beta
5 law 0.030
5 english 0.014
id lda_topic gamma
226795 5 0.93
226795 1 0.01

Which statement is correct?

  • The beta values say how important each word is for a topic; the gamma values say how much of each document belongs to each topic.
  • The beta values say how much of each document belongs to each topic; the gamma values say how important each word is for a topic.
  • Document 226795 belongs only to topic 5, as each document is assigned to exactly one topic.
  • The word law occurs 0.03 times in topic 5.
Q4: A different seed

You run the topic model twice, with a different seed, and look at the top words:

run lda_topic top words
seed = 99 9 asylum, seekers, health, services, community, …
seed = 123 10 asylum, seekers, rights, women, human, …

What does this tell you?

  • Probably nothing much: topic numbers are arbitrary, so the same cluster can get a different number in each run.
  • The model found completely different topics, so one of the runs is wrong.
  • The seed determines the number of topics, so the second run has more topics.
  • The asylum topic is less important in the second run, as it has a higher number.
Q5: Finding immigration sentences

You want to find all sentences about immigration in a corpus of manifestos. In this chapter, two methods were compared on the PImPo sentences:

method precision recall
dictionary 99% 56%
LDA topics 2, 3, 5 and 9 62% 57%

What is the best conclusion?

  • Topic modeling is mostly useful for exploring an unknown corpus, not for finding a topic you already know you’re looking for.
  • LDA would beat the dictionary if you used a larger k, such as 20 topics.
  • LDA is better, because it does not need a hand-crafted word list.
  • Both methods are good enough, so it does not matter which one you use.