9  Dealing with Text: stringr

Many interesting (computational) social science data sets have textual elements. This can e.g. be the contents of online posts or reviews, official documents, reports, or party manifestoes, but also open survey questions or question metadata.

Very often, we need to clean, preprocess and/or analyse this textual data. This chapter will take the first steps in this by introducing the tidyverse stringr package, which can handle most of the cleaning functions such as normalizing, filtering and extracting parts of textual columns. In the next chapters we will look at other parts of the pipeline: analysing Text as Data (Chapter 10), using LLMs as an analysis tool (Chapter 11), embeddings and clustering (Chapter 12), and dealing with image data (Chapter 13).

Chapter objectives

In this chapter, you will learn how to use various stringr functions for dealing with textual data:

• Basic text cleaning operations (Section 9.1)
• Filter data based on the occurrence of specific text (Section 9.1.1)
• Using basic ‘regular expression’ patterns to find, replace or remove parts of a text (Section 9.2)
• Making simple word plots (Section 9.3)
• Optionally, learn regular expression syntax in more detail to make powerful patterns (Section 9.4)

As always, the chapter ends with a multiple choice quiz organized per section that you can use to test your knowledge and understanding. You can also expand the table below to see which functions are covered in this chapter.

NoteFunctions introduced in this chapter

Function	What it does	Section
str_detect()	does the text contain the pattern?	Section 9.1.1
str_glue()	build a text, filling in R code between curly braces	Section 9.1.1
str_c(collapse=)	collapse all values of a vector into a single string	Section 9.1.1
str_to_lower()	convert text to lower case	Section 9.1.1
str_length()	the number of characters	Section 9.1.2
str_count()	count how often the pattern occurs in each text	Section 9.1.2
str_trim()	remove whitespace at the start and end	Section 9.1.2
str_squish()	trim and normalize all whitespace in the text	Section 9.1.2
str_extract()	extract the first match in each text	Section 9.2.1
str_remove_all()	remove all matches	Section 9.2.2
str_replace_all()	find and replace	Section 9.2.3
fixed()	treat the whole pattern as literal text	Section 9.2.3
str_extract_all()	extract all matches in each text	Section 9.2.4
str_view()	show the matches of a pattern	Section 9.4.1
str_c()	combine texts into one	Section 9.4.2
str_subset()	only the texts that contain the pattern (for a vector, not a data frame)	Section 9.4.2
regex()	wrap a regular expression to specify options	Section 9.4.2

Data for this chapter: US Elections on Bluesky

As an example data set for this chapter, we will use a collection of Bluesky posts about the 2024 US elections (see the book’s github repository for the R code used to scrape these posts). To make things easier, we will only keep the date, author and text columns:

9.1 Using stringr for basic text cleaning and processing

As a first step, we will look at a number of functions from the stringr package for detecting and counting whether a text contains a specific word or substring. This is very useful for cleaning and exploring data.

Who do people post about? Using str_detect in filter

A first question we might ask is how much people post about a candidate, for example Donald Trump. We can use str_detect to check whether a text contains a specific substring, and use that to filter the posts:

Tipstr_glue

In the example above, we used str_glue to display the number of posts in a nice way. You’ve actually already seen this in Chapter 6, where we used it to create a nice label in a graph.

str_glue is a very useful function for building a text by ‘glueing’ calculated values (such as {nrow(posts)}) into a text. Technically, all R code within curly braces ({}) will be evaluated and inserted into the text.

A small caveat: if the code produces multiple values, this will result in multiple texts being produced. This is useful if you use str_glue within a mutate call, as it will produce a value for each row, but might be surprising if you use it to quickly print something like above. You can use str_glue("Count to three: {str_c(1:3, collapse=',')}") if you want to include multiple values in a single string: this ‘collapses’ multiple values (1, 2 and 3 in this case) into a single string "1,2,3".

As you can see, about 25% of the posts mention Trump – or rather, contain the specific sequence of characters T-r-u-m-p. This will not find trump, but will find #Trump, DonaldTrump. To also find matches of trump, an easy trick is to call str_to_lower(text) to lowercase the whole text column:

Which finds about 300 extra mentions (which could of course also include false positives, e.g. trump used a verb or trumpet as a musical instrument)

NoteAll the str_ functions

As you might have noticed, all functions introduced above started with str_: str_detect, str_glue, and str_to_lower. This naming convention holds for (almost) all functions in stringr. This makes it easy to find them in Rstudio: just type str_ and press tab or control_space, and you get an overview of all the functions for dealing with text. See the table near the end of this chapter for an overview of useful stringr functions.

Basic string functions in a data frame

Next to filtering, it can be useful to use stringr within a mutate call to calculate properties of the data. For example, we could count the length of all posts, and check if longer posts get more likes:

We can also use str_count to count how often a fragment occurs within a text. If we count the number of spaces, we get a decent approximation of the number of words in each post (but see the callout below on characters vs words).

Before we do this, we should probably normalize the whitespace in the posts. The function str_trim can be used to remove (‘trim’) all whitespace at the start and end of a text. We will go a step further and use str_squish, which trims the text but also replaces all consecutive whitespaces by a single space, i.e. it replaces newlines, tabs, multiple spaces or any combination of these by a single space.

Exercise: Talk of the town?

A single count over the whole three months hides a lot: does attention to a candidate change as the campaign goes on? We already have all the tools we need to find out – str_count from the previous section, and floor_date, group_by/summarize and ggplot from earlier chapters.

The US election took place on November 5, 2024 – right in the middle of our three months of data. Count how often Trump is mentioned per week, to see whether attention to him changes over the course of the campaign. Ignore the difference between lowercase and uppercase, i.e.  count trump or TRUMP as well as Trump.

TipHint:

• Which function did we use earlier in this chapter to make matching case-insensitive?
• floor_date() rounds a timestamp down to a chosen unit – which unit do you need to get one value per week?
• str_count() tells you how often a pattern occurs within a single text. To get the total per week, you still need to add those numbers up per group, e.g. with group_by(week) |> summarize(n = sum(...)).

Important🎓 Solution:

First lowercase the text so that case doesn’t matter, then round each timestamp down to the week with floor_date(), and finally add up the mentions per week:

counts_per_week <- posts |> mutate(week = floor_date(date, unit = "week"), mentions = str_count(str_to_lower(text), "trump")) |> group_by(week) |> summarize(n = sum(mentions)) print(counts_per_week, n=15)

counts_per_week <- posts |>
  mutate(week = floor_date(date, unit = "week"),
         mentions = str_count(str_to_lower(text), "trump")) |>
  group_by(week) |>
  summarize(n = sum(mentions))
print(counts_per_week, n=15)

Note that str_count() counts mentions, not posts: a post that says “Trump” twice adds 2 to the total. That’s a deliberate choice here (we want the volume of Trump-talk, not the number of posts about him), but it does mean a single very repetitive post could inflate a week’s count.

Extend the analysis to compare Trump and Harris:

• Add a second column counting mentions of “Harris” per week (again, case-insensitive).
• Reshape the result so you have one row per week per candidate – see pivot_longer in Section 7.1.3 if you need a refresher.
• Plot both lines in a single graph, using colour to distinguish the two candidates.

As an additional challenge, compute the number of posts (instead of total mentions) for each candidate, and/or the share of posts that mention each candidate.

Some questions to think about:

• Do the two candidates get similar amounts of attention throughout the campaign, or does one dominate? What could cause this?
• Bluesky’s userbase leans strongly to the left compared to the wider US population. How might that affect how well this pattern would generalize to, say, X/Twitter data from the same period?

TipAnalysis of texts: characters or words?

In the examples and exercise above, we use the str_ functions to perform simple text analyses by counting mentions of candidates.

This works, but has very clear limits because stringr fundamentally works on characters, while a lot of our analyses talk about words. As mentioned, str_detect(str_to_lower(text), "trump") will match trumpet (as well as "#trump", "STRUMP" and "Strumpfhose"). The reason is that stringr sees Strumpfhose as a sequence of characters S-t-r-u-m-p-f..., and these characters contain the sequence t-r-u-m-p, so it matches. stringr doesn’t really know what a word is, and doesn’t really care either.

This can be solved to a degree by using more sophisticated matching rules, as explained below, but these solutions are relatively brittle and address the sympton rather than the cause: often, we would like to analyse words rather than characters. Chapter 10 will address this problem head-on by first tokenizing a text into words, and then using analyses on the words.

None of this makes the stringr package any less relevant: Functions like str_detect, str_trim and str_count are still very useful for filtering and cleaning texts – and in fact many of the analyses in Chapter 10 will use these functions either before or after the tokenization to clean the text and analyse the extracted words.

9.2 Finding, removing and replacing simple patterns

The code above showed how you can detect and count literal substrings. The stringr functions are much more powerful than that, however, as you can also search for patterns rather than literal strings.

For now, we will just introduce some very basic patterns, that are however already quite powerful. For example, the code below uses the patterns Trump|Vance and Harris|Walz, which detect mentions of either of the names:

So, it looks like the republican ticket gets more attention on Bluesky than the democratic ticket - does that necessarily mean that coverage is more favourable for the Trump ticket?

Extracting a pattern with str_extract

Another pattern is #\\w+, which matches any word starting with a #.
Using the function str_extract, we can use it to extract the (first) hashtag in each post, and store it as a new column:

As you can see, the corpus is dominated by the hashtag #Election2024. This is actually very unsurprising, since that tag was used to select the posts, so each post will contain this tag.

Note that the reason why the total is lower than the total number of posts is that the code above only extracts the first occurrence of a pattern, so in the other posts another tag was used before that tag (and see Section 9.2.4 for str_extract_all to extract all tags)

TipRegular expressions?

The pattern introduced above (#\\w+) may look weird, overly complex, or suspiciously like a cartoon curseword. But it’s actually a very standardized and useful way to express character patterns called regular expressions (or regex, for friends)

Although this will be explained in much more detail in Section 9.4, it is useful to explain at least a little bit of what is going on. In a regex, most characters simply match that character, which is why we could use "trump" earlier to simply search for the word “trump”, or actually the sequence of characters t-r-u-m-p.

We also saw the special character |, which was used to list multiple alternatives. So, angry|mad|annoyed would match any of these strings, but be aware that they match characters, and not words: This pattern would also match armada, since it contains the characters m-a-d. See Section 9.4.2 below for a solution.

The \\w used in the last example, functions as a kind of wildcard called a character class. Rather than matching a literal sequence of \ and w characters, it matches ‘word characters’ defined as letters and numbers (and curiously the underscore _). So, #\\w would match a literal hash sign (#), followed by a letter, number, or underscore. Similarly, \\d matches digits and \\s matches whitespace.

The last part, +, is a quantifier which reads “one or more”: it means that the preceding character (in this case, the \\w word character class) is required but may be repeated as often as desired. So, the total pattern #\\w+ matches a literal hash sign, followed by one or more word characters, which is a fair approximation of a hash tag.

By using combinations of character classes and quantifiers, you can make patterns for many useful things such as hashtags, zip codes, phone numbers, email addresses, etc. See Section 9.4 below for more detail.

Removing parts of a text with str_remove_all

Since the occurrence of this particular hashtag is not informative, let’s clean the data first by lowercasing it, and then removing that hashtag using str_remove_all, followed by the same code as above:

This is more informative: a number of posts had no other hashtags (giving a NA for the extraction), and after the somewhat generic #politics and #vote tags we see hashtags for the two main candidates, mostly balanced.

Replacing text with str_replace_all

Next to removing text, we can replace parts of a text with str_replace_all(text, pattern, replacement). For example, we can ‘anonymize’ the at-mentions by replacing all word characters after an at sign by an ANONYMIZED placeholder (using str_squish to make it easier to read the results):

Note that this only replaces the at-mention until a non-word character appears. Since bluesky names often contain domain names (e.g. @example.bsky.social) this will only replace the first part and leave the domain intact, which might or might not be a desired effect. For more powerful ways of specifying patterns, see Section 9.4 below.

TipMatching periods: escaping and fixed()

As noted above, the \\w pattern matches letters, numbers, and underscore, but not the dots in a domain name. Confusingly, the period character (.) also does not (just) match a literal dot: it is a character class itself, and matches every possible character (apart from line breaks). So, @\\w+.bsky.social matches all at-mentions in this domain, but would also match @\\w+_bsky_social and even @\\w+ bsky social.

For this case it’s probably not a big deal, as those patterns are unlikely to appear. But let’s try searching for U.S.:

As you can see, this gives 130 matches. But not all of them actually contain the string U.S., for example the 23rd match printed here. Why did that post match?

The answer, of course, is that . is a wildcard, so the pattern U.S. matched the word RUSSIAN!

To remedy this, you have two options. The easiest is to enclose the whole pattern in a fixed() function: str_detect(text, fixed('U.S.')) tells R to actually not use a regular expression pattern, but a fixed substring. Note that within a fixed() everyhing matches literal characters, including | and +. To match literal periods within a more complex pattern, you will have to to place a double backslash before each period: U\\.S\\.|U\\.\\K\\., a practice known as escaping (again, see Section 9.4 below for more detail). Try either fixed('U.S.') or U\\.S\\. in the code above - it should return 102 rows instead of the original 130.

Note that the same logic holds for other ‘special’ characters, such as brackets (both () and []) and the quantifiers + and *, so you also need to either wrap the whole pattern in fixed() or escape each of those characters if you want to match literal brackets or plus-signs.

Finding multiple occurrences with str_extract_all

Earlier, we tried to analyse hashtag usage in the US Election posts, noting the limitation that str_extract only extracts the first occurrence of a pattern.

After seeing the examples with str_remove_all and str_replace_all, you may have guessed that there is also a function called str_extract_all – and you would be right.

The complication is that this of course can return multiple results per row, which does not really fit with the idea of a column in a data frame: a cell normally contains only a single value. This can be solved by using unnest_longer, for example to do a complete count of all hashtags:

Technically, str_extract_all produces a ‘list column’, i.e. a column with a data type list. Since that is not very useful for analysis, we unnest the list column into a long format: every value becomes a row, so if two hashtags were found the row will be repeated for each hashtag, with only the tag column being different and containing a single tag.

To make this more explicit, the following toy example shows the results for each step:

Exercise: Extracting urls from posts

An essential role of social media in political communication is sharing links to news and other websites. Let’s see if we can track which .com domains get shared most.

The code below extracts a pattern and shows the most frequent occurrences. Your job is to fill in the pattern. For this exercise, we will only find .com domains. Can you make a pattern for urls that start with https:// and end with .com/?

TipHint:

• str_extract(text, pattern) pulls out the (first) match of pattern in each post – posts without a link will get an NA, which is fine, we’ll just see it as one of the rows.
• The pattern itself: a literal https://, then “anything” (.*), then a literal .com/.

Important🎓 Solution:

posts |> mutate(url = str_extract(text, 'https://.*.com/')) |> count(url) |> arrange(-n)

posts |>
  mutate(url = str_extract(text, 'https://.*.com/')) |>
  count(url) |>
  arrange(-n)

youtube.com comes out on top, with x.com and theguardian.com next.

Look at the table you just made. Two things about it should bother you:

• The code above only finds the first url in each post. Can you modify it to find all urls? Does it change the results?
• Try str_detect(text, "https://") to count how many posts really have a link, and compare to the domains found above. What causes many links to be missed? How could we solve this?
• The pattern contains .com, but you’ve already seen that . doesn’t actually mean “a literal dot” – it means “any character at all.” So strictly speaking, this pattern doesn’t require a dot before “com” either. Why doesn’t that cause a visible problem in this dataset? Can you think of a (hypothetical, or real) link where it would?

Both are fixable, but not with the patterns we discussed so far. Section 9.4 will explain this in more detail, but try the pattern https://.*\\.(com|net|org|be|social)/. Can you understand how this regex works?

9.3 Word plots with ggrepel

Word clouds are a staple of (bad) text analysis blogs, but that doesn’t mean we should never make them.

A simple word cloud

In ggplot, a word cloud is simply a normal plot, with an aesthetic mapping for the dimensions of the data we want to use. The main difference is that we use the geom_text_repel from the ggrepel package which ‘repels’ words to try to avoid them overlapping each other. For example, using the hashtags data frame created earlier, the following code plots a traditional word cloud:

Technicalities aside, how informative is the plot we made above? Could we have made a bar plot with the same information? Would that make it easier to see what’s going on, or harder?

NoteWhy are word clouds so complicated?

You might have noticed we are doing quite a lot of work to produce a word cloud:

• set.seed(1) is used to fix the random seed so the plot comes out (mostly) deterministically.
• we filter out #election2024 as this tag was used to collect the data, so every post in the data set will contain that tag.
• we manually calculate a semi-random x and y coordinate for each word, where we use rnorm(n()) to create a normal distribution around 0, and then divide by sqrt(n) to nudge more frequent words closer to the center of the plot.
• The ggplot is fairly standard apart from the geom_text_repel. This geom essentially places each word on its x and y position, checks if it overlaps a previous word, and then starts moving it away from that word until it doesn’t overlap anymore (or it gives up).
  
• We set min.segment.length=Inf to suppress the use of segments. By default, ggrepel will draw a line from the label to the actual x and y position where it started. We don’t want to see those lines so we set the minimum length to infinity.

The main reason for this is that the tool we introduced, ggrepel, is meant more for flexible plotting of texts than for pure word clouds. As we will see below, we can use it to encode more meaning in word plots, but it is also very useful for annotating e.g. scatter plots by labeling specific points, without creating overlap on the labels. In fact, this is why the default draws a line from the ‘repelled’ label to its intended position.

If all you want is a word cloud, ggwordcloud can probably do this more easily (and perhaps more beautifully). But we hope that after reading this section you won’t actually (just) want a word cloud!

A more informative word plot

The main criticism of word clouds is that they only use a single dimension of the data (frequency), and the rest of the variation (e.g. on the x and y axes) is just random noise. However, we can also use other dimensions of the data, such as the date a word is used or the topic or cluster it was used in. For example, the code below creates an over-time usage word plot of the hashtags dataset created above:

What can you see in this plot? Do you see different tags before and after the election? Is this plot more informative than a ‘regular’ word cloud?

This code looks very complicated, but most of it builds on the code used earlier. Have a good look at the individual steps, which should all look familiar.

That said, we used some particular tricks to make this graph informative:

1. We calculate the median date for each word. This has a tendency to place words near the middle, so the modal date or the date in which it was most dominant would be good alternatives. We also count distinct authors per tag to avoid having a single author dominate the findings.
   
2. Since most posts occur around election day, plotting the same top-75 most frequent tags will cause most tags to crowd around that day, and leave especially the post-election coverage almost empty. For that reason, we take the top-5 tags per week rather than the top-75 tags in total, assuring an even distribution over the graph.
3. We tweak the geom to give it more time to converge. Since it functions by moving each individual word until it reaches a place without overlap, it can take quite a number of iterations (‘nudges’) before each word found a suitable location (and even with these settings, it’s quite possible there will still be some overlap). We also set direction='y' - this causes the words to be nudged only on the vertical axis, guaranteeing that they are displayed on the correct date.

9.4 A deep dive into regular expressions

In the sections above we have seen a number of stringr functions to deal with textual data in R. For example, we’ve used str_detect to filter data on whether a term occurs in it, str_replace_all and str_remove_all to replace or remove specific words or patterns, and str_extract to extract patterns such as hashtags. As explained above, these functions all use regular expression patterns as their driver, but although we used some patterns (such as #\\w+ for hashtags), we did not really spend a lot of time on how regular expressions work.

This section will remedy this, and by the end of this section you should have a very good idea of how regular expressions work. You will also get a fairly complete overview of all the various character classes and other options.

This section uses str_view as an important visualization and debugging tool: it simply shows all matches of a pattern in a specified input, making it easy to check and debug regular expressions. There are also many useful online tools for analysing regular expressions, such as regex101.com, which explains the components of a pattern and lets you test your own patterns, and visualizers such as regex-vis.com, which show a visualization of a pattern. Note that for such tools you should use single backslashes rather than double backslashes, so #\w instead of #\\w.

Note that this section will be a bit more of a reference work and explainer than an interactive tutorial. So, you might wish to skip the section now, and simply return to it when you need to use more complicated patterns. But if you were getting curious about questions like “how do we match words instead of characters”, “what other things are there except for \w and \d”, or “can we make a perfect pattern for matching emails”, read on!

TipSo how do regular expressions actually work?

Each element of a regular expression pattern can have one of three functions:

• It can specify what to match, e.g. a literal character (e.g. w matches “w”) or a character class (e.g. \\w matches any word character).
• It can anchor a match at a specific position: ^cat matches “cat” only at the start of the text, and \\bcat matches “cat” only as the start of a word.
  
• It can quantify the preceding character: + means one or more of the preceding match, so w+ is one or more “w”s, while \\w+ means one or more word characters.
• It can combine parts of a pattern using parentheses () and |-alternatives

These can be combined in various ways. For example, you can read #(trump|vance)\\d+ as “A literal hash sign, followed by either trump or vance, followed by one or more digits”, so it would match e.g. #trump2024 or #vance24.

The remainder of this section will discuss each of those functions in turn, and end with a reference list of stringr functions and regular expression syntax.

Specifiers: Matching literals and character classes

In the earlier sections of this chapter, we’ve seen that alphanumerical characters match that literal character, so c matches “c” and cat matches “cat”.

We’ve also introduced character classes such as ., which matches everything (except newline), and \\w which matches word characters. There are a couple more classes like \\w, such as \\d for digits and \\s for whitespace. They can also be negated by capitalizing them: \\W matches everything except for word characters.

A flexible way to make character classes is to use square brackets: [abc] matches a, b, or c, and [A-Z] matches all capital letters (but not accented letters like Ç or É). You can also negate these patterns: [^A-Z] matches everything except capital letters.

Every specifier can be quantified, so \\w+ means one or more word characters, and [A-Z]{2} means two upper case Latin letters. We can use this for example to recognize things like at-mentions, email address, and zip codes:

Note: this uses the str_view function to show all matches in the specified input texts. The option html=TRUE creates html output that works well in this interactive website, and match=NA indicates that we want to see all input texts, even if they didn’t match the pattern.

Can you edit the chunk above to make patterns for:

• American zip codes (5 digits)
• Extended American zip codes (5 digits, a dash, 4 digits)
• Dutch postal codes: 4 digits, a space, two upper case Latin letters
• Bluesky mentions with domain: An at-sign, one or more word characters, a literal period, one or more word characters
• Generic mentions: An at sign, and one or more word characters or literal periods. Hint: you can use abbreviations like \\w within a []-character class. Make sure to escape the period, even within a []-class.
• Simplified email: One or more word characters or periods, an at sign, one or more word characters or period, a literal period, and two or more word characters (hint: two or more can be specified as one followed by one or more)

What would be needed to find at-mentions without also matching the second half of the email address?

Another powerful way to specify characters is using \\p{} for Unicode categories. For example, use \\p{letter} for letters (including accented or non-Latin letters), \\p{uppercase_letter} for upper case letters or \\p{punctuation} for punctuation. Besides the general categories listed in the link above, there are also properties for specific scripts. For example, \\p{hangul} matches letters in Korean script. Other properties include \\p{extended_pictographic} which matches emoji (confusingly \\p{emoji} also matches digits as these can be used in constructing emoji like 1️⃣). Unicode properties can be negated by capitalizing: \\P{letter} matches everything except for letter characters

We saw earlier that you need to escape special characters such as periods: . matches all characters, but \\. matches a literal period, and the same for \\*, \\[ etc. Confusingly, to produce a literal backslash you need to use \\\\: you escape the backslash itself, and then double up all the backslashes. The reason you need to double the backslashes is that R also parses the string before passing it on to the regular expression engine, so R parses \\\\ as two backslashes, which is then interpreted by the regex engine as a literal backslash. To ease this burden for complex patterns you can also use raw strings: r"(U\.S\.)" is read by R without interpreting the escapes, so it results in the string U\.S\..

As an example of how hard it can become to read heavily escaped text, let’s search for shrug emoticons in the Bluesky posts:

Can you rewrite this to use fixed() or r"(...)"?

Anchors: Word and text boundaries

Earlier we showed that you could use | to make a simple dictionary of terms to match. However, something like angry|mad|annoyed is problematic as it matches characters, not words. So, mad also matches “armada”. You could of course start the pattern with a whitespace character, but that would fail if it is at the start of the text or following punctuation.

To solve this, we can use \\b to match a word boundary, defined as the boundary between a word character and a non-word character (or the start or end of a text), which for languages that separate words with spaces works fairly well.

So, \\b(angry|mad|annoyed)\\b would match these three words only if they are separate words, and if you omit the closing \\b it will match any words starting with these terms. Let’s build a dictionary containing a set of Trump’s favorite words:

Now, we can use that dictionary with str_extract or str_count as above to do a sentiment analysis. But let’s first check the top 10 matches of this dictionary to do a face validity check:

In this example, we first use str_subset to select those posts$text that contain the pattern, then take the top-10 with head and display the matches with str_view as above.

Did it look like the dictionary was correctly capturing posts mentioning negative emotions? Can you improve it?

If you replace head(10) by sample(10), you get a new sample of matched posts every time you run it. Does that help in estimating the validity of the dictionary?

Can you count the percentage of posts that contain a ‘sad word’? Hint: use mutate and str_detect or str_count to determine matches, and then summarize to compute the mean.

Next to the word boundary anchor, there are two special anchors that match only the start of a text (^) or the end of a text ($). So, if you want to know which posts start with ‘RT’, you can use ^RT to match those two characters at the start of a post only (an older Twitter convention to mark a retweet).

Note that it may seem that anchors are very similar to character classes. However, what sets them apart is that anchors don’t actually match (or consume) a character. Instead, they point to a point between two characters. For that reason, you also cannot quantify anchors, and within [] their meaning shifts: as we saw earlier, the ^ in [^A-Z] actually negates the character class, and the $ in [0-9$] is just a literal dollar sign.

Noteregex flags

We noted earlier that . matches everything except newlines. In fact, whether it should match newlines or not is controlled by an option you can set. To set it, you need to wrap your pattern in a regex() function. So, while normally <script>.*</script> would only match a script that is fully contained on a single line, regex("<script>.*</script>", dotall=TRUE) sets the dotall flag, which causes . to also match newlines. Two other useful flags are multiline, which if set makes ^ and $ also match the start and end of lines; and ignore_case, which makes the regular expression ignore case.

These options seem very useful, but in most cases preprocessing your data with str_squish and/or str_to_lower achieves the same aims while re-using a general part of your text cleaning pipeline. So practically, the regex flags are not actually used that much.

Quantifiers: How many cats do we need?

We earlier used #\\w+ to match hashtags, reading the + as “one or more (word characters)”. + is an example of a quantifier, which indicates how many of the preceding part of the pattern should be matched. Next to +, you can also use *, meaning “zero or more”: #trump\\d* would match #trump24, #trump2020 but also plain #trump as the digits are optional.

In addition, ? means “zero or one”, making the preceding element optional. So, if we are afraid that people might misspell Tim Walz’ last name, we could use walt?z, meaning w-a-l, followed optionally by a t, followed by a z.

Finally, you can specify exactly how many matches you expect using curly braces: \\d{2} means exactly two digits, \\d{2,4} means two to four digits, and \\d{2,} means two of more digits. Note that you can also apply quantifiers to character classes, so e.g. [A-Z]{2} means exactly two upper case (Latin) letters.

Exercise: British postal codes

Let’s put quantifiers to work by making a pattern for British postal codes. Valid British postal codes consist of three parts:

1. First, one or two upper case letters specify the name of a city or area (e.g. M for Manchester or EH for Edinburgh)
2. Next, the district in or near the city is specified with either one digit, two digits, or (in London) a digit plus a letter. This is appended directly to the city code, so EH8 is an area in or around Edinburgh, and EC1A is a district in east central London.
3. Finally, a second part separated by a space marks a specific unit tailored to the mail routes. This consists of a digit (the sector), followed by two upper case letters.

So, EH8 9YL, M13 9PL and WC2A 2AE are all valid postal codes (corresponding to the addresses of British universities - can you guess which?), but 8EH 9YL, M13A 9PL and WC2A AE2 are not.

Can you change the pattern below so it matches all valid codes, without matching any invalid codes? Note the use of anchors (^ and $) to ensure the pattern must match the whole example text. When you submit, we will also test your pattern on a number of other valid and invalid codes.

TipHint:

• Build the pattern one part at a time, and keep the ^ at the start and the $ at the end.
• One or two upper case letters: [A-Z] with a quantifier that allows one to two repetitions.
• The district always starts with a digit, which can be followed by one optional digit or letter. A character class can combine both: [A-Z\\d]. Which quantifier makes something optional?
• The second part is simply a digit followed by exactly two upper case letters.

Important🎓 Solution:

The pattern consists of the three parts described above, separated by a space before the last part:

• [A-Z]{1,2}: one or two upper case letters for the city or area
• \\d[A-Z\\d]?: a digit, optionally followed by a second digit or a letter
• \\d[A-Z]{2}: a space, a digit, and two upper case letters

The anchors ^ and $ make sure the pattern has to match the whole text, so that e.g. EH8 9YLX is not accepted just because it contains a valid code.

valid <- c("EH8 9YL", "M13 9PL", "WC2A 2AE") invalid <- c("8EH 9YL", "M13A 9PL", "WC2A AE2", "eh8 9yl", "EH8 9YLX") examples <- c("Valid codes (should be matched):", valid, "Invalid codes (should not be matched):", invalid) pattern <- "^[A-Z]{1,2}\\d[A-Z\\d]? \\d[A-Z]{2}$" str_view(examples, pattern, html=TRUE, match=NA)

valid <- c("EH8 9YL", "M13 9PL", "WC2A 2AE")
invalid <- c("8EH 9YL", "M13A 9PL", "WC2A AE2", "eh8 9yl", "EH8 9YLX")
examples <- c("Valid codes (should be matched):", valid,
              "Invalid codes (should not be matched):", invalid)
pattern <- "^[A-Z]{1,2}\\d[A-Z\\d]? \\d[A-Z]{2}$"
str_view(examples, pattern, html=TRUE, match=NA)

Note that this pattern checks the format of a postal code, not whether the code actually exists.

Suppose we would look for a .com website domain using a pattern like https://.*\\.com. This may look reasonable, but in practice this can lead to unexpected results:

As you can see, this matches from the first https:// to the second .com, rather than matching each URL separately. The reason for this is that quantifiers are by default greedy. This means that, given the choice, they try to match as much as possible. Since it can validly match .* all the way to (just before) the second .com, this is what it will do.

The remedy is to add a question mark next to the quantifier: .*? is a non-greedy version of .*. Try changing it in the code above!

Note that https://.*?\\.com is still a pretty bad pattern (try changing vanatteveldt.com to vanatteveldt.nl above to see what happens). A better pattern would probably be adding a word boundary anchor next to it and/or restricting the character class to e.g. only words and periods: (https://[\\w.]+\\.com\\b). Note that the . within a []-class does not need to be escaped, and that the ? is no longer needed as the restricted character class already prevents a ‘runaway’ greedy match.

Groups and alternatives

Earlier, we’ve seen that #(trump|harris)\\b would match either #trump or #harris. This is a combination of two mechanisms: | is used to indicate two or more alternative patterns that can be matched, and () groups these alternatives so the parts outside the brackets (# and \\b) are not included in the scope of the |.

Another use of groups is that they can be quantified. So, #(lol)+\\b would match “#lol” as well as “#lollollol” (but not “#lolll”), and #trump(2024)?\\b would match either “#trump” or “#trump2024”.

Finally, groups can be used in str_extract to specify what should be extracted. For example, let’s construct a (somewhat complicated) pattern to extract the domains that are linked to in the Bluesky posts:

This pattern looks complicated, but as always if you read it step by step from the left, it starts making sense:

• First, the start of the match needs to be a ‘word’ starting with either “www.”, “https://”, “http://” (with ? making the “s” optional).
• Next, we allow one or more subdomains to be specified with [\\w.-]+, i.e. one or more word characters, dots, or hyphens, and the whole subdomain string has to be followed by a dot. This whole part is made optional: the ? quantifier applies to the whole parenthesized group.
• The next part is the actual domain, which we define here as one or more word characters or hyphens, followed by a dot, followed by two or more word characters.

This is then merged into a single pattern with str_c. Note that this is equivalent to writing the whole expression as a single pattern directly, but by splitting it up it remains at least somewhat readable.

As you can see, the actual result of str_extract in this case is not the whole hyperlink, but only the domain part. The reason for this is that we wrapped the domain part in parentheses, turning it into a group; and then we specified group=3 in str_extract, which points to this group. It is the third group because both start and subdomains also use groups (to scope the | and ?), so these are groups 1 and 2.

Two final things to note:

• You can also use so-called non-capturing groups by using (?: ) instead of (). For example, try changing start to \\b(?:www\\.|https?://) in the code above, and do the same for subdomains, and then change the group into group=1 – it should give the same result. The benefit is that the number of the group we care about (domain) no longer shifts around depending on the rest of the pattern, but perhaps at the cost of further complicating the pattern.
• Capturing groups can also be used in str_replace and str_detect by using \\1 to refer back to group 1. For example, str_replace_all(text, pattern, "[\\3]") would use the pattern above to replace all hyperlinks to their domain contained in square brackets, so “Watch https://www.youtube.com now!” becomes “Watch [youtube.com] now!”, which could be useful for e.g. anonymizing or normalizing posts or open ended survey responses.

NoteWhat is ‘regular’ about these expressions?

As a complete aside: Why are ‘regular expressions’ called ‘regular’?

The answer goes all the way back to the early days of computers and computer science in the 1950s. A mathematician named Stephen Kleene proposed ‘regular languages’ and proved that they can be identified by what is called a ‘finite state machine’: a simple program that has a finite number of states, and can go from one state to another by consuming an input.

What this means for regular expressions is that we can visualize a regex as a finite state machine, which provides a nice visual idea of what a pattern does. For example, the pattern #(kam[ae]la|walt?z)\\d* can be visualised as follows:

flowchart LR
  S(( )) -->|"#35;"| H(( ))
  H -->|k| K1(( ))
  K1 -->|a| K2(( ))
  K2 -->|m| K3(( ))
  K3 -->|a| K4(( ))
  K3 -->|e| K4(( ))
  K4 -->|l| K5(( ))
  K5 -->|a| F((( )))
  H -->|w| W1(( ))
  W1 -->|a| W2(( ))
  W2 -->|l| W3(( ))
  W3 -->|t| W4(( ))
  W4 -->|z| F
  W3 -->|z| F
  F -->|"\d"| F

So, you start from the left by matching a ‘#’. Then, the pattern diverges: you match either a ‘k’ or a ‘w’, and then you have to continue in that branch. Options or alternatives ([ae] or t?) are depicted as small subbranches, and the final zero or more digits (\\d*) is visualized by optionally going into an infinite loop of \\ds, with the double circle indicating valid end nodes.

Because regular expressions can be matched in a single forward pass through the text, regular expression matching is generally quite efficient (with the small note that later additions such as the \\1 backreferences briefly mentioned in Section 9.4.4 actually mean that modern regular expression engines work a bit differently, so some patterns can actually cause matches to be somewhat slow)

9.5 Regular expression reference and example patterns

Below are tables with an overview of the most commonly used functions and regular expression syntax. See also the stringr cheat sheet.

Example patterns

Below are some examples of regular expressions for common patterns like dates or HTML tags.

Table 9.1: Example patterns

Goal	Pattern	Example
US zip code	\\d{5}	“90210”
US phone number	(\\d{3}-)?\\d{3}-\\d{4}	“202-456-1111”, “456-1111”
Dutch postal code	\\d{4} ?[A-Za-z]{2}	“1015 GK”
ISO date	\\d{4}-\\d{2}-\\d{2}	“2020-07-20”
German date	\\d{1,2}\\.\\d{1,2}\\.\\d{4}	“25.6.1988”
Year (1900–2099)	\\b(19|20)\\d{2}\\b	“2024”
Percentage	\\d+(\\.\\d+)?%	“48.3%”
Amount in dollars	\\$\\d+(,\\d{3})*(\\.\\d{2})?	"$1,250.00"
International phone number	\\+(\\d[- ]?){7,}\\d	“+1 555-1234567”
URL	https?://\\S+	“https://example.com?a=b”
E-mail address	[\\w.-]+@[\\w.-]+\\.\\w+	“jane.doe@example.com”
Hashtag	#\\w+	“#election2024”
At-mention (including Bluesky domains)	@[\\w.-]+	“@vanatteveldt.com”
Emoji	\\p{Extended_Pictographic}	“😀”
HTML tags	</?\\w[^>]*>	"</html>"
HTML character escapes	&[^;]+;	"&nbsp;"

Table 9.2: What to match: character classes and anchors

Pattern	Matches	Example
a	the literal character a	cat matches “cat”
\\.	a literal special character (also \\*, \\(, \\\\, …)	U\\.S\\. matches “U.S.”
.	any character except a newline	c.t matches “cat”, “cot”, “c t”
\\w	a word character: letter, digit or underscore	#\\w matches “#e” in “#election”
\\d	a digit	\\d\\d matches “12” or “99”
\\s	whitespace: space, tab, newline	a\\sb matches “a b”
\\W, \\D, \\S	anything except a word character, digit or whitespace	\\W matches the period in “U.S”
[...]	any one of the listed characters or ranges	[aeiou] matches a vowel, [A-Z0-9] a capital or digit
[^...]	any character except the listed ones	[^aeiou] matches anything but a vowel
\\p{...}	a character from a Unicode category	\\p{letter} matches “a”, but also “É” and “조”
\\P{...}	any character not in a Unicode category	\\P{letter} matches “1” and “!”
\\b	word boundary anchor	\\bcat\\b matches “cat” as a word, but not in “cats” or “muscat”
^	start of text anchor	^RT matches RT at the start of a text
$	end of text anchor	!$ matches an exclamation mark at the end of a text

Table 9.3: How many and in what combination: quantifiers, alternatives and groups

Pattern	Matches	Example
*	zero or more of the preceding part	#trump\\d* matches “#trump” and “#trump2024”
+	one or more of the preceding part	#\\w+ matches “#election2024”
?	zero or one of the preceding part (i.e. optional)	walt?z matches “walz” and “waltz”
{n}	exactly n of the preceding part	\\d{4} matches “2024”
{n,m}	between n and m of the preceding part	[A-Z]{1,2} matches “M” and “EH”
{n,}	n or more of the preceding part	\\w{2,} matches words of at least two characters
*?, +?, ??	a lazy quantifier: match as little as possible	https://.*?\\.com stops at the first “.com”
a|b	either the pattern on the left or the one on the right	trump|vance matches “trump” and “vance”
(...)	a group, to scope |, apply a quantifier, or extract a part	#(trump|harris)\\b, #(lol)+, str_extract(text, pattern, group=1)
(?:...)	a non-capturing group: grouping without a group number	\\b(?:www\\.|https?://)

Quiz questions

Section 9.1 quiz: using stringr for basic text cleaning and processing

Q1: Keeping only matching posts

You have this small data frame of posts:

id	text
1	“I love Trump’s rally today”
2	“trump supporters gathered downtown”
3	“TRUMP announced a new policy”

You want to use filter() to keep only the rows where text contains the word “Trump”. Which stringr function do you put inside filter() to do that?

• str_detect()
• str_extract()
• str_count()
• str_length()

Q2: Which posts match?

Using the same posts data frame as Q1, what does posts |> filter(str_detect(text, “Trump”)) return?

• Only the row with id 1.
• All three rows (id 1, 2 and 3).
• No rows.
• The rows with id 1 and 2, but not id 3.

Q3: Counting words

You want to count words per post by counting spaces with str_count(text, ” “). Why does the chapter first apply str_squish() rather than str_trim() before counting?

• str_trim() only removes whitespace at the start and end of the text, so extra spaces or line breaks in the middle of a post would still be counted; str_squish() collapses those too.
• str_trim() removes punctuation as well as whitespace, which would throw off the count.
• str_count() only works on columns produced by str_squish(), not by str_trim().
• str_trim() doesn’t exist in stringr - only str_squish() does.

Q4: An unexpected match

A student runs posts |> filter(str_detect(str_to_lower(text), “trump”)) hoping to find posts about Donald Trump, but one result turns out to be a post about a school’s trumpet section. What’s the best explanation?

• str_detect() matches sequences of characters, not whole words - “trumpet” contains the letters t-r-u-m-p, so it matches too.
• The student forgot to lowercase the text, so casing differences caused the false match.
• str_detect() requires an exact full-string match, so this must be a bug in stringr.
• str_to_lower() altered the spelling of “trumpet” so that it accidentally matched.

Section 9.2 quiz: finding, removing and replacing simple patterns

Q1: Replacing hashtags

You want to replace every hashtag in a post’s text with the literal placeholder “[TAG]”, keeping the rest of the post unchanged. Which function should you use?

• str_replace_all(text, “#\\w+”, “[TAG]”)
• str_remove_all(text, “#\\w+”)
• str_extract(text, “#\\w+”)
• str_detect(text, “#\\w+”)

Q2: Extracting hashtags

Given toy <- tibble(text = “Three hashtags: #trump #harris #biden”), what does str_extract(toy$text, “#\w+”) return?

• “#trump” — only the first match.
• c(“#trump”, “#harris”, “#biden”) — all three matches.
• NA — the pattern doesn’t match anything.
• “Three hashtags: #trump #harris #biden” — the whole string, unchanged.

Q3: Searching for “U.S.”

A student runs posts |> filter(str_detect(text, “U.S.”)) expecting only posts that literally contain “U.S.”, but one match is a post that only mentions “RUSSIAN” — no “U.S.” anywhere. Why?

• In a regex, . matches almost any character, not just a literal period — so U.S. also matches unrelated four-character sequences like part of “RUSSIAN”.
• str_detect() is case-insensitive by default, so “U.S.” matched lowercase letters hiding inside “RUSSIAN”.
• R silently ignores periods in patterns, so the search was really just for “US” anywhere in the text.
• str_detect() matches on whole words only, and “RUSSIAN” was incorrectly counted as one of those words.

Q4: One row per hashtag

You want a data frame with one row per hashtag — so a post with three hashtags should contribute three rows. Which of these achieves that?

• posts |> mutate(tag = str_extract_all(text, “#\\w+”)) |> unnest_longer(tag)
• posts |> mutate(tag = str_extract(text, “#\\w+”))
• posts |> mutate(tag = str_extract_all(text, “#\\w+”))
• posts |> filter(str_detect(text, “#\\w+”))