2 Regular expressions

Note🎯 Learning goals

After working through Tutorial 2, you’ll be able to…

  • Find, count, and replace words in texts with stringr functions
  • Write regular expressions using logical operators, character classes, and quantifiers
  • Combine regular expressions and stringr functions to clean and filter text data

1 Working with text in R: stringr

A simple text analysis would be to check which words texts contain. For example, …

  • Which texts mention the word “crime”?
  • How often does this word appear?
  • Can we replace or remove this word?

For this, we use the package stringr. It is part of the tidyverse, so you do not need to install it separately. As in Tutorial 1, we first load our packages:

All stringr functions start with str_ and work the same way: The first argument is the text we want to search (string), the second argument is what we are looking for (pattern).

Important stringr functions
Function What it does Example
str_detect() checks whether a pattern occurs in a text (TRUE) or not (FALSE) str_detect(x, "crime")
str_count() counts how often a pattern occurs in a text str_count(x, "crime")
str_extract() extracts the first match of a pattern str_extract(x, "crime")
str_replace_all() replaces all occurrences of a pattern with something else str_replace_all(x, "crime", "mystery")
str_remove_all() removes all occurrences of a pattern str_remove_all(x, "crime")

Let’s try these functions with three short example texts:

example <- c("A crime drama set in London.",
             "A comedy about a family.",
             "Crime, crime, and more crime!")

# does the text contain "crime"?
str_detect(example, "crime")
[1]  TRUE FALSE  TRUE
# how often does "crime" occur?
str_count(example, "crime")
[1] 1 0 2
# replace "crime" with "mystery"
str_replace_all(example, "crime", "mystery")
[1] "A mystery drama set in London."    "A comedy about a family."         
[3] "Crime, mystery, and more mystery!"
# remove "crime"
str_remove_all(example, "crime")
[1] "A  drama set in London."  "A comedy about a family."
[3] "Crime, , and more !"

Did you notice something? In the third text, str_count() only counts “crime” twice - although the text contains the word three times.

The reason: stringr functions are case-sensitive. “Crime” (with a capital C) is not the same as “crime”. We will learn how to deal with this later in this tutorial.

2 Regular expressions

So far, we have searched for exact words, such as “crime”. What if we want to find…

  • “crime” and “Crime”?
  • texts mentioning “drama” or “crime”?

For this, we use regular expressions.

You already know the logic of regular expressions! Think of how you search for content in Google or a library database, e.g., climate chang* OR global warming. In text analysis, regular expressions help us to…

  • identify texts (e.g., texts about crime)
  • clean texts (e.g., fix encoding problems, see Tutorial 1)
  • analyze texts (e.g., search for positive words such as “good|great|fantastic”)

2.1 What are regular expressions?

When we search for a word in a text, we usually search for a string pattern, i.e., an exact sequence of characters: "crime" finds the word “crime” - and nothing else.

Regular expressions go one step further: They are string patterns in which some characters have a special, non-literal meaning. For example, "[Cc]rime" finds both “Crime” and “crime”, because [Cc] means “either C or c”.

Regular expressions consist of different building blocks. Here are the most important ones:

  • Logical operators, e.g., | (which means OR)
  • Character classes, e.g., [Cc] (matches “C” or “c”) or [a-z] (matches any lowercase letter)
  • Quantifiers, e.g., *, +, or ? (which specify how often the previous expression occurs)

In the following, we will again work with the TV series dataset. Make sure to load it!

# read in the data
data <- read.csv2("data/data_tvseries.csv", encoding = "UTF-8")

2.2 Logical operators

With logical operators, we can combine several patterns. The most important one is |, which means OR: The pattern "drama|crime" finds texts that contain “drama” or “crime” (or both). We can also combine two str_detect() commands with & (AND) or put a ! in front of str_detect() to say that a certain string should not be matched.

Logical operators
Operator Meaning Example
| OR: finds the expression on the left or on the right “drama|crime” finds texts containing “drama” or “crime” (or both)
& AND: both patterns have to occur str_detect(x, "drama") & str_detect(x, "crime") finds texts containing both words
! NOT: the pattern must not occur !str_detect(x, "drama") finds texts not containing “drama”

For example, how many TV series descriptions mention “drama” or “crime”?

data |>
  filter(str_detect(Description, "drama|crime")) |>
  nrow()
[1] 65

And how many mention “drama” and “crime”?

data |>
  filter(str_detect(Description, "drama") & str_detect(Description, "crime")) |>
  nrow()
[1] 2

2.3 Character classes

Regular expressions allow us to search for specific types of characters, so-called character classes. A character class is always written in square brackets and stands for exactly one character. For example, the pattern [a-z] matches any one lowercase letter (i.e., “a” or “b” or “c”, etc.). We can also define our own classes: [Dd] matches either “D” or “d”.

Character classes are especially useful because regular expressions are case-sensitive (remember Section 1!). Compare how many descriptions we find with "drama" vs. "[Dd]rama":

# only lowercase "drama"
data |>
  filter(str_detect(Description, "drama")) |>
  nrow()
[1] 21
# "drama" or "Drama"
data |>
  filter(str_detect(Description, "[Dd]rama")) |>
  nrow()
[1] 25

Here is a (limited) selection of frequently used character classes:

Character classes
Character class Meaning Example
[a-z] finds any lowercase letter “[a-z]eadline” finds “headline” and “deadline”
[A-Z] finds any uppercase letter “[A-Z]BC” finds “ABC” and “NBC”
[[:alpha:]] finds any letter (lowercase and uppercase) “[[:alpha:]]rime” finds “Crime” and “crime”
[0-9] finds any digit “Season [0-9]” finds “Season 1” and “Season 2”
[[:blank:]] finds spaces and tabs “New[[:blank:]]York” finds “New York”
[[:punct:]] finds punctuation, e.g., ! # : ; . “[[:punct:]]” finds “.” and “!”

For example, we can find all TV series whose title contains an abbreviation, i.e., at least two uppercase letters in a row (like “IT” or “NCIS”):

data |>
  filter(str_detect(Title, "[A-Z][A-Z]")) |>
  select(Title) |>
  head(3)
              Title
1 144. The IT Crowd
2         145. NCIS
3       224. The OA

2.4 Quantifiers

Regular expressions also allow us to specify how often a letter, number, or other character should occur. For this, we use so-called quantifiers.

A quantifier always refers to the expression directly before it: In "superheroe?s", the ? only refers to the “e” before it.

Quantifiers
Quantifier Meaning Example
? the preceding expression occurs zero times or once (i.e., it is optional) “superheroe?s” finds “superheros” and “superheroes”
+ the preceding expression occurs at least once “[0-9]+” finds “7” and “2023”
* the preceding expression occurs zero or more times “Go*al” finds “Goal” and “Goooooal”
{n} the preceding expression occurs exactly n times “[0-9]{4}” finds “2023”, but not “23”
{n,} the preceding expression occurs at least n times “[A-Z]{2,}” finds “FBI” and “NYPD”
{n,m} the preceding expression occurs at least n times and at most m times “[0-9]{2,4}” finds “23” and “2023”

Quantifiers become especially powerful in combination with character classes. For example, [A-Z]{2,} finds sequences of at least two uppercase letters - which helps us identify abbreviations such as “FBI” or “NYPD”.

Let’s search the titles again - this time with a quantifier instead of writing [A-Z][A-Z]. We get the same result, but with a shorter pattern:

data |>
  filter(str_detect(Title, "[A-Z]{2,}")) |>
  select(Title) |>
  head(3)
              Title
1 144. The IT Crowd
2         145. NCIS
3       224. The OA

Anchors do not match a character, but a position in a text. With anchors, we can specify that a pattern should only be found at the beginning or at the end of a text:

  • ^ matches the beginning of a string
  • $ matches the end of a string

Let’s look at three short example texts:

example_anchors <- c("Crown Jewels", "Inside the Crown", "The Crown is back")

# "Crown" anywhere in the text
str_detect(example_anchors, "Crown")
[1] TRUE TRUE TRUE
# "Crown" at the beginning of the text
str_detect(example_anchors, "^Crown")
[1]  TRUE FALSE FALSE
# "Crown" at the end of the text
str_detect(example_anchors, "Crown$")
[1] FALSE  TRUE FALSE

3 Combining stringr and regular expressions

Now we can bring everything together: We use stringr functions (Section 1) with regular expressions (Section 2) to clean and filter our data.

For example, combined with filter(), str_detect() keeps only rows where a pattern occurs. Let’s keep only TV series whose description mentions “drama” or “Drama”:

data |>
  filter(str_detect(Description, "[Dd]rama")) |>
  nrow()
[1] 25

With !, we can also do the opposite and keep only series whose description does not mention “drama” or “Drama”:

data |>
  filter(!str_detect(Description, "[Dd]rama")) |>
  nrow()
[1] 875

💡 Take-Aways

  • string patterns: exact sequences of characters, e.g., "crime"
  • regular expressions: string patterns with a non-literal meaning, built from logical operators (|), character classes (e.g., [a-z], [0-9], [[:punct:]]), quantifiers (e.g., *, +, ?, {n}), and anchors (^, $)
  • stringr functions: str_detect() (does a pattern occur?), str_count() (how often?), str_extract() (extract it), str_replace_all() and str_remove_all() (replace/remove it)

🤓 Smart Hacks

Writing regular expressions often involves trial and error. Websites like regex101.com let you test a pattern on example text and explain step by step what each part of the pattern does.

Some characters, such as ?, +, *, |, (, [, or ., are not interpreted literally in regular expressions - for example, ? is a quantifier and . stands for “any character”. These characters are called metacharacters.

If you want to search for a metacharacter itself (e.g., a full stop), you have two options:

  • Escape it with \\, e.g., "\\.": R then interprets the next character literally.
  • Wrap the pattern in fixed(), e.g., fixed("."): R then interprets the whole pattern literally (but you can no longer use any regular expressions in this pattern).

For example, if we simply count "." in the first description, R counts every character - because . means “any character”:

data |>
  slice(1) |>
  mutate(n_dots = str_count(Description, ".")) |>
  pull(n_dots)
[1] 131

Once we escape the full stop as a metacharacter or use fixed(), R counts only actual full stops:

# escaping the metacharacter
data |>
  slice(1) |>
  mutate(n_dots = str_count(Description, "\\.")) |>
  pull(n_dots)
[1] 1
# interpreting the pattern literally
data |>
  slice(1) |>
  mutate(n_dots = str_count(Description, fixed("."))) |>
  pull(n_dots)
[1] 1

🎲 Quiz

Note🎲 Question 1

Which of the following statements about regular expressions are correct?

Note🎲 Question 2

You want to keep only TV series whose description mentions “drama” or “Drama”. Type in the correct command!

📚 More tutorials on this

You still have questions? The following tutorials & papers can help you with that:

📌 Test your knowledge

Task 1 (Easy🔥)

Use the TV series data. Can you identify the number of TV series whose description mentions London?

Task 2 (Medium🔥🔥)

Use the TV series data.

  • Identify all TV series about superheroes, i.e., whose description contains “superhero” or “superheroes” (upper- or lowercase).
  • In these descriptions, replace “superhero”/“superheroes” with “fancy R programmers”.

How many series are about superheroes? What does the first description look like after your replacement?

Task 3 (Hard🔥🔥🔥)

Use the TV series data.

In the variable Title, each title starts with a number, a dot, and a space:

data |>
  select(Title) |>
  head(5)
                Title
1  1. Game of Thrones
2     2. Breaking Bad
3  3. Stranger Things
4          4. Friends
5 5. The Walking Dead

We want to remove these characters with str_remove_all(). The result should look like this:

             Title
1  Game of Thrones
2     Breaking Bad
3  Stranger Things
4          Friends
5 The Walking Dead

How would you clean the data this way using regular expressions?

Hint: Have a look at the box “More on regular expressions: Anchors” in Section 2.4.