3 Preprocessing text

Note🎯 Learning goals

After working through Tutorial 3, you’ll be able to…

  • Explain what preprocessing is
  • Explain the bag-of-words assumption
  • Normalize text with quanteda (e.g., tokenization, lowercasing, etc.)

1 Preprocessing and the bag-of-words assumption

What is preprocessing?

Preprocessing reduces the complexity of text data through cleaning and normalization - without losing its substantive meaning.

Preprocessing has two goals:

  • Cleaning: reducing (systematic) errors, e.g., encoding problems → Tutorial 2
  • Normalization: making texts comparable across documents, languages, etc. → this tutorial

This involves a central trade-off: The more we reduce the complexity of texts for automated analysis, the more we risk losing their substantive meaning (which we may need for in-depth analyses).

Let’s start with a simple example. Imagine a news article that reads:

On the state of the German economy: Will we have another financial crisis in Germany in 2023?

In a first step, we break the text down into features, our unit of analysis. Most often, features are words. The process of breaking texts down into features is called tokenization. Afterwards, we can count how often each feature occurs (“On”: 1, “the”: 2, “state”: 1, …).

By reducing texts to the frequency of features, we make an important assumption - the bag-of-words assumption:

We assume that we can interpret texts without considering the order and context of features.

For preprocessing, we use the package quanteda. Make sure that you install it first (you only need to do this once). Then, we load quanteda and the tidyverse:

# install.packages("quanteda")

# load packages (every time you restart R)
library(tidyverse)
library(quanteda)

In quanteda, we tokenize texts with tokens():

example <- "On the state of the German economy: Will we have another financial crisis in Germany in 2023?"

tokens(example)
Tokens consisting of 1 document.
text1 :
 [1] "On"      "the"     "state"   "of"      "the"     "German"  "economy"
 [8] ":"       "Will"    "we"      "have"    "another"
[ ... and 7 more ]

2 Lowercasing

Next, we often convert all features to lowercase, so that “Will” and “will” are counted as the same feature. In quanteda, we use tokens_tolower():

tokens(example) |>
  tokens_tolower()
Tokens consisting of 1 document.
text1 :
 [1] "on"      "the"     "state"   "of"      "the"     "german"  "economy"
 [8] ":"       "will"    "we"      "have"    "another"
[ ... and 7 more ]

Often, lowercasing does not change the meaning of features: “Learning R” and “learning R” mean the same thing. But there are exceptions, where upper- and lowercase letters signal different meanings, for example:

  • “Apple” (the company) vs. “apple” (the fruit)
  • “US” (United States) vs. “us” (the pronoun)
  • “Bild” (German tabloid) vs. “bild” (German for picture)

After lowercasing, R can no longer tell these features apart:

tokens("Apple apple US us") |>
  tokens_tolower()
Tokens consisting of 1 document.
text1 :
[1] "apple" "apple" "us"    "us"

If such differences matter for your research question (e.g., when analyzing coverage of companies or countries), think twice before lowercasing.

3 Removing punctuation, numbers, and symbols

Punctuation, numbers, and symbols are often not of substantive interest. We can remove them directly when tokenizing, using the arguments remove_punct, remove_numbers, and remove_symbols in tokens():

tokens(example,
       remove_punct = TRUE,
       remove_numbers = TRUE,
       remove_symbols = TRUE) |>
  tokens_tolower()
Tokens consisting of 1 document.
text1 :
 [1] "on"        "the"       "state"     "of"        "the"       "german"   
 [7] "economy"   "will"      "we"        "have"      "another"   "financial"
[ ... and 4 more ]

Often, punctuation, numbers, and symbols carry little substantive meaning. But sometimes, they are exactly what we are interested in, for example:

  • emotions: “Is this true???!!!” or emojis like 👍🤬🥳
  • numbers with a meaning: “9/11”, “G7”, or years like “2023”
  • social media content: hashtags like “#metoo” or URLs like “www.zeit.de”

4 Removing stopwords

Stopwords are very frequent words that (often) carry little substantive meaning, e.g., “the”, “of”, “in”. quanteda includes ready-made stopword lists in many languages, which you can access via stopwords():

stopwords("english") |>
  head(20)
 [1] "i"          "me"         "my"         "myself"     "we"        
 [6] "our"        "ours"       "ourselves"  "you"        "your"      
[11] "yours"      "yourself"   "yourselves" "he"         "him"       
[16] "his"        "himself"    "she"        "her"        "hers"

We remove stopwords with tokens_remove():

tokens(example,
       remove_punct = TRUE,
       remove_numbers = TRUE,
       remove_symbols = TRUE) |>
  tokens_tolower() |>
  tokens_remove(stopwords("english"))
Tokens consisting of 1 document.
text1 :
[1] "state"     "german"    "economy"   "another"   "financial" "crisis"   
[7] "germany"

There is no consensus definition of stopwords - what counts as a stopword strongly depends on your context and research question. Use off-the-shelf lists with care! For example, if you study how politicians construct collective identities, words like “we”, “our”, or “they” are of substantive interest and should not be removed.

Often, it makes sense to adapt existing stopword lists or to define your own, “organic” stopwords.

5 Stemming and lemmatization

Finally, we often reduce features to their base form, so that “running”, “runs”, and “run” are counted as the same feature. There are two approaches:

  • Stemming reduces features to their word stem by cutting off suffixes, e.g., “running”, “runs”, “run” → “run”. This is fast, but does not always work well: “ran” stays “ran”, and “Germany” becomes “germani”.
  • Lemmatization reduces features to their dictionary form (lemma), e.g., “running”, “ran” → “run”. This is more accurate, but requires additional tools (e.g., the R packages udpipe or spacyr).

In quanteda, we use tokens_wordstem() for stemming:

tokens(example,
       remove_punct = TRUE,
       remove_numbers = TRUE,
       remove_symbols = TRUE) |>
  tokens_tolower() |>
  tokens_remove(stopwords("english")) |>
  tokens_wordstem()
Tokens consisting of 1 document.
text1 :
[1] "state"   "german"  "economi" "anoth"   "financi" "crisi"   "germani"

Stemming and lemmatization reduce the number of features - but they may also merge words with different meanings. For example, stemming reduces both “university” and “universe” to “univers”, and both “organization” and “organ” to “organ”:

tokens("university universe organization organ") |>
  tokens_wordstem()
Tokens consisting of 1 document.
text1 :
[1] "univers" "univers" "organ"   "organ"

Lemmatization works better in such cases, but it can also make mistakes - for example, if a word has several meanings (“saw” can be a tool or the past tense of “see”). And stems such as “univers” or “germani” are harder to read and interpret when presenting results.

6 Example: Preprocessing the IMDb data

Now let’s apply all steps to our TV series data. First, we load the data:

data <- read.csv2("data/data_tvseries.csv", encoding = "UTF-8")

quanteda cannot work with a data frame directly. Instead, it uses a specific object type: the corpus. A corpus stores the texts we want to analyze together with their metadata (in quanteda, these are called document variables or docvars, e.g., the title or rating of a series).

We create a corpus with corpus(). With the argument text_field, we tell quanteda which variable contains the texts:

corpus_tv <- corpus(data, text_field = "Description")

Our corpus contains 900 documents, i.e., 900 TV series descriptions. All other variables of our data set are saved as docvars:

docvars(corpus_tv) |>
  head(3)
               Title      Year Parental.Rating Rating Number.of.Votes
1 1. Game of Thrones 2011–2019           TV-MA    9.2            2.3M
2    2. Breaking Bad 2008–2013           TV-MA    9.5            2.1M
3 3. Stranger Things 2016–2025           TV-14    8.7            1.3M

With summary(), we get a first overview of our texts, e.g., the number of tokens (all words, numbers, and punctuation characters) and types (unique tokens) per document:

summary(corpus_tv, n = 3)
Corpus consisting of 900 documents, showing 3 documents:

  Text Types Tokens Sentences              Title      Year Parental.Rating
 text1    22     23         1 1. Game of Thrones 2011–2019           TV-MA
 text2    24     26         1    2. Breaking Bad 2008–2013           TV-MA
 text3    22     25         1 3. Stranger Things 2016–2025           TV-14
 Rating Number.of.Votes
    9.2            2.3M
    9.5            2.1M
    8.7            1.3M

Now we can apply preprocessing steps. We combine them in one pipe and save the result as tokens_tv:

tokens_tv <- corpus_tv |>

  # tokenization & removing punctuation, numbers, symbols
  tokens(remove_punct = TRUE,
         remove_numbers = TRUE,
         remove_symbols = TRUE) |>

  # lowercasing
  tokens_tolower() |>

  # removing stopwords
  tokens_remove(stopwords("english")) |>

  # stemming
  tokens_wordstem()

This is what our first text looked like before preprocessing:

data$Description[1]
[1] "Nine noble families fight for control over the lands of Westeros, while an ancient enemy returns after being dormant for millennia."

…and this is what it looks like afterwards:

tokens_tv[1]
Tokens consisting of 1 document and 5 docvars.
text1 :
 [1] "nine"      "nobl"      "famili"    "fight"     "control"   "land"     
 [7] "westero"   "ancient"   "enemi"     "return"    "dormant"   "millennia"

💡 Take-Aways

  • preprocessing: reduces the complexity of text through cleaning and normalization, without losing substantive meaning
  • features: the unit of analysis texts are reduced to (often: words)
  • corpus: texts + metadata (docvars) in one quanteda object: corpus()
  • tokenization: breaking texts down into features: tokens()
  • bag-of-words assumption: the order and context of features do not influence their meaning (unlikely!)
  • normalization: tokens_tolower(), tokens_remove(stopwords("english")), tokens_wordstem()

🤓 Smart Hacks

stopwords() includes lists for many languages. For example, here are the first German stopwords:

stopwords("german") |>
  head(10)
 [1] "aber"  "alle"  "allem" "allen" "aller" "alles" "als"   "also"  "am"   
[10] "an"

Similarly, tokens_wordstem() can stem texts in other languages via the argument language, e.g., tokens_wordstem(language = "german").

🎲 Quiz

Note🎲 Question 1

Which of the following statements about preprocessing are correct?

Note🎲 Question 2

Which of the following statements about preprocessing are correct?

📚 More tutorials on this

You still have questions? The following tutorials & papers can help you with that:

📌 Test your knowledge

Task 1 (Easy🔥)

Use the TV series data. Preprocess only the description of “Breaking Bad” (the second document) as in this tutorial.

How many tokens does the description have before and after preprocessing?

Task 2 (Medium🔥🔥)

Use the TV series data. In this tutorial, we first removed stopwords and then applied stemming. Repeat the preprocessing from this tutorial, but switch the order of these two steps: First apply stemming, then remove stopwords.

Why does the order of preprocessing steps matter here?