# load packages
library(tidyverse)
library(quanteda)
# read in the data
data <- read.csv2("data/data_tvseries.csv", encoding = "UTF-8")
# create corpus (see Tutorial 3)
corpus_tv <- corpus(data, text_field = "Description")
# preprocessing (see Tutorial 3)
tokens_tv <- corpus_tv |>
tokens(remove_punct = TRUE,
remove_numbers = TRUE,
remove_symbols = TRUE) |>
tokens_tolower() |>
tokens_remove(stopwords("english")) |>
tokens_wordstem()4 Text-as-data representations
Note🎯 Learning goals
After working through Tutorial 4, you’ll be able to…
- Explain why we need to represent text numerically
- Explain what a document-feature matrix (DFM) is
- Create and inspect a DFM with
quanteda - Name other text-as-data representations beyond the bag-of-words
1 Why do we need text-as-data representations?
Computers cannot “read” text the way we do. For computers to analyze text, we need to transform it into a numerical format. How we do this is called a text-as-data representation.
A classic representation is the bag-of-words, which builds on the bag-of-words assumption (see Tutorial 3): We reduce texts to the frequency of their features and ignore their order and context. The most common way of storing a bag-of-words representation is the document-feature matrix (DFM).
First, we load our packages and data, and repeat the preprocessing from Tutorial 3:
2 The document-feature matrix
2.1 What is a document-feature matrix?
A document-feature matrix (DFM) is a matrix in which…
- Rows = documents
- Columns = features
- Cells = how often a feature occurs in a document
Let’s look at an example with three short news headlines:
headlines <- c(
doc1 = "On the state of the German economy: Will we have another financial crisis in Germany in 2023?",
doc2 = "The GDP is going down, unemployment is up: Crisis ahead in Germany?",
doc3 = "German economy tumbles into crisis: Government under pressure"
)
headlines |>
tokens(remove_punct = TRUE, remove_numbers = TRUE) |>
tokens_tolower() |>
tokens_remove(stopwords("english")) |>
tokens_wordstem() |>
dfm()Document-feature matrix of: 3 documents, 14 features (54.76% sparse) and 0 docvars.
features
docs state german economi anoth financi crisi germani gdp go unemploy
doc1 1 1 1 1 1 1 1 0 0 0
doc2 0 0 0 0 0 1 1 1 1 1
doc3 0 1 1 0 0 1 0 0 0 0
[ reached max_nfeat ... 4 more features ]For example, the feature “crisi” (the stem of “crisis”) occurs once in every document, while “economi” occurs in doc1 and doc3, but not in doc2.
A DFM can be used as input for many methods, e.g., dictionaries or machine learning. With little preprocessing, it becomes very large and contains many zeros (because most features occur in only a few documents) - it is sparse.
2.2 Creating a DFM for our data
We create a DFM from our tokens with dfm():
dfm_tv <- tokens_tv |>
dfm()
dfm_tvDocument-feature matrix of: 900 documents, 4,234 features (99.65% sparse) and 5 docvars.
features
docs nine nobl famili fight control land westero ancient enemi return
text1 1 1 1 1 1 1 1 1 1 1
text2 0 0 1 0 0 0 0 0 0 0
text3 0 0 0 0 0 0 0 0 0 0
text4 0 0 0 0 0 0 0 0 0 0
text5 0 0 0 0 0 0 0 0 0 0
text6 0 0 0 0 0 0 0 0 0 0
[ reached max_ndoc ... 894 more documents, reached max_nfeat ... 4,224 more features ]We see that…
- the DFM consists of 900 documents,
- after preprocessing, it still contains 4,234 unique features,
- it is 99.65% sparse, i.e., more than 99% of all cells contain a 0.
2.3 Inspecting a DFM
quanteda offers several functions to inspect a DFM:
-
ndoc(): number of documents -
nfeat(): number of features -
sparsity(): share of cells containing a 0 -
featnames(): names of all features -
docvars(): metadata of all documents (e.g., title, rating)
3 Other text-as-data representations
The bag-of-words representation is simple and often useful. But because it ignores the order and context of features, it has clear limitations. This is why there are other text-as-data representations, for example:
- N-grams: sequences of n features (e.g., “new york” as a bigram), which keep some information about word order
- Syntax: information about the grammatical role of features (e.g., subject vs. object)
- Semantic representations, e.g., word embeddings, which represent features based on the contexts they occur in, so that similar words (e.g., “series” and “show”) get similar representations
We will learn more about these later in the course.
💡 Take-Aways
-
document-feature matrix: rows = documents, columns = features, cells = frequencies:
dfm() -
inspect a DFM:
ndoc(),nfeat(),sparsity(),featnames(),docvars()
🤓 Smart Hacks
TipSmart Hack 1: Weighting a DFM
By default, the cells of a DFM contain absolute frequencies. With dfm_weight(), we can change this, e.g., to the proportion of each feature within a document. This is useful if documents differ in length:
dfm_tv |>
dfm_weight(scheme = "prop") |>
head(3)Document-feature matrix of: 3 documents, 4,234 features (99.65% sparse) and 5 docvars.
features
docs nine nobl famili fight control land
text1 0.08333333 0.08333333 0.08333333 0.08333333 0.08333333 0.08333333
text2 0 0 0.06250000 0 0 0
text3 0 0 0 0 0 0
features
docs westero ancient enemi return
text1 0.08333333 0.08333333 0.08333333 0.08333333
text2 0 0 0 0
text3 0 0 0 0
[ reached max_nfeat ... 4,224 more features ]Another common weighting is tf-idf (dfm_tfidf()), which gives more weight to features that occur often in a document, but rarely across all documents.
🎲 Quiz
📚 More tutorials on this
You still have questions? The following tutorials & papers can help you with that:
- quanteda tutorials: Document-feature matrix
- quanteda quick start guide
- Grimmer, J., Roberts, M. E., & Stewart, B. M. (2022). Text as data: A new framework for machine learning and the social sciences. Princeton University Press.