1 Reading in text

Note🎯 Learning goals

After working through Tutorial 1, you’ll be able to…

• Read text data into R
• Inspect text data and recognize (and fix) encoding problems

1 Reading text data into R

1.1 Reading in Texts

Throughout the next tutorials, we work with a data set of the 900 most popular TV series on IMDb. For each series, it contains the title, the years it aired, its parental rating, its IMDb rating, the number of votes, and - most important for us - a short description of the series.

You can download the data as a data_tvseries.csv file from Moodle (see Session 2). Save it in your project folder, for example in your subfolder called data.

Before we do anything else, we install and load the packages we need for this tutorial. Remember: You only need to run install.packages() once. Assuming that you already have these packages installed, the hashtag # will skip their installation.

# install (only once)
# install.packages("tidyverse")

# load packages (every time you restart R)
library(tidyverse)

The file is separated by semicolons (;) and uses commas as decimal separators. This is why we use read.csv2() instead of read.csv(). With encoding = "UTF-8", we tell R how the characters in the file are encoded (more on this in Section 3). We save the data as an object called data:

data <- read.csv2("data/data_tvseries.csv", encoding = "UTF-8")

WarningIf R cannot find your file

If you get an error such as cannot open file 'data/data_tvseries.csv': No such file or directory, R cannot find your file. Check that…

• you are working in your R project (see the top right corner of RStudio)
• the file is saved in the folder you specified (here: the subfolder data)
• the file name is spelled correctly (including the file extension .csv)

1.2 Reading in texts in other formats

Text data comes in many formats. Here is a (small) selection of formats and R packages that help you read them into R:

Format	Packages
Spreadsheets (.csv, .xlsx)	base R (e.g., read.csv2()), readr, readxl
Text documents (.txt, .pdf, .doc)	readtext
Files from other statistics programs (.sav, .dta)	haven, foreign
Content from URLs (download, scraping)	base R, httr, rvest, polite

2 Inspecting text data

After reading in data, always check whether everything worked. Let’s get an overview of our TV series data. How many rows (documents) and columns (variables) does it have?

nrow(data)

[1] 900

ncol(data)

[1] 6

What do the first rows look like?

data |>
  as_tibble() |>
  head()

# A tibble: 6 × 6
  Title               Year    Parental.Rating Rating Number.of.Votes Description
  <chr>               <chr>   <chr>            <dbl> <chr>           <chr>      
1 1. Game of Thrones  2011–2… TV-MA              9.2 2.3M            "Nine nobl…
2 2. Breaking Bad     2008–2… TV-MA              9.5 2.1M            "A chemist…
3 3. Stranger Things  2016–2… TV-14              8.7 1.3M            "When a yo…
4 4. Friends          1994–2… TV-14              8.9 1.1M            "Follows t…
5 5. The Walking Dead 2010–2… TV-MA              8.1 1.1M            "Sheriff D…
6 6. Sherlock         2010–2… TV-14              9.1 1M              "The quirk…

The variable we are most interested in is Description, which contains the text we want to analyze. Let’s look at the first description. With slice(), we keep only the first row, and with pull(), we extract the content of the variable as a character string:

data |>
  slice(1) |>
  pull(Description)

[1] "Nine noble families fight for control over the lands of Westeros, while an ancient enemy returns after being dormant for millennia."

Looks good! We can also check how long our texts are. str_length() from the tidyverse package stringr counts the number of characters per text. We use mutate to create the variable length. With summarise, we then get summary statistics of the shortest, longest, and average characters of each TV series description.

data |>
  mutate(length = str_length(Description)) |>
  summarise(min = min(length),
            mean = mean(length),
            max = max(length))

  min     mean max
1  35 153.0189 351

Descriptions are short: on average, they have about 153 characters.

3 Encoding problems

3.1 What is encoding?

Computers store characters (e.g., the letters “w”, “o”, “r”, “d”) as numbers (bytes). For example, “word” is stored as 01110111 01101111 01110010 01100100.

An encoding is the “key” that translates these numbers back into characters. The problem: There are different encodings (e.g., UTF-8, ASCII, latin1). If a text is saved with one encoding but read with another, characters get “translated” incorrectly. This affects especially…

• language-specific characters such as “ü”, “ß”, “é”, or “ç”
• special characters such as typographic quotation marks (“…”) or emojis

Let’s look at an example: We create a string and then change its encoding to see what happens.

# example string
string <- "Arsène Lupin & Saga Norén"

# check encoding
Encoding(string)

[1] "UTF-8"

# change encoding (for testing purposes)
Encoding(string) <- "latin1"
string

[1] "Arsène Lupin & Saga Norén"

Oops! “è” turned into “è”. This is a typical encoding problem.

3.2 How can encoding cause problems?

Our TV series data also contains special characters. Let’s simulate what happens if we read the data with the wrong encoding (here: latin1) and look at three descriptions containing special characters:

data_wrong <- read.csv2("data/data_tvseries.csv", encoding = "latin1")

data_wrong |>
  slice(180, 348, 676) |>
  pull(Description)

[1] "Inspired by the adventures of Arsène Lupin, gentleman thief Assane Diop sets out to avenge his father for an injustice inflicted by a wealthy family."                                                    
[2] "When a body is found on the bridge between Denmark and Sweden, right on the border, Danish inspector Martin Rohde and Swedish Saga Norén have to share jurisdiction and work together to find the killer."
[3] "Betrayed by his trusted friends and the woman he loved, Ömer Uçar returns as Ezel to exact his vengeance."                                                                                               

Names like “Arsène”, “Norén”, or “Ömer Uçar” are now broken. Compare this with the data we read in using the correct encoding:

data |>
  slice(180, 348, 676) |>
  pull(Description)

[1] "Inspired by the adventures of Arsène Lupin, gentleman thief Assane Diop sets out to avenge his father for an injustice inflicted by a wealthy family."                                                    
[2] "When a body is found on the bridge between Denmark and Sweden, right on the border, Danish inspector Martin Rohde and Swedish Saga Norén have to share jurisdiction and work together to find the killer."
[3] "Betrayed by his trusted friends and the woman he loved, Ömer Uçar returns as Ezel to exact his vengeance."                                                                                                

3.3 How can we fix encoding problems?

There are two main strategies:

1. Read in texts with the correct encoding (recommended). Most functions have an argument for this, e.g., encoding = "UTF-8" or fileEncoding = "UTF-8" in read.csv2(), or encoding = "UTF-8" in readtext(). Today, UTF-8 is the most common encoding.
2. If this is not possible: Clean the texts afterwards, for example by replacing wrongly encoded characters with regular expressions. We will learn more about this in Tutorial 2. For now, here is a sneak peek using str_replace_all() from the tidyverse package stringr, which replaces one pattern with another:

string |>
  str_replace_all(pattern = "è", replacement = "è") |>
  str_replace_all(pattern = "é", replacement = "é")

[1] "Arsène Lupin & Saga Norén"

💡 Take-Aways

• read in data: read.csv2() or read.csv() (spreadsheets)
• inspect data: nrow(), ncol(), head(), slice(), pull(), str_length()
• encoding: specify the correct encoding when reading in data (e.g., encoding = "UTF-8"), check Encoding(), clean with str_replace_all() if necessary

🤓 Smart Hacks

TipSmart Hack 1: Figuring out encodings with readr

The tidyverse package readr can help with encoding issues. If you do not know a file’s encoding, readr::guess_encoding() can help you find out:

guess_encoding("data/data_tvseries.csv")

# A tibble: 2 × 2
  encoding     confidence
  <chr>             <dbl>
1 UTF-8              1   
2 windows-1252       0.64

🎲 Quiz

round = (n, places) => {
  if (!places) return Math.round(n);
  const d = 10 ** places;
  return Math.round(n * d) / d;
}

function textReplacementWithText({ text, placeholder, correctText, fontFamily = "monospace", color = "#000000" }) {
  const parts = text.split(/\{\{input\}\}/); // Teilt den Text an den Eingabestellen

  const root = htl.html`<div style="font-size: HUGE; margin-top: -40; margin-bottom: -40; margin-left: -50px; line-height: 1.8; font-family: ${fontFamily}; color: ${color}; white-space: pre-wrap; text-indent: 0; display: flex; align-items: baseline;">
    <div style="flex: 1;">
      ${parts.map((part, index) =>
        htl.html`${part}${index < parts.length - 1 ? htl.html`<span style="display: inline-flex; align-items: baseline;">
          <input 
            type="text"
            placeholder="${placeholder}"
            style="padding: 3px; font-size: HUGE; border: 1px solid #ccc; margin-left: 0px; border-radius: 4px; width: calc(${correctText.length}ch + 1em); font-family: ${fontFamily}; color: ${color}; text-align: right; box-sizing: border-box; margin-right: 0.2em;"
            oninput=${(e) => {
              const feedbackElement = root.querySelector(`#feedback-final`);
              const allInputsCorrect = Array.from(root.querySelectorAll("input")).every((input) => input.value.trim() === correctText.trim());
              if (allInputsCorrect) {
                feedbackElement.textContent = "✅";
                feedbackElement.style.color = "green";
              } else {
                feedbackElement.textContent = "🔴";
                feedbackElement.style.color = "red";
              }
            }}
          /></span>` : ""}`
      )}
    </div>
    <span id="feedback-final" style="font-weight: bold; font-size: HUGE; margin-left: 0.5em; color: inherit;"></span>
  </div>`;

  return root;
}



function textReplacementTask({ placeholder, correctText, width = "300px", fontFamily = "monospace", color = "black" }) {
  let enteredText = ""; 

  const root = htl.html`<div style="font-size: 14px; line-height: 1.5; margin-bottom: 20px;">
    <h4>Bitte ersetzen Sie den Text:</h4>
    <!-- Eingabefeld -->
    <div style="display: flex; align-items: center;">
      <input
        type="text"
        placeholder="${placeholder}"
        style="padding: 5px; font-size: 14px; border: 1px solid #ccc; border-radius: 4px; width: ${width}; font-family: ${fontFamily}; color: ${color};"
        onInput=${(e) => {
          enteredText = e.target.value;
          root.value = enteredText;
          root.dispatchEvent(new CustomEvent("input"));
          updateFeedback();
        }}
      />
      <!-- Feedback -->
      <div id="feedback" style="font-weight: bold; font-size: 14px; margin-left: 10px; text-align: left; white-space: nowrap; color: inherit;"></div>
    </div>
  </div>`;

// Funktion zur Aktualisierung des Feedbacks
  function updateFeedback() {
    const feedbackElement = root.querySelector("#feedback");
    if (enteredText.trim() === correctText.trim()) {
      feedbackElement.textContent = "✅";
      feedbackElement.style.color = "green";
    } else if (enteredText.trim() === "") {
      feedbackElement.textContent = ""; 
    } else {
      feedbackElement.textContent = "🔴";
      feedbackElement.style.color = "red";
    }
  }

  root.value = null; 
  return root;
}


function singleChoiceInput({ options, correctIndex }) {
  let selectedAnswer = null;

  const root = htl.html`<div style="font-size: HUGE; line-height: 1.5; margin-bottom: 0px;">
    <!-- Antworten -->
    <div style="display: grid; row-gap: 12px;">
      ${options.map(
        (option, index) =>
          htl.html`<div style="display: flex; align-items: center;">
            <!-- Radiobutton und Option -->
            <label style="display: flex; align-items: center; padding-left: 4pt; font-weight: normal;">
              <input
                type="radio"
                name="singleChoice"
                value=${index}
                style="margin-right: 10px;"
                onChange=${() => {
                  selectedAnswer = index;
                  root.value = selectedAnswer;
                  root.dispatchEvent(new CustomEvent("input"));
                  updateFeedback();
                }}
              />
              <span>${option}</span>
            </label>
            <!-- Feedback -->
            <div id="feedback-${index}" style="font-weight: bold; font-size: 14px; margin-left: 10px; text-align: left; white-space: nowrap; color: inherit;"></div>
          </div>`
      )}
    </div>
  </div>`;

// Funktion zur Aktualisierung des Feedbacks
  function updateFeedback() {
    options.forEach((_, index) => {
      const feedbackElement = root.querySelector(`#feedback-${index}`);
      if (selectedAnswer === index) {
        feedbackElement.textContent =
          index === correctIndex ? "✅" : "🔴";
        feedbackElement.style.color = index === correctIndex ? "green" : "red";
      } else {
        feedbackElement.textContent = ""; // Löscht Feedback für nicht ausgewählte Optionen
      }
    });
  }

  root.value = null;
  return root;
}


function quizInput({ questions, options}) {
  let answers = questions.map(() => null);
  let root = htl.html`<div
      style="
        display: grid;
        grid-template-columns: 10% 10% 70% 10%;"
    >
      ${options.map(
        (opt) => htl.html`<div style="font-weight: bold; font-size: HUGE">${opt}</div>`
      )}
      <div style="font-weight: bold">Statement</div>
      <div style="font-weight: bold"></div>
      ${Array.from(questions.entries(), ([i, [question, correct]]) =>
        quizInputRow({
          question,
          options,
          correct,
          onChange: (newAnswer) => {
            answers[i] = newAnswer;
            root.value = answers;
            root.dispatchEvent(new CustomEvent("input"));
          }
        })
      )}
    </div>`;
  root.value = answers;
  return root;
}

function quizInputRow({
  question,
  options,
  correct,
  onChange = () => {}
}) {
  let root = htl.html`<div>`;

  function setAnswer(answer, initial = false) {
    morph(
      root,
      htl.html`<div style="display: contents"> 
      <form style="display: contents">
        ${options.map(
          (opt) =>
            htl.html`<label>&emsp;</label> 
            <input  
              name=${question} &emsp;
              type="radio"
              value="${opt}"
              checked=${opt === answer}
              onChange=${() => setAnswer(opt)}
            >
            </input>`
        )}
      </form>
      <div>${question}</div>
      <div> &emsp; ${
       answer === null ? "" : answer === correct ? "✅" : "🔴"
      }</div>
    </div>`
    );

    root.value = answer;
    if (!initial) {
      root.dispatchEvent(new CustomEvent("input"));
      onChange(answer);
    }
  }

  setAnswer(null, true);
  return root;
}

morph = require("https://bundle.run/nanomorph@5.4.2")

Note🎲 Question 1

You read in a data set and notice that the name “Arsène” is displayed as “Arsène”. What is the most likely reason?

viewof answer_encoding_1 = singleChoiceInput({
  options: [
    "The original text contains a typo.",
    "The text was saved with one encoding (e.g., UTF-8) but read in with another one (e.g., latin1).",
    "R cannot display letters with accents.",
    "read.csv2() automatically removes special characters."
  ],
  correctIndex: 1
})

📚 More tutorials on this

You still have questions? The following tutorials & papers can help you with that:

• quanteda tutorials: Import data
• readtext documentation
• R for Data Science by Wickham et al., Chapter “Data import”

📌 Test your knowledge

Task 1 (Easy🔥)

Use the TV series data. How many TV series have an IMDb Rating of at least 9?