Session 7: Scraping Static Web Pages

Introduction to Web Scraping and Data Management for Social Scientists

Johannes B. Gruber

2026-07-28

Introduction

This Course

Day	Session
1	Introduction
2	Data Structures and Wrangling
3	Working with Files
4	Linking and joining data & SQL
5	Database Software and Scaling your Analyses
6	Introduction to the Web
7	Static Web Pages
8	Application Programming Interface (APIs)
9	Interactive Web Pages
10	Building a Reproducible Research Project

The Plan for Today

In this session, we trap some docile data that wants to be found. We will:

Go over some parsing examples:
- Wikipedia: World World Happiness Report
Discuss some examples of good approaches to data wrangling
Go into a bit more detail on requesting raw data

Original Image Source: prowebscraper.com

Joe Caione via unsplash.com

Correction: xpaths

When I told you it’s not worth learning xpath, I was thinking about static xpaths like this:

/html/body/div[3]/div/div[3]/main/div[3]/div[3]/div[2]/section[4]/section/table/tbody/tr[1]/td[3]/div[1]/a

However, I forgot that dynmic xpath also exists, which looks like this:

"descendant-or-self::table/descendant::a"

Which is actually what rvest always uses under the hood by translating CSS to xpath with the selectr package:

selectr::css_to_xpath("table a")

[1] "descendant-or-self::table/descendant::a"

This dynamic xpath can be very useful and contains more functionality than CSS selectors, but it’s harder to learn and only useful in special cases

Example: World Happiness Report

Use your Browser to Scout

Use your Browser’s `Inspect` tool

Note: Might not be available on all browsers; use Chromium-based or Firefox.

Use `rvest` to scrape

library(rvest)
library(tidyverse)

# 1. Request & collect raw html
html <- read_html(
  "https://en.wikipedia.org/w/index.php?title=World_Happiness_Report&oldid=1165407285"
)

# 2. Parse
happy_table <- html |>
  html_elements(".wikitable") |> # select the right element
  html_table() |> # special function for tables
  pluck(3) # select the third table

# 3. No wrangling necessary
happy_table

# A tibble: 153 × 9
   `Overall rank` `Country or region` Score `GDP per capita` `Social support`
            <int> <chr>               <dbl>            <dbl>            <dbl>
 1              1 Finland              7.81             1.28             1.5 
 2              2 Denmark              7.65             1.33             1.50
 3              3 Switzerland          7.56             1.39             1.47
 4              4 Iceland              7.50             1.33             1.55
 5              5 Norway               7.49             1.42             1.50
 6              6 Netherlands          7.45             1.34             1.46
 7              7 Sweden               7.35             1.32             1.43
 8              8 New Zealand          7.3              1.24             1.49
 9              9 Austria              7.29             1.32             1.44
10             10 Luxembourg           7.24             1.54             1.39
# ℹ 143 more rows
# ℹ 4 more variables: `Healthy life expectancy` <dbl>,
#   `Freedom to make life choices` <dbl>, Generosity <dbl>,
#   `Perceptions of corruption` <dbl>

## Plot relationship wealth and life expectancy
ggplot(happy_table, aes(x = `GDP per capita`, y = `Healthy life expectancy`)) +
  geom_point() +
  geom_smooth(method = 'lm')

Exercises 1

Get the table with 2025 opinion polling for the next United Kingdom general election from https://en.wikipedia.org/w/index.php?title=Opinion_polling_for_the_next_United_Kingdom_general_election&oldid=1366502041
Wrangle and plot the data opinion polls

Example: UK prime ministers on Wikipedia

Use your Browser to Scout

Use `rvest` to scrape

# 1. Request & collect raw html
html <- read_html(
  "https://en.wikipedia.org/w/index.php?title=List_of_prime_ministers_of_the_United_Kingdom&oldid=1366130861"
) # I pinned this to the version 13:01, 26 July 2026 so that it still works, even if the site is updated

# 2. Parse
pm_table <- html |>
  html_elements(".wikitable") |>
  keep(function(x) str_detect(html_text(x), "List of prime ministers")) |>
  map(function(x) as_tibble(html_table(x), .name_repair = "unique")) |>
  bind_rows() |>
  as_tibble(.name_repair = "unique") |>
  filter(!duplicated(`Prime ministerOffice(lifespan)`))

# 3. No wrangling necessary
pm_table

# A tibble: 78 × 11
   Portrait...1 Portrait...2 Prime ministerOffice(lifespa…¹ `Term of office...4`
   <chr>        <chr>        <chr>                          <chr>               
 1 "Portrait"   "Portrait"   Prime ministerOffice(lifespan) Start               
 2 ""           ""           Robert Walpole[27]MP for King… 3 April 1721        
 3 ""           ""           Spencer Compton[28]1st Earl o… 16 February 1742    
 4 ""           ""           Henry Pelham[29]MP for Sussex… 27 August 1743[e]   
 5 ""           ""           Thomas Pelham-Holles[30]1st D… 16 March 1754       
 6 ""           ""           William Cavendish[31]4th Duke… 16 November 1756    
 7 ""           ""           Thomas Pelham-Holles[32]1st D… 29 June 1757        
 8 ""           ""           John Stuart[33]3rd Earl of Bu… 26 May 1762         
 9 ""           ""           George Grenville[34]MP for Bu… 16 April 1763       
10 ""           ""           Charles Watson-Wentworth[35]2… 13 July 1765        
# ℹ 68 more rows
# ℹ abbreviated name: ¹`Prime ministerOffice(lifespan)`
# ℹ 7 more variables: `Term of office...5` <chr>, `Term of office...6` <chr>,
#   `Mandate[a]` <chr>, `Ministerial offices held as prime minister` <chr>,
#   `Party[b]` <chr>, Government <chr>, MonarchReign <chr>

<td rowspan="4">
  <span class="anchor" id="18th_century"></span>
   <b>
     <a href="/wiki/Robert_Walpole" title="Robert Walpole">Robert Walpole</a>
   </b>
   <sup id="cite_ref-FOOTNOTEEccleshallWalker20021,_5EnglefieldSeatonWhite19951–5PrydeGreenwayPorterRoy199645–46_28-0" class="reference">
     <a href="#cite_note-FOOTNOTEEccleshallWalker20021,_5EnglefieldSeatonWhite19951–5PrydeGreenwayPorterRoy199645–46-28">[27]</a>
   </sup>
   <br>
   <span style="font-size:85%;">MP for <a href="/wiki/King%27s_Lynn_(UK_Parliament_constituency)" title="King's Lynn (UK Parliament constituency)">King's Lynn</a>
   <br>(1676–1745)
  </span>
</td>

links <- html |>
  html_elements(".wikitable") |>
  keep(function(x) str_detect(html_text(x), "List of prime ministers")) |>
  html_elements("td [style=\"font-size:100%;\"] a[title]") |>
  html_attr("href")
title <- html |>
  html_elements(".wikitable") |>
  keep(function(x) str_detect(html_text(x), "List of prime ministers")) |>
  html_elements("td [style=\"font-size:100%;\"] a[title]") |>
  html_text()
tibble(name = title, link = links)

# A tibble: 81 × 2
   name                     link                                                
   <chr>                    <chr>                                               
 1 Robert Walpole           https://en.wikipedia.org/wiki/Robert_Walpole        
 2 Spencer Compton          https://en.wikipedia.org/wiki/Spencer_Compton,_1st_…
 3 Henry Pelham             https://en.wikipedia.org/wiki/Henry_Pelham          
 4 Thomas Pelham-Holles     https://en.wikipedia.org/wiki/Thomas_Pelham-Holles  
 5 William Cavendish        https://en.wikipedia.org/wiki/William_Cavendish,_4t…
 6 Thomas Pelham-Holles     https://en.wikipedia.org/wiki/Thomas_Pelham-Holles  
 7 John Stuart              https://en.wikipedia.org/wiki/John_Stuart,_3rd_Earl…
 8 George Grenville         https://en.wikipedia.org/wiki/George_Grenville      
 9 Charles Watson-Wentworth https://en.wikipedia.org/wiki/Charles_Watson-Wentwo…
10 William Pitt the Elder   https://en.wikipedia.org/wiki/William_Pitt_the_Elder
# ℹ 71 more rows

Exercises 2

For extracting text, rvest has two functions: html_text and html_text2. Explain the difference. You can test your explanation with the example html below.

html <- "<p>This is some text
         some more text</p><p>A new paragraph!</p>
         <p>Quick Question, is web scraping:

         a) fun
         b) tedious
         c) I'm not sure yet!</p>" |>
  read_html()

Wikipedia recently started using absolute links. Before that, they used relative links, such as many other websites still do. Relative links look like this:

links <- c(
  "/wiki/Robert_Walpole",
  "/wiki/Spencer_Compton,_1st_Earl_of_Wilmington",
  "/wiki/Henry_Pelham",
  "/wiki/Thomas_Pelham-Holles",
  "/wiki/William_Cavendish,_4th_Duke_of_Devonshire",
  "/wiki/Thomas_Pelham-Holles",
  "/wiki/John_Stuart,_3rd_Earl_of_Bute",
  "/wiki/George_Grenville",
  "/wiki/Charles_Watson-Wentworth",
  "/wiki/William_Pitt_the_Elder",
  "/wiki/Augustus_FitzRoy,_3rd_Duke_of_Grafton",
  "/wiki/Frederick_North,_Lord_North",
  "/wiki/Charles_Watson-Wentworth",
  "/wiki/William_Petty,_2nd_Earl_of_Shelburne",
  "/wiki/William_Cavendish-Bentinck,_3rd_Duke_of_Portland",
  "/wiki/William_Pitt_the_Younger",
  "/wiki/Henry_Addington",
  "/wiki/William_Pitt_the_Younger",
  "/wiki/William_Grenville",
  "/wiki/William_Cavendish-Bentinck,_3rd_Duke_of_Portland",
  "/wiki/Spencer_Perceval",
  "/wiki/Robert_Jenkinson,_2nd_Earl_of_Liverpool",
  "/wiki/George_Canning",
  "/wiki/Frederick_John_Robinson",
  "/wiki/Arthur_Wellesley,_1st_Duke_of_Wellington",
  "/wiki/Charles_Grey,_2nd_Earl_Grey",
  "/wiki/William_Lamb,_2nd_Viscount_Melbourne",
  "/wiki/Arthur_Wellesley,_1st_Duke_of_Wellington",
  "/wiki/Robert_Peel",
  "/wiki/William_Lamb,_2nd_Viscount_Melbourne",
  "/wiki/Robert_Peel",
  "/wiki/Lord_John_Russell",
  "/wiki/Edward_Smith-Stanley,_14th_Earl_of_Derby",
  "/wiki/George_Hamilton-Gordon,_4th_Earl_of_Aberdeen",
  "/wiki/Henry_John_Temple",
  "/wiki/Edward_Smith-Stanley,_14th_Earl_of_Derby",
  "/wiki/Henry_John_Temple",
  "/wiki/John_Russell,_1st_Earl_Russell",
  "/wiki/Edward_Smith-Stanley,_14th_Earl_of_Derby",
  "/wiki/Benjamin_Disraeli",
  "/wiki/William_Ewart_Gladstone",
  "/wiki/Benjamin_Disraeli",
  "/wiki/William_Ewart_Gladstone",
  "/wiki/Robert_Gascoyne-Cecil,_3rd_Marquess_of_Salisbury",
  "/wiki/William_Ewart_Gladstone",
  "/wiki/Robert_Gascoyne-Cecil,_3rd_Marquess_of_Salisbury",
  "/wiki/William_Ewart_Gladstone",
  "/wiki/Archibald_Primrose,_5th_Earl_of_Rosebery",
  "/wiki/Robert_Gascoyne-Cecil,_3rd_Marquess_of_Salisbury",
  "/wiki/Arthur_Balfour",
  "/wiki/Henry_Campbell-Bannerman",
  "/wiki/H._H._Asquith",
  "/wiki/David_Lloyd_George",
  "/wiki/Bonar_Law",
  "/wiki/Stanley_Baldwin",
  "/wiki/Ramsay_MacDonald",
  "/wiki/Stanley_Baldwin",
  "/wiki/Ramsay_MacDonald",
  "/wiki/Stanley_Baldwin",
  "/wiki/Neville_Chamberlain",
  "/wiki/Winston_Churchill",
  "/wiki/Clement_Attlee",
  "/wiki/Winston_Churchill",
  "/wiki/Anthony_Eden",
  "/wiki/Harold_Macmillan",
  "/wiki/Alec_Douglas-Home",
  "/wiki/Harold_Wilson",
  "/wiki/Edward_Heath",
  "/wiki/Harold_Wilson",
  "/wiki/James_Callaghan",
  "/wiki/Margaret_Thatcher",
  "/wiki/John_Major",
  "/wiki/Tony_Blair",
  "/wiki/Gordon_Brown",
  "/wiki/David_Cameron",
  "/wiki/Theresa_May",
  "/wiki/Boris_Johnson",
  "/wiki/Liz_Truss",
  "/wiki/Rishi_Sunak",
  "/wiki/Keir_Starmer",
  "/wiki/Andy_Burnham"
)

How could you convert the links object above so that it contains actual URLs?

How could you add the links we extracted above to the pm_table to keep everything together?

Example: thomann customer reviews

Goal

Collect all reviews from a given Thomann page
Identify and extract relevant variables for each review

Scout

After clicking on reviews, then “Read all reviews”, these are the links to the first three pages:

https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm
https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?order=&page=2&rating=0
https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?order=&page=3&rating=0

URLs explained

Uniform Resource Locator (URL) is a key mechanisms used to identify resources on a website for retrieval

Getting all URLs to product reviews

We can spot two things:
- there are 2906 reviews for this product (478 in English)
- there are 45 pages with reviews
(here you can get all reviews. E.g., Thomann limits results to 10 pages per query)

pages <- 1:45
links <- paste0(
  "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm",
  "?page=",
  pages,
  "&order=&rating=0"
)
links

 [1] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=1&order=&rating=0" 
 [2] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=2&order=&rating=0" 
 [3] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=3&order=&rating=0" 
 [4] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=4&order=&rating=0" 
 [5] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=5&order=&rating=0" 
 [6] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=6&order=&rating=0" 
 [7] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=7&order=&rating=0" 
 [8] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=8&order=&rating=0" 
 [9] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=9&order=&rating=0" 
[10] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=10&order=&rating=0"
[11] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=11&order=&rating=0"
[12] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=12&order=&rating=0"
[13] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=13&order=&rating=0"
[14] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=14&order=&rating=0"
[15] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=15&order=&rating=0"
[16] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=16&order=&rating=0"
[17] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=17&order=&rating=0"
[18] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=18&order=&rating=0"
[19] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=19&order=&rating=0"
[20] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=20&order=&rating=0"
[21] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=21&order=&rating=0"
[22] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=22&order=&rating=0"
[23] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=23&order=&rating=0"
[24] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=24&order=&rating=0"
[25] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=25&order=&rating=0"
[26] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=26&order=&rating=0"
[27] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=27&order=&rating=0"
[28] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=28&order=&rating=0"
[29] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=29&order=&rating=0"
[30] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=30&order=&rating=0"
[31] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=31&order=&rating=0"
[32] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=32&order=&rating=0"
[33] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=33&order=&rating=0"
[34] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=34&order=&rating=0"
[35] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=35&order=&rating=0"
[36] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=36&order=&rating=0"
[37] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=37&order=&rating=0"
[38] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=38&order=&rating=0"
[39] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=39&order=&rating=0"
[40] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=40&order=&rating=0"
[41] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=41&order=&rating=0"
[42] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=42&order=&rating=0"
[43] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=43&order=&rating=0"
[44] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=44&order=&rating=0"
[45] "https://www.thomann.co.uk/beyerdynamic_dt770_pro80_ohm_reviews.htm?page=45&order=&rating=0"

Collecting the reviews from 1 page

# make sure to read the HTML in only once
html <- read_html(links[1])

# we select review elements first
reviews <- html |>
  html_elements(".customer-review")

# then extract the information from these reviews
title <- reviews |>
  html_elements(".review-intro__headline.js-headline-original") |>
  html_text2()

date <- reviews |>
  html_elements(".review-intro__author") |>
  html_text2() |>
  str_extract("\\d{2}.\\d{2}.\\d{4}") |>
  lubridate::dmy()

text <- reviews |>
  html_elements(".review-text .fx-text-collapsible__fallback") |>
  html_text2()

# the rating system on this page is very interesting, instead of showing you
# a number of stars, it overlays greyed out stars with a colour and sets a
# width. 60% width of filled stars means 3/5 stars
star_ratings <- function(pct) {
  as.integer(pct) / 100 * 5
}
rating <- reviews |>
  html_elements(".fx-rating-stars__filler") |>
  html_attr("style") |>
  str_extract("(?<=width: )\\d+?(?=%)") |>
  star_ratings()


tibble(date, rating, title, text)

# A tibble: 10 × 4
   date       rating title                                                 text 
   <date>      <dbl> <chr>                                                 <chr>
 1 2021-03-17      5 "Delivery to UK and Brexit"                           "I o…
 2 2020-05-29      5 "Absolutely gorgeous ..."                             "I a…
 3 2020-09-11      5 "YES YES YES"                                         "Fin…
 4 2017-12-19      5 "Great sound and quality"                             "I u…
 5 2022-12-11      1 "Unfortunately, Beyer is bought because others are b… "Onc…
 6 2024-12-20      5 "Great headphones for the price"                      "Bou…
 7 2019-08-23      5 "Used them for a few weeks now, love them."           "Fir…
 8 2026-02-24      4 "The Treble Trap"                                     "The…
 9 2022-04-08      5 "King in headphones world"                            "Tho…
10 2021-07-27      5 ""                                                    "For…

(Un)helpful reviews rating

helpful <- reviews |>
  html_elements(".action [data-vote-rating-type=\"up\"] .action__value") |>
  html_text2() |>
  as.integer()
helpful

 [1] 35 28 20 25 17  0  3  1  1  0

unhelpful <- reviews |>
  html_elements(".action [data-vote-rating-type=\"down\"] .action__value") |>
  html_text2() |>
  as.integer()
unhelpful

 [1] 7 1 5 0 1 0 1 0 0 0

tibble(date, rating, helpful, unhelpful, title, text)

# A tibble: 10 × 6
   date       rating helpful unhelpful title                               text 
   <date>      <dbl>   <int>     <int> <chr>                               <chr>
 1 2021-03-17      5      35         7 "Delivery to UK and Brexit"         "I o…
 2 2020-05-29      5      28         1 "Absolutely gorgeous ..."           "I a…
 3 2020-09-11      5      20         5 "YES YES YES"                       "Fin…
 4 2017-12-19      5      25         0 "Great sound and quality"           "I u…
 5 2022-12-11      1      17         1 "Unfortunately, Beyer is bought be… "Onc…
 6 2024-12-20      5       0         0 "Great headphones for the price"    "Bou…
 7 2019-08-23      5       3         1 "Used them for a few weeks now, lo… "Fir…
 8 2026-02-24      4       1         0 "The Treble Trap"                   "The…
 9 2022-04-08      5       1         0 "King in headphones world"          "Tho…
10 2021-07-27      5       0         0 ""                                  "For…

Make it safer: Iterate over cases, rather than variable values

you can go over a nodeset and extract every matching value, then assemble results
as soon as one element is missing in one of the nodesets, it errors
safer: treat nodeset of reviews as list and iterate over each element

check_len <- function(x, len = 1, .default = NA) {
  if (length(x) != len) {
    return(rep(.default, len))
  } else {
    return(x)
  }
}

parse_review <- function(r) {
  date <- r |>
    html_elements(".review-intro__author") |>
    html_text2() |>
    str_extract("\\d{2}.\\d{2}.\\d{4}") |>
    lubridate::dmy() |>
    check_len()

  rating <- r |>
    html_elements(".fx-rating-stars__filler") |>
    html_attr("style") |>
    str_extract("(?<=width: )\\d+?(?=%)") |>
    star_ratings() |>
    check_len()

  helpful <- r |>
    html_elements(".action [data-vote-rating-type=\"up\"] .action__value") |>
    html_text2() |>
    as.integer() |>
    check_len()

  unhelpful <- r |>
    html_elements(".action [data-vote-rating-type=\"down\"] .action__value") |>
    html_text2() |>
    as.integer() |>
    check_len()

  title <- r |>
    html_elements(".review-intro__headline.js-headline-original") |>
    html_text2() |>
    check_len()

  text <- r |>
    html_elements(".review-text .fx-text-collapsible__fallback") |>
    html_text2() |>
    check_len()

  tibble(date, rating, helpful, unhelpful, title, text)
}

map(reviews, parse_review) |>
  bind_rows()

# A tibble: 10 × 6
   date       rating helpful unhelpful title                               text 
   <date>      <dbl>   <int>     <int> <chr>                               <chr>
 1 2021-03-17      5      35         7 "Delivery to UK and Brexit"         "I o…
 2 2020-05-29      5      28         1 "Absolutely gorgeous ..."           "I a…
 3 2020-09-11      5      20         5 "YES YES YES"                       "Fin…
 4 2017-12-19      5      25         0 "Great sound and quality"           "I u…
 5 2022-12-11      1      17         1 "Unfortunately, Beyer is bought be… "Onc…
 6 2024-12-20      5       0         0 "Great headphones for the price"    "Bou…
 7 2019-08-23      5       3         1 "Used them for a few weeks now, lo… "Fir…
 8 2026-02-24      4       1         0 "The Treble Trap"                   "The…
 9 2022-04-08      5       1         0 "King in headphones world"          "Tho…
10 2021-07-27      5       0         0 ""                                  "For…

Iterate over pages to collect all reviews

get_reviews_from_page <- function(link) {
  html <- try(read_html(link))

  while (is(html, "try-error")) {
    Sys.sleep(3)
    html <- try(read_html(link))
  }

  reviews <- html |>
    html_elements(".customer-review")

  map(reviews, parse_review) |>
    bind_rows()
}

all_reviews <- map(
  links[1:3],
  get_reviews_from_page,
  .progress = interactive()
) |>
  bind_rows()
glimpse(all_reviews)

Rows: 30
Columns: 6
$ date      <date> 2021-03-17, 2020-05-29, 2020-09-11, 2017-12-19, 2022-12-11,…
$ rating    <dbl> 5, 5, 5, 5, 1, 5, 5, 4, 5, 5, 5, 5, 5, 5, 5, 5, 3, 4, 5, 5, …
$ helpful   <int> 35, 28, 20, 25, 17, 0, 3, 1, 1, 0, 1, 0, 4, 2, 0, 1, 1, 0, 0…
$ unhelpful <int> 7, 1, 5, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
$ title     <chr> "Delivery to UK and Brexit", "Absolutely gorgeous ...", "YES…
$ text      <chr> "I ordered these, together with a headband and a pair of Sen…

Exercises 3

We might be interested in product specific rating scales (sound, comfort, and quality)

Extract that information from the first review page (hint: it works similar to the star rating and you can reuse that function)
Each review has 3 feature ratings, extract the labels to those ratings and combine them with the ratings themselves into a data.frame
Add the variables sound, comfort, and quality to the parse_review function (hint: you will need pivot_wider)
Create all_reviews again, but with the feature ratings this time

Example: Getting content from embedded json

Goal

Collect news articles from news.sky.com
Get the text of an article, the headline, date, and author

Scout

Let’s look for the date information in the page source

A Wild JSON string Appears!

JavaScript Object Notation (json) is a way of storing complicated nested data in plain text (see session 3)
data is put into a character string that indicates object types and relation of objects
R knows how to read json strings/files and can easily process them

library(jsonlite)
json_string <- list(x = 1:10, y = list(z = 1:10, a = LETTERS[1:10])) |>
  toJSON()
json_string

{"x":[1,2,3,4,5,6,7,8,9,10],"y":{"z":[1,2,3,4,5,6,7,8,9,10],"a":["A","B","C","D","E","F","G","H","I","J"]}}

fromJSON(json_string)

$x
 [1]  1  2  3  4  5  6  7  8  9 10

$y
$y$z
 [1]  1  2  3  4  5  6  7  8  9 10

$y$a
 [1] "A" "B" "C" "D" "E" "F" "G" "H" "I" "J"

essentially we seem to get pre-packaged data here

Obtain the JSON string

# 1. Request & collect raw html
html <- read_html(
  "https://web.archive.org/web/20250819084103/https://news.sky.com/story/crowdstrike-company-that-caused-global-techno-meltdown-offers-partners-10-vouchers-to-say-sorry-and-they-dont-work-13184488"
)

# 2. Parse
json_string <- html |>
  rvest::html_element("[type=\"application/ld+json\"]") |>
  rvest::html_text()

# 3. wrangling (part 1)
data <- jsonlite::fromJSON(json_string)

Why I use the Internet Archive’s version:

html <- read_html(
  "https://news.sky.com/story/crowdstrike-company-that-caused-global-techno-meltdown-offers-partners-10-vouchers-to-say-sorry-and-they-dont-work-13184488"
)

Error in `open.connection()`:
! cannot open the connection

Wrangling part 2

From here it is really straightforward to extract (most of) the relevant information:

# datetime
datetime <- pluck(data, "datePublished") |>
  lubridate::as_datetime()

# headline
headline <- pluck(data, "headline")

# author
author <- pluck(data, "author", "name")

The only thing missing from this data is the article itself…

Getting the article

text <- html |>
  rvest::html_elements(".sdc-article-body p") |>
  rvest::html_text2() |>
  paste(collapse = "\n")

sky_article <- tibble(datetime, author, headline, text)
glimpse(sky_article)

Rows: 1
Columns: 4
$ datetime <dttm> 2024-07-24 20:33:00
$ author   <chr> "Claire Gilbody Dickerson, news reporter"
$ headline <chr> "CrowdStrike: Company that caused global techno meltdown off…
$ text     <chr> "The firm behind the global IT outage that cost companies bil…

Exercises 4

Get the author, publication datetime, headline and text from this site: https://www.cnet.com/tech/services-and-software/facebook-hopes-to-normalize-idea-of-data-scraping-leaks-says-leaked-internal-memo/ (hint: it works in a very similar way, but you have to apply one extra data wrangling step)

Example: zeit.de

Special Requests: Behind Paywall

Let’s get this cool data journalism article.

html <- read_html(
  "https://www.zeit.de/mobilitaet/2024-04/deutschlandticket-klimaschutz-oeffentliche-verkehrsmittel-autos-verkehrswende"
)
html |>
  html_elements(".article-body p") |>
  html_text2()

[1] "Ganz Deutschland fährt Bahn. So fühlte sich das im Sommer 2022 zumindest an, als das 9-Euro-Ticket für drei Monate für überfüllte Züge sorgte. Die Bundesregierung und viele Menschen zeigten sich begeistert: So leicht war es also, Bürgerinnen und Bürger für die umweltfreundlichen öffentlichen Verkehrsmittel zu begeistern, man muss nur ein günstiges Ticket für ganz Deutschland anbieten."
[2] "Aber als die Bundesregierung den Nachfolger vorstellte, waren viele enttäuscht. 49 Euro monatlich kostet das Deutschlandticket und ist nur im Abo erhältlich. Euphorisch war nur noch die Bundesregierung. Doch jetzt, ein Jahr nach dem Start, kann man sagen: zu Recht. Zumindest, was die Fahrgastzahlen angeht."

🤔 Wait, that’s only the first two paragraphs!

Tip

Websites use cookies to remember users (including logged in ones)

What are browser cookies

Small pieces of data stored on the user’s device by the web browser while browsing websites
Purpose:
- Session Management: Maintain user sessions by storing login information and keeping users logged in as they navigate a website.
- Personalization: Save user preferences, such as language settings or theme choices, to enhance user experience.
- Tracking and Analytics: Track user behavior across websites for analytics and targeted advertising.
We can use them in scraping:
- to get content from websites that require consent before giving access
- to authenticate as a user with content access privileges
- to access personalized content
- to simulate real user behavior, reducing the chances of getting blocked by websites with anti-scraping measures
You can use browser extensions like “Get cookies.txt” for Chromium-based browsers or “cookies.txt” for Firefox to save your cookies to a file
Implications:
- You need to keep cookies secure as they can authenticate others as you!

Special Requests: Behind Paywall Cookies!

library(cookiemonster)
add_cookies("cookies.txt")

library(httr2)
html <- request(
  "https://www.zeit.de/mobilitaet/2024-04/deutschlandticket-klimaschutz-oeffentliche-verkehrsmittel-autos-verkehrswende"
) |> # start a request
  req_options(cookie = get_cookies("zeit.de", as = "string")) |> # add cookies to be sent with it
  req_perform() |>
  resp_body_html() # extract html from response

text <- html |>
  html_elements(".article-body p") |>
  html_text2()
length(text)

Example: South African Parliament (a special case)

Goal

Collect information about any financial interest of South Africa’s Members of Parliament

Collect links to PDF

library(httr2)
# I use the internet archive here as the real website is incredibly slow
html <- request(
  "https://web.archive.org/web/20240519142346/https://www.parliament.gov.za/register-members-Interests"
) |>
  req_timeout(100) |>
  req_perform() |>
  resp_body_html()

links <- html |>
  html_elements(".parly-h2+ul a") |>
  html_attr("href") |>
  # some links are malformed
  str_remove("^/web/20240519142346/")

years <- html |>
  html_elements(".parly-h2+ul a") |>
  html_text()

interest_pdfs <- tibble(
  link = links,
  year = years
) |>
  mutate(file_name = paste0("data/za/", year, ".pdf"))
interest_pdfs

# A tibble: 15 × 3
   link                                                          year  file_name
   <chr>                                                         <chr> <chr>    
 1 https://www.parliament.gov.za/storage/app/media/Register%20o… 2023  data/za/…
 2 https://www.parliament.gov.za/storage/app/media/Register%20o… 2022  data/za/…
 3 https://www.parliament.gov.za/storage/app/media/Register%20o… 2021  data/za/…
 4 https://www.parliament.gov.za/storage/app/media/Register%20o… 2020  data/za/…
 5 https://www.parliament.gov.za/storage/app/media/Register%20o… 2019  data/za/…
 6 https://web.archive.org/web/20240519142346/https://www.parli… 2018  data/za/…
 7 https://web.archive.org/web/20240519142346/https://www.parli… 2017  data/za/…
 8 https://web.archive.org/web/20240519142346/https://www.parli… 2016  data/za/…
 9 https://web.archive.org/web/20240519142346/https://www.parli… 2015  data/za/…
10 https://web.archive.org/web/20240519142346/https://www.parli… 2014  data/za/…
11 https://web.archive.org/web/20240519142346/https://www.parli… 2013  data/za/…
12 https://web.archive.org/web/20240519142346/https://www.parli… 2012  data/za/…
13 https://web.archive.org/web/20240519142346/https://www.parli… 2011  data/za/…
14 https://web.archive.org/web/20240519142346/https://www.parli… 2010  data/za/…
15 https://web.archive.org/web/20240519142346/https://www.parli… 2009  data/za/…

Download files for processing

dir.create("data/za", showWarnings = FALSE)
if (!file.exists("data/za/2018.pdf")) {
  # multi_download is a neat little function that parallesises file download
  curl::multi_download(
    urls = interest_pdfs$link,
    destfiles = interest_pdfs$file_name
  )
}

Scraping data from PDFs?

Data inside a PDF is actually not such an uncommon case
Many institutions share PDFs with tables, images and lists of data
We can use some of our new pattern finding skills to scrape data from these PDFs as well though
- Session names seem to be in a larger font and bold
- Paper titles are in italics
- Authors are either bold or plain font

Let’s investigate the PDF a little

library(pdftools)
interests_pdf <- pdf_data("data/za/2018.pdf", font_info = TRUE)
glimpse(interests_pdf[[2]])

Rows: 172
Columns: 8
$ width     <int> 78, 31, 24, 6, 31, 16, 26, 40, 42, 25, 6, 29, 6, 57, 52, 33,…
$ height    <int> 9, 9, 9, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, …
$ x         <int> 35, 115, 150, 41, 54, 87, 106, 134, 177, 54, 82, 90, 41, 54,…
$ y         <int> 65, 65, 65, 83, 83, 83, 83, 83, 83, 91, 91, 91, 109, 109, 10…
$ space     <lgl> TRUE, TRUE, FALSE, TRUE, TRUE, TRUE, TRUE, TRUE, FALSE, TRUE…
$ text      <chr> "Abraham-Ntantiso,", "Phoebe", "(ANC)", "1.", "SHARES", "AND…
$ font_name <chr> "Arial-BoldMT", "Arial-BoldMT", "Arial-BoldMT", "Arial-BoldM…
$ font_size <dbl> 8.775, 8.775, 8.775, 7.500, 7.500, 7.500, 7.500, 7.500, 7.50…

We see here that:

each page is an element in a list
each word is in one row of the table
it contains the font_size and font_name
the position of each word on tha page is given with x and y coordinates

Let’s investigate the PDF a little

Let’s investigate a few words we saw above:

# a politician name
interests_pdf[[2]] |>
  filter(str_detect(text, "Abrahams,"))

# A tibble: 1 × 8
  width height     x     y space text      font_name    font_size
  <int>  <int> <int> <int> <lgl> <chr>     <chr>            <dbl>
1    45      9    35   473 TRUE  Abrahams, Arial-BoldMT      8.78

# an item header
interests_pdf[[2]] |>
  filter(str_detect(text, "1"))

# A tibble: 7 × 8
  width height     x     y space text  font_name    font_size
  <int>  <int> <int> <int> <lgl> <chr> <chr>            <dbl>
1     6      8    41    83 TRUE  1.    Arial-BoldMT      7.50
2    10      8    39   350 TRUE  10.   Arial-BoldMT      7.50
3    10      8    40   376 TRUE  11.   Arial-BoldMT      7.50
4    10      8    39   403 TRUE  12.   Arial-BoldMT      7.50
5    10      8    39   429 TRUE  13.   Arial-BoldMT      7.50
6     6      8    41   492 TRUE  1.    Arial-BoldMT      7.50
7    10      8    39   749 TRUE  10.   Arial-BoldMT      7.50

# a disclose
interests_pdf[[2]] |>
  filter(str_detect(text, "disclose"))

# A tibble: 19 × 8
   width height     x     y space text      font_name font_size
   <int>  <int> <int> <int> <lgl> <chr>     <chr>         <dbl>
 1    29      8    90    91 FALSE disclose. ArialMT        7.50
 2    29      8    90   118 FALSE disclose. ArialMT        7.50
 3    29      8    90   144 FALSE disclose. ArialMT        7.50
 4    29      8    90   170 FALSE disclose. ArialMT        7.50
 5    29      8    90   196 FALSE disclose. ArialMT        7.50
 6    29      8    90   268 FALSE disclose. ArialMT        7.50
 7    29      8    90   295 FALSE disclose. ArialMT        7.50
 8    29      8    90   358 FALSE disclose. ArialMT        7.50
 9    29      8    90   385 FALSE disclose. ArialMT        7.50
10    29      8    90   411 FALSE disclose. ArialMT        7.50
11    29      8    90   437 FALSE disclose. ArialMT        7.50
12    29      8    90   500 FALSE disclose. ArialMT        7.50
13    29      8    90   526 FALSE disclose. ArialMT        7.50
14    29      8    90   553 FALSE disclose. ArialMT        7.50
15    29      8    90   579 FALSE disclose. ArialMT        7.50
16    29      8    90   605 FALSE disclose. ArialMT        7.50
17    29      8    90   631 FALSE disclose. ArialMT        7.50
18    29      8    90   658 FALSE disclose. ArialMT        7.50
19    29      8    90   684 FALSE disclose. ArialMT        7.50

# a word inside table
interests_pdf[[2]] |>
  filter(str_detect(text, "Pringle"))

# A tibble: 1 × 8
  width height     x     y space text    font_name font_size
  <int>  <int> <int> <int> <lgl> <chr>   <chr>         <dbl>
1    23      8   188   721 TRUE  Pringle ArialMT        7.50

# a table header
interests_pdf[[2]] |>
  filter(str_detect(text, "Description"))

# A tibble: 3 × 8
  width height     x     y space text        font_name    font_size
  <int>  <int> <int> <int> <lgl> <chr>       <chr>            <dbl>
1    41      8    56   223 FALSE Description Arial-BoldMT      7.50
2    41      8    56   322 FALSE Description Arial-BoldMT      7.50
3    41      8    56   711 FALSE Description Arial-BoldMT      7.50

Findings:

It looks like we can say relatively easily where a new politician entry starts based on the font
The item header has the same font name, but a different size
We can tell quite easily on which items there is nothing to disclose
The table colnames are similar to item headers, but start at a different x location

Test extract info from one page

p1 <- interests_pdf[[2]]

# add whether politician name
p1 |>
  mutate(
    is_name = font_name == "Arial-BoldMT" &
      round(font_size, 3) == 8.775
  ) |>
  # add whether header
  mutate(
    is_header = font_name == "Arial-BoldMT" &
      round(font_size, 1) == 7.5
  )

# A tibble: 172 × 10
   width height     x     y space text     font_name font_size is_name is_header
   <int>  <int> <int> <int> <lgl> <chr>    <chr>         <dbl> <lgl>   <lgl>    
 1    78      9    35    65 TRUE  Abraham… Arial-Bo…      8.78 TRUE    FALSE    
 2    31      9   115    65 TRUE  Phoebe   Arial-Bo…      8.78 TRUE    FALSE    
 3    24      9   150    65 FALSE (ANC)    Arial-Bo…      8.78 TRUE    FALSE    
 4     6      8    41    83 TRUE  1.       Arial-Bo…      7.50 FALSE   TRUE     
 5    31      8    54    83 TRUE  SHARES   Arial-Bo…      7.50 FALSE   TRUE     
 6    16      8    87    83 TRUE  AND      Arial-Bo…      7.50 FALSE   TRUE     
 7    26      8   106    83 TRUE  OTHER    Arial-Bo…      7.50 FALSE   TRUE     
 8    40      8   134    83 TRUE  FINANCI… Arial-Bo…      7.50 FALSE   TRUE     
 9    42      8   177    83 FALSE INTERES… Arial-Bo…      7.50 FALSE   TRUE     
10    25      8    54    91 TRUE  Nothing  ArialMT        7.50 FALSE   FALSE    
# ℹ 162 more rows

Wrangle into shape

p1_df <- p1 |>
  # one line in the PDF is all on the same y position
  group_by(y) |>
  # so we can summarise, ie. make one row out of one line
  summarise(
    # we retain only the first value from font_name, font_size and x
    # since they are all the same anyway
    font_name = head(font_name, 1),
    font_size = head(font_size, 1),
    x = head(x, 1),
    # we use paste with collapse to get several character values into
    # one string per line
    text = paste(text, collapse = " "),
    # dropping groups as we don't need them
    .groups = "drop"
  ) |>
  # we check whether a line is a name
  mutate(
    is_name = font_name == "Arial-BoldMT" &
      round(font_size, 3) == 8.775
  ) |>
  # and add a unique ID per person
  mutate(id = cumsum(is_name)) |>
  # now we do the same per disclosure item
  mutate(
    is_header = font_name == "Arial-BoldMT" &
      round(font_size, 1) == 7.5 &
      x < 50
  ) |>
  mutate(item_id = cumsum(is_header))

Wrangle into shape

p1_df_tidy <- p1_df |>
  # we group by person
  group_by(id) |>
  # and add a new variable with their name
  mutate(
    name = text[is_name],
  ) |>
  ungroup() |>
  # now we can remove the rows that contain the name in the text
  filter(!is_name) |>
  # we do the same per item
  group_by(item_id) |>
  mutate(
    item = text[is_header],
    content = paste(text[!is_header], collapse = "\n")
  ) |>
  # this produces a lot of duplicates, which we can get rid of with distinct
  distinct(name, item, content)
glimpse(p1_df_tidy)

Rows: 23
Columns: 4
Groups: item_id [23]
$ item_id <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,…
$ name    <chr> "Abraham-Ntantiso, Phoebe (ANC)", "Abraham-Ntantiso, Phoebe (A…
$ item    <chr> "1. SHARES AND OTHER FINANCIAL INTERESTS", "2. REMUNERATED EMP…
$ content <chr> "Nothing to disclose.", "Nothing to disclose.", "Nothing to di…

Apply to whole PDF

First, we have to add a page number to each site:

for (p in seq_along(interests_pdf)) {
  interests_pdf[[p]]$page <- p
}

Apply to whole PDF

I’m sorry for this long code 😅

interests_df <- bind_rows(interests_pdf) |>
  # one line in the PDF is all on the same y position
  group_by(page, y) |>
  # so we can summarise, ie. make one row out of one line
  summarise(
    # we retain only the first value from font_name, font_size and x
    # since they are all the same anyway
    font_name = head(font_name, 1),
    font_size = head(font_size, 1),
    x = head(x, 1),
    # we use paste with collapse to get several character values into
    # one string per line
    text = paste(text, collapse = " "),
    # dropping groups as we don't need them
    .groups = "drop"
  ) |>
  # we check whether a line is a name
  mutate(
    is_name = font_name == "Arial-BoldMT" &
      round(font_size, 3) == 8.775
  ) |>
  # and add a unique ID per person
  mutate(id = cumsum(is_name)) |>
  # everything before the first name can be removed
  filter(id != 0) |>
  # now we do the same per disclosure item
  mutate(
    is_header = font_name == "Arial-BoldMT" &
      round(font_size, 1) == 7.5 &
      x < 50
  ) |>
  mutate(item_id = cumsum(is_header)) |>
  # we group by person
  group_by(id) |>
  # and add a new variable with their name
  mutate(
    name = text[is_name],
  ) |>
  ungroup() |>
  # now we can remove the rows that contain the name in the text
  filter(!is_name) |>
  # we do the same per item
  group_by(item_id) |>
  mutate(
    item = text[is_header],
    content = paste(text[!is_header], collapse = "\n")
  ) |>
  # this produces a lot of duplicates, which we can get rid of with distinct
  ungroup() |>
  distinct(id, name, item, content)

Exercises 5

First, review the material. What you should take from this section:

websites might contain files containing the data you are interested in. Scraping the links and then downloading the files is a quick way to get access
It is possible to extract data from PDFs, but not trivial
The package pdftools has a pdf_data function which gives you much more flexibility to extract information systematically than pdf_text

That said, you can practice working with PDFs a litte

In the folder /data (relative to this document) there is a PDF with some text. Read it into R
The PDF has two columns, parse the left column of the first page into one object and the right into another
Now combine them and bring the text in the right order as a human would read it

Bonus:

Let’s assume you wanted to have this text in a table with one column indicating the section and one having the text of the section
Now let’s assume you wanted to parse this on the paragraph level instead (hint: remember str_split)

Optional Homework

You have seen some tools and tricks to scrape websites now. But your best ally in web scraping is experience! Now your task is to find a page on Wikipedia you find interesting and scrape content from there. Even if you don’t fully succeed, document the steps you take and note down where the information can be found. If you want to try to get some data you actually need from a different website, your’re also welcome! But note that if you collect raw html in R and the data is not where it should be (e.g., the html elements containing panel names do not exist), you might have discovered a more advanced site, which we will cover later. Note that down and try another website.

Deadline: Friday before class

Wrap Up

Save some information about the session for reproducibility.

Show Session Info

sessionInfo()

R version 4.6.1 (2026-06-24)
Platform: x86_64-pc-linux-gnu
Running under: EndeavourOS

Matrix products: default
BLAS:   /usr/lib/libblas.so.3.12.0 
LAPACK: /usr/lib/liblapack.so.3.12.0  LAPACK version 3.12.0

locale:
 [1] LC_CTYPE=en_GB.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_GB.UTF-8        LC_COLLATE=en_GB.UTF-8    
 [5] LC_MONETARY=en_GB.UTF-8    LC_MESSAGES=en_GB.UTF-8   
 [7] LC_PAPER=en_GB.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C       

time zone: Europe/Berlin
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] pdftools_3.9.0   httr2_1.2.3      jsonlite_2.0.0   rvest_1.0.4     
 [5] lubridate_1.9.5  forcats_1.0.1    stringr_1.6.0    dplyr_1.2.1     
 [9] purrr_1.2.2      readr_2.2.0      tidyr_1.3.2      tibble_3.3.1    
[13] ggplot2_4.0.3    tidyverse_2.0.0  tinytable_0.17.0

loaded via a namespace (and not attached):
 [1] gtable_0.3.6       xfun_0.59          websocket_1.4.4    processx_3.9.0    
 [5] lattice_0.22-9     tzdb_0.5.0         vctrs_0.7.3        tools_4.6.1       
 [9] ps_1.9.3           generics_0.1.4     curl_7.1.0         pkgconfig_2.0.3   
[13] Matrix_1.7-5       RColorBrewer_1.1-3 S7_0.2.2           lifecycle_1.0.5   
[17] compiler_4.6.1     farver_2.1.2       chromote_0.5.1     codetools_0.2-20  
[21] htmltools_0.5.9    yaml_2.3.12        later_1.4.8        pillar_1.11.1     
[25] nlme_3.1-169       tidyselect_1.2.1   digest_0.6.39      stringi_1.8.7     
[29] labeling_0.4.3     splines_4.6.1      fastmap_1.2.0      grid_4.6.1        
[33] cli_3.6.6          magrittr_2.0.5     utf8_1.2.6         withr_3.0.3       
[37] scales_1.4.0       promises_1.5.0     rappdirs_0.3.4     timechange_0.4.0  
[41] rmarkdown_2.31     httr_1.4.8         otel_0.2.0         qpdf_1.4.1        
[45] askpass_1.2.1      hms_1.1.4          evaluate_1.0.5     knitr_1.51        
[49] mgcv_1.9-4         rlang_1.2.0        Rcpp_1.1.1-1.1     glue_1.8.1        
[53] selectr_0.4-2      xml2_1.6.0         R6_2.6.1

Session 7: Scraping Static Web Pages

Introduction

This Course

The Plan for Today

Correction: xpaths

Example: World Happiness Report

Use your Browser to Scout

Use your Browser’s Inspect tool

Use rvest to scrape

Exercises 1

Example: UK prime ministers on Wikipedia

Use your Browser to Scout

Use rvest to scrape

Exercises 2

Example: thomann customer reviews

Goal

Scout

URLs explained

Getting all URLs to product reviews

Collecting the reviews from 1 page

(Un)helpful reviews rating

Make it safer: Iterate over cases, rather than variable values

Iterate over pages to collect all reviews

Exercises 3

Example: Getting content from embedded json

Goal

Scout

A Wild JSON string Appears!

Obtain the JSON string

Wrangling part 2

Getting the article

Exercises 4

Example: zeit.de

Special Requests: Behind Paywall

What are browser cookies

Special Requests: Behind Paywall Cookies!

Example: South African Parliament (a special case)

Goal

Collect links to PDF

Download files for processing

Scraping data from PDFs?

Let’s investigate the PDF a little

Let’s investigate the PDF a little

Test extract info from one page

Wrangle into shape

Wrangle into shape

Apply to whole PDF

Apply to whole PDF

Exercises 5

Optional Homework

Wrap Up

Use your Browser’s `Inspect` tool

Use `rvest` to scrape

Use `rvest` to scrape