Session 9: Scraping Interactive Web Pages (part 2)
Introduction to Web Scraping and Data Management for Social Scientists
Johannes B. Gruber
2026-07-30
Browser automation
What is Browser Automation?
Definition: the process of using software to control web browsers and interact with web elements programmatically
Tools Involved: Common tools include Selenium, Puppeteer, and Playwright
These tools allow scripts to perform actions like clicking, typing, and navigating through web pages automatically
Common Uses of Browser Automation
Testing: Widely used in software development for automated testing of web applications to ensure they perform as expected across different environments and browsers
Task Automation: Simplifies repetitive tasks such as form submissions, account setups, or any routine that can be standardized across web interfaces
Browser Automation in Web Scraping
Dynamic Content Handling: Essential for scraping websites that load content dynamically with JavaScript. Automation tools can interact with the webpage, wait for content to load, and then scrape the data.
Simulation of User Interaction: Can mimic human browsing patterns to interact with elements (like dropdowns, sliders, etc.) that need to be manipulated to access data
Avoiding Detection: More sophisticated than basic scraping scripts, browser automation can help mimic human-like interactions, reducing the risk of being detected and blocked by anti-scraping technologies
Example: Google Maps
Goal
Check the communte time programatically
Extract the distance and time it takes to make a journey
The new read_html_live from rvest solves this by emulating a browser:
# loads a real web browsersess <-read_html_live("https://www.google.de/maps/dir/Armadale+St,+Glasgow,+UK/Lilybank+House,+Glasgow,+UK/@55.8626667,-4.2712892,14z/data=!3m1!4b1!4m14!4m13!1m5!1m1!1s0x48884155c8eadf03:0x8f0f8905398fcf2!2m2!1d-4.2163615!2d55.8616765!1m5!1m1!1s0x488845cddf3cffdb:0x7648f9416130bcd5!2m2!1d-4.2904601!2d55.8740368!3e0?entry=ttu")
You can have a look at the browser with:
sess$view()
Unfortunately, we do not get content yet. We first have to click on “Accept all”
Let’s use browser Automation
After manipulating something about the session, you need to read it into R again:
html <- page$body |>as.character() |>read_html()html |>html_text() |> stringr::str_detect("Drug suspect")
[1] TRUE
headline <- html |>html_elements(".article-title") |>html_text2()author <- html |>html_elements(".author-section a") |>html_text2()date <- html |>html_elements(".issue_date") |>html_text2() |> stringr::str_extract("\\w{3} \\d{1,2}, \\d{4}") |> lubridate::mdy()text <- html |>html_elements(".article-full-body p") |>html_text2() |>paste0(collapse ="\n\n")tibble::tibble(url = mb_link, date, author, headline, text)
# A tibble: 1 × 5
url date author headline text
<chr> <date> <chr> <chr> <chr>
1 https://mb.com.ph/2021/04/14/drug-suspect-ki… 2021-04-14 Fredd… Drug su… "A d…
Mission: success!
Summary: Browser Automation
What is it
remote control a browser to perform pre-defined steps
several available tools:
native R: chromote, used in read_html_live
native Python: Playwright, can be used from R with playwrightr (experimental)
native Java: Selenium, can be used from R with RSelenium (buggy and outdated)
native JavaScript: Puppeteer not R bindings
Opinionated tool collections:
scrapling: combines different tools for “stealthy” scraping
What are they good for?
Get content from pages which you can’t otherwise access
Load more content through automated scrolling on dynamic pages
Automate tasks like downloading files
Issues
Companies have mechanisms to counter scraping:
rate limiting requests per second/minute/day and user/IP(Twitter)
captchas (can be solved but quite complex)
Won’t get you around very obscure HTML code (Facebook)
Quite heavy and very slow compared to requests
Wrap Up
Save some information about the session for reproducibility.
