Introduction to Web Scraping and Data Management for Social Scientists

Essex Summer School in Social Science Data Analysis 2026

Author

Welcome to the Course

This is the course material for the Essex Summer School in Social Science Data Analysis course 2H Introduction to Web Scraping and Data Management for Social Scientists, Monday 20 July–Friday 31 July 2026.

The course provides an introduction to the automated collection of data from the internet, as well as methods for efficient and safe data management. The first week covers different modes of storing data for individual researchers and research teams, and the theory and good practices behind them. The second week focuses on methods for collecting different kinds of data from the internet through various means.

Schedule

Slides are uploaded here as the course progresses.

Day	Date	Session	Materials	Exercises
1	Mon, 20 July	Introduction	Slides	Exercises
2	Tue, 21 July	Data Structures and Wrangling	Slides	Exercises
3	Wed, 22 July	Working with Files	Slides	Exercises
4	Thu, 23 July	Linking and Joining Data & SQL	Slides	Exercises
5	Fri, 24 July	Database Software and Scaling your Analyses	Slides	Exercises
6	Mon, 27 July	Introduction to the Web	Slides	Exercises
7	Tue, 28 July	Static Web Pages	Slides	Exercises
8	Wed, 29 July	Application Programming Interfaces (APIs)	Slides	Exercises
9	Thu, 30 July	Interactive Web Pages	Part 1 / Part 2	–
10	Fri, 31 July	Building a Reproducible Research Project	Slides	–

Required Software

You should have several software applications installed before the start of the course:

R (version 4.1.0+)
An IDE, preferably RStudio Desktop, but you can also use VSCodium or VS Code or Positron
Quarto

Download Course Materials

Clone the repository using RStudio:

Create a new project (top left corner)
Select “Version Control” → “Git”
Use URL: https://github.com/JBGruber/ess-web-scraping-data-management.git

Install Required Packages

After cloning, you can install all required R packages with:

if (!requireNamespace("rlang", quietly = TRUE)) {
  install.packages("rlang", dependencies = TRUE)
}
rlang::check_installed("remotes")
if (!requireNamespace("jbgtemplates", quietly = TRUE)) {
  remotes::install_github("JBGruber/jbgtemplates")
}
jbgtemplates::install_deps(
  gh_packages = c(
    "JBGruber/paperboy",
    "JBGruber/playwrightr",
    "ccs-amsterdam/amcat4r"
  )
)

Course by Dr Johannes B. Gruber • GitHub Repository