This Course

The Plan for Today

Forms to get data from the web

Web Scraping

Why Should You Learn Web Scraping?

Found vs Designed Data

What is HTML

• HTML (HyperText Markup Language) is the standard markup language for documents designed to be displayed in a web browser
• Contains the raw data (text, URLs to pictures and videos) plus defines the layout and some of the styling of text

Image Source: Wikipedia.org

Example: Simple

<!DOCTYPE html>
<html>
<head>
    <title>My Simple HTML Page</title>
</head>
<body>
    <p>This is the body of the text.</p>
</body>
</html>

Example: With headline and author

<!DOCTYPE html>
<html>
<head>
    <title>My Simple HTML Page</title>
</head>
<body>
    <h1>My Headline</h1>
    <p class="author" href="https://www.johannesbgruber.eu/">Me</p>
    <p>This is the body of the text.</p>
</body>
</html>

Example: With some data

<!DOCTYPE html>
<html>
<head>
    <title>My Simple HTML Page</title>
</head>
<body>
    <h1>My Headline</h1>
    <p class="author">Me</p>
    <p>This is the body of the text.</p>
    <p>Consider this data:</p>
    <table>
        <tr>
            <th>Name</th>
            <th>Age</th>
        </tr>
        <tr>
            <td>John</td>
            <td>25</td>
        </tr>
        <tr>
            <td>Mary</td>
            <td>26</td>
        </tr>
    </table>
</body>
</html>

Example: With an image

<!DOCTYPE html>
<html>
<head>
    <title>My Simple HTML Page</title>
</head>
<body>
    <h1>My Headline</h1>
    <p class="author">Me</p>
    <p>This is the body of the text.</p>
    <p>Consider this image:</p>
    <img src="https://upload.wikimedia.org/wikipedia/commons/thumb/1/11/3_month_old_Corgi_puppy.jpg/960px-3_month_old_Corgi_puppy.jpg" alt="About The Dog.">
</body>
</html>

Example: With a link

<!DOCTYPE html>
<html>
<head>
    <title>My Simple HTML Page</title>
</head>
<body>
    <h1>My Headline</h1>
    <a href="https://www.johannesbgruber.eu/">
      <p class="author">Me</p>
    </a>
    <p>This is the body of the text.</p>
</body>
</html>

What is CSS

• CSS (Cascading Style Sheets) is very often used in addition to HTML to control the presentation of a document
• Designed to enable the separation of content from things concerning the look, such as layout, colours, and fonts.
• The reason it is interesting for web scraping is that certain information often get the same styling

Example: CSS

<!DOCTYPE html>
<html>
<head>
    <title>My Simple HTML Page</title>
    <link rel="stylesheet" type="text/css" href="example.css">
</head>
<body>
  <h1 class="headline">My Headline</h1>
  <p class="author">Me</p>
  <div class="content">
    <p>This is the body of the text.</p>
    <img src="https://upload.wikimedia.org/wikipedia/commons/thumb/1/11/3_month_old_Corgi_puppy.jpg/960px-3_month_old_Corgi_puppy.jpg" alt="About The Dog.">
    <p>Consider this data:</p>
    <table>
      <tr class="top-row">
          <th>Name</th>
          <th>Age</th>
      </tr>
      <tr>
          <td>John</td>
          <td>25</td>
      </tr>
      <tr>
          <td>Mary</td>
          <td>26</td>
      </tr>
    </table>
  </div>
</body>
</body>
</html>

/* CSS file */

.headline {
  color: red;
}

.author {
  color: grey;
  font-style: italic;
  font-weight: bold;
}

.top-row {
  background-color: lightgrey;
}

.content img {
  border: 2px solid black;
}

table, th, td {
  border: 1px solid black;
}

Using HTML tags:

You can select HTML elements by their tags

library(rvest)
read_html("data/example.html") |>
  html_elements("p") |>
  html_text2()

[1] "Me"                            "This is the body of the text."
[3] "Consider this image:"          "Consider this data:"          

• to select them, tags are written without the <>
• in theory, arbitrary tags are possible, but commonly people use <p> (paragraph), <br> (line break), <h1>, <h2>, <h3>, … (first, second, third, … level headline), <b> (bold), <i> (italic), <img> (image), <a> (hyperlink), and a couple more.

Using attributes

You can select elements by an attribute, including the class:

read_html("data/example.html") |>
  html_element("[class=\"headline\"]") |>
  html_text()

[1] "My Headline"

For class, there is also a shorthand:

read_html("data/example.html") |>
  html_element(".headline") |>
  html_text()

[1] "My Headline"

Another important shorthand is #, which selects the id attribute:

read_html("data/example.html") |>
  html_element("#table-1") |>
  html_table()

# A tibble: 2 × 2
  Name    Age
  <chr> <int>
1 John     25
2 Mary     26

read_html("data/example.html") |>
  html_element("#table-1 > tr")

{html_node}
<tr class="top-row">
[1] <th>Name</th>
[2] <th>Age</th>

Extracting attributes

Instead of selecting by attribute, you can also extract one or all attributes:

read_html("data/example.html") |>
  html_elements("a") |>
  html_attr("href")

[1] "https://www.johannesbgruber.eu/"   "https://en.wikipedia.org/wiki/Dog"

read_html("data/example.html") |>
  html_elements("a") |>
  html_attrs()

[[1]]
                             href 
"https://www.johannesbgruber.eu/" 

[[2]]
                               href 
"https://en.wikipedia.org/wiki/Dog" 

Chaining selectors

If there is more than one element that fits your selector, but you only want one of them, see if you can make your selection more specific by chaining selectors with > (for the immediate children) or an empty space (for any children and descendants of an element):

read_html("data/example.html") |>
  html_elements(".author>a") |>
  html_attr("href")

[1] "https://www.johannesbgruber.eu/"

read_html("data/example.html") |>
  html_elements(".author a") |>
  html_attr("href")

[1] "https://www.johannesbgruber.eu/"

Tip: there is also no rule against doing this instead:

read_html("data/example.html") |>
  html_elements(".author") |>
  html_elements("a") |>
  html_attr("href")

[1] "https://www.johannesbgruber.eu/"

Family Relations

Each html tag can contain other tags. To keep track of the relations we speak of ancestors, descendants, parents, children and siblings.

<book>
  <chapter>
    <section>
      <subsection>
        This is a subsection.
      </subsection>
      <subsection>
        This is another subsection.
      </subsection>
    </section>
    <section>
      This is a section.
    </section>
  </chapter>
  <chapter>
    <section>
      This is a section.
    </section>
    <section>
      This is a section.
    </section>
  </chapter>
  <chapter>
    This is a chapter without sections.
  </chapter>
</book>

Four Sets of Considerations

Before starting a scraping project, Brown et al. (2025) suggest working through four sets of questions:

• Legal: what am I allowed to access, collect, store and share?
• Ethical: what should I access, collect, store and share?
• Institutional: who do I need to talk to – and who has my back if something goes wrong?
• Scientific: is the data I end up with good enough to answer my question?

Legal: Personal Data

Collecting personal data of people in the EU might violate GDPR (General Data Protection Regulation)

• The GDPR defines personal data as “any information relating to an identified or identifiable natural person.” (Art. 4 GDPR)
• It applies if you are based in the EU or if you “monitor the behaviour” of people in the EU (Art. 3) – so it reaches researchers outside Europe as well
• You need a legal basis (Art. 6 GDPR):
  • legitimate interests (Art. 6(1)(f)): the usual route for academic scraping – the benefit of your research has to outweigh the risks for the people in the data
  • public interest task: processing is legitimate when “necessary for the performance of a task carried out in the public interest” (Art. 6(1)(e))
  • consent from the people whose data it is – the cleanest option, but almost never feasible at scale
• Special categories of data (Art. 9) – health, sexuality, religion, ethnicity, political opinions – get extra protection, though scientific research is privileged if the safeguards of Art. 89(1) are in place

Legal: Personal Data (continued)

Research is privileged under the GDPR – but the privileges come with homework (Brown et al. 2025)

• You usually do not have to inform every single person whose data you scraped, if that would take “disproportionate effort” or undermine the research aims (Art. 14(5)(b))
• Data subject rights can be limited where they would make the research impossible – e.g., where opt-outs would systematically skew the results (Art. 89)
• But the burden of proof is on you: you have to be able to show appropriate technical and organisational measures
  • Data minimisation: only collect the fields you actually need
  • Pseudonymisation: strip or hash identifiers as early in the pipeline as you can
  • Security: encrypted, access-controlled storage – not a folder in your Dropbox
  • A deletion plan: when does the raw data go away?
• For larger projects, a Data Protection Impact Assessment (Article 35 of the GDPR, see also this template) is a good way to write all of this down (and your data protection officer will ask for it anyway)

Legal: Hacking Laws

• The US Computer Fraud and Abuse Act (CFAA) is the law companies reach for when they want scraping stopped
  • violating a website’s terms of service alone is not “unauthorised access” (Sandvig v. Barr, 2020)
  • breaking technical controls can be: bypassing a login, evading IP blocks, or carrying on after a cease-and-desist letter
• The rule of thumb generalises well beyond the US – the more a site does to keep you out, the riskier it is to get in:
  • freely accessible page, no account needed → low risk
  • account needed → grey zone
  • technical barrier you had to defeat → high risk

ToS and Robots.txt

Twitter ToS

User-agent: *                                       # the rules apply to all user agents
Disallow: /episerver/                               # do not crawl any URLs that start with /episerver/
Disallow: /utils/                                   # do not crawl any URLs that start with /episerver/
Sitemap: https://www.parliament.uk/sitemapindex.xml # where to find the map of the site

https://www.parliament.uk/robots.txt

• Many companies have ToS that might prohibit you from scraping (see above: these are not laws, might not be binding and whether they can be enforced is a separate question)
• /robots.txt is often where guidelines are communicated to automated crawlers (these are not legally binding either)

Summary: Managing Legal Risk

• There is no one-size-fits-all answer – it depends on where you are, where the server is, and whose data you collect
• Publicly accessible and unauthenticated is much safer than logged-in and paywalled
• Collect the minimum, secure it properly, and write down what you did
• The goal is to manage legal risk, not to eliminate it – the only way to get zero risk is if you do nothing at all
• Brown et al. (2025) make an interesting point here: commercial scrapers (including AI companies) decided long ago to live with this uncertainty. If researchers are systematically more cautious than that, research falls behind the very companies it wants to study

Ethical

What is legal and what is decent are two different questions.

• Are there other means available to get to the data (e.g., via an API)?
• robots.txt might not be legally binding, but it is not nice to ignore it
• Scraping can put a heavy load on a website (if you make 1000s of requests) which costs the hosts money and might bring down a site (DDoS attack)
  • be a polite scraper: rate limit yourself, cache what you already downloaded so you don’t request it twice, and identify yourself in the user agent
• Think twice before scraping personal data. You should ask yourself:
  • is it necessary for your research?
  • are you harming anyone by obtaining (or distributing) the data?
  • do you really need everything or are parts of the data sufficient (e.g., can you preselect cases or ignore variables)

Ethical: “But it’s Public!”

Public availability is where the ethical question starts, not where it ends (Brown et al. 2025)

• People do not expect it: few users are aware that their public posts are used in research, and most think researchers should ask first (Fiesler and Proferes 2018)
• Context matters: data was shared in one context (a support forum, a local Facebook group) – is your use appropriate to that context? (Zimmer 2018)
• Third parties: replies, social graphs, images and tags drag in people who never posted anything themselves and cannot consent
• Vulnerable groups and sensitive topics deserve extra care – minors in particular are unlikely to understand that their data is used for research
• The whole lifecycle: your dataset can be repurposed for things you never intended (facial recognition, predictive policing, profiling)
  • limit what you share, document the intended use, add licence restrictions, and do not publish raw data when the risk of misuse is high

Institutional: Who to Talk To

You are not supposed to figure all of this out on your own – but the support structures are scattered and often do not talk to each other (Brown et al. 2025)

• Ethics committee / IRB: engage them early and often, as a partner rather than as a gatekeeper
  • most treat publicly available online data as “not human subjects research” or exempt it
  • but boards differ enormously between institutions, and few have scraping-specific expertise
• Legal counsel: your university’s legal office exists to protect the university, not you personally
  • for genuinely risky projects, external organisations (e.g., the Knight First Amendment Institute) can be the better address
• Research IT and data stewards: secure storage, access control, deletion protocols
  • technical rigour is what actually lowers your ethical and legal risk
• Data archives (e.g., GESIS, the Social Media Archive at ICPSR) can host data and advise on sharing

Scientific: Is the Data Any Good?

The most under-discussed set of considerations – scraped data has systematic quality problems (Brown et al. 2025)

• Sampling frame: you usually cannot draw a random sample, and you often cannot tell what your sample represents
  • what a search box or a feed returns is ranked, filtered and moderated by the platform – not by you
  • “everything with the hashtag #x” is not a population, it is whatever the platform decided to show you
• Missing data, missing non-randomly: rate limits and timeouts create gaps, and results usually arrive in the order they were created – so gaps cluster in time
  • platform metrics (“views”, “impressions”) are often estimated or sampled by the platform itself
• Temporal instability: sites change layout, algorithms and A/B tests without notice
  • best case your scraper breaks, worst case it keeps running and silently collects something else
• Construct validity: can you measure what you actually mean?
  • an impression is not a “read”; logged-out search results are not what a logged-in user sees
• Reproducibility: posts get deleted, accounts get banned, sites get redesigned – your study may not be repeatable a year later

Scientific: What to Do About It

• Ask first whether scraping is the right way to answer your question, or whether a better source exists
• Define the population you can actually describe up front and sample from it – avoid convenience and snowball samples
  • a complete, well-defined subsample (all MPs, all national newspapers, all municipalities) often beats a “random” sample of everything
• Test your pipeline: look for duplicates, missing pages and silent errors, scrape a few pages twice, and compare against something you can verify by hand
• Caveat your claims to what the data can support – “posts we could retrieve through X’s search interface in March 2026” is a very different population from “public discourse”
• Document everything: version control the scraper, publish code and metadata, and archive the raw responses where you are allowed to