Session 9: Scraping Interactive Web Pages

Introduction to Web Scraping and Data Management for Social Scientists

Johannes B. Gruber

2026-07-30

Introduction

This Course

Day Session
1 Introduction
2 Data Structures and Wrangling
3 Working with Files
4 Linking and joining data & SQL
5 Database Software and Scaling your Analyses
6 Introduction to the Web
7 Static Web Pages
8 Application Programming Interface (APIs)
9 Interactive Web Pages
10 Building a Reproducible Research Project

The Plan for Today

In this session, we learn how to hunt down wild data. We will:

  • Learn how to find secret APIs
  • Emulate a Browser
  • We focus specifically on step 1 below

Original Image Source: prowebscraper.com

Philipp Pilz via unsplash.com

Request & Collect Raw Data: a closer look

Common Problems

Imagine you wanted to scrape researchgate.net, since it contains self-created profiles of many researchers. However, when you try to get the html content:

library(rvest)
read_html("https://www.researchgate.net/profile/Johannes-Gruber-2")
Error in `open.connection()`:
! cannot open the connection

If you don’t know what an HTTP error means, you can go to https://http.cat and have the status explained in a fun way. Below I use a little convenience function:

error_cat <- function(error) {
  link <- paste0("https://http.cat/images/", error, ".jpg")
  knitr::include_graphics(link)
}
error_cat(403)

So what’s going on?

  • If something like this happens, the server essentially did not fullfill our request
  • This is because the website seems to have some special requirements for serving the (correct) content. These could be:
    • specific user agents
    • other specific headers
    • login through browser cookies
  • To find out how the browser manages to get the correct response, we can use the Network tab in the inspection tool

Strategy 1: Emulate what the Browser is Doing

Open the Inspect Window Again:

Strategy 1: Emulate what the Browser is Doing

But this time, we focus on the Network tab:

Here we get an overview of all the network activity of the browser and the individual requests for data that are performed. Clear the network log first and reload the page to see what is going on. Finding the right call is not always easy, but in most cases, we want:

  • a call with status 200 (OK/successful)
  • a document type
  • something that is at least a few kB in size
  • Initiator is usually “other” (we initiated the call by refreshing)

When you identified the call, you can right click -> copy -> copy as cURL

Refresher: cURL Calls

What is cURL:

  • cURL is a library that can make HTTP requests.
  • it is widely used for API calls from the terminal.
  • it lists the parameters of a call in a pretty readable manner:
    • the unnamed argument in the beginning is the Uniform Resource Locator (URL) the request goes to
    • -H arguments describe the headers, which are arguments sent with the call
    • -d is the data or body of a request, which is used e.g., for uploading things
    • -o/-O can be used to write the response to a file (otherwise the response is returned to the screen)
    • --compressed means to ask for a compressed response which is unpacked locally (saves bandwith)
curl 'https://www.researchgate.net/profile/Johannes-Gruber-2' \
  -H 'authority: www.researchgate.net' \
  -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7' \
  -H 'accept-language: en-GB,en;q=0.9' \
  -H 'cache-control: max-age=0' \
  -H '[Redacted]' \
  -H 'sec-ch-ua: "Chromium";v="115", "Not/A)Brand";v="99"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Linux"' \
  -H 'sec-fetch-dest: document' \
  -H 'sec-fetch-mode: navigate' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-user: ?1' \
  -H 'upgrade-insecure-requests: 1' \
  -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36' \
  --compressed

httr2::curl_translate()

  • We have seen httr2::curl_translate() in action yesterday (and the alternative https://curlconverter.com/r-httr2/)
  • It can also convert more complicated API calls that make look R no different from a regular browser
  • (Remember: you need to escape all " in the call, press ctrl + F to open the Find & Replace tool and put " in the find \" in the replace field and go through all matches except the first and last):
library(httr2)
httr2::curl_translate(
  "curl 'https://www.researchgate.net/profile/Johannes-Gruber-2' \
  -H 'authority: www.researchgate.net' \
  -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7' \
  -H 'accept-language: en-GB,en;q=0.9' \
  -H 'cache-control: max-age=0' \
  -H 'cookie: [Redacted]' \
  -H 'sec-ch-ua: \"Chromium\";v=\"115\", \"Not/A)Brand\";v=\"99\"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: \"Linux\"' \
  -H 'sec-fetch-dest: document' \
  -H 'sec-fetch-mode: navigate' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-user: ?1' \
  -H 'upgrade-insecure-requests: 1' \
  -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36' \
  --compressed"
)
request("https://www.researchgate.net/profile/Johannes-Gruber-2") |>
  req_cookies_set(
    `[Redacted]` = "",
  ) |>
  req_headers(
    authority = "www.researchgate.net",
    accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
    `accept-language` = "en-GB,en;q=0.9",
    `cache-control` = "max-age=0",
    cookie = "[Redacted]",
    `upgrade-insecure-requests` = "1",
    `user-agent` = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36",
  ) |>
  req_perform()

‘Emulating’ the Browser Request

resp <- request("https://www.researchgate.net/profile/Johannes-Gruber-2") |>
  req_headers(
    authority = "www.researchgate.net",
    accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
    `accept-language` = "en-GB,en;q=0.9",
    `cache-control` = "max-age=0",
    cookie = "[Redacted]",
    `sec-ch-ua` = "\"Chromium\";v=115\", \"Not/A)Brand\";v=\"99",
    `sec-ch-ua-mobile` = "?0",
    `sec-ch-ua-platform` = "\"Linux\"",
    `sec-fetch-dest` = "document",
    `sec-fetch-mode` = "navigate",
    `sec-fetch-site` = "cross-site",
    `sec-fetch-user` = "?1",
    `upgrade-insecure-requests` = "1",
    `user-agent` = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36",
  ) |>
  req_perform()
resp
<httr2_response>
GET https://www.researchgate.net/profile/Johannes-Gruber-2
Status: 200 OK
Content-Type: text/html
Body: In memory (326535 bytes)

Mission outcome: success

  • the result can be converted into an rvest object and parsed
resp |>
  resp_body_html() |>
  html_elements("[data-testid=\"publicProfileStatsCitations\"]") |>
  html_text()
[1] "269"

Example: anyflip

Goal

  • get information stored as “flipping book”
  • don’t take individual screenshots/pages, but get whole book

Scout

  • hovering over the page, we find this element

    <div class="side-image"   style="position:absolute;z-index:0;width:100%;height:100%;top:0;lef  t:0;">
  <img id="c7ARbTjMdA" alt="c7ARbTjMdA"   src="../files/large/ae105d2ec767250c5e9a4e2b0a48efaf.webp?174011361  2" style="pointer-events: none; width: 100%; height: 100%;   position: absolute; top: 0px; left: 0px; transform-origin: 0% 0%   0px; transform: scale(1);" width="100%" height="100%">
</div>

  • looks easy enough! .side-image img should give us the link!

  • hold down ctrl and click on the link in the inspect

Scout

Can we get the actual file?

dir.create("data/anyflip", showWarnings = FALSE)
out_file <- paste0("data/anyflip/1_ae105d2ec767250c5e9a4e2b0a48efaf.webp")
curl::curl_download(
  url = "https://online.anyflip.com/ogboc/rozd/files/large/ae105d2ec767250c5e9a4e2b0a48efaf.webp?1740113612",
  destfile = out_file
)
file.size(out_file)
[1] 166322

We can!

get images

dl_urls <- paste0(
  "https://online.anyflip.com/ogboc/rozd/files/large/",
  file_names
)
out_files <- paste0(
  "data/anyflip/",
  sprintf("%03d", seq_along(file_names)),
  "_",
  file_names
)
curl::multi_download(
  dl_urls,
  out_files
)
# A tibble: 136 × 10
   success status_code resumefrom url   destfile error type  modified           
   <lgl>         <dbl>      <dbl> <chr> <chr>    <chr> <chr> <dttm>             
 1 TRUE            200          0 http… /home/j… <NA>  imag… 2025-02-21 05:53:27
 2 TRUE            200          0 http… /home/j… <NA>  imag… 2025-02-21 05:53:27
 3 TRUE            200          0 http… /home/j… <NA>  imag… 2025-02-21 05:53:27
 4 TRUE            200          0 http… /home/j… <NA>  imag… 2025-02-21 05:53:27
 5 TRUE            200          0 http… /home/j… <NA>  imag… 2025-02-21 05:53:27
 6 TRUE            200          0 http… /home/j… <NA>  imag… 2025-02-21 05:53:27
 7 TRUE            200          0 http… /home/j… <NA>  imag… 2025-02-21 05:53:27
 8 TRUE            200          0 http… /home/j… <NA>  imag… 2025-02-21 05:53:27
 9 TRUE            200          0 http… /home/j… <NA>  imag… 2025-02-21 05:53:27
10 TRUE            200          0 http… /home/j… <NA>  imag… 2025-02-21 05:53:27
# ℹ 126 more rows
# ℹ 2 more variables: time <dbl>, headers <list>

Finally: get a single PDF

# convert each image to a pdf
for (page in out_files) {
  magick::image_read(page) |>
    magick::image_write(path = paste0(page, ".pdf"), format = "pdf")
}

# combine into one pdf
title <- read_html("https://anyflip.com/ogboc/rozd/") |>
  html_elements("title") |>
  html_text2() |>
  janitor::make_clean_names()

pdftools::pdf_combine(
  input = paste0(out_files, ".pdf"),
  output = paste0("data/anyflip/", title, ".pdf")
)
[1] "/home/johannes/Dropbox/Teaching/ess-web-scraping-data-management/09_Interactive_Web_Pages/data/anyflip/rayngan_sthankarn_thang_sangkhm_ssw_11_pi_2567_tpso_11_flip_pdf_any_flip.pdf"

Mission outcome: success!

To summarise the steps:

  1. poking around the website with the inspect tool
  2. found an image link that we could download
  3. the link was not actually in the HTML source, but must come from somewhere
  4. found the request that loads the ‘config’ JSON, containing all the image names in order
  5. guessed the format of the image links from the one link we got + the image names
  6. Downloaded the files -> success!

Extra: getting the text data?

  • since this is a pdf that was made from combining images, we have no actual text data
  • we can use Optical Character Recognition (and early version of visual ML models)
pdftools::pdf_ocr_text(
  "data/anyflip/002_ca317b05c3eeca7309063ce966b4b933.webp.pdf",
  language = "tha"
)
Converting page 1 to 002_ca317b05c3eeca7309063ce966b4b933.webp_1.png... done!
[1] "ค ํ า น ํ า\n\nส ํ า น ั ก ง า น ป ล ั ด ก ร ะ ท ร ว ง ก า ร พ ั ฒ น า ส ั ง ค ม แล ะ ค ว า ม ม ั น ค ง ขอ ง ม น ุ ษ ย ์ ได ้ ม อ บ ห ม า ย ให ้\nส ํ า น ั ก ง า น ส ่ ง เส ร ิ ม แล ะ ส น ั บ ส น ุ น ว ิ ชา ก า ร 1-11 (ส ส ว . 1-11) ด ํ า เน ิ น ก า ร โค ร ง ก า ร จ ั ด ท ํ า ร า ย ง า น ส ถา น ก า ร ณ์ ท า ง\nส ั ง ค ม ใน ร ะ ด ั บ ก ล ุ ่ ม จ ั ง ห ว ั ด เพ ื ่ อ ร ว บ ร ว ม ว ิ เค ร า ะ ห ์ ข้ อ ม ู ล ส ถา น ก า ร ณ์ ท า ง ส ั ง ค ม ท ี ่ เก ิ ด ขึ ้ น ใน เข ต ร ั บ ผิ ด ชอบ ขอ ง\nส ํ า น ั ก ง า น ส ่ ง เส ร ิ ม แล ะ ส น ั บ ส น ุ น ว ิ ชา ก า ร 11 ใน ก า ร ค า ด ก า ร ณ์ แน ว โน ้ ม ส ถา น ก า ร ณ์ ท า ง ส ั ง ค ม แล ะ ผล ก ร ะ ท บ\nพ ร ้ อ ม ท ั ้ ง เส น อ แน ะ แน ว ท า ง ใน ก า ร แก ้ ไข ป ั ญ ห า ท า ง ส ั ง ค ม ใน พ ื ้ น ท ี ่ ก ล ุ ่ ม จ ั ง ห ว ั ด ใน เข ต พ ื ้ น ท ี ่ ร ั บ ผิ ด ชอบ ขอ ง\nส ํ า น ั ก ง า น ส ่ ง เส ร ิ ม แล ะ ส น ั บ ส น ุ น ว ิ ชา ก า ร 11\n\nส ํ า น ั ก ง า น ส ่ ง เส ร ิ ม แล ะ ส น ั บ ส น ุ น ว ิ ชา ก า ร 11 (ส ส ว .11) ได ้ ด ํ า เน ิ น ก า ร จ ั ด ท ํ า ร า ย ง า น\nส ถา น ก า ร ณ์ ท า ง ส ั ง ค ม ร ะ ด ั บ ก ล ุ ่ ม จ ั ง ห ว ั ด ภา ค ไต ้ ต อ น ล ่ า ง ใน พ ื ้ น ท ี ่ เข ต ร ั บ ผิ ด ชอบ ขอ ง ส ํ า น ั ก ง า น ส ่ ง เส ร ิ ม แล ะ\nส น ั บ ส น ุ น ว ิ ชา ก า ร 11 ได ้ แก ่ จ ั ง ห ว ั ด ต ร ั ง น ร า ธิ ว า ส ป ั ต ต า น ี พ ั ท ล ุ ง ย ะ ล า ส ง ข ล า แล ะ ส ต ู ล ซึ ่ ง ป ร ะ ก อ บ ด ้ ว ย\n6 ส ่ ว น ได ้ แก ่ ส ่ ว น ท ี ่ 1 บ ท น ํ า ส ่ ว น ท ี ่ 2 ข้ อ ม ู ล พ ื ้ น ฐา น ใน พ ื ้ น ท ี ่ ก ล ุ ่ ม จ ั ง ห ว ั ด ส ่ ว น ท ี ่ 3 ส ถา น ก า ร ณ์ ก ล ุ ่ ม เป ้ า ห ม า ย\nท า ง ส ั ง ค ม ร ะ ด ั บ ก ล ุ ่ ม จ ั ง ห ว ั ด ส ่ ว น ท ี ่ 4 ส ถา น ก า ร ณ์ เช ิ ง ป ร ะ เด ็ น ท า ง ส ั ง ค ม แล ะ ส ถา น ก า ร ณ์ เร ่ ง ด ่ ว น (| ๐ 1 |55 น ๑ 5)\nใน ร ะ ด ั บ ก ล ุ ่ ม จ ั ง ห ว ั ด ส ่ ว น ท ี ่ 5 ก า ร ว ิ เค ร า ะ ห ์ แน ว โน ้ ม ขอ ง ส ถา น ก า ร ณ์ ท า ง ส ั ง ค ม ก ล ุ ่ ม จ ั ง ห ว ั ด ส ่ ว น ท ี ่ 6 บ ท ส ร ุ ป\nแล ะ ข้ อ เส น อ แน ะ\n\nผู ้ จ ั ด ท ํ า ห ว ั ง เป ็ น อ ย ่ า ง ย ิ ง ว ่ า ร า ย ง า น ส ถา น ก า ร ณ์ ท า ง ส ั ง ค ม ก ล ุ ่ ม จ ั ง ห ว ั ด ภา ค ใต ้ ต อ น ล ่ า ง\nป ร ะ จ ํ า ป ี 2567 ฉบับ น ี ้ จ ะ เป ็ น ป ร ะ โย ชน ์ ต ่ อ ห น ่ ว ย ง า น ร ะ ด ั บ ท ้ อ ง ถิ ่ น แล ะ ร ะ ด ั บ จ ั ง ห ว ั ด ส า ม า ร ถ น ํ า ข้ อ ม ู ล ใน\nพ ื ้ น ท ี ่ ไป ใช ้ ใน ก า ร ก ํ า ห น ด น โย บ า ย แผ น ง า น โค ร ง ก า ร ใน ก า ร ค ุ ้ ม ค ร อ ง ป ้ อ ง ก ั น แล ะ แก ้ ไข ป ั ญ ห า ท า ง ส ั ง ค ม ใน\nร ะ ด ั บ พ ื ้ น ท ี ่ แล ะ ห น ่ ว ย ง า น ร ะ ด ั บ ก ร ะ ท ร ว ง ส า ม า ร ถ น ํ า ข้ อ ม ู ล ใน ภา พ ร ว ม ไป ใช ้ ป ร ะ โย ชน ์ ว ิ เค ร า ะ ห ์ ส ถา น ก า ร ณ์\nป ั ญ ห า ท า ง ส ั ง ค ม ท ี ่ ส ํ า ค ั ญ แ ล ะ ก ํ า ห น ด น โย บ า ย แผ น ง า น ใน ก า ร ป ้ อ ง ก ั น แล ะ แก ้ ไข ป ั ญ ห า ส ั ง ค ม ภา พ ร ว ม ต ่ อ ไป\n\nส ํ า น ั ก ง า น ส ่ ง เส ร ิ ม แล ะ ส น ั บ ส น ุ น ว ิ ชา ก า ร 11\nก ร ก ฎา ค ม 2567\nร า ย ง า น ส ถา น ก า ร ณ์ ท า ง ส ั ง ค ม ก ล ุ ่ ม จ ั ง ห ว ั ด ภา ค ไต ้ ต อ น ล ่ า ง ป ร ะ จ ํ า ป ี 2567 | 1\n"

Example: X-Twitter

Goal

  1. Tweets from a Twitter profile
  2. Get the text, likes, shares and comments

Can we use rvest?

xhtml <- read_html("https://x.com/elonmusk")

At least one of these elements should be here!

xhtml |>
  html_text() |>
  str_detect("Harold Clarke Speaker Series")
[1] FALSE

Probing the hidden/internal API

Translating a request

curl_translate(
  "curl 'https://x.com/i/api/graphql/eoJ5zbv51Z_KVl81v9PmLQ/UserTweets?variables=%7B%22userId%22%3A%221525016244%22%2C%22count%22%3A20%2C%22includePromotedContent%22%3Atrue%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22withVoice%22%3Atrue%7D&features=%7B%22rweb_video_screen_enabled%22%3Afalse%2C%22rweb_cashtags_enabled%22%3Atrue%2C%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22responsive_web_profile_redirect_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Afalse%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22premium_content_api_read_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22responsive_web_grok_analyze_button_fetch_trends_enabled%22%3Afalse%2C%22responsive_web_grok_analyze_post_followups_enabled%22%3Atrue%2C%22rweb_cashtags_composer_attachment_enabled%22%3Atrue%2C%22responsive_web_jetfuel_frame%22%3Atrue%2C%22responsive_web_grok_share_attachment_enabled%22%3Atrue%2C%22responsive_web_grok_annotations_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22rweb_conversational_replies_downvote_enabled%22%3Afalse%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22content_disclosure_indicator_enabled%22%3Atrue%2C%22content_disclosure_ai_generated_indicator_enabled%22%3Atrue%2C%22responsive_web_grok_show_grok_translated_post%22%3Afalse%2C%22responsive_web_grok_analysis_button_from_backend%22%3Atrue%2C%22post_ctas_fetch_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Afalse%2C%22responsive_web_grok_image_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_imagine_annotation_enabled%22%3Atrue%2C%22responsive_web_grok_community_note_auto_translation_is_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D&fieldToggles=%7B%22withArticlePlainText%22%3Afalse%7D' \
  -H 'accept: */*' \
  -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
  -H 'authorization: Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA' \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -H 'Cookie: guest_id=v1%3A177825193349654388; __cuid=660c69ccdfa44dbbbdec8b0f8def2bfb; __cuid=660c69ccdfa44dbbbdec8b0f8def2bfb; gt=2082732292510752843; g_state={\"i_l\":0,\"i_ll\":1785398028357,\"i_b\":\"uuJazTIyCsGGnRpAueC4M3AP+qMsCZftFnl0ROWwGPo\",\"i_e\":{\"enable_itp_optimization\":24},\"i_et\":1785398028357}; twid=u%3D1632536605; auth_token=af61bd5f3081f1de1176b9f45b891e0bd32e7737; lang=en-gb; ct0=469c42f2e7ec07eeb835393c6807fd8823568e8d5d23a364b181fd8f11a8656e4ffc625a3af9d65d75347e61c67e28d374c570da3eafccdaeb03c38760b6602cf232a4cf4d7295c6541bf8ba87b911ca; d_prefs=MToxLGNvbnNlbnRfdmVyc2lvbjoyLHRleHRfdmVyc2lvbjoxMDAw; guest_id_ads=v1%3A177825193349654388; guest_id_marketing=v1%3A177825193349654388; personalization_id=\"v1_sr99TBLB43XWJFbObEbmhQ==\"; __eoi=ID=901e4be8a5cce0ab:T=1785398148:RT=1785398148:S=AA-Afjbc6IIvw0USuoei_0jNyzMd; __cf_bm=tEvicPKziyzqTVniInO0z.1NU3c8am.icSXR5BkYjOM-1785398232.84372-1.0.1.1-8KGcI1axtOgIeLU.q.8aQ.MX1ZxrKJLivbwivHIYB.gQGtLrSXdv2FBQJMViiBYLdgAmE.SiWHkaeAJN2TwUM.uoIFej3QEPjHb9Pwwv4CRJfNssiBvvr82fZ8xWWmOa' \
  -H 'pragma: no-cache' \
  -H 'priority: u=1, i' \
  -H 'referer: https://x.com/EssexSumSchool' \
  -H 'sec-ch-ua: \"Not;A=Brand\";v=\"8\", \"Chromium\";v=\"150\"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: \"Linux\"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36' \
  -H 'x-client-transaction-id: wnIueAigYpZD63Ns4zTPCTUi2jZeRCzdtpcpu1xfwJRPgXk9kjN89FUzwr2SvC6vcKli2cRfxIQ7DjM62HySvhlPs3ABwQ' \
  -H 'x-csrf-token: 469c42f2e7ec07eeb835393c6807fd8823568e8d5d23a364b181fd8f11a8656e4ffc625a3af9d65d75347e61c67e28d374c570da3eafccdaeb03c38760b6602cf232a4cf4d7295c6541bf8ba87b911ca' \
  -H 'x-twitter-active-user: yes' \
  -H 'x-twitter-auth-type: OAuth2Session' \
  -H 'x-twitter-client-language: en-GB'"
)
request("https://x.com/i/api/graphql/eoJ5zbv51Z_KVl81v9PmLQ/UserTweets") |>
  req_url_query(
    variables = '{"userId":"1525016244","count":20,"includePromotedContent":true,"withQuickPromoteEligibilityTweetFields":true,"withVoice":true}',
    features = '{"rweb_video_screen_enabled":false,"rweb_cashtags_enabled":true,"profile_label_improvements_pcf_label_in_post_enabled":true,"responsive_web_profile_redirect_enabled":true,"rweb_tipjar_consumption_enabled":false,"verified_phone_label_enabled":false,"creator_subscriptions_tweet_preview_api_enabled":true,"responsive_web_graphql_timeline_navigation_enabled":true,"premium_content_api_read_enabled":false,"communities_web_enable_tweet_community_results_fetch":true,"c9s_tweet_anatomy_moderator_badge_enabled":true,"responsive_web_grok_analyze_button_fetch_trends_enabled":false,"responsive_web_grok_analyze_post_followups_enabled":true,"rweb_cashtags_composer_attachment_enabled":true,"responsive_web_jetfuel_frame":true,"responsive_web_grok_share_attachment_enabled":true,"responsive_web_grok_annotations_enabled":true,"articles_preview_enabled":true,"responsive_web_edit_tweet_api_enabled":true,"rweb_conversational_replies_downvote_enabled":false,"graphql_is_translatable_rweb_tweet_is_translatable_enabled":true,"view_counts_everywhere_api_enabled":true,"longform_notetweets_consumption_enabled":true,"responsive_web_twitter_article_tweet_consumption_enabled":true,"content_disclosure_indicator_enabled":true,"content_disclosure_ai_generated_indicator_enabled":true,"responsive_web_grok_show_grok_translated_post":false,"responsive_web_grok_analysis_button_from_backend":true,"post_ctas_fetch_enabled":false,"freedom_of_speech_not_reach_fetch_enabled":true,"standardized_nudges_misinfo":true,"tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled":true,"longform_notetweets_rich_text_read_enabled":true,"longform_notetweets_inline_media_enabled":false,"responsive_web_grok_image_annotation_enabled":true,"responsive_web_grok_imagine_annotation_enabled":true,"responsive_web_grok_community_note_auto_translation_is_enabled":false,"responsive_web_enhance_cards_enabled":false}',
    fieldToggles = '{"withArticlePlainText":false}',
  ) |>
  req_cookies_set(
    guest_id = "v1:177825193349654388",
    `__cuid` = "660c69ccdfa44dbbbdec8b0f8def2bfb",
    `__cuid` = "660c69ccdfa44dbbbdec8b0f8def2bfb",
    gt = "2082732292510752843",
    g_state = '{"i_l":0,"i_ll":1785398028357,"i_b":"uuJazTIyCsGGnRpAueC4M3AP+qMsCZftFnl0ROWwGPo","i_e":{"enable_itp_optimization":24},"i_et":1785398028357}',
    twid = "u=1632536605",
    auth_token = "af61bd5f3081f1de1176b9f45b891e0bd32e7737",
    lang = "en-gb",
    ct0 = "469c42f2e7ec07eeb835393c6807fd8823568e8d5d23a364b181fd8f11a8656e4ffc625a3af9d65d75347e61c67e28d374c570da3eafccdaeb03c38760b6602cf232a4cf4d7295c6541bf8ba87b911ca",
    d_prefs = "MToxLGNvbnNlbnRfdmVyc2lvbjoyLHRleHRfdmVyc2lvbjoxMDAw",
    guest_id_ads = "v1:177825193349654388",
    guest_id_marketing = "v1:177825193349654388",
    personalization_id = '"v1_sr99TBLB43XWJFbObEbmhQ=="',
    `__eoi` = "ID=901e4be8a5cce0ab:T=1785398148:RT=1785398148:S=AA-Afjbc6IIvw0USuoei_0jNyzMd",
    `__cf_bm` = "tEvicPKziyzqTVniInO0z.1NU3c8am.icSXR5BkYjOM-1785398232.84372-1.0.1.1-8KGcI1axtOgIeLU.q.8aQ.MX1ZxrKJLivbwivHIYB.gQGtLrSXdv2FBQJMViiBYLdgAmE.SiWHkaeAJN2TwUM.uoIFej3QEPjHb9Pwwv4CRJfNssiBvvr82fZ8xWWmOa",
  ) |>
  req_headers(
    accept = "*/*",
    `accept-language` = "en-GB,en-US;q=0.9,en;q=0.8",
    authorization = "Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA",
    `cache-control` = "no-cache",
    `content-type` = "application/json",
    priority = "u=1, i",
    `user-agent` = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
    `x-client-transaction-id` = "wnIueAigYpZD63Ns4zTPCTUi2jZeRCzdtpcpu1xfwJRPgXk9kjN89FUzwr2SvC6vcKli2cRfxIQ7DjM62HySvhlPs3ABwQ",
    `x-csrf-token` = "469c42f2e7ec07eeb835393c6807fd8823568e8d5d23a364b181fd8f11a8656e4ffc625a3af9d65d75347e61c67e28d374c570da3eafccdaeb03c38760b6602cf232a4cf4d7295c6541bf8ba87b911ca",
    `x-twitter-active-user` = "yes",
    `x-twitter-auth-type` = "OAuth2Session",
    `x-twitter-client-language` = "en-GB",
  ) |>
  req_perform()
twitter_resp <- request(
  "https://x.com/i/api/graphql/eoJ5zbv51Z_KVl81v9PmLQ/UserTweets"
) |>
  req_url_query(
    variables = '{"userId":"1525016244","count":20,"includePromotedContent":true,"withQuickPromoteEligibilityTweetFields":true,"withVoice":true}',
    features = '{"rweb_video_screen_enabled":false,"rweb_cashtags_enabled":true,"profile_label_improvements_pcf_label_in_post_enabled":true,"responsive_web_profile_redirect_enabled":true,"rweb_tipjar_consumption_enabled":false,"verified_phone_label_enabled":false,"creator_subscriptions_tweet_preview_api_enabled":true,"responsive_web_graphql_timeline_navigation_enabled":true,"premium_content_api_read_enabled":false,"communities_web_enable_tweet_community_results_fetch":true,"c9s_tweet_anatomy_moderator_badge_enabled":true,"responsive_web_grok_analyze_button_fetch_trends_enabled":false,"responsive_web_grok_analyze_post_followups_enabled":true,"rweb_cashtags_composer_attachment_enabled":true,"responsive_web_jetfuel_frame":true,"responsive_web_grok_share_attachment_enabled":true,"responsive_web_grok_annotations_enabled":true,"articles_preview_enabled":true,"responsive_web_edit_tweet_api_enabled":true,"rweb_conversational_replies_downvote_enabled":false,"graphql_is_translatable_rweb_tweet_is_translatable_enabled":true,"view_counts_everywhere_api_enabled":true,"longform_notetweets_consumption_enabled":true,"responsive_web_twitter_article_tweet_consumption_enabled":true,"content_disclosure_indicator_enabled":true,"content_disclosure_ai_generated_indicator_enabled":true,"responsive_web_grok_show_grok_translated_post":false,"responsive_web_grok_analysis_button_from_backend":true,"post_ctas_fetch_enabled":false,"freedom_of_speech_not_reach_fetch_enabled":true,"standardized_nudges_misinfo":true,"tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled":true,"longform_notetweets_rich_text_read_enabled":true,"longform_notetweets_inline_media_enabled":false,"responsive_web_grok_image_annotation_enabled":true,"responsive_web_grok_imagine_annotation_enabled":true,"responsive_web_grok_community_note_auto_translation_is_enabled":false,"responsive_web_enhance_cards_enabled":false}',
    fieldToggles = '{"withArticlePlainText":false}',
  ) |>
  req_cookies_set(
    guest_id = "v1:177825193349654388",
    `__cuid` = "660c69ccdfa44dbbbdec8b0f8def2bfb",
    `__cuid` = "660c69ccdfa44dbbbdec8b0f8def2bfb",
    gt = "2082732292510752843",
    g_state = '{"i_l":0,"i_ll":1785398028357,"i_b":"uuJazTIyCsGGnRpAueC4M3AP+qMsCZftFnl0ROWwGPo","i_e":{"enable_itp_optimization":24},"i_et":1785398028357}',
    twid = "u=1632536605",
    auth_token = "af61bd5f3081f1de1176b9f45b891e0bd32e7737",
    lang = "en-gb",
    ct0 = "469c42f2e7ec07eeb835393c6807fd8823568e8d5d23a364b181fd8f11a8656e4ffc625a3af9d65d75347e61c67e28d374c570da3eafccdaeb03c38760b6602cf232a4cf4d7295c6541bf8ba87b911ca",
    d_prefs = "MToxLGNvbnNlbnRfdmVyc2lvbjoyLHRleHRfdmVyc2lvbjoxMDAw",
    guest_id_ads = "v1:177825193349654388",
    guest_id_marketing = "v1:177825193349654388",
    personalization_id = '"v1_sr99TBLB43XWJFbObEbmhQ=="',
    `__eoi` = "ID=901e4be8a5cce0ab:T=1785398148:RT=1785398148:S=AA-Afjbc6IIvw0USuoei_0jNyzMd",
    `__cf_bm` = "tEvicPKziyzqTVniInO0z.1NU3c8am.icSXR5BkYjOM-1785398232.84372-1.0.1.1-8KGcI1axtOgIeLU.q.8aQ.MX1ZxrKJLivbwivHIYB.gQGtLrSXdv2FBQJMViiBYLdgAmE.SiWHkaeAJN2TwUM.uoIFej3QEPjHb9Pwwv4CRJfNssiBvvr82fZ8xWWmOa",
  ) |>
  req_headers(
    accept = "*/*",
    `accept-language` = "en-GB,en-US;q=0.9,en;q=0.8",
    authorization = "Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA",
    `cache-control` = "no-cache",
    `content-type` = "application/json",
    priority = "u=1, i",
    `user-agent` = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
    `x-client-transaction-id` = "wnIueAigYpZD63Ns4zTPCTUi2jZeRCzdtpcpu1xfwJRPgXk9kjN89FUzwr2SvC6vcKli2cRfxIQ7DjM62HySvhlPs3ABwQ",
    `x-csrf-token` = "469c42f2e7ec07eeb835393c6807fd8823568e8d5d23a364b181fd8f11a8656e4ffc625a3af9d65d75347e61c67e28d374c570da3eafccdaeb03c38760b6602cf232a4cf4d7295c6541bf8ba87b911ca",
    `x-twitter-active-user` = "yes",
    `x-twitter-auth-type` = "OAuth2Session",
    `x-twitter-client-language` = "en-GB",
  ) |>
  req_perform()

Parsing the Twitter data

This is the code we developed in session 2. We can use it again to get a clean table with some interesting information

ess_tweets <- twitter_resp |>
  resp_body_json()

entries <- pluck(
  ess_tweets,
  "data",
  "user",
  "result",
  "timeline",
  "timeline",
  "instructions",
  3L,
  "entries"
)

tweets <- map(entries, function(x) {
  pluck(x, "content", "itemContent", "tweet_results", "result", "legacy")
})

tweets_df <- map(tweets, function(t) {
  tibble(
    id = t$id_str,
    user_id = t$user_id_str,
    created_at = lubridate::parse_date_time(t$created_at, "a b d H M S z Y"),
    full_text = t$full_text,
    favorite_count = t$favorite_count,
    retweet_count = t$retweet_count,
    bookmark_count = t$bookmark_count
  )
}) |>
  bind_rows()
tweets_df

Next page?

  • We only got the first 12 tweets
  • But when scrolling down and meanwhile filter the network tab for “UserTweets”, we see more requests are made
  • Checking the difference, we see a URL query parameter "cursor":"DAAHCgABHOdeg83___ALAAIAAAATMjA3MTYwNDkxODIzNDgxNjcyNggAAwAAAAIAAA"
  • This is similar to the “next” value we saw the UK parliament API uses

Next pages!

get_x_user_json <- function(userId, cursor = NULL) {
  request("https://x.com/i/api/graphql/eoJ5zbv51Z_KVl81v9PmLQ/UserTweets") |>
    req_url_query(
      variables = paste0(
        '{"userId":"',
        userId,
        '","count":20,"cursor":"',
        cursor,
        '","includePromotedContent":true,"withQuickPromoteEligibilityTweetFields":true,"withVoice":true}'
      )
    ) |>
    req_cookies_set(
      guest_id = "v1:177825193349654388",
      `__cuid` = "660c69ccdfa44dbbbdec8b0f8def2bfb",
      `__cuid` = "660c69ccdfa44dbbbdec8b0f8def2bfb",
      gt = "2082732292510752843",
      g_state = '{"i_l":0,"i_ll":1785398028357,"i_b":"uuJazTIyCsGGnRpAueC4M3AP+qMsCZftFnl0ROWwGPo","i_e":{"enable_itp_optimization":24},"i_et":1785398028357}',
      twid = "u=1632536605",
      auth_token = "af61bd5f3081f1de1176b9f45b891e0bd32e7737",
      lang = "en-gb",
      ct0 = "469c42f2e7ec07eeb835393c6807fd8823568e8d5d23a364b181fd8f11a8656e4ffc625a3af9d65d75347e61c67e28d374c570da3eafccdaeb03c38760b6602cf232a4cf4d7295c6541bf8ba87b911ca",
      d_prefs = "MToxLGNvbnNlbnRfdmVyc2lvbjoyLHRleHRfdmVyc2lvbjoxMDAw",
      guest_id_ads = "v1:177825193349654388",
      guest_id_marketing = "v1:177825193349654388",
      personalization_id = '"v1_sr99TBLB43XWJFbObEbmhQ=="',
      `__eoi` = "ID=901e4be8a5cce0ab:T=1785398148:RT=1785398148:S=AA-Afjbc6IIvw0USuoei_0jNyzMd",
      `__cf_bm` = "tEvicPKziyzqTVniInO0z.1NU3c8am.icSXR5BkYjOM-1785398232.84372-1.0.1.1-8KGcI1axtOgIeLU.q.8aQ.MX1ZxrKJLivbwivHIYB.gQGtLrSXdv2FBQJMViiBYLdgAmE.SiWHkaeAJN2TwUM.uoIFej3QEPjHb9Pwwv4CRJfNssiBvvr82fZ8xWWmOa",
    ) |>
    req_headers(
      accept = "*/*",
      `accept-language` = "en-GB,en-US;q=0.9,en;q=0.8",
      authorization = "Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA",
      `cache-control` = "no-cache",
      `content-type` = "application/json",
      priority = "u=1, i",
      `user-agent` = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
      `x-client-transaction-id` = "wnIueAigYpZD63Ns4zTPCTUi2jZeRCzdtpcpu1xfwJRPgXk9kjN89FUzwr2SvC6vcKli2cRfxIQ7DjM62HySvhlPs3ABwQ",
      `x-csrf-token` = "469c42f2e7ec07eeb835393c6807fd8823568e8d5d23a364b181fd8f11a8656e4ffc625a3af9d65d75347e61c67e28d374c570da3eafccdaeb03c38760b6602cf232a4cf4d7295c6541bf8ba87b911ca",
      `x-twitter-active-user` = "yes",
      `x-twitter-auth-type` = "OAuth2Session",
      `x-twitter-client-language` = "en-GB",
    ) |>
    req_perform(path = "test.json") |>
    resp_body_json()
}

Find the cursor

parse_path <- function(ix) {
  out <- as.list(ix$p)
  out[which(ix$p == as.character(ix$pos))] <- ix$pos[
    ix$p == as.character(ix$pos)
  ]
  gsub("list(", "purrr::pluck(DATA, ", deparse1(out), fixed = TRUE)
}

#' Search a list
#'
#' @param l a list
#' @param f a function to identify the element you are searching
#'
#' @return an object containing the searched element with the function to extract it as a name
#' @export
list_search <- function(l, f) {
  paths <- rrapply::rrapply(
    object = l,
    condition = f,
    f = function(x, .xparents, .xname, .xpos) {
      list(p = .xparents, n = .xname, pos = .xpos)
    },
    how = "flatten"
  )

  out <- purrr::map(paths, function(p) purrr::pluck(l, !!!p$pos))
  names(out) <- purrr::map_chr(paths, parse_path)
  return(out)
}
initial_resp <- get_x_user_json(userId = "1525016244")
list_search(initial_resp, function(x) str_detect(x, "___"))
$`purrr::pluck(DATA, "data", "user", "result", "timeline", "timeline", "instructions", 3L, "entries", 17L, "content", "value")`
[1] "DAAHCgABHOd21qR___ALAAIAAAATMjA3MTYwNDkxODIzNDgxNjcyNggAAwAAAAIAAA"
purrr::pluck(
  initial_resp,
  "data",
  "user",
  "result",
  "timeline",
  "timeline",
  "instructions",
  3L,
  "entries",
  17L,
  "content"
)
$`__typename`
[1] "TimelineTimelineCursor"

$cursorType
[1] "Bottom"

$entryType
[1] "TimelineTimelineCursor"

$value
[1] "DAAHCgABHOd21qR___ALAAIAAAATMjA3MTYwNDkxODIzNDgxNjcyNggAAwAAAAIAAA"

Add cursor to next request

next_resp <- get_x_user_json(
  userId = "1525016244",
  cursor = pluck(
    initial_resp,
    "data",
    "user",
    "result",
    "timeline",
    "timeline",
    "instructions",
    3L,
    "entries",
    17L,
    "content",
    "value"
  )
)
parse_x_user <- function(resp_data) {
  entries <- pluck(
    resp_data,
    "data",
    "user",
    "result",
    "timeline",
    "timeline",
    "instructions",
    3L,
    "entries"
  )

  if (is.null(entries)) {
    entries <- pluck(
      resp_data,
      "data",
      "user",
      "result",
      "timeline",
      "timeline",
      "instructions",
      2L,
      "entries"
    )
  }

  tweets <- map(entries, function(x) {
    pluck(x, "content", "itemContent", "tweet_results", "result", "legacy")
  })

  map(tweets, function(t) {
    tibble(
      id = t$id_str,
      user_id = t$user_id_str,
      created_at = lubridate::parse_date_time(t$created_at, "a b d H M S z Y"),
      full_text = t$full_text,
      favorite_count = t$favorite_count,
      retweet_count = t$retweet_count,
      bookmark_count = t$bookmark_count
    )
  }) |>
    bind_rows()
}
bind_rows(
  parse_x_user(initial_resp),
  parse_x_user(next_resp)
)
# A tibble: 16 × 7
   id         user_id created_at          full_text favorite_count retweet_count
   <chr>      <chr>   <dttm>              <chr>              <int>         <int>
 1 208166848… 152501… 2026-07-27 09:10:17 "This We…              2             1
 2 208065257… 152501… 2026-07-24 13:53:26 "https:/…              1             1
 3 208031926… 152501… 2026-07-23 15:48:57 "🎓 A big…              6             4
 4 208022516… 152501… 2026-07-23 09:35:03 "📌 Frida…              3             1
 5 207984948… 152501… 2026-07-22 08:42:12 "📌 TODAY…              1             1
 6 207737300… 152501… 2026-07-15 12:41:34 "📢 Today…              2             1
 7 207664436… 152501… 2026-07-13 12:26:13 "⏳ Less …              5             1
 8 207451181… 152501… 2026-07-07 15:12:14 "We are …              4             1
 9 207412995… 152501… 2026-07-06 13:54:52 "From su…              0             0
10 207308329… 152501… 2026-07-03 16:35:49 "Last da…              4             1
11 207306638… 152501… 2026-07-03 15:28:37 "☀️The Es…              3             2
12 207160491… 152501… 2026-06-29 14:41:15 "We are …              2             0
13 206977350… 152501… 2026-06-24 13:23:52 "Want to…              0             0
14 206906996… 152501… 2026-06-22 14:48:15 "Need be…              2             1
15 206641555… 152501… 2026-06-15 07:00:35 "☕️Women’…              1             0
16 206475044… 152501… 2026-06-10 16:44:03 "⏳ Final…              0             2
# ℹ 1 more variable: bookmark_count <int>

Now get the next page. Right?

list_search(next_resp, function(x) str_detect(x, "___"))
named list()

This time we got no cursor :(

Looks promising, but…

I stopped at this point since there are three issue that are unclear to resolve:

  1. Can we reliably get the cursor?
  2. How do we find the user ID?
  3. We have to send several identifiers: It is not clear how stable x-csrf-token, authorization, and the cookies are

Summary: hidden APIs

What are they

  • used by services of a company to communicate with each other
  • code on a website often uses one to download additional conent
  • the browser logs them and provides them to us as cURL calls

What are they good for?

  • We can often use them to get content that is otherwise unavailable
  • We can study them to find out what requests the website server accepts
  • Some websites allow access just using a special header or cookies
  • If they are somewhat flexible we can wrap them in a function or package
  • This can allow us to gather data on scale

Issues

  • Companies have mechanisms to counter scraping:
    • signing specific requests (TikTok)
    • obscuring pagination (Twitter)
    • rate limiting requests per second/minute/day and user/IP(Twitter)
    • expiring session tokens (telegraaf.nl)

Wrap Up

Save some information about the session for reproducibility.

sessionInfo()
R version 4.6.1 (2026-06-24)
Platform: x86_64-pc-linux-gnu
Running under: EndeavourOS

Matrix products: default
BLAS:   /usr/lib/libblas.so.3.12.0 
LAPACK: /usr/lib/liblapack.so.3.12.0  LAPACK version 3.12.0

locale:
 [1] LC_CTYPE=en_GB.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_GB.UTF-8        LC_COLLATE=en_GB.UTF-8    
 [5] LC_MONETARY=en_GB.UTF-8    LC_MESSAGES=en_GB.UTF-8   
 [7] LC_PAPER=en_GB.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C       

time zone: Europe/Berlin
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] rvest_1.0.4      httr2_1.2.3      lubridate_1.9.5  forcats_1.0.1   
 [5] stringr_1.6.0    dplyr_1.2.1      purrr_1.2.2      readr_2.2.0     
 [9] tidyr_1.3.2      tibble_3.3.1     ggplot2_4.0.3    tidyverse_2.0.0 
[13] tinytable_0.17.0

loaded via a namespace (and not attached):
 [1] gtable_0.3.6       xfun_0.59          websocket_1.4.4    processx_3.9.0    
 [5] tzdb_0.5.0         vctrs_0.7.3        tools_4.6.1        ps_1.9.3          
 [9] generics_0.1.4     curl_7.1.0         pkgconfig_2.0.3    pdftools_3.9.0    
[13] RColorBrewer_1.1-3 S7_0.2.2           tesseract_5.2.5    lifecycle_1.0.5   
[17] compiler_4.6.1     farver_2.1.2       chromote_0.5.1     janitor_2.2.1     
[21] rrapply_1.2.8      snakecase_0.11.1   litedown_0.9       htmltools_0.5.9   
[25] yaml_2.3.12        later_1.4.8        pillar_1.11.1      crayon_1.5.3      
[29] magick_2.9.1       tidyselect_1.2.1   digest_0.6.39      stringi_1.8.7     
[33] fastmap_1.2.0      grid_4.6.1         cli_3.6.6          magrittr_2.0.5    
[37] utf8_1.2.6         withr_3.0.3        scales_1.4.0       promises_1.5.0    
[41] rappdirs_0.3.4     timechange_0.4.0   rmarkdown_2.31     httr_1.4.8        
[45] lobstr_1.2.1       otel_0.2.0         qpdf_1.4.1         askpass_1.2.1     
[49] hms_1.1.4          evaluate_1.0.5     knitr_1.51         rlang_1.2.0       
[53] Rcpp_1.1.2         docopt_0.7.2       glue_1.8.1         selectr_0.4-2     
[57] xml2_1.6.0         jsonlite_2.0.0     R6_2.6.1