| Day | Session |
|---|---|
| 1 | Introduction |
| 2 | Data Structures and Wrangling |
| 3 | Working with Files |
| 4 | Linking and joining data & SQL |
| 5 | Database Software and Scaling your Analyses |
| 6 | Introduction to the Web |
| 7 | Static Web Pages |
| 8 | Application Programming Interface (APIs) |
| 9 | Interactive Web Pages |
| 10 | Building a Reproducible Research Project |
Session 8: Application Programming Interface (APIs)
Introduction to Web Scraping and Data Management for Social Scientists
Johannes B. Gruber
2026-07-29
Introduction
This Course
The Plan for Today
In this session, we learn how to adopt data from someone else. We will:
- Learn what an API is and what parts it consists of
- Learn about
httr2, a modern intuitive package to communicate with APIs - Discuss some examples:
- A simple first API: The Guardian API
- UK Parliament API
- Semantic Scholar API
- Go into a bit more detail on requesting raw data
What are APIs?
What is An Application Programming Interface (API)?
- An Application Programming Interface (API) is a way for two computer programs to speak to each other
- In modern software development they are used extensively when:
- two programs are not on the same machine
- two applications are not in the same language
- when the inner workings of a software should be obscured, but its functionality is offered for customization
- when a graphic user interface would be inconvenient at scale
- Several important types (SOAP, GraphQL, etc.), but we will focus on REST (Representational state transfer) APIs
- Commonly used to distribute data or do many other things
- A few prominent examples:
- the Twitter and Facebook APIs (both effectively defunct)
- the ChatGPT API, which is used to build many additional services
- news APIs like The Guardian and NYT
- financial APIs
- translation APIs (Google, Bing and DeepL)
Parts of an API call
API calls usually combine several elements:
- a base URL of the service (e.g.,
https://api.openai.com/) - an endpoint for a specific service, usually accessed through a sub-directory (e.g.,
/v1/completions) - an API method:
GET,POST,PUT,DELETE,PATCHQUERY1 (onlyGETand sometimesPOSTare important for us ) - headers containing some settings, e.g., what format you want to receive the data in (JSON, XML, HTML etc.), and communicating who you are through user-agent, cookies, device and software information that is usually used for debugging
- query parameters, i.e., your search term, filters, what fields/columns you want to access, how many results you want to receive, how results are ordered etc (
?q=parliament%20AND%20debate) - a body if your request contains some more complicated instructions (not for
GETrequests) - authentication, usually in form of a token (a standardized string, similar to a password)
Parts of an API response
APIs respond to a call. The response usually also contains several elements:
- a status code: 200s mean success, 300s mean success with some caveat, 400+ are request errors (not found, forbidden), 500 is a server error
- headers provide additional information about the response (e.g., type of data returned, size of the data, time stamp)
- body: the main response containing the requested data (e.g., as JSON or HTML)
- response metadata: more information about the response (e.g., pagination information, version numbers, remaining rate limit allowance, link to next page)
- error messages: when unsuccessful, the API might include an error message on top of the status code
Accessing APIs from R
The httr2 package
- rewrite of the
httrwhich was the de-factor default to develop API packages inR - developed by Hadley Wickham
- tidyverse programming principles
- telling verbs are used in a pipe
- requests are build up using
req_*functions - responses are deconstructed using
resp_* - makes wrapping an API in a few functions or a package straightforward
Example: The Guardian API
Background
- The newspaper
The Guardianoffers all its articles through an open API for free 🤓 - To access the API, you first need to obtain an API key by filling out a small form here
- The API key should arrive within seconds per mail
- This is unfortunately very rare in the world of news media ☹️
- To figure out how to use the API, we can use its documentation
Your task: get a key and use usethis::edit_r_environ(scope = "project") to open your .Renviron file. Save the API key as the variable GUARDIAN_KEY.
Building Requests
Let’s build our first httr2 request!
library(httr2)
library(tidyverse, warn.conflicts = FALSE)
req <- request("https://content.guardianapis.com") |> # start the request with the base URL
req_url_path("search") |> # navigate to the endpoint you want to access
req_method("GET") |> # specify the method
req_timeout(seconds = 60) |> # how long to wait for a response
req_headers("User-Agent" = "httr2 guardian test") |> # specify request headers
# req_body_json() |> # since this is a GET request the body stays empty
req_url_query(
# instead the query is added to the URL
q = "parliament AND debate",
"show-blocks" = "all"
) |>
req_url_query(
# in this case, the API key is also added to the query
"api-key" = Sys.getenv("GUARDIAN_KEY") # but httr2 also has req_auth_* functions for other
) # authentication procedures
print(req)
We now built the request. But this doesn’t yet do anything until you also perform it.
Performing the request
Printing the request tells us several important things:
- the status of the response is OK (hurray!)
- the response carries data in the JSON format
- however, you probably don’t want to manually inspect each response…
Parsing the response: a first look
We can automatically check if the response has the form we expect:
If we’re happy with the status of the response, we can start to look at the body by transforming it with the correct resp_body_* function:
<list>
└─response: <list>
├─status: "ok"
├─userTier: "developer"
├─total: 33393
├─startIndex: 1
├─pageSize: 10
├─currentPage: 1
├─pages: 3340
├─orderBy: "relevance"
└─results: <list>
├─<list>
│ ├─id: "politics/2026/apr/16/officials-d..."
│ ├─type: "article"
│ ├─sectionId: "politics"
│ ├─sectionName: "Politics"
│ ├─webPublicationDate: "2026-04-16T17:39:59Z"
│ ├─webTitle: "Officials debate withholding Man..."
│ ├─webUrl: "https://www.theguardian.com/poli..."
│ ├─apiUrl: "https://content.guardianapis.com..."
│ ├─blocks: <list>
│ │ ├─main: <list>
│ │ │ ├─id: "69e0d9738f08b9e0aac013e2"
│ │ │ ├─bodyHtml: "<figure class="element element-i..."
│ │ │ ├─bodyTextSummary: ""
... Parsing the response: a first look
We already see some useful information about the the result. We could extract that information either with pluck from the tidyverse or using square brackets:
[1] 33393[1] 10[1] 3340Parsing the response: extracting the data
So far we only got the results for page 1, which is a common way to return results from an API. To get to the other pages that contain results, we would need to loop through all of these pages (by adding the query page = i). For now, we can have a closer look at the articles on the first results page.
We can have a closer look at this using the Viewer in RStudio:
In typical fashion, this API returns the data in a rather complicated format. This is probably the main reason why people dislike working with APIs in R, as it can be very frustrating to get this into a format that makes sense for us.
Parsing the response: building a data wrangling function
Let’s build a function to select just some important information. We start by writing a few lines of code to parse the first article:
$id
[1] "politics/2026/apr/16/officials-debate-withholding-mandelson-vetting-documents-from-parliament"
$type
[1] "article"
$sectionId
[1] "politics"
$sectionName
[1] "Politics"
$webPublicationDate
[1] "2026-04-16T17:39:59Z"
$webTitle
[1] "Officials debate withholding Mandelson vetting documents from parliament"
$webUrl
[1] "https://www.theguardian.com/politics/2026/apr/16/officials-debate-withholding-mandelson-vetting-documents-from-parliament"
$apiUrl
[1] "https://content.guardianapis.com/politics/2026/apr/16/officials-debate-withholding-mandelson-vetting-documents-from-parliament"
$blocks
$blocks$main
$blocks$main$id
[1] "69e0d9738f08b9e0aac013e2"
$blocks$main$bodyHtml
[1] "<figure class=\"element element-image element--showcase\" data-media-id=\"ec69467fc26d2e96e444f8fe08e98215c3e05b61\"> <img src=\"https://media.guim.co.uk/ec69467fc26d2e96e444f8fe08e98215c3e05b61/0_0_5000_3000/1000.jpg\" alt=\"Peter Mandelson\" width=\"1000\" height=\"600\" class=\"gu-image\" /> <figcaption> <span class=\"element-image__caption\">Mandelson was sacked in September 2025.</span> <span class=\"element-image__credit\">Illustration: Guardian Design/Getty</span> </figcaption> </figure>"
$blocks$main$bodyTextSummary
[1] ""
$blocks$main$attributes
named list()
$blocks$main$published
[1] TRUE
$blocks$main$createdDate
[1] "2026-04-16T14:06:40Z"
$blocks$main$firstPublishedDate
[1] "2026-04-16T14:11:30Z"
$blocks$main$publishedDate
[1] "2026-04-16T14:11:30Z"
$blocks$main$lastModifiedDate
[1] "2026-04-16T14:11:30Z"
$blocks$main$contributors
list()
$blocks$main$elements
$blocks$main$elements[[1]]
$blocks$main$elements[[1]]$type
[1] "image"
$blocks$main$elements[[1]]$assets
$blocks$main$elements[[1]]$assets[[1]]
$blocks$main$elements[[1]]$assets[[1]]$type
[1] "image"
$blocks$main$elements[[1]]$assets[[1]]$mimeType
[1] "image/jpeg"
$blocks$main$elements[[1]]$assets[[1]]$file
[1] "https://media.guim.co.uk/ec69467fc26d2e96e444f8fe08e98215c3e05b61/0_0_5000_3000/1000.jpg"
$blocks$main$elements[[1]]$assets[[1]]$typeData
$blocks$main$elements[[1]]$assets[[1]]$typeData$width
[1] 1000
$blocks$main$elements[[1]]$assets[[1]]$typeData$height
[1] 600
$blocks$main$elements[[1]]$assets[[2]]
$blocks$main$elements[[1]]$assets[[2]]$type
[1] "image"
$blocks$main$elements[[1]]$assets[[2]]$mimeType
[1] "image/jpeg"
$blocks$main$elements[[1]]$assets[[2]]$file
[1] "https://media.guim.co.uk/ec69467fc26d2e96e444f8fe08e98215c3e05b61/0_0_5000_3000/500.jpg"
$blocks$main$elements[[1]]$assets[[2]]$typeData
$blocks$main$elements[[1]]$assets[[2]]$typeData$width
[1] 500
$blocks$main$elements[[1]]$assets[[2]]$typeData$height
[1] 300
$blocks$main$elements[[1]]$assets[[3]]
$blocks$main$elements[[1]]$assets[[3]]$type
[1] "image"
$blocks$main$elements[[1]]$assets[[3]]$mimeType
[1] "image/jpeg"
$blocks$main$elements[[1]]$assets[[3]]$file
[1] "https://media.guim.co.uk/ec69467fc26d2e96e444f8fe08e98215c3e05b61/0_0_5000_3000/140.jpg"
$blocks$main$elements[[1]]$assets[[3]]$typeData
$blocks$main$elements[[1]]$assets[[3]]$typeData$width
[1] 140
$blocks$main$elements[[1]]$assets[[3]]$typeData$height
[1] 84
$blocks$main$elements[[1]]$imageTypeData
$blocks$main$elements[[1]]$imageTypeData$caption
[1] "Mandelson was sacked in September 2025."
$blocks$main$elements[[1]]$imageTypeData$displayCredit
[1] TRUE
$blocks$main$elements[[1]]$imageTypeData$credit
[1] "Illustration: Guardian Design/Getty"
$blocks$main$elements[[1]]$imageTypeData$source
[1] "Getty"
$blocks$main$elements[[1]]$imageTypeData$photographer
[1] "Guardian Design"
$blocks$main$elements[[1]]$imageTypeData$alt
[1] "Peter Mandelson"
$blocks$main$elements[[1]]$imageTypeData$mediaId
[1] "ec69467fc26d2e96e444f8fe08e98215c3e05b61"
$blocks$main$elements[[1]]$imageTypeData$imageType
[1] "Illustration"
$blocks$main$elements[[1]]$imageTypeData$role
[1] "showcase"
$blocks$body
$blocks$body[[1]]
$blocks$body[[1]]$id
[1] "69e0b7b18f08fbc58317e1db"
$blocks$body[[1]]$bodyHtml
[1] "<p>Senior government officials have been considering whether to withhold from parliament sensitive documents that show Peter Mandelson <a href=\"https://www.theguardian.com/politics/2026/apr/16/revealed-mandelson-failed-vetting-but-foreign-office-overruled-decision\">failed security vetting</a> before he assumed the role of US ambassador, the Guardian can reveal.</p> <p>Any such decision could amount to an extraordinary breach of a parliamentary vote, known as a humble address, that ordered the release of “all papers” relevant to Mandelson’s appointment.</p> <p>The Guardian has revealed that Mandelson did not receive vetting clearance from security officials, but that their decision was overruled by the Foreign Office to ensure he could take up his post.</p> <p>According to multiple sources, officials across government have been in dispute over whether to release documents that would reveal those facts, and other information about Mandelson’s security vetting, to the parliamentary intelligence and security committee (ISC).</p> <aside class=\"element element-rich-link element--thumbnail\"> <p> <span>Related: </span><a href=\"https://www.theguardian.com/politics/2026/apr/16/five-key-questions-decision-peter-mandelson-security-clearance\">Five key questions: who overruled decision to deny Mandelson security clearance?</a> </p> </aside> <p>The committee has been entrusted by parliament with the role of assessing the most sensitive papers relating to Mandelson’s appointment. Ministers have assured parliament that there will be no block on what the committee gets to see.</p> <figure class=\"element element-atom\"> <gu-atom data-atom-id=\"f4bcbd72-f585-4822-aa3f-789b3f582803\" data-atom-type=\"media\" > </gu-atom> </figure> <p>However, as of Thursday morning, a decision had not been made about whether the committee should have access to documents about Mandelson’s vetting by UK Security Vetting (UKSV) and the decision by the Foreign, Commonwealth and Development Office (FCDO) to override it.</p> <p>The ultimate decision will rest with the Cabinet Office. But sources said officials were split over how to proceed, with some arguing the materials should be withheld, despite a parliamentary motion to release all relevant materials. One source said: “There is no consensus.”</p> <p>Late on Thursday, after publication of the Guardian’s revelations, Downing Street indicated the documents would be released to the committee.</p> <p>“The government is committed to complying with the humble address in full as soon as possible,” a government spokesperson said. “Any documentation within the scope of the humble address that requires redaction on the basis of national security or international relations will be provided to the ISC. This will include documents provided to the FCDO by UK Security Vetting.”</p> <p>Lord Beamish, chair of the ISC, noted the government’s role in providing documents to the committee. But he added: “The committee would take a very dim view if documents relating to the humble address are withheld from the committee, one which I think would be shared by parliament.”</p> <p>Mandelson <a href=\"https://www.theguardian.com/politics/2025/sep/11/keir-starmer-sacks-peter-mandelson-over-jeffrey-epstein-revelations\">was sacked in September 2025</a> after the extent of his friendship with the convicted sex offender Jeffrey Epstein was revealed, in documents released to the public by the US department of justice.</p> <p>A second tranche of DoJ documents released in late January 2026 suggested Mandelson had passed market-sensitive information and Downing Street emails to Epstein while serving in Gordon Brown’s government, prompting fresh parliamentary scrutiny of Keir Starmer’s decision to appoint him.</p> <p>In February 2026, parliament voted for a rare motion brought by the Tories to compel the government to publicly release “all papers relating to Mandelson’s appointment”. The language in the motion, called <a href=\"https://www.theguardian.com/politics/2026/feb/04/peter-mandelson-disclosures-what-is-humble-address-parliament\">a humble address</a>, made an exception for papers “prejudicial to UK national security or international relations”, which it said should be provided to the ISC.</p> <p>The committee comprises nine MPs and peers, including Jeremy Wright, a former attorney general, and Alan West, a retired Royal Navy admiral. Its members are sworn to secrecy under the Official Secrets Act and are given access to highly classified material. Tasked with overseeing the UK intelligence community on parliament’s behalf, they are now likely to demand answers over the precise basis for the UKSV decision.</p> <p>During the debate on the humble address, Derek Twigg, a member of the ISC, asked a minister, Chris Ward, for assurances there would be “no block whatsoever” on the material provided to the committee. Ward said: “Yes, I can confirm that.”</p> <p>According to an <a href=\"https://isc.independent.gov.uk/wp-content/uploads/2026/02/20260226-ISC-Press-Notice.docx-1.pdf\">ISC press notice</a>, the committee met top officials from the Cabinet Office after the humble address and it was agreed that to “fulfil the will of parliament”, it would be for the committee alone to assess material provided to it by the government and decide if it should be published.</p> <p>However, sources have said officials have been considering a different approach, and wrestling with whether there could be grounds to withhold highly sensitive UKSV vetting documents from the committee entirely, despite the vote.</p> <p>One source said officials were wary that disclosure of details of an individual’s vetting, even if access was restricted to a trusted group of parliamentarians, would be “unprecedented”. They pointed to strong concerns, including among the security services, about revealing documents.</p> <p>A second source said officials were grappling with a “live tactical question” over the release, and searching for loopholes that might enable them to withhold documents from the ISC. They said top officials were “looking for ways to comply with the letter of the humble address while potentially bypassing its spirit”.</p> <p>The second source said some in government were arguing that any disclosure of documents revealing Mandelon’s UKSV vetting had failed, only to be overruled by the Foreign Office, would “reflect poorly on the government’s overall integrity”. Others, they said, had been arguing that “precedent should be set aside” to comply with parliament’s wishes.</p> <p>Another option under consideration would be to show the redacted documents to a select number of the nine-person committee.</p> <p>The Guardian’s revelation that Mandelson had failed the developed vetting process is likely to put pressure on the government to release the documents to the public in full.</p> <div class=\"element element-callout\" data-campaign-id=\"10d669d8-b414-449c-8a7c-6319cb7b8eb8\" > </div> <p>Details of an individual’s UKSV developed vetting results have never before been publicly disclosed, and some top officials have been arguing that doing so would jeopardise national security.</p> <p>According to <a href=\"https://www.gov.uk/government/publications/united-kingdom-security-vetting-clearance-levels/dv-guidance-pack-for-applicants\">publicly available government documents</a>, the vetting process includes a questionnaire and interviews requiring disclosure of highly private information, including about personal finances, business connections and sexual history. The security services are also consulted.</p> <p>Regardless of the final decision by the Cabinet Office, the notion that officials have even contemplated whether to withhold the Mandelson vetting documents, in what would appear to be a breach of parliament’s wishes, could prove hugely controversial.</p> <p>Such a move could raise a constitutional conflict, with officialdom and the security services challenging the supremacy of parliament. It would also raise questions about whether the government is adhering to its word.</p> <p>In a letter on 6 February 2026 to the chair of the committee, Starmer said: “Above all else, the government wishes to engage constructively with the ISC, and to ensure that parliament’s instruction is met with the urgency and transparency it deserves.”</p> <p>The Foreign Office, Downing Street and the Cabinet Office have been approached for comment.</p>"
$blocks$body[[1]]$bodyTextSummary
[1] "Senior government officials have been considering whether to withhold from parliament sensitive documents that show Peter Mandelson failed security vetting before he assumed the role of US ambassador, the Guardian can reveal. Any such decision could amount to an extraordinary breach of a parliamentary vote, known as a humble address, that ordered the release of “all papers” relevant to Mandelson’s appointment. The Guardian has revealed that Mandelson did not receive vetting clearance from security officials, but that their decision was overruled by the Foreign Office to ensure he could take up his post. According to multiple sources, officials across government have been in dispute over whether to release documents that would reveal those facts, and other information about Mandelson’s security vetting, to the parliamentary intelligence and security committee (ISC).\nThe committee has been entrusted by parliament with the role of assessing the most sensitive papers relating to Mandelson’s appointment. Ministers have assured parliament that there will be no block on what the committee gets to see.\nHowever, as of Thursday morning, a decision had not been made about whether the committee should have access to documents about Mandelson’s vetting by UK Security Vetting (UKSV) and the decision by the Foreign, Commonwealth and Development Office (FCDO) to override it. The ultimate decision will rest with the Cabinet Office. But sources said officials were split over how to proceed, with some arguing the materials should be withheld, despite a parliamentary motion to release all relevant materials. One source said: “There is no consensus.” Late on Thursday, after publication of the Guardian’s revelations, Downing Street indicated the documents would be released to the committee. “The government is committed to complying with the humble address in full as soon as possible,” a government spokesperson said. “Any documentation within the scope of the humble address that requires redaction on the basis of national security or international relations will be provided to the ISC. This will include documents provided to the FCDO by UK Security Vetting.” Lord Beamish, chair of the ISC, noted the government’s role in providing documents to the committee. But he added: “The committee would take a very dim view if documents relating to the humble address are withheld from the committee, one which I think would be shared by parliament.” Mandelson was sacked in September 2025 after the extent of his friendship with the convicted sex offender Jeffrey Epstein was revealed, in documents released to the public by the US department of justice. A second tranche of DoJ documents released in late January 2026 suggested Mandelson had passed market-sensitive information and Downing Street emails to Epstein while serving in Gordon Brown’s government, prompting fresh parliamentary scrutiny of Keir Starmer’s decision to appoint him.\nIn February 2026, parliament voted for a rare motion brought by the Tories to compel the government to publicly release “all papers relating to Mandelson’s appointment”. The language in the motion, called a humble address, made an exception for papers “prejudicial to UK national security or international relations”, which it said should be provided to the ISC. The committee comprises nine MPs and peers, including Jeremy Wright, a former attorney general, and Alan West, a retired Royal Navy admiral. Its members are sworn to secrecy under the Official Secrets Act and are given access to highly classified material. Tasked with overseeing the UK intelligence community on parliament’s behalf, they are now likely to demand answers over the precise basis for the UKSV decision. During the debate on the humble address, Derek Twigg, a member of the ISC, asked a minister, Chris Ward, for assurances there would be “no block whatsoever” on the material provided to the committee. Ward said: “Yes, I can confirm that.” According to an ISC press notice, the committee met top officials from the Cabinet Office after the humble address and it was agreed that to “fulfil the will of parliament”, it would be for the committee alone to assess material provided to it by the government and decide if it should be published. However, sources have said officials have been considering a different approach, and wrestling with whether there could be grounds to withhold highly sensitive UKSV vetting documents from the committee entirely, despite the vote. One source said officials were wary that disclosure of details of an individual’s vetting, even if access was restricted to a trusted group of parliamentarians, would be “unprecedented”. They pointed to strong concerns, including among the security services, about revealing documents. A second source said officials were grappling with a “live tactical question” over the release, and searching for loopholes that might enable them to withhold documents from the ISC. They said top officials were “looking for ways to comply with the letter of the humble address while potentially bypassing its spirit”. The second source said some in government were arguing that any disclosure of documents revealing Mandelon’s UKSV vetting had failed, only to be overruled by the Foreign Office, would “reflect poorly on the government’s overall integrity”. Others, they said, had been arguing that “precedent should be set aside” to comply with parliament’s wishes. Another option under consideration would be to show the redacted documents to a select number of the nine-person committee. The Guardian’s revelation that Mandelson had failed the developed vetting process is likely to put pressure on the government to release the documents to the public in full.\nDetails of an individual’s UKSV developed vetting results have never before been publicly disclosed, and some top officials have been arguing that doing so would jeopardise national security. According to publicly available government documents, the vetting process includes a questionnaire and interviews requiring disclosure of highly private information, including about personal finances, business connections and sexual history. The security services are also consulted. Regardless of the final decision by the Cabinet Office, the notion that officials have even contemplated whether to withhold the Mandelson vetting documents, in what would appear to be a breach of parliament’s wishes, could prove hugely controversial. Such a move could raise a constitutional conflict, with officialdom and the security services challenging the supremacy of parliament. It would also raise questions about whether the government is adhering to its word. In a letter on 6 February 2026 to the chair of the committee, Starmer said: “Above all else, the government wishes to engage constructively with the ISC, and to ensure that parliament’s instruction is met with the urgency and transparency it deserves.” The Foreign Office, Downing Street and the Cabinet Office have been approached for comment."
$blocks$body[[1]]$attributes
named list()
$blocks$body[[1]]$published
[1] TRUE
$blocks$body[[1]]$createdDate
[1] "2026-04-16T14:06:40Z"
$blocks$body[[1]]$firstPublishedDate
[1] "2026-04-16T14:08:46Z"
$blocks$body[[1]]$publishedDate
[1] "2026-04-17T16:09:34Z"
$blocks$body[[1]]$lastModifiedDate
[1] "2026-04-17T16:09:34Z"
$blocks$body[[1]]$contributors
list()
$blocks$body[[1]]$elements
$blocks$body[[1]]$elements[[1]]
$blocks$body[[1]]$elements[[1]]$type
[1] "text"
$blocks$body[[1]]$elements[[1]]$assets
list()
$blocks$body[[1]]$elements[[1]]$textTypeData
$blocks$body[[1]]$elements[[1]]$textTypeData$html
[1] "<p>Senior government officials have been considering whether to withhold from parliament sensitive documents that show Peter Mandelson <a href=\"https://www.theguardian.com/politics/2026/apr/16/revealed-mandelson-failed-vetting-but-foreign-office-overruled-decision\">failed security vetting</a> before he assumed the role of US ambassador, the Guardian can reveal.</p> \n<p>Any such decision could amount to an extraordinary breach of a parliamentary vote, known as a humble address, that ordered the release of “all papers” relevant to Mandelson’s appointment.</p> \n<p>The Guardian has revealed that Mandelson did not receive vetting clearance from security officials, but that their decision was overruled by the Foreign Office to ensure he could take up his post.</p> \n<p>According to multiple sources, officials across government have been in dispute over whether to release documents that would reveal those facts, and other information about Mandelson’s security vetting, to the parliamentary intelligence and security committee (ISC).</p>"
$blocks$body[[1]]$elements[[2]]
$blocks$body[[1]]$elements[[2]]$type
[1] "rich-link"
$blocks$body[[1]]$elements[[2]]$assets
list()
$blocks$body[[1]]$elements[[2]]$richLinkTypeData
$blocks$body[[1]]$elements[[2]]$richLinkTypeData$url
[1] "https://www.theguardian.com/politics/2026/apr/16/five-key-questions-decision-peter-mandelson-security-clearance"
$blocks$body[[1]]$elements[[2]]$richLinkTypeData$originalUrl
[1] "https://www.theguardian.com/politics/2026/apr/16/five-key-questions-decision-peter-mandelson-security-clearance"
$blocks$body[[1]]$elements[[2]]$richLinkTypeData$linkText
[1] "Five key questions: who overruled decision to deny Mandelson security clearance?"
$blocks$body[[1]]$elements[[2]]$richLinkTypeData$linkPrefix
[1] "Related: "
$blocks$body[[1]]$elements[[2]]$richLinkTypeData$role
[1] "thumbnail"
$blocks$body[[1]]$elements[[3]]
$blocks$body[[1]]$elements[[3]]$type
[1] "text"
$blocks$body[[1]]$elements[[3]]$assets
list()
$blocks$body[[1]]$elements[[3]]$textTypeData
$blocks$body[[1]]$elements[[3]]$textTypeData$html
[1] "<p>The committee has been entrusted by parliament with the role of assessing the most sensitive papers relating to Mandelson’s appointment. Ministers have assured parliament that there will be no block on what the committee gets to see.</p>"
$blocks$body[[1]]$elements[[4]]
$blocks$body[[1]]$elements[[4]]$type
[1] "contentatom"
$blocks$body[[1]]$elements[[4]]$assets
list()
$blocks$body[[1]]$elements[[4]]$contentAtomTypeData
$blocks$body[[1]]$elements[[4]]$contentAtomTypeData$atomId
[1] "f4bcbd72-f585-4822-aa3f-789b3f582803"
$blocks$body[[1]]$elements[[4]]$contentAtomTypeData$atomType
[1] "media"
$blocks$body[[1]]$elements[[4]]$contentAtomTypeData$isMandatory
[1] FALSE
$blocks$body[[1]]$elements[[5]]
$blocks$body[[1]]$elements[[5]]$type
[1] "text"
$blocks$body[[1]]$elements[[5]]$assets
list()
$blocks$body[[1]]$elements[[5]]$textTypeData
$blocks$body[[1]]$elements[[5]]$textTypeData$html
[1] "<p>However, as of Thursday morning, a decision had not been made about whether the committee should have access to documents about Mandelson’s vetting by UK Security Vetting (UKSV) and the decision by the Foreign, Commonwealth and Development Office (FCDO) to override it.</p> \n<p>The ultimate decision will rest with the Cabinet Office. But sources said officials were split over how to proceed, with some arguing the materials should be withheld, despite a parliamentary motion to release all relevant materials. One source said: “There is no consensus.”</p> \n<p>Late on Thursday, after publication of the Guardian’s revelations, Downing Street indicated the documents would be released to the committee.</p> \n<p>“The government is committed to complying with the humble address in full as soon as possible,” a government spokesperson said. “Any documentation within the scope of the humble address that requires redaction on the basis of national security or international relations will be provided to the ISC. This will include documents provided to the FCDO by UK Security Vetting.”</p> \n<p>Lord Beamish, chair of the ISC, noted the government’s role in providing documents to the committee. But he added: “The committee would take a very dim view if documents relating to the humble address are withheld from the committee, one which I think would be shared by parliament.”</p> \n<p>Mandelson <a href=\"https://www.theguardian.com/politics/2025/sep/11/keir-starmer-sacks-peter-mandelson-over-jeffrey-epstein-revelations\">was sacked in September 2025</a> after the extent of his friendship with the convicted sex offender Jeffrey Epstein was revealed, in documents released to the public by the US department of justice.</p> \n<p>A second tranche of DoJ documents released in late January 2026 suggested Mandelson had passed market-sensitive information and Downing Street emails to Epstein while serving in Gordon Brown’s government, prompting fresh parliamentary scrutiny of Keir Starmer’s decision to appoint him.</p>"
$blocks$body[[1]]$elements[[6]]
$blocks$body[[1]]$elements[[6]]$type
[1] "text"
$blocks$body[[1]]$elements[[6]]$assets
list()
$blocks$body[[1]]$elements[[6]]$textTypeData
$blocks$body[[1]]$elements[[6]]$textTypeData$html
[1] "<p>In February 2026, parliament voted for a rare motion brought by the Tories to compel the government to publicly release “all papers relating to Mandelson’s appointment”. The language in the motion, called <a href=\"https://www.theguardian.com/politics/2026/feb/04/peter-mandelson-disclosures-what-is-humble-address-parliament\">a humble address</a>, made an exception for papers “prejudicial to UK national security or international relations”, which it said should be provided to the ISC.</p> \n<p>The committee comprises nine MPs and peers, including Jeremy Wright, a former attorney general, and Alan West, a retired Royal Navy admiral. Its members are sworn to secrecy under the Official Secrets Act and are given access to highly classified material. Tasked with overseeing the UK intelligence community on parliament’s behalf, they are now likely to demand answers over the precise basis for the UKSV decision.</p> \n<p>During the debate on the humble address, Derek Twigg, a member of the ISC, asked a minister, Chris Ward, for assurances there would be “no block whatsoever” on the material provided to the committee. Ward said: “Yes, I can confirm that.”</p> \n<p>According to an <a href=\"https://isc.independent.gov.uk/wp-content/uploads/2026/02/20260226-ISC-Press-Notice.docx-1.pdf\">ISC press notice</a>, the committee met top officials from the Cabinet Office after the humble address and it was agreed that to “fulfil the will of parliament”, it would be for the committee alone to assess material provided to it by the government and decide if it should be published.</p> \n<p>However, sources have said officials have been considering a different approach, and wrestling with whether there could be grounds to withhold highly sensitive UKSV vetting documents from the committee entirely, despite the vote.</p> \n<p>One source said officials were wary that disclosure of details of an individual’s vetting, even if access was restricted to a trusted group of parliamentarians, would be “unprecedented”. They pointed to strong concerns, including among the security services, about revealing documents.</p> \n<p>A second source said officials were grappling with a “live tactical question” over the release, and searching for loopholes that might enable them to withhold documents from the ISC. They said top officials were “looking for ways to comply with the letter of the humble address while potentially bypassing its spirit”.</p> \n<p>The second source said some in government were arguing that any disclosure of documents revealing Mandelon’s UKSV vetting had failed, only to be overruled by the Foreign Office, would “reflect poorly on the government’s overall integrity”. Others, they said, had been arguing that “precedent should be set aside” to comply with parliament’s wishes.</p> \n<p>Another option under consideration would be to show the redacted documents to a select number of the nine-person committee.</p> \n<p>The Guardian’s revelation that Mandelson had failed the developed vetting process is likely to put pressure on the government to release the documents to the public in full.</p>"
$blocks$body[[1]]$elements[[7]]
$blocks$body[[1]]$elements[[7]]$type
[1] "callout"
$blocks$body[[1]]$elements[[7]]$assets
list()
$blocks$body[[1]]$elements[[7]]$calloutTypeData
$blocks$body[[1]]$elements[[7]]$calloutTypeData$campaignId
[1] "10d669d8-b414-449c-8a7c-6319cb7b8eb8"
$blocks$body[[1]]$elements[[7]]$calloutTypeData$isNonCollapsible
[1] FALSE
$blocks$body[[1]]$elements[[8]]
$blocks$body[[1]]$elements[[8]]$type
[1] "text"
$blocks$body[[1]]$elements[[8]]$assets
list()
$blocks$body[[1]]$elements[[8]]$textTypeData
$blocks$body[[1]]$elements[[8]]$textTypeData$html
[1] "<p>Details of an individual’s UKSV developed vetting results have never before been publicly disclosed, and some top officials have been arguing that doing so would jeopardise national security.</p> \n<p>According to <a href=\"https://www.gov.uk/government/publications/united-kingdom-security-vetting-clearance-levels/dv-guidance-pack-for-applicants\">publicly available government documents</a>, the vetting process includes a questionnaire and interviews requiring disclosure of highly private information, including about personal finances, business connections and sexual history. The security services are also consulted.</p> \n<p>Regardless of the final decision by the Cabinet Office, the notion that officials have even contemplated whether to withhold the Mandelson vetting documents, in what would appear to be a breach of parliament’s wishes, could prove hugely controversial.</p> \n<p>Such a move could raise a constitutional conflict, with officialdom and the security services challenging the supremacy of parliament. It would also raise questions about whether the government is adhering to its word.</p> \n<p>In a letter on 6 February 2026 to the chair of the committee, Starmer said: “Above all else, the government wishes to engage constructively with the ISC, and to ensure that parliament’s instruction is met with the urgency and transparency it deserves.”</p> \n<p>The Foreign Office, Downing Street and the Cabinet Office have been approached for comment.</p>"
$blocks$totalBodyBlocks
[1] 1
$isHosted
[1] FALSE
$pillarId
[1] "pillar/news"
$pillarName
[1] "News"[1] "politics/2026/apr/16/officials-debate-withholding-mandelson-vetting-documents-from-parliament"[1] "article"[1] "2026-04-16 17:39:59 UTC"[1] "Officials debate withholding Mandelson vetting documents from parliament"Parsing the response: building a data wrangling function
So far so good, but where is the text? It seems it is stored in these “blocks” -> “body” elements. Let’s have a look:
[[1]]
[[1]]$id
[1] "69e0b7b18f08fbc58317e1db"
[[1]]$bodyHtml
[1] "<p>Senior government officials have been considering whether to withhold from parliament sensitive documents that show Peter Mandelson <a href=\"https://www.theguardian.com/politics/2026/apr/16/revealed-mandelson-failed-vetting-but-foreign-office-overruled-decision\">failed security vetting</a> before he assumed the role of US ambassador, the Guardian can reveal.</p> <p>Any such decision could amount to an extraordinary breach of a parliamentary vote, known as a humble address, that ordered the release of “all papers” relevant to Mandelson’s appointment.</p> <p>The Guardian has revealed that Mandelson did not receive vetting clearance from security officials, but that their decision was overruled by the Foreign Office to ensure he could take up his post.</p> <p>According to multiple sources, officials across government have been in dispute over whether to release documents that would reveal those facts, and other information about Mandelson’s security vetting, to the parliamentary intelligence and security committee (ISC).</p> <aside class=\"element element-rich-link element--thumbnail\"> <p> <span>Related: </span><a href=\"https://www.theguardian.com/politics/2026/apr/16/five-key-questions-decision-peter-mandelson-security-clearance\">Five key questions: who overruled decision to deny Mandelson security clearance?</a> </p> </aside> <p>The committee has been entrusted by parliament with the role of assessing the most sensitive papers relating to Mandelson’s appointment. Ministers have assured parliament that there will be no block on what the committee gets to see.</p> <figure class=\"element element-atom\"> <gu-atom data-atom-id=\"f4bcbd72-f585-4822-aa3f-789b3f582803\" data-atom-type=\"media\" > </gu-atom> </figure> <p>However, as of Thursday morning, a decision had not been made about whether the committee should have access to documents about Mandelson’s vetting by UK Security Vetting (UKSV) and the decision by the Foreign, Commonwealth and Development Office (FCDO) to override it.</p> <p>The ultimate decision will rest with the Cabinet Office. But sources said officials were split over how to proceed, with some arguing the materials should be withheld, despite a parliamentary motion to release all relevant materials. One source said: “There is no consensus.”</p> <p>Late on Thursday, after publication of the Guardian’s revelations, Downing Street indicated the documents would be released to the committee.</p> <p>“The government is committed to complying with the humble address in full as soon as possible,” a government spokesperson said. “Any documentation within the scope of the humble address that requires redaction on the basis of national security or international relations will be provided to the ISC. This will include documents provided to the FCDO by UK Security Vetting.”</p> <p>Lord Beamish, chair of the ISC, noted the government’s role in providing documents to the committee. But he added: “The committee would take a very dim view if documents relating to the humble address are withheld from the committee, one which I think would be shared by parliament.”</p> <p>Mandelson <a href=\"https://www.theguardian.com/politics/2025/sep/11/keir-starmer-sacks-peter-mandelson-over-jeffrey-epstein-revelations\">was sacked in September 2025</a> after the extent of his friendship with the convicted sex offender Jeffrey Epstein was revealed, in documents released to the public by the US department of justice.</p> <p>A second tranche of DoJ documents released in late January 2026 suggested Mandelson had passed market-sensitive information and Downing Street emails to Epstein while serving in Gordon Brown’s government, prompting fresh parliamentary scrutiny of Keir Starmer’s decision to appoint him.</p> <p>In February 2026, parliament voted for a rare motion brought by the Tories to compel the government to publicly release “all papers relating to Mandelson’s appointment”. The language in the motion, called <a href=\"https://www.theguardian.com/politics/2026/feb/04/peter-mandelson-disclosures-what-is-humble-address-parliament\">a humble address</a>, made an exception for papers “prejudicial to UK national security or international relations”, which it said should be provided to the ISC.</p> <p>The committee comprises nine MPs and peers, including Jeremy Wright, a former attorney general, and Alan West, a retired Royal Navy admiral. Its members are sworn to secrecy under the Official Secrets Act and are given access to highly classified material. Tasked with overseeing the UK intelligence community on parliament’s behalf, they are now likely to demand answers over the precise basis for the UKSV decision.</p> <p>During the debate on the humble address, Derek Twigg, a member of the ISC, asked a minister, Chris Ward, for assurances there would be “no block whatsoever” on the material provided to the committee. Ward said: “Yes, I can confirm that.”</p> <p>According to an <a href=\"https://isc.independent.gov.uk/wp-content/uploads/2026/02/20260226-ISC-Press-Notice.docx-1.pdf\">ISC press notice</a>, the committee met top officials from the Cabinet Office after the humble address and it was agreed that to “fulfil the will of parliament”, it would be for the committee alone to assess material provided to it by the government and decide if it should be published.</p> <p>However, sources have said officials have been considering a different approach, and wrestling with whether there could be grounds to withhold highly sensitive UKSV vetting documents from the committee entirely, despite the vote.</p> <p>One source said officials were wary that disclosure of details of an individual’s vetting, even if access was restricted to a trusted group of parliamentarians, would be “unprecedented”. They pointed to strong concerns, including among the security services, about revealing documents.</p> <p>A second source said officials were grappling with a “live tactical question” over the release, and searching for loopholes that might enable them to withhold documents from the ISC. They said top officials were “looking for ways to comply with the letter of the humble address while potentially bypassing its spirit”.</p> <p>The second source said some in government were arguing that any disclosure of documents revealing Mandelon’s UKSV vetting had failed, only to be overruled by the Foreign Office, would “reflect poorly on the government’s overall integrity”. Others, they said, had been arguing that “precedent should be set aside” to comply with parliament’s wishes.</p> <p>Another option under consideration would be to show the redacted documents to a select number of the nine-person committee.</p> <p>The Guardian’s revelation that Mandelson had failed the developed vetting process is likely to put pressure on the government to release the documents to the public in full.</p> <div class=\"element element-callout\" data-campaign-id=\"10d669d8-b414-449c-8a7c-6319cb7b8eb8\" > </div> <p>Details of an individual’s UKSV developed vetting results have never before been publicly disclosed, and some top officials have been arguing that doing so would jeopardise national security.</p> <p>According to <a href=\"https://www.gov.uk/government/publications/united-kingdom-security-vetting-clearance-levels/dv-guidance-pack-for-applicants\">publicly available government documents</a>, the vetting process includes a questionnaire and interviews requiring disclosure of highly private information, including about personal finances, business connections and sexual history. The security services are also consulted.</p> <p>Regardless of the final decision by the Cabinet Office, the notion that officials have even contemplated whether to withhold the Mandelson vetting documents, in what would appear to be a breach of parliament’s wishes, could prove hugely controversial.</p> <p>Such a move could raise a constitutional conflict, with officialdom and the security services challenging the supremacy of parliament. It would also raise questions about whether the government is adhering to its word.</p> <p>In a letter on 6 February 2026 to the chair of the committee, Starmer said: “Above all else, the government wishes to engage constructively with the ISC, and to ensure that parliament’s instruction is met with the urgency and transparency it deserves.”</p> <p>The Foreign Office, Downing Street and the Cabinet Office have been approached for comment.</p>"
[[1]]$bodyTextSummary
[1] "Senior government officials have been considering whether to withhold from parliament sensitive documents that show Peter Mandelson failed security vetting before he assumed the role of US ambassador, the Guardian can reveal. Any such decision could amount to an extraordinary breach of a parliamentary vote, known as a humble address, that ordered the release of “all papers” relevant to Mandelson’s appointment. The Guardian has revealed that Mandelson did not receive vetting clearance from security officials, but that their decision was overruled by the Foreign Office to ensure he could take up his post. According to multiple sources, officials across government have been in dispute over whether to release documents that would reveal those facts, and other information about Mandelson’s security vetting, to the parliamentary intelligence and security committee (ISC).\nThe committee has been entrusted by parliament with the role of assessing the most sensitive papers relating to Mandelson’s appointment. Ministers have assured parliament that there will be no block on what the committee gets to see.\nHowever, as of Thursday morning, a decision had not been made about whether the committee should have access to documents about Mandelson’s vetting by UK Security Vetting (UKSV) and the decision by the Foreign, Commonwealth and Development Office (FCDO) to override it. The ultimate decision will rest with the Cabinet Office. But sources said officials were split over how to proceed, with some arguing the materials should be withheld, despite a parliamentary motion to release all relevant materials. One source said: “There is no consensus.” Late on Thursday, after publication of the Guardian’s revelations, Downing Street indicated the documents would be released to the committee. “The government is committed to complying with the humble address in full as soon as possible,” a government spokesperson said. “Any documentation within the scope of the humble address that requires redaction on the basis of national security or international relations will be provided to the ISC. This will include documents provided to the FCDO by UK Security Vetting.” Lord Beamish, chair of the ISC, noted the government’s role in providing documents to the committee. But he added: “The committee would take a very dim view if documents relating to the humble address are withheld from the committee, one which I think would be shared by parliament.” Mandelson was sacked in September 2025 after the extent of his friendship with the convicted sex offender Jeffrey Epstein was revealed, in documents released to the public by the US department of justice. A second tranche of DoJ documents released in late January 2026 suggested Mandelson had passed market-sensitive information and Downing Street emails to Epstein while serving in Gordon Brown’s government, prompting fresh parliamentary scrutiny of Keir Starmer’s decision to appoint him.\nIn February 2026, parliament voted for a rare motion brought by the Tories to compel the government to publicly release “all papers relating to Mandelson’s appointment”. The language in the motion, called a humble address, made an exception for papers “prejudicial to UK national security or international relations”, which it said should be provided to the ISC. The committee comprises nine MPs and peers, including Jeremy Wright, a former attorney general, and Alan West, a retired Royal Navy admiral. Its members are sworn to secrecy under the Official Secrets Act and are given access to highly classified material. Tasked with overseeing the UK intelligence community on parliament’s behalf, they are now likely to demand answers over the precise basis for the UKSV decision. During the debate on the humble address, Derek Twigg, a member of the ISC, asked a minister, Chris Ward, for assurances there would be “no block whatsoever” on the material provided to the committee. Ward said: “Yes, I can confirm that.” According to an ISC press notice, the committee met top officials from the Cabinet Office after the humble address and it was agreed that to “fulfil the will of parliament”, it would be for the committee alone to assess material provided to it by the government and decide if it should be published. However, sources have said officials have been considering a different approach, and wrestling with whether there could be grounds to withhold highly sensitive UKSV vetting documents from the committee entirely, despite the vote. One source said officials were wary that disclosure of details of an individual’s vetting, even if access was restricted to a trusted group of parliamentarians, would be “unprecedented”. They pointed to strong concerns, including among the security services, about revealing documents. A second source said officials were grappling with a “live tactical question” over the release, and searching for loopholes that might enable them to withhold documents from the ISC. They said top officials were “looking for ways to comply with the letter of the humble address while potentially bypassing its spirit”. The second source said some in government were arguing that any disclosure of documents revealing Mandelon’s UKSV vetting had failed, only to be overruled by the Foreign Office, would “reflect poorly on the government’s overall integrity”. Others, they said, had been arguing that “precedent should be set aside” to comply with parliament’s wishes. Another option under consideration would be to show the redacted documents to a select number of the nine-person committee. The Guardian’s revelation that Mandelson had failed the developed vetting process is likely to put pressure on the government to release the documents to the public in full.\nDetails of an individual’s UKSV developed vetting results have never before been publicly disclosed, and some top officials have been arguing that doing so would jeopardise national security. According to publicly available government documents, the vetting process includes a questionnaire and interviews requiring disclosure of highly private information, including about personal finances, business connections and sexual history. The security services are also consulted. Regardless of the final decision by the Cabinet Office, the notion that officials have even contemplated whether to withhold the Mandelson vetting documents, in what would appear to be a breach of parliament’s wishes, could prove hugely controversial. Such a move could raise a constitutional conflict, with officialdom and the security services challenging the supremacy of parliament. It would also raise questions about whether the government is adhering to its word. In a letter on 6 February 2026 to the chair of the committee, Starmer said: “Above all else, the government wishes to engage constructively with the ISC, and to ensure that parliament’s instruction is met with the urgency and transparency it deserves.” The Foreign Office, Downing Street and the Cabinet Office have been approached for comment."
[[1]]$attributes
named list()
[[1]]$published
[1] TRUE
[[1]]$createdDate
[1] "2026-04-16T14:06:40Z"
[[1]]$firstPublishedDate
[1] "2026-04-16T14:08:46Z"
[[1]]$publishedDate
[1] "2026-04-17T16:09:34Z"
[[1]]$lastModifiedDate
[1] "2026-04-17T16:09:34Z"
[[1]]$contributors
list()
[[1]]$elements
[[1]]$elements[[1]]
[[1]]$elements[[1]]$type
[1] "text"
[[1]]$elements[[1]]$assets
list()
[[1]]$elements[[1]]$textTypeData
[[1]]$elements[[1]]$textTypeData$html
[1] "<p>Senior government officials have been considering whether to withhold from parliament sensitive documents that show Peter Mandelson <a href=\"https://www.theguardian.com/politics/2026/apr/16/revealed-mandelson-failed-vetting-but-foreign-office-overruled-decision\">failed security vetting</a> before he assumed the role of US ambassador, the Guardian can reveal.</p> \n<p>Any such decision could amount to an extraordinary breach of a parliamentary vote, known as a humble address, that ordered the release of “all papers” relevant to Mandelson’s appointment.</p> \n<p>The Guardian has revealed that Mandelson did not receive vetting clearance from security officials, but that their decision was overruled by the Foreign Office to ensure he could take up his post.</p> \n<p>According to multiple sources, officials across government have been in dispute over whether to release documents that would reveal those facts, and other information about Mandelson’s security vetting, to the parliamentary intelligence and security committee (ISC).</p>"
[[1]]$elements[[2]]
[[1]]$elements[[2]]$type
[1] "rich-link"
[[1]]$elements[[2]]$assets
list()
[[1]]$elements[[2]]$richLinkTypeData
[[1]]$elements[[2]]$richLinkTypeData$url
[1] "https://www.theguardian.com/politics/2026/apr/16/five-key-questions-decision-peter-mandelson-security-clearance"
[[1]]$elements[[2]]$richLinkTypeData$originalUrl
[1] "https://www.theguardian.com/politics/2026/apr/16/five-key-questions-decision-peter-mandelson-security-clearance"
[[1]]$elements[[2]]$richLinkTypeData$linkText
[1] "Five key questions: who overruled decision to deny Mandelson security clearance?"
[[1]]$elements[[2]]$richLinkTypeData$linkPrefix
[1] "Related: "
[[1]]$elements[[2]]$richLinkTypeData$role
[1] "thumbnail"
[[1]]$elements[[3]]
[[1]]$elements[[3]]$type
[1] "text"
[[1]]$elements[[3]]$assets
list()
[[1]]$elements[[3]]$textTypeData
[[1]]$elements[[3]]$textTypeData$html
[1] "<p>The committee has been entrusted by parliament with the role of assessing the most sensitive papers relating to Mandelson’s appointment. Ministers have assured parliament that there will be no block on what the committee gets to see.</p>"
[[1]]$elements[[4]]
[[1]]$elements[[4]]$type
[1] "contentatom"
[[1]]$elements[[4]]$assets
list()
[[1]]$elements[[4]]$contentAtomTypeData
[[1]]$elements[[4]]$contentAtomTypeData$atomId
[1] "f4bcbd72-f585-4822-aa3f-789b3f582803"
[[1]]$elements[[4]]$contentAtomTypeData$atomType
[1] "media"
[[1]]$elements[[4]]$contentAtomTypeData$isMandatory
[1] FALSE
[[1]]$elements[[5]]
[[1]]$elements[[5]]$type
[1] "text"
[[1]]$elements[[5]]$assets
list()
[[1]]$elements[[5]]$textTypeData
[[1]]$elements[[5]]$textTypeData$html
[1] "<p>However, as of Thursday morning, a decision had not been made about whether the committee should have access to documents about Mandelson’s vetting by UK Security Vetting (UKSV) and the decision by the Foreign, Commonwealth and Development Office (FCDO) to override it.</p> \n<p>The ultimate decision will rest with the Cabinet Office. But sources said officials were split over how to proceed, with some arguing the materials should be withheld, despite a parliamentary motion to release all relevant materials. One source said: “There is no consensus.”</p> \n<p>Late on Thursday, after publication of the Guardian’s revelations, Downing Street indicated the documents would be released to the committee.</p> \n<p>“The government is committed to complying with the humble address in full as soon as possible,” a government spokesperson said. “Any documentation within the scope of the humble address that requires redaction on the basis of national security or international relations will be provided to the ISC. This will include documents provided to the FCDO by UK Security Vetting.”</p> \n<p>Lord Beamish, chair of the ISC, noted the government’s role in providing documents to the committee. But he added: “The committee would take a very dim view if documents relating to the humble address are withheld from the committee, one which I think would be shared by parliament.”</p> \n<p>Mandelson <a href=\"https://www.theguardian.com/politics/2025/sep/11/keir-starmer-sacks-peter-mandelson-over-jeffrey-epstein-revelations\">was sacked in September 2025</a> after the extent of his friendship with the convicted sex offender Jeffrey Epstein was revealed, in documents released to the public by the US department of justice.</p> \n<p>A second tranche of DoJ documents released in late January 2026 suggested Mandelson had passed market-sensitive information and Downing Street emails to Epstein while serving in Gordon Brown’s government, prompting fresh parliamentary scrutiny of Keir Starmer’s decision to appoint him.</p>"
[[1]]$elements[[6]]
[[1]]$elements[[6]]$type
[1] "text"
[[1]]$elements[[6]]$assets
list()
[[1]]$elements[[6]]$textTypeData
[[1]]$elements[[6]]$textTypeData$html
[1] "<p>In February 2026, parliament voted for a rare motion brought by the Tories to compel the government to publicly release “all papers relating to Mandelson’s appointment”. The language in the motion, called <a href=\"https://www.theguardian.com/politics/2026/feb/04/peter-mandelson-disclosures-what-is-humble-address-parliament\">a humble address</a>, made an exception for papers “prejudicial to UK national security or international relations”, which it said should be provided to the ISC.</p> \n<p>The committee comprises nine MPs and peers, including Jeremy Wright, a former attorney general, and Alan West, a retired Royal Navy admiral. Its members are sworn to secrecy under the Official Secrets Act and are given access to highly classified material. Tasked with overseeing the UK intelligence community on parliament’s behalf, they are now likely to demand answers over the precise basis for the UKSV decision.</p> \n<p>During the debate on the humble address, Derek Twigg, a member of the ISC, asked a minister, Chris Ward, for assurances there would be “no block whatsoever” on the material provided to the committee. Ward said: “Yes, I can confirm that.”</p> \n<p>According to an <a href=\"https://isc.independent.gov.uk/wp-content/uploads/2026/02/20260226-ISC-Press-Notice.docx-1.pdf\">ISC press notice</a>, the committee met top officials from the Cabinet Office after the humble address and it was agreed that to “fulfil the will of parliament”, it would be for the committee alone to assess material provided to it by the government and decide if it should be published.</p> \n<p>However, sources have said officials have been considering a different approach, and wrestling with whether there could be grounds to withhold highly sensitive UKSV vetting documents from the committee entirely, despite the vote.</p> \n<p>One source said officials were wary that disclosure of details of an individual’s vetting, even if access was restricted to a trusted group of parliamentarians, would be “unprecedented”. They pointed to strong concerns, including among the security services, about revealing documents.</p> \n<p>A second source said officials were grappling with a “live tactical question” over the release, and searching for loopholes that might enable them to withhold documents from the ISC. They said top officials were “looking for ways to comply with the letter of the humble address while potentially bypassing its spirit”.</p> \n<p>The second source said some in government were arguing that any disclosure of documents revealing Mandelon’s UKSV vetting had failed, only to be overruled by the Foreign Office, would “reflect poorly on the government’s overall integrity”. Others, they said, had been arguing that “precedent should be set aside” to comply with parliament’s wishes.</p> \n<p>Another option under consideration would be to show the redacted documents to a select number of the nine-person committee.</p> \n<p>The Guardian’s revelation that Mandelson had failed the developed vetting process is likely to put pressure on the government to release the documents to the public in full.</p>"
[[1]]$elements[[7]]
[[1]]$elements[[7]]$type
[1] "callout"
[[1]]$elements[[7]]$assets
list()
[[1]]$elements[[7]]$calloutTypeData
[[1]]$elements[[7]]$calloutTypeData$campaignId
[1] "10d669d8-b414-449c-8a7c-6319cb7b8eb8"
[[1]]$elements[[7]]$calloutTypeData$isNonCollapsible
[1] FALSE
[[1]]$elements[[8]]
[[1]]$elements[[8]]$type
[1] "text"
[[1]]$elements[[8]]$assets
list()
[[1]]$elements[[8]]$textTypeData
[[1]]$elements[[8]]$textTypeData$html
[1] "<p>Details of an individual’s UKSV developed vetting results have never before been publicly disclosed, and some top officials have been arguing that doing so would jeopardise national security.</p> \n<p>According to <a href=\"https://www.gov.uk/government/publications/united-kingdom-security-vetting-clearance-levels/dv-guidance-pack-for-applicants\">publicly available government documents</a>, the vetting process includes a questionnaire and interviews requiring disclosure of highly private information, including about personal finances, business connections and sexual history. The security services are also consulted.</p> \n<p>Regardless of the final decision by the Cabinet Office, the notion that officials have even contemplated whether to withhold the Mandelson vetting documents, in what would appear to be a breach of parliament’s wishes, could prove hugely controversial.</p> \n<p>Such a move could raise a constitutional conflict, with officialdom and the security services challenging the supremacy of parliament. It would also raise questions about whether the government is adhering to its word.</p> \n<p>In a letter on 6 February 2026 to the chair of the committee, Starmer said: “Above all else, the government wishes to engage constructively with the ISC, and to ensure that parliament’s instruction is met with the urgency and transparency it deserves.”</p> \n<p>The Foreign Office, Downing Street and the Cabinet Office have been approached for comment.</p>"Parsing the response: building a data wrangling function
It seems the API returns articles as HTML strings. Luckily, we know how to extract text from that 😎
[1] "Senior government officials have been considering whether to withhold from parliament sensitive documents that show Peter Mandelson failed security vetting before he assumed the role of US ambassador, the Guardian can reveal.\n\nAny such decision could amount to an extraordinary breach of a parliamentary vote, known as a humble address, that ordered the release of “all papers” relevant to Mandelson’s appointment.\n\nThe Guardian has revealed that Mandelson did not receive vetting clearance from security officials, but that their decision was overruled by the Foreign Office to ensure he could take up his post.\n\nAccording to multiple sources, officials across government have been in dispute over whether to release documents that would reveal those facts, and other information about Mandelson’s security vetting, to the parliamentary intelligence and security committee (ISC).\n\nRelated: Five key questions: who overruled decision to deny Mandelson security clearance?\n\nThe committee has been entrusted by parliament with the role of assessing the most sensitive papers relating to Mandelson’s appointment. Ministers have assured parliament that there will be no block on what the committee gets to see.\n\nHowever, as of Thursday morning, a decision had not been made about whether the committee should have access to documents about Mandelson’s vetting by UK Security Vetting (UKSV) and the decision by the Foreign, Commonwealth and Development Office (FCDO) to override it.\n\nThe ultimate decision will rest with the Cabinet Office. But sources said officials were split over how to proceed, with some arguing the materials should be withheld, despite a parliamentary motion to release all relevant materials. One source said: “There is no consensus.”\n\nLate on Thursday, after publication of the Guardian’s revelations, Downing Street indicated the documents would be released to the committee.\n\n“The government is committed to complying with the humble address in full as soon as possible,” a government spokesperson said. “Any documentation within the scope of the humble address that requires redaction on the basis of national security or international relations will be provided to the ISC. This will include documents provided to the FCDO by UK Security Vetting.”\n\nLord Beamish, chair of the ISC, noted the government’s role in providing documents to the committee. But he added: “The committee would take a very dim view if documents relating to the humble address are withheld from the committee, one which I think would be shared by parliament.”\n\nMandelson was sacked in September 2025 after the extent of his friendship with the convicted sex offender Jeffrey Epstein was revealed, in documents released to the public by the US department of justice.\n\nA second tranche of DoJ documents released in late January 2026 suggested Mandelson had passed market-sensitive information and Downing Street emails to Epstein while serving in Gordon Brown’s government, prompting fresh parliamentary scrutiny of Keir Starmer’s decision to appoint him.\n\nIn February 2026, parliament voted for a rare motion brought by the Tories to compel the government to publicly release “all papers relating to Mandelson’s appointment”. The language in the motion, called a humble address, made an exception for papers “prejudicial to UK national security or international relations”, which it said should be provided to the ISC.\n\nThe committee comprises nine MPs and peers, including Jeremy Wright, a former attorney general, and Alan West, a retired Royal Navy admiral. Its members are sworn to secrecy under the Official Secrets Act and are given access to highly classified material. Tasked with overseeing the UK intelligence community on parliament’s behalf, they are now likely to demand answers over the precise basis for the UKSV decision.\n\nDuring the debate on the humble address, Derek Twigg, a member of the ISC, asked a minister, Chris Ward, for assurances there would be “no block whatsoever” on the material provided to the committee. Ward said: “Yes, I can confirm that.”\n\nAccording to an ISC press notice, the committee met top officials from the Cabinet Office after the humble address and it was agreed that to “fulfil the will of parliament”, it would be for the committee alone to assess material provided to it by the government and decide if it should be published.\n\nHowever, sources have said officials have been considering a different approach, and wrestling with whether there could be grounds to withhold highly sensitive UKSV vetting documents from the committee entirely, despite the vote.\n\nOne source said officials were wary that disclosure of details of an individual’s vetting, even if access was restricted to a trusted group of parliamentarians, would be “unprecedented”. They pointed to strong concerns, including among the security services, about revealing documents.\n\nA second source said officials were grappling with a “live tactical question” over the release, and searching for loopholes that might enable them to withhold documents from the ISC. They said top officials were “looking for ways to comply with the letter of the humble address while potentially bypassing its spirit”.\n\nThe second source said some in government were arguing that any disclosure of documents revealing Mandelon’s UKSV vetting had failed, only to be overruled by the Foreign Office, would “reflect poorly on the government’s overall integrity”. Others, they said, had been arguing that “precedent should be set aside” to comply with parliament’s wishes.\n\nAnother option under consideration would be to show the redacted documents to a select number of the nine-person committee.\n\nThe Guardian’s revelation that Mandelson had failed the developed vetting process is likely to put pressure on the government to release the documents to the public in full.\n\nDetails of an individual’s UKSV developed vetting results have never before been publicly disclosed, and some top officials have been arguing that doing so would jeopardise national security.\n\nAccording to publicly available government documents, the vetting process includes a questionnaire and interviews requiring disclosure of highly private information, including about personal finances, business connections and sexual history. The security services are also consulted.\n\nRegardless of the final decision by the Cabinet Office, the notion that officials have even contemplated whether to withhold the Mandelson vetting documents, in what would appear to be a breach of parliament’s wishes, could prove hugely controversial.\n\nSuch a move could raise a constitutional conflict, with officialdom and the security services challenging the supremacy of parliament. It would also raise questions about whether the government is adhering to its word.\n\nIn a letter on 6 February 2026 to the chair of the committee, Starmer said: “Above all else, the government wishes to engage constructively with the ISC, and to ensure that parliament’s instruction is met with the urgency and transparency it deserves.”\n\nThe Foreign Office, Downing Street and the Cabinet Office have been approached for comment."Parsing the response: finising the data wrangling function
Let’s put this all together:
# A tibble: 1 × 5
id type time headline text
<chr> <chr> <dttm> <chr> <chr>
1 politics/2026/apr/16/officials-debat… arti… 2026-04-16 17:39:59 Officia… "Sen…We can loop over all articles returned by the API and apply this function to it:
# A tibble: 10 × 5
id type time headline text
<chr> <chr> <dttm> <chr> <chr>
1 politics/2026/apr/16/officials-deba… arti… 2026-04-16 17:39:59 "Offici… "Sen…
2 news/2026/may/21/jamaica-parliament… arti… 2026-05-21 12:38:35 "‘It’s … "Whe…
3 society/2026/may/04/parliament-must… arti… 2026-05-04 15:56:51 "Parlia… "The…
4 uk-news/2026/mar/03/parliament-suic… arti… 2026-03-03 21:05:52 "UK par… "Par…
5 world/2026/jul/22/india-cockroach-j… arti… 2026-07-22 10:04:40 "Opposi… "Ind…
6 society/2026/mar/17/scottish-parlia… arti… 2026-03-17 22:44:37 "Scotti… "The…
7 politics/2026/apr/22/robbins-says-o… arti… 2026-04-22 05:00:09 "Robbin… "Oll…
8 politics/2026/jul/08/andy-burnham-l… arti… 2026-07-08 21:44:43 "Burnha… "And…
9 politics/live/2026/apr/24/keir-star… live… 2026-04-24 15:11:33 "Assist… "Tha…
10 australia-news/live/2026/mar/10/ira… live… 2026-03-10 07:42:20 "Fuel s… "Tha…Exercises 1
First, review the material and make sure you have a broad understanding how to:
- build a request to an API
- perform the request
- handle the response
httr2has several more functions to customize how a request is performed. What do these functions do?
req_throttle:req_error:req_retry:
Make your own request to the API with a different search term
You might want to add more information to the data.frame. Adapt the function parse_response to also extract: apiUrl, lastModifiedDate, pillarId
Request page 2 from the API. You can search for the correct query parameter in the documentation here: https://open-platform.theguardian.com/documentation/
Example: The UK Parliament API
Background
- The UK parliament offers several APIs
- You can get data on members, constituencies votes etc.
- The documentation is generated from OpenAPI specifications and rendered with swagger, which is quite convenient
Exploring the Docs
We can look for an endpoint that interests us and even run an example right here!
We even get a Curl call, which makes this really convenient!
Note: what are cURL calls
cURLis a library that can make HTTP requests- think of it as a general non-R-specific
httr2 - it is widely used for API calls from the terminal.
- it lists the parameters of a call in a pretty readable manner:
- the unnamed argument in the beginning is the Uniform Resource Locator (URL) the request goes to
-Harguments describe the headers, which are arguments sent with the call-dis the data or body of a request, which is used e.g., for uploading things--compressedmeans to ask for a compressed response which is unpacked locally (saves bandwidth)
A more advanced curl call
Translating the example request
What’s great about curl calls is that httr2 has a way to translate them into R code:
request("https://members-api.parliament.uk/api/Members/Search") |>
req_method("GET") |>
req_url_query(
Name = "Major",
skip = "0",
take = "20",
) |>
req_headers(
accept = "text/plain",
) |>
req_perform()Some pointers:
- make sure to escape
"when translating curl calls. You can use the search and replace tool in RStudio and turn"inside the curl string into\" - when you call just
curl_translate(), it uses what is currently in your clipboard, parses it, and copies the result back to your clipboard - alternatively, use https://curlconverter.com/r-httr2/
Making the first request from R
We can copy the output from curl_translate() and run it in R. I also added the resp_body_json() since we already know the returned data will be json.
[1] 1<list>
├─value: <list>
│ ├─id: 119
│ ├─nameListAs: "Major, Mr John"
│ ├─nameDisplayAs: "Mr John Major"
│ ├─nameFullTitle: "Rt Hon John Major"
│ ├─nameAddressAs: "Mr Major"
│ ├─latestParty: <list>
│ │ ├─id: 4
│ │ ├─name: "Conservative"
│ │ ├─abbreviation: "Con"
│ │ ├─backgroundColour: "0063ba"
│ │ ├─foregroundColour: "ffffff"
│ │ ├─isLordsMainParty: TRUE
│ │ ├─isLordsSpiritualParty: FALSE
│ │ ├─governmentType: 3
│ │ └─isIndependentParty: FALSE
│ ├─gender: "M"
│ ├─latestHouseMembership: <list>
│ │ ├─membershipFrom: "Huntingdon"
│ │ ├─membershipFromId: 1530
│ │ ├─house: 1
│ │ ├─membershipStartDate: "1979-05-03T00:00:00"
│ │ ├─membershipEndDate: "2001-06-07T00:00:00"
│ │ ├─membershipEndReason: "Dissolution"
... Wrangling the data
As usual, we get some meta information like totalResults and the data in a list. To make the items more useful, we can bring them into a tabular format.
items <- pluck(search, "items")
tibble(
id = map_int(items, function(i) pluck(i, "value", "id")),
nameListAs = map_chr(items, function(i) pluck(i, "value", "nameListAs")),
nameDisplayAs = map_chr(items, function(i) {
pluck(i, "value", "nameDisplayAs")
}),
nameFullTitle = map_chr(items, function(i) {
pluck(i, "value", "nameFullTitle")
}),
nameAddressAs = map_chr(items, function(i) {
pluck(i, "value", "nameAddressAs")
}),
gender = map_chr(items, function(i) pluck(i, "value", "gender")),
latestParty = map(items, function(i) pluck(i, "value", "latestParty")),
latestHouseMembership = map(items, function(i) {
pluck(i, "value", "latestHouseMembership")
}),
test = map_chr(items, function(i) pluck(i, "value", "test", .default = NA))
)# A tibble: 1 × 9
id nameListAs nameDisplayAs nameFullTitle nameAddressAs gender latestParty
<int> <chr> <chr> <chr> <chr> <chr> <list>
1 119 Major, Mr… Mr John Major Rt Hon John … Mr Major M <named list>
# ℹ 2 more variables: latestHouseMembership <list>, test <chr>This code is relatively busy, so let’s deconstruct it a little:
tibblewraps the results in a tibbleitemsis a list, to extract the first element from it, we usedpluck(search, "items", 1), but usually we have more than 1 result, so we need to loop over the results using amap_*function- We know what types to expect from our first request, so we choose
map_intfor integer fields,map_chrfor character fields andmapfor lists
Wrangling the data
As usual, we get some meta information like totalResults and the data in a list. To make the items more useful, we can bring them into a tabular format.
items <- pluck(search, "items")
tibble(
id = map_int(items, function(i) pluck(i, "value", "id")),
nameListAs = map_chr(items, function(i) pluck(i, "value", "nameListAs")),
nameDisplayAs = map_chr(items, function(i) {
pluck(i, "value", "nameDisplayAs")
}),
nameFullTitle = map_chr(items, function(i) {
pluck(i, "value", "nameFullTitle")
}),
nameAddressAs = map_chr(items, function(i) {
pluck(i, "value", "nameAddressAs")
}),
gender = map_chr(items, function(i) pluck(i, "value", "gender")),
latestParty = map(items, function(i) pluck(i, "value", "latestParty")),
latestHouseMembership = map(items, function(i) {
pluck(i, "value", "latestHouseMembership")
}),
test = map_chr(items, function(i) pluck(i, "value", "test", .default = NA))
)# A tibble: 1 × 9
id nameListAs nameDisplayAs nameFullTitle nameAddressAs gender latestParty
<int> <chr> <chr> <chr> <chr> <chr> <list>
1 119 Major, Mr… Mr John Major Rt Hon John … Mr Major M <named list>
# ℹ 2 more variables: latestHouseMembership <list>, test <chr>This code is relatively busy, so let’s deconstruct it a little:
- I included the test column simply to show why we use
pluckhere instead of e.g.,i[["value"]][["id"]]: we can set a default value if nothing is found- many APIs are inconsistent in what they return
- if you try to extract a field deep in a list with
[[]], you will get an error that the field does not exist orNULL(which causes an error withtibble()) - returning
NAinstead makes the parsing safer and is good practice
Wrapping the endpoint in a function
The reason why APIs are useful is because you can request all kinds of information using a few parameters. This lends itself very well to wrapping specific calls in functions.
# make a new function with different default
safe_pluck <- function(...) {
pluck(..., .default = NA)
}
search_members <- function(name) {
# request
resp <- request("https://members-api.parliament.uk/api/Members/Search") |>
req_method("GET") |>
req_url_query(
Name = name
) |>
req_headers(
accept = "text/plain",
) |>
req_perform() |>
resp_body_json()
# wrangle
items <- pluck(resp, "items")
return(tibble(
id = map_int(items, function(i) safe_pluck(i, "value", "id")),
nameListAs = map_chr(items, function(i) {
safe_pluck(i, "value", "nameListAs")
}),
nameDisplayAs = map_chr(items, function(i) {
safe_pluck(i, "value", "nameDisplayAs")
}),
nameFullTitle = map_chr(items, function(i) {
safe_pluck(i, "value", "nameFullTitle")
}),
nameAddressAs = map_chr(items, function(i) {
safe_pluck(i, "value", "nameAddressAs")
}),
gender = map_chr(items, function(i) safe_pluck(i, "value", "gender")),
latestParty = map(items, function(i) safe_pluck(i, "value", "latestParty")),
latestHouseMembership = map(items, function(i) {
safe_pluck(i, "value", "latestHouseMembership")
})
))
}Next Step: testing our function
# A tibble: 4 × 8
id nameListAs nameDisplayAs nameFullTitle nameAddressAs gender latestParty
<int> <chr> <chr> <chr> <chr> <chr> <list>
1 512 Blair, Mr… Mr Tony Blair Rt Hon Tony … Mr Blair M <named list>
2 4182 Blair of … Lord Blair o… The Lord Bla… The Lord Bla… M <named list>
3 4377 Donaldson… Stuart Blair… Stuart Blair… Stuart Blair… M <named list>
4 5076 McDougall… Blair McDoug… Blair McDoug… <NA> M <named list>
# ℹ 1 more variable: latestHouseMembership <list># A tibble: 20 × 8
id nameListAs nameDisplayAs nameFullTitle nameAddressAs gender
<int> <chr> <chr> <chr> <chr> <chr>
1 5368 Booth-Smith, L. Lord Booth-S… The Lord Boo… <NA> M
2 727 Buchanan-Smith, Alick Alick Buchan… Rt Hon Alick… <NA> M
3 4756 Clarke-Smith, Brendan Brendan Clar… Brendan Clar… <NA> M
4 2723 Delacourt-Smith of Al… Baroness Del… The Baroness… <NA> F
5 2713 Dixon-Smith, L. Lord Dixon-S… The Lord Dix… The Lord Dix… M
6 152 Duncan Smith, Sir Iain Sir Iain Dun… Rt Hon Sir I… Sir Iain Dun… M
7 2490 Goldsmith, L. Lord Goldsmi… The Rt Hon. … <NA> M
8 4062 Goldsmith of Richmond… Lord Goldsmi… The Rt Hon. … <NA> M
9 29 Johnson Smith, Sir Ge… Sir Geoffrey… Sir Geoffrey… <NA> M
10 5341 Kyrke-Smith, Laura Laura Kyrke-… Laura Kyrke-… <NA> F
11 4554 McGregor-Smith, B. Baroness McG… The Baroness… <NA> F
12 5273 Naismith, Connor Connor Naism… Connor Naism… <NA> M
13 216 Naysmith, Dr Doug Dr Doug Nays… Dr Doug Nays… Dr Naysmith M
14 4738 Smith, Alyn Alyn Smith Alyn Smith <NA> M
15 95 Smith, Mr Andrew Mr Andrew Sm… Rt Hon Andre… Mr Smith M
16 1564 Smith, Angela Angela Smith Angela Smith Angela Smith F
17 30 Smith, Angela E. Angela E. Sm… Rt Hon Angel… <NA> F
18 4436 Smith, Cat Cat Smith Cat Smith MP Cat Smith F
19 1609 Smith, Chloe Chloe Smith Rt Hon Chloe… Chloe Smith F
20 1292 Smith, Sir Cyril Sir Cyril Sm… Sir Cyril Sm… <NA> M
# ℹ 2 more variables: latestParty <list>, latestHouseMembership <list>The Smith search is a little odd since there are surely more than 20 results for this common name.
Wrapping the endpoint in a function: add pagination
- Most APIs use pagination when the data matching a query becomes too big
- In that case you need to iterate through the pages to get everything
- The UK parliament APIs handles pagination through two parameters:
- skip: The number of records to skip from the first, default is 0
- take: The number of records to return, default is 20. Maximum is 20
So to get the second page with the next 20 results, we need to adapt the call:
resp <- request("https://members-api.parliament.uk/api/Members/Search") |>
req_method("GET") |>
req_url_query(
Name = "Smith",
take = 20,
skip = 20
) |>
req_headers(
accept = "text/plain",
) |>
req_perform() |>
resp_body_json()
# wrangle
items <- pluck(resp, "items")
tibble(
id = map_int(items, function(i) safe_pluck(i, "value", "id")),
nameListAs = map_chr(items, function(i) safe_pluck(i, "value", "nameListAs")),
nameDisplayAs = map_chr(items, function(i) {
safe_pluck(i, "value", "nameDisplayAs")
}),
nameFullTitle = map_chr(items, function(i) {
safe_pluck(i, "value", "nameFullTitle")
}),
nameAddressAs = map_chr(items, function(i) {
safe_pluck(i, "value", "nameAddressAs")
}),
gender = map_chr(items, function(i) safe_pluck(i, "value", "gender")),
latestParty = map(items, function(i) safe_pluck(i, "value", "latestParty")),
latestHouseMembership = map(items, function(i) {
safe_pluck(i, "value", "latestHouseMembership")
})
)# A tibble: 20 × 8
id nameListAs nameDisplayAs nameFullTitle nameAddressAs gender
<int> <chr> <chr> <chr> <chr> <chr>
1 5218 Smith, David David Smith David Smith MP <NA> M
2 1267 Smith, Sir Dudley Sir Dudley Smith Sir Dudley Smi… <NA> M
3 4609 Smith, Eleanor Eleanor Smith Eleanor Smith Eleanor Smith F
4 471 Smith, Geraldine Geraldine Smith Geraldine Smith <NA> F
5 4778 Smith, Greg Greg Smith Greg Smith MP <NA> M
6 3960 Smith, Henry Henry Smith Henry Smith Henry Smith M
7 4456 Smith, Jeff Jeff Smith Jeff Smith MP Jeff Smith M
8 681 Smith, John John Smith John Smith John Smith M
9 564 Smith, Mr John Mr John Smith Rt Hon John Sm… <NA> M
10 4118 Smith, Sir Julian Sir Julian Smith Rt Hon Sir Jul… <NA> M
11 2852 Smith, L. The Lord Smith The Rt Hon. th… <NA> M
12 4648 Smith, Laura Laura Smith Laura Smith Laura Smith F
13 541 Smith, Llew Llew Smith Llew Smith <NA> M
14 3928 Smith, Nick Nick Smith Nick Smith MP Nick Smith M
15 4042 Smith, Owen Owen Smith Owen Smith Owen Smith M
16 5301 Smith, Rebecca Rebecca Smith Rebecca Smith … <NA> F
17 639 Smith, Sir Robert Sir Robert Smith Sir Robert Smi… Sir Robert M
18 4478 Smith, Royston Royston Smith Royston Smith Royston Smith M
19 5117 Smith, Sarah Sarah Smith Sarah Smith MP <NA> F
20 1245 Smith, Timothy Timothy Smith Timothy Smith <NA> M
# ℹ 2 more variables: latestParty <list>, latestHouseMembership <list>Wrapping the endpoint in a function: add pagination
Based on this we can adapt the function
search_members <- function(name) {
# make the initial request
resp <- request("https://members-api.parliament.uk/api/Members/Search") |>
req_method("GET") |>
req_url_query(
Name = name,
take = 20,
skip = 0
) |>
req_headers(
accept = "text/plain",
) |>
req_perform() |>
resp_body_json()
# checking the total and setting things up for pagination
total <- resp$totalResults
message(total, " results found")
skip <- 0
page <- 1
# extract initial results
items <- pluck(resp, "items")
# while loops are repeated until the condition inside is FALSE
while (total > skip) {
skip <- skip + 20
page <- page + 1
# we print a little status message to let the user know work is ongoing
message("\t...fetching page ", page)
# we retrieve the next page by adding an increasing skip
resp <- request("https://members-api.parliament.uk/api/Members/Search") |>
req_method("GET") |>
req_url_query(
Name = name,
skip = skip,
take = 20
) |>
req_headers(
accept = "text/plain",
) |>
req_throttle(rate = 1) |> # do not make more than one request per second
req_perform() |>
resp_body_json()
# we append the original result with the new items
items <- c(items, pluck(resp, "items"))
}
# wrangle
return(tibble(
id = map_int(items, function(i) safe_pluck(i, "value", "id")),
nameListAs = map_chr(items, function(i) {
safe_pluck(i, "value", "nameListAs")
}),
nameDisplayAs = map_chr(items, function(i) {
safe_pluck(i, "value", "nameDisplayAs")
}),
nameFullTitle = map_chr(items, function(i) {
safe_pluck(i, "value", "nameFullTitle")
}),
nameAddressAs = map_chr(items, function(i) {
safe_pluck(i, "value", "nameAddressAs")
}),
gender = map_chr(items, function(i) safe_pluck(i, "value", "gender")),
latestParty = map(items, function(i) safe_pluck(i, "value", "latestParty")),
latestHouseMembership = map(items, function(i) {
safe_pluck(i, "value", "latestHouseMembership")
})
))
}# A tibble: 52 × 8
id nameListAs nameDisplayAs nameFullTitle nameAddressAs gender
<int> <chr> <chr> <chr> <chr> <chr>
1 5368 Booth-Smith, L. Lord Booth-S… The Lord Boo… <NA> M
2 727 Buchanan-Smith, Alick Alick Buchan… Rt Hon Alick… <NA> M
3 4756 Clarke-Smith, Brendan Brendan Clar… Brendan Clar… <NA> M
4 2723 Delacourt-Smith of Al… Baroness Del… The Baroness… <NA> F
5 2713 Dixon-Smith, L. Lord Dixon-S… The Lord Dix… The Lord Dix… M
6 152 Duncan Smith, Sir Iain Sir Iain Dun… Rt Hon Sir I… Sir Iain Dun… M
7 2490 Goldsmith, L. Lord Goldsmi… The Rt Hon. … <NA> M
8 4062 Goldsmith of Richmond… Lord Goldsmi… The Rt Hon. … <NA> M
9 29 Johnson Smith, Sir Ge… Sir Geoffrey… Sir Geoffrey… <NA> M
10 5341 Kyrke-Smith, Laura Laura Kyrke-… Laura Kyrke-… <NA> F
# ℹ 42 more rows
# ℹ 2 more variables: latestParty <list>, latestHouseMembership <list>Adding more parameters
- The documentation lists a whole lot of other parameters.
- We can copy them into the function to employ them when calling the API.
- We can set the defaults to
NULL, which means they are ignored byreq_url_querywhen not used - Documentations usually list the required parameters, for which you shouldn’t set a default
search_members <- function(
name = NULL,
location = NULL,
posttitle = NULL,
partyid = NULL,
house = NULL,
constituencyid = NULL,
namestartswith = NULL,
gender = NULL,
membershipstartedsince = NULL,
membershipended_membershipendedsince = NULL,
membershipended_membershipendreasonids = NULL,
membershipindaterange_wasmemberonorafter = NULL,
membershipindaterange_wasmemberonorbefore = NULL,
membershipindaterange_wasmemberofhouse = NULL,
iseligible = NULL,
iscurrentmember = NULL,
policyinterestid = NULL,
experience = NULL
) {
# 1. request
resp <- request("https://members-api.parliament.uk/api/Members/Search") |>
req_method("GET") |>
req_url_query(
Name = name,
Location = location,
PostTitle = posttitle,
PartyId = partyid,
House = house,
ConstituencyId = constituencyid,
NameStartsWith = namestartswith,
Gender = gender,
MembershipStartedSince = membershipstartedsince,
MembershipEnded.MembershipEndedSince = membershipended_membershipendedsince,
MembershipEnded.MembershipEndReasonIds = membershipended_membershipendreasonids,
MembershipInDateRange.WasMemberOnOrAfter = membershipindaterange_wasmemberonorafter,
MembershipInDateRange.WasMemberOnOrBefore = membershipindaterange_wasmemberonorbefore,
MembershipInDateRange.WasMemberOfHouse = membershipindaterange_wasmemberofhouse,
IsEligible = iseligible,
IsCurrentMember = iscurrentmember,
PolicyInterestId = policyinterestid,
Experience = experience,
take = 20
) |>
req_headers(
accept = "text/plain",
) |>
req_perform() |>
# 2. Parse
resp_body_json()
# checking the total and setting things up for pagination
total <- resp$totalResults
message(total, " results found")
skip <- 20
page <- 1
# extract initial results
items <- pluck(resp, "items")
# while loops are repeated until the condition inside is FALSE
while (total > skip) {
page <- page + 1
# we print a little status message to let the user know work is ongoing
message("\t...fetching page ", page)
# we retrieve the next page by adding an increasing skip
resp <- request("https://members-api.parliament.uk/api/Members/Search") |>
req_method("GET") |>
req_url_query(
Name = name,
Location = location,
PostTitle = posttitle,
PartyId = partyid,
House = house,
ConstituencyId = constituencyid,
NameStartsWith = namestartswith,
Gender = gender,
MembershipStartedSince = membershipstartedsince,
MembershipEnded.MembershipEndedSince = membershipended_membershipendedsince,
MembershipEnded.MembershipEndReasonIds = membershipended_membershipendreasonids,
MembershipInDateRange.WasMemberOnOrAfter = membershipindaterange_wasmemberonorafter,
MembershipInDateRange.WasMemberOnOrBefore = membershipindaterange_wasmemberonorbefore,
MembershipInDateRange.WasMemberOfHouse = membershipindaterange_wasmemberofhouse,
IsEligible = iseligible,
IsCurrentMember = iscurrentmember,
PolicyInterestId = policyinterestid,
Experience = experience,
take = 20,
skip = skip
) |>
req_headers(
accept = "text/plain",
) |>
req_perform() |>
# 2. Parse
resp_body_json()
# we append the original result with the new items
items <- c(items, pluck(resp, "items"))
# increase the skip number
skip <- skip + 20
}
# wrangle
return(tibble(
id = map_int(items, function(i) safe_pluck(i, "value", "id")),
nameListAs = map_chr(items, function(i) {
safe_pluck(i, "value", "nameListAs")
}),
nameDisplayAs = map_chr(items, function(i) {
safe_pluck(i, "value", "nameDisplayAs")
}),
nameFullTitle = map_chr(items, function(i) {
safe_pluck(i, "value", "nameFullTitle")
}),
nameAddressAs = map_chr(items, function(i) {
safe_pluck(i, "value", "nameAddressAs")
}),
gender = map_chr(items, function(i) safe_pluck(i, "value", "gender")),
latestParty = map(items, function(i) safe_pluck(i, "value", "latestParty")),
latestHouseMembership = map(items, function(i) {
safe_pluck(i, "value", "latestHouseMembership")
})
))
}# A tibble: 3 × 8
id nameListAs nameDisplayAs nameFullTitle nameAddressAs gender latestParty
<int> <chr> <chr> <chr> <chr> <chr> <list>
1 152 Duncan Sm… Sir Iain Dun… Rt Hon Sir I… Sir Iain Dun… M <named list>
2 4778 Smith, Gr… Greg Smith Greg Smith MP <NA> M <named list>
3 4118 Smith, Si… Sir Julian S… Rt Hon Sir J… <NA> M <named list>
# ℹ 1 more variable: latestHouseMembership <list>Adding documentation
In its current form, the function is working well, but to find out what the parameters do, you would have to visit the documentation website, which isn’t great. To make this more useful, we should add some documentation. In R, the roxygen2 package handles parsing documentation for packages We can use it here to add explanations to the parameters. You can easily add roxygen code to a function using the Code menu in RStudio and Insert Roxygen Skeleton:
#' Search for members of the UK Parliament
#'
#' @param name Members where name contains term specified
#' @param location Members where postcode or geographical location matches the term specified
#' @param posttitle Members which have held the post specified
#' @param partyid Members which are currently affiliated with party with party ID
#' @param house Members where their most recent house is the house specified (1 for Commons, 2 for Lords)
#' @param constituencyid Members which currently hold the constituency with constituency id
#' @param namestartswith Members with surname beginning with letter(s) specified
#' @param gender Members with the gender specified
#' @param membershipstartedsince Members who started on or after the date given
#' @param membershipended_membershipendedsince Members who left the House on or after the date given
#' @param membershipended_membershipendreasonids Members who left the House on or after the date given
#' @param membershipindaterange_wasmemberonorafter Members who were active on or after the date specified
#' @param membershipindaterange_wasmemberonorbefore Members who were active on or before the date specified
#' @param membershipindaterange_wasmemberofhouse Members who were active in the house specified (1 for Commons, 2 for Lords)
#' @param iseligible Members currently Eligible to sit in their House
#' @param iscurrentmember TRUE gives you members who are current
#' @param policyinterestid Members with specified policy interest
#' @param experience Members with specified experience
#'
#' @return a tibble
#' @export#'
#'
#' @examples
#' # example code
#' search_members(name = \"Smith\")
#'
search_members <- function(
name = NULL,
location = NULL,
posttitle = NULL,
partyid = NULL,
house = NULL,
constituencyid = NULL,
namestartswith = NULL,
gender = NULL,
membershipstartedsince = NULL,
membershipended_membershipendedsince = NULL,
membershipended_membershipendreasonids = NULL,
membershipindaterange_wasmemberonorafter = NULL,
membershipindaterange_wasmemberonorbefore = NULL,
membershipindaterange_wasmemberofhouse = NULL,
iseligible = NULL,
iscurrentmember = NULL,
policyinterestid = NULL,
experience = NULL
) {
# ...
}You could look at this documentation using this little helper function. Usually inside an R package, you just use devtools::document().
render_roxygen <- function(text) {
# 1. roxygen block -> Rd topic object -> .Rd source
rd <- roxygen2::roc_proc_text(roxygen2::rd_roclet(), text)[[1]]
rd_file <- tempfile(fileext = ".Rd")
writeLines(format(rd), rd_file)
# 2. render and display in browser
html <- tempfile(fileext = ".html")
tools::Rd2HTML(rd_file, out = html)
browseURL(html)
}Exercises 2
First, review the material and make sure you have a broad understanding how to:
- read the documentation of the UK Parliament API (the documentation is specific to the API, but the Swagger format they use is very common)
- how to translate a curl call
- What the individual parts of the
search_membersfunction are doing
To get more information about an MP, we can use the endpoint “/api/Members/{id}/Biography”
- Search for an MP you are interested in with the function above and use the id on the documentation website with “Try it out”
- Copy the Curl call and translate it into
httr2code - Wrangle the
governmentPostsreturned in the data into a tabular format
Bonus:
- Write a function which lets you request information given an ID and which wrangles the results
- Two more interesting endpoints are “/api/Posts/GovernmentPosts” and “/api/Posts/OppositionPosts”. What do they do and how can you request data from them
Example: Semantic Scholar
What do we want
- Get information about scholars and the papers they write
Exploring the documentation
- The documentation for the API can be found here: https://api.semanticscholar.org/api-docs/graph
- It is shown in the other common documentation format called ReDoc
- I personally prefer swagger, however, this format can be produced by the OpenAPI specification linked on the website (you can use ReDoc though if you want)
- There is a tool in
Rwhich opens a small server on your computer that can display OpenAPI specifications in the swagger format (url for semnatic scholar: https://api.semanticscholar.org/graph/v1/swagger.json)
Making a first request
We can use one of the examples and convert it into httr2:
Parsing the initial request
We note two meta information that are helpful later on:
The actual data sits in data and is a pretty well behaved list that we can just convert to a tibble:
# A tibble: 100 × 2
authorId name
<chr> <chr>
1 39765778 Adam D. Smith
2 2109352648 Adam W. Smith
3 2118081729 A. Smith
4 48961800 A. Smith
5 47450760 Adam E. Smith
6 2218400362 Adam C. Smith
7 2109352801 Adam J. Smith
8 39765787 Adam D. Smith
9 2290435364 Adam E. Smith
10 2109352863 Adam N. Smith
# ℹ 90 more rowsHowever, the information seems a bit sparse… But we’ll look at that later.
Wrapping the endpoint in a function and add pagination
First we wrap this in a function and add pagination to get all results:
find_scholar <- function(name, verbose = TRUE) {
# make initial request
res <- request("https://api.semanticscholar.org/graph/v1/author/search") |>
req_url_query(query = name, limit = 100) |>
req_retry(
is_transient = function(resp) resp_status(resp) == 429,
max_tries = 50,
backoff = function(tries) tries^2
) |>
req_perform() |>
resp_body_json()
# note total
total <- pluck(res, "total")
# display user message
if (verbose) {
message("Found ", total, " authors")
}
# note offset
nxt <- pluck(res, "next")
# wrangle initial data
data <- pluck(res, "data") |>
bind_rows()
page <- 1
#----- New Stuff -----#
# loop through pages until no new ones exist
while (!is.null(nxt)) {
# if there are not more results next is empty
page <- page + 1
message("\t...fetching page ", page)
res <- request("https://api.semanticscholar.org/graph/v1/author/search") |>
req_url_query(query = name, limit = 100, offset = nxt) |>
req_throttle(rate = 30 / 60) |> # make only 30 requests per minute
req_retry(
is_transient = function(resp) resp_status(resp) == 429,
max_tries = 50,
backoff = function(tries) tries^2
) |>
req_perform() |>
resp_body_json()
# get next offset; will be NULL on the last page
nxt <- pluck(res, "next")
data_new <- pluck(res, "data") |>
bind_rows()
data <- data |>
bind_rows(data_new)
}
return(data)
}# A tibble: 489 × 2
authorId name
<chr> <chr>
1 39765778 Adam D. Smith
2 2118081729 A. Smith
3 2109352648 Adam W. Smith
4 2218400362 Adam C. Smith
5 2109352801 Adam J. Smith
6 39765787 Adam D. Smith
7 2290435364 Adam E. Smith
8 2109352863 Adam N. Smith
9 2387402541 A. Smith
10 2109352675 Adam C. Smith
# ℹ 479 more rowsSo where is the rest of the data?
- Semantic scholar only returns authorId and name by default.
- But we also want papers.
- The API handles this through the fields parameter and you can request additional fields
- The given example is
https://api.semanticscholar.org/graph/v1/author/search?query=adam+smith&fields=name,aliases,url,papers.title,papers.year
We are only interested in some of the fields, so let’s build a new request and see what we get:
resp <- request("https://api.semanticscholar.org/graph/v1/author/search") |>
req_url_query(query = "Adam Smith") |>
req_url_query(
fields = "name,papers.title,papers.year,papers.fieldsOfStudy,papers.authors",
limit = 10
) |>
req_headers(accept = "application/json") |>
req_retry(
is_transient = function(resp) resp_status(resp) == 429,
max_tries = 50,
backoff = function(tries) tries^2
) |>
req_perform() |>
resp_body_json()This structure is a lot more demanding since we have nested content (authors inside papers inside scholars).
wrangle the data
For most of the wrangling here, we can use the unnest_ functions from the tidyverse:
adam_search <- pluck(resp, "data") |>
# bind initial data into a tibble
bind_rows() |>
# unnest papers list into columns
unnest_wider(papers) |>
# unnest authors into rows
unnest(authors) |>
# unnest the new authors into columns
unnest_wider(authors, names_sep = "_") |>
# fieldsOfStudy is a list within a list, so we call unnest twice
unnest(fieldsOfStudy, keep_empty = TRUE) |>
unnest(fieldsOfStudy, keep_empty = TRUE)We now get several useful columns including the field of study of a paper (which we could use to differentiate between different authors with the same name).
# A tibble: 2,539 × 8
authorId name paperId title year fieldsOfStudy authors_authorId
<chr> <chr> <chr> <chr> <int> <chr> <chr>
1 39765778 Adam D. Smith 01670b5c78… The … 2023 <NA> 15089134
2 39765778 Adam D. Smith 01670b5c78… The … 2023 <NA> 39765778
3 39765778 Adam D. Smith 01670b5c78… The … 2023 <NA> 7430051
4 39765778 Adam D. Smith 01670b5c78… The … 2023 <NA> 4704115
5 39765778 Adam D. Smith 01670b5c78… The … 2023 <NA> 3429443
6 39765778 Adam D. Smith 01670b5c78… The … 2023 <NA> 32546788
7 39765778 Adam D. Smith 0671806ef9… Fort… 2023 <NA> 49608903
8 39765778 Adam D. Smith 0671806ef9… Fort… 2023 <NA> 39765778
9 39765778 Adam D. Smith 44fc62276b… US A… 2023 <NA> 2221731705
10 39765778 Adam D. Smith 44fc62276b… US A… 2023 <NA> 17317553
# ℹ 2,529 more rows
# ℹ 1 more variable: authors_name <chr>Let’s put it all together in an extended function
find_scholar <- function(
name,
fields = "name,papers.title,papers.year,papers.fieldsOfStudy,papers.authors",
limit = 100
) {
# make initial request
res <- request("https://api.semanticscholar.org/graph/v1/author/search") |>
req_url_query(query = name) |>
req_url_query(fields = fields, limit = limit) |>
req_headers(accept = "application/json") |>
req_retry(
is_transient = function(resp) resp_status(resp) == 429,
max_tries = 50,
backoff = function(tries) tries^2
) |>
req_perform() |>
resp_body_json()
# note total
total <- pluck(res, "total")
# display user message
message("Found ", total, " authors")
# note offset
nxt <- pluck(res, "next")
# wrangle initial data
data <- parse_response(res)
page <- 1
# loop through pages until no new ones exist
while (!is.null(nxt)) {
page <- page + 1
message("\t...fetching page ", page)
res <- request("https://api.semanticscholar.org/graph/v1/author/search") |>
req_url_query(
query = name,
offset = nxt,
fields = fields,
limit = limit
) |>
req_throttle(rate = 30 / 60) |> # make only 30 requests per minute
req_headers(accept = "application/json") |>
req_retry(
is_transient = function(resp) resp_status(resp) == 429,
max_tries = 50,
backoff = function(tries) tries^2
) |>
req_perform() |>
resp_body_json()
# get next offset; will be NULL on the last page
nxt <- pluck(res, "next")
data_new <- pluck(res, "data") |>
bind_rows()
data <- data |>
bind_rows(data_new)
}
return(data)
}Let’s put it all together in an extended function
I separated the parsing function from this to make it easier to read.
parse_response <- function(resp) {
pluck(resp, "data") |>
# bind initial data into a tibble
bind_rows() |>
# unnest papers list into columns
unnest_wider(papers) |>
# unnest authors into rows
unnest(authors) |>
# unnest the new authors into columns
unnest_wider(authors, names_sep = "_") |>
# fieldsOfStudy is a list within a list, so we call unnest twice
unnest(fieldsOfStudy, keep_empty = TRUE) |>
unnest(fieldsOfStudy, keep_empty = TRUE)
}Let’s test it with Carolina (director of the summer school):
# A tibble: 285 × 8
authorId name paperId title year fieldsOfStudy authors_authorId
<chr> <chr> <chr> <chr> <int> <chr> <chr>
1 77113776 C. Garriga d1b9811456812… Revi… 2022 Medicine 2257761876
2 77113776 C. Garriga d1b9811456812… Revi… 2022 Medicine 2188893608
3 77113776 C. Garriga d1b9811456812… Revi… 2022 Medicine 77113776
4 77113776 C. Garriga d1b9811456812… Revi… 2022 Medicine 104185484
5 77113776 C. Garriga d1b9811456812… Revi… 2022 Medicine 77776275
6 77113776 C. Garriga d1b9811456812… Revi… 2022 Medicine 1397929278
7 77113776 C. Garriga d1b9811456812… Revi… 2022 Medicine 2139759754
8 77113776 C. Garriga 2b6ede1dc0d2d… Asso… 2015 Medicine 2238677829
9 77113776 C. Garriga 2b6ede1dc0d2d… Asso… 2015 Medicine 145003519
10 77113776 C. Garriga 2b6ede1dc0d2d… Asso… 2015 Medicine 77113776
# ℹ 275 more rows
# ℹ 1 more variable: authors_name <chr>Exercises 3
First, review the material. This example is pretty similar to the last one. But:
- it uses a different documentation style called ReDoc, which does not give you curl calls to copy
- it uses a different pagination: instead of using the total number of items, we look for new ones until nothing new is returned
- we throttle the number of requests
- Document the function we just created. This is mainly to let you think about the parameters and how you would describe their working to someone else
- Use the function to search for a couple of scholars of your choice. Who has the most co-authors and unique papers?
- Say you found an authors ID with the search function. How could you use “/author/{author_id}” and “/author/{author_id}/papers” to request more information about them?
Bonus:
- Write a function that wraps “/author/{author_id}”
Bonus 2:
- <
Wrap Up
Save some information about the session for reproducibility.
R version 4.6.1 (2026-06-24)
Platform: x86_64-pc-linux-gnu
Running under: EndeavourOS
Matrix products: default
BLAS: /usr/lib/libblas.so.3.12.0
LAPACK: /usr/lib/liblapack.so.3.12.0 LAPACK version 3.12.0
locale:
[1] LC_CTYPE=en_GB.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_GB.UTF-8 LC_COLLATE=en_GB.UTF-8
[5] LC_MONETARY=en_GB.UTF-8 LC_MESSAGES=en_GB.UTF-8
[7] LC_PAPER=en_GB.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C
time zone: Europe/Berlin
tzcode source: system (glibc)
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] rvest_1.0.4 tinytable_0.17.0 lubridate_1.9.5 forcats_1.0.1
[5] stringr_1.6.0 dplyr_1.2.1 purrr_1.2.2 readr_2.2.0
[9] tidyr_1.3.2 tibble_3.3.1 ggplot2_4.0.3 tidyverse_2.0.0
[13] httr2_1.2.3
loaded via a namespace (and not attached):
[1] utf8_1.2.6 rappdirs_0.3.4 generics_0.1.4 xml2_1.6.0
[5] stringi_1.8.7 hms_1.1.4 digest_0.6.39 magrittr_2.0.5
[9] evaluate_1.0.5 grid_4.6.1 timechange_0.4.0 RColorBrewer_1.1-3
[13] fastmap_1.2.0 jsonlite_2.0.0 processx_3.9.0 chromote_0.5.1
[17] ps_1.9.3 promises_1.5.0 httr_1.4.8 scales_1.4.0
[21] codetools_0.2-20 cli_3.6.6 rlang_1.2.0 withr_3.0.3
[25] yaml_2.3.12 otel_0.2.0 tools_4.6.1 tzdb_0.5.0
[29] curl_7.1.0 vctrs_0.7.3 R6_2.6.1 lifecycle_1.0.5
[33] pkgconfig_2.0.3 pillar_1.11.1 later_1.4.8 gtable_0.3.6
[37] glue_1.8.1 Rcpp_1.1.1-1.1 xfun_0.59 tidyselect_1.2.1
[41] knitr_1.51 farver_2.1.2 htmltools_0.5.9 websocket_1.4.4
[45] rmarkdown_2.31 compiler_4.6.1 S7_0.2.2 