This repository contains materials for the workshop “Collecting Data from the Web: Foundation of Web Scraping.” The workshop introduces participants to the fundamentals of web scraping for research, covering both conceptual and practical aspects. Examples are provided in R and Python, allowing participants to follow along in their preferred language.
This intermediate workshop will teach you how to scrape data from the Internet using R and Python. This is an intermediate-level course. Students must have a basic background in R/Pyhton . This includes understanding the basic data types in R/Python; how to install and load packages; and how to use functions, pipes, and apply/map functions.
The workshop is delivered in a single session with a pre-reading.
Please read Collecting Data from the Web: Legal and Ethical Foundations before attending the workshop.
The document introduces the main legal and ethical considerations involved in collecting web data, including copyright, terms of service, data protection and research ethics.
Before the session, please complete both exercises at the end of the document (pages 13–14). We will discuss your answers at the beginning of the workshop, so brief notes for each question will be sufficient.
This session first introduces key concepts and considerations before writing any code. Participants will:
- Explore web page structures and how data is organised in HTML.
- Use SelectorGadget to identify elements on a web page for extraction.
This hands-on session then guides participants through scraping static websites. Participants can choose to work in either R or Python, with parallel code examples provided.
You'll need to install the SelectorGadget extension for the Chrome browser in addition to ensuring you have access to R/RStudio. You can use R/RStudio on the University's Noteable service, which requires no installation. You can also install R/Rstudio locally on your computer.
- Open the Chrome browser. If you don't have Chrome, you can install it here.
- Log in to the Chrome browser with your Google account. If you don't have a Google account, you'll need to sign up for one.
- Follow this link to download the SelectorGadget browser extension.
- Make sure SelectorGadget is enabled in your Chrome extension toolbar by clicking on the 'Customize and control Google Chrome' icon in the top righthand corner of your browser pane. Mouse over to 'Extensions' on the dropdown menu and select 'Manage Extensions.' Make sure SelectorGadget is toggled 'on.'
- Open the login page here.
- Login with your EASE credentials.
- Select RStudio as a personal notebook server and press start.
- Go to File > New Project> Version Control > Git
- Copy and Paste this repository URL https://github.com/EFI-training/Web-Scraping/ as the Repository URL. (The Project directory name will be filled in automatically, but you can change it if you want your folder in Notable to have a different name).
- Decide where you'd like to store the folder. By default, it will be located in your home directory.
- Select 'Create Project'.
Congratulations! You have now pulled the content of this repository to your Notable server space.
- Go to https://www.r-project.org/
- Go to the download link
- Choose the closest CRAN mirror (if you're in Edinburgh, select either Bristol or Imperial College London)
- Download the version compatible with your operating system (Windows/Mac/Linux)
- For Windows: click on install R for the first time. Then download R for Windows and follow the installation widget. If you get stuck, follow this video tutorial
- For Mac: download the most recent pkg file and follow the installation widget. If you get stuck, follow this video tutorial
- Once R is installed, you can install R studio (R interface)
- Go to www.rstudio.com
- Find the download page
- Download the version compatible with your operating system
You'll need to install the SelectorGadget extension for the Chrome browser in addition to ensuring you have access to Jupyter Notebook. You can use Jupyter Notebook on the University's Noteable service, which requires no installation. You can also install Jupyter Notebook locally on your computer.
- Open the Chrome browser. If you don't have Chrome, you can install it here.
- Log in to the Chrome browser with your Google account. If you don't have a Google account, you'll need to sign up for one.
- Follow this link to download the SelectorGadget browser extension.
- Make sure SelectorGadget is enabled in your Chrome extension toolbar by clicking on the 'Customize and control Google Chrome' icon in the top righthand corner of your browser pane. Mouse over to 'Extensions' on the dropdown menu and select 'Manage Extensions.' Make sure SelectorGadget is toggled 'on.'
- Open the login page here.
- Login with your EASE credentials.
- Select Standard Python 3 as a personal notebook server and press start.
- Go to Git > Clone a Repository
- Copy and Paste this repository URL [https://github.com/EFI-training/Web-Scraping] as the Repository URL. (The Project directory name will be filled in automatically, but you can change it if you want your folder in Notable to have a different name).
- Click 'Clone'
Congratulations! You have now pulled the content of this repository to your Notable server space.
- Make sure you have Python installed. If you use Windows, I recommend installing Anaconda. For other operating systems, see https://wiki.python.org/moin/BeginnersGuide/Download.
- If you use Anaconda, install Jupyter via Anaconda (if it is not installed yet). If not, you can use
pipto install Jupyter Notebook. Type the following into your terminal:
python3 -m pip install notebook- Launch your Jupyter Notebook either via Anaconda Navigator, or via your terminal:
jupyter notebookReddit community focused on web scraping
Programming Historian, Fetching and Parsing Data from the Web with OpenRefine and Automated Downloading with Wget
University of Southampton Web Data Research Assistant
Public APIs GitHub repo
Tutorial for browser automation in Python using Selenium