Introduction

Web scrapping / web data extraction is a method used to acquire data from websites around the globe. There are numerous commercial and open source tools that can be deployed to interact with websites either directly or through a hypertext transfer protocol (HTTP). Extraction of data from websites require an understanding of html file configuration or other web-based file formats. Web pages are designed to be static or dynamic, and this configuration impacts the way a web page will be scrapped.

Here I will show examples of the web pages that I have scraped using different python packages. The commonly used python scrapping packages are scrapy, selenium, beautiful soup, and request. You can read more about the applicability of these packages within the python frame work.

Note: It should be noted that web scrapping is not the preferred method of extracting data from the web. Use of API if available or requesting the data directly from the owner would be preferred. Finally, follow the terms and conditions before you start to web scrape any site.