Python Web Scraping

Libraries for Python Web Scraping:

Requests

•  using it to get the raw ingredients (i.e. raw HTML)

http://docs.python-requests.org/en/master/user/quickstart/

BeautifulSoup  

• a parsing library that can use different parsers. A parser is simply a program that can extract data from HTML and XML documents.

https://www.crummy.com/software/BeautifulSoup/bs4/doc/

https://gist.github.com/bradmontgomery/1872970

lxml

• a high-performance, production-quality HTML and XML parsing library

http://lxml.de/index.html#introduction

http://docs.python-guide.org/en/latest/scenarios/scrape/

Selenium

• to scrape sites with data tucked away by JavaScript.

http://selenium-python.readthedocs.io/

http://thiagomarzagao.com/2013/11/12/webscraping-with-selenium-part-1/

https://www.scrapehero.com/tutorial-web-scraping-hotel-prices-using-selenium-and-python/

Scrapy

• if you need to build a real spider or web-crawler

https://doc.scrapy.org/en/latest/topics/architecture.html

https://medium.com/@kaismh/extracting-data-from-websites-using-scrapy-e1e1e357651a#.sw7c9ycio

https://scrapinghub.com/

Reference

https://elitedatascience.com/python-web-scraping-libraries