Books scraper for Google Scholar and Goodreads

1. Install Python for your operating. You can download Python 3.8.2 from here.
2. This program makes use of Selenium WebDriver for fetching GoodReads book shelf data. You should have a driver installed for your browser. Currently supported browsers are: Chrome, Firefox, Edge and Safari. We have tested with Firefox and Safari (on macOSX 10.14.6).

1. Open a shell
2. cd some_folder_where_you_want_this_code
3. git clone https://github.com/bsodhi/books_scraper.git
4. cd books_scraper
5. python3 -m venv give_some_name
6. source give_some_name/bin/activate
7. pip install -r requirements.txt
8. python3 books_scraper/scraper.py -h

The output is written as a csv file. For Goodreads data following columns are written to the csv file: ["author", "title", "isbn", "language", "avg_rating", "ratings", "pub_year", "book_format", "pages", "genre"]

For Google Scholar data, the columns are: ["author", "title", "citedby", "url", "abstract"]

This code is written by taking lot of help from StackOverflow community and Python API documentation. Greatly appreciated!