Skip to content

arnavn101/WebXplore

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

WebXplore (v1.0.3)

Build Status PyPI - License codecov PyPI - Python Version

WebXplore offers multitude of tools for web scraping, crawling and performing computations on scraped information to determine sentiment values or tone of the author.

This package helps in retrieving information from these sources:

  1. Google Search: Get links from any google search query.

  2. Website Text: Use an intelligent parser to strip all the HTML tags from webpage contents.

  3. Twitter: Given a word or phrase, get related tweets.

  4. Reddit: Get the hottest posts given the subreddit and a key phrase.

  5. NewsAPI: Retrieve News Articles given topic or phrase.

Installation

$ pip install webxplore

or clone the repository.

$ git clone https://github.com/arnavn101/WebXplore.git

Getting Started

Here are steps for using webxplore.

1. Get Links from Google Search

from webxplore.web_searcher import SearchWeb

search_query = SearchWeb('Artificial Intelligence', 5)
print(search_query.returnListLinks())

2. Scrape a Website

from webxplore.web_scraper import ScrapeWebsite

scrape_query = ScrapeWebsite('https://en.wikipedia.org/wiki/Artificial_intelligence')
print(scrape_query.return_article())

3. Get Sentiments from Text

from webxplore.utils.sentiment import RetrieveSentiments

sentiment_analyzer = RetrieveSentiments('This is a good situation.')
print(sentiment_analyzer.returnFinalSentiment())

4. Get Summary of the Text

from webxplore.utils.summarizer import SummarizeText

textSummarizer = SummarizeText('He feels very scared. He wants to protect himself.', 1)
print(textSummarizer.returnFinalSummary())

5. Get Tone of the Text (for each sentence)

from webxplore.utils.analyzer import ToneAnalysis

textTone = ToneAnalysis('Laugh and the world laughs with you.' +
                        'Weep and you weep alone.', "watsonApiKey")
print(textTone.returnTone())

6. Use the news api to get the latest articles

from webxplore.search.news import RetrieveNewsArticle

newsArticles = RetrieveNewsArticle('Politics', 5, 'newsApiKey')
print(newsArticles.return_articleSentences())

7. Get Posts from a SubReddit

from webxplore.search.reddit import CrawlSubReddit

redditPosts = CrawlSubReddit('stocks', 'amazon', 10, 'RedditClientId',
                                          'RedditClientSecret', 'RedditUserAgent')
print(redditPosts.return_listSentences())

8. Get Tweets that have a key word

from webxplore.search.twitter import CrawlTwitter

retrieveTweets = CrawlTwitter('tesla', 10, 'TwitterConsumerKey', 'TwitterConsumerSecret',
                                        'TwitterAccountKey', 'TwitterAccountSecret')
print(retrieveTweets.return_tweets())

Contributions

Anyone is welcome to add any contribution to this repository. All good changes are welcome. Please create a pull request and ensure that it passes all the CI tests.

License

MIT License Copyright (c) 2020, Arnav Nidumolu

About

Web Scraping and Crawling Tools

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages