talkytrend.handler.scraper.ScraperHandler#
- class talkytrend.handler.scraper.ScraperHandler(**kwargs)[source]#
Bases:
ClientA class that handles scraping webpages.
Initialize the object with the given keyword arguments.
- Parameters:
kwargs – keyword arguments
- Returns:
None
Methods
fetch()Asynchronously scrapes a webpage and retrieves the content specified by the scraper_page_url and scraper_page_id attributes.
get_news()Asynchronously retrieves the latest news from various sources based on the configured settings.
monitor()Asynchronously monitors the system and retrieves various data sources based on the configured settings.
- async fetch()[source]#
Asynchronously scrapes a webpage and retrieves the content specified by the scraper_page_url and scraper_page_id attributes.
- Returns:
The content of the webpage as a string,
formatted using BeautifulSoup. If the scraper_page_id is not specified, the entire webpage is returned.
- Return type:
str
- async monitor()[source]#
Asynchronously monitors the system and retrieves various data sources based on the configured settings. Cover Events, Feed, and Signal.
- Returns:
- A string containing the concatenated results
of the retrieved data sources.
- Return type:
str
- async get_news()#
Asynchronously retrieves the latest news from various sources based on the configured settings.
- Returns:
- A string containing the concatenated
results of the retrieved news sources.
- Return type:
str