Collect, Enrich, & Store News Data

1. Create a list of Publisher Real Simple Syndication (RSS) feeds to aggregate news which includes

• publisher
• article title
• short summary

2. Developed with the company, Python scripts to retrieve feed items daily and enhance them with additional metadata.

3. Learn and use Java-based Stanford Natural Language Processing (NLP) tools and server to extract interesting entities from the title and short summary.

Stanford CoreNLP (“part of speech” classifier) splits sentences into tokens and performs Named Entity Recognition (NER) against the token stream (PERSON/PLACE/etc.)
A Part-Of-Speech Tagger (POS Tagger), like Stanford CoreNLP, is a piece of software that reads the text in some language and assigns parts of speech to each word.

For example:
New PLACE
York PLACE
Timothy PERSON
Geithner PERSON

4. Index/verify enhanced documents with the Apache Solr search platform. Python script runs and submits enhanced documents to Solr API for subsequent searching.

An example query could look like this:

{
"id":"d8cb2778-9e6d-4ff2-977f-5b83196a3220",
"publisher_name":"FOX News",
"neutrality_pubRSS_ID":"a0ed2b36-ea4b-403e-b02c-9b84fc5ce2c1",
"neutrality_article_thumbnail_url: ["https://a57.foxnews.com/static.foxnews.com/foxnews.com/content/uploads/2021/06/931/523/East-Coast-beaches-sharks.jpg?ve=1&tl=1{delim}https://static.foxnews.com/foxnews.com/content/uploads/2021/06/East-Coast-beaches-sharks.jpg"],
"neutrality_article_description":"A woman was attacked by a shark in the Maldives while free diving, and it was all caught on camera. The video of the bite has resurfaced on the internet.",
"path":"feeds.foxnews.com",
"neutrality_article_title":"Video of shark attacking snorkeler in Maldives resurfaces; tourist was left with 6-inch wound",
"neutrality_article_author":"Ryan Morik",
"neutrality_article_normpubDate":"2023-04-14T18:34:16Z",
"neutrality_article_url":"https://www.foxnews.com/sports/video-shark-attacking-snorkeler-maldives-resurfaces-tourist-left-6-inch-wound",
"sentiment":1.25,
"RSSmap":"General News",
}

This is where we would verify in Solr that the query looks good to use for training.