Download Term Report PDF - VTechWorks

5 Project Requirements
Our goal for the project, at a high level, can be described as the following:
1. Identify and remove the “noise” in the data 2. Process and standardize the “sound” in the data 3. Extract and organize the data into a usable format As the Noise Reduction team, we are to clean up the collection of documents (tweets and web pages) by firstly identifying and removing character sequences that are irrelevant to various components of the IR system. After that, we standardize the remaining text using popular Natural Language Processing techniques to convert the text to a common structure/format. We then extract any information stored within the text that are valuable, such as twitter handles, URLs that are out links in web pages, etc. and store all of this information in a schema/format that aids the other teams that are responsible for building the remaining components of the IR system. 10