Data-Mining (Sources and Methods)

[felixhoffmnn]

Describe the Problem

For the thesis, it is important to get an overview over the state of the art and what method to use. For that, we need to follow the steps below.

• Define and structure important topics
• Define and structure important methods
• Search and discuss books, papers, and other sources

[l7f19]

1. Brainstorming → Project Structure

Sources shall cover the following topics:

• Business Understanding
  • 💡 Understanding the correlation between a political event and the reaction on social media
  • Analyze a political statement or tweet to check if the content complies with the party
  • ❓ Related Work / State of the Art
    • Sentiment politic analysis
    • Similarity of the topic of two texts
    • General social media and news analytics
  • (Characteristics of Social Media)
    • Biases
    • Geography and demography
• Data Understanding
  • 💾 Evaluation and Collection of Media Sources
    • Quality, neutrality, complexity of data collection, and target group (check with papers and complete list accordingly)
    • Check for existing datasets
    • Search for scientific method (e.g., matrix evaluation)
  • Choosing data platform
    • See Business Understanding #18
  • (Ensuring proper relation between news and social media reactions)
• Data Preparation
  • ⏳ We already can ensure that the data not only needs to be cleaned but also needs to be filtered
    • Search for preparation pipelines and standards (e.g., lower case can be processed more effective)
  • Define minimum requirements for both news and social media data
    • Check for metadata
    • Decide on language (e.g., en or de)
• Modeling
  • Check for NLP including large language models (e.g., SBERT, GPT-2)
  • Classification model to categorize based on different parties
• Evaluation
  • Statistical evaluation (significants of results)
    • Search for scientific method (e.g., matrix evaluation)
• Deployment
  • See Business Understanding #18
  • Visual UI similar to applications like streamlit or jupyter notebooks