AI Web Scrapper

Uses following

1. Ollama LLM, for calling the AI to parse the content.
2. Selenium, for the webscraping.
3. Streamlit, to give web interface
4. BrightData, for unblocking websites with Captcha. Its not free but lets you have free trial.
5. BeatifulSoup
6. ChromeDriver, if captcha is not involved, then local version of ChromeDriver can be used instead of BrightData.

In order to run the application, execute following in the terminal - streamlit run main.py