jusText

Program jusText is a tool for removing boilerplate content, such as navigation links, headers, and footers from HTML pages. It is designed to preserve mainly text containing full sentences and it is therefore well suited for creating linguistic resources such as Web corpora. You can try it online.

This is a fork of original (currently unmaintained) code of jusText hosted on Google Code.

Adaptations of the algorithm to other languages:

• C++
• Go
• Java

Some libraries using jusText:

• chirp
• lazynlp
• off-topic-memento-toolkit
• pears
• readability calculator
• sky

Some currently (Jan 2020) maintained alternatives:

• dragnet
• html2text
• inscriptis
• newspaper
• python-readability
• trafilatura

Installation

Make sure you have Python 2.7+/3.4+ and pip (Windows, Linux) installed. Run simply:

$ [sudo] pip install justext

Dependencies

lxml (version depends on your Python version)

Usage

$ python -m justext -s Czech -o text.txt http://www.zdrojak.cz/clanky/automaticke-zabezpeceni/
$ python -m justext -s English -o plain_text.txt english_page.html
$ python -m justext --help # for more info

Python API

import requests
import justext

response = requests.get("http://planet.python.org/")
paragraphs = justext.justext(response.content, justext.get_stoplist("English"))
for paragraph in paragraphs:
  if not paragraph.is_boilerplate:
    print paragraph.text

Testing

Run tests via

$ py.test-2.7 && py.test-3.4 && py.test-3.5 && py.test-3.6 && py.test-3.7 && py.test-3.8

Acknowledgements

This software has been developed at the Natural Language Processing Centre of Masaryk University in Brno with a financial support from PRESEMT and Lexical Computing Ltd. It also relates to PhD research of Jan Pomikálek.