[Suggestion] filter non-html page from collector

[ghost]

Hi,

Hope you are all well !

It would be interesting to exclude non-html page from indexing content into elasticsearch, or to create a mime/type detector to detect images or pdf documents and create dedicated sub-processing for the binary types.

For now, I just added:

		if title == "" {
			spider.Logger.Error(errors.New("not an html page"))
			return
		}

Cheers,
X