go-microdata-extract

A Go package for extracting structured data from HTML.

Formats supported

For currently supported formats, see Statistics

Statistics

Usage statistics of structured data formats for websites

(from https://w3techs.com/technologies/overview/structured_data, 2024-10-31)

Format	Usage	Supported
None	23.7%
OpenGraph	67.2%	✔
X Cards	51.9%	✔
JSON-LD	49.3%	✔
RDFa	39.3%	-
Microdata	24.2%	✔
Dublin Core	0.9%	-
Microformats	0.4%	-

Installation

go get github.com/aafeher/go-microdata-extract

import "github.com/aafeher/go-microdata-extract"

Usage

Create instance

To create a new instance with default settings, you can simply call the New() function.

e := extract.New()

Configuration defaults

• syntaxes: []Syntax{extract.SyntaxOpenGraph, extract.SyntaxXCards, extract.SyntaxJSONLD, extract.SyntaxMicrodata}
• userAgent: "go-microdata-extract (+https://github.com/aafeher/go-microdata-extract/blob/main/README.md)"
• fetchTimeout: 3 seconds

Overwrite defaults

Syntaxes

To set the syntaxes whose results you want to retrieve after processing, use the SetSyntaxes() function.

e := extract.New()
e = e.SetSyntaxes([]Syntax{extract.SyntaxOpenGraph, extract.SyntaxJSONLD})

... or ...

e := extract.New().SetSyntaxes([]Syntax{extract.SyntaxOpenGraph, extract.SyntaxJSONLD})

User Agent

To set the user agent, use the SetUserAgent() function.

e := extract.New()
e = e.SetUserAgent("YourUserAgent")

... or ...

e := extract.New().SetUserAgent("YourUserAgent")

Fetch timeout

To set the fetch timeout, use the SetFetchTimeout() function. It should be specified in seconds as an uint8 value.

e := extract.New()
e = e.SetFetchTimeout(10)

... or ...

e := extract.New().SetFetchTimeout(10)

Chaining methods

In both cases, the functions return a pointer to the main object of the package, allowing you to chain these setting methods in a fluent interface style:

e := extract.New()
     .SetSyntaxes([]Syntax{extract.SyntaxOpenGraph, extract.SyntaxJSONLD})
     .SetUserAgent("YourUserAgent")
     .SetFetchTimeout(10)

Extract

Once you have properly initialized and configured your instance, you can extract structured data using the Extract() function.

The Extract() function takes in two parameters:

• url: the URL of the webpage,
• urlContent: an optional string pointer for the content of the URL

If you wish to provide the content yourself, pass the content as the second parameter. If not, simply pass nil and the function will fetch the content on its own. The Extract() function performs concurrent extracting and fetching optimized by the use of Go's goroutines and sync package, ensuring efficient structured data handling.

e, err := e.Extract("https://github.com/aafeher/go-microdata-extract", nil)

In this example, structured data is extracted from "https://github.com/aafeher/go-microdata-extract". The function fetches the content itself, as we passed nil as the urlContent.

Examples

Examples can be found in /examples.