Skip to content

Commit

Permalink
Add ScraperClient class for web scraping functionality
Browse files Browse the repository at this point in the history
  • Loading branch information
transitive-bullshit committed Feb 25, 2024
1 parent 458db5b commit d8c6f0f
Showing 1 changed file with 8 additions and 0 deletions.
8 changes: 8 additions & 0 deletions src/services/scraper-client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,14 @@ export type ScrapeResult = {
title: string
}

/**
* This is a single endpoint API for scraping websites. It returns the HTML,
* markdown, and plaintext for main body content of the page, as well as
* metadata like title and description.
*
* It tries the simplest and fastest methods first, and falls back to slower
* proxies and JavaScript rendering if needed.
*/
export class ScraperClient {
readonly apiBaseUrl: string
readonly ky: KyInstance
Expand Down

0 comments on commit d8c6f0f

Please sign in to comment.