v1.6.15 | Crawlit Engine - Dofus & Dofus-Touch encyclopedia data parser | API builder

Changelog

Added

• Parsing limitations bypassed because of new parsing method
• You can now parse with command line and pass arguments/options to the command (categories, maxItem...)
• Structure of Crawlit has completely changed to a more modular one.
• Crawlit provide you MongoDb file ready to be imported in your DB.
• Full parse with 1 command : you can now parse the entire encyclopedia with 1 command
• Full english version of the encyclopedia can be parsed (for Dofus & Dofus-Touch)

Changed

• Categories scrappers are now based on "Models" (for structure definition).
• Structure of categories json changed to be more flexible.
• Crawlit is now based on "request-promise-native" instead of "request-promise" because of ES6 support.
• Error management changed : we have now a module "ErrorHandler" to centralize error msg and code.

Deleted

• No more possibility to package the app with "pkg" module.

v1.3.1 | Crawlit - Dofus & Dofus-Touch encyclopedia scraper | API builder

Changelog

Added

• No more items of a set in the array, only key (id) of items are added.

Changed

• We use from now a new way to parse items : we have created helpers to alleviate the code from each category scraped.
• Many bugs corrected : misplaced conditions, conditions with "or", confusion between "dammage (boost)" and "dammage (attack)...
• Title and emotes are correctly displayed.
• Exchangeable date added properly.

v1.2.1 | Crawlit - Dofus & Dofus-Touch encyclopedia scraper | API builder

Changelog

Added

• Since now, you can ask to scrap all items of a main category / subcategory at once from the CLI.

Changed

• Files of items are now stored in respective game's folder ("data/dofus/" or "data/dofus-touch").
• Links files are removed after the scraping.

Removed

• Auto resume feature temporarily removed because of unhandled behavior of the app.

v1.1.0 | Crawlit - Dofus & Dofus-Touch encyclopedia scraper | API builder

Changelog

Added

• Resume parsing feature.
• Error 429, ETIMEDOUT, ECONNRESET and ECONNABORTED are handled, the app proposed to resume parsing after few minutes.
• Ressources, and Consumables are supported.

Changed

• New parsing method based on item's number and no more on pages.
• Recipes added on Equipment, Weapon, Ressource and Consumable.

Removed

• Parsing method based on pages was removed.

v1.0.1 | Crawlit - Dofus & Dofus-Touch encyclopedia scraper | API builder

The tool is compatible with windows, linux & macos. Just launch the executable and enjoy.
This version RELEASE support Dofus-Touch scraping & Allow you to scrap English or French Dofus Encyclopedia.

Correction : I also correct on this version an error due to a wrong link (http instead of https).

⚠️ Be aware : use the tool for only 24 pages/hour to avoid an IP blocked by Ankama aka Dofus.

v1.0.0 | Crawlit - Dofus & Dofus-Touch encyclopedia scraper | API builder

The tool is compatible with windows, linux & macos. Just launch the executable and enjoy.
This version RELEASE support Dofus-Touch scraping & Allow you to scrap English or French Dofus Encyclopedia.

⚠️ Be aware : use the tool for only 24 pages/hour to avoid an IP blocked by Ankama aka Dofus.