Skip to content

MichaPau/wiktionary_to_sqlite

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 

Repository files navigation

Wikitionary jsonl dump to sqlite:

(https://kaikki.org/dictionary/English/kaikki.org-dictionary-English.jsonl)

Narrow subset -> 2 main tables + 2 helper tables (words, part-of-speech, lang) + (word_forms for word, tag)

lines in source jsonl : 1306029

parse time: 2.8119661000000002 secs. db insert time: 8.8602965 secs.

main time: 10.4464539 secs. (2 seperate threads for parsing and db commits) 5000 -> bulk size for transactions

db inserts (words + word_forms): 1276286 + 965908

About

parse wikitionary dump files

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages