-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Cleanup, refactorings, re-generated CLDF, added tests.
- Loading branch information
Showing
12 changed files
with
2,855 additions
and
2,756 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| ## Specification of form manipulation | ||
|
|
||
|
|
||
| Specification of the value-to-form processing in Lexibank datasets: | ||
|
|
||
| The value-to-form processing is divided into two steps, implemented as methods: | ||
| - `FormSpec.split`: Splits a string into individual form chunks. | ||
| - `FormSpec.clean`: Normalizes a form chunk. | ||
|
|
||
| These methods use the attributes of a `FormSpec` instance to configure their behaviour. | ||
|
|
||
| - `brackets`: `{'(': ')'}` | ||
| Pairs of strings that should be recognized as brackets, specified as `dict` mapping opening string to closing string | ||
| - `separators`: `(';', '/', ',')` | ||
| Iterable of single character tokens that should be recognized as word separator | ||
| - `missing_data`: `('?', '-')` | ||
| Iterable of strings that are used to mark missing data | ||
| - `strip_inside_brackets`: `True` | ||
| Flag signaling whether to strip content in brackets (**and** strip leading and trailing whitespace) | ||
| - `replacements`: `[]` | ||
| List of pairs (`source`, `target`) used to replace occurrences of `source` in formswith `target` (before stripping content in brackets) | ||
| - `first_form_only`: `False` | ||
| Flag signaling whether at most one form should be returned from `split` - effectively ignoring any spelling variants, etc. | ||
| - `normalize_whitespace`: `True` | ||
| Flag signaling whether to normalize whitespace - stripping leading and trailing whitespace and collapsing multi-character whitespace to single spaces | ||
| - `normalize_unicode`: `None` | ||
| UNICODE normalization form to use for input of `split` (`None`, 'NFD' or 'NFC') |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,9 +1,9 @@ | ||
| ID,Name,Glottocode,Glottolog_Name,ISO639P3code,Macroarea,Latitude,Longitude,Family,ChineseName,SubGroup | ||
| Duhumbi,Duhumbi,chug1252,,,,27.42,92.23,,,Western Kho-Bwa | ||
| Jerigaon,Jerigaon,jeri1243,,,,27.340629,92.486595,,,Western Kho-Bwa | ||
| Khispi,Khispi,lish1235,,,,27.37,92.23,,,Western Kho-Bwa | ||
| Khoina,Khoina,khoi1253,,,,27.334981,92.52994,,,Western Kho-Bwa | ||
| Khoitam,Khoitam,khoi1252,,,,27.327487,92.439455,,,Western Kho-Bwa | ||
| Rahung,Rahung,rahu1234,,,,27.310778,92.395028,,,Western Kho-Bwa | ||
| Rupa,Rupa,rupa1234,,,,27.203065,92.398757,,,Western Kho-Bwa | ||
| Shergaon,Shergaon,sher1261,,,,27.105018,92.272133,,,Western Kho-Bwa | ||
| Duhumbi,Duhumbi,chug1252,Chug,cvg,Eurasia,27.42,92.23,Sino-Tibetan,,Western Kho-Bwa | ||
| Jerigaon,Jerigaon,jeri1243,Jerigaon,,,27.340629,92.486595,Sino-Tibetan,,Western Kho-Bwa | ||
| Khispi,Khispi,lish1235,Lish,lsh,Eurasia,27.37,92.23,Sino-Tibetan,,Western Kho-Bwa | ||
| Khoina,Khoina,khoi1253,Khoina,,,27.334981,92.52994,Sino-Tibetan,,Western Kho-Bwa | ||
| Khoitam,Khoitam,khoi1252,Khoitam,,,27.327487,92.439455,Sino-Tibetan,,Western Kho-Bwa | ||
| Rahung,Rahung,rahu1234,Rahung,,,27.310778,92.395028,Sino-Tibetan,,Western Kho-Bwa | ||
| Rupa,Rupa,rupa1234,Rupa,,,27.203065,92.398757,Sino-Tibetan,,Western Kho-Bwa | ||
| Shergaon,Shergaon,sher1261,Shergaon,,,27.105018,92.272133,Sino-Tibetan,,Western Kho-Bwa |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,49 +1,45 @@ | ||
| appdirs==1.4.3 | ||
| atomicwrites==1.2.1 | ||
| certifi==2018.8.24 | ||
| atomicwrites==1.3.0 | ||
| certifi==2019.9.11 | ||
| chardet==3.0.4 | ||
| -e git+https://github.com/cldf/cldfbench.git@f373855e3b9cde029578e77c26136f0df26a82fa#egg=cldfbench | ||
| cldfbench==1.0.0 | ||
| cldfcatalog==1.3.0 | ||
| colorlog==3.1.4 | ||
| configparser==3.5.0 | ||
| -e git+https://github.com/cldf/csvw.git@bcd398856fdfe6408567cc02c7ff8b67ba1c8e38#egg=csvw | ||
| decorator==4.3.0 | ||
| defusedxml==0.5.0 | ||
| idna==2.7 | ||
| clldutils==3.3.0 | ||
| colorlog==4.0.2 | ||
| csvw==1.6.0 | ||
| decorator==4.4.1 | ||
| idna==2.8 | ||
| isodate==0.6.0 | ||
| jdcal==1.4.1 | ||
| -e git+https://github.com/lingpy/bodtkhobwa/@e80544c6a001e3ce30ef197463be03834e5a3704#egg=lexibank_bodtkhobwa | ||
| -e git+https://github.com/lexibank/castrosui.git@a2d13314e81682f6619c6af90a6ecf4bff401d00#egg=lexibank_castrosui | ||
| -e git+https://github.com/lexibank/lieberherrkhobwa.git@1985e1f912b2454cfcaa45611b488c95e31ff04c#egg=lexibank_lieberherrkhobwa | ||
| -e git+https://github.com/digling/sinotibetan-data.git@9488b83bc5cb4d443bf9f3b81c20711a95b772c6#egg=lexibank_sagartst | ||
| -e git+https://github.com/LinguList/lingpy.git@2a1671c1b65886e1d33eccd74818b29bc4ce73dd#egg=lingpy | ||
| lxml==4.2.4 | ||
| Markdown==2.6.11 | ||
| networkx==2.2 | ||
| newick==0.9.2 | ||
| numpy==1.15.1 | ||
| openpyxl==2.6.4 | ||
| pathlib2==2.3.2 | ||
| pluggy==0.8.0 | ||
| lingpy==2.6.5 | ||
| Markdown==3.1.1 | ||
| networkx==2.1 | ||
| newick==1.0.0 | ||
| numpy==1.17.4 | ||
| openpyxl==3.0.0 | ||
| packaging==19.2 | ||
| pluggy==0.13.0 | ||
| purl==1.5 | ||
| py==1.7.0 | ||
| py==1.8.0 | ||
| pybtex==0.22.2 | ||
| -e git+https://github.com/cldf-clts/pyclts@4842f1fd9613de6ef20a917dbc3bd723e8d0ffbb#egg=pyclts | ||
| -e git+https://github.com/concepticon/pyconcepticon.git@615a048b11cc6e5f8a3fe92619ad7790b23154db#egg=pyconcepticon | ||
| pycountry==18.12.8 | ||
| -e git+https://github.com/clld/pyglottolog.git@0f24f24a46d1f510c975337e4c0d8c23b357c8bd#egg=pyglottolog | ||
| -e git+https://github.com/lexibank/pylexibank/@b2b24aafcbe618f08ef994d8e0ce2d29f601bff1#egg=pylexibank | ||
| pytest==4.0.0 | ||
| regex==2018.8.29 | ||
| requests==2.20.0 | ||
| rfc3986==1.1.0 | ||
| scipy==1.2.0 | ||
| six==1.11.0 | ||
| SQLAlchemy==1.3.0 | ||
| tabulate==0.8.2 | ||
| pycldf==1.8.2 | ||
| pyclts==2.0.0 | ||
| pyconcepticon==2.5.1 | ||
| pycountry==19.8.18 | ||
| pyglottolog==2.2.1 | ||
| pylexibank==2.1.0 | ||
| pytest==5.2.2 | ||
| regex==2019.11.1 | ||
| requests==2.22.0 | ||
| rfc3986==1.3.2 | ||
| segments==2.1.2 | ||
| six==1.13.0 | ||
| SQLAlchemy==1.3.10 | ||
| tabulate==0.8.5 | ||
| termcolor==1.1.0 | ||
| tqdm==4.25.0 | ||
| tqdm==4.38.0 | ||
| uritemplate==3.0.0 | ||
| urllib3==1.24.2 | ||
| urllib3==1.25.6 | ||
| wcwidth==0.1.7 | ||
| xlrd==1.1.0 | ||
| xlrd==1.2.0 | ||
| zipp==0.6.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.