[idea] Generic code tokenizer

[EgorBu]

Feature extraction for source code heavily relies on tokenization of source code and structure information in many tasks. If we want to use suggestion feature at GitHub we must use tokenized code.
This part is very important for everybody in MLonCode area and still it's quite complicated to do.
Proposal - extend bblfsh client or make new module that could be used by many different projects.

TLDR: information required by feature extractor

• tokenization: code -> nodes -> "".join(nodes) == code
• parent information Add parent field into UAST Node in clients. bblfsh/sdk#252 - we use this info, these people use this info too https://github.com/tech-srl/ - this feature may increase usage of bblfsh by researchers in this area
• reserved keywords
• roles that are used in particular language

And this module could be used by many researchers in this area. Related issue bblfsh/bblfshd#231

[zurk]

Right now we have tokenizer for JS in style-analyzer:
https://github.com/src-d/style-analyzer/blob/a0eaafd5b371433e3c2e3dc9d113710814912f99/lookout/style/format/feature_extractor.py#L655-L656

If we start this project, this code should be considered as an entry point.

[m09]

Should this be transferred to src-d/feature-idea? @vmarkovtsev @EgorBu

[EgorBu]

Good idea, @m09

[m09]

Somehow I cannot transfer it, GitHub does not find the feature-idea repo 😕

Edit: it seems we need someone admin in both ml-backlog and feature-idea to transfer the issue.

[m09]

Calling @smola to the rescue to transfer the issue to feature-idea :)