UASTv2 has no tokens for JSX Nodes #58
Comments
|
Also, I am not sure if we can count JSX as the same language like JS. Can I kindly ask you do give some understanding why is it so? |
|
If you want to reproduce this trees: Python Code# my bblfsh is v2.12.6
import bblfsh
from bblfsh import BblfshClient, role_id, role_name
def node2raw(node, internal_type=True):
token = "|%s|" % node.token[:20].replace("\n", "\\n")
roles = ", ".join([role_name(k) for k in node.roles])
internal_role = node.internal_type
positions = str(node.start_position.line) if node.start_position.line != 0 else ""
return [positions, token, internal_role, roles] if internal_type else [positions, token, roles]
def uast_roles(uast) -> iter:
stack = [(0, uast)]
while stack:
level, node = stack.pop()
yield level, node
stack.extend((level+1, c) for c in node.children[::-1])
def rreplace(s, old, new, times):
li = s.rsplit(old, times)
return new.join(li)
def print_table(table):
col_size = [max(len(cell) for cell in table_col) for table_col in zip(*table)]
for raw in table:
print(" ".join(cell.ljust(size) for cell, size in zip(raw, col_size)))
def print_uast_structure(uast, internal_type=True):
if internal_type:
table = [["#", "Token", "Internal Role", "Roles Tree"], ["", "", "", ""]]
else:
table = [["#", "Token", "Roles Tree"], ["", "", ""]]
old_level = 0
for level, node in uast_roles(uast):
raw = node2raw(node, internal_type)
if level != 0:
raw[-1] = "┃ " * (level-1) + "┣ " + raw[-1]
if level < old_level:
table[-1][-1] = rreplace(table[-1][-1].replace("┣", "┗"), "┃", "┗", old_level-level-1)
pass
table.append(raw)
old_level = level
table[-1][-1] = table[-1][-1].replace("┣", "┗").replace("┃", "┗")
print_table(table)
def print_nodes(nodes):
table = [["#", "Token", "Roles"]]
for node in nodes:
table.append(node2raw(node, False))
print_table(table)
code = """
class TestIdTestApp extends React.Component {
render() {
return (
<View>
<Text testID="Text">text</Text>
</View>
);
}
}
""".strip().encode()
bc = bblfsh.BblfshClient("localhost:9432")
response = bc.parse("", language="JavaScript", contents=code)
print_uast_structure(response.uast) |
|
@zurk the second example you gave uses bblfsh protocol.v2 but it is in "annotated" mode, or equivalent of Full UASTv2, which is a "semantic" mode, is only supported by bblfsh/python-client#128 So here is an output of your script with 1 line modified to play better in this mode: The only change is token = "|%s|" % (node.token[:20].replace("\n", "\\n") if node.token else node.properties["name"])and, as you can see, now it includes the text of all From my experience, |
|
To your second question
Long story short: it depends on the convention :) In So that way, on bblfsh side a single driver now does support multiple languages which makes sense (e.g as there is no such language as JavaScript - there are many ECMAScript language standards that all are parsed by the same tool that you mention If you are interested in sneak-peeking into the future - there is a proposal in RFC stage on project's mailing list for language aliases support that covers exactly this problem. Let me know if that answers your questions! |
|
Answering to #58 (comment)
If you are worried about style-analyzer I think we will not use the annotated layer. We have an implementation for code tokenizer based on UAST tokens. So, we actually don't need just identifiers but all tokens that UAST have. About JSX #58 (comment) |
Would you be so kind to also check that it in the CLI following the instructions that I posted above and let me know if you think something is missing in there?
I belive lookout does not add anything to that and is just using Right now |
|
For me this issue is solved, thanks @bzz! Feel free to close it. |
JSX is a syntax extension to JavaScript. It can be converted to a proper JS with babel tool.
The bug I found is that UASTv1 provides a lot of tokens while UASTv2 don't
Here is the data to reproduce:
The JSX code
UASTv1
UASTv2
Should be easy to fix because all positional information is saved.
drivers are v1.2.0 and v.2.6.0
The text was updated successfully, but these errors were encountered: