Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IndexError when extracting optional items from a repeat #66

Open
jpsnyder opened this issue Mar 19, 2021 · 0 comments
Open

IndexError when extracting optional items from a repeat #66

jpsnyder opened this issue Mar 19, 2021 · 0 comments

Comments

@jpsnyder
Copy link

First of all. Thanks for making this excellent library. I find it much more approachable, and yet more customizable and elegant than the other libraries on the market. It's a huge improvement over the PLY library. I hope to see this project grow in the future.

I'm having a small issue where I run into an IndexError when accessing an item within an optional which is within a repeat.
Please see the following simplified example.

import sly


class Lexer(sly.Lexer):
    tokens = {A, B}
    literals = {"[", "]", ","}

    A = "a"
    B = "b"


class Parser(sly.Parser):
    tokens = Lexer.tokens

    @_("'[' [ expr ] { ',' [ expr ] } ']'")
    def array(self, p):
        items = [p.expr0, *p.expr1]
        # Each None accounts for an Elision
        items = [item or ("elision", None) for item in items]
        return "array", items

    @_("A", "B")
    def expr(self, p):
        return "expr", p[0]


lexer = Lexer()
parser = Parser()
code = "[a,,a,b,,]"

tokens = list(lexer.tokenize(code))  # pulling full tokens, just so we can look at them.
print("\n".join(map(str, tokens)))
tree = parser.parse(iter(tokens))
print(repr(tree))
Token(type='[', value='[', lineno=1, index=0)
Token(type='A', value='a', lineno=1, index=1)
Token(type=',', value=',', lineno=1, index=2)
Token(type=',', value=',', lineno=1, index=3)
Token(type='A', value='a', lineno=1, index=4)
Token(type=',', value=',', lineno=1, index=5)
Token(type='B', value='b', lineno=1, index=6)
Token(type=',', value=',', lineno=1, index=7)
Token(type=',', value=',', lineno=1, index=8)
Token(type=']', value=']', lineno=1, index=9)
Traceback (most recent call last):
  File "/home/jon/.config/JetBrains/PyCharmCE2020.3/scratches/scratch_36.py", line 40, in <module>
    tree = parser.parse(iter(tokens))
  File "/home/jon/.virtualenvs/test/lib/python3.8/site-packages/sly/yacc.py", line 2082, in parse
    value = p.func(self, pslice)
  File "/home/jon/.config/JetBrains/PyCharmCE2020.3/scratches/scratch_36.py", line 18, in array
    items = [p.expr0, *p.expr1]
  File "/home/jon/.virtualenvs/test/lib/python3.8/site-packages/sly/yacc.py", line 148, in __getattr__
    return self._namemap[name](self._slice)
  File "/home/jon/.virtualenvs/test/lib/python3.8/site-packages/sly/yacc.py", line 239, in <lambda>
    namemap[k] = lambda s,i=index,n=n: ([x[n] for x in s[i].value]) if isinstance(s[i].value, list) else s[i].value[n]
  File "/home/jon/.virtualenvs/test/lib/python3.8/site-packages/sly/yacc.py", line 239, in <listcomp>
    namemap[k] = lambda s,i=index,n=n: ([x[n] for x in s[i].value]) if isinstance(s[i].value, list) else s[i].value[n]
IndexError: tuple index out of range

Process finished with exit code 1

With some debugging, I discovered this is occuring when I try to access p.expr1.
I can access it with indexing, but it seems like it is due to the second expr term being incorrectly wrapped in a tuple? Below is the workaround I implemented, but I'm curious on your take on this.

    @_("'[' [ expr ] { ',' [ expr ] } ']'")
    def array(self, p):
        items = [p.expr0]
        for item in p[2]:
            comma, expr_tuple = item
            # expr_tuple is a single element tuple for some reason
            items.append(expr_tuple[0])
        items = [item or ("elision", None) for item in items]
        return "array", items
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant