Comments at the end of file don't work

[tusharsadhwani]

The current tokenizer has a flaw that it crashes if a file ends with a comment.

[Agent-Hellboy]

Hi @tusharsadhwani
It crashes when a newline is found after a comment, I am not able to figure out why it is not failing into this loop of skipping tokens.

interpreted/src/interpreted/tokenizer.py

Line 195 in ad57787

self.skip_token()

interpreted/src/interpreted/tokenizer.py

Line 111 in ad57787

elif char in "\f\v\t\r\n ":

should we need an EOL token and later a branch to parse that?

interpreted/src/interpreted/parser.py

Line 530 in ad57787

def parse_literal(self) -> Expression:

By the way, Is there a write-up about how this works? I will try to come up with a write-up and you can review