Bug/prefix wildcard with fullpath

[tuxnco]

It appears that the escape_url helper's call to Url::fullpath inserts forcibly a leading slash / to directives, which prevents a rule such as Disallow: */bla to work as expected.

In this PR, we simply don't call the fullpath method to circumvent the issue, but this could have been an other way (by patching the fullpath method, for example)

[tuxnco]

Fix #34

[dlecocq]

Seems reasonable to me. If memory serves, directives are supposed to start with / in general (aside from this wildcard case), so I think this should be fine. If we wanted to keep the behavior as much as possible, we could test whether it starts with * and if not, use the existing code. I'm not sure how much that comes up in the wild, though.

I'm not sure who at Moz owns this at this point. @lindseyreno - do you know?

[lindseyreno]

I'm pretty sure Pro Services owns it since we own Reppy. We actually have a ticket in our backlog for this issue.

[tammybailey]

@dlecocq I've been looking at the issue and this seems to be the cleanest fix (at least without changing url-cpp). I haven't yet found a test case that would fail if this change was made, which is my only concern.

If we wanted to treat the leading wildcard as a special case with the existing code, we would need to add add a second directive in the Agent::Allow and Agent::Disallow functions which is the result of evaluating the input query stripped of leading '*'s. That is, we would evaluate Disallow: */test as Disallow: /test and Disallow: /*/test.

You know the code much better than I - which approach do you think is best?

[dlecocq]

@tammybailey - oh, it's been a minute since I really was digging into all this code, but I agree that this seems cleanest.

[tammybailey]

@dlecocq thanks!