Options instead of word boundary

[yonderblue]

Here is a small runnable example. Changing the OR pattern to (?i)or\b solves the issue, but is another form of lookahead that relies on the definition of \w. What if I wanted - to also be considered in \w? Is there a difference approach?

package main

import (
	"fmt"

	"github.com/alecthomas/participle/v2"
	"github.com/alecthomas/participle/v2/lexer"
)

func main() {
	s := "mars or earth Orbit"

	dumpLex(lexDef, s)

	p, err := parser.ParseString("", s)
	if err != nil {
		panic(err)
	}
	fmt.Printf("PARSE %+v\n", p)
}

type parse struct {
	Texts []text `@@*`
}

type text struct {
	Or    bool     `( @Or`
	Texts []string `| @Plain+ )`
}

var (
	lexDef = lexer.MustSimple([]lexer.SimpleRule{
		{"Or", `(?i)or`},
		{"Plain", `\S+`},
		{"whitespace", `\s+`},
	})
	parser = participle.MustBuild[parse](participle.Lexer(lexDef))
)

func dumpLex(def *lexer.StatefulDefinition, s string) {
	fmt.Println("STRING", s)
	fmt.Println("SYMBOLS", def.Symbols())

	lex, err := def.LexString("", s)
	if err != nil {
		panic(err)
	}

	tokens, err := lexer.ConsumeAll(lex)
	if err != nil {
		panic(err)
	}

	for _, t := range tokens {
		fmt.Printf("TOKEN type=%v value=%v\n", t.Type, t.Value)
	}
}

[yonderblue]

@alecthomas What I'd really like to do is match at a unicode word boundary, but I am not aware of anyway to do that without lookahead. Considering the SQL example in here uses word boundary in the lexer, I can't be too far off in my quest? :D Maybe a pluggable regex impl feature?

[alecthomas]

The lexer uses Go's regex engine, so it supports everything it does. If you need something beyond that you can write your own lexer that implements the lexer interface.