The specific issue is that -w causes the regex to be wrapped in Unicode
word boundaries. Regrettably, Unicode word boundaries are the one thing
our regex engine can't handle well in the presence of non-ASCII text. We
work around its slowness by stripping word boundaries in some
circumstances, and using the resulting expression as a way to produce match
candidates that are then verified by the full original regex.
This doesn't fix all cases, but it should fix all cases where -w is used.
In particular, if we had an inner literal and were doing a case insensitive
search, then the literals are dropped because we previously only allowed
a single inner literal to have an effect. Now we allow alternations of
inner literals, but still don't quite take full advantage.
We really need functionality like this when memory maps aren't suitable,
either because they're too slow or because they just aren't available (like
for reading stdin). However, this particular approach was completely bunk.
Namely, the interface was all wrong. The caller needs to maintain some kind
of control over the search buffers for special output features (like
contexts or inverted matching), but this interface as written doesn't
support that kind of pattern at all.
So... back to the drawing board.
