build is failing in 1.8 because a Unicode feature isn't enabled, but previously worked in 1.7

[DevNulPavel]

Hello

When i updated regex version from 1.7.3 up to 1.8.0 i've received a problem with regex parsing.

Output:

(\d+\s?(years|year|y))?\s?(\d+\s?(months|month|m))?\s?(\d+\s?(weeks|week|w))?\s?(\d+\s?(days|day|d))?\s?(\d+\s?(hours|hour|h))?
       ^^
error: Unicode-aware Perl class not found (make sure the unicode-perl feature is enabled)

Even if i enable unicode feature manually it still does not work:

regex = {version = "1.8", features = ["unicode"]}

But if i downgrade version to 1.7.3, everything works good:

regex = "~1.7.3"

[BurntSushi]

Could you please include a complete reproduction here? I cannot reproduce it:

use regex::Regex;

fn main() {
    let pat = r"(\d+\s?(years|year|y))?\s?(\d+\s?(months|month|m))?\s?(\d+\s?(weeks|week|w))?\s?(\d+\s?(days|day|d))?\s?(\d+\s?(hours|hour|h))?";
    let result = Regex::new(pat);
    assert!(result.is_ok());
}

And my Cargo.toml:

[package]
name = "i982"
version = "0.1.0"
edition = "2021"

[dependencies]
regex = "1.8.0"

[[bin]]
name = "i982"
path = "main.rs"

And the result:

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.00s
     Running `target/debug/i982`
$

Trying to work backwards from the error, the message you're getting is here:

regex/regex-syntax/src/hir/mod.rs

Lines 125 to 128 in 93316a3

	UnicodePerlClassNotFound => {
	"Unicode-aware Perl class not found \
	(make sure the unicode-perl feature is enabled)"
	}

Which corresponds to this error kind variant:

regex/regex-syntax/src/hir/mod.rs

Lines 97 to 100 in 93316a3

	/// This occurs when a Unicode-aware Perl character class (`\w`, `\s` or
	/// `\d`) could not be found. This can occur when the `unicode-perl`
	/// crate feature is not enabled.
	UnicodePerlClassNotFound,

This error variant is constructed in precisely one place in non-test code:

regex/regex-syntax/src/hir/translate.rs

Lines 1071 to 1073 in 93316a3

	unicode::Error::PerlClassNotFound => {
	self.error(sp, ErrorKind::UnicodePerlClassNotFound)
	}

Which occurs only when an internal Unicode error occurs indicating that a Perl class could not be found:

regex/regex-syntax/src/unicode.rs

Line 22 in 93316a3

PerlClassNotFound,

That variant in turn is constructed in three places. Here:

regex/regex-syntax/src/unicode.rs

Lines 389 to 392 in 93316a3

	#[cfg(not(feature = "unicode-perl"))]
	fn imp() -> Result<hir::ClassUnicode, Error> {
	Err(Error::PerlClassNotFound)
	}

here:

regex/regex-syntax/src/unicode.rs

Lines 407 to 410 in 93316a3

	#[cfg(not(any(feature = "unicode-perl", feature = "unicode-bool")))]
	fn imp() -> Result<hir::ClassUnicode, Error> {
	Err(Error::PerlClassNotFound)
	}

and here:

regex/regex-syntax/src/unicode.rs

Lines 431 to 434 in 93316a3

	#[cfg(not(any(feature = "unicode-perl", feature = "unicode-gencat")))]
	fn imp() -> Result<hir::ClassUnicode, Error> {
	Err(Error::PerlClassNotFound)
	}

You use \d and \s, but not \w, so it can't be the first. This must mean that, somehow, one of unicode-perl, unicode-bool or unicode-gencat is not present when building regex-syntax. But the default is to enable those features.

So I can't quite figure out what's going on here without more details. Please do try to include a complete reproduction here. Thanks.

[BratSinot]

tracing-subscriber have regex in deps and after upgrades to 1.8 have same error:

Downgrade to 1.7 work fine.

tokio-rs/tracing#2565

[BurntSushi]

Folks, I need a reproduction. Please find a way to hand me a set of commands and source code that I can enter and run. As I mentioned above, I still don't understand how this error is appearing.

[BurntSushi]

@BratSinot And the error you're showing is different from the error in the OP. It's not the same.

[BurntSushi]

I have a guess as to what is happening. This PR tipped me off to it: tokio-rs/tracing#2566

Unfortunately, this is one of those cases where folks were able to under specify regex crate features and still have their patterns work even though the features specified weren't actually sufficient to do so. The way this manifests is likely going to be a little different in each case, but taking tracing-subscriber as one example... I ran cargo tree on it and noticed that it depends on matchers and that in turn depends on regex-automata 0.1. That in turn depends on regex-syntax 0.6. And it just brings in all of regex-syntax's Unicode features unconditionally.

Now tracing-subscriber also depends on regex directly, and note that it disables all features except for std: https://github.com/tokio-rs/tracing/blob/8aae1c37b091963aafdd336b1168fe5a24c0b4f0/tracing-subscriber/Cargo.toml#L72

But because of how feature unification works in Cargo, this means that even though the regex dependency on tracing-subscriber doesn't ask for anything more than std, in practice, it ends up pulling in everything because of the transitive dependency on regex-syntax with its default features enabled. This in turn permitted tracing-subscriber to use a regex with Unicode case insensitive matching enabled, even though the regex dependency did not enable the required features to do so. It worked because the required features were enabled implicitly via a transitive dependency.

With the release of regex-syntax 0.7, the feature unification no longer applies because regex-syntax 0.6 and regex-syntax 0.7 are semver incompatible. As a result, regex 1.8 now really is only using std in tracing-subscriber, and that in turn leads to a regex compilation error because the required features are not enabled. In other words, the dependency specification in tracing-subscriber is incorrect but "happened" to work.

So... probably everything is actually working correctly here and folks are just now getting hit over the head with the fact that they under-specified the regex crate features.

If you think this analysis is wrong or have a case where this doesn't apply, please let me know. And please provide as many details as possible. Ideally a reproduction.

[brymko]

Here is a minimal repro:
(edit: set regex-syntax version to 0.7)

Cargo.toml:

[package]
name = "regex_repro"
version = "0.1.0"
edition = "2021"

[dependencies]
regex = { version = "=1.8.1", default_features = false, features = ["std"] }
regex-syntax = { version = "0.7", default_features = false }

main.rs:

use regex::Regex;

fn main() {
    let pat = r"(\d+\s?(years|year|y))?\s?(\d+\s?(months|month|m))?\s?(\d+\s?(weeks|week|w))?\s?(\d+\s?(days|day|d))?\s?(\d+\s?(hours|hour|h))?";
    let result = Regex::new(pat).unwrap();
}

Ouput:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Syntax(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
regex parse error:
    (\d+\s?(years|year|y))?\s?(\d+\s?(months|month|m))?\s?(\d+\s?(weeks|week|w))?\s?(\d+\s?(days|day|d))?\s?(\d+\s?(hours|hour|h))?
     ^^
error: Unicode-aware Perl class not found (make sure the unicode-perl feature is enabled)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
)', src/main.rs:5:34
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Judging from your comment above this looks like it's one of those times where we have relied on incorrect assumptions. Even tho it hurts in the short term I'd be in favor of correctness over legacy hacks and backwards compatibility hacks like you see in webkit.

[BurntSushi]

So I actually think that example isn't quite right. You want default-features = true for regex-syntax 0.6. And you do want 0.6 in there. Basically, the key is that you want something that works with regex 1.7.3 but doesn't with regex 1.8 without changing the dependencies. Here:

[dependencies]
regex = { version = "1.6.0", default_features = false, features = ["std"] }
regex-syntax = "0.6.29"

With the same main.rs as you, running that gives:

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.00s
     Running `target/debug/regexdepissue`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Syntax(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
regex parse error:
    (\d+\s?(years|year|y))?\s?(\d+\s?(months|month|m))?\s?(\d+\s?(weeks|week|w))?\s?(\d+\s?(days|day|d))?\s?(\d+\s?(hours|hour|h))?
     ^^
error: Unicode-aware Perl class not found (make sure the unicode-perl feature is enabled)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
)', main.rs:5:34
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Now let's downgrade to regex 1.7.3 via the lock file:

$ cargo update -p regex --precise 1.7.3
    Updating crates.io index
 Downgrading regex v1.8.1 -> v1.7.3
    Removing regex-syntax v0.7.1

And re-run:

$ cargo run                            
   Compiling regex v1.7.3
   Compiling regexdepissue v0.1.0 (/tmp/regexdepissue)
    Finished dev [unoptimized + debuginfo] target(s) in 0.54s
     Running `target/debug/regexdepissue`
$

(I've snipped out unrelated warnings from the above output.

Anyway, that's the right repro, because it simulates real world conditions:

1. You've got a dependency on regex with its crate features under specified.
2. Somewhere else there is a dependency on regex-syntax that enables everything.
3. The under specified features in (1) gets papered over by feature unification.
4. Everything works.

And then when you upgrade to regex 1.8, you get regex-syntax 0.7 and the feature unification no longer applies.