Add RegexFullMatch operator #5401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

xadupre merged 45 commits into onnx:main from adityagoel4512:regexfullmatch-operator

Aug 4, 2023

Contributor

adityagoel4512 commented Jul 9, 2023 •

edited

Loading

Description

This PR introduces the RegexFullMatch operator, as originally proposed in #5317.

The RegexFullMatch operator takes one string tensor as input and returns a bool tensor of identical shape indicating if each element fully matches the regex pattern encoded in the pattern string attribute. This attribute is a string and we expect valid re2 regex.

Some examples are as follows:

RegexFullMatch(["www.google.com", "www.facebook.com", "www.bbc.co.uk"], "www\.[\w.-]+\.\bcom\b")
=> [True, True, False]
RegexFullMatch([["account@gmail.com", "account@hotmail.com"], ["not email", "account2@yahoo.com"]], "(\W|^)[\w.\-]{0,25}@(yahoo|gmail)\.com(\W|$)")
=> [[True, False], [False, True]]

Motivation and Context

Closes #5317.

Following discussion at the last Operators SIG Weekly the "engine" attribute has been dropped in favour of simply using re2 syntax for now. This reflects the fact that both Tensorflow and PyTorch operators requiring regex use re2 already.


          Add RegexFullMatch operator

00bfa0a

Signed-off-by: Aditya Goel <agoel4512@gmail.com>

adityagoel4512 marked this pull request as ready for review

July 9, 2023 23:11

adityagoel4512 requested review from a team as code owners

July 9, 2023 23:11


          Fix docs

02c32d1

Signed-off-by: Aditya Goel <agoel4512@gmail.com>

adityagoel4512 force-pushed the regexfullmatch-operator branch from e109fac to 02c32d1 Compare

July 9, 2023 23:16

github-advanced-security bot found potential problems

View reviewed changes

onnx/reference/ops/_op_list.py

@@ @@ -175,6 +175,7 @@ @@
                   ReduceSumSquare_1,
                   ReduceSumSquare_18,
               )
+              from onnx.reference.ops.op_regex_full_match import RegexFullMatch

Check notice

Code scanning / CodeQL

Unused import

Import of 'RegexFullMatch' is not used.

xadupre reviewed

View reviewed changes

requirements.txt Outdated Show resolved Hide resolved

xadupre reviewed

View reviewed changes

onnx/reference/ops/op_regex_full_match.py Outdated Show resolved Hide resolved

xadupre reviewed

View reviewed changes

onnx/defs/text/defs.cc Outdated Show resolved Hide resolved

adityagoel4512 and others added 2 commits

July 10, 2023 13:54


          Exclude regex full match from backend test

f9fb94d

Signed-off-by: Aditya Goel <agoel4512@gmail.com>


          [Spec] output_shape for ConvTranspose should not have batch and chann…

574e0a5

…els (onnx#5400)

### Description
<!-- - Describe your changes. -->
Add a sentence to highlight that output_shape for ConvTranspose should
not have batch and channels.

### Motivation and Context
<!-- - Why is this change required? What problem does it solve? -->
<!-- - If it fixes an open issue, please link to the issue here. -->
It's quite confusing that whether output_shape needs to be fully
provided. For instance, onnx#1437. An
explicit description should help prevent confusion.

cc @satyajandhyala

---------

Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>
Signed-off-by: Aditya Goel <agoel4512@gmail.com>

adityagoel4512 force-pushed the regexfullmatch-operator branch from 88dfc73 to 574e0a5 Compare

July 10, 2023 12:54

adityagoel4512 and others added 2 commits

July 10, 2023 13:55


          Merge branch 'main' into regexfullmatch-operator

9d8ca99


          Add docs

754cc99

Signed-off-by: Aditya Goel <agoel4512@gmail.com>

adityagoel4512 force-pushed the regexfullmatch-operator branch from f3a3a96 to 754cc99 Compare

July 10, 2023 13:17


          Make re2 optional dependency

29a5b4a

Signed-off-by: Aditya Goel <agoel4512@gmail.com>

adityagoel4512 force-pushed the regexfullmatch-operator branch from 66309bb to 29a5b4a Compare

July 10, 2023 14:10

adityagoel4512 and others added 3 commits

July 10, 2023 16:04


          Install with text

d4f84bc

Signed-off-by: Aditya Goel <agoel4512@gmail.com>


          Merge branch 'main' into regexfullmatch-operator

6aff1a4

Signed-off-by: Aditya Goel <48102515+adityagoel4512@users.noreply.github.com>


          Run test coverage

b976bd8

Signed-off-by: Aditya Goel <agoel4512@gmail.com>

adityagoel4512 force-pushed the regexfullmatch-operator branch from 1ae97ce to b976bd8 Compare

July 11, 2023 10:31

gramalingam added the topic: operator label

cbourjau mentioned this pull request

Move StringNormalizer into onnx/defs/text/defs.cc #5420

Closed


          Merge remote-tracking branch 'upstream/main' into regexfullmatch-oper…

d9613be

…ator

Signed-off-by: Aditya Goel <agoel4512@gmail.com>

Contributor Author

adityagoel4512 commented Jul 15, 2023

Would it be possible to receive a review/approval on this PR please? @gramalingam @xadupre


          Fix merge

4949d4d

Signed-off-by: Aditya Goel <agoel4512@gmail.com>

jcwchen added the run release CIs label

jcwchen reviewed

View reviewed changes

Member

jcwchen left a comment

I just added run-release-CIs label to run release CIs (it will take effect if there is a new commit). You will need to add google-re2 in requirements-release.txt to make release CIs pass I believe.


          Add re2 to requirements-release.txt

a0a8ddc

Signed-off-by: Aditya Goel <agoel4512@gmail.com>

adityagoel4512 added 4 commits

July 27, 2023 16:47


          x86 install

f282c0d

Signed-off-by: Aditya Goel <agoel4512@gmail.com>


          x64 install

93761e1

Signed-off-by: Aditya Goel <agoel4512@gmail.com>


          x64 and x86 install

Signed-off-by: Aditya Goel <agoel4512@gmail.com>


          Conditionally import google-re2

5b93576

Signed-off-by: Aditya Goel <agoel4512@gmail.com>

adityagoel4512 force-pushed the regexfullmatch-operator branch from 8aee616 to 5b93576 Compare

July 27, 2023 19:58

adityagoel4512 added 4 commits

July 27, 2023 21:58


          Conditionally import google-re2

4af3eb6

Signed-off-by: Aditya Goel <agoel4512@gmail.com>


          Merge branch 'regexfullmatch-operator' of github.com:adityagoel4512/o…

1b1eb38

…nnx into regexfullmatch-operator

Signed-off-by: Aditya Goel <agoel4512@gmail.com>


          fix merge

Signed-off-by: Aditya Goel <agoel4512@gmail.com>


          Clean diff

485bb79

Signed-off-by: Aditya Goel <agoel4512@gmail.com>

Contributor Author

adityagoel4512 commented Jul 27, 2023 •

edited

Loading

Updates:

we lazily import re2 in the RegexFullMatch reference operator
google-re2 is an optional dependency under the "reference" name (pip install onnx[reference]). New dependencies exclusive to the reference implementation and not used elsewhere can be placed in requirements-reference.txt if it makes sense.
tests that use the reference implementation of RegexFullMatch are skipped when the platform is win32 - (edit: after asking, the next google-re2 release will now support win32

adityagoel4512 added 2 commits

July 28, 2023 17:42


          Raise explicitly for invalid regex patterns in reference implementations

060d433

Signed-off-by: Aditya Goel <agoel4512@gmail.com>


          Skip on win32

dace543

Signed-off-by: Aditya Goel <agoel4512@gmail.com>

adityagoel4512 force-pushed the regexfullmatch-operator branch from 8206f2b to dace543 Compare

August 1, 2023 15:22

adityagoel4512 and others added 3 commits

August 1, 2023 16:22


          Merge branch 'main' into regexfullmatch-operator

7bb78c1


          Fix coverage

c2def01

Signed-off-by: Aditya Goel <agoel4512@gmail.com>


          Merge branch 'main' into regexfullmatch-operator

87e5110

Contributor Author

adityagoel4512 commented Aug 2, 2023

Is there anything holding back this PR? @gramalingam @xadupre @jcwchen


          Merge remote-tracking branch 'origin' into regexfullmatch-operator

d0dfc67

Signed-off-by: Aditya Goel <agoel4512@gmail.com>

xadupre reviewed

View reviewed changes

onnx/test/test_backend_reference.py Show resolved Hide resolved

xadupre reviewed

View reviewed changes

onnx/reference/ops/op_regex_full_match.py Outdated Show resolved Hide resolved

adityagoel4512 added 2 commits

August 3, 2023 18:08


          Add comment and rm issue_test_ai_onnx_ml_label_encoder_tensor_mapping…

2a8e896

….onnx format string

Signed-off-by: Aditya Goel <agoel4512@gmail.com>


          Fix typo

91058c3

Signed-off-by: Aditya Goel <agoel4512@gmail.com>

xadupre approved these changes

View reviewed changes

xadupre enabled auto-merge

August 4, 2023 10:28


          Merge branch 'main' into regexfullmatch-operator

f340297

xadupre added this pull request to the merge queue

Merged via the queue into onnx:main with commit 2e0908d

adityagoel4512 deleted the regexfullmatch-operator branch

August 6, 2023 10:32

liqunfu mentioned this pull request

Add compute kernel for RegexFullMatch microsoft/onnxruntime#17594

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run release CIs topic: operator