Add expected outputs for light models to make the backend test faster #4960
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: xadupre <xadupre@microsoft.com>
Signed-off-by: xadupre <xadupre@microsoft.com>
Signed-off-by: xadupre <xadupre@microsoft.com>
Signed-off-by: xadupre <xadupre@microsoft.com>
Comment on lines
+73
to
+74
| # if isinstance(model, InferenceSessionBackendRep): | ||
| # return model |
Check notice
Code scanning / CodeQL
Commented-out code
jcwchen
reviewed
Mar 3, 2023
Signed-off-by: xadupre <xadupre@microsoft.com>
|
The test are running if onnxruntime is installed on CI. It is not the case right now. This file makes it easier for developer to check onnxruntime on existing tests. |
| return name | ||
| i += 1 | ||
| # The function should never go through that line. | ||
| raise RuntimeError("The function should never go through that line.") |
Check warning
Code scanning / CodeQL
Unreachable code
yuanyao-nv
reviewed
Mar 16, 2023
Signed-off-by: xadupre <xadupre@microsoft.com>
Signed-off-by: xadupre <xadupre@microsoft.com>
Signed-off-by: xadupre <xadupre@microsoft.com>
Signed-off-by: xadupre <xadupre@microsoft.com>
Signed-off-by: xadupre <xadupre@microsoft.com>
Signed-off-by: xadupre <xadupre@microsoft.com>
Signed-off-by: xadupre <xadupre@microsoft.com>
Signed-off-by: xadupre <xadupre@microsoft.com>
gramalingam
approved these changes
Apr 3, 2023
Signed-off-by: xadupre <xadupre@microsoft.com>
Signed-off-by: xadupre <xadupre@microsoft.com>
smk2007
pushed a commit
to smk2007/onnx
that referenced
this pull request
Apr 5, 2023
…onnx#4960) * Add expected outputs for light models to make the backend test faster Signed-off-by: xadupre <xadupre@microsoft.com> * add expected outputs Signed-off-by: xadupre <xadupre@microsoft.com> * add missing outputs Signed-off-by: xadupre <xadupre@microsoft.com> * add missing expected output Signed-off-by: xadupre <xadupre@microsoft.com> * lint Signed-off-by: xadupre <xadupre@microsoft.com> * lint Signed-off-by: xadupre <xadupre@microsoft.com> * skip failing test for onnxruntime Signed-off-by: xadupre <xadupre@microsoft.com> * lint Signed-off-by: xadupre <xadupre@microsoft.com> * use range as well Signed-off-by: xadupre <xadupre@microsoft.com> * fix a bug Signed-off-by: xadupre <xadupre@microsoft.com> * exception Signed-off-by: xadupre <xadupre@microsoft.com> * fix light models by lower the fill value of ConstantOfShape nodes Signed-off-by: xadupre <xadupre@microsoft.com> * lint Signed-off-by: xadupre <xadupre@microsoft.com> * fix type issue in deform implementation Signed-off-by: xadupre <xadupre@microsoft.com> * exclude deformconv from onnxruntime backend test series Signed-off-by: xadupre <xadupre@microsoft.com> * disable test when numpy is not recent enough Signed-off-by: xadupre <xadupre@microsoft.com> * move one test Signed-off-by: xadupre <xadupre@microsoft.com> * avoid python 3.7 Signed-off-by: xadupre <xadupre@microsoft.com> * add missing import Signed-off-by: xadupre <xadupre@microsoft.com> * fix import Signed-off-by: xadupre <xadupre@microsoft.com> * add missing license Signed-off-by: xadupre <xadupre@microsoft.com> * add copyright for binary file Signed-off-by: xadupre <xadupre@microsoft.com> --------- Signed-off-by: xadupre <xadupre@microsoft.com> Co-authored-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: Sheil Kumar <smk2007@gmail.com>
smk2007
pushed a commit
to smk2007/onnx
that referenced
this pull request
Apr 5, 2023
…onnx#4960) * Add expected outputs for light models to make the backend test faster Signed-off-by: xadupre <xadupre@microsoft.com> * add expected outputs Signed-off-by: xadupre <xadupre@microsoft.com> * add missing outputs Signed-off-by: xadupre <xadupre@microsoft.com> * add missing expected output Signed-off-by: xadupre <xadupre@microsoft.com> * lint Signed-off-by: xadupre <xadupre@microsoft.com> * lint Signed-off-by: xadupre <xadupre@microsoft.com> * skip failing test for onnxruntime Signed-off-by: xadupre <xadupre@microsoft.com> * lint Signed-off-by: xadupre <xadupre@microsoft.com> * use range as well Signed-off-by: xadupre <xadupre@microsoft.com> * fix a bug Signed-off-by: xadupre <xadupre@microsoft.com> * exception Signed-off-by: xadupre <xadupre@microsoft.com> * fix light models by lower the fill value of ConstantOfShape nodes Signed-off-by: xadupre <xadupre@microsoft.com> * lint Signed-off-by: xadupre <xadupre@microsoft.com> * fix type issue in deform implementation Signed-off-by: xadupre <xadupre@microsoft.com> * exclude deformconv from onnxruntime backend test series Signed-off-by: xadupre <xadupre@microsoft.com> * disable test when numpy is not recent enough Signed-off-by: xadupre <xadupre@microsoft.com> * move one test Signed-off-by: xadupre <xadupre@microsoft.com> * avoid python 3.7 Signed-off-by: xadupre <xadupre@microsoft.com> * add missing import Signed-off-by: xadupre <xadupre@microsoft.com> * fix import Signed-off-by: xadupre <xadupre@microsoft.com> * add missing license Signed-off-by: xadupre <xadupre@microsoft.com> * add copyright for binary file Signed-off-by: xadupre <xadupre@microsoft.com> --------- Signed-off-by: xadupre <xadupre@microsoft.com> Co-authored-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: Sheil Kumar <smk2007@gmail.com>
gramalingam
pushed a commit
that referenced
this pull request
Apr 5, 2023
* Bump ossf/scorecard-action from 2.1.2 to 2.1.3 (#5084) Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 2.1.2 to 2.1.3. - [Release notes](https://github.com/ossf/scorecard-action/releases) - [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md) - [Commits](ossf/scorecard-action@e38b190...80e868c) --- updated-dependencies: - dependency-name: ossf/scorecard-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Sheil Kumar <smk2007@gmail.com> * [CI] Fix weekly mac release CI failure and cover onnxruntime tests on AzurePipelines (#5057) * clean up CIs Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * rename .yml instead of .yaml for consistency Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * windows CI Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> --------- Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Co-authored-by: Andreas Fehlner <fehlner@arcor.de> Signed-off-by: Sheil Kumar <smk2007@gmail.com> * Fix reference implementation for Pad and Reduce ops (#5008) * Fix reference implementation for Pad Signed-off-by: xadupre <xadupre@microsoft.com> * add unit test Signed-off-by: xadupre <xadupre@microsoft.com> * use parameterized Signed-off-by: xadupre <xadupre@microsoft.com> * check type Signed-off-by: xadupre <xadupre@microsoft.com> --------- Signed-off-by: xadupre <xadupre@microsoft.com> Signed-off-by: Sheil Kumar <smk2007@gmail.com> * Add expected outputs for light models to make the backend test faster (#4960) * Add expected outputs for light models to make the backend test faster Signed-off-by: xadupre <xadupre@microsoft.com> * add expected outputs Signed-off-by: xadupre <xadupre@microsoft.com> * add missing outputs Signed-off-by: xadupre <xadupre@microsoft.com> * add missing expected output Signed-off-by: xadupre <xadupre@microsoft.com> * lint Signed-off-by: xadupre <xadupre@microsoft.com> * lint Signed-off-by: xadupre <xadupre@microsoft.com> * skip failing test for onnxruntime Signed-off-by: xadupre <xadupre@microsoft.com> * lint Signed-off-by: xadupre <xadupre@microsoft.com> * use range as well Signed-off-by: xadupre <xadupre@microsoft.com> * fix a bug Signed-off-by: xadupre <xadupre@microsoft.com> * exception Signed-off-by: xadupre <xadupre@microsoft.com> * fix light models by lower the fill value of ConstantOfShape nodes Signed-off-by: xadupre <xadupre@microsoft.com> * lint Signed-off-by: xadupre <xadupre@microsoft.com> * fix type issue in deform implementation Signed-off-by: xadupre <xadupre@microsoft.com> * exclude deformconv from onnxruntime backend test series Signed-off-by: xadupre <xadupre@microsoft.com> * disable test when numpy is not recent enough Signed-off-by: xadupre <xadupre@microsoft.com> * move one test Signed-off-by: xadupre <xadupre@microsoft.com> * avoid python 3.7 Signed-off-by: xadupre <xadupre@microsoft.com> * add missing import Signed-off-by: xadupre <xadupre@microsoft.com> * fix import Signed-off-by: xadupre <xadupre@microsoft.com> * add missing license Signed-off-by: xadupre <xadupre@microsoft.com> * add copyright for binary file Signed-off-by: xadupre <xadupre@microsoft.com> --------- Signed-off-by: xadupre <xadupre@microsoft.com> Co-authored-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: Sheil Kumar <smk2007@gmail.com> * merge batch dimension fix to STFT Signed-off-by: Sheil Kumar <smk2007@gmail.com> * Add clang tidy review in Lint CI (#5041) * add clang-tidy review Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * more configs Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * add post yml Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * rename Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * post Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * orkflow_dispatch Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Post clang-tidy review comments Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * remove workflow Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * test Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * revert demo Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> --------- Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Co-authored-by: Andreas Fehlner <fehlner@arcor.de> Signed-off-by: Sheil Kumar <smk2007@gmail.com> * Upgrade Protobuf to v21.12 to accept Python Protobuf 4.21 (#4956) * [WIP] Upgrade Protobuf to v21.1 to accept Python Protobuf 4.21 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * then Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Same as previous PR Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * fix CIs Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * 3.21.12 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * build protobuf from source on Linux Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * static protobuf Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * protoc --version Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * $BUILD_TYPE Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * printenv PATH Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * path Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * onnx-protobufv21 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * test 3.20.2 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Protobuf 4 and Protobuf 3 will output different order of protobuf fields. Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * skip Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * %%script --no-raise-error false Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * updated Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * test lower protoc Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * export CMAKE_ARGS="-DONNX_WERROR=ON -DONNX_USE_PROTOBUF_SHARED_LIBS=ON" Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * fix Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * still test protobuf==3.20.2 at least in a CI Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * update Protobufs.ipynb Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * ["nbreg"] = {"skip": True, "skip_reason": "Not ready for testing."} Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * NBVAL_SKIP Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * use " instead of ' Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> --------- Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: Sheil Kumar <smk2007@gmail.com> * add reuse badge (#5085) * Update README.md --------- Signed-off-by: Andreas Fehlner <fehlner@arcor.de> Signed-off-by: Sheil Kumar <smk2007@gmail.com> * Update reuse.yml (#5087) Signed-off-by: Andreas Fehlner <fehlner@arcor.de> Signed-off-by: Sheil Kumar <smk2007@gmail.com> * onnx (#5089) Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Co-authored-by: Andreas Fehlner <fehlner@arcor.de> Signed-off-by: Sheil Kumar <smk2007@gmail.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Sheil Kumar <smk2007@gmail.com> Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: xadupre <xadupre@microsoft.com> Signed-off-by: Andreas Fehlner <fehlner@arcor.de> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chun-Wei Chen <jacky82226@gmail.com> Co-authored-by: Andreas Fehlner <fehlner@arcor.de> Co-authored-by: Xavier Dupré <xadupre@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Current backend test use ReferenceEvaluator to compute the expected output for random input. However, implementation for operator Conv is far too slow to make them usable. This PR creates explicit inputs for those models and add the expected outputs to the reposity to avoid relying on ReferenceEvaluator to compute them.
Motivation and Context
Agility.