Skip to content

Conversation

tonypottera24
Copy link
Contributor

@tonypottera24 tonypottera24 commented Oct 27, 2024

Description

Raise an exception when the external file exists during onnx.save.

Note: we can also consider to overwrite it.

Motivation and Context

If the external file is not removed, it will not be overwritten. The new content is always appended to it, making it larger and larger forever.

Signed-off-by: Po-Chu Hsu <tonypottera@gmail.com>
@tonypottera24 tonypottera24 requested a review from a team as a code owner October 27, 2024 06:12
Copy link

codecov bot commented Oct 27, 2024

Codecov Report

Attention: Patch coverage is 33.33333% with 2 lines in your changes missing coverage. Please review.

Project coverage is 57.39%. Comparing base (52dc967) to head (5ef9e1d).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
onnx/external_data_helper.py 33.33% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6497      +/-   ##
==========================================
- Coverage   57.40%   57.39%   -0.01%     
==========================================
  Files         507      507              
  Lines       31526    31528       +2     
  Branches     3548     3549       +1     
==========================================
  Hits        18096    18096              
- Misses      12582    12583       +1     
- Partials      848      849       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Po-Chu Hsu <tonypottera@gmail.com>
@xadupre
Copy link
Contributor

xadupre commented Oct 28, 2024

Should we remove it or raise an exception? cc @gramalingam

@tonypottera24
Copy link
Contributor Author

tonypottera24 commented Oct 28, 2024

Should we remove it or raise an exception? cc @gramalingam

I moved the check to convert_model_to_external_data.
Let me know if you think we should overwrite it instead.

Signed-off-by: Po-Chu Hsu <tonypottera@gmail.com>
Signed-off-by: Po-Chu Hsu <tonypottera@gmail.com>
Signed-off-by: Po-Chu Hsu <tonypottera@gmail.com>
@tonypottera24 tonypottera24 changed the title Remove external file if exists in extract_model Remove/raise exception when external file exists during onnx.save Oct 28, 2024
@justinchuby

This comment was marked as resolved.

Copy link
Member

@justinchuby justinchuby left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks. Please ignore my previous comments

@justinchuby
Copy link
Member

Could you add the Raises section in the docstring to describe this new behavior. Thanks! ref: https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html

@justinchuby
Copy link
Member

Overwriting can be unsafe if the model proto has reference to the existing data file. Maybe we can provide an option? But I also don't prefer overcrowding the options we have. hmm

Copy link
Member

@justinchuby justinchuby left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deferring to @gramalingam

Signed-off-by: Po-Chu Hsu <tonypottera@gmail.com>
@tonypottera24
Copy link
Contributor Author

Could you add the Raises section in the docstring to describe this new behavior. Thanks! ref: https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html

Thank you @justinchuby, I added the explanations.

@tonypottera24
Copy link
Contributor Author

tonypottera24 commented Nov 16, 2024

Deferring to @gramalingam

@gramalingam any thoughts?

@gramalingam
Copy link
Contributor

I think it is safer not to overwrite it silently. I am fine with using a user option to overwrite or leaving in the PR's current form (with an exception).

Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
@justinchuby justinchuby added the release notes Important changes to call out in release notes label Nov 21, 2024
@justinchuby justinchuby added this to the 1.18 milestone Nov 21, 2024
@justinchuby justinchuby temporarily deployed to testpypi_onnxweekly November 21, 2024 17:30 — with GitHub Actions Inactive
@justinchuby justinchuby added this pull request to the merge queue Nov 21, 2024
Merged via the queue into onnx:main with commit 80a281d Nov 21, 2024
46 checks passed
@justinchuby justinchuby temporarily deployed to testpypi_onnxweekly November 21, 2024 18:48 — with GitHub Actions Inactive
@tonypottera24 tonypottera24 deleted the patch-external-data branch November 21, 2024 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release notes Important changes to call out in release notes topic: bc breaking
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants