Skip to content

Improve mismatched sizes management when loading a pretrained model #17257

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
May 17, 2022
Merged

Improve mismatched sizes management when loading a pretrained model #17257

merged 3 commits into from
May 17, 2022

Conversation

regisss
Copy link
Contributor

@regisss regisss commented May 14, 2022 •
edited
Loading

What does this PR do?

Examples currently fail when the loaded model head has different dimensions from the expected ones. For instance, in the image classification example, if a pretrained classification head has different dimensions from the classification head to fine-tune, the current implementation will lead to this error:

Traceback (most recent call last):
  File "run_image_classification.py", line 377, in <module>
    main()
  File "run_image_classification.py", line 267, in main
    model = AutoModelForImageClassification.from_pretrained(
  File "/home/regis/HuggingFace/dev/transformers/venv/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 446, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/home/regis/HuggingFace/dev/transformers/venv/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2067, in from_pretrained
    model, missing_keys, unexpected_keys, mismatched_keys, error_msgs = cls._load_pretrained_model(
  File "/home/regis/HuggingFace/dev/transformers/venv/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2276, in _load_pretrained_model
    raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
RuntimeError: Error(s) in loading state_dict for SwinForImageClassification:
	size mismatch for classifier.weight: copying a param with shape torch.Size([1000, 1024]) from checkpoint, the shape in current model is torch.Size([3, 1024]).
	size mismatch for classifier.bias: copying a param with shape torch.Size([1000]) from checkpoint, the shape in current model is torch.Size([3]).

To reproduce this, you can run the image classification example with Swin, such as:

python run_image_classification.py \
  --dataset_name beans \
  --output_dir /tmp/beans_outputs/ \
  --remove_unused_columns False \
  --do_train \
  --per_device_train_batch_size 8 \
  --model_name_or_path microsoft/swin-base-patch4-window7-224

The solution is to add the argument ignore_mismatched_sizes=True to the AutoModelForXXX.from_pretrained method. Thus, this PR does the following:

  • expand the error message and suggest a solution when the error is raised
  • for all classification examples, an argument --ignore_mismatched_sizes can now be given to adapt the dimensions of the classification head when they are different from the expected ones

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@regisss regisss requested review from LysandreJik and sgugger May 14, 2022 21:48
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented May 14, 2022 •
edited
Loading

The documentation is not available anymore as the PR was closed or merged.

@regisss
Copy link
Contributor Author

regisss commented May 14, 2022

I see that all tests corresponding to mismatched sizes failed. Looking at test_modeling_common.py, I guess the current behaviour is expected.

Shall I close this PR and open a new issue regarding the fact that some models will fail in examples (such as Swin for image classification as explained in the first message)?

@sgugger
Copy link
Collaborator

sgugger commented May 16, 2022

I'm not sure why you would open a new issue: this is not a bug and there is an option to load a model with a pretrained head that has different shapes from the checkpoint. Or do you mean the examples should have a flag to activate that option?

@regisss
Copy link
Contributor Author

regisss commented May 16, 2022

I'm not sure why you would open a new issue: this is not a bug and there is an option to load a model with a pretrained head that has different shapes from the checkpoint. Or do you mean the examples should have a flag to activate that option?

After using it a bit more I realized it's not a bug indeed, sorry for the confused wording. The problem is that just changing the model used in the example may break it and I couldn't fine anywhere in the doc or in the READMEs how to solve this, I had to take a look at the code. I suggest one of the following to make it more user-friendly:

  • the error message suggests to add the argument ignore_mismatched_sizes=True to AutoModelForXXX.from_pretrained
  • adding a flag to activate that option as you propose, with a mention in the README
  • changing the default value of ignore_mismatched_sizes to True since a warning is displayed when sizes are different, but I guess I'm lacking of context here and I'm just considering this example use case

Again I'm certainly lacking of context here but I would be happy to modify this PR so that it makes using a different model in examples less tedious when the pretrained head has different dimensions :)

@sgugger
Copy link
Collaborator

sgugger commented May 16, 2022 

* the error message suggests to add the argument `ignore_mismatched_sizes=True` to `AutoModelForXXX.from_pretrained`

This is definitely something we can add and would help the user!

* adding a flag to activate that option as you propose, with a mention in the README

Yes, another welcome improvement!

* changing the default value of `ignore_mismatched_sizes` to `True` since a warning is displayed when sizes are different, but I guess I'm lacking of context here and I'm just considering this example use case

This can't be done for backward compatibility reasons. In further work in from_pretrained, we might have a default that does this, but only for the head of the model. It's dangerous to have it enabled by default on the whole body.

@regisss
Copy link
Contributor Author

regisss commented May 16, 2022

Great, I'm going to modify this PR accordingly!

@regisss
Add --ignore_mismatched_sizes argument to classification examples
27192d3 
- Expand the error message when loading a model whose head dimensions are different from expected dimensions
@regisss regisss changed the title Fix mismatched keys extraction when loading from a pretrained model Improve mismatched sizes management when loading a pretrained model May 17, 2022
@regisss
Copy link
Contributor Author

regisss commented May 17, 2022

I just modified the classification examples because I'm not sure about the types of head used in other scenarios.

Also, I noticed that VSCode automatically trimmed extra whitespaces (because I configured it this way). Let me know if this is an issue and I'll revert that.

sgugger
sgugger approved these changes May 17, 2022
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice thanks for adding all of this! I think you covered all the examples where the parameter is useful.

@regisss @sgugger
Update error message
14c9de5 
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
@regisss regisss merged commit 28a0811 into huggingface:main May 17, 2022
@regisss regisss deleted the fix_ignore_mismatched_sizes branch May 17, 2022 15:58
ArthurZucker added a commit to ArthurZucker/transformers that referenced this pull request May 20, 2022
@ArthurZucker
Squashed commit of the following:
55b1309 
commit 5419205
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Thu May 19 23:46:26 2022 +0200

    [Test OPT] Add batch generation test opt (huggingface#17359)

    * up

    * up

commit 48c2269
Author: ddobokki <44228269+ddobokki@users.noreply.github.com>
Date:   Fri May 20 05:42:44 2022 +0900

    Fix bug in Wav2Vec2 pretrain example (huggingface#17326)

commit 5d6feec
Author: Nathan Dahlberg <58701810+nadahlberg@users.noreply.github.com>
Date:   Thu May 19 16:21:19 2022 -0400

    fix for 17292 (huggingface#17293)

commit 518bd02
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Thu May 19 22:17:02 2022 +0200

    [Generation] Fix Transition probs (huggingface#17311)

    * [Draft] fix transition probs

    * up

    * up

    * up

    * make it work

    * fix

    * finish

    * update

commit e8714c0
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Thu May 19 22:15:36 2022 +0200

    [OPT] Run test in lower precision on GPU (huggingface#17353)

    * [OPT] Run test only in half precision

    * up

    * up

    * up

    * up

    * finish

    * fix on GPU

    * Update tests/models/opt/test_modeling_opt.py

commit 2b28229
Author: Nicolas Patry <patry.nicolas@protonmail.com>
Date:   Thu May 19 20:28:12 2022 +0200

    Adding `batch_size` test to QA pipeline. (huggingface#17330)

commit a4386d7
Author: Nicolas Patry <patry.nicolas@protonmail.com>
Date:   Thu May 19 10:29:16 2022 +0200

    [BC] Fixing usage of text pairs (huggingface#17324)

    * [BC] Fixing usage of text pairs

    The BC is actually preventing users from misusing the pipeline since
    users could have been willing to send text pairs and the pipeline would
    instead understand the thing as a batch returning bogus results.

    The correct usage of text pairs is preserved in this PR even when that
    makes the code clunky.

    Adds support for {"text":..,, "text_pair": ...} inputs for both dataset
    iteration and more explicit usage to pairs.

    * Updating the doc.

    * Update src/transformers/pipelines/text_classification.py

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

    * Update src/transformers/pipelines/text_classification.py

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

    * Update tests/pipelines/test_pipelines_text_classification.py

    Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

    * quality.

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
    Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

commit 3601aa8
Author: Stas Bekman <stas00@users.noreply.github.com>
Date:   Wed May 18 16:00:47 2022 -0700

    [tests] fix copy-n-paste error (huggingface#17312)

    * [tests] fix copy-n-paste error

    * fix

commit 1b20c97
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Wed May 18 21:49:08 2022 +0200

    Fix ci_url might be None (huggingface#17332)

    * fix

    * Update utils/notification_service.py

    Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
    Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>

commit 6aad387
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Wed May 18 21:26:44 2022 +0200

    fix (huggingface#17337)

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

commit 1762ded
Author: Zachary Mueller <muellerzr@gmail.com>
Date:   Wed May 18 14:17:40 2022 -0400

    Fix metric calculation in examples and setup tests to run on multi-gpu for no_trainer scripts (huggingface#17331)

    * Fix length in no_trainer examples

    * Add setup and teardown

    * Use new accelerator config generator to automatically make tests able to run based on environment

commit 6e195eb
Author: Jader Martins <jadermcs@users.noreply.github.com>
Date:   Wed May 18 14:18:43 2022 -0300

    docs for typical decoding (huggingface#17186)

    Co-authored-by: Jader Martins <jadermcs94@gmail.com>

commit 060fe61
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Wed May 18 19:07:48 2022 +0200

    Not send successful report (huggingface#17329)

    * send report only if there is any failure

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

commit b3b9f99
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Wed May 18 17:57:23 2022 +0200

    Fix test_t5_decoder_model_past_large_inputs (huggingface#17320)

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

commit 6da76b9
Author: Jingya HUANG <44135271+JingyaHuang@users.noreply.github.com>
Date:   Wed May 18 17:52:13 2022 +0200

    Add onnx export cuda support (huggingface#17183)

    Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

    Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>

commit adc0ff2
Author: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Date:   Wed May 18 17:47:18 2022 +0200

    Add CvT (huggingface#17299)

    * Adding cvt files

    * Adding cvt files

    * changes in init file

    * Adding cvt files

    * changes in init file

    * Style fixes

    * Address comments from code review

    * Apply suggestions from code review

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

    * Format lists in docstring

    * Fix copies

    * Apply suggestion from code review

    Co-authored-by: AnugunjNaman <anugunjjha@gmail.com>
    Co-authored-by: Ayushman Singh <singhayushman13@protonmail.com>
    Co-authored-by: Niels Rogge <nielsrogge@Nielss-MacBook-Pro.local>
    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

commit 4710702
Author: Sylvain Gugger <Sylvain.gugger@gmail.com>
Date:   Wed May 18 10:46:40 2022 -0400

    Fix style

commit 5fdb54e
Author: mraunak <83710963+mraunak@users.noreply.github.com>
Date:   Wed May 18 10:39:02 2022 -0400

    Add Information Gain Filtration algorithm (huggingface#16953)

    * Add information gain filtration algorithm

    * Complying with black requirements

    * Added author

    * Fixed import order

    * flake8 corrections

    Co-authored-by: Javier Turek <javier.turek@intel.com>

commit 91ede48
Author: Kamal Raj <kamalraj97@gmail.com>
Date:   Wed May 18 19:59:53 2022 +0530

    Fix typo (huggingface#17328)

commit fe28eb9
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Wed May 18 16:06:41 2022 +0200

    remove (huggingface#17325)

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

commit 2cb2ea3
Author: Nicolas Patry <patry.nicolas@protonmail.com>
Date:   Wed May 18 16:06:24 2022 +0200

    Accepting real pytorch device as arguments. (huggingface#17318)

    * Accepting real pytorch device as arguments.

    * is_torch_available.

commit 1c9d1f4
Author: Nicolas Patry <patry.nicolas@protonmail.com>
Date:   Wed May 18 15:46:12 2022 +0200

    Updating the docs for `max_seq_len` in QA pipeline (huggingface#17316)

commit 60ad734
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Wed May 18 15:08:56 2022 +0200

    [T5] Fix init in TF and Flax for pretraining (huggingface#17294)

    * fix init

    * Apply suggestions from code review

    * fix

    * finish

    * Update src/transformers/modeling_tf_utils.py

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

commit 7ba1d4e
Author: Joaq <55513213+jQuinRivero@users.noreply.github.com>
Date:   Wed May 18 09:23:47 2022 -0300

    Add type hints for ProphetNet (Pytorch) (huggingface#17223)

    * added type hints to prophetnet

    * reformatted with black

    * fix bc black misformatted some parts

    * fix imports

    * fix imports

    * Update src/transformers/models/prophetnet/configuration_prophetnet.py

    Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>

    * update OPTIONAL type hint and docstring

    Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>

commit d6b8e9c
Author: Carl <carl.cochet@gmail.com>
Date:   Wed May 18 01:07:43 2022 +0200

    Add trajectory transformer (huggingface#17141)

    * Add trajectory transformer

    Fix model init

    Fix end of lines for .mdx files

    Add trajectory transformer model to toctree

    Add forward input docs

    Fix docs, remove prints, simplify prediction test

    Apply suggestions from code review

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
    Apply suggestions from code review

    Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
    Update docs, more descriptive comments

    Apply suggestions from code review

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
    Update readme

    Small comment update and add conversion script

    Rebase and reformat

    Fix copies

    Fix rebase, remove duplicates

    Fix rebase, remove duplicates

    * Remove tapex

    * Remove tapex

    * Remove tapex

commit c352640
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Wed May 18 00:34:31 2022 +0200

    fix (huggingface#17310)

commit d9050dc
Author: Cesare Campagnano <cesare.campagnano@gmail.com>
Date:   Tue May 17 23:44:37 2022 +0200

    [LED] fix global_attention_mask not being passed for generation and docs clarification about grad checkpointing (huggingface#17112)

    * [LED] fixed global_attention_mask not passed for generation + docs clarification for gradient checkpointing

    * LED docs clarification

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    * [LED] gradient_checkpointing=True should be passed to TrainingArguments

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    * [LED] docs: remove wrong word

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    * [LED] docs fix typo

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

commit bad3583
Author: Jean Vancoppenolle <jean.vcop@gmail.com>
Date:   Tue May 17 23:42:14 2022 +0200

    Add support for pretraining recurring span selection to Splinter (huggingface#17247)

    * Add SplinterForSpanSelection for pre-training recurring span selection.

    * Formatting.

    * Rename SplinterForSpanSelection to SplinterForPreTraining.

    * Ensure repo consistency

    * Fixup changes

    * Address SplinterForPreTraining PR comments

    * Incorporate feedback and derive multiple question tokens per example.

    * Update src/transformers/models/splinter/modeling_splinter.py

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    * Update src/transformers/models/splinter/modeling_splinter.py

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    Co-authored-by: Jean Vancoppenole <jean.vancoppenolle@retresco.de>
    Co-authored-by: Tobias Günther <tobias.guenther@retresco.de>
    Co-authored-by: Tobias Günther <github@tobigue.de>
    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

commit 0511305
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Tue May 17 18:56:58 2022 +0200

    Add PR author in CI report + merged by info (huggingface#17298)

    * Add author info to CI report

    * Add merged by info

    * update

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

commit 032d63b
Author: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Date:   Tue May 17 12:56:24 2022 -0400

    Fix dummy creation script (huggingface#17304)

commit 986dd5c
Author: Sylvain Gugger <Sylvain.gugger@gmail.com>
Date:   Tue May 17 12:50:14 2022 -0400

    Fix style

commit 38ddab1
Author: Karim Foda <35491698+KMFODA@users.noreply.github.com>
Date:   Tue May 17 09:32:12 2022 -0700

    Doctest longformer (huggingface#16441)

    * Add initial doctring changes

    * make fixup

    * Add TF doc changes

    * fix seq classifier output

    * fix quality errors

    * t

    * swithc head to random init

    * Fix expected outputs

    * Update src/transformers/models/longformer/modeling_longformer.py

    Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>

    Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>

commit 10704e1
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Tue May 17 18:20:36 2022 +0200

    [Test] Fix W2V-Conformer integration test (huggingface#17303)

    * [Test] Fix W2V-Conformer integration test

    * correct w2v2

    * up

commit 28a0811
Author: regisss <15324346+regisss@users.noreply.github.com>
Date:   Tue May 17 17:58:14 2022 +0200

    Improve mismatched sizes management when loading a pretrained model (huggingface#17257)

    - Add --ignore_mismatched_sizes argument to classification examples

    - Expand the error message when loading a model whose head dimensions are different from expected dimensions

commit 1f13ba8
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Tue May 17 15:48:23 2022 +0200

    correct opt (huggingface#17301)

commit 349f1c8
Author: Matt <Rocketknight1@users.noreply.github.com>
Date:   Tue May 17 14:36:23 2022 +0100

    Rewrite TensorFlow train_step and test_step (huggingface#17057)

    * Initial commit

    * Better label renaming

    * Remove breakpoint before pushing (this is your job)

    * Test a lot more in the Keras fit() test

    * make fixup

    * Clarify the case where we flatten y dicts into tensors

    * Clarify the case where we flatten y dicts into tensors

    * Extract label name remapping to a method

commit 651e48e
Author: Matt <Rocketknight1@users.noreply.github.com>
Date:   Tue May 17 14:14:17 2022 +0100

    Fix tests of mixed precision now that experimental is deprecated (huggingface#17300)

    * Fix tests of mixed precision now that experimental is deprecated

    * Fix mixed precision in training_args_tf.py too

commit 6d21142
Author: SaulLu <55560583+SaulLu@users.noreply.github.com>
Date:   Tue May 17 14:33:13 2022 +0200

    fix retribert's `test_torch_encode_plus_sent_to_model` (huggingface#17231)
elusenji pushed a commit to elusenji/transformers that referenced this pull request Jun 12, 2022
@regisss @elusenji
Improve mismatched sizes management when loading a pretrained model (h…
eebf780 
…uggingface#17257)

- Add --ignore_mismatched_sizes argument to classification examples

- Expand the error message when loading a model whose head dimensions are different from expected dimensions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LysandreJik LysandreJik Awaiting requested review from LysandreJik

1 more reviewer

@sgugger sgugger sgugger approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@regisss @HuggingFaceDocBuilderDev @sgugger