Skip to content

[Generation] Fix Transition probs #17311

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
May 19, 2022
Merged

[Generation] Fix Transition probs #17311

merged 8 commits into from
May 19, 2022

Conversation

patrickvonplaten
Copy link
Contributor

@patrickvonplaten patrickvonplaten commented May 17, 2022 •
edited
Loading

What does this PR do?

Fixes #15869

This PR fixes incorrectly computed transition probabilties for beam search.
In beam search it's common to want to know the transition probability between tokens -> see: https://discuss.huggingface.co/t/generation-probabilities-how-to-compute-probabilities-of-output-scores-for-gpt2/3175/15?u=patrickvonplaten

Currently there is a bug for every beam search that ends before max_length as discovered by #15869 .

This PR fixes this behavior and adds a couple of tests.

🚨 There is a tiny breaking change in the output format of beam_indices. Instead of returning a tuple(tuple(int)) a torch.LongTensor is returned. Since the feature was broken before (beam_indices were incorrect), this is hardly a breaking change and necessary to make the functionality more userfriendly and robustu🚨

@patil-suraj @gante , it would be super nice if you could do an in-depth review here also to understand this feature.
Note: people do seem to be interested in output scores for generation as can be seen by the large number of views here: https://discuss.huggingface.co/t/generation-probabilities-how-to-compute-probabilities-of-output-scores-for-gpt2/3175

Feel free to drop any question in the PR if you don't understand something or something is unclear - thanks 🙏

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented May 17, 2022 •
edited
Loading

The documentation is not available anymore as the PR was closed or merged.

@patrickvonplaten patrickvonplaten changed the title Fix transition probs [WIP]Fix transition probs May 17, 2022
@patrickvonplaten
Copy link
Contributor Author

I'll talk about this feature in more-detail in a tutorial

@patrickvonplaten patrickvonplaten changed the title [WIP]Fix transition probs [Generation] Fix Transition probs May 18, 2022
sgugger
sgugger approved these changes May 19, 2022
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this with new tests! LGTM at a high level :-)

gante
gante approved these changes May 19, 2022
View reviewed changes
Copy link
Member

@gante gante left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 👍

(I see from this PR that TF is lacking many useful things like sequence_scores, which are not being set)

@patrickvonplaten patrickvonplaten merged commit 518bd02 into huggingface:main May 19, 2022
ArthurZucker added a commit to ArthurZucker/transformers that referenced this pull request May 20, 2022
@ArthurZucker
Squashed commit of the following:
55b1309 
commit 5419205
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Thu May 19 23:46:26 2022 +0200

    [Test OPT] Add batch generation test opt (huggingface#17359)

    * up

    * up

commit 48c2269
Author: ddobokki <44228269+ddobokki@users.noreply.github.com>
Date:   Fri May 20 05:42:44 2022 +0900

    Fix bug in Wav2Vec2 pretrain example (huggingface#17326)

commit 5d6feec
Author: Nathan Dahlberg <58701810+nadahlberg@users.noreply.github.com>
Date:   Thu May 19 16:21:19 2022 -0400

    fix for 17292 (huggingface#17293)

commit 518bd02
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Thu May 19 22:17:02 2022 +0200

    [Generation] Fix Transition probs (huggingface#17311)

    * [Draft] fix transition probs

    * up

    * up

    * up

    * make it work

    * fix

    * finish

    * update

commit e8714c0
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Thu May 19 22:15:36 2022 +0200

    [OPT] Run test in lower precision on GPU (huggingface#17353)

    * [OPT] Run test only in half precision

    * up

    * up

    * up

    * up

    * finish

    * fix on GPU

    * Update tests/models/opt/test_modeling_opt.py

commit 2b28229
Author: Nicolas Patry <patry.nicolas@protonmail.com>
Date:   Thu May 19 20:28:12 2022 +0200

    Adding `batch_size` test to QA pipeline. (huggingface#17330)

commit a4386d7
Author: Nicolas Patry <patry.nicolas@protonmail.com>
Date:   Thu May 19 10:29:16 2022 +0200

    [BC] Fixing usage of text pairs (huggingface#17324)

    * [BC] Fixing usage of text pairs

    The BC is actually preventing users from misusing the pipeline since
    users could have been willing to send text pairs and the pipeline would
    instead understand the thing as a batch returning bogus results.

    The correct usage of text pairs is preserved in this PR even when that
    makes the code clunky.

    Adds support for {"text":..,, "text_pair": ...} inputs for both dataset
    iteration and more explicit usage to pairs.

    * Updating the doc.

    * Update src/transformers/pipelines/text_classification.py

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

    * Update src/transformers/pipelines/text_classification.py

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

    * Update tests/pipelines/test_pipelines_text_classification.py

    Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

    * quality.

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
    Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

commit 3601aa8
Author: Stas Bekman <stas00@users.noreply.github.com>
Date:   Wed May 18 16:00:47 2022 -0700

    [tests] fix copy-n-paste error (huggingface#17312)

    * [tests] fix copy-n-paste error

    * fix

commit 1b20c97
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Wed May 18 21:49:08 2022 +0200

    Fix ci_url might be None (huggingface#17332)

    * fix

    * Update utils/notification_service.py

    Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
    Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>

commit 6aad387
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Wed May 18 21:26:44 2022 +0200

    fix (huggingface#17337)

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

commit 1762ded
Author: Zachary Mueller <muellerzr@gmail.com>
Date:   Wed May 18 14:17:40 2022 -0400

    Fix metric calculation in examples and setup tests to run on multi-gpu for no_trainer scripts (huggingface#17331)

    * Fix length in no_trainer examples

    * Add setup and teardown

    * Use new accelerator config generator to automatically make tests able to run based on environment

commit 6e195eb
Author: Jader Martins <jadermcs@users.noreply.github.com>
Date:   Wed May 18 14:18:43 2022 -0300

    docs for typical decoding (huggingface#17186)

    Co-authored-by: Jader Martins <jadermcs94@gmail.com>

commit 060fe61
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Wed May 18 19:07:48 2022 +0200

    Not send successful report (huggingface#17329)

    * send report only if there is any failure

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

commit b3b9f99
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Wed May 18 17:57:23 2022 +0200

    Fix test_t5_decoder_model_past_large_inputs (huggingface#17320)

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

commit 6da76b9
Author: Jingya HUANG <44135271+JingyaHuang@users.noreply.github.com>
Date:   Wed May 18 17:52:13 2022 +0200

    Add onnx export cuda support (huggingface#17183)

    Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

    Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>

commit adc0ff2
Author: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Date:   Wed May 18 17:47:18 2022 +0200

    Add CvT (huggingface#17299)

    * Adding cvt files

    * Adding cvt files

    * changes in init file

    * Adding cvt files

    * changes in init file

    * Style fixes

    * Address comments from code review

    * Apply suggestions from code review

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

    * Format lists in docstring

    * Fix copies

    * Apply suggestion from code review

    Co-authored-by: AnugunjNaman <anugunjjha@gmail.com>
    Co-authored-by: Ayushman Singh <singhayushman13@protonmail.com>
    Co-authored-by: Niels Rogge <nielsrogge@Nielss-MacBook-Pro.local>
    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

commit 4710702
Author: Sylvain Gugger <Sylvain.gugger@gmail.com>
Date:   Wed May 18 10:46:40 2022 -0400

    Fix style

commit 5fdb54e
Author: mraunak <83710963+mraunak@users.noreply.github.com>
Date:   Wed May 18 10:39:02 2022 -0400

    Add Information Gain Filtration algorithm (huggingface#16953)

    * Add information gain filtration algorithm

    * Complying with black requirements

    * Added author

    * Fixed import order

    * flake8 corrections

    Co-authored-by: Javier Turek <javier.turek@intel.com>

commit 91ede48
Author: Kamal Raj <kamalraj97@gmail.com>
Date:   Wed May 18 19:59:53 2022 +0530

    Fix typo (huggingface#17328)

commit fe28eb9
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Wed May 18 16:06:41 2022 +0200

    remove (huggingface#17325)

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

commit 2cb2ea3
Author: Nicolas Patry <patry.nicolas@protonmail.com>
Date:   Wed May 18 16:06:24 2022 +0200

    Accepting real pytorch device as arguments. (huggingface#17318)

    * Accepting real pytorch device as arguments.

    * is_torch_available.

commit 1c9d1f4
Author: Nicolas Patry <patry.nicolas@protonmail.com>
Date:   Wed May 18 15:46:12 2022 +0200

    Updating the docs for `max_seq_len` in QA pipeline (huggingface#17316)

commit 60ad734
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Wed May 18 15:08:56 2022 +0200

    [T5] Fix init in TF and Flax for pretraining (huggingface#17294)

    * fix init

    * Apply suggestions from code review

    * fix

    * finish

    * Update src/transformers/modeling_tf_utils.py

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

commit 7ba1d4e
Author: Joaq <55513213+jQuinRivero@users.noreply.github.com>
Date:   Wed May 18 09:23:47 2022 -0300

    Add type hints for ProphetNet (Pytorch) (huggingface#17223)

    * added type hints to prophetnet

    * reformatted with black

    * fix bc black misformatted some parts

    * fix imports

    * fix imports

    * Update src/transformers/models/prophetnet/configuration_prophetnet.py

    Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>

    * update OPTIONAL type hint and docstring

    Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>

commit d6b8e9c
Author: Carl <carl.cochet@gmail.com>
Date:   Wed May 18 01:07:43 2022 +0200

    Add trajectory transformer (huggingface#17141)

    * Add trajectory transformer

    Fix model init

    Fix end of lines for .mdx files

    Add trajectory transformer model to toctree

    Add forward input docs

    Fix docs, remove prints, simplify prediction test

    Apply suggestions from code review

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
    Apply suggestions from code review

    Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
    Update docs, more descriptive comments

    Apply suggestions from code review

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
    Update readme

    Small comment update and add conversion script

    Rebase and reformat

    Fix copies

    Fix rebase, remove duplicates

    Fix rebase, remove duplicates

    * Remove tapex

    * Remove tapex

    * Remove tapex

commit c352640
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Wed May 18 00:34:31 2022 +0200

    fix (huggingface#17310)

commit d9050dc
Author: Cesare Campagnano <cesare.campagnano@gmail.com>
Date:   Tue May 17 23:44:37 2022 +0200

    [LED] fix global_attention_mask not being passed for generation and docs clarification about grad checkpointing (huggingface#17112)

    * [LED] fixed global_attention_mask not passed for generation + docs clarification for gradient checkpointing

    * LED docs clarification

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    * [LED] gradient_checkpointing=True should be passed to TrainingArguments

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    * [LED] docs: remove wrong word

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    * [LED] docs fix typo

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

commit bad3583
Author: Jean Vancoppenolle <jean.vcop@gmail.com>
Date:   Tue May 17 23:42:14 2022 +0200

    Add support for pretraining recurring span selection to Splinter (huggingface#17247)

    * Add SplinterForSpanSelection for pre-training recurring span selection.

    * Formatting.

    * Rename SplinterForSpanSelection to SplinterForPreTraining.

    * Ensure repo consistency

    * Fixup changes

    * Address SplinterForPreTraining PR comments

    * Incorporate feedback and derive multiple question tokens per example.

    * Update src/transformers/models/splinter/modeling_splinter.py

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    * Update src/transformers/models/splinter/modeling_splinter.py

    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

    Co-authored-by: Jean Vancoppenole <jean.vancoppenolle@retresco.de>
    Co-authored-by: Tobias Günther <tobias.guenther@retresco.de>
    Co-authored-by: Tobias Günther <github@tobigue.de>
    Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

commit 0511305
Author: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Date:   Tue May 17 18:56:58 2022 +0200

    Add PR author in CI report + merged by info (huggingface#17298)

    * Add author info to CI report

    * Add merged by info

    * update

    Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

commit 032d63b
Author: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Date:   Tue May 17 12:56:24 2022 -0400

    Fix dummy creation script (huggingface#17304)

commit 986dd5c
Author: Sylvain Gugger <Sylvain.gugger@gmail.com>
Date:   Tue May 17 12:50:14 2022 -0400

    Fix style

commit 38ddab1
Author: Karim Foda <35491698+KMFODA@users.noreply.github.com>
Date:   Tue May 17 09:32:12 2022 -0700

    Doctest longformer (huggingface#16441)

    * Add initial doctring changes

    * make fixup

    * Add TF doc changes

    * fix seq classifier output

    * fix quality errors

    * t

    * swithc head to random init

    * Fix expected outputs

    * Update src/transformers/models/longformer/modeling_longformer.py

    Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>

    Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>

commit 10704e1
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Tue May 17 18:20:36 2022 +0200

    [Test] Fix W2V-Conformer integration test (huggingface#17303)

    * [Test] Fix W2V-Conformer integration test

    * correct w2v2

    * up

commit 28a0811
Author: regisss <15324346+regisss@users.noreply.github.com>
Date:   Tue May 17 17:58:14 2022 +0200

    Improve mismatched sizes management when loading a pretrained model (huggingface#17257)

    - Add --ignore_mismatched_sizes argument to classification examples

    - Expand the error message when loading a model whose head dimensions are different from expected dimensions

commit 1f13ba8
Author: Patrick von Platen <patrick.v.platen@gmail.com>
Date:   Tue May 17 15:48:23 2022 +0200

    correct opt (huggingface#17301)

commit 349f1c8
Author: Matt <Rocketknight1@users.noreply.github.com>
Date:   Tue May 17 14:36:23 2022 +0100

    Rewrite TensorFlow train_step and test_step (huggingface#17057)

    * Initial commit

    * Better label renaming

    * Remove breakpoint before pushing (this is your job)

    * Test a lot more in the Keras fit() test

    * make fixup

    * Clarify the case where we flatten y dicts into tensors

    * Clarify the case where we flatten y dicts into tensors

    * Extract label name remapping to a method

commit 651e48e
Author: Matt <Rocketknight1@users.noreply.github.com>
Date:   Tue May 17 14:14:17 2022 +0100

    Fix tests of mixed precision now that experimental is deprecated (huggingface#17300)

    * Fix tests of mixed precision now that experimental is deprecated

    * Fix mixed precision in training_args_tf.py too

commit 6d21142
Author: SaulLu <55560583+SaulLu@users.noreply.github.com>
Date:   Tue May 17 14:33:13 2022 +0200

    fix retribert's `test_torch_encode_plus_sent_to_model` (huggingface#17231)
elusenji pushed a commit to elusenji/transformers that referenced this pull request Jun 12, 2022
@patrickvonplaten @elusenji
[Generation] Fix Transition probs (huggingface#17311)
6714fcc 
* [Draft] fix transition probs

* up

* up

* up

* make it work

* fix

* finish

* update
@adamkhakhar
Copy link

Hello,
Regarding the follow up to these 3 posts: https://discuss.huggingface.co/t/generation-probabilities-how-to-compute-probabilities-of-output-scores-for-gpt2/3175/23, https://discuss.huggingface.co/t/retrieving-probability-over-tokens-during-beam-search/21217, https://discuss.huggingface.co/t/get-top-k-tokens-for-each-time-step-instead-of-the-highest-probability-token/14032:

Can you show us, step by step, how to retrieve the probability for each token generated for each prediction in a transformers.generation_utils.BeamSearchEncoderDecoderOutput?

Much appreciated.

@gante
Copy link
Member

gante commented Aug 3, 2022

Hi @adamkhakhar -- it is in our plans to write a tutorial for that :) It probably won't happen in the next 1 or 2 months, but stay tuned 🙏

@MotzWanted
Copy link

Hi @patrickvonplaten and @gante; what is the perspective on the tutorial?

@gante
Copy link
Member

gante commented Sep 28, 2022

No news yet :)

@MotzWanted
Copy link

Thanks for the quick response @gante. I will stay tuned. Can you tell me whether you can expect the first generated sequence to always have the highest probability?

@gante
Copy link
Member

gante commented Sep 28, 2022

@MotzWanted yeah, if you use num_return_sequences >1 with beam search, the output sequences are sorted by their score (from highest to lowest)

@MotzWanted
Copy link

But if I replicate this showcase of @patrickvonplaten to compute the sequence probabilities of the output scores, I don't get that output sequences are sorted by their probability (from highest to lowest). Can you explain to me why this is so @gante?

@gante
Copy link
Member

gante commented Sep 29, 2022 •
edited
Loading

@MotzWanted my reply was given in the context of this thread, which is related to beam_search. Any call with do_sample=True, as in the example you linked, will lose the sorting property.

A working example:

import torch
from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer


gpt2 = AutoModelForCausalLM.from_pretrained("gpt2", return_dict_in_generate=True)
tokenizer = AutoTokenizer.from_pretrained("gpt2")

input_ids = tokenizer("Today is a nice day", return_tensors="pt").input_ids

generated_outputs = gpt2.generate(input_ids, num_return_sequences=3, num_beams=3, output_scores=True)
print(generated_outputs.sequences_scores)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gante gante gante approved these changes

@patil-suraj patil-suraj Awaiting requested review from patil-suraj

+1 more reviewer

@sgugger sgugger sgugger approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Mismatch between beam search score transition probabilities and beam sequence scores
6 participants
@patrickvonplaten @HuggingFaceDocBuilderDev @adamkhakhar @gante @MotzWanted @sgugger