-
Notifications
You must be signed in to change notification settings - Fork 441
Reduce number of discarded generated data samples #919
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Today `ilab generate` creates a large percentage (> 50% in my testing) of discarded examples because the examples do not match our expected format. Typically, the model fails to follow our complicated numbering scheme for instructions, inputs, and outputs. This change simplifies that scheme, taking inspiration from Emacs org-mode without any rigid adherence to it. Before this change, we expected the model to continue a format like: ``` 1. Instruction: Generate a joke involving three horses. 1. Input: <noinput> 1. Output: There once were three horses... ``` After this change, we now give the model: ``` * Task 1 ** Instruction Generate a joke involving three horses. ** Input <noinput> ** Output There once were three horses... ``` In my local testing, the model is able to replicate our new format with a much higher accuracy than the previous format, resulting in substantially fewer discarded generations due to format. Other formats I considered but decided against: - JSON was excluded because the model was not able to reliably handle quotes or special characters when generating JSON, even though overall it did seem to understand the expected format. - XML was excluded because it resulted in a more verbose prompt, consuming quite a bit more of our available context window. The model did seem to have a good grasp of XML with CDATA elements used to handle special characters. - Markdown was excluded because knowledge documents are in Markdown format, and I wanted to minimize the chances of the embedded knowledge document interfering with the expected parsing structure. Signed-off-by: Ben Browning <bbrownin@redhat.com>
Occasionally the model inserts a trailing colon in the generated example headings, so optionally allow that in the regular expression that splits based on those headings. This is another incremental decrease in the overall number of discarded instructions with this new format. Signed-off-by: Ben Browning <bbrownin@redhat.com>
I missed this in the prior commits, but this updates the stop sequence passed to the Completions API to match our new task format so that the model only generates two new instructions per run. We ask it to generate five, but seed it with two examples, it generates two, and it stops as soon as it starts generating the fifth. Signed-off-by: Ben Browning <bbrownin@redhat.com>
For the sake of anyone newly reviewing this PR, there is some additional useful context and data in the comments of #857 that demonstrates the large reduction in discarded samples this achieves. |
looks pretty promising to me, File 1 orig
new
File 2 orig
new
|
anik120
approved these changes
Apr 22, 2024
Thanks for reviewing/merging! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today
ilab generate
creates a large percentage (> 50% in my testing) of discarded examples because the examples do not match our expected format. Typically, the model fails to follow our complicated numbering scheme for instructions, inputs, and outputs. This change simplifies that scheme, taking inspiration from Emacs org-mode without any rigid adherence to it.Before this change, we expected the model to continue a format like:
After this change, we now give the model:
In my local testing, the model is able to replicate our new format with a much higher accuracy than the previous format, resulting in substantially fewer discarded generations due to format.
Other formats I considered but decided against:
Resolves #466
This is a reopened version of #857 which accidentally got closed.