skill: Add skills for problem solving #428
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
7956699 to
b33cf94
Compare
mingxzhao
requested changes
Mar 25, 2024
69a02c9 to
9feb6d6
Compare
@mingxzhao Appreciate the review. I added two more examples to the skill. |
|
|
Thank you for updating, I will go ahead and mark as approved! There does seem to be a couple of warnings that are marked in the file just due to the formatting around some of the colons, I do not think it will affect the training, but if you are able to fix those just to get rid of the warnings that would be appreciated, thanks. |
6b06063 to
af12fac
Compare
|
@instruct-lab-bot generate |
1 similar comment
|
@instruct-lab-bot generate |
|
Beep, boop 🤖, Generating test data for your PR with the job type: sdg-svc. Your Job ID is 162. The results will be presented below in the pull request status box. This may take several minutes... Caution Migration Alert There is an existing github Status (Generate SDG Status) present on your PR. |
|
Results for job ID: 162 using the model sdg service backend! Results can be found here. |
|
@instruct-lab-bot precheck |
|
Beep, boop 🤖, Generating test data for your PR with the job type: precheck. Your Job ID is 166. The results will be presented below in the pull request status box. This may take several minutes... |
|
Results for job ID: 166 using the model merlinite-7b! Results can be found here. |
|
@instruct-lab-bot precheck |
|
Beep, boop 🤖, Generating test data for your PR with the job type: precheck. Your Job ID is 169. The results will be presented below in the pull request status box. This may take several minutes... |
|
Results for job ID: 169 using the model merlinite-7b! Results can be found here. |
|
@instruct-lab-bot precheck |
|
Beep, boop 🤖, Generating test data for your PR with the job type: precheck. Your Job ID is 170. The results will be presented below in the pull request status box. This may take several minutes... |
|
@instruct-lab-bot precheck |
|
Beep, boop 🤖, Generating test data for your PR with the job type: precheck. Your Job ID is 171. The results will be presented below in the pull request status box. This may take several minutes... |
|
Results for job ID: 171 using the model merlinite-7b! Results can be found here. |
|
@instruct-lab-bot precheck |
|
Beep, boop 🤖, Generating test data for your PR with the job type: precheck. Your Job ID is 173. The results will be presented below in the pull request status box. This may take several minutes... |
|
Results for job ID: 173 using the model merlinite-7b! Results can be found here. |
|
@instruct-lab-bot precheck |
|
Beep, boop 🤖, Generating test data for your PR with the job type: precheck. Your Job ID is 174. The results will be presented below in the pull request status box. This may take several minutes... |
|
Results for job ID: 174 using the model merlinite-7b! Results can be found here. |
|
@instruct-lab-bot precheck |
|
Beep, boop 🤖, Generating test data for your PR with the job type: precheck. Your Job ID is 175. The results will be presented below in the pull request status box. This may take several minutes... |
|
Results for job ID: 175 using the model merlinite-7b! Results can be found here. |
|
Thank you for your contribution to InstructLab! Unfortunately, once a Github repo is made public, all open PRs are automatically closed since they link against a private repo. We have detected that your PR might have been one of the ones affected by this change. If you are still interested in contributing your improvement, please fill out the following short form by no later than May 3rd, and we will get back to you with the additional steps necessary once we have had time to assess the PRs of those still interested: |
jjasghar
pushed a commit
that referenced
this pull request
Jun 5, 2024
Reopening : **#428 **Describe the contribution to the taxonomy** - This PR adds skill to solve logical and inference riddles **Input given at the prompt & Response that was received** Attempted two riddles and also tried to question the model to fix the output Example 1: ``` >>> can you solve this riddle ? "Two turkeys in front of the turkey, two turkeys behind the turkey, tell me how many total turkeys?" ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Certainly! This is an interesting riddle. The answer to "how many total turkeys" would be 10. Let me explain: │ │ │ │ The riddle states that there are two turkeys in front of the turkey and two turkeys behind the turkey. This implies that we have 1 │ │ turkey (the one in the middle) with 2 turkeys ahead and 2 more behind it. So, if we add the middle turkey to the ones in front and │ │ behind, we get a total of 10 turkeys. │ │ │ │ Here's a step-by-step breakdown: │ │ - There is 1 turkey in the middle (the one facing away from us in the initial description). │ │ - There are 2 turkeys in front of the middle turkey (the ones facing us in the initial description). │ │ - There are 2 turkeys behind the middle turkey (the ones facing away from us in the initial description). │ │ - Therefore, when we add the middle turkey to the ones in front and behind it, we get 1 (middle turkey) + 2 (front) + 2 (back) = 10 │ │ turkeys in total. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 9.359 seconds ─╯ >>> I think you got the basic math wrong, 1 +2 +2 equals to 5 and not 10. [S][default] ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Ah, I see where the confusion might have arisen. Thank you for pointing it out! You are correct that 1 (middle turkey) + 2 (front) │ │ + 2 (back) should equal 5, not 10. The initial response included an error in the explanation. I apologize for any confusion caused. │ │ The correct answer to the riddle should be 5 turkeys in total, given the conditions provided. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 3.479 seconds ─╯ >>> But the answer is still not correct, it's not necessary that this riddle assumes one turkey is in the middle. [S][default] ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ You're right; the initial riddle description did not explicitly specify the positioning of the turkeys. Let's reconsider the riddle │ │ without assuming a middle turkey. Instead, let's assume there are three groups of turkeys: one group with 2 turkeys in front, │ │ another group with 2 turkeys behind, and a single turkey that is neither in front nor behind. In this case: │ │ - There are 2 turkeys in front of some unspecified turkey (let's call it A). │ │ - There are 2 turkeys behind some unspecified turkey (let's call it B). │ │ - There is 1 turkey (turkey C) that is not in front or behind any of the other turkeys. │ │ - Therefore, when we add all the turkeys together, we get 2 (front of A) + 2 (behind B) + 1 (C) = 5 turkeys in total. │ │ │ │ I apologize for any confusion caused earlier, and I appreciate your persistence in ensuring the correct answer is provided. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 9.162 seconds ─╯ ``` It got the basic math wrong in the first response, although after hinting the model about wrong math, it does corrected it, but It was not able to solve the riddle. Example 2: ``` >>> Tom's mother has 3 sons. The names of the two sons are dick and hardy, what's the name of the third son? [S][default] ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Interesting question! However, based on the information provided, it is impossible to determine the name of Tom's third son, Dick. │ │ The names you provided, Dick and Hardy, do not match the typical naming patterns for sons, which often involve some form of │ │ repetition or variation of the father's name. Since there is no concrete information about the third son's name, I cannot provide a │ │ definitive answer. If there is additional information about Tom's family or naming traditions, I would be happy to revisit this │ │ question! │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 4.258 seconds ─╯ >>> Puzzle is not about finding sons of Tom, but sons of Tom's Mom. Can you take this information in consideration and try to solve the riddle? ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Of course! Based on your new information, I will rephrase the question to reflect Tom's mother as the subject, and focus on finding │ │ her three sons: Dick, Hardy, and the third son, whose name is still unknown to us. Thank you for providing this clarification! │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 2.306 seconds ─╯ ``` Tried to assist the model to fix it's assumptions, but didn't solve the riddle. **Response that is now received instead** After training the model with the seed examples and answer, received the following responses from the models Example 1: ``` >>> can you solve the following riddle? Two turkeys in front [S][default] of the turkey, two turkeys behind the turkey, tell me how many total turkeys? ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The answer is 5, because there are two turkeys in front, two turkeys behind, and one original turkey, making a total of five │ │ turkeys. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.254 seconds ─╯ >>> are you sure about it? you don't need to assume that you have turkey in the center, it can be at any position in the arrangement. ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ Yes, I am sure. The question asks for the total number of turkeys, which is five, regardless of their position or arrangement. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.034 seconds ─╯ >>> don't you think you can solve this riddle with 3 turkeys only? [S][default] ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The question states that there are two turkeys in front and two turkeys behind, which adds up to four turkeys. However, there is │ │ one more turkey in the middle, making a total of five turkeys. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.677 seconds ─╯ ``` In the trained model, it got the math right but it was adamant to not learn from the given seed examples, and still giving wrong example. _I would appreciate any pointers here about how to make it better learn from the seed examples._ Example 2: ``` >>> can you solve the following riddle? Tom's mother has 3 sons. [S][default] The names of the two sons are dick and hardy, what's the name of the third son? ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The answer is Tom, because the question asks for the name of the third son, and the names of the two other sons are Dick and Harry, │ │ making Tom the third son. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.360 seconds ─╯ >>> can you solve the following riddle? Ajay’s mother has 3 sons. [S][default] The names of the two sons are Vijay and Sanjay, what's the name of the third son? ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The answer is Ajay, because the question asks for the name of the third son, and the names of the two other sons are Vijay and │ │ Sanjay, making Ajay the third son. ``` **Contribution checklist** - [x ] Contribution was tested with `lab generate` - [ x] No errors or warnings were produced by `lab generate` - [ x] All [commits are signed off](https://github.com/instruct-lab/taxonomy/blob/main/CONTRIBUTING.md#legal) (DCO) - [x ] The `qna.yaml` file was [linted](https://yamllint.com) and [prettified](https://onlineyamltools.com/prettify-yaml) ([yaml-validator](https://jsonformatter.org/yaml-validator) can do both) --------- Signed-off-by: Anil Kumar Vishnoi <avishnoi@redhat.com> Signed-off-by: Anil Vishnoi <vishnoianil@gmail.com> Signed-off-by: BJ Hargrave <hargrave@us.ibm.com>
jjasghar
pushed a commit
to luke-inglis/taxonomy
that referenced
this pull request
Jun 7, 2024
) Reopening : **instructlab#428 **Describe the contribution to the taxonomy** - This PR adds skill to solve logical and inference riddles **Input given at the prompt & Response that was received** Attempted two riddles and also tried to question the model to fix the output Example 1: ``` >>> can you solve this riddle ? "Two turkeys in front of the turkey, two turkeys behind the turkey, tell me how many total turkeys?" ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Certainly! This is an interesting riddle. The answer to "how many total turkeys" would be 10. Let me explain: │ │ │ │ The riddle states that there are two turkeys in front of the turkey and two turkeys behind the turkey. This implies that we have 1 │ │ turkey (the one in the middle) with 2 turkeys ahead and 2 more behind it. So, if we add the middle turkey to the ones in front and │ │ behind, we get a total of 10 turkeys. │ │ │ │ Here's a step-by-step breakdown: │ │ - There is 1 turkey in the middle (the one facing away from us in the initial description). │ │ - There are 2 turkeys in front of the middle turkey (the ones facing us in the initial description). │ │ - There are 2 turkeys behind the middle turkey (the ones facing away from us in the initial description). │ │ - Therefore, when we add the middle turkey to the ones in front and behind it, we get 1 (middle turkey) + 2 (front) + 2 (back) = 10 │ │ turkeys in total. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 9.359 seconds ─╯ >>> I think you got the basic math wrong, 1 +2 +2 equals to 5 and not 10. [S][default] ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Ah, I see where the confusion might have arisen. Thank you for pointing it out! You are correct that 1 (middle turkey) + 2 (front) │ │ + 2 (back) should equal 5, not 10. The initial response included an error in the explanation. I apologize for any confusion caused. │ │ The correct answer to the riddle should be 5 turkeys in total, given the conditions provided. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 3.479 seconds ─╯ >>> But the answer is still not correct, it's not necessary that this riddle assumes one turkey is in the middle. [S][default] ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ You're right; the initial riddle description did not explicitly specify the positioning of the turkeys. Let's reconsider the riddle │ │ without assuming a middle turkey. Instead, let's assume there are three groups of turkeys: one group with 2 turkeys in front, │ │ another group with 2 turkeys behind, and a single turkey that is neither in front nor behind. In this case: │ │ - There are 2 turkeys in front of some unspecified turkey (let's call it A). │ │ - There are 2 turkeys behind some unspecified turkey (let's call it B). │ │ - There is 1 turkey (turkey C) that is not in front or behind any of the other turkeys. │ │ - Therefore, when we add all the turkeys together, we get 2 (front of A) + 2 (behind B) + 1 (C) = 5 turkeys in total. │ │ │ │ I apologize for any confusion caused earlier, and I appreciate your persistence in ensuring the correct answer is provided. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 9.162 seconds ─╯ ``` It got the basic math wrong in the first response, although after hinting the model about wrong math, it does corrected it, but It was not able to solve the riddle. Example 2: ``` >>> Tom's mother has 3 sons. The names of the two sons are dick and hardy, what's the name of the third son? [S][default] ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Interesting question! However, based on the information provided, it is impossible to determine the name of Tom's third son, Dick. │ │ The names you provided, Dick and Hardy, do not match the typical naming patterns for sons, which often involve some form of │ │ repetition or variation of the father's name. Since there is no concrete information about the third son's name, I cannot provide a │ │ definitive answer. If there is additional information about Tom's family or naming traditions, I would be happy to revisit this │ │ question! │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 4.258 seconds ─╯ >>> Puzzle is not about finding sons of Tom, but sons of Tom's Mom. Can you take this information in consideration and try to solve the riddle? ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Of course! Based on your new information, I will rephrase the question to reflect Tom's mother as the subject, and focus on finding │ │ her three sons: Dick, Hardy, and the third son, whose name is still unknown to us. Thank you for providing this clarification! │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 2.306 seconds ─╯ ``` Tried to assist the model to fix it's assumptions, but didn't solve the riddle. **Response that is now received instead** After training the model with the seed examples and answer, received the following responses from the models Example 1: ``` >>> can you solve the following riddle? Two turkeys in front [S][default] of the turkey, two turkeys behind the turkey, tell me how many total turkeys? ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The answer is 5, because there are two turkeys in front, two turkeys behind, and one original turkey, making a total of five │ │ turkeys. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.254 seconds ─╯ >>> are you sure about it? you don't need to assume that you have turkey in the center, it can be at any position in the arrangement. ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ Yes, I am sure. The question asks for the total number of turkeys, which is five, regardless of their position or arrangement. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.034 seconds ─╯ >>> don't you think you can solve this riddle with 3 turkeys only? [S][default] ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The question states that there are two turkeys in front and two turkeys behind, which adds up to four turkeys. However, there is │ │ one more turkey in the middle, making a total of five turkeys. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.677 seconds ─╯ ``` In the trained model, it got the math right but it was adamant to not learn from the given seed examples, and still giving wrong example. _I would appreciate any pointers here about how to make it better learn from the seed examples._ Example 2: ``` >>> can you solve the following riddle? Tom's mother has 3 sons. [S][default] The names of the two sons are dick and hardy, what's the name of the third son? ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The answer is Tom, because the question asks for the name of the third son, and the names of the two other sons are Dick and Harry, │ │ making Tom the third son. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.360 seconds ─╯ >>> can you solve the following riddle? Ajay’s mother has 3 sons. [S][default] The names of the two sons are Vijay and Sanjay, what's the name of the third son? ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The answer is Ajay, because the question asks for the name of the third son, and the names of the two other sons are Vijay and │ │ Sanjay, making Ajay the third son. ``` **Contribution checklist** - [x ] Contribution was tested with `lab generate` - [ x] No errors or warnings were produced by `lab generate` - [ x] All [commits are signed off](https://github.com/instruct-lab/taxonomy/blob/main/CONTRIBUTING.md#legal) (DCO) - [x ] The `qna.yaml` file was [linted](https://yamllint.com) and [prettified](https://onlineyamltools.com/prettify-yaml) ([yaml-validator](https://jsonformatter.org/yaml-validator) can do both) --------- Signed-off-by: Anil Kumar Vishnoi <avishnoi@redhat.com> Signed-off-by: Anil Vishnoi <vishnoianil@gmail.com> Signed-off-by: BJ Hargrave <hargrave@us.ibm.com>
jjasghar
pushed a commit
to acidonper/taxonomy
that referenced
this pull request
Jun 10, 2024
) Reopening : **instructlab#428 **Describe the contribution to the taxonomy** - This PR adds skill to solve logical and inference riddles **Input given at the prompt & Response that was received** Attempted two riddles and also tried to question the model to fix the output Example 1: ``` >>> can you solve this riddle ? "Two turkeys in front of the turkey, two turkeys behind the turkey, tell me how many total turkeys?" ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Certainly! This is an interesting riddle. The answer to "how many total turkeys" would be 10. Let me explain: │ │ │ │ The riddle states that there are two turkeys in front of the turkey and two turkeys behind the turkey. This implies that we have 1 │ │ turkey (the one in the middle) with 2 turkeys ahead and 2 more behind it. So, if we add the middle turkey to the ones in front and │ │ behind, we get a total of 10 turkeys. │ │ │ │ Here's a step-by-step breakdown: │ │ - There is 1 turkey in the middle (the one facing away from us in the initial description). │ │ - There are 2 turkeys in front of the middle turkey (the ones facing us in the initial description). │ │ - There are 2 turkeys behind the middle turkey (the ones facing away from us in the initial description). │ │ - Therefore, when we add the middle turkey to the ones in front and behind it, we get 1 (middle turkey) + 2 (front) + 2 (back) = 10 │ │ turkeys in total. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 9.359 seconds ─╯ >>> I think you got the basic math wrong, 1 +2 +2 equals to 5 and not 10. [S][default] ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Ah, I see where the confusion might have arisen. Thank you for pointing it out! You are correct that 1 (middle turkey) + 2 (front) │ │ + 2 (back) should equal 5, not 10. The initial response included an error in the explanation. I apologize for any confusion caused. │ │ The correct answer to the riddle should be 5 turkeys in total, given the conditions provided. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 3.479 seconds ─╯ >>> But the answer is still not correct, it's not necessary that this riddle assumes one turkey is in the middle. [S][default] ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ You're right; the initial riddle description did not explicitly specify the positioning of the turkeys. Let's reconsider the riddle │ │ without assuming a middle turkey. Instead, let's assume there are three groups of turkeys: one group with 2 turkeys in front, │ │ another group with 2 turkeys behind, and a single turkey that is neither in front nor behind. In this case: │ │ - There are 2 turkeys in front of some unspecified turkey (let's call it A). │ │ - There are 2 turkeys behind some unspecified turkey (let's call it B). │ │ - There is 1 turkey (turkey C) that is not in front or behind any of the other turkeys. │ │ - Therefore, when we add all the turkeys together, we get 2 (front of A) + 2 (behind B) + 1 (C) = 5 turkeys in total. │ │ │ │ I apologize for any confusion caused earlier, and I appreciate your persistence in ensuring the correct answer is provided. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 9.162 seconds ─╯ ``` It got the basic math wrong in the first response, although after hinting the model about wrong math, it does corrected it, but It was not able to solve the riddle. Example 2: ``` >>> Tom's mother has 3 sons. The names of the two sons are dick and hardy, what's the name of the third son? [S][default] ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Interesting question! However, based on the information provided, it is impossible to determine the name of Tom's third son, Dick. │ │ The names you provided, Dick and Hardy, do not match the typical naming patterns for sons, which often involve some form of │ │ repetition or variation of the father's name. Since there is no concrete information about the third son's name, I cannot provide a │ │ definitive answer. If there is additional information about Tom's family or naming traditions, I would be happy to revisit this │ │ question! │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 4.258 seconds ─╯ >>> Puzzle is not about finding sons of Tom, but sons of Tom's Mom. Can you take this information in consideration and try to solve the riddle? ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Of course! Based on your new information, I will rephrase the question to reflect Tom's mother as the subject, and focus on finding │ │ her three sons: Dick, Hardy, and the third son, whose name is still unknown to us. Thank you for providing this clarification! │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 2.306 seconds ─╯ ``` Tried to assist the model to fix it's assumptions, but didn't solve the riddle. **Response that is now received instead** After training the model with the seed examples and answer, received the following responses from the models Example 1: ``` >>> can you solve the following riddle? Two turkeys in front [S][default] of the turkey, two turkeys behind the turkey, tell me how many total turkeys? ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The answer is 5, because there are two turkeys in front, two turkeys behind, and one original turkey, making a total of five │ │ turkeys. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.254 seconds ─╯ >>> are you sure about it? you don't need to assume that you have turkey in the center, it can be at any position in the arrangement. ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ Yes, I am sure. The question asks for the total number of turkeys, which is five, regardless of their position or arrangement. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.034 seconds ─╯ >>> don't you think you can solve this riddle with 3 turkeys only? [S][default] ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The question states that there are two turkeys in front and two turkeys behind, which adds up to four turkeys. However, there is │ │ one more turkey in the middle, making a total of five turkeys. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.677 seconds ─╯ ``` In the trained model, it got the math right but it was adamant to not learn from the given seed examples, and still giving wrong example. _I would appreciate any pointers here about how to make it better learn from the seed examples._ Example 2: ``` >>> can you solve the following riddle? Tom's mother has 3 sons. [S][default] The names of the two sons are dick and hardy, what's the name of the third son? ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The answer is Tom, because the question asks for the name of the third son, and the names of the two other sons are Dick and Harry, │ │ making Tom the third son. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.360 seconds ─╯ >>> can you solve the following riddle? Ajay’s mother has 3 sons. [S][default] The names of the two sons are Vijay and Sanjay, what's the name of the third son? ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The answer is Ajay, because the question asks for the name of the third son, and the names of the two other sons are Vijay and │ │ Sanjay, making Ajay the third son. ``` **Contribution checklist** - [x ] Contribution was tested with `lab generate` - [ x] No errors or warnings were produced by `lab generate` - [ x] All [commits are signed off](https://github.com/instruct-lab/taxonomy/blob/main/CONTRIBUTING.md#legal) (DCO) - [x ] The `qna.yaml` file was [linted](https://yamllint.com) and [prettified](https://onlineyamltools.com/prettify-yaml) ([yaml-validator](https://jsonformatter.org/yaml-validator) can do both) --------- Signed-off-by: Anil Kumar Vishnoi <avishnoi@redhat.com> Signed-off-by: Anil Vishnoi <vishnoianil@gmail.com> Signed-off-by: BJ Hargrave <hargrave@us.ibm.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Describe the contribution to the taxonomy
Input given at the prompt & Response that was received
Attempted two riddles and also tried to question the model to fix the output
Example 1:
It got the basic math wrong in the first response, although after hinting the model about wrong math, it does corrected it, but It was not able to solve the riddle.
Example 2:
Tried to assist the model to fix it's assumptions, but didn't solve the riddle.
Response that is now received instead
After training the model with the seed examples and answer, received the following responses from the models
Example 1:
In the trained model, it got the math right but it was adamant to not learn from the given seed examples, and still giving wrong example. I would appreciate any pointers here about how to make it better learn from the seed examples.
Example 2:
Contribution checklist
lab generatelab generateqna.yamlfile was linted and prettified (yaml-validator can do both)