skill: Add skills for problem solving (Clone of PR 428) #737

vishnoianil · 2024-04-22T05:59:40Z

Reopening : #428

Describe the contribution to the taxonomy

This PR adds skill to solve logical and inference riddles

Input given at the prompt & Response that was received

Attempted two riddles and also tried to question the model to fix the output

Example 1:

>>> can you solve this riddle ? "Two turkeys in front of the turkey, two turkeys behind the turkey, tell me how many total turkeys?"
╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮
│ Certainly! This is an interesting riddle. The answer to "how many total turkeys" would be 10. Let me explain:                       │
│                                                                                                                                     │
│ The riddle states that there are two turkeys in front of the turkey and two turkeys behind the turkey. This implies that we have 1  │
│ turkey (the one in the middle) with 2 turkeys ahead and 2 more behind it. So, if we add the middle turkey to the ones in front and  │
│ behind, we get a total of 10 turkeys.                                                                                               │
│                                                                                                                                     │
│ Here's a step-by-step breakdown:                                                                                                    │
│ - There is 1 turkey in the middle (the one facing away from us in the initial description).                                         │
│ - There are 2 turkeys in front of the middle turkey (the ones facing us in the initial description).                                │
│ - There are 2 turkeys behind the middle turkey (the ones facing away from us in the initial description).                           │
│ - Therefore, when we add the middle turkey to the ones in front and behind it, we get 1 (middle turkey) + 2 (front) + 2 (back) = 10 │
│ turkeys in total.                                                                                                                   │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 9.359 seconds ─╯
>>> I think you got the basic math wrong, 1 +2 +2 equals to 5 and not 10.                                                  [S][default]
╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮
│ Ah, I see where the confusion might have arisen. Thank you for pointing it out! You are correct that 1 (middle turkey) + 2 (front)  │
│ + 2 (back) should equal 5, not 10. The initial response included an error in the explanation. I apologize for any confusion caused. │
│ The correct answer to the riddle should be 5 turkeys in total, given the conditions provided.                                       │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 3.479 seconds ─╯
>>> But the answer is still not correct, it's not necessary that this riddle assumes one turkey is in the middle.          [S][default]
╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮
│ You're right; the initial riddle description did not explicitly specify the positioning of the turkeys. Let's reconsider the riddle │
│ without assuming a middle turkey. Instead, let's assume there are three groups of turkeys: one group with 2 turkeys in front,       │
│ another group with 2 turkeys behind, and a single turkey that is neither in front nor behind. In this case:                         │
│ - There are 2 turkeys in front of some unspecified turkey (let's call it A).                                                        │
│ - There are 2 turkeys behind some unspecified turkey (let's call it B).                                                             │
│ - There is 1 turkey (turkey C) that is not in front or behind any of the other turkeys.                                             │
│ - Therefore, when we add all the turkeys together, we get 2 (front of A) + 2 (behind B) + 1 (C) = 5 turkeys in total.               │
│                                                                                                                                     │
│ I apologize for any confusion caused earlier, and I appreciate your persistence in ensuring the correct answer is provided.         │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 9.162 seconds ─╯

It got the basic math wrong in the first response, although after hinting the model about wrong math, it does corrected it, but It was not able to solve the riddle.

Example 2:

>>> Tom's mother has 3 sons. The names of the two sons are dick and hardy, what's the name of the third son?               [S][default]
╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮
│ Interesting question! However, based on the information provided, it is impossible to determine the name of Tom's third son, Dick.  │
│ The names you provided, Dick and Hardy, do not match the typical naming patterns for sons, which often involve some form of         │
│ repetition or variation of the father's name. Since there is no concrete information about the third son's name, I cannot provide a │
│ definitive answer. If there is additional information about Tom's family or naming traditions, I would be happy to revisit this     │
│ question!                                                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 4.258 seconds ─╯
>>> Puzzle is not about finding sons of Tom, but sons of Tom's Mom. Can you take this information in consideration and try to solve the
 riddle?
╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮
│ Of course! Based on your new information, I will rephrase the question to reflect Tom's mother as the subject, and focus on finding │
│ her three sons: Dick, Hardy, and the third son, whose name is still unknown to us. Thank you for providing this clarification!      │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 2.306 seconds ─╯

Tried to assist the model to fix it's assumptions, but didn't solve the riddle.

Response that is now received instead

After training the model with the seed examples and answer, received the following responses from the models

Example 1:

>>> can you solve the following riddle? Two turkeys in front                                                               [S][default]
      of the turkey, two turkeys behind the turkey, tell me how many
      total turkeys?
╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮
│ The answer is 5, because there are two turkeys in front, two turkeys behind, and one original turkey, making a total of five        │
│ turkeys.                                                                                                                            │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.254 seconds ─╯
>>> are you sure about it? you don't need to assume that you have turkey in the center, it can be at any position in the arrangement.
╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮
│ Yes, I am sure. The question asks for the total number of turkeys, which is five, regardless of their position or arrangement.      │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.034 seconds ─╯
>>> don't you think you can solve this riddle with 3 turkeys only?                                                         [S][default]
╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮
│ The question states that there are two turkeys in front and two turkeys behind, which adds up to four turkeys. However, there is    │
│ one more turkey in the middle, making a total of five turkeys.                                                                      │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.677 seconds ─╯

In the trained model, it got the math right but it was adamant to not learn from the given seed examples, and still giving wrong example. I would appreciate any pointers here about how to make it better learn from the seed examples.

Example 2:

>>> can you solve the following riddle? Tom's mother has 3 sons.                                                           [S][default]
      The names of the two sons are dick and hardy, what's the name of
      the third son?
╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮
│ The answer is Tom, because the question asks for the name of the third son, and the names of the two other sons are Dick and Harry, │
│ making Tom the third son.                                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.360 seconds ─╯

>>> can you solve the following riddle? Ajay’s mother has 3 sons.                                                          [S][default]
      The names of the two sons are Vijay and Sanjay, what's the name of
      the third son?
╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮
│ The answer is Ajay, because the question asks for the name of the third son, and the names of the two other sons are Vijay and      │
│ Sanjay, making Ajay the third son.

Contribution checklist

[x ] Contribution was tested with lab generate
[ x] No errors or warnings were produced by lab generate
[ x] All commits are signed off (DCO)
[x ] The qna.yaml file was linted and prettified (yaml-validator can do both)

instruct-lab-bot · 2024-04-22T05:59:54Z

Beep, boop 🤖, Hi, I'm @instructlab-bot and I'm going to help you with your pull request. Thanks for you contribution! 🎉

I support the following commands:

@instructlab-bot precheck -- Check existing model behavior using the questions in this proposed change.
@instructlab-bot generate -- Generate a sample of synthetic data using the synthetic data generation backend infrastructure.
@instructlab-bot generate-local -- Generate a sample of synthetic data using a local model.
@instructlab-bot help -- Print this help message again.

Note

Results or Errors of these commands will be posted as a pull request check in the Checks section below

Note

Currently only maintainers belongs to [[taxonomy-triagers taxonomy-approvers taxonomy-maintainers labrador-org-maintainers instruct-lab-bot-maintainers]] teams are allowed to run these commands.

instruct-lab-bot · 2024-04-22T05:59:55Z

Beep, boop 🤖, Hi, I'm @instructlab-bot and I'm going to help you with your pull request. Thanks for you contribution! 🎉

I support the following commands:

@instructlab-bot precheck -- Check existing model behavior using the questions in this proposed change.
@instructlab-bot generate -- Generate a sample of synthetic data using the synthetic data generation backend infrastructure.
@instructlab-bot generate-local -- Generate a sample of synthetic data using a local model.
@instructlab-bot help -- Print this help message again.

Note

Results or Errors of these commands will be posted as a pull request check in the Checks section below

Note

Currently only maintainers belongs to [[taxonomy-triagers taxonomy-approvers taxonomy-maintainers labrador-org-maintainers instruct-lab-bot-maintainers]] teams are allowed to run these commands.

vishnoianil · 2024-04-22T06:01:47Z

>>> If you are driving the bus, and bus initially has 10 passengers.                                                         [S][default]
       At first bus stop 4 passenger gets off the bus and 3 get on to the bus.
       At second bus stop, 3 passenger get off the bus and 5 get on the bus.
       Now bus reached to it's final stop. Answer the following questions -
      Who is driving the bus and how many total passenger will get off from
      the bus at final stop?
╭───────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ─────────────────────────────────────────────────────────╮
│ The response should be improved by providing more context and clarifying the question:                                                │
│                                                                                                                                       │
│ Question: If you are driving the bus, and bus initially has 10 passengers. At first bus stop, 4 passengers get off the bus and 3 get  │
│ on to the bus. At second bus stop, 3 passengers get off the bus and 5 get on the bus. Now bus reached to its final stop. Who is       │
│ driving the bus and how many total passengers will get off from the bus at the final stop?                                            │
│                                                                                                                                       │
│ Response: The bus driver is the person responsible for operating the vehicle during its entire journey, including the final stop.     │
│ Initially, the bus had 10 passengers. After the first bus stop, there were 10 - 4 + 3 = 9 passengers remaining on the bus. After the  │
│ second bus stop, there were 9 - 3 + 5 = 7 passengers remaining on the bus. At the final stop, no further passengers got on or off the │
│ bus, so the number of passengers getting off at the final stop is equal to the number of passengers remaining on the bus, which is 7. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 8.821 seconds ─╯
>>> are you sure 9-3+5 equals to 7?                                                                                          [S][default]
╭───────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ─────────────────────────────────────────────────────────╮
│ Yes, I apologize for the confusion in my previous response. The correct calculation is as follows:                                    │
│                                                                                                                                       │
│ Initial number of passengers: 10                                                                                                      │
│ At first bus stop:                                                                                                                    │
│ -4 passengers got off: 10 - 4 = 6                                                                                                     │
│ +3 passengers got on: 6 + 3 = 9                                                                                                       │
│ At second bus stop:                                                                                                                   │
│ -3 passengers got off: 9 - 3 = 6                                                                                                      │
│ +5 passengers got on: 6 + 5 = 11                                                                                                      │
│                                                                                                                                       │
│ At the final stop, there were 11 passengers on the bus. However, since we were asked to find out how many total passengers will get   │
│ off from the bus at the final stop, we need to consider the number of passengers who got on at the beginning of the journey and will  │
│ get off at the final stop. In this case, there were no new passengers who got on at the final stop, so the number of passengers       │
│ getting off at the final stop is equal to the number of passengers who got on at the beginning of the journey and remained on the bus │
│ until the final stop. Therefore, 11 - 10 = 1 passenger will get off at the final stop.                                                │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 9.555 seconds ─╯
>>> did you figure out who is driving the bus?                                                                               [S][default]
╭───────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ─────────────────────────────────────────────────────────╮
│ The question does not provide information about who is driving the bus, so I cannot provide an answer to that question. The question  │
│ only asks about the number of passengers getting off at the final stop, which I have already answered.                                │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.767 seconds ─╯
>>> question says you are driving the bus, so it's it that you are the bus driver?                                           [S][default]
╭───────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ─────────────────────────────────────────────────────────╮
│ I apologize for any confusion in my previous responses. The question does not explicitly state that I am the bus driver. It simply    │
│ states that "if you are driving the bus." Therefore, I cannot definitively say that I am the bus driver based on the information      │
│ provided in the question.                                                                                                             │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 2.297 seconds ─╯

>>> A building has 10 floors. At the ground floor, 7 people enter the elevator. At the second floor, 3 people get off and 2 people enter.
 At the fourth floor, 5 people get off and 2 people enter. At the seventh floor, 1 person gets off and 3 people enter. Finally, at the te
nth floor, everyone remaining in the elevator gets off.

The question is: How many people were in the elevator when it started its journey, and how many total people exited the elevator by the e
nd of its journey?
╭───────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ─────────────────────────────────────────────────────────╮
│ To determine how many people were in the elevator when it started its journey, we need to track the number of people entering and     │
│ exiting at each floor.                                                                                                                │
│                                                                                                                                       │
│ At the ground floor, 7 people enter the elevator.                                                                                     │
│ At the second floor, 3 people get off, leaving 7 - 3 = 4 people in the elevator.                                                      │
│ At the fourth floor, 5 people get off, leaving 4 - 5 = -1 people in the elevator (we can't have a negative number of people, so we    │
│ must have missed something).                                                                                                          │
│                                                                                                                                       │
│ Going back to the question, at the third floor, 2 people enter the elevator. Therefore, there were actually 4 + 2 = 6 people in the   │
│ elevator when it started its journey.                                                                                                 │
│                                                                                                                                       │
│ By the end of its journey at the tenth floor, everyone remaining in the elevator got off. So, the total number of people who exited   │
│ the elevator by the end of its journey is equal to the number of people who got off at the tenth floor plus the number of people who  │
│ got off at each previous floor.                                                                                                       │
│                                                                                                                                       │
│ At the tenth floor, 1 person got off, leaving 6 - 1 = 5 people in the elevator.                                                       │
│ At the seventh floor, 3 people got off, leaving 5 - 3 = 2 people in the elevator.                                                     │
│ At the fourth floor, 5 people got off, leaving 2 - 5 = -3 people in the elevator (again, we can't have a negative number of people).  │
│                                                                                                                                       │
│ Going back to the question, at the third floor, 2 people entered the elevator. Therefore, a total of 5 + 2 = 7 people exited the      │
│ elevator by the end of its journey.                                                                                                   │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 15.411 seconds ─╯

jjasghar · 2024-04-22T17:36:53Z

@instructlab-bot precheck

instruct-lab-bot · 2024-04-22T17:36:55Z

Beep, boop 🤖, Generating test data for your PR with the job type: precheck. Your Job ID is 185. The results will be presented below in the pull request status box. This may take several minutes...

jjasghar · 2024-04-22T17:39:36Z

Can you please fix the DCO on this PR?

instruct-lab-bot · 2024-04-22T17:39:50Z

Beep, boop 🤖, Hi, I'm @instructlab-bot and I'm going to help you with your pull request. Thanks for you contribution! 🎉

I support the following commands:

@instructlab-bot precheck -- Check existing model behavior using the questions in this proposed change.
@instructlab-bot generate -- Generate a sample of synthetic data using the synthetic data generation backend infrastructure.
@instructlab-bot generate-local -- Generate a sample of synthetic data using a local model.
@instructlab-bot help -- Print this help message again.

Note

Results or Errors of these commands will be posted as a pull request check in the Checks section below

Note

Currently only maintainers belongs to [[taxonomy-triagers taxonomy-approvers taxonomy-maintainers labrador-org-maintainers instruct-lab-bot-maintainers]] teams are allowed to run these commands.

instruct-lab-bot · 2024-04-22T17:53:57Z

Beep, boop 🤖, Hi, I'm @instructlab-bot and I'm going to help you with your pull request. Thanks for you contribution! 🎉

I support the following commands:

@instructlab-bot precheck -- Check existing model behavior using the questions in this proposed change.
@instructlab-bot generate -- Generate a sample of synthetic data using the synthetic data generation backend infrastructure.
@instructlab-bot generate-local -- Generate a sample of synthetic data using a local model.
@instructlab-bot help -- Print this help message again.

Note

Results or Errors of these commands will be posted as a pull request check in the Checks section below

Note

Currently only maintainers belongs to [[taxonomy-triagers taxonomy-approvers taxonomy-maintainers labrador-org-maintainers instruct-lab-bot-maintainers]] teams are allowed to run these commands.

vishnoianil · 2024-04-22T17:55:11Z

Can you please fix the DCO on this PR?

Done

jjasghar · 2024-04-22T18:12:24Z

@instructlab-bot precheck

instruct-lab-bot · 2024-04-22T18:12:27Z

Beep, boop 🤖, Generating test data for your PR with the job type: precheck. Your Job ID is 191. The results will be presented below in the pull request status box. This may take several minutes...

instruct-lab-bot · 2024-04-22T18:50:59Z

Results for job ID: 191 using the model merlinite-7b!

Results can be found here.

Signed-off-by: Anil Kumar Vishnoi <avishnoi@redhat.com> Signed-off-by: Anil Vishnoi <vishnoianil@gmail.com> Signed-off-by: BJ Hargrave <hargrave@us.ibm.com>

instruct-lab-bot · 2024-04-24T19:41:22Z

Beep, boop 🤖, Hi, I'm @instructlab-bot and I'm going to help you with your pull request. Thanks for you contribution! 🎉

I support the following commands:

@instructlab-bot precheck -- Check existing model behavior using the questions in this proposed change.
@instructlab-bot generate -- Generate a sample of synthetic data using the synthetic data generation backend infrastructure.
@instructlab-bot generate-local -- Generate a sample of synthetic data using a local model.
@instructlab-bot help -- Print this help message again.

Note

Results or Errors of these commands will be posted as a pull request check in the Checks section below

Note

Currently only maintainers belongs to [[taxonomy-triagers taxonomy-approvers taxonomy-maintainers labrador-org-maintainers instruct-lab-bot-maintainers]] teams are allowed to run these commands.

alinaryan · 2024-06-05T19:29:22Z

Thank you for your submission. Here are your results:
Your PR improved the granite-7b-lab model's performance performance by 1.0 points.
The model's new score is: 6.64
The old score was: 5.64
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 66.39999999999999%

alinaryan · 2024-06-06T15:52:07Z

Thank you for your submission. Here are your results:
Your PR decreased the merlinite-7b-lab model's performance by 0.4800000000000004 points.
The model's new score is: 5.06
The old score was: 5.54
||||||||||||||||||||||||||||||||||||||||||||||||||| 50.6%

) Reopening : **instructlab#428 **Describe the contribution to the taxonomy** - This PR adds skill to solve logical and inference riddles **Input given at the prompt & Response that was received** Attempted two riddles and also tried to question the model to fix the output Example 1: ``` >>> can you solve this riddle ? "Two turkeys in front of the turkey, two turkeys behind the turkey, tell me how many total turkeys?" ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Certainly! This is an interesting riddle. The answer to "how many total turkeys" would be 10. Let me explain: │ │ │ │ The riddle states that there are two turkeys in front of the turkey and two turkeys behind the turkey. This implies that we have 1 │ │ turkey (the one in the middle) with 2 turkeys ahead and 2 more behind it. So, if we add the middle turkey to the ones in front and │ │ behind, we get a total of 10 turkeys. │ │ │ │ Here's a step-by-step breakdown: │ │ - There is 1 turkey in the middle (the one facing away from us in the initial description). │ │ - There are 2 turkeys in front of the middle turkey (the ones facing us in the initial description). │ │ - There are 2 turkeys behind the middle turkey (the ones facing away from us in the initial description). │ │ - Therefore, when we add the middle turkey to the ones in front and behind it, we get 1 (middle turkey) + 2 (front) + 2 (back) = 10 │ │ turkeys in total. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 9.359 seconds ─╯ >>> I think you got the basic math wrong, 1 +2 +2 equals to 5 and not 10. [S][default] ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Ah, I see where the confusion might have arisen. Thank you for pointing it out! You are correct that 1 (middle turkey) + 2 (front) │ │ + 2 (back) should equal 5, not 10. The initial response included an error in the explanation. I apologize for any confusion caused. │ │ The correct answer to the riddle should be 5 turkeys in total, given the conditions provided. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 3.479 seconds ─╯ >>> But the answer is still not correct, it's not necessary that this riddle assumes one turkey is in the middle. [S][default] ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ You're right; the initial riddle description did not explicitly specify the positioning of the turkeys. Let's reconsider the riddle │ │ without assuming a middle turkey. Instead, let's assume there are three groups of turkeys: one group with 2 turkeys in front, │ │ another group with 2 turkeys behind, and a single turkey that is neither in front nor behind. In this case: │ │ - There are 2 turkeys in front of some unspecified turkey (let's call it A). │ │ - There are 2 turkeys behind some unspecified turkey (let's call it B). │ │ - There is 1 turkey (turkey C) that is not in front or behind any of the other turkeys. │ │ - Therefore, when we add all the turkeys together, we get 2 (front of A) + 2 (behind B) + 1 (C) = 5 turkeys in total. │ │ │ │ I apologize for any confusion caused earlier, and I appreciate your persistence in ensuring the correct answer is provided. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 9.162 seconds ─╯ ``` It got the basic math wrong in the first response, although after hinting the model about wrong math, it does corrected it, but It was not able to solve the riddle. Example 2: ``` >>> Tom's mother has 3 sons. The names of the two sons are dick and hardy, what's the name of the third son? [S][default] ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Interesting question! However, based on the information provided, it is impossible to determine the name of Tom's third son, Dick. │ │ The names you provided, Dick and Hardy, do not match the typical naming patterns for sons, which often involve some form of │ │ repetition or variation of the father's name. Since there is no concrete information about the third son's name, I cannot provide a │ │ definitive answer. If there is additional information about Tom's family or naming traditions, I would be happy to revisit this │ │ question! │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 4.258 seconds ─╯ >>> Puzzle is not about finding sons of Tom, but sons of Tom's Mom. Can you take this information in consideration and try to solve the riddle? ╭──────────────────────────────────────────────────────── merlinite-7b-Q4_K_M ────────────────────────────────────────────────────────╮ │ Of course! Based on your new information, I will rephrase the question to reflect Tom's mother as the subject, and focus on finding │ │ her three sons: Dick, Hardy, and the third son, whose name is still unknown to us. Thank you for providing this clarification! │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 2.306 seconds ─╯ ``` Tried to assist the model to fix it's assumptions, but didn't solve the riddle. **Response that is now received instead** After training the model with the seed examples and answer, received the following responses from the models Example 1: ``` >>> can you solve the following riddle? Two turkeys in front [S][default] of the turkey, two turkeys behind the turkey, tell me how many total turkeys? ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The answer is 5, because there are two turkeys in front, two turkeys behind, and one original turkey, making a total of five │ │ turkeys. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.254 seconds ─╯ >>> are you sure about it? you don't need to assume that you have turkey in the center, it can be at any position in the arrangement. ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ Yes, I am sure. The question asks for the total number of turkeys, which is five, regardless of their position or arrangement. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.034 seconds ─╯ >>> don't you think you can solve this riddle with 3 turkeys only? [S][default] ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The question states that there are two turkeys in front and two turkeys behind, which adds up to four turkeys. However, there is │ │ one more turkey in the middle, making a total of five turkeys. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.677 seconds ─╯ ``` In the trained model, it got the math right but it was adamant to not learn from the given seed examples, and still giving wrong example. _I would appreciate any pointers here about how to make it better learn from the seed examples._ Example 2: ``` >>> can you solve the following riddle? Tom's mother has 3 sons. [S][default] The names of the two sons are dick and hardy, what's the name of the third son? ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The answer is Tom, because the question asks for the name of the third son, and the names of the two other sons are Dick and Harry, │ │ making Tom the third son. │ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────── elapsed 1.360 seconds ─╯ >>> can you solve the following riddle? Ajay’s mother has 3 sons. [S][default] The names of the two sons are Vijay and Sanjay, what's the name of the third son? ╭───────────────────────────────────── ./ibm-merlinite-7b-mlx-q-fused-pt/ggml-model-Q4_K_M.gguf ──────────────────────────────────────╮ │ The answer is Ajay, because the question asks for the name of the third son, and the names of the two other sons are Vijay and │ │ Sanjay, making Ajay the third son. ``` **Contribution checklist** - [x ] Contribution was tested with `lab generate` - [ x] No errors or warnings were produced by `lab generate` - [ x] All [commits are signed off](https://github.com/instruct-lab/taxonomy/blob/main/CONTRIBUTING.md#legal) (DCO) - [x ] The `qna.yaml` file was [linted](https://yamllint.com) and [prettified](https://onlineyamltools.com/prettify-yaml) ([yaml-validator](https://jsonformatter.org/yaml-validator) can do both) --------- Signed-off-by: Anil Kumar Vishnoi <avishnoi@redhat.com> Signed-off-by: Anil Vishnoi <vishnoianil@gmail.com> Signed-off-by: BJ Hargrave <hargrave@us.ibm.com>

vishnoianil requested a review from a team as a code owner April 22, 2024 05:59

github-actions bot added triage-needed (Auto labeled) skill is ready to be triaged skill (Auto labeled) labels Apr 22, 2024

vishnoianil changed the title ~~Riddles~~ skill: Add skills for problem solving (Clone of PR 428) Apr 22, 2024

jjasghar added triage-requested-changes skill has been reviewed; changes requested from contributor and removed triage-needed (Auto labeled) skill is ready to be triaged labels Apr 22, 2024

github-actions bot added the triage-needed (Auto labeled) skill is ready to be triaged label Apr 22, 2024

jjasghar removed the triage-requested-changes skill has been reviewed; changes requested from contributor label Apr 22, 2024

vishnoianil marked this pull request as draft April 23, 2024 04:47

RobotSail force-pushed the main branch from 825fde4 to 9ae7bfa Compare April 23, 2024 17:40

instructlab deleted a comment from nerdalert Apr 23, 2024