I think the functional test failure is flakiness in the tests. See #748.

Updated to use patterns instead of pattern for knownledge schema. See instructlab/taxonomy#654 and instructlab/schema#3 (comment).

Updated to use patterns instead of pattern for knownledge schema. See instruct-lab/taxonomy#654 and instruct-lab/schema#3 (comment).

@bjhargrave this needs to be updated too

I'm adding this to the next milestone

this needs to be updated too

Done in b3669de#diff-c930cb39432fe5403f7d83393641f63b81edd36cdd0a7788eb6f3548dee1a13fR186

@bjhargrave doing a quick PR #791 to get us moving with patterns

doing a quick PR #791 to get us moving with patterns

Great. I rebased.

@xukai92 The packaging issue which caused the functional test failure is now fixed.

Capturing a discussion with @shivchander:

I was writing up a test case for the cli to test knowledge workflow, but the way that I laid out my qna.yaml as follows:

test_knowledge_valid = b"""created_by: test-bot
seed_examples:
- question: What is Operator Framework? 
  answer: 'The Operator Framework is a set of Kubernetes components and developer tools, 
  that aid in Operator development and central management on a multi-tenant cluster.'
- question: What is an Operator? 
  answer: 'The goal of an Operator is to put operational knowledge into software. 
  Previously this knowledge only resided in the minds of administrators, 
  various combinations of shell scripts or automation software like Ansible. 
  It was outside of your Kubernetes cluster and hard to integrate. 
  With Operators, CoreOS changed that. Operators implement and automate 
  common Day-1 (installation, configuration, etc.) and Day-2 (re-configuration, 
  update, backup, failover, restore, etc.) activities in a piece of software running 
  inside your Kubernetes cluster, by integrating natively with Kubernetes concepts and APIs. 
  We call this a Kubernetes-native application. 
  With Operators you can stop treating an application as a collection of primitives like Pods, 
  Deployments, Services or ConfigMaps, but instead as a single object that only exposes the knobs 
  that make sense for the application.'
- question: What is Operator Lifecycle Manager? 
  answer: 'OLM is a component of the Operator Framework, 
  an open source toolkit to manage Kubernetes native applications, 
  called Operators, in an effective, automated, and scalable way. 
  OLM extends Kubernetes to provide a declarative way to install, 
  manage, and upgrade Operators and their dependencies in a cluster.
task_description: to teach a large language model about the Operator Framework
document:
  repo: https://github.com/anik120/knowledge-doc-test
  commit: bf78d868f544e55d8e1d99f68d9105fc3b8751bd
  patterns:
  - operator-framework*.md

Essentially, the seed_example question/answers I have there are from our overarching project websites https://operatorframework.io, https://olm.operatorframework.io and https://sdk.operatorframework.io, and the documents I have in https://github.com/anik120/knowledge-doc-test are README.mds from our components' GitHub repository. In other words, the seed_example question/answers do not actually come from the documents hosted in document.repo.

The way I laid things out, the seed_examples are "product pitch/summary description" and document.repo contains all the docs I want the model to learn about.

Shiv tells me that that's the wrong way of thinking about it, and the verb document should be source in reality, and seed_examples are examples of questions and answers that can be answered once the model is trained on the docs hosted in docs.repo.

Eureka moment: Even after learning (only a little while ago, ie fresh info being processed by brain still) how the taxonomy interacts with the model, I was thinking about the structure of my knowledge doc, the wrong way. It's likely that other users will also confuse the taxonomy/model interactions and lay out the qna.yaml files the wrong way, leading to PR submissions that'll likely not improve model quality.

Proposed fix: Change document to source