Thanks, @lalitpagaria . As per the discussion on #153
First thing to do is integrate the #160 TextSplitter changes with Analyzers , change the Analyzer to work with TextSplitter for texts with length greater than 512 . Right now texts greater than max length are truncated ( see #113)

Then For InterfaceAggregator, I have two approaches in mind:

merge the output of the analyzer by simply averaging out the results of splits, I want to try out a simple count approach. The output label of text would be the label with the highest count among splits.
Another approach would be to combine the softmax logits outputs of N*K classes as mentioned in https://discuss.huggingface.co/t/new-pipeline-for-zero-shot-text-classification/681/84 discussion. This approach would need more clarity and thinking.

So let's have two tasks in this Issue :

Integrating TextSplitter and Analyzer
InterfaceAggregator
The first is something I've already started and then will continue on InterfaceAgrregator . Will share the results of both integration approaches so we can decide which one to move forward with.
Kindly assign this to me

Add InferenceAggregator node #163

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions