Possible issue with DynamicTimeWarp kernel class #717
Description
Sorry for the long post in advance.
First of all let me start by saying that this is an issue being opened after a brief interaction with Cesar in stackoverflow that can be fount in the following link.
Some parts of this post will be from that thread.
Just to summarize.
I'm trying to create a prototype of something that can be compared to a touchpad, with the ability to identify gestures like swipe left, swipe right, double tap, etc.
While researching for a way to do this I came accross machine learning and Dynamic Time Warping.
Since my project was in .NET I decided to give Accord.NET a shot.
I started by creating a blank project and following the example on this link.
http://accord-framework.net/docs/html/T_Accord_Statistics_Kernels_DynamicTimeWarping.htm
With the help of Cesar I adapted that binary learning problem into a multi-class one.
Next, to be able to reject accidental gestures, I had to use the .Probability or .Score methods of the SVM and accept results above a certain threshold. Cesar suggested calibrating the machine
using ProbabilisticOutputCalibration.
The exampe code I used is from the second example on the following link.
After doing this I came accross some inconsistent results.
The best way to understand what I'm about to explain is to refer back to the code on the bottom.
Let's say I have an array of gestures that I use to train my algorythm.
If after the calibration I calculate the probability of the first gesture using
machine.Probability(gestures[0], out decision1);
I get the right decision, although the probability is only approximatelly 0.67.
If after this I try to evaluate gestures[0] again I get the same result, as expected.
Now comes the weird part, if I create a new instance of a gesture (named swipeLeftGesture in the code) with the same values as gestures[0] the probability lowers to around 0.35.
If I use .Score instead of .Probability I also get inconsistent results.
You can use the following code to see this happening.
I didn't print the values to console, just place a breakpoint at the end of the Main method and inspect de variables.
************************ Source Code ************************
namespace DynamicTimeWarpingTest
{
public class Program
{
public static void Main(string[] args)
{
double[][][] gestures =
{
new double[][] // Swipe left
{
new double[] { 1, 1, 1 },
new double[] { 1, 2, 1 },
new double[] { 1, 2, 2 },
new double[] { 2, 2, 2 },
},
new double[][] // Swipe right
{
new double[] { 1, 10, 6 },
new double[] { 1, 5, 6 },
new double[] { 6, 7, 1 },
},
new double[][] // Double tap
{
new double[] { 8, 2, 5 },
new double[] { 1, 50, 4 },
}
};
int[] outputs =
{
0, // Swipe left
1, // Swipe right
2 // Double tap
};
var teacher = new MulticlassSupportVectorLearning<DynamicTimeWarping, double[][]>()
{
// Configure the learning algorithm to use SMO to train the
// underlying SVMs in each of the binary class subproblems.
Learner = (param) => new SequentialMinimalOptimization<DynamicTimeWarping, double[][]>
{
Complexity = 1.5,
Kernel = new DynamicTimeWarping(alpha: 1, degree: 1),
//UseKernelEstimation = true
}
};
// Learn a machine
var machine = teacher.Learn(gestures, outputs);
// Create the multi-class learning algorithm for the machine
var calibration = new MulticlassSupportVectorLearning<DynamicTimeWarping, double[][]>()
{
Model = machine, // We will start with an existing machine
// Configure the learning algorithm to use Platt's calibration
Learner = (param) => new ProbabilisticOutputCalibration<DynamicTimeWarping, double[][]>()
{
Model = param.Model // Start with an existing machine
}
};
// Configure parallel execution options
calibration.ParallelOptions.MaxDegreeOfParallelism = 1;
// Learn a machine
calibration.Learn(gestures, outputs);
// Validate Results
double decision1, decision2, decision3, decision4, decision5, decision6;
// First create new instances for the gestures with the same values as in the sequences array
var swipeLeftGesture = new double[][]
{
new double[] { 1, 1, 1 },
new double[] { 1, 2, 1 },
new double[] { 1, 2, 2 },
new double[] { 2, 2, 2 },
};
var swipeRightGesture = new double[][]
{
new double[] { 1, 10, 6 },
new double[] { 1, 5, 6 },
new double[] { 6, 7, 1 },
};
var doubleTapGesture = new double[][]
{
new double[] { 8, 2, 5 },
new double[] { 1, 50, 4 },
};
// Check what are the decisions and probabilities of the original sequences array
var res1 = machine.Probability(gestures[0], out decision1); // decision 0 - Probability 0.66666666570779487 - Score 0.69314717840248374
var res2 = machine.Probability(gestures[1], out decision2); // decision 1 - Probability 0.57647717384694053 - Score 0.6931471784024833
var res3 = machine.Probability(gestures[2], out decision3); // decision 2 - Probability 0.66666666570779465 - Score 0.6931471784024833
// Check what are the decisions and probabilities of the newly created instances
var res4 = machine.Probability(swipeLeftGesture, out decision4); // decision 0 - Probability 0.34881631999941815 - Score 0.035525143563626807
var res5 = machine.Probability(swipeRightGesture, out decision5); // decision 1 - Probability 0.38414838905152121 - Score 0.13802120233728751
var res6 = machine.Probability(doubleTapGesture, out decision6); // decision 2 - Probability 0.607525583968486 - Score 0.56625552124018352
}
}
}