-
Notifications
You must be signed in to change notification settings - Fork 338
Remove the knowledge of ILocalKernelFinder and IRemoteKernelFinder from Notebook Controller Manager #9694
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Dang it, how do I turn it back into a draft? |
// Finding a kernel is the same no matter what the source | ||
@traceDecoratorVerbose('Find kernel spec', TraceOptions.BeforeCall | TraceOptions.Arguments) | ||
@captureTelemetry(Telemetry.KernelFinderPerf) | ||
public async findKernel( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Find is now common to local/remote
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And I think it probably didn't work correctly before? It did them separately.
).then((l) => this.finishListingKernels(l, useCache, 'remote')); | ||
} | ||
|
||
private async listKernelsUsingFinder( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Caching happens in this function. Should be the same logic you had before.
const key = `${kind}:${useCache}`; | ||
if (this.startTimeForFetching && !this.fetchingTelemetrySent.has(key)) { | ||
this.fetchingTelemetrySent.add(key); | ||
sendTelemetryEvent(Telemetry.FetchControllers, this.startTimeForFetching.elapsedTime, { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This telemetry used to be the in the NotebookControllerManager
) {} | ||
@traceDecoratorVerbose('Find kernel spec', TraceOptions.BeforeCall | TraceOptions.Arguments) | ||
@captureTelemetry(Telemetry.KernelFinderPerf) | ||
public async findKernel( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Find and caching were moved up a level.
if (this.isLocalLaunch) { | ||
return; | ||
} | ||
if (this.failedToFetchRemoteKernels) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I removed this altogether for now. I assume you're doing this differently anyway.
@@ -262,8 +263,8 @@ function sendTelemetryEventInternal<P extends IEventNamePropertyMapping, E exten | |||
|
|||
reporter.sendTelemetryEvent(eventNameSent, customProperties, measures); | |||
} | |||
|
|||
traceVerbose( | |||
traceWithLevel( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh moved this to an even higher trace level. Telemetry spew was bothering me too.
Codecov Report
@@ Coverage Diff @@
## main #9694 +/- ##
=====================================
- Coverage 63% 63% -1%
=====================================
Files 204 204
Lines 9840 9843 +3
Branches 1556 1557 +1
=====================================
+ Hits 6242 6243 +1
+ Misses 3335 3100 -235
- Partials 263 500 +237
|
@@ -486,6 +486,11 @@ export function compareKernels( | |||
return 1; | |||
} else if (b === activeInterpreterConnection) { | |||
return -1; | |||
// Interpreter should trump kernelspec | |||
} else if (a.kind === 'startUsingPythonInterpreter' && b.kind !== 'startUsingPythonInterpreter') { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Out of interest, why would this be the case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well the tests were guaranteeing that so I assume that was the wanted behavior. I think before it was just luck that it was happening.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the test that was failing it would get to this point and the two kernelspecs would be equivalent (start from kernelspec and start from interpreter). There was nothing that would distinguish them.
} | ||
|
||
// Send telemetry related to fetching the kernel connections | ||
// Send telemetry related to fetching the kernel connections. Do it here |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This part feels a lot better with your change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if you are still in Draft. But seems pretty peachy to me. NotebookControllerManager feels a good bit cleaner as the big plus.
Sorry there is one small downside that I forgot to mention. Previously we used to only validate the preferred kernel when searching through the cache. Now we validate the entire cache before returning the preferred kernel. So imagine you have 20 kernels in your cache, We will now make sure the file for all 20 intepreters exists before returning the preferred one. We used to only check for file existence on the first one. I don't think a STAT call on 20 files is going to take very long, but it is a consequence of moving the caching around. |
I think I could make it so the cache is not validated during the preferred search though. It's all internal to the kernel finder anyway. |
I think the algorithm would have to be tweaked though. Like pass the validation into the findPreferredKernel function and instead of it returning the top of the list, it validates the top until it finds one that's valid. |
But it seems like premature optimization to me. I wrote some code to test that hypothesis: var glob = require("glob");
var fsExtra = require("fs-extra");
async function run() {
// options is optional
var files = await new Promise((resolve, reject) => {
glob('./src/**/*.ts', (ex, res) => {
if (ex) {
reject(ex);
} else {
resolve(res);
}
});
});
let start = new Date().getTime();
await Promise.all(files.map(async (f) => {
await fsExtra.pathExists(f);
}));
let end = new Date().getTime();
console.log(`Total time for ${files.length} files is ${end-start}`);
}
run(); That prints out this:
So 15 ms for 771 files. So even if the user has a ridiculous number of interpreters, it's still super fast to check for existence. |
Fixes #9689
@DonJayamanne wanted to know your opinion on this.
The genesis for this is that the notebook controller manager has explicit knowledge of local kernels. For this to work in web we need to remove this knowledge.
This is my attempt to do that by:
This is just an idea, wanted to know what you thought of it.
I think right now I have validation of the cached kernels in the wrong spot. This could probably move to the ILocalKernelFinder and the IRemoteKernelFinder.