Originally reported at sage-devel:

Some machines suffer from a massive overhead in the following test.

def test(n):
    st = singular.cputime()
    ct = cputime()
    wt = walltime()
    for i in range(n):
        a = singular(i)
    print "Wall time:", walltime(wt)
    print "Total CPU:", cputime(ct)+singular.cputime(st)

On bsd.math, one gets something like

sage: test(1000)
Wall time: 0.261538028717
Total CPU: 0.331918 

On some other machines, one may get up to

sage: test(1000)
Wall time: 59.9999949932
Total CPU: 0.05 

The reports indicate that the overhead is bad on Debian and worst on Ubuntu; it seems to be independent of the CPU. So far, there is no other variety of Linux or Unix known that shows such a massive overhead.

Studying the interface code reveals that the overhead is caused by at least two calls to select.select, that are done when pexpect is waiting for the prompt:

1. In singular._synchronize()
2. If garbage collection occurs, it is waited for a prompt once for each variable that is to be deleted.
3. When the actual code is sent to singular.

This is why the overhead of the Gap interface is only one third of the above: It does not use synchronization, and if a variable is to be deleted then simply it may be overwritten when creating the next Gap element. There must be ways to improve the Singular interface, so that one or two calls to select() can be avoided.

David Kirkby suggests to try and upgrade pexpect - Sage uses version 2.0, but the current pexpect is 2.3.

I suggest that the Singular-specific problems and the pexpect upgrade are dealt with on different tickets, while this ticket focuses on the general problem of slow select() calls on some systems.

The disadvantage of my suggestion is: What could we do to overcome a system-dependent performance problem? Would there be a way to work around select()? Would it help to leave pexpect and use expect instead?

I reported the problem to the pexpect developer, but there was no answer yet. Who knows, perhaps this ticket will eventually be a "wontfix", but we should at least try...

Upstream: Reported upstream. No feedback yet.

CC: @alexanderdreyer @wjp @nexttime

Component: interfaces

Keywords: pexpect select performance Ubuntu

Issue created by migration from https://trac.sagemath.org/ticket/10294