Both these notations result in incorrect output:
(function(){return"test"}())
(function(){return"test"})()
The compressed output is:
!function(){return"test"}();
Which evaluates to false while the original code evaluates to "test".
Note that the output length is the same as the input length.
I guess the output makes sense because the result of the function call isn't assigned to anything so it doesn't matter what the code evaluates to. The problem is that running the compressed code through eval() will return the wrong result.

I believe the right solution would be to leave the parentheses untouched in those cases where it's the last unnested expression.