About the tests that fail:

AssertionError: 
Arrays are not almost equal to 6 decimals

Mismatched elements: 5 / 5 (100%)
Max absolute difference: 0.002462
Max relative difference: 0.00493456
 x: array([0.5     , 0.501392, 0.499438, 0.5     , 0.501738], dtype=float32)
 y: array([0.499986, 0.49893 , 0.499301, 0.500023, 0.500554], dtype=float

If the error is caused by the difference, I think we should introduce a threshold here. Since CNN / most neural have randomly initialised initial state, it is not uncommon to get different results (and in this case its a very minor difference).

Just a note, there are 2 tests which are failing, and both of them are related to idempotent matrices. It feels odd that only those two cases are failing, but I do not understand why its just those two matrices.

Something I found out about reproduciblity in keras.
official docs, stack overflow answer

In short, to be absolutely sure that you will get reproducible results with your python script on one computer's/laptop's CPU then you will have to do the following:

1. Set PYTHONHASHSEED environment variable at a fixed value
2. Set python built-in pseudo-random generator at a fixed value
3. Set numpy pseudo-random generator at a fixed value
4. Set tensorflow pseudo-random generator at a fixed value
5. Configure a new global tensorflow session

I can test this out to see if it also works on windows 🤞🏽
Though I do think we need to redo our tests since:

Moreover, when running on a GPU, some operations have non-deterministic outputs, in particular tf.reduce_sum(). This is due to the fact that GPUs run many operations in parallel, so the order of execution is not always guaranteed. Due to the limited precision of floats, even adding several numbers together may give slightly different results depending on the order in which you add them. You can try to avoid the non-deterministic operations, but some may be created automatically by TensorFlow to compute the gradients

So when we do implement the added functionality of GPU compute, the test is meant to fail.

hi, sorry for the delay in looking at this. I have looked back at CNNClassifier, and basically I found the same. keras is non deterministic, and even if you could fix all seeds and reproduce results, fixing a global variable like numpy.random in one classifier seems like it may have nasty ramifications down the line.

My decision was to just not do the correctness tests for CNNClassifier. Instead, I have run the whole lot on the UCR data and can show no significant difference to the published results with a fixed version of sktime. If we ever have a correctness problem, we can compare back to this. I can do the same for the Regressor, and the other deep learners. Note I also excluded these two test

# test fail with deep problem with pickling inside tensorflow.
"CNNClassifier": [
    "test_fit_idempotent",
    "test_persistence_via_pickle",
],

I suggest you do the same for regressors

@TonyBagnall My commit seems to fail the test for soft dependencies

RuntimeError: Estimator CNNRegressor does not require soft dependencies according to tags, but raises ModuleNotFoundError on __init__. Any required soft dependencies should be added to the "python_dependencies" tag, and python version bouds should be added to the "python_version" tag. Exception text: tensorflow

I have added the required dependency in test_softdeps.py but it still is not reflecting that. Any idea what could be going wrong?

Estimator CNNRegressor does not require soft dependencies according to tags

so just looking at CNNClassifier, which does run, the BaseDeepClassifier has these tags,
_tags = {
"X_inner_mtype": "numpy3D",
"capability:multivariate": True,
"python_dependencies": "tensorflow",
}
I've not seen the python_dependencies tag before. Does the base regressor have it? If not, try adding it. Ask franz if you want an overview of tags logic.

Done, @fkiraly