@soumith is the method documentation at lines 145/6 in a satisfactory format? Can't find a module with a similar example.

Everything looks good to me now 👍 I'll leave Soumith to help with that bit of documentation and how to sort out testing.

@jvmancuso I think you just need to add some test cases to the list here: https://github.com/pytorch/pytorch/blob/master/test/test_nn.py#L3114

you can follow the style of Linear here: https://github.com/pytorch/pytorch/blob/master/test/common_nn.py#L28

@jvmancuso as for the docstring, you should prefix the """ with an r i.e.

r"""
   this is my docstring which is now treated as a raw string.
   This means escape characters like \ wont be converted and instead
   treated as a literal \
"""

@alykhantejani pretty sure I fixed the math line in the docs at L123.

I'm not sure how a test case like Linear's will work in this instance, since the output of the layer will be different for each forward pass by definition. Specifically, there is no stable reference_fn to use, and I'm not sure how to test the functionality without a custom TestNN object that checks for AssertNotEqual instead of AssertEqual.

@jvmancuso You don't have to add the reference_fn field but adding the entry to the dict will make sure the jacobian checks are done (numerical vs analytical)

@alykhantejani Apart from the dict tests I think 2 custom tests would be good:

• In training mode, show that output 1, given a fixed input, is reasonably different from output 2, given the same input, but after reset_noise has been called. There is already an assertNotEqual function available to use.
• In evaluation mode, the output matches a linear layer with the same weights and biases.

It is pretty critical that the module shows these behaviours beyond simply passing the automatic differentiation checks.

@Kaixhin agreed, these can just be added as regular test functions in test_nn.py

Dug into these most recent failures. Question: what do I do about test_noncontig? It appears to be checking that using deepcopy to copy the layer doesn't change the gradient of the parameters. I think deepcopy might be resampling the noise somewhere, which would definitely trigger assertEqual to fail on parameter grads if that's the case. Can somebody else please take a look at this?

Also, no idea why test_Conv2d_backward_twice is failing. Nothing I've done changes the Conv2d module or that test case, and I don't see how any changes I've made would cause it to fail.

@Kaixhin @alykhantejani

@jvmancuso sorry I've been away on vacation. Will try to take a look at this, this week.

@jvmancuso The issue with test_noncontig is because the function tries to zero the gradients of the params here which calls this snippet of code which only zeros gradients for weight and bias and not the other params in this module (weight_mu etc.)

I'm not quite sure why _zero_grad_parameters explicitly names weight and bias, but perhaps this function can be replaced with a call to module. zero_grad (). Although there is a detach() call in _zero_grad_parameters, so if this is actually needed this function can instead loop through the modules parameters and manually zero the grads and detach.

@soumith @apaszke wdyt?

@jvmancuso in terms of the other test failing, try and pull in changes from upstream/master as you currently have merge conflicts anyway.

Hi all, it's been awhile, and I haven't forgotten about this. I'm doing an analysis of DeepMind's Noisy Nets versus OpenAI's Parameter Space Noise, and wanted to revisit this when that's done. I'll be implementing the OpenAI version shortly, and wanted to investigate including that in this PR. I'll close this for now and reopen when I've thought through that a bit more.

@jvmancuso for future reference, this doesn't work with CUDA. I think the best solution is to register weight_epsilon and bias_epsilon as buffers so that when the model is cast to CUDA they are cast as well - and then the generated noise needs to be copied over. You can see an example here.

@Kaixhin I had made a few changes to the code to accommodate for that but hadn't committed them. Your solution is more elegant though, I'll integrate it into my work. Thanks!