Currently, after calling graphics.defineEffect() successfully, the effect in question persists while the program does: until close or relaunch. This is almost always perfectly fine.

There are some trouble spots, however.

One of them: editing shaders. I've dabbled with something where you link nodes together, à la Blender or Unreal Editor. An in-app take on Shader Playground would be another example. Without being able to replace obsolete ones, new effects (with unique names) will continue to accumulate as we make updates, wasting many CPU- and GPU-side resources. This is especially egregious since an edit might only change a line or two!

@XeduR also mentioned this being relevant to his Solar2D Playground, if one wants to write custom shaders. Since a "true" relaunch never actually occurs, some steps need to be emulated, but this falls flat with permanently registered effects.

The details:

A few concerns needed addressing.

Removing the effect is one thing, but what if an object is still using it? With most effects, this actually "just works", since resources doled out to instances are either cloned or ref-counted.

Multi-pass effects complicate matters. In general, effect resources are only instantiated on demand, when we first assign the effect. With "simple" effects this is straightforward, since the defined one is what we want.

When defining a multi-pass effect, we list several sub-effects by name in graph.nodes. Presumably, we want these to be effects that are already defined, or soon will be. That "will be" speaks to the problem: since resources are instantiated lazily, one of these child effects might be undefined and even replaced before the actual multi-pass one is ever brought into being. Simply grabbing whatever goes by a given name is now a matter of uncertainty.

To address this, a "stub" system has been introduced. If a multi-pass effect uses any custom shaders, we map a stub table to each name, keeping these in a local list. We also add them to a global list. (If a name is already present in the latter, we instead add the corresponding stub to the local list.) Since both lists see the same table references, the stubs are shared.

If we undefine a child effect, it will see that it has a stub in the global list. It registers its lazy-loader here; the stub being shared, this will show up in the multi-pass effects' local lists. The entry is then evicted from the global list, since it no longer exists to the world at large.

While a multi-pass effect is being instantiated and connecting its child effects, it consults its local stubs. Any lazy-loader found in this way is used instead, so the proper choices are made despite any undefinitions.

This is my test:

--- Test for graphics.undefineEffect().

-- Permission is hereby granted, free of charge, to any person obtaining
-- a copy of this software and associated documentation files (the
-- "Software"), to deal in the Software without restriction, including
-- without limitation the rights to use, copy, modify, merge, publish,
-- distribute, sublicense, and/or sell copies of the Software, and to
-- permit persons to whom the Software is furnished to do so, subject to
-- the following conditions:
--
-- The above copyright notice and this permission notice shall be
-- included in all copies or substantial portions of the Software.
--
-- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-- EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-- MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
-- IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
-- CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
-- TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
-- SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
--
-- [ MIT license: http://www.opensource.org/licenses/mit-license.php ]

graphics.defineEffect{
  category = "filter", name = "effect1",
  isTimeDependent = true,

  fragment = [[
    P_COLOR vec4 FragmentKernel (P_UV vec2 uv)
    {
      return vec4(uv.x * .5 + sin(CoronaTotalTime * 3.7) * .5, 0.0, uv.y * fract(CoronaTotalTime * 18.), 1.0);
    }
  ]]
}

local r1 = display.newRect(display.contentCenterX, display.contentCenterY, 250, 250)

r1.fill.effect = "filter.custom.effect1"

for k, v in pairs(graphics.listEffects("generator")) do
--  print("GENERATOR, STEP #1: ", k, v)
end

print("")

graphics.defineEffect{
  category = "filter", name = "quantized",

  fragment = [[
    P_COLOR vec4 FragmentKernel (P_UV vec2 texCoord)
    {
      texCoord = floor(4. * texCoord) / 4.;
      
      return texture2D(CoronaSampler0, texCoord);
    }
  ]]
}

graphics.defineEffect{
  category = "filter", name = "blurred",

  graph = {
	   nodes = {
		  blur = { effect = "filter.custom.quantized", input1 = "effect1" },
		  effect1 = { effect = "filter.custom.effect1", input1 = "paint1" },
	   }, output = "blur"
   }
}

graphics.undefineEffect("filter.custom.effect1")

for k, v in pairs(graphics.listEffects("generator")) do
--  print("GENERATOR, STEP #2: ", k, v)
end

print("")

graphics.defineEffect{
  category = "filter", name = "effect1",

  fragment = [[
    P_COLOR vec4 FragmentKernel (P_UV vec2 uv)
    {
      return vec4(uv, 0.0, 1.0);
    }
  ]]
}

local r2 = display.newRect(250, 200, 300, 300)

r2.fill.effect = "filter.custom.effect1"

for k, v in pairs(graphics.listEffects("generator")) do
--  print("GENERATOR, STEP #3: ", k, v)
end

timer.performWithDelay(100, function()
  r2.x = math.random(249, 251)
end, 0)

print("")

timer.performWithDelay(3000, function()
  r1:removeSelf()
end)

local DummyFill = { type = "image", filename = "Image1.jpg" } -- image not actually used, just to get uvs in the effect

timer.performWithDelay(700, function()
  local x, y = math.random(display.contentWidth), math.random(display.contentHeight)
  local rect = display.newRoundedRect(x, y, 250, 250, 12)

  rect.fill = DummyFill
  rect.fill.effect = "filter.custom.blurred"
end, 5)

I tested with this image, but anything would do, since it was only used to get the fill to generate texture coordinates.

They were getting noisy during testing, but those print()s in the graphics.listEffects() loops can be uncommented to show everything currently (still) defined. Most of these are Solar's built-ins but you'll also see (or not) the ones written for the sample.

I create a first object, r1, and assign it an effect, filter.custom.effect; it sticks around briefly before being removed. In the midst of this, graphics.defineEffect() is called, but r1 is fine and even performs time-dependent updates.

Another "effect1" is then defined and assigned to a new object, r2.

A multi-pass effect incorporating the original filter.custom.effect1 is defined early on and then some rounded rects using it are gradually added via a timer. By the time these happen, the old "effect1" has been undefined.

A couple more cases may be tested:

If we never assign it to r1, filter.custom.effect1 gets undefined before ever being instantiated. In one of the middle commits I had been registering prototypes in the stubs, but this situation will undermine that approach; the lazy-loaders
exist immediately after definition, so proved to be the right way to go.

Removing r1 actually came from an older test: if you comment out the timer that creates the rounded rects, nothing more will reference the original filter.custom.effect1 once r1 is gone, and its GPU resources will soon be cleaned up. (I had some CoronaLog()s in Rtt_GLProgram.cpp code verifying as much, since removed.)