Perhaps explain difference with opacity a bit?

Opacity is primarily used for alpha blending, while the transmission property is used for physically-based transparency. When transmission is not 0, opacity generally should be set to 1. I will update the relevant documentation later.

What of objects that have opacity 1 (i.e. is_transparent is False), but are still transparent, or partially transparent?

This issue is a bit complicated, and I originally intended to raise a separate issue for it. In fact, it’s quite difficult for the engine to automatically determine whether an object is transparent (and it probably shouldn’t attempt to do so). Ideally, there should be a boolean property on the material that indicates whether the object is transparent, allowing the user to explicitly set it.

This is because the rendering logic for transparent and non-transparent objects is quite different. When rendering transparent (or potentially transparent) objects, the user should manually set this flag to intentionally use a different rendering process (after all opaque objects, from far to near, etc.), which is maybe more performance-intensive. In the glTF specification, materials have an "alpha_mode" property, which can be set to "opaque," "alpha_blend," or "alpha_test." Generally, when it's set to "alpha_blend," it indicates that the "transparent object" rendering process should be used.

Ideally, there should be a boolean property on the material that indicates whether the object is transparent, allowing the user to explicitly set it.

I tend to agree with this sentiment.

It's easy to see the scientific user counter argument though. Someone's rendering a Points object to create e.g. a scatter plot, they set opacity to 0.8, and nothing happens, because they forgot to also set transparent=True.

🤷 Personally I don't mind making transparent=True an explicit required configuration on the user's part.

You can also have objects that have some fragments that are opaque and others that are transparent. This can be e.g. rendering markers with a solid edge, but a semi-transparent face. But it already happens with anti-aliasing.

Partially transparent mesh objects should be treated as transparent object usually.

Generally, If an object requires color blending (e.g., glass, fire), it must be handled as a transparent object (sorted from back to front + Alpha Blend).

However, if the transparency is used primarily for anti-aliasing or edge softening (e.g., grass, leaves), it is typically treated as an opaque object, using Alpha-to-Coverage (in conjunction with MSAA) or Alpha Test.

I don't think this special case is a good idea. With the refactoring of the renderer and blender that I'm working on, overlays will be taken into account in some form. So maybe this will do for the time being ... But let's at least move the import out of the loop.

Yes, I believe that such UI elements should not be added in the scene, as they are not part of the 3D scene and should not participate in the sorting of renderable objects or any interactions with scene objects. For example, they should not be involved in generating transmission light sampling maps or environment maps for the scene. Perhaps we should have a dedicated UI layer to store these objects and handle them separately.

I wonder what the implications are when there are a lot of objects in the scene. The previous code avoided sorting by grouping by render-order during flattening. I'm ok with using this approach with a call to sorted, but some questions must be answred: if there are 10000 objects with the same render_order, will this sort call be very fast? What if there's a couple of objects with a different render order?

This is a topic worth discussing.

My suggestion is that, in most cases, Z-sorting should be the default behavior of the renderer.

First, in most cases, after performing frustum culling (which we haven’t implemented yet, but will eventually), there shouldn’t be too many world objects requiring sorting during rendering. For advanced rendering, sorting can improve performance. Modern GPUs typically support Early-Z functionality.


• For non-transparent objects, sorting by Z value from near to far can maximize the use of Early-Z. By rendering objects closer to the camera first, all subsequent occluded fragments won’t perform shading computings. However, if we render distant objects first and then closer objects, any distant objects occluded by closer ones will result in unnecessary overdraw.
  
• For transparent objects using alpha blending, sorting from far to near is necessary to achieve the correct blend result. (By the way, the default "ordered2 blender" method we currently use actually doesn’t sort, which is not correct for transparent rendering.)
  

In 3D game development, when creating scene assets, it's generally advisable to avoid having a large number of small mesh objects in the scene. Typically, geometries with the same material are merged where possible. If there are many mesh objects, it usually indicates poor optimization, and we should consider using instancing or other methods to optimize rendering.

Lastly, if there really are a large number of simple objects to render, we could provide an API to allow users to disable sorting (with the user ensuring the correctness of the rendering results). But as I mentioned earlier, the default behavior should be " enbale Z-sorting."

Returning to the previous issue, if there are indeed 10,000 objects that require sorting, it's clearly a poor optimized scene, and we should avoid this situation. There are various techniques to address this (such as instanced rendering, particle/sprite systems, etc.). However, if it's absolutely necessary, we can manually disable sorting.

For cases where multiple objects have the same render order, since in most situations, Z-sorting should be applied, this isn't a major concern. However, to ensure stable and consistent sorting results, we can additionally compare the object's ID beyond the render order and Z value.

When sorting is disabled, the render order will also be ineffective. Additionally, we can provide an API that allows users to define their own sorting methods to accommodate specific logic.

• By the way, the default "ordered2 blender" method we currently use actually doesn’t sort, which is not correct for transparent rendering

I want to make sure that it is clear to @panxinmiao that we do not necessarily consider this to be "incorrect". In pygfx you can independently pick a blending method, and toggle scene sorting. If you enable scene sorting, and select the ordered2 blender, you'll get the expected result. What I can see though is that perhaps sort_objects should be True by default, since ordered2 is the default blender.

I have to agree with @almarklein though, there will commonly be cases where there is no transparency and no use of render order in the scene at all, and the new code will then still run a full sorting algorithm over all world objects. The old code would not perform any sorting at all in that case, and support very high amounts of world objects just fine.

Well, I'm convinced :) I'm okay with your change

Ok, sorting is faster than I anticipated. I did find that using a sort function make its more than 10x slower.

print(timeit.timeit('sorted(a, key=lambda i: i)', globals=globals(), number=1000))

Still fast enough not to affect performance, in the case of sorting by render_order.

Sorting by z is a completely different story, because it will do matrix multiplications. Rendering 10k objects is IMO a perfectly viable use-case. Sure, game devs should probably do some optimizations if they have such scenes, but for scientific use-cases such optimizations are out of scope.

I like Pygfx to be fast by default, so my preference is to keep the z-sorting opt-in (i.e. off by default).

Sorting also does not guarantee correct results, because (transparent) objects may cross each-other.

Sorting by z is a completely different story, because it will do matrix multiplications.

Yes, that's correct. However, transforming objects into camera (view) space is a necessary step. The difference is that we’ve moved this matrix multiplication (view_matrix @ model_matrix) into the shader, whereas many engines perform this step before render and only pass the product of the multiplication (view_matrix @ model_matrix) to the shader (meaning the shader uses camera space with the camera as the origin).
This has several benefits.:

• First, it makes Z-sorting easier.
• Second, each object only needs to execute the matrix multiplication once per frame, avoiding the need to perform it for every vertex in the shader.
• Additionally, using camera space in the shader helps improve precision. This is because when the camera is far from the origin of the world coordinate system, the world matrix values become large, which can lead to precision issues. Using camera space mitigates this, as objects that pass frustum culling are always close to the camera (the origin of the coordinate system).

Of course, this is beyond the scope of this PR. Back to this question, Although Z-sorting currently requires an additional matrix multiplication (a 4x4 matrix multiplication is actually quite fast), considering the impact of Early-Z, I believe the overall performance will still improve, especially for advanced rendering that requires complex shading calculations.

Sorting also does not guarantee correct results, because (transparent) objects may cross each-other.

Yes. For complex scenes, achieving correct transparent rendering remains challenging. However, rendering transparent objects in intricate scenes is inherently difficult. We should consider the performance and feasibility, aiming to meet the needs of most use cases.

In typical scenarios, we avoid intersections of transparent objects (e.g., by performing mesh geometry segmentation to minimize overlaps). Z-sorting and alpha blending suffice for most situations.

For truly complex scenes containing transparent objects, advanced techniques are necessary. Methods such as weighted blending (approximate effects), depth peeling algorithms with multiple render passes, or ray tracing can be employed. These scenes often require custom renderers and algorithms tailored to specific rendering needs.

? How does stuff still render?

In the following _render_objects() method.

I have to admit, I don't understand the internals of the renderer and blender well enough to comment on the changes here. I'm just worried that existing functionality may be broken or damaged. Best I can do is ask some questions to check:

• Do all the existing transparency blend modes still work?
• Are we making more roundtrips to the GPU than before (to get a single frame drawn)?
• Is pipeline state management still robust (reconstructing wgpu pipelines and recompiling shaders reactively)?

Can you explain this special treatment of double_sided transmissive objects? Why need to copy the color texture after each one here, but not for single-sided transmissive objects?

Well, it's a bit complex, let me try to explain.

Rendering complex transmissive objects correctly is not an easy task. We need to balance performance, feasibility, and visual accuracy, often resorting to approximations for the effect.

First, we render all opaque objects and generate a transmission light sampling texture, which will later be used by the transmissive objects.

You might quickly realize that when there are multiple transmissive objects in the scene, the transmission light sampling texture of a transmissive object located closer to the camera should include the transmissive objects' effects behind it — this is the physically correct result. However, for real-time rendering, it is too costly to generate a separate transmission light sampling texture for each transmissive object. So, we let "alpha blending" handle the overlap between them.

But for some transmissive objects with volume, the transmission light not only affect by the light Exit Surface. The ligth Entry Surface should have an effect on the transmitted light map too, but it doesn't, Because it is the backside and be obscured. For a more realistic effect, we can set it to double-sided rendering. When rendering these objects, we first draw the back face, and update the transmission light sampling texture using the frame containing the back face. Then, when we render the front face, we get a more physically accurate transmission light sampling texture, resulting in a more realistic rendering effect.

Let's give an example, Imagine a glass sphere where light enters from the rear (the side opposite the camera) and exits from the front into the camera. The following processes occur:

1. Refraction at the Entry Surface: Light enters the object from the rear (incident surface) and undergoes refraction (from air to glass). This corresponds to the first step, where the back surface of the transparent glass is rendered, and the result is updated in the transmitted light map.
2. Internal Propagation: Light propagates inside the object, potentially being absorbed or scattered.
3. Refraction at the Exit Surface: Light exits the object from the front (exit surface), refracting again (from glass to air) and carrying previous transmission information (color, brightness). This corresponds to the second step, where the front surface is rendered using the transmitted light map that includes the results from the first step.

This approach yields more accurate and realistic results, but more costly too.

Thanks for the explanation! So IIUC this copying of the transmission_framebuffer should in theory be done after rendering each object, but to balance performance with quality, you only do it for double-sided geometry, to make sure that this use-case works correctly. Is that more or less correct? 😄

Yes, to be precise, this should occurs each time light passes through the surface of a transmissive object (both entering and exiting). However, this process is computationally expensive. Therefore, we've simplified it.
But if users wish to achieve high-quality rendering of a transmissive object and set it to be double-sided, we can provide more accurate results. This requires coordination with the modeling process, as using single-sided or double-sided rendering may necessitate adjustments to material properties such as transmission and IOR. If you model with double-sided rendering in mind, it will be more physically accurate, though it will also be more computationally demanding.

This sort function can be much faster luckily! :)

Import it like this:

from operator import attrgetter

The transmission framebuffer is not placed on the shared object, i.e. it's a global texture. And it's re-created if the size does not match. This means that if there are two renderers with a different internal size, it gets re-created on each draw.

I think it makes sense to put it in the blender. There is already logic in place to only create the texture if its needed; if there are no objects that need picking there is no picking texture. We could do the same for the transmission texture. We already have flat.has_transmissive_objects 👍

This is now handled by the blender, so we can remove this.

Is the transmission_alpha intentionally multiplied after setting opacity=1 for opaque objects here?

This code looks scary to me, when I think it being applied to say 5000 objects on each draw...

Using the bounding box is a nice idea, and likely more precise in some cases, but I would like to know more how this affects performance. I don't think we've so far considered that the bounding box for all objects is queried at each draw. There's potential matrix multiplications to the get the world bounding box. We really want to make sure we have the caching correct etc.

Maybe turn this into a method for re-use?

This changes the behavior of render_order so that there's an extra level of sorting based on the render_order of any group ancestor. I'm fine with changing this behavior, but let's discuss what makes the most sense. And we should update the docs.

In current main, the render_order (an int) is summed with that of all its ancestors. The idea is that you usually want the child objects to be rendered "together" with the parent.

With the current change (if I read it correctly), there are now 4 sort keys: (render_queue, group_order, render_order, z). Where the group_order is the render_order of the first ancestor that is a Group.
I think this is the same as what ThreeJS does? In any case it still allows changing the order of the objects withing that group. I can see how it can give some more control.

One could argue about what should happen if there are multiple group in the ancestors.