Another issue to pay attention to is whether the renderer defaults to enabling z-sort.
I strongly recommend enabling it by default, at least for transparent objects.

At present, I have maintained the behavior of not enabling it by default, so that the behavior of certain transparent examples remains consistent with before. But in fact, the rendering result may be incorrect.

Looks good to me. Curious to hear Almar's thoughts.

Is transparency still properly supported in points, lines and text?

The material.transparent looks a lot like WorldObject.rendermask, let's use that instead, or unify these api's somehow?

What currently happens with rendermask is that it first iterates over all objects that are either fully or partially opaque, and renders these. Then it iterates over all objects again, and renders all objects that are either fully or partially transparent.

If we sort the objects by z, we can iterate them over them in reverse in one of these cases, so that we go near-to-far for opaque and far-to-near for transparent objects.

The material.transparent looks a lot like WorldObject.rendermask, let's use that instead, or unify these api's somehow?

I'm not quite sure because theoretically, setting transparent only tells the renderer to process transparent objects according to their flow, and it doesn't necessarily have to be a truly "transparent" object. If it's not set to True, it doesn't necessarily mean it's not a transparent object (such as typical tranimissive objects based on physics). It is an identifier for reference by the renderer and has no direct relationship with the rendering process (shader).

Is transparency still properly supported in points, lines and text?

I think there should be no problem, and the test doesn't seem to have any abnormalities.
The only issue is Pygfx.Stats, which is essentially not a part of the scene and should not be involved in sorting objects in the scene. At present, we have dealt with it separately, but I think that in the future, this type of component should be placed in a separate UI layer and not together with the world objects in the scene.

Another issue to pay attention to is whether the renderer defaults to enabling z-sort. I strongly recommend enabling it by default, at least for transparent objects.

A major consideration in deciding whether to enable z-sorting by default is the additional performance overhead. Because for each object, z-sorting requires an additional position conversion (from world space to camera space).

I have thought about this issue again today.
Currently, we use the la.vec_transform() method for calculation, but it seems that directly using numpy's matrix multiplication speed will be much faster. Moreover, most importantly, because the same camera matrix is used, they can be calculated in batches.

I did a test:

import numpy as np
import pylinalg as la
import timeit

v = np.array([1, 2, 3], dtype=np.float32)
v4 = la.vec_homogeneous(v)
m = la.mat_compose(np.array([1, 2, 3]), la.quat_from_euler((1, 2, 3)), np.array([1, 2, 1]))

print(la.vec_transform(v, m))


def apply_matrix(v, m):
    vv = m @ v.T
    return vv.T

print(apply_matrix(v4, m))

v4_batch = np.stack([v4] * 100000)

vo = apply_matrix(v4_batch, m)
print(vo[:5])


print(timeit.timeit(lambda: la.vec_transform(v, m), number=100000))
print(timeit.timeit(lambda: apply_matrix(v4, m), number=100000))
print(timeit.timeit(lambda: apply_matrix(v4_batch, m), number=1))

[-3.02585975  2.94074763  0.01546533]
[-3.02585975  2.94074763  0.01546533  1.        ]
[[-3.02585975  2.94074763  0.01546533  1.        ]
 [-3.02585975  2.94074763  0.01546533  1.        ]
 [-3.02585975  2.94074763  0.01546533  1.        ]
 [-3.02585975  2.94074763  0.01546533  1.        ]
 [-3.02585975  2.94074763  0.01546533  1.        ]]
0.45319540007039905
0.13281040010042489
0.0017979999538511038

If I haven't made any mistakes, there seems to be a lot of room for optimization here, so the performance cost of z-sorting can be almost negligible. There is almost only the performance cost of sorting itself.

I believe vec_transform also accepts batches.

But I see the major difference is that you are not performing the division operation by w. Is that right?

Otherwise the code looks identical...

https://github.com/pygfx/pylinalg/blob/f162fcb8e65fd5d8d2c3cbe1ea58742758c990d5/pylinalg/vector.py#L105

I'm not quite sure, but it seems like there's more to it. The performance gap is still quite noticeable.

I'm not quite sure, but it seems like there's more to it. The performance gap is still quite noticeable.

I looked into this, see pygfx/pylinalg#102

Awesome how discussions like this result in performance boosts 😄 🚀. Making sorting the default feels better now.

I want to propose:

• We try to merge this PR as is
• We enable sorting by default and improve its performance in a separate PR

@almarklein how do you feel about that?

This PR highlights multiple issues with the current blending, and possibilities for improvements. This is very valuable indeed! But I also have a few objections to (the current state) of this pr. Let's discuss these.

The alpha_test makes sense, no objections.

The depth_write is also a good addition, but I think it perhaps needs a default "auto" value, because transparent objects should by default not write their depth?

I believe the stats object does not need special handing, because it's always rendered in a separate pass anyway.

You make good points about the currently incorrect sorting, and I can see it being enabled by default now.

Also good point to try and make the discard conditional to help early-z.

The greater objection that I have: the new transparent prop, and categorizing objects based on it sort of adds a new system next to the one based around render_mask. The render_mask system is already pretty powerful, because the shader can in many detect whether an object is opaque or transparent. And users can also explicitly override it via object.render_mask. It feels to me like you're implementing a better version of the ordered2 blender, but hard-coded in the renderer. I respond in more detail in #985

The depth_write is also a good addition, but I think it perhaps needs a default "auto" value, because transparent objects should by default not write their depth?

Er, The purpose of adding this setting is to provide users with better control over rendering behavior. Although transparent rendering typically disables depth_write, there is no inherent connection between transparent rendering and depth writing. Regardless, depth_test should remain enabled.

Previously, we only had the depth_test option. Disabling depth_test also disabled depth_write, which was incorrect.

Note that some transparent scenes may require depth_write (for example, if you consider transmissive objects as transparent, they need to enable depth_write), depending on the user's logic.

My initial idea is that when users render transparent objects and explicitly set transparent=True, they are generally aware of the need to set depth_write=False. If they don't set depth_write, it could be because they forgot (in which case, setting it by default would be helpful), or they might intentionally want to enable depth_write. Moreover, if we follow the rendering order of opaque objects first, followed by transparent objects from far to near, even if the user forgets to set depth_write=False, there shouldn't be errors in most cases(If errors occur, it indicates that transparent objects intersect, and even with depth_write=False, there would still be color blending issues).

However, if we automatically set depth_write=True when the user sets an object to transparent=True, I think that's acceptable.

I believe the stats object does not need special handing, because it's always rendered in a separate pass anyway.

I have tried it before because Stats contains both opaque (text) and transparent (background) objects. If you participate in sorting scene objects, you will render the text first and then the background, and the rendering result will be incorrect. In addition, if they are treated as ordinary scene objects, there are other issues, such as their participation in the generation of transmitted light sampling textures, which is incorrect in any case. My idea is that they should never be equated with ordinary renderable objects in the scene.

You make good points about the currently incorrect sorting, and I can see it being enabled by default now.

This PR does not have default enable sorting (by z-value), only distinguishes between transparent and opaque objects, and renders opaque objects first and then transparent objects.

The greater objection that I have: the new transparent prop, and categorizing objects based on it sort of adds a new system next to the one based around render_mask. The render_mask system is already pretty powerful, because the shader can in many detect whether an object is opaque or transparent. And users can also explicitly override it via object.render_mask.

I don't think so. transparent is just an identifier that tells the renderer to treat the object as transparent, without discarding any fragments or affecting the shader rendering process. In fact, if there is only one object in the scene, it doesn't matter whether you set it or not. transparent=True does not necessarily mean that the object is visually transparent (alpha<1). For example, if a partially transparent mesh object is set to transparent=True, it will not cause its non-transparent parts to be discarded.

It is meaningful to distinguish between transparent and opaque objects before rendering, and the format of the model file will also clearly indicate whether the material is transparent or not.

It feels to me like you're implementing a better version of the ordered2 blender, but hard-coded in the renderer.

I actually want to get rid of Blender. And use the Blender mode of ordered1, just because it actually has no effect, 😓

I have always felt that Blender is a somewhat strange design. We have defined many object related properties (specifically, material related attributes) in Blender, but we associate them with the renderer, which is strange.

For example, in Blender, we define "depth_descriptor" to describe the behavior of objects related to DepthTest and StencilTest. This is clearly an object or material related property, not a renderer property. Different materials or objects in the scene have their own depth_test logic and stencil_test logic (and it is important for the implementation of certain functions and effects, such as in post-processing where stencil _test information is needed for object edge drawing).

The "color_descriptor" is the same, and its blend method is a property of the material itself, not the behavior of the renderer. For example, in the scene, ordinary transparent objects are generally using "Translucent Blend Mode" (through alpha value blending), flame particles, etc. are using "Additive Blend Mode" (direct color addition), etc.
In the WGPU RenderPipeline, the "blend" method can be flexibly set through parameters such as "src_factor", "dst_factor", "operation", etc., but they are defined by materials according to requirements (just like each object (material) corresponds to its own RenderPipeline), rather than a renderer.

In the UE engine, there is also an explanation for the "Material Blend Mode".

The approach of this PR assumes ordered1 and ignores other blend modes. Which makes sense, since you say you want to get rid of the blender. But the blender has a purpose. Let me explain.

I strive to make things Just Work for our users. In the ideal case, a user populates a scene with different types of objects (meshes, lines, volumes, etc.) and everything looks as one would expect. That way scientists can explore data without having to be viz experts. We got a lot of things right in Pygfx, but making things Just Work for transparency is particularly tricky. It always has been.

The blender is my attempt to resolve this, by providing alternative blend modes, like weighted blending. In order to implement these advanced blend modes, they need control over the blending ops and whether or not depth is written. Maybe this helps understand the reason why the blender defines them and not the material.

I admit that the blender is not really successful at making things plug and play in regard to transparency. Users have to be aware of it, and chose (and understand) between the different modes. And even then, the results vary. E.g. weighted blending works well in some cases, but not so much in others.

I can also see how this inhibits the more game-engine-like feature, like specifying additive blending on a specific object.

But I have an idea ...

The blend mode which I find most interesting is the one based on dither (stochastic transparency), since its the only one that always produces correct results, regardless of ordering. And it's a single pass. And it can deal with objects that are partially transparent. Ok, it looks noisy, but we may be able to improve that.

Another big advantage (I realize now), is that it's compatible with classic blending; you can mix objects that use dither and alpha blending (you simply consider the dithered-object as opaque when you sort the objects).

So, a proposal ...

• We get rid of the weighted blending modes and the blender.
• The material gets a blend_mode property, by which the user can chose between classic alpha blending, additive, or dithered. And maybe allow specifying the full blend function.
• The shader still examines the color sources and determine whether the object is opaque, transparent, or possibly both (the render_mask), and will expose this attribute better.
• The shader can disable the dither code (if in use) when the object is known to be opaque, enabling early-z.
• The detected opaque/transparent-ness is used for sorting the objects by the renderer.
• Users can explicitly state the opaqueness too (as in object.render_mask or material.transparent).

With this:

• There's always a single "pass" (no double-rendering as ordered2 does for some objects).
• Simpler code.
• Enables gltf compatibility / classic blending.
• Transparency that just works with dithering.

But the blender has a purpose. Let me explain...

I strive to make things Just Work for our users ...

Thank you for your detailed explanation. Taking this opportunity, I would like to share some of my thoughts here, and I hope it doesn't come across as too presumptuous. 😅

As a rendering engine, I think Pygfx's core value lies in its high-level encapsulation of wgpu functionalities and rendering pipelines. It abstracts and simplifies wgpu capabilities and rendering processes through Pygfx-specific concepts and data structures.

In the rendering pipeline, we've introduced several key abstractions. The traditional programmable rendering pipeline typically consists of a geometry processing stage (vertex shader), rasterization stage, and pixel processing stage (fragment shader). For the geometry processing stage, we've designed the Geometry class as an abstraction, while for the rasterization and pixel processing stages, we use the Material class. Therefore, our renderable objects (world objects) consist of geometry and material components.

The core module of the engine should automatically handle the mapping between these abstract concepts and their underlying implementations, freeing users from dealing with the low-level details of WGPU API. This includes managing the lifecycle of various GPU objects, assembling rendering pipelines, data transfer, byte alignment, and other complex technical details.

Ideally, when users define vertex attributes in Geometry, these attributes should be automatically available in the vertex shader without additional effort. Similarly, when users define uniform properties or textures in Material, these resources should also be automatically accessible in shaders. Corresponding WGSL structure definitions, GPU object generation, and binding should all be handled automatically by the engine.

In essence, Pygfx users can conveniently define, configure, and implement their rendering pipelines using intuitive concepts like Geometry and Material. And it can cover all or most of the capabilities provided by wgpu. With these features, developing various applications based on Pygfx becomes straightforward, without requiring in-depth knowledge of wgpu's underlying implementation.

This level of functionality primarily targets developers who use Pygfx as a rendering engine.

Additionally, our various built-in Material classes (with accompanying shaders) and objects can be seen as predefined "classic rendering pipelines" based on the core functionalities mentioned above. These components are ready-to-use, and by adjusting parameters, they can meet the needs of most general scenarios and typical tasks. This level of functionality is aimed at more "front-end" users or developers who may not need to deeply understand the details of rendering pipelines.

As for higher-level functionalities targeting specific scenarios (such as scientific computing and data visualization), I believe these are better suited for implementation by libraries built on Pygfx (like fastplotlib). Even if some of these capabilities are to be included in Pygfx, they should be positioned as “core capability demonstrations and applications” rather than framework essentials.

Thanks for sharing your view on Pygfx. It's very helpful. And I mostly agree 😄

As for higher-level functionalities targeting specific scenarios (such as scientific computing and data visualization), I believe these are better suited for implementation by libraries built on Pygfx (like fastplotlib). Even if some of these capabilities are to be included in Pygfx, they should be positioned as “core capability demonstrations and applications” rather than framework essentials.

I do think that Pygfx's core framework can include features that make it more suitable for scientific purposes. We've always positioned Pygfx as a scientific-able render engine.

Which parts of my proposal are you referring to exactly?

As for dithering, making it possible in the core means that a library/users that wants to use it, does not have to subclass the shader for every object they want to support.

As for the shader determining whether the object is opaque or transparent, this is a task that the shader can do relatively easily, but is much harder to implement by higher-level code (because it's shader-specific). It's a relatively small effort from the engine's end that makes the user-experience a lot more friendly.

I'm not sure if it was clear from what I wrote, but I'm basically proposing what you are in this PR and comments, with two additions: 1) in addition to the common options of material.blend_mode, also enable stochastic transparency; 2) keep the auto-detection of transparent/opaque to the user (usually) does not have to specify it.

Which parts of my proposal are you referring to exactly?

The comments above are not directed towards a specific proposal you have put forward, just taking this opportunity to express my thoughts, 😅

My idea is that we should expose and map more comprehensive wgpu capabilities through basic abstract concepts such as Material, rather than hiding them (such as depth and blend related configurations), in order to provide maximum flexibility. Ideally, for all configurable and programmable places in wgpu's programmable rendering pipeline, we should provide corresponding APIs through this level of conceptual abstraction (Geometry and Material, etc.), rather than embedding some fixed pattern.

This is superseded by #1002. The discussions in this PR initiated that work 😉 @panxinmiao I think the main thing to salvage here is the addition in load_gltf.py. Could you submit that in a new PR?