ok so for the skybox, you basically sample with a direction vector, which isn't super intuitive and I'm not sure how you perfer to return this. The basics are in here. I need to review with a clear head to get something "right".

Some questions I still need to answer for myself:

• Always return a tuple of length 3?
• Is it ok to return a "direction vector" for the skybox? or do we want "u, v, face_index" or something of that sort.

Oh, I now realize I assumed that the varyings.texcoord always represented the full quad coords, and the .z field indicating the face of the cube, but this is not the case; it's the direction vector. Sorry, my earlier comments may have been confusing because of this.

I propose to keep it simple:

• In the vertex shader always do varyings.texcoord = vec3<f32>(pos * 0.5 + 0.5, 0.0);
• For cube texture use new varyings.cubecoord (instead of using textcoord for it).
• Send this texcoord as picking info and expose it as a coord field in the pick info dict.
• Forget about cube-specific pick info for now.

I propose to keep it simple:

• In the vertex shader always do varyings.texcoord = vec3(pos * 0.5 + 0.5, 0.0);
• For cube texture use new varyings.cubecoord (instead of using textcoord for it).
• Send this texcoord as picking info and expose it as a coord field in the pick info dict.
• Forget about cube-specific pick info for now.

Are these mutually exclusive options? Sorry the choice of words makes it seem like they could be sequential.

As it stand "right now" (i might force push to cleanup the commit history), the normalized direction vector is being sent. After sleeping on it, it makes sense to me for a direction vector to get sent because in a "skybox" the most important thing might be "the direction where the user clicked".

The "6 sided cube trick" to me feels like a neat graphics trick, but I wonder if it helps or hinders the downstream application from being able to use the information effectively.

For example, I could see an application placing markers:

• Current position + skybox vector * 10

away as the user clicks on the skybox.

It seems that the picking info is super material specific, so I'm not too worried about that.
I might suggest that the BackgroundImage material loose its ability to take in Cube matrial. When "inputs" change return values, to me that is where the API gets messy.

If:

• BackgroundImageMaterial throws an error on skybox textures, then it becomes a clear rule that the returned pick info is a 2D coordinate.
• BackgroundSkyboxMaterial always returns a 3D unit vector.

Are these mutually exclusive options? Sorry the choice of words makes it seem like they could be sequential.

No I meant these as a package of choices to keep things simple. In other words, I said: let's forget about the skybox and always return a vector that can be used to see where on the screen the user clicked.

Now that I think about it, the position where the user clicked should (eventually) be part of the info always, the renderer can do that (once we figure out what to do with viewports). So maybe ... forget about any pick info for now! 🤷

So maybe ... forget about any pick info for now! 🤷

I'm down, i just want it to "respond" to "mouse move". Right now, moving the mouse over the Background just "ignores it"

Keeping it so simple it seems my top level comment is still appropriate.

I added a comment pointing to this discussion. Not sure if you find that appropriate.

Thanks for your time again!