[Merged by Bors] - Add z-index support with a predictable UI stack #5877

oceantume · 2022-09-04T16:30:33Z

Objective

Add consistent UI rendering and interaction where deep nodes inside two different hierarchies will never render on top of one-another by default and offer an escape hatch (z-index) for nodes to change their depth.

The problem with current implementation

The current implementation of UI rendering is broken in that regard, mainly because it sets the Z value of the Transform component based on a "global Z" space shared by all nodes in the UI. This doesn't account for the fact that each node's final GlobalTransform value will be relative to its parent. This effectively makes the depth unpredictable when two deep trees are rendered on top of one-another.

At the moment, it's also up to each part of the UI code to sort all of the UI nodes. The solution that's offered here does the full sorting of UI node entities once and offers the result through a resource so that all systems can use it.

Solution

New ZIndex component

This adds a new optional ZIndex enum component for nodes which offers two mechanism:

ZIndex::Local(i32): Overrides the depth of the node relative to its siblings.
ZIndex::Global(i32): Overrides the depth of the node relative to the UI root. This basically allows any node in the tree to "escape" the parent and be ordered relative to the entire UI.

Note that in the current implementation, omitting ZIndex on a node has the same result as adding ZIndex::Local(0). Additionally, the "global" stacking context is essentially a way to add your node to the root stacking context, so using ZIndex::Local(n) on a root node (one without parent) will share that space with all nodes using Index::Global(n).

New UiStack resource

This adds a new UiStack resource which is calculated from both hierarchy and ZIndex during UI update and contains a vector of all node entities in the UI, ordered by depth (from farthest from camera to closest). This is exposed publicly by the bevy_ui crate with the hope that it can be used for consistent ordering and to reduce the amount of sorting that needs to be done by UI systems (i.e. instead of sorting everything by global_transform.z in every system, this array can be iterated over).

New z_index example

This also adds a new z_index example that showcases the new ZIndex component. It's also a good general demo of the new UI stack system, because making this kind of UI was very broken with the old system (e.g. nodes would render on top of each other, not respecting hierarchy or insert order at all).

Changelog

Added the ZIndex component to bevy_ui.
Added the UiStack resource to bevy_ui, and added implementation in a new stack.rs module.
Removed the previous Z updating system from bevy_ui, because it was replaced with the above.
Changed bevy_ui rendering to use UiStack instead of z ordering.
Changed bevy_ui focus/interaction system to use UiStack instead of z ordering.
Added a new z_index example.

ZIndex demo

Here's a demo I wrote to test these features
https://user-images.githubusercontent.com/1060971/188329295-d7beebd6-9aee-43ab-821e-d437df5dbe8a.mp4

oceantume · 2022-09-04T16:46:20Z

The local and global z-index solution in this PR is an attempt to bring flexibility without adding too much complexity.

Local z-index allows a node to change its depth compared to its siblings regardless of the order they were added to their parent, while the Global z-index allows nodes to "escape" the parent and be ordered relative to the entire UI.

Those two use-cases are the most common in my opinion, but I can see some benefits in other more complex solutions. For example, having the ZIndex::Local depth be relative to the nearest parent node that has a StackingContext component attached to it (effectively creating contexts dynamically instead of basing it solely on siblings).

bjorn3 · 2022-09-04T16:48:15Z

This basically allows any node in the tree to

The sentence cut off.

djeedai

I really like this PR. I think it not only provides a relatively simple and intuitive solution to a design limitation which currently blocks entire classes of use cases, but it's also a step in what I'd argue is the right direction for UI hierarchies:

support for multiple UI roots, which this PR helps greatly with its StackingContext
getting rid of the use of Transform in UI hierarchies, which unlike other use cases is for UI not user controlled but auto-generated (the source of truth is Style for UI), which makes it very unintuitive for users. Unity has the same approach; for UI their Transform is replaced with a RectTransform.

@alice-i-cecile what are the arguments for making this PR as controversial? Can we link a discussion or post some rationale?

TimJentzsch

Nice, while z-index ordering might not be perfect for all cases, it's definitely a good basis to build on (and familiar to a lot of people).

examples/ui/z_index.rs

oceantume · 2022-09-06T15:08:37Z

I think this system works great with how Bevy UI works at the moment, because it assumes one window and viewport. However, the UiStack will need to change slightly after or before #5892 gets merged. The UiStack struct will be even more useful if it contains one Vec<Entity> per UI root, and each of those roots comes with information on where they're getting rendered to.

For example, the UI interaction system will be able to filter on the root's target information to only update UI that's getting rendered on the current active window. This will also be useful during UI rendering when multiple render targets are supported.

rparrett · 2022-09-06T16:07:07Z

I think that this PR could be improved by

Addressing how this solution relates to some of the discussion / design from the previous attempt in Allow multiple root UI Node #1211
Adding a test. This changeset puts us at -1 tests for "ui z stuff."
Some benchmarking to demonstrate that there are no major perf regressions.

rparrett · 2022-09-06T16:19:26Z

The included example serves as a good demo, but putting myself in the shoes of a user who is trying to learn how ZIndex works, it seems pretty difficult to follow.

alice-i-cecile · 2022-09-06T21:08:39Z

Addressing how this solution relates to some of the discussion / design from the previous attempt in #1211

This was what motivated me to add the Controversial label. So long as we can avoid the problems there, I'm happy to move forward on this. I think we have consensus that this needs to happen, and this seems like a good step forward.

Much of the related camera-driven rendering + hierarchy mess that plagued #1211 has since been dramatically improved.

oceantume · 2022-09-07T00:54:40Z

Addressing how this solution relates to some of the discussion / design from the previous attempt in #1211

This was what motivated me to add the Controversial label. So long as we can avoid the problems there, I'm happy to move forward on this. I think we have consensus that this needs to happen, and this seems like a good step forward.

Much of the related camera-driven rendering + hierarchy mess that plagued #1211 has since been dramatically improved.

I made a summary of some important points from that PR and how they're improved upon with this one.

We need to ensure that layers never "intersect" with each other.

The UiStack concept helps with this since it's a sorted vector built from the hierarchies (and the z-index escape hatch). You can only iterate over it in one way or the other depending on what you need to do.

I think it probably makes sense to make it relative. Maybe we can do something similar to CSS.

The global and local options helps giving some flexibility here and the result is somewhat similar to CSS with less rules and exceptions.

I pretty strongly feel that it should be a part of Style (defaulting to a None or Auto value). I don't like requiring a separate component for roots because that is "redundant information".

There's nothing root-centric in this PR, so that helps. As for the ZIndex component, it's fully optional. However, we should probably add it to UI bundles as an Option<ZIndex> for clarity/discovery.

Z precision issues

The z precision issues discussed on there are gone with this solution because we don't move ui nodes on the z axis anymore.

WindowId, UI to texture concerns

This PR doesn't change anything in that regard.

My PR also doesn't address the fact that multiple root nodes share the same internal taffy root and will share screen space. However, I have a separate PR #5892 which does.

oceantume · 2022-09-07T02:33:04Z

I just realized that adding Option<ZIndex> to the bundles is not yet supported. Do you think it's important enough that it should be added to the node bundles to justify adding it with a default value of ZIndex::Local(0)? It would remain an optional component, but would be present by default when using bundles to help with discovery.

alice-i-cecile · 2022-09-27T15:33:26Z

bors try

cart · 2022-09-29T06:49:19Z

@alice-i-cecile I would still call this controversial as this is a pretty foundational piece of UI that will be hard to change once people start depending on its behaviors. I'd like to give this a look, but I also don't want to hold back progress. Lets "start the merge clock" on this, but with the agreement that it can be merged for Bevy 0.9 if I don't get to it in time (so slightly accelerated).

alice-i-cecile · 2022-09-29T12:14:09Z

Sounds good!

bors · 2022-10-03T18:37:53Z

try

Merge conflict.

cart

I'm on board for this! The sorting algorithm is usable and roughly in line with the web spec. There may be potential here for optimization in various places, but I'm cool with moving forward on this as-is (once merge conflicts are resolved).

cart · 2022-10-26T20:38:11Z

crates/bevy_ui/src/stack.rs

+    }
+
+    *ui_stack = UiStack {
+        uinodes: Vec::<Entity>::with_capacity(total_entry_count),


We should consider reusing the vector here to avoid allocations

I just made changes to do that. Would just need a confirmation that it makes sense as-is.

cart · 2022-10-27T21:12:08Z

crates/bevy_ui/src/render/mod.rs

    extracted_uinodes
        .uinodes
-        .sort_by(|a, b| FloatOrd(a.transform.w_axis[2]).cmp(&FloatOrd(b.transform.w_axis[2])));
+        .sort_by_key(|node| node.stack_index);


Its a bit unfortunate that we already have an absolute order in UiStack, which we're then re-establishing here in uinodes. The sort is redundant. That being said, if we want a "distributed" extract (ex: multiple systems with node-type-specific extraction logic), then the only real alternative is extracting nodes as entities (instead of pushing them onto an unsorted ExtractedUiNodes list), then using an extracted UiStack value to query/iterate over them in prepare_uinodes. The extra cost of "per-entity extraction" might outweigh the time saved on sorting.

Worth measuring, but not something we need to block on here, especially given that this is following the pre-existing pattern.

alice-i-cecile

Merge looks correct.

james7132 · 2022-11-01T09:37:35Z

@alice-i-cecile @oceantume does this address #1275?

alice-i-cecile · 2022-11-01T10:30:32Z

IMO no, not fully. It doesn't work for 2D, and doesn't allow a direct and automatic translation from an ordered enum.

cart · 2022-11-02T21:48:37Z

Just calling out that the recent merge broke this code / it needs fixing.

cart · 2022-11-02T22:05:47Z

bors r+

# Objective Add consistent UI rendering and interaction where deep nodes inside two different hierarchies will never render on top of one-another by default and offer an escape hatch (z-index) for nodes to change their depth. ## The problem with current implementation The current implementation of UI rendering is broken in that regard, mainly because [it sets the Z value of the `Transform` component based on a "global Z" space](https://github.com/bevyengine/bevy/blob/main/crates/bevy_ui/src/update.rs#L43) shared by all nodes in the UI. This doesn't account for the fact that each node's final `GlobalTransform` value will be relative to its parent. This effectively makes the depth unpredictable when two deep trees are rendered on top of one-another. At the moment, it's also up to each part of the UI code to sort all of the UI nodes. The solution that's offered here does the full sorting of UI node entities once and offers the result through a resource so that all systems can use it. ## Solution ### New ZIndex component This adds a new optional `ZIndex` enum component for nodes which offers two mechanism: - `ZIndex::Local(i32)`: Overrides the depth of the node relative to its siblings. - `ZIndex::Global(i32)`: Overrides the depth of the node relative to the UI root. This basically allows any node in the tree to "escape" the parent and be ordered relative to the entire UI. Note that in the current implementation, omitting `ZIndex` on a node has the same result as adding `ZIndex::Local(0)`. Additionally, the "global" stacking context is essentially a way to add your node to the root stacking context, so using `ZIndex::Local(n)` on a root node (one without parent) will share that space with all nodes using `Index::Global(n)`. ### New UiStack resource This adds a new `UiStack` resource which is calculated from both hierarchy and `ZIndex` during UI update and contains a vector of all node entities in the UI, ordered by depth (from farthest from camera to closest). This is exposed publicly by the bevy_ui crate with the hope that it can be used for consistent ordering and to reduce the amount of sorting that needs to be done by UI systems (i.e. instead of sorting everything by `global_transform.z` in every system, this array can be iterated over). ### New z_index example This also adds a new z_index example that showcases the new `ZIndex` component. It's also a good general demo of the new UI stack system, because making this kind of UI was very broken with the old system (e.g. nodes would render on top of each other, not respecting hierarchy or insert order at all). ![image](https://user-images.githubusercontent.com/1060971/189015985-8ea8f989-0e9d-4601-a7e0-4a27a43a53f9.png) --- ## Changelog - Added the `ZIndex` component to bevy_ui. - Added the `UiStack` resource to bevy_ui, and added implementation in a new `stack.rs` module. - Removed the previous Z updating system from bevy_ui, because it was replaced with the above. - Changed bevy_ui rendering to use UiStack instead of z ordering. - Changed bevy_ui focus/interaction system to use UiStack instead of z ordering. - Added a new z_index example. ## ZIndex demo Here's a demo I wrote to test these features https://user-images.githubusercontent.com/1060971/188329295-d7beebd6-9aee-43ab-821e-d437df5dbe8a.mp4 Co-authored-by: Carter Anderson <mcanders1@gmail.com>

bors · 2022-11-02T22:23:28Z

Pull request successfully merged into main.

Build succeeded:

oceantume · 2022-11-02T22:44:10Z

Thanks for fixing it @cart. I was hoping to look into this today, but didn't find the time.

# Objective Add consistent UI rendering and interaction where deep nodes inside two different hierarchies will never render on top of one-another by default and offer an escape hatch (z-index) for nodes to change their depth. ## The problem with current implementation The current implementation of UI rendering is broken in that regard, mainly because [it sets the Z value of the `Transform` component based on a "global Z" space](https://github.com/bevyengine/bevy/blob/main/crates/bevy_ui/src/update.rs#L43) shared by all nodes in the UI. This doesn't account for the fact that each node's final `GlobalTransform` value will be relative to its parent. This effectively makes the depth unpredictable when two deep trees are rendered on top of one-another. At the moment, it's also up to each part of the UI code to sort all of the UI nodes. The solution that's offered here does the full sorting of UI node entities once and offers the result through a resource so that all systems can use it. ## Solution ### New ZIndex component This adds a new optional `ZIndex` enum component for nodes which offers two mechanism: - `ZIndex::Local(i32)`: Overrides the depth of the node relative to its siblings. - `ZIndex::Global(i32)`: Overrides the depth of the node relative to the UI root. This basically allows any node in the tree to "escape" the parent and be ordered relative to the entire UI. Note that in the current implementation, omitting `ZIndex` on a node has the same result as adding `ZIndex::Local(0)`. Additionally, the "global" stacking context is essentially a way to add your node to the root stacking context, so using `ZIndex::Local(n)` on a root node (one without parent) will share that space with all nodes using `Index::Global(n)`. ### New UiStack resource This adds a new `UiStack` resource which is calculated from both hierarchy and `ZIndex` during UI update and contains a vector of all node entities in the UI, ordered by depth (from farthest from camera to closest). This is exposed publicly by the bevy_ui crate with the hope that it can be used for consistent ordering and to reduce the amount of sorting that needs to be done by UI systems (i.e. instead of sorting everything by `global_transform.z` in every system, this array can be iterated over). ### New z_index example This also adds a new z_index example that showcases the new `ZIndex` component. It's also a good general demo of the new UI stack system, because making this kind of UI was very broken with the old system (e.g. nodes would render on top of each other, not respecting hierarchy or insert order at all). ![image](https://user-images.githubusercontent.com/1060971/189015985-8ea8f989-0e9d-4601-a7e0-4a27a43a53f9.png) --- ## Changelog - Added the `ZIndex` component to bevy_ui. - Added the `UiStack` resource to bevy_ui, and added implementation in a new `stack.rs` module. - Removed the previous Z updating system from bevy_ui, because it was replaced with the above. - Changed bevy_ui rendering to use UiStack instead of z ordering. - Changed bevy_ui focus/interaction system to use UiStack instead of z ordering. - Added a new z_index example. ## ZIndex demo Here's a demo I wrote to test these features https://user-images.githubusercontent.com/1060971/188329295-d7beebd6-9aee-43ab-821e-d437df5dbe8a.mp4 Co-authored-by: Carter Anderson <mcanders1@gmail.com>

oceantume added 3 commits September 4, 2022 11:26

Add ZIndex component and consistent UI stack

bf32dbe

Merge branch 'main' into feature/zindex-and-uistack

f5ac1e4

Fix small issues after merging

1bd3fe1

oceantume changed the title ~~Feature/zindex and uistack~~ Add z-index support and predictable UI rendering and interaction Sep 4, 2022

oceantume changed the title ~~Add z-index support and predictable UI rendering and interaction~~ Add z-index support with a predictable UI stack Sep 4, 2022

Run fmt and clippy

04e5d7a

Fix clippy errors

321de09

oceantume force-pushed the feature/zindex-and-uistack branch from a48628a to 321de09 Compare September 4, 2022 16:53

Fix example error

12cc199

oceantume force-pushed the feature/zindex-and-uistack branch from c5b64e1 to 12cc199 Compare September 4, 2022 17:26

alice-i-cecile added C-Bug An unexpected or incorrect behavior C-Feature A new feature, making something new possible A-UI Graphical user interfaces, styles, layouts, and widgets labels Sep 4, 2022

alice-i-cecile self-requested a review September 4, 2022 18:09

alice-i-cecile added the X-Controversial There is active debate or serious implications around merging this PR label Sep 4, 2022

djeedai approved these changes Sep 6, 2022

View reviewed changes

TimJentzsch approved these changes Sep 6, 2022

View reviewed changes

examples/ui/z_index.rs Outdated Show resolved Hide resolved

alice-i-cecile removed the X-Controversial There is active debate or serious implications around merging this PR label Sep 6, 2022

oceantume added 2 commits September 6, 2022 23:56

Simplify the z_index example

0c1bbab

Replace one color in z-index example

2d2d335

oceantume force-pushed the feature/zindex-and-uistack branch from 7b439e4 to 2d2d335 Compare September 7, 2022 11:50

oceantume mentioned this pull request Sep 7, 2022

UI render layers #5414

Closed

alice-i-cecile added X-Controversial There is active debate or serious implications around merging this PR S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it labels Sep 20, 2022

Remove unused component from z_index example

c0ff6f5

alice-i-cecile added this to the Bevy 0.9 milestone Sep 29, 2022

cart reviewed Oct 27, 2022

View reviewed changes

Merge branch 'main' into feature/zindex-and-uistack

bd18d78

alice-i-cecile approved these changes Oct 31, 2022

View reviewed changes

oceantume added 2 commits October 31, 2022 18:44

Fix format

36dfd25

Fix example and reuse uinodes vector

32766b3

oceantume force-pushed the feature/zindex-and-uistack branch from 3b27b67 to 32766b3 Compare October 31, 2022 22:58

Fix merge errors

2f793cc

bors bot changed the title ~~Add z-index support with a predictable UI stack~~ [Merged by Bors] - Add z-index support with a predictable UI stack Nov 2, 2022

bors bot closed this Nov 2, 2022

oceantume mentioned this pull request Nov 2, 2022

Add support for multi viewports and windows to UI #5892

Closed

3 tasks

ManevilleF mentioned this pull request Nov 5, 2022

Texture Atlas rework #5103

Merged

james7132 mentioned this pull request Nov 12, 2022

Unstable z-order sorting when using multiple root UI nodes #2569

Closed

rparrett mentioned this pull request Jan 10, 2023

Tagging ImageBundle makes it disappear #1329

Closed

james7132 mentioned this pull request Sep 20, 2023

Remove ZIndex::Global #9877

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - Add z-index support with a predictable UI stack #5877

[Merged by Bors] - Add z-index support with a predictable UI stack #5877

oceantume commented Sep 4, 2022 •

edited

Loading

oceantume commented Sep 4, 2022

bjorn3 commented Sep 4, 2022

djeedai left a comment

TimJentzsch left a comment

oceantume commented Sep 6, 2022 •

edited

Loading

rparrett commented Sep 6, 2022

rparrett commented Sep 6, 2022

alice-i-cecile commented Sep 6, 2022

oceantume commented Sep 7, 2022 •

edited

Loading

oceantume commented Sep 7, 2022 •

edited

Loading

alice-i-cecile commented Sep 27, 2022

cart commented Sep 29, 2022

alice-i-cecile commented Sep 29, 2022

bors bot commented Oct 3, 2022

cart left a comment

cart Oct 26, 2022

oceantume Oct 31, 2022

cart Oct 27, 2022

alice-i-cecile left a comment

james7132 commented Nov 1, 2022 •

edited

Loading

alice-i-cecile commented Nov 1, 2022

cart commented Nov 2, 2022

cart commented Nov 2, 2022

bors bot commented Nov 2, 2022

oceantume commented Nov 2, 2022

[Merged by Bors] - Add z-index support with a predictable UI stack #5877

[Merged by Bors] - Add z-index support with a predictable UI stack #5877

Conversation

oceantume commented Sep 4, 2022 • edited Loading

Objective

The problem with current implementation

Solution

New ZIndex component

New UiStack resource

New z_index example

Changelog

ZIndex demo

oceantume commented Sep 4, 2022

bjorn3 commented Sep 4, 2022

djeedai left a comment

Choose a reason for hiding this comment

TimJentzsch left a comment

Choose a reason for hiding this comment

oceantume commented Sep 6, 2022 • edited Loading

rparrett commented Sep 6, 2022

rparrett commented Sep 6, 2022

alice-i-cecile commented Sep 6, 2022

oceantume commented Sep 7, 2022 • edited Loading

oceantume commented Sep 7, 2022 • edited Loading

alice-i-cecile commented Sep 27, 2022

cart commented Sep 29, 2022

alice-i-cecile commented Sep 29, 2022

bors bot commented Oct 3, 2022

try

cart left a comment

Choose a reason for hiding this comment

cart Oct 26, 2022

Choose a reason for hiding this comment

oceantume Oct 31, 2022

Choose a reason for hiding this comment

cart Oct 27, 2022

Choose a reason for hiding this comment

alice-i-cecile left a comment

Choose a reason for hiding this comment

james7132 commented Nov 1, 2022 • edited Loading

alice-i-cecile commented Nov 1, 2022

cart commented Nov 2, 2022

cart commented Nov 2, 2022

bors bot commented Nov 2, 2022

oceantume commented Nov 2, 2022

oceantume commented Sep 4, 2022 •

edited

Loading

oceantume commented Sep 6, 2022 •

edited

Loading

oceantume commented Sep 7, 2022 •

edited

Loading

oceantume commented Sep 7, 2022 •

edited

Loading

james7132 commented Nov 1, 2022 •

edited

Loading