Parallel Masks Causing Deadlock With Brushes #2448

Zeranny · 2023-10-06T23:35:31Z

Server Implementation

Paper

Server Version

1.20

Describe the bug

I hope I haven't gone down the wrong rabbit hole but;

It seems that when using a large enough brush with a mask that checks blocks surrounding the test block (e.g. angle mask) a scenario can arise where two threads will deadlock waiting on the same getChunkLock.lock().

I haven't yet worked out why this is limited to brushes and not also regions.

From VisualVM:

Found one Java-level deadlock:
=============================
"AsyncNotifyKeyedQueue - 0":
  waiting for ownable synchronizer 0x00000007d1464f60, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "pool-23-thread-11"

"pool-23-thread-11":
  waiting for ownable synchronizer 0x00000007d134bfa0, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "AsyncNotifyKeyedQueue - 0"

Java stack information for the threads listed above:
===================================================
"AsyncNotifyKeyedQueue - 0":
        at jdk.internal.misc.Unsafe.park(java.base@17.0.6/Native Method)
        - parking to wait for  <0x00000007d1464f60> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(java.base@17.0.6/LockSupport.java:211)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@17.0.6/AbstractQueuedSynchronizer.java:715)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@17.0.6/AbstractQueuedSynchronizer.java:938)
        at java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@17.0.6/ReentrantLock.java:153)
        at java.util.concurrent.locks.ReentrantLock.lock(java.base@17.0.6/ReentrantLock.java:322)
        at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.checkAndWaitOnCalledLock(ChunkHolder.java:96)
        at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.getBlock(ChunkHolder.java:1128)
        at com.fastasyncworldedit.core.queue.IChunkExtent.getBlock(IChunkExtent.java:62)
        at com.sk89q.worldedit.extent.Extent.getNearestSurfaceTerrainBlock(Extent.java:445)
        at com.sk89q.worldedit.extent.Extent.getNearestSurfaceTerrainBlock(Extent.java:414)
        at com.sk89q.worldedit.extent.Extent.getNearestSurfaceTerrainBlock(Extent.java:397)
        at com.fastasyncworldedit.core.extent.PassthroughExtent.getNearestSurfaceTerrainBlock(PassthroughExtent.java:67)
        at com.fastasyncworldedit.core.extent.PassthroughExtent.getNearestSurfaceTerrainBlock(PassthroughExtent.java:67)
        at com.fastasyncworldedit.core.function.mask.AngleMask.getHeight(AngleMask.java:78)
        at com.fastasyncworldedit.core.function.mask.AngleMask.testSlope(AngleMask.java:87)
        at com.fastasyncworldedit.core.function.mask.AngleMask.test(AngleMask.java:166)
        at com.sk89q.worldedit.extent.MaskingExtent.applyBlock(MaskingExtent.java:114)
        at com.fastasyncworldedit.core.extent.filter.block.CharFilterBlock.filter(CharFilterBlock.java:176)
        - locked <0x00000007d1485058> (a com.fastasyncworldedit.core.extent.filter.block.CharFilterBlock)
        at com.fastasyncworldedit.core.extent.filter.block.ChunkFilterBlock.filter(ChunkFilterBlock.java:77)
        - locked <0x00000007d1485058> (a com.fastasyncworldedit.core.extent.filter.block.CharFilterBlock)
        at com.sk89q.worldedit.extent.MaskingExtent.processSet(MaskingExtent.java:109)
        at com.fastasyncworldedit.core.extent.processor.MultiBatchProcessor.processSet(MultiBatchProcessor.java:120)
        at com.fastasyncworldedit.core.extent.processor.MultiBatchProcessor.processSet(MultiBatchProcessor.java:106)
        at com.fastasyncworldedit.core.extent.processor.BatchProcessorHolder.processSet(BatchProcessorHolder.java:27)
        at com.fastasyncworldedit.core.extent.processor.BatchProcessorHolder.processSet(BatchProcessorHolder.java:27)
        at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.call(ChunkHolder.java:1075)
        - locked <0x00000007d14850c8> (a com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder)
        at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.call(ChunkHolder.java:1058)
        - locked <0x00000007d14850c8> (a com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder)
        at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.call(ChunkHolder.java:34)
        at java.util.concurrent.FutureTask.run(java.base@17.0.6/FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(java.base@17.0.6/ThreadPoolExecutor.java:2037)
        at java.util.concurrent.ThreadPoolExecutor.reject(java.base@17.0.6/ThreadPoolExecutor.java:833)
        at java.util.concurrent.ThreadPoolExecutor.execute(java.base@17.0.6/ThreadPoolExecutor.java:1365)
        at java.util.concurrent.AbstractExecutorService.submit(java.base@17.0.6/AbstractExecutorService.java:145)
        at com.fastasyncworldedit.core.queue.implementation.QueueHandler.submit(QueueHandler.java:379)
        at com.fastasyncworldedit.core.queue.implementation.SingleThreadQueueExtent.submitUnchecked(SingleThreadQueueExtent.java:251)
        at com.fastasyncworldedit.core.queue.implementation.SingleThreadQueueExtent.flush(SingleThreadQueueExtent.java:466)
        - locked <0x00000007d1800720> (a com.fastasyncworldedit.core.queue.implementation.SingleThreadQueueExtent)
        at com.fastasyncworldedit.core.queue.IQueueExtent.commit(IQueueExtent.java:104)
        at com.fastasyncworldedit.core.extent.PassthroughExtent.commit(PassthroughExtent.java:221)
        at com.fastasyncworldedit.core.extent.PassthroughExtent.commit(PassthroughExtent.java:221)
        at com.sk89q.worldedit.EditSession.flushQueue(EditSession.java:1293)
        at com.sk89q.worldedit.LocalSession.remember(LocalSession.java:529)
        at com.sk89q.worldedit.LocalSession.remember(LocalSession.java:443)
        at com.sk89q.worldedit.command.tool.BrushTool.act(BrushTool.java:485)
        at com.sk89q.worldedit.command.tool.BrushTool.actPrimary(BrushTool.java:367)
        at com.sk89q.worldedit.extension.platform.PlatformManager.lambda$handlePlayerInput$6(PlatformManager.java:501)
        at com.sk89q.worldedit.extension.platform.PlatformManager$$Lambda$8307/0x0000000801e068b8.run(Unknown Source)
        at com.sk89q.worldedit.extension.platform.AbstractPlayerActor.lambda$runAction$1(AbstractPlayerActor.java:672)
        at com.sk89q.worldedit.extension.platform.AbstractPlayerActor$$Lambda$8261/0x0000000801df8498.run(Unknown Source)
        at com.fastasyncworldedit.core.util.task.AsyncNotifyKeyedQueue.lambda$run$0(AsyncNotifyKeyedQueue.java:48)
        at com.fastasyncworldedit.core.util.task.AsyncNotifyKeyedQueue$$Lambda$8262/0x0000000801df86b8.call(Unknown Source)
        at com.fastasyncworldedit.core.util.task.AsyncNotifyKeyedQueue.lambda$call$1(AsyncNotifyKeyedQueue.java:58)
        at com.fastasyncworldedit.core.util.task.AsyncNotifyKeyedQueue$$Lambda$8263/0x0000000801df88d8.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(java.base@17.0.6/FutureTask.java:264)
        at com.fastasyncworldedit.core.util.task.KeyQueuedExecutorService$KeyRunner.lambda$run$0(KeyQueuedExecutorService.java:150)
        at com.fastasyncworldedit.core.util.task.KeyQueuedExecutorService$KeyRunner$$Lambda$8264/0x0000000801df8d00.run(Unknown Source)
        at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(java.base@17.0.6/ForkJoinTask.java:1395)
        at java.util.concurrent.ForkJoinTask.doExec(java.base@17.0.6/ForkJoinTask.java:373)
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(java.base@17.0.6/ForkJoinPool.java:1182)
        at java.util.concurrent.ForkJoinPool.scan(java.base@17.0.6/ForkJoinPool.java:1655)
        at java.util.concurrent.ForkJoinPool.runWorker(java.base@17.0.6/ForkJoinPool.java:1622)
        at java.util.concurrent.ForkJoinWorkerThread.run(java.base@17.0.6/ForkJoinWorkerThread.java:165)
"pool-23-thread-11":
        at jdk.internal.misc.Unsafe.park(java.base@17.0.6/Native Method)
        - parking to wait for  <0x00000007d134bfa0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(java.base@17.0.6/LockSupport.java:211)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@17.0.6/AbstractQueuedSynchronizer.java:715)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@17.0.6/AbstractQueuedSynchronizer.java:938)
        at java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@17.0.6/ReentrantLock.java:153)
        at java.util.concurrent.locks.ReentrantLock.lock(java.base@17.0.6/ReentrantLock.java:322)
        at com.fastasyncworldedit.core.queue.implementation.SingleThreadQueueExtent.getOrCreateChunk(SingleThreadQueueExtent.java:291)
        at com.fastasyncworldedit.core.queue.implementation.SingleThreadQueueExtent.getOrCreateChunk(SingleThreadQueueExtent.java:46)
        at com.fastasyncworldedit.core.queue.IChunkExtent.getBlock(IChunkExtent.java:61)
        at com.sk89q.worldedit.extent.InputExtent.getBlock(InputExtent.java:53)
        at com.fastasyncworldedit.core.extent.PassthroughExtent.getBlock(PassthroughExtent.java:194)
        at com.fastasyncworldedit.core.extent.PassthroughExtent.getBlock(PassthroughExtent.java:194)
        at com.sk89q.worldedit.function.mask.BlockMask.test(BlockMask.java:207)
        at com.fastasyncworldedit.core.function.mask.CachedMask.test(CachedMask.java:71)
        at com.fastasyncworldedit.core.function.mask.AngleMask.test(AngleMask.java:154)
        at com.sk89q.worldedit.extent.MaskingExtent.applyBlock(MaskingExtent.java:114)
        at com.fastasyncworldedit.core.extent.filter.block.CharFilterBlock.filter(CharFilterBlock.java:176)
        - locked <0x00000007d1a32cb8> (a com.fastasyncworldedit.core.extent.filter.block.CharFilterBlock)
        at com.fastasyncworldedit.core.extent.filter.block.ChunkFilterBlock.filter(ChunkFilterBlock.java:77)
        - locked <0x00000007d1a32cb8> (a com.fastasyncworldedit.core.extent.filter.block.CharFilterBlock)
        at com.sk89q.worldedit.extent.MaskingExtent.processSet(MaskingExtent.java:109)
        at com.fastasyncworldedit.core.extent.processor.MultiBatchProcessor.processSet(MultiBatchProcessor.java:120)
        at com.fastasyncworldedit.core.extent.processor.MultiBatchProcessor.processSet(MultiBatchProcessor.java:106)
        at com.fastasyncworldedit.core.extent.processor.BatchProcessorHolder.processSet(BatchProcessorHolder.java:27)
        at com.fastasyncworldedit.core.extent.processor.BatchProcessorHolder.processSet(BatchProcessorHolder.java:27)
        at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.call(ChunkHolder.java:1075)
        - locked <0x00000007d1464fc0> (a com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder)
        at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.call(ChunkHolder.java:1058)
        - locked <0x00000007d1464fc0> (a com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder)
        at com.fastasyncworldedit.core.queue.implementation.chunk.ChunkHolder.call(ChunkHolder.java:34)
        at java.util.concurrent.FutureTask.run(java.base@17.0.6/FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.6/ThreadPoolExecutor.java:1136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.6/ThreadPoolExecutor.java:635)
        at java.lang.Thread.run(java.base@17.0.6/Thread.java:833)

Found 1 deadlock.

To Reproduce

//mask #angle[0d][45d] - Seemingly a mask that checks blocks other than the exact test(BlockVector3) block
//brush sphere stone 50 - I believe the size increases the odds of the crash.
Use brush
Server will have deadlocked threads and FAWE will be locked (Paper may also thread dump and close the server - seems to happen if the user that used the brush leaves)

Note that making an equivalent sized region selection and performing the same masked (gmask or replace etc) action will not crash/deadlock.

Expected behaviour

Parallel masks should not enter a thread unsafe deadlock scenario.

Screenshots / Videos

No response

Error log (if applicable)

https://paste.gg/p/anonymous/381e68e542c940208676aa5a76cfcfdc

Fawe Debugpaste

https://athion.net/ISPaster/paste/view/69b8fe8c9eb2437e97aba97df0299cd6

Fawe Version

FastAsyncWorldEdit-Bukkit-2.8.1-SNAPSHOT-574

Checklist

I have included a Fawe debugpaste.
I am using the newest build from https://ci.athion.net/job/FastAsyncWorldEdit/ and the issue still persists.

Anything else?

I first encountered this issue on the Bakery Creative server where the Arceon angle mask caused the same crash.
This mask, like the FAWE angle mask, checks blocks surrounding the block being tested.

The text was updated successfully, but these errors were encountered:

…vert code to previous behavior.

- addresses crashing of #2448

dordsor21 · 2023-10-14T19:13:18Z

The parallel masking is a duplicate of #2388 but the actual crash has a different cause so leaving this open

- remove ChunkHolder locking concept as this is no longer needed - previously we obtained the copy from chunk GET on finalize, meaning the copy could be replaced by a "newer" one (bad) - work around this issue by introducing concept of "unique" keys to map chunk GET copies to - correctly handle resetting of various chunk-related classes to actually allow pooling to work - remove chunks as they are submitted when flushing a SingleThreadQueueExtenting - Fixes #2459 - Addresses #2448 - Fixes #2388

* fix: clear player's history away from main thread if lock locked - addresses crashing of #2448 * Correct lock usage * remove possibility for race condition

Zeranny added the Requires Testing This is a new issue which needs to be approved before labeled with "bug" label Oct 6, 2023

TheMeinerLP added a commit to TheMeinerLP/FastAsyncWorldEdit that referenced this issue Oct 10, 2023

[IntellectualSites#2448] Remove from recycle method lock calls and re…

527b95a

…vert code to previous behavior.

OneLiteFeather added Bug Something isn't working and removed Requires Testing This is a new issue which needs to be approved before labeled with "bug" labels Oct 10, 2023

TheMeinerLP mentioned this issue Oct 10, 2023

Fixes mask deadlock for angle mask #2456

Closed

dordsor21 added a commit that referenced this issue Oct 11, 2023

fix: clear player's history away from main thread if lock locked

bca2d92

- addresses crashing of #2448

dordsor21 mentioned this issue Oct 11, 2023

fix: clear player's history away from main thread if lock locked #2457

Merged

SirYwell linked a pull request Oct 11, 2023 that will close this issue

fix: clear player's history away from main thread if lock locked #2457

Merged

dordsor21 mentioned this issue Oct 14, 2023

fix: vastly superier handling of queue chunks #2461

Merged

NotMyFault closed this as completed in #2457 Oct 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallel Masks Causing Deadlock With Brushes #2448

Parallel Masks Causing Deadlock With Brushes #2448

Zeranny commented Oct 6, 2023 •

edited by dordsor21

Loading

dordsor21 commented Oct 14, 2023

Parallel Masks Causing Deadlock With Brushes #2448

Parallel Masks Causing Deadlock With Brushes #2448

Comments

Zeranny commented Oct 6, 2023 • edited by dordsor21 Loading

Server Implementation

Server Version

Describe the bug

To Reproduce

Expected behaviour

Screenshots / Videos

Error log (if applicable)

Fawe Debugpaste

Fawe Version

Checklist

Anything else?

dordsor21 commented Oct 14, 2023

Zeranny commented Oct 6, 2023 •

edited by dordsor21

Loading