From 096a0cd148448281bce86320625d3e5255e85a13 Mon Sep 17 00:00:00 2001 From: Pierre Chevalier Date: Fri, 15 Mar 2024 08:18:57 -0700 Subject: [PATCH] derived_data_utils: Fix duplicated derived data dependency graph Summary: Recently, we had a report from AOSP that they saw this error: ``` D0314 14:55:17.055516 307 fbcode/scm/grepo/branch_forest/src/derive_data.rs:98] derived graph in 14.239055403s: Err(failed to derive bssm_v3 batch (start:9ff0c1071da6397408981fab35b69323c8f2900fc76b4b323bdb7edace0dc849, end:7b9d79e5f2ab5dacb04e7c7a2e0354163be485e98c6f726978d2cba960cc421c) Caused by: dependency 'skeleton_manifests' of 66a9c86eb6dc728992307e5f7d83bb54c290246405b508f9dabaa3ae30f1c11a was not already derived ``` Full trace: P1196413182 At the moment, `branch_forest` uses the old "derived_data/utils" interface to interact with data derivation. This code duplicates the dependency graph that is established at compile time in the definition of each derivable data type and performs runtime traversals. For this reason, it is error prone and we want to use recent refactorings in derived data to replace it with the `BulkDerivation` interface and delete it. We are not quite there in terms of refactorings, yet, so for now patch the bug in `derived_data/utils`. This is the source of truth for this particular derived data dependency: https://www.internalfb.com/code/fbsource/[97d3f14298ef3390a03e4f5051b093f9813f119d]/fbcode/eden/mononoke/derived_data/basename_suffix_skeleton_manifest_v3/mapping.rs?lines=75 Wihout this, the ordering of derivations won't deterministically ensure we derive `skeleton_manifests` before `bssm_v3`, so we can hit a race condition that results in the error observed in prod above. Reviewed By: RajivTS Differential Revision: D54949896 fbshipit-source-id: b8c5bbe28e9754845b79a3a71f1f9ee2ab239cb0 --- eden/mononoke/derived_data/utils/lib.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/eden/mononoke/derived_data/utils/lib.rs b/eden/mononoke/derived_data/utils/lib.rs index 6a57611bd348e..3abd2f5256035 100644 --- a/eden/mononoke/derived_data/utils/lib.rs +++ b/eden/mononoke/derived_data/utils/lib.rs @@ -144,7 +144,7 @@ lazy_static! { dag.insert(fsnodes, vec![]); dag.insert(deleted_mf_v2, vec![unodes]); dag.insert(skeleton_mf, vec![]); - dag.insert(bssm_v3, vec![]); + dag.insert(bssm_v3, vec![skeleton_mf]); dag.insert(git_tree, vec![]); dag.insert(git_commit, vec![git_tree]); dag.insert(git_delta_manifest, vec![git_tree, git_commit, unodes]);