Models recommended for each Preprocessors

[brunogcar]

i tried to compile a list of models recommended for each preprocessor, to include in a pull request im preparing and a wiki i plan to help expand for controlnet

some are obvious, but others aren't that much, specially to new users

canny -> control_canny - t2iadapter_canny
mlsd -> control_mlsd
hed -> control_hed
scribble -> control_scribble - t2iadapter_sketch
fake_scribble -> control_scribble - t2iadapter_sketch
openpose -> control_openpose - t2iadapter_openpose - t2iadapter_keypose
openpose_hand -> control_openpose - t2iadapter_openpose
segmentation -> control_seg - t2iadapter_seg
depth -> control_depth - t2iadapter_depth
depth_leres -> control_depth - t2iadapter_depth
depth_leres_boost -> control_depth - t2iadapter_depth
normal_map -> control_normal
binary -> control_scribble - t2iadapter_sketch
color -> t2iadapter_color
pidinet -> control_hed
clip_vision -> t2iadapter_style

please let me know if i missed something or got it wrong, idea isnt to make a list of everything that woks, but which models are ideal for each preprocessor

[KurozumiGH]

@brunogcar
Hi,
I added the binary preprocessor, and the combination of this list is correct 👍

Thanks a lot

[alexopus]

Tried to update this mapping for ControlNet 1.1:

model -> preprocessor

model	preprocessor(s)
control_v11p_sd15_canny	canny
control_v11p_sd15_mlsd	mlsd
control_v11f1p_sd15_depth	depth_midas, depth_leres, depth_zoe
control_v11p_sd15_normalbae	normal_bae
control_v11p_sd15_seg	seg_ofade20k, seg_ofcoco, seg_ufade20k
control_v11p_sd15_inpaint	inpaint_global_harmonious?
control_v11p_sd15_lineart	lineart_standard (?), lineart_realistic, lineart_coarse
control_v11p_sd15s2_lineart_anime	lineart_anime
control_v11p_sd15_openpose	openpose (body), openpose_face, openpose_faceonly, openpose_full (body+hand+face), openpose_hand
control_v11p_sd15_scribble	scribble_hed, scribble_pidinet
control_v11p_sd15_softedge	softedge_pidinet, softedge_pidisafe, softedge_hed, softedge_hed_safe
control_v11e_sd15_shuffle (experimental)	shuffle
control_v11e_sd15_ip2p (experimental)	-
control_v11u_sd15_tile (unfinished)	tile_gaussian?

preprocessor -> model

preprocessor	ControlNet 1.1 model
invert (from white bg & black line)
canny	control_v11p_sd15_canny
depth_leres	control_v11f1p_sd15_depth
depth_midas	control_v11f1p_sd15_depth
depth_zoe	control_v11f1p_sd15_depth
inpaint_global_harmonious	control_v11p_sd15_inpaint
lineart_anime	control_v11p_sd15s2_lineart_anime
lineart_coarse	control_v11p_sd15_lineart
lineart_realistic	control_v11p_sd15_lineart
lineart_standard (from white bg & black line)	control_v11p_sd15_lineart (?)
mediapipe_face
mlsd	control_v11p_sd15_mlsd
normal_bae	control_v11p_sd15_normalbae
normal_midas	abandoned in 1.1
openpose	control_v11p_sd15_openpose
openpose_face	control_v11p_sd15_openpose
openpose_faceonly	control_v11p_sd15_openpose
openpose_full	control_v11p_sd15_openpose
openpose_hand	control_v11p_sd15_openpose
scribble_hed	control_v11p_sd15_scribble
scribble_pidinet	control_v11p_sd15_scribble
scribble_xdog
seg_ofade20k	control_v11p_sd15_seg
seg_ofcoco	control_v11p_sd15_seg
seg_ufade20k	control_v11p_sd15_seg
shuffle	control_v11e_sd15_shuffle
softedge_hed	control_v11p_sd15_softedge
softedge_hedsafe	control_v11p_sd15_softedge
softedge_pidinet	control_v11p_sd15_softedge
softedge_pidisafe	control_v11p_sd15_softedge
threshold
tile_gaussian	control_v11u_sd15_tile?

What about mediapipe_face? scribble_xdog? threshold?
Are there some preprocessors that are only relevant for ControlNet 1.0, like normal_midas? Is lineart_standard one of those?

@lllyasviel could you check if this makes sense and if anything is missing?

[lllyasviel]

since 1.1, cn begin to use standard naming, and memorizing correspondence is not necessary now. if model and preprocessor match a full word, they match. otherwise, they do not match. if no preprocessor matches, use none. threshold is for advanced users who understand what is happening and sometimes they want to threshold photos of scribbles drawn on paper. note that now t2i adapter also follow the word matching rule and it is not necessary to remember matches.

[andres885]

What's the difference between softedge_hed, softedge_hedsafe, softedge_pidinet and softedge_pidisafe?

[Walterever]