Incorrect texture rendering for 3D meshes from .obj file

[anoopsonar30]

I've been trying to render 3D meshes from the S3DIS database which can be found at - http://buildingparser.stanford.edu/dataset.html

The render for area_3 in pybullet looks like this -

where the textures are all messed up and in the incorrect places. Moreover the roof isn't rendered at all.
On importing the same .obj file into a software like 3DS Max I got the following perfect renders -

Moreover, I've been trying to render a husky robot on the floor of this 3D environment. However, pybullet renders both area_3 and husky for a split second after which area_3 disappears and only the husky is left. Could someone look into this? The 3D environment above is rendered using the createMultiBody() function using the .obj file and the husky is rendered from it's URDF.

[erwincoumans]

PyBullet (and the underlying Bullet Robotics APIs) rendering has some limitations:

1. wavefront obj files can have at most 1 texture
2. there is a limited pre-allocated memory for visual meshes

You can use obj2sdf utility to get around (1), it creates additional obj files for each material/texture. Use premake to build Bullet, you get the App_obj2sdf executable (it is in the 'Extras' folder).

You can try increasing the default memory allocation to get around (2), or reduce/simplify the mesh/obj file. See https://github.com/bulletphysics/bullet3/blob/master/examples/OpenGLWindow/SimpleOpenGL3App.h#L19, increase the maxShapeCapacityInBytes.

Can you share the full reproduction case, including PyBullet python file and the actual obj file and textures in some github repo?

By the way, the Stanford GIBSON project also uses PyBullet and its own renderer, which may be able to render those datasets better. Still, it would be good to fix PyBullet to allow to render this too.

[Akshay6077]

Can you give me a little more info about how can I use the App_obj2sdf executable I am also facing similar issues while rendering

[danielpmorton]

Hi @erwincoumans, this seems to be an old thread but I'll revive it to ask a very similar question. I have an environment with a complicated OBJ with multiple textures (just like in the above thread), and to get it to load, I've used the obj2sdf program and then loaded each individual component separately.

However, I'd like to combine these into one entity (MultiBody?) where I have these multiple visual components (from the obj2sdf results) and one collision component (which I already computed with VHACD on the full OBJ).

Is this possible? I tried a VisualShapeArray, but it seems this has the same issue where it can only handle 1 texture. If this is not possible, I was thinking about just loading all of the visual elements with no collision (somehow, not totally sure how), and then loading an invisible body for my VHACD result with collision enabled.

Wondering if you have any advice on approaching this? Thanks!

(as a side note, it seems that the loadSDF() function doesn't load the textures even when the OBJs can each be loaded individually with their textures - if this isn't supposed to happen, let me know)

[Akshay6077]

HI @danielpmorton did you get any solution for this did the use of obj2sdf work for you I am also trying to simulate a terrain like structure
which has little complex materials, so wondering did you figure this out?

[danielpmorton]

Yeah, I figured this out. I wrote a bunch of info in a docs page for a still-unpublished repository. I'll paste what I wrote below, since it's not public yet. I made a few small changes to make it a bit more general, but there might still be some artifacts of my specific workflow.

Loading complex meshes into Pybullet

Loading complex meshes with lots of textures into pybullet is tricky. Ideally, we'd be able to just load a URDF and call it a day, but that just doesn't seem to be possible if you want all of the textures to appear. So, we have to resort to loading multiple OBJs.

If you load an OBJ file through createVisualShape(), this will look for an MTL file in the same directory and apply it to the OBJ. However, if the OBJ has multiple bodies each with different textures, pybullet will mess this up and apply only one of the textures to every body in the OBJ. To fix this problem, we have to separate the complex mesh into multiple OBJs, with each OBJ corresponding to a single texture. We can use Erwin's obj2sdf tool for this, even if we're not using an SDF.

For the collision side of things, if we just import a mesh into pybullet, the collision body will not be the mesh itself, but rather the convex hull of the entire body (which is not useful if you want to go inside the mesh). Running VHACD is the solution here - this will give us a decomposition of the mesh into multiple convex hulls and allow for it to be hollow.

Once we have the OBJs for each texture, and the VHACD OBJ for collision info, we can start loading these into pybullet. We now have some number n different obj files which represent the visual components of the mesh, but we only have 1 VHACD file for the mesh. So, we'll load the first visual component paired with the VHACD collision information, and for the remainder of the bodies, we'll load them strictly as visual elements by setting the collision ID in the multibody as -1. Note: it does not appear to be possible to create an "invisible" collision body, because if you set the visual ID in the multibody as -1, it will still visualize the collision mesh but in a bunch of random colors.

A full workflow of the steps required to go from meshes to a correct pybullet environment can be found below:

0. Folder structure

This is what the directory will look like at the end of the process:

meshes
└── obj
    ├── your_part_1
    │   ├── your_part_1.mtl # From Blender OBJ export
    │   ├── your_part_1.obj # From Blender OBJ export
    │   ├── decomp.mtl # From vhacd: Unused
    │   ├── decomp.obj # From vhacd: Collision body
    │   ├── decomp.stl # From vhacd: Unused
    │   ├── newsdf.sdf # From obj2sdf: Unused
    │   └── part*.obj # From obj2sdf: One obj per texture
    ├── your_part_2
    ├── your_part_3
    ├── ...
    └── textures
        └── *.png

1. Fix problematic texture images

Certain texture PNG images are unable to be loaded into pybullet. After some debugging, this appears to be an issue with the PNG filetype - some were exported as PNG8 whereas others are PNG16/24. Pybullet seems to need PNGs to be 8-bit, so these need to be converted.

To do so, import each problematic PNG into Gimp and then export/overwrite the image to the same location, with the pixelformat dropdown set to 8bpc RGBA

2. OBJ2SDF

(Ensure that the C++ version of Bullet is locally built first)

• For each part OBJ in its respective folder (i.e. obj/your_part_x/your_part_x.obj), run obj2sdf as follows:
• For example:

  cd obj/your_part_x/
  ./BULLET_PARENT_DIR/bullet3/build_cmake/Extras/obj2sdf/App_obj2sdf --fileName="cupola.obj"
  

This will add newsdf.sdf and multiple part*.obj files, one per texture, into the obj/your_part_x folder.

3. VHACD

• For each part OBJ in its respective folder (i.e. obj/your_part_x/your_part_x.obj), run VHACD as follows:
• For example:

  cd obj/your_part_x/
  ./VHACD_PARENT_DIR/v-hacd-4.1.0/app/build/TestVHACD cupola.obj
  

This will add decomp.obj, decomp.mtl, and decomp.stl to the obj/part_x folder.

4. Final adjustments

Refine VHACD

• The VHACD output will most likely not be perfect. There seems to be no exact combination of parameters that leads to an exact decomposition of the part (see the vhacd readme for more info on these). In particular, VHACD tends to poorly represent the thin walls and circular regions. So, to fix this, open the decomp.obj VHACD result for each part in Blender and update the file manually (as in, replace any "bad" convex hulls with simple objects in Blender, like rectangular prisms).

Update the paths

• The paths at the top of the OBJ/MTL files from this output (part0, part1, ...) will be absolute paths, but pybullet works best with these as relative paths. These will need to be manually modified with a quick directory search/replace. For MTL files especially, ensure that these are pointing to files inside the obj/textures directory

5. Importing into Pybullet

For each part,

• For the first OBJ file, load it as a visual shape. Pybullet will see the associated MTL file in the same directory and apply the texture. For the collision body, load the full VHACD result for the entire part.
• For each other OBJ file, load the visual shape as before, but set the collision body to -1, so that it doesn't load any collision info

[Akshay6077]

My structure is ind of this I want to load this into the bullet environment. The problem is down sampled it still I was facing the similar issue. Do you think following all of these steps will help?

[danielpmorton]

I did this because I had an OBJ with multiple bodies and textures. You can open the OBJ in a text editor and see if this is the case for you, and if so, then yes these steps should help. But, if all you have is a single body and a single texture, then I don't know what the issue is.

[Akshay6077]

I'm trying to install the obj2sdf executable. Can you tell me how to do that?