Segmentation on Javascript

[hpssjellis]

Anyone know how to get the segmentation MASK working as an overlay for a javascript image on a canvas.

See my Demo here

[radames]

hi @hpssjellis I've done this work on this notebook, but here's a fragment you can adapt

  const COLORS = [
      { r: 166, g: 206, b: 227 },
      { r: 31, g: 120, b: 180 },
      { r: 178, g: 223, b: 138 },
      { r: 51, g: 160, b: 44 },
      { r: 251, g: 154, b: 153 },
      { r: 227, g: 26, b: 28 },
      { r: 253, g: 191, b: 111 },
      { r: 255, g: 127, b: 0 },
      { r: 202, g: 178, b: 214 },
      { r: 106, g: 61, b: 154 },
      { r: 255, g: 255, b: 153 },
      { r: 177, g: 89, b: 40 },
  ];

  const imageSegRes = await hf.imageSegmentation({
    data: await FileAttachment("cats.png").blob(),
    model: "facebook/detr-resnet-50-panoptic"
  })

  const img = document.querySelector("#image");
  const canvas = document.querySelector("#canvas");
  const ctx = canvas.getContext("2d");
  canvas.width = img.width;
  canvas.height = img.height;

  const masks = await Promise.all(
    imageSegRes.map(
      (obj, index) =>
        new Promise((resolve, _) => {
          const image = new Image();
          image.src = "data:image/png;base64," + obj.mask;
          // make mask transparent
          image.onload = () => {
            const maskcolor = COLORS[index];
            const imgcanvas = document.createElement("canvas");
            const imgcanvasctx = imgcanvas.getContext("2d");
            imgcanvas.width = image.width;
            imgcanvas.height = image.height;

            imgcanvasctx.drawImage(image, 0, 0);

            const imageData = imgcanvasctx.getImageData(
              0,
              0,
              imgcanvas.width,
              imgcanvas.height
            );
            const data = imageData.data;

            for (let i = 0; i < data.length; i += 4) {
              const white = data[i] === 255;
              data[i] = maskcolor.r;
              data[i + 1] = maskcolor.g;
              data[i + 2] = maskcolor.b;
              data[i + 3] = white ? 255 : 0;
            }
            imgcanvasctx.putImageData(imageData, 0, 0);
            resolve(imgcanvas);
          };
        })
    )
  );
  ctx.drawImage(img, 0, 0);
  ctx.globalAlpha = 0.5;
  masks.forEach((mask) => {
    ctx.drawImage(mask, 0, 0);
  });

[hpssjellis]

@radames can you explain what this line does?

const maskcolor = d3.rgb(d3.schemePaired[index]);

The rest of it makes sense. I assume it just picks a mask color in rgb format, but not sure if it is a vanilla javascript funciton.

On second thought it is probably a d3.js funciton using <script src="//d3js.org/d3.v3.min.js"></script> I will give that a try.

[radames]

sorry that's just an array of colors, I'll fix here. You don't need necessarily d3 for that.

[hpssjellis]

That's a great notebook @radames !

[hpssjellis]

@radames All good. Not really sure why it works, but it works! demo here