Spans detected in page.get_text("dict") fails in a weird pdf format

[lnmduc2]

Description of the bug

As far as I know, the page.get_text("dict") API can access to spans bounding boxes (which contain texts with the same font styles). However, some pdfs are somewhat strangely encoded, and it seems like PyMuPDF cannot detect spans for these files.

How to reproduce the bug

When i use the span detection with the below code for the
no_bug.pdf file, the code works just fine and the spans are detected relatively accurate (as shown in this image)

)

import fitz  # PyMuPDF
import sys
import io
from PIL import Image, ImageDraw

def visualize_span_bbox(page, img_path, dpi=200):
		"""Visualize the span bounding boxes on the image."""
		# Get the page dimensions and set scale factor for DPI
		zoom_x = dpi / 72.0
		zoom_y = dpi / 72.0
		mat = fitz.Matrix(zoom_x, zoom_y)  # Create a transformation matrix for DPI scaling
		
		# Convert PDF page to image with specified DPI
		pix = page.get_pixmap(matrix=mat)
		img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)
		draw = ImageDraw.Draw(img)
		
		blocks = page.get_text("dict")["blocks"]
		for block in blocks:
				try:
					for line in block["lines"]:
							for span in line['spans']:
									span_bbox = [int(coord * zoom_x) for coord in span['bbox']]
									text = span['text']
									fontname = span["font"]
									font_flags = span["flags"]
									size = span["size"]

									is_bold = "Bold" in fontname
									is_italic = "Italic" in fontname or "Oblique" in fontname
									print(f"{text} : {fontname}: {size} : ({is_bold}, {is_italic})")
									draw.rectangle(span_bbox, outline="blue", width=2)
				except:
					print("ERROR")
								
		# Save the image with span bounding boxes
		img.save(img_path)

if __name__ == "__main__":
		doc = fitz.open(sys.argv[1])
		for page in doc:
				print("Text from page %i:" % page.number)
				visualize_span_bbox(page, f"result.png", dpi=200)

but when i changed into this pdf file (bug.pdf), it breaks (the bounding boxes are separated in a weird way, and the font styles bold/italic are also inaccurate):

Can someone tell me why does this happen?

PyMuPDF version

1.24.9

Operating system

Linux

Python version

3.10

[JorjMcKie]

1. When spans overlap each other then this happens because the PDF creator made it so. It is legal to do that!
2. Some creators actively do this: writing the same text twice to almost the same position lets the text look like bold, but saves the necessity to add a separate (bold) version of a font: resulting in smaller file size.
3. There are techniques to let text look bold or italic using only one font with no special properties:

import pymupdf

# simulate font effects
doc = pymupdf.open()
page = doc.new_page()
text1 = "This is normal text."
text2 = "This text simulates italic."
text3 = "This text simulates bold."
mat = pymupdf.Matrix(1, 0, 0.5, 1, 0, 0)
p = pymupdf.Point(100, 100)
page.insert_text(p, text1)
p += (0, 20)
page.insert_text(p, text2, morph=(p, mat))  # use a matrix to morph the text
p += (0, 20)
page.insert_text(p, text3)
page.insert_text(p + (0.5, 0), text3)  # write a second time with horizontal shift
doc.ez_save(__file__.replace(".py", ".pdf"))

Gives you this - all with the same font:

[lnmduc2]

Ok I understand, but as i visualized some more, I noticed that the spans are not overlapped with each other. However, it seems like the author uses many custom fonts in the pdf, and they are randomly spreaded like in this image (each color corresponds to a different base font name):

Color palette for the image above:
{'TTE3E6E5F0t00': (153, 246, 138), 'TTE2ACA850t00': (139, 236, 179), 'Times': (236, 104, 118), 'Helvetica': (206, 233, 105), 'TTE3E186A8t00': (177, 125, 121)}

Code to reproduce:

import fitz  # PyMuPDF
import sys
import io
import random
from PIL import Image, ImageDraw

def get_base_font_name(fontname):
    """Extract the base font name without style indicators."""
    base_name = fontname.split('+')[-1].split('-')[0]
    return base_name

def get_color_for_font(base_fontname, font_colors):
    """Get a consistent color for a given base font name."""
    if base_fontname not in font_colors:
        font_colors[base_fontname] = (random.randint(100, 255), random.randint(100, 255), random.randint(100, 255))
    return font_colors[base_fontname]

def visualize_span_bbox(page, img_path, dpi=200):
    """Visualize the span bounding boxes on the image."""
    zoom_x = zoom_y = dpi / 72.0
    mat = fitz.Matrix(zoom_x, zoom_y)
    
    pix = page.get_pixmap(matrix=mat)
    img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)
    draw = ImageDraw.Draw(img)
    
    font_colors = {}  # Dictionary to store colors for each base font name
    
    blocks = page.get_text("dict")["blocks"]
    for block in blocks:
        try:
            for line in block["lines"]:
                for span in line['spans']:
                    span_bbox = [int(coord * zoom_x) for coord in span['bbox']]
                    text = span['text']
                    fontname = span["font"]
                    font_flags = span["flags"]
                    size = span["size"]

                    base_fontname = get_base_font_name(fontname)
                    is_bold = "Bold" in fontname
                    is_italic = "Italic" in fontname or "Oblique" in fontname
                    print(f"{text} : {fontname} (Base: {base_fontname}): {size} : ({is_bold}, {is_italic})")
                    
                    # Get a consistent color for this base font name
                    color = get_color_for_font(base_fontname, font_colors)
                    
                    # Draw the bounding box with the font's color
                    draw.rectangle(span_bbox, outline=color, width=2)
        except:
            print("ERROR")
                        
    # Save the image with span bounding boxes
    img.save(img_path)
    print(font_colors)

if __name__ == "__main__":
    doc = fitz.open(sys.argv[1])
    for page in doc:
        print("Text from page %i:" % page.number)
        visualize_span_bbox(page, f"result.png", dpi=200)

1. How did the weird fonts (like the "TTE3E6E5F0t00" font, which is a custom font made by the author of the pdf, and not installed on my machine) are somehow recognized and displayed with the exact same styles as other fonts?
For example:
Nguyên t : Times-Bold (Base: Times): 11.269189834594727 : (True, False)
and
ắ : TTE3E186A8t00 (Base: TTE3E186A8t00): 11.269189834594727 : (False, False)

As can be seen, "Nguyên t" and "ắ" has different base fonts, but they are displayed as the exact same style on the image.

2. Are there any ways to "map" or "approximate" these weird fonts to a list of predefined fonts?
Like, i may want to map the TTE3E186A8t00 font to Times-Bold, because when I tested on a few "convert pdf to docx" tools, I noticed that TTE3E186A8t00 is somehow mapped to Calibri font.

[lnmduc2]

@JorjMcKie Thanks for your immediate reply. I've used various tools to parse the bug.pdf files such as pdf2docx, PyMuPDF, and PDF readers like PDFgear, but they all encounter the issue where the original text 'Bất động sản đầu tư' is falsely extracted as 'B t ñng s n ñu tư,' although some of the other words are extracted with the correct characters. On the other hand, when using the Aspose-Words package it detects 'Bất đống sản đầu tư' which is more correctly recognized though the font is slightly off. Do you have any suggestions to help accurately recognize characters while still using PyMuPDF?

[JorjMcKie]

Whatever Aspose-Words (which I don't know) does: there is no way to do it better if the usual PDF viewers including PyMuPDF are not able to extract the text properly.
It is a font or encoding problem in the PDF.