Overlapping text

[sttpvvoyer]

Hi,

I'm struggling to determine the best approach. Any help is appreciated. I got some overlapping text. See column Comments that could spawn in the background theorically over all columns depending of it's variable length.

I don't care about losing data out of Comments. But i need to make sure other columns data is valid.

How can i do that using get_textbox?

Here what i have so far:
civic_data = {'MYKEY' : mykey, 'CIVIC': page.get_textbox(fitz.Rect(posCivic.x0, firstElemHeight, posPdrType.x0, nextElemHeight-1)) ,'COMMENTS' : page.get_textbox(fitz.Rect(posCommentsPoc.x0-5, firstElemHeight,posCivic.x0-2, nextElemHeight-1)) ,'PDRTYPE' : page.get_textbox(fitz.Rect(posPdrType.x0, firstElemHeight,posAm.x0-5, nextElemHeight-1)) ,'AM' : page.get_textbox(fitz.Rect(posAm.x0-5, firstElemHeight,posNWPV.x0-5, nextElemHeight-1)) ,'LNM_V' : page.get_textbox(fitz.Rect(posNWPV.x0-5, firstElemHeight,posCc.x0-5, nextElemHeight-1)) ,'CC' : page.get_textbox(fitz.Rect(posCc.x0+5, firstElemHeight,posMMV.x0-5, nextElemHeight-1))}

For example the line with Civic 7125-6 is returning me the following with a break line between "HE ARTS" and "7125-6":

HE ARTS

7125-6

How can i have text_box only returning "7125-6"?

[JorjMcKie]

The table creator did not care about confining table cells content inside the designated table cell. So all I can think of is doing a text extraction by words get_text("words"):
This will deliver you rectangle coordinates for every string surrounded by space. So one of the items of that list should be (x0,y0,x1,y1, "7125-6", ...)`.

[sttpvvoyer]

Ok thanks! Will try it in the hope that it will not slow down the extract.

So no way for the extractor to determine what is "on top" or "visible" in the pdf versus what is in the background?

I'm also wondering why a break line is inserted? Why is PyMuPDF inserting that?

Could i do something like a split on the line break and be confident that it will give the good data you think?
Something like:
page.get_textbox(fitz.Rect(posCivic.x0, firstElemHeight, posPdrType.x0, nextElemHeight-1)).split('\n')[-1]

[JorjMcKie]

Oh sure, you can determine which object is above another one: it simply is determined by the extraction sequence (you mustn't have used sorting in any way of course).

I'm also wondering why a break line is inserted? Why is PyMuPDF inserting that?

This is done by MuPDF (not PyMuPDF). During extraction (in MuPDF, which means when building the intermediate object TextPage), acomplex heuristics-based algorithm determined which characters are making up words, where to generate artificial spaces (which often are not written at all) ... all that based on things like font characteristics (bold, italic specifically) with font size and more. If text pieces are too far apart horizontally or vertically (bottom coord.), then always line breaks will be inserted.
You need some own logics to determine that this is not always a new line "below", but may just be the content of the next table cell.

Our package pymupdf4llm has a few functions to do that for you.

import pymupdf
from pymupdf4llm.helpers.get_text_lines import get_text_lines

doc = pymupdf.open("input.pdf")
page = doc[0]
text = get_text_lines(page)
...

This extracts text in reading order (works in most cases ...).
Text spans in the same line having a large-enough distance are separated by \t characters.
Note that the underlying approach sorts the text multiple times, so it can't be used to find out "above"/"below".

If you also need rectangles, e.g. for the full line or the spans within the line, you also use

...
from pymupdf4llm.helpers.get_text_lines import get_raw_lines

This guy delivers (bbox, spans) for each line. Parameter here is a TextPage (not a page). The function is internally used by get_text_lines, too.