test function : ParseTab

[Larbo53]

Hello,

I have just tested the ParseTab function and the result is not there. Where is my error?
environment :
PyMuPDF 1.19.4: Python bindings for the MuPDF 1.19.0 library.
Version date: 2022-01-01 00:00:01.
Built for Python 3.9 on darwin (64-bit).

Thanks for your feedback.
Sincerely

source : SBG_MGL_3310911671.png
result :ParseTab_27Jan2022.txt
function : test_ParseTab_27Janv2022.txt

[JorjMcKie]

there are no files attached

[Larbo53]

sorry

ParseTab_27Jan2022.txt
test_ParseTab_27Janv2022.txt

[Larbo53]

I don't understand what's going on with the png file.
Can you delete it so I can send it back to you in jpeg format?

Thank you.

Sincerely

[Larbo53]

the file

[JorjMcKie]

Thought about your problem:
I determined that these scripts are not a good starting point in your case for the following reasons:

1. They are several years old (and I never overhauled them). Many new features and methods have been added since there creation.
2. Your file is actually a case that can be solved more directly. There also is the lucky circumstance, that each page has the table layout.

Why not use this approach:
I think you already have the coordinates of the table lines and columns.
I suggest that you take these and define a 2-dimensional list of rectangles like these:

It seems that the same layout can be reused for each page ...

Then simply extract the text for each of the table cells like this:

tab = [[""] * cols] * rows # extracted cell text lands here
for j in rows:
    for i in cols:
        tab[j][i] = page.get_textbox(cell[j][i])

[Larbo53]

Good morning,

thank you. But I can't define the cells from the table (bbox) and the vertical rows (columns) to extract the content with a row by row return.
Can you give me the code?
Thank you.
Thank you very much.

[JorjMcKie]

I saw you have the coordinates of the vertical lines that separate the columns.
If you can find out also the horizontal line coordinates, then you have what you need:
From the column coordinates, make column rectangles. For example, the first one would be fitz.Rect(24.7, 272.2, 69.71, 710.82), etc.
From the horizontal coordinates make row rectangles in a similar way.
At the end you have two lists of rectangles cols and rows.
Then the above code snippet would look like this:

tab = [[""] * cols] * rows # extracted cell text lands here
for j, row in enumerate(rows):
    for i, col in enumerate(cols):
        tab[j][i] = page.get_textbox(row & col)  # look at the intersections of each row and column

You could of course also try to do without row coordinates. The problem with your PDF is however, that many table cells consist of multiple lines. This will make it hard or impossible to find the correct row / cell for any piece of text.

[Larbo53]

Good morning,

thank you for your help. In fact, I have the solution with the pdfplumber library. But it's an old library and Fitz has more features that interest me, that's why I'm looking for the equivalent with Fitz's functions. I attach the function "recup_schema_tableau(Identifier,doc)" where identifier is the sender of the invoice and doc the pdf file. The SQL query returns the position of each zone, i.e. the position of the vertical lines. It is a function like this one that I am looking for in Fitz.

Sincerely.

extraction_tab.txt

[JorjMcKie]

You can extract drawings (lines, rectangles, ...) with PyMuPDF as well!
If you let me have the PDF, I can give it a try.

[Larbo53]

Here is. Tanks a lot. Yves Larbodiere EVARD 3, rue des Courtes Terres 95220 HERBLAY portable : 07 81 08 41 00 mail : ***@***.***

…

Le 29 janv. 2022 à 14:20, Jorj X. McKie ***@***.***> a écrit : You can extract drawings (lines, rectangles, ...) with PyMuPDF as well! If you let me have the PDF, I can give it a try. — Reply to this email directly, view it on GitHub <#39 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AU2RHDFNQBONXA3AS2KYWTLUYPSQNANCNFSM5M6AR3IQ>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.

[JorjMcKie]

file was not attached 😒

[Larbo53]

here is

Than
SBG_MGL_3310911671.pdf
ks a lot

[JorjMcKie]

Here is script that should be good enough for a start:
test.zip

[Larbo53]

Good morning,

thank you for this script which works perfectly. I will test it on a dozen different invoices. This script solves the problem of the general conditions which are for some on the last page.
Congratulations for this solution.
Another question: in which country do you live?
Thanks again for this. Your scripts allow me to discover all the potentialities of pymupdf.
Best regards.

[JorjMcKie]

Glad it works for you.

Another question: in which country do you live?

I am a German living in beautiful Venezula since my retirement (more than 12 years now).

[JorjMcKie]

I can only encourage you to uncover more of PyMuPDF. For example, instead of using page.get_textbox(), you could also use other text extraction variants, that may offer more flexibility in some situations.

For example, page.get_text("dict") delivers a hierarchy of blocks, lines and so-called text "spans" with a lot of information beyond the sheer position information, like also fontsize, fontname, text color and more.
You could take such a span and determine if it almost fits in one of the table cells by comparing the intersection size it has, and so on.

[Larbo53]

Good morning,

among the dozen invoices, there is the one attached for which there is no horizontal line in the table, on the one hand. I also tried to make an approach by the blocks. The text area of the blocks appears without separator for each cell concerned.
I try to deepen the page.get_drawings() command in order to understand and solve the extraction of the zones of this pdf.

Kind regards.
blocks_chomette.txt
test_chomette.txt
CHOMETTE__FACTURE 3110943.pdf

[Larbo53]

Hi,
test_chomette_V2.txt
CHOMETTE__FACTURE 3110943.pdf
to solve the problem of invoices without horizontal lines, I will use the function page.get_text("blocks") to get the y0 and y1 in a list from blocks filtered by the VAT code for example, here the value '03'. I did a test that returns no information!
I attach the program and the pdf.
Thank you.
Sincerely.

[JorjMcKie]

Yes, very good.
If there are tables without borders that are easily detectable via lines, you always are dependent on some kind of other heuristics. At times, you may know the text of the column headings, which then helps to at least locate the table start.
Or you can perform some statistical analysis like detecting that many words, text spans or text blocks have the same left (or right) coordinate values, etc.
But it remains an area of uncertainty. I have seen others who use AI for text analysis, like @zeusrb - althouh I believe not explicitely for table detection.
Part of PDF's specification also formally covers table embedding. But this is not often used, not explicitely supported by MuPDF, and, as usual in PDF, very complex to understand and use.