Importing parts

[4jag]

I have a list of parts as a CSV or XLSX file, with fields SAP#, Manufacturers PN#, Manufacturers name, description, value, package, etc (35 fields/columns)

When I try to import it the system waits ~10 seconds, then I get http://localhost:1337/admin/part/part/import/ -- 502 Bad Gateway nginx/1.20.2. Do you think this is related to the file format, or is something else wrong?

Do I need to remove the first row of my bulk import file, which is field labels? Or does Inventree use the 1st-row text strings to match the database fields, so they can be in any order? And/or do I need to rearrange the columns to match the order that the fields are listed in the Docs? (In which case they fields would have to match precisely).

I'm confused about "parameters" -- We have a lot of fields (columns) for our parts that are related to lead-time, testing, classes, last modification, etc., that don't match Inventree's default fields. How do i add the necessary fields before I do a bulk import of parts? I tried following the process for adding a parameter, but that requires a template, then I'm supposed to add the parameter to a specific part or part template. ??? That doesn't work for just adding a generic field that's nonspecific to a particular part, template, or category.

Inventree, out of the box, has the following fields (below); and docs say that when importing data Inventree looks for an ID field. I tried a new, empty, left-most "id" column, and I tried changing my SAP# column to ID. Same Bad Gateway result.

category, default_location, default_supplier, category_name, variant_of, suppliers, in_stock, on_order, used_in, allocated, building, id, name, is_template, description, keywords, IPN, revision, link, image, default_expiry, minimum_stock, units, assembly, component, trackable, purchaseable, salable, active, virtual, notes, creation_date, creation_user, responsible, base_cost, multiple

When I tried importing a BOM the system give me the opportunity to match fields, but those fields weren't the same list for parts.

Do you have any walk-thrus on this type of data importing?

Thanks

EDIT:
I tried again to import a BOM (even though I think I need to import parts first), and after opening a file via browse-file dialog the system seemed to read the file... but then when I hit submit the result was "bad request 400". Prior to submit I got the opportunity to match fields, but the set of choices doesn't match the part fields I listed above.

[SchrodingersGat]

The current method of uploading part information via the web front end has some issues. This is very similar to the problem we had with uploading BOM data:

• [BUG] 400 Bad Request when working with a larger BOM #2568
• [BUG] Worker timeout when importing BOM #2497
• [BUG] 502 Proxy Error when uploading BOM #2331

Which was (eventually) fixed in #2603

Work is planned to migrate these same improvements / fixes to the part import tool. However, it's a long way off (limited developer resources).

The best bet for bulk import currently is to use the importer supplied in the "admin" interface:

[4jag]

@matmair I created the category "3033" prior to trying to import (See the photo I attached in my previous post) -- is that what you mean? I did that.

From EEITech's comment that fields with an asterisk (using the function "Create Part" as a guide) are needed in the import file, I deduced that several fields are necessary. And from the error I received I deduced that at least "category" must have an identical category name defined in the database for all values that appearing in the category column of my input file prior to import.

You say that my category column can have empty values? (null? just can't have a non-blank value that there isn't a defined category with the same name?) I just tried clearing that column, and now "Bad Gateway". This happens at exactly 30-seconds, so i assume it's a timeout while processing the input file. I tried stripping my data down to just Name and Description columns, and still get "Bad Gateway" with 1500+ rows... I reduced this to 1250 rows, and find that when it can parse UNDER 30 SECONDS it returns a page (doesn't give Bad Gateway).

Now the problem is it complains about blank descriptions. (Even if description is blank, it should simply create a new db record for every row, with an incrementing primary key, regardless of whether there's a string in that field, except for fields you've define as needing to be matched to another table, such as category.) ... but you just said that category can be blank...

Reducing to 500 rows, with just "name" and "description", and blank "category" columns:

This error occurs 500 times -- "Description - This field cannot be blank". So, it's not reading the description?... This seems to be completely broken.
Could you please provide a sample file to try to import, just to prove that this import feature is not broken.

Our part data has many columns.

1. Can my import file have columns with headings that don't match any Inventree's set of columns, and those would just be ignored? right? (That's what the "none"'s mean?)
2. Must I change my input columns names to match the fields that Inventree uses (category, description, name, suppliers, keywords, link, image, notes, etc)? Would that work?
3. Is it true I can't create new columns (free-form fields) in the part table? I fully expected to be able to import our data with all our columns (whether or not I have to create those fields in Inventree tables prior to import), and be able to search, sort, and compare multiple BOMs based on that data.
4. Is adding our own fields to Parts and BOMs not possible?
5. Is changing the titles of Inventree's Part fields not possible?

Just curious -- you use the spelling "categorie"... isn't the Inentree expecting the spelling of the category header column text as "CATEGORY"?

All the while I'm trying to import an xlsx file. Are any of the other import formats working -- tsv, json, yaml? Could you please tell me what you know does work, rather than my fiddling with fields, row-lengths, formats, and tables.

[eeintech]

When I import an xlsx file, every line generates this error -- "PartCategory matching query does not exist". My file has X lines, and I get this error X-1 times, showing the data on line 2 with the first error, so it appears to understand that the first line is the table header/column names. I did create the category "3033", and ALL of my lines have this exact string in the category column. Any idea why this error is thrown? PS, I also tried using my "sap" column as id, but Inventree wants to store the id as the table key, so it requires a float (presumably with no decimal places). (PSS, I abandoned WSL-2 in favor of RHEL7 -- night and day)

Errors

    Line number: 1 - PartCategory matching query does not exist.
    None, 3033, 0566-2-15-01-21-27-10-0, None, None, 0566-2-15-01-21-27-10-0, MILL-MAX, CONN, PIN, RECEPT, SINGLE-PIN, 0.031 HOLE, None, None, None, None, M, None, 0, None, None, None, None, None, None, None

    Traceback (most recent call last):
    File "/home/inventree/.local/lib/python3.8/site-packages/import_export/resources.py", line 658, in import_row
    self.import_obj(instance, row, dry_run)

You are almost there!

Next, in this screenshot:

Click on the name (it's a link to the category details).

Here you can find out the category ID in the URL, for instance the following category has ID=13:

In your spreadsheet, in the category column, replace the name of the category (eg. 3033) with its ID.
Save it and re-try uploading it, it should be able to match the category now!

I just tested it with this spreadsheet (note that category with ID=13 exists in my test database):
Test_parts_import.zip

This is what I get when I import it:

And after confirming the import, here it is:

[4jag]

@eeintech Thank you for that! Success--changing the data in category column to the Inventree category db ID gets past that error! But why they require the ID and not the string-name of the category is dumbfounding--it seems importing will take considerable reformatting. This isn't really a problem for one-time imports, but our engineers are NOT going to do this type of data conversion when they import BOMs several times per day.

Ok, so now the Description field still isn't being read. I get an error for every line, so I assume it's just not accepting certain characters (we have %, #, commas, etc in this field.)

I'm also getting an error on field "default_expiry" -- seems Inventree wants a float... (epoch timestamp?!?)

[eeintech]

Have you tried to change the header from "Description" to "description"?

Also not sure why you have the name of the company as name of your part. I would think you want the description field to be the name of your part.

[4jag]

Changing "Description" to "description" on the input file column header fixed the error. Yay! Thank you.
(I had tried different iterations of case on headings when I was struggling with the category problem -- but that was category-ID vs category-name)

The file I used in this latest sample wasn't as stripped down as others I had tried, that's all. We also have very specific fields that need to be part of basic reporting/calculations (not category-specific) -- e.g. Manufacturer's Part Number, Manufacturer's Equivalent PN, Lead Finish, Package Leak Testing, ESD Class, ESD (kVolts), radiation testing, thermal testing... I need to figure out how to incorporate all of these fields so that when the engineers upload one or more BOMs they can compare deltas among them.

Baby steps :)