Issue #264: Update qviz

[jorgeMarin1]

Description

Fixes #264 .

With the new qbeast format, when we try to run the OTree Index Visualization (qviz) it crashes, so we need to update it.

Type of change

Bug fix.

Checklist:

• New feature / bug fix has been committed following the Contribution guide.
• Add tests.
• Your branch is updated to the main branch (dependent changes have been merged).

How Has This Been Tested? (Optional)

The OTree Visualization (qviz) was run locally on my PC.

Also, a pyspark shell was created in order to do some tests at the implemented fix. Then, a huge table of 4.000.000 rows was created, full of fake data. From this table, three Qbeast tables were created and read as Delta tables.

From these tables, we compared each of their total number of elements with the sumatory of the element_count of their respective cubes. Since these values matched, we concluded it worked as it should.

[jorgeMarin1]

Cubes are obtained by using: cubes = df["tags.blocks"]

These cubes are in a panda Series and have the following structure (this is one of these cubes):
[{"cubeId":"wggg","minWeight":-103008559,"maxWeight":2147483647,"elementCount":9521,"replicated":false},{"cubeId":"wggQ","minWeight":-100165008,"maxWeight":2147483647,"elementCount":880,"replicated":false},{"cubeId":"wgg","minWeight":-1401341541,"maxWeight":-103459089,"elementCount":9759,"replicated":false},{"cubeId":"wggA","minWeight":-102673026,"maxWeight":2147483647,"elementCount":5982,"replicated":false},{"cubeId":"wggw","minWeight":-91591261,"maxWeight":2147483647,"elementCount":1242,"replicated":false}]

Each of these are blocks, which are related to a cube through the cubeId.

[jorgeMarin1]

This is the structure of the metadata for revision_id = 1 :

Metadata: {'revisionID': 1, 'timestamp': 1726134712571, 'tableID': '/Users/jorgemarin/Documents/qviz-bug/qbeast-spark/utils/visualizer/tests/resources/test_table_delta_multiples_writes/', 'desiredCubeSize': 10000, 'columnTransformers': [{'className': 'io.qbeast.core.transform.LinearTransformer', 'columnName': 'user_id', 'dataType': 'IntegerDataType'}, {'className': 'io.qbeast.core.transform.StringHistogramTransformer', 'columnName': 'brand', 'dataType': 'StringDataType'}], 'transformations': [{'className': 'io.qbeast.core.transform.LinearTransformation', 'minNumber': 274969076, 'maxNumber': 566347286, 'nullValue': 391116828, 'orderedDataType': 'IntegerDataType'}, {'className': 'io.qbeast.core.transform.StringHistogramTransformation', 'histogram': ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']}]}

[Jiaweihu08]

Can you add ruff for code formatting? @jorgeMarin1

[Jiaweihu08]

Also, remove the redundant test tables. You can use the most complex one to cover all the test cases.
You don't need the JSON log files that end with crc.

[osopardo1]

Some small things that I noticed while reviewing !

[Jiaweihu08]

I'm closing this PR as I'm not allowed to update the branch. We will be working on this issue with #437 instead.