Allow missing values in CategoricalMatrix

[MartinStancsicsQC]

Checklist

• Added a CHANGELOG.rst entry

This PR adds support for missing values in CategoricalMatrix. The default behavior does not change (raise on missing categoricals). However, if a matrix is constructed with the new cat_missing_method parameter set to zero, then missing categorical values are allowed, and are treated as all-zero indicator rows in subsequent method calls. This new behavior is consistent with pandas.get_dummies' way of handling missing values.

Notes about performance/templating functions:

• When a categorical matrix has missing values, the *_complex (formerly *_drop_first) suite of functions is used, so this feature has no performance impact in the most important case (no drop_first and no missing values).
• Most of the changes on the C++/Cython side are very minor. The only bigger differences are in the _sandwich_cat_dense* and _sandwich_cat_cat* functions, which were not templated by drop_first before, but are now.
• Benchmarks show that performance differences between the *_fast and *_complex functions is negligible in most cases. The only exception is CategoricalMatrix.tocsr (and subsequently CategoricalMatrix.multiply), where the _*fast function is indeed faster (by 25% and 10%, respectively, on a 10,000,000 x 5 categorical matrix). This might imply that having two versions of these functions (like the newly templated Categorical._sandwich_cat_dense* and Categorical._sandwich_cat_cat, but maybe even Categorical.sandwich, Categorical.matvec and Categorical.transpose_matvec) may add more complexity than the benefit it brings.

[lbittarello]

What about adding the option to treat missing values as category in their own? :)

[MartinStancsicsQC]

Yes, that's a great idea, and I think it only takes a few extra lines in the __init__ methods to implement. And I believe CategoricalMatrix would then have full feature parity with pandas.get_dummies. I'll add it this afternoon.

[MartinStancsicsQC]

I've added the option to treat missings in categorical columns as their own category. What do you think, @lbittarello?

Couple of notes:

• This is implemented as a pre-processing step instead of complicating things on the Cython/C++ side. The only drawback is that we need to make a copy of categorical columns with missing values, but I don't think that is a big issue.
• In contrast to pandas.get_dummies, a new category is only added if the column actually contains missing values. I think this is more reasonable for our purposes, as all-zero columns make no sense when estimating a model.
• At the moment the name of the missing category is always '(MISSING)' (even when it already exists). It would be simple to make it parametrezable, if you think it's worth the extra __init__ parameter. The name of the missing category is (MISSING) by default, but parametrizable through the cat_missing_name argument. If cat_missing_name is already an existing category in the input vector and the latter also contains missing values, a ValueError is raised.

[MatthiasSchmidtblaicherQC]

We should clarify under which circumstances we want to have the same behaviour as pd.get_dummies (except for not expanding the dummy matrix), and where we want to deviate. Otherwise, looks great!

[MartinStancsicsQC]

@MarcAntoineSchmidtQC, let me know if we should merge it into main or aim for the next major release/refactor

[MarcAntoineSchmidtQC]

Amazing. The tests seem to cover a lot. I don't have anything to add here.

[MarcAntoineSchmidtQC]

We can merge to main right now. My understanding is that it's fully backward compatible.