Locales, Pickles, and VisiData

[jgarte]

Hi,

I tried downloading punkt and inspecting the pickles with vd but I get the following error message:

~/Downloads/punkt$ vd english.pickle 
saul.pw/VisiData v3.0.2
unsupported locale setting
opening english.pickle as pickle

Can anyone advise on a good approach to debug this? I'll start reading the source code that raised that message and look into locales, a topic that always leaves me in a state of mystery when I try to read docs on GNU or other locales 🦆

Any advice is much appreciated.

[saulpw]

First of all, try using --debug, which should show a stack trace for more exceptions. With a loader it may be useful to also use -b (batch mode) so that it's single-threaded. And please report back, others may have similar issues!

[midichef]

The unsupported locale setting message is coming from here:

visidata/visidata/main.py

Lines 186 to 189 in 6fff636

	try:
	locale.setlocale(locale.LC_ALL, '')
	except locale.Error as e:
	vd.warning(e)

I don't have a good understanding of locales, but one place to start is to check your LANG environment variable:

% echo $LANG
en_US.UTF-8
% python
>>> import locale; locale.getlocale()
('en_US', 'UTF-8')

Or run locale.getlocale() in Python and search your shell's environment variables for those strings by running env in the shell.

In any case, I do not think Visidata can open this particular kind of .pickle file, because vd english.pickle tries to use the pandas loader to load it as a pandas DataFrame. It fails with this error:
pandas loader can only unpickle dataframes
The english.pickle file contains a specific kind of Punkt object: <nltk.tokenize.punkt.PunktSentenceTokenizer object>, as can be seen with:

#!/usr/bin/python3

import pickle
with open('english.pickle', 'rb') as fp:
    obj = pickle.load(fp)
print(repr(obj))