how to cast value when using df.func.where?

[Hasham04]

Hello, I have a file that I am r.eading where one of the columns has values in scientific notation. like 4.503021124e+10, My goal is to output this file I am reading as a CSV.

I have tried using.

df[my_col]=df[my].astype('str')
df[my_col]=df.func.where(df[my_col].str.contains('e')==True, str(int(df[my_col])),df[my_col])

I basically want :

4.503021124e+10 to look like 45030211240

However, this gives me an error saying int() does not take an Expression which makes sense. Any help is greatly appreciated. Thank you.

[JovanVeljanoski]

Can you provide a small example file that you use for input? or describe its contents. Hard to offer suggestions without reproducing your exact usecase.

[JovanVeljanoski]

Hey

Maybe you misunderstand things.. how the number appears in the csv is up to the "writer" (i.e. the pandas to_csv or the equivalent pyarrow function). How it looks like in vaex has no impact on that. So you need to find the settings in pyarrow that allows you to format the number as you see fit.

Btw, we do plan on supporting the pyarrow csv writer from vaex, and here is a general discussion about writing csv files faster from vaex: #1687

Btw, how often do you plan on exporting 100GB csv files? If it is a one time thing.. you can consider just waiting it out.. if you need to do it often then yeah.. it is worth considering faster alternatives.

[Hasham04]

@JovanVeljanoski

x = ['-129.66','-120','15','30','40','4.503021124e+10','5.503021124e+10']
df = vaex.from_arrays(x=x)
df['new_col']=df.func.where(df.x.str.contains('e')==True,df.x,'0')
df['new_col']=df['new_col'].astype('float')
df['new_col']=df['new_col'].astype('int64')
df['new_col']=df['new_col'].astype('str')

df['final']=df.func.where(df['x'].str.contains('e')==False,df['x'],df['new_col'])

shouldn't the values in column final be 45030211584 and 55030210560 instead of 4.50302e+10 and 5.50302e+10. Any help is greatly appeeciated,

[JovanVeljanoski]

Why did you just ignore my answer before? As i said.. the way vaex represents data on screen when you print or display things is not related to how they are saved when you write them to a csv file..

How vaex prints stuff to screen is related to some internal formatting..

[Hasham04]

Hi Jovan, the final column in my last comment is of type string. why is a string being formatted to be in scientific notation? We are no longer talking about float formatting when generating CSV. A string should be represented as is correct.

x = ['-129.66','-120','15','30','40','4.503021124e+10','5.503021124e+10']
df = vaex.from_arrays(x=x)
df['new_col']=df.func.where(df.x.str.contains('e')==True,df.x+'','0')
df['new_col']=df['new_col'].astype('float')
df['new_col']=df['new_col'].astype('int64')
df['new_col']=df['new_col'].astype('str')

df['final']=df.func.where(df['x'].str.contains('e')==False,df.x+'-',df['new_col']+'')

if I add '-' to the final column then the number is represented as I want it.

when I remove the '-' it goes back to being scientific notation.

[JovanVeljanoski]

"Why a strong is formatted in a scientific notation" is a different question then your original question in this thread. If you want to open a separate discussion on this, feel free to do so.

In short..

Say in stead of numbers you had text.. and one sample was a text entry with 10_000 characters. Do you want to see all of them? Would not be very nice.. you would probably see something like "Lorem ipsum dolor est ...". Something similar is done in pandas as well.

Here we recognize the string is integer so instead of doing 1234.... we can actually show you the whole thing!.

At least this is my assumption. This part of vaex is quite complicated (for me), so if you are reallly interested in it.. you'll have to look in the code yourself..

I will repeat again. The way the strings are formatted when you do print(df) or display(df) (in Jupyter notebook for example), has nothing to do with the actual representation internally, or when you do operations to them.

For instance, for saving data to CSV, vaex uses pandas under the hood. So you can easily do df.to_pandas_df() and see that the strings are now formatted the way pandas formats them. And since pandas is the writer, you need to make sure you format your data in the way you want to. It would be similar when you use arrow. Again this is the same explanation as in my first reply.. does not matter if you are talking about strings, ints etc..