Adding with_row_index() to a CSV LazyFrame does not always add the column

[kgutwin]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import io
import polars as pl

csvf = "a,b\n1,10\n2,20\n3,30"
lf = pl.scan_csv(io.BytesIO(csvf.encode()))

lf.collect()  # necessary to trigger the bug
print(lf.with_row_index().collect())

Output:

shape: (3, 2)
┌─────┬─────┐
│ a   ┆ b   │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1   ┆ 10  │
│ 2   ┆ 20  │
│ 3   ┆ 30  │
└─────┴─────┘

---

Alternative misbehavior:

import io
import polars as pl

csvf = "a,b\n1,10\n2,20\n3,30"
lf = pl.scan_csv(io.BytesIO(csvf.encode()))

lf.with_row_index().collect()  # necessary to trigger the bug
print(lf.collect())

Output:

shape: (3, 3)
┌───────┬─────┬─────┐
│ index ┆ a   ┆ b   │
│ ---   ┆ --- ┆ --- │
│ u32   ┆ i64 ┆ i64 │
╞═══════╪═════╪═════╡
│ 0     ┆ 1   ┆ 10  │
│ 1     ┆ 2   ┆ 20  │
│ 2     ┆ 3   ┆ 30  │
└───────┴─────┴─────┘

This output has the row index, but it shouldn't.

Log output

read files in parallel
file < 128 rows, no statistics determined
no. of chunks: 1 processed by: 1 threads.

Issue description

When using with_row_index() on a LazyFrame derived from scan_csv(), making a call to collect() seems to "freeze" the LazyFrame in that state and no further calls to with_row_index() have the desired effect.

• If you create the LazyFrame via scan_csv() and then immediately call lf.collect(), a subsequent call to lf.with_row_index() will not add the index column.
• If you create the lazy frame and then immediately call lf.with_row_index(), the LazyFrame will always have the index column, even if you call lf.collect() on the original LazyFrame reference.

This was only tested to be found with LazyFrames from CSV files; a LazyFrame created from a dict will not show this behavior.

This behavior is not present in polars 1.7.1.

Expected behavior

Python 3.11.9 (main, Apr  2 2024, 08:25:04) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import io
>>> import polars as pl
>>> pl.__version__
'1.7.1'
>>> 
>>> csvf = "a,b\n1,10\n2,20\n3,30"
>>> lf = pl.scan_csv(io.BytesIO(csvf.encode()))
>>> 
>>> lf.collect()  # try to trigger the bug
shape: (3, 2)
┌─────┬─────┐
│ a   ┆ b   │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1   ┆ 10  │
│ 2   ┆ 20  │
│ 3   ┆ 30  │
└─────┴─────┘
>>> print(lf.with_row_index().collect())  # expected output
shape: (3, 3)
┌───────┬─────┬─────┐
│ index ┆ a   ┆ b   │
│ ---   ┆ --- ┆ --- │
│ u32   ┆ i64 ┆ i64 │
╞═══════╪═════╪═════╡
│ 0     ┆ 1   ┆ 10  │
│ 1     ┆ 2   ┆ 20  │
│ 2     ┆ 3   ┆ 30  │
└───────┴─────┴─────┘
>>> # reset
>>> lf = pl.scan_csv(io.BytesIO(csvf.encode()))
>>> 
>>> lf.with_row_index().collect()  # try to trigger the bug
shape: (3, 3)
┌───────┬─────┬─────┐
│ index ┆ a   ┆ b   │
│ ---   ┆ --- ┆ --- │
│ u32   ┆ i64 ┆ i64 │
╞═══════╪═════╪═════╡
│ 0     ┆ 1   ┆ 10  │
│ 1     ┆ 2   ┆ 20  │
│ 2     ┆ 3   ┆ 30  │
└───────┴─────┴─────┘
>>> print(lf.collect())  # expected output
shape: (3, 2)
┌─────┬─────┐
│ a   ┆ b   │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1   ┆ 10  │
│ 2   ┆ 20  │
│ 3   ┆ 30  │
└─────┴─────┘

Installed versions

--------Version info---------
Polars:              1.8.1
Index type:          UInt32
Platform:            macOS-14.3-arm64-arm-64bit
Python:              3.11.9 (main, Apr  2 2024, 08:25:04) [Clang 15.0.0 (clang-1500.3.9.4)]

----Optional dependencies----
adbc_driver_manager  <not installed>
altair               <not installed>
cloudpickle          <not installed>
connectorx           <not installed>
deltalake            <not installed>
fastexcel            0.11.6
fsspec               <not installed>
gevent               <not installed>
great_tables         <not installed>
matplotlib           <not installed>
nest_asyncio         <not installed>
numpy                2.1.0
openpyxl             3.1.2
pandas               2.2.2
pyarrow              17.0.0
pydantic             1.10.18
pyiceberg            <not installed>
sqlalchemy           <not installed>
torch                <not installed>
xlsx2csv             <not installed>
xlsxwriter           3.2.0

[coastalwhite]

Not 100% sure but this might be related to the file metadata caching work by @nameexhaustion.

[nameexhaustion]

Not 100% sure but this might be related to the file metadata caching work by @nameexhaustion.

Yes, with_row_index() mutates the DSL but still keeps the old cached IR