We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
To reduce the memory required for writing large dataframes, a new mode sync_filediffs is being implemented in the mysql.Connection class.
sync_filediffs
The approach is to do as much as possible out of memory. On receiving a dataframe, the df is written to disk.
The db table which should be updated is also downloaded chunkwise to disk.
Then the filediffs package is used to find the differences between the two dataframes and save them to disk.
filediffs
After that the update part and the delete part are read back into memory and the database is updated.
A first version is already implemented on the sync_filediffs branch.
Still open Issues are
query
The text was updated successfully, but these errors were encountered:
No branches or pull requests
To reduce the memory required for writing large dataframes, a new mode
sync_filediffs
is being implemented in the mysql.Connection class.The approach is to do as much as possible out of memory.
On receiving a dataframe, the df is written to disk.
The db table which should be updated is also downloaded chunkwise to disk.
Then the
filediffs
package is used to find the differences between the two dataframes and save them to disk.After that the update part and the delete part are read back into memory and the database is updated.
A first version is already implemented on the sync_filediffs branch.
Still open Issues are
query
method's output format. Changing it seems to be a breaking change.The text was updated successfully, but these errors were encountered: