Transaction ingester aborted due to broken data in data store?

[coxchen]

Hi

I'm testing Crux + JDBC + Postgres, and I've encountered some problem ingest data and recover, further ingestion seems to be blocked by some broken data in data store. The exception stacktrace is as below:

Exception in thread "crux-polling-tx-consumer" clojure.lang.ExceptionInfo: Thaw failed: Decryption/decompression failure, or data unfrozen/damaged. {:opts {:compressor :auto, :encryptor :auto}}
	at taoensso.nippy$thaw$fn__14031$ex__14032.invoke(nippy.clj:1859)
	at taoensso.nippy$thaw$fn__14031$fn__14053.invoke(nippy.clj:1910)
	at taoensso.nippy$thaw$fn__14031$thaw_data__14034.invoke(nippy.clj:1885)
	at taoensso.nippy$thaw$fn__14031.invoke(nippy.clj:1909)
	at taoensso.nippy$call_with_bindings.invokeStatic(nippy.clj:1274)
	at taoensso.nippy$call_with_bindings.invoke(nippy.clj:1270)
	at taoensso.nippy$thaw.invokeStatic(nippy.clj:1852)
	at taoensso.nippy$thaw.invoke(nippy.clj:1828)
	at taoensso.nippy$thaw.invokeStatic(nippy.clj:1841)
	at taoensso.nippy$thaw.invoke(nippy.clj:1828)
	at crux.jdbc$eval23514$fn__23515.invoke(jdbc.clj:32)
	at clojure.lang.MultiFn.invoke(MultiFn.java:234)
	at crux.jdbc.JdbcDocumentStore$fn__23589$fn__23590.invoke(jdbc.clj:111)
	at clojure.core$juxt$fn__5822.invoke(core.clj:2594)
	at clojure.core$map$fn__5866.invoke(core.clj:2753)
	at clojure.lang.LazySeq.sval(LazySeq.java:42)
	at clojure.lang.LazySeq.seq(LazySeq.java:51)
	at clojure.lang.RT.seq(RT.java:535)
	at clojure.core$seq__5402.invokeStatic(core.clj:137)
	at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:24)
	at clojure.core.protocols$fn__8146.invokeStatic(protocols.clj:75)
	at clojure.core.protocols$fn__8146.invoke(protocols.clj:75)
	at clojure.core.protocols$fn__8088$G__8083__8101.invoke(protocols.clj:13)
	at clojure.core$reduce.invokeStatic(core.clj:6828)
	at clojure.core$into.invokeStatic(core.clj:6895)
	at clojure.core$into.invoke(core.clj:6887)
	at crux.jdbc.JdbcDocumentStore$fn__23589.invoke(jdbc.clj:112)
	at crux.io$with_nippy_thaw_all_STAR_.invokeStatic(io.clj:153)
	at crux.io$with_nippy_thaw_all_STAR_.invoke(io.clj:151)
	at crux.jdbc.JdbcDocumentStore._fetch_docs(jdbc.clj:104)
	at crux.document_store.CachedDocumentStore._fetch_docs(document_store.clj:54)
	at crux.fork.ForkedDocumentStore._fetch_docs(fork.clj:176)
	at crux.db$fetch_docs.invokeStatic(db.clj:84)
	at crux.db$fetch_docs.invoke(db.clj:78)
	at crux.tx.InFlightTx.index_tx_events(tx.clj:293)
	at crux.tx$index_tx_log$fn__22093$fn__22098.invoke(tx.clj:438)
	at crux.tx$index_tx_log$fn__22093.invoke(tx.clj:429)
	at crux.tx$index_tx_log.invokeStatic(tx.clj:421)
	at crux.tx$index_tx_log.invoke(tx.clj:419)
	at crux.tx$__GT_polling_tx_consumer$fn__22114.invoke(tx.clj:464)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.lang.Thread.run(Thread.java:745)
Caused by: clojure.lang.ExceptionInfo: Thaw failed against type-id: 112 {:type-id 112}
	at taoensso.nippy$thaw_from_in_BANG_.invokeStatic(nippy.clj:1773)
	at taoensso.nippy$thaw_from_in_BANG_.invoke(nippy.clj:1583)
	at taoensso.nippy$thaw$fn__14031$thaw_data__14034.invoke(nippy.clj:1883)
	... 39 more
Caused by: java.io.EOFException
	at java.io.DataInputStream.readByte(DataInputStream.java:267)
	at taoensso.nippy$thaw_from_in_BANG_.invokeStatic(nippy.clj:1590)
	at taoensso.nippy$thaw_from_in_BANG_.invoke(nippy.clj:1583)
	at taoensso.nippy$read_kvs_into$fn__13984.invoke(nippy.clj:1436)
	at clojure.lang.LongRange.reduce(LongRange.java:233)
	at clojure.core$reduce.invokeStatic(core.clj:6827)
	at clojure.core$reduce.invoke(core.clj:6810)
	at taoensso.encore$reduce_n.invokeStatic(encore.cljc:910)
	at taoensso.encore$reduce_n.invoke(encore.cljc:909)
	at taoensso.nippy$read_kvs_into.invokeStatic(nippy.clj:1436)
	at taoensso.nippy$read_kvs_into.invokePrim(nippy.clj)
	at taoensso.nippy$thaw_from_in_BANG_.invokeStatic(nippy.clj:1668)
	... 41 more

I can start over with a clean database. But I'd like to learn if there is any way to recover/detect/prevent such situation in production. Please help. Thank you,

• Cox

[refset]

Hi @coxchen, sorry for the delayed response, are you able to summarise the kinds of values your documents contain? Is it all regular edn types?

Also, which version of Crux are you using? Have you only been using that same version, or did you upgrade at some point? And, just in case, which version of Postgres?

I think the only ways to recover from this situation are:

1. upgrade to a patched version of Crux, once we've identified and solved the issue
2. manually insert an eviction tombstone in place of the offending document
3. start over with a clean database

[refset]

Hi again, in case you're still interested in solving this, you should be able to locate the problematic document using latest-completed-tx. The EOF error suggests there has been truncated doc submitted somewhere along the line. Did you still have a copy of the older data store?

For future reference:
https://github.com/ptaoussanis/nippy/blob/ba8827708ea9da828735562244d151b59eb692a7/src/taoensso/nippy.clj#L161

[jarohen]

Created #1422 to track this - thanks @coxchen for the report 🙏