Crashing exception after some time

[richiesgr]

I've deployed secor in K8s using the helm charts.

After some time completely random I get this this exception crash the pod:

java.lang.RuntimeException: Failed to apply upload policy
	at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:198)
	at com.pinterest.secor.consumer.Consumer.run(Consumer.java:179)
Caused by: java.util.concurrent.ExecutionException: com.google.cloud.storage.StorageException: java.lang.NullPointerException
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at com.pinterest.secor.uploader.FutureHandle.get(FutureHandle.java:36)
	at com.pinterest.secor.uploader.Uploader.uploadFiles(Uploader.java:164)
	at com.pinterest.secor.uploader.Uploader.checkTopicPartition(Uploader.java:317)
	at com.pinterest.secor.uploader.Uploader.checkTopicPartition(Uploader.java:339)
	at com.pinterest.secor.uploader.Uploader.applyPolicy(Uploader.java:363)
	at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:196)
	... 1 more
Caused by: com.google.cloud.storage.StorageException: java.lang.NullPointerException
	at com.google.cloud.storage.StorageException.translateAndThrow(StorageException.java:81)
	at com.google.cloud.storage.BlobWriteChannel.flushBuffer(BlobWriteChannel.java:180)
	at com.google.cloud.BaseWriteChannel.close(BaseWriteChannel.java:151)
	at com.pinterest.secor.uploader.GsUploadManager$1.run(GsUploadManager.java:125)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.getCurrentUploadOffset(HttpStorageRpc.java:783)
	at com.google.cloud.storage.BlobWriteChannel$1.run(BlobWriteChannel.java:125)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.storage.BlobWriteChannel.flushBuffer(BlobWriteChannel.java:70)
	... 7 more
2020-12-14 12:07:30,234 [Thread-1] (com.pinterest.secor.common.ShutdownHookRegistry:55) INFO  Running hooks for priority 1
2020-12-14 12:07:30,257 [Thread-1] (com.pinterest.secor.common.ShutdownHookRegistry:55) INFO  Running hooks for priority 10 

Any idea ?

[richiesgr]

Hi
Is there any idea what causing this ?
I can see that issue googleapis/java-storage#666
Is there any relation with secor.gs.upload.direct parameter ?
What should I use true or false ?
Thanks