Getting StreamWriterClosedException in BQ Storage Write Api

[SubhamSinghal]

I have been using java service which write records to BQ using storage api. Its been running fine for almost a week and then suddenly i see below StreamWriterClosedException in application log.

at bq.DataWriter.append(DataWriter.java:61)
	at bq.DataWriter$AppendCompleteCallback.onFailure(DataWriter.java:109)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:67)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
	at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:94)
	at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:76)
	at com.google.api.core.SettableApiFuture.setException(SettableApiFuture.java:51)
	at com.google.cloud.bigquery.storage.v1.StreamWriter.cleanupInflightRequests(StreamWriter.java:585)
	at com.google.cloud.bigquery.storage.v1.StreamWriter.appendLoop(StreamWriter.java:496)
	at com.google.cloud.bigquery.storage.v1.StreamWriter.access$1000(StreamWriter.java:51)
	at com.google.cloud.bigquery.storage.v1.StreamWriter$1.run(StreamWriter.java:221)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.google.cloud.bigquery.storage.v1.Exceptions$StreamWriterClosedException: FAILED_PRECONDITION: Connection is closed due to com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException: INTERNAL: An error occurred while verifying authorization. Entity: projects/project_id/datasets/dataset_name/tables/table_name/ Entity: projects/project_id/datasets/dataset_name/tables/table_name/_default
	at com.google.cloud.bigquery.storage.v1.StreamWriter.appendInternal(StreamWriter.java:327)
	at com.google.cloud.bigquery.storage.v1.StreamWriter.append(StreamWriter.java:287)
	at com.google.cloud.bigquery.storage.v1.JsonStreamWriter.append(JsonStreamWriter.java:166)
	at com.google.cloud.bigquery.storage.v1.JsonStreamWriter.append(JsonStreamWriter.java:109)

This is my build.gradle dependency for storage write api

implementation platform('com.google.cloud:libraries-bom:26.1.0')
implementation 'com.google.cloud:google-cloud-bigquerystorage'
implementation 'com.google.cloud:google-cloud-bigquery'

I am using spring boot version 2.7.2

here is issue link raised in bq forum as well: https://www.googlecloudcommunity.com/gc/Data-Analytics/Getting-StreamWriterClosedException-in-BQ-Storage-Write-Api/m-p/476293#M766

[Neenu1995]

Hi @SubhamSinghal,
We need more information to debug the issue. Can you provide a code snippet and details on what changed in the project setup: change in permissions, what resource is encountering the issue etc, authentication method ?

[SubhamSinghal]

Hi @Neenu1995
We are using gcp service account for authentication. There is no change in permission as i have observed that issue got resolved after restart.

Here is code snippet for BQ default storage api client:
BQClient.java

public void loadData(JSONArray jsonArray, String tableName)
            throws DescriptorValidationException, InterruptedException, IOException {

        TableName parentTable = TableName.of(projectId, bqDataSetName, tableName);
        DataWriter writer = bqStreamRepository.getDataWriter(parentTable);

        writer.append(new AppendContext(jsonArray, 0));
        int size = jsonArray.length();
        log.info("Appended {} records successfully.", size);
    }

DataWriter code is same as mentioned here storage api java client

We are managing DataWriter objects in a hashMap, this to have a single DataWriter instance per table for project lifecycle.
Code:

BqStreamRepository.java

  private Map<String, DataWriter> dataWriterMap = new ConcurrentHashMap<>();

     public DataWriter getDataWriter(TableName tableName) throws Descriptors.DescriptorValidationException, IOException, InterruptedException {
         DataWriter dataWriter = dataWriterMap.getOrDefault(tableName.getTable(), null);
         if(dataWriter == null) {
             synchronized (this) {
                 if(dataWriter == null) {
                     dataWriter = new DataWriter();
                     // One time initialization for the worker.
                     dataWriter.initialize(tableName);
                     dataWriterMap.put(tableName.getTable(), dataWriter);
                 }
             }
         }
         return dataWriter;
     }

     @PreDestroy
    public void cleanUp() {
         // Final cleanup for the stream during worker teardown.
         log.info("Cleaning up data writer");
         dataWriterMap.forEach((tableName, dataWriter) -> dataWriter.cleanup());
     }

[SubhamSinghal]

I have recently updated com.google.cloud:libraries-bom version to 26.1.3. I am facing one more issue related to connection close exception:

java.lang.RuntimeException: com.google.cloud.bigquery.storage.v1.Exceptions$StreamWriterClosedException: FAILED_PRECONDITION: Connection is closed due to com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException: INTERNAL: Internal error encountered. Entity: projects/project_name/datasets/dataset_name/tables/table_name/_default
at bq.DataWriter.append(DataWriter.java:61)
at bq.DataWriter.append(DataWriter.java:61)
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:67)
at com.google.cloud.bigquery.storage.v1.ConnectionWorker$1.run(ConnectionWorker.java:233)

Caused by: com.google.cloud.bigquery.storage.v1.Exceptions$StreamWriterClosedException: FAILED_PRECONDITION: Connection is closed due to com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException: INTERNAL: Internal error encountered. Entity: projects/project_name/datasets/dataset_name/tables/table_name/_default
at com.google.cloud.bigquery.storage.v1.ConnectionWorker.appendInternal(ConnectionWorker.java:309)
at com.google.cloud.bigquery.storage.v1.JsonStreamWriter.append(JsonStreamWriter.java:113)
at com.google.api.core.ApiFutures.addCallback(ApiFutures.java:62)
at com.google.api.core.SettableApiFuture.setException(SettableApiFuture.java:51)
at com.google.cloud.bigquery.storage.v1.ConnectionWorker.appendLoop(ConnectionWorker.java:515)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:94)
at com.google.cloud.bigquery.storage.v1.ConnectionWorker.append(ConnectionWorker.java:267)
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:67)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:761)
at bq.DataWriter$AppendCompleteCallback.onFailure(DataWriter.java:109)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
at bq.DataWriter$AppendCompleteCallback.onFailure(DataWriter.java:123)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.cloud.bigquery.storage.v1.StreamWriter.append(StreamWriter.java:354)
at com.google.cloud.bigquery.storage.v1.ConnectionWorker.access$000(ConnectionWorker.java:58)

[Neenu1995]

Hi @SubhamSinghal,
Thanks for the information. The issue has been identified and we are working on a fix. There are no action items required from your side to fix this.
Thanks for your patience.

[Neenu1995]

Hi @SubhamSinghal ,
While we are working on the fix, retrying should reduce the frequency of the Failed Precondition errors.