response exhaust report getting failed due to unable to upload file whose file size more than 2Gb into OCI object storage.

[shridhar95]

Hi Team,
As we are working on the response exhaust report, These reports are getting failed for those file requests whose file size is more than 2 Gb.
we did the analysis and we had tried to optimize the code and tried to change the memory configuration of the big data cluster, but still our job are getting failed. We had tried to changing the Driver memory, driver memory overload, executor memory, executor memory overload with different values like 200Gb, 20Gb, 40Gb and 4Gb Respectively.
we have analyzed, if we tried to write a multiple partitioned files in object storage our job is able to write and the CopyMerge method will also able to merge these multiples files and tried to upload in object storage bucket at that moment the the job getting failed.

Post this we have been gone through the logs and observed that the connector unable to write a large size data into the bucket.
Here we are using the org.jets3t.service connector. also we had tried to change into jets3t.properties files configuration upto 10Gb but still not working.

Requesting you to please help us on same.
For your reference attaching the screenshot

[kumarks1122]

@shridhar95 can you please share the full log?

[shridhar95]

Hi @kumarks1122 ,

please find the attachment containing the full logs.

response_failed_job_14Nov.txt

[kumarks1122]

@shridhar95 As per discussion forum for aws sdk, s3 sdk's default maximum streaming size is 5 GB. Can you try adding the below property in OCIS3StorageService and run the job.

Kindly build and deploy all the required libraries(analytics-core, core-data-products, data-products) since the change is happening in the cloud-storage-sdk.

FYI: Discussion link

cc: @AmiableAnil

[shridhar95]

Thanks @kumarks1122 for your kind response.
We had made some changes in configuration level as suggested by OCI team and tried with multipart file as well in single and it has been executed successfully. Now our all the reports are getting succeed without any fail. We had not changed the s3 sdk's max streaming size.