Skip to content

Commit

Permalink
Docs: Update smart_open example code to prevent decompression error (f…
Browse files Browse the repository at this point in the history
…ixes #59). (#60)
  • Loading branch information
junhaoliao authored May 27, 2024
1 parent 0e5b287 commit 156914b
Showing 1 changed file with 8 additions and 2 deletions.
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,15 +203,21 @@ session = boto3.Session(

url = 's3://clp-example-s3-bucket/example.clp.zst'
# Using `smart_open.open` to stream the encoded CLP IR:
with smart_open.open(url, "rb", transport_params={'client': session.client('s3')}) as istream:
with smart_open.open(
url, mode="rb", compression="disable", transport_params={"client": session.client("s3")}
) as istream:
with ClpIrStreamReader(istream, allow_incomplete_stream=True) as clp_reader:
for log_event in clp_reader:
# Print the log message with its timestamp properly formatted.
print(log_event.get_formatted_message())
```

Note:
When `allow_incomplete_stream` is set to False (default), the reader will raise
- Setting `compression="disable"` is necessary so that `smart_open` doesn't
undo the IR file's Zstandard compression (based on the file's extension) before
streaming it to `ClpIrStreamReader`; `ClpIrStreamReader` expects the input
stream to be Zstandard-compressed.
- When `allow_incomplete_stream` is set to False (default), the reader will raise
`clp_ffi_py.ir.IncompleteStreamError` if the stream is incomplete (it doesn't end
with the byte sequence indicating the stream's end). In practice, this can occur
if you're reading a stream that is still being written or wasn't properly
Expand Down

0 comments on commit 156914b

Please sign in to comment.