This sub-package provides the capability to compress and decompress data using the stream specification, especially the stream specification based on a double buffer.
Because the LZ4 stream format does not define a container format, the Python bindings will by default insert the compressed data size as an integer at the start of the compressed payload. However, it is possible to set the bit depth of this compressed data size.
So far, only the double-buffer based approach is implemented.
To use the lz4 stream format bindings is straightforward:
>>> from lz4.stream import LZ4StreamCompressor, LZ4StreamDecompressor
>>> import os
>>> block_size_length = 2 # LZ4 compressed block size stored on 2 bytes
>>> page_size = 8192 # LZ4 context double buffer page size
>>> origin_stream = 10 * 1024 * os.urandom(1024) # 10MiB
>>> # LZ4 stream compression of origin_stream into compressed_stream:
>>> compressed_stream = bytearray()
>>> with LZ4StreamCompressor("double_buffer", page_size, store_comp_size=block_size_length) as proc:
... offset = 0
... while offset < len(origin_stream):
... chunk = origin_stream[offset:offset + page_size]
... block = proc.compress(chunk)
... compressed_stream.extend(block)
... offset += page_size
>>> # LZ4 stream decompression of compressed_stream into decompressed_stream:
>>> decompressed_stream = bytearray()
>>> with LZ4StreamDecompressor("double_buffer", page_size, store_comp_size=block_size_length) as proc:
... offset = 0
... while offset < len(compressed_stream):
... block = proc.get_block(compressed_stream[offset:])
... chunk = proc.decompress(block)
... decompressed_stream.extend(chunk)
... offset += block_size_length + len(block)
>>> decompressed_stream == origin_stream
True
A Python wrapper for the LZ4 stream protocol.
lz4.stream.
LZ4StreamCompressor
(strategy, buffer_size, mode='default', acceleration=True, compression_level=9, return_bytearray=False, store_comp_size=4, dictionary='')¶LZ4 stream compressing context.
__enter__
()¶Enter the LZ4 stream context.
__exit__
(exc_type, exc, exc_tb)¶Exit the LZ4 stream context.
compress
(chunk)¶Stream compress given chunk
of data.
Compress the given chunk
, using the given LZ4 stream context,
returning the compressed data as a bytearray
or as a bytes
object.
chunk (str, bytes or buffer-compatible object) – Data to compress
Compressed data.
bytes or bytearray
Exceptions occuring during compression. –
OverflowError – raised if the source is too large for being compressed in the given context.
LZ4StreamError – raised if the call to the LZ4 library fails.
lz4.stream.
LZ4StreamDecompressor
(strategy, buffer_size, return_bytearray=False, store_comp_size=4, dictionary='')¶LZ4 stream decompression context.
__enter__
()¶Enter the LZ4 stream context.
__exit__
(exc_type, exc, exc_tb)¶Exit the LZ4 stream context.
decompress
(chunk)¶Decompress streamed compressed data.
Decompress the given chunk
, using the given LZ4 stream context,
Raises an exception if any error occurs.
chunk (str, bytes or buffer-compatible object) – Data to decompress
Decompressed data.
bytes or bytearray
Exceptions occuring during decompression. –
ValueError – raised if the source is inconsistent with a finite LZ4 stream block chain.
MemoryError – raised if the work output buffer cannot be allocated.
OverflowError – raised if the source is too large for being decompressed in the given context.
LZ4StreamError – raised if the call to the LZ4 library fails. This can be
caused by decompressed_size
being too small, or invalid data.
get_block
(stream)¶Return the first LZ4 compressed block from stream
.
stream (str, bytes or buffer-compatible object) – LZ4 compressed stream.
LZ4 compressed data block.
bytes or bytearray
Exceptions occuring while getting the first block from stream. –
BufferError – raised if the function cannot return a complete LZ4 compressed block from the stream (i.e. the stream does not hold a complete block).
MemoryError – raised if the output buffer cannot be allocated.
OverflowError – raised if the source is too large for being handled by the given context.