ASCL.net

Astrophysics Source Code Library

Making codes discoverable since 1999

Bitshuffle: Filter for improving compression of typed binary data

Discussion topics for individual codes
Ada Coda
ASCL Robot
Posts: 1077
Joined: Thu May 08, 2014 5:37 am

Bitshuffle: Filter for improving compression of typed binary data

Postby Ada Coda » Mon Jan 01, 2018 3:06 am

Bitshuffle: Filter for improving compression of typed binary data

Abstract: Bitshuffle rearranges typed, binary data for improving compression; the algorithm is implemented in a python/C package within the Numpy framework. The library can be used alongside HDF5 to compress and decompress datasets and is integrated through the dynamically loaded filters framework. Algorithmically, Bitshuffle is closely related to HDF5's Shuffle filter except it operates at the bit level instead of the byte level. Arranging a typed data array in to a matrix with the elements as the rows and the bits within the elements as the columns, Bitshuffle "transposes" the matrix, such that all the least-significant-bits are in a row, etc. This transposition is performed within blocks of data roughly 8kB long; this does not in itself compress data, but rearranges it for more efficient compression. A compression library is necessary to perform the actual compression. This scheme has been used for compression of radio data in high performance computing.

Credit: Masui, Kiyoshi

Site: https://github.com/kiyo-masui/bitshuffle
http://adsabs.harvard.edu/abs/2015A%26C....12..181M

Bibcode: 2017ascl.soft12004M

Preferred citation method: http://adsabs.harvard.edu/abs/2015A%26C....12..181M

ID: ascl:1712.004

Return to “Astrophysics Source Code Library”

Who is online

Users browsing this forum: No registered users and 8 guests

cron