linkBytes Utility Library for Java

Bytes is a utility library that makes it easy to create, parse, transform, validate and convert byte arrays in Java. It's main class Bytes is a collections of bytes and the main API. It supports endianness as well as copy-on-write and mutable access, so the caller may decide to favor performance. This can be seen as combination of the features provided by BigInteger, ByteBuffer but providing a lot of additional features on the micro and macro level of byte arrays (similar to Okio's ByteString). The main goal is to minimize the need to blindly paste code snippets from s t a c k o v e r f l o w . c o m

It's main features include:

• Creation from a wide variety of sources: multiple arrays, integers, streams, random, strings, files, uuid, ...
• Transformation with many built-in: append, xor, and, hash, shifts, shuffle, reverse, checksum, ...
• Validators with the ability to arbitrarily combine multiple ones with logical expressions
• Parsing and Encoding in most common binary-to-text-encodings: hex, base32, base64, ...
• Immutable, Mutable and Read-Only versions
• Handling Strings with encoding and normalizing strings for arbitrary charset
• Utility Features like indexOf, count, isEmpty, bitAt, contains ...
• Flexibility provide your own Transformers, Validators and Encoders

The code is compiled with target Java 7 to keep backwards compatibility with Android and older Java applications. It is lightweight as it does not require any additional dependencies.

linkQuickstart

Add dependency to your pom.xml (check latest release):

1<dependency>

2    <groupId>at.favre.lib</groupId>

3    <artifactId>bytes</artifactId>

4    <version>{latest-version}</version>

5</dependency>

Note: There is a byte-code optimized version (powered by ProGuard) which can be used with classifier 'optimized'. This may have issues so use at your own risk.

Some simple examples:

1Bytes b = Bytes.wrap(someByteArray);  //reuse given reference

2b.copy().reverse(); //reverse the bytes on a copied instance

3String hex = b.encodeHex(); //encode base16/hex

1Bytes b = Bytes.parseHex("0ae422f3");  //parse from hex string

2int result = b.toInt(); //get as signed int

1Bytes b = Bytes.from(array1);  //create from copy of array1

2b.resize(2).xor(array2); //shrink to 2 bytes and xor with other array

3byte[] result = b.array(); //get as byte array

linkAPI Description

Per default the instance is semi-immutable, which means any transformation will create a copy of the internal array (it is, however, possible to get and modify the internal array). There is a mutable version which supports in-place modification for better performance and a read-only version which restricts the access to the internal array.

linkConstructors

There are 3 basic constructors:

• wrap() which reuses the given array reference; this is equivalent to ByteBuffer.wrap()
• from() which always creates a new internal array reference (i.e. a copy of the passed reference)
• parse() which parses from binary-text-encoded strings (see other section)

Here is a simple example to show the difference:

1byte[] myArray = ...

2Bytes bWrap = Bytes.wrap(myArray);

3assertSame(myArray, bWrap.array());

4

5byte[] myArray2 = ...

6Bytes bFrom = Bytes.from(myArray2);

7assertNotSame(myArray2, bFrom.array());

8assertArrayEquals(myArray2, bFrom.array());

The following code is equivalent:

1Bytes.wrap(myArray).copy() ~ Bytes.from(myArray)

linkMore Constructors

For a null-safe version, which uses the empty array in case of a null byte array:

1Bytes.wrapNullSafe(null);

2Bytes.fromNullSafe(null);

Concatenating of multiple byte arrays or bytes:

1Bytes.from(array1, array2, array3);

2Bytes.from((byte) 0x01, (byte) 0x02, (byte) 0x03);

Creating byte arrays from primitive integer types and arrays:

1Bytes.from(8);  //00000000 00000000 00000000 00001000

2Bytes.from(1897621543227L);

3Bytes.from(1634, 88903, 77263);

4Bytes.from(0.7336f, -87263.0f);

5Bytes.from(0.8160183296, 3984639846.0);

Initializing empty arrays of arbitrary length:

1Bytes.allocate(16);

2Bytes.allocate(4, (byte) 1); //fill with 0x01

3Bytes.empty(); //creates zero length byte array

Creating cryptographically secure random byte arrays:

1Bytes.random(12);

Creating cryptographically unsecure random byte arrays for e.g. testing:

1Bytes.unsecureRandom(12, 12345L); // using seed makes it deterministic

Reading byte content of encoded Strings:

1Bytes.from(utf8String)

2Bytes.from(utf8StringToNormalize, Normalizer.Form.NFKD) //normalizes unicode

3Bytes.from(asciiString, StandardCharset.US_ASCII) //any charset

And other types:

1Bytes.from(byteInputStream); //read whole java.io.InputStream

2Bytes.from(byteInputStream, 16); //read java.io.InputStream with length limitation

3Bytes.from(byteList); //List<Byte> byteList = ...

4Bytes.from(myBitSet); //java.util.BitSet myBitSet = ...

5Bytes.from(bigInteger); //java.math.BigInteger

6Bytes.from(file); //reads bytes from any java.io.File

7Bytes.from(dataInput, 16); //reads bytes from any java.io.DataInput

8Bytes.from(UUID.randomUUID()); //read 16 bytes from UUID

For parsing binary-text-encoded strings, see below.

linkTransformers

Transformers transform the internal byte array. It is possible to create custom transformers if a specific feature is not provided by the default implementation (see BytesTransformer). Depending on the type (mutable vs immutable) and transformer it will overwrite the internal byte array or always create a copy first.

1Bytes result = Bytes.wrap(array1).transform(myCustomTransformer);

linkBuilt-In Transformers

For appending byte arrays or primitive integer types to current instances. Note: this will create a new copy of the internal byte array; for dynamically growing byte arrays see ByteArrayOutputStream.

1Bytes result = Bytes.wrap(array1).append(array2);

2Bytes result = Bytes.wrap(array1).append(1341);

3Bytes result = Bytes.wrap(array1).append((byte) 3);

4Bytes result = Bytes.wrap(array1).append("some string");

Bitwise operations: XOR, OR, AND, NOT as well as left and right shifts and switching bits:

1Bytes.wrap(array).xor(array2); // 0010 0011 xor() 1011 1000 = 1001 1011

2Bytes.wrap(array).or(array2); // 0010 0011 or() 1101 0100 = 1111 0111

3Bytes.wrap(array).and(array2); // 0010 0011 and() 1011 1000 = 0010 0000

4Bytes.wrap(array).not(); // 0010 0011 negate() = 1101 1100

5Bytes.wrap(array).leftShift(8);

6Bytes.wrap(array).rightShift(8);

7Bytes.wrap(array).switchBit(3, true);

Copy operations, which copies the internal byte array to a new instance:

1Bytes copy = Bytes.wrap(array).copy();

2Bytes copy = Bytes.wrap(array).copy(3, 17); //copy partial array

Resizing the internal byte array:

1Bytes resized = Bytes.wrap(array).resize(3); //from {3, 9, 2, 1} to {9, 2, 1}

Hashing the internal byte array using the MessageDigest Java crypto API:

1Bytes hash = Bytes.wrap(array).hashSha256();

2Bytes hash = Bytes.wrap(array).hashSha1();

3Bytes hash = Bytes.wrap(array).hashMd5();

4Bytes hash = Bytes.wrap(array).hash("SHA-512");

Reversing of the byte order in the array

1Bytes result = Bytes.wrap(array).reverse();

linkAdditional Transformers

More transformers can be accessed through the BytesTransformers, which can be statically imported for a less verbose syntax:

1import static at.favre.lib.bytes.BytesTransformers.*;

HMAC used to calculate keyed-hash message authentication code:

1Bytes.wrap(array).transform(hmacSha256(macKey32Byte));

2Bytes.wrap(array).transform(hmacSha1(macKey20Byte));

3Bytes.wrap(array).transform(hmac(macKey16Byte,"HmacMd5"));

Checksum can be calculated or automatically appended:

1Bytes.wrap(array).transform(checksumAppendCrc32());

2Bytes.wrap(array).transform(checksumCrc32());

3Bytes.wrap(array).transform(checksum(new Adler32(), ChecksumTransformer.Mode.TRANSFORM, 4));

GZip compression is supported by GZIPInputStream:

1Bytes compressed = Bytes.wrap(array).transform(compressGzip());

2Bytes decompressed = compressed.transform(decompressGzip());

Sorting of individual bytes with either Comparator or natural order:

1Bytes.wrap(array).transform(sort()); // 0x00 sorts after 0xff

2Bytes.wrap(array).transform(sortUnsigned()); // 0xff sorts after 0x00

3Bytes.wrap(array).transform(sort(byteComparator));

Shuffling of individual bytes:

1Bytes.wrap(array).transform(shuffle());

linkParser and Encoder for Binary-Text-Encodings

This library can parse and encode a variety of encodings: binary, decimal, octal, hex and base64. Additionally custom parsers are supported by providing your own implementation:

1Bytes.parse("8sK;S*j=r", base85Decoder);

2Bytes.encode(base85Encoder);

Hex can be upper and lowercase and also supports 0x prefix for parsing:

1Bytes.parseHex("a0e13eaa1a")

2Bytes.parseHex("0xA0E1")

3

4Bytes.from(array).encodeHex() //a0e13eaa1a

This lib has it's own build in Base64 encoder:

1Bytes.parseBase64("SpT9/x6v7Q==");

2

3Bytes.from(array).encodeBase64(); //"SpT9/x6v7Q=="

4Bytes.from(array).encodeBase64Url(); //"SpT9_x6v7Q=="

also a Base32 encoder (using the RFC4648 non-hex alphabet):

1Bytes.parseBase32("MZXQ====");

2Bytes.from(array).encodeBase32();

Additionally the following radix encodings are supported:

1Bytes.from(array).encodeBinary(); //1110110110101111

2Bytes.from(array).encodeDec(); //20992966904426477

3Bytes.from(array).encodeOctal(); //1124517677707527755

4Bytes.from(array).encodeRadix(36); //5qpdvuwjvu5

linkHandling Strings

You can easily get the UTF-8 encoded version of a string with

1String s = "...";

2Bytes.from(s);

or get the normalized version, which is the recommended way to convert e.g. user names

1String pwd = "ℌH";

2Bytes.from(pwd, Normalizer.Form.NFKD); //would be "HH" normalized

or get as any other character encodings

1String asciiString = "ascii";

2Bytes.from(asciiString, StandardCharsets.US_ASCII);

To easily append a string to an byte array you can do

1String userPwdHash = ...;

2Bytes.from(salt).append(userPwd).hashSha256();

linkUtility Methods

Methods that return additional information about the instance.

Finding occurrence of specific bytes:

1Bytes.wrap(array).contains((byte) 0xE1);

2Bytes.wrap(array).indexOf((byte) 0xFD);

3Bytes.wrap(array).indexOf(new byte[] {(byte) 0xFD, 0x23});

4Bytes.wrap(array).indexOf((byte) 0xFD, 5); //search fromIndex 5

5Bytes.wrap(array).lastIndexOf((byte) 0xAE);

6Bytes.wrap(array).startsWith(new byte[] {(byte) 0xAE, 0x32});

7Bytes.wrap(array).endsWidth(new byte[] {(byte) 0xAE, 0x23});

Length checks:

1Bytes.wrap(array).length();

2Bytes.wrap(array).lengthBit(); //8 * array.length

3Bytes.wrap(array).isEmpty();

Accessing part of the array as primitives from arbitrary position:

1Bytes.wrap(array).bitAt(4); // 0010 1000 -> false

2Bytes.wrap(array).byteAt(14); // 1111 1111 -> -1

3Bytes.wrap(array).unsignedByteAt(14); // 1111 1111 -> 255

4Bytes.wrap(array).intAt(4);

5Bytes.wrap(array).longAt(6);

And others:

1Bytes.wrap(array).count(0x01); //occurrences of 0x01

2Bytes.wrap(array).count(new byte[] {0x01, 0xEF}); //occurrences of pattern [0x01, 0xEF]

3Bytes.wrap(array).entropy();

Of course all standard Java Object methods are implemented including: hashCode(), equals(), toString() as well as it being Comparable. In addition there is a constant time equalsConstantTime() method, see here why this might be useful.

The toString() methods only shows the length and a preview of maximal 8 bytes:

116 bytes (0x7ed1fdaa...12af000a)

Bytes also implements the Iterable interface, so it can be used in a foreach loop:

1for (Byte aByte : bytesInstance) {

2    ...

3}

The equals method has overloaded versions for byte[], Byte[] and ByteBuffer which can be used to directly compare the inner array:

1byte[] primitiveArray1 = ...

2byte[] primitiveArray2 = ...

3Bytes.wrap(primitiveArray1).equals(primitiveArray2); //compares primitiveArray1 with primitiveArray2

linkValidation

A simple validation framework which can be used to check the internal byte array:

1import static at.favre.lib.bytes.BytesValidators.*;

2

3Bytes.wrap(new byte[]{8, 3, 9}).validate(startsWith((byte) 8), atLeast(3)); // true

This is especially convenient when combining validators:

1Bytes.wrap(new byte[]{0, 1}).validate(atMost(2), notOnlyOf((byte)  0)); // true

Validators also support nestable logical expressions AND, OR as well as NOT:

1Bytes.allocate(0).validate(or(exactLength(1), exactLength(0))) //true

2Bytes.allocate(19).validate(and(atLeast(3), atMost(20))) //true

3Bytes.allocate(2).validate(not(onlyOf((byte) 0))); //false

Nesting is also possible:

1assertTrue(Bytes.allocate(16).validate(

2                or(

3                   and(atLeast(8),not(onlyOf(((byte) 0)))),

4                   or(exactLength(16), exactLength(12))))); // true

linkConverting

The internal byte array can be converted or exported into many different formats. There are 2 different kinds of converters:

• Ones that create a new type which reuses the same shared memory
• Ones that create a copy of the internal array, which start with to*

linkShared Memory Conversion

Not technically a conversation, but it is of course possible to access the internal array:

1Bytes.wrap(array).array();

Conversion to InputStream and ByteBuffer:

1Bytes.wrap(array).inputStream();

2Bytes.wrap(array).byteBuffer();

If you just want a duplicated instance, sharing the same array:

1Bytes.wrap(array).duplicate();

For the conversion to read-only and mutability, see below.

linkCopy Conversion

To primitives (if the internal array is not too long)

1Bytes.wrap(array).toByte();

2Bytes.wrap(array).toUnsignedByte();

3Bytes.wrap(array).toInt();

4Bytes.wrap(array).toDouble();

To primitive arrays

1Bytes.wrap(array).toIntArray(); // of type int[]

2Bytes.wrap(array).toLongArray(); // of type long[]

To other collections

1Bytes.wrap(array).toList(); // of type List<Byte>

2Bytes.wrap(array).toBoxedArray(); // of type Byte[]

3Bytes.wrap(array).toBitSet(); //of type java.util.BitSet

to BigInteger of course

1Bytes.wrap(array).toBigInteger();

and others

1Bytes.wrap(array).toUUID(); // convert 16 byte to UUID

2Bytes.wrap(array).toCharArray(StandardCharsets.UTF-8); // converts to encoded char array

linkMutable and Read-Only

Per default the instance is immutable, i.e. every transformation will create a a new internal byte array (very similar to the API of BigInteger). While this is usually the default way to design such a construct because it shows various advantages this can introduce a major performance issue when handling big arrays or many transformations.

linkMutable Bytes

All transformers (if possible) reuse or overwrite the same internal memory to avoid unneeded array creation to minimize time and space complexity. To create a mutable instance just do:

1MutableBytes b = Bytes.from(array).mutable();

Mutable classes also enable further APIs for directly modify the internal array:

1b.setByteAt(3, (byte) 0xF1)

2b.overwrite(anotherArray) //directly overwrite given array

3b.fill(0x03) // fills with e.g. 3

4b.wipe() //fills with zeros

5b.secureWipe() //fills with random data

Create a immutable version again with:

1Bytes b2 = b.immutable();

Note: a copy will inherit mutability/read-only properties:

1Bytes b = Bytes.from(array).mutable().copy();

2assertTrue(b.isMutable());

linkAutoClosable for try-with-resources

In security-relevant environments it is best practice to wipe the memory of secret data, such as secret keys. This can be used with Java 7 feature try-with-resource like this:

1try (MutableBytes b = Bytes.wrap(aesBytes).mutable()) {

2    SecretKey s = new SecretKeySpec(b.array(), "AES");

3    ...

4}

linkReadonly Bytes

On the other hand, if you want a export a instance with limited access, especially no easy way to alter the internal byte array, read-only instances may be created by:

1Bytes b = Bytes.from(array).readOnly();

Every call to the following conversation methods will throw a ReadOnlyBufferException:

1readOnlyBytes.array();

2readOnlyBytes.byteBuffer();

3readOnlyBytes.inputStream();

linkDownload

The artifacts are deployed to jcenter and Maven Central.

linkMaven

Add the dependency of the latest version to your pom.xml:

1<dependency>

2    <groupId>at.favre.lib</groupId>

3    <artifactId>bytes</artifactId>

4    <version>{latest-version}</version>

5</dependency>

linkGradle

Add to your build.gradle module dependencies:

1implementation group: 'at.favre.lib', name: 'bytes', version: '{latest-version}'

linkLocal Jar Library

Grab jar from latest release.

linkOSGi

The library should be prepared to be used with the OSGi framework with the help of the bundle plugin.

linkDigital Signatures

linkSigned Jar

The provided JARs in the Github release page are signed with my private key:

1CN=Patrick Favre-Bulle, OU=Private, O=PF Github Open Source, L=Vienna, ST=Vienna, C=AT

2Validity: Thu Sep 07 16:40:57 SGT 2017 to: Fri Feb 10 16:40:57 SGT 2034

3SHA1: 06:DE:F2:C5:F7:BC:0C:11:ED:35:E2:0F:B1:9F:78:99:0F:BE:43:C4

4SHA256: 2B:65:33:B0:1C:0D:2A:69:4E:2D:53:8F:29:D5:6C:D6:87:AF:06:42:1F:1A:EE:B3:3C:E0:6D:0B:65:A1:AA:88

Use the jarsigner tool (found in your $JAVA_HOME/bin folder) folder to verify.

linkSigned Commits

All tags and commits by me are signed with git with my private key:

1GPG key ID: 4FDF85343912A3AB

2Fingerprint: 2FB392FB05158589B767960C4FDF85343912A3AB

linkBuild

linkJar Sign

If you want to jar sign you need to provide a file keystore.jks in the root folder with the correct credentials set in environment variables ( OPENSOURCE_PROJECTS_KS_PW and OPENSOURCE_PROJECTS_KEY_PW); alias is set as pfopensource.

If you want to skip jar signing just change the skip configuration in the pom.xml jar sign plugin to true:

1true

linkBuild with Maven

Use the Maven wrapper to create a jar including all dependencies

1mvnw clean install

linkCheckstyle Config File

This project uses my common-parent which centralized a lot of the plugin versions aswell as providing the checkstyle config rules. Specifically they are maintained in checkstyle-config. Locally the files will be copied after you mvnw install into your target folder and is called target/checkstyle-checker.xml. So if you use a plugin for your IDE, use this file as your local configuration.

linkTech Stack

• Java 7 (+ errorprone static analyzer)
• Maven

linkCredits

• Byte util methods derived from primitives.Bytes from Google Guava (Apache v2)
• Entropy class derived from Twitter Commons (Apache v2)
• Base64 implementation and some util methods from Okio (Apache v2)

linkLicense

Copyright 2017 Patrick Favre-Bulle

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

1http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.