Codecs: custom formats and types

A field’s type usually determines how it is written to bytes. If you need a different representation, assign the field a codec: an object that defines how to encode and decode one value.

Use a built-in codec

Suppose a packet signature occupies exactly 4 bytes and must not have a length prefix:

from bytespec import ProtoModel, field
from bytespec.codecs import FixedBytesCodec

class Packet(ProtoModel):
    signature: bytes = field(codec=FixedBytesCodec(4))

packet = Packet(signature=b"ABCD")
print(Packet.decode(packet.encode()).signature)  # b'ABCD'

FixedBytesCodec(4) already implements this format. It checks the value’s length when encoding and the availability of four bytes when decoding. field(codec=...) accepts a ready-to-use codec instance; you do not need to write a class here. Other built-in options are listed in Codecs.

Compose a codec from existing parts

By default, IntEnum uses VarInt. If the protocol requires one unsigned byte, combine EnumCodec with UInt8Codec:

from enum import IntEnum
from bytespec.codecs import EnumCodec, UInt8Codec

class Status(IntEnum):
    READY = 1
    BUSY = 2

class Message(ProtoModel):
    status: Status = field(codec=EnumCodec(Status, UInt8Codec()))

message = Message(status=Status.BUSY)
print(Message.decode(message.encode()).status.name)  # BUSY

EnumCodec extracts .value, and UInt8Codec writes the number as one byte. Similarly, ListCodec accepts an element codec. You can combine existing objects without reimplementing byte decoding.

Replacing a codec also changes the decode result

An explicit codec completely replaces automatic selection. The field annotation alone does not wrap the result back into the logical type:

from typing import Annotated
from bytespec.types import CodecSpec

class RawStatus(ProtoModel):
    __header__ = ()
    first: Status = field(codec=UInt8Codec())
    second: Annotated[Status, CodecSpec(codec=UInt8Codec())]

raw = RawStatus(first=Status.BUSY, second=Status.BUSY)
decoded = RawStatus.decode(raw.encode())
assert type(decoded.first) is int
assert type(decoded.second) is int
print(raw.encode().hex(" "))

02 02

Both fields contain int after decoding, despite the Status annotation. To preserve the enum, use EnumCodec from the previous example. The same rule applies to CodecSpec and any other logical type: the codec itself reconstructs the object.

Support a custom type

Suppose the application stores an identifier in a separate class:

from dataclasses import dataclass

@dataclass(frozen=True)
class UserId:
    value: int

To encode UserId, pass its number to the built-in UInt32Codec; when decoding, wrap the number again:

from bytespec import ByteOrder
from bytespec.codecs import ICodec, UInt32Codec

class UserIdCodec(ICodec[UserId]):
    def encode(self, value: UserId, byte_order: ByteOrder) -> bytes:
        return UInt32Codec().encode(value.value, byte_order)

    def decode(self, buffer: bytes, byte_order: ByteOrder,
               offset: int) -> tuple[UserId, int]:
        value, end = UInt32Codec().decode(buffer, byte_order, offset)
        return UserId(value), end

encode() returns the bytes for one value. decode() receives the starting position and returns the new absolute position in the same buffer. byte_order is the model’s byte order; here we pass it to the numeric codec.

Now attach the implementation to a field:

class User(ProtoModel):
    id: UserId = field(codec=UserIdCodec())

user = User(id=UserId(42))
decoded = User.decode(user.encode())
print(decoded.id.value)  # 42

ICodec[T] describes the codec interface. Inheriting from it is convenient for typing, but any object with encode and decode methods also works. Checking that these methods exist does not verify the implementation’s correctness.

In this example, UInt32Codec checks sizes and numeric ranges. If you read bytes directly, check that the data is available and report malformed data with DecodeError and unrepresentable values with EncodeError. Arbitrary exceptions from custom codecs are not wrapped. For offset requirements, including those for list elements, see Implementation requirements.

Assign a codec to every field of your type

If you use UserId often, you can avoid repeating field(codec=...). Add a rule through configure_codecs():

class User(ProtoModel):
    @classmethod
    def configure_codecs(cls):
        return {UserId: lambda annotation, field_info: UserIdCodec()}

    id: UserId

user = User(id=UserId(7))
print(User.decode(user.encode()).id.value)  # 7

The key is the field type. The value is a factory that returns a codec for it. The factory receives the annotation and field settings and is called when the model class is declared. This example has no additional settings, so the factory simply creates UserIdCodec.

The returned rules extend the built-in rules. An existing key replaces the rule for that type; subclasses inherit the rules. See Extension types for factory annotations and the exact precedence rules. You do not need to work with the internal registry directly.

Reuse settings through Annotated

Like a string prefix, an explicit codec can be placed in CodecSpec:

from typing import Annotated
from bytespec.types import CodecSpec

Signature = Annotated[bytes, CodecSpec(codec=FixedBytesCodec(4))]

class Message(ProtoModel):
    signature: Signature

message = Message(signature=b"ABCD")
print(Message.decode(message.encode()).signature)  # b'ABCD'

You can use this annotation in several models and in list[Signature]. An explicit field(codec=...) takes precedence over the annotation. A ready-made codec is configured through its constructor: field(prefix_length=1, codec=StrCodec()) does not reconfigure the supplied StrCodec().

Call a codec without a model

from bytespec.codecs import ListCodec, StrCodec

codec = ListCodec(StrCodec(prefix_length=1), prefix_length=2)
encoded = codec.encode(["red", "blue"], ByteOrder.BIG)
values, end = codec.decode(encoded, ByteOrder.BIG, 0)

print(values)         # ['red', 'blue']
print(encoded[end:])  # b''

This call returns only the value’s representation, without a model header. The caller is responsible for any remaining bytes. ModelCodec is an exception: it uses the nested model’s framing, skipping its Constructor. Flags and PayloadLength remain if declared. For a comparison of standalone and nested encoding, see How nested model headers are written.

Next: Model inheritance for reusing settings, or Validating model values for validating values. For the exact byte layout, see Binary / wire format.