Asyncio And Pyzmq - 'utf-8' Codec Can't Decode Byte 0xff In Position 0

I have a asyncio server, which is an example from the TCP Doc. However I'm connecting to it using pyzmq and when the reader on the server tries to read I get a decode error. Any h

Solution 1:

Zeromq uses the ZMTP protocol. It is a binary protocol so you won't be able to decode it directly.

If you're curious about it, check the ZMTP frames using wireshark and the ZMTP plugin:

You can see that the bytes you got actually corresponds to the greeting message signature.

---

In order to receive the messages from a ZMQ socket in asyncio, use a dedicated project like aiozmq:

import aiozmq
import asyncio

asyncdefmain(port=5555):
    bind = "tcp://*:%s" % port
    rep = await aiozmq.create_zmq_stream(aiozmq.zmq.REP, bind=bind)
    message, = await rep.read()
    print(message.decode())
    rep.write([message])

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())
    loop.close()

Solution 2:

The byte ff is the first byte of a little-endian UTF-16 BOM, it has no place in a UTF-8 stream, where the maximum number of 1-bits at the start of a codepoint is four.

See an earlier answer of mine for more detail on the UTF-8 encoding.

As to fixing it, you'll need to receive what was sent. That will involve either fixing the transmission side to do UTF-8, or the reception side to do UTF-16.

You may want to look into the differences between strings in Python 2 and 3, this may well be what's causing your issue (see here).