Structures, Serialisation, XOR Checksums etc.

@pugwash

LOL 🙂

@frogandtoad

I have been considering piping the data from the struct to Python and found a little-used conversion method to handle incoming strings of bytes.

In my naivety, I thought that the following would work, but doesn't even compile!

Serial.write(&myObject, sizeof(myObject));

What am I missing here?

A type cast of &myObject to an array of chars or bytes maybe ?

@zeferby

Do you mean something like this?

char* bytes_to_serial = static_cast<char*>(static_cast<void*>(&myObject));
Serial.write(my_s_bytes, sizeof(bytes_to_serial));

It compiles, at least ? 

@pugwash

Let me get my Arduino IDE to try something simpler.

Ok, looking at C:\Users\username>, in AppData\Local\Arduino15\packages\arduino\hardware\avr\1.8.2\cores\arduino/Print.h i find these two :

 virtual size_t write(const uint8_t *buffer, size_t size);
size_t write(const char *buffer, size_t size) {
return write((const uint8_t *)buffer, size);
}

So this should work I guess ?

Serial.write((const uint8_t *)(&myObject), sizeof(myObject));

or :

Serial.write((const char *)(&myObject), sizeof(myObject));

I did compile this sample :

void loop() {
// put your main code here, to run repeatedly:
void * ptr;
Serial.write((const uint8_t *)ptr, 12345);
Serial.write((const char *)ptr, 12345);
}

@pugwash

Posted by: @pugwash

Serial.write(&myObject, sizeof(myObject));

What am I missing here?

According to the following reference:

serial.write()

... it has 3 overloads, and the one you're using is expecting an array.

Try using your buffer created from "malloc" and see how that goes.

@frogandtoad, @zeferby

On the data receiving end, I am looking at using this python code and method(struct.unpack):

import serial

my_comm_port = '/dev/cu.usbserial-00000000'; # *nix comm port notation

try:

   # Change the baud rate here if different than 9600
   ser = serial.Serial(port=my_comm_port, baudrate=115200)

except IOError:

   print("Invalid comm port!")
   exit(1)

def main():

   while 1:

      data = ser.readline()

      # struct.unpack(format, bytestring)
      tup = struct.unpack(Lcffffc, data)

      print(tup)

????? I am not entirely sure whether "q" is the short form for an unsigned long integer!

Edited: "qbffffb" changed to "Lcffffc" This seems to be the correct format!

@pugwash

Sorry I never used Python. According to the doc, "q" is for signed, 8-byte/64-bit integers.  I don't know what structure you are passing between Arduino/C++ and Python though.

https://docs.python.org/2/library/struct.html

https://docs.python.org/3/library/struct.html

@pugwash

Posted by: @pugwash

Edited: "qbffffb" changed to "Lcffffc" This seems to be the correct format!

Interesting, so did your update work?

Similarly, it sounds like you may want to work with JSON objects, which have been mostly adopted for portable communication between different systems over a network, and quite versatile:

python-json

@frogandtoad

Interesting, so did your update work?

No, not yet, too much pre-Xmas stress. I have to modify the receiver sketch first and then write the data catcher in Python.

JSON did cross my mind but uncrossed it just as fast! If I had gone down the javascript route and used Electron as a front end, it might have been interesting but I guess I will just stick with Python and Tkinter for the time being.

@frogandtoad

Progress so far is quite interesting. The story so far!!

The data throwing sketch is this:

typedef struct{
  uint8_t abyte = 12;
  float afloat = 15.4;
  uint8_t bbyte = 5;
  }data;

data myData; 


void setup() {
  Serial.begin(115200);

}

void loop() { 

  delay(5000);
  Serial.write((const byte*)&myData, sizeof(myData));

}

And the receiving program is this (without indents):

import serial
import struct

my_comm_port = '/dev/cu.wchusbserial1460'; # *nix comm port notation

try:

# Change the baud rate here if different than 115200
ser = serial.Serial(
port='/dev/cu.wchusbserial1460',\
baudrate=115200,\
parity=serial.PARITY_NONE,\
stopbits=serial.STOPBITS_ONE,\
bytesize=serial.EIGHTBITS,\
timeout=None
)

print("connected to: " + ser.portstr)

except IOError:

print("Invalid comm port!")
exit(1)


while True:

data = ser.read(6)

#struct.unpack(format, bytestring)
tup = struct.unpack('cfc', data)

print(tup)

ser.close()

'cfc' is theoretically 6 bytes long but I am getting "struct.error: unpack requires a bytes object of length 9"

when I change to ser.read(9), I am getting the following screen output

seaswallow:serial_read stevec$ Python3 struct_catcher.py

connected to: /dev/cu.wchusbserial1460

(b'\x0c', 1.6530705626643284e+23, b'f')

(b'v', 15.399999618530273, b'\x05')

(b'\x0c', 1.6530705626643284e+23, b'f')

(b'v', 15.399999618530273, b'\x05')

(b'\x0c', 1.6530705626643284e+23, b'f')

(b'v', 15.399999618530273, b'\x05')

As you can see on every alternate line the float is correct and the two bytes are correct, so I guess there has to be some fine-tuning to sync the byte strings.

@frogandtoad

I ran this and it also came back 9

size = struct.calcsize('cfc') 
print(size)

Ergo, I have size problem (don't even think it ? )

The Arduino is sending out 6 bytes and Python wants 9 bytes!

@pugwash

Hum, I think there are "alignment" considerations here...

>>> import struct
>>> size = struct.calcsize('c')
>>> print(size)
1
>>> size = struct.calcsize('f')
>>> print(size)
4
>>> size = struct.calcsize('cc')
>>> print(size)
2
>>> size = struct.calcsize('cf')
>>> print(size)
8
>>> size = struct.calcsize('fc')
>>> print(size)
5
>>> size = struct.calcsize('ccf')
>>> print(size)
8
>>> size = struct.calcsize('fcc')
>>> print(size)
6
>>> size = struct.calcsize('cfc')
>>> print(size)
9
>>> exit()

From Python 3 "struct" doc :

Note

 

By default, the result of packing a given C struct includes pad bytes in order to maintain proper alignment for the C types involved; similarly, alignment is taken into account when unpacking. This behavior is chosen so that the bytes of a packed struct correspond exactly to the layout in memory of the corresponding C struct. To handle platform-independent data formats or omit implicit pad bytes, use standard size and alignment instead of native size and alignment: see Byte Order, Size, and Alignment for details.

So, using the "=" modifier to specify No Alignment :

>>> import struct
>>> size = struct.calcsize('=cf')
>>> print(size)
5
>>> size = struct.calcsize('=fc')
>>> print(size)
5
>>> size = struct.calcsize('=ccf')
>>> print(size)
6
>>> size = struct.calcsize('=fcc')
>>> print(size)
6
>>> size = struct.calcsize('=cfc')
>>> print(size)
6
>>> exit()

@zeferby

Excellent, that worked!

The output is aligned now.

(b'\x0c', 15.399999618530273, b'\x05')
(b'\x0c', 15.399999618530273, b'\x05')
(b'\x0c', 15.399999618530273, b'\x05')

Thanks for the other tip, it wasn't as difficult as I first thought!

Serial.write((const byte*)&myData, sizeof(myData));

@pugwash, @ZeFerby

Good one, ZeFerby!

Professor pugwash... you've taken this to the next level 🙂
Here's a bit of info on struct packing, as it can be done on the C/C++ side too:

struct-packing

Note: - There are also maco preprocessor directives (compiler specific), that you can include in your code to pack structs for specific needs.