Notifications
Clear all
Components & Programming
21
Posts
5
Users
1
Likes
2,450
Views
2020-09-22 3:31 pm
Wow, Byron, wow. I expected something, but wow!
Thank you so much man. I guess if this is not helpful, then noone can ever help. 🤣
I saw your message coming in, but when I tried to open I received a 401. So last message came in first.
Anyway... it may take some while until I get through with that. But wow, thank you.
Wolfgang
If I am not here, then I am most probably somewhere else
2020-09-26 11:04 am
I haven't got much time last two days to get deep into your code. Murphy's Law beat me hard, when my computer was litteraly broken and I had to order a new one. Now weekend I guess will take most of my time to make the new one in a way I am used to it.
I anyway had little time to dive through your example, try to understand and I guess - so far I can judge until now - that this is usefull for me.
Thanks again and have a nice weekend.
Wolfgang
If I am not here, then I am most probably somewhere else
2020-09-26 2:26 pm
Topic starter
2020-09-26 5:22 pm
What do you think?
The code was quickly cobbled together to illustrate mqtt messages going to and fro and as you can see there was hardly any error checking. The document I included on the mqtt library for the arduino indicates all the return codes. Your suggested amendments are all valid and, as the program is very short I suggest you submit an amended example of the arduino code in case it may help someone.
On the subject of secure networks it is worth pointing that out. Also to be wary of is the use of the mysql_connector, which I usually use for connecting to my mariadb, which could be the subject of an sql injection attack. However all my programs only work with my home network and are not used over any internet connection. If anyone is intending to use communications outside of their home network then security should be an important consideration.
And a note to anyone looking at the python program, this is again just a quickly written program for illustrative purposes so don't take it as a finished program. It was the first time I used the inbuilt python sqlite3 database so I was interested in seeing what it could do (and it confirmed that I should stick with a proper database for my usual db work 🤨 ). Im still working on a much larger program that interacts with mqtt messages with some gui windows, and I may take a different approach to checking if, when a mqtt message is published that is expecting a response, that a response has actually been received.
Thus far I have found, in my home network, the mqtt to be completely reliable and fast so much so that I only use mqtt QOS 0 (i.e. just send and hope 😀). I may revise this as I'm now doing a time critical (serval minutes, not u_seconds) program for my heating control.
A final thought for @wolfgangw, the mqtt is a lightweight and fast protocol and is useful for sending info over an ethernet network like temperature readings or sending a command to switch on a light that is triggered by a remote microprocessor etc. If your only desire is to store the reading in a database then, of course, you could establish a database connection direct from your esp8266 to update the database with the likes of a mysql_connetor. There are libraries for this available for the arduino environment. But if you are building an IOT home-brewed solution I think you will find mqtt to be a useful tool with a lot less faffing about than with ip tcp web type libraries as a solution of passing data. (but like everything - it depends...) However, now I think of it, @robo-pi wrote something in the past about how he likes to do this stuff with ip udp in a very lightweight way so it maybe worthwhile giving him a ping.
Topic starter
2020-09-27 12:31 am
@wolgangw
a further few thoughts for you before I turn in for the night whilst I wait for swimbo to finish watching some silly movie and my thoughts turn to my project. I'm on to doing a gui in python and it sure makes me wish I had a good old visual basic type of program to do this with.
But on to mqtt and @frogandtoad remarks on error handling. The Last Will and Testimony facility is to do with errors in the MQTT pub/sub process where connections are unexpectedly lost. With the arduino mqtt library I used this may not be much use as this lib appears to disconnect by design on a regular basis. The MQTT library I used for the rpi python code expects to be connected until the program closes.
Of course you will still have to report errors like incorrect readings, suspect readings, or the sensor not being available etc via published MQTT messages. A catch all for MQTT pub/sub errors is as per example in the in the rpi receiver program that sets and removes flags to check if the requested response was received. (the code was just a somewhat crude example of using a single global variable for this and can be improved) We must remember that publishing a MQTT message does not mean that a subscriber is expected to pick it up. The last message published can be retained on the broker if you desire for a new subscriber to pick it up when it connects.
And another comment is on the fact that I have had an mqtt broker running on an old rpi2 for some years now. Its not heavily bombarded but it pub/subs messages from a bunch of sensors giving messages on temperature, wind speed, wind direction, rainfall data, plus commands to switch on lights, and front gate opening etc. It's proved to be fast and reliable in this sort of environment so how far you wish to take checking for MQTT pub/sub errors is up to how critical the data is. I log stuff every 15 minutes to the database, but it would not mean much for historic trends if the logging stopped for an hour.
I know you are an experience C# programmer who will bring you own perspective to code written for microprocessors, and I know you have put some comments in some other thread about global variables etc. However for these small programs I think you will find it advantageous not to take the 'big program' thought process to the task. A very pythonic thought it is, but readability for you (or me in my case) it what should rule. Global variables, functions that do three things at once, I happily break best practice rules all the time 😎 . I suggest to folk, especially beginners, that they should think twice about obfuscating their code with attempting to do C++ classes when coding in the Arduino environment where simple functions will do. Admittedly as the programs we can fit on our microprocessors gets larger and we put in more sophisticated code one would need to tighten up on ones approach. But most programs I do, and see other do, in this hobby environment are really quite small. @frogandtoad is the forums code policeman and a stickler for the rules and I've received several violation tickets from him, but I've yet to pay the fine 😀
2020-09-27 6:28 am
What do you think?
The code was quickly cobbled together to illustrate mqtt messages going to and fro and as you can see there was hardly any error checking. The document I included on the mqtt library for the arduino indicates all the return codes. Your suggested amendments are all valid and, as the program is very short I suggest you submit an amended example of the arduino code in case it may help someone.
The whole point is that the code you provided is very unsafe, where safety was one of the key points @wolfgangw was inquiring about, and of which you also spoke about in previous posts. As for return codes, the function you wrote with it’s given signature is not even capable of returning any of them, so I was highlighting that for your oversight as well.
Aside from providing all the return codes for the status function, I also provided code examples for how each of the three functions I discussed should be used?
On the subject of secure networks it is worth pointing that out. Also to be wary of is the use of the mysql_connector, which I usually use for connecting to my mariadb, which could be the subject of an sql injection attack. However all my programs only work with my home network and are not used over any internet connection. If anyone is intending to use communications outside of their home network then security should be an important consideration.
It’s not so much the connector, it’s a host of other setups and or configurations, for example: turning off installation defaults, user access, passwords, encryption, stored procedures and prepared statements to try and avert SQL injection attacks.
And a note to anyone looking at the python program, this is again just a quickly written program for illustrative purposes so don't take it as a finished program. It was the first time I used the inbuilt python sqlite3 database so I was interested in seeing what it could do (and it confirmed that I should stick with a proper database for my usual db work :). Im still working on a much larger program that interacts with mqtt messages with some gui windows, and I may take a different approach to checking if, when a mqtt message is published that is expecting a response, that a response has actually been received.
Thus far I have found, in my home network, the mqtt to be completely reliable and fast so much so that I only use mqtt QOS 0 (i.e. just send and hope :). I may revise this as I'm now doing a time critical (serval minutes, not u_seconds) program for my heating control.
Indeed, MQTT is faster than HTTP requests, and a good choice for home automation. I also think as I mentioned in my previous post, that QoS should be incorporated to guarantee message delivery for obvious safety reasons.
A final thought for @wolfgangw, the mqtt is a lightweight and fast protocol and is useful for sending info over an ethernet network like temperature readings or sending a command to switch on a light that is triggered by a remote microprocessor etc. If your only desire is to store the reading in a database then, of course, you could establish a database connection direct from your esp8266 to update the database with the likes of a mysql_connetor. There are libraries for this available for the arduino environment. But if you are building an IOT home-brewed solution I think you will find mqtt to be a useful tool with a lot less faffing about than with ip tcp web type libraries as a solution of passing data. (but like everything - it depends...) However, now I think of it, @robo-pi wrote something in the past about how he likes to do this stuff with ip udp in a very lightweight way so it maybe worthwhile giving him a ping.
It might require a little more setup time (and depending on how far one wants to take home automation), but one may be better off installing Home Assistant, with Node-RED, InfluxDB and a host of other tools to make things easier in the long run.
Cheers.
2020-09-27 6:49 am
But on to mqtt and @frogandtoad remarks on error handling. The Last Will and Testimony facility is to do with errors in the MQTT pub/sub process where connections are unexpectedly lost. With the arduino mqtt library I used this may not be much use as this lib appears to disconnect by design on a regular basis. The MQTT library I used for the rpi python code expects to be connected until the program closes.
That disconnection is more to do with the TCP protocol. All the libraries work in the same way really, it’s just that some libraries provide functions to handle such disconnections for you, whereas others expect the user to handle it directly.
Of course you will still have to report errors like incorrect readings, suspect readings, or the sensor not being available etc via published MQTT messages. A catch all for MQTT pub/sub errors is as per example in the in the rpi receiver program that sets and removes flags to check if the requested response was received. (the code was just a somewhat crude example of using a single global variable for this and can be improved) We must remember that publishing a MQTT message does not mean that a subscriber is expected to pick it up. The last message published can be retained on the broker if you desire for a new subscriber to pick it up when it connects
Going back to my previous post, I don’t really see this kind of error handling as warranted by each client, or in the spirit of using MQTT. For example, I think a sensor should only implement a last will message upon subscription for hard disconnections, and simply publish temperature readings or a NaN otherwise, that’s it.
Ideally you want to run a separate client acting as a dashboard subscribed to all of the topics in your home. It can then provide real time monitoring of your devices, provide alerting services and even delegate to another client to handle specific errors.
A very pythonic thought it is, but readability for you (or me in my case) it what should rule.
Everyone should strive for readability, and one of the key points of readability is providing good meaningful identifier, function and class names. Doing so makes ones code self documenting, reducing the need for silly comments, for example:
int t_n[6]; // An array to store tattslotto numbers, vs... int tattslotto_numbers[6];
Global variables, functions that do three things at once, I happily break best practice rules all the time 🙂 . I suggest to folk, especially beginners, that they should think twice about obfuscating their code with attempting to do C++ classes when coding in the Arduino environment where simple functions will do.
On the contrary… I know you don’t like classes, not even in python, but they become essential when programs start to gain complexity, and in fact… they make ones code and program logic that much cleaner and easier to understand, let alone easier to debug code, etc… there are many advantages to using classes, and they even become necessary to write programs you could never write procedurally.
Admittedly as the programs we can fit on our microprocessors gets larger and we put in more sophisticated code one would need to tighten up on ones approach. But most programs I do, and see other do, in this hobby environment are really quite small.
That’s not entirely true. Take DB1 for example. Just one of the modules I’ve seen floating around is already around 1,000 lines of code alone, and not at all finished. We’ve discussed the need for classes in the past, and again, IMO, you cannot program a robot of this complexity without classes.
@frogandtoad is the forums code policeman and a stickler for the rules and I've received several violation tickets from him, but I've yet to pay the fine :).
Hardly a policeman or stickler… I simply point out for the benefit of everyone (especially beginners), when someone posts poor or wrong code or information. Likewise, I am always more than happy to be corrected if I do the same.
Would you stay silent if you saw someone provided an example that connects two AC 400V Active and Neutral wires together?
Silence is Compliance!
My advice to you is to learn how to program using classes in both C++ and Python, and you’ll thank me for it in the long run 😉
Cheers.
Topic starter
2020-09-27 10:51 am
The whole point is that the code you provided is very unsafe, where safety was one of the key points @wolfgangw was inquiring about
I think wolfgang was looking to see how to handle errors that can occur when reading a temperature sensor. MQTT is a good way of communicating between different microprocessors and computers and introducing MQTT for these communications means one has to consider the extra errors that may be caused by using the MQTT pub/sub. I have found that MQTT is reliable enough for messaging temperature type data without resorting to higher Quality of Service level provided by MQTT, but they are there to be used if desired. If you feel that a better example client program should be provided then why not just do so.
I know you don’t like classes, not even in python,
No, that is not correct. In python I caution about the creating user classes when they are not necessary which is often done by folks coming to python from other languages. In python everything is an object. Take the creation of a variable to hold a string. For example if you create myVar = 'hello'. and the type dir(myVar) you will see all the methods your newly created object has inherited. So when I see folks say 'I use classes to break up my code', I say have you considered breaking up your code into separate files. You can use the functions in a separate file in python by >> import 'myfile' as MF - and then using the functions in that file with MF.myfunction() in the main file. The namespace in python is very powerful. Yes python classes can do similar but at the expense of some readability. Classes are not to be ignored, just not used unnecessarily. I am in the process of coding a python GUI and I've been getting up to speed with tkinter. In looking at examples I've seen some that create a class for tkinter, but do nothing about modifying the methods. tkinter is a class anyway so creating a user class (a subclass) is not necessary. There are good cases to use classes and I like them very much.
When it comes to C++ however I find that classes make the code a whole magnitude more unreadable for the usually small chunks of code we do for microprocessors.
That’s not entirely true. Take DB1 for example.
That was in response to when I said most code written for microprocessors..... Most indicates a large majority but not all. However you choose the example of the so far published DB1 code. I did not count the number of lines but it was an easy readable bit of code that did not appear to warrant the use of classes. You indicated you thought is should be broken into classes way back when, but your suggested code improvements did not include a single class. I don't think this particular code will grow much as the whole way of working with DB1 appears to be discrete code running on a number of microprocessors that communicate via i2c
It might require a little more setup time (and depending on how far one wants to take home automation), but one may be better off installing Home Assistant, with Node-RED, InfluxDB and a host of other tools to make things easier in the long run.
Yes, each to his own. I've played about with them but these frameworks don't appeal to me, good though they are, as I prefer a more home brewed solution.
Would you stay silent if you saw someone provided an example that connects two AC 400V Active and Neutral wires together?
Now you are being silly.
2020-09-27 2:16 pm
Topic starter
2020-09-27 3:16 pm
...but then you concede by what you've written, that you're not actually using a "last will message" at all,
I have not used it up to now, but may do so in my latest more critical program. My example was to show MQTT messages going between an esp8266 and a rpi in a simple way and to indicate and provide links to documents containing examples, for a more robust use.
I already provided you with improved code examples
Some simple and obvious improvements you have suggested, but not on the thing you harp on about, namely the last will and QOS. I dont think an example of their use is needed as the doc links suffice. You have stated that this think this should be provided, but obviously don't feel this should be for you to provide. Just more noise.
I'm not sure what you mean by "modifying the methods"?
I mean inheriting the class methods and adding to, or overriding them. Isn't this obvious?
I'd be surprised if anyone viewing the following example (the class will be hidden from view in a header file), would consider it to be more difficult to understand than your procedural example
You are of course entitled to feel that way
For a simple example I might agree, but for MQTT where a connection to a broker is established and subscriptions made with 3 or 4 lines code early in the procedural code, then one only needs to use a single line like client1.publish("/lights/kitchen", "on"). (well that was python I let you think in C). No need to create a class, instantiate it and then use its methods. But no harm done if you choose to write a class for this of course, but why bother for a short bit of code to read a temperature sensor. You are, of course, entitled to feel differently.
I'm pretty sure you cautioned Robo Pi against using them in tkinter.
Nope. Read the thread again. Its not decent to put words into others mouth when you have not bothered to check on their accuracy. Naughty boy froggy.
It was the perfect analogy
No, it was silly.
This post was modified 4 years ago by byron
2020-09-27 4:57 pm
I have not used it up to now, but may do so in my latest more critical program.
Yes, I know... that's why I highlighted it. Let's put it this way... if you don't employ such features, then what benefit are you actually getting out of using MQTT over and above other communication methods? Many tutorial's I've read discuss the need to use these features in a positive sense, and advocate their use for good reasons.
Some simple and obvious improvements you have suggested, but not on the thing you harp on about, namely the last will and QOS. I dont think an example of their use is needed as the doc links suffice. You have stated that this think this should be provided, but obviously don't feel this should be for you to provide. Just more noise.
If I do harp on the need to use these features, then it's for good reason, and backed up by many articles on the subject, let alone many years of experience. Why is it that you're always negative against any suggestions for improvement, without actually having any real facts to back up your opposing views?
Once again, it's not my job to code your programs for you, and unless you are able to contribute with valid responses and reasons, rather than negative responses without reasons, then you're the only person making noise here!
I'd be surprised if anyone viewing the following example (the class will be hidden from view in a header file), would consider it to be more difficult to understand than your procedural example
Ok, you chose not to comment on whether the class example I provided was more clear and understandable, but it clearly is. What you fail to understand about OOP (and this is not language specific), is that it is perfectly suited for use with MQTT and home automation. Why? Because many people who employ MQTT want to have many clients... might have 10 light switches, 3 heaters, 3 television's, 8 camera's, 2 toasters, 4 garage lights, 1 door bell ... you get the picture!
The example client class I created can now be used to create all these clients very easily, and with valid names, and without duplicating any code at all, whereas in your procedural code, you will have to copy and paste every client connection function that you wrote, and your code would grow up to 4 times larger, harder to maintain, understand and debug... I could go on. If you don't see the benefit in that, then you got a lot to learn. But just because you can't see it, you should not be making preemptive negative comments for newbies to keep away from using classes and OOP... that's just plain bad advice!
I'm pretty sure you cautioned Robo Pi against using them in tkinter.
Nope. Read the thread again. Its not decent to put words into others mouth when you have not bothered to check on their accuracy. Naughty boy froggy.
Actually, I did read it, and here is your quote to Robo Pi:
More than once I see classes in python being promoted as useful for breaking down large program files, so I caution that one can do this by simply placing functions in separate files too.
Robo Pi even challenged you on your caution, and then you backed down from what I can see.
No, it was silly.
No, it was the perfect analogy... remember, silence is compliance 😉
Topic starter
2020-09-27 6:40 pm
Yes, I know... that's why I highlighted it
Alluding to MQTT and Last Will etc, yes, I know too, thats why I highlighted it in my post and provided links to docs that show its use. It is not necessary to use these features at all though in any professional type of program they would be. Whilst it may have been apposite to re-emphasis if you feel I did not highlight their use sufficiently, your additional continual highlighting is more to grandiose I think.
Why is it that you're always negative against any suggestions for improvement
It seems you like to create noise, and as we know, this has be pointed out to you by another before.
Robo Pi even challenged you on your caution, and then you backed down from what I can see.
Robo wondered why I use the word 'caution' which maybe was a poor choice of words I did admit I explained further with some other backup docs where I was coming from, and Robo came to the conclusion that there was indeed no need to create classes just for the purpose of splitting up python code. So you have endeavoured to pick a small snippet out a thread that does not show its essence, and wonder why I find some of your posts negative. Was it to grandiose?
What you fail to understand about OOP (and this is not language specific), is that it is perfectly suited for use with MQTT and home automation.
No, I do not fail to understand that OOP is well suited for the use of MQTT and home automation. I do much of this programming in Python. Do you not understand that using python one is mostly using OOP as all the python libraries are classes, largely written in C and C++, including the MQTT library. The only thing not relevant to classes and OOP programming are small user constructed programs in the Arduino environment. The Arduino libraries are often C++ classes anyway and user created classes, for the purposes of smallish programs like those to read sensors will, in my opinion, obfuscate the code making it harder to understand. Period. You continually fantasise by extrapolating about this beyond what I actually write. So read the words above, and that which I previously stated in this thread about the use of classes in python and my position on all of this should be perfectly clear. Well it is to me anyway. 😀 I've no wish to change you mind in anyway and I don't really see why you continually try change mine by alluding to some fantasy about what you think I understand.
No, it was the perfect analogy... remember, silence is compliance
Ok, so let's just say that some code froggy writes I think is rather bad. So bad its like risking electrocution from an incorrectly wired high voltage cable, so one cannot be silent about that or risk putting someone at mortal peril. That is silly and your analogy was silly. When trying for an OTT analogy you should endeavour at least to find a humorous example. Now if I said some of your posts were like a croaking frog, at least that would have some sort of relatively appropriate analogy with humorous connotations. (assuming you have a sense of humour) 🤔
croak croak
This post was modified 4 years ago by byron
2020-09-28 10:15 am
Hey guys,
I have no idea, what is this debate for. I was asking Byron, whether he want to give me some code example, which allow me to get into the basics of the MQTT. He did and I am happy.
If this is helpful or not, I can not judge yet, because I haven't had time to watch and try much.
Later, when I do, I will do it in my way after study the content and I will use it in a way which is matching to my needs, I think most people would do it in such way.
I asked for help, and I got help. This - in my understanding - is what a forum is standing for.
Wolfgang
If I am not here, then I am most probably somewhere else
Page 1 / 2
Next