Notifications
Clear all

MQTT Broker

125 Posts
8 Users
61 Likes
7,126 Views
byron
(@byron)
No Title
Joined: 5 years ago
Posts: 1112
Topic starter  

@inq @zander

Probably my final post in this subject, though feel free to ask me on anything I have posted.  I get the feeling my post is a bit shallow and may not have explained various bits in the proper depth, but then again I may have waffled on far too much.

So this is a follow up from last nights small snippet of a program that connects to a MQTT broker and subscribes to some topics, and I hope a better explanation is now given.   

Before turning in for the night I started on that program which was meant to be a quick cut and paste from another larger program,  a quick bit of deleting the unnecessary stuff, a recompile and load and a quick test is what was intended.  What should have taken only about 20 minutes turned into an over 2 hour session and a late night.  But the number of times that has happened I really should know better by now.

But to business.  A MQTT message consists of a topic and a payload.  A message is published to a broker, and all the programs that subscribe to the topic on whatever computer connect to the network, will be sent the message by the broker.  When publishing a message a Quality of Service level can be stipulated. Basically this indicates if the broker will acknowledge the message receipt at the broker to the sender.  However it does not indicate that a subscriber program has received the message.  This has to be handled at the application level.

For example I have a program that controls my home heating currently running on an ESP32.  This program receives room temperature readings via a MQTT message, makes decisions on whether to turn the boiler on or off depending on its on/off time schedule and the room desired temperature setting.  It then sends a MQTT message to the relay controlling the boiler, and sends  another MQTT message to be picked up by the Heating Status Display program in the format of '22.4,22.0,True,False,off'  (current temp, desired temp, heat allowed per time schedule, heat allowed per temperature setting, boiler on or off).  This status message is picked up and displayed as dials and coloured blobs by the display program.  

The ESP32 heating control program runs an asynio based MQTT client program that will automatically reconnect to the wifi/broker should it go down  and one of its async functions is to blink a blue light to indicate that the async 'round robin' scheduler is working and a green or red light (a neopixel led) to show if the wifi/broker is up or down.  

But the display program, far removed from the control program on the ESP32 board,  has no way of knowing if all is well except that those status messages cease to arrive.  So the time a status message arrives is logged, and periodically (every 2 minutes) a check is made on the time the last message was received to the current time is compared and warnings are displayed along with a sound buzzer.

Of course these sorts of program resilience checks need to be programmed, but using my rpi based MQTT broker from various programs to publish and subscribe data such as weather station readings to be stored in a mysql database, home heating control, green house and workshop heating control, a host of sensor readings, front gate monitoring (open or closed), switching on/off outside lighting etc, all with a QoS of 0 (no broker ack) I have never noticed a missed message. The messages are all sent and received, in human terms, seemingly instantly.  (press the button on a board to send an message to switch on a light connected to a relay on a board way down the garden to switch on a light, and the light is on just as quick as operating a mains light switch).  I've also yet to bother with telling the broker about a 'last will and testimony' as I not felt the need in my programs.

Now to put more meat on the MQTT bones I hand over to some better documentation than my waffle.  Have a look at the following 

https://www.hivemq.com/mqtt-essentials/?utm_source=adwords&utm_campaign=&utm_term=mqtt&utm_medium=ppc&hsa_tgt=kwd-28940447691&hsa_cam=2053640785&hsa_src=g&hsa_net=adwords&hsa_kw=mqtt&hsa_ad=513050862223&hsa_grp=76047219100&hsa_ver=3&hsa_acc=3585854406&hsa_mt=b&gclid=Cj0KCQjwgYSTBhDKARIsAB8Kuks5aaH3JNdWnMVV4WUQnUXopkZrwaFwA1xef238vVTRz2HBnh3FS0IaAtlwEALw_wcB

http://www.steves-internet-guide.com/mqtt-protocol-messages-overview/

I have mainly followed MQTT tutorials and examples in python with steves-internet-guides being some of the best for MQTT, but my links may not be the best for you.   I did a quick google of C++  Mqtt tutorials and theres a bunch out there but I'm sticking with python.

Finally a quick note on my python 'prejudice'.   Way back in the early 70's I did a short spell at earning my living being a C programmer which soon transmuted into being a business analyst (Id previously qualified as an Accountant so that was a natural progression and the programming spell was just because I wanted to have a go).  Some time in the early 80's from hazy memory, for a few years I made my living as a freelance VB and SQL Server programmer before again moving on.  When I retired in 2014 in my early sixties I came across the rpi and arduino as hobby play things.  To start with I found I was more comfortable in C on the arduino, though my C was a long long time ago but I did go through the C++ tutorials.  At the same time I took to learning python on the Rpi.  It soon became very apparent it was much easier to get up and running with the more advanced programming requirements in Python and I was mightily impressed that after one evening going through some tutorials on multi threading and multi processing my multi threading programs were up and running the very next day.  At that time it settled down to python on the rpi and c / c++ for the arduino boards.   But then, of course, along came micropython to run on some of the new and cheap microcontroller boards, and when the larger capacity boards based around the ESP 32 came out, and micropython advanced to the stage of having a very nice asyncio implementation, switching to micropython and microcontroller boards that could run it became a no brainer for me.   But there's still a very good use of arduino C for the very many program examples and some of the many arduino boards that dont run micropthon, or boards like the ESP8266 that typically don't come with enough memory to properly make use of micropython.  But if I compare the very good asyncio MQTT library for micropython to the MQTT libraries found for the arduino uno or the like, then no contest at all.   Which was why my interest in @inq 's webserver library running on an ESP8266 was sparked as that is very possibly the best way to go for the Arduino related programming.   For those who are up and running with C++ I'm not suggesting a switch to python as theres still a large learning curve required  that I probably underestimate as I've absorbed the learning of it all over several years now and I'm no where near the competent professional level (and don't feel the need to be either, as long as I can get my programs to work ok for me)

But no more time for all this fireside chat, I've finished my cuppa, the sun is up, and I need to get my tractor out to turn over the horse manure pile thats been building up for a few months.

 


   
Inq, Lee G, Ron and 1 people reacted
ReplyQuote
Inq
 Inq
(@inq)
Member
Joined: 2 years ago
Posts: 1900
 

@byron - I truly appreciate all the trouble you are going to for the forum.  I don't know if it was my request for a link or two, but this is above and beyond.  I don't want it to seem like I'm not interested or appreciative.  🤩 

We are now finally experiencing some good weather here without nightly freezing and I need to pour some concrete.  At my age, it'll be a slow process. 😉 In the evenings, I expect to be suffering in an easy-chair... even the brain will hurt.  

But... I have a blank RaspPi IV-8GB dedicated to this project.  And... I have my fingers crossed that I'm going to FINALLY get some un-metered Internet at the house so I can actually download the image here instead of at the Library.  I want to try the 64bit version of the OS.  This will eventually be the top-level brain for my ten-year plan 🙄 robot. 

I trust you'll be available for the Inquisition that will soon follow. 😆 

VBR,

Inq

3 lines of code = InqPortal = Complete IoT, App, Web Server w/ GUI Admin Client, WiFi Manager, Drag & Drop File Manager, OTA, Performance Metrics, Web Socket Comms, Easy App API, All running on ESP8266...
Even usable on ESP-01S - Quickest Start Guide


   
ReplyQuote
Ron
 Ron
(@zander)
Father of a miniature Wookie
Joined: 3 years ago
Posts: 6664
 

@byron Thanks for all the effort, I will dig into it as soon as my 90% project is finished, today is ear day.

First computer 1959. Retired from my own computer company 2004.
Hardware - Expert in 1401, and 360, fairly knowledge in PC plus numerous MPU's and MCU's
Major Languages - Machine language, 360 Macro Assembler, Intel Assembler, PL/I and PL1, Pascal, Basic, C plus numerous job control and scripting languages.
Sure you can learn to be a programmer, it will take the same amount of time for me to learn to be a Doctor.


   
ReplyQuote
frogandtoad
(@frogandtoad)
Member
Joined: 5 years ago
Posts: 1458
 
Posted by: @zander

@byron Thanks for all the effort, I will dig into it as soon as my 90% project is finished, today is ear day.

For anyone interested in MQTT, you can actually just install an MQTT broker (better referred to as just an MQTT server these days) on any machine, raspberry pi, or dedicated desktop (windows included), and do any testing you like without attaching any external devices whatsoever... every new client device can be emulated by just running a new shell (CMD etc...).

Thus... all of your testing can take place locally on your machine without installing too much confusion to get started!

Cheers


   
Lee G reacted
ReplyQuote
byron
(@byron)
No Title
Joined: 5 years ago
Posts: 1112
Topic starter  
Posted by: @inq

I don't know if it was my request for a link or two, but this is above and beyond.

Just to go a teeny bit further, I now see I left out some code to show the mqtt publishing, but I guess the connection and subscribe example will suffice, but here is the code to publish: (in the context of using the arduino pubsubclient as per code snippet I provided in a previous post.

mqttClient.publish("Topic/SubTopic/SubSubTopic", myjson );  

So thats easy enough.  One way often used to send serialised data is to send it in the json format which is what the myjson variable represents.  Both the topic and payload are transmitted as binary data, but the format to use in your program can be mqtt library dependent and most I have used want the topic and payload to be input as strings or a structured format such  json.

I hope it gives a starter, and my aim was not to provide much code, but rather just to show the preliminaries that I found a great help when getting going with MQTT.  i.e  (1) get yourself a local broker which can run on a small linux box such as a rpi and for which I provided the mqtt installation steps. (and of course mqtt will run on PC's, any linux box, and mac's as well but I expect all wiil have their own installation requirements that you will need to research). (2) get MQTT connect/publish/subscribe program (which are free) such as MQTT-fx which gives a head start to play with messages.  But also remember if you cannot get a local broker set up to start with then its also very easy and free to use a public broker to have an initial play.  


   
Lee G and Ron reacted
ReplyQuote
Inq
 Inq
(@inq)
Member
Joined: 2 years ago
Posts: 1900
 

Looks like my option for real Internet bandwidth fell through.  But, I am making progress on the MQTT front.  I'm downloading the Raspberry Pi image as I write.  A local restaurant/bar has pretty good Internet.  That's my story and I'm sticking to it.  😆 

I forgot to bring the RaspPi, so I'll have to install and set up the Remote desktop and bring it back down to do all the updates and install the broker.  Ahhhh... such a shame to have to come back a second time in the same week.  😋 

3 lines of code = InqPortal = Complete IoT, App, Web Server w/ GUI Admin Client, WiFi Manager, Drag & Drop File Manager, OTA, Performance Metrics, Web Socket Comms, Easy App API, All running on ESP8266...
Even usable on ESP-01S - Quickest Start Guide


   
ReplyQuote
Ron
 Ron
(@zander)
Father of a miniature Wookie
Joined: 3 years ago
Posts: 6664
 

@inq Are you not able to get Starlink, or are you waiting for the 'on the move' version supposedly out later this year. I start a cross country trip in about a year and would really like to have starlink since in Canada cellular internet is horribly expensive. Another issue is there are 2 big players, and they divided the country up so if you sign up with one you get second tier service in the other guy's area. I am on the west coast, but I may just opt for the east coast provider since we will be spending much more time there and are not coming back here when the trip is over.

First computer 1959. Retired from my own computer company 2004.
Hardware - Expert in 1401, and 360, fairly knowledge in PC plus numerous MPU's and MCU's
Major Languages - Machine language, 360 Macro Assembler, Intel Assembler, PL/I and PL1, Pascal, Basic, C plus numerous job control and scripting languages.
Sure you can learn to be a programmer, it will take the same amount of time for me to learn to be a Doctor.


   
ReplyQuote
Inq
 Inq
(@inq)
Member
Joined: 2 years ago
Posts: 1900
 

@zander - I got notice that my Starlink number came up about a week ago.  Checked with the app and I have WAY too many trees and mountains in the way.  I don't know what standard it is, but there is a wireless, tower based Internet standard.  It goes by many names.  My local provider goes by the name SkyFi.  They said no a couple of years back, but if they get a customer that can "relay" it opens an option.  A new house near me went to get service and I have direct line of sight to it.  Unfortunately, the verdict came back they don't have line of sight to the tower.  Bummed! 😕  

This is a particularly bad time of day... I started at 5:10 trying to send this one pid'ln post.

3 lines of code = InqPortal = Complete IoT, App, Web Server w/ GUI Admin Client, WiFi Manager, Drag & Drop File Manager, OTA, Performance Metrics, Web Socket Comms, Easy App API, All running on ESP8266...
Even usable on ESP-01S - Quickest Start Guide


   
ReplyQuote
Ron
 Ron
(@zander)
Father of a miniature Wookie
Joined: 3 years ago
Posts: 6664
 

@inq Strange, don't know why mountains would have any effect, most dishes I have seen are pointed almost straight up. He does have a few satellites. The on the move version will use lasers to communicate between the satellites so a ground station isn't needed. This is for the sailors and RVers (me) and anyone on the move.Trees however can be a problem, what I saw when I lived in satellite country (no other option) was dishes placed roadside aimed at a single geosynchronous satellite (before starlink) then a very long cable back to the house. I don't know how long it can be for a starlink though, it needs power to move the dish.

First computer 1959. Retired from my own computer company 2004.
Hardware - Expert in 1401, and 360, fairly knowledge in PC plus numerous MPU's and MCU's
Major Languages - Machine language, 360 Macro Assembler, Intel Assembler, PL/I and PL1, Pascal, Basic, C plus numerous job control and scripting languages.
Sure you can learn to be a programmer, it will take the same amount of time for me to learn to be a Doctor.


   
ReplyQuote
byron
(@byron)
No Title
Joined: 5 years ago
Posts: 1112
Topic starter  

@inq

A dicky internet is not good, you must be out in the sticks somewhere 😎   Your post on your rpi triggered me to remember I was going to give the 64 bit version a go.  I'm putting it on my test rpi4 that has an SD card to boot off so the download was finished after just a minute or so.  I'm about to find out if I can load and run all my usual programs.  The new version of the Rpi imager also seems to work well, and I see one can pre-load the un and pw to use and the and wifi connection info.  I just hope Im not in for another late night 😀 (probably will be though)


   
ReplyQuote
Inq
 Inq
(@inq)
Member
Joined: 2 years ago
Posts: 1900
 

At your latitude, it's easier.  At my latitude... not so much.  That's why it just opened up in my area only a week ago.

25 degrees elevation is well below my "effective" horizon over 70% of my azimuth... even if I don't count my trees.  And trees count.

3 lines of code = InqPortal = Complete IoT, App, Web Server w/ GUI Admin Client, WiFi Manager, Drag & Drop File Manager, OTA, Performance Metrics, Web Socket Comms, Easy App API, All running on ESP8266...
Even usable on ESP-01S - Quickest Start Guide


   
ReplyQuote
Inq
 Inq
(@inq)
Member
Joined: 2 years ago
Posts: 1900
 

Once they get the constellation closer to being filled, the degree elevation is supposed to go up to 40 degrees.  At that point, I might have a chance.  

3 lines of code = InqPortal = Complete IoT, App, Web Server w/ GUI Admin Client, WiFi Manager, Drag & Drop File Manager, OTA, Performance Metrics, Web Socket Comms, Easy App API, All running on ESP8266...
Even usable on ESP-01S - Quickest Start Guide


   
Ron reacted
ReplyQuote
Ron
 Ron
(@zander)
Father of a miniature Wookie
Joined: 3 years ago
Posts: 6664
 

@byron Keep in mind that the 64 bit OS is not any faster, it just allows insane memory spaces. I can't imagine ever using anything close to that on a Pi, maybe one of the M2 super Macs though.

First computer 1959. Retired from my own computer company 2004.
Hardware - Expert in 1401, and 360, fairly knowledge in PC plus numerous MPU's and MCU's
Major Languages - Machine language, 360 Macro Assembler, Intel Assembler, PL/I and PL1, Pascal, Basic, C plus numerous job control and scripting languages.
Sure you can learn to be a programmer, it will take the same amount of time for me to learn to be a Doctor.


   
ReplyQuote
Ron
 Ron
(@zander)
Father of a miniature Wookie
Joined: 3 years ago
Posts: 6664
 

@inq What lat are you at? I thought it would be more difficult way up north where I am.

First computer 1959. Retired from my own computer company 2004.
Hardware - Expert in 1401, and 360, fairly knowledge in PC plus numerous MPU's and MCU's
Major Languages - Machine language, 360 Macro Assembler, Intel Assembler, PL/I and PL1, Pascal, Basic, C plus numerous job control and scripting languages.
Sure you can learn to be a programmer, it will take the same amount of time for me to learn to be a Doctor.


   
ReplyQuote
byron
(@byron)
No Title
Joined: 5 years ago
Posts: 1112
Topic starter  
Posted by: @zander

Keep in mind that the 64 bit OS is not any faster

My main reason for trying it out is that some software is only provided for 64 bit os, and there is a general trend for dropping 32 bit options (or so I read)  Maybe in future one only gets 64 bit versions of the mqtt broker for example.  


   
ReplyQuote
Page 2 / 9