MQTT Broker

Posted by: @byron

It worked just fine for a week before refusing to work.  I then found that Apple would only allow my own program on my own iPhone to work for a week for 'testing purposes' and after that I would need a developers licence with an annual subscription.  (FU Apple)

Aren't they special like that?  That is precisely why I use Android.  But I read this morning that Europe is giving the finger back!  "Apple, you shall use the USB-C plug or you are not welcome here."  Love it!

From my insomniac read last night, I had your "suggested readings" bookmarked.  I can tell you're well versed in this subject as well as Internet workings (in general)... especially that last post to @dualfuel.  I'll start digging into those links.  Thanks again for your fine tutorial.

VBR,

Inq

@inq

Posted by: @inq

When a sleeping consumer of the message wakes up... does it automatically receive the delayed/deferred message... or do I have to make the consumer request it

It depends on the QoS setting... if the QoS is set to 2, then yes... the MQTT server will resend the message until the consumer (client) receives it.

@byron

Posted by: @byron

There is a retained message facility that will keep the last message sent on the broker and will send it out to new connected clients that subscribe to the topic.  But its not that useful.  For example say the last message published on a topic had the message 'turn water on'.  The new client would receive that message and would probably initiate a process to do just that and then disconnect and go back to sleep.  Ten minutes later the client wakes up again, connects and subscribes to the topic, and, if the last message published was still 'turn water on' it would again receive the same message.  

As you mentioned, this is for newly connected clients, which should consume that message upon connection, and then that message should be removed for that new client, but I'm not sure the behavior is as you describe for an awakening client.

Can you point me to the documentation that describes that behavior?

@inq

One thing that may be worth mentioning is that not all mqtt clients are equal in that some do not implement all the Qos levels.  For example one I like to use for micropython, the excellent resilient asynio library by Peter Hinch only implements up to Qos 1.  Not that this has proved an impediment as I only find I need to use level 0 as I've never failed to receive a message 

Also, as you will probably note if you get to read to doc's,  Qos 1 and 2 message queuing only works with persistent sessions.

And I will confirm the behaviour of retained messages as described in case doubt was cast by other posters.

@bryon - Making some progress... or that olde dogs can learn new tricks.

I went through your first link to HiveMQ.  Went through the first 5 just fine.  Very useful.  On the more detailed versions of QoS (Episode 6) and on... I think I'm getting bogged down in the weeds with when I get the replies and or storing and deferred reception.  I'll get it... just need to fiddle with the behavior.

I also noted on their index page a topic explaining using Websockets... That answers my question above... that a web server page can be created to expose all the functionality of all my IoT devices on one dashboard!  👍 👍 

Here is what I've implemented... My Windows laptop is hosting VNC viewers of two RasPi Zeros and one RasPi 4.  Here are a few things along the way.

1. The RasPi 4 64bit image had MQTT already preloaded and the daemon started.
2. The RasPi Zero 32 bit image did not.  Those I loaded and started using your steps.
3. I was able to use the RasPi 4 as THE MQTT server and route all publish request through it to drive your LED on/off program on the ESP8266.
4. I was NOT able to use the RasPi Zeroes as the MQTT server.  I get "Error:  Connection refused" when I try to set up a subscription from another computer or the ESP8266.  I'm sure it's some kind of Linux permissions I need to set somewhere... (Linux is not my bag) but interesting to note that the RasPI 4 already had those permissions opened up.

I think, I'll write this up and use it in my User's Group class tonight.  Thanks again for your tutelage!  

Posted by: @inq

Making some progress... or that olde dogs can learn new tricks.

Good progress I see. 

A couple of points.  I'm pretty sure that no version of RPI OS comes with a MQTT broker installed.  Maybe you forgot that when you installed the 64bit image you then issued a 'sudo apt install mosquitto mosquitto-clients' command as you may have done this some time back.  The older version of the mosquitto MQTT broker would work after the install, but the latter version need to have an additional security check.  In my MQTT set up post I put in the 'allow_anonymous true' in the config file to allow all connections.  My property is not in wifi range of another so this is fine for me, but you may wish to refine the ip address of those that can connect.   Also a note that the MQTT version I use is 3.x  There is a newer 5.x version of MQTT but I've not looked at that as 3.x works just fine for me and I'm not looking to complicate my life.

I think a rpi zero may be a little underpowered to be a MQTT broker especially as it just has a wifi connection. (Just fine for an MQTT client though)   I checked on the old rpi that happily still runs a MQTT broker and its a pi 2 model B with and connected with a wired ethernet socket.  The newer version of the rpi Zero (the Zero 2) with its 4 cores may work ok, but I would rather be thinking of a wired ethernet connection for a broker.  Whatever version of RPI OS you use on whatever rpi you select, the actual config settings should be the same and no different or extra permission settings should be required.

Whatever version of Qos you may use, don't forget the Qos levels are for communication between the publisher and broker, and subscriber and broker.  For example a publisher has no way on knowing who/what is subscribing to the message topics it publishes and a broker has no way of knowing that an unconnected client would be subscribing to a topic if only it was connected.  

When I need to be sure a particular subscribing client has received and acted on a published message I get the subscribing program to publish a message back to the original publisher program (which of course must be subscribing to the return message topic)

For example I use MQTT (Qos 0) where a heating control program publishes a 'boiler on' message to a boiler control program.  I follow this up with another message sent from the heating control to the boiler control program requesting the current state of the relay the boiler program controls and it reverts with a message as to the state of the relay.  If this return message is not received or the relay state is not as expected then alerts are generated.

Another method I use for the same system where I expect temperature sensor nodes to publish temperature readings every 30 seconds is to note the time the temp readings are received.  A periodic function runs every two minutes and checks the current time to the last temperature received time to raise alerts if necessary.

When I first set up a MQTT broker I gave it a test by having a client publish a message in a loop (Qos 0) with no delays that contained an incremented a number along with some other bumph in its payload.  Another client program subscribing to the same topic received the messages which then stored them in an array.  I think I simply sent about 500 messages in the loop.  When the last message was received I then had another function that looped through the array to check if all 500 messaged were received and that they were appended to the array in the correct numerical sequence.  All were received OK and in the correct order.

I was not attempting to find the limits of my MQTT set up, but just to check if it was OK for my use, and I have no requirement to have to hammer it like my test did.   But you could easily run some similar test on your eventual set up of broker and clients publishing and subscribing to a variety of messages all at various Qos levels.  I think that most wifi based comms issues arise when a node is getting towards the edge of the wifi range capabilities.  Raising the Qos level may well assist for dodgy comms but probably will not be needed otherwise.  

I expect your endeavours will soon exceed my experience of using MQTT so a write up of how to get a broker up and running and using it with sleeping ESP8266 or ESP32 and having a browser based control setup would be a very interesting to see.