MQTT Broker

Posted by: @inq

The Internet is already starting to hang up as I write this.

Which may mean you don't see this post until later, but just to remind that halfway through this thread I installed the rpi os 64 bit and reinstalled the mosquitto mqtt and did another set of notes on it while it was very fresh in my mind.  Very similar to the first install notes, but maybe a bit clearer especially about using the nano editor to update the necessary files if you are not so familiar with linux

@byron - Thanks for the heads-up... I'll find it.  The one redeeming factor in this.  If I work on it at 3AM here when I have good Internet, it'll be during your normal work day.  🤣 

Ok, I am back from a journey in the weeds. @inq I was changing the Time to Line programming in my WRT54G router so that my Visible ZTE R2 would provide a hotspot link to the router and then the router would provide multiple devices with internet access. I learned a lot. I didn't succeed but I still have one thing to check. I AM able to use the R2 as an AP when it connects wirelessly to my TP-Link AC750 Travel Router. I had been using the ethernet port on the AC750 to provide AP to the WRT54G. I had been using The DD-WRT firmware on the WRT54G in WDS mode, providing lowlevel internet around parts of my property. So, I have to compare what the TP-Link is doing right with what the settings on the WRT54G are doing wrong.

Why all this?

Well byron, it appears if you want to use a public broker, you better have an AP. So I was shooting for an inexpensive AP source. Visible at $25/month plus a $10 used WRT54G v.2 w/DD-WRT, and external antennas is a ridiculously cheap unlimited data plan with a 5mps down.

Setting that aside for a minute. I am also interested in MQTT on a non-AP private LAN.

So, I wanna run through how I think it should go.

1. Broker needs to be attached to the master router in a WDS. So lets say, a RPI3 ethered up to a WRT54G who has both a parabolic antenna, and some sort of isotropic antenna like a J-pole.

2. Clients have to be within range of the WDS, and be able to find it.

3. I have to learn about MQTT packet size.

4 I have to learn how to configure MQTT to handle 1200baud ax.25 packet speeds.

ok, off to Steve's to get me sum learnin.

 

" halfway through this thread I installed the rpi os 64 bit "

When you do cool things like that, can you hyperlink to it when you talk about it? We are at 8 pages already on this thread and its slow slogging through to find the nuggets.

THAHNKS!

Posted by: @dualfuel

I am also interested in MQTT on a non-AP private LAN.

 

@dualfuel

If you want someone to be notified of the fact you have referenced then in a post then you need to use the @name form of address or they will not get an email alerting them to a post.  I've only just noticed your post.

MQTT on a non-AP private LAN indicated a wired ethernet connection, but maybe we are not using the AP term in the same way.  So bear with me while I go explain what I understand of the term Access Point.

A home network can just be made from ethernet cables where each computer (or TV or whatever) is wired through via an ethernet hub or ethernet switch to each other.  There can also be a wifi connection (even just a wifi connected network) where the connections are made to a wifi network router through an Access Point.  There can be multiple Access Points on a network especially if the network is spread out and each individual access point does not provide the desired range.  Multiple Access Points can communicate via ethernet cables or meshed together where they communicate to each other via wifi. 

A typical home router will provide an Access Point for wifi connections and probably some additional wired ethernet cable connections.    

A router can also be connected to a Wide Area Network (though not necessarily so) and this would typically mean an internet connection through some means, such as a wired connection (like a telephone), a cell phone type of connection (4G 5G etc), optic fibre cable, satellite link cable etc. 

So in my terms the Access Point is a wifi connection to a local network and its typically provided by a router.  If the router has a connection to the internet  then it can send and receive messages to other things linked to the internet.  

To connect to a public MQTT broker over the internet you will need to connect your device via an ethernet cable or via an Access Point to make a wifi connection, to access your home network, and your router will need access to the internet.

To connect to your own broker then the computer on which it resides will need to be connected to your home network either via an ethernet cable or via a wifi connection to your networks Access Point (probably on your router).  Then other computers or things connected to your home network, again either via an ethernet connection or via a wifi connection to an Access Point on your home network, will be able to talk to your local mqtt broker.

If your wifi AP connection is out of range then you can add additional wifi Access Points.  For example where I live I have extended the reach of wifi by laying ethernet cables to outlying buildings and putting in routers with wireless Access Points at those outlying locations and thus get full network speed at those remote spots.

There are of course other ways to send some data to and fro between your computers.  Just using HTTP over the local (or internet) connection will suffice.  Also there are other means to communicate locally that do not use a home network connection (wifi or ethernet), but can also communicate wirelessly.  Here I am thinking of the likes of LoRa, cell phone types of connection, zwave, ZigBee etc.

The case for using MQTT is that its a lightweight messaging facility, its easy to set up your own local broker,  and it very easy and simple to use in your programs.  It also has the advantage that before going to the trouble of creating your own Broker its easy to see whats involved by initially using an internet Broker (if you are connected to the internet of course)

Posted by: @byron

When you do cool things like that, can you hyperlink to it when you talk about it? We are at 8 pages already on this thread and its slow slogging through to find the nuggets.

THAHNKS!

Not that cool, I just used the rpi imager software (as download from the rpi site) to upload the 64 bit in place of the 32 bit offering.  You will not notice any difference and I only put in on my rpi 4 in case I wanted to load any 64bit software that is not available with 32 bit offerings.  

I made the additional post on loading an mqtt broker to an rpi as I noted that inq was also in the process of loading the 64 bit version and I took some notes about what I did to load it in case there were any differences.  There were no differences, but the notes were perhaps more succinctly made and they did include a slightly more step by step approach to creating the config file for someone not familiar with using a rpi.  When I wrote the reminder post to inq that post on the 64bit installation was by then several pages back.  To find the actual page to hyperlink to it would have taken me as much time as it would take you to find it.  

Which is why as @Zander often mentions, creating this sort of information is not best suited to this style of forum and he would like a section for documents that cannot be polluted by comments.  But I doubt if his desired forum additional facility will come to fruition though I support his desires.  You will have to continue to slog for your nuggets I think.

Posted by: @byron

When developing mqtt programs it can be of great assistance to have an mqtt message sniffer on hand.  The one I like to use is MQTT fx which can be downloaded from here

https://www.jensd.de/wordpress/

but there are others.  

Got broker installed, conf file...

Got down to this.  Apparently he sold it out to some other company.  The old links for downloads go to 404 pages.  The new company is selling it for 50 Euros.  Can you recommend any others?  You said sniffing above...

Do these tools actually sniff it at the network level like WireShark or are they connecting through standard MQTT standards and watching the traffic?

VBR,

Inq

@inq  Oh thats a shame 50 Euros...  I have also used the mqttbox I mentioned, but that was some while back and I've only used it on my mac. I have not downloaded any recent versions as MQTT fx was my preferred one.    

It is also possible just to send mqtt messages from the command line,  but I would have to lookup up the necessary.   I guess WireShark would be a proper network sniffer, and I rather loosely used that term sniffer when referring to the other MQTT 'testers' which are all really just applications that make it easy to connect to a broker to publish and subscribe messages that your program in development may be sending or receiving.  Presumably WireShark would also work but maybe at the expense of a lot of other extraneous info and it may not be geared to letting you simply enter MQTT messages for publishing.  I've not used it.

I wonder if my python snippet of an MQTT publish and subscribe program would be of any use.  Its short and sweet and  I use it as a template to use when I need make a program that requires the use of MQTT.  .  It will run on the rpi4 and you can easily change the MQTT message text by firing up the Thonny python IDE included with raspi os to amend the program.  It will prove that your rpi broker is working, but it is python so it may not be suitable for you.  Let me know.  Otherwise have a go with mqttbox or mqtt explorer,   I'm hoping these will still be free 😎.  I dont know if they run on the rpi, but there should be versions that run on a mac or PC.

@byron mqtt explorer is also available on Mac.

Posted by: @byron

It is also possible just to send mqtt messages from the command line,  but I would have to lookup up the necessary.

Please don't go to the trouble.  I'll continue with the tutorial without the sniffing.  

Posted by: @byron

I rather loosely used that term sniffer when referring to the other MQTT 'testers' which are all really just applications that make it easy to connect to a broker to publish and subscribe messages that your program in development may be sending or receiving.

This is good clarification of what is going on with inspection of the MQTT.  Don't want to use an EKG machine when a stethoscope will work fine.

Posted by: @byron

I wonder if my python snippet of an MQTT publish and subscribe program would be of any use.

Maybe, maybe not.  Don't want to learn Greek to learn MQTT... yet. 🤣 

VBR,

Inq

@byron, et al - I've not used the Sleep Modes of an ESP8266 or ESP32.  How, in general, can those be used in conjunction with MQTT? 

Say... I have a trivial sensor node that I want to make battery operated.  I've read that with long sleep intervals, such a device can run on a AA battery for over a year.  It also seems I've read that MQTT can be randomly connected... The ESP comes out of sleep, get it's sensor reading, does its MQTT thing, reports the value to some server and goes back to sleep.

I'm assuming that MQTT is just using TCP under the covers to perform all the communications??? What is the advantage of adding MQTT instead of just doing those steps with a TCP connection?

VBR,

Inq

Posted by: @inq

What is the advantage of adding MQTT instead of just doing those steps with a TCP connection?

Not a lot of difference and both comms would work just fine for you I'm sure.  I suspected this answer would have been asked before so I did a quick google and the following link will provide a good rational for you.

https://www.electromaker.io/blog/article/http-vs-mqtt

I've used both and find I prefer the publish / subscribe methodology but your milage may vary.  Its so easy to set up a MQTT broker that just runs for ever without a hitch.  I've got a broker running on an old rpi 2 or is it 3, no matter, I set it up 4 or 5 years ago, and have never updated the rpi and I've just let it run continuously.  Its never failed and once I've set up a sensor, or whatever, to publish data, its also very easy to then choose to subscribe to that topic on whatever device to receive the data with a really simple program construct.  

But one can also say the same for using http.  Whilst I found a preference for MQTT it a bit like being asked if one prefers blonds or brunettes.  At the end of the day I would happily get in the sack with either. 😀 (well in my long distance youth anyway 😎 )

Posted by: @byron

Its so easy to set up a MQTT broker that just runs for ever without a hitch.  I've got a broker running on an old rpi 2 or is it 3, no matter, I set it up 4 or 5 years ago, and have never updated the rpi and I've just let it run continuously.

Perfect - I think that is a great justification!  I hadn't thought of that.  I could add projects around the property willy-nilly and never have to touch the "server" and I have plenty of old RasPi's 2s, 3s just gathering dust.  In an HTTP or TCP system, I'd constantly be modifying both ends.  

Posted by: @byron

if one prefers blonds or brunettes.  At the end of the day I would happily get in the sack with either. (well in my long distance youth anyway )

🤣 My buddies would never have asked that question - we didn't discriminate.  😎 

Been at the library all day to get Internet that can do more than make a post to the forum.  It appears the 64bit install I did on the RaspPi 4 already had all MQTT Broker.  Just before leaving, I got the client side libraries you recommended downloaded and compiled your sample INO.  Didn't get a chance to see it working... yet.

You mentioned having a broker on old Ras 2, 3's.  I downloaded the latest 32 bit before leaving, want to try it on a Ras Zero (1st Gen).  Is there really anything performance oriented things, that I'll run into down the road?  Obviously, there is nothing I'm going to do real-time.  

I know how to put web servers / MySQL on the RaspPi so I can do all the history things I might want and serve webpages.  Is there some easy/clean way of feeding input from a browser front-end dashboard into the MQTT system so I can monitor and control all these IoT nodes?

VBR,

Inq

P.S. This MQTT is growing on me so far.  Thanks for your tutorial.

Posted by: @byron

The mqtt broker could also be a public broker to which you can publish and subscribe to messages from your computers or microprocessor board linked to your home network (assuming an internet connection from you home network).   But you will be limited in the amount of data you can transmit and your data would not be secure (others can read it).  You can use secure public mqtt brokers but those will like to charge you for the privilege.

Correct me or elaborate if I'm wrong or missing something important.

I think I see some great advantages even within my own little LAN.  I want to start exploring remote/battery operated clients.  Obviously to make them last any length of time, they have to use sleep modes judiciously.  This eliminates me from using my web server library on them, since it's not known when they'll be awake and accessible.  As a MQTT client, it can "do it's thing", notify the broker, use the QoS feature to get confirmation, and then go to sleep.  The broker being AC powered, is always available.  When a sleeping consumer of the message wakes up... does it automatically receive the delayed/deferred message... or do I have to make the consumer request it?

Public Broker Questions

Before retirement and still living in the "big city" my ISP provided me with a static IP.  I could easily use port forwarding on the router and set up an ESP8266/InqPortal as a server exposed on the Internet.  It makes it very convenient to see what's going on AND exposes any nice web page GUI I care to make to be seen anywhere in the world simply by browsing to my $1 server.  

Now, that I'm behind crappy Verizon Internet, remote access is impossible by the above method.

Is this correct? -

1. I can have, say... a sensor client posting message topic/payload and other consumer clients that will receive that message topic/payload and do something with it.  But instead of having an internal LAN broker, I can just as easily use a public broker and do the exact same thing???
2. With the intent, that I could also have, say... a MQTT client on my cellphone while away from the house that could receive those messages from that public broker and make messages that will control the clients within my house, behind my firewalls???
3. Does a cellphone MQTT client imply it must be an native App?  Can a webpage be created that is a MQTT client (getting/sending messages)???
4. If (2) is correct... do you know how they do that???  I know public facing HTTP servers can only support a relatively small, finite number of connections.  It works fine since HTTP (usually) releases the connection after serving the web page.  Since the public broker can not initiate a connection to my MQTT clients inside my house, they must be initiated by my MQTT clients.  That public broker would have to keep those connections for the off chance a message needs to be sent down to the clients.  That would mean the public broker might have to maintain thousands (millions???) of connections.  How is that done... or is there some other mechanism going on here besides strait TCP???

😎 Inquisitive minds want to know.  🤔 

VBR,

Inq

Posted by: @inq

When a sleeping consumer of the message wakes up... does it automatically receive the delayed/deferred message... or do I have to make the consumer request it

There is a retained message facility that will keep the last message sent on the broker and will send it out to new connected clients that subscribe to the topic.  But its not that useful.  For example say the last message published on a topic had the message 'turn water on'.  The new client would receive that message and would probably initiate a process to do just that and then disconnect and go back to sleep.  Ten minutes later the client wakes up again, connects and subscribes to the topic, and, if the last message published was still 'turn water on' it would again receive the same message.  This may not be what you desire.  So its probably better for your remote client to publish a message to your broker that says 'Im ready - what to do' and the command mqtt client program running on your main control computer, that is subscribed to messages sent from the remote client, has the appropriate logic to decide what command to then publish to the remote client.  So a long winded way of saying make the consumer request it.

Probably public mqtt brokers will not keep connections alive for long and will need the client to reconnect to publish or receive new messages. I've not used public brokers apart from some testing as for my purposes my local mqtt broker is what I need so I've not really looked into this.   

Also, to receive published messages you will need a client program that is connected to the broker and subscribed to the topic.   On a mobile phone this means a program will need to be up and running to receive such messages.  A few years back I followed along an online tutorial on getting a Swift program (the apple version of C) running on my iPhone.  It worked just fine for a week before refusing to work.  I then found that Apple would only allow my own program on my own iPhone to work for a week for 'testing purposes' and after that I would need a developers licence with an annual subscription.  (FU Apple)

A better method of communicating with your mobile phone may be email or SMS are these programs are always up on the the mobile and will alert you of new messages and these mail messages can be sent from your control program.   If your home email is running on a PC and is using outlook as its email client then you could also program the outlook with VBA to initiate something depending on the email subject or content.  Or a least one could, as I remember doing this round about 1999 - 2000 time for a project for a client.  I don't know how to do this with a Mac or Linux mail client but I guess it could also be possible.

To more fully understand the MQTT client and broker applications and the use of public MQTT brokers then I suggest a good read of the following links may prove productive

https://www.hivemq.com/mqtt-essentials/?utm_source=adwords&utm_campaign=&utm_term=mqtt&utm_medium=ppc&hsa_tgt=kwd-329131232284&hsa_cam=17295502954&hsa_src=g&hsa_net=adwords&hsa_kw=mqtt&hsa_ad=594094363642&hsa_grp=138486496202&hsa_ver=3&hsa_acc=3585854406&hsa_mt=p&gclid=CjwKCAjwkYGVBhArEiwA4sZLuCfp7glM_QMyyLzHGWw4-_3qvgYRvtbhs-8AwIxREPv4r9hiLFwWFhoCpRgQAvD_BwE

and

http://www.steves-internet-guide.com/mqtt/