MQTT Broker

Posted by: @zander

What I want to do is have my esp32 program send me status messages. Can you point me at an example and also the Mac version of a client

The esp32 can be programed with arduino c / c++ in which case you can use the Arduino mqtt pubsubclient library (as per my example in the first few posts of this topic).  The esp32 can also have the micropython or circuitpython installed instead and then one would use their respective micropython or circuitpython mqtt client libraries.

On my mac I also use python and python mqtt client libraries.  If you intend to program in C on your mac then I have not gone there so I dont know what mqtt client libraries would be available.

If you intend to use python on your mac does your new mac come with a version of python3  already installed?  If not  then you will need to download and install python3 which is now at version 3.11 I think.  Let me know if this is the route you intend to take and I can provide an example python mqtt client program.

You can also run the mqtt broker on you mac if you wish (though the broker on your rpi would suffice). And I would note that you can run mqtt client programs on the same computer that runs the broker.  To find instruction on how to install the mqtt broker on your mac a google of 'install mqtt broker on mac' will bring up a load of info including youtube videos.

@zander That's ok, I don't care for Scotch either, and as for lite beer.. as I stated before, I'm not a beer drinker, so our friends in Canada, UK, Australia , and other places around the world can have our share of the beers...lol

Although when the wife and I did a river cruise from BudaPest to Amsterdam a few years ago ( 2019 I think).. I did drink some very good beers & wine  in Austria and Germany.. they served it every day lunch and diner, as much as you wanted.(all part of the package) so we got a taste of the different regions as the boat made port almost every day for the 15 day cruise.

regards,

LouisR

@inst-tech

Posted by: @inst-tech

@zander I thought I'd made that very clear in my previous post.. I'm a bourbon man..I also enjoy Southern Comfort, and Apple cider beer on occasion.

regards,

LouisR 

You da man!

I'm also a bourbon man... Wild Turkey (fav), Jack Daniels(another fav, though not technically a bourbon), and a few others that a poor man like me can sometimes afford!

Cheers.

@byron

Whew! This is a juicy one! Dense!

Ok, I think I am getting dragged into this because of the wifi scan sketch. I asked myself how to send the scan data to receiving computer with out the serial monitor. It appears, from reading your code in the Wemos-MQTT-example, that I can do just that. Now, to work out the details.

1. I am trying to work out the differences involved in using MQTT with a WAN and a LAN. I assume the test site, and public broker only work with a WAN. In using the WAN, does the MQTT _SERVER variable in the example, get set to the AP? or the laptop acting as the broker?

2. Lets say I am trying to set this up using a LAN with no internet access. In this case, the broker has to be the laptop or RPI, and the broker is the server? hmm

@dualfuel

To send mqtt messages (publish and subscribe) between programs you need to route these messages via a mqtt broker.   Mqtt client programs send and receive messages via a broker.  Client programs publishing and receive mqtt messages can all be running on the same computer or different computers (more likely) and by computers I also include microprocessor boards such as the wemos esp8266 board. 

As the mqtt messages are sent over a network (local or internet) your computers or boards need to be connected to your home network to publish or receive messages to which they subscribe.   The broker can remote over the internet, or run on a computer attached to your home network.  A home network based broker is the recommended way to go and you can set one up on a linux computer, a raspberry pi (which of course runs a version of linux), a mac or a windows PC computer.  

The computer that runs your home network based mqtt broker does not have to be dedicated to this task.  For example a rpi could be running mqtt broker software as a service, at the same time as it runs a program that subscribes to a topic that collects data published by, say, a wemos esp8266 that sends (publishes)  temperature readings from an attached temperature sensor.

The mqtt broker could also be a public broker to which you can publish and subscribe to messages from your computers or microprocessor board linked to your home network (assuming an internet connection from you home network).   But you will be limited in the amount of data you can transmit and your data would not be secure (others can read it).  You can use secure public mqtt brokers but those will like to charge you for the privilege. 

I have an old rpi 3 which happily runs a mqtt broker so its not necessary to run the broker on a powerful computer.  I also run a mqtt broker on an old pc which now runs linux and is also ideal for the task.

But to have a play to start with, to publish a message from a program on your wemos and then to receive the message on another wemos (or whatever) that runs a program that subscribes to the same message topic that the publisher used, the initial use of a public free to use broker will get you started.

However, there are other ways to send and receive data between 2 boards and the use of ESP Now or perhaps the web server that @inq recently gave a link to might also be the way to go for your requirements.

Hi @inq,

 Re: writing bootable images on  SD cards from Windows. Sorry to read about your experience - hope you have recovered the situation. .. Apologies, it is a side issue to this topic.

---

For writing to USB stick memories in Windows (10), I have found 'Rufus' useful of a number of occasions - easy to use & so far it has always played nicely! (There isn't a Linux or Apple variant.)

https://rufus.ie/en/

I think that it includes SDcards, but it is a long time since I wrote to one, so I can't be certain I used it.

It's fairly small program ... For my occasional use needs, I keep a downloaded copy in its own directory, and copy the 'iso image' into the same area before booting ... then the 'browse' to Iso is easy ... and it has always found the USB stick automatically ... though of course, check carefully in case you have two plugged in at the same time!

Just a suggestion - best wishes, Dave

Posted by: @dualfuel

. I am trying to work out the differences involved in using MQTT with a WAN and a LAN. I assume the test site, and public broker only work with a WAN. In using the WAN, does the MQTT _SERVER variable in the example, get set to the AP?

I forgot to address that point in my long spiel 😀 - the MQTT_SERVER is the ip address of your mqtt broker wherever its is.  So if its on your local network it would have an address in the form of "10.0.1.141"  or "192.168.0.100"   If you were using a public server over the internet then it would be in the form of "test.mosquitto.org" and the internet DNS lookup service would translate that address into the appropriate ip numbers.

I did this part...

mosquitto

mosquitto_sub -t 'Important_Messages'

mosquitto_pub -t 'Important_Messages' -m 'This is an Important Service announcement, Mosquitto is working'

@dualfuel

Seems you are making good progress.  I understand you have loaded a mqtt broker onto one of your computers.  You need to find the network address of that computer if you dont already know it, perhaps by looking at your router administration or a network sniffer that shows everything connected to your network.  This address will be the one you define as the MQTT_SERVER

As you are not using an ESP8266 then you will need the appropriate wifi library for your ESP32.

I suggest you start really simple and on your ESP32 just connect to the wifi, connect to your mqtt broker and publish a message such as

mqttClient.publish("Test/Test1", "hello from ESP32" ); 

assuming you are using mqttClient as the name for your connection.  Then, on your computers subscribe terminal you should see the message appear.  Then move on to the blinky LED's 😎 

Edit: as long as your subscribe terminal is subscribing to "Test/Test1" or is subscribing to everything as a wildcard or you will see nowt.

@byron 

Ok, I am reading through the "Steves-internet-guide" explaination. I am picking up a few gems...like the server ip address IS the broker.

YES YES YES, I am in full retreat to the simplest possible sketch. In fact I have an ESP8622 around here, and am setting that up to learn with.

Just looking ahead, it appears the callback function is the very thing I want to end up with. EX: lets say we have the ESP32 scanning for wifi signals, and aiming its yagi antenna towards one in particular...then I publish the name of a different SSID, and the ESP32 reads the signal strength of that one and starts aiming (using PID) the yagi to maximise signal strength at the new SSID. Its doable, just overly ambitious.

alright! I got the mqtt_esp8266 example up and working with a D1 working as a generic esp8266. Its using a public broker in the example. I have the esp8266 connected to the WAN and sending output to the serial terminal (of the laptop). The public broker has tutorials too....so I might get up to speed sooner then I thought.

@byron I just discovered the arduino cloud is using mqtt, cuz when I tried to compile my first attempt it complained about not including the ArduinoMqttClient.h.

@dualfuel - I leant most of info about mqtt from Steves-internet-guide, I found them very good and easy to understand.   And that the thing about mqtt, once the relatively small learning curve is completed its vey nice and easy to use for a whole lot of messaging applications between different computers using the network.  If the computers are in close proximity you could use other ways of course, but its definitely a good tool to have in the box.  

I would be interested to see how you PID works out for tuning in your yagi and it will turn out to be an excellent example of PID use from the usual examples of making wheels turn at the same speed. 👍 

Posted by: @zander

I just discovered the arduino cloud is using mqtt,

Thats interesting and highlights that mqtt messaging is used for high volume applications as I expect there is a lot of traffic on the Arduino cloud.  Im not sure I like that Arduino cloud though as I have an Arduino RP2030 connect where the wifi firmware needed to be updated.  Arduino advised that the best way was to connect it to their Arduino cloud  and that would automatically update the firmware.  So I signed up for an account (just the free one) but I had a lot of trouble to get it to update my Arduino.  It eventually did complete the update after many attempts over several hours.

The data display facilities of the Arduino Cloud are rather limited and the only advantage over a home brewed display that you connect to via a local mqtt broker is the ability to use the Acloud from a mobile device from an App. (as least I think there is an App, but I've not tried it)   Maybe some hybrid approach could be useful where a pretty display is engineered locally but data is also sent to the Acloud just for liaising with a mobile phone.  I also note, that like most of the IOT cloud providers, the is a cost as soon as the data frequency and volume increases.    

@byron, et al...

A Little Cheese With My Whine

First, I want to explain my tardiness to the party (this thread).

1. Internet (via Verizon Wireless) is basically the only game in town here.  Mine's metered, and with tourist in the area it goes to dial-up speeds during the day... if I'm lucky.  The last five days just looking at a forum post takes five minutes or is totally denied as Verizon isn't working at all.
2. Getting Raspberry Pi up and running - Linux doesn't come to me naturally.  I have extensive cheat-sheets that still let me down regularly.  Because of (1) I thought I could take it to library/bar to install it headless.  They kind of frown if I bring in the 27" display and keyboard and plop it down on the high-top.  I had hoped to keep the RaspPi, a travel router and all the power supplies and wires inside my backpack.  That way, I could simply plug-in the tail (extension cord) coming from my pack and have the laptop on the bar-table with my beer.
3. @davee @zander - My earlier complaints about the RaspPi Imager trying to write an SD image turns out to be unfounded.  Fortunately, I did keep an image of the OS on my laptop.  Once I got home, but before trying other imagers, I simply plugged in a separate USB to SD writer into the laptop.  Flawless image.  Culprit - the built-in SD writer in this Dell Laptop, not the RaspPi Imager.  
4. So, as of now, I have a working RaspPi IV with 64 bit OS.  I must say, I'm impressed.  They've come a long way.  I've loaded the Arduino IDE on it and benchmarked it against my Windows i7 Desktop.  Compiling one of my projects too exactly the same amount of time on both - 30 seconds!  UI feels just as snappy.  

This Thread is Fantastic... 

Where can you learn about world-wide beer habits and Starlink communications along with MQTT.  Apologies for my contributions to the tangents.  I've gone back through and skimmed the whole thread for your (@byron) actual tutorial posts. And starting through them now.

I am now just starting (finally) the very first post and working through.  I was able to do the first steps...

Install MQTT Broker:

sudo apt install mosquitto mosquitto-clients

verify installation

sudo systemctl status mosquitto

... while I still had significant Internet last night.  The Internet is already starting to hang up as I write this.  More questions to come...

Thanks again for your effort.

VBR,

Inq