InqBee - LoRa, Battery Powered, Remote Sensor Station

After some minor changes to Bill's Sender and Receiver Sketches...

1. Using standard ESP8266 Serial usage.
2. Using standard ESP8266 pin mappings for SPI

...both failed to boot.

I'm used to this in my wire-it and try-it mentality.  At least I didn't have a RUD... or smoke.  I'm sure I could have determined ahead of time if I had only read the RFM95W documentation.  Here are some things I've discovered.

• I can now confirm now that the Adafruit breakout board pin G0 does go to the RFM95 pin labeled DIO0.  
• I can also confirm this pin is pulled LOW on the RFM95.  I had it going to the D3 (GPIO0) pin of the WeMos ESP8266.  The ESP8266 will not boot if the D3 is pulled low.  I've switched it to the D1 pin.  The wiring diagram has been updated and is below.

After that fix, they both booted up and start talking to each other.

Wild, Wild West

From looking at the library, there is no hint of security or any type of TCP/IP like ACK/NAK behavior.  Also, there is no connection hand-shaking like TCP/IP or a router connection does.  This indicates to me that its more like a UDP/IP Broadcast... in that its a fire and forget communications to the world.  Anyone within range can (and will) receive it.  For instance my WiFi can't even reach my closest neighbor... but InqBee might reach the next county.  🤔  Not that I think InqBee data is Top-Secret, but it might give someone else pause about using a clear language protocol.  

This post was triggered while I sit here programming the next modification to InqBee.  Both have been running with the Serial Monitor parked in the corner of my screen.  This showed up, confirming my suspicions...

Received packet 'Packet 31' with RSSI -6
Received packet 'Packet 32' with RSSI -7
Received packet 'Packet 33' with RSSI -6
Received packet 'Packet 34' with RSSI -7
Received packet 'Packet 35' with RSSI -7
Received packet '⸮⸮'&4⸮$⸮#!⸮⸮⸮⸮⸮$}⸮6⸮⸮⸮UH,:)6⸮⸮ҙ⸮⸮⸮⸮⸮⸮⸮⸮~⸮᷎⸮⸮
⸮⸮⸮s⸮BЈ⸮⸮l1-⸮' with RSSI -110
Received packet 'Packet 36' with RSSI -7
Received packet 'Packet 37' with RSSI -6
Received packet 'Packet 38' with RSSI -7
Received packet 'Packet 54' with RSSI -7
Received packet 'Packet 55' with RSSI -7
Received packet '⸮⸮w⸮⸮\⸮⸮!⸮⸮⸮⸮ďW⸮⸮4**0⸮⸮⸮w⸮qKM5#⸮&L<⸮%
⸮o/⸮⸮I⸮w⸮⸮⸮xLW⸮t|[⸮GQ⸮qlc{⸮=⸮}⸮N⸮⸮⸮⸮A⸮⸮r⸮L3⸮}⸮⸮M⸮⸮k搄9⸮-ј᭓⸮c⸮
m ⸮⸮43⸮r⸮⸮⸮⸮D⸮!Q+⸮⸮Y⸮W⸮X$⸮⸮⸮⸮`g⸮G|ȣ⸮⸮$⸮k⸮a8⸮A⸮C⸮⸮Xmwo⸮
⸮U⸮⸮⸮Z⸮v##⸮⸮⸮⸮⸮⸮l⸮4⸮⸮0⸮p⸮⸮⸮⸮⸮ with RSSI -111
Received packet 'Packet 60' with RSSI -8
Received packet 'Packet 61' with RSSI -6
Received packet 'Packet 62' with RSSI -8
Received packet 'Packet 63' with RSSI -7
Received packet 'Packet 64' with RSSI -7
Received packet 'Packet 65' with RSSI -6
Received packet 'Packet 66' with RSSI -7

 I'm wondering if I'm flooding their receiver and causing havoc with their program. 😘 Anyway, this leads to having to analyze the packets:

1. to see if they're true InqBee clients and reject alien packets.
2. having some kind of client ID for multiple bee hives.
3. having some kind packet ID to confirm sequence and possibly have the sender store packets and re-send ones the receiver didn't get. 
4. Still need to determine if reception of two different packets at the same time causes a conflict.
   1. does one get lost?
   2. is the second one buffered and passed on after the first?
   3. is the data intermingled and thus useless?

InqBee - Version 0.8.0

This version of InqBee (0.8.0) uses two ESP8266 WeMos R1, D2 Minis.  Both wired the same.  One is the client, and only sends its supply voltage.  Once on a battery, this will give an indication when it is time to change the battery.  The second Wemos gets the server sketch, receives the messages and exposes a web server.  I had concerns that the there might be interrupt conflicts between the LoRa and WiFi going on, on the server, but that appears to be OK. 

The server will be converted to a web server running on a Raspberry Pi Zero-W as I need far more data in a database and the multi-tasking ability to run reports against a large database.  However, at this point of testing, the WeMos will suffice and any webpages that I make will simply move over to the RasPi.  With either server, it will allow me monitor, alert, and print history reports.

The server also counts any packets coming in that it doesn't recognized and also counts if it missed any messages coming from the client.

 Here is the InqBee dashboard giving current status

Here is the Histogram of the Client's Voltage and RSSI strength

Here is the source code.

LoRa Range in the Mountains

Walked around the property to see if I'd have any range problems using my test rigs.  Right now, they're only using the 8.19cm long single strand copper wire.  

The client sender LoRa remained on my desk at the back of the house, not near any windows. The house sits on the top of the property, has a small flat area and then drops off about 40', then comes close to leveling out, but still drops away.  The property is between two ridges and runs about 200 yards.  It is called a Holler in this neck of the woods.  Great for peace and quite, bad for any kind of radio.  The road running down the Holler isn't even strait and as about 300 yards, makes a bend and more ridge blocks heading further down toward main roads.

Crude Side View

The important points:

1. One antenna is inside, with no attempt to expose it more to the outside.
2. Both antennas were completely vertical.  No attempt was made to angle them so that the radiation plane would aim toward each other. 
3. The drop-off essentially fully blocks the top half of the property from direct line-of-sight.  

• At 200 yards, the RSSI was -110 dB and no packets were missed.
• At 500 yards and behind the ridge (I'm guessing the waves can bounce around corners) the RSSI was at -118 dB and still not losing packets.
• At 800 yards, the RSSI was at -120 dB and I notices some packets starting to get lost.
• At 1000 yards, the RSSI was at -123 dB and most packets were getting lost.

I drove around the area within a couple of miles hoping that some fluke of bouncing around and over ridges might find me.  It was in vain. 

It will be quite servicable for my project in mind, but I can't imagine enough improvement to reach the farm about 8 miles away.  

I've continued the experiments with LoRa.  Here are some things I'm running into and learning about the technology.

LoRa

• These use the SX1278 LoRa chip which is the defining aspect of LoRa.  The RFM95W is an implementation, break-out board if you will.  There are other boards using the same SX1278 chip. 
• I ran into a bug in the library that @dronebot-workshop's is recommending in his YT tutorial (LoRa by Sandeep Mistry).  As a result, I was considering opening a bug report on the Github page for the library.  I find it has 135 open issues with 388 closed.  I started searches and reading on the bug-base to see if the bug I found had been reported.  Reading through some entries, I find many of the so-called bugs were simply people not understanding what LoRa is and is not.  As is typical, people seem to think  they paid the open-source developer and have the right to be rude and expect a perfect product and perfect support.  One got all belligerent about how the library was not receiving packets from "some" senders.  It didn't seem to cross his mind that his setup was problem and not the library.  Anyway, the developer tried to explain, but in the end, I think his frustration has reached an end and he doesn't plan on releasing any more fixes.  He did mention that others might be taking branches and doing upgrades.  I did not find or submit the bug I found.  Shame on people that bitch and moan to someone trying to offer something that took a lot of time and effort and knowledge FOR FREE.  If you didn't pay for, don't act like an ass!  Ask nicely, give full facts on how to reproduce it and you're likely to be the first one fixed.  
• As a result, here are some important aspects to consider with LoRa
  • It is not secure.  If you want security, you must do the encryption / decryption.
  • It is not connection based.  It is not a private communication between your sender and your receiver.
  • It is simply a broadcast to the world.  If you sent a string with a readable password in it, everyone a receiver within range got it and can read it!
  • It is not WiFi with only 100 meters range or BT with 10 meters range.  It has kilometers range and may be hitting receivers all over creation.  
  • If there are many senders in your area (yours and/or others) broadcasting, your receivers will be getting all of them and if two packets arrive at the same time, you will miss at least one of the packets. 
  • If your message must get there, it is up to you to send confirmations back to your senders that it was received.  If your sender doesn't get a acknowledgement (ACK), it's up to you to implement a re-send ability.  
  • If you have lots of senders, you should also implement a staggered sending schedule.  IOW... don't have all 100 senders wake up precisely on the hour and send their data.  Staggering them by even some small time will allow your receiver to receive all of them.  If you're in a location with lots of other sender banging away at the airwaves, that you have no control... well, you're kind of screwed.
  • On regular wired and WiFi networks, ACK/NAK and staggered send timing is what the TCP/IP protocol does for you.  On LoRa, you must implement both in your own protocol.

LoRaWan

I've tried to find information on the differences between LoRa and LoRaWan.  Presumably, LoRaWan is to LoRa what TCP/IP is the MAC layer on WiFi networks.  The material seems to be a little thin on the subject.  I have found that it at least incorporates some encryption.  This appears to be the standard's owner:

https://lora-developers.semtech.com/documentation/tech-papers-and-guides/lora-and-lorawan/

Here are some excerpts about LoRaWan.  

• LoRa is purely a physical (PHY), or “bits” layer implementation, ...
• There is no one-to-one relationship between LoRa-based devices and gateways in a LoRaWAN network; messages sent to and from end devices travel through all gateways within range. Deduplication is handled by the network server.
• going through this code sequence multiplication buys you a higher RF link budget, so you can transmit over a longer range.
• However, two packets with the same spreading factor arriving at the same time on the same channel might result in a collision. However if one of the two packets is stronger by six dB, it will survive.

I will admit, I have not thoroughly studied this document yet as it appears much of the long range ability requires Gateways.  I'm not likely to have these in my area for years, maybe decades.  Also, doing a search on the document for "ACK", "NAK" and "resend", I also didn't find any hits.  They might have used different terminology, but they'd be remiss to not include those keywords if it was supposed to handle some of the most fundamental problems of communications that TCP does for us.  IOW, I feel if I have to implement these types of abilities anyway, and for my sum kilometer range requirements, I'd rather not have the overhead of the LoRaWan aspect.  Maybe further research will change that opinion. 

For your case "You mileage may vary" (YMMV) is applicable.  😉 

VBR,

Inq

I may make many sensor modules that might be all over the property reporting on bees, soil moisture, streams water flow, ponds depth and temperature, weather... all kinds of things.  I have lot's of trees and live between two ridges and want to know when and for how long the sun hits a spot throughout the year.  Might even put one in the mailbox at 300 yards away to let me know when the mail has arrived.

As such, I want to tackle some of the issues LoRa doesn't. 

• Although right now, I'm testing with two WeMos ESP8266, the sensor modules will likely be ESP8266-07 because I can use better voltage regulators and have no LED's using power for Deep Sleep efficiency.  We did a thread somewhere around here on that.  Thanks guys for helping out.
• The server will eventually be on Raspberry Pi Zero-W.  If it can't handle it, I can always upgrade to any number of newer/better ones I have.  But, those only sip the electricity - less than $1 per year where I live.  It'll keep the database of all the disparate sensor nodes data and allow me to track it over the years.  It'll also host a webserver to expose that data to any device in the house.
• The sensor nodes being on batteries, I want to minimize their up time, but I want to make sure the data gets to the server.  This is the plan at the moment.
  • In Arduino setup() method - All this is done in setup.
    • If powering on for the first time or from a battery replacement, it'll load (1) the node's ID and (2) a default sleep interval from a configuration file on a LittleFS partition.  It'll also start (3) the Sent Sequence Number to 0.  This is power hungry so, we'll shove those three pieces of information into RTC memory.  This is memory that is usable while in Deep Sleep.  All normal ram is flushed during deep sleep.  Unfortunately, RTC memory is not persisted during a power off, so both cases have to be handled.
    • If waking from Deep Sleep, it'll fetch the ID/Default Sleep Duration/SeqNo from RTC...  Saving power. 😊 
    • Start-up and gather all sensor data.
    • CRC - TBD - I need to research if this is needed.  There was some CRC functions in the library, but it has been deprecated.  Maybe, it is not needed... maybe they let the user decide to use their own.  It may need to CRC the data and send that so the server can confirm the data hasn't been corrupted.
    • Turn on LoRa module.  It will be in sleep mode - (1 μA)
    • Send the sensor data - (120 mA)
    • Switch it over to receive mode - (12 mA)
    • Increment the SeqNo
    • Update the RTC memory.
    • Right now with both units on my desk this all takes 53 milliseconds.  However, I only have the supply voltage number being gathered and sent out.
  • In Arduino loop() method
    • Wait for a confirmation and/or some other command.  Right now, I'm seeing it take another 38 milliseconds get a reply if the server is not overloaded.  I'll need to see how (1) range, (2) many sensors hitting the server and (3) contention might affect this reply time.  I expect I can keep the wait time to well under 1 second. 
    • If it gets a confirmation, it'll go to sleep.
    • If the time expires, it'll put the sensor data packet in RTC memory and then go to sleep.  The assumption being the server is powered down due to power outage (common here) or I'm upgrading it.  When it comes up, it can retrieve this missed data. 
• The server will be up all the time and powered by a USB AC adapter.  It will:
  • Continuously wait for incoming sensor node data.
  • A sensor node's data coming will have several options for replies.
    • It can reply immediately that it has been received.  It returns the ID of the sensor node of the received message.  Remember - all nodes hear all messages.  The node will have to sift through all replies to see if its data has been received.
    • If the server wants to adjust the sleep schedule, it can return a new sleep duration.  This is for:
      • The sensor nodes will not have time-clocks on them.  It wastes power.
      • It is reported that the Sleep duration is not consistent and accurate.  It differs mainly because of temperature.  Being outside, it'll range from below -10 to over 100 °F.
      • The server will be responsible for dynamically changing the sleep time to reduce the chance of the nodes checking in at the same time.
      • For manual control, for things like wanting to see second-by-second updates for the next ten minutes when the normal interval might be 15 minutes.
    • The node will update its sleep duration and go to sleep.
    • The server will recognize it missed some SeqNo's and request they be sent.  The node will be expected to send those and then go back to sleep.  
  • The server will then process the data into the database... typically a time consuming process that we don't want to hold up the node from sleeping.
  • Other processes on the RaspPi will handle the Web Server pumping out real-time data coming in to waiting browser GUI's and generating historical reports.

With this design, the nodes and server will handle most of the issues that TCP/IP communications typically handle for us.  I'll try to write it in a modular, documented, generic way in case someone might be interested in using parts or all of it.

VBR,

Inq

@inq Sounds like you have a good start on the overall system design. One thing that I think I heard is that power outages are somewhat common. Be aware there are several UPS options available for a Pi, but they are not cheap. I think there are also Aliexpress UPS solutions but I can not vouch for them. I think I have a few but have yet to test them. Here is a link to one UPS for any Pi.

https://www.pishop.ca/product/piz-uptime-plus/

@inq Here is the link to the bigger brother of the UPS I posted earlier. It is more oriented to the Pi 4 or 5 but will work with any 40 pin model. https://www.pishop.ca/product/pi-uptime-ups-board-for-raspberry-pi/

Posted by: @zander

↑

@inq Here is the link to the bigger brother of the UPS I posted earlier. It is more oriented to the Pi 4 or 5 but will work with any 40 pin model. https://www.pishop.ca/product/pi-uptime-ups-board-for-raspberry-pi/

Thanks Ron.  If and when you get a chance to test them, let me know your thoughts.  I'm not so worried about the power outages and missing sensor data.  I hope that the design above will take care of it. 

But at one time I did a media server on a RasPi and the power outage trashed the SD card.  VERY FRUSTRATING... building a media server takes a lot of time and D.A. me didn't make a backup image! 😡  

Having a tiny UPS like you suggest that has the ability to keep it running for a while is great.  I'd also want one that has the ability when it nears running out of juice to gracefully shutdown the Raspberry Pi so it'll boot back up when power comes on. 👍 

VBR,

Inq

@inq That is what a UPS does. SD card? I left those behind several years ago, The Pi 'BIOS' was upgraded to boot from a USB drive so I have been using nothing but SSD's for quite a while.

@inq That is what a UPS does. SD card? I left those behind several years ago, The Pi 'BIOS' was upgraded to boot from a USB drive so I have been using nothing but SSD's for quite a while.

Posted by: @zander

↑

@inq That is what a UPS does.

OK... seemed like I looked at some dedicated RasPi ones before and they didn't mention being able to trigger the OS to shutdown.  Must have been the real cheap ones! 😆

@inq I think a shutdown is accomplished via an analogue pin monitoring the UPS battery voltage. A background task initiates a soft shutdown when it drops below a certain point. The power back on is done when grid power resumes or when the UPS battery is sufficiently recharged. The UPS board then activates the Pi power on pin. 

The cheaper units are not a true UPS as described above, they are just battery backup and when the battery runs out the Pi suffers a hard shutdown. As you know, if an SD card is still the boot device, a hard shutdown can damage the SD card as it is also being used for swap space.

I've been running a test overnight with the sender and receiver right next to each other... 4 inches.  It is not reliable in the sense like TCP/IP where you know it was successfully delivered.  Here... you have no idea if the receiver is running, if it got it, if it came through un-corrupted.  Once I get a battery operated sensor node, I'll need to do the same test with some distance.  Either way, this confirms my suspicions that some kind of robust protocol is necessary to up the reliability of the communications.

The percentage is tiny (0.2%)  This is at 4" range and was only run with 6 byte packets at 1 sample per second.  It can only get worse with distance, many senders vying for attention and larger packets.  Either way, it is still significant if you want all your data.

BUT... near 1000 meter range with just a tiny wire antenna is compelling!

I also found the Invalid (un-recognized packets) interesting.  I didn't think there was anyone computer literate, much less LoRa literate around here.  I might have to see if I can triangulate and meet a fellow enthusiast.  🤣  Then again, it might simply be some fluke in these transceivers.  

@inq I'm not sure if this will be helpful, but who knows?

Back in the days when I was paid to code, my system had to process ALL the financial markets communications lines in the Americas. We were using 68010 8mHz processors in a 6-pair and spare configuration, BUT I think the communications boards had their own processors to handle the lines. These lines were ONE WAY. The communications lines were mostly BiSynch but some SDLC and even one old 6-bit Baudot.

My gut feeling is you can get some number of sensor nodes to work with a central server, but at some point, the data loss may become intolerable. You could introduce concentrator nodes, for example, a Pi Zero for every six sensor nodes then using compression and scheduled transmissions, have the concentrators send to the central server. I would first try to find a low-power, very low-cost board that can receive and buffer the LORA data. You may only need to buffer a small number of LORA messages before passing them to the main board (at high speed or some direct memory access through shared memory)

Sorry, these are not well-thought-out points, but maybe you can find something to inspire you. I agree that you need to create a protocol and some way to ensure two sensors do NOT try to transmit simultaneously.

I would look at the Swiss Guy's videos to see if he has any projects that might inspire.