How to program an alarm trigger from audio input?

@fransk

Not an area I have any direct experience in but perhaps you could utilize something like the librosa library python.

@fransk,

For the door buzzer my initial attempt would be to make a simple ESP32 volume detector that sends a message to the Home Assistant software. This can sit next to the buzzer speaker. You tweak the threshold to the buzzer volume and duration. (You're interested in the presence of the sound, not the tone.) Note that any audio solution will pick up ambient noise and may generate false signals. The duration would be the deciding factor. (IIRC, Bill did such a project but there should be one on the net.)

The water problems...I think a cost analysis would point to a repair rather than an automation solution. The cost of replacing the float in the toilet versus developing a sensor I think would highly favor the replacement. Not to mention the maintenance issues of a sensor in a wet environment.

The water tap...I'm unclear what the solution would do. I don't think the solution would need an audio component; it would simply perform the action every time the tap was turned off. Again, though, this seems like over-engineering. Get the tap adjusted is cheaper.

@robotbuilder Thanks, the librosa library looks quite capable, but I don't think it will run on micropython or ESP32, and it needs to run nearly real-time (delay till my trigger event less then 1 sec)

@tfmccarthy Thanks, you made me think about a digital soloution for the doorbell 👍

The signal of the mic attached to the intercom case is veeery much higher than any common audio picked up from the house. A diode and resistor (and 3V zener as input protection, perhaps a (darlington) transistor as pre-amp) makes it almost a square wave of which the leading edge can trigger an interrupt on a digital input. Then use time(rs) to detemine frequency and filter non-repeating signals like me touching the case. I recorded the bell sounds using a handheld audio recorder: one bell is a mix of 962 and 1260 Hz for 0.45 sec, the other 10 repetitions of 2 alternating tones: 974 and 1270 Hz lasting 0.3 resp. 0.08 sec.

The water related sensor issues are put in the freezer for now, or solved with a 10 cent rubber ring and a trip to the hardware store.

@fransk 

You have more information than I do, and I don't want to confuse the issue but...

I'm under the impression that you can't access the intercom unit, e.g.,

"I can't do any tinkering on my doorbell ... So the bell sound must be picked up by microphone from the ...  high buzzer in the intercom box"

So, I don't understand what you mean by,

"The signal of the mic attached to the intercom case..."

unless you mean the microphone that's embedded in the intercom unit, which is inaccessible. The ESP32 microphone would be a separate microphone from the intercom microphone.

If you can access the microphone in the intercom, then you should be able to tap the speaker as a switch.

"...much higher than any common audio picked up from the house."

How sure are you about that? TV, radio, mp3 player, even your audio recorder are potential sources.

I'm not an audio buff so I can't speak to this, but you've been able to record a sound and then separate it into specific frequencies.

How might you have done that?

Can you leverage that for this system?

My idea was more like a "Clapper" than a frequency analyzer.

@tfmccarthy 

Indeed my first thought was to connect the input of an optocoupler to the buzzer contacts, but I couldn't reach the buzzer by simply pulling off the intercom's cover.

To clarify things, here is a photo shows my intercom box and on top of it the handheld recorder for my first measurements.

The good old Audacity audio editor let show the recorded audio wave form of the recorded bell.

Here you see me twice touching the case sligthly with my hand, followed by the sound of the bell.

In the recording I can hear very faint music playing in my living room. In the wave form this shows just as noise.

Zooming in on the recorded audio, this is the start of a bell signal.
There is a very strong signal around the zero-axis. Some small signals are visible in the tops of the waveform. Beeing recorded with a audio tool (automatic volume control), you can see almost no clipping at the top of the waves.
For the ESP32 , this will be one-sided rectified with a single diode to make it a DC signal. If this is amplified to a top value of 3.3V, it will trigger the ESP32 input nicely, I expect.
You see a little shift in the signal, that I will explain below.

Using the spectrum tool (menu: analyse) on each distinct section of the bell sound showed me the main frequencies in the sound. 
When I move the mouse cursor in the spectrum, Audacity shows the frequency of a local peak indicated by a read vertical line, and values in the "Peak" textbox below the spectrum.
The main frequency of this bell is 962 Hz, equals B6 as musical note.
But there appears to be a second tone in this bell as the close second peak at 1260 Hz, a D-sharp6 note, indicates.

I plan to tie the mic (and a case for the ESP32) on the outside in direct contact with the plastic casing, using rubber band or double-sided tape. This way the sound is transferred from the casing to the mic directly, without first having to travel through the air. The mechanical contact will make that environmental sound reaching the mic by air will be very much lower in volume the the mechanical vibrations coming from the case.

Next step is to get the actual mic, bind it to the casing and verify the level of the signal it gives using an oscilloscope. Then I also know if, and what kind op amplification I need for the signal to trigger a digital input on the esp32. Normally you would use a schmitt trigger after some amplification of the mic signal, but I think that the esp32 input can take care of that.

The software will distinguish 2 different bell signals (my other bell button gives a different waveform and frequencies) using only timing or timers. That is much easier, using much less cpu power, than an audio library calculating frequencies at nearly real time (the trigger should happen within a second from somebody pressing the doorbell).

I hope this will clarify things for you. I will keep you posted on the little project.

@fransk 

Not exactly sure of your setup but maybe the use of sound module?

https://esp32io.com/tutorials/esp32-sound-sensor#google_vignette

I knew someone with hard of hearing where the lights would turn on/off for a short duration after the door bell was pressed.

Searching the internet should come up with practical ideas?

https://iotassistant.io/esp32/smart-door-bell-noise-meter-using-fft-esp32/

You might also get some ideas from ChatGPT.

I suspect there are over complicated solutions and simpler but still workable solutions.

@fransk 

Rather than have software or hardware recognize the door bell and filter noise another solution might simply be a wireless transmitter near the doorbell and a connection to a receiver you carry with you (like a mobile phone or hearing aids) and let your brain do all the "filtering"? You can be omnipresent at any location even not in the house at all. Such setups are common now. You can use your mobile phone to to talk to your dog left in the house while on holidays and work an auto feeder.

In one of my first homes we had two doors and each had doorbells which were also intercoms so we could talk to the person pressing the bell. These were always on and we could hear the kids outside as they ran around the house. 

@robotbuilder 

Interesting links  👍. No, in house I don't carry my smart phone with me mostly, people can leave a message if they need to talk to me.

I am looking for a cristal mic that I haven't found yet, to determine the signal level. I found an undocumented module that came with an ESP32 "smart" kit and book of 6 years ago, and I still have to find the book. But your first link uses a similar analog mic module, same pins; now I know what they mean! Good. But unfortunately the big potentiometer on the module makes it impossible to bring the mic into mechanical contact with the intercom case. Have to get out my soldering iron.
A digital MEMS mic is expected later this week; but I don't know yet how to employ it to detect the signal frequencies I need. The MEMS mic has quite a lot of control registers to play with. Otherwise the FFT approach used in your second link will be the solution.

The FFT approch is now stored together with the other project in my "project freezer" for future use.

Posted by: @fransk

↑

I read/saw Bill's highly interesting articles/videos about ESP32 audio output and mems microphone input using I2S. But what me intrigues is the a step in audio processing: how to trigger an alarm to my Home Assistant using audio input from a microphone at several places in my house?
With my growing age and less sensitive hearing I start to miss some small, annoying problems in my house that I used to hear immediately: a doorbell when the living room door is closed, a water tap that I forgot to close firmly, a leaking toilet cistern or water valve.

Question:
are there audio or filter libraries that can (help a program to) detect these situations so that a trigger can be sent to my Home Assistant?
Preferrably for micropython on ESP32 or Raspberry Pi Pico W.
Or should I better switch to tiny-AI techniques, and train my own AI filter models for my sounds?

Note: I can't do any tinkering on my doorbell as I'm living in a rental flat with an old (1980's) bell/intercom that is using some bus structure thru the whole building. So the bell sound must be picked up by microphone from the (loud and horribly sounding) high buzzer in the intercom box near my door (it has 2 different tones).

I would appreciate any hints that allow me to program such a audio trigger.
(I do know how to program messages to/from Home Asistant running on a rPi5 with M.2 SSD, it's the audio analysis that I'm not familiar with)

Posted by: @fransk

↑

I read/saw Bill's highly interesting articles/videos about ESP32 audio output and mems microphone input using I2S. But what me intrigues is the a step in audio processing: how to trigger an alarm to my Home Assistant using audio input from a microphone at several places in my house?
With my growing age and less sensitive hearing I start to miss some small, annoying problems in my house that I used to hear immediately: a doorbell when the living room door is closed, a water tap that I forgot to close firmly, a leaking toilet cistern or water valve.

Question:
are there audio or filter libraries that can (help a program to) detect these situations so that a trigger can be sent to my Home Assistant?
Preferrably for micropython on ESP32 or Raspberry Pi Pico W.
Or should I better switch to tiny-AI techniques, and train my own AI filter models for my sounds?

Note: I can't do any tinkering on my doorbell as I'm living in a rental flat with an old (1980's) bell/intercom that is using some bus structure thru the whole building. So the bell sound must be picked up by microphone from the (loud and horribly sounding) high buzzer in the intercom box near my door (it has 2 different tones).

I would appreciate any hints that allow me to program such a audio trigger.
(I do know how to program messages to/from Home Asistant running on a rPi5 with M.2 SSD, it's the audio analysis that I'm not familiar with)

I went for hearing aids.  Now I hear every sound that the house makes, every creaky board, every snap, crackle and pop from every joint in my body.  

could you put a coil around the bell feed?

Posted by: @tabcomary

↑

could you put a coil around the bell feed?

Unfortunately not, the speaker is on backside of the case and covered by a pcb completely. If I had access to the speaker, I could have used a suitable opto-coupler on the speaker terminals and get a digital input signal for a ESP32.

For a simple hard sound an adjustable sound detector module could do the job, as @robotbuilder pointed out: https://esp32io.com/tutorials/esp32-sound-sensor#google_vignette

However, I want to distinguish 2 different sounds (and ignore any other sounds).  
Now I wait for delivery of an adjustable amplified electret microphone module that can deliver 3.3v pulses - I hope - directly to a digital input of the ESP32. Then the software can detect the bell sounds using a digital time filter.

Posted by: @tabcomary

↑

could you put a coil around the bell feed?

In general, yes, but only around one of the wires, not both. And you need a lot of windings around that wire to get a usable voltage out of the coil (it's a 1:N transformer). Too complex for me.

This is a current transformer, measuring the magnetic field created by the current flowing through the speaker. Used to measure a current with complete electrical isolation from the other circuit, used for 'high-side' measurements and for mains power (120/230V and higher).
There are chips and modules for that, some using a Hall-effect sensor needing no actual transformer, most need to be soldered in the current circuit itself, some have a 2-part iron core that can be clipped over one of the wires. All will need amplifiers to interface to a microprocessor.

Posted by: @fransk

↑

I read/saw Bill's highly interesting articles/videos about ESP32 audio output and mems microphone input using I2S. But what me intrigues is the a step in audio processing: how to trigger an alarm to my Home Assistant using audio input from a microphone at several places in my house?
With my growing age and less sensitive hearing I start to miss some small, annoying problems in my house that I used to hear immediately: a doorbell when the living room door is closed, a water tap that I forgot to close firmly, a leaking toilet cistern or water valve.

Question:
are there audio or filter libraries that can (help a program to) detect these situations so that a trigger can be sent to my Home Assistant?
Preferrably for micropython on ESP32 or Raspberry Pi Pico W.
Or should I better switch to tiny-AI techniques, and train my own AI filter models for my sounds?

Note: I can't do any tinkering on my doorbell as I'm living in a rental flat with an old (1980's) bell/intercom that is using some bus structure thru the whole building. So the bell sound must be picked up by microphone from the (loud and horribly sounding) high buzzer in the intercom box near my door (it has 2 different tones).

I would appreciate any hints that allow me to program such a audio trigger.
(I do know how to program messages to/from Home Asistant running on a rPi5 with M.2 SSD, it's the audio analysis that I'm not familiar with)

You can detect household sounds without heavy AI models. Simple DSP works very well. Use an ESP32 or Pico W with an I2S MEMS microphone. Capture short audio frames and run a fast FFT using ulab. Look for energy peaks at known buzzer tones. Doorbell signals have stable and narrow frequencies. Water leaks show steady broadband noise. Dripping taps create short sharp spikes. Each pattern is easy to track.

You only need three steps. First read audio from the microphone. Then run an FFT. Finally check the frequency bands that match your target sound. Once detected, send an MQTT message to Home Assistant.

Many builders use similar methods. A great example is the AquaPing acoustic leak detector, which uses simple spectral tricks. You can find it here: https://www.pcbway.com/blog/Engineering_Technical/AquaPing_Open_Source_Battery_Powered_Stand_off_Acoustic_Water_Leak_Detection_4f240e75.html .

It shows how lightweight audio analysis can catch quiet water leaks.

Start with one sound. Tune thresholds slowly. Test in your room. Then expand to other sounds. You can always add small ML models later. But DSP usually works well for these tasks.