Notifications
Clear all
Topic starter
2025-05-01 4:56 am
Hi y'all I've been working on a project for quite a while now and recently made a bit of progress thanks to the esp32 s3 sense video that Bill made and it gave me some new idea and to try some old ideas that I already had but I've recently hit another wall and I'm not really sure what the issue is. I have created a smaller portion of my code to provide here and is only a portion of the code but I believe this is where the issue lies as I still had an issue when trying to run it by itself. I am trying to send raw audio data to be played from a web browser and from what I understand it should be possible using the audio worklet. At one point I did get like a static noise in chrome (not using this snippet) but no sound in Firefox. At one point I thought maybe it was because it wasn't being done through https but I am not certain as I did see proof that that was the reason at one point but not during other tests and am wondering if it is something to do with the data being just raw audio. I was just wondering if maybe someone could test this code out for themselves and see what they think and to give me some helpful tips as to what I should do next. I am trying to stay away from any encoding to minimize the processing power on the board as much as possible to help enable some other features in my project. Below is the audio portion of my code,
#include <WiFi.h>
#include <ESP_I2S.h>
#include <WebServer.h>
#include <ArduinoJson.h>
I2SClass I2S;
WebServer Audioserver(82);
const char* ssid = "your wifi name";
const char* password = "your wifi password";
const int sampleRate = 16000;
const int bitsPerSample = 16;
const int numChannels = 1;
#define VOLUME_GAIN 3
#define CHUNK_SIZE 1024
void mic_i2s_init() {
Serial.begin(115200);
while (!Serial);
I2S.setPinsPdmRx(42, 41); // Adjust for your hardware
if (!I2S.begin(I2S_MODE_PDM_RX, sampleRate, I2S_DATA_BIT_WIDTH_16BIT, I2S_SLOT_MODE_MONO)) {
Serial.println("Failed to initialize I2S!");
while (1);
}
}
void handleAudioStream() {
WiFiClient Audioclient = Audioserver.client();
Audioclient.print("HTTP/1.1 200 OK\r\n");
Audioclient.print("Content-Type: application/octet-stream\r\n");
Audioclient.print("Access-Control-Allow-Origin: *\r\n");
Audioclient.print("\r\n");
Serial.println("Client connected for audio stream");
while (Audioclient.connected()) {
int sample = I2S.read();
if (sample && sample != -1 && sample != 1) {
long amplifiedSample = (long)sample * VOLUME_GAIN;
amplifiedSample = max(min(amplifiedSample, 32767L), -32768L);
int16_t sendSample = (int16_t)amplifiedSample;
Audioclient.write(reinterpret_cast<const uint8_t*>(&sendSample), sizeof(sendSample));
}
}
Serial.println("Client disconnected");
I2S.end();
}
void audio_http_stream() {
Audioserver.on("/audio", HTTP_GET, handleAudioStream);
Audioserver.begin();
}
void setup() {
Serial.begin(115200);
WiFi.begin(ssid, password);
WiFi.setSleep(false);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("\nWiFi connected");
Serial.println("Audio streaming on http://" + WiFi.localIP().toString() + ":82/audio");
Audioserver.on("/", HTTP_GET, handleAudioPage);
Audioserver.on("/pcm-processor.js", HTTP_GET, handlePCMProcessorJS);
mic_i2s_init();
audio_http_stream();
}
void handleAudioPage() {
String html = R"rawliteral(
<!DOCTYPE html>
<html>
<head>
<title>ESP32 Audio Stream</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>
<h1>ESP32 Audio Stream</h1>
<button onclick="startAudio()">Play Audio</button>
<button onclick="stopAudio()">Stop Audio</button>
<script>
let audioContext;
let audioWorkletNode;
let reader;
let isPlaying = false;
async function startAudio() {
if (!audioContext) {
audioContext = new (window.AudioContext || window.webkitAudioContext)({ sampleRate: 16000 });
await audioContext.audioWorklet.addModule('/pcm-processor.js');
audioWorkletNode = new AudioWorkletNode(audioContext, 'pcm-processor');
audioWorkletNode.connect(audioContext.destination);
}
const response = await fetch('/audio');
reader = response.body.getReader();
isPlaying = true;
while (isPlaying) {
const { done, value } = await reader.read();
if (done) break;
const int16 = new Int16Array(value.buffer);
const float32 = Float32Array.from(int16, s => s / 32768);
audioWorkletNode.port.postMessage(float32);
}
}
function stopAudio() {
isPlaying = false;
if (reader) reader.cancel();
if (audioWorkletNode) audioWorkletNode.disconnect();
}
</script>
</body>
</html>
)rawliteral";
Audioserver.send(200, "text/html", html);
}
void handlePCMProcessorJS() {
String js = R"rawliteral(
class PCMProcessor extends AudioWorkletProcessor {
constructor() {
super();
this.buffer = [];
this.port.onmessage = event => {
this.buffer.push(...event.data);
};
}
process(inputs, outputs) {
const output = outputs[0][0];
for (let i = 0; i < output.length; i++) {
output[i] = this.buffer.length ? this.buffer.shift() : 0;
}
return true;
}
}
registerProcessor('pcm-processor', PCMProcessor);
)rawliteral";
Audioserver.send(200, "application/javascript", js);
}
void loop() {
Audioserver.handleClient();
}
I'm not completely sure if there is an issue with it being not over https or if the issue is because of raw audio data or if there is another issue. However, if someone is able to point out the issue and give a pointer with a small (basic) code snippet on how to adjust my code I would really be thankful. Also I did try to make it work over https using the esp32_https_server library but I wasn't sure how to connect the audio end point for it. Lastly, sorry for the bad java-script I'm pretty new to coding a java-script more-so I wouldn't be surprised if the issue was in there.
Topic starter
2025-05-03 6:32 am
I meant to revert this before I posted this but if anyone tries this then you should adjust this line,
Serial.println("Audio streaming on http://" + WiFi.localIP().toString() + ":82/audio");
to this when trying to visit the page,
Serial.println("Audio streaming on http://" + WiFi.localIP().toString() + ":82/");
2025-05-04 1:01 am
Hello,
I am working on kind of the same thing. But like yourself, am also having to learn javascript to do what I want to do. In the end I want to be able to speak into a microphone connected to an esp32 and stream voice audio to the client (browser).
I am taking it piece by piece for now. I got the esp32 (server) sending audio data successfully to postman which I use for testing the server (esp32) . I am working on the client now but I am using websockets not http and using node.js on my pc as the server. Once i get all the parts working individually then i hope it will make it easier to get the whole thing working together.
I am also using web audio api but not web workers so my javascript does not look like yours.
I will take a look at your code and maybe try it out. I don't have an s3 but I dont think that matters.
last week was able to read stream a audio from a file to a browser and hear my voice perfectly so for me that was a big step. I also have been using AI for generating some of the javascript which helps. It may not work at first but kind of gets you going in the right direction.
Javascript is a crazy complicated language compared to C. Very big and lots of parts and some of the concepts are strange as heck.
Good luck, its a good project but a tough one to get everything working.
Cheers
Topic starter
2025-05-04 8:24 am
@noweare I did something pretty similar to what you are trying to do and I might be able to help you out with your project if you want some help with it. I was able to get that to work. The only reason I'm trying to do mine the way I am right now is simply because I believe it might save some memory and processing power. I tried to send a private message to let you know so I hope you see this and get back to me!
also check out this link I think this youtube video will likely help you out quite a bit, it helped me out a little with the code I was just referring too.
This post was modified 6 months ago by SkiddieNomad
2025-05-05 12:50 pm
That's a good video. He is a very smart guy and has done a lot of unique and interesting videos.
Thank you.