raspberry pi i2s-mems S32_LE still reported as S16_BE by vlc stream with repeating noise

[walterav1984]

Updated:2018-05-10

On a raspbery pi zero with "i2s mems" mono microphone on raspbian lite strech I'm able to record hearable sound using:
arecord -D plughw:0 -c1 -r 48000 -f S32_LE -t wav -V mono -v recordS32_LE.wav
Although a S16_LE flag is accepted it will not record any sound than, it will also not error so S16_LE is not an option... Further investigation shows that the i2s mems microphone doesn't produce a clean S32 signal but somehow padded 24bit to 32bit signal, this might explain the white noise sound when used v4l2rtspserver.

When trying to run "v4l2server" (compiled with alsa support) to stream video and audio I tried with the following command but gives interrupted/repeated noise/hiss every second without recognizable sound in vlc player:

./v4l2rtspserver -W 1280 -H 720 -F 25 /dev/video0,hw:0 -A 48000 -a S32_LE -p 8555

Omiting -C 1 for single channel gives error:

2018-03-09 14:30:35,298 [NOTICE] - /home/pi/v4l2rtspserver/src/main.cpp:581
	Create V4L2 Source.../dev/video0

[NOTICE] src/V4l2Device.cpp:117
	driver:bm2835 mmal capabilities:85200005 mandatory:4000001
[NOTICE] src/V4l2Device.cpp:120
	/dev/video0 support capture
[NOTICE] src/V4l2Device.cpp:122
	/dev/video0 support read/write
[NOTICE] src/V4l2Device.cpp:123
	/dev/video0 support streaming
[NOTICE] src/V4l2Device.cpp:147
	/dev/video0:H264 size:1280x720
[NOTICE] src/V4l2Device.cpp:229
	fps:1/25
[NOTICE] src/V4l2Device.cpp:230
	nbBuffer:1
[NOTICE] src/V4l2MmapDevice.cpp:71
2018-03-09 14:30:35,873 [NOTICE] - /home/pi/v4l2rtspserver/src/main.cpp:604
	Create Source .../dev/video0
2018-03-09 14:30:35,881 [NOTICE] - /home/pi/v4l2rtspserver/src/DeviceSource.cpp:93
	begin thread
2018-03-09 14:30:35,970 [NOTICE] - /home/pi/v4l2rtspserver/src/main.cpp:634
	Create ALSA Source...hw:0
2018-03-09 14:30:35,975 [NOTICE] - /home/pi/v4l2rtspserver/src/ALSACapture.cpp:49
	Open ALSA device: "hw:0"
2018-03-09 14:30:35,979 [NOTICE] - /home/pi/v4l2rtspserver/src/ALSACapture.cpp:117
	set sample format device: hw:0 to:10 ok
2018-03-09 14:30:35,982 [ERROR ] - /home/pi/v4l2rtspserver/src/ALSACapture.cpp:80
	cannot set channel count device: hw:0 error:Invalid argument
2018-03-09 14:30:35,984 [NOTICE] - /home/pi/v4l2rtspserver/src/ALSACapture.cpp:104
	ALSA device: "hw:0" buffer_size:0 period_size:0 rate:48000
2018-03-09 14:30:35,986 [NOTICE] - /home/pi/v4l2rtspserver/src/main.cpp:161
	Play this stream using the URL "rtsp:https://192.168.54.5:8554/unicast"

Changing S32_LE to S16_LE still shows S16_BE in vlc stream info but without hearable sound or continuous interrupted noise.

I read that earlier fixes enabled S16_LE to S16_BE conversion and later S32_LE to S32_BE conversion by your server and pull request #55 also does some audio fixes.

Could it be that S32_LE to BE is broken? Further testing with a music file based virtual alsa input device shows that S32_LE to S32_BE conversion is done right and is not broken, however mpromonet mentions it is the rtp payload that still shows S16 when S32 is used. Changes in commit 41589c8 didn't fix this for me yet.

Github user @lbdroid posted a usable workaround which even creates a single channel mono signal -C 1 when needed. YMMV depending on the amount of sound adapters I had the change "plughw:1,0" into "plughw:0,0" before it started working.

[mpromonet]

Hi,
PCM format transitting over network are always BE, the purpose of the code that was fixed in PR #55 is to convert LE capture for network transport.
The RTP payload is actually set to audio/L16, maybe it should be set to audio/L32 ?
Best Regards,
Michel.

[walterav1984]

Updated:2018-05-10

The RTP payload is actually set to audio/L16, maybe it should be set to audio/L32 ?

Could this be achieved by a command option/flag or does the code has to be recompiled? Sorry I didn't see the commit 41589c8 earlier when I wrote this, but specifying S32_LE or S32_BE with v4l2rtspserver doesn't produce hearable sound or noise at all, even VLC codec info doesn't show any audio stream info, not even wrong S16BE like before.

Furthermore I've also tested virtualmic input with alsa file input using a ffmpeg transcoded raw pcm file instead of the raspberry i2s microphone to rule out hardware problems. Tested multiple file formats as alsa sources both 16bitLE/BE and 32bitLE/BE I noticed that the 32bit examples streamed via v4l2rtspserver with the same interrupting/stuttering/gaps as the i2smems, however it sounded like recognizeable music/sound instead of white/buzz noise. Altering the the payload identification from 16>32bit sounds may indeed bring that fix. If specified correctly the 32LE and 32BE examples files sound the same, so I guess the LE>BE is done right, except for the stuttering that indeed maybe RTP payload identifaction related.

Since the S32_LE to S32_BE conversion doesn't seem the problem, the noise thing with i2s mems might be because of its fake 32bit (padded 16/24 bit) signal.

Are you interested in step by step reproducible alsa virtual input test/debug case, thats shows the default is 16bitLE tot BE conversion not BE as you suggest in readme.MD? It might also assist in debugging this 32bit stuff.
For my own reference and help to others these are the steps to create a virtual alsa source. The virtmic code was inspired from stackoverflow poster akostadinov.

Add the following code to your ~/.asoundrc that's in your /home/pi folder, /etc/asound.conf is not used in my example!

pcm.virtmica {
    type file
    format "raw"
    file "/dev/null"
    infile "/home/pi/music16le.raw"
    slave.pcm "default"
}
pcm.virtmicb {
    type file
    format "raw"
    file "/dev/null"
    infile "/home/pi/music16be.raw"
    slave.pcm "default"
}
pcm.virtmicc {
    type file
    format "raw"
    file "/dev/null"
    infile "/home/pi/music32le.raw"
    slave.pcm "default"
}
pcm.virtmicd {
    type file
    format "raw"
    file "/dev/null"
    infile "/home/pi/music32be.raw"
    slave.pcm "default"
}

Create some raw pcm music sources with ffmpeg, which I use a minute(s) long mp3 music file for (16bit 44100Hz stereo) in my example. Use filenames that resemble code above and use long enough samples as in minutes not seconds, since as soon as the v4l2rtspserver is started the music file starts playing and will not loop!

sudo apt-get install ffmpeg
ffmpeg -i music.mp3 -f s16le -acodec pcm_s16le /home/pi/music16le.raw
ffmpeg -i music.mp3 -f s16be -acodec pcm_s16be /home/pi/music16be.raw
ffmpeg -i music.mp3 -f s32le -acodec pcm_s32le /home/pi/music32le.raw
ffmpeg -i music.mp3 -f s32be -acodec pcm_s32be /home/pi/music32be.raw

Test v4l2rtspserver (before commit 41589c8) with the virtual alsa devices and sample files you just made:

./v4l2rtspserver -F 25 -W 1280 -H 720 -P 8555 /dev/video0,virtmica #you hear music
./v4l2rtspserver -F 25 -W 1280 -H 720 -P 8555 /dev/video0,virtmica -a S16_LE #you hear music
./v4l2rtspserver -F 25 -W 1280 -H 720 -P 8555 /dev/video0,virtmica -a S16_BE #noise

./v4l2rtspserver -F 25 -W 1280 -H 720 -P 8555 /dev/video0,virtmicb #noise, default is LE>BE!!!
./v4l2rtspserver -F 25 -W 1280 -H 720 -P 8555 /dev/video0,virtmicb -a S16_LE #noise
./v4l2rtspserver -F 25 -W 1280 -H 720 -P 8555 /dev/video0,virtmicb -a S16_BE #you hear music

./v4l2rtspserver -F 25 -W 1280 -H 720 -P 8555 /dev/video0,virtmicc #slow/offpitch music :-)
./v4l2rtspserver -F 25 -W 1280 -H 720 -P 8555 /dev/video0,virtmicc -a S32_LE #recognizable interupted music with gaps
./v4l2rtspserver -F 25 -W 1280 -H 720 -P 8555 /dev/video0,virtmicc -a S32_BE #noise with gaps

./v4l2rtspserver -F 25 -W 1280 -H 720 -P 8555 /dev/video0,virtmicd #noise
./v4l2rtspserver -F 25 -W 1280 -H 720 -P 8555 /dev/video0,virtmicd -a S32_LE #noise with gaps
./v4l2rtspserver -F 25 -W 1280 -H 720 -P 8555 /dev/video0,virtmicd -a S32_BE #recognizable interupted music with gaps

After commit 41589c8 there is no recognizable sound or audio stream info in VLC when using S32 except 32LE without no format specifying creates slow/offpitch music. Others 32bit options crash VLC!

[lbdroid]

Run "apt-get install ladspa-sdk", then create a /etc/asound.conf file containing the following;

pcm.pluglp {
    type ladspa
    slave.pcm "plughw:1,0"
    path "/usr/lib/ladspa"
    capture_plugins [
	{	
		label hpf
		id 1042
	}
        {
                label amp_mono
                id 1048
                input {
                    controls [ 30 ]
                }
        }
    ]
}

pcm.lp {
    type plug
    slave.pcm pluglp
}

Then you can happily run this and it will work nicely;

arecord -D lp -c1 -r 48000 -f S16_LE -t wav -V mono -v recording.wav
v4l2rtspserver -F 25 -W 1280 -H 720 -P 8555 /dev/video0,lp -A 48000 -a S16_LE

There are actually a couple of problems with that microphone (I got a couple of them in the mail last night, and have been fiddling with it all day to make it work nice, much to the annoyance of my wife...)

1. There is a DC offset that severely interferes with adding gain,
2. The rpi sound driver (and I'm talking about the main one, not the fake one used for the microphone) is buggy and doesn't read the S24_LE properly (as you have noted), which means that we are stuck with S32_LE, the consequence of which is that everything recorded is incredibly QUIET.

The above asound.conf solves both problems. The high pass filter (I didn't specify the cutoff, which defaults to 440 Hz) handles the DC offset, and the amp_mono adds a gain of 30 dB.

[walterav1984]

Thanks @lbdroid for your comprehensive input for both covering the volume-bug and streaming solution, I will test this solution asap.

About the highpass filter, I will be using it to record bird chirps/activity in house so I need to test if it needs adjustment, but I guess lowpass may also be an option now like humm/buzz generator/contruction works?.

Do you happen to have a compressed alsa/ffmpeg solution for audio, since v4l2 can also handle compressed/uncompressed formats maybe alsa can do something like 64Kb mono aac?

[lbdroid]

Low pass will not be an option. The high pass algorithm "sees" a DC offset as an oscillation of low frequency. Such a low frequency that one cycle is longer than the entire length of the audio. You probably could lower the frequency or play around with it, but honestly, I doubt you'll need to. I've never heard a bird that chirps under 440 Hz. I've been running mine in the house with kids running around making noise, and at 440 Hz, it seems to pick up everything. If you happen to have a subway running under your house, it will probably filter that out.

I have not looked into compressing the audio, but I am almost positive that that would have to be done within v4l2rtspstreamer rather than in alsa.

[walterav1984]

@lbdroid thanks I verified your workaround.

@mpromonet I added virtual alsa file source examples that show different behaviour depending on 16/32 le/be sources and specifying format or relying on default behaviour. Notice the slow offpitch music when S32LE gets default/auto interpreted as S16LE.

When using newer/latest commit 41589c8 the 32bit samples with specifying format just crash VLC, although notice that 32LE without format still produces slow/offpitch music.

[walterav1984]

@lbdroid
v4l2rtspserver -F 25 -W 1280 -H 720 -P 8555 /dev/video0,lp -A 48000 -a S16_LE

Do you also encounter a/v sync/drift issues? In less than half an hour its already more than a few seconds? I'm running on a Pi zero with v2 cam noir, using the v4l2 driver not the uvl4 prop thing.

I'm sure its not the client, since restarting vlc client on the same or other machine on the network has the same audio delay. Restarting the server does resync the audio, until it drifts again...

[lbdroid]

I haven't noticed any drift, but I'm not running exactly this setup, I have another couple of stages in between where I read the audio from 'lp' with a custom program that analyzes the audio for certain patterns in order to generate notifications, and then writes it out to aloop. I have v4l2rtspserver read the audio from the other side of aloop.

If your a/v is going out of sync, it could be that you are running too long of buffers.

[walterav1984]

Could you hint where to address these buffer settings as in v4l2/alsa/v4l2rtspserver?

I'm mostly running a headless default Raspbian Lite 4.14 image from 2018-04-18 on a rpi0 with 512MB RAM and 128MB for GPU with a v2 noIR camera with v4l2 driver.

However the rpi0 does has some other gpio pins active/solderd for sensors DS18b20/DHT11/TCRT5000 although they are not actively being polled by any software right now. The tcrt5000 works without overlay changes using tracking pin11. Maybe these sensors interfere with video&audio recording?

#/boot/config.txt
dtoverlay=w1-gpio,gpiopin=4
dtoverlay=dht11,gpiopin=22

In the following days I hope I have a more comprehensive instruction to replicate symptoms.

[lbdroid]

Buffers would be addressed in;

1. specific alsa driver,
2. asound.conf,
3. application.

[lbdroid]

Use of other pins should be no issue as long as you aren't trying to use any pin for multiple purposes simultaneously.