UVC #4

edbek · 2020-12-16T15:12:52Z

Is it possible with the help of this stack to implement interaction with Webcamera through UVC ? If so, how?

xiaocq2001 · 2020-12-17T02:52:17Z

Unfortunately there is no online document for now. The basic UVC work flow is as follow:

Initialize USBX with UVC

    /*========================================================================*/
    /*= Initialize USBX with UVC supported.  */
    /*========================================================================*/

    /* Initialize USBX system. */
    status = ux_system_initialize(memory_pointer, USBX_MEMORY_SIZE, usbx_cache_safe_memory, USBX_CACHE_SAFE_MEMORY_SIZE);
    if (status != UX_SUCCESS)
        error_handler();

    /* Initialize USBX Host Stack.  */
    status =  ux_host_stack_initialize(NULL);
    if (status != UX_SUCCESS)
        error_handler();

    /* Register video class.  */
    status =  ux_host_stack_class_register(_ux_system_host_class_video_name, _ux_host_class_video_entry);
    if (status != UX_SUCCESS)
        error_handler();

    /* Register EHCI HCD.  */
    status = ux_host_stack_hcd_register(_ux_system_host_hcd_ehci_name, _ux_hcd_ehci_initialize, EHCI_BASE, 0x0);
    if (status != UX_SUCCESS)
        error_handler();

Wait a UVC device connection

    /*========================================================================*/
    /*= Wait until UVC device is connected.  */
    /*========================================================================*/

    /* Find the main video container.  */
    status = ux_host_stack_class_get(_ux_system_host_class_video_name, &host_class);
    if (status != UX_SUCCESS)
        error_handler();

    /* We get the first instance of the video device.  */
    while (1)
    {
        status = ux_host_stack_class_instance_get(host_class, 0, (void **) &inst);
        if (status == UX_SUCCESS)
            break;

        tx_thread_sleep(10);
    }

    /* We still need to wait for the video status to be live */
    while (inst -> ux_host_class_video_state != UX_HOST_CLASS_INSTANCE_LIVE)
    {
        tx_thread_sleep(10);
    }
    video = inst;

Setup parameters

    /* Set video parameters to MJPEG, W x H resolution, .. fps. */
    status = ux_host_class_video_frame_parameters_set(video,
                                                      UX_HOST_CLASS_VIDEO_VS_FORMAT_MJPEG,
                                                      CAMERA_RESOLUTION_WIDTH,
                                                      CAMERA_RESOLUTION_HEIGHT,
                                                      TEN_FRAMES_PER_SECOND);
    if (status != UX_SUCCESS)
        error_handler();

Start streaming

    /*========================================================================*/
    /*= Start UVC streaming.  */
    /*========================================================================*/

    /* Start video transfer. */
    status = ux_host_class_video_start(video);
    if (status != UX_SUCCESS)
        error_handler();

#if HIGH_BANDWIDTH_EHCI /* Driver HCD must support adding requests list.  */
    /* Build buffer list.  */
    for (i = 0; i < VIDEO_BUFFER_NB; i ++)
        video_buffers[i] = video_buffer[i];

    /* Issue transfer request list to start streaming.  */
    status = ux_host_class_video_transfer_buffers_add(video, video_buffers, VIDEO_BUFFER_NB);
    if (status != UX_SUCCESS)
        error_handler();
#elif NORMAL_BANDWIDTH_OHCI /* Driver adds request one by one.  */
    /* Allocate space for video buffer. */
    for(buffer_index = 0; buffer_index < VIDEO_BUFFER_NB; buffer_index++)
    {
        /* Add buffer to the video device for video streaming data. */
        status = ux_host_class_video_transfer_buffer_add(video,
                                                         video_buffer[buffer_index]);
        if (status != UX_SUCCESS)
            error_handler();
    }
#endif

Handle frame data and reuse frame buffers, assuming there is transfer done callback putting semaphore:

    /* Set transfer callback (do before start transfer). */
    ux_host_class_video_transfer_callback_set(video,
                                              video_transfer_done);

    /* Wait transfer done and re-use buffers.  */
    buffer_index = 0;
    while (1)
    {

        /* Suspend here until a transfer callback is called. */
        status = tx_semaphore_get(&data_received_semaphore, TX_WAIT_FOREVER);
        if (status != UX_SUCCESS)
            error_handler();

        /* Received data. The callback function needs to obtain the actual
           number of bytes received, so the application routine can read the
           correct amount of data from the buffer. */

        /* Application can now consume video data while the video device stores
           the data into the other buffer. */

        /* Add the buffer back for video transfer. */
        status = ux_host_class_video_transfer_buffer_add(video,
                                                         video_buffer[buffer_index]);
        if (status != UX_SUCCESS)
            error_handler();

        /* Increment the buffer_index, and wrap to zero if it exceeds the
           maximum number of buffers. */
        buffer_index = (buffer_index + 1);
        if(buffer_index >= VIDEO_BUFFER_NB)
            buffer_index = 0;
    }

edbek · 2020-12-18T16:25:12Z

Thank you for the detailed answer !

yuxin-azrtos · 2021-01-28T22:35:20Z

Close this issue. Feel free to reopen if you have questions.

bSquaredOlea · 2021-12-13T16:52:30Z

Are there anymore complete examples for this yet? I used the above code to get to the point where I can set parameters, but my device doesn't stream.

xiaocq2001 · 2021-12-14T06:35:44Z

@bSquaredOlea There is no complete example project yet. For the stream, it actually depends on your hardware and host controller driver, since the ISO transfer is quite different to bulk and interrupt transfer. If your HCD is not ready for ISO transfer there is no stream.

xiaocq2001 · 2021-12-14T07:31:54Z

You can try the steps to build a video example that enumerates and start streaming on a USB 2.0 high speed webcam (tested with "Microsoft LifeCam Studio(TM)"):

Get MIMXRT1060 Examples

Download: https://github.com/azure-rtos/samples/releases/download/v6.1_rel/Azure_RTOS_6.1_MIMXRT1060_IAR_Samples_2021_11_03.zip
Extract
Confirm project sample_usbx_host_mass_storage works, we are modifying it

Modifications in `sample_usbx_host_mass_storage.c`

Add include file

#include "ux_host_class_video.h"

Add global variables

/* Define the number of buffers used in this demo. */
#define VIDEO_BUFFER_NB (UX_HOST_CLASS_VIDEO_TRANSFER_REQUEST_COUNT - 1)

UX_HOST_CLASS_VIDEO                 *video;

#pragma location="NonCacheable"
UCHAR                               video_buffer[UX_HOST_CLASS_VIDEO_TRANSFER_REQUEST_COUNT][3072];

/* This semaphore is used for the callback function to signal application thread
 that video data is received and can be processed. */
TX_SEMAPHORE data_received_semaphore;

Add instance check function (before demo_thread_entry)

static UINT  demo_class_video_check()
{

UINT status;
UX_HOST_CLASS               *host_class;
UX_HOST_CLASS_VIDEO         *inst;


    /* Find the main video container.  */
    status = ux_host_stack_class_get(_ux_system_host_class_video_name, &host_class);
    if (status != UX_SUCCESS)
        while(1); /* Error Halt  */

    /* We get the first instance of the video device.  */
    while (1)
    {
        status = ux_host_stack_class_instance_get(host_class, 0, (void **) &inst);
        if (status == UX_SUCCESS)
            break;

        tx_thread_sleep(10);
    }

    /* We still need to wait for the video status to be live */
    while (inst -> ux_host_class_video_state != UX_HOST_CLASS_INSTANCE_LIVE)
    {
        tx_thread_sleep(10);
    }

    video = inst;
    return(UX_SUCCESS);
}

Add video transfer done callback (before demo_thread_entry)

/* video data received callback function. */
static VOID video_transfer_done (UX_TRANSFER * transfer_request)
{

UINT status;

    status = tx_semaphore_put(&data_received_semaphore);
    if (status != UX_SUCCESS)
        while(1); /* Error Halt.  */
}

Add class registration (in demo_thread_entry)

    /* Register video class.  */
    status =  ux_host_stack_class_register(_ux_system_host_class_video_name, _ux_host_class_video_entry);
    if (status != UX_SUCCESS)
        return;

Replace the while loop code block in demo_thread_entry

UINT i;
UINT buffer_index;
UCHAR *video_buffers[VIDEO_BUFFER_NB];

    /* Assume video points to a valid video instance. */
    /* Create the semaphore for signaling video data received. */
    status = tx_semaphore_create(&data_received_semaphore, "payload semaphore", 0);
    if (status != UX_SUCCESS)
        while(1); /* Error Halt.  */

    /* Wait for camera to be plugged in.  */
    demo_class_video_check();

    /* Set transfer callback. */
    ux_host_class_video_transfer_callback_set(video,
                                              video_transfer_done);

    /* Set video parameters to MJPEG, 640x480 resolution, 30fps. */
    status = ux_host_class_video_frame_parameters_set(video,
                                                      UX_HOST_CLASS_VIDEO_VS_FORMAT_MJPEG, 176, 144, 333333);
    if (status != UX_SUCCESS)
        while(1); /* Error Halt.  */

    /* Start video transfer. */
    status = ux_host_class_video_start(video);
    if (status != UX_SUCCESS)
        while(1); /* Error Halt.  */

    /* Build buffer list.  */
    for (i = 0; i < VIDEO_BUFFER_NB; i ++)
        video_buffers[i] = video_buffer[i];

    /* Issue transfer request list to start streaming.  */
    ux_host_class_video_transfer_buffers_add(video, video_buffers, VIDEO_BUFFER_NB);

    buffer_index = 0;
    while (1)
    {

        /* Suspend here until a transfer callback is called. */
        status = tx_semaphore_get(&data_received_semaphore, TX_WAIT_FOREVER);
        if (status != UX_SUCCESS)
            while(1); /* Error Halt.  */

        /* Received data. The callback function needs to obtain the actual
           number of bytes received, so the application routine can read the
           correct amount of data from the buffer. */

        /* Application can now consume video data while the video device stores
           the data into the other buffer. */

        /* Add the buffer back for video transfer. */
        status = ux_host_class_video_transfer_buffer_add(video,
                                                         video_buffer[buffer_index]);
        if (status != UX_SUCCESS)
            while(1); /* Error Halt.  */

        /* Increment the buffer_index, and wrap to zero if it exceeds the
           maximum number of buffers. */
        buffer_index = (buffer_index + 1);
        if(buffer_index >= VIDEO_BUFFER_NB)
            buffer_index = 0;
    }

yuxin-azrtos · 2022-02-08T06:13:21Z

@bSquaredOlea : does the sample code help?

bSquaredOlea · 2022-02-08T15:07:23Z

Thanks for your responses. I was able to get to a point where I started a transaction, but never got any video data from the device (I could get meta data from the device though). However, we are moving in a different direction now. Thanks, Ben

…

On Tue, Feb 8, 2022 at 12:13 AM yuxin-azrtos ***@***.***> wrote: @bSquaredOlea <https://github.com/bSquaredOlea> : does the sample code help? — Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWZZBYBAHZJQA6BUBLRXRLLU2CYAZANCNFSM4U6EMFVA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

xiaocq2001 · 2022-02-09T02:29:09Z

@bSquaredOlea Thanks for sharing the progress.
For the transaction, please note UVC transaction is based on isochronous endpoint, which works different against the control requests and bulk transfer, so only HCD with isochronous transfer support can get video data from the device (which has been done in EHCI HCD for 1060). If you are working on some other chip, your HCD still needs modification on isochronous transfer to make things right.

xianghui-renesas · 2023-04-28T00:36:16Z

Hi, we can collect stream video through ethernet port to the PC. Is there a recommended PC side application that can collect from the Ethernet port and display the video? thanks!

xiaocq2001 · 2023-04-28T03:12:30Z

No for the raw video stream.

xianghui-renesas · 2023-04-28T03:21:26Z

Hi @xiaocq2001 , thanks for the comment. What about mpeg format? we can output mpeg format to PC.
Can some of the webcam application be used to display the video collected through USBX? eg. webcamiod. If not, what are some of the limiting factors? thanks!

xiaocq2001 · 2023-04-28T06:53:48Z

@xianghui-renesas, I'm not sure directly forwarded USB Video stream can be recognized by webcamiod, maybe you can try. I think the video stream must be rearrange/packaged by some web streaming protocol to allow PC application to play.

xianghui-renesas · 2023-05-05T18:16:37Z

Hi @xiaocq2001, thanks! I had a quick try with webcamiod and found they are primarily looking for the USB Video streaming device and is unaware of the host video packet format.
I have a specific question on the definition of TEN_FRAMES_PER_SECOND. How does it convert with fps? Appreciate any comment you can provide. Thanks!
status = ux_host_class_video_frame_parameters_set(video,
UX_HOST_CLASS_VIDEO_VS_FORMAT_MJPEG,
CAMERA_RESOLUTION_WIDTH,
CAMERA_RESOLUTION_HEIGHT,
TEN_FRAMES_PER_SECOND);

xiaocq2001 · 2023-05-08T01:37:59Z

Yes. That's the framerate.

xianghui-renesas · 2023-05-08T05:09:36Z

Hi @xiaocq2001, thanks! how is an input of 333333 converted 30fps, what is the unit of this argument in the API? thanks!
/* Set video parameters to MJPEG, 640x480 resolution, 30fps. */
status = ux_host_class_video_frame_parameters_set(video,
UX_HOST_CLASS_VIDEO_VS_FORMAT_MJPEG, 176, 144, 333333);
I can see 10000000/30=333333. It seems the unit of this argument is 10th of a microsecond. Could you explain?

xiaocq2001 · 2023-05-09T05:35:25Z

Please refer to [Universal Serial Bus Device Class Definition for Video Devices:
Frame Based Payload], where you can see frame intervals are in 100ns unit.

xianghui-renesas · 2023-05-15T17:50:59Z

Thanks @xiaocq2001 for the reference information. We are trying to stream the video to PC through UDP port. The PC end application we are trying to use is VLC:
https://docs.videolan.me/vlc-user/3.0/en/advanced/streaming/stream_over_udp.html
Do you have any experience using host video class and VLC?
One of the video format we identified using VLC is UX_HOST_CLASS_VIDEO_VS_FORMAT_H264, how do we set up the bandwidth?
Also a general question is how are we defining the color channel format? We do not seem to see there is information on the stack for this.

xiaocq2001 · 2023-05-16T00:36:55Z

Unfortunately, there is no H264 format demo for now, maybe you can trace some existing H264 camera for reference also there is H264 format spec on usb.org (Universal Serial Bus Device Class Definition for Video Devices: H.264 Payload).

General for USB bandwidth: done through changing different alternate settings.

xianghui-renesas · 2023-05-18T08:28:17Z

Thanks @xiaocq2001 , could you explain how the color channel encoding is defined in the USBX host video stack? If we collect image using uncompressed format, what is the format of data in the video buffer collected?
For example, with this configuration, how are the color coding and buffer data format defined?
ux_host_class_video_frame_parameters_set(video,
UX_HOST_CLASS_VIDEO_VS_FORMAT_UNCOMPRESSED,
160, 120,
333333);
The max payload for this setting is 384 (which is identified from (ux_host_class_video_max_payload_get), can you help to explain the format of this data so we can repackage to send to the PC program?

xiaocq2001 · 2023-05-18T09:39:50Z

Checking uncompressed format spec in https://usb.org/sites/default/files/USB_Video_Class_1_5.zip.

The supported pixel coding is as follow:

The format is reported by GUID:

The each payload is composed by a header and actual data, the header is like:

You can refer to the spec for more details.

BluesharkPD · 2023-05-18T11:11:43Z

Hi @xiaocq2001 , is there any application of uvc host running on stm32h7 target ??

xianghui-renesas · 2023-06-01T20:30:48Z

Hi @xiaocq2001 , does the usbx video host stack support still image collection? Do you have an example code if it is supported.

xiaocq2001 · 2023-06-02T01:51:01Z

@xianghui-renesas , still image collection is not supported currently.

xianghui-renesas · 2023-06-05T17:42:02Z

Hi @xiaocq2001, I tried to piece together the packets collected in the MCU to display them using a feature in our e2studio IDE and found the packets are out of order in the packet buffers. I have 96 packet buffers. If the MCU is not providing the buffer fast enough for the frame rate, will the MCU start to skip packet? Do you have experience with this? thanks!

xianghui-renesas · 2023-06-05T18:30:55Z

Hi @xiaocq2001, your example so far uses stream-based protocol, does the stack support frame-based protocol, do you have an example for the frame-based implementation? I think it may be easier to look at the raw image from the MCU buffer with frame-based implementation.

xiaocq2001 · 2023-06-06T03:26:08Z

@xianghui-renesas , do you mean to see how a video frame is detected in USB packets? For motion jpeg, if you check the spec about payload header for each USB packet, there is EOF bit to indicate a frame end.

MaheshAvula-Alifsemi · 2023-10-09T06:51:49Z

Hi @xiaocq2001 ,@yuxin-azrtos,
I am working on USB host Isochrnous support, I have a one question that single transaction per microframe, I am able to see the video streaming and it's working fine but, multitransactions per miocroframe is not working, and i am no seeing any valid frames,

I am using demo app as above one, I would like to know that, do i need to change application..? or could you please suggest how i can approach further on this.

I really appriciate your help on this.

Thanks
Mahesh

xiaocq2001 · 2023-10-11T08:51:51Z

@Mahesh-dev-avula I think the application is fine for multitransactions per microframe. Maybe you can check if multiple transactions per microframe is supported by your host controller, or the host controller driver needs modification to support it.

MaheshAvula-Alifsemi · 2023-10-11T09:28:34Z

@xiaocq2001 ,
Thank you for responding on my query,
Yes my host controller will support multi transactions per microframe,same hardware working on linux, my driver code also implemented based on linux reference(xhci driver).
the difference i found that linux and azure rtos, in the linux they are preparing multiple bufferes at time and sending command to the hardware,but in RTOS from the application we are requesting only one buffere at time and it's working for single transaction per microframe, I would thinking that may be we need to prepare multiple bufferes at time for multi transactions.
please correct me if I am wrong.

Thanks
Mahesh

xiaocq2001 · 2023-10-12T07:19:56Z

For isochronous request, it's supposed to support request list input, that is, the requests are linked by its ux_transfer_request_next_transfer_request, this means linked multiple requests can be accepted which includes multiple buffers.

In current EHCI implementation, single buffer is used for multiple transactions, max 3072 (3 * 1024) bytes can be transferred in the request buffer.

MaheshAvula-Alifsemi · 2023-10-12T07:41:43Z

@xiaocq2001 ,
Thank you for the responding my query,
Actually my hardware doesn't support EHCI , I have implemented xhci driver and trying on it.
there are few scenorios which i have tested:

When UVC device sends payload length as less than or equals 1024 bytes, in this scenorio, I am requesting payload buffer as 3072 bytes from my application and it's working.
When we increase resolution as 1280*720 with 30fps from my application and at this time UVC device commited payload length as more than 1024 bytes and it's three transactions per microframe, in this scenorio, first 1024 bytes only my host controller receiving and then it's throwing DATA BUFFER ERROR and RING OVERRUN ERROR.

Any suggestion on this to resolve the issue.

Thanks
Mahesh

xiaocq2001 · 2023-10-13T05:08:27Z

I'm not checking xHCI spec in detail, but I see following may relates to the high bandwidth multiple transactions support:

Mult, Max Packet Size and Max Burst Size in 6.2.3 Endpoint Context.
TRB Transfer Length in 6.4.1.3 Isoch TRB.

From description TRB Transfer Length can be 3072.
From description Max Burst Size shall be set to the number of additional transactions and Mult shall be 0 for high-speed.

MaheshAvula-Alifsemi · 2023-10-16T04:52:43Z

@xiaocq2001 ,
Thank you so much for your inputs and it's working fine after programming Endpoint Context array.

I really appriciating your help. thanks

Mahesh

chrisrhayden84 · 2024-05-07T02:21:13Z

Hi @xiaocq2001 is there any update on when/if still image collection & H264 payload will be supported?

xiaocq2001 · 2024-05-16T05:48:33Z

For still image you can extract YUV frame from isochronous video stream.
For H264 payload, you can handle that in application.

yuxin-azrtos closed this as completed Jan 28, 2021

yuxin-azrtos added the help wanted Extra attention is needed label Feb 8, 2021

yuxin-azrtos reopened this Dec 14, 2021

AYEDMSTM mentioned this issue Feb 10, 2023

Is there an example of a USB host connected to a USB camera? STMicroelectronics/x-cube-azrtos-h7#30

Open

UVC #4

UVC #4

Comments

edbek commented Dec 16, 2020

xiaocq2001 commented Dec 17, 2020

edbek commented Dec 18, 2020

yuxin-azrtos commented Jan 28, 2021

bSquaredOlea commented Dec 13, 2021

xiaocq2001 commented Dec 14, 2021

xiaocq2001 commented Dec 14, 2021

Get MIMXRT1060 Examples

Modifications in sample_usbx_host_mass_storage.c

yuxin-azrtos commented Feb 8, 2022

bSquaredOlea commented Feb 8, 2022 via email

xiaocq2001 commented Feb 9, 2022

xianghui-renesas commented Apr 28, 2023

xiaocq2001 commented Apr 28, 2023

xianghui-renesas commented Apr 28, 2023 • edited Loading

xiaocq2001 commented Apr 28, 2023 • edited Loading

xianghui-renesas commented May 5, 2023

xiaocq2001 commented May 8, 2023

xianghui-renesas commented May 8, 2023 • edited Loading

xiaocq2001 commented May 9, 2023 • edited Loading

xianghui-renesas commented May 15, 2023

xiaocq2001 commented May 16, 2023

xianghui-renesas commented May 18, 2023

xiaocq2001 commented May 18, 2023

BluesharkPD commented May 18, 2023

xianghui-renesas commented Jun 1, 2023

xiaocq2001 commented Jun 2, 2023

xianghui-renesas commented Jun 5, 2023 • edited Loading

xianghui-renesas commented Jun 5, 2023

xiaocq2001 commented Jun 6, 2023

MaheshAvula-Alifsemi commented Oct 9, 2023 • edited Loading

xiaocq2001 commented Oct 11, 2023

MaheshAvula-Alifsemi commented Oct 11, 2023

xiaocq2001 commented Oct 12, 2023

MaheshAvula-Alifsemi commented Oct 12, 2023

xiaocq2001 commented Oct 13, 2023

MaheshAvula-Alifsemi commented Oct 16, 2023

chrisrhayden84 commented May 7, 2024

xiaocq2001 commented May 16, 2024

Modifications in `sample_usbx_host_mass_storage.c`

xianghui-renesas commented Apr 28, 2023 •

edited

Loading

xiaocq2001 commented Apr 28, 2023 •

edited

Loading

xianghui-renesas commented May 8, 2023 •

edited

Loading

xiaocq2001 commented May 9, 2023 •

edited

Loading

xianghui-renesas commented Jun 5, 2023 •

edited

Loading

MaheshAvula-Alifsemi commented Oct 9, 2023 •

edited

Loading