Planned USB host video implementation

[MarkWangChinese]

This is a meta issue for tracking the usb host video implementation.

Rough task items

• Describe the design of USB host video including:

  • host stack enhancement of class interface
  • host video class to implement the video interface (include\zephyr\drivers\video.h).
  • how to use it from application. I think video interface (include\zephyr\drivers\video.h) should be used from application,

• Enhance the class interface (usbh_class_api) between host core and host class. (WIP @MarkWangChinese @AidenHu)

  • usbh_class_api is an interface between host core and host class, it is draft defined, never implemented and never used now.
  • need to implement the class initialization during host initialization.
  • implement the notification of connection/disconnection status to host class.

• Implement the host video class diver and video interface.

  • implement UVC class driver. (WIP @MarkWangChinese @AidenHu)
  • provide video interface (include\zephyr\drivers\video.h) in UVC class driver. (WIP @josuah)
  • support both uncompressed and mjpeg formats.

• Implement Video capture sample.

  • Based on samples/drivers/video/capture sample, implement to connect with an USB device camera.
  • Display the video streaming from USB device camera to the displayer.

[MarkWangChinese]

Hi @josuah I see that you are working on the USB host video. As David's comment (#42066 (comment)), NXP also works on it for a customer requirement. I create this meta ticket to describe the task list from my understand. Could you help explain what parts you will work on? and what's your plan? Thanks.

[ngphibang]

Hi,

To support UVC host functionality in Zephyr, the main points I see are:

1. A generic UVC Class Driver

• This driver belongs to the video subsystem and is implemented on top of the USB host stack and video4zephyr (v4z) APIs.

• It should handle a wide variety of USB camera models (as in Linux, which supports ~250 models). Note that some real-world USB cameras may violate the UVC specification (which need to handled).

• It should support compressed (MJPEG) and uncompressed non-RGB (YUY2, NV12) formats as many USB cameras output these formats. To convert these to RGB:

Use software libraries (something like libjpeg, libyuv in Linux) or libmpix in Zephyr (https://github.com/libmpix/libmpix/)
Or leverage hardware accelerators if available.

• It should support various frame rates and resolutions. The v4z's frame interval API is already helpful for this.

• UVC controls (e.g., brightness, exposure, gain) should be mapped to v4z controls. The v4z's control framework is already helpful for this.

• It should handle metadata:
  For JPEG: EXIF metadata (if present).
  For IR cameras (e.g., FLIR Lepton): thermal metadata

2. Extension of the v4z subsystem if needed
   The v4z subsystem is already capable of supporting a UVC camera driver I think. It may need some extensions to handle metadata (?)

3. USB host Stack enhancements if needed
   Zephyr’s USB host stack may lack some full support for something we need for UVC host ?

[josuah]

Thank you for creating this issue and outline @MarkWangChinese and @ngphibang. This matches my impression and also list a few details I did not think of.

[josuah]

@MarkWangChinese
Could you help explain what parts you will work on? and what's your plan?

It seems like a proof of concept that can access the device descriptors would help: then by scanning the various descriptors, into a dev->data, a video device would immediately usable with the samples. This makes it possible to test webcams or UVB early-on. Then, USB API improvements can be inserted inside the PoC to insert dedicated API instead of anything hard-coded.

I agree that native video APIs would be great, like on Linux. The challenge becomes dynamically creating Zephyr devices at runtime, but it is possible to instantiate one particular UVC device on the devicetree and let the USB Host fill the configuration at runtime inside this existing device.

Support for several devices instances can eventually be handled via multiple struct video_interface for a single struct device, avoiding the need to dynamically create Zephyr devices even for multiple webcams plugged.

As far as I understand, to get UVC working, features would need to be available in this order:

1. Descriptor access (create and configure a device instance)
2. Control messages on endpoint 0 (stream and format negotiation)
3. Data transfer (once a stream is selected, request the buffers containing the data).

[jfischer-no]

• It should handle a wide variety of USB camera models (as in Linux, which supports ~250 models). Note that some real-world USB cameras may violate the UVC specification (which need to handled).

To stay on the ground and avoid later misunderstandings, the implementation should follow the standard and be at least at the level of UVC device support, also to enable testing in the loop. Everything else is optional and non-compliant behavior likely unacceptable.