This is the sample code for Leopard USB3.0 camera linux camera tool. It is a simple interface for capturing, viewing, controlling video stream from Leopard's uvc compatible devices, with a special emphasis for the linux v4l2 driver.

For Leopard USB3.0 Firmware Update Tool, please refer to firmware update tool README

• Leopard USB3.0 Camera Linux Camera Tool
  • Table of Contents
  • Installation
    • Support OS Platforms
    • Support OpenCV Versions
    • Hardware Prerequisites
    • Dependencies
      • OpenCV Prerequisites
      • OpenCV CUDA Support
    • Build Camera Tool
    • Install leopard_cam
    • Uninstall leopard_cam
  • Usages
    • Run Linux Camera Tool on Host PC
    • Run Linux Camera Tool on Docker
    • Exit Linux Camera Tool
    • Kill Linux Camera Tool Zombie Process
  • Examples
    • Test on RAW sensor 12 Megapixel Sensor IMX477
    • Test on YUV 12 Megapixel AR1335-ICP3 MJPG output
    • Test on OV9782-OV580 stereo RAW8 1 Megapixel Cam
  • Code Structure
  • Declarations
    • Image Processing Functionalities
      • Benchmark Results
      • Datatype Definition for Data Stream
    • Auto Exposure
    • Exposure Time Calculation
  • Known Bugs
    • Exposure & Gain Control Momentarily Split Screen
    • SerDes Camera Experiences First Frame Bad When Uses Trigger
  • Reporting Bugs
  • Contributing

This application uses v4l2(video for linux) API for linux. The related system calls are also UNIX based, e.g. open(), close(), fork() etc. It doesn't support running on Mac OS and Windows. For Windows Camera Tool for your Leopard USB3 camera, please contact leopard support for application/source code. Docker hasn't supported USB video device pass through on Windows. So even running docker, it is suggested you have a linux operating system workstation there.

• 5.0.0-29-generic #31~18.04.1-Ubuntu SMP
• 4.18.0-17-generic #18~18.04.1-Ubuntu SMP
• 4.15.0-20-generic #21~Ubuntu 18.04.1 LTS
• 4.15.0-32-generic #35~16.04.1-Ubuntu

Due to OpenCV CUDA support hasn't been updated to 4.0.X, it is suggested to install 3.4.3 for OpenCV CUDA support. The non-OpenCV CUDA support part has been tested and verified on the following OpenCV versions

• 3.4.2
• 3.4.3
• 4.0.1
• 4.1.1 Doesn't work with OpenCV 4.2.0

• Camera: any Leopard USB3.0/2.0 cameras

• Host PC: with USB3.0 port connectivity

  Some boards like OV580-STEREO has more strict voltage requirements toward USB3.0 port. It is highly suggested to use a USB3.0 powered hub to regulate the voltage out from PC's USB3.0 port if a OV580-STEREO USB3 device is used.

  The underlying issue is that different host PCs/USB3 PCIe manufacturers have different USB3 controllers. Some of the USB3 cameras are not able to handle transient voltage droop from the USB3 port from PC. So a USB3.0 powered hub is highly recommended if this issue is repeatable with direct USB3 connection on your host PC. (Any serdes USB3 camera which have an external 12V power supply will be immune to this as you assumed :D)

  • Do a lsusb and found your Leopard USB3.0 camera device after connection: VID of our USB device is usually 2a0b (for more hardware information like OV580-STEREO, please refer to hardware.h).
  • Do a ls /dev/vi* and make sure there is video device listed.
  • Once you know which /dev/videoX the camera is at, use ./leopard_cam -d X when you run your camera

• USB3.0, USB2.0 Connectivity: If you plug in your USB camera device slow enough to a USB3.0 port, your host PC might still recognize the USB3.0 device as USB2.0. To be able to tell if your host PC has been recognized this USB3.0 camera truly as a USB3.0 device, do a v4l2-ctl --device=/dev/videoX --all, where X is the device number that being recognized by your PC. And check if its resolution, frame rate listed is the right one, (USB2.0 camera will have much lower bandwidth, so usually small resolution and frame rate).

foo@bar:~$ lsusb ## list all your USB device on your host PC
Bus 002 Device 034: ID 05e3:0612 Genesys Logic, Inc. 
Bus 002 Device 035: ID 2a0b:00cd  
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 035: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 034: ID 046d:c53d Logitech, Inc. 
Bus 001 Device 031: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
foo@bar:~$ lsusb -v -d 2a0b:00cd ## list verbose information on connected Leopard USB3 camera 
Bus 002 Device 035: ID 2a0b:00cd  
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.00
  bDeviceClass          239 Miscellaneous Device
  bDeviceSubClass         2 ?
  bDeviceProtocol         1 Interface Association
  bMaxPacketSize0         9
  idVendor           0x2a0b 
  idProduct          0x00cd 
  bcdDevice            0.00
  iManufacturer           1 
  iProduct                2 
  iSerial                 3 
  bNumConfigurations      1
...
foo@bar:~$ ls /dev//vi* ## list all the video device in your host PC
/dev/video0  /dev/video1

Make sure the libraries have installed. Run configure.sh in current directory for completing installing all the required dependencies.

$ chmod +X configure.sh
$ ./configure.sh

Make sure you have GTK 3 and OpenCV (3 at least) installed. The way you do to install this package varies by operational system.

Gtk3 and Gtk2 don't live together peaceful. If you try to run camera tool and got this error message:

Gtk-ERROR **: GTK+ 2.x symbols detected. Using GTK+ 2.x and GTK+ 3 in the same proc

It is mostly you have OpenCV build with GTk2 support. The only way to fix it is rebuild OpenCV without GTk2:

opencv_dir/release$ cmake [other options] -D WITH_GTK_2_X=OFF ..

in order to disable Gtk2 from OpenCV.

# rebuild OpenCV, this might take a while
# this is just a sample of rebuilding OpenCV, modify them to fit your need
$ rm -rf build/
$ mkdir build && cd build/
$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_TBB=ON -D BUILD_NEW_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON -D WITH_QT=ON -D WITH_OPENGL=ON -D WITH_GTK=ON -D WITH_GTK3=ON -D WITH_GTK_2_X=OFF ..
$ make -j6                     # do a $lscpu to see how many cores for CPU
$ sudo make install -j6

# link OpenCV
$ sudo sh -c 'echo "/usr/local/lib" >> /etc/ld.so.conf.d/opencv.conf'
$ sudo ldconfig

To build OpenCV CUDA support, please refer to INSTALL_OPENCV_CUDA.md

1. Make
   • Use make when you don't need OpenCV CUDA support

$ make

2. CMake
   • Use cmake when you need OpenCV CUDA support

$ mkdir build
$ cd build
########################################################################
$ cmake ..                      # USE_OPENCV_CUDA is turned off by default
$ cmake -DUSE_OPENCV_CUDA=ON .. # turn on USE_OPENCV_CUDA for OpenCV CUDA support
########################################################################
$ make

# Add to your project
add_subdirectory(linux_camera_tool)
include_directories(
	linux_camera_tool/src/
)

...

target_link_libraries(<Your Target>
	leopard_tools
)

$ sudo cp ./leopard_cam /usr/bin

$ sudo rm -f /usr/bin/leopard_cam 

• Run your Linux Camera Tool after installation

$ leopard_cam

• Run your Linux Camera Tool when you want to assign a specific video device /dev/videoX(usually when you have more than one camera in your host PC, and the Linux Camera Tool detected/opened a different device than you want)

# either the following four ways
$ leopard_cam -d X
$ leopard_cam -d /dev/videoX
$ leopard_cam --device X
$ leopard_cam --device /dev/videoX

• Run your Linux Camera Tool with a different resolution and frame rate than the default one

$ leopard_cam -s 1280x720 -t 15

• For detailed usage of commands in Linux Camera Tool, please run

$ leopard_cam -h 

Please refer to Docker README for more information

In the following five ways:

1. Press ESC, q, Q in your keyboard to exit the whole program
2. Use EXIT button on the grid1 control panel to exit the whole program
3. Use Help -> Exit in the menu bar to exit the whole program
4. User Ctrl + C to exit the control panel after exit from camera streaming GUI
5. Unexpected quit results in defunct process, use

   $ ps

   to see if the leopard_cam still alive, if it is, use

   $ killall -9 leopard_cam

   to kill the zombie process

If you forget to exit both windows and tried to run the camera tool again, it will give you the error of VIDIOC_DQBUF: Invalid argument, or when something goes wrong unexpected when running Linux Camera Tool, please check your available processes and kill leopard_cam, then restart the camera tool application

$ ps
$ killall -9 leopard_cam

1. Original streaming 12M(4056x3120) after debayer for IMX477 raw sensor
2. Use generic register read/write to talk to IMX477 sensor(8-bit slave addr:0x20)
3. Switch resolution to 8M(3840x2050), change bayer pattern, enable awb, auto brightness & contrast with OpenCV CUDA support
4. Apply gamma correction(1/2.2) and black level correction for debayered image

1. Original streaming for 12M AR1335-ICP3 full resolution MJPEG @10fps
2. Capture a bmp and raw image, save in the current directory
3. Flip and mirror image streaming
4. Enable/disable showing RGB color histogram
5. Change streaming contrast and brightness
6. Start and stop capturing video, save in the current directory

1. Original streaming two 1M(1280x800) OV9782 raw sensors after debayer
2. Enabled AWB, CLAHE(Contrast Limited Adaptive Histogram Equalization) algorithm) with non OpenCV CUDA support
3. Find device information and software info in menu bar
4. Flip and mirror image streaming, display left and right camera separately
5. Enable edge detection and change edge threshold

$ linux_camera_tool .
├── Makefile                              # for make
├── configure.sh                          
├── CMakeLists.txt                        # for CMake
├── gitlog-to-changelog.md                # automate generating CHANGELOG.md, version number in software
│
├── README.md                             # THIS FILE
├── INSTALL_OPENCV_CUDA.md
├── CHANGELOG.md
├── LICENSE.md
├── AUTHOR.md
│
├── config.json                           # configuration files for automating
├── BatchCmd.txt                          # camera controls
|
├── includes
│   ├── shortcuts.h                       # common used macro functions  
│   ├── hardware.h                        # specific hardware feature support
│   ├── gitversion.h                      # auto-generated header file for revision
│   ├── utility.h                         # timer for benchmark
│   ├── cam_property.h
│   ├── extend_cam_ctrl.h
|   ├── isp_lib.h
│   ├── core_io.h
│   ├── fd_socket.h
│   ├── gui_callbacks.h
|   ├── json_parser.h
│   ├── batch_cmd_parser.h
│   ├── ui_control.h
│   ├── uvc_extension_unit_ctrl.h
│   └── v4l2_devices.h
│
├── src
│   ├── cam_property.cpp                  # gain, exposure, ptz ctrl etc
│   ├── extend_cam_ctrl.cpp               # camera stream, capture ctrl etc
|   ├── isp_lib.cpp                       # ISP using OpenCV
|   ├── core_io.cpp                       # string manipulation for parser
|   ├── fd_socket.cpp                     # file descriptor socket for IPC
│   ├── gui_callbacks.cpp                 # part of the GUI callbacks
|   ├── json_parser.cpp                   # json parser
|   ├── batch_cmd_parser.cpp              # BatchCmd.txt parser
|   ├── ui_control.cpp                    # all GUI related
│   ├── uvc_extension_unit_ctrl.cpp       # all uvc extension unit ctrl
│   └── v4l2_device.cpp                   # udev ctrl
│   
└── test
    └── main.c                              

Most of the image processing jobs in Linux Camera Tool(code for ISP) are done by using OpenCV. Since there are histogram matrix calculations and many other heavy image processing involved, enabling these features will slow down the streaming a lot. If you build Linux Camera Tool with OpenCV CUDA support, these feature speed won't be compromised a lot. Detailed benchmark result is listed below.

Benchmark test is performed on a 12M pixel RAW12 bayer camera (IMX477) with host PC OpenCV build TBB on.

Auto exposure is usually implemented on sensor/ISP, which when enabled, won't further slow down the streaming. You need to check with camera driver for auto exposure support. If some sensor don't have AE support built-in, the check button in the first tab of Camera Control (Enable auto exposure)won't work.

There is a software generated auto exposure support in the second tab of Camera Control (Software AE), you can enable that if you really need AE support for the camera. Since adjusting gain and exposure under linux will experience split screen issue, same thing will happen to this software AE.

Since V4L2 doesn't provide a easier way to tell you what exact exposure time in ms like what Windows does, here is the explanation for helping you figuring out your camera's current exposure time in ms:

Exposure time = exposure_time_line_number / (frame_rate * total_line_number)

, where

1. exposure_time_line_number is the value that display is linux camera tool exposure control.
2. you can get frame_rate from the log message Get Current Frame Rate =, don't refer to Current Fps that displays in the Camera View, that's the frame rate your PC can process, not the physically USB received frame rate from camera.
3. total_line_number is usually around the height of the camera's current resolution, usually slightly larger than that since VD needs blanking. To get the exact total_line_number, you might want to read VTS register value for your sensor, which might be a hassle for you so I will only tell you this roughly calculation method...
4. Noticing for exposure time change in Linux Camera Tool Control GUI, the exposure time range will usually be over the camera's resolution height. If you have the exposure value set to be over camera's resolution height, it might affect your frame rate. Since when camera exposure time line number is greater than the total line number, that will slow down your camera's frame rate. This varies from different sensors. Some sensors don't allow that happen, so the exposure time will stick the same when it is over total line number.

When changing exposure and gain under linux, camera tool will experience split screen issue at the moment change is happened. This issue happens for the USB3 camera that use manual DMA in the FX3 driver. For the camera that utilizes auto DMA, the image will be ok when exposure and gain change happens.

For updating driver from manual DMA to auto DMA, you need to ensure:

1. PTS embedded information is not needed in each frame
2. The camera board has a Crosslink FPGA on the tester board that is able to add uvc header to qualify for auto DMA
3. FX3 driver need to be updated

When use triggering mode instead of master free running mode for USB3 SerDes camera, the very first frame received will be bad and should be tossed away. It is recommended to use an external function generator or a dedicated triggering signal for triggering the cameras for the purpose of syncing different SerDes cameras.

The included shot 1 trigger function is only a demonstration on generating one pulse and let camera output one frame after shot 1 trigger is clicked once. User should not fully rely on this software generated trigger but use a hardware trigger for sync the camera streaming.

Report bugs by creating a new issue on this Github page: https://github.com/danyu9394/linux_camera_tool/issues

If it is clearly a firmware issue: likely some features hasn't been implemented in the driver, please contact Leopard support: [email protected]

Fork the project and send pull requests, or send patches by creating a new issue on this Github page: https://github.com/danyu9394/linux_camera_tool/issues