Camera-ISP Driver

Introduction
The OMAP3 Camera-ISP driver is a Video4Linux2 (V4L2) driver, which makes use of v4l2-int-device framework, which is now marked as obsolete by v4l2 community.

Overall hardware structure
The Camera-ISP contains the following features:


 * Image sensor
 * CSI1/CCP2 serial interface
 * One/Two MIPI CSI2 serial interfaces (one for OMAP34xx/35xx, and two for OMAP36xx/37xx)
 * Parallel interface
 * SYNC mode
 * ITU mode
 * Video processing hardware
 * Video processing front end (VPFE)
 * CCDC
 * Video processing back end (VPBE)
 * Preview module
 * Resizer module
 * Statistic collection modules (SCM)
 * 3A metrics
 * Histogram
 * Central-resource shared logic (SBL)
 * Circular buffer
 * Memory management unit (MMU)
 * Clock generator
 * Timing control

And the way they are interconnected, can be described in the Public Technical Reference manual, found here (public TRM)

NOTE: Please check Chapter #6: Camera Image Signal Processor

Current driver structure
The HW blocks supported are:
 * SEN (Sensor)
 * CSI2
 * CCP2
 * CCDC
 * Preview
 * Resizer
 * SCM (H3A and Histogram)

The VPFE is supported by the CCDC module and performs signal processing operations on RAW image input data. Based on the pipeline used, the input RAW data can either be sent to the video port or can be saved into memory. The CCDC is also responsible for LSC (Lens Shading Compensation).

The VPBE consists of the Preview and Resizer module. The RAW input data is converted into YUV422 by the Preview module. The preview also performs other operations like white balance, luma enhance, black adjustment, etc. The Resizer module resizes the image into the resolution required by user space. The OMAP3 ISP resizer can only scale (up-scale and down-scale) by a factor of 4x.

The SCM or Statistics Collection Module consists of the H3A and Histogram blocks. Once the H3A stats (AEWB and AF) are collected, they are sent to the 3A engine. The 3A engine then processes these stats and returns back values to reprogram the ISP to improve the quality of the capture images.

Camera ISP driver files and locations
The files which contain the implementation of Camera ISP modules reside in drivers/media/video/isp/
 * Camera Driver implementation for OMAP34XX: omap34xxcam.c and omap34xxcam.h (Also valid and tested for 36xx chip family)
 * ISP library implementation and header file: isp/isp.c and isp/isp.h
 * ISP CCDC library implementation and header file: isp/ispccdc.c and isp/ispccdc.h
 * ISP preview library implementation and header file: isp/isppreview.c and isp/isppreview.h
 * ISP resizer library implementation and header file: isp/ispresizer.c and isp/ispresizer.h
 * ISP H3A library implementation and header file: isp/isph3a.c isp/isph3a.h (Mainly supports AEWB stat buffer collection)
 * Auto Focus: isp/isp_af.c and isp/isp_af.h
 * Histogram: isp/isphist.c and isp/isphist.h
 * Statistics: isp/ispstat.c and isp/ispstat.h
 * These files store the various table needed by ISP for image processing: isp/bluegamma_table.h, cfa_coef_table.h, greengamma_table.h, luma_enhance_table.h, noise_filter_table.h, redgamma_table.h
 * Header containing register address and field macros: ispreg.h
 * ISP interface library control all phy: isp/isp_csi.c, isp_csi.h, ispcsi2.c, ispcsi2.h
 * Auxiliary library to support DSS driver in case of 720p use case (when needed to use ISP resizer): isp/ispdss.c
 * Preview Wrapper and Resizer Wrapper Files: omap_previewer.c, omap_resizer.c, omap_prev2resz.c
 * Sensor driver located in drivers/media/video/: imx046.c and imx046_regs.h
 * Virtual Sensor Driver: isp_csi_memvs.c. This is the sensor implementation of virtual sensor used for all Memory to Memory pipes. This is an interface helper to feed images from memory into the CSI/CCP2 Memory read channel, which also registers with the omap34xxcam master as a slave sensor.
 * Camera Board Configure File located in arch/arm/mach-omap2: board-zoom2-camera.c. This is a platform specific board_config file to configure sensor, lens and power management for the camera driver.

Camera States
From a power management perspective, the Camera Driver supports active and suspended states that are entered by the execution of resume and suspend callbacks respectively. While in the active state, the Camera Driver manages the state machine of the camera device in five states: Configuration, Buffer Allocation & Mapping, Buffer Prepare and Streaming Capture as depicted in the adjoining figure.

An application can issue any sensor image, preview window, image cropping and control IOCTL in the Configuration state. Once the proper configuration is done, an application can move the state machine to an intermediate state to allocate and/or map the streaming buffers. Then it moves to a buffer prepare state through the V4L2 QBUF IOCTL call. After then it reaches at streaming capture state through the V4L2 STREAMON IOCTL call. V4L2 control IOCTLs can be made even in the Streaming Capture states. Any state can be suspended or resumed by the Linux power management framework.

Imaging Pipelines
The camera driver supports different data paths or pipelines. The pipes are classified as High Performance and High Quality, based on the system performance and image quality of the images captured.

Some of the Pipes that we support are:
 * SEN-CSI2-CCDC
 * SEN-CSI2-CCDC-PREV-RESZ-MEM
 * MEM-PREV-MEM: can be accessed using the preview wrapper
 * MEM-RESZ-MEM: can be accessed using the resizer wrapper
 * MEM-PREV-RESZ-MEM: can be accessed using the prev2resz wrapper
 * SEN-CSI2-MEM
 * MEM-CCP2-CCDC-PREV-RESZ-MEM

High Performance
In the HP pipe, (for all cases except 720p) the data path used is SEN-CSI2-CCDC-PREV-RESZ-MEM. The resized buffers are then sent to the Display subsystem. For the 720p use case, the path used is SEN-CSI2-CCDC-PREV-MEM. The buffers stored in memory are then sent to the DSS; the DSS then scales those buffers using the ISP resizer. There is an ongoing discussion to remove this '720p hack'.

High Quality
Additional pipelines were added to support HQ capture. These pipes are selected when the Android Camera app requests for an HQ image capture. The pipe SEN-CSI2-MEM-CCP2-CCDC-PREV-RESZ-MEM is used for for HQ capture. To select the SEN-CSI2-MEM data path, a private IOCTL must be used i.e. V4L2_CID_PRIVATE_OMAP3ISP_CSI2MEM.

The SEN-CSI2-MEM data path is used to receive directly a raw image with the most appropriate parameters for exposure and gain. This path is selected during raw capture. For this purpose, we use /dev/video5 with the configuration for BAYER format. The Sensor-MEM-CCP2-CCDC-H3-MEM path is used to gather H3A statistic from the RAW image. For this purpose, /dev/video10 is used as virtual sensor to provide input of the image in memory; and /dev/video6 is used to configure the CCDC and H3A engine and receive the H3A statistics buffers.

Camera Driver Kernel Source
The OMAP3 ISP driver is part of the Linux Android Kernel. The source can be found here (android kernel source)

To build the kernel, refer to this page

Camera Driver Feature Trees
Akwasi

Christina

Camera Driver Testing
The camera driver test code can be found here (test code source)

Refer to the README for build instructions.

$ git clone git://dev.omapzoom.org/pub/scm/richo/device_driver_test.git $ cd device_driver_test/ $ git checkout master $ git checkout --track -b ddt_master origin/master

Add this to your .bashrc: $ export TESTROOT= $ export CROSS_COMPILE=arm-none-linux-gnueabi- $ export KDIR= $ export HOST=arm-none-linux-gnueabi- $ export TESTSUITES="camera" $ make

Note: If you are using the latest version of the git tree, then make "all" defaults to "APPLICABLE_TESTS". In that case, camera tests will not be built. So, to build camera, try make TESTSUITES=camera.

The  is a location where the driver test suites will be copied. Typically a user will point  to a location of their NFS file system.

If you want to build test apps statically linked (for use in Android FS) then export following before build: $ export FSTYPE=android

Execute a specific Camera Test Group:

$ cd camera/test_code/scripts $ ./test_runner.sh -p L_DD_CAMERA_0102

Execute a specific Camera Test Case ID (within a Group):

$ cd camera/test_code/scripts $ ./test_runner.sh -T 0002  -p L_DD_CAMERA_0102

Background for Media Controller Framework
As the HW for multimedia embedded devices is becoming complex, and can be configured in many different ways, current Video4Linux specification wasn't enough for covering the required flexibility by applications to truly exploit the hardware.

As a consequence, the driver ended up being too complex, and using a lot of non-standard, "hackish" paths to export that functionality to user-space.

So, as many other multimedia embedded silicon vendors have very similar experiences, the Media Controller concept was formulated.

A little more background on it can be seen here.

Also, it's very helpful to look at one of the presentations done by Laurent Pinchart in the last V4L2 mini summit, here.

The OMAP3 Camera-ISP driver is a work in progress towards being the first driver with the Media controller framework (see above), and contains entities for these HW blocks: Sensor, CSI2, CCP2, CCDC, Preview, Resizer, H3A AEWB, H3A AF and Histogram

And the way those are interconnected is intended to be done in a user-space library.

In few words, the kernel will just provide entities, and the basic validations to be done to configure and link them.

And userspace will be the one deciding, in an abstracted fashion, what pipelines to build, like:

... etc.
 * Sensor -> CSI2 -> CCDC -> PREV -> RESZ -> MEM
 * MEM -> CCP2 -> CCDC -> PREV -> MEM
 * Sensor -> CSI2 -> MEM

Also, the main camera driver associates a sensor driver, with a flash device and a lens motor, to provide a fully integrated camera solution.

Mandatory items
Already done by Laurent Pinchart, and published on 6 Nov
 * 1) Migrate Camera-ISP driver from v4l2-int-device to v4l2_subdev framework:

Done for Nokia N900, Zoom2/3 & 3630SDP platforms
 * 1) Migrate Sensor drivers from v4l2-int-device to v4l2_subdev framework

Already started, and N900 is the first functional device. The driver can be found in the 'Source' section below.
 * 1) Migrate Camera-ISP driver to Media Controller framework

This is in progress, and these are the revisions so far:
 * 1) Submit v4l2_subdev preparations for Media controller for review.
 * 1) * V1
 * 2) * V2 (included as part of the latest OMAP3 ISP RFC): 1, 2, 3

This is in progress, and these are the revisions so far:
 * 1) Submit Media controller patches for review.
 * 1) * V1
 * 2) * V2
 * 3) * V3
 * 4) * V4
 * 5) * V5


 * 1) After above is completed, submit Camera-ISP driver, adapted to Media Controller for review. Meanwhile, Laurent has been posting snapshots of the driver to function as a reference to Media Controller patches:
 * 2) * For V3 of Media Controller: refer to this
 * 3) * For V5 of Media Controller: 1, 2, 3.

Desirable items

 * 1) Migrate other platform's sensor drivers to v4l2_subdev:


 * 1) * Aptina MT9P012 5MP RAW10 parallel sensor (3430SDP)
 * 2) * Tps61059 Flash (3430SDP)
 * 3) * DW9710 Absolute Lens (3430SDP)
 * 4) * OmniVision OV3640 3MP YUYV/RAW10 CSI2 sensor (3430SDP, LDP)
 * 5) * Aptina MT9T111 3MP YUYV/JPEG parallel sensor (Beagleboard xM)
 * 6) * Aptina MT9V113 VGA YUYV parallel sensor (Beagleboard xM)


 * 1) Prepare Documentation on steps to adapt driver to any custom platform.

People involved in migration to Media Controller Framework
The people involved in these plans are mainly:


 * Laurent Pinchart 
 * Sakari Ailus 
 * Vaibhav Hiremath 
 * Sergio Aguirre 

Source
You can find the development repository at ('devel' branch), maintained by Laurent Pinchart.

Also, Laurent keeps a clean tree meant only for consolidated and rebased patches for submission located here

Please base all your contributions on above trees _only_.

Testing
Media controller can be tested with Laurent's 'media-ctl' tool, located here

Build it with:

$ KDIR= make

and copy media-ctl binary in your filesystem.

Add "CFLAGS=-static" to compile statically, in case your target filesystem lacks libraries (Like a busybox based filesystem).

For help, run it with -h option.

Some sample instructions:

Show topology of subdevices (-p) for /dev/media0 media controller device (-d):

./media-ctl -d /dev/media0 -p

Reset links (-r) and create a link (-l) for CSI2 subdev with CSI2 output video node:

./media-ctl -d /dev/media0 -r -l '"OMAP3 ISP CSI2a":1 -> "OMAP3 ISP CSI2a output":0 [1]'

Configure imx046 sensor output pad to 3280x2464 SRGGB10, and CSI2a input as the same size (CSI2a output size is propagated automatically):

./media-ctl -d /dev/media0 -f '"imx046 2-001a":0 [SRGGB10 3280x2464]','"OMAP3 ISP CSI2a":0 [SRGGB10 3280x2464]'

For actually capturing the resulting images, you should be able to use any v4l2 compatible capture app (mplayer w/v4l2 plugin, Laurent's yavta app, etc.)

Appendix A: Sergio Aguirre's tree
This tree's main development focus is Zoom3 boards, and this small guide in order to help you try the code easily.

Source
You can find the tree here

Which contains the following branches:


 * master: Clone of Laurent's tree, 'media-0004-omap3isp' branch + Zoom2/3 camera sensor support & wip patches.
 * forLaurent: Rebased branch of Pending patches for Laurent's 'media-0004-omap3isp' branch. This includes only changes to omap3isp code laying in my master branch.

Compiling
After cloning, and while in master branch, do:

$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- omap2plus_defconfig $ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- menuconfig

1. Select "Device Drivers->Multimedia support" (CONFIG_MEDIA_SUPPORT) and enter to the submenu.

2. Select "Video For Linux" (CONFIG_VIDEO_DEV)

3. Go back to main kernel configuration page

4. Select "System Type->TI OMAP2/3/4 Specific Features->Zoom 2/3 & 3630SDP camera support" (CONFIG_VIDEO_MACH_OMAP_ZOOM)

That will select: "Sony IMX046 sensor driver (8MP)" (CONFIG_VIDEO_IMX046) and "Piezo Actuator Lens driver for LV8093" (CONFIG_VIDEO_LV8093)

NOTE: If you want to start adding patches, I'll suggest you to follow the topic branches philosophy.

Compile as usual, with:

$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- uImage

Patches in queue for Laurent's tree

 * YUV support for CCDC + cleanups: (Acked V2 on Tue, 16 Nov 2010 12:44:44 +0100, pending for merge)
 * V1
 * V2


 * Improve inter subdev interaction:
 * V1
 * V2

WIP not yet submitted

 * Improve subdev isolation to facilitate subdev selection (Major)


 * JPEG Input Support in CCDC (Minor)

Appendix B: TI MC migration tree
This tree's main development focus is Zoom3 boards, and most importantly, is based on TI omap3 integration tree (found here).

IMPORTANT: This tree is only helpful if you want a stable Zoom3 & 3630SDP support tree, but it is NOT meant for upstreaming.

Links
Please visit it's dedicated Gitorious project page here.

For help, visit the project's Wiki here.