Android, QEMU and the Camera - Emulating the Camera Hardware in Android (Part II)

In my previous post I reviewed the "administrative" background related to camera emulation on Android.
Now let's trace the code backwards, from the point of loading the Camera HAL module and until we open an android.hardware.Camera instance in the application.  The diagrams below shows the top-down control flow of loading a camera HALv1 module and initializing a camera HALv1 device.  But this is all fairly standard and we are interested in getting some insight into the particulars of the emulated camera, so let's I start at the end: the loading of the HAL module.

The generic Android emulated device created by the AVD tool is called goldfish and, following the HAL naming convention,  the dynamically linked (shared object) library containing the goldfish camera HAL  is located in /system/lib/hw/camera.goldfish.so.  You can 'adb shell' into an emulated device instance and search for camera.goldfish.so in the device file system just like you would do on a real device: 

nzmora@~/Dev/aosp:$ adb -s emulator-5554 shell
root@generic_x86:/ #  ls -la /system/lib/hw/
-rw-r--r-- root     root            9464  2014-03-08  16:20  audio.primary.default.so
-rw-r--r-- root     root          13560  2014-03-08  16:20  audio.primary.goldfish.so
-rw-r--r-- root     root        144868  2014-03-08  16:21  audio_policy.default.so
-rw-r--r-- root     root      2309860  2014-03-08  16:21  bluetooth.default.so
-rw-r--r-- root   root            5204  2014-03-08  16:23  camera.goldfish.jpeg.so
-rw-r--r-- root   root        288668  2014-03-08  16:22  camera.goldfish.so
-rw-r--r-- root     root          13652  2014-03-08  16:21  gps.goldfish.so
-rw-r--r-- root     root          13872  2014-03-08  16:20  gralloc.default.so
-rw-r--r-- root     root          21836  2014-03-08  16:21  gralloc.goldfish.so
-rw-r--r-- root     root            5360  2014-03-08  16:21  keystore.default.so
-rw-r--r-- root     root            9456  2014-03-08  16:20  lights.goldfish.so
-rw-r--r-- root     root            5364  2014-03-08  16:20  local_time.default.so
-rw-r--r-- root     root            5412  2014-03-08  16:17  power.default.so
-rw-r--r-- root     root          13660  2014-03-08  16:20  sensors.goldfish.so
-rw-r--r-- root     root            5360  2014-03-08  16:17  vibrator.default.so

-rw-r--r-- root     root            5364  2014-03-08  16:21  vibrator.goldfish.so

The code itself is found in the Android source tree, at /device/generic/goldfish/camera so let's turn our attention there.

When approaching new code, to get a quick high-level understanding of what's what, I like to start with the makefile, Android.mk, and give it a quick scan: examining the input files, flags and output files.  The Android makefile format is fairly self describing and easier to grasp than "true" make files because it hides most of the gory details.  In any case, in this particular makefile the LOCAL_SOURCE_FILES variable (which contains the list of files to compile) is listed in a sort of hierarchy - and this gives us a first clue as to how the source files relate to one another.  

LOCAL_SRC_FILES := \
        EmulatedCameraHal.cpp \
        EmulatedCameraFactory.cpp \
        EmulatedCameraHotplugThread.cpp \
        EmulatedBaseCamera.cpp \
        EmulatedCamera.cpp \
                EmulatedCameraDevice.cpp \
                EmulatedQemuCamera.cpp \
                EmulatedQemuCameraDevice.cpp \
                EmulatedFakeCamera.cpp \
                EmulatedFakeCameraDevice.cpp \
                Converters.cpp \
                PreviewWindow.cpp \
                CallbackNotifier.cpp \
                QemuClient.cpp \
                JpegCompressor.cpp \
        EmulatedCamera2.cpp \
                EmulatedFakeCamera2.cpp \
                EmulatedQemuCamera2.cpp \
                fake-pipeline2/Scene.cpp \
                fake-pipeline2/Sensor.cpp \
                fake-pipeline2/JpegCompressor.cpp \
        EmulatedCamera3.cpp \
                EmulatedFakeCamera3.cpp

In the first file, EmulatedCameraHal.cpp, I find the definition of the HAL module structure: HAL_MODULE_INFO_SYM. This is the symbol that /hardware/libhardware/hardware.c loads when CameraService is first referenced (see flow diagram above), and therefore it is the entry way into the HAL.

camera_module_t HAL_MODULE_INFO_SYM = {
    common: {
         tag:                            HARDWARE_MODULE_TAG,
         module_api_version:  CAMERA_MODULE_API_VERSION_2_1,
         hal_api_version:         HARDWARE_HAL_API_VERSION,
         id:                              CAMERA_HARDWARE_MODULE_ID,
         name:                        "Emulated Camera Module",
         author:                       "The Android Open Source Project",
         methods:                    &android::EmulatedCameraFactory::mCameraModuleMethods,
         dso:                           NULL,
         reserved:                   {0},
    },
    get_number_of_cameras:  android::EmulatedCameraFactory::get_number_of_cameras,
    get_camera_info:              android::EmulatedCameraFactory::get_camera_info,
    set_callbacks:                   android::EmulatedCameraFactory::set_callbacks,
};

The first thing that we learn is that this is a HAL module v2.1.  There are 4 function pointers listed in this structure and cgrep'ing these functions leads us to file EmulatedCameraFactory.cpp, where these functions are defined.  We quickly learn from the code documentation in EmulatedCameraFactory.cpp that "A global instance of EmulatedCameraFactory is statically instantiated and initialized when camera emulation HAL is loaded".
When the CameraService invokes camera_module_t::get_camera_info, it actually performs a call to gEmulatedCameraFactory.getCameraInfo.  In other words, the three function pointers in camera_module_t just forward the work to gEmulatedCameraFactory (the global singleton factory instance I mentioned above):

int EmulatedCameraFactory::get_camera_info(int camera_id, struct camera_info* info)
{
    return gEmulatedCameraFactory.getCameraInfo(camera_id, info);
}

Let's refocus our attention at where the action is: the constructor of EmulatedCameraFactory.  The first thing the EmulatedCameraFactory constructor (device/generic/goldfish/camera/EmulatedCameraFactory.cpp) does is connect to the camera service in the Android emulator.  Please notice that this is not the CameraService of the Android framework!  This is a very important distinction.
I will describe the emulator's camera service in the third post in this series.

The code documentation does a very good job at explaining the EmulatedCameraFactory class responsibility:

    /* Class EmulatedCameraFactory - Manages cameras available for the emulation.
     *
     * When the global static instance of this class is created on the module load,
     * it enumerates cameras available for the emulation by connecting to the
     * emulator's 'camera' service. For every camera found out there it creates an
     * instance of an appropriate class, and stores it an in array of emulated
     * cameras. In addition to the cameras reported by the emulator, a fake camera
     * emulator is always created, so there is always at least one camera that is
     * available.
     *
     * Instance of this class is also used as the entry point for the camera HAL API,
     * including:
     *  - hw_module_methods_t::open entry point
     *  - camera_module_t::get_number_of_cameras entry point
     *  - camera_module_t::get_camera_info entry point
     *
     */

Usually, when I'm trying to quickly get familiar with new code I either draw for myself some call flows, or I write them down.  This helps me understand the code, and this is also a quick way to re-familiarize myself with the code if I put it away for a prolonged time and then need to reference it.  In the case of the EmulatedCameraFactory constructor I used another technique, which I usually use less often.  It is a stripped-down syntax-incomplete version of the code.  This technique is useful when there's a method like the EmulatedCameraFactory constructor which packs a lot of action.  This particular code is self-explanatory, except for the call to mQemuClient.connectClient, but I'll return to that later - for now I choose to do a breadth-wise scanning of the code.

EmulatedCameraFactory::EmulatedCameraFactory()
{
    /* Connect to the factory service in the emulator, and create Qemu cameras. */
    if (mQemuClient.connectClient(NULL) == NO_ERROR) {
        /* Connection has succeeded. Create emulated cameras for each camera
         * device, reported by the service. */
        createQemuCameras();
    }
     
    if (isBackFakeCameraEmulationOn()) {
        switch (getBackCameraHalVersion()):
            1: new EmulatedFakeCamera(camera_id, true, &HAL_MODULE_INFO_SYM.common);
            2: new EmulatedFakeCamera2(camera_id, true, &HAL_MODULE_INFO_SYM.common);
            3: new EmulatedFakeCamera3(camera_id, true, &HAL_MODULE_INFO_SYM.common);
        mEmulatedCameras[camera_id]->Initialize()
    }
    if (isFrontFakeCameraEmulationOn()) {
        switch (getBackCameraHalVersion()):
            1: new EmulatedFakeCamera(camera_id, true, &HAL_MODULE_INFO_SYM.common);
            2: new EmulatedFakeCamera2(camera_id, true, &HAL_MODULE_INFO_SYM.common);
            3: new EmulatedFakeCamera3(camera_id, true, &HAL_MODULE_INFO_SYM.common);
        mEmulatedCameras[camera_id]->Initialize()
    }
 
    mHotplugThread = new EmulatedCameraHotplugThread(&cameraIdVector[0],                                                                                                                          mEmulatedCameraNum);
    mHotplugThread->run();
  }
 
When you review the pseudo-code above, remember that FakeCamera refers to a camera device fully emulated in SW, while QemuCamera refers to a real web-camera that is wrapped by the emulation code.

After I got a bit of understanding of the initialization dynamics, I turn to study the structure of rest of the classes. When there are many classes involved (46 files in this case), I find that a class diagram of the overall structure can help me identify the more important classes (I have no intention of going over the code of all these classes). I extract the class hierarchy structure by scanning the .h files looking for class relationships and key value members. Usually I hand sketch some UML on paper - this doesn't have to be complete since I am just trying to get a quick grasp of things.

There are a few structural points to note:

• The class structure is similar to the structure of the class listing in the makefile so whoever wrote the makefile was nice enough to be professional all the way (adding pseudo documentation to the makefile, if you will).
• EmulatedFakeCamera types are classes that actually simulate the camera frames and behavior, along with a simulated sensor and processing pipeline.  Their implementation is interesting in and by itself and I'll return to this in a different post.  
• EmulatedQemuCamera types are a gateway to actual cameras connected to the host - i.e. webcams connected to your workstation or built into your laptop.  I visually differentiate between the Qemu and Fake cameras by giving them different colors.
• There are Camera types and CameraDevice types.  CameraDevice types are more important as they contain more code.
• The EmulatedCameraFactory represents the camera HAL module and contains handles to EmulatedCameras.
• There are two classes which abstract the connection to the QEMU emulator.  You can see that the EmulatedQemuCameraDevice holds a reference to CameraQemuClient and clearly this is required for communicating with the webcam on the emulator (more on this later).  There are three related classes: EmulatedCamera1, EmulatedCamera2, EmulatedCamera3 which represent cameras that are exposed through HALv1, HALv2, and HALv3, respectively.  Obviously HALv2 is of no significance by now because Android does not support it.  HALv3 does not exist for the webcam, most likely because the new HALv3 does not add any new features to a simple point & shoot webcam.

Now back to the dynamic view (my discovery process is a back & forth dance to discover new components and interactions) - when an application calls android.hardware.Camera.open(cameraId) this call is propagated through the code layers and ends with a call to camera_module_t::methods.open(cameraId) which is actually a call to EmulatedCameraFactory::device_open.  You can trace this flow in the first diagram in this blog post.

EmulatedCameraFactory::device_open
    ==> gEmulatedCameraFactory.cameraDeviceOpen
        ==> mEmulatedCameras[camera_id]->connectCamera(hw_device_t** device)

There's a whole lot of interesting stuff going on in the emulation code, especially in the emulated fake camera code, but in this blog post I want to look at the emulated QEMU camera code (EmulatedQemuCamera) and the communication with QEMU.

So, on to QemuClient.  This class "encapsulates a connection to the 'camera' service in the emulator via qemu pipe". The pipe connection is established by invoking qemu_pipe_open(pipename) which is implemented in /hardware/libhardware/include/hardware/qemu_pipe.h: First a device of type /dev/qemu_pipe is opened, and then the concatenation of the strings "pipe:" and pipename is written to the device.  In the kernel, we find the other side of this pipe (i.e. the qemu_pipe device driver) in /drivers/platform/goldfish/goldfish_pipe.c.  The header of the driver does an excellent job of describing this driver, so I bring it forth without much ado:

    /* This source file contains the implementation of a special device driver
     * that intends to provide a *very* fast communication channel between the
     * guest system and the QEMU emulator.
     *
     * Usage from the guest is simply the following (error handling simplified):
     *
     *    int  fd = open("/dev/qemu_pipe",O_RDWR);
     *    .... write() or read() through the pipe.
     *
     * This driver doesn't deal with the exact protocol used during the session.
     * It is intended to be as simple as something like:
     *
     *    // do this _just_ after opening the fd to connect to a specific
     *    // emulator service.
     *    const char*  msg = "";
     *    if (write(fd, msg, strlen(msg)+1) < 0) {
     *       ... could not connect to service
     *       close(fd);
     *    }
     *
     *    // after this, simply read() and write() to communicate with the
     *    // service. Exact protocol details left as an exercise to the reader.
     *
     * This driver is very fast because it doesn't copy any data through
     * intermediate buffers, since the emulator is capable of translating
     * guest user addresses into host ones.
     *
     * Note that we must however ensure that each user page involved in the
     * exchange is properly mapped during a transfer.
     */

QEMU pipes are further described in external/qemu/docs/ANDROID-QEMU-PIPE.TXT.

QemuClient::sendMessage and QemuClient::receiveMessage are wrappers for the pipe operations qemud_fd_write and qemud_fd_read, respectively.  QemuClient::doQuery is slightly more involved and I'll get to it in the third and final blog post in this series.

To recap, so far I've shown that the camera HAL of the emulated goldfish device contains classes that abstract an emulated QEMU camera (EmulatedQemuCameraDevice) which holds a reference to an instance of CameraQemuClient which it uses to communicate with a device named /dev/qemu_pipe.  This character device represents a virtual device with (emulated) MMIO register space and IRQ line, and it belongs to the emulated goldfish platform.  On the "other side" of the pipe device is the QEMU emulator and more specifically the goldfish pipe which is implemented in /external/qemu/hw/goldfish_pipe.c
You can think of this pipe as the conduit for communication between the Android Guest kernel and the emulator code.  In the Android QEMU codebase, file /external/qemu/android/hw-qemud.c implements a sort of bridge between various Android QEMU services and the goldfish pipe device.  One of these Android QEMU services is the camera-service that I briefly mentioned earlier.  This camera-service is the topic of the third blog post.  I'll wrap up with a diagram showing the relationships between the various components.

Android, QEMU and the Camera - Emulating the Camera Hardware in Android (Part I)

I was playing around with a simple Android application that makes use of the camera, in order to prove a point about the rotation and mirroring of camera preview frames.  I tested the code on my Nexus 4 and Nexus 7 and after making sure that the Android API performed as I expected, I decided to give it a go on an Android virtual device (AVD) on Eclipse.  I quickly configured a Nexus 4 AVD with Webcam0 as the front camera, but the emulator did not behave as I expected: the preview picture was rotated 270 degrees!  I figured this was due to the unknown scanning order of the webcam on my laptop and therefore this is the resulting quirk.  Still, when I switched the front camera to use the emulated camera, I get the same strange results from android.hardware.Camera.CameraInfo.orientation - a 270 degree orientation which is unexpected and not like the behavior of the camera on the actual Nexus 4 device that I own.

OK, so this obviously called for an investigation of how the camera is emulated and why I am getting these strange orientation values.  It turned out that the camera orientation is hard coded - but by the time I discovered that code, my curiosity about how the camera works in emulation mode led me to dig deeper.

This three-part blog post is an attempt to review the AVD camera emulation code and the process that I usually use for reverse engineering code. Along the way I added to the Android emulator code to emulate Intel's Atom SoC camera and some other neat (geek talk) stuff.  The first blog post will lay all the infrastructure details before I get to the interesting implementation details in the second post.

First thing, first: when you create a new AVD you configure the camera emulation for both the front and back cameras of the emulated device.  There are three choice, None, Emulated, and Webcam0.  As we will see later, Emulated refers to a completely fake emulated camera which uses synthetically generated image frames.  Webcam0 refers to the webcam attached or built-in to your PC or laptop.

The AVD configuration is saved in a file on your workstation (on Windows it is located in %USERPROFILE%\.android\avd\AtomAVD.avd\hardware-qemu.ini) and I'll refer to it later when I describe how the emulator chooses which emulated camera to create.  Here's the snippet from my AVD's camera configuration:
    hw.camera.back = emulated
    hw.camera.front = webcam0

I'll show later that this configuration can be extended with private variables if you need to configure your proprietary emulator.

To wrap up this post I want to give a very high level of the emulation environment.  As with all virtual environments, we have three interacting components: the Host OS, the Guest OS, and the emulator.  The Google SDK AVD uses the QEMU emulator in "computer emulation mode".  To quote Wikipedia:

QEMU (short for "Quick EMUlator") is a free and open-source hosted hypervisor that performs     hardware virtualization.
QEMU is a hosted virtual machine monitor: It emulates central processing units through dynamic binary translation and provides a set of device models, enabling it to run a variety of unmodified guest operating systems. It also provides an accelerated mode for supporting a mixture of binary translation (for kernel code) and native execution (for user code), in the same fashion as VMware Workstation and VirtualBox do. QEMU can also be used purely for CPU emulation for user-level processes, allowing applications compiled for one architecture to be run on another.

The diagram below depicts QEMU in full system emulation mode running in parallel to a few other Host applications.  QEMU runs as a Host OS process and emulates a full Android mobile device, including an Atom x86 or ARM processor, SoC IPs and various peripherals.  On a Linux Host OS, the KVM (Kernel Virtual Machine) driver can be used for virtualizing the CPU and memory.  On an x86 Host machine, HAXM (Hardware Accelerated Execution Manager) can be used for further acceleration.

There is a very good description with many further details in this site about Android Emulator Under the Hood.