Camera Framework 分析

Camera Framework 分析,本文主要介绍 Camera API2 相关。

类文件速查表

类文件目录

1
2
3
4
5
6
1. Framework Java API1:frameworks/base/core/java/android/hardware/Camera.java
2. Framework Java API2:frameworks/base/core/java/android/hardware/camera2
3. Framework JNI: frameworks/base/core/jni/
4. AIDL: frameworks/av/camera/aidl
5. Framework Native: frameworks/av/camera
6. Framework Service: frameworks/av/services/camera/libcameraservice

JNI 相关

1
2
3
4
5
6
7
8
// frameworks/base/core/jni
./android_hardware_camera2_legacy_LegacyCameraDevice.cpp
./android_hardware_Camera.cpp
./android/graphics/Camera.cpp
./include/android_runtime/android_hardware_camera2_CameraMetadata.h
./android_hardware_camera2_DngCreator.cpp
./android_hardware_camera2_CameraMetadata.cpp
./android_hardware_camera2_legacy_PerfMeasurement.cpp

API 1 中,使用 jni 通过 Binder 机制和 CameraService 通信。
API 2 中,直接在 CameraManager.java 中通过 Binder 机制和 CameraService 通信。

AIDL 相关

Framework Camere AIDLCamera 中客户端和服务端跨进程通信时使用的 AIDL 文件,代码都在 frameworks/av/camera/ 目录下,其中 aidl 文件一共有 16 个:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
xmt@server005:~/frameworks/av/camera/aidl/android/hardware$ tree
.
├── camera2
│   ├── CaptureRequest.aidl
│   ├── ICameraDeviceCallbacks.aidl
│   ├── ICameraDeviceUser.aidl
│   ├── impl
│   │   ├── CameraMetadataNative.aidl
│   │   └── CaptureResultExtras.aidl
│   ├── params
│   │   ├── OutputConfiguration.aidl
│   │   ├── VendorTagDescriptor.aidl
│   │   └── VendorTagDescriptorCache.aidl
│   └── utils
│   └── SubmitInfo.aidl
├── CameraInfo.aidl
├── CameraStatus.aidl
├── ICamera.aidl
├── ICameraClient.aidl
├── ICameraService.aidl
├── ICameraServiceListener.aidl
└── ICameraServiceProxy.aidl

4 directories, 16 files

frameworks/av/camera/aidl/ 目录下的 aidl 文件有两种类型:

  • 作为 Binder 中的 IInterface 跨进程通信中能提供的方法
  • 作为 Binder 中的 parcelable 跨进程通信数据传输的数据结构

很容易从名字上区分这两种类型的文件,IInterface 类型的文件都是以 I 开头的,比如:ICameraService.aidl, ICameraDeviceUser.aidl 等。不管是哪种类型的 aidl 文件,它们都会生成对应的 .java, .h, .cpp 文件,分别供 Java 层和 CPP 层调用。

IInterface 类型文件

IInterface 类型文件一共有 7 个,它们的 .java, .h, .cpp 文件,绝大部分都是自动生成的。

Java 文件是在 frameworks/base/Android.mk 中定义规则,在编译时自动生成:

1
2
3
4
5
6
7
8
9
10
11
// frameworks/base/Android.mk
LOCAL_SRC_FILES += \
...
../av/camera/aidl/android/hardware/ICameraService.aidl \
../av/camera/aidl/android/hardware/ICameraServiceListener.aidl \
../av/camera/aidl/android/hardware/ICameraServiceProxy.aidl \
../av/camera/aidl/android/hardware/ICamera.aidl \
../av/camera/aidl/android/hardware/ICameraClient.aidl \
../av/camera/aidl/android/hardware/camera2/ICameraDeviceUser.aidl \
../av/camera/aidl/android/hardware/camera2/ICameraDeviceCallbacks.aidl \
...

out/target/common/obj/JAVA_LIBRARIES/framework_intermediates/dotdot/ 目录下生成对应的 Java 文件:

1
2
3
4
5
6
7
8
// out/target/common/obj/JAVA_LIBRARIES/framework_intermediates/dotdot/
av/camera/aidl/android/hardware/ICameraService.java
av/camera/aidl/android/hardware/ICameraServiceListener.java
av/camera/aidl/android/hardware/ICameraServiceProxy.java
av/camera/aidl/android/hardware/ICamera.java
av/camera/aidl/android/hardware/ICameraClient.java
av/camera/aidl/android/hardware/camera2/ICameraDeviceUser.java
av/camera/aidl/android/hardware/camera2/ICameraDeviceCallbacks.java

.h, .cpp 文件中,ICamera.aidl, ICameraClient.aidl 两个文件是直接以代码形式手动实现的:

1
2
3
4
5
6
7
8
9
// 1. ICameraClient.aidl
frameworks/av/camera/aidl/android/hardware/ICameraClient.aidl
frameworks/av/camera/include/camera/android/hardware/ICameraClient.h
frameworks/av/camera/ICameraClient.cpp

// 2. ICamera.aidl
frameworks/av/camera/aidl/android/hardware/ICamera.aidl
frameworks/av/camera/include/camera/android/hardware/ICamera.h
frameworks/av/camera/ICamera.cpp

其他 5 个 aidl 文件是在 frameworks/av/camera/Android.bp 中定义规则,编译时自动生成对应的 .h, .cpp 文件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
// frameworks/av/camera/Android.bp
cc_library_shared {
name: "libcamera_client",

aidl: {
export_aidl_headers: true,
local_include_dirs: ["aidl"],
include_dirs: [
"frameworks/native/aidl/gui",
],
},

srcs: [
// AIDL files for camera interfaces
// The headers for these interfaces will be
// available to any modules that
// include libcamera_client, at the path "aidl/package/path/BnFoo.h"
"aidl/android/hardware/ICameraService.aidl",
"aidl/android/hardware/ICameraServiceListener.aidl",
"aidl/android/hardware/ICameraServiceProxy.aidl",
"aidl/android/hardware/camera2/ICameraDeviceCallbacks.aidl",
"aidl/android/hardware/camera2/ICameraDeviceUser.aidl",


// Source for camera interface parcelables,
// and manually-written interfaces
"Camera.cpp",
"CameraMetadata.cpp",
"CameraParameters.cpp",
...
}

out/soong/.intermediates/frameworks/av/camera/libcamera_client/ 目录下生成对应的 .h, .cpp 文件,通常在该目录下会同时生成 32 和 64 位两套代码,但实际两份代码是一样的,这里选取 64 位的:

  • 64 位:android_arm64_armv8-a_shared_core
  • 32 位:android_arm_armv7-a-neon_cortex-a53_shared_core
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// 目录 out/soong/.intermediates/frameworks/av/camera/libcamera_client
// 64 位 android_arm64_armv8-a_shared_core/gen/aidl/
android/hardware/ICameraService.h
android/hardware/BnCameraService.h
frameworks/av/camera/aidl/android/hardware/ICameraService.cpp

android/hardware/ICameraServiceListener.h
android/hardware/BnCameraServiceListener.h
frameworks/av/camera/aidl/android/hardware/ICameraServiceListener.cpp

android/hardware/ICameraServiceProxy.h
android/hardware/BnCameraServiceProxy.h
frameworks/av/camera/aidl/android/hardware/ICameraServiceProxy.cpp

android/hardware/camera2/ICameraDeviceUser.h
android/hardware/camera2/BnCameraDeviceUser.h
frameworks/av/camera/aidl/android/hardware/camera2/ICameraDeviceUser.cpp

android/hardware/camera2/ICameraDeviceCallbacks.h
android/hardware/camera2/BnCameraDeviceCallbacks.h
frameworks/av/camera/aidl/android/hardware/camera2/ICameraDeviceCallbacks.cpp

parcelable 类型文件

parcelable 类型文件一共有 9 个,它们都是手动编写的代码。

Java 文件目录为 frameworks/base/core/java/android/hardware/

1
2
3
4
5
6
7
8
9
10
// frameworks/base/core/java/android/hardware/
camera2/CaptureRequest.java
camera2/impl/CameraMetadataNative.java
camera2/impl/CaptureResultExtras.java
camera2/params/OutputConfiguration.java
camera2/params/VendorTagDescriptor.java
camera2/params/VendorTagDescriptorCache.java
camera2/utils/SubmitInfo.java
CameraInfo.java
CameraStatus.java

.h, .cpp 文件并不一定是和 aidl 文件名称一一对应的,而是在 aidl 文件中定义的,比如 CameraStatus.aidl 定义如下:

1
2
3
4
package android.hardware;

/** @hide */
parcelable CameraStatus cpp_header "camera/CameraBase.h";

parcelable 类型的 aidl 文件对应的 .h, .cpp 文件目录为 frameworks/av/camera ,对应关系整理如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
// .h, .cpp 文件目录 frameworks/av/camera
// CaptureRequest.aidl
include/camera/camera2/CaptureRequest.h
camera2/CaptureRequest.cpp

// CameraMetadataNative.aidl
include/camera/CameraMetadata.h
CameraMetadata.cpp

// CaptureResultExtras.aidl
include/camera/CaptureResult.h
CaptureResult.cpp

// OutputConfiguration.aidl
include/camera/camera2/OutputConfiguration.h
camera2/OutputConfiguration.cpp

// VendorTagDescriptor.aidl 和 VendorTagDescriptorCache.aidl
include/camera/VendorTagDescriptor.h
VendorTagDescriptor.cpp

// SubmitInfo.aidl
include/camera/camera2/SubmitInfo.h
camera2/SubmitInfo.cpp

// CameraInfo.aidl 和 CameraStatus.aidl
include/camera/CameraBase.h
CameraBase.cpp

ICameraService 相关

分为客户端向服务端的请求 ICameraService.aidl 和客户端监听服务端的变化 ICameraServiceListener.aidl 。这两个 AIDL 是在 CameraService.cpp 中实现对应功能的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
// 1. ICameraService.aidl
interface ICameraService
{
...
const int CAMERA_TYPE_BACKWARD_COMPATIBLE = 0;
const int CAMERA_TYPE_ALL = 1;

// 返回指定类型的相机设备数量
int getNumberOfCameras(int type);
// 根据 id 返回当前相机设备信息
CameraInfo getCameraInfo(int cameraId);

...
const int CAMERA_HAL_API_VERSION_UNSPECIFIED = -1;
// api1 + hal1
ICamera connect(ICameraClient client,
int cameraId,
String opPackageName,
int clientUid, int clientPid);

// api2 + hal3
ICameraDeviceUser connectDevice(ICameraDeviceCallbacks callbacks,
String cameraId,
String opPackageName,
int clientUid);

// api1 + 指定 hal 版本(通常为 hal1)
ICamera connectLegacy(ICameraClient client,
int cameraId,
int halVersion,
String opPackageName,
int clientUid);

// 添加和移除 ICameraServiceListener 监听
CameraStatus[] addListener(ICameraServiceListener listener);
void removeListener(ICameraServiceListener listener);

// 根据 id 返回相机支持的属性
CameraMetadataNative getCameraCharacteristics(String cameraId);

// 获取 vendor tag
VendorTagDescriptor getCameraVendorTagDescriptor();
VendorTagDescriptorCache getCameraVendorTagCache();

// camera api 1 获取参数信息
String getLegacyParameters(int cameraId);

const int API_VERSION_1 = 1;
const int API_VERSION_2 = 2;
// 指定 id 支持的 API 版本
boolean supportsCameraApi(String cameraId, int apiVersion);
// 指定 id 设置手电筒模式
void setTorchMode(String cameraId, boolean enabled,
IBinder clientBinder);

// 服务端向系统打印系统消息
const int EVENT_NONE = 0;
const int EVENT_USER_SWITCHED = 1;
oneway void notifySystemEvent(int eventId, in int[] args);
}

// 2. ICameraServiceListener.aidl
interface ICameraServiceListener
{
const int STATUS_NOT_PRESENT = 0;
const int STATUS_PRESENT = 1;
const int STATUS_ENUMERATING = 2;
const int STATUS_NOT_AVAILABLE = -2;
const int STATUS_UNKNOWN = -1;
// 相机设备状态变化事件
oneway void onStatusChanged(int status, String cameraId);

const int TORCH_STATUS_NOT_AVAILABLE = 0;
const int TORCH_STATUS_AVAILABLE_OFF = 1;
const int TORCH_STATUS_AVAILABLE_ON = 2;
const int TORCH_STATUS_UNKNOWN = -1;
// 手电筒状态变化事件
oneway void onTorchStatusChanged(int status, String cameraId);
}

ICameraServiceProxy.aidl 文件

CameraServiceProxy 服务是在 Java 层注册的:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
interface ICameraServiceProxy
{
// CameraService 向代理服务发送消息,通知用户更新
oneway void pingForUserUpdate();

const int CAMERA_STATE_OPEN = 0;
const int CAMERA_STATE_ACTIVE = 1;
const int CAMERA_STATE_IDLE = 2;
const int CAMERA_STATE_CLOSED = 3;
const int CAMERA_FACING_BACK = 0;
const int CAMERA_FACING_FRONT = 1;
const int CAMERA_FACING_EXTERNAL = 2;

// CameraService 向代理服务发送消息,通知相机设备状态更新
oneway void notifyCameraState(String cameraId, int facing,
int newCameraState, String clientName);
}

ICamera 相关

Camera API1 才会使用到,分为 ICamera.aidl, ICameraClient.aidl
它们的代码是手动实现的,参考:CameraClient.h/cpp, Camera.h/cpp

ICameraDevice 相关

Camera API2 才会使用到,分为客户端向服务端的请求 ICameraDeviceUser.aidl 和服务端发给客户端的回调 ICameraDeviceCallbacks.aidl
表示相机设备具备的能力,能够提供的函数;这两个 AIDL 是在 CameraDeviceClient 中实现对应功能的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
// 1. ICameraDeviceUser.aidl 
interface ICameraDeviceUser
{
void disconnect();

const int NO_IN_FLIGHT_REPEATING_FRAMES = -1;
// 向设备提交捕获请求
SubmitInfo submitRequest(in CaptureRequest request, boolean streaming);
SubmitInfo submitRequestList(in CaptureRequest[] requestList,
boolean streaming);
// 取消置顶 id 的重复请求,并返回上次请求的帧 id
long cancelRequest(int requestId);

const int NORMAL_MODE = 0;
const int CONSTRAINED_HIGH_SPEED_MODE = 1;
const int VENDOR_MODE_START = 0x8000;

// 在流处理前执行配置请求
void beginConfigure();
// 根据指定输出配置,创建流
int createStream(in OutputConfiguration outputConfiguration);
void endConfigure(int operatingMode);
void deleteStream(int streamId);

// 创建输入流,返回流 id
int createInputStream(int width, int height, int format);
// 返回输入流的 Surface
Surface getInputSurface();

// Keep in sync with public API in
// frameworks/base/core/java/android/hardware/camera2/CameraDevice.java
const int TEMPLATE_PREVIEW = 1;
const int TEMPLATE_STILL_CAPTURE = 2;
const int TEMPLATE_RECORD = 3;
const int TEMPLATE_VIDEO_SNAPSHOT = 4;
const int TEMPLATE_ZERO_SHUTTER_LAG = 5;
const int TEMPLATE_MANUAL = 6;
// 根据模板创建默认请求,返回相机参数信息
CameraMetadataNative createDefaultRequest(int templateId);
// 获取相机参数信息
CameraMetadataNative getCameraInfo();
void waitUntilIdle();
long flush();
void prepare(int streamId);
void tearDown(int streamId);
void prepare2(int maxCount, int streamId);
void finalizeOutputConfigurations(int streamId,
in OutputConfiguration outputConfiguration);
}

// 2. ICameraDeviceCallbacks.aidl
interface ICameraDeviceCallbacks
{
...
oneway void onDeviceError(int errorCode,
in CaptureResultExtras resultExtras);
oneway void onDeviceIdle();
oneway void onCaptureStarted(in CaptureResultExtras resultExtras,
long timestamp);
oneway void onResultReceived(in CameraMetadataNative result,
in CaptureResultExtras resultExtras);
oneway void onPrepared(int streamId);
// 重复请求引起的错误回调
oneway void onRepeatingRequestError(in long lastFrameNumber,
in int repeatingRequestId);
oneway void onRequestQueueEmpty();
}

Services 目录下的文件介绍

frameworks/av/services/camera/libcameraservice
AOSP 中这个目录下是 87 个文件,而 Qcom 的基线中增加了 27 个文件,分别为 api1/qticlient2 目录下的 25 个文件,以及 QTICamera2Client.cpp, QTICamera2Client.h 两个文件。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
.
├── Android.mk
├── api1
│   ├── client2
│   └── qticlient2
├── api2
├── CameraFlashlight.cpp
├── CameraFlashlight.h
├── CameraService.cpp
├── CameraService.h
├── common
├── device1
├── device3
├── gui
├── MODULE_LICENSE_APACHE2
├── NOTICE
├── tests
└── utils

从目录结构上可以看出,API1/2HAL1/3 就是在这一层体现的。

API1/API2

APP Java 客户端调用服务端方法时,Camera API1/2 接口对应功能都是在 CameraService 中实现的,而这里的 API1/2 目录对应的就是对上层不同版本接口的处理。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
api1
├── Camera2Client.cpp
├── Camera2Client.h
├── CameraClient.cpp
├── CameraClient.h
├── client2
│   ├── CallbackProcessor.cpp
│   ├── CallbackProcessor.h
│   ├── Camera2Heap.h
│   ├── CaptureSequencer.cpp
│   ├── CaptureSequencer.h
│   ├── FrameProcessor.cpp
│   ├── FrameProcessor.h
│   ├── JpegCompressor.cpp
│   ├── JpegCompressor.h
│   ├── JpegProcessor.cpp
│   ├── JpegProcessor.h
│   ├── Parameters.cpp
│   ├── Parameters.h
│   ├── StreamingProcessor.cpp
│   ├── StreamingProcessor.h
│   ├── ZslProcessor.cpp
│   └── ZslProcessor.h
├── QTICamera2Client.cpp
├── QTICamera2Client.h
└── qticlient2
├── CallbackProcessor.cpp
├── CallbackProcessor.h
├── Camera2Heap.h
├── CaptureSequencer.cpp
├── CaptureSequencer.h
├── FrameProcessor.cpp
├── FrameProcessor.h
├── JpegCompressor.cpp
├── JpegCompressor.h
├── JpegProcessor.cpp
├── JpegProcessor.h
├── Parameters.cpp
├── Parameters.h
├── QTICaptureSequencer.cpp
├── QTICaptureSequencer.h
├── QTIFrameProcessor.cpp
├── QTIFrameProcessor.h
├── QTIParameters.cpp
├── QTIParameters.h
├── RawProcessor.cpp
├── RawProcessor.h
├── StreamingProcessor.cpp
├── StreamingProcessor.h
├── ZslProcessor.cpp
└── ZslProcessor.h
api2
├── CameraDeviceClient.cpp
└── CameraDeviceClient.h

BasicClient 有三个重要的子类:

  • CameraClient
    如果平台仅支持 HAL 1,即 CAMERA_DEVICE_API_VERSION_1_0 ;使用 API 1/2 + HAL 1 都会对应该客户端。
  • Camera2Client
    如果平台支持 HAL 3 ,即 CAMERA_DEVICE_API_VERSION_3_0 及以上版本;使用 API 1 + HAL 3 对应的客户端。Camera2Client 会将 API1 中的接口转换为 API2 中对应的功能。
  • CameraDeviceClient
    如果平台支持 HAL 3 ,使用 API 2 + HAL 3 对应的客户端。

平台仅支持 HAL 1 时,API 2openCamera 时,通过 CameraDeviceUserShimAPI 2 转换为 API 1 ,即 HAL 1 + API 1 向下发起请求。
LegacyCameraDevice 会将 CAMERA API2 转换为 CAMERA API1 ,而 CameraDeviceUserShim 封装了 LegacyCameraDevice

QTICamera2Client

Qcom 的基线中增加了 27 个文件,分别为 api1/qticlient2 目录下的 25 个文件,以及 QTICamera2Client.cpp, QTICamera2Client.h 两个文件。
QTICamera2Client 是高通针对 API1 做的优化?在什么情况下会转换为 QTICamera2Client 呢?看如下源码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
// 1. Camera2Client.h
class Camera2Client :
public Camera2ClientBase<CameraService::Client>
{
#ifdef USE_QTI_CAMERA2CLIENT
friend class QTICamera2Client;
#endif
...
#ifdef USE_QTI_CAMERA2CLIENT
sp<camera2::RawProcessor> mRawProcessor;
#endif
...
#ifdef USE_QTI_CAMERA2CLIENT
sp<QTICamera2Client> mQTICamera2Client;
#endif
...
}

// 2. Camera2Client.cpp
template<typename TProviderPtr>
status_t Camera2Client::initializeImpl(TProviderPtr providerPtr)
{
...
#ifdef USE_QTI_CAMERA2CLIENT
mQTICamera2Client = new QTICamera2Client(this);
#endif
...
#ifdef USE_QTI_CAMERA2CLIENT
mRawProcessor = new RawProcessor(this, mCaptureSequencer);
threadName = String8::format("C2-%d-RawProc", mCameraId);
mRawProcessor->run(threadName.string());
#endif
...
}

QTICamera2Client 是高通对 API 1Camera2Client 做的一层封装,添加了部分功能,主要是向上提供 raw 数据。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// 1. QTICamera2Client.h
class QTICamera2Client: public virtual RefBase{
private:
wp<Camera2Client> mParentClient;
status_t stopPreviewExtn();

public:
QTICamera2Client(sp<Camera2Client> client);
~QTICamera2Client();
...
}

// 2. QTICamera2Client.cpp
QTICamera2Client::QTICamera2Client(sp<Camera2Client> client):
mParentClient(client) {
}

device1/device3

device1/device3 可以理解为 Framework 层对应 HAL 层的 HAL 1/3

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
device1
├── CameraHardwareInterface.cpp
└── CameraHardwareInterface.h
device3
├── Camera3BufferManager.cpp
├── Camera3BufferManager.h
├── Camera3Device.cpp
├── Camera3Device.h
├── Camera3DummyStream.cpp
├── Camera3DummyStream.h
├── Camera3InputStream.cpp
├── Camera3InputStream.h
├── Camera3IOStreamBase.cpp
├── Camera3IOStreamBase.h
├── Camera3OutputStream.cpp
├── Camera3OutputStream.h
├── Camera3OutputStreamInterface.h
├── Camera3SharedOutputStream.cpp
├── Camera3SharedOutputStream.h
├── Camera3StreamBufferFreedListener.h
├── Camera3StreamBufferListener.h
├── Camera3Stream.cpp
├── Camera3Stream.h
├── Camera3StreamInterface.h
├── Camera3StreamSplitter.cpp
├── Camera3StreamSplitter.h
├── StatusTracker.cpp
└── StatusTracker.h
  • API1/device1/HAL1 的连接过程
1
2
3
4
5
// API1: CameraClient.h
sp<CameraHardwareInterface> mHardware;
// device1: CameraHardwareInterface.h
sp<hardware::camera::device::V1_0::ICameraDevice> mHidlDevice;
// 这里的 ICameraDevice 即为 HAL1

API1 的客户端 CameraClient 对应的 device1: CameraHardwareInterface,而它直接包含了 HAL1ICameraDevice

  • API1/3/device3/HAL3 的连接过程
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    // API1: Camera2Client.h
    class Camera2Client :
    public Camera2ClientBase<CameraService::Client>{...}

    // API2: CameraDeviceClient.h
    class CameraDeviceClient :
    public Camera2ClientBase<CameraDeviceClientBase>,
    public camera2::FrameProcessorBase::FilteredListener{...}

    // Camera2ClientBase.h
    sp<CameraDeviceBase> mDevice;

    // Camera2ClientBase.cpp
    template <typename TClientBase>
    Camera2ClientBase<TClientBase>::Camera2ClientBase(
    const sp<CameraService>& cameraService,
    const sp<TCamCallbacks>& remoteCallback,
    const String16& clientPackageName,
    const String8& cameraId,
    int cameraFacing,
    int clientPid,
    uid_t clientUid,
    int servicePid):
    TClientBase(cameraService, remoteCallback, clientPackageName,
    cameraId, cameraFacing, clientPid, clientUid, servicePid),
    mSharedCameraCallbacks(remoteCallback),
    mDeviceVersion(cameraService->getDeviceVersion(
    TClientBase::mCameraIdStr)),
    mDeviceActive(false)
    {
    ...
    mInitialClientPid = clientPid;
    // 只要是 HAL3 ,则 device 都是对应的 Camera3Device
    mDevice = new Camera3Device(cameraId);
    ...
    }

从源码可以看出,不管是 API1/2 ,只要是 HAL 3Camera2Client, CameraDeviceClient 两个客户端对应的都是 device3: Camera3Device

Camera3Device::HalInterface 内部类,用于和 HAL 层通信,实现了 HALICameraDeviceSession.hal 部分代码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// Camera3Device.h
class Camera3Device :
...
class HalInterface : public camera3::Camera3StreamBufferFreedListener {
public:
...
// Calls into the HAL interface

// Caller takes ownership of requestTemplate
status_t constructDefaultRequestSettings(
camera3_request_template_t templateId,
/*out*/ camera_metadata_t **requestTemplate);
status_t configureStreams(
/*inout*/ camera3_stream_configuration *config);
status_t processCaptureRequest(camera3_capture_request_t *request);
status_t flush();
status_t close();
...
}
...
}

cameraserver 进程

cameraserver 进程的源码在 frameworks/av/camera/cameraserver 目录下,该目录只有三个文件:

1
2
3
4
.
├── Android.mk
├── cameraserver.rc // rc 文件
└── main_cameraserver.cpp // 主进程

cameraserver 进程在启动时,做了三件事:

  • 设置 Socket 通信时,对端关闭读取时进程不退出,返回错误信息(Socket 用在了哪?)
  • HIDL 通信初始化
  • Native Binder 初始化,CameraServiceservice_manager 注册服务
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
// frameworks/av/camera/cameraserver/main_cameraserver.cpp
int main(int argc __unused, char** argv __unused)
{
// 1. Socket 通信时,对端关闭读取时进程不退出,返回错误信息
signal(SIGPIPE, SIG_IGN);

// 2. HIDL 通信初始化
// Set 3 threads for HIDL calls
hardware::configureRpcThreadpool(3, /*willjoin*/ false);

// 3. Native Binder 初始化,CameraService 是具体的服务
sp<ProcessState> proc(ProcessState::self());
sp<IServiceManager> sm = defaultServiceManager();
ALOGI("ServiceManager: %p", sm.get());
CameraService::instantiate();
ProcessState::self()->startThreadPool();
IPCThreadState::self()->joinThreadPool();
}
1
2
3
4
5
6
7
8
9
10
// init 进程启动名字为 cameraserver 的进程及对应路径
service cameraserver /system/bin/cameraserver
// class 表示类别,同一类别的进程同时启动
class main
// 用户名
user cameraserver
// 分组
group audio camera input drmrpc
ioprio rt 4
writepid /dev/cpuset/camera-daemon/tasks /dev/stune/top-app/tasks

CameraService 启动服务注册流程图:
0109-android-camera-4-framework-CameraService-addService.png

CameraService 服务

CameraService 服务的名称为:media.camera ,主要有两个功能:

  • 作为服务端
    实现 AIDL 对应功能,当 API1/2 客户端发出请求后,作为服务端响应并处理这些功能。
  • 作为客户端
    实现 HIDL 回调,用于响应 HAL 层发回的回调。并且通过 CameraProviderManagerHAL 层实现双向通信。

服务名称

CameraService 继承了 BinderService<CameraService> ,将 CameraService::instantiate(); 代码展开:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
// BinderService.h
template<typename SERVICE>
class BinderService
{
public:
static status_t publish(bool allowIsolated = false) {
sp<IServiceManager> sm(defaultServiceManager());
return sm->addService(
String16(SERVICE::getServiceName()),
new SERVICE(), allowIsolated);
}
...
static void instantiate() { publish(); }
...
}

// IServiceManager.h
class IServiceManager : public IInterface
{
public:
...
virtual status_t addService( const String16& name,
const sp<IBinder>& service,
bool allowIsolated = false) = 0;
...
}

// CameraService.h
class CameraService :
public BinderService<CameraService>,
public virtual ::android::hardware::BnCameraService,
public virtual IBinder::DeathRecipient,
public camera_module_callbacks_t,
public virtual CameraProviderManager::StatusListener
{
...
// Implementation of BinderService<T>
static char const* getServiceName() { return "media.camera"; }
...
}

从继承关系及 CameraService.h 源码,getServiceName 设置了 CameraService 服务的名称为 media.camera

注册流程图

CameraService 注册流程,查看原图

0109-android-camera-4-framework-CameraService-init.jpg

源码分析

先来看 CameraService.h 头文件相关定义:

1
2
3
4
5
6
7
8
9
10
11
// CameraService.h
class CameraService :
public BinderService<CameraService>,
public virtual ::android::hardware::BnCameraService,
public virtual IBinder::DeathRecipient,
public camera_module_callbacks_t,
public virtual CameraProviderManager::StatusListener
{

static char const* getServiceName() { return "media.camera"; }
}
  • BinderService
    继承了 BinderService ,用于注册服务。服务名称为 media.camera
  • camera_module_callbacks_t
    继承了 camera_module_callbacks_t ,它是在 HAL 中定义的,用于 HALFramework 发送通知。
  • StatusListener
    继承了 StatusListener ,它是在 CameraProviderManager.h 中定义的,用于 CameraProviderManagerCameraService 发送通知。

现在查看 CameraService 的构造方法,因为在注册服务时 BinderService 会对 CameraService 强指针引用,所以会调用对应函数 onFirstRef

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
// CameraService.cpp
CameraService::CameraService() :
mEventLog(DEFAULT_EVENT_LOG_LENGTH),
mNumberOfCameras(0), mNumberOfNormalCameras(0),
mSoundRef(0), mInitialized(false) {
// camera_module_callbacks_t 结构体的函数指针赋值
this->camera_device_status_change =
android::camera_device_status_change;
this->torch_mode_status_change = android::torch_mode_status_change;
...
}

void CameraService::onFirstRef()
{
...
BatteryNotifier& notifier(BatteryNotifier::getInstance());
notifier.noteResetCamera();
notifier.noteResetFlashlight();

status_t res = INVALID_OPERATION;

// 实例化 CameraProviderManager ,并连接 Hardware
res = enumerateProviders();
if (res == OK) {
mInitialized = true;
}

// CameraServiceProxy 服务是 Java 代码注册的
// 但是 CameraService 启动时间很早,CameraServiceProxy 可能还并没有注册
// 实际调试结果也是,这段代码实际不会调用 CameraServiceProxy 对应方法
CameraService::pingCameraServiceProxy();
}

构造函数中非常简单,仅仅是将 camera_module_callbacks_t 结构体的函数指针赋值;在 onFirstRef 中,主要通过 enumerateProviders 来实例化对应的 CameraProviderManager 并连接 HAL ,最后去 ping 一次 CameraServiceProxy 代理服务,实际上是 ping 不通的,因为 CameraService.cpp 一定是比 CameraServiceProxy.java 启动的早。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
// CameraService.cpp
status_t CameraService::enumerateProviders() {
...
// 实例化 CameraProviderManager
if (nullptr == mCameraProviderManager.get()) {
mCameraProviderManager = new CameraProviderManager();
res = mCameraProviderManager->initialize(this);
...
}

mNumberOfCameras = mCameraProviderManager->getCameraCount();
mNumberOfNormalCameras =
mCameraProviderManager->getAPI1CompatibleCameraCount();
mCameraProviderManager->setUpVendorTags();

if (nullptr == mFlashlight.get()) {
mFlashlight = new CameraFlashlight(mCameraProviderManager, this);
}
res = mFlashlight->findFlashUnits();
...
for (auto& cameraId : mCameraProviderManager->getCameraDeviceIds()){
...
onDeviceStatusChanged(id8, CameraDeviceStatus::PRESENT);
...
}

return OK;
}

如果 mCameraProviderManager 为空,则实例化并调用 initialize ;接着实例化 CameraFlashlight ;先看头文件 CameraProviderManager.h 中定义的几个重要数据结构和函数:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
// CameraProviderManager.h
class CameraProviderManager :
virtual public hidl::manager::V1_0::IServiceNotification {
public:
...
// 定义纯虚函数
struct ServiceInteractionProxy {
virtual bool registerForNotifications(
const std::string &serviceName,
const sp<hidl::manager::V1_0::IServiceNotification>
&notification) = 0;
virtual sp<hardware::camera::provider::V2_4::ICameraProvider>
getService(const std::string &serviceName) = 0;
virtual ~ServiceInteractionProxy() {}
};

// 调用 ICameraProvider 实现这些方法
struct HardwareServiceInteractionProxy :
public ServiceInteractionProxy {
virtual bool registerForNotifications(
const std::string &serviceName,
const sp<hidl::manager::V1_0::IServiceNotification>
&notification) override {
return hardware::camera::provider::V2_4::
ICameraProvider::registerForNotifications(
serviceName, notification);
}
virtual sp<hardware::camera::provider::V2_4::ICameraProvider>
getService(const std::string &serviceName) override {
return hardware::camera::provider::V2_4::
ICameraProvider::getService(serviceName);
}
};

struct StatusListener : virtual public RefBase {
~StatusListener() {}
virtual void onDeviceStatusChanged(const String8 &cameraId,
hardware::camera::common::V1_0::CameraDeviceStatus
newStatus) = 0;
virtual void onTorchStatusChanged(const String8 &cameraId,
hardware::camera::common::V1_0::TorchModeStatus
newStatus) = 0;
virtual void onNewProviderRegistered() = 0;
};

virtual hardware::Return<void> onRegistration(
const hardware::hidl_string& fqName,
const hardware::hidl_string& name,
bool preexisting) override;

status_t initialize(wp<StatusListener> listener,
ServiceInteractionProxy *proxy
= &sHardwareServiceInteractionProxy);

private:
static HardwareServiceInteractionProxy sHardwareServiceInteractionProxy;
  • ServiceInteractionProxy
    定义了几个纯虚函数,用于向 HAL 系统服务中注册 registerForNotifications 监听 ICameraProvider.hal 的消息;getService 返回 ICameraProvider 的实例。
  • HardwareServiceInteractionProxy
    ServiceInteractionProxy 的实现结构体,具体调用 ICameraProvider 对应的 registerForNotifications, getService ;也就是 CameraProviderManager 持有 ICameraProvider 的远程实例。
  • onRegistration
    registerForNotifications 的回调函数,注册成功后回调。
  • StatusListener
    状态监听接口,这些接口是在 CameraService 中实现的;用于 CameraProviderManager 回调 CameraService
  • sHardwareServiceInteractionProxy
    静态变量,是初始化 initialize 函数形参 ServiceInteractionProxy 的默认值。

CameraService 中调用 CameraProviderManager::initialize 时,传入的是 CameraService 的实例,仅仅一个参数,所以 ServiceInteractionProxy 使用的是默认的 sHardwareServiceInteractionProxy 实例。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
// CameraProviderManager.cpp
// 实例化 HAL 代理
CameraProviderManager::HardwareServiceInteractionProxy
CameraProviderManager::sHardwareServiceInteractionProxy{};

status_t CameraProviderManager::initialize(
wp<CameraProviderManager::StatusListener> listener,
ServiceInteractionProxy* proxy) {
...
mListener = listener;
mServiceProxy = proxy;

bool success = mServiceProxy->registerForNotifications(
/* instance name, empty means no filter */ "",
this);
...

addProviderLocked(kLegacyProviderName, /*expected*/ false);
return OK;
}

CameraProviderManager::initialize 中主要是初始化赋值 mListener, mServiceProxy ,并通过 sHardwareServiceInteractionProxy->registerForNotificationsHIDL 服务管理注册了自己,最后调用 addProviderLocked

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
// CameraProviderManager.cpp
status_t CameraProviderManager::addProviderLocked(
const std::string& newProvider, bool expected) {
for (const auto& providerInfo : mProviders) {
if (providerInfo->mProviderName == newProvider) {
ALOGW(...);
return ALREADY_EXISTS;
}
}

// HIDL 通信,通过 ICameraProvider 和 HAL 层通信
sp<provider::V2_4::ICameraProvider> interface;
interface = mServiceProxy->getService(newProvider);

if (interface == nullptr) {
...
}

sp<ProviderInfo> providerInfo =
new ProviderInfo(newProvider, interface, this);
status_t res = providerInfo->initialize();
if (res != OK) {
return res;
}

mProviders.push_back(providerInfo);

return OK;
}

addProviderLocked 中有如下信息:

  • 通过代理获取 ICameraProvider 实例,用于和 HAL 通信
  • 新建 ProviderInfo 并初始化,保存 ICameraProvider 实例
  • mProviders 保存所有的 ProviderInfo (实测只有一个实例元素,名称为 legacy/0
1
2
3
4
5
6
7
8
9
10
11
12
// CameraProviderManager.h
struct ProviderInfo :
virtual public
hardware::camera::provider::V2_4::ICameraProviderCallback,
virtual public hardware::hidl_death_recipient
{
const std::string mProviderName;
const sp<hardware::camera::provider::V2_4::ICameraProvider>
mInterface;
const metadata_vendor_id_t mProviderTagid;
...
}

ProviderInfo 继承了 ICameraProviderCallback, hidl_death_recipient ,它会处理来着 ICameraProvider 的回调。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
// CameraProviderManager.cpp
CameraProviderManager::ProviderInfo::ProviderInfo(
const std::string &providerName,
sp<provider::V2_4::ICameraProvider>& interface,
CameraProviderManager *manager) :
mProviderName(providerName),
mInterface(interface),
mProviderTagid(generateVendorTagId(providerName)),
mUniqueDeviceCount(0),
mManager(manager) {
(void) mManager;
}

status_t CameraProviderManager::ProviderInfo::initialize() {
status_t res = parseProviderName(mProviderName, &mType, &mId);
...
// 设置回调
hardware::Return<Status> status = mInterface->setCallback(this);
...
// HIDL 连接
hardware::Return<bool> linked =
mInterface->linkToDeath(this, /*cookie*/ mId);
...
// Get initial list of camera devices, if any
std::vector<std::string> devices;
// 获取 CameraIdList ,实际是获取的一组设备名
hardware::Return<void> ret =
mInterface->getCameraIdList([&status, &devices](
Status idStatus,
const hardware::hidl_vec<hardware::hidl_string>&
cameraDeviceNames) {
status = idStatus;
if (status == Status::OK) {
for (size_t i = 0; i < cameraDeviceNames.size(); i++) {
devices.push_back(cameraDeviceNames[i]);
}
} });
...
for (auto& device : devices) {
std::string id;
// 添加从 HAL 返回的每个设备名
status_t res = addDevice(device,
hardware::camera::common::V1_0::CameraDeviceStatus::PRESENT,
&id);
...
}
...
return OK;
}

ProviderInfo::initialize 初始化,主要是从 HAL 获取设备名后,添加具体的设备信息。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
// CameraProviderManager.h
struct DeviceInfo {
const std::string mName; // Full instance name
const std::string mId; // ID section of full name
const hardware::hidl_version mVersion;
const metadata_vendor_id_t mProviderTagid;
...
protected:
bool mHasFlashUnit;
template<class InterfaceT>
static status_t setTorchMode(InterfaceT& interface, bool enabled);
};

// HALv1-specific camera fields, including the actual device interface
struct DeviceInfo1 : public DeviceInfo {
typedef hardware::camera::device::V1_0::ICameraDevice InterfaceT;
const sp<InterfaceT> mInterface;
...
private:
CameraParameters2 mDefaultParameters;
};

// HALv3-specific camera fields, including the actual device interface
struct DeviceInfo3 : public DeviceInfo {
typedef hardware::camera::device::V3_2::ICameraDevice InterfaceT;
const sp<InterfaceT> mInterface;
...
private:
CameraMetadata mCameraCharacteristics;
};

头文件中可以看出,DeviceInfo 有两个子类,分别对应 HAL 1HAL 3 ,并将具体的 ICameraDevice 版本保存到 mInterface 中;所以设备添加时也会根据不同版本分别添加:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
// CameraProviderManager.cpp
status_t CameraProviderManager::ProviderInfo::addDevice(
const std::string& name,
CameraDeviceStatus initialStatus,
/*out*/ std::string* parsedId) {
...
status_t res = parseDeviceName(name, &major, &minor, &type, &id);
...
std::unique_ptr<DeviceInfo> deviceInfo;
switch (major) {
case 1:
deviceInfo = initializeDeviceInfo<DeviceInfo1>(
name, mProviderTagid, id, minor);
break;
case 3:
deviceInfo = initializeDeviceInfo<DeviceInfo3>(
name, mProviderTagid, id, minor);
break;
default:
ALOGE(...);
return BAD_VALUE;
}
...
return OK;
}

template<class DeviceInfoT>
std::unique_ptr<CameraProviderManager::ProviderInfo::DeviceInfo>
CameraProviderManager::ProviderInfo::initializeDeviceInfo(
const std::string &name, const metadata_vendor_id_t tagId,
const std::string &id, uint16_t minorVersion) const {
Status status;

auto cameraInterface =
getDeviceInterface<typename DeviceInfoT::InterfaceT>(name);
if (cameraInterface == nullptr) return nullptr;

CameraResourceCost resourceCost;
cameraInterface->getResourceCost([&status, &resourceCost](
Status s, CameraResourceCost cost) {
status = s;
resourceCost = cost;
});
...
return std::unique_ptr<DeviceInfo>(
new DeviceInfoT(name, tagId, id, minorVersion, resourceCost,
cameraInterface));
}

根据传入的 deviceName 解析版本号、类型、设备 Id (前后摄),并根据 major 版本号(表示 HAL 1 或者 HAL 3) 分别初始化对应的 DeviceInfo ;在 initializeDeviceInfo 中通过 getDeviceInterface 获取对应的 ICameraDevice 版本,在对应版本 DeviceInfo 实例化时保存;也就是将 DeviceInfoHAL 层的 ICameraDevice 绑定。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
// CameraProviderManager.cpp
CameraProviderManager::ProviderInfo::DeviceInfo3::DeviceInfo3(
const std::string& name,
const metadata_vendor_id_t tagId, const std::string &id,
uint16_t minorVersion,
const CameraResourceCost& resourceCost,
sp<InterfaceT> interface) :
DeviceInfo(name, tagId, id,
hardware::hidl_version{3, minorVersion}, resourceCost),
mInterface(interface) {
// Get camera characteristics and initialize flash unit availability
Status status;
hardware::Return<void> ret;
// 获取 Camera 设备配置信息
ret = mInterface->getCameraCharacteristics([&status, this](Status s,
device::V3_2::CameraMetadata metadata) {
status = s;
if (s == Status::OK) {
camera_metadata_t *buffer =
reinterpret_cast<camera_metadata_t*>(metadata.data());
size_t expectedSize = metadata.size();
int res = validate_camera_metadata_structure(buffer,
&expectedSize);
if (res==OK||res==CAMERA_METADATA_VALIDATION_SHIFTED) {
set_camera_metadata_vendor_id(buffer, mProviderTagid);
mCameraCharacteristics = buffer;
} else {
ALOGE(...);
status = Status::INTERNAL_ERROR;
}
}
});
...
camera_metadata_entry flashAvailable =
mCameraCharacteristics.find(ANDROID_FLASH_INFO_AVAILABLE);
if (flashAvailable.count == 1 &&
flashAvailable.data.u8[0] == ANDROID_FLASH_INFO_AVAILABLE_TRUE) {
mHasFlashUnit = true;
} else {
mHasFlashUnit = false;
}
}

这里分析的是 DeviceInfo3 的构造函数,它会向 HAL 层请求当前设备的配置信息,并保存 mCameraCharacteristics ,后续查看属性时都会通过这个变量查询。
CameraService::enumerateProviders 中,首先新建并初始化 CameraProviderManager ,其持有和 HAL 通信的实例;接着新建并初始化 CameraFlashlight ,用于控制闪光灯。先看头文件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
// CameraFlashlight.h
class FlashControlBase : public virtual VirtualLightRefBase {
public:
...
virtual status_t hasFlashUnit(const String8& cameraId,
bool *hasFlash) = 0;
virtual status_t setTorchMode(const String8& cameraId,
bool enabled) = 0;
};

// HAL 3 闪光灯控制
class ProviderFlashControl : public FlashControlBase {
public:
...
// FlashControlBase
status_t hasFlashUnit(const String8& cameraId, bool *hasFlash);
status_t setTorchMode(const String8& cameraId, bool enabled);
private:
sp<CameraProviderManager> mProviderManager;
...
};

// HAL 1 闪光灯控制,通过 CameraHardwareInterface 向下调用
class CameraHardwareInterfaceFlashControl : public FlashControlBase {
public:
...
// FlashControlBase
status_t setTorchMode(const String8& cameraId, bool enabled);
status_t hasFlashUnit(const String8& cameraId, bool *hasFlash);
private:
sp<CameraProviderManager> mProviderManager;
const camera_module_callbacks_t *mCallbacks;
sp<CameraHardwareInterface> mDevice;
String8 mCameraId;
CameraParameters mParameters;
...
}

class CameraFlashlight : public virtual VirtualLightRefBase {
public:
...
bool hasFlashUnit(const String8& cameraId);
status_t setTorchMode(const String8& cameraId, bool enabled);
private:
sp<FlashControlBase> mFlashControl;
sp<CameraProviderManager> mProviderManager;
const camera_module_callbacks_t *mCallbacks;

头文件定义的几个信息:

  • CameraHardwareInterfaceFlashControl
    HAL 1 闪光灯控制类,通过 CameraHardwareInterface 向下调用。
  • ProviderFlashControl
    HAL 3 闪光灯控制类。
  • FlashControlBase
    基类。
  • CameraProviderManager
    主要用于 ProviderFlashControl 向下发送信息。
  • camera_module_callbacks_t
    HAL 层的回调。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
// CameraFlashlight.cpp
CameraFlashlight::CameraFlashlight(
sp<CameraProviderManager> providerManager,
camera_module_callbacks_t* callbacks) :
mProviderManager(providerManager),
mCallbacks(callbacks),
mFlashlightMapInitialized(false) {
}

status_t CameraFlashlight::findFlashUnits() {
...
mFlashControl.clear();
for (auto &id : cameraIds) {
ssize_t index = mHasFlashlightMap.indexOfKey(id);
if (0 <= index) {
continue;
}

bool hasFlash = false;
res = createFlashlightControl(id);
...
}
...
return OK;
}

status_t CameraFlashlight::createFlashlightControl(
const String8& cameraId) {
...
if (mProviderManager->supportSetTorchMode(cameraId.string())) {
mFlashControl = new ProviderFlashControl(mProviderManager);
} else {
// Only HAL1 devices do not support setTorchMode
mFlashControl =
new CameraHardwareInterfaceFlashControl(mProviderManager,
*mCallbacks);
}
return OK;
}

CameraFlashlight 的构造函数仅仅初始化了几个本地变量,CameraService 中调用 CameraFlashlight::findFlashUnits 时,会根据 HAL 1/3 分别来创建对应的闪光灯控制类。至此整个 CameraService 注册流程结束。

小结

CameraService 初始化和注册流程中,实例化了两个对象:

  • CameraProviderManager mCameraProviderManager 对象
  • Flashlight mFlashlight 对象

CameraProviderManager 初始化完后:

  • mProviders 保存了 ProviderInfo 对象;并关联了 ICameraProvider ,用于和 HAL 通信
  • ProviderInfomDevices 保存了所有的 DeviceInfo1, DeviceInfo3 设备信息,并关联 ICameraDevice 实例,用于直接通信
  • DeviceInfo1 中保存了 CameraParameters2 mDefaultParameters 参数信息
  • DeviceInfo3 中保存了 CameraMetadata mCameraCharacteristics 参数信息

CameraFlashlight 新建和初始化后:

  • 如果是 HAL 1 会实例化控制类 CameraHardwareInterfaceFlashControl
  • 如果是 HAL 3 会实例化控制类 ProviderFlashControl

Camera Open 流程

API

Camera API 2 开启摄像头设备时,通过 CameraManager.openCamera 来打开:

1
2
3
4
5
6
7
8
9
// CameraManager.java
@RequiresPermission(android.Manifest.permission.CAMERA)
public void openCamera(@NonNull String cameraId,
@NonNull final CameraDevice.StateCallback callback,
@Nullable Handler handler)
throws CameraAccessException {

openCameraForUid(cameraId, callback, handler, USE_CALLING_UID);
}
  • String cameraId
    表示前后摄的 ID ,通常 0 表示后摄。
  • CameraDevice.StateCallback callback
    打开设备时,状态回调接口。
  • Handler handler
    表示回调接口在哪个线程执行。

示例

打开一个设备,在回调中保存 CameraDevice

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
CameraDevice.StateCallback mCameraDeviceStateCallback 
= new CameraDevice.StateCallback() {
@Override
public void onOpened(@NonNull CameraDevice camera) {
mCameraDevice = camera;
//createCameraCaptureSession();
}

@Override
public void onDisconnected(@NonNull CameraDevice camera) {
camera.close();
mCameraDevice = null;
}

@Override
public void onError(@NonNull CameraDevice camera, int error) {
camera.close();
mCameraDevice = null;
}
};

try {
mCameraManager.openCamera(mCameraId, mCameraDeviceStateCallback,
mBackHandler);
} catch (CameraAccessException e) {
e.printStackTrace();
}

CameraDevice.StateCallback 接口

在打开设备时,会传入 StateCallback 回调接口,它有四个方法,都是在 CameraDeviceImpl 中回调的:

  • onOpened
    CameraManager.openCameraDeviceUserAsync 方法中,CameraDeviceImpl.setRemoteDevice(cameraUser); 会触发 StateCallback.onOpened 回调。
  • onClosed
    CameraDevice.close 是在 CameraDeviceImpl.close 中实现的,同时会触发 StateCallback.onClosed 回调。
  • onDisconnected
    CameraDeviceImpl.setRemoteDevice(cameraUser); 中如果远程连接断开,或者 ICameraDeviceCallbacks.onDeviceError 返回了 ERROR_CAMERA_DISCONNECTED 错误码,都会触发 StateCallback.onDisconnected 回调。
  • onError
    Binder 通信中绑定失败 binderDiedsetRemoteFailure 以及 ICameraDeviceCallbacks.onDeviceError 返回了 ERROR_CAMERA_DEVICE/ERROR_CAMERA_SERVICE 错误码,都会触发 StateCallback.onError 回调。

在设备打开时,会通过 StateCallback 回调返回打开状态,从代码可以看出,只要 ICameraService.connectDevice 成功后,直接调用 CameraDeviceImpl.setRemoteDevice(cameraUser); 来触发 StateCallback.onOpened ,表示设备打开成功。
StateCallbackJava 接口,它的 onDisconnected, onError 两个回调方法,需要真实的与物理设备交互;所以需要通过 ICameraDeviceCallbacks.aidlFramework Service 中获取真实的信息回调。

流程图

Camera API 2 开启相机设备流程图:

0109-android-camera-4-framework-Camera2-open.png

0109-android-camera-4-framework-Camera2-open-CameraService_connectDevice.jpg

源码分析

通过 CameraManager.openCamera 打开设备,我们重点分析如下代码,代码执行路径为 :
openCamera -> openCameraForUid -> openCameraDeviceUserAsync

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
// CameraManager.java
@RequiresPermission(android.Manifest.permission.CAMERA)
public void openCamera(@NonNull String cameraId,
@NonNull final CameraDevice.StateCallback callback,
@Nullable Handler handler)
throws CameraAccessException {

openCameraForUid(cameraId, callback, handler, USE_CALLING_UID);
}

private CameraDevice openCameraDeviceUserAsync(String cameraId,
CameraDevice.StateCallback callback,
Handler handler, final int uid)
throws CameraAccessException {
CameraCharacteristics characteristics =
getCameraCharacteristics(cameraId);
CameraDevice device = null;

synchronized (mLock) {
ICameraDeviceUser cameraUser = null;
// 新建 CameraDeviceImpl 实例
android.hardware.camera2.impl.CameraDeviceImpl deviceImpl =
new android.hardware.camera2.impl.CameraDeviceImpl(
cameraId,
callback,
handler,
characteristics,
mContext.getApplicationInfo().targetSdkVersion);
// 获取 ICameraDeviceCallbacks 回调
ICameraDeviceCallbacks callbacks = deviceImpl.getCallbacks();

try {
if (supportsCamera2ApiLocked(cameraId)) {
// Use cameraservice's cameradeviceclient
// implementation for HAL3.2+ devices
ICameraService cameraService =
CameraManagerGlobal.get().getCameraService();
if (cameraService == null) {
throw new ServiceSpecificException(...);
}
// 连接设备,并获取 ICameraDeviceUser
cameraUser = cameraService.connectDevice(callbacks,
cameraId, mContext.getOpPackageName(), uid);
} else {
// Use legacy camera implementation for HAL1 devices
int id;
try {
id = Integer.parseInt(cameraId);
} catch (NumberFormatException e) {
throw new IllegalArgumentException(...);
}

Log.i(TAG, "Using legacy camera HAL.");
cameraUser =
CameraDeviceUserShim.connectBinderShim(callbacks, id);
}
} catch (ServiceSpecificException e) {
...
} catch (RemoteException e) {
...
}
// 关联 CameraDeviceImpl 和 ICameraDeviceUser ,方便直接通信
deviceImpl.setRemoteDevice(cameraUser);
device = deviceImpl;
}

return device;
}

// CameraDeviceImpl.java
public void setRemoteDevice(ICameraDeviceUser remoteDevice)
throws CameraAccessException {
synchronized(mInterfaceLock) {
...
// 新建包装类,包装接口并处理对应访问异常
mRemoteDevice = new ICameraDeviceUserWrapper(remoteDevice);
...
}
}

从上面展示的 API 部分代码中可以看出:

  • 支持 API 2
    如果系统开启了 HAL 3 ,则支持 API 2 ;此时通过 ICameraService 访问服务。
  • 不支持 API 2
    如果系统仅支持 HAL 1 ,则 API 2 需要通过 CameraDeviceUserShim 转换为对应的 API 1 + HAL 1 来实现对应功能。CameraDeviceUserShimICameraDeviceUser 的实现类;整个 frameworks/base/core/java/android/hardware/camera2/legacy 目录下的代码都是为了实现这个转换功能。

整个打开设备的动作有如下功能:

  • 新建了 CameraDeviceImpl 实例,它是 CameraDevice 的实现类
  • CameraManager 通过 CameraService.connectDevice 连接设备,获取到 ICameraDeviceUser, ICameraDeviceCallbacks 对象,它们用于后续 CameraDeviceImpl.javaCameraDeviceClient.cpp 绑定通信
  • 新建 ICameraDeviceUserWrapper 实例,它是对 ICameraDeviceUser 的包装类,捕获并处理远程访问异常等

这里需要重点分析 connectDevice

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
// CameraService.cpp
Status CameraService::connectDevice(
const sp<hardware::camera2::ICameraDeviceCallbacks>& cameraCb,
const String16& cameraId,
const String16& clientPackageName,
int clientUid,
/*out*/
sp<hardware::camera2::ICameraDeviceUser>* device) {

ATRACE_CALL();
Status ret = Status::ok();
String8 id = String8(cameraId);
sp<CameraDeviceClient> client = nullptr;
ret = connectHelper<hardware::camera2::ICameraDeviceCallbacks,
CameraDeviceClient>(cameraCb, id,
CAMERA_HAL_API_VERSION_UNSPECIFIED, clientPackageName,
clientUid, USE_CALLING_PID, API_2,
/*legacyMode*/ false, /*shimUpdateOnly*/ false,
/*out*/client);

...
*device = client;
return ret;
}

template<class CALLBACK, class CLIENT>
Status CameraService::connectHelper(const sp<CALLBACK>& cameraCb,
const String8& cameraId, int halVersion,
const String16& clientPackageName, int clientUid, int clientPid,
apiLevel effectiveApiLevel, bool legacyMode, bool shimUpdateOnly,
/*out*/sp<CLIENT>& device) {
...
{
sp<BasicClient> tmp = nullptr;
if(!(ret = makeClient(this, cameraCb, clientPackageName,
cameraId, facing, clientPid, clientUid, getpid(),
legacyMode, halVersion, deviceVersion, effectiveApiLevel,
/*out*/&tmp)).isOk()) {
return ret;
}
...
err = client->initialize(mCameraProviderManager);
...
if (shimUpdateOnly) {
mServiceLock.unlock();
client->disconnect();
mServiceLock.lock();
} else {
// Otherwise, add client to active clients list
finishConnectLocked(client, partial);
}
} // lock is destroyed, allow further connect calls

device = client;
return ret;
}

CameraService::connectDevice 函数调用了模板函数 connectHelper ,而该模板主要的两个功能就是:makeClient 新建客户端,initialize 初始化客户端。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
Status CameraService::makeClient(const sp<CameraService>& cameraService,
const sp<IInterface>& cameraCb, const String16& packageName,
const String8& cameraId, int facing, int clientPid,
uid_t clientUid, int servicePid, bool legacyMode,
int halVersion, int deviceVersion, apiLevel effectiveApiLevel,
/*out*/sp<BasicClient>* client) {

if (halVersion < 0 || halVersion == deviceVersion) {
switch(deviceVersion) {
case CAMERA_DEVICE_API_VERSION_1_0:
if (effectiveApiLevel == API_1) { // Camera1 API route
sp<ICameraClient> tmp =
static_cast<ICameraClient*>(cameraCb.get());
*client = new CameraClient(cameraService, tmp,
packageName, cameraIdToInt(cameraId), facing,
clientPid, clientUid, getpid(), legacyMode);
} else { // Camera2 API route
ALOGW("Camera using old HAL version: %d", deviceVersion);
return STATUS_ERROR_FMT(...);
}
break;
case CAMERA_DEVICE_API_VERSION_3_0:
case CAMERA_DEVICE_API_VERSION_3_1:
case CAMERA_DEVICE_API_VERSION_3_2:
case CAMERA_DEVICE_API_VERSION_3_3:
case CAMERA_DEVICE_API_VERSION_3_4:
if (effectiveApiLevel == API_1) { // Camera1 API route
sp<ICameraClient> tmp =
static_cast<ICameraClient*>(cameraCb.get());
*client = new Camera2Client(cameraService, tmp,
packageName, cameraIdToInt(cameraId), facing,
clientPid, clientUid, servicePid, legacyMode);
} else { // Camera2 API route
sp<hardware::camera2::ICameraDeviceCallbacks> tmp =
static_cast<hardware::camera2::
ICameraDeviceCallbacks*>(cameraCb.get());
*client = new CameraDeviceClient(cameraService, tmp,
packageName, cameraId, facing, clientPid,
clientUid, servicePid);
}
break;
default:
// Should not be reachable
ALOGE("Unknown camera device HAL version:%d", deviceVersion);
return STATUS_ERROR_FMT(...);
}
} else {
if (deviceVersion > CAMERA_DEVICE_API_VERSION_1_0 &&
halVersion == CAMERA_DEVICE_API_VERSION_1_0) {
sp<ICameraClient> tmp =
static_cast<ICameraClient*>(cameraCb.get());
*client = new CameraClient(cameraService, tmp, packageName,
cameraIdToInt(cameraId), facing, clientPid,
clientUid, servicePid, legacyMode);
} else {
ALOGE("Invalid camera HAL version ..");
return STATUS_ERROR_FMT(...;
}
}
return Status::ok();
}

makeClient 主要是根据 device, HAL 版本和调用 API 的版本来创建对应的客户端:

  • HAL 1 + API 1 :新建 CameraClient
  • HAL 1 + API 2 :不支持
  • HAL 3 + API 1 :新建 Camera2Client
  • HAL 3 + API 2 :新建 CameraDeviceClient

这里的三个变量 effectiveApiLevel, legacyMode=0, halVersion ,主要是有三个连接函数决定: connect, connectLegacy, connectDevice ,其中 connectLegacy 可以指定 HAL 版本(来决定到底使用哪个 client):

  • 使用系统自带相机
    effectiveApiLevel=1, legacyMode=1, halVersion=256(HAL 1) ,系统自带应用使用的是 connectLegacy
  • 使用标准 API2 接口
    effectiveApiLevel=2, legacyMode=0, halVersion=-1 ,其中 -1 表示 CAMERA_HAL_API_VERSION_UNSPECIFIED

所谓的 HAL 版本,实际指的就是 Device 的版本:其中 HAL 1 对应 CAMERA_DEVICE_API_VERSION_1_0HAL 3 对应的是 CAMERA_DEVICE_API_VERSION_3_0 及以上版本。而 HAL 2CAMERA_DEVICE_API_VERSION_2_0 已经废弃。
因为手机平台使用 HAL 3 时,为了满足部分应用中使用了 API 1 的接口,常常需要兼容 HAL 1,所以支持 HAL 3 即意味着同时会支持 HAL 1

这里流程跟踪的是新建 CameraDeviceClient ,先看头文件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
// CameraService.h
class BasicClient : public virtual RefBase {
public:
virtual status_t initialize(sp<CameraProviderManager> manager) = 0;
...
private:
...
class OpsCallback : public BnAppOpsCallback {
public:
explicit OpsCallback(wp<BasicClient> client);
virtual void opChanged(int32_t op, const String16& packageName);

private:
wp<BasicClient> mClient;

}; // class OpsCallback

sp<OpsCallback> mOpsCallback;
...
}

// Camera3Device.h
class Camera3Device :
public CameraDeviceBase,
virtual public hardware::camera::device::V3_2::ICameraDeviceCallback,
private camera3_callback_ops
{...}

// Camera2ClientBase.h
template <typename TClientBase>
class Camera2ClientBase :
public TClientBase,
public CameraDeviceBase::NotificationListener
{
public:
typedef typename TClientBase::TCamCallbacks TCamCallbacks;
...
protected:
// 实例为 Camera3Device
sp<CameraDeviceBase> mDevice;
...
private:
...
template<typename TProviderPtr>
status_t initializeImpl(TProviderPtr providerPtr);
};

// CameraDeviceClient.h
struct CameraDeviceClientBase :
public CameraService::BasicClient,
public hardware::camera2::BnCameraDeviceUser
{
typedef hardware::camera2::ICameraDeviceCallbacks TCamCallbacks;
...

protected:
CameraDeviceClientBase(const sp<CameraService>& cameraService,
const sp<hardware::camera2::ICameraDeviceCallbacks>&
remoteCallback,
const String16& clientPackageName,
const String8& cameraId,
int cameraFacing,
int clientPid,
uid_t clientUid,
int servicePid);

sp<hardware::camera2::ICameraDeviceCallbacks> mRemoteCallback;
};

class CameraDeviceClient :
public Camera2ClientBase<CameraDeviceClientBase>,
public camera2::FrameProcessorBase::FilteredListener
{...}

从类图结构来看:BasicClient 是三个客户端 CameraClient, Camera2Client, CameraDeviceClient 的基类;而 Camera2ClientBase 中的变量 CameraDeviceBase 实际的子类是 Camera3Device 。来看构造函数的流程:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
// CameraDeviceClient.cpp
CameraDeviceClient::CameraDeviceClient(
const sp<CameraService>& cameraService,
const sp<hardware::camera2::ICameraDeviceCallbacks>& remoteCallback,
const String16& clientPackageName,
const String8& cameraId,
int cameraFacing,
int clientPid,
uid_t clientUid,
int servicePid) :
Camera2ClientBase(cameraService, remoteCallback, clientPackageName,
cameraId, cameraFacing, clientPid, clientUid, servicePid),
mInputStream(),
mStreamingRequestId(REQUEST_ID_NONE),
mRequestIdCounter(0),
mPrivilegedClient(false) {
...
}

// Camera2ClientBase.cpp
template <typename TClientBase>
Camera2ClientBase<TClientBase>::Camera2ClientBase(
const sp<CameraService>& cameraService,
const sp<TCamCallbacks>& remoteCallback,
const String16& clientPackageName,
const String8& cameraId,
int cameraFacing,
int clientPid,
uid_t clientUid,
int servicePid):
TClientBase(cameraService, remoteCallback, clientPackageName,
cameraId, cameraFacing, clientPid, clientUid, servicePid),
mSharedCameraCallbacks(remoteCallback),
mDeviceVersion(cameraService->getDeviceVersion(
TClientBase::mCameraIdStr)),
mDeviceActive(false)
{
...
// 实例化 Camera3Device
mDevice = new Camera3Device(cameraId);
}

// CameraDeviceClient.cpp
CameraDeviceClientBase::CameraDeviceClientBase(
const sp<CameraService>& cameraService,
const sp<hardware::camera2::ICameraDeviceCallbacks>& remoteCallback,
const String16& clientPackageName,
const String8& cameraId,
int cameraFacing,
int clientPid,
uid_t clientUid,
int servicePid) :
BasicClient(cameraService,
IInterface::asBinder(remoteCallback),
clientPackageName,
cameraId,
cameraFacing,
clientPid,
clientUid,
servicePid),
mRemoteCallback(remoteCallback) {
}

// CameraService.cpp
CameraService::BasicClient::BasicClient(
const sp<CameraService>& cameraService,
const sp<IBinder>& remoteCallback,
const String16& clientPackageName,
const String8& cameraIdStr, int cameraFacing,
int clientPid, uid_t clientUid,
int servicePid):
mCameraIdStr(cameraIdStr), mCameraFacing(cameraFacing),
mClientPackageName(clientPackageName),
mClientPid(clientPid), mClientUid(clientUid),
mServicePid(servicePid),
mDisconnected(false),
mRemoteBinder(remoteCallback)
{
...
}

根据类继承关系,一条链路实例化各个子类,最终会新建 Camera3Device 实例。makeClient 新建完客户端后,调用客户端的初始化:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// CameraDeviceClient.cpp
template<typename TProviderPtr>
status_t CameraDeviceClient::initializeImpl(TProviderPtr providerPtr) {
...
res = Camera2ClientBase::initialize(providerPtr);
...
String8 threadName;
mFrameProcessor = new FrameProcessorBase(mDevice);
threadName = String8::format("CDU-%s-FrameProc",
mCameraIdStr.string());
mFrameProcessor->run(threadName.string());

mFrameProcessor->registerListener(FRAME_PROCESSOR_LISTENER_MIN_ID,
FRAME_PROCESSOR_LISTENER_MAX_ID,
/*listener*/this,
/*sendPartials*/true);

return OK;
}

CameraDeviceClient::initializeImpl 是一个模板函数,主要有两个功能:调用 Camera2ClientBase 及其父类初始化;新建 FrameProcessorBase 实例,它主要功能是在发出预览、拍照、录像请求后,HAL 层向 Framework 层返回结果的回调类,后面讲预览流程时会详细分析。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// Camera2ClientBase.cpp
template <typename TClientBase>
status_t Camera2ClientBase<TClientBase>::initialize(
sp<CameraProviderManager> manager) {
return initializeImpl(manager);
}

template <typename TClientBase>
template <typename TProviderPtr>
status_t Camera2ClientBase<TClientBase>::initializeImpl(
TProviderPtr providerPtr) {
...
res = mDevice->initialize(providerPtr);
...
return OK;
}

Camera2ClientBase::initialize 也是一个模板函数,最终会调用 Camera3Device 的初始化:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
// Camera3Device.cpp
status_t Camera3Device::initialize(sp<CameraProviderManager> manager) {
...
sp<ICameraDeviceSession> session;
// 打开会话,获取 ICameraDeviceSession
status_t res = manager->openSession(mId.string(), this,
/*out*/ &session);
...
// 获取当前设备的配置信息,并保存到 CameraMetadata mDeviceInfo
res = manager->getCameraCharacteristics(mId.string(), &mDeviceInfo);
...
// 通过 ICameraDeviceSession 获取请求队列
std::shared_ptr<RequestMetadataQueue> queue;
auto requestQueueRet = session->getCaptureRequestMetadataQueue(
[&queue](const auto& descriptor) {
queue = std::make_shared<RequestMetadataQueue>(descriptor);
...
});
...
// 通过 ICameraDeviceSession 获取结果队列
std::unique_ptr<ResultMetadataQueue>& resQueue=mResultMetadataQueue;
auto resultQueueRet = session->getCaptureResultMetadataQueue(
[&resQueue](const auto& descriptor) {
resQueue=std::make_unique<ResultMetadataQueue>(descriptor);
...
});
...
// 新建 HalInterface 实例,并绑定 ICameraDeviceSession 以及请求队列
mInterface = new HalInterface(session, queue);
...
return initializeCommonLocked();
}

Camera3Device::initialize 初始化中,重点实现的功能为打开物理设备,并获取 ICameraDeviceSession 用于后续直接和 HAL 通信,并通过它从 HAL 获取请求队列和结果队列;最后新建 HalInterface 实例,并将 ICameraDeviceSession 保存并绑定。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// CameraProviderManager.cpp
status_t CameraProviderManager::openSession(const std::string &id,
const sp<hardware::camera::device::V3_2::ICameraDeviceCallback>&
callback, /*out*/
sp<hardware::camera::device::V3_2::ICameraDeviceSession> *session){

std::lock_guard<std::mutex> lock(mInterfaceMutex);

auto deviceInfo = findDeviceInfoLocked(id,
/*minVersion*/ {3,0}, /*maxVersion*/ {4,0});
if (deviceInfo == nullptr) return NAME_NOT_FOUND;

auto *deviceInfo3=static_cast<ProviderInfo::DeviceInfo3*>(deviceInfo);

Status status;
hardware::Return<void> ret;
// 向 HAL 打开设备,并返回 ICameraDeviceSession
ret = deviceInfo3->mInterface->open(callback,
[&status, &session] (Status s,
const sp<device::V3_2::ICameraDeviceSession>& cameraSession) {
status = s;
if (status == Status::OK) {
*session = cameraSession;
}
});
...
}

CameraProviderManager::openSession 打开设备时,会向 HAL 打开设备,将 ICameraDeviceCallback 传入 HAL 并获取 ICameraDeviceSession 实例。接着看 Camera3Device::initializeCommonLocked

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
// Camera3Device.cpp
status_t Camera3Device::initializeCommonLocked() {

/** Start up status tracker thread */
mStatusTracker = new StatusTracker(this);
status_t res = mStatusTracker->run(
String8::format("C3Dev-%s-Status", mId.string()).string());
...

/** Register in-flight map to the status tracker */
mInFlightStatusId = mStatusTracker->addComponent();

/** Create buffer manager */
mBufferManager = new Camera3BufferManager();

mTagMonitor.initialize(mVendorTagId);

/** Start up request queue thread */
mRequestThread = new RequestThread(this, mStatusTracker, mInterface);
res = mRequestThread->run(
String8::format("C3Dev-%s-ReqQueue", mId.string()).string());
...
mPreparerThread = new PreparerThread();
...
return OK;
}

initializeCommonLocked 中新建了很多实例:

  • StatusTracker :状态跟踪线程
  • Camera3BufferManager :输出流的图形缓冲区管理,比如 Camera3OutputStream 的管理
  • TagMonitor :相机元数据 metadata 的监视器,比如 3A 信息等
  • RequestThread :请求线程,比如拍照、录像、预览的数据请求
  • PreparerThread :监测数据已经准备好流的线程

小结

以上流程图都是基于 API 2 + HAL 3,当 Camera Open 流程结束后:

  • 客户端调用 API 时,得到了 CameraDevice 的实例,并将 ICameraDeviceUserCameraDeviceImpl 绑定
  • 根据 HAL 1/3 生成了对应的 Device 客户端,当前生成的是 CameraDeviceClient 实例
  • Camera3Device 在初始化时,调用 CameraProviderManager.openSession ,它会通过 HIDL 通知 HAL 层打开摄像头物理设备;打开成功会 Camera3Device::HalInterfaceICameraDeviceSession 实例绑定
  • 新建 RequestThread 对象,后台运行线程,用于监听 API 发起的请求 CaptureRequest :预览、拍照、录像等
  • 新建 FrameProcessorBase 对象,后台运行线程,用于监听 HAL 返回的请求结果 CaptureResult

打开设备时,实际上 Framework, HAL 已经创建好会话 ICameraDeviceSession ;而下面分析的 API 创建会话流程,实际是根据不同需求(预览、拍照、录像)来创建和配置输出流。

创建会话流程

API

在打开设备后,获取到了 CameraDevice 的实例,通过它来创建会话 Session

1
2
3
4
5
// CameraDevice.java
public abstract void createCaptureSession(@NonNull List<Surface> outputs,
@NonNull CameraCaptureSession.StateCallback callback,
@Nullable Handler handler)
throws CameraAccessException;
  • List<Surface> outputs
    表示有多少个输出流,通常为预览流和拍照/录像,两个输出流。
  • CameraCaptureSession.StateCallback callback
    创建会话状态回调。
  • Handler handler
    回调方法使用哪个线程响应,如果为 null 表示当前线程。

API 创建会话过程源码分析:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
// CameraDeviceImpl.java
@Override
public void createCaptureSession(List<Surface> outputs,
CameraCaptureSession.StateCallback callback, Handler handler)
throws CameraAccessException {
// 将 Surface 转换为 OutputConfiguration
List<OutputConfiguration> outConfigurations =
new ArrayList<>(outputs.size());
for (Surface surface : outputs) {
outConfigurations.add(new OutputConfiguration(surface));
}
createCaptureSessionInternal(null, outConfigurations, callback,
handler, ICameraDeviceUser.NORMAL_MODE);
}

private void createCaptureSessionInternal(InputConfiguration inputConfig,
List<OutputConfiguration> outputConfigurations,
CameraCaptureSession.StateCallback callback, Handler handler,
int operatingMode) throws CameraAccessException {
...
// 创建会话时,输入 Surface 为空
Surface input = null;
try {
// configure streams and then block until IDLE
// 向 Framework, HAL 发送信息,配置设备
configureSuccess = configureStreamsChecked(inputConfig,
outputConfigurations, operatingMode);
if (configureSuccess == true && inputConfig != null) {
input = mRemoteDevice.getInputSurface();
}
} catch (CameraAccessException e) {
...
}
...
CameraCaptureSessionCore newSession = null;
// 根据模式来实例化对应的 Session
if (isConstrainedHighSpeed) {
newSession = new CameraConstrainedHighSpeedCaptureSessionImpl(
mNextSessionId++, callback, handler, this,
mDeviceHandler, configureSuccess, mCharacteristics);
} else {
// 假设实例化 CameraCaptureSessionImpl
newSession = new CameraCaptureSessionImpl(mNextSessionId++,
input, callback, handler, this,
mDeviceHandler, configureSuccess);
}

mCurrentSession = newSession;
...
}
  • List<Surface> 转换为 List<OutputConfiguration>
  • createCaptureSession 创建会话时,输入 Surface, InputConfiguration 都为空,即只有输出流
  • 根据 isConstrainedHighSpeed 来创建 CameraCaptureSession 实例;如果支持高速模式,则创建 CameraConstrainedHighSpeedCaptureSessionImpl 实例;否则创建普通 CameraCaptureSessionImpl 实例

示例

创建预览 mTextureSurface 和拍照 ImageReader.getSurface 两个输出流的会话,使用当前线程处理回调接口:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
mCameraDevice.createCaptureSession(
Arrays.asList(mTextureSurface, mImageReader.getSurface()),
new CameraCaptureSession.StateCallback() {
@Override
public void onConfigured(@NonNull CameraCaptureSession session) {
Log.d(TAG, "onConfigured: ");
mCameraCaptureSession = session;
preview();
}

@Override
public void onConfigureFailed(
@NonNull CameraCaptureSession session) {
Log.e(TAG, "onConfigureFailed: ");
}
},
null);
} catch (CameraAccessException e) {
e.printStackTrace();
}

CameraCaptureSession.StateCallback 回调

CameraCaptureSession.StateCallback 回调用来处理 createCaptureSession 创建会话过程中出现的各种状态,比如创建成功、失败等,这些回调处理直接在 API Java 层实现的;回调接口中会获取到 CameraCaptureSession 实例。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
public static abstract class StateCallback {
public abstract void onConfigured(
@NonNull CameraCaptureSession session);
public abstract void onConfigureFailed(
@NonNull CameraCaptureSession session);
public void onReady(@NonNull CameraCaptureSession session) {
// default empty implementation
}
public void onActive(@NonNull CameraCaptureSession session) {
// default empty implementation
}
public void onCaptureQueueEmpty(
@NonNull CameraCaptureSession session) {
// default empty implementation
}
public void onClosed(@NonNull CameraCaptureSession session) {
// default empty implementation
}
public void onSurfacePrepared(@NonNull CameraCaptureSession session,
@NonNull Surface surface) {
// default empty implementation
}
}

createCaptureSession 创建会话时,会创建 CameraCaptureSessionImpl 实例,而 CameraCaptureSession.StateCallback 接口的回调都是在 CameraCaptureSessionImpl 中实现的:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
// CameraCaptureSessionImpl.java
public class CameraCaptureSessionImpl extends CameraCaptureSession
implements CameraCaptureSessionCore {
CameraCaptureSessionImpl(int id, Surface input,
CameraCaptureSession.StateCallback callback,
Handler stateHandler,
android.hardware.camera2.impl.CameraDeviceImpl deviceImpl,
Handler deviceStateHandler, boolean configureSuccess) {
...
mStateCallback =
createUserStateCallbackProxy(mStateHandler, callback);
...
// 根据传入的参数,响应 CameraCaptureSession.StateCallback 回调
if (configureSuccess) {
mStateCallback.onConfigured(this);
if (DEBUG) Log.v(...);
mConfigureSuccess = true;
} else {
mStateCallback.onConfigureFailed(this);
mClosed = true;
Log.e(...);
mConfigureSuccess = false;
}
}
...

// 用户指定并传入的回调实现及对应线程
private final CameraCaptureSession.StateCallback mStateCallback;
private final Handler mStateHandler;

private StateCallback createUserStateCallbackProxy(Handler handler,
StateCallback callback) {
InvokeDispatcher<StateCallback> userCallbackSink
= new InvokeDispatcher<>(callback);
HandlerDispatcher<StateCallback> handlerPassthrough =
new HandlerDispatcher<>(userCallbackSink, handler);
// 创建代理类
return new CallbackProxies.SessionStateCallbackProxy(
handlerPassthrough);
}
}

用户指定的 StateCallback 传入后,在方法 createUserStateCallbackProxy 中,通过 CallbackProxies 重新生成一个代理 mStateCallback 对象,通过反射的方式,完成所有回调响应过程。

  • 如果 configureStreamsChecked 创建 Stream 成功,则响应回调 mStateCallback.onConfigured
  • 如果失败则响应 mStateCallback.onConfigureFailed ,其他场景会产生剩余的回调

动态代理类 CallbackProxies 源码注释(JDK 中的动态代理只支持接口 interface,对于抽象类只能自己实现了):

1
2
3
4
5
6
7
8
9
10
/**
* Proxy out invocations to the camera2 API callbacks into
* a {@link Dispatchable}.
*
* <p>Since abstract classes do not support Java's dynamic {@code Proxy},
* we have to to use our own proxy mechanism.</p>
*/
public class CallbackProxies {

}

流程图

创建会话流程,查看原图

0109-android-camera-4-framework-CameraService-addService-CameraDevice_createCaptureSession.jpg

源码分析

API 中创建捕获会话 createCaptureSession 时,CameraDeviceImpl.configureStreamsChecked 源码中可以看到;CameraDeviceImpl 是通过 ICameraDeviceUser 来向 Framework, HAL 层发送配置信息的:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
// CameraDeviceImpl.java
public boolean configureStreamsChecked(InputConfiguration inputConfig,
List<OutputConfiguration> outputs, int operatingMode)
throws CameraAccessException {
...
// createCaptureSession 时,imputConfig 为空
checkInputConfiguration(inputConfig);
boolean success = false;
synchronized(mInterfaceLock) {
...
mDeviceHandler.post(mCallOnBusy);
stopRepeating();

try {
waitUntilIdle();
mRemoteDevice.beginConfigure();
...
// Delete all streams first (to free up HW resources)
for (Integer streamId : deleteList) {
mRemoteDevice.deleteStream(streamId);
mConfiguredOutputs.delete(streamId);
}

// Add all new streams
for (OutputConfiguration outConfig : outputs) {
if (addSet.contains(outConfig)) {
int streamId = mRemoteDevice.createStream(outConfig);
mConfiguredOutputs.put(streamId, outConfig);
}
}
operatingMode = (operatingMode | (customOpMode << 16));
mRemoteDevice.endConfigure(operatingMode);

success = true;
} catch (...)
}

return success;
}

configureStreamsChecked 配置流有三个主要过程:beginConfigure, createStream, endConfigure ,都是通过 ICameraDeviceUser 向下发送信息。 native 代码中由 CameraDeviceClient.cpp 实现了 ICameraDeviceUser 中的所有功能,这里重点分析 createStream 函数:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
// CameraDeviceClient.cpp
binder::Status CameraDeviceClient::createStream(
const hardware::camera2::params::OutputConfiguration
&outputConfiguration,
/*out*/int32_t* newStreamId) {
...
// 获取 IGraphicBufferProducer 的个数
const std::vector<sp<IGraphicBufferProducer>>& bufferProducers =
outputConfiguration.getGraphicBufferProducers();
size_t numBufferProducers = bufferProducers.size();
...
std::vector<sp<Surface>> surfaces;
std::vector<sp<IBinder>> binders;
...
OutputStreamInfo streamInfo;
bool isStreamInfoValid = false;
for (auto& bufferProducer : bufferProducers) {
...
// 创建 Native Surface
sp<Surface> surface;
res = createSurfaceFromGbp(streamInfo, isStreamInfoValid,
surface, bufferProducer);
if (!res.isOk())
return res;
...
binders.push_back(IInterface::asBinder(bufferProducer));
surfaces.push_back(surface);
}

int streamId = camera3::CAMERA3_STREAM_ID_INVALID;
// 调用 Camera3Device 创建流
err = mDevice->createStream(surfaces, deferredConsumer,
streamInfo.width, streamInfo.height,
streamInfo.format, streamInfo.dataSpace,
static_cast<camera3_stream_rotation_t>(
outputConfiguration.getRotation()),
&streamId, outputConfiguration.getSurfaceSetID(), isShared);

if (err != OK) {
...
} else {
...
mStreamInfoMap[streamId] = streamInfo;
...
// Set transform flags to ensure preview to be rotated correctly.
res = setStreamTransformLocked(streamId);
*newStreamId = streamId;
}

return res;
}

CameraDeviceClient.createStream 中,将 API 传入的 OutputConfiguration 数据,转换成 native Surface, OutputStreamInfo ;根据 OutputConfigurationIGraphicBufferProducer 的个数创建对应的 native Surface ,并最终通过设备来创建流。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
binder::Status CameraDeviceClient::createSurfaceFromGbp(
OutputStreamInfo& streamInfo, bool isStreamInfoValid,
sp<Surface>& surface, const sp<IGraphicBufferProducer>& gbp) {

...
// 根据 IGraphicBufferProducer 创建 Surface
surface = new Surface(gbp, useAsync);
ANativeWindow *anw = surface.get();

int width, height, format;
android_dataspace dataSpace;
// 查询对应的长、宽、格式、数据空间
if ((err = anw->query(anw, NATIVE_WINDOW_WIDTH, &width)) != OK) {
...
}
if ((err = anw->query(anw, NATIVE_WINDOW_HEIGHT, &height)) != OK) {
...
}
if ((err = anw->query(anw, NATIVE_WINDOW_FORMAT, &format)) != OK) {
...
}
if ((err = anw->query(anw, NATIVE_WINDOW_DEFAULT_DATASPACE,
reinterpret_cast<int*>(&dataSpace))) != OK) {
...
}

...
// 赋值给输出流
if (!isStreamInfoValid) {
streamInfo.width = width;
streamInfo.height = height;
streamInfo.format = format;
streamInfo.dataSpace = dataSpace;
streamInfo.consumerUsage = consumerUsage;
return binder::Status::ok();
}
...
}

这里的 NATIVE_WINDOW_FORMAT 格式代表着不同流的类型,在 system\core\libsystem\include\system\graphics-base.h 文件中定义:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
typedef enum {
HAL_PIXEL_FORMAT_RGBA_8888 = 1,
HAL_PIXEL_FORMAT_RGBX_8888 = 2,
HAL_PIXEL_FORMAT_RGB_888 = 3,
HAL_PIXEL_FORMAT_RGB_565 = 4,
HAL_PIXEL_FORMAT_BGRA_8888 = 5,
HAL_PIXEL_FORMAT_RGBA_1010102 = 43, // 0x2B
HAL_PIXEL_FORMAT_RGBA_FP16 = 22, // 0x16
HAL_PIXEL_FORMAT_YV12 = 842094169, // 0x32315659
HAL_PIXEL_FORMAT_Y8 = 538982489, // 0x20203859
HAL_PIXEL_FORMAT_Y16 = 540422489, // 0x20363159
HAL_PIXEL_FORMAT_RAW16 = 32, // 0x20
HAL_PIXEL_FORMAT_RAW10 = 37, // 0x25
HAL_PIXEL_FORMAT_RAW12 = 38, // 0x26
HAL_PIXEL_FORMAT_RAW_OPAQUE = 36, // 0x24
HAL_PIXEL_FORMAT_BLOB = 33, // 0x21
HAL_PIXEL_FORMAT_IMPLEMENTATION_DEFINED = 34, // 0x22
HAL_PIXEL_FORMAT_YCBCR_420_888 = 35, // 0x23
HAL_PIXEL_FORMAT_YCBCR_422_888 = 39, // 0x27
HAL_PIXEL_FORMAT_YCBCR_444_888 = 40, // 0x28
HAL_PIXEL_FORMAT_FLEX_RGB_888 = 41, // 0x29
HAL_PIXEL_FORMAT_FLEX_RGBA_8888 = 42, // 0x2A
HAL_PIXEL_FORMAT_YCBCR_422_SP = 16, // 0x10
HAL_PIXEL_FORMAT_YCRCB_420_SP = 17, // 0x11
HAL_PIXEL_FORMAT_YCBCR_422_I = 20, // 0x14
HAL_PIXEL_FORMAT_JPEG = 256, // 0x100
} android_pixel_format_t;
  • HAL_PIXEL_FORMAT_BLOB 拍照流
    值为 33 ,通常对应 mImageReader.getSurface()
  • HAL_PIXEL_FORMAT_IMPLEMENTATION_DEFINED 预览和录像流
    值为 34 ,通常对应预览 new Surface(mTextureView.getSurfaceTexture()) 和录像 mMediaRecorder.getSurface

CameraDeviceClient::createStream 最终会调用 Camera3Device::createStream ,它会根据 NATIVE_WINDOW_FORMAT 格式创建不同配置的流:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
// Camera3Device.cpp
status_t Camera3Device::createStream(
const std::vector<sp<Surface>>& consumers,
bool hasDeferredConsumer, uint32_t width, uint32_t height,
int format, android_dataspace dataSpace,
camera3_stream_rotation_t rotation, int *id,
int streamSetId, bool isShared, uint64_t consumerUsage) {
...
// 拍照流
if (format == HAL_PIXEL_FORMAT_BLOB) {
ssize_t blobBufferSize;
if (dataSpace != HAL_DATASPACE_DEPTH) {
blobBufferSize = getJpegBufferSize(width, height);
if (blobBufferSize <= 0) {
SET_ERR_L(...);
return BAD_VALUE;
}
} else {
blobBufferSize = getPointCloudBufferSize();
if (blobBufferSize <= 0) {
SET_ERR_L(...);
return BAD_VALUE;
}
}
newStream = new Camera3OutputStream(mNextStreamId, consumers[0],
width, height, blobBufferSize, format, dataSpace,
rotation, mTimestampOffset, streamSetId);
} else if (format == HAL_PIXEL_FORMAT_RAW_OPAQUE) {
ssize_t rawOpaqueBufferSize =
getRawOpaqueBufferSize(width, height);
if (rawOpaqueBufferSize <= 0) {
SET_ERR_L(...);
return BAD_VALUE;
}
newStream = new Camera3OutputStream(mNextStreamId, consumers[0],
width, height, rawOpaqueBufferSize, format, dataSpace,
rotation, mTimestampOffset, streamSetId);
}
...
else {
newStream = new Camera3OutputStream(mNextStreamId, consumers[0],
width, height, format, dataSpace, rotation,
mTimestampOffset, streamSetId);
}
newStream->setStatusTracker(mStatusTracker);
newStream->setBufferManager(mBufferManager);

res = mOutputStreams.add(mNextStreamId, newStream);
if (res < 0) {
SET_ERR_L(...);
return res;
}

*id = mNextStreamId++;
mNeedConfig = true;

...
ALOGV("Camera %s: Created new stream", mId.string());
return OK;
}

注意:每配置一个输出 Surface ,都会创建对应的输出流 Camera3OutputStream ,这是一个 for 循环过程。
API 调用过程中,CameraDeviceImpl.configureStreamsChecked 的第三步为 endConfigure,而 CameraDeviceClient::endConfigure 代码流程如下:

1
2
3
4
5
6
7
// CameraDeviceClient.cpp
binder::Status CameraDeviceClient::endConfigure(int operatingMode) {
...
status_t err = mDevice->configureStreams(operatingMode);
...
return res;
}

它的主要作用就是通过 Camera3Device 来配置流,configureStreamsLocked 配置流主要有三个过程:startConfiguration, configureStreams, endConfigure

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
// Camera3Device.cpp
status_t Camera3Device::configureStreams(int operatingMode) {
...
return configureStreamsLocked(operatingMode);
}

status_t Camera3Device::configureStreamsLocked(int operatingMode) {
...
camera3_stream_configuration config;
config.operation_mode = mOperatingMode;
config.num_streams = (mInputStream != NULL) + mOutputStreams.size();

Vector<camera3_stream_t*> streams;
streams.setCapacity(config.num_streams);

if (mInputStream != NULL) {
camera3_stream_t *inputStream;
inputStream = mInputStream->startConfiguration();
if (inputStream == NULL) {
CLOGE("Can't start input stream configuration");
cancelStreamsConfigurationLocked();
return INVALID_OPERATION;
}
streams.add(inputStream);
}

for (size_t i = 0; i < mOutputStreams.size(); i++) {
if (mOutputStreams[i].get() ==
static_cast<Camera3StreamInterface*>(mInputStream.get())) {

config.num_streams--;
continue;
}

camera3_stream_t *outputStream;
outputStream = mOutputStreams.editValueAt(i)->startConfiguration();
if (outputStream == NULL) {
CLOGE("Can't start output stream configuration");
cancelStreamsConfigurationLocked();
return INVALID_OPERATION;
}
streams.add(outputStream);
}

config.streams = streams.editArray();

// Do the HAL configuration; will potentially touch stream
// max_buffers, usage, priv fields.
// 向 HAL 层发送配置信息
res = mInterface->configureStreams(&config);
...
if (mInputStream != NULL && mInputStream->isConfiguring()) {
res = mInputStream->finishConfiguration();
if (res != OK) {
CLOGE("Can't finish configuring input stream %d: %s (%d)",
mInputStream->getId(), strerror(-res), res);
cancelStreamsConfigurationLocked();
return BAD_VALUE;
}
}

for (size_t i = 0; i < mOutputStreams.size(); i++) {
sp<Camera3OutputStreamInterface> outputStream =
mOutputStreams.editValueAt(i);
if (outputStream->isConfiguring() &&
!outputStream->isConsumerConfigurationDeferred()) {
res = outputStream->finishConfiguration();
if (res != OK) {
CLOGE(...);
cancelStreamsConfigurationLocked();
return BAD_VALUE;
}
}
}
...
}

最终通过 Camera3Device::HalInterface::configureStreamsHAL 层发起配置信息:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
// Camera3Device.cpp
status_t Camera3Device::HalInterface::configureStreams(
camera3_stream_configuration *config) {
...
// Invoke configureStreams

device::V3_3::HalStreamConfiguration finalConfiguration;
common::V1_0::Status status;

// See if we have v3.3 HAL
sp<device::V3_3::ICameraDeviceSession> hidlSession_3_3;
auto castResult =
device::V3_3::ICameraDeviceSession::castFrom(mHidlSession);
if (castResult.isOk()) {
hidlSession_3_3 = castResult;
} else {
ALOGE(...);
}
if (hidlSession_3_3 != nullptr) {
// We do; use v3.3 for the call
ALOGV("%s: v3.3 device found", __FUNCTION__);
auto err = hidlSession_3_3->configureStreams_3_3(
requestedConfiguration,
[&status, &finalConfiguration]
(common::V1_0::Status s, const
device::V3_3::HalStreamConfiguration& halConfiguration) {
finalConfiguration = halConfiguration;
status = s;
});
if (!err.isOk()) {
ALOGE(...);
return DEAD_OBJECT;
}
} else {
// We don't; use v3.2 call and
// construct a v3.3 HalStreamConfiguration
ALOGV("%s: v3.2 device found", __FUNCTION__);
HalStreamConfiguration finalConfiguration_3_2;
auto err = mHidlSession->configureStreams(requestedConfiguration,
[&status, &finalConfiguration_3_2]
(common::V1_0::Status s, const
HalStreamConfiguration& halConfiguration) {
finalConfiguration_3_2 = halConfiguration;
status = s;
});
if (!err.isOk()) {
ALOGE(...);
return DEAD_OBJECT;
}
finalConfiguration.streams.resize(
finalConfiguration_3_2.streams.size());
for (size_t i = 0; i<finalConfiguration_3_2.streams.size(); i++){
finalConfiguration.streams[i].v3_2 =
finalConfiguration_3_2.streams[i];
finalConfiguration.streams[i].overrideDataSpace =
requestedConfiguration.streams[i].dataSpace;
}
}
...
}

这里需要注意的是 HAL 3.3, 3.2 的配置是有区别的;执行完配置后,Camera3Stream::finishConfiguration 结束配置:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
// Camera3Stream.cpp
status_t Camera3Stream::finishConfiguration() {
....
res = configureQueueLocked();
...
}

// Camera3OutputStream.cpp
status_t Camera3OutputStream::configureConsumerQueueLocked() {
...
// Configure consumer-side ANativeWindow interface.
// to notify buffer manager (if it is used) of the returned buffers.
res = mConsumer->connect(NATIVE_WINDOW_API_CAMERA,
/*listener*/mBufferReleasedListener,
/*reportBufferRemoval*/true);
if (res != OK) {
ALOGE(...);
return res;
}

mConsumerName = mConsumer->getConsumerName();

res = native_window_set_usage(mConsumer.get(), mUsage);
if (res != OK) {
ALOGE(...);
return res;
}

res = native_window_set_scaling_mode(mConsumer.get(),
NATIVE_WINDOW_SCALING_MODE_SCALE_TO_WINDOW);
if (res != OK) {
ALOGE(...);
return res;
}
...
}

这里 mConsumernative Surface,也就是将生产者-消费者模型连接起来;同时 configureConsumerQueueLocked 有非常多的 native window 配置。

小结

创建会话 createCaptureSession 过程中,小结如下:

  • API 调用时,最终通过 CameraCaptureSession.StateCallback 获取到 CameraCaptureSessionImpl 实例
  • ICameraDeviceUser.createStream 由输入的 Surface 信息,根据不同的 format 创建对应输出流 Camera3OutputStream
  • ICameraDeviceUser.endConfigure 最终通过 CameraDeviceSession.configureStream_3_3 会向 HAL 层发送配置信息

相机预览过程中,如果 session 创建成功,会出现正常的预览界面;如果 session 创建失败,则预览会出现黑屏。

预览/拍照/录像流程

API

创建会话 createCaptureSession 成功后,通过拿到的 CameraCaptureSession 来预览、拍照、录像:

1
2
3
4
5
6
7
8
9
10
// CameraCaptureSession.java
// 预览和录像使用同一个 API
public abstract int setRepeatingRequest(@NonNull CaptureRequest request,
@Nullable CaptureCallback listener,
@Nullable Handler handler)
throws CameraAccessException;
// 拍照
public abstract int capture(@NonNull CaptureRequest request,
@Nullable CaptureCallback listener, @Nullable Handler handler)
throws CameraAccessException;
  • CaptureRequest request
    捕获请求,比如创建一个预览模板的请求 CameraDevice.TEMPLATE_PREVIEW ;拍照模板的请求 CameraDevice.TEMPLATE_STILL_CAPTURE ;录像模板的请求 CameraDevice.TEMPLATE_RECORD
  • CameraCaptureSession.CaptureCallback listener
    捕获状态的回调接口。
  • Handler handler
    回调接口使用哪个线程响应,如果是 null 表示当前线程。

CameraDevice 请求模板是一组常量:

1
2
3
4
5
6
7
// CameraDevice.java
public static final int TEMPLATE_PREVIEW = 1;
public static final int TEMPLATE_STILL_CAPTURE = 2;
public static final int TEMPLATE_RECORD = 3;
public static final int TEMPLATE_VIDEO_SNAPSHOT = 4;
public static final int TEMPLATE_ZERO_SHUTTER_LAG = 5;
public static final int TEMPLATE_MANUAL = 6;

各模板对应的含义:

  • TEMPLATE_PREVIEW
    创建适合相机预览的窗口,高帧率优于高质量的后期处理。
  • TEMPLATE_STILL_CAPTURE
    创建适合拍照的请求,优先考虑帧速率的图像质量。
  • TEMPLATE_RECORD
    创建适合录像的请求,使用稳定的帧率。
  • TEMPLATE_VIDEO_SNAPSHOT
    创建录像时快照的请求,在不中断录像的前提下最大化图像质量。
  • TEMPLATE_ZERO_SHUTTER_LAG
    创建 ZSL 零延时拍照请求,也就是连拍功能,在不影响帧率的前提下最大化图像质量,并开启 3A 算法。
  • TEMPLATE_MANUAL
    手动控制模板,禁用所有的自动控制 3A 算法。

示例

mTextureSurface 创建预览请求 TEMPLATE_PREVIEW ,使用后台线程处理回调接口:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
CaptureRequest.Builder previewRequestBuilder = 
mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
previewRequestBuilder.addTarget(mTextureSurface);
CameraCaptureSession.CaptureCallback captureCallback
= new CameraCaptureSession.CaptureCallback() {
@Override
public void onCaptureCompleted(
@NonNull CameraCaptureSession session,
@NonNull CaptureRequest request,
@NonNull TotalCaptureResult result) {
//Log.d(TAG, "preview, onCaptureCompleted: ");
}
};
mCameraCaptureSession.setRepeatingRequest(
previewRequestBuilder.build(), captureCallback, mBackHandler);

ImageReader 创建拍照请求 TEMPLATE_STILL_CAPTURE ,使用后台线程处理回调:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
CaptureRequest.Builder captureRequestBuild = mCameraDevice
.createCaptureRequest(CameraDevice.TEMPLATE_STILL_CAPTURE);
captureRequestBuild.addTarget(mImageReader.getSurface());
captureRequestBuild.set(CaptureRequest.CONTROL_AF_MODE,
CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE);
captureRequestBuild.set(CaptureRequest.JPEG_ORIENTATION,
mImageOrientation);
CameraCaptureSession.CaptureCallback captureCallback =
new CameraCaptureSession.CaptureCallback() {
@Override
public void onCaptureCompleted(
@NonNull CameraCaptureSession session,
@NonNull CaptureRequest request,
@NonNull TotalCaptureResult result) {
// Log.d(TAG, "takePicture, onCaptureCompleted: ");
}
};
mCameraCaptureSession.stopRepeating();
mCameraCaptureSession.abortCaptures();
mCameraCaptureSession.capture(captureRequestBuild.build(),
captureCallback, mBackHandler);

MediaRecorder 创建录像请求 TEMPLATE_RECORD ,不处理回调:

1
2
3
4
5
6
7
CaptureRequest.Builder recordCaptureBuild = mCameraDevice
.createCaptureRequest(CameraDevice.TEMPLATE_RECORD);
recordCaptureBuild.addTarget(mMediaRecorder.getSurface());
recordCaptureBuild.set(CaptureRequest.CONTROL_MODE,
CameraMetadata.CONTROL_MODE_AUTO);
mCameraCaptureSession.setRepeatingRequest(
recordCaptureBuild.build(), null, null);

CameraCaptureSession.CaptureCallback 回调

CameraCaptureSession 在请求预览、拍照、录像等功能时,出现的各种状态通过 CameraCaptureSession.CaptureCallback 回调来处理,回调是由 HAL 层发起向上传递的;回调接口中通常包含当前会话信息 CameraCaptureSession ,捕获请求 CaptureRequest ,捕获的结果 CaptureResult 等。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
// CameraCaptureSession.java
public static abstract class CaptureCallback {

public static final int NO_FRAMES_CAPTURED = -1;

// 设备开始拍照时
public void onCaptureStarted(@NonNull CameraCaptureSession session,
@NonNull CaptureRequest request, long timestamp,
long frameNumber) {
// default empty implementation
}
// 返回部分数据
public void onCapturePartial(CameraCaptureSession session,
CaptureRequest request, CaptureResult result) {
// default empty implementation
}
// 返回部分数据时,表示正在捕获数据过程中
public void onCaptureProgressed(@NonNull CameraCaptureSession session,
@NonNull CaptureRequest request,
@NonNull CaptureResult partialResult) {
// default empty implementation
}
// 数据捕获已经完成,回调最终总的结果集
public void onCaptureCompleted(@NonNull CameraCaptureSession session,
@NonNull CaptureRequest request,
@NonNull TotalCaptureResult result) {
// default empty implementation
}
// 捕获失败
public void onCaptureFailed(@NonNull CameraCaptureSession session,
@NonNull CaptureRequest request,
@NonNull CaptureFailure failure) {
// default empty implementation
}
// 所有捕获结果已经发送完毕
public void onCaptureSequenceCompleted(
@NonNull CameraCaptureSession session,
int sequenceId, long frameNumber) {
// default empty implementation
}
// 请求捕获被中止
public void onCaptureSequenceAborted(
@NonNull CameraCaptureSession session,
int sequenceId) {
// default empty implementation
}
// 捕获的 buffer 不能成功显示
public void onCaptureBufferLost(@NonNull CameraCaptureSession session,
@NonNull CaptureRequest request,
@NonNull Surface target, long frameNumber) {
// default empty implementation
}
}

API 在发起请求 CameraCaptureSession.setRepeatingRequest/capture 时,用户会创建 CaptureCallback 的实例,这些接口都在 CameraDeviceImpl 中实现回调。

  • onCaptureStarted
    该回调接口从 HAL 回调路径为:CameraDeviceClient::notifyShutter -> CameraDeviceImpl.onCaptureStarted -> onCaptureStarted
  • onCapturePartial
    该回调接口搜索整个 framework ,发现没有任何地方会回调它。
  • onCaptureProgressed, onCaptureCompleted, onCaptureSequenceCompleted
    三个接口都是在 CameraDeviceImpl.onResultReceived 中回调的。
  • onCaptureSequenceAborted
    CameraDeviceClient.cpp 中的 submitCaptureRequest, stopRepeating, flush 这三个函数会回调该接口。
  • onCaptureFailed, onCaptureBufferLost
    HAL 回调路径为 CameraDeviceClient::notifyError -> CameraDeviceImpl.onDeviceError ,而这两个接口在 CameraDeviceImpl 中的回调路径为 onDeviceError -> onCaptureErrorLocked -> onCaptureFailed/onCaptureBufferLost

流程图

创建捕获请求流程,查看原图

0109-android-camera-4-framework-CameraDevice_createCaptureRequest.jpg

预览/拍照/录像流程基本一致,这里仅给出预览的流程图:预览流程,查看原图

0109-android-camera-4-framework-CameraCaptureSession_setRepeatingRequest.jpg

源码分析

在分析预览、拍照、录像流程前,先回顾下打开设备 openCamera 时,做的一些初始化:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
// CameraDeviceClient.cpp
status_t CameraDeviceClient::initialize(sp<CameraProviderManager> manager) {
return initializeImpl(manager);
}

template<typename TProviderPtr>
status_t CameraDeviceClient::initializeImpl(TProviderPtr providerPtr) {
...
res = Camera2ClientBase::initialize(providerPtr);
if (res != OK) {
return res;
}

String8 threadName;
mFrameProcessor = new FrameProcessorBase(mDevice);
threadName = String8::format("CDU-%s-FrameProc",
mCameraIdStr.string());
mFrameProcessor->run(threadName.string());

mFrameProcessor->registerListener(FRAME_PROCESSOR_LISTENER_MIN_ID,
FRAME_PROCESSOR_LISTENER_MAX_ID,
/*listener*/this,
/*sendPartials*/true);

return OK;
}

CameraDeviceClient::initializeImpl 中,调用了 Camera2ClientBase::initialize 的初始化,以及实例化一个 FrameProcessorBase 对象;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// Camera2ClientBase.cpp
template <typename TClientBase>
status_t Camera2ClientBase<TClientBase>::initialize(
sp<CameraProviderManager> manager) {
return initializeImpl(manager);
}

template <typename TClientBase>
template <typename TProviderPtr>
status_t Camera2ClientBase<TClientBase>::initializeImpl(
TProviderPtr providerPtr) {
...
res = mDevice->initialize(providerPtr);
...
}

这里 Camera2ClientBase::initializeImpl 中主要是调用了 Camera3Device::initialize 函数,下面只关心和捕获请求有关的代码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// Camera3Device.cpp
status_t Camera3Device::initialize(sp<CameraProviderManager> manager) {
...
return initializeCommonLocked();
}

status_t Camera3Device::initializeCommonLocked() {
...
/** Start up request queue thread */
mRequestThread = new RequestThread(this, mStatusTracker, mInterface);
res = mRequestThread->run(
String8::format("C3Dev-%s-ReqQueue", mId.string()).string());
...
return OK;
}

Camera3Device::initializeCommonLocked 中实例化了 RequestThread 对象。至此,捕获流程中的发起请求的对象 RequestThread 和响应回调的对象 FrameProcessorBase 都实例化完毕,并开始运行。他们两个都继承的是线程,参看 system 目录下的 Thread.h/Threads.cpp 源码,可以看到 threadLoop 是在一个 while 中被循环调用的。当 threadLoop 返回 true 时就会不停的循环;返回 false 时会退出循环:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
// Threads.cpp
int Thread::_threadLoop(void* user)
{
Thread* const self = static_cast<Thread*>(user);
...
bool first = true;
do {
bool result;
if (first) {
first = false;
self->mStatus = self->readyToRun();
result = (self->mStatus == NO_ERROR);

if (result && !self->exitPending()) {
result = self->threadLoop();
}
} else {
result = self->threadLoop();
}

...
if (result == false || self->mExitPending) {
...
break;
}
...
} while(strong != 0);

return 0;
}

先来看发送捕获请求的线程 RequestThread

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
// Camera3Device.h
class RequestThread : public Thread {
...
protected:
...
virtual bool threadLoop();
private:
...
// Used to prepare a batch of requests.
struct NextRequest {
sp<CaptureRequest> captureRequest;
camera3_capture_request_t halRequest;
Vector<camera3_stream_buffer_t> outputBuffers;
bool submitted;
};
Vector<NextRequest> mNextRequests;
...
Condition mRequestSignal;
RequestList mRequestQueue;
RequestList mRepeatingRequests;
...
}

这里只关注 RequestThread 类中几个关键函数和变量,NextRequest 结构体包含了请求信息,逐个向 HAL 发送这些信息;类中定义了多个条件变量,重点关注 mRequestSignal 条件变量, threadLoop 运行时,会通过 mRequestSignal.waitRelative 阻塞等待 50 ms;直到等到捕获请求后 mRequestSignal.signal 发出通知,threadLoop 继续运行。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
// Camera3Device.cpp
bool Camera3Device::RequestThread::threadLoop() {
...
status_t res;

// Handle paused state.
if (waitIfPaused()) {
return true;
}

// Wait for the next batch of requests.
waitForNextRequestBatch();
if (mNextRequests.size() == 0) {
return true;
}
...
// Prepare a batch of HAL requests and output buffers.
res = prepareHalRequests();
...
}

void Camera3Device::RequestThread::waitForNextRequestBatch() {
...
NextRequest nextRequest;
nextRequest.captureRequest = waitForNextRequestLocked();
if (nextRequest.captureRequest == nullptr) {
return;
}
...
}

sp<Camera3Device::CaptureRequest>
Camera3Device::RequestThread::waitForNextRequestLocked() {
status_t res;
sp<CaptureRequest> nextRequest;

while (mRequestQueue.empty()) {
if (!mRepeatingRequests.empty()) {
const RequestList &requests = mRepeatingRequests;
RequestList::const_iterator firstRequest =
requests.begin();
nextRequest = *firstRequest;
mRequestQueue.insert(mRequestQueue.end(),
++firstRequest,
requests.end());

mRepeatingLastFrameNumber = mFrameNumber+requests.size()-1;

break;
}
// 条件变量 mRequestSignal 阻塞等待 kRequestTimeout
res = mRequestSignal.waitRelative(mRequestLock, kRequestTimeout);
...
}
}

RequestThread 在没有捕获请求时,会循环调用 threadLoop ,并阻塞等待 mRequestSignal 的通知。再看响应回调的线程 FrameProcessorBase

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
// FrameProcessorBase.h
class FrameProcessorBase: public Thread {
public:
...
struct FilteredListener: virtual public RefBase {
virtual void onResultAvailable(const CaptureResult &result) = 0;
};
protected:
static const nsecs_t kWaitDuration = 10000000; // 10 ms
wp<CameraDeviceBase> mDevice;
virtual bool threadLoop();
...
}

// FrameProcessorBase.cpp
bool FrameProcessorBase::threadLoop() {
status_t res;

sp<CameraDeviceBase> device;
{
device = mDevice.promote();
if (device == 0) return false;
}

res = device->waitForNextFrame(kWaitDuration);
if (res == OK) {
processNewFrames(device);
} else if (res != TIMED_OUT) {
ALOGE(...);
}

return true;
}

FrameProcessorBase::threadLoop 代码非常简单,device->waitForNextFrame 阻塞等待 10ms ,这里 CameraDeviceBase 实际类型为 Camera3Device

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
// Camera3Device.h
class Camera3Device :
public CameraDeviceBase,
virtual public hardware::camera::device::V3_2::ICameraDeviceCallback,
private camera3_callback_ops {
...
private:
List<CaptureResult> mResultQueue;
Condition mResultSignal;
...
}

// Camera3Device.cpp
status_t Camera3Device::waitForNextFrame(nsecs_t timeout) {
...
while (mResultQueue.empty()) {
res = mResultSignal.waitRelative(mOutputLock, timeout);
if (res == TIMED_OUT) {
return res;
} else if (res != OK) {
ALOGW(...);
return res;
}
}
return OK;
}

Camera3Device::waitForNextFrame 代码也很简单,调用条件变量 mResultSignal.waitRelative 实现阻塞等待 10 ms。
至此初始化过程中,捕获请求线程 RequestThread 循环执行 threadLoop ,并会阻塞等待 mRequestSignal 的通知;回调响应线程 FrameProcessorBase 循环执行 threadLoop ,并会阻塞等待 mResultSignal 的通知。

当用户调用 API 创建捕获请求时,mRequestSignal 会发出通知;因为预览、拍照、录像流程基本一样,一起分析:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
// CameraCaptureSessionImpl.java
// 预览和录像
@Override
public int setRepeatingRequest(CaptureRequest request,
CaptureCallback callback,
Handler handler)
throws CameraAccessException {
...
synchronized (mDeviceImpl.mInterfaceLock) {
...
return addPendingSequence(mDeviceImpl.setRepeatingRequest(
request,
createCaptureCallbackProxy(handler, callback),
mDeviceHandler));
}
}

// 拍照
@Override
public int capture(CaptureRequest request, CaptureCallback callback,
Handler handler) throws CameraAccessException {
...
synchronized (mDeviceImpl.mInterfaceLock) {
...
return addPendingSequence(mDeviceImpl.capture(request,
createCaptureCallbackProxy(handler, callback),
mDeviceHandler));
}
}

createCaptureCallbackProxy 创建了一个回调动态代理,通过 CameraDeviceImpl.setRepeatingRequest/capture 下发预览或者拍照的捕获请求:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
// CameraDeviceImpl.java
// 预览和录像
public int setRepeatingRequest(CaptureRequest request,
CaptureCallback callback,
Handler handler) throws CameraAccessException {
List<CaptureRequest> requestList = new ArrayList<CaptureRequest>();
requestList.add(request);
return submitCaptureRequest(requestList, callback,
handler, /*streaming*/true);
}
// 拍照
public int capture(CaptureRequest request, CaptureCallback callback,
Handler handler) throws CameraAccessException {
List<CaptureRequest> requestList = new ArrayList<CaptureRequest>();
requestList.add(request);
return submitCaptureRequest(requestList, callback,
handler, /*streaming*/false);
}

private int submitCaptureRequest(List<CaptureRequest> requestList,
CaptureCallback callback,
Handler handler, boolean repeating)
throws CameraAccessException {
...
synchronized(mInterfaceLock) {
checkIfCameraClosedOrInError();
if (repeating) {
stopRepeating();
}
SubmitInfo requestInfo;
CaptureRequest[] requestArray = requestList.toArray(
new CaptureRequest[requestList.size()]);
requestInfo = mRemoteDevice.submitRequestList(requestArray,
repeating);
...
if (callback != null) {
mCaptureCallbackMap.put(requestInfo.getRequestId(),
new CaptureCallbackHolder(
callback, requestList, handler,
repeating, mNextSessionId - 1));
} else {
...
}

if (repeating) {
if (mRepeatingRequestId != REQUEST_ID_NONE) {
checkEarlyTriggerSequenceComplete(mRepeatingRequestId,
requestInfo.getLastFrameNumber());
}
mRepeatingRequestId = requestInfo.getRequestId();
} else {
mRequestLastFrameNumbersList.add(
new RequestLastFrameNumbersHolder(requestList,
requestInfo));
}
...
}
}

从代码流程来看,预览和录像使用同一个接口;预览和拍照的主要区别是 repeating 的值;当为 true 时,表示预览/录像;当为 false 时,表示为拍照。通过 ICameraDeviceUser.submitRequestList 向下发送请求:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
// CameraDeviceClient.cpp
binder::Status CameraDeviceClient::submitRequestList(
const std::vector<hardware::camera2::CaptureRequest>& requests,
bool streaming,
/*out*/
hardware::camera2::utils::SubmitInfo *submitInfo) {
...
if (streaming) {
err = mDevice->setStreamingRequestList(metadataRequestList,
surfaceMapList, &(submitInfo->mLastFrameNumber));
...
} else {
err = mDevice->captureList(metadataRequestList, surfaceMapList,
&(submitInfo->mLastFrameNumber));
...
}
return res;
}

如果是预览/录像,则调用 Camera3Device->setStreamingRequestList ;如果是拍照,则调用 Camera3Device->captureList

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
//Camera3Device.cpp
// 预览和录像
status_t Camera3Device::setStreamingRequestList(
const List<const CameraMetadata> &requests,
const std::list<const SurfaceMap> &surfaceMaps,
int64_t *lastFrameNumber) {
...
return submitRequestsHelper(requests, surfaceMaps,
/*repeating*/true, lastFrameNumber);
}
// 拍照
status_t Camera3Device::captureList(
const List<const CameraMetadata> &requests,
const std::list<const SurfaceMap> &surfaceMaps,
int64_t *lastFrameNumber) {
...
return submitRequestsHelper(requests, surfaceMaps,
/*repeating*/false, lastFrameNumber);
}

status_t Camera3Device::submitRequestsHelper(
const List<const CameraMetadata> &requests,
const std::list<const SurfaceMap> &surfaceMaps,
bool repeating, /*out*/ int64_t *lastFrameNumber) {
...
RequestList requestList;
res = convertMetadataListToRequestListLocked(requests, surfaceMaps,
repeating, /*out*/&requestList);
...

if (repeating) {
res = mRequestThread->setRepeatingRequests(requestList,
lastFrameNumber);
} else {
res = mRequestThread->queueRequestList(requestList,
lastFrameNumber);
}
...
return res;
}

同样,预览/录像和拍照请求在 Camera3Device 中的区别也主要是 repeating 的值,都会调用 Camera3Device::submitRequestsHelper ,并通过 RequestThread 发起捕获请求;当预览/录像时,调用 setRepeatingRequests ;当拍照时,调用 queueRequestList

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
// Camera3Device.cpp
// 预览和录像
status_t Camera3Device::RequestThread::setRepeatingRequests(
const RequestList &requests,
/*out*/
int64_t *lastFrameNumber) {
...
if (lastFrameNumber != NULL) {
*lastFrameNumber = mRepeatingLastFrameNumber;
}
mRepeatingRequests.clear();
mRepeatingRequests.insert(mRepeatingRequests.begin(),
requests.begin(), requests.end());

unpauseForNewRequests();

mRepeatingLastFrameNumber =
...ICameraDeviceUser::NO_IN_FLIGHT_REPEATING_FRAMES;
return OK;
}
// 拍照
status_t Camera3Device::RequestThread::queueRequestList(
List<sp<CaptureRequest> > &requests,
/*out*/
int64_t *lastFrameNumber) {
...
for (List<sp<CaptureRequest> >::iterator it = requests.begin();
it != requests.end(); ++it) {
mRequestQueue.push_back(*it);
}
if (lastFrameNumber != NULL) {
*lastFrameNumber = mFrameNumber + mRequestQueue.size() - 1;
ALOGV(...);
}

unpauseForNewRequests();
return OK;
}

void Camera3Device::RequestThread::unpauseForNewRequests() {
...
mRequestSignal.signal();
...
}

预览/录像时会将捕获请求存入 mRepeatingRequests 列表中;拍照时会将捕获请求存入 mRequestQueue 列表中;它们最终都会调用 unpauseForNewRequests ,而该函数的核心功能就是通过 mRequestSignal.signal 发出消息,通知在开启设备初始化过程中 waitForNextRequestLocked 的阻塞等待。我们重新进入 RequestThread::threadLoop 中,继续向下分析:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
// Camera3Device.cpp
bool Camera3Device::RequestThread::threadLoop() {
...
status_t res;
...
// Wait for the next batch of requests.
// 阻塞等待
waitForNextRequestBatch();
if (mNextRequests.size() == 0) {
return true;
}
...
bool submitRequestSuccess = false;
nsecs_t tRequestStart = systemTime(SYSTEM_TIME_MONOTONIC);
if (mInterface->supportBatchRequest()) {
submitRequestSuccess = sendRequestsBatch();
} else {
submitRequestSuccess = sendRequestsOneByOne();
}
...
return submitRequestSuccess;
}

bool Camera3Device::RequestThread::sendRequestsBatch() {
status_t res;
size_t batchSize = mNextRequests.size();
std::vector<camera3_capture_request_t*> requests(batchSize);
uint32_t numRequestProcessed = 0;
for (size_t i = 0; i < batchSize; i++) {
requests[i] = &mNextRequests.editItemAt(i).halRequest;
}

...
res = mInterface->processBatchCaptureRequests(requests,
&numRequestProcessed);
...
}

status_t Camera3Device::HalInterface::processBatchCaptureRequests(
std::vector<camera3_capture_request_t*>& requests,
/*out*/uint32_t* numRequestProcessed) {
...
hardware::hidl_vec<device::V3_2::CaptureRequest> captureRequests;
size_t batchSize = requests.size();
captureRequests.resize(batchSize);
...
auto err = mHidlSession->processCaptureRequest(captureRequests,
cachesToRemove,
[&status, &numRequestProcessed] (auto s, uint32_t n) {
status = s;
*numRequestProcessed = n;
});
...
return CameraProviderManager::mapToStatusT(status);
}

waitForNextRequestBatch 拿到请求通知后,会将捕获请求存入 mNextRequests 中,当前平台支持批量请求处理,sendRequestsBatch -> processBatchCaptureRequests 流程,向 HAL 层发送捕获请求 mHidlSession->processCaptureRequest ,至此捕获请求从 API 发送到 HAL 整个流程全部分析完毕。

HAL 拿到捕获的结果后,会从 ICameraDeviceSession.processCaptureResult 回调到 Framework 层:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
// Camera3Device.cpp
hardware::Return<void> Camera3Device::processCaptureResult(
const hardware::hidl_vec<
hardware::camera::device::V3_2::CaptureResult>& results) {
...
for (const auto& result : results) {
processOneCaptureResultLocked(result);
}
mProcessCaptureResultLock.unlock();
return hardware::Void();
}

void Camera3Device::processOneCaptureResultLocked(
const hardware::camera::device::V3_2::CaptureResult& result) {
camera3_capture_result r;
...
processCaptureResult(&r);
}

void Camera3Device::processCaptureResult(
const camera3_capture_result *result) {
...
if (result->partial_result != 0)
request.resultExtras.partialResultCount = result->partial_result;

// Check if this result carries only partial metadata
if (mUsePartialResult && result->result != NULL) {
...
if (isPartialResult && request.hasCallback) {
// Send partial capture result
sendPartialCaptureResult(result->result, request.resultExtras,
frameNumber);
}
}
...
if (result->result != NULL && !isPartialResult) {
if (shutterTimestamp == 0) {
request.pendingMetadata = result->result;
request.collectedPartialResult = collectedPartialResult;
} else if (request.hasCallback) {
CameraMetadata metadata;
metadata = result->result;
sendCaptureResult(metadata, request.resultExtras,
collectedPartialResult, frameNumber,
hasInputBufferInRequest);
}
}
...
}

void Camera3Device::sendPartialCaptureResult(
const camera_metadata_t * partialResult,
const CaptureResultExtras &resultExtras, uint32_t frameNumber) {
...
CaptureResult captureResult;
captureResult.mResultExtras = resultExtras;
captureResult.mMetadata = partialResult;
insertResultLocked(&captureResult, frameNumber);
}


void Camera3Device::sendCaptureResult(CameraMetadata &pendingMetadata,
CaptureResultExtras &resultExtras,
CameraMetadata &collectedPartialResult,
uint32_t frameNumber,
bool reprocess) {
...
CaptureResult captureResult;
captureResult.mResultExtras = resultExtras;
captureResult.mMetadata = pendingMetadata;
...
insertResultLocked(&captureResult, frameNumber);
}

void Camera3Device::insertResultLocked(CaptureResult *result,
uint32_t frameNumber) {
...
camera_metadata_t *meta = const_cast<camera_metadata_t *>(
result->mMetadata.getAndLock());
set_camera_metadata_vendor_id(meta, mVendorTagId);
result->mMetadata.unlock(meta);
...
// Valid result, insert into queue
List<CaptureResult>::iterator queuedResult =
mResultQueue.insert(mResultQueue.end(),
CaptureResult(*result));
...
mResultSignal.signal();
}

从代码流程来看,从 HAL 传过来的捕获结果,不管是发回部分结果 sendPartialCaptureResult 还是最终结果 sendCaptureResult ,最终都会调用 insertResultLocked ,它的主要功能就是将捕获结果放入 mResultQueue 队列,并由 mResultSignal.signal 发出消息,通知在开启设备初始化过程中 waitForNextFrame 的阻塞等待。一旦 FrameProcessorBase::threadLoop 获取到捕获结果后,逐个处理:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
// FrameProcessorBase.cpp
void FrameProcessorBase::processNewFrames(
const sp<CameraDeviceBase> &device) {
status_t res;
...
while ( (res = device->getNextResult(&result)) == OK) {
camera_metadata_entry_t entry;
entry = result.mMetadata.find(ANDROID_REQUEST_FRAME_COUNT);
...
if (!processSingleFrame(result, device)) {
break;
}
...
}
...
return;
}

bool FrameProcessorBase::processSingleFrame(CaptureResult &result,
const sp<CameraDeviceBase> &device) {
...
return processListeners(result, device) == OK;
}

status_t FrameProcessorBase::processListeners(
const CaptureResult &result,
const sp<CameraDeviceBase> &device) {
...
List<sp<FilteredListener> >::iterator item = listeners.begin();
for (; item != listeners.end(); item++) {
(*item)->onResultAvailable(result);
}
return OK;
}

代理流程可以看出,逐个取出 CaptureResult 并处理,最终调用 CameraDeviceClient::onResultAvailableAPI 发送捕获结果:

1
2
3
4
5
6
7
8
9
10
// CameraDeviceClient.cpp
void CameraDeviceClient::onResultAvailable(const CaptureResult& result) {
...
sp<hardware::camera2::ICameraDeviceCallbacks> remoteCb
= mRemoteCallback;
if (remoteCb != NULL) {
remoteCb->onResultReceived(result.mMetadata,
result.mResultExtras);
}
}

API 中的回调是在 CameraDeviceImpl.java 中实现的:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
// CameraDeviceImpl.java
@Override
public void onResultReceived(CameraMetadataNative result,
CaptureResultExtras resultExtras) throws RemoteException {
int requestId = resultExtras.getRequestId();
long frameNumber = resultExtras.getFrameNumber();
...
synchronized(mInterfaceLock) {
...
final CaptureCallbackHolder holder =
CameraDeviceImpl.this.mCaptureCallbackMap.get(requestId);
final CaptureRequest request =
holder.getRequest(resultExtras.getSubsequenceId());
boolean isPartialResult = (resultExtras.getPartialResultCount()
< mTotalPartialCount);
...
if (isPartialResult) {
final CaptureResult resultAsCapture =
new CaptureResult(result, request, resultExtras);
// Partial result
resultDispatch = new Runnable() {
@Override
public void run() {
if (!CameraDeviceImpl.this.isClosed()) {
if (holder.hasBatchedOutputs()) {
for (int i = 0; i < holder.getRequestCount(); i++) {
CameraMetadataNative resultLocal =
new CameraMetadataNative(resultCopy);
CaptureResult resultInBatch = new CaptureResult(
resultLocal, holder.getRequest(i), resultExtras);

holder.getCallback().onCaptureProgressed(
CameraDeviceImpl.this,
holder.getRequest(i),
resultInBatch);
}
} else {
holder.getCallback().onCaptureProgressed(
CameraDeviceImpl.this,
request,
resultAsCapture);
}
}
}
};
finalResult = resultAsCapture;
} else {
List<CaptureResult> partialResults =
mFrameNumberTracker.popPartialResults(frameNumber);
...
final TotalCaptureResult resultAsCapture =
new TotalCaptureResult(result, request, resultExtras,
partialResults, holder.getSessionId());
// Final capture result
resultDispatch = new Runnable() {
@Override
public void run() {
if (!CameraDeviceImpl.this.isClosed()){
if (holder.hasBatchedOutputs()) {
for (int i = 0; i < holder.getRequestCount(); i++) {
...
TotalCaptureResult resultInBatch = new TotalCaptureResult(
resultLocal, holder.getRequest(i), resultExtras,
partialResults, holder.getSessionId());

holder.getCallback().onCaptureCompleted(
CameraDeviceImpl.this,
holder.getRequest(i),
resultInBatch);
}
} else {
holder.getCallback().onCaptureCompleted(
CameraDeviceImpl.this,
request,
resultAsCapture);
}
}
}
};
finalResult = resultAsCapture;
}
...
// Fire onCaptureSequenceCompleted
if (!isPartialResult) {
checkAndFireSequenceComplete();
}
}
}

CameraDeviceImpl 中处理 CameraCaptureSession.CaptureCallback 各回调结果:如果返回的是部分结果,则回调 onCaptureProgressed ;如果返回最终结果,则回调 onCaptureCompleted 。整个预览、拍照、录像流程及回调分析完毕。

小结

  • Camera3Device::RequestThread
    这个类主要是处理预览、拍照、录像的请求 CaptureRequest
  • FrameProcessorBase.cpp
    这个类主要是处理请求后的回调函数,回调中会包含捕获的结果 CaptureResult
  • Condition
    不管是请求还是结果回调,因为是多线程处理,都涉及到条件变量的阻塞等待和通知机制。
  • CameraDeviceSession.CaptureCallback
    该回调接口都是在 CameraDeviceImpl 中实现的。

三者异同

预览、拍照、录像三者的流程基本一致,它们之间有如下异同:

  • 预览
    捕获请求模板为 CameraDevice.TEMPLATE_PREVIEWAPI 接口为 CameraCaptureSession.setRepeatingRequestrepeating 值为 true
  • 拍照
    捕获请求模板为 CameraDevice.TEMPLATE_STILL_CAPTUREAPI 接口为 CameraCaptureSession.Capturerepeating 值为 false
  • 录像
    捕获请求模板为 CameraDevice.TEMPLATE_RECORDAPI 接口为 CameraCaptureSession.setRepeatingRequestrepeating 值为 true

也就是说,预览和录像仅仅是捕获请求模板不一样;而预览和拍照不管是模板,接口,repeating 值都不一样;但是它们三者最终在 Framework 中代码流程基本一致。

CameraServiceProxy 注册服务

AIDL

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
interface ICameraServiceProxy
{
// CameraService 向代理服务发送消息,通知用户更新
oneway void pingForUserUpdate();

const int CAMERA_STATE_OPEN = 0;
const int CAMERA_STATE_ACTIVE = 1;
const int CAMERA_STATE_IDLE = 2;
const int CAMERA_STATE_CLOSED = 3;
const int CAMERA_FACING_BACK = 0;
const int CAMERA_FACING_FRONT = 1;
const int CAMERA_FACING_EXTERNAL = 2;

// CameraService 向代理服务发送消息,通知相机设备状态更新
oneway void notifyCameraState(String cameraId, int facing,
int newCameraState, String clientName);
}

AIDL 文件看出,CameraServiceProxy 主要是响应 CameraService 的请求,也就是向 Framework Java 发送消息。

流程图

CameraServiceProxy 服务名称:media.camera.proxyCameraServiceProxy 继承了 SystemService ,注册流程如下:

0109-android-camera-framework-CameraServiceProxy-addService.png

源码分析

先来看注册流程的源码,服务的标准注册流程:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
// CameraServiceProxy.java
public class CameraServiceProxy extends SystemService
implements Handler.Callback, IBinder.DeathRecipient {
...
private static final String CAMERA_SERVICE_BINDER_NAME
= "media.camera";
public static final String CAMERA_SERVICE_PROXY_BINDER_NAME
= "media.camera.proxy";
...
// 构造方法中,初始化线程相关
public CameraServiceProxy(Context context) {
super(context);
mContext = context;
mHandlerThread = new ServiceThread(TAG,
Process.THREAD_PRIORITY_DISPLAY, /*allowTo*/false);
mHandlerThread.start();
mHandler = new Handler(mHandlerThread.getLooper(), this);

mNotifyNfc = SystemProperties.getInt(NFC_NOTIFICATION_PROP, 0)>0;
if (DEBUG) Slog.v(...);
}

// onStart 主要是注册服务,并监听 User 相关广播
@Override
public void onStart() {
mUserManager = UserManager.get(mContext);
...
IntentFilter filter = new IntentFilter();
filter.addAction(Intent.ACTION_USER_ADDED);
filter.addAction(Intent.ACTION_USER_REMOVED);
filter.addAction(Intent.ACTION_USER_INFO_CHANGED);
filter.addAction(Intent.ACTION_MANAGED_PROFILE_ADDED);
filter.addAction(Intent.ACTION_MANAGED_PROFILE_REMOVED);
mContext.registerReceiver(mIntentReceiver, filter);

// 注册 CameraServiceProxy 服务
publishBinderService(CAMERA_SERVICE_PROXY_BINDER_NAME,
mCameraServiceProxy);
publishLocalService(CameraServiceProxy.class, this);
CameraStatsJobService.schedule(mContext);
}
...
}

类中的 User 指的是 Android 多用户;再看回调接口的实现:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// CameraServiceProxy.java
private final ICameraServiceProxy.Stub mCameraServiceProxy
= new ICameraServiceProxy.Stub() {
@Override
public void pingForUserUpdate() {
notifySwitchWithRetries(30);
}

@Override
public void notifyCameraState(String cameraId, int newCameraState,
int facing, String clientName) {
String state = cameraStateToString(newCameraState);
String facingStr = cameraFacingToString(facing);
if (DEBUG) Slog.v(...);
updateActivityCount(cameraId, newCameraState,facing,clientName);
}
};

小结

通常情况下 API 中,CameraManager 通过 ICameraService.aidlCameraService 下发请求;而 CameraService 通过 ICameraServiceListener.aidl 发回回调。
而如果没有 API 请求的情况下,CameraService 无法向 Framework Java 发送信息,所以系统开机时注册了 CameraServiceProxy 服务,用于响应 CameraService 的回调。

其他

Camera 相关声音

1
2
3
4
5
6
7
8
9
10
11
12
13
14
void CameraService::loadSound() {
ATRACE_CALL();

Mutex::Autolock lock(mSoundLock);
LOG1("CameraService::loadSound ref=%d", mSoundRef);
if (mSoundRef++) return;

mSoundPlayer[SOUND_SHUTTER] = newMediaPlayer(
"/system/media/audio/ui/camera_click.ogg");
mSoundPlayer[SOUND_RECORDING_START] = newMediaPlayer(
"/system/media/audio/ui/VideoRecord.ogg");
mSoundPlayer[SOUND_RECORDING_STOP] = newMediaPlayer(
"/system/media/audio/ui/VideoStop.ogg");
}

CameraMetadata

CameraMetadataNativeCameraMetadata 是同一个类型,只是命名空间不一样。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
namespace android {

...
class CameraMetadata: public Parcelable {
...
}

namespace hardware {
namespace camera2 {
namespace impl {
using ::android::CameraMetadata;
typedef CameraMetadata CameraMetadataNative;
}
}
}

常见问题

API 2 + HAL 1

平台仅支持 HAL 1 时,API 2openCamera 时,通过 CameraDeviceUserShimAPI 2 转换为 API 1 ,即 HAL 1 + API 1 向下发起请求。
LegacyCameraDevice 会将 CAMERA API2 转换为 CAMERA API1 ,而 CameraDeviceUserShim 封装了 LegacyCameraDevice

AIDL 生成多类型文件

AIDL 可以同时生成 .java, .h, .cpp 文件,编译规则在 Android.bp 中配置。

总结

后续

  • 数据传递
    Binder 通信机制,数据传输限制在 1M ,那整个通信机制是如何传递图片的呢?以及预览的呢?传递的是什么?
    createCaptureSession 创建捕获会话时,配置输出流;当 setRepeatingRequest 发起预览请求时,回调结果为 CaptureResult ,它是如何和输出流关联的呢?
  • SurfaceFlinger 显示相关知识
  • Buffer 相关管理
    这些都是 API 1 模式下的数据流,不过有参考意义。

参考文档

0%