一、前提回顾
稍微回顾一下之前的内容，以编码DefaultVideoEncoderFactory为入口，createEncoder是由HardwareVideoEncoderFactory 和 SoftwareVideoEncoderFactory各自创建出来，然后回传到PeerConnectionClient。
@Nullable
public VideoEncoder createEncoder(VideoCodecInfo info) {
VideoEncoder softwareEncoder = this.softwareVideoEncoderFactory.createEncoder(info);
VideoEncoder hardwareEncoder = this.hardwareVideoEncoderFactory.createEncoder(info);
if (hardwareEncoder != null && softwareEncoder != null) {
return new VideoEncoderFallback(softwareEncoder, hardwareEncoder);
} else {
return hardwareEncoder != null ? hardwareEncoder : softwareEncoder;
}
}对于HardwareVideoEncoderFactory createEncoder创建硬解器，大概如下：
@Nullable
@Override
public VideoEncoder createEncoder(VideoCodecInfo input) {
VideoCodecMimeType type = VideoCodecMimeType.valueOf(input.name);
MediaCodecInfo info = findCodecForType(type);
String codecName = info.getName();
String mime = type.mimeType();
Integer surfaceColorFormat = MediaCodecUtils.selectColorFormat(
MediaCodecUtils.TEXTURE_COLOR_FORMATS, info.getCapabilitiesForType(mime));
Integer yuvColorFormat = MediaCodecUtils.selectColorFormat(
MediaCodecUtils.ENCODER_COLOR_FORMATS, info.getCapabilitiesForType(mime));
if (type == VideoCodecMimeType.H264) {
boolean isHighProfile = H264Utils.isSameH264Profile(
input.params, MediaCodecUtils.getCodecProperties(type, /* highProfile= */ true));
boolean isBaselineProfile = H264Utils.isSameH264Profile(
input.params, MediaCodecUtils.getCodecProperties(type, /* highProfile= */ false));
if (!isHighProfile && !isBaselineProfile) {
return null;
}
if (isHighProfile && !isH264HighProfileSupported(info)) {
return null;
}
}
return new HardwareVideoEncoder(new MediaCodecWrapperFactoryImpl(), codecName, type,
surfaceColorFormat, yuvColorFormat, input.params, getKeyFrameIntervalSec(type),
getForcedKeyFrameIntervalMs(type, codecName), createBitrateAdjuster(type, codecName),
sharedContext);
}createEncoder的流程在 Android-RTC-6 分析VideoEncoderFactory进行过较为详细的分析，这次深入HardwareVideoEncoder。
二、BitrateAdjuster
事不宜迟，赶紧进入分析，首先是初始化操作。
@Override
public VideoCodecStatus initEncode(Settings settings, Callback callback) {
// ...
useSurfaceMode = canUseSurface();
if (settings.startBitrate != 0 && settings.maxFramerate != 0) {
bitrateAdjuster.setTargets(settings.startBitrate * 1000, settings.maxFramerate);
}
adjustedBitrate = bitrateAdjuster.getAdjustedBitrateBps();
return initEncodeInternal();
}
private VideoCodecStatus initEncodeInternal() {
lastKeyFrameNs = -1;
try {
codec = mediaCodecWrapperFactory.createByCodecName(codecName);
} catch (IOException | IllegalArgumentException e) {
Logging.e(TAG, "Cannot create media encoder " + codecName);
return VideoCodecStatus.FALLBACK_SOFTWARE;
}
final int colorFormat = useSurfaceMode ? surfaceColorFormat : yuvColorFormat;
try {
MediaFormat format = MediaFormat.createVideoFormat(codecType.mimeType(), width, height)
format.setInteger(MediaFormat.KEY_BIT_RATE, adjustedBitrate);
format.setInteger(KEY_BITRATE_MODE, VIDEO_ControlRateConstant);
format.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat);
format.setInteger(MediaFormat.KEY_FRAME_RATE, bitrateAdjuster.getCodecConfigFramerate()
format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, keyFrameIntervalSec);
if (codecType == VideoCodecMimeType.H264) {
String profileLevelId = params.get(VideoCodecInfo.H264_FMTP_PROFILE_LEVEL_ID);
if (profileLevelId == null) {
profileLevelId = VideoCodecInfo.H264_CONSTRAINED_BASELINE_3_1;
}
switch (profileLevelId) {
case VideoCodecInfo.H264_CONSTRAINED_HIGH_3_1:
format.setInteger("profile", VIDEO_AVC_PROFILE_HIGH);
format.setInteger("level", VIDEO_AVC_LEVEL_3);
break;
case VideoCodecInfo.H264_CONSTRAINED_BASELINE_3_1:
break;
default:
Logging.w(TAG, "Unknown profile level id: " + profileLevelId);
}
}
codec.configure(
format, null /* surface */, null /* crypto */, MediaCodec.CONFIGURE_FLAG_ENCODE);
if (useSurfaceMode) {
textureEglBase = EglBase.createEgl14(sharedContext, EglBase.CONFIG_RECORDABLE);
textureInputSurface = codec.createInputSurface();
textureEglBase.createSurface(textureInputSurface);
textureEglBase.makeCurrent();
}
codec.start();
outputBuffers = codec.getOutputBuffers();
} catch (IllegalStateException e) {
Logging.e(TAG, "initEncodeInternal failed", e);
release();
return VideoCodecStatus.FALLBACK_SOFTWARE;
}
running = true;
outputThreadChecker.detachThread();
outputThread = createOutputThread();
outputThread.start();
return VideoCodecStatus.OK;
}1、initEncode都是一些MediaCodec的常规操作，就是封装得比较完善，看见一个bitrateAdjuster，主要意义用于码率调整适配器。是HardwareVideoEncoderFactory->createEncoder创建HardwareVideoEncoder对象的时候通过createBitrateAdjuster传递的一个对象。
2、initEncode还根据profileLevelId设置了profile / level，关于profileLevelId的相关知识，已在 Android-RTC-6 分析过，大家可以去看看。
这次让我们深入浅出的看看这个BitrateAdjuster是如何工作的。
private BitrateAdjuster createBitrateAdjuster(VideoCodecMimeType type, String codecName) {
if (codecName.startsWith(EXYNOS_PREFIX)) {
if (type == VideoCodecMimeType.VP8) {
// Exynos VP8 encoders need dynamic bitrate adjustment.
return new DynamicBitrateAdjuster();
} else {
// Exynos VP9 and H264 encoders need framerate-based bitrate adjustment.
return new FramerateBitrateAdjuster();
}
}
// Other codecs don't need bitrate adjustment.
return new BaseBitrateAdjuster();
}一图看懂BitrateAdjuster的继承关系。主要工作是FramerateBitrateAdjuster 和 DynamicBitrateAdjuster。
class FramerateBitrateAdjuster extends BaseBitrateAdjuster {
private static final int INITIAL_FPS = 30;
@Override
public void setTargets(int targetBitrateBps, int targetFps) {
if (this.targetFps == 0) {
// Framerate-based bitrate adjustment always initializes to the same framerate.
targetFps = INITIAL_FPS;
}
super.setTargets(targetBitrateBps, targetFps);
this.targetBitrateBps = this.targetBitrateBps * INITIAL_FPS / this.targetFps;
}
@Override
public int getCodecConfigFramerate() {
return INITIAL_FPS;
}
}先看FramerateBitrateAdjuster，写法有点绕，其实就是固定帧率=30。然后根据设置的码率求 调整后的帧率对应的码率。（占百分比的关系，多数用于固定带宽的场景。）
class DynamicBitrateAdjuster extends BaseBitrateAdjuster {
private static final double BITS_PER_BYTE = 8.0;
// Change the bitrate at most once every three seconds.
private static final double BITRATE_ADJUSTMENT_SEC = 3.0;
// Maximum bitrate adjustment scale - no more than 4 times.
private static final double BITRATE_ADJUSTMENT_MAX_SCALE = 4;
// Amount of adjustment steps to reach maximum scale.
private static final int BITRATE_ADJUSTMENT_STEPS = 20;
// How far the codec has deviated above (or below) the target bitrate (tracked in bytes).
private double deviationBytes;
private double timeSinceLastAdjustmentMs;
private int bitrateAdjustmentScaleExp;
@Override
public void setTargets(int targetBitrateBps, int targetFps) {
if (this.targetBitrateBps > 0 && targetBitrateBps < this.targetBitrateBps) {
// Rescale the accumulator level if the accumulator max decreases
deviationBytes = deviationBytes * targetBitrateBps / this.targetBitrateBps;
}
super.setTargets(targetBitrateBps, targetFps);
}
@Override
public void reportEncodedFrame(int size) {
if (targetFps == 0) {
return;
}
// Accumulate the difference between actual and expected frame sizes.
double expectedBytesPerFrame = (targetBitrateBps / BITS_PER_BYTE) / targetFps;
deviationBytes += (size - expectedBytesPerFrame);
timeSinceLastAdjustmentMs += 1000.0 / targetFps;
// Adjust the bitrate when the encoder accumulates one second's worth of data in excess or
// shortfall of the target.
double deviationThresholdBytes = targetBitrateBps / BITS_PER_BYTE;
// Cap the deviation, i.e., don't let it grow beyond some level to avoid using too old data for
// bitrate adjustment. This also prevents taking more than 3 "steps" in a given 3-second cycle.
double deviationCap = BITRATE_ADJUSTMENT_SEC * deviationThresholdBytes;
deviationBytes = Math.min(deviationBytes, deviationCap);
deviationBytes = Math.max(deviationBytes, -deviationCap);
// Do bitrate adjustment every 3 seconds if actual encoder bitrate deviates too much
// from the target value.
if (timeSinceLastAdjustmentMs <= 1000 * BITRATE_ADJUSTMENT_SEC) {
return;
}
if (deviationBytes > deviationThresholdBytes) {
// Encoder generates too high bitrate - need to reduce the scale.
int bitrateAdjustmentInc = (int) (deviationBytes / deviationThresholdBytes + 0.5);
bitrateAdjustmentScaleExp -= bitrateAdjustmentInc;
// Don't let the adjustment scale drop below -BITRATE_ADJUSTMENT_STEPS.
// This sets a minimum exponent of -1 (bitrateAdjustmentScaleExp / BITRATE_ADJUSTMENT_STEPS).
bitrateAdjustmentScaleExp = Math.max(bitrateAdjustmentScaleExp, -BITRATE_ADJUSTMENT_STEPS);
deviationBytes = deviationThresholdBytes;
} else if (deviationBytes < -deviationThresholdBytes) {
// Encoder generates too low bitrate - need to increase the scale.
int bitrateAdjustmentInc = (int) (-deviationBytes / deviationThresholdBytes + 0.5);
bitrateAdjustmentScaleExp += bitrateAdjustmentInc;
// Don't let the adjustment scale exceed BITRATE_ADJUSTMENT_STEPS.
// This sets a maximum exponent of 1 (bitrateAdjustmentScaleExp / BITRATE_ADJUSTMENT_STEPS).
bitrateAdjustmentScaleExp = Math.min(bitrateAdjustmentScaleExp, BITRATE_ADJUSTMENT_STEPS);
deviationBytes = -deviationThresholdBytes;
}
timeSinceLastAdjustmentMs = 0;
}
private double getBitrateAdjustmentScale() {
return Math.pow(BITRATE_ADJUSTMENT_MAX_SCALE,
(double) bitrateAdjustmentScaleExp / BITRATE_ADJUSTMENT_STEPS);
}
@Override
public int getAdjustedBitrateBps() {
return (int) (targetBitrateBps * getBitrateAdjustmentScale());
}
}再看DynamicBitrateAdjuster，复写了reportEncodedFrame，此方法传入的参数 是 编码输出的实际字节大小。我们来看看其中的算法逻辑。
// 期望的每帧分配码率，转换大Byte为单位。
double expectedBytesPerFrame = (targetBitrateBps / BITS_PER_BYTE) / targetFps;
// 统计实际帧大小和预期帧大小之间的差异。
deviationBytes += (size - expectedBytesPerFrame);
// 期望的码率值(Byte)
double deviationThresholdBytes = targetBitrateBps / BITS_PER_BYTE;/**
* 限制偏差，也就是说，不要让偏差超过某个级别，以避免使用太旧的数据进行比特率调整。
* 这还可以防止在给定的3秒周期内采取超过3个“调整级别”。
*/
// 3s内的期望码率
double deviationCap = BITRATE_ADJUSTMENT_SEC * deviationThresholdBytes;
// 把实际与期望的偏差控制在-3s ~ +3s的期望码率内。
deviationBytes = Math.min(deviationBytes, deviationCap);
deviationBytes = Math.max(deviationBytes, -deviationCap);// 统计时间，调整码率调控时间 (3s一次)
timeSinceLastAdjustmentMs += 1000.0 / targetFps;
if (timeSinceLastAdjustmentMs <= 1000 * BITRATE_ADJUSTMENT_SEC) {
return;
}if (deviationBytes > deviationThresholdBytes) {
// 编码器产生的比特率太高-需要按比例减小。
int bitrateAdjustmentInc = (int) (deviationBytes / deviationThresholdBytes + 0.5);
bitrateAdjustmentScaleExp -= bitrateAdjustmentInc;
// 不要让调整比例降到低于 -BITRATE_ADJUSTMENT_STEPS.
// 这保证(bitrateAdjustmentScaleExp / BITRATE_ADJUSTMENT_STEPS)最小指数设置为-1
bitrateAdjustmentScaleExp = Math.max(bitrateAdjustmentScaleExp, -BITRATE_ADJUSTMENT_STEPS);
deviationBytes = deviationThresholdBytes;
} else if (deviationBytes < -deviationThresholdBytes) {
// 编码器生成的比特率太低-需要按比例增加。
int bitrateAdjustmentInc = (int) (-deviationBytes / deviationThresholdBytes + 0.5);
bitrateAdjustmentScaleExp += bitrateAdjustmentInc;
// 不要让调整比例超过 BITRATE_ADJUSTMENT_STEPS.
// 这保证(bitrateAdjustmentScaleExp / BITRATE_ADJUSTMENT_STEPS)最大指数设置为1.
bitrateAdjustmentScaleExp = Math.min(bitrateAdjustmentScaleExp, BITRATE_ADJUSTMENT_STEPS);
deviationBytes = -deviationThresholdBytes;
}
timeSinceLastAdjustmentMs = 0;
关键逻辑就是根据情况算出bitrateAdjustmentScaleExp，得出合适的getBitrateAdjustmentScale。这个DynamicBitrateAdjuster场景在哪呢？就是客户端千变万化的网络场景，特别是弱网网络，根据探测的上行网络带宽，适当的调整编码输出码率，便于传输。
三、Encode
在AppWebRTC的demo当中，我们可以看到HardwareVideoEncoder的createOutputThread。执行deliverEncodedImage函数，其中就是在dequeueOutputBuffer输出编码数据后，通过bitrateAdjuster.getAdjustedBitrateBps()实时的调整码率。
private Thread createOutputThread() {
return new Thread() {
@Override
public void run() {
while (running) {
deliverEncodedImage();
}
releaseCodecOnOutputThread();
}
};
}
// Visible for testing.
protected void deliverEncodedImage() {
outputThreadChecker.checkIsOnValidThread();
try {
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
int index = codec.dequeueOutputBuffer(info, DEQUEUE_OUTPUT_BUFFER_TIMEOUT_US);
if (index < 0) {
if (index == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
outputBuffersBusyCount.waitForZero();
outputBuffers = codec.getOutputBuffers();
}
return;
}
ByteBuffer codecOutputBuffer = outputBuffers[index];
codecOutputBuffer.position(info.offset);
codecOutputBuffer.limit(info.offset + info.size);
if ((info.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
Logging.d(TAG, "Config frame generated. Offset: " + info.offset + ". Size: " + info.size);
configBuffer = ByteBuffer.allocateDirect(info.size);
configBuffer.put(codecOutputBuffer);
} else {
bitrateAdjuster.reportEncodedFrame(info.size);
if (adjustedBitrate != bitrateAdjuster.getAdjustedBitrateBps()) {
updateBitrate();
}
final boolean isKeyFrame = (info.flags & MediaCodec.BUFFER_FLAG_SYNC_FRAME) != 0;
if (isKeyFrame) {
Logging.d(TAG, "Sync frame generated");
}
// ...
}
} catch (IllegalStateException e) {
Logging.e(TAG, "deliverOutput failed", e);
}
}有输入才有输出，我们看看encode的输入：
@Override
public VideoCodecStatus encode(VideoFrame videoFrame, EncodeInfo encodeInfo) {
encodeThreadChecker.checkIsOnValidThread();
if (codec == null) {
return VideoCodecStatus.UNINITIALIZED;
}
final VideoFrame.Buffer videoFrameBuffer = videoFrame.getBuffer();
final boolean isTextureBuffer = videoFrameBuffer instanceof VideoFrame.TextureBuffer;
// If input resolution changed, restart the codec with the new resolution.
final int frameWidth = videoFrame.getBuffer().getWidth();
final int frameHeight = videoFrame.getBuffer().getHeight();
final boolean shouldUseSurfaceMode = canUseSurface() && isTextureBuffer;
if (frameWidth != width || frameHeight != height || shouldUseSurfaceMode != useSurfaceMode) {
VideoCodecStatus status = resetCodec(frameWidth, frameHeight, shouldUseSurfaceMode);
if (status != VideoCodecStatus.OK) {
return status;
}
}
if (outputBuilders.size() > MAX_ENCODER_Q_SIZE) {
// Too many frames in the encoder. Drop this frame.
Logging.e(TAG, "Dropped frame, encoder queue full");
return VideoCodecStatus.NO_OUTPUT; // See webrtc bug 2887.
}
boolean requestedKeyFrame = false;
for (EncodedImage.FrameType frameType : encodeInfo.frameTypes) {
if (frameType == EncodedImage.FrameType.VideoFrameKey) {
requestedKeyFrame = true;
}
}
if (requestedKeyFrame || shouldForceKeyFrame(videoFrame.getTimestampNs())) {
requestKeyFrame(videoFrame.getTimestampNs());
}
// Number of bytes in the video buffer. Y channel is sampled at one byte per pixel; U and V are
// subsampled at one byte per four pixels.
int bufferSize = videoFrameBuffer.getHeight() * videoFrameBuffer.getWidth() * 3 / 2;
EncodedImage.Builder builder = EncodedImage.builder()
.setCaptureTimeNs(videoFrame.getTimestampNs())
.setEncodedWidth(videoFrame.getBuffer().getWidth())
.setEncodedHeight(videoFrame.getBuffer().getHeight())
.setRotation(videoFrame.getRotation());
outputBuilders.offer(builder);
final VideoCodecStatus returnValue;
if (useSurfaceMode) {
returnValue = encodeTextureBuffer(videoFrame);
} else {
returnValue = encodeByteBuffer(videoFrame, videoFrameBuffer, bufferSize);
}
// Check if the queue was successful.
if (returnValue != VideoCodecStatus.OK) {
// Keep the output builders in sync with buffers in the codec.
outputBuilders.pollLast();
}
return returnValue;
}其中BlockingDeque<EncodedImage.Builder> outputBuilders；是输出对象队列，用于打时间戳、旋转角度等各种非必要的编码信息。其中看到encodeTextureBuffer / encodeByteBuffer两个输入方法。
encodeTextureBuffer代码的模式和上一节的 Android-RTC-7 介绍的是一样的。这里就不再详细展开；encodeByteBuffer 也比较简单，着重关注方法里面的fillInputBuffer，根据当前的yuvColorFormat，底层调用libyuv进行格式转换，然后填入MediaCodec。
下一章分析WebRTC的关键点： PeerConnection。新年假期简单码了一篇文章，也算是对得起自己之前承诺的保持学习，每月保持原创输出。在这新的虎年打算保持Android-RTC的产出，还打算输出Android有关于HDR实战专栏。希望能帮助大家。