标签：return RTC BitrateAdjuster int deviationBytes bitrateAdjustmentScaleExp Android
一、前提回顾
稍微回顾一下之前的内容，以编码DefaultVideoEncoderFactory为入口，createEncoder是由HardwareVideoEncoderFactory 和 SoftwareVideoEncoderFactory各自创建出来，然后回传到PeerConnectionClient。
@Nullable
public VideoEncoder createEncoder(VideoCodecInfo info) {
VideoEncoder softwareEncoder = this.softwareVideoEncoderFactory.createEncoder(info);
VideoEncoder hardwareEncoder = this.hardwareVideoEncoderFactory.createEncoder(info);
if (hardwareEncoder != null && softwareEncoder != null) {
return new VideoEncoderFallback(softwareEncoder, hardwareEncoder);
} else {
return hardwareEncoder != null ? hardwareEncoder : softwareEncoder;
}
}对于HardwareVideoEncoderFactory createEncoder创建硬解器，大概如下：
@Nullable
@Override
public VideoEncoder createEncoder(VideoCodecInfo input) {
VideoCodecMimeType type = VideoCodecMimeType.valueOf(input.name);
MediaCodecInfo info = findCodecForType(type);
String codecName = info.getName();
String mime = type.mimeType();
Integer surfaceColorFormat = MediaCodecUtils.selectColorFormat(
MediaCodecUtils.TEXTURE_COLOR_FORMATS, info.getCapabilitiesForType(mime));
Integer yuvColorFormat = MediaCodecUtils.selectColorFormat(
MediaCodecUtils.ENCODER_COLOR_FORMATS, info.getCapabilitiesForType(mime));
if (type == VideoCodecMimeType.H264) {
boolean isHighProfile = H264Utils.isSameH264Profile(
input.params, MediaCodecUtils.getCodecProperties(type, /* highProfile= */ true));
boolean isBaselineProfile = H264Utils.isSameH264Profile(
input.params, MediaCodecUtils.getCodecProperties(type, /* highProfile= */ false));
if (!isHighProfile && !isBaselineProfile) {
return null;
}
if (isHighProfile && !isH264HighProfileSupported(info)) {
return null;
}
}
return new HardwareVideoEncoder(new MediaCodecWrapperFactoryImpl(), codecName, type,
surfaceColorFormat, yuvColorFormat, input.params, getKeyFrameIntervalSec(type),
getForcedKeyFrameIntervalMs(type, codecName), createBitrateAdjuster(type, codecName),
sharedContext);
}createEncoder的流程在 Android-RTC-6 分析VideoEncoderFactory进行过较为详细的分析，这次深入HardwareVideoEncoder。
二、BitrateAdjuster
事不宜迟，赶紧进入分析，首先是初始化操作。
@Override
public VideoCodecStatus initEncode(Settings settings, Callback callback) {
// ...
useSurfaceMode = canUseSurface();
if (settings.startBitrate != 0 && settings.maxFramerate != 0) {
bitrateAdjuster.setTargets(settings.startBitrate * 1000, settings.maxFramerate);
}
adjustedBitrate = bitrateAdjuster.getAdjustedBitrateBps();
return initEncodeInternal();
}
private VideoCodecStatus initEncodeInternal() {
lastKeyFrameNs = -1;
try {
codec = mediaCodecWrapperFactory.createByCodecName(codecName);
} catch (IOException | IllegalArgumentException e) {
Logging.e(TAG, "Cannot create media encoder " + codecName);
return VideoCodecStatus.FALLBACK_SOFTWARE;
}
final int colorFormat = useSurfaceMode ? surfaceColorFormat : yuvColorFormat;
try {
MediaFormat format = MediaFormat.createVideoFormat(codecType.mimeType(), width, height)
format.setInteger(MediaFormat.KEY_BIT_RATE, adjustedBitrate);
format.setInteger(KEY_BITRATE_MODE, VIDEO_ControlRateConstant);
format.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat);
format.setInteger(MediaFormat.KEY_FRAME_RATE, bitrateAdjuster.getCodecConfigFramerate()
format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, keyFrameIntervalSec);
if (codecType == VideoCodecMimeType.H264) {
String profileLevelId = params.get(VideoCodecInfo.H264_FMTP_PROFILE_LEVEL_ID);
if (profileLevelId == null) {
profileLevelId = VideoCodecInfo.H264_CONSTRAINED_BASELINE_3_1;
}
switch (profileLevelId) {
case VideoCodecInfo.H264_CONSTRAINED_HIGH_3_1:
format.setInteger("profile", VIDEO_AVC_PROFILE_HIGH);
format.setInteger("level", VIDEO_AVC_LEVEL_3);
break;
case VideoCodecInfo.H264_CONSTRAINED_BASELINE_3_1:
break;
default:
Logging.w(TAG, "Unknown profile level id: " + profileLevelId);
}
}
codec.configure(
format, null /* surface */, null /* crypto */, MediaCodec.CONFIGURE_FLAG_ENCODE);
if (useSurfaceMode) {
textureEglBase = EglBase.createEgl14(sharedContext, EglBase.CONFIG_RECORDABLE);
textureInputSurface = codec.createInputSurface();
textureEglBase.createSurface(textureInputSurface);
textureEglBase.makeCurrent();
}
codec.start();
outputBuffers = codec.getOutputBuffers();
} catch (IllegalStateException e) {
Logging.e(TAG, "initEncodeInternal failed", e);
release();
return VideoCodecStatus.FALLBACK_SOFTWARE;
}
running = true;
outputThreadChecker.detachThread();
outputThread = createOutputThread();
outputThread.start();
return VideoCodecStatus.OK;
}1、initEncode都是一些MediaCodec的常规操作，就是封装得比较完善，看见一个bitrateAdjuster，主要意义用于码率调整适配器。是HardwareVideoEncoderFactory->createEncoder创建HardwareVideoEncoder对象的时候通过createBitrateAdjuster传递的一个对象。
2、initEncode还根据profileLevelId设置了profile / level，关于profileLevelId的相关知识，已在 Android-RTC-6 分析过，大家可以去看看。
这次让我们深入浅出的看看这个BitrateAdjuster是如何工作的。
private BitrateAdjuster createBitrateAdjuster(VideoCodecMimeType type, String codecName) {
if (codecName.startsWith(EXYNOS_PREFIX)) {
if (type == VideoCodecMimeType.VP8) {
// Exynos VP8 encoders need dynamic bitrate adjustment.
return new DynamicBitrateAdjuster();
} else {
// Exynos VP9 and H264 encoders need framerate-based bitrate adjustment.
return new FramerateBitrateAdjuster();
}
}
// Other codecs don't need bitrate adjustment.
return new BaseBitrateAdjuster();
}
一图看懂BitrateAdjuster的继承关系。主要工作是FramerateBitrateAdjuster 和 DynamicBitrateAdjuster。
class FramerateBitrateAdjuster extends BaseBitrateAdjuster {
private static final int INITIAL_FPS = 30;
@Override
public void setTargets(int targetBitrateBps, int targetFps) {
if (this.targetFps == 0) {
// Framerate-based bitrate adjustment always initializes to the same framerate.
targetFps = INITIAL_FPS;
}
super.setTargets(targetBitrateBps, targetFps);
this.targetBitrateBps = this.targetBitrateBps * INITIAL_FPS / this.targetFps;
}
@Override
public int getCodecConfigFramerate() {
return INITIAL_FPS;
}
}先看FramerateBitrateAdjuster，写法有点绕，其实就是固定帧率=30。然后根据设置的码率求 调整后的帧率对应的码率。（占百分比的关系，多数用于固定带宽的场景。）
class DynamicBitrateAdjuster extends BaseBitrateAdjuster {
private static final double BITS_PER_BYTE = 8.0;
// Change the bitrate at most once every three seconds.
private static final double BITRATE_ADJUSTMENT_SEC = 3.0;
// Maximum bitrate adjustment scale - no more than 4 times.
private static final double BITRATE_ADJUSTMENT_MAX_SCALE = 4;
// Amount of adjustment steps to reach maximum scale.
private static final int BITRATE_ADJUSTMENT_STEPS = 20;
// How far the codec has deviated above (or below) the target bitrate (tracked in bytes).
private double deviationBytes;
private double timeSinceLastAdjustmentMs;
private int bitrateAdjustmentScaleExp;
@Override
public void setTargets(int targetBitrateBps, int targetFps) {
if (this.targetBitrateBps > 0 && targetBitrateBps < this.targetBitrateBps) {
// Rescale the accumulator level if the accumulator max decreases
deviationBytes = deviationBytes * targetBitrateBps / this.targetBitrateBps;
}
super.setTargets(targetBitrateBps, targetFps);
}
@Override
public void reportEncodedFrame(int size) {
if (targetFps == 0) {
return;
}
// Accumulate the difference between actual and expected frame sizes.
double expectedBytesPerFrame = (targetBitrateBps / BITS_PER_BYTE) / targetFps;
deviationBytes += (size - expectedBytesPerFrame);
timeSinceLastAdjustmentMs += 1000.0 / targetFps;
// Adjust the bitrate when the encoder accumulates one second's worth of data in excess or
// shortfall of the target.
double deviationThresholdBytes = targetBitrateBps / BITS_PER_BYTE;
// Cap the deviation, i.e., don't let it grow beyond some level to avoid using too old data for
// bitrate adjustment. This also prevents taking more than 3 "steps" in a given 3-second cycle.
double deviationCap = BITRATE_ADJUSTMENT_SEC * deviationThresholdBytes;
deviationBytes = Math.min(deviationBytes, deviationCap);
deviationBytes = Math.max(deviationBytes, -deviationCap);
// Do bitrate adjustment every 3 seconds if actual encoder bitrate deviates too much
// from the target value.
if (timeSinceLastAdjustmentMs <= 1000 * BITRATE_ADJUSTMENT_SEC) {
return;
}
if (deviationBytes > deviationThresholdBytes) {
// Encoder generates too high bitrate - need to reduce the scale.
int bitrateAdjustmentInc = (int) (deviationBytes / deviationThresholdBytes + 0.5);
bitrateAdjustmentScaleExp -= bitrateAdjustmentInc;
// Don't let the adjustment scale drop below -BITRATE_ADJUSTMENT_STEPS.
// This sets a minimum exponent of -1 (bitrateAdjustmentScaleExp / BITRATE_ADJUSTMENT_STEPS).
bitrateAdjustmentScaleExp = Math.max(bitrateAdjustmentScaleExp, -BITRATE_ADJUSTMENT_STEPS);
deviationBytes = deviationThresholdBytes;
} else if (deviationBytes < -deviationThresholdBytes) {
// Encoder generates too low bitrate - need to increase the scale.
int bitrateAdjustmentInc = (int) (-deviationBytes / deviationThresholdBytes + 0.5);
bitrateAdjustmentScaleExp += bitrateAdjustmentInc;
// Don't let the adjustment scale exceed BITRATE_ADJUSTMENT_STEPS.
// This sets a maximum exponent of 1 (bitrateAdjustmentScaleExp / BITRATE_ADJUSTMENT_STEPS).
bitrateAdjustmentScaleExp = Math.min(bitrateAdjustmentScaleExp, BITRATE_ADJUSTMENT_STEPS);
deviationBytes = -deviationThresholdBytes;
}
timeSinceLastAdjustmentMs = 0;
}
private double getBitrateAdjustmentScale() {
return Math.pow(BITRATE_ADJUSTMENT_MAX_SCALE,
(double) bitrateAdjustmentScaleExp / BITRATE_ADJUSTMENT_STEPS);
}
@Override
public int getAdjustedBitrateBps() {
return (int) (targetBitrateBps * getBitrateAdjustmentScale());
}
}再看DynamicBitrateAdjuster，复写了reportEncodedFrame，此方法传入的参数 是 编码输出的实际字节大小。我们来看看其中的算法逻辑。
// 期望的每帧分配码率，转换大Byte为单位。
double expectedBytesPerFrame = (targetBitrateBps / BITS_PER_BYTE) / targetFps;
// 统计实际帧大小和预期帧大小之间的差异。
deviationBytes += (size - expectedBytesPerFrame);
// 期望的码率值(Byte)
double deviationThresholdBytes = targetBitrateBps / BITS_PER_BYTE;/**
* 限制偏差，也就是说，不要让偏差超过某个级别，以避免使用太旧的数据进行比特率调整。
* 这还可以防止在给定的3秒周期内采取超过3个“调整级别”。
*/
// 3s内的期望码率
double deviationCap = BITRATE_ADJUSTMENT_SEC * deviationThresholdBytes;
// 把实际与期望的偏差控制在-3s ~ +3s的期望码率内。
deviationBytes = Math.min(deviationBytes, deviationCap);
deviationBytes = Math.max(deviationBytes, -deviationCap);// 统计时间，调整码率调控时间 (3s一次)
timeSinceLastAdjustmentMs += 1000.0 / targetFps;
if (timeSinceLastAdjustmentMs <= 1000 * BITRATE_ADJUSTMENT_SEC) {
return;
}if (deviationBytes > deviationThresholdBytes) {
// 编码器产生的比特率太高-需要按比例减小。
int bitrateAdjustmentInc = (int) (deviationBytes / deviationThresholdBytes + 0.5);
bitrateAdjustmentScaleExp -= bitrateAdjustmentInc;
// 不要让调整比例降到低于 -BITRATE_ADJUSTMENT_STEPS.
// 这保证(bitrateAdjustmentScaleExp / BITRATE_ADJUSTMENT_STEPS)最小指数设置为-1
bitrateAdjustmentScaleExp = Math.max(bitrateAdjustmentScaleExp, -BITRATE_ADJUSTMENT_STEPS);
deviationBytes = deviationThresholdBytes;
} else if (deviationBytes < -deviationThresholdBytes) {
// 编码器生成的比特率太低-需要按比例增加。
int bitrateAdjustmentInc = (int) (-deviationBytes / deviationThresholdBytes + 0.5);
bitrateAdjustmentScaleExp += bitrateAdjustmentInc;
// 不要让调整比例超过 BITRATE_ADJUSTMENT_STEPS.
// 这保证(bitrateAdjustmentScaleExp / BITRATE_ADJUSTMENT_STEPS)最大指数设置为1.
bitrateAdjustmentScaleExp = Math.min(bitrateAdjustmentScaleExp, BITRATE_ADJUSTMENT_STEPS);
deviationBytes = -deviationThresholdBytes;
}
timeSinceLastAdjustmentMs = 0;
关键逻辑就是根据情况算出bitrateAdjustmentScaleExp，得出合适的getBitrateAdjustmentScale。这个DynamicBitrateAdjuster场景在哪呢？就是客户端千变万化的网络场景，特别是弱网网络，根据探测的上行网络带宽，适当的调整编码输出码率，便于传输。
三、Encode
在AppWebRTC的demo当中，我们可以看到HardwareVideoEncoder的createOutputThread。执行deliverEncodedImage函数，其中就是在dequeueOutputBuffer输出编码数据后，通过bitrateAdjuster.getAdjustedBitrateBps()实时的调整码率。
private Thread createOutputThread() {
return new Thread() {
@Override
public void run() {
while (running) {
deliverEncodedImage();
}
releaseCodecOnOutputThread();
}
};
}
// Visible for testing.
protected void deliverEncodedImage() {
outputThreadChecker.checkIsOnValidThread();
try {
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
int index = codec.dequeueOutputBuffer(info, DEQUEUE_OUTPUT_BUFFER_TIMEOUT_US);
if (index < 0) {
if (index == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
outputBuffersBusyCount.waitForZero();
outputBuffers = codec.getOutputBuffers();
}
return;
}
ByteBuffer codecOutputBuffer = outputBuffers[index];
codecOutputBuffer.position(info.offset);
codecOutputBuffer.limit(info.offset + info.size);
if ((info.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
Logging.d(TAG, "Config frame generated. Offset: " + info.offset + ". Size: " + info.size);
configBuffer = ByteBuffer.allocateDirect(info.size);
configBuffer.put(codecOutputBuffer);
} else {
bitrateAdjuster.reportEncodedFrame(info.size);
if (adjustedBitrate != bitrateAdjuster.getAdjustedBitrateBps()) {
updateBitrate();
}
final boolean isKeyFrame = (info.flags & MediaCodec.BUFFER_FLAG_SYNC_FRAME) != 0;
if (isKeyFrame) {
Logging.d(TAG, "Sync frame generated");
}
// ...
}
} catch (IllegalStateException e) {
Logging.e(TAG, "deliverOutput failed", e);
}
}有输入才有输出，我们看看encode的输入：
@Override
public VideoCodecStatus encode(VideoFrame videoFrame, EncodeInfo encodeInfo) {
encodeThreadChecker.checkIsOnValidThread();
if (codec == null) {
return VideoCodecStatus.UNINITIALIZED;
}
final VideoFrame.Buffer videoFrameBuffer = videoFrame.getBuffer();
final boolean isTextureBuffer = videoFrameBuffer instanceof VideoFrame.TextureBuffer;
// If input resolution changed, restart the codec with the new resolution.
final int frameWidth = videoFrame.getBuffer().getWidth();
final int frameHeight = videoFrame.getBuffer().getHeight();
final boolean shouldUseSurfaceMode = canUseSurface() && isTextureBuffer;
if (frameWidth != width || frameHeight != height || shouldUseSurfaceMode != useSurfaceMode) {
VideoCodecStatus status = resetCodec(frameWidth, frameHeight, shouldUseSurfaceMode);
if (status != VideoCodecStatus.OK) {
return status;
}
}
if (outputBuilders.size() > MAX_ENCODER_Q_SIZE) {
// Too many frames in the encoder. Drop this frame.
Logging.e(TAG, "Dropped frame, encoder queue full");
return VideoCodecStatus.NO_OUTPUT; // See webrtc bug 2887.
}
boolean requestedKeyFrame = false;
for (EncodedImage.FrameType frameType : encodeInfo.frameTypes) {
if (frameType == EncodedImage.FrameType.VideoFrameKey) {
requestedKeyFrame = true;
}
}
if (requestedKeyFrame || shouldForceKeyFrame(videoFrame.getTimestampNs())) {
requestKeyFrame(videoFrame.getTimestampNs());
}
// Number of bytes in the video buffer. Y channel is sampled at one byte per pixel; U and V are
// subsampled at one byte per four pixels.
int bufferSize = videoFrameBuffer.getHeight() * videoFrameBuffer.getWidth() * 3 / 2;
EncodedImage.Builder builder = EncodedImage.builder()
.setCaptureTimeNs(videoFrame.getTimestampNs())
.setEncodedWidth(videoFrame.getBuffer().getWidth())
.setEncodedHeight(videoFrame.getBuffer().getHeight())
.setRotation(videoFrame.getRotation());
outputBuilders.offer(builder);
final VideoCodecStatus returnValue;
if (useSurfaceMode) {
returnValue = encodeTextureBuffer(videoFrame);
} else {
returnValue = encodeByteBuffer(videoFrame, videoFrameBuffer, bufferSize);
}
// Check if the queue was successful.
if (returnValue != VideoCodecStatus.OK) {
// Keep the output builders in sync with buffers in the codec.
outputBuilders.pollLast();
}
return returnValue;
}其中BlockingDeque<EncodedImage.Builder> outputBuilders；是输出对象队列，用于打时间戳、旋转角度等各种非必要的编码信息。其中看到encodeTextureBuffer / encodeByteBuffer两个输入方法。
encodeTextureBuffer代码的模式和上一节的 Android-RTC-7 介绍的是一样的。这里就不再详细展开；encodeByteBuffer 也比较简单，着重关注方法里面的fillInputBuffer，根据当前的yuvColorFormat，底层调用libyuv进行格式转换，然后填入MediaCodec。
下一章分析WebRTC的关键点： PeerConnection。新年假期简单码了一篇文章，也算是对得起自己之前承诺的保持学习，每月保持原创输出。在这新的虎年打算保持Android-RTC的产出，还打算输出Android有关于HDR实战专栏。希望能帮助大家。
标签：return,RTC,BitrateAdjuster,int,deviationBytes,bitrateAdjustmentScaleExp,Android, 来源： https://blog.csdn.net/a360940265a/article/details/122583855
本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享； 2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关； 3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关； 4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除； 5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。