Audio Codec Encoder 中包含大量错误重传、减少传输量、QoS分析、网络质量等的优化部分。如:
- FEC(前向纠错forward error correction),
- VAD(静音检测,Voice Activity Detector),
- DTX(不连续传输Discontinuous Transmission).
- RTT(往返时延 round-trip time )
- ANA 。。。
真正的编码在EncodeImpl()函数中。
音频帧率计算方法:
- 格式(编码字节数、采样一位所占的字节数) format = s16(格式)=16(bit)
- 声道数 channels = 2
- 一次采样(一秒中所占的位数)TotalBit = sampling * channels * format = 1411200
- 一次采样(一秒中所占的字节数)TotalByte = TotalBit/8 = 176400
- AAC:
- nb_samples和frame_size = 1024
- 一帧数据量:10242s16/8 = 4096个字节。
- ACC帧率 (一秒播放帧数)= TotalByte/4096 = 43.06640625帧
- MP3:
- nb_samples和frame_size = 1152
- 一帧数据量:11522s16/8 = 4608个字节。
- MP3帧率 (一秒播放帧数)= TotalByte/4608 = 38.28125帧
webrtc\src\api\audio_codecs\audio_encoder.h
// This is the interface class for encoders in AudioCoding module. Each codec
// type must have an implementation of this class.
class AudioEncoder {
public:
virtual ~AudioEncoder() = default;
// 设置采样率和通道数。
virtual int SampleRateHz() const = 0;
virtual size_t NumChannels() const = 0;
// 返回采样率 SampleRate Hz. 采样率从8 kHz(窄带)到48 kHz(全频)
// 人对频率的识别范围是 20HZ ~ 20kHZ
// 电话采样率 8kHZ
virtual int RtpTimestampRateHz() const;
// 下一个编码包中,10 ms内的编码帧数。
// 每个包编码出来的帧数可能会不一样。
virtual size_t Num10MsFramesInNextPacket() const = 0;
// 10ms编码 最大帧数
virtual size_t Max10MsFramesInAPacket() const = 0;
// 当前的比特率 bits/s (码率).
//Opus 支持恒定比特率(CBR)和可变比特率(VBR)
virtual int GetTargetBitrate() const = 0;
// 编码动作,实际会调用 EncodeImpl()。
// 输入一个10 ms音频包
// 多通道音频需要交叉编码。
// Audio.size() == SampleRateHz() * NumChannels / 100
EncodedInfo Encode(uint32_t rtp_timestamp,
rtc::ArrayView<const int16_t> audio,
rtc::Buffer* encoded);
// 编码结果包发出之前,重新编码
virtual void Reset() = 0;
// Enables or disables codec-internal FEC (forward error correction).
// 名词:NACK重传, FEC(前向纠错)
// 题外:如果视频Codec选择为H264的时候, FEC,RED是被关闭的。
// 参见rtp_video_sender.cc
virtual bool SetFec(bool enable);
// DTX(不连续传输)
// Enables or disables codec-internal VAD(静音检测,Voice Activity Detector)/DTX(不连续传输Discontinuous Transmission)/.
// 主要应用在不活跃的语音周期中,降低传输速率,同时保持可接受的输出质量。
// VAD将输入信号分类为活动语音、非活动语音和背景噪声。
// 基于VAD决策,DTX在静默期间插入静默插入描述符(SID)帧。
// 在静默期间,SID帧被周期性地发送到CNG(舒适噪音生成)模块,该模块在接收端的非活动语音期间产生环境噪声。
virtual bool SetDtx(bool enable);
// Returns the status of codec-internal DTX.
virtual bool GetDtx() const;
// Sets the application mode.
enum class Application { kSpeech, kAudio };
virtual bool SetApplication(Application application);
// 设置播放(decoder)的最大采样率。
virtual void SetMaxPlaybackRate(int frequency_hz);
RTC_DEPRECATED virtual void SetTargetBitrate(int target_bps);
// NOTE: This method is subject to change. Do not call or override it.
virtual rtc::ArrayView<std::unique_ptr<AudioEncoder>>
ReclaimContainedEncoders();
// Enables audio network adaptor. Returns true if successful.
virtual bool EnableAudioNetworkAdaptor(const std::string& config_string,
RtcEventLog* event_log);
// Disables audio network adaptor.
virtual void DisableAudioNetworkAdaptor();
// Provides uplink packet loss fraction to this encoder to allow it to adapt.
// 上行链路包丢失片断处理
// |uplink_packet_loss_fraction| is in the range [0.0, 1.0].
// The uplink packet loss fractions as set by the ANA FEC controller.
virtual void OnReceivedUplinkPacketLossFraction(
float uplink_packet_loss_fraction);
// 可以FEC前向纠错的部分。
// Provides 1st-order-FEC-recoverable uplink packet loss rate to this encoder
// to allow it to adapt.
// |uplink_recoverable_packet_loss_fraction| is in the range [0.0, 1.0].
virtual void OnReceivedUplinkRecoverablePacketLossFraction(
float uplink_recoverable_packet_loss_fraction);
// Provides target audio bitrate to this encoder to allow it to adapt.
virtual void OnReceivedTargetAudioBitrate(int target_bps);
// Provides target audio bitrate and corresponding probing interval of
// the bandwidth estimator to this encoder to allow it to adapt.
virtual void OnReceivedUplinkBandwidth(int target_audio_bitrate_bps,
absl::optional<int64_t> bwe_period_ms);
// Provides target audio bitrate and corresponding probing interval of
// the bandwidth estimator to this encoder to allow it to adapt.
virtual void OnReceivedUplinkAllocation(BitrateAllocationUpdate update);
// RTT:round-trip time(往返时延),是指从数据包发送开始,到接收端确认接收,
// 然后发送确认给发送端总共经历的延时,注意:不包括接收端处理需要的耗时。
// Sender:s(t0)-------------------------------------->Receiver:r(t1)
// Sender:r(t3)<---------------------------------------Receiver:s(t2)
// rtt时间=t1-t0+t3-t2=t3-t0-(t2-t1)=t3-t0-d d(接收端处理耗时)
// Provides RTT to this encoder to allow it to adapt.
virtual void OnReceivedRtt(int rtt_ms);
// Overhead 每个包中Encoder带来的字节大小。
// Provides overhead to this encoder to adapt. The overhead is the number of
// bytes that will be added to each packet the encoder generates.
virtual void OnReceivedOverhead(size_t overhead_bytes_per_packet);
// To allow encoder to adapt its frame length, it must be provided the frame
// length range that receivers can accept.
virtual void SetReceiverFrameLengthRange(int min_frame_length_ms,
int max_frame_length_ms);
// 网络适配 统计用
// Get statistics related to audio network adaptation.
virtual ANAStats GetANAStats() const;
protected:
// 真正的Encoding部分
// Subclasses implement this to perform the actual encoding. Called by
// Encode().
virtual EncodedInfo EncodeImpl(uint32_t rtp_timestamp,
rtc::ArrayView<const int16_t> audio,
rtc::Buffer* encoded) = 0;
};