[WebRTC] Audio Codec Encoder 基类注解

Audio Codec Encoder 中包含大量错误重传、减少传输量、QoS分析、网络质量等的优化部分。如:

  • FEC(前向纠错forward error correction),
  • VAD(静音检测,Voice Activity Detector),
  • DTX(不连续传输Discontinuous Transmission).
  • RTT(往返时延 round-trip time )
  • ANA 。。。

真正的编码在EncodeImpl()函数中。

音频帧率计算方法:

  • 格式(编码字节数、采样一位所占的字节数) format = s16(格式)=16(bit)
  • 声道数 channels = 2
  • 一次采样(一秒中所占的位数)TotalBit = sampling * channels * format = 1411200
  • 一次采样(一秒中所占的字节数)TotalByte = TotalBit/8 = 176400
  1. AAC:
  • nb_samples和frame_size = 1024
  • 一帧数据量:10242s16/8 = 4096个字节。
  • ACC帧率 (一秒播放帧数)= TotalByte/4096 = 43.06640625帧
  1. MP3:
  • nb_samples和frame_size = 1152
  • 一帧数据量:11522s16/8 = 4608个字节。
  • MP3帧率 (一秒播放帧数)= TotalByte/4608 = 38.28125帧

webrtc\src\api\audio_codecs\audio_encoder.h

// This is the interface class for encoders in AudioCoding module. Each codec
// type must have an implementation of this class.
class AudioEncoder {
 public:
  

  virtual ~AudioEncoder() = default;

  // 设置采样率和通道数。
  virtual int SampleRateHz() const = 0;
  virtual size_t NumChannels() const = 0;

  // 返回采样率 SampleRate Hz. 采样率从8 kHz(窄带)到48 kHz(全频)
  // 人对频率的识别范围是 20HZ ~ 20kHZ
  // 电话采样率 8kHZ
  virtual int RtpTimestampRateHz() const;

  // 下一个编码包中,10 ms内的编码帧数。
  // 每个包编码出来的帧数可能会不一样。
  virtual size_t Num10MsFramesInNextPacket() const = 0;

  // 10ms编码 最大帧数
  virtual size_t Max10MsFramesInAPacket() const = 0;

  // 当前的比特率 bits/s (码率). 
  //Opus 支持恒定比特率(CBR)和可变比特率(VBR)
  virtual int GetTargetBitrate() const = 0;


  // 编码动作,实际会调用 EncodeImpl()。
  // 输入一个10 ms音频包
  // 多通道音频需要交叉编码。
  // Audio.size() == SampleRateHz() * NumChannels / 100
  EncodedInfo Encode(uint32_t rtp_timestamp,
                     rtc::ArrayView<const int16_t> audio,
                     rtc::Buffer* encoded);

  // 编码结果包发出之前,重新编码
  virtual void Reset() = 0;

  // Enables or disables codec-internal FEC (forward error correction).
  // 名词:NACK重传, FEC(前向纠错)
  // 题外:如果视频Codec选择为H264的时候, FEC,RED是被关闭的。
  // 		参见rtp_video_sender.cc
  virtual bool SetFec(bool enable);

  // DTX(不连续传输)
  // Enables or disables codec-internal VAD(静音检测,Voice Activity Detector)/DTX(不连续传输Discontinuous Transmission)/. 
  // 主要应用在不活跃的语音周期中,降低传输速率,同时保持可接受的输出质量。
  // VAD将输入信号分类为活动语音、非活动语音和背景噪声。
  // 基于VAD决策,DTX在静默期间插入静默插入描述符(SID)帧。
  // 在静默期间,SID帧被周期性地发送到CNG(舒适噪音生成)模块,该模块在接收端的非活动语音期间产生环境噪声。
  virtual bool SetDtx(bool enable);

  // Returns the status of codec-internal DTX. 
  virtual bool GetDtx() const;

  // Sets the application mode. 
  enum class Application { kSpeech, kAudio };
  virtual bool SetApplication(Application application);

  // 设置播放(decoder)的最大采样率。
  virtual void SetMaxPlaybackRate(int frequency_hz);


  RTC_DEPRECATED virtual void SetTargetBitrate(int target_bps);

  // NOTE: This method is subject to change. Do not call or override it.
  virtual rtc::ArrayView<std::unique_ptr<AudioEncoder>>
  ReclaimContainedEncoders();

  // Enables audio network adaptor. Returns true if successful.
  virtual bool EnableAudioNetworkAdaptor(const std::string& config_string,
                                         RtcEventLog* event_log);

  // Disables audio network adaptor.
  virtual void DisableAudioNetworkAdaptor();

  // Provides uplink packet loss fraction to this encoder to allow it to adapt.
  // 上行链路包丢失片断处理
  // |uplink_packet_loss_fraction| is in the range [0.0, 1.0].
  // The uplink packet loss fractions as set by the ANA FEC controller.
  virtual void OnReceivedUplinkPacketLossFraction(
      float uplink_packet_loss_fraction);

  // 可以FEC前向纠错的部分。
  // Provides 1st-order-FEC-recoverable uplink packet loss rate to this encoder
  // to allow it to adapt.
  // |uplink_recoverable_packet_loss_fraction| is in the range [0.0, 1.0].
  virtual void OnReceivedUplinkRecoverablePacketLossFraction(
      float uplink_recoverable_packet_loss_fraction);

  // Provides target audio bitrate to this encoder to allow it to adapt.
  virtual void OnReceivedTargetAudioBitrate(int target_bps);

  // Provides target audio bitrate and corresponding probing interval of
  // the bandwidth estimator to this encoder to allow it to adapt.
  virtual void OnReceivedUplinkBandwidth(int target_audio_bitrate_bps,
                                         absl::optional<int64_t> bwe_period_ms);

  // Provides target audio bitrate and corresponding probing interval of
  // the bandwidth estimator to this encoder to allow it to adapt.
  virtual void OnReceivedUplinkAllocation(BitrateAllocationUpdate update);

  // RTT:round-trip time(往返时延),是指从数据包发送开始,到接收端确认接收,
  // 然后发送确认给发送端总共经历的延时,注意:不包括接收端处理需要的耗时。
  // Sender:s(t0)-------------------------------------->Receiver:r(t1)
  // Sender:r(t3)<---------------------------------------Receiver:s(t2)
  //  rtt时间=t1-t0+t3-t2=t3-t0-(t2-t1)=t3-t0-d      d(接收端处理耗时)
  // Provides RTT to this encoder to allow it to adapt.
  virtual void OnReceivedRtt(int rtt_ms);

  // Overhead 每个包中Encoder带来的字节大小。
  // Provides overhead to this encoder to adapt. The overhead is the number of
  // bytes that will be added to each packet the encoder generates.
  virtual void OnReceivedOverhead(size_t overhead_bytes_per_packet);

  // To allow encoder to adapt its frame length, it must be provided the frame
  // length range that receivers can accept.
  virtual void SetReceiverFrameLengthRange(int min_frame_length_ms,
                                           int max_frame_length_ms);

  // 网络适配 统计用
  // Get statistics related to audio network adaptation.
  virtual ANAStats GetANAStats() const;

 protected:
  // 真正的Encoding部分
  // Subclasses implement this to perform the actual encoding. Called by
  // Encode().
  virtual EncodedInfo EncodeImpl(uint32_t rtp_timestamp,
                                 rtc::ArrayView<const int16_t> audio,
                                 rtc::Buffer* encoded) = 0;
};

[WebRTC] Audio Codec Encoder 基类注解

上一篇:javascript – 在JS中检测双Ctrl按键


下一篇:javascript – 处理鼠标点击和keydown或按键事件(非修饰键)的事件