17.3 Real-Time Interactive Audio and Video

Real-time multimedia applications face unique challenges in computer networks. Low latency, high bandwidth, and synchronization are crucial for smooth video conferencing and streaming. These apps must handle jitter and tolerate some errors while maintaining quality.

Protocols like SIP and RTP enable multimedia communication. SIP manages session setup and teardown, while RTP handles media transport. Quality of Service techniques like forward error correction and adaptive bitrate streaming help overcome network issues and ensure a good user experience.

Real-Time Multimedia Challenges and Requirements

Challenges of real-time multimedia applications

Top images from around the web for Challenges of real-time multimedia applications

• 

  Latency lags bandwidth - Marc's Blog View original

  Is this image relevant?

• 

  5G Technology View original

  Is this image relevant?

• 

  Latency lags bandwidth - Marc's Blog View original

  Is this image relevant?

• 

  5G Technology View original

  Is this image relevant?

1 of 2

Top images from around the web for Challenges of real-time multimedia applications

• 

  Latency lags bandwidth - Marc's Blog View original

  Is this image relevant?

• 

  5G Technology View original

  Is this image relevant?

• 

  Latency lags bandwidth - Marc's Blog View original

  Is this image relevant?

• 

  5G Technology View original

  Is this image relevant?

1 of 2

• Low latency required to enable real-time interaction between participants (video conferencing)
  • End-to-end delay should be minimized, typically less than 150 ms for acceptable user experience
• High bandwidth requirements compared to traditional data applications
  • Video conferencing requires bandwidth ranging from hundreds of kbps to several Mbps depending on video quality and resolution (HD, 4K)
• Synchronization of audio and video streams necessary to maintain lip-sync
  • Timestamps and sequence numbers used to ensure proper synchronization between media streams
• Error tolerance higher compared to traditional data applications
  • Limited packet loss can be concealed using techniques such as forward error correction and interpolation (frame duplication)
• Jitter handling essential to ensure smooth playback of media streams
  • Jitter buffers used to smooth out variations in packet arrival times caused by network delays

Signaling and Transport Protocols for Real-Time Multimedia

Role of signaling protocols

• Session Initiation Protocol (SIP) used for establishing, modifying, and terminating multimedia sessions
  • SIP messages negotiate session parameters such as codecs, transport protocols, and IP addresses
  • Utilizes a client-server architecture with user agents (UAs) acting as clients and servers (SIP proxy)
  • SIP URIs identify users and resources (sip:user@domain.com)
• Session Description Protocol (SDP) describes multimedia session parameters
  • Includes information such as media types, codecs, transport protocols, and IP addresses
  • Carried as a payload in SIP messages during session establishment
• SIP manages the lifecycle of multimedia sessions
  • Messages such as INVITE, ACK, and BYE used to establish, modify, and terminate sessions
  • Supports advanced call management features like call transfer and conference calling

Transport protocols for multimedia

• Real-time Transport Protocol (RTP) designed for real-time multimedia applications
  • Provides features such as timestamping, sequence numbering, and payload type identification
  • Typically runs on top of UDP to minimize latency and overhead
  • Does not guarantee quality of service (QoS) or reliable delivery
• RTP Control Protocol (RTCP) serves as a companion protocol to RTP for monitoring and control purposes
  • Provides feedback on the quality of the RTP session such as packet loss, jitter, and round-trip time
  • RTCP reports sent periodically by each participant in the RTP session
  • Helps in adapting to network conditions and maintaining QoS
• Comparison between RTP and RTCP
  • RTP used for media transport, while RTCP used for monitoring and control
  • RTP and RTCP use different port numbers, typically with RTCP using the next higher odd port number
  • RTP and RTCP packets multiplexed on the same transport-layer connection to minimize latency and overhead

Quality of Service Techniques for Real-Time Multimedia

Quality of service techniques

• Forward error correction (FEC) mitigates the impact of packet loss on real-time multimedia
  • Adds redundant data to the media stream, allowing the receiver to reconstruct lost packets
  • FEC schemes such as Reed-Solomon and Raptor codes generate redundant packets based on the original media packets
  • Introduces additional latency and bandwidth overhead, requiring a trade-off between error resilience and efficiency
• Jitter buffering smooths out variations in packet arrival times
  • Jitter buffers temporarily store incoming packets and release them at a constant rate to the decoder
  • Adaptive jitter buffers dynamically adjust their size based on observed network conditions
  • Introduces additional latency, requiring a trade-off between smoothness and responsiveness
• Adaptive bitrate streaming adapts the media bitrate to the available network bandwidth
  • Media content encoded at multiple bitrates, and the appropriate bitrate selected based on network conditions
  • Adaptive bitrate algorithms such as MPEG-DASH and HLS use feedback from the client to make bitrate decisions
  • Helps in maintaining QoS by avoiding buffer underruns and overruns (stalling, buffering)
• Quality of Experience (QoE) monitoring assesses the perceived quality of the real-time multimedia session
  • QoE metrics such as Mean Opinion Score (MOS) and Video Quality Metric (VQM) provide a quantitative measure of user experience
  • Helps in identifying and troubleshooting quality issues in real-time multimedia applications (blurry video, audio distortion)