The 'State of Play' series is designed to offer insight into new evolutions and significant issues in the online video industry. The following episodes have already been published :
State of Play
This is an installment of our ongoing State of Play series, designed to offer insight into new evolutions and significant issues in the online video industry.
About a year ago, we wrote about video formats as part of an introduction to multimedia containers and codecs. Since then, the online video landscape has changed drastically by Google's acquisition of On2 and the subsequent launch of WebM as an open source video format. In this post, we'll try to shed some light on the ongoing 'WebM vs H.264' debate.
WebM / H264 Formats
On May 9, 2010 Google released WebM as an open and royalty-free video compression format for the web. The WebM project defines a container format based on a subset of the Matroska media container, which is designed to hold VP8 video and Vorbis audio streams.
H.264 is a video codec standard developed by the ITU-T Video Coding Experts Group (as H.264) together with the ISO/IEC Moving Picture Experts Group (as MPEG-4 Part 10, Advanced Video Coding, or AVC). During the last couple of years, it has become the de-facto compression standard for online video. When used in Flash or HTML5 video, it is often being used together with the AAC audio codec and stored inside of an MPEG-4 container. Other possible container formats include .mov (QuickTime), .3gp and .3gp2 (mobile) and .f4v (Flash).
Openness and Licensing
A lot of attention has been drawn to the free and open nature of the WebM format, as opposed to H.264 which has a patent license administered by a private organization known as MPEG LA. On August 26, 2010 MPEG LA announced that H.264 encoded internet video will perpetually remain free for non-commercial use, which means that you will never have to pay royalties for distributing free internet video encoded via H.264. However, license fees are required for other types of usage such as subscription based content services and device manufacturers.
The WebM project, on the other hand, is offering its code under a standard BSD license. Google has irrevocably released all of its patents on VP8 as a royalty-free format, but provides no indemnification for costs incurred should a patent lawsuit arise. That means that anyone distributing WebM/VP8 could be on the hook for any patent-related fees that might come up.
On February 11, 2011 MEPG LA announced that it is forming a patent pool and gathering claims from companies that believe they have patents essential to the VP8 codec, spurring uncertainty about a royalty-free future for WebM. The move resulted in an investigation by the US Justice Department to see whether MPEG LA is trying to stifle competition. Google reacted by launching a new community cross-licensing initiative, which was joined by consumer electronics manufacturers like Samsung, LG Electronics and Cisco.
There has also been some debate surrounding Google's assertion that it wants to 'enable open innovation'. There are now two viable branches of VP8, the one Google has retained in house, and another one expanded on by the open-source community. However, there's no third party entity that manages the evolution of the specification. And when it comes to important decisions like choosing the code to be used by hardware manufacturers for implementing silicon, Google will always have the final say - which is, incidentally, the same policy that Google is applying to other 'open source' projects/products like Android and Honeycomb.
Quality
H.264 is widely considered as today's most advanced video codec. By defining a set of encoding profiles and levels, the H.264 standard allows for playback on platforms with very divergent decoding capabilities. These profiles include a baseline profile (BP) aimed at applications with limited computing resources, a main profile (MP) intended as the mainstream consumer profile and a high profile (HiP) for HD and FullHD which requires lots of memory and processing power.
Quality seems to be the main point of criticism regarding WebM. Compression-wise, the standard is said to be comparable to the H.264 baseline profile. In reality, the encoder implementations are as important as the standard itself for evaluating quality. The Moscow State University’s Graphics and Media lab evaluated both the x264 decoder and the VP8 encoder and concluded that the latter shows “20-30% lower quality at average", while requiring slightly longer encoding times to reach the “normal” settings of x264 (baseline profile). However, WebM is judged to be performing significantly better than Theora or xvid. In contrast, a comparison by Streaming Media concluded that WebM is almost as good as H.264 "at most relevant data rates" for web video.
In terms of decoding speed, VP8 and H.264 seem to be relatively comparable. But H.264 already has a great deal of hardware support, which makes it decode faster in most real life circumstances. Particularly for portable devices, hardware acceleration is essential for video playback. In January 2011, the WebM project announced the availability of VP8 Hardware IP Designs and Chinese chip designer Rockchip announced the first processor with support for VP8. However, it will take time for WebM hardware acceleration to reach critical mass.
HTML5
The new HTML5 standard supports embedded multimedia, including video, audio, and dynamic graphics. The new <video> tag allows browsers to natively playback video, without needing plugins like Flash, Silverlight or QuickTime. However, disagreements in the W3C standards group resulted in an omission of a codec specification for the <video> tag. This has led to a stalemate, which is not likely to be resolved very soon. All browser vendors agree that their software should support the HTML5 <video> tag, but they can't agree on the video codec that will allow it. As a result, different browsers have native support for different video codecs.
On January 11, 2011 Google decided to drop support for H.264 in the Chromium open source browser (from which Chrome is built), in favor of WebM. This led to the current situation, in which Microsoft (IE 9) and Apple (Safari 5) support H.264, while Google (Chrome 12), Mozilla (Firefox 5) and Opera (10) support VP8 and Theora.
Both Apple and Microsoft, determined not to give Google control over the future of web video, have declared to be fully behind H.264 as the codec for HTML5 video going forward. Microsoft has even gone a step further, releasing plugins for Firefox and Chrome which allow them to play H.264 on Windows. Google in its turn has released a WebM plugin for Internet Explorer.
This division has been a major obstacle for the progress of HTML5 video. The majority of web video is currently encoded in H.264 and being played using Adobe Flash Player. Since this is still a viable option for all major browsers and H.264 also remains dominant on devices (Google has added WebM support to Android, but didn't remove support for H.264), there's no real incentive for video content owners to switch to HTML5.
VIDEO IN A Two-codec world
Even though H.264 is still dominant - according to MeFeedia, H.264 now accounts for 69% of all HTML5 video - WebM is expected to gradually gain market share, which may result in a two-codec world for the coming years. If your online videos are designed to stick around, you may want to start encoding into both formats. To do this transparently, Rambla customers can use their default hotfolder: simply upload your video files and they will get automatically published in both formats on the CDN. Developers can use our API's instead. For more information, take a look at our user guides or contact us.
This is an installment of our ongoing State of Play series, designed to offer insight into new evolutions and significant issues in the online video industry.
In a previous article, we presented you with an overview of the main streaming protocols. In this edition, we take a closer look at the new technique of adaptive streaming (aka bitrate switching) for on-demand media. In a future edition, we will further examine support for live streaming and discover how a single broadcast can result in multiple simultaneously live streams, using different streaming protocols.
TECHNOLOGY OVERVIEW
Adaptive streaming is a technology in which the bitrate of a stream is dynamically adapted to the user's needs. The technique consists in detecting the user's bandwidth and CPU capacity in real time and adjusting the quality of the video stream accordingly. If the viewer's bandwidth or CPU capacity drops, the player can automatically switch to a lower resolution without interrupting the viewing experience.
Basically, all adaptive bitrate streaming technologies work in the same way. In advance, the source video file (or live event) must be encoded to multiple resolutions and/or data rates. Before delivery, these encoded files are divided into segments - sometimes referred to as "chunks" - each representing two to ten seconds of video. When the client (= video player) first contacts the streaming server, it receives a table of URLs in which each URL points to a specific time interval (the columns) of a specific quality (the rows). A client then starts requesting segments from the Web server, downloading them via HTTP. As the segments are downloaded to the client, it plays back the segments in the order received. While playing, the client monitors playback-related indicators such as file download time, buffer levels, and CPU utilization. If desirable, the client can switch to a higher or lower quality stream, which should result in a seamless user experience.
MULTIPLE PROTOCOLS
One issue with Adaptive Streaming is that implementations were not born out of a widely accepted standard, but instead every vendor has defined its own proprietary protocol. Recently, the Motion Pictures Experts Group (MPEG) has started work on a common standard called DASH, but this still has a long way to go.
Currently, there are three main flavors of HTTP adaptive streaming - by Apple, Microsoft and Adobe - each targeting their own platform. All three specifications are codec agnostic in theory, but the current implementations are limited to certain video and audio codecs. The delivery mechanisms have a lot in common, but are nevertheless incompatible; for instance, each protocol provides the table of available URLs to the client in a different fashion.
Apple HTTP Adaptive Streaming
- Available on iPhone, iPad, and iPod touch (requires iOS version 3.0 or later) or any device with QuickTime X or later installed.
- The current implementation requires H.264 video streams and AAC or MP3 audio streams which are stored inside a MPEG-2 Transport Stream (MPEG-2 TS) container.
- The streaming server communicates the available streams and their URLs to the client by exposing an m3u playlist (.m3u8).
Microsoft Smooth Streaming
- Supported by Microsoft's Silverlight framework, which is also included on Windows Phone 7.
- The current Silverlight framework requires the H.264 or VC-1 video codec, and the AAC or WMA audio codec. The container format is “fragmented” MPEG-4, a variant of the ISO specification in which not all the data in a regular MP4 file is included.
- The streaming server sends an XML-based .ismc manifest back to the client, which describes the available streams and their URLs.
Adobe HTTP Dynamic Streaming
- Supported by Flash Player 10.1 (or higher) and Adobe AIR 2.
- The Flash player currently supports H.264 or VP6 video and AAC or MP3 audio. The container format can either be FLV or “fragmented” MPEG-4.
- The streaming server sends an XML-based .F4M manifest back to the client, which describes the available streams and their URLs.
Adobe RTMP Dynamic Streaming
Before Adobe released their version of HTTP Dynamic Streaming, they had already included support in Flash Player (version 10 or higher) for adaptive streaming over RTMP. From the client's perspective, the bitrate switching mechanism is similar. The major difference is that RTMP is a stateful protocol with a persistent client-server connection. See our overview of streaming protocols for the benefits / drawbacks of using RTMP instead of HTTP.
BENEFITS
Adaptive streaming is particularly well suited for internet devices, which have different requirements regarding memory and processing power. Connections may vary from high-end (e.g. broadband TV) to low-end (e.g. 3G networks) and are often fluctuating. Adaptive streaming allows these devices to dynamically choose a stream with a suitable resolution and bitrate.
To get a fast start-up time, clients will usually begin by downloading the lowest available quality. During this first download, the client is able to calculate the available network bandwidth. The client then uses this knowledge to switch to the highest available quality, considering the available network bandwidth and processing power. Clients should also be able to quickly jump to a different part of the video by calculating the time interval and requesting the corresponding segment.
From an end-user's point of view, HTTP adaptive streaming is fully transparent. The client just needs the URL that points to the playlist or manifest file (which will be different for each flavor of HTTP streaming). After making its initial request to this URL, the client retrieves all necessary data to dynamically stream the video.
For RTMP adaptive streaming, some additional client-side work is needed. This set of articles by 'Adobe Developer Connection' covers the details of how to achieve dynamic streaming in Flash. However, embeddable video players like JW Player and Flowplayer have already implemented the bulk of these mechanisms and only require some basic client-side configuration.
Should You Consider Adaptive streaming ?
As more and more internet devices appear on the market with greater capabilities for video browsing and playback, adaptive streaming is starting to become a mainstream technology. However, it is still a young technology and a lot of its (added) value depends on the intelligence that is built into client applications. Although some recent studies have shown that clients can react quite effectively to bandwidth variations, player implementations are also said to be "still at their infancy“, which in some cases may lead to heterogeneous user experiences.
In our own experience HTTP adaptive streaming generally works well, although there are some differences :
- Apple's adaptive streaming looks very solid. It switches transparently from one bitrate to another and is capable of selecting the most suitable stream in different circumstances. Moreover, the Apple client devices can rely on hardware acceleration support when rendering video, which allows them to playback high-quality video's more easily.
- Adobe's HTTP dynamic streaming still has some limitations. These are caused by the overhead of the Flash platform, and the lack of hardware acceleration support in some mobile devices (we'll elaborate on these issues in a future edition of 'state of play'). This makes it sometimes more difficult for the client to choose the most suitable stream. Another consequence is that a buffering delay may occur when the client has to switch back to a lower bitrate. This being said, the average Flash client performs well under sufficient bandwidth conditions and most modern devices are capable of rendering decent quality video.
- The Microsoft Smooth Streaming player is quite effective under sufficient and/or stable bandwidth conditions. After having made a connection, it quickly tries to build a large playback buffer at the highest possible bitrate: this way, it can continue to play video at the same bitrate for quite some time after the connection has deteriorated. Once this buffer is exhausted however, the player may experience difficulties when switching back to a lower bitrate. This may result in buffering delay(s) or unnecessary bitrate reductions.
Anyhow, player implementations are rapidly getting better and - certainly as far as mobile devices are concerned - HTTP adaptive streaming seems to be the technology of the future. Should you decide to use it, Rambla takes care of the deployment issues for you. We have automated the whole workflow process of transcoding your files into multiple resolutions/bitrates and creating the necessary configuration files. All you have to do is upload your source file via FTP to a so-called 'hotfolder' and wait for the notification that provides you with your URL's for adaptive streaming. A default hotfolder is now available for every user account, free of charge. For developers, our RATS API includes the same functionality.
This is an installment of our ongoing State of Play series, designed to offer insight into new evolutions and significant issues in the online video industry.
In this post, we'll try to give you an overview of the main methods and protocols which are currently employed to stream online video (for instance, we are not covering Peer-to-Peer streaming since it is still being considered as experimental in a CDN context). In our next post, we will look into more detail at the advanced technique of adaptive bitrate streaming (a.k.a. bitrate switching) which is quickly gaining popularity.
HTTP Pseudostreaming
HTTP Pseudostreaming is not 'true streaming', but a form of progressive download that allows viewers to seek to not-yet downloaded parts of a video. The delivery protocol is regular HTTP, combined with a mechanism that allows the player to include a start time parameter in the HTTP request's query-string. Using this mechanism, the client can do random seeking to any part of the timeline at any time.
Pseudostreaming has some advantages over true streaming:
- It leads to fewer hassles with firewalls and proxy servers (corporate environments).
- The initial loading time is usually shorter.
- It results in a better viewing experience if the local connection is below average.
There are also a number of drawbacks:
- In contrast to 'true streaming', pseudostreaming results in your digital files being downloaded to the physical drive on the end user's device.
- It leads to higher bandwidth consumption, since the client will try to download the file in its entirety (as opposed to true streaming where only the parts that you actually watch are being delivered).
- When seeking in large video's (> 15 minutes), pseudostreaming causes relatively long wait times before play is being resumed.
- For high-bandwidth videos, it may slow down (or even crash) a viewer's browser.
- Progressive download results in less precise/detailed statistics, since not all downloaded data has necessarily been viewed.
Stateful Streaming
In 'true streaming' protocols, the file is being sent to the end-user in a (more or less) constant stream and the user watches it as it arrives. For this purpose, traditional video streaming protocols maintain a persistent client-server connection that allows for real-time communication. The actual video and audio streams are being delivered via a dedicated streaming protocol built on top of TCP or UDP.
The main problem with this approach is that streams are frequently blocked by (mainly corporate) firewalls, since they are not being sent over HTTP (port 80). To circumvent this problem, protocols have been extended to allow for a stream to be encapsulated within HTTP requests. However, this technique - called 'tunneling' - comes at a performance cost and is therefore often only deployed as a fallback solution. Most of these protocols also have secure variants that use encryption algorithms to protect the stream.
Well known stateful streaming protocols include:
- Real Time Streaming Protocol (RTSP) : developed by the IETF and published in 1998 as rfc 2326. Most RTSP servers use the Real-time Transport Protocol (RTP) for media stream delivery, which supports a range of multimedia formats (such as H.264, MPEG-4, MJPEG, MPEG, etc.). Client applications include QuickTime, Skype and Windows Media Player. Most smartphone platforms (e.g. Android, Blackberry, but not iOS) also include support for RTSP as part of the 3GPP standard.
- Real Time Messaging Protocol (RTMP) : developed by Adobe Systems, which has recently released the specification for public use. RTMP is primarily used to stream audio and video over the internet to Adobe's Flash Player client. Due to the success of Flash Player, the bulk of streaming videos on the internet is currently delivered via RTMP or one of its variants (RTMPT, RTMPS, RTMPE, ...).
- MPEG transport stream (MPEG-TS) : a mechanism, defined in the MPEG-2 protocol, which controls the transmission and storage of audio, video, and data. It is mainly used in broadcast systems such as DVB and ATSC and enables connectivity with many of the conventional IPTV set-top boxes.
HTTP Streaming
HTTP streaming is a recent technology which is still maturing and would benefit from a widely accepted standard. Currently, separate solutions have been instituted by different vendors, which all use the same mechanism but are incompatible. From a client perspective, they all require the vendor's own software or platform:
Apple HTTP Adaptive Streaming : an HTTP-based media streaming communications protocol implemented by Apple Inc. as part of their QuickTime X framework and iOS 3.0 system. Apple has submitted its solution to the IETF for consideration as an Internet standard.- Microsoft Smooth Streaming : an IIS Media Services extension that enables adaptive streaming of media to clients over HTTP. Microsoft provides Smooth Streaming Client software development kits for Silverlight and Windows Phone 7.
- Adobe's HTTP Dynamic Streaming : Adobe released their own version of HTTP streaming (requires Flash Player 10.1) in June 2010 and is now trying to catch up with Apple and Microsoft. Since Flash Player is the main client platform targeted, this is also being referred to as HTTP Flash Streaming.
Streaming over HTTP works by breaking the overall stream up into a sequence of small HTTP-based file downloads, each download loading one short chunk of an overall potentially unbounded transport stream. By using HTTP as the stream transfer protocol, firewall issues are generally avoided. Downsides of this approach are a lesser (default) security and the absence of metadata, which makes it harder to ensure quality of service.
All flavors of HTTP streaming include support for adaptive streaming (a.k.a. bitrate switching) which allows clients to dynamically switch between different streams of varying quality and size during playback, in order to adapt to changing bandwidth conditions and available CPU resources. While mainly associated with HTTP streaming, Adobe has also included support for adaptive streaming into the RTMP protocol.
Another would-be advantage of HTTP streaming is that it allows the HTTP chunks to be cached within ISPs or corporations, in contrast to video streamed via RTMP. This would reduce the overall bandwidth required to deliver HTTP streams. However, research by Streaming Media concluded that "practically speaking, it appears that the benefit of cache servers is so unquantifiable that it is irrelevant".
Recently, the Motion Pictures Experts Group (MPEG) has started work on a specification for Dynamic Adaptive Streaming over HTTP (DASH) in an effort to standardize video streaming over HTTP. However, lots of preliminary issues still need to be resolved - including legal ones (patent claims) - before DASH can become a widely used standard.
A Client Perspective
Whereas most clients allow video's to play via (progressive) download, support for streaming protocols varies strongly depending on your choice of player, device and/or platform.
- iOS 3.0+ devices (iPhone, iPad, iPod touch) support Apple HTTP Adaptive Streaming.
- Windows Phone 7 client libraries support Microsoft Smooth Streaming.
- Most smartphones (3GPP) support RTSP.
- Flash Player supports RTMP and, starting with version 10.1, also Adobe's HTTP Dynamic Streaming
- QuickTime supports RTSP, QuickTime X (OS X 10.6) also supports Apple HTTP Adaptive Streaming.
- Silverlight player (3.0 or later) supports Microsoft Smooth Streaming
- ...
This fragmentation has to be taken into account when streaming to multiple platforms (e.g. in case of a live-stream, where it is impossible to fall back to progressive download). As a Rambla customer, you can opt for a mix of different techniques to reach your target audience. Our CDN supports all streaming methods described in this post, both for on-demand and live-streaming. Moreover, it suffices to publish a single .mp4 file (H.264/AAC) on the CDN to make it accessible to all players and devices mentioned here. For more details, see our documentation or contact us.
This is an installment in our ongoing State of Play series, designed to offer insight into new evolutions and significant issues in the online video industry.
Lack of Common Standard
During the last decade, lots of standardization efforts have gone into extending the Semantic Web. 
However, most efforts seem to have concentrated on the interchange of data and the representation of documents or HTML pages. When it comes to multimedia objects, there has been a lack of widely accepted protocols and/or ontologies.
There have been standardization efforts like MPEG-7 and MPEG-21 but they are known for their complexity and have largely been ignored on the web. Other open protocols, like those developed by the Open Archives Initiative (OAI-ORE, OAI-PMH) are being used for one specific purpose only. In practice, organisations have often resorted to the generic Dublin Core protocol, which is not particularly suited for representing multimedia. This has resulted in some industry standards like EBU-CORE or the IPTC protocols, but their usage is mostly limited to industry applications. More recently, there have been open initiatives like Microformats and RDFa vocabularies but none of them gained wide adoption. Finally, some corporations have defined their own proprietary standards: these include Google's YouTube Data API, Yahoo's Media RSS and Facebook's Open Graph Protocol.
Recent Activity
Following the Video on the Web Workshop in December 2007, the W3C released a Video in the Web Activity Statement: "The goal of this activity is to make video a 'first class citizen' of the Web.
Video on the Web (and this includes audio, as the two are typically used together) has seen explosive growth, improving the richness of the user experience but leading to challenges in content discovery, searching, indexing and accessibility. Enabling users (from individuals to large organizations) to put video in the Web requires that we build a solid architectural foundation that enables people to create, navigate, search, link and distribute video, effectively making video part of the Web instead of an extension that doesn't take full advantage of the Web architecture."
As a result of the Video in the Web Activity, the Media Annotations Working Group was instituted. Its mission is to "provide an ontology and API designed to facilitate cross-community data integration of information related to media objects in the Web, such as video, audio and images". The ontology draft defines a core vocabulary of media properties and a set of mappings between the existing metadata formats. The API specification focuses on the methodology; it defines a client-side API to access the metadata. For both specifications, a Last Call Working Draft has been published.
Conclusion
Currently, there is no prevailing protocol or ontology for attaching metadata to multimedia resources. The 'Media Annotations Working Group' initiative could eventually lead to a widely accepted standard, but it still has a long way to go. In the meantime, users will have to look for a pragmatic approach based on existing standards and their own particular needs.
In order to permit Rambla customers to benefit from storing and managing their metadata at CDN level, we have released a metadata extension to our RAWS API's. These new metadata services - originally built into RAWS as the result of a research project funded by the Flemish Institute of Science and Technology (IWT) - are designed to give you maximum freedom and flexibility in choosing or defining your ontologies and can be extended to support different API's (like the one specified by the 'Media Annotations Working Group') in the future.
