Intro to Video Codecs

By Xah Lee. Date:

This page provides a survey of the current video/audio codecs, file formats, streaming tech, for streaming multimedia.

I need to setup embedded videos on my site as well sites i work for. So, today, i started a comprehensive study on video streaming. Here's some learning notes. (The last time i worked with video is in 1995 to 1997. At the time video streaming is still pretty much science fiction)

The technology involved for playing a video from a browser from a site, such as youtube, involves several separate technologies. Here's a summary of the basics:

Video Codecs

Here is the most popular or important ones.


H.264 (aka MPEG-4 part 10, AVC (Advanced Video Coding)). It is currently the most popular video codec. It is used in Blu-ray Disc, YouTube, iTunes, and many nation's broadcasting and other video related applications.

First version of H.264 was completed in 2003-05.


VP8 is a competitor to H.264. Currently owned by Google and released as open source.

The associated container format with VP8 is WebM. WebM format is also open source, and is based on the free Matroska container format. WebM file format is competing to be the default video format for HTML Video.

[see HTML Video Tag]

WMV (VC-1)

Windows Media Video (WMV), refers to several video codecs from Microsoft, but mostly the latest WMV 9 (aka VC-1). WMV 9 is released to standard body and standardized as VC-1, in 2006.

It is widely supported, and is used in Blue-ray Discs, Xbox 360, PlayStation 3. It's a competitor to H.264.

Besides WMV 9, there's 2 other codecs: WMV Screen and VMV Image. The screen one is optimized for screenshots, for example, tutorials on using a application. The Image one is optimized for slideshows.


Another widely used one is from Apple's QuickTime. (see below)


Sorenson codec

Sorenson codec refers to 2 proprietary codecs.


Theora is a free lossy video compression codec. The technical quality of Theora is not as good as H.264 or VP8. It is based on VP2 format of 2002, then a proprietary format by On2 released as free. It is not widely supported. Theora is usually stored in the Ogg container format, together with the free lossy audio codec Vorbis.

DivX and xvid

DivX started as a open source project in ~2000 but became proprietary, and Xvid is forked from it. Most pirated movies from DVDs are saved in DivX or Xvid formats. Both do not particularly define new codecs or container formats, rather, they are based on some subset of MPEG-4 standard and other container formats. It began as a reverse-engineer of Microsoft's MPEG-4 version 3 codec.


There are tens of audio codecs, some are lossy, some lossless. Here's some popular ones.



wav and aiff both support compression, but audio stored in these formats are almost always not compressed.

Audio Formats Do Not Matter Much Anymore

Note that 300 kilo bits per sec gets you CD quality audio (using a lossy compression). While a DVD quality video is about 5 mega bits/s. That's about 17 times more.

The need for audio codec research has past. Computer storage and processing power today can deal with audio no problem, and use of lossless codec for audio is increasingly popular. So, for issues of movie streaming, the video part is the primary concern.

Multimedia Container Formats

QuickTime (“.mov” or “.qt”) is Apple's container format. Widely used.

AVI is Microsoft's tech, fairly old, started in early 1990s. Widely used.

Advanced Systems Format (ASF) is Microsoft's container format, part of the Microsoft's Windows Media framework.

MPEG-4 Part 14 (.mp4) is a standardized container format. Variant file name extensions are M4V (.m4v) for video, “.m4a” for files containing audio only. The “.m4v” and “.m4a” suffixes are popularized by Apple's iTunes, iPod, iPhone etc.

3GP and 3G2 (.3gp, .3g2) are video/audio container formats for mobile phones. They are also based on MPEG-4 Part 14 (.mp4).

Adobe Flash is a widely used for online video. Flash files have several extension variations, most popular is “.swf”. (Adobe Flash is a application, which supports multimedia. Adobe Flash by itself isn't just a container format.) Flash files dedicated to video are Flash Video files, which is container format, with these extensions: “.flv”, “.f4v”, “.f4p”, “.f4a”, “.f4b”. Flash Video is used by for example YouTube, Google Video, “Yahoo! Video”. The FLV format is old, started in 2002 and is being phased out due to limitations. FLV is also commonly used with older codecs such as Sorenson and VP6. Now the recommended format is F4V, created in 2007.

File ExtensionInternet Media TypeMeaning
.f4vvideo/mp4Video for Adobe Flash Player
.f4pvideo/mp4Protected Video for Adobe Flash Player
.f4aaudio/mp4Audio for Adobe Flash Player
.f4baudio/mp4Audio Book for Adobe Flash Player

Matroska (“.mkv”) is free container format. Recently adopted by Google and re-branded as WebM, to be used together with VP8, proposed to be a standard video file format for html5.

Ogg is another free multimedia container format. Its tech quality is often in dispute. It is used by Wikipedia.


QuickTime (QT) is Apple's multimedia framework. It supports audio and video, as well as interactive panoramic images, and including such things as midi. It supports many codecs for audio and video.

The file format of QT is “.mov”. Quote:

The QuickTime (.mov) file format functions as a multimedia container file that contains one or more tracks, each of which stores a particular type of data: audio, video, effects, or text (For example, for subtitles). Each track either contains a digitally-encoded media stream (using a specific codec) or a data reference to the media stream located in another file. Tracks are maintained in a hierarchical data structure consisting of objects called atoms. An atom can be a parent to other atoms or it can contain media or edit data, but it cannot do both.[11]

QT 7.x is around from 2005 (OS X 10.4) to version 7.6 in 2009 (OS X 10.6). After that, the next version is QT X (10), which is supposedly completely rewritten for 64-bit computing and somewhat incompatable with past QT versions. Though, QT X relies on QT 7 for dealing with older codecs and other files such as MIDI.

Some more Wikipedia quotes:

QuickTime X is a combination of two technologies: QuickTime Kit Framework (QTKit) and QuickTime X Player.

… many Apple products (such as iTunes and Apple TV) still use the older QuickTime 7 engine.

QT Streaming

QuickTime Streaming Server (QTSS) is a server or service daemon built into Apple's Mac OS X Server that delivers video and audio on request to users over a computer network, including the Internet. Its primary GUI configuration tool is QTSS Publisher and its web-based administration port is 1220.

QuickTime Broadcaster is an audio and video RTP/RTSP server by Apple Computer for Mac OS X. It is separate from Apple's QuickTime Streaming Server, as it is not a service daemon but a desktop application.


FFmpeg and VLC

FFmpeg is a open source project on video and audio tech. Three notable component from FFmpeg are:

One interesting thing about the project is that it has a command line tool “ffmpeg” that lets you convert one video format to another.

VLC is a movie player. Originally designed as a server/client for streaming multimedia, but now is just a single application the VLC. Was at one point used by Google at Google Video until they switched to Flash. VLC can also be used on the command line.

Streaming Technologies

The following are the most commonly used protocols for Streaming media. Each with a particular purpose:

The above combo are usually referred to as “RTSP/RTP”.

Adobe Flash uses its own Real Time Messaging Protocol (RTMP).

Microsoft was using Microsoft Media Server (MMS), but is preprecated in 2003. Now Microsoft uses RTSP.

HTTP Live Streaming is Apple's tech, new in 2009 with QuickTime X. It is different than others because it is HTTP based. Proposed as a internet standard.

Some detail:

Here is a list of video hosting services: Comparison of video services. Contains some detail of what protocol they use.

See also: Comparison of streaming media systems .

Some References

Besides Wikipedia, here's some other articles i used for this article.

Comparison of codecs:

Comparison of container formats:

Audio codecs comparison: