International Defense Security & Technology Your trusted Source for News, Research and Analysis
Multimedia includes media text, audio, still image, video, and metadata. These files need to be acquired for example through the camera, then stored on the hard drive, along with compression, searched by description, played, and edited for example video editing.
A multimedia framework is a software framework that handles media on a computer and through a network. Computers contain a variety of media hardware including different sound cards, different sound hardware, and therefore different hardware drivers which are needed. Providing an interface to handle the complexity of the hardware is not an easy task. Most devices have multiple ICs with different communication protocols… so the device drivers tends to be very complex as well.
To avoid the problem of implementing each driver for each piece of hardware into every program we need to abstract the hardware drivers into one kind of “standard driver” which can be accessed by all other programs. This abstraction layer can then implement all other hardware drivers and can talk to the hardware. This abstracted hardware can accept all audio streams and commands which would go directly to the hardware in other cases. This is called a multimedia framework that can understand all the different media formats like MP3, OGG, WAV, ACC, WMA, RMA, etc.
Dealing with multimedia is not an easy task. Therefore, selecting the proper framework to process audio and video streams becomes a key point to ensuring a successful project. Developers facing this challenge should consider the following factors before making their choice for the right multimedia framework: coding complexity (architecture), cross-platform support, multimedia technologies coverage (codecs, filters, etc.), documentation and support available.
A good multimedia framework offers an intuitive API and a modular architecture to easily add support for new audio, video and container formats and transmission protocols. It is meant to be used by applications such as media players and audio or video editors, but can also be used to build videoconferencing applications, media converters and other multimedia tools. Data is processed among modules automatically, it is unnecessary for app to pass buffers between connected modules one by one.
In contrast to function libraries, a multimedia framework provides a run time environment for the media processing. Ideally such an environment provides execution contexts for the media processing blocks separated from the application using the framework. The separation supports the independent processing of multimedia data in a timely manner. These separate contexts can be implemented as threads.
Common multimedia frameworks are Quicktime, GStreamer, NMM, Xine, Helix and also the old aRts from KDE 3.
Quicktime
Quicktime was one of the first multimedia frameworks to come out, way back in 1991. Despite being so old, it was quite powerful and well designed right from the beginning. Apple has continued to improve the framework since it’s initial release, so it’s still a very good framework even today. For a long time, Macs were considered to be one of the best choices for a computer if you needed to edit video. This was largely based on the power and flexibility that the Quicktime multimedia framework offered.
GStreamer
GStreamer is an open source multimedia framework mainly used to create media applications (streaming, media playback, non-linear editing, etc.). The GStreamer framework is designed to make it easy to write applications that handles audio or video or both. It uses plugins that will provide the various codec and other functionalities.
It works on all major operating systems (ie. Linux, Android, Windows, Max OS X, iOS) and runs on all major hardware architectures including x86, ARM, MIPS, SPARC. It counts with an extensive list of multimedia plugins (encoders, decoders, content filters…) plus it allows an easy integration of 3rd party ones. Lastly, GStreamer counts with a wide and well-structured documentation available for developers and a big community whose contributions ensure a continuous improvement of this framework.
Intel Open Source Media Framework
Increase the speed and performance of video decoding, encoding, processing, and transcoding on compressed digital video and imaging. Support workload offloading from the CPU to the GPU. The latest features from Intel mean decreased power use, increased mobile battery life, and reduced power requirements for other types of devices.
Benefits
- GPU and CPU hardware acceleration in widely adopted open source Linux* video frameworks (FFmpeg* and GStreamer).
- Open source video processing algorithms in libXCam and Linux video frameworks (FFmpeg and GStreamer).
- Supports major media codecs such as H.264/AVC, H.265/HEVC, VP8, VP9, JPEG, MJPEG, MPEG-2, and VC-1
- Written primarily in C and C++ programming languages, and distributed under an open source license
- Used by Google* for Chrome OS* and Android*
- Incorporated into SteamOS* (from Valve*) and customized Linux distributions
The leading and most widely adopted open source multimedia framework is able to decode, encode, transcode, mux, demux, stream, filter, and play almost anything that humans and machines have created. FFmpeg supports the most obscure, old formats up to the most advanced stage. It’s also highly portable under a wide variety of build environments, machine architectures, and configurations.
Advanced Linux Sound Architecture (ALSA)
The Advanced Linux Sound Architecture (ALSA) provides kernel driven sound card drivers. It replaces the original Open Sound System (OSS).
Besides the sound device drivers, ALSA also bundles a user space driven library for application developers. They can then use those ALSA drivers for high level API development. This enables direct (kernel) interaction with sound devices through ALSA libraries.
