Developing
multimedia applications with the
JavaTM Media Framework
( includes JMF 2.1.1 and MMAPI )
Organization of the talk
|
|
|
|
|
Internet, Java and multimedia
applications |
|
Overview of JMF architecture |
|
Media playback |
|
Media capture |
|
Media processing |
|
Media storage |
|
Simple media transmission and reception |
|
JMF applet |
|
JMF RTP API |
|
JMF control |
|
Extending JMF |
|
JMStudio, PlayerBean and JMFCustomizer |
|
MMAPI |
|
Case study on multimedia project
development. |
Internet and Java
|
|
|
Exponential growth of Internet. |
|
Internet-based distributed applications
: robust, secure, platform-independent. |
|
Java for Internet Computing -
object-oriented, extensive set of APIs, multi-threaded, exception handling. |
|
Java compilers create Java byte codes. |
|
Byte codes are interpreted by Java
Virtual Machine. |
|
Just in time compilers. |
|
Java Applets - make web pages dynamic. |
Multimedia applications
|
|
|
|
|
Increase in speed of PCs and bandwidth of Internet - exciting multimedia
applications. |
|
Example multimedia applications |
|
video telephony, |
|
video conferencing, |
|
media-on-demand, |
|
voice and video mailing system, |
|
Remote lecturing, |
|
live broadcast over internet, |
|
media rich e-commerce applications. |
|
|
Java Media APIs
|
|
|
JMF is a part of Java Media APIs that
include, |
|
JavaTM 2D API |
|
JavaTM 3D API |
|
JavaTM Advanced Imaging API |
|
JavaTM Image I/O |
|
JavaTM Shared Data Toolkit
(JSDT) |
|
JavaTM Sound API |
|
JavaTM Speech API |
|
|
What is JMF ?
|
|
|
|
JMF is a framework for handling
streaming media in Java programs |
|
Java programs can be Java applications
or applets or JavaBeans. |
|
JMF enables Java programs to |
|
Present ( playback) multimedia contents, |
|
capture audio through microphone and
video through Camera, |
|
do real-time streaming of media over
the Internet, |
|
process media ( such as changing media
format, adding special effects etc. ), |
|
store media into a file. |
Java Platforms & JMF
Features of JMF
|
|
|
JMF supports popular media formats ( JPEG,
MPEG-1, MPEG-2, QuickTime, AVI,
WAV, MP3, GSM,
G723, H263, and MIDI. ) |
|
JMF supports popular media access
protocols ( file, HTTP, HTTPS, FTP,
RTP, and RTSP ). |
|
JMF uses a plug-in architecture that
allows JMF to be customized and extended. |
|
JMF uses a robust event reporting
mechanism that follows the “Observer” design pattern. |
|
JMF uses the “Factory” design pattern that
simplifies creation of JMF objects. |
|
JMF supports the reception and
transmission of media streams using the Real-time Transport Protocol ( RTP ). |
Criticisms on JMF
|
|
|
|
Multimedia processing and presentation
is compute-intensive. |
|
Existing desktop media players
implement media encoding, decoding,
and rendering using native codes. |
|
But native implementations are not
portable across platforms - hence unsuitable for web-deployment. |
|
Java offers platform neutrality |
|
However pure Java based implementation lack performance as
compared to native codes. |
|
|
|
We need portability as well as
performance. |
Why JMF ?
|
|
|
|
|
JMF API provides a layer of abstraction
that makes multimedia applications platform neutral. |
|
However a particular JMF implementation
can delegate compute-intensive tasks to native implementations to
improve performance. |
|
Windows performance pack, Solaris
performance pack and cross platform
implementation. |
|
Internet-based multimedia application
designer has to simply adhere to JMF API. |
|
If client machine has JMF performance
pack application will run faster. |
|
Otherwise … applications can still run
on cross platform JMF. |
JMF architecture
Generic Media
Processing scenario
The JMF model
JMF building blocks
The JMF Time Model
|
|
|
Media is any form of data that changes
meaningfully with respect to time. |
|
Time class- represents a time instant |
|
Duration interface is implemented by
classes that can compute and advertise the duration of the media they handle. |
|
For example Players and Processors
extend Duration interface. |
|
Interface TimeBase - uncontrolled
source of clock ticks |
|
Interface Clock - controlled source of
clock ticks |
|
Clock can be stopped, restarted, rate
can be adjusted. |
|
Analogy: crystal oscillator Vs digital
clock. |
AudioFormat
|
|
|
|
The class Format stores some of the
most common attributes that define a particular format . |
|
Fields of the class AudioFormat : |
|
SampleRate - sampling rate ( 8 Khz, 44100 Hz etc) |
|
SampleSizeInBits -
bits per sample ( 8 / 16 bits per sample ) |
|
channels, -
Number of channels ; 1 ( mono) , 2 (stereo) |
|
endian,
- Little endian or Big endian. |
|
signed - signed /
unsigned samples |
|
FrameSizeInBits -
size of the frame in bits, |
|
linear, ULAW, DVI, G723, GSM, IMA4, and
MPEG are defined as fields of AudioFormat. |
|
AudioFormat af
= new AudioFormat(
AudioFormat.LINEAR, 44100, 16, 1, AudioFormat.LITTLE_ENDIAN, AudioFormat.SIGNED, 16,
Format.NOT_SPECIFIED, Format.byteArray ); |
VideoFormat
|
|
|
VideoFormat class defines video format
attributes such as encoding scheme, dimension, frame rate, etc. |
|
include fields defining RGB, YUV, JPEG,
MJPG, MPEG, H263, CINEPAK, and INDEO50. |
|
Specialized classes : H261Format,
H263Format, JPEGFormat, RGBFormat, and YUVFormat. |
|
H263Format can have its attributes (i)
arithmetic coding, (ii) advanced Prediction, (iii) Error Compensation etc
enabled or disabled. |
|
|
JMF Managers
|
|
|
|
Manager class : Factory for creating |
|
DataSource, |
|
Players, |
|
Processors, and |
|
DataSink. |
|
PlugInManager maintains all the
available PlugIns |
|
CaptureDeviceManager registers all
media capture devices |
|
PackageManager : registry for package
prefix list for packages that contain custom Players, Processors,
DataSources, and DataSinks. |
|
|
|
|
Sources of Media
MediaLocator and URL
|
|
|
|
URL - Uniform Resource Locator |
|
MediaLocator locates a media source |
|
MediaLocator and URL can be constructed
using LocatorString. |
|
Examples for Locator String |
|
file://ring.wav |
|
javasound://44100, vfw://0 |
|
rtp://144.16.251.120:9000/audio/1 |
|
http:// www.mywebsite.com/clip.mov |
|
rtsp:// host:port/mediafile |
|
Unlike an URL or a MediaLocator a DataSource provides control over media access. |
MediaLocator and URL
DataSource
|
|
|
Media source are varied in nature. So
is the media access protocol ( file,
HTTP, HTTPS, FTP, RTP, and RTSP ). |
|
We need an abstraction to hide the
complexity of media access protocol. |
|
DataSource : encapsulates media
location as well as media access protocol |
|
DataSource offers a simple connect
protocol. |
|
Connect, start, stop, and disconnect
methods. |
|
DataSource can be constructed from a
MediaLocator or URL. |
DataSource
|
|
|
|
Pull DataSources : client initiates and
controls data transfer ( HTTP, file protocol) |
|
Push DataSource : Server initiates and
controls data transfer ( RTP protocol, capture devices) |
|
Push and pull data stream |
|
DataSource may implement following
interfaces: |
|
Positionable |
|
RateConfigurable |
|
SourceCloneable |
|
CaptureDevice |
|
|
Constructing a
DataSource
Media Presentation
Player methods
|
|
|
|
|
Start() |
|
setStopTime( ) |
|
setRate( ) |
|
setMediaTime( ) |
|
get and set TimeBase, |
|
public
GainControl getGainControl( ) |
|
public java.awt.Component getVisualComponent( ), |
|
public
java.awt.Component getControlPanelComponent( ) |
|
public
Time getStartLatency( ). |
Player Life cycle
|
|
|
|
|
UnRealized state : Player has no
knowledge of the media it is supposed to handle. |
|
|
|
Realizing state : Player identifies and acquires the resources it needs to playback the
media. |
|
|
|
Prefetching state : Player prefetches
the media data |
Constructing a
Realized Player
Presenting the visual
and control component
Starting and stopping
the Player
Media Presentation
Media capture
Registering
capture devices
Classes related to
media capture
|
|
|
|
|
|
CaptureDeviceInfo class : stores name,
MediaLocator, supported output formats |
|
CaptureDeviceManager |
|
captureDeviceInfo
getDevice(java.lang.String deviceName) |
|
getDeviceList( Format format) |
|
addDevice, removeDevice |
Searching for a
capture device
Media Processing
Stages of processing
|
|
|
|
Demultiplexing : demultiplexes the input media stream into various tracks. |
|
Transcoding. convert the media from one
format to another. |
|
Encoding: uncompressed or raw media
stream to compressed ( for storage or transmission. |
|
Decoding: for presentation |
|
Adding special effects (video contrast
enhancement, echo effect or graphic equalizer effect on audio ) |
|
Multiplexing : multiplex various tracks
processed by the Processor into a single track. |
|
Renderering : performs presentation of
the media. |
|
|
|
|
|
|
PlugIns
|
|
|
|
PlugIn abstracts a media processing
stage. |
|
Interface PlugIn extends interface
Control ( has getControl, getControls method ). |
|
PlugIn has open, close, and reset
methods. |
|
Types of PlugIns : DEMULTIPLEXER,
CODEC, EFFECT, RENDERER, MULTIPLEXER |
|
PlugInManager |
|
public static java.util.Vector
getPlugInList (Format input, Format output, int type). |
|
public static Format[]
getSupportedInputFormats (java.lang.String className, int type). |
|
public static boolean addPlugIn(
java.lang.String classname, Format [] in, Format [] out, int type) |
JMFRegistry editor
showing PlugIns.
PlugIn viewer
|
|
|
Fig : (i) capturing audio & video,
(ii) playing a quickTime file,
(iii) RTP transmission of audio |
Processor
|
|
|
Reads from a DataSource |
|
Processes the media |
|
Renders the media on output devices OR |
|
The output of a Processor is available
as a DataSource that can feed data to
another Player / Processor. |
JMF Processor model
Processor Model
|
|
|
|
abstracts essential information needed
by Manager factory class to construct a Processor. |
|
has following three Constructors |
|
takes, input DataSource, array of
Format objects one per track, output Content Descriptor. |
|
same as above, but takes MediaLocator
instead of DataSouce. |
|
same as above with no input DataSource
( uses suitable capture device as DataSource ) |
|
To create Processor invoke
createRealizedProcessor ( ProcessorModel pm) |
|
Disadvantages of ProcessorModel. |
|
It does not allow you to add Effect
PlugIns. |
|
You have no control over which PlugIn
will be used. |
|
createRealizedProcessor () is a
blocking call. |
|
|
|
|
Processor state diagram
Creating a Processor
Configuring a Processor
( setting trackFormat)
Setting Codecs of a
Processor
Setting Renderer of
Processor
Displaying visual and
control
panel of Processor
The DataSink
|
|
|
|
Destination of the processed media can
be rendering devices or storage file or network stream. |
|
The DataSink is an interface
abstracting classes that can read media and render it to some destination (
file or network stream). |
|
Created using the Manager’s method : |
|
public
static DataSink createDataSink
( DataSource datasource, MediaLocator destLocator ) throws NoDataSinkException. |
|
DataSink
can also control the sinking process ( open, start, stop, close methods ) |
|
DataSinkEvents & DataSinkListener |
|
|
|
|
Steps involved in media
storage
|
|
|
|
|
Following are the steps for storing the
media |
|
Construct MediaLocator / URL /
DataSource |
|
Construct an Unrealized Processor. |
|
Configure the Processor |
|
The output ContentDescriptor of the
processor is then set to FileTypeDescriptor. ( For example FileTypeDescriptor
.WAVE / FileTypeDescriptor.MSVIDEO |
|
A given file format may support
different encoding schemes for each of the media it can store. |
|
Set the format of different tracks
compatible with the file format. |
|
For example AudioFormat.GSM_MS for a
wav file. |
Steps involved in media
storage
|
|
|
|
Realize the processor |
|
Get the output DataSource of the
Processor. |
|
Construct
MediaLocator for the media destination. |
|
construct a DataSink using the Manager |
|
Open and start the DataSink, |
|
Connect and start to the output-
DataSource of the Processor |
|
Start the Processor. |
|
When you want to stop the file storage |
|
Stop and close the Processor |
|
Disconnect and stop the DataSink. |
|
You may contain the size of the
strorage file using the StreamWriterControl |
Media transmission &
Reception
|
|
|
|
|
Steps involved in Media transmission
are as follows. |
|
During configuration set the audio and
/ or video tracks of the Processor to be RTP formatted tracks. |
|
Example : VideoFormat format
= new VideoFormat ( VideoFormat.H263_RTP
); |
|
Realize and prefetch the Processor. |
|
Get the output DataSource of the
Processor. |
|
Construct a MediaLocator representing
the media stream that you want to generate. |
|
Example: String locatorString = "rtp://192.168.0.12:9000/video/1"
; |
|
MediaLocator m = new MediaLocator(
locatorString ); |
Media transmission &
Reception
|
|
|
|
|
Create a DataSink using the method
createDataSink of the Manager. |
|
Pass the DataSource and the
MediaLocator created as the parameters to the createDataSink method. |
|
Open and start the DataSink. |
|
Start the Processor. |
|
Media reception is similar to playback |
|
construct an URL or MediaLocator representing the stream. |
|
String url =
"rtp://192.168.0.12:9000/video/1" ; |
|
construct a player for this URL and
playback the media. |
JMF Applet
JMF applet running in a
browser
JMF Control
|
|
|
|
The JMF API has defined a number of
interfaces meant for controlling
attributes of various JMF objects. |
|
Example ... |
|
GainControl : enables adjustment of
audio signal gain. |
|
TrackControl : This interface is
provided by the Processor to query, and manipulate the data of individual
media tracks. |
|
BitRateControl : used for specifying parameters to control
media bit rate. |
|
BufferControl : used to control the
size and threshold of buffers maintained by certain JMF objects. |
|
CachingControl : an interface supported
by Players that are capable of reporting download progress. |
|
FormatControl : This interfaces is
implemented by objects which supports
setting of media format. |
JMF Control
|
|
|
|
FrameGrabbingControl : This control is used to grab a still video frame
from the video stream. |
|
FrameRateControl : You can specify the
frame rate using this control. |
|
MonitorControl : Some capture devices
or encoders may allow you to monitor ( view/ listen ) to the capture or
encoding progress through this control. |
|
QualityControl : Some JMF objects such
as encoders allow you to control the quality of the media they deliver using
this control. |
|
SilenceSuppressionControl : is a
Control for specifying the parameters for silence suppression. |
|
StreamWriterControl : This interface is
implemented by a Multiplexer or DataSink to enable controlling the number of
bytes generated as output. |
|
Method of using the control : |
|
QualityControl qc =
(QualityControl)
processor.getControl (
"javax.media.control.QualityControl” ); |
|
float quality = qc.setQuality( 0.3f ) ; |
|
|
Real-time Transport protocol
|
|
|
|
What is RTP ? |
|
Defined by Audio Video Transport (AVT)
working group of the Internet Engineering Task Force (IETF) . |
|
It is a transport protocol for
streaming media. Such a protocol
requires … |
|
low latency ( less overhead TCP
unsuitable) |
|
Sequencing ( UDP insufficient) |
|
Timing information. |
|
Field to identify payload type ( format
of the media transported) |
|
RTP packets are typically carried by
UDP datagrams. |
|
RTCP ( RTP Control Protocol ) |
|
a companion protocol to monitor the
quality of data delivery offered by the RTP. |
JMF RTP API
|
|
|
|
RTP session: A collection of
participants exchanging media using the RTP constitutes a RTP session. |
|
Participant : abstracts an application
that sends and / or receives RTP encoded stream in a RTP session. |
|
local and remote participants. |
|
Identified by a CNAME( cononical name) |
|
RTPManager :Each participant use a
RTPManager object to co-ordinates the
RTPsession on participant’s behalf.. |
|
RTPStream : defines a media stream exchanged in a RTP
session. ReceiveStream and SendStream. |
|
SessionAddress : a transport level
end-point in JMF RTP API. |
|
Has the InetAddress and port number to
be used by the RTP for the data transfer |
|
the InetAddress and port number to be
used by the RTCP for the purpose of control. |
JMF RTP API
|
|
|
|
RTPEvents : events posted by the
RTPManager related |
|
to the RTP session. |
|
Types of RTPEvents |
|
ReceiveStreamEvents |
|
SendStreamEvents |
|
SesssionEvents |
|
RemoteEvents |
|
To handle ReceiveStreamEvents : |
|
Define a class implementing the
interface ReceiveStreamListener. |
|
Add this class as a
ReceiveStreamListener to the RTPManager |
|
JMF
RTP API related packages, |
|
javax.media.rtp |
|
javax.media.rtp.event |
|
javax.media.rtp.rtcp |
Media Reception using
RTPManager
|
|
|
|
The steps involved in receiving media
streams using a RTPManager are as follows. |
|
Construct an instance of a RTPManager, |
|
RTPManager.newInstance(); |
|
Initialize RTPManager with a local
transport end point, |
|
rtpManager.initialize( localaddr); |
|
Add a ReceiveStreamListener to the RTPManager that can receive and
handle the media stream(s). |
|
rtpManager.addReceiveStreamListener(
receiver ); |
|
Specify the remote transport end
point(s). |
|
rtpManager.addTarget( remoteAddr ); |
|
Handle NewReceiveStreamEvent |
|
Get DataSource from media stream - stream.getDataSource(); |
|
Construct Player / Processor from
DataSource |
Media Transmission using
RTPManager
|
|
|
|
Prepare
RTPManager as follows |
|
Instantiate RTPManager - RTPManager.newInstance(); |
|
Intialize RTPManager to add local session
address |
|
rtpManager.initialize( localaddr); |
|
Add Listeners ( Receive, Session,
Send) - Optional. |
|
rtpManager.addSendStreamListener( ); |
|
Specify remote session addresses using
addTarget() method. - rtpManager.addTarget( remoteAddr ); |
|
Prepare a DataSource to stream the data
for transmission |
|
source = Manager.createDataSource(
aLocator ); |
|
instantiate and realize the Processor
from DataSource. |
Media Transmission using
RTPManager
|
|
|
|
Configure the Processor as follows. |
|
Set ContentDesriptor |
|
ContentDescriptor cd = new
ContentDescriptor(ContentDescriptor.RAW); |
|
processor.setContentDescriptor(cd); |
|
set track format as a RTP format |
|
opformat = new
VideoFormat(VideoFormat.H263_RTP); |
|
(track[i] ).setFormat( opFormat ); |
|
Get processor output DataSource. |
|
output
= processor.getDataOutput(); |
|
|
Media Transmission using
RTPManager
|
|
|
|
Transmit |
|
Create a send stream using this
DataSource and RTPManager. |
|
Stream =
rtpManager.createSendStream(output,0); |
|
stream.start(); |
|
Stopping transmission |
|
stream.close(); |
|
processor.close(); |
|
rtpManager.removeTarget( remoteAddr
,"Stop requested"); |
|
rtpManager.dispose(); |
Extending JMF
|
|
|
JMF supports a set of open standards
for audio and video compression formats, media access protocols, file storage
format, etc. |
|
You may want to use an existing open
standard or an emerging standard or a proprietary standard which is not
supported by a JMF reference implementation. |
|
You can extend JMF and do the
following. |
|
Define new AudioFormat and VideoFormat
classes. |
|
Implement Codecs for these newly
defined formats and integrate with JMF. |
|
New media access protocols can be
supported by defining, implementing and integrating DataSource classes. |
|
|
Extending JMF
|
|
|
|
|
New protocol for sinking the data can
be supported by defining, implementing and integrating DataSink class. |
|
Special effects can be applied on the
media by implementing Effect PlugIns. |
|
New media transmission format can be
supported by implementing Packetizer and Depacketizer PlugIns codecs and
integrating with JMF. |
|
Change the default transport protocol
used by the RTP and RTCP using RTPConnector. |
|
High performance Codec or MediaHandler
(using faster algorithm or a hardware accelerator ) can be implemented and
used in place of default implementation.. |
VolumeBarEffect PlugIn
JMStudio
|
|
|
|
A
stand alone general purpose JMF application. |
|
Using JMStudio you can |
|
play local /remote file, |
|
capture audio and video, |
|
transcode media from one format to
another, |
|
store media in files, |
|
receive and transmit media streams
across the network. |
|
Extremely useful for developing and
debugging JMF applications. |
|
|
JMStudio main menus.
The Multi media Player
Bean
|
|
|
|
Component Based Software Engineering
(CBSE) |
|
A bean is a reusable software component
based on Sun’s JavaBeans specification that can be manipulated visually in a
builder tool. |
|
MediaPlayer is a JMF player
encapsulated in a JavaBean. |
|
Manipulate a number of properties
through get and set methods. |
|
setCachingControlVisible(true); |
|
setPlaybackLoop(true); |
|
setMediaLocationVisible( true); |
|
setControlPanelVisible( true); |
|
setFixedAspectRatio ( true); |
|
MediaPlayer can integrated with Beans
Development Kit (BDK). |
JMF Customizer
Mobile Media API 1.0(JSR-135)
|
|
|
There are many portable / mobile
devices that have limited by memory, power, and network bandwidth. |
|
Example devices :smart cards, cellular
phones, pagers PDAs, palmtops, screen phones, set-top boxes, game machines,
VCRs and CD players. |
|
JavaTM 2 Platform, Micro
Edition ( J2ME ) is a framework for deploying Java applications over such
devices. |
|
Mobile Media API 1.0 ( MMAPI) is an API
providing multimedia support for the
J2METM platform. |
|
MMAPI is targeted at Connected Device
Limited Configuration (CLDC ) and Mobile Information Device Profile (MIDP )
based device. |
The MMAPI architecture
MMAPI Features
|
|
|
|
MMAPI does not mandate that certain
media type and access protocols should be supported by an implementation. |
|
The media type and access protocol
support depends upon |
|
whether the native code underneath
supports them, |
|
whether the MMAPI implementers exposes
those native services through the API. |
|
Sun’s Reference Implementation supports
monotone, tone sequence, Wav, MIDI, MPEG playback and audio capture. |
|
Wireless ToolKit (WTK) with MMEmulator
can be used to develop multimedia MIDlets. |
WTK with MMEmulator
|
|
|
MPEG player |
|
Plays a file in the network using HTTP
protocol |
|
Volume control |
|
Rate control |
|
Frame positioning |
|
Pause and restart |
|
Rewind or forward. |
Smart phones supporting
MMAPI
Nokia 3650 and Sony Erricson T610
Multimedia projects
-Cosmophone
|
|
|
|
|
Written using Java and JMF. |
|
Video telephony over Internet |
|
Answering machine |
|
GSM or G723 Audio format |
|
H.263 or JPEG video format |
|
Video Preview |
|
Text ChattingVoice and Video mail over
Internet |
|
Video Conferencing in an Intranet. |
|
Auto detection of capture devices |
|
|
Multimedia projects
-Cosmophone
Thank you… for more
information http://business.vsnl.com/cosmos_software