Developing multimedia    applications with the
JavaTM Media Framework
( includes JMF 2.1.1 and MMAPI )
T.G.Venkatesh

Organization of the talk
Internet, Java and multimedia applications
Overview of JMF architecture
Media playback
Media capture
Media processing
Media storage
Simple media transmission and reception
JMF applet
JMF RTP API
JMF control
Extending JMF
JMStudio, PlayerBean and JMFCustomizer
MMAPI
Case study on multimedia project development.

 Internet and Java
Exponential growth of Internet.
Internet-based distributed applications : robust, secure, platform-independent.
Java for Internet Computing - object-oriented, extensive set of APIs, multi-threaded, exception handling.
Java compilers create Java byte codes.
Byte codes are interpreted by Java Virtual Machine.
Just in time compilers.
Java Applets - make web pages dynamic.

 Multimedia applications
Increase in speed of PCs and  bandwidth of Internet - exciting multimedia applications.
Example multimedia applications
video telephony,
video conferencing,
 media-on-demand,
voice and video mailing system,
Remote lecturing,
live broadcast over internet,
media rich e-commerce applications.

  Java Media APIs
JMF is a part of Java Media APIs that include,
JavaTM 2D API
JavaTM 3D API
JavaTM Advanced Imaging API
JavaTM Image I/O
JavaTM Shared Data Toolkit (JSDT)
JavaTM Sound API
JavaTM Speech API

What is JMF ?
JMF is a framework for handling streaming media in Java programs
Java programs can be Java applications or applets or JavaBeans.
JMF enables Java programs to
Present ( playback)  multimedia contents,
capture audio through microphone and video through Camera,
do real-time streaming of media over the Internet,
process media ( such as changing media format, adding  special effects etc. ),
store media into a file.

   Java Platforms & JMF

  Features of JMF
JMF supports popular media formats ( JPEG, MPEG-1, MPEG-2,  QuickTime,  AVI,  WAV,  MP3,  GSM,  G723, H263, and  MIDI. )
JMF supports popular media access protocols ( file, HTTP,  HTTPS,  FTP,  RTP, and  RTSP ).
JMF uses a plug-in architecture that allows JMF to be customized and extended.
JMF uses a robust event reporting mechanism that follows the “Observer” design pattern.
 JMF uses the “Factory” design pattern that simplifies  creation of JMF objects.
JMF supports the reception and transmission of media streams using the Real-time Transport Protocol ( RTP ).


 Criticisms on JMF
Multimedia processing and presentation is compute-intensive.
Existing desktop media players implement  media encoding, decoding, and rendering using native codes.
But native implementations are not portable across platforms - hence unsuitable for web-deployment.
Java offers platform neutrality
However pure Java  based implementation lack performance as compared to native codes.
We need portability as well as performance.

Why JMF ?
JMF API provides a layer of abstraction that makes multimedia applications platform neutral.
However a particular JMF implementation can delegate compute-intensive tasks to native implementations to improve  performance.
Windows performance pack, Solaris performance pack and cross platform  implementation.
Internet-based multimedia application designer has to simply adhere to JMF API.
If client machine has JMF performance pack application will run faster.
Otherwise … applications can still run on cross platform JMF.

   JMF architecture

    Generic Media
     Processing scenario

The JMF model

    JMF building blocks

  The JMF Time Model
Media is any form of data that changes meaningfully with respect to time.
Time class- represents a time instant
Duration interface is implemented by classes that can compute and advertise the duration of the media they handle.
For example Players and Processors extend Duration interface.
Interface TimeBase - uncontrolled source of clock ticks
Interface Clock - controlled source of clock ticks
Clock can be stopped, restarted, rate can be adjusted.
Analogy: crystal oscillator Vs digital clock.

 AudioFormat
The class Format stores some of the most common attributes that define a particular format .
Fields of the class AudioFormat :
SampleRate           -    sampling rate (  8 Khz, 44100 Hz etc)
SampleSizeInBits  -  bits per sample ( 8 / 16 bits per sample  )
channels,           -   Number of channels ; 1 ( mono) , 2 (stereo)
endian,                  -    Little endian or Big endian.
signed                 -    signed /  unsigned samples
FrameSizeInBits   -    size of the frame in bits,
linear, ULAW, DVI, G723, GSM, IMA4, and MPEG are defined as fields of AudioFormat.
AudioFormat   af  =  new AudioFormat( AudioFormat.LINEAR, 44100, 16, 1, AudioFormat.LITTLE_ENDIAN,  AudioFormat.SIGNED, 16, Format.NOT_SPECIFIED, Format.byteArray );

VideoFormat
VideoFormat class defines video format attributes such as encoding scheme, dimension, frame rate, etc.
include fields defining RGB, YUV, JPEG, MJPG, MPEG, H263, CINEPAK, and INDEO50.
Specialized classes : H261Format, H263Format, JPEGFormat, RGBFormat, and YUVFormat.
H263Format can have its attributes (i) arithmetic coding, (ii) advanced Prediction, (iii) Error Compensation etc enabled or disabled.

JMF Managers
Manager class : Factory for creating
DataSource,
Players,
Processors, and
DataSink.
PlugInManager maintains all the available PlugIns
CaptureDeviceManager registers all media capture devices
PackageManager : registry for package prefix list for packages that contain custom Players, Processors, DataSources, and DataSinks.

  Sources of Media

  MediaLocator and URL
URL - Uniform Resource Locator
MediaLocator  locates a media source
MediaLocator and URL can be constructed using LocatorString.
Examples for Locator String
file://ring.wav
javasound://44100, vfw://0
rtp://144.16.251.120:9000/audio/1
http:// www.mywebsite.com/clip.mov
rtsp:// host:port/mediafile
Unlike an URL or a MediaLocator  a DataSource  provides control over media access.

    MediaLocator and URL

DataSource
Media source are varied in nature. So is the  media access protocol ( file, HTTP, HTTPS, FTP, RTP, and RTSP ).
 We need an abstraction to hide the complexity of media access protocol.
DataSource : encapsulates media location as well as media access protocol
DataSource offers a simple connect protocol.
Connect, start, stop, and disconnect methods.
DataSource can be constructed from a MediaLocator or URL.

DataSource
Pull DataSources : client initiates and controls data transfer ( HTTP, file protocol)
Push DataSource : Server initiates and controls data transfer ( RTP protocol, capture devices)
Push and pull data stream
DataSource may implement following interfaces:
Positionable
RateConfigurable
SourceCloneable
CaptureDevice

Constructing a
DataSource

    Media Presentation

Player methods
Start()
setStopTime( )
setRate( )
setMediaTime( )
get and set TimeBase,
public  GainControl  getGainControl( )
public java.awt.Component  getVisualComponent( ),
public  java.awt.Component getControlPanelComponent( )
public  Time  getStartLatency( ).

Player Life cycle
UnRealized state : Player has no knowledge of the media it is supposed to handle.
Realizing state : Player  identifies and acquires  the resources it needs to playback the media.
Prefetching state : Player prefetches the media data

  Constructing a
 Realized Player

    Presenting the visual
    and control component

Starting and stopping
the Player

Media Presentation

Media capture

Registering
capture devices

Classes related to
media capture
CaptureDeviceInfo class : stores name, MediaLocator, supported output formats
CaptureDeviceManager
captureDeviceInfo getDevice(java.lang.String deviceName)
getDeviceList( Format format)
addDevice, removeDevice

Searching for a
capture device

    Media Processing

   Stages of processing
Demultiplexing : demultiplexes  the input media stream into various tracks.
Transcoding. convert the media from one format to another.
Encoding: uncompressed or raw media stream to compressed ( for storage or transmission.
Decoding: for presentation
Adding special effects (video contrast enhancement, echo effect or graphic equalizer effect on audio )
Multiplexing : multiplex various tracks processed by the Processor into a single track.
Renderering : performs presentation of the media.

PlugIns
PlugIn abstracts a media processing stage.
Interface PlugIn extends interface Control ( has getControl, getControls method ).
PlugIn has open, close, and reset methods.
Types of PlugIns : DEMULTIPLEXER, CODEC, EFFECT, RENDERER, MULTIPLEXER
PlugInManager
public static java.util.Vector getPlugInList (Format input, Format output, int type).
public static Format[] getSupportedInputFormats (java.lang.String className,  int type).
public static boolean addPlugIn( java.lang.String classname, Format [] in, Format [] out, int type)

JMFRegistry editor
showing PlugIns.

PlugIn viewer
Fig : (i)  capturing audio & video,
(ii) playing a quickTime file,
(iii) RTP transmission of audio

Processor
Reads from a DataSource
Processes the media
Renders the media on output devices OR
The output of a Processor is available as a DataSource  that can feed data to another Player / Processor.

JMF Processor model

Processor Model
abstracts essential information needed by Manager factory class to construct a Processor.
has following three Constructors
takes, input DataSource, array of Format objects one per track, output Content Descriptor.
same as above, but takes MediaLocator instead of DataSouce.
same as above with no input DataSource ( uses suitable capture device as DataSource )
To create Processor invoke createRealizedProcessor ( ProcessorModel pm)
Disadvantages of ProcessorModel.
It does not allow you to add Effect PlugIns.
You have no control over which PlugIn will  be used.
createRealizedProcessor () is a blocking call.

Processor state diagram

Creating a Processor

Configuring a Processor
( setting trackFormat)

Setting Codecs of a Processor

Setting Renderer of Processor

Displaying visual and control
panel of Processor

The DataSink
Destination of the processed media can be rendering devices or storage file or network stream.
The DataSink is an interface abstracting classes that can read media and render it to some destination ( file or network stream).
Created using the Manager’s method :
public  static DataSink  createDataSink (   DataSource datasource,  MediaLocator destLocator )    throws NoDataSinkException.
 DataSink can also control the sinking process ( open, start, stop, close methods )
DataSinkEvents & DataSinkListener

Steps involved in media storage
Following are the steps for storing the media
Construct MediaLocator / URL / DataSource
Construct an Unrealized Processor.
Configure the Processor
The output ContentDescriptor of the processor is then set to FileTypeDescriptor. ( For example FileTypeDescriptor .WAVE / FileTypeDescriptor.MSVIDEO
A given file format may support different encoding schemes for each of the media it can store.
Set the format of different tracks compatible with the file format.
For example AudioFormat.GSM_MS for a wav file.

Steps involved in media storage
Realize the processor
Get the output DataSource of the Processor.
 Construct  MediaLocator for the media destination.
construct a DataSink using the Manager
Open and start the DataSink,
Connect and start to the output- DataSource of the Processor
Start the Processor.
When you want to stop the file storage
Stop and close the Processor
Disconnect and stop the DataSink.
You may contain the size of the strorage file using the StreamWriterControl

Media transmission & Reception
Steps involved in Media transmission are as follows.
During configuration set the audio and / or video tracks of the Processor to be RTP formatted tracks.
Example :  VideoFormat    format   =      new           VideoFormat ( VideoFormat.H263_RTP );
 Realize and prefetch the Processor.
Get the output DataSource of the Processor.
Construct a MediaLocator representing the media stream that you want to generate.
Example: String  locatorString =  "rtp://192.168.0.12:9000/video/1" ;
MediaLocator m = new MediaLocator( locatorString );

Media transmission & Reception
Create a DataSink using the method createDataSink of the Manager.
Pass the DataSource and the MediaLocator created as the parameters to the createDataSink method.
Open and start the DataSink.
Start the Processor.
Media reception is similar to playback
construct an URL or MediaLocator  representing the stream.
String url = "rtp://192.168.0.12:9000/video/1" ;
construct a player for this URL and playback the media.

JMF Applet

JMF applet running in a browser

JMF Control
The JMF API has defined a number of interfaces meant for controlling  attributes of various JMF objects.
Example ...
GainControl : enables adjustment of audio signal gain.
TrackControl : This interface is provided by the Processor to query, and manipulate the data of individual media tracks.
BitRateControl  : used for specifying parameters to control media bit rate.
BufferControl : used to control the size and threshold of buffers maintained by certain JMF objects.
CachingControl : an interface supported by Players that are capable of reporting download progress.
FormatControl : This interfaces is implemented by objects which supports  setting of media format.

JMF Control
FrameGrabbingControl : This  control is used to grab a still video frame from the video stream.
FrameRateControl : You can specify the frame rate using this control.
MonitorControl : Some capture devices or encoders may allow you to monitor ( view/ listen ) to the capture or encoding progress through this control.
QualityControl : Some JMF objects such as encoders allow you to control the quality of the media they deliver using this control.
SilenceSuppressionControl : is a Control for specifying the parameters for silence suppression.
StreamWriterControl : This interface is implemented by a Multiplexer or DataSink to enable controlling the number of bytes generated as output.
Method of using the control :
QualityControl    qc =  (QualityControl)  processor.getControl         ( "javax.media.control.QualityControl” );
float quality = qc.setQuality( 0.3f ) ;

 Real-time Transport protocol
What is RTP  ?
Defined by Audio Video Transport (AVT) working group of the Internet Engineering Task Force (IETF) .
It is a transport protocol for streaming media.  Such a protocol requires 
low latency ( less overhead TCP unsuitable)
Sequencing ( UDP insufficient)
Timing information.
Field to identify payload type ( format of the media transported)
RTP packets are typically carried by UDP datagrams.
RTCP ( RTP Control Protocol )
a companion protocol to monitor the quality of data delivery offered by the RTP.

JMF RTP API
RTP session: A collection of participants exchanging media using the RTP constitutes a RTP session.
Participant : abstracts an application that sends and / or receives RTP encoded stream in a RTP session.
local and remote participants.
Identified by a CNAME( cononical name)
RTPManager :Each participant use a RTPManager object to  co-ordinates the RTPsession on participant’s behalf..
RTPStream  : defines a media stream exchanged in a RTP session. ReceiveStream and SendStream.
SessionAddress : a transport level end-point in JMF RTP API.
Has the InetAddress and port number to be used by the RTP for the data transfer
the InetAddress and port number to be used by the RTCP for the purpose of control.

JMF RTP API
RTPEvents : events posted by the RTPManager related
         to the RTP session.
Types of RTPEvents
ReceiveStreamEvents
SendStreamEvents
SesssionEvents
RemoteEvents
To handle ReceiveStreamEvents :
Define a class implementing the interface ReceiveStreamListener.
Add this class as a ReceiveStreamListener to the RTPManager
JMF  RTP API  related packages,
javax.media.rtp
 javax.media.rtp.event
javax.media.rtp.rtcp

Media Reception using RTPManager
The steps involved in receiving media streams using a RTPManager are as follows.
Construct an instance of a RTPManager,
RTPManager.newInstance();
Initialize RTPManager with a local transport end point,
rtpManager.initialize( localaddr);
Add a ReceiveStreamListener  to the RTPManager that can receive and handle the media stream(s).
rtpManager.addReceiveStreamListener( receiver );
Specify the remote transport end point(s).
rtpManager.addTarget( remoteAddr );
Handle    NewReceiveStreamEvent
Get DataSource from media stream - stream.getDataSource();
Construct Player / Processor from DataSource

Media Transmission using RTPManager
Prepare  RTPManager as follows
Instantiate RTPManager - RTPManager.newInstance();
 Intialize RTPManager to add local session address
rtpManager.initialize( localaddr);
Add Listeners ( Receive, Session, Send)  - Optional.
rtpManager.addSendStreamListener( );
Specify remote session addresses using addTarget() method. - rtpManager.addTarget( remoteAddr );
Prepare a DataSource to stream the data for transmission
source = Manager.createDataSource( aLocator );
instantiate and realize the Processor from DataSource.

Media Transmission using RTPManager
Configure the Processor as follows.
Set ContentDesriptor
ContentDescriptor cd = new ContentDescriptor(ContentDescriptor.RAW);
processor.setContentDescriptor(cd);
set track format as a RTP format
opformat = new VideoFormat(VideoFormat.H263_RTP);
(track[i] ).setFormat( opFormat );
Get processor output DataSource.
output  = processor.getDataOutput();

Media Transmission using RTPManager
Transmit
Create a send stream using this DataSource and RTPManager.
Stream = rtpManager.createSendStream(output,0);
stream.start();
Stopping transmission
stream.close();
processor.close();
rtpManager.removeTarget( remoteAddr ,"Stop requested");
rtpManager.dispose();

Extending JMF
JMF supports a set of open standards for audio and video compression formats, media access protocols, file storage format, etc.
You may want to use an existing open standard or an emerging standard or a proprietary standard which is not supported by a JMF reference implementation.
You can extend JMF and do the following.
Define new AudioFormat and VideoFormat classes.
Implement Codecs for these newly defined formats and integrate with JMF.
New media access protocols can be supported by defining, implementing and integrating DataSource classes.

Extending JMF
New protocol for sinking the data can be supported by defining, implementing and integrating DataSink class.
Special effects can be applied on the media by implementing Effect PlugIns.
New media transmission format can be supported by implementing Packetizer and Depacketizer PlugIns codecs and integrating with JMF.
Change the default transport protocol used by the RTP and RTCP using RTPConnector.
High performance Codec or MediaHandler (using faster algorithm or a hardware accelerator ) can be implemented and used in place of default implementation..

VolumeBarEffect PlugIn

JMStudio
 A stand alone general purpose JMF application.
Using JMStudio you can
play local /remote file,
capture audio and video,
transcode media from one format to another,
store media in files,
receive and transmit media streams across the network.
Extremely useful for developing and debugging JMF applications.

JMStudio main menus.

The Multi media Player Bean
Component Based Software Engineering (CBSE)
A bean is a reusable software component based on Sun’s JavaBeans specification that can be manipulated visually in a builder tool.
MediaPlayer is a JMF player encapsulated in a JavaBean.
Manipulate a number of properties through  get and set methods.
setCachingControlVisible(true);
 setPlaybackLoop(true);
setMediaLocationVisible( true);
setControlPanelVisible( true);
setFixedAspectRatio ( true);
MediaPlayer can integrated with Beans Development Kit (BDK).

JMF Customizer

Mobile Media API 1.0(JSR-135)
There are many portable / mobile devices that have limited by memory, power, and network bandwidth.
Example devices :smart cards, cellular phones, pagers PDAs, palmtops, screen phones, set-top boxes, game machines, VCRs and CD players.
JavaTM 2 Platform, Micro Edition ( J2ME ) is a framework for deploying Java applications over such devices.
Mobile Media API 1.0 ( MMAPI) is an API providing  multimedia support for the J2METM platform.
MMAPI is targeted at Connected Device Limited Configuration (CLDC ) and Mobile Information Device Profile (MIDP ) based device.

The MMAPI architecture

MMAPI Features
MMAPI does not mandate that certain media type and access protocols should be supported by an implementation.
The media type and access protocol support depends upon
whether the native code underneath supports them,
whether the MMAPI implementers exposes those native services through the API.
Sun’s Reference Implementation supports monotone, tone sequence, Wav, MIDI, MPEG playback and audio capture.
Wireless ToolKit (WTK) with MMEmulator can be used to develop multimedia MIDlets.

WTK with MMEmulator
MPEG player
Plays a file in the network using HTTP protocol
Volume control
Rate control
Frame positioning
Pause and restart
Rewind or forward.

Smart phones supporting MMAPI
Nokia 3650  and Sony Erricson  T610

Multimedia projects -Cosmophone
Written using Java and JMF.
Video telephony over Internet
Answering machine
GSM or G723 Audio format
H.263 or JPEG video format
Video Preview
Text ChattingVoice and Video mail over Internet
Video Conferencing in an Intranet.
Auto detection of capture devices

Multimedia projects -Cosmophone

Thank you… for more information http://business.vsnl.com/cosmos_software