Notes
Slide Show
Outline
1
Developing multimedia    applications with the
JavaTM Media Framework
( includes JMF 2.1.1 and MMAPI )

  • T.G.Venkatesh
2
Organization of the talk

  • Internet, Java and multimedia applications
  • Overview of JMF architecture
  • Media playback
  • Media capture
  • Media processing
  • Media storage
  • Simple media transmission and reception
  • JMF applet
  • JMF RTP API
  • JMF control
  • Extending JMF
  • JMStudio, PlayerBean and JMFCustomizer
  • MMAPI
  • Case study on multimedia project development.
3
 Internet and Java
  • Exponential growth of Internet.
  • Internet-based distributed applications : robust, secure, platform-independent.
  • Java for Internet Computing - object-oriented, extensive set of APIs, multi-threaded, exception handling.
  • Java compilers create Java byte codes.
  • Byte codes are interpreted by Java Virtual Machine.
  • Just in time compilers.
  • Java Applets - make web pages dynamic.
4
 Multimedia applications
  • Increase in speed of PCs and  bandwidth of Internet - exciting multimedia applications.
  • Example multimedia applications
      • video telephony,
      • video conferencing,
      •  media-on-demand,
      • voice and video mailing system,
      • Remote lecturing,
      • live broadcast over internet,
      • media rich e-commerce applications.

5
  Java Media APIs
  • JMF is a part of Java Media APIs that include,
  • JavaTM 2D API
  • JavaTM 3D API
  • JavaTM Advanced Imaging API
  • JavaTM Image I/O
  • JavaTM Shared Data Toolkit (JSDT)
  • JavaTM Sound API
  • JavaTM Speech API


6
What is JMF ?
  • JMF is a framework for handling streaming media in Java programs
  • Java programs can be Java applications or applets or JavaBeans.
  • JMF enables Java programs to
    • Present ( playback)  multimedia contents,
    • capture audio through microphone and video through Camera,
    • do real-time streaming of media over the Internet,
    • process media ( such as changing media format, adding  special effects etc. ),
    • store media into a file.
7
   Java Platforms & JMF
8
  Features of JMF
  • JMF supports popular media formats ( JPEG, MPEG-1, MPEG-2,  QuickTime,  AVI,  WAV,  MP3,  GSM,  G723, H263, and  MIDI. )
  • JMF supports popular media access protocols ( file, HTTP,  HTTPS,  FTP,  RTP, and  RTSP ).
  • JMF uses a plug-in architecture that allows JMF to be customized and extended.
  • JMF uses a robust event reporting mechanism that follows the “Observer” design pattern.
  •  JMF uses the “Factory” design pattern that simplifies  creation of JMF objects.
  • JMF supports the reception and transmission of media streams using the Real-time Transport Protocol ( RTP ).
9

 Criticisms on JMF
    • Multimedia processing and presentation is compute-intensive.
    • Existing desktop media players implement  media encoding, decoding, and rendering using native codes.
    • But native implementations are not portable across platforms - hence unsuitable for web-deployment.
    • Java offers platform neutrality
    • However pure Java  based implementation lack performance as compared to native codes.


    • We need portability as well as performance.
10
Why JMF ?

  • JMF API provides a layer of abstraction that makes multimedia applications platform neutral.
  • However a particular JMF implementation can delegate compute-intensive tasks to native implementations to improve  performance.
  • Windows performance pack, Solaris performance pack and cross platform  implementation.
  • Internet-based multimedia application designer has to simply adhere to JMF API.
  • If client machine has JMF performance pack application will run faster.
  • Otherwise … applications can still run on cross platform JMF.
11
   JMF architecture
12
    Generic Media
     Processing scenario
13
The JMF model
14
    JMF building blocks
15
  The JMF Time Model
  • Media is any form of data that changes meaningfully with respect to time.
  • Time class- represents a time instant
  • Duration interface is implemented by classes that can compute and advertise the duration of the media they handle.
  • For example Players and Processors extend Duration interface.
  • Interface TimeBase - uncontrolled source of clock ticks
  • Interface Clock - controlled source of clock ticks
  • Clock can be stopped, restarted, rate can be adjusted.
  • Analogy: crystal oscillator Vs digital clock.
16
 AudioFormat
  • The class Format stores some of the most common attributes that define a particular format .
  • Fields of the class AudioFormat :
    • SampleRate           -    sampling rate (  8 Khz, 44100 Hz etc)
    • SampleSizeInBits  -  bits per sample ( 8 / 16 bits per sample  )
    • channels,           -   Number of channels ; 1 ( mono) , 2 (stereo)
    • endian,                  -    Little endian or Big endian.
    • signed                 -    signed /  unsigned samples
    • FrameSizeInBits   -    size of the frame in bits,
  • linear, ULAW, DVI, G723, GSM, IMA4, and MPEG are defined as fields of AudioFormat.
  • AudioFormat   af  =  new AudioFormat( AudioFormat.LINEAR, 44100, 16, 1, AudioFormat.LITTLE_ENDIAN,  AudioFormat.SIGNED, 16, Format.NOT_SPECIFIED, Format.byteArray );
17
VideoFormat
  • VideoFormat class defines video format attributes such as encoding scheme, dimension, frame rate, etc.
  • include fields defining RGB, YUV, JPEG, MJPG, MPEG, H263, CINEPAK, and INDEO50.
  • Specialized classes : H261Format, H263Format, JPEGFormat, RGBFormat, and YUVFormat.
  • H263Format can have its attributes (i) arithmetic coding, (ii) advanced Prediction, (iii) Error Compensation etc enabled or disabled.


18
JMF Managers
  • Manager class : Factory for creating
    • DataSource,
    • Players,
    • Processors, and
    • DataSink.
  • PlugInManager maintains all the available PlugIns
  • CaptureDeviceManager registers all media capture devices
  • PackageManager : registry for package prefix list for packages that contain custom Players, Processors, DataSources, and DataSinks.



19
  Sources of Media
20
  MediaLocator and URL
  • URL - Uniform Resource Locator
  • MediaLocator  locates a media source
  • MediaLocator and URL can be constructed using LocatorString.
  • Examples for Locator String
    • file://ring.wav
    • javasound://44100, vfw://0
    • rtp://144.16.251.120:9000/audio/1
    • http:// www.mywebsite.com/clip.mov
    • rtsp:// host:port/mediafile
  • Unlike an URL or a MediaLocator  a DataSource  provides control over media access.
21
    MediaLocator and URL
22
DataSource
  • Media source are varied in nature. So is the  media access protocol ( file, HTTP, HTTPS, FTP, RTP, and RTSP ).
  •  We need an abstraction to hide the complexity of media access protocol.
  • DataSource : encapsulates media location as well as media access protocol
  • DataSource offers a simple connect protocol.
  • Connect, start, stop, and disconnect methods.
  • DataSource can be constructed from a MediaLocator or URL.
23
DataSource
  • Pull DataSources : client initiates and controls data transfer ( HTTP, file protocol)
  • Push DataSource : Server initiates and controls data transfer ( RTP protocol, capture devices)
  • Push and pull data stream
  • DataSource may implement following interfaces:
    • Positionable
    • RateConfigurable
    • SourceCloneable
    • CaptureDevice

24
Constructing a
DataSource
25
    Media Presentation
26
Player methods

  • Start()
  • setStopTime( )
  • setRate( )
  • setMediaTime( )
  • get and set TimeBase,
  • public  GainControl  getGainControl( )
  • public java.awt.Component  getVisualComponent( ),
  • public  java.awt.Component getControlPanelComponent( )
  • public  Time  getStartLatency( ).
27
Player Life cycle

  • UnRealized state : Player has no knowledge of the media it is supposed to handle.


  • Realizing state : Player  identifies and acquires  the resources it needs to playback the media.


  • Prefetching state : Player prefetches the media data
28
  Constructing a
 Realized Player
29
    Presenting the visual
    and control component
30
Starting and stopping
the Player
31
Media Presentation
32
Media capture
33
Registering
capture devices
34
Classes related to
media capture

  • CaptureDeviceInfo class : stores name, MediaLocator, supported output formats
  • CaptureDeviceManager
    • captureDeviceInfo getDevice(java.lang.String deviceName)
    • getDeviceList( Format format)
    • addDevice, removeDevice
35
Searching for a
capture device
36
    Media Processing
37
   Stages of processing
  • Demultiplexing : demultiplexes  the input media stream into various tracks.
  • Transcoding. convert the media from one format to another.
    • Encoding: uncompressed or raw media stream to compressed ( for storage or transmission.
    • Decoding: for presentation
  • Adding special effects (video contrast enhancement, echo effect or graphic equalizer effect on audio )
  • Multiplexing : multiplex various tracks processed by the Processor into a single track.
  • Renderering : performs presentation of the media.




38
PlugIns
  • PlugIn abstracts a media processing stage.
  • Interface PlugIn extends interface Control ( has getControl, getControls method ).
  • PlugIn has open, close, and reset methods.
  • Types of PlugIns : DEMULTIPLEXER, CODEC, EFFECT, RENDERER, MULTIPLEXER
  • PlugInManager
    • public static java.util.Vector getPlugInList (Format input, Format output, int type).
    • public static Format[] getSupportedInputFormats (java.lang.String className,  int type).
    • public static boolean addPlugIn( java.lang.String classname, Format [] in, Format [] out, int type)
39
JMFRegistry editor
showing PlugIns.
40
PlugIn viewer
  • Fig : (i)  capturing audio & video,
    (ii) playing a quickTime file,
    (iii) RTP transmission of audio
41
Processor
  • Reads from a DataSource
  • Processes the media
  • Renders the media on output devices OR
  • The output of a Processor is available as a DataSource  that can feed data to another Player / Processor.
42
JMF Processor model
43
Processor Model
  • abstracts essential information needed by Manager factory class to construct a Processor.
  • has following three Constructors
    • takes, input DataSource, array of Format objects one per track, output Content Descriptor.
    • same as above, but takes MediaLocator instead of DataSouce.
    • same as above with no input DataSource ( uses suitable capture device as DataSource )
  • To create Processor invoke createRealizedProcessor ( ProcessorModel pm)
  • Disadvantages of ProcessorModel.
    • It does not allow you to add Effect PlugIns.
    • You have no control over which PlugIn will  be used.
    • createRealizedProcessor () is a blocking call.


44
Processor state diagram
45
Creating a Processor
46
Configuring a Processor
( setting trackFormat)
47
Setting Codecs of a Processor
48
Setting Renderer of Processor
49
Displaying visual and control
panel of Processor
50
The DataSink
  • Destination of the processed media can be rendering devices or storage file or network stream.
  • The DataSink is an interface abstracting classes that can read media and render it to some destination ( file or network stream).
  • Created using the Manager’s method :
    • public  static DataSink  createDataSink (   DataSource datasource,  MediaLocator destLocator )    throws NoDataSinkException.
  •  DataSink can also control the sinking process ( open, start, stop, close methods )
  • DataSinkEvents & DataSinkListener



51
Steps involved in media storage

  • Following are the steps for storing the media
  • Construct MediaLocator / URL / DataSource
  • Construct an Unrealized Processor.
  • Configure the Processor
  • The output ContentDescriptor of the processor is then set to FileTypeDescriptor. ( For example FileTypeDescriptor .WAVE / FileTypeDescriptor.MSVIDEO
  • A given file format may support different encoding schemes for each of the media it can store.
  • Set the format of different tracks compatible with the file format.
  • For example AudioFormat.GSM_MS for a wav file.
52
Steps involved in media storage
  • Realize the processor
  • Get the output DataSource of the Processor.
  •  Construct  MediaLocator for the media destination.
  • construct a DataSink using the Manager
  • Open and start the DataSink,
  • Connect and start to the output- DataSource of the Processor
  • Start the Processor.
  • When you want to stop the file storage
    • Stop and close the Processor
    • Disconnect and stop the DataSink.
  • You may contain the size of the strorage file using the StreamWriterControl
53
Media transmission & Reception

  • Steps involved in Media transmission are as follows.
  • During configuration set the audio and / or video tracks of the Processor to be RTP formatted tracks.
  • Example :  VideoFormat    format   =      new           VideoFormat ( VideoFormat.H263_RTP );
  •  Realize and prefetch the Processor.
  • Get the output DataSource of the Processor.
  • Construct a MediaLocator representing the media stream that you want to generate.
  • Example: String  locatorString =  "rtp://192.168.0.12:9000/video/1" ;
  • MediaLocator m = new MediaLocator( locatorString );
54
Media transmission & Reception
  • Create a DataSink using the method createDataSink of the Manager.
  • Pass the DataSource and the MediaLocator created as the parameters to the createDataSink method.
  • Open and start the DataSink.
  • Start the Processor.
  • Media reception is similar to playback
  • construct an URL or MediaLocator  representing the stream.
      • String url = "rtp://192.168.0.12:9000/video/1" ;
  • construct a player for this URL and playback the media.
55
JMF Applet
56
JMF applet running in a browser
57
JMF Control
  • The JMF API has defined a number of interfaces meant for controlling  attributes of various JMF objects.
  • Example ...
    • GainControl : enables adjustment of audio signal gain.
    • TrackControl : This interface is provided by the Processor to query, and manipulate the data of individual media tracks.
    • BitRateControl  : used for specifying parameters to control media bit rate.
    • BufferControl : used to control the size and threshold of buffers maintained by certain JMF objects.
    • CachingControl : an interface supported by Players that are capable of reporting download progress.
    • FormatControl : This interfaces is implemented by objects which supports  setting of media format.
58
JMF Control
    • FrameGrabbingControl : This  control is used to grab a still video frame from the video stream.
    • FrameRateControl : You can specify the frame rate using this control.
    • MonitorControl : Some capture devices or encoders may allow you to monitor ( view/ listen ) to the capture or encoding progress through this control.
    • QualityControl : Some JMF objects such as encoders allow you to control the quality of the media they deliver using this control.
    • SilenceSuppressionControl : is a Control for specifying the parameters for silence suppression.
    • StreamWriterControl : This interface is implemented by a Multiplexer or DataSink to enable controlling the number of bytes generated as output.
  • Method of using the control :
  • QualityControl    qc =  (QualityControl)  processor.getControl         ( "javax.media.control.QualityControl” );
  • float quality = qc.setQuality( 0.3f ) ;


59
 Real-time Transport protocol
  • What is RTP  ?
  • Defined by Audio Video Transport (AVT) working group of the Internet Engineering Task Force (IETF) .
  • It is a transport protocol for streaming media.  Such a protocol requires  …
    • low latency ( less overhead TCP unsuitable)
    • Sequencing ( UDP insufficient)
    • Timing information.
    • Field to identify payload type ( format of the media transported)
  • RTP packets are typically carried by UDP datagrams.
  • RTCP ( RTP Control Protocol )
    • a companion protocol to monitor the quality of data delivery offered by the RTP.
60
JMF RTP API
  • RTP session: A collection of participants exchanging media using the RTP constitutes a RTP session.
  • Participant : abstracts an application that sends and / or receives RTP encoded stream in a RTP session.
    • local and remote participants.
    • Identified by a CNAME( cononical name)
  • RTPManager :Each participant use a RTPManager object to  co-ordinates the RTPsession on participant’s behalf..
  • RTPStream  : defines a media stream exchanged in a RTP session. ReceiveStream and SendStream.
  • SessionAddress : a transport level end-point in JMF RTP API.
    • Has the InetAddress and port number to be used by the RTP for the data transfer
    • the InetAddress and port number to be used by the RTCP for the purpose of control.
61
JMF RTP API
  • RTPEvents : events posted by the RTPManager related
  •          to the RTP session.
  • Types of RTPEvents
    • ReceiveStreamEvents
    • SendStreamEvents
    • SesssionEvents
    • RemoteEvents
  • To handle ReceiveStreamEvents :
    • Define a class implementing the interface ReceiveStreamListener.
    • Add this class as a ReceiveStreamListener to the RTPManager
  • JMF  RTP API  related packages,
    • javax.media.rtp
    •  javax.media.rtp.event
    • javax.media.rtp.rtcp
62
Media Reception using RTPManager
  • The steps involved in receiving media streams using a RTPManager are as follows.
  • Construct an instance of a RTPManager,
    • RTPManager.newInstance();
  • Initialize RTPManager with a local transport end point,
    • rtpManager.initialize( localaddr);
  • Add a ReceiveStreamListener  to the RTPManager that can receive and handle the media stream(s).
    • rtpManager.addReceiveStreamListener( receiver );
  • Specify the remote transport end point(s).
    • rtpManager.addTarget( remoteAddr );
  • Handle    NewReceiveStreamEvent
  • Get DataSource from media stream - stream.getDataSource();
  • Construct Player / Processor from DataSource
63
Media Transmission using RTPManager
  • Prepare  RTPManager as follows
  • Instantiate RTPManager - RTPManager.newInstance();
  •  Intialize RTPManager to add local session address
    • rtpManager.initialize( localaddr);
  • Add Listeners ( Receive, Session, Send)  - Optional.
    • rtpManager.addSendStreamListener( );
  • Specify remote session addresses using addTarget() method. - rtpManager.addTarget( remoteAddr );
  • Prepare a DataSource to stream the data for transmission
  • source = Manager.createDataSource( aLocator );
  • instantiate and realize the Processor from DataSource.
64
Media Transmission using RTPManager
  • Configure the Processor as follows.
  • Set ContentDesriptor
    • ContentDescriptor cd = new ContentDescriptor(ContentDescriptor.RAW);
    • processor.setContentDescriptor(cd);
  • set track format as a RTP format
    • opformat = new VideoFormat(VideoFormat.H263_RTP);
    • (track[i] ).setFormat( opFormat );
  • Get processor output DataSource.
    • output  = processor.getDataOutput();

65
Media Transmission using RTPManager
  • Transmit
  • Create a send stream using this DataSource and RTPManager.
    • Stream = rtpManager.createSendStream(output,0);
    • stream.start();
  • Stopping transmission
    • stream.close();
    • processor.close();
    • rtpManager.removeTarget( remoteAddr ,"Stop requested");
    • rtpManager.dispose();
66
Extending JMF
  • JMF supports a set of open standards for audio and video compression formats, media access protocols, file storage format, etc.
  • You may want to use an existing open standard or an emerging standard or a proprietary standard which is not supported by a JMF reference implementation.
  • You can extend JMF and do the following.
  • Define new AudioFormat and VideoFormat classes.
  • Implement Codecs for these newly defined formats and integrate with JMF.
  • New media access protocols can be supported by defining, implementing and integrating DataSource classes.


67
Extending JMF

  • New protocol for sinking the data can be supported by defining, implementing and integrating DataSink class.
  • Special effects can be applied on the media by implementing Effect PlugIns.
  • New media transmission format can be supported by implementing Packetizer and Depacketizer PlugIns codecs and integrating with JMF.
  • Change the default transport protocol used by the RTP and RTCP using RTPConnector.
  • High performance Codec or MediaHandler (using faster algorithm or a hardware accelerator ) can be implemented and used in place of default implementation..
68
VolumeBarEffect PlugIn
69
JMStudio
  •  A stand alone general purpose JMF application.
  • Using JMStudio you can
    • play local /remote file,
    • capture audio and video,
    • transcode media from one format to another,
    • store media in files,
    • receive and transmit media streams across the network.
  • Extremely useful for developing and debugging JMF applications.


70
JMStudio main menus.
71
The Multi media Player Bean
  • Component Based Software Engineering (CBSE)
  • A bean is a reusable software component based on Sun’s JavaBeans specification that can be manipulated visually in a builder tool.
  • MediaPlayer is a JMF player encapsulated in a JavaBean.
  • Manipulate a number of properties through  get and set methods.
    • setCachingControlVisible(true);
    •  setPlaybackLoop(true);
    • setMediaLocationVisible( true);
    • setControlPanelVisible( true);
    • setFixedAspectRatio ( true);
  • MediaPlayer can integrated with Beans Development Kit (BDK).
72
JMF Customizer
73
Mobile Media API 1.0(JSR-135)
  • There are many portable / mobile devices that have limited by memory, power, and network bandwidth.
  • Example devices :smart cards, cellular phones, pagers PDAs, palmtops, screen phones, set-top boxes, game machines, VCRs and CD players.
  • JavaTM 2 Platform, Micro Edition ( J2ME ) is a framework for deploying Java applications over such devices.
  • Mobile Media API 1.0 ( MMAPI) is an API providing  multimedia support for the J2METM platform.
  • MMAPI is targeted at Connected Device Limited Configuration (CLDC ) and Mobile Information Device Profile (MIDP ) based device.
74
The MMAPI architecture
75
MMAPI Features
  • MMAPI does not mandate that certain media type and access protocols should be supported by an implementation.
  • The media type and access protocol support depends upon
    • whether the native code underneath supports them,
    • whether the MMAPI implementers exposes those native services through the API.
  • Sun’s Reference Implementation supports monotone, tone sequence, Wav, MIDI, MPEG playback and audio capture.
  • Wireless ToolKit (WTK) with MMEmulator can be used to develop multimedia MIDlets.
76
WTK with MMEmulator
  • MPEG player
  • Plays a file in the network using HTTP protocol
  • Volume control
  • Rate control
  • Frame positioning
  • Pause and restart
  • Rewind or forward.
77
Smart phones supporting MMAPI
Nokia 3650  and Sony Erricson  T610
78
Multimedia projects -Cosmophone

  • Written using Java and JMF.
  • Video telephony over Internet
  • Answering machine
  • GSM or G723 Audio format
  • H.263 or JPEG video format
  • Video Preview
  • Text ChattingVoice and Video mail over Internet
  • Video Conferencing in an Intranet.
  • Auto detection of capture devices


79
Multimedia projects -Cosmophone
80
Thank you… for more information http://business.vsnl.com/cosmos_software