1
|
|
2
|
- Internet, Java and multimedia applications
- Overview of JMF architecture
- Media playback
- Media capture
- Media processing
- Media storage
- Simple media transmission and reception
- JMF applet
- JMF RTP API
- JMF control
- Extending JMF
- JMStudio, PlayerBean and JMFCustomizer
- MMAPI
- Case study on multimedia project development.
|
3
|
- Exponential growth of Internet.
- Internet-based distributed applications : robust, secure,
platform-independent.
- Java for Internet Computing - object-oriented, extensive set of APIs,
multi-threaded, exception handling.
- Java compilers create Java byte codes.
- Byte codes are interpreted by Java Virtual Machine.
- Just in time compilers.
- Java Applets - make web pages dynamic.
|
4
|
- Increase in speed of PCs and
bandwidth of Internet - exciting multimedia applications.
- Example multimedia applications
- video telephony,
- video conferencing,
- media-on-demand,
- voice and video mailing system,
- Remote lecturing,
- live broadcast over internet,
- media rich e-commerce applications.
|
5
|
- JMF is a part of Java Media APIs that include,
- JavaTM 2D API
- JavaTM 3D API
- JavaTM Advanced Imaging API
- JavaTM Image I/O
- JavaTM Shared Data Toolkit (JSDT)
- JavaTM Sound API
- JavaTM Speech API
|
6
|
- JMF is a framework for handling streaming media in Java programs
- Java programs can be Java applications or applets or JavaBeans.
- JMF enables Java programs to
- Present ( playback) multimedia
contents,
- capture audio through microphone and video through Camera,
- do real-time streaming of media over the Internet,
- process media ( such as changing media format, adding special effects etc. ),
- store media into a file.
|
7
|
|
8
|
- JMF supports popular media formats ( JPEG, MPEG-1, MPEG-2, QuickTime, AVI,
WAV, MP3, GSM,
G723, H263, and MIDI. )
- JMF supports popular media access protocols ( file, HTTP, HTTPS,
FTP, RTP, and RTSP ).
- JMF uses a plug-in architecture that allows JMF to be customized and
extended.
- JMF uses a robust event reporting mechanism that follows the “Observer”
design pattern.
- JMF uses the “Factory” design
pattern that simplifies creation
of JMF objects.
- JMF supports the reception and transmission of media streams using the Real-time
Transport Protocol ( RTP ).
|
9
|
- Multimedia processing and presentation is compute-intensive.
- Existing desktop media players implement media encoding, decoding, and
rendering using native codes.
- But native implementations are not portable across platforms - hence
unsuitable for web-deployment.
- Java offers platform neutrality
- However pure Java based
implementation lack performance as compared to native codes.
- We need portability as well as performance.
|
10
|
- JMF API provides a layer of abstraction that makes multimedia
applications platform neutral.
- However a particular JMF implementation can delegate compute-intensive
tasks to native implementations to improve performance.
- Windows performance pack, Solaris performance pack and cross
platform implementation.
- Internet-based multimedia application designer has to simply adhere to
JMF API.
- If client machine has JMF performance pack application will run faster.
- Otherwise … applications can still run on cross platform JMF.
|
11
|
|
12
|
|
13
|
|
14
|
|
15
|
- Media is any form of data that changes meaningfully with respect to
time.
- Time class- represents a time instant
- Duration interface is implemented by classes that can compute and
advertise the duration of the media they handle.
- For example Players and Processors extend Duration interface.
- Interface TimeBase - uncontrolled source of clock ticks
- Interface Clock - controlled source of clock ticks
- Clock can be stopped, restarted, rate can be adjusted.
- Analogy: crystal oscillator Vs digital clock.
|
16
|
- The class Format stores some of the most common attributes that define a
particular format .
- Fields of the class AudioFormat :
- SampleRate - sampling rate ( 8 Khz, 44100 Hz etc)
- SampleSizeInBits - bits per sample ( 8 / 16 bits per
sample )
- channels, - Number of channels ; 1 ( mono) , 2
(stereo)
- endian, - Little endian or Big endian.
- signed - signed / unsigned samples
- FrameSizeInBits - size of the frame in bits,
- linear, ULAW, DVI, G723, GSM, IMA4, and MPEG are defined as fields of
AudioFormat.
- AudioFormat af =
new AudioFormat( AudioFormat.LINEAR, 44100, 16, 1,
AudioFormat.LITTLE_ENDIAN, AudioFormat.SIGNED,
16, Format.NOT_SPECIFIED, Format.byteArray );
|
17
|
- VideoFormat class defines video format attributes such as encoding
scheme, dimension, frame rate, etc.
- include fields defining RGB, YUV, JPEG, MJPG, MPEG, H263, CINEPAK, and
INDEO50.
- Specialized classes : H261Format, H263Format, JPEGFormat, RGBFormat, and
YUVFormat.
- H263Format can have its attributes (i) arithmetic coding, (ii) advanced
Prediction, (iii) Error Compensation etc enabled or disabled.
|
18
|
- Manager class : Factory for creating
- DataSource,
- Players,
- Processors, and
- DataSink.
- PlugInManager maintains all the available PlugIns
- CaptureDeviceManager registers all media capture devices
- PackageManager : registry for package prefix list for packages that
contain custom Players, Processors, DataSources, and DataSinks.
|
19
|
|
20
|
- URL - Uniform Resource Locator
- MediaLocator locates a media
source
- MediaLocator and URL can be constructed using LocatorString.
- Examples for Locator String
- file://ring.wav
- javasound://44100, vfw://0
- rtp://144.16.251.120:9000/audio/1
- http:// www.mywebsite.com/clip.mov
- rtsp:// host:port/mediafile
- Unlike an URL or a MediaLocator a
DataSource provides control over
media access.
|
21
|
|
22
|
- Media source are varied in nature. So is the media access protocol ( file, HTTP,
HTTPS, FTP, RTP, and RTSP ).
- We need an abstraction to hide
the complexity of media access protocol.
- DataSource : encapsulates media location as well as media access
protocol
- DataSource offers a simple connect protocol.
- Connect, start, stop, and disconnect methods.
- DataSource can be constructed from a MediaLocator or URL.
|
23
|
- Pull DataSources : client initiates and controls data transfer ( HTTP,
file protocol)
- Push DataSource : Server initiates and controls data transfer ( RTP
protocol, capture devices)
- Push and pull data stream
- DataSource may implement following interfaces:
- Positionable
- RateConfigurable
- SourceCloneable
- CaptureDevice
|
24
|
|
25
|
|
26
|
- Start()
- setStopTime( )
- setRate( )
- setMediaTime( )
- get and set TimeBase,
- public GainControl getGainControl( )
- public java.awt.Component
getVisualComponent( ),
- public java.awt.Component
getControlPanelComponent( )
- public Time getStartLatency( ).
|
27
|
- UnRealized state : Player has no knowledge of the media it is supposed
to handle.
- Realizing state : Player
identifies and acquires
the resources it needs to playback the media.
- Prefetching state : Player prefetches the media data
|
28
|
|
29
|
|
30
|
|
31
|
|
32
|
|
33
|
|
34
|
- CaptureDeviceInfo class : stores name, MediaLocator, supported output
formats
- CaptureDeviceManager
- captureDeviceInfo getDevice(java.lang.String deviceName)
- getDeviceList( Format format)
- addDevice, removeDevice
|
35
|
|
36
|
|
37
|
- Demultiplexing : demultiplexes
the input media stream into various tracks.
- Transcoding. convert the media from one format to another.
- Encoding: uncompressed or raw media stream to compressed ( for storage
or transmission.
- Decoding: for presentation
- Adding special effects (video contrast enhancement, echo effect or
graphic equalizer effect on audio )
- Multiplexing : multiplex various tracks processed by the Processor into
a single track.
- Renderering : performs presentation of the media.
|
38
|
- PlugIn abstracts a media processing stage.
- Interface PlugIn extends interface Control ( has getControl, getControls
method ).
- PlugIn has open, close, and reset methods.
- Types of PlugIns : DEMULTIPLEXER, CODEC, EFFECT, RENDERER, MULTIPLEXER
- PlugInManager
- public static java.util.Vector getPlugInList (Format input, Format
output, int type).
- public static Format[] getSupportedInputFormats (java.lang.String
className, int type).
- public static boolean addPlugIn( java.lang.String classname, Format []
in, Format [] out, int type)
|
39
|
|
40
|
- Fig : (i) capturing audio &
video,
(ii) playing a quickTime file,
(iii) RTP transmission of audio
|
41
|
- Reads from a DataSource
- Processes the media
- Renders the media on output devices OR
- The output of a Processor is available as a DataSource that can feed data to another Player /
Processor.
|
42
|
|
43
|
- abstracts essential information needed by Manager factory class to
construct a Processor.
- has following three Constructors
- takes, input DataSource, array of Format objects one per track, output
Content Descriptor.
- same as above, but takes MediaLocator instead of DataSouce.
- same as above with no input DataSource ( uses suitable capture device
as DataSource )
- To create Processor invoke createRealizedProcessor ( ProcessorModel pm)
- Disadvantages of ProcessorModel.
- It does not allow you to add Effect PlugIns.
- You have no control over which PlugIn will be used.
- createRealizedProcessor () is a blocking call.
|
44
|
|
45
|
|
46
|
|
47
|
|
48
|
|
49
|
|
50
|
- Destination of the processed media can be rendering devices or storage
file or network stream.
- The DataSink is an interface abstracting classes that can read media and
render it to some destination ( file or network stream).
- Created using the Manager’s method :
- public static DataSink createDataSink ( DataSource datasource, MediaLocator destLocator ) throws NoDataSinkException.
- DataSink can also control the
sinking process ( open, start, stop, close methods )
- DataSinkEvents & DataSinkListener
|
51
|
- Following are the steps for storing the media
- Construct MediaLocator / URL / DataSource
- Construct an Unrealized Processor.
- Configure the Processor
- The output ContentDescriptor of the processor is then set to
FileTypeDescriptor. ( For example FileTypeDescriptor .WAVE /
FileTypeDescriptor.MSVIDEO
- A given file format may support different encoding schemes for each of
the media it can store.
- Set the format of different tracks compatible with the file format.
- For example AudioFormat.GSM_MS for a wav file.
|
52
|
- Realize the processor
- Get the output DataSource of the Processor.
- Construct MediaLocator for the media
destination.
- construct a DataSink using the Manager
- Open and start the DataSink,
- Connect and start to the output- DataSource of the Processor
- Start the Processor.
- When you want to stop the file storage
- Stop and close the Processor
- Disconnect and stop the DataSink.
- You may contain the size of the strorage file using the
StreamWriterControl
|
53
|
- Steps involved in Media transmission are as follows.
- During configuration set the audio and / or video tracks of the
Processor to be RTP formatted tracks.
- Example : VideoFormat format =
new VideoFormat (
VideoFormat.H263_RTP );
- Realize and prefetch the
Processor.
- Get the output DataSource of the Processor.
- Construct a MediaLocator representing the media stream that you want to
generate.
- Example: String locatorString
=
"rtp://192.168.0.12:9000/video/1" ;
- MediaLocator m = new MediaLocator( locatorString );
|
54
|
- Create a DataSink using the method createDataSink of the Manager.
- Pass the DataSource and the MediaLocator created as the parameters to
the createDataSink method.
- Open and start the DataSink.
- Start the Processor.
- Media reception is similar to playback
- construct an URL or MediaLocator
representing the stream.
- String url = "rtp://192.168.0.12:9000/video/1" ;
- construct a player for this URL and playback the media.
|
55
|
|
56
|
|
57
|
- The JMF API has defined a number of interfaces meant for
controlling attributes of various
JMF objects.
- Example ...
- GainControl : enables adjustment of audio signal gain.
- TrackControl : This interface is provided by the Processor to query,
and manipulate the data of individual media tracks.
- BitRateControl : used for
specifying parameters to control media bit rate.
- BufferControl : used to control the size and threshold of buffers
maintained by certain JMF objects.
- CachingControl : an interface supported by Players that are capable of
reporting download progress.
- FormatControl : This interfaces is implemented by objects which
supports setting of media
format.
|
58
|
- FrameGrabbingControl : This
control is used to grab a still video frame from the video
stream.
- FrameRateControl : You can specify the frame rate using this control.
- MonitorControl : Some capture devices or encoders may allow you to
monitor ( view/ listen ) to the capture or encoding progress through
this control.
- QualityControl : Some JMF objects such as encoders allow you to control
the quality of the media they deliver using this control.
- SilenceSuppressionControl : is a Control for specifying the parameters
for silence suppression.
- StreamWriterControl : This interface is implemented by a Multiplexer or
DataSink to enable controlling the number of bytes generated as output.
- Method of using the control :
- QualityControl qc = (QualityControl) processor.getControl ( "javax.media.control.QualityControl”
);
- float quality = qc.setQuality( 0.3f ) ;
|
59
|
- What is RTP ?
- Defined by Audio Video Transport (AVT) working group of the Internet
Engineering Task Force (IETF) .
- It is a transport protocol for streaming media. Such a protocol requires …
- low latency ( less overhead TCP unsuitable)
- Sequencing ( UDP insufficient)
- Timing information.
- Field to identify payload type ( format of the media transported)
- RTP packets are typically carried by UDP datagrams.
- RTCP ( RTP Control Protocol )
- a companion protocol to monitor the quality of data delivery offered by
the RTP.
|
60
|
- RTP session: A collection of participants exchanging media using the RTP
constitutes a RTP session.
- Participant : abstracts an application that sends and / or receives RTP
encoded stream in a RTP session.
- local and remote participants.
- Identified by a CNAME( cononical name)
- RTPManager :Each participant use a RTPManager object to co-ordinates the RTPsession on
participant’s behalf..
- RTPStream : defines a media
stream exchanged in a RTP session. ReceiveStream and SendStream.
- SessionAddress : a transport level end-point in JMF RTP API.
- Has the InetAddress and port number to be used by the RTP for the data
transfer
- the InetAddress and port number to be used by the RTCP for the purpose
of control.
|
61
|
- RTPEvents : events posted by the RTPManager related
- to the RTP session.
- Types of RTPEvents
- ReceiveStreamEvents
- SendStreamEvents
- SesssionEvents
- RemoteEvents
- To handle ReceiveStreamEvents :
- Define a class implementing the interface ReceiveStreamListener.
- Add this class as a ReceiveStreamListener to the RTPManager
- JMF RTP API related packages,
- javax.media.rtp
- javax.media.rtp.event
- javax.media.rtp.rtcp
|
62
|
- The steps involved in receiving media streams using a RTPManager are as
follows.
- Construct an instance of a RTPManager,
- RTPManager.newInstance();
- Initialize RTPManager with a local transport end point,
- rtpManager.initialize( localaddr);
- Add a ReceiveStreamListener to
the RTPManager that can receive and handle the media stream(s).
- rtpManager.addReceiveStreamListener( receiver );
- Specify the remote transport end point(s).
- rtpManager.addTarget( remoteAddr );
- Handle NewReceiveStreamEvent
- Get DataSource from media stream - stream.getDataSource();
- Construct Player / Processor from DataSource
|
63
|
- Prepare RTPManager as follows
- Instantiate RTPManager - RTPManager.newInstance();
- Intialize RTPManager to add local
session address
- rtpManager.initialize( localaddr);
- Add Listeners ( Receive, Session, Send)
- Optional.
- rtpManager.addSendStreamListener( );
- Specify remote session addresses using addTarget() method. - rtpManager.addTarget(
remoteAddr );
- Prepare a DataSource to stream the data for transmission
- source = Manager.createDataSource( aLocator );
- instantiate and realize the Processor from DataSource.
|
64
|
- Configure the Processor as follows.
- Set ContentDesriptor
- ContentDescriptor cd = new ContentDescriptor(ContentDescriptor.RAW);
- processor.setContentDescriptor(cd);
- set track format as a RTP format
- opformat = new VideoFormat(VideoFormat.H263_RTP);
- (track[i] ).setFormat( opFormat );
- Get processor output DataSource.
- output =
processor.getDataOutput();
|
65
|
- Transmit
- Create a send stream using this DataSource and RTPManager.
- Stream = rtpManager.createSendStream(output,0);
- stream.start();
- Stopping transmission
- stream.close();
- processor.close();
- rtpManager.removeTarget( remoteAddr ,"Stop requested");
- rtpManager.dispose();
|
66
|
- JMF supports a set of open standards for audio and video compression
formats, media access protocols, file storage format, etc.
- You may want to use an existing open standard or an emerging standard or
a proprietary standard which is not supported by a JMF reference
implementation.
- You can extend JMF and do the following.
- Define new AudioFormat and VideoFormat classes.
- Implement Codecs for these newly defined formats and integrate with JMF.
- New media access protocols can be supported by defining, implementing
and integrating DataSource classes.
|
67
|
- New protocol for sinking the data can be supported by defining,
implementing and integrating DataSink class.
- Special effects can be applied on the media by implementing Effect
PlugIns.
- New media transmission format can be supported by implementing
Packetizer and Depacketizer PlugIns codecs and integrating with JMF.
- Change the default transport protocol used by the RTP and RTCP using
RTPConnector.
- High performance Codec or MediaHandler (using faster algorithm or a
hardware accelerator ) can be implemented and used in place of default
implementation..
|
68
|
|
69
|
- A stand alone general purpose JMF
application.
- Using JMStudio you can
- play local /remote file,
- capture audio and video,
- transcode media from one format to another,
- store media in files,
- receive and transmit media streams across the network.
- Extremely useful for developing and debugging JMF applications.
|
70
|
|
71
|
- Component Based Software Engineering (CBSE)
- A bean is a reusable software component based on Sun’s JavaBeans
specification that can be manipulated visually in a builder tool.
- MediaPlayer is a JMF player encapsulated in a JavaBean.
- Manipulate a number of properties through get and set methods.
- setCachingControlVisible(true);
- setPlaybackLoop(true);
- setMediaLocationVisible( true);
- setControlPanelVisible( true);
- setFixedAspectRatio ( true);
- MediaPlayer can integrated with Beans Development Kit (BDK).
|
72
|
|
73
|
- There are many portable / mobile devices that have limited by memory,
power, and network bandwidth.
- Example devices :smart cards, cellular phones, pagers PDAs, palmtops,
screen phones, set-top boxes, game machines, VCRs and CD players.
- JavaTM 2 Platform, Micro Edition ( J2ME ) is a framework for
deploying Java applications over such devices.
- Mobile Media API 1.0 ( MMAPI) is an API providing multimedia support for the J2METM
platform.
- MMAPI is targeted at Connected Device Limited Configuration (CLDC ) and
Mobile Information Device Profile (MIDP ) based device.
|
74
|
|
75
|
- MMAPI does not mandate that certain media type and access protocols
should be supported by an implementation.
- The media type and access protocol support depends upon
- whether the native code underneath supports them,
- whether the MMAPI implementers exposes those native services through
the API.
- Sun’s Reference Implementation supports monotone, tone sequence, Wav,
MIDI, MPEG playback and audio capture.
- Wireless ToolKit (WTK) with MMEmulator can be used to develop multimedia
MIDlets.
|
76
|
- MPEG player
- Plays a file in the network using HTTP protocol
- Volume control
- Rate control
- Frame positioning
- Pause and restart
- Rewind or forward.
|
77
|
|
78
|
- Written using Java and JMF.
- Video telephony over Internet
- Answering machine
- GSM or G723 Audio format
- H.263 or JPEG video format
- Video Preview
- Text ChattingVoice and Video mail over Internet
- Video Conferencing in an Intranet.
- Auto detection of capture devices
|
79
|
|
80
|
|