Developing multimedia    applications with the

JavaTM Media Framework

( includes JMF 2.1.1 & MMAPI )

 

First Edition

 

 

T.G.Venkatesh

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

CONTENTS

                                                                           Page

1.      Introduction

1.1     Java for Internet computing -----------------------------------------------------------------                     15

1.2     Java Media APIs ----------------------------------------------------------------------------------                      16

1.3     Java Media Framework -----------------------------------------------------------------------                      18

   What is JMF?------------------------------------------------------------------------------------                       18

   Criticisms on JMF ----------------------------------------------------------------------------                       19

   Why JMF? ---------------------------------------------------------------------------------------                        19    

1.4     Organization of the book ---------------------------------------------------------------------                      20

                                          

2.      Getting started with JMF                                           

2.1     History and various releases of JMF ------------------------------------------------------                    25

2.2      Versions of JMF  -----------------------------------------------------------------------------------                    25

2.3     Hardware platform requirements -----------------------------------------------------------                   26

2.4     Software requirements --------------------------------------------------------------------------                    26

    Operating system  ------------------------------------------------------------------------------                     26

                 Java platform requirements (JDK) ------------------------------------------------------                   26

                 Optional browser requirements ----------------------------------------------------------                   26

                 Optional Integrated development environments --------------------------------                   27

2.5     Media formats supported by the JMF -----------------------------------------------------                    27

2.6     Installing JMF ----------------------------------------------------------------------------------------                    27

2.7     Using multimedia devices with JMF -------------------------------------------------------                   27

                 Multimedia devices supported by JMF --------------------------------------                   27

                 Adding new devices ---------------------------------------------------------------------------                   28

2.8     The JMF Registry -----------------------------------------------------------------------------------                    28

2.9     The JMStudio ----------------------------------------------------------------------------------------                    29

2.10 HelloJMF.java ---------------------------------------------------------------------------------------                    34

2.11 Debugging JMF programs: the jmf.log file --------------------------------------------                    36

 

3.      Multimedia : an introduction

3.1     General media compression techniques -----------------------------------------------                    39

            Quantization ------------------------------------------------------------------------------------                     40

            Run length coding ---------------------------------------------------------------------------                     40

            Entropy coding --------------------------------------------------------------------------------                     40

            Predictive coding ----------------------------------------------------------------------------                     40

            Transform coding  ---------------------------------------------------------------------------                     40

            Discrete Cosine Transformation -------------------------------------------------------                     41

            Zigzag scanning of DCT coefficients -----------------------------------------------                      41

            Motion estimation ญ---------------------------------------------------------------------------                      41

3.2     Audio coding techniques and standards ญญญญญญญญญญญญญญญญญญญญญญญญญญ----------------------------------------------                      41

           Speech codecs --------------------------------------------------------------------------------                      42

             Vocoders ---------------------------------------------------------------------------------------                      42

             Waveform codecs --------------------------------------------------------------------------                      42

             Pulse code modulation ------------------------------------------------------------------                      42

             DPCM ---------------------------------------------------------------------------------------------                      43

             ADPCM ญญ------------------------------------------------------------------------------------------                      43

             Sub-band coding ---------------------------------------------------------------------------                      43

             Hybrid codecs --------------------------------------------------------------------------------                      43

             Code Excited Linear Prediction ----------------------------------------------------                       44

3.3     Image coding standard -----------------------------------------------------------------------                       44

           Color formats ----------------------------------------------------------------------------------                       44

          JPEG format ------------------------------------------------------------------------------------                       45

3.4 Video coding standards -----------------------------------------------------------------------                       45

          The CIF and QCIF standard -------------------------------------------------------------                       45

           H261 standard --------------------------------------------------------------------------------                        45

           H.263 videoconferencing standard for POTS ---------------------------------                        46

           Moving Picture Expert Group standards -----------------------------------------                       46

           ( MPEG-1, MPEG-2, MPEG-4, MPEG audio standard)

3.5    Proprietary standards ญ-------------------------------------------------------------------------                       47

 

4        A quick tour to JMF

    4.1 The JMF Model  ------- -----------------------------------------------------------------------------                       53

    4.2 Representing media ------------------------------------------------------------------------------                        53

    4.3 Media source -----------------------------------------------------------------------------------------                       53              

         MediaLocator -----------------------------------------------------------------------------------                       54

         DataSource -------------------------------------------------------------------------------------                        54

         DataSink -----------------------------------------------------------------------------------------                         54

   4.4  Media Playback -------------------------------------------------------------------------------------                       55

MediaHandler ----------------------------------------------------------------------------------                         55

  Controller -----------------------------------------------------------------------------------------                        55

               Player ----------------------------------------------------------------------------------------------                        55

   4.5  Media Processing ----------------------------------------------------------------------------------                        55

         PlugIn ----------------------------------------------------------------------------------------------                        56

         Processor ------------------------------------------------------------------------------------------                       56

  4.6  Managers ญญญญญ-----------------------------------------------------------------------------------------------                        57

        Manager ญญ--------------------------------------------------------------------------------------------                       57

        PlugInManager ----------------------------------------------------------------------------------                       57

        CaptureDeviceManager ---------------------------------------------------------------------                       57

        PackageManager -------------------------------------------------------------------------------                      57

  4.7  JMF Controls -------------------------------------------------------------------------------------------                       57

  4.8  JMF RTP API -------------------------------------------------------------------------------------------                       57

       RTP ---------------------------------------------------------------------------------------------------                         58

       RTCP -------------------------------------------------------------------------------------------------                        58

       RTPSession ---------------------------------------------------------------------------------------                        58

4.9 The JMF Time model --------------------------------------------------------------------------------                        58

        Time -------------------------------------------------------------------------------------------------                        58

        TimeBase ------------------------------------------------------------------------------------------                       59

        Clock -------------------------------------------------------------------------------------------------                       59

        Duration --------------------------------------------------------------------------------------------                        62

4.10 Media Format ------------------------------------------------------------------------------------------                        62

 Format ญ----------------------------------------------------------------------------------------------                         62

 AudioFormat --------------------------------------------------------------------------------------                        63

 VideoFormat---------------------------------------------------------------------------------------                        66    

              Buffer ------------------------------------------------------------------------------------------------                        66

       ContentDescriptor -------------------------------------------------------------------------------                       66

       The RGB Format ---------------------------------------------------------------------------------                       68

       The YUV Format ----------------------------------------------------------------------------------                      68

 

     

5        Media source

5.1   Specifying media source using MediaLocator  ------------------------------------                       70

5.2 Specifying media source using URL -----------------------------------------------------                      74     

5.3    DataSource -----------------------------------------------------------------------------------------                       76

5.4    DataSource interfaces -------------------------------------------------------------------------                      83

     Positionable------------------------------------------------------------------------------------ -                     83

           SourceCloneable -----------------------------------------------------------------------------                      83

          CaptureDevice ---------------------------------------------------------------------------------                       85

          RateConfigurable -----------------------------------------------------------------------------                      85

5.5 Push and Pull DataSources -------------------------------------------------------------------                      86

            Buffer ----------------------------------------------------------------------------------------------                      87

            DataSource Classification ---------------------------------------------------------------                      87

5.6  SourceStreamClassification ------------------------------------------------------------------                     89

            PullSourceStream ---------------------------------------------------------------------------                     89

            PushSourceStream -------------------------------------------------------------------------                      89

                  Pull and Push BufferStreams -----------------------------------------------------------                     92

                      

6. Media presentation

6.1    The Controller  interface -----------------------------------------------------------------------                      95

6.2    Controller states -----------------------------------------------------------------------------------                       97

                Unrealized, Realizing and Realized states ---------------------------------------                      97

                Prefetching and Prefetched state -----------------------------------------------------                      97

                Starting and stopping a Controller ---------------------------------------------------                      98

                Deallocating and closing a Controller ----------------------------------------------                      98

                Startup latency ---------------------------------------------------------------------------------                      98

6.3    The Player interface -----------------------------------------------------------------------------                      99

6.4    ControllerListener --------------------------------------------------------------------------------                       104

6.5    An experiment with the asynchronous Controller methods -------------------                     107

6.6    ControllerAdaptor ---------------------------------------------------------------------------------                      110

6.7    Anatomy of a Player -----------------------------------------------------------------------------                      112

     6.8  Controller Events ----------------------------------------------------------------------------------                       115

 

7. Media Capture

7.1    Supported capture  devices ------------------------------------------------------------------                     124

7.2    Registering capture devices ------------------------------------------------------------------                     124

7.3    JMF API support for capturing media ----------------------------------------------------                     126

                CaptureDevice interface --------------------------------------------------------------------                     126

                Class CaptureDeviceInfo -------------------------------------------------------------------                      126

                Class CaptureDeviceManager -----------------------------------------------------------                      128

7.4    Example to find registered capture devices -----------------------------------------                      128

7.5    Capturing  media and presenting it to the user ------------------------------------                      130

          Example for capturing media ----------------------------------------------------------                       131

          Example for captured media ----------------------------------------------------------                        133                                          

 

8        Media processing

8.1 Media processing stages ญ------------------------------------------------------------------------                     137             

8.2 PlugInManager -------------------------------------------------------------------------------------                      140

          PlugInManager methods ------------------------------------------------------------------                       140

           Example to list registered PlugIns ---------------------------------------------------                      142

8.3 ProccesorModel -------------------------------------------------------------------------------------                     143

8.4 Controlling the media processing ---------------------------------------------------------                      147

          Configured state ------------------------------------------------------------------------------                       149

          Track ----------------------------------------------------------------------------------------------                        150

          TrackControl -----------------------------------------------------------------------------------                        150

          Processor’s output --------------------------------------------------------------------------                        151

          Buffer-----------------------------------------------------------------------------------------------                        152

          ContentDescriptor --------------------------------------------------------------------------                         152

 

9  Processor configuration

9.1 The asynchronous nature of the configure() method. ----------------------------                      155

9.2 Using the configureCompleteEvent handler ------------------------------------------                      160

9.3 Invoking realize() in ConfigureCompleteEvent handler -------------------------                     163

9.4 Making Processor state transitions synchronous ------------------------------------                     165

9.5 Waiting for the Processor to reach the target state --------------------------------                     168

9.6 Setting codecs of a Processor ----------------------------------------------------------------                     174

9.7 Setting Renderers of a Processor ----------------------------------------------------------                      177

      9.8 The utility ProcUtil.java -------------------------------------------------------------------------                      179

 

10 Media storage

10.1 DataSink ------------------------------------------------------------------------------------------                          188

10.2 DataSinkEvents & DataSinkListeners --------------------------------------------------                       190

      10.3 Storing captured media into  a file -----------------------------------------------------                      190

      10.4 Storing both audio and video in the same  file -----------------------------------                      194        

 

11 Simple media transmission and reception                                  

 11.1 Media transmitting and receiving using JMStudio ------------------------------                     199

 11.2 Media transmission using a DataSink -------------------------------------------------                      201   

 11.3 Media reception using a MediaLocator -----------------------------------------------                     206      

 

12   JMF Applet

   12.1  JMF Player in an Applet ------------------------------------------------------------------                      210

    12.2  Media capture and storage in an applet -----------------------------------------                       213

         12.3  Browser support  for JMF-----------------------------------------------------------------                      213

 

ADVANCED TOPICS

                  

13    JMF RTP API   

      13.1 Real-time Transport Protocol --------------------------------------------------------------                      215

       13.2  RTP Specification -----------------------------------------------------------------------------                       216

       13.3  RTP Control Protocol ------------------------------------------------------------------------                       217

 13.4  The JMF RTP API Terminology ----------------------------------------------------------                     218

       13.5  The javax.media.rtp package ------------------------------------------------------------                     220

 13.6  The package javax.media.rtp.rtcp -----------------------------------------------------                     223

       13.7  The package javax.media.rtp.events -------------------------------------------------                     224

       13.8  RTPEvent Listeners ----------------------------------------------------------------------------                     228

       13.9  RTPEvent listener adaptors ----------------------------------------------------------------                     229

 

 

 

 

14 Multimedia communication using JMF RTP API- Part 1

          14.1 Media reception using JMF RTP API --------------------------------------------------                   234

Media reception through a Processor ----------------------------------------------                  245        

A GUI for media reception --------------------------------------------------------------                   245        

         14.2 Media transmission using JMF RTP API ญญญญญญญญญญญญญญญญญญญญญญญญญ----------------------------------------------                   248

                     Transmitting a media stream ----------------------------------------------                   249
                     Creating a media stream ----------------------------------------------------------------                   253

                    A GUI for media transmission ----------------------------------------------------------                   256

 

15. Multimedia communication using JMF RTP API- Part 2

      15.1 Simultaneous transmission and reception of media -----------------------------                    260

      15.2 Transfer of audio and video ------------------------------------------------------------------                    264

                   Audio and video transmission ----------------------------------------------------------                   264

                   Audio and video reception ---------------------------------------------------------------                   271

      15.3 Processing the received media. -------------------------------------------------------------                   273

                   Storing the received media --------------------------------------------------------------                   274

                   Retransmission of received media ----------------------------------------------------                  278

 

16     JMF Control

      16.1   The interface Control and Controls ------------------------------------------------------                   281

      16.2   Gain, mute, and rate control            -----------------------------------------------------                   284

      16.3   Buffer control --------------------------------------------------------------------------------------                    291

 16.4   Quality control ------------------------------------------------------------------------------------                     295

 16.5   Stream writer control -------------------------------------------------------------------------                     298

      16.6  Monitor control -----------------------------------------------------------------------------------                      298

16.7     Bit rate control ----------------------------------------------------------------------------------                      300

16.8     Frame controls ----------------------------------------------------------------------------------                      301

 

17 Multiple DataSources and Players

17.1 Cloning DataSources  ---------------------------------------------------------------------------                     302

17.2 Merging DataSources ---------------------------------------------------------------------------                     312

17.3 Controlling multiple players -----------------------------------------------------------------                     318

 

18    Multimedia Player Bean

18.1     JavaBeans component architecture ------------------------------------------                    323

18.2     Using MediaPlayer in JMF programs ----------------------------------------------------                   325

 

19    Extending JMF

19.1     Extending PlugIns ญญ------------------------------------------------------------------------------                     329

19.2     Using PlugIn as a Processing stage ----------------------------------------------------                     335

19.3     Registering a PlugIn with the PlugIn Manager -----------------------------------                     339

19.4     An audio Effect PlugIn -----------------------------------------------------------------------                      342

19.5     A video Effect PlugIn --------------------------------------------------------------------------                     346

19.6     Defining new control interfaces for a PlugIn --------------------------------------                     351

19.7     Extending Multiplexer,Demultiplexer and Renderer --------------------------                     357

19.8     Extending DataSource, DataSink, and Controller --------------------------------                    358

19.9     Extentions related to JMF RTP API ----------------------------------------------------                      358

                     Extending Packetizer and DePacketizers ------------------------------------                       358

                     Dynamic payload -------------------------------------------------------------------------                      359

                     RTPConnector ------------------------------------------------------------------------------                     360

 

JMF-based Multimedia Projects

 

20. Multimedia Projects

Aim -----------------------------------------------------------------------------------------------------                        362

Design specifications --------------------------------------------------------------------------                        362

 

21. Video phone-----------------------------------------------------ญ------------------------------------                       364

 21.1 Detailed design specifications ------------------------------------------------------------                      366

       21.2 Design of videophone ------------------------------------------------------------------------                      371

       21.3 Implementation of videophone ----------------------------------------------------------                      371

                  User interface module ---------------------------------------------------------------------                     374

                  Connection manager ---------------------------------------------------------------------                       374

                  Videophone module -----------------------------------------------------------------------                      382

                  Chat manager ---------------------------------------------------------------------------------                      385

                  Overall co-ordination -----------------------------------------------------------------------                     388

       21.4 Source code of JMF classes -----------------------------------------------------------------                    390

 

22. Video & Voice mailing system ญญ

 22.1 Detailed design specifications --------------------------------------------------------------                    391

       22.2 Design of video & voice mailing system ------------------------------------------------                   391

       22.3 Implementation of video & voice mailing system ---------------------------------                    395

                  Mail Transmission module ----------------------------------------------------------------                    396

                   Mail Reception module --------------------------------------------------------------------                   399

                  Mail box module ---------------------------------------------------------------                    401

                  Mail management module ---------------------------------------------------                    403

      22.4 Source code of JMF classes   ----------------------------------------------------------------                    404

 

23. A simple Video Conferencing system

 23.1 Design of a simple video conferencing system --------------------------------------                  405

       23.2 Implementation of the video conferencing system --------------------------------                   406

       23.3 Source Code of JMF classes ------------------------------------------------------------------                   409

24. Cosmophone

 24.1 Integration of subsystems -------------------------------------------------------------------                      410

       24.2 The help facility ----------------------------------------------------------------------------------                     410

24.3 Debugging ------------------------------------------------------------------------------------------                      411

24.4 Auto detection of capture devices ------------------------------------------------------                      411

24.5 Serial number ------------------------------------------------------------------------------------                       411

24.6 Documentation -----------------------------------------------------------------------------------                      412

24.7 Deployment of Cosmophone ----------------------------------------------------------------                     412

 

Special topics of JMF

1        Mobile media API

   The J2ME platform -----------------------------------------------------------------------------------                      414

   The Connected Limited Device Configuration (CLDC) ------------------------------                     414

   The Mobile Information Device Profile (MIDP)------------------------------------------                      415

   The MIDlet -----------------------------------------------------------------------------------------------                      416

   What is MMAPI? --------------------------------------------------------------------------------------                      416

   The MMAPI architecture --------------------------------------------------------------------------                      417

   The DataSource -------------------------------------------------------------------------------------                        418

   The Player ---------------------------------------------------------------------------------------------                        418

   The Manager ------------------------------------------------------------------------------------------                       420

          Supported Media formats and protocols ------------------------------------------------                       421

          The Player Control ---------------------------------------------------------------------------------                      422

        Developing multimedia MIDlets ---------------------------------------------------------------                       422

              The Wireless ToolKit (WTK) ---------------------------------------------------------------                       422

              The MM Emulator -------------------------------------------------------------------------------                      425

        Example multimedia MIDlets --------------------------------------------------------------------                      426

               Playing mono tone ----------------------------------------------------------------------------                      426

              Playing a tone sequence --------------------------------------------------------------------                      428

              Playing a wav file with simple playback control -------------------------------                       430

              Playing a MIDI file ------------------------------------------------------------------------------                      432

              Capturing audio and playing back --------------------------------------------------                        434

              Capturing audio and recording --------------------------------------------------------                       434

               Developing  a MPEG Player ------------------------------------------------------------                        435

 

2         IBM’s HotMedia -----------------------------------------------------------------------------------------                        446

 

3.    JMF Customizer ---------------------------------------------------------------------------------------                        447

 

4.    Anatomy of the JMF ---------------------------------------------------------------------------------                        450

 

APPENDIX

 

Appendix A : Brief introduction to the UML notation used in the book. -----------------                 454

 

Appendix B : The hardware and software requirements of the various  ----------------                457

                       versions of JMF

 

Appendix C : The various media formats and capture devices supported -------------                 460

                       by JMF

 

Appendix D : Various Errors and exceptions thrown by the JMF objects. ------------                   465

 

Appendix E : Java-enabled smartphones supporting MMAPI --------------------------------                 467

 

References

 

Index