Fast Access to Video Information

National Science Foundation Grant No. IRI-9502702
Principal Investigator: Thomas D.C. Little

Project Summary

The use of information retrieval mechanisms (e.g., Mosaic, Gopher) that allow the exchange and display of audio, stills, and video is rapidly becoming a major consumer of network bandwidth and storage. Without filtering mechanisms or hierarchical coding schemes one must process a great deal of data to yield sought information. To support future interactive information delivery services such as entertainment, distance learning, home health care, and telecommuting, there is a need to provide efficient means for locating, accessing, and delivering enormous amounts of video content. In this National Science Foundation-sponsored work, we seek to enable fast access during browsing and retrieval of this video content.

The research addresses an integrated view of multiple technologies necessary to support rapid and high availability access to information. We have focused on the specific areas involving (1) domain-specific information models for video content and capture for hierarchical coding and scalable delivery, (2) video query interfaces tailored to the information models, and (3) mapping of information content to developed delivery protocols for network bandwidth management. The end results are techniques and proof of concept for rapid and efficient delivery of video information while maintaining the ability to scale to many users.

Results were favorable in attaining the proposed research objectives. Particularly interesting results include the development of domain-specific information models for video content using metadata and ontologies; the development of an SGML-based annotation system that supports the data models, capable of GUI reconfiguration based on the captured DTD; the development of video composition techniques for information retrieval and personalization of video content; the development of evaluation metrics for quantifying video composition for news delivery applications; and the development of a proof-of-concept prototype news composition system.

In addition to the PI, seven students students received financial support under this grant. These include:

• G. Ahanger, PhD recipient, investigated access techniques for news video data, cohesive composition, and constraint-based customization of composition. Dr. Ahanger completed her dissertation in December of 1998.
• D. Venkatesh, PhD recipient, investigated data distribution techniques for large-scale video, general information storage, and distance learning technologies. Dr. Venkatesh completed his dissertation in August of 1997.
• R. Krishnan, PhD Candidate, is investigating the service aggregation problem and developed logical and physical video data models for the project.
• J. Begin, undergraduate engineering student, worked on digitization and annotation of news video content. He was also involved in populating a relational database with the annotated content.
• L. Kuczynski, MS recipient in computer engineering, implemented software components for the distance learning and news-on-demand applications.
• J. Tran, BS recipient, was supported in part by the work-study program (1996). He was involved in the digitization, collection, and annotation of video content via Vane.
• K. Srinivasan, MS student, was involved in the integration of the composition and evaluation schemes for Canvass. He also redesigned Canvass to function on the wide-area Internet using a SMIL-based delivery format.

In addition, the Vane project was implemented in part by two exchange students under the supervision of the PI. J. Martin, a senior computer engineering student has also been involved in research as a volunteer. He has contributed significantly to the collection and digitization of analog video for the distance learning and customized-news-service applications. These three students received no financial support under this grant.

Two students from the University of Padua were active in investigating topics related to the grant.

The major research activities, findings, training, and outreach activities of the project are described below.

• The development of an object model to support the creation of metadata for video annotation and the incorporation of the model in the annotation tool called Vane (Video Annotation Engine) for populating our system [Carrer et al., 1997].
• Interpretation of user access data for data server design. [Venkatesh and Little, 1995, Venkatesh and Little 1996].
• Development of a generic database model for video data. Application of the model to educational and news video data [Ahanger and Little, Webnet'97] [Ahanger and Little, ICMCS'97].
• Implementation of an application (SGML2SQL) to populate the Vane raw metadata content into application specific relational database model [Carrer et al., 1997].
• Development of an ontology for educational and news video data to aid in collection of metadata, query formulation, and composition of information [Ahanger and Little, ICMCS'97] [Klippgen et al., 1996].
• Development of a language for composition and customization of news video delivery [Ahanger and Little, CIT'98].
• The investigation of the distance learning environment and current technology to support this objective [Little and Venkatesh, 1995].
• Implementation of a prototype distance learning application called Easy Ed (Easy Education) as proof-of-concept [Ahanger and Little, Webnet'97].
• Investigation of technologies for a customized-news-service and design of system architecture [Ahanger and Little, ICMCS'97].
• Implementation of a prototype customized news service application called Canvass (Customized Access to News Video Archive Storage System) as proof-of-concept [Ahanger and Little, TKDE'99].
• Modeling of closed caption data associated with news video data to aid in video retrieval [Ahanger and Little, DS8'99].
• Investigation of automatic association of closed-caption data with corresponding video clips and automatic identification of segments according to the model of closed-captioned news data [Ahanger and Little, TKDE'98].
• Investigation of manual editing techniques for television newscast and how these can be adapted in automatic composition of newscast while simultaneously achieving customization [Ahanger and Little, TKDE'99].
• Digitization of 60 hours of analog educational and news video data into MPEG-1 system format.
• Annotation and population of 15 hours of educational and news video data.
• Recording and digitization of 240 hours of broadcast news video for study, investigation, and use in Canvass.
• Conversion of news video database content into SMIL-compliant format using Real video compression for wide-area Internet dissemination.
• Investigation of existing text retrieval techniques to use for retrieval of closed caption data associated with news video [Ahanger and Little, DS8'99].
• Investigation of the effect on recall and precision by using a hybrid technique. The technique utilizes transitive search and annotated metadata [Ahanger and Little, DS8'99].
• Study of the re-use pattern of existing video data in broadcast news. This study was valuable for understanding the automatic replacement of superfluous segments with interesting segments (transpositioning) without changing the context of the presentation.
• Composition of a questionnaire and MPEG-1 video clips for subjective feedback on quality of automatic news video composition techniques and validation of proposed evaluation metrics.
• Development of a methodology for transforming raw video data to query-usable form [Ahanger and Little CRC'98].
• Integration of the proposed set of composition schemes into the Canvass prototype.
• Development of evaluation metrics for characterizing the quality of composed video [Ahanger and Little, TKDE'99].

A great deal of the research conducted focused on synthesis. As such, the research findings consist of new models and techniques to achieve rapid and personalized video delivery. Major results of this research are:

• An object model to support the creation of metadata for video annotation and its incorporation in the annotation tool called Vane for populating our system [Carrer et al., 1997].
• A generic database model for video data and its application to educational and news video data [Ahanger and Little, Webnet'97] [Ahanger and Little, ICMCS'97].
• An application (SGML2SQL) to populate the Vane raw metadata content into application specific relational database model [Carrer et al., 1997].
• An ontology for educational and news video data to aid in collection of metadata, query formulation, and composition of information [Ahanger and Little, ICMCS'97] [Klippgen et al., 1996].
• A hybrid technique to improve recall and precision on video data. The technique utilizes transitive search and annotated metadata [Ahanger and Little, DS8'99].
• A language for composition and customization of news video delivery [Ahanger and Little, CIT'98].
• A prototype customized news service application called Canvass (Customized Access to News Video Archive Storage System) as proof-of-concept [Ahanger and Little, TKDE'99].
• A model for closed caption data associated with news video data to aid in video retrieval [Ahanger and Little, DS8'99].
• A methodology for transforming raw video data to query-usable form [Ahanger and Little CRC '98].
• A set of evaluation metrics for characterizing the quality of composed video [Ahanger and Little, TKDE'99].

Graduate student participants have gained a variety of independent research skills while undertaking literature research, mathematical analysis, simulation, and experimentation as part of the project. In addition, students have been expected to produce regular project reports and oral presentations that ultimately have been refined into conference and journal publications. This experience has proven to be invaluable in developing the students into independent investigators.

The research sponsored under this grant is promoted via the Web and via various open-house-type activities on campus, including support of Pathways, a program designed to encourage young women to pursue careers in science and engineering.

The project has produced a variety of reports, publications, and prototypes, described below.

• G. Ahanger and T.D.C. Little, "Data Semantics for Improving Retrieval Performance of Digital News Video Systems," IEEE Trans. on Knowledge and Data Engineering, Vol. 13, No. 3, May/June 2001, pp. 352-360 [pdf].
• G. Ahanger, "Techniques for Automatic Digital Video Composition,"Ph.D. Dissertation, Dept. of Electrical and Computer Engineering, Boston University, December 11, 1998 [pdf].
• G. Ahanger and T.D.C. Little, "Automatic Composition Techniques for Video Production, " IEEE Trans. on Knowledge and Data Engineering, Vol. 10, No. 6, December 1998 [pdf].
• G. Ahanger and T.D.C. Little, "Automatic Digital Video Production Concepts," Handbook on Internet and Multimedia Systems and Applications, CRC Press, Boca Raton, December 1998 [pdf].
• G. Ahanger and T.D.C. Little, "A Language to Support Automatic Composition of Newscasts," Journal of Computer and Information Technology, Vol. 6, No. 3, 1998, pp. 297-310 [pdf].
• M. Carrer, L. Ligresti, G. Ahanger, and T.D.C. Little, "An Annotation Engine for Supporting Video Database Population," Multimedia Tools and Applications Vol. 5, No. 3, November 1997, pp. 233-258 [pdf].
• W. Klippgen, T.D.C. Little, G. Ahanger, and D. Venkatesh, "The Use of Metadata for the Rendering of Personalized Video Delivery,'' in Managing Multimedia Data: Using Metadata to Integrate and Apply Digital Data, Amit Sheth and Wolfgang Klas (eds.), McGraw Hill, March 1998, pp. 287-318 [pdf].
• M. Carrer, L. Ligresti, T.D.C. Little, "A Tcl/Tk-Based Video Annotation Engine,'' Proc. 5th Annual Tcl/Tk Workshop '97, Boston, MA, July 1997 (extended abstract and poster) [pdf].
• M. Carrer, Environment for the Annotation of Video of Video Information via Metadata Collection and Management, Thesis, Department of Electronics and Informatics, University of Padova, March 1996.
• L. Ligresti Environment for Capture, Analysis, and Annotation of Video Information, Thesis, Department of Electronics and Informatics, University of Padova, March 1996.
• G. Ahanger and T.D.C. Little, A Survey of Technologies for Parsing and Indexing Digital Video, Journal of Visual Communication and Image Representation, special issue on Digital Libraries, Vol. 7, No. 1, March 1996, pp. 28-43 [pdf].
• T.D.C. Little and D. Venkatesh, "The Use of Multimedia Technology in Distance Learning," Proc. IEEE Intl. Conf. on Multimedia and Networking, Aizu-Wakamatsu, Fukushima, Japan, September 1995, pp. 3-17 [pdf].
• Venkatesh and T.D.C. Little, "Investigation of Web Server Access as a Basis for Designing Video-on-Demand Systems," Proc. 1st Intl. Symposium on Photonics Technologies and Systems for Voice, Video, and Data Communications, October 1995, Philadelphia, SPIE Vol. 2617-06 [pdf].
• D. Venkatesh and T.D.C. Little, A Model for Evaluating the Cost-Performance Characteristics of Single Disk Storage Systems for Supporting Digital Video Content, Proc. 6th Intl. Workshop on Network and Operating System Support for Digital Audio and Video, Zushi, Japan, April 1996, pp. 139-146 [pdf].
• G. Ahanger and T.D.C. Little, "A System for Customized News Delivery from Video Archives," Proc. 4th Intl. Conf. on Multimedia Computing and Systems, Ottawa, Canada, June 1997, pp. 526-533 [pdf].
• G. Ahanger and T.D.C. Little, "Easy Ed: An Integration of Technologies for Multimedia Education," Proc. of WebNet '97 Toronto, Canada, October, 1997 [pdf].
• T.D.C. Little, ``Fast Access Techniques for Video Information Retrieval,'' IDMS '98 Workshop Report.
• G. Ahanger and T.D.C. Little, "Automatic Digital Video Production Concepts," Handbook on Internet and Multimedia Systems and Applications, CRC Press, Boca Raton, December 1998 [pdf].
• G. Ahanger and T.D.C. Little, "Data Semantics for Improving Retrieval Performance of Digital News Video Systems," Proc. 8th IFIP 2.6 Working Conference on Database Semantics, Rotorua, New Zealand, January 1999 [pdf].
• G. Ahanger and T.D.C. Little, "The Canvass News Video Composition System," MCL Technical Report, TR-08-25-1999, August 1999 [pdf].

In addition to publication, the majority of the aforementioned papers appear as technical reports accessible from the project Web address (http://hulk.bu.edu/projects/fap/summary.html).

A variety of software was developed as proof-of-concepts under the grant. As such, its value has been in the demonstration of ideas rather than as deployable implementations. These ideas are described in detail in the listed publications. As appropriate, developed software is available via the aforementioned Web address.

Canvass, or Customized Access to News Video Archive Storage System, is a proof-of concept prototype video composition system that allows users to retrieve automatically-composed video stories based on input selection criteria. Composition is achieved at the shot level. In essence, users are provided with a set of news stories that are composed, on-the-fly, from available news content, considering their individual needs. We envision this concept to be applicable to future personalized video delivery systems as video storage and delivery systems become more ubiquitous. The Canvass prototype software demonstration system is no longer accessible from the project web site; however, it is described in MCL Technical Report TR-08-25-1999 [pdf] and the source code canvass-src.tar.gz for this application is available as well.

The project contributed to the multimedia computing domain as well as to other core scientific and engineering disciplines. These, and other contributions educational or societal goals are described below.

In the past few years there has been an unusual amount of activity in the commercial sector in the multimedia area, especially related to Web technology. This can be interpreted either as competition to the project or as a demonstration of increased significance of the work. Although we have produced prototypes to demonstrate proof of concepts, we view the major work as a conceptual substrate in the form of mathematics, techniques, and algorithms for the development of a variety of tools and applications that will be more robust than our prototypes.

The overall work is largely interdisciplinary, major contributions to various disciplines include (see ``Findings'' for additional details): (1) the development of domain-specific information models for video content using metadata and ontologies, (2) the development of an SGML-based annotation system that supports the data models, capable of GUI reconfiguration based on the captured DTD, (3) the development of video composition techniques for information retrieval and personalization ov video content, (4) the development of evaluation metrics for quantifying video composition for news delivery applications, and (5) the development of a proof-of-concept prototype news composition system.

Our work under this grant is inherently multidisciplinary, affecting fields of computer science, engineering, communications, and media production. It is also expected to influence the development of future commercial media production systems. Major interdisciplinary contributions are largely the integration of diverse technologies to yield a functional system that embraces text analysis, database/information system modeling, data storage and delivery, video (media) production, network media delivery, and the use of the Web.

Primary contributions are the development of new researchers in engineering through course development, independent graduate student research, and the support of outreach programs.

Teaching activities include the development of courses to reflect the changing needs of computer engineers and the use, evaluation, and promotion of computer-based instructional tools. A hands-on course in multimedia computing was developed proposed that addresses recent advances in computing technology.

In the Spring of 1996, thanks to the recent acquisition of new computer equipment the department for our Software Engineering Lab, the PI introduced a laboratory component consisting of seven two-week laboratory design/experimental elements. These labs were combined with existing student-led seminars and instructor-led lectures. Ultimately these materials have been be polished for use at the mixed graduate and undergraduate level. In the spring of 1997, these materials were augmented with a complete on-line (web-based) environment comprised of lab materials, lectures, and assignments. In subsequent years the material has been refined and refurbished. Although experimentation with video-based instruction was performed in this forum, it was not adopted due to its distraction from the primary educational goals.

The research activities supported by this grant will continue to serve as a substrate for additional research activities in the lab and in undergraduate and graduate teaching. In particular, spin-off projects are encouraged in the course SC748, multimedia computer design, towards related topic areas.

The work under this grant is intended to improve access to information in the video medium. We hope that through promotion and dissemination of our techniques we will ultimately contribute to the development of robust tools and products to achieve this goal in the greater scientific community. Our contributions to date potentially enable a new class of video-based applications that require high-bandwith network delivery. These are anticipated within five years.

Our work is intended to improve the ability to edit and deliver personalized video information content. We expect that our results will lead to more efficient and user-tolerable systems that may see widespread deployment within the next few years, much in the same manner that we have seen an explosion in text-based personalization systems on the Internet.

This research addresses an integrated view of multiple technologies necessary to support rapid and high availability access to video information. Major results include: (1) the development of domain-specific information models for video content using metadata and ontologies, (2) the development of an SGML-based annotation system that supports the data models, capable of GUI reconfiguration based on the captured DTD, (3) the development of video composition techniques for information retrieval and personalization ov video content, (4) the development of evaluation metrics for quantifying video composition for news delivery applications, and (5) the development of a proof-of-concept prototype news composition system.