John Kender | Indexing Videos Automatically

John Kender
Professor of Computer Science
This profile is included in the publication Excellentia, which features current research of Columbia Engineering faculty members.
                                    Photo by Eileen Barroso

Many colleges videotape classes so students can review lectures, notes, equations, pictures, presentations, computer screens, and simulations. Yet students rarely use videos to review for exams. Why? Because it takes too long to find the topics and references they need.

John Kender hopes to solve that problem with software that automatically indexes videos. Just like in a book, his index enables people to find exactly what they want in a video.

“It is hard to index a video,” said Kender. “Most presenters move around and change the subject. Those taping them often lack training. There are none of the classical clues, like fades or establishing shots, to indicate a change in topic. We are developing software to find those clues and create an index.”

Kender has tested the software on videotaped lectures at Columbia. “We have shown that our tools helped students effectively locate the parts of lectures they wanted to study,” he said. “After we gave them the tools, their grades improved between midterm and final exams. If you have a good way of reviewing educational videos, it pays off in your grades.”

Indexing videos is no simple task. “A professor may start a lecture with a slide, move to a website, then stop to answer a question. That may trigger something he or she forgot to say earlier. They may start four new ideas without finishing previous ones,” Kender related.

Kender’s team found several ways to keep track of this convoluted discourse. One student developed software that recognized and indexed programming languages when they flashed across the screen. “Students can ask for all examples of software code and quickly page through them to find what they want,” Kender said.

Another program reads words from presentations, handwriting from white boards, and captures spoken words using speech-to-text software. Although handwriting and speech-to-text identification is not highly accurate, speakers repeat key concepts often enough to locate them in the video. Another program matches the resulting index with textbook chapters and articles that cover the same material.

A current project involves gestures. Instructors typically use different gestures when reviewing old material, introducing concepts, or working through difficult problems. These gestures vary from teacher to teacher. Kender hopes to decode their meaning by correlating them with such actions as showing new slides, writing on a board, or introducing new words.

Visual indexing could change how people use videos. “It’s like providing a table of contents and index for a book that didn’t have them before,” Kender concluded.

B.S., University of Detroit, 1970; M.S., University of Michigan, 1972; Ph.D., Carnegie Mellon University, 1980

500 W. 120th St., Mudd 510, New York, NY 10027    212-854-2993