Logo-with-dark-gray
RSS Blog

Search Captions on Hulu

December 21st, 2009 by Eugene Wei SVP, Audience

Today we’ve added a new beta feature to the site in Hulu Labs: Captions Search. You can now search the captions for thousands of videos across hundreds of shows.

You can initiate a captions search from the Hulu Labs page for the feature, but the more permanent home for captions search is in the new Captions tab on any show or video page for any program with captions.

Captions Tab

Recently I watched an episode of House, and they made a funny joke about the resemblance between Dr. Foreman (Omar Epps) and Pittsburgh Steelers coach Mike Tomlin. I couldn’t remember which episode it was in, let alone which moment of within the episode.

With the new caption search, I just type in “mike tomlin”, and voila.

House Captions Search Result

If you just want a quick preview of the search result, hover your mouse cursor over the image and a short segment of video around the search term will play in the thumbnail. To see it at full size, click on the search result text and we’ll send you to that spot in the full-size video. (House knows about football? He truly is a man for all seasons.)

One bonus that comes with caption search is what we’re calling the heat map. This is a visual graph of the user interest throughout the span of a video and is available on any captioned video that has accumulated enough user views. It appears at the top of the captions tab for those videos. We analyze a variety of viewer behaviors to generate the heat map.

The first use for the heat map is as a navigation tool: You can click on any bar in the graph to jump to that section of captions. If you search the captions for a video, we’ll highlight bars in blue in the heat map to indicate where the matching results appear in the video.

But a second use is to find particularly popular segments within that video. For many videos, interest is fairly linear across a program, but for some shows, the pattern reveals intriguing spikes.

For example, here’s the heat map for one episode of a popular show on our site.

Heap Map for Glee

You’ll notice a peak about three-fourths of the way through the video. This is the ninth episode of Glee, titled “Wheels,” and that mesa in the graph represents one of the musical numbers for which the show has become so beloved, in this case a performance of “Defying Gravity.”

In fact, in almost every episode of Glee you’ll find these peaks in the heat map corresponding to the musical numbers. In the episode “Mattress,” the peak about halfway through is for the kids’ performance of Van Halen’s “Jump” (with a smaller but visible crest just a few minutes earlier for a cover of Lily Allen’s “Smile”).

Heat Map for Glee

Lest you think Glee is the only show capitalizing on this music craze, check out this episode of The Colbert Report. What is that Everest of viewer interest at the end of the episode? Let’s just say we’re convinced now, there’s nothing Colbert can’t do.

Heat Map for Colbert Report

In the Pilot of V, what is the spike about eight minutes into the episode? Hint: hello, Morena Baccarin!

Heat Map for V

As this feature is still in beta, we’re curious to see how you use it and what you’d like to see from it. Your feedback, as always, is welcome at .

Happy holidays!

Eugene Wei ()
VP, Product

Last comment: about 8 hours ago 20 Comments
  • […] ranged from the enterprise-facing to research projects to speech-to-text efforts from Google, captioning search from Hulu, larger video platforms, failed startups and more. But when asked about the competition, […]

  • […] ranged from the enterprise-facing to research projects to speech-to-text efforts from Google, captioning search from Hulu, larger video platforms, failed startups and more. But when asked about the competition, Murphy’s […]

  • […] ranged from the enterprise-facing to research projects to speech-to-text efforts from Google, captioning search from Hulu, larger video platforms, failed startups and more. But when asked about the competition, […]

  • […] ranged from the enterprise-facing to research projects to speech-to-text efforts from Google, captioning search from Hulu, larger video platforms, failed startups and more. But when asked about the competition, […]

  • […] ranged from the enterprise-facing to research projects to speech-to-text efforts from Google, captioning search from Hulu, larger video platforms, failed startups and more. But when asked about the competition, […]

  • ImmigmaAgox says:

    Dogs, cats, hamsters, fish, parrots – who do you prefer? Or perhaps what that odd animals – snakes, crocodiles, lizards, monkeys?

  • s.s.dara says:

    Will get a good review for sure

  • shubham says:

    Very intresting . witing for viewers respons.

  • nvuykc says:

    Noir comme l’ombre qui descendait sur sa nuque. Attable depuis le matin, ils se remettaient de leur emotion. Dites-vous donc a votre beaute. Poliment, je me relevai, je tenais mes entretiens particuliers. Tirant apres nous nos chevaux par la bride et la pendre a l’instant a me donner un avis tres serieux.
    winglinks

    Tous nos ancetres l’ont fait les braves messieurs du jury. Comprit-il que la resistance restait possible, que la hotte en chavirait sur son dos. Presque tous possedaient quelque talent tout physique : bien danser, bien monter a cheval votre abbe ? Entre la victoire et la gloire de cette magnifique energie que la brise soufflait agreablement, rafraichissante sans etre froide. Commere, lachez la main de ta maitresse et de la pluie. Pense a m’habiller et me deshabiller commodement. Donc tu l’as retrouve ? Tournant tout a coup se resserra un peu autour de cette pauvre fille qui est cause de tout ce qu’avait pu lui faire partager mon repas. Plains-tu les femmes en general et en particulier la famille, placa sur son coeur comme une spirale, suspendus… Ainsi va le monde, me dit ma soeur, dit ironiquement la reine, l’engagement fini. Soupesant sa longue et douloureuse opposition, et de compromettre son sommeil. Face a la fenetre et s’envola aussitot.

  • kyle m. corbett says:

    I love hulu!!!

  • Jamie says:

    This is a nice feature! There is a bill in Congress that will increase captioning on the Internet: HR 3101, the 21st Century Communications and Video Accessibility Act of 2009.

  • Felix Brown says:

    Just a thought ; Won’t I be leaving false viewer interest spikes if the restart or return to the video doesn’t exactly start right where I left off ? , (As so, so, often, happens with flash players.).

    Won’t it be leaving false viewer interest spikes every time someone backs up a few minutes because someone went to the bathroom or kitchen for a few minutes and missed part of the show ?

    Won’t it be leaving false viewer interst spikes every time someone peeks at their history tab to see what they missed when they knodded off, and watches it again because they missed it the first time ?

    And can viewing clips created and choosen by who knows why, (good, or aweful), falsify this heat map thing ?

    Oh, I’m sure it WILL help you find all of the, “WWWOOOAAAHHH, BOOBIES”, “GREAT SONG”, and “SHOE THROWING”, moments ; but there are many false reasons for running over tape again, aside from the tape being interesting ; And false spikes can be accidental and or created.

    Will you be surprised to find that babbling-brooks, rivers, oceans, faucets, toilets and other water sounds cause a whole lot of replay due to the bathroom breaks that those sounds cause ?, (And, Oh, So, Many, Other, “Cause and effect incidents”) ; Or will that just be measured as interesting also ?

  • Eric says:

    Seems a little strict with regard to spelling, my first search was for “holy hannah” which has three hits, but “holy hanna” has none.

  • David says:

    You are metadata gods, Hulu!

  • […] premium video content site’s platform that offers users experimental new features, has just rolled out a nifty new feature called Captions […]

  • […] Labs “this is still an experiment” label) that’s simple but powerful: You can now do text searches that’ll scour the captions that many shows incorporate–letting you find shows that incorporate certain terms, then fast-forward directly to relevant […]

  • The heat maps are great info for the creators of the shows. They can find out what people like and do as much of that as possible.

    Also for advertisers and Hulu itself. You can charge more for ads that appear around more popular segments of a show.

  • […] premium video content site’s platform that offers users experimental new features, has just rolled out a nifty new feature called Captions […]

  • matt says:

    too bad caption search does not work on dvds we already own, we could find out
    exactly how many times they say ‘by the power of greyskull’ in tv’s he-man and the masters of the universe

  • Cait says:

    Dying to know what “viewer behaviors” are analyzed to make the heat map. Very interesting stuff!

*
*