Analyzing video, audio & text with AI

We helped automate the laborious process of generating metadata for longform content, empowering content creators to find their audience

BEFORE

Broadcasting experts frustrated that most high-quality media was effectively unsearchable due to the labor required to categorize it

Wanted to create an automated solution for extracting meaningful information from longform content

Needed a flexible tool able to identify and deliver relevant, actionable information to content creators and distributors

AFTER

Developed MVP for an AI-powered meta-data generator to extract topics, keywords, and more, tying it to relevant segments and optimizing for domain-specific content

Released a versatile API solution easily integrated with video, audio, and blog hosting platforms

Productionalized by a major US sports organization and international content distribution

Challenge

Filtering a media firehose

As leaders during the meteoric rise of radio giant SiriusXM, Dan Benjamin and Marlon Osbourne (DBMO) had front-row seats to the digital broadcasting revolution (and the content explosion it sparked). But while platforms for pushing out videos, podcasts, and blogs had proliferated and evolved, tools for keeping track of all that information had fallen behind – to the great detriment of creators and audiences. Dan and Marlon believed an AI solution could deliver this critical information, and they asked us to help bring their vision to life.

Process

Adapting AI for multimedia analysis

We helped bring the DBMO platform from ideation to fruition. We combined Dan and Marlon’s industry expertise with our own research to identify the data categories most valuable to content creators: entities, topics, themes, and keyword data. We then ideated and prioritized features to maximize utility for our target users. Since so much of the project’s success depended on finding innovative ways to leverage AI, we took an iterative approach to design and development. We explored ways to apply AI analysis techniques to different types of media, adjusting the UI and UX along the way as we refined the product’s capabilities towards a robust MVP.

Solution

Text as an anchor

To create one tool capable of analyzing various media types, our first priority was converting them to a common denominator: text. Written language was already the primary information vehicle for blogs and news outlets, and videos with audio and podcasts could be automatically converted into time-stamped transcripts that captured most of their meaning. Homogenizing the content’s format freed us to refine an AI application designed to pull relevant information out of text-based files (and to deliver its findings in an actionable, versatile format).

Solution

Image recognition

Since auto-transcription captures only the audio portion of multimedia content, the DBMO product also needed visual interpretation abilities to provide high quality video analysis. We incorporated image recognition features able to supplement voice transcriptions to provide a more complete catalog of where entities, themes, and topics are present throughout a piece of video content.

Solution

Entities, topics & themes

We combined the text/transcript and image data streams and used AI analyses to identify entities (location names, people), topics (events, news items), themes (media genre). Using similarity matching via Doc2Vec and other text-focused AI recognition and evaluation techniques, we trained DBMO to generate time-associated metadata ready to be leveraged for search engine and content optimization.

Solution

Actionable insights

The metadata scraped by the DBMO platform can enable powerful positive feedback loops for content creators and publishers. Entities, topics, and themes can be leveraged for keyword research that can help tag and reference content in SEO-friendly ways, improving searchability. The corresponding increase in views or listens can, in turn, provide content makers with invaluable insights about their audiences’ preferences, driving content strategy in the future. The auto transcription and image recognition features can also be used to generate closed captions and image/video alt text, enhancing content accessibility and further increasing the potential consumer pool.

Praise from our clients

Partnering with Twenty Ideas was incredible for the development of our product. They are highly collaborative, communicative, and deeply invested in creating the best work possible, and that came through beautifully in both the process and the product!