Topic Tagging Using Large Language Models

Topic Tagging

Topic tagging is an important and widely applicable problem in and Latent Dirichlet Analysis [1], topic tagging has historically been a labor-intensive task, especially when there are many fine-grained topics. There are numerous applications to topic-tagging, including:

• Content organization, to help users of websites, libraries, and other sources of large amounts of content to navigate through the content


• Recommender systems, where suggestions for products to buy, articles to read, or videos to watch are generated wholly or in part using their topics or topic tags


• Data analysis and social media management — to understand the popularity of topics and subjects to prioritize

Large Language Models (LLMs) have greatly simplified topic tagging by leveraging their multimodal and long-context capabilities to process large documents effectively. However, LLMs are computationally expensive and require the user to understand the trade-offs between the quality of the LLM and the computational or dollar cost of using them.