Apple Researchers Propose Cut Cross-Entropy (CCE): A Machine Learning Method that Computes the Cross-Entropy Loss without Materializing the Logits for all Tokens into Global Memory

" data-medium-file="https://www.marktechpost.com/wp-content/uploads/2024/11/Screenshot-2024-11-15-at-12.01.53 PM-300x204.png" data-large-file="https://www.marktechpost.com/wp-content/uploads/2024/11/Screenshot-2024-11-15-at-12.01.53 PM-1024x695.png" tabindex="0" role="button"> " data-medium-file="https://www.marktechpost.com/wp-content/uploads/2024/11/Screenshot-2024-11-15-at-12.01.53 PM-300x204.png" data-large-file="https://www.marktechpost.com/wp-content/uploads/2024/11/Screenshot-2024-11-15-at-12.01.53 PM-1024x695.png" tabindex="0" role="button">Advancements in large language models (LLMs) have revolutionized natural language processing, with applications spanning text generation, translation, and summarization. These models rely on large amounts of data, large parameter counts, and expansive vocabularies, necessitating sophisticated techniques to manage computational and memory requirements. A critical component of LLM training is the cross-entropy loss computation, which, while […]

The post .