The Fable model, a cutting-edge language model, has been making waves in the AI community. However, its high cost has been a significant barrier to adoption. Recently, a novel approach has been proposed to reduce Fable costs by converting code to images and leveraging Optical Character Recognition (OCR) technology [1]. This method achieves a remarkable 60% cost reduction without compromising model performance.

The idea is to render bulky context as images, which are then read by the model using OCR. This approach takes advantage of the fact that images have a fixed token cost, regardless of the amount of text they contain [2]. The pxpipe library, developed by teamchong, implements this approach and has been shown to reduce Fable costs significantly [3]. The library works by intercepting requests, rewriting eligible bulk history into image blocks, and splicing them back into the request [4].

The economics of this approach are clear: a 1928×1928 image costs approximately 4,761 vision tokens and can hold up to 92,000 characters, making it a more cost-effective option than plain text [5]. While this approach is not without its limitations, including the potential for silent confabulation and the need for careful handling of exact strings [6], it represents a significant step forward in reducing Fable costs. As the AI community continues to evolve, it will be exciting to see how this approach is refined and improved.

Sources

  1. https://news.ycombinator.com/item?id=48776464
  2. https://pub.towardsai.net/deepseek-ocr-a-picture-is-worth-a-thousand-words-e2a8b9d74c7f
  3. https://github.com/teamchong/pxpipe
  4. https://medium.com/@pcb.it18/cutting-llm-costs-by-converting-long-text-into-images-28eebc61656d
  5. https://www.threads.com/@marcslove/post/DZYBDVYHVUo/the-fable-is-me-paying-that-much-for-tokens
  6. https://www.instagram.com/reel/DZZEuRbMRtv