Mixtral

The use case is exploratory literature review in a specific scientific field. I have a setup that takes pdfs and does some OCR and layout detection with Amazon, and then bunch them with some internal reports. Then, I have a pipeline to write summaries of each document and another one to slice them into chunks, get embeddings and set up a vector store for a RAG chat bot. At the moment it’s using Mixtral and the command line. But I like being able to swap LLMs to experiments with different models and quantisation without hassle, and I more or less plan to set this up on a remote server to free some resources on my workstation so the web UI could come in handy. Running this locally is a must for confidentiality reasons. I’d like to get rid of Textract as well, but unfortunately I haven’t found a solution that’s even close. Tesseract in particular was very disappointing.

Very well researched! I'd add the MR review use case. I have limited success with feeding a LLM (dolphin finetune of mixtral) a content of a merge request coming from my team. It was few thousand lines of added integration test code and I just couldn't be bothered/had little time to really delve. I slapped the diff and used about 10 prompt strategies to get anything meaningful. So my first initial impressions were: clearly it was finetuned on too short responses. It kept putting in "etc.", "and other input parameters", "and other relevant information". At one point I was ready to give up; it clearly hallucinated. Or that's what I thought: turned out there was some new edge case of a existing functionality added that was added, without ever me noticing (despite being on the same meetings). I think it actually saved me a lot of hours or pestering other team members.

Dolphin-mixtral is incredible for the size that it is. But I'm curious, have you tried Goliath-120b or the new `Mixtral_34Bx2_MoE_60B` (it's named Mixtral but the base is actually Yi). Goliath is too big for my system but Mixtral_34Bx2_MoE_60B[1] is giving me some really good results. PSA to anyone that does not understand what we're talkign about: I was new to all of this until two weeks ago as well. If you want to get up to speed with the incredible innovation and home-tinkering happening with LLMs, you have to checkout - https://www.reddit.com/r/LocalLLaMA/ I believe we should be at GPT4 levels of intelligence locally sometime later this year (Possibly with the release of Llama3 or Mistral Medium open-model). [1] - https://huggingface.co/TheBloke/Mixtral_34Bx2_MoE_60B-GGUF