Repository navigation
Docs on using TypeChat with other LLMs #61
Description
What I understand is Typechat can be an alternative to lang chain library but in whole documentation I only see this being used for OpenAI's GPT Model only. So, is there any option to use this with any Open source model like LLAMA or Falcon ?
Activity
DKFN commentedon Jul 26, 2023
You can implement any
TypeChatLanguageModelthat satisfies the interface.It is not documented aside from the code source but if you need an example you can probably use my early Quivr implementation for typechat here as a reference:
https://github.com/DKFN/quivr-typechat/blob/e1c68c2c5e9d9479386cf66f6b6438dcac32e05b/src/main.ts
This is how I run typechat on my Quivr instance using GPT4All
I hope this helps !
EDIT: Reference exact commit sha and file. For now, the code is basic enough to be an example but it will become more complex in the future and may become confusing for ppl searching for examples.
Underewarrr commentedon Jul 26, 2023
Why not make some type of documentation for these steps ? I see as valuable resource!
If complexity start getting bigger, documentation can be upgraded also!
DKFN commentedon Jul 26, 2023
Thanks Rafhael Oliveira (@Underewarrr) that is a great suggestion !
I will start working on documentation about it after work
DanielRosenwasser commentedon Jul 26, 2023
Hey all, we'll probably pursue some specific docs on plugging in a specific model.
Until then, I've also put together a doc over at #65 which specifically points out
TypeChatLanguageModelas something you can implement to make TypeChat work with any language model.[-]How to use it with any other open source LLM like LLAMA ?[/-][+]Docs on using TypeChat with other LLMs[/+]DanielRosenwasser commentedon Jul 27, 2023
Speaking of which, there are at least 2 PRs (#34 and #55) which are entirely about adding a proxy to the built-in
create*LanguageModelfunctions. Maybe we should include proxies in our documentation alongside plugging into specific language models.As an aside, over time I don't know how many external language models we will have "out-of-the-box" support for. We might opt to keep things as-is, add more support, or remove all the built-in helpers. Others on the team might feel differently.
renambot commentedon Aug 2, 2023
DKFN (@DKFN) thanks for your example. It seems to work when adapting to Text Generation Inference (TGI) serving a Llama model.
DKFN commentedon Aug 2, 2023
Awesome! I am happy it worked for you :)
I have dropped the documentation effort since Daniel mentionned works was already on the way for it.
In the meantime, I am glad you found my little code useful :)
robgruen commentedon Oct 12, 2026
Closing the original documentation request as completed. Basic TypeScript Usage: Providing a Model now explains the bring-your-own-model contract. A model adapter implements
TypeChatLanguageModel.complete; it does not need to use OpenAI. The current public interface describes the supported prompt shapes.The concrete Ollama walkthrough remains tracked in #183 rather than keeping this general missing-documentation request open.