We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation .
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Hello - excellent project, I'm super excited about any option for decentralized AI.
I'm extremely green to this domain - I was hoping someone could help me understand GPU requirements for deploying this. I see many other in the issues boasting 4090s, dual 3080s, etc. - I'm wondering if it's viable for me to try using the 70b with my paltry 6700xt and ryzen 3700x (or what the max model to use with no GPU at all if any, and what the state of AMD support is). I recognize all this info is out there - I'd greatly appreciate links to resources for my own research.
The text was updated successfully, but these errors were encountered:
I can't speak to the state of support with AMD is, but currently for GPU usage, CUDA is required. Largely I believe the larger number of cuda cores has a pretty large impact on performance (as does the amount of ram in each card).
As for the 70b, I am running that as a test on a i5-9400k with 64G of ram and no GPU support. Its fairly slow, especially compared to running the 13b model. I am running a 13b model with decent performance in a vm on top of a ryzen 5 3600x, so from that perspective you should be good.
Sorry, something went wrong.
I run 34b model with 13900k CPU only, 32 threads 100% running still taking minutes in order to get full reply. I'm not sure if this speed is what I was expecting.
Feels like 34b is out of reach for mere mortals.
No branches or pull requests
