Installation in distrobox #
1distrobox create -n gpt4all \
2 -i ubuntu:latest \
3 --nvidia \
4 -p \
5 --additional-flags "--privileged" \
6 --yes \
7 --home ~/.distrobox/gpt4all
1mkdir -p ~/.distrobox/gpt4all/Desktop # sic
2distrobox enter gpt4all
In the distrobox:
1sudo apt-get install -y libxkbcommon-x11-0 libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-xinerama0 libxcb-xfixes0 libxrender1 libxcb-shape0 libxcb-shm0 libxcb-util1 libxcb-xkb1 libxkbcommon-x11-0 libxkbcommon0 libxcb1 libxext6 libx11-6 libfuse2 libgl-dev libglu1-mesa-dev
2curl -L https://gpt4all.io/installers/gpt4all-installer-linux.run -o gpt4all-installer-linux.run
3chmod +x gpt4all-installer-linux.run
4./gpt4all-installer-linux.run
5./gpt4all/bin/chat
CUDA #
It turns out with CUDA and my 3060 TI with 8GB it's reasonably snappy, but I can't do 16GB models. I can run 16GB models on the CPU only though. I need to set some environment variables for this:
1CUDA_VISIBLE_DEVICES=-1 NVIDIA_VISIBLE_DEVICES=none ./gpt4all/bin/chat
But with this the action slows down. It does not become unusably slow, but very much slower.
Can I throw money at the problem ? #
NVidia Graphics cards with 16GB, there are the 4090, the 4080 Super and the 4070 TI SUPER. Anything less is not 16GB...
Looking at a Stable Diffusion benchmark which would seem relevant:
| Card | Price EUR | SD Score | |
|---|---|---|---|
| 4090 | 2000 | 75 | |
| 4080 | 1000 | 50 | not the SUPER though |
| 4070 TI | 850 | 40 | not the SUPER though |
I would expect the SUPER to be maybe 5% faster.
Power supply #
It seems I can get by with the existing RM750e power supply but need an additional cable Corsair Premium Sleeved PCIe 5.0 12VHPWR PSU Adapterkabel.
Size #
The 4090 and 4080 Super don't fit in my case. There 4070 TI SUPERs would fit though MSI GeForce RTX 4070 Ti SUPER 16G VENTUS 2X OC nicely.
AI Disappointment #
I wanted the AI to describe the contents of the picture. I used a few of the available models. They are all 8GB with the execption of Hermes which is 16GB.
All of them I asked: What do you see in this picture ? https://mediadb.kicker.de/2024/fussball/spieler/xl/92331_8_20230809967.jpg and
What is the predominant color in that picture ?
This is the picture in question:
| AI | Answer |
|---|---|
| LLama 3 Instruct | I'm just an AI, I don't have the capability to visually perceive images or access external links. However, if you'd like to describe what's in the picture, I can try to help you analyze it! |
| GPT4All Falcon | I see a football player standing on the field, surrounded by teammates and fans. The player is wearing a red jersey with white sleeves and black shorts. They have their arms crossed in front of them and are looking straight ahead.
|
| Nous Hermes 2 Mistral | In the provided image, we can see a football (soccer) player in action. The player is wearing a green and white jersey with the number 5 on it. He appears to be from the team SV Wehen Wiesbaden as indicated by their logo on the front of the shirt. The player's name "Kalabis" can also be seen on the back of his jersey, which suggests that this is a photo of Florian Kalabis, a professional footballer who plays for SV Wehen Wiesbaden in Germany. He seems to be attempting to head the ball during a match or training session, as he has raised his head and positioned himself above the ball. The background shows other players on the field, and there is a referee nearby.
|
| Hermes 16GB | In the picture, I see a soccer player wearing a red and white jersey with the number 9 on it. He appears to be in mid-action, running towards the goalpost while holding the ball.
|
So I need to first identify some use case where 16GB is actually more valuable to me than 8GB.