moritz/infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Adding current state that was previously created to make the repo cleaner

Browse Source
This commit is contained in:
Moritz Graf
2025-04-21 16:11:22 +02:00
parent 201aefe8da
commit 1357498b2e
8 changed files with 321 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
k8s/llm_hosting.md
+24
View File
@@ -0,0 +1,24 @@
# llm-hosting
This is an extended article to not have to write everything in the main readme. This chapter takes care of hosting llm models on the server.
## deploy
```sh
kubectl apply -f llm/llama_cpp_hosting.yaml
```
## development
```sh
```
## links
Two examples of model files that are currently tried out:
* [https://huggingface.co/MaziyarPanahi/Meta-Llama-3-70B-Instruct-GGUF/resolve/main/Meta-Llama-3-70B-Instruct.IQ1_S.gguf?download=true](https://huggingface.co/MaziyarPanahi/Meta-Llama-3-70B-Instruct-GGUF/resolve/main/Meta-Llama-3-70B-Instruct.IQ1_S.gguf?download=true)
* From [this page](https://huggingface.co/MaziyarPanahi/Meta-Llama-3-70B-Instruct-GGUF/tree/main).
* [https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF/resolve/main/Meta-Llama-3-8B-Instruct.Q8_0.gguf?download=true](https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF/resolve/main/Meta-Llama-3-8B-Instruct.Q8_0.gguf?download=true)
* From [this page](https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF/tree/main).