PrivateGPT vs LocalGPT: Which One Actually Works?

# In settings.yaml:
llama:
  llama_cpp:
    model_path: models/mistral-7b-instruct-v0.2.Q4_K_M.gguf
    n_gpu_layers: 20  # Not 35! Let CPU handle some layers
    n_batch: 1024  # Increase from 512 for better GPU utilization
    n_ctx: 4096  # Reduce from 8192
    f16_kv: true  # Use FP16 for KV cache (saves VRAM)
    use_mmap: true  # Memory-map model file
    use_mlock: false  # Disable on GPU systems
    numa: false  # Disable NUMA for single-GPU systems

# Alternative: Use smaller model
llama:
  llama_cpp:
    model_path: models/mistral-7b-instruct-v0.2.Q3_K_M.gguf  # More aggressive quantization
    n_gpu_layers: 30  # Can fit more layers with Q3

# In settings.yaml:
chunks:
  size: 256  # Reduce from 512 (smaller chunks process faster)
  overlap: 50

# Add custom chunking for large docs in code:
# Create ingest_custom.py

from private_gpt.components.ingest import chunking
from langchain.text_splitter import RecursiveCharacterTextSplitter

def create_optimized_chunker():
    return RecursiveCharacterTextSplitter(
        chunk_size=256,
        chunk_overlap=50,
        length_function=len,
        separators=[
            "\n\n",  # Paragraphs first
            "\n",    # Lines
            " ",     # Words
            ""       # Characters (last resort)
        ],
        keep_separator=False  # Don't include separator in chunk
    )

# Use batch ingestion to prevent memory buildup
PGPT_PROFILES=chroma pgpt ingest documents/*.pdf --batch-size 10

# Use code-specific embedding
embedding:
  mode: huggingface
  huggingface:
    # Better for code/technical content
    model_name: BAAI/bge-base-en-v1.5
    # Or for mixed code+text:
    # model_name: intfloat/e5-large-v2

# Enable hybrid search (keyword + semantic)
vectorstore:
  database: chroma
  chroma:
    collection_name: my_documents
    # Enable BM25 hybrid search
    hybrid_search: true
    bm25_weight: 0.3  # 30% keyword, 70% semantic

# Adjust retrieval
retrieval:
  mode: chroma
  k: 10  # Retrieve more candidates
  score_threshold: 0.2  # Lower threshold
  rerank: true  # Enable reranking
  rerank_model: cross-encoder  # Better final ranking

# Option 1: Use HuggingFace for both (recommended)
embedding:
  mode: huggingface
  huggingface:
    model_name: BAAI/bge-small-en-v1.5
    # Dimension: 384

llm:
  mode: llama-cpp
  llama_cpp:
    # Use LLM only for generation, not embedding
    embedding_mode: false
    n_gpu_layers: -1

# Option 2: Use llama-cpp for both
embedding:
  mode: llama-cpp
  llama_cpp:
    model_path: models/mistral-7b-instruct-v0.2.Q4_K_M.gguf
    # Dimension: 4096 (Mistral)

llm:
  mode: llama-cpp

# CRITICAL: Must delete old ChromaDB when changing embedding model
# Different dimensions = incompatible vectors
rm -rf chroma_db  # Delete and re-ingest

# Create custom prompt template
# In prompts/custom_system_prompt.txt

You are a helpful assistant that answers questions based on the provided context.
If the answer cannot be found in the context, say "I don't have enough information to answer this."

Context information is below.
---------------------
{context}
---------------------

Given the context information and not prior knowledge, answer the query.
Query: {query}

Answer:

# In settings.yaml, point to custom prompt:
prompt:
  mode: custom
  template_file: prompts/custom_system_prompt.txt

# Or use built-in with context injection
prompt:
  mode: default
  # Ensure context is actually passed
  include_context: true
  max_context_tokens: 2048  # Increase from default 1024