Nirman
second-state

second-state/MiniCPM-V-2_6-GGUF

No description available.

Model Documentation

header start --> 200823 -->
header end -->

MiniCPM-V-2_6-GGUF



Original Model



openbmb/MiniCPM-V-2_6

Run with LlamaEdge



  • LlamaEdge version: v0.14.17 and above


  • Prompt template


  • Prompt type: minicpmv


  • Prompt string


    • text
    <|system|>
    {system_message}<|end|>
    <|user|>
    {user_message_1}<|end|>
    <|assistant|>
    {assistant_message_1}<|end|>
    <|user|>
    {user_message_2}<|end|>
    <|assistant|>


    The {user_message_n} has the format: {image_base64_encoding_string}\n{user_question}.

  • Context size: 128000


  • Run as LlamaEdge service


    • bash
  wasmedge --dir .:. \
    --nn-preload default:GGML:AUTO:MiniCPM-V-2_6-Q5_K_M.gguf \
    llama-api-server.wasm \
    --prompt-template minicpmv \
    --ctx-size 128000 \
    --llava-mmproj mmproj-model-f16.gguf \
    --model-name minicpmv-26


    Quantized GGUF Models



    | Name | Quant method | Bits | Size | Use case | | ---
  • | ---- | ---- | ---- | ----- |
    • | MiniCPM-V-2_6-Q2_K.gguf | Q2_K | 2 | 3.01 GB| smallest, significant quality loss
  • not recommended for most purposes |
    • | MiniCPM-V-2_6-Q3_K_L.gguf | Q3_K_L | 3 | 4.09 GB| small, substantial quality loss | | MiniCPM-V-2_6-Q3_K_M.gguf | Q3_K_M | 3 | 3.81 GB| very small, high quality loss | | MiniCPM-V-2_6-Q3_K_S.gguf | Q3_K_S | 3 | 3.49 GB| very small, high quality loss | | MiniCPM-V-2_6-Q4_0.gguf | Q4_0 | 4 | 4.43 GB| legacy; small, very high quality loss
  • prefer using Q3_K_M |
    • | MiniCPM-V-2_6-Q4_K_M.gguf | Q4_K_M | 4 | 4.68 GB| medium, balanced quality
  • recommended |
    • | MiniCPM-V-2_6-Q4_K_S.gguf | Q4_K_S | 4 | 4.46 GB| small, greater quality loss | | MiniCPM-V-2_6-Q5_0.gguf | Q5_0 | 5 | 5.31 GB| legacy; medium, balanced quality
  • prefer using Q4_K_M |
    • | MiniCPM-V-2_6-Q5_K_M.gguf | Q5_K_M | 5 | 5.44 GB| large, very low quality loss
  • recommended |
    • | MiniCPM-V-2_6-Q5_K_S.gguf | Q5_K_S | 5 | 5.31 GB| large, low quality loss
  • recommended |
    • | MiniCPM-V-2_6-Q6_K.gguf | Q6_K | 6 | 6.25 GB| very large, extremely low quality loss | | MiniCPM-V-2_6-Q8_0.gguf | Q8_0 | 8 | 8.10 GB| very large, extremely low quality loss
  • not recommended |
    • | MiniCPM-V-2_6-f16.gguf | f16 | 16 | 15.2 GB| |

    *Quantized with llama.cpp b4120.*

    Files & Weights

    FilenameSizeAction
    MiniCPM-V-2_6-Q2_K.gguf 2.81 GB
    MiniCPM-V-2_6-Q3_K_L.gguf 3.81 GB
    MiniCPM-V-2_6-Q3_K_M.gguf 3.55 GB
    MiniCPM-V-2_6-Q3_K_S.gguf 3.25 GB
    MiniCPM-V-2_6-Q4_0.gguf 4.13 GB
    MiniCPM-V-2_6-Q4_K_M.gguf 4.36 GB
    MiniCPM-V-2_6-Q4_K_S.gguf 4.15 GB
    MiniCPM-V-2_6-Q5_0.gguf 4.95 GB
    MiniCPM-V-2_6-Q5_K_M.gguf 5.07 GB
    MiniCPM-V-2_6-Q5_K_S.gguf 4.95 GB
    MiniCPM-V-2_6-Q6_K.gguf 5.82 GB
    MiniCPM-V-2_6-Q8_0.gguf 7.54 GB
    MiniCPM-V-2_6-f16.gguf 14.19 GB
    mmproj-model-f16.gguf 0.97 GB