[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-2ac6f7d5-3787-49b3-8bf4-9cb95befa9da":3,"$fErkcCmIzJkMeyYNW9ayVGYk_BZV5i-2qMgqQbgeRcmI":42},{"id":4,"title":5,"description":6,"categoryId":7,"moduleId":8,"tags":9,"prompt":10,"icon":11,"source":12,"sourceUrl":13,"authorId":14,"authorName":15,"isPublic":16,"stars":17,"runs":18,"createdAt":19,"updatedAt":19,"module":20,"category":27,"packages":33},"2ac6f7d5-3787-49b3-8bf4-9cb95befa9da","embedding-strategies","向量搜索应用中嵌入模型选择与优化指南。","cat_coding_backend","mod_coding","sickn33,coding","---\nname: embedding-strategies\ndescription: \"Guide to selecting and optimizing embedding models for vector search applications.\"\nrisk: unknown\nsource: community\ndate_added: \"2026-02-27\"\n---\n\n# Embedding Strategies\n\nGuide to selecting and optimizing embedding models for vector search applications.\n\n## Do not use this skill when\n\n- The task is unrelated to embedding strategies\n- You need a different domain or tool outside this scope\n\n## Instructions\n\n- Clarify goals, constraints, and required inputs.\n- Apply relevant best practices and validate outcomes.\n- Provide actionable steps and verification.\n- If detailed examples are required, open `resources\u002Fimplementation-playbook.md`.\n\n## Use this skill when\n\n- Choosing embedding models for RAG\n- Optimizing chunking strategies\n- Fine-tuning embeddings for domains\n- Comparing embedding model performance\n- Reducing embedding dimensions\n- Handling multilingual content\n\n## Core Concepts\n\n### 1. Embedding Model Comparison\n\n| Model | Dimensions | Max Tokens | Best For |\n|-------|------------|------------|----------|\n| **text-embedding-3-large** | 3072 | 8191 | High accuracy |\n| **text-embedding-3-small** | 1536 | 8191 | Cost-effective |\n| **voyage-2** | 1024 | 4000 | Code, legal |\n| **bge-large-en-v1.5** | 1024 | 512 | Open source |\n| **all-MiniLM-L6-v2** | 384 | 256 | Fast, lightweight |\n| **multilingual-e5-large** | 1024 | 512 | Multi-language |\n\n### 2. Embedding Pipeline\n\n```\nDocument → Chunking → Preprocessing → Embedding Model → Vector\n                ↓\n        [Overlap, Size]  [Clean, Normalize]  [API\u002FLocal]\n```\n\n## Templates\n\n### Template 1: OpenAI Embeddings\n\n```python\nfrom openai import OpenAI\nfrom typing import List\nimport numpy as np\n\nclient = OpenAI()\n\ndef get_embeddings(\n    texts: List[str],\n    model: str = \"text-embedding-3-small\",\n    dimensions: int = None\n) -> List[List[float]]:\n    \"\"\"Get embeddings from OpenAI.\"\"\"\n    # Handle batching for large lists\n    batch_size = 100\n    all_embeddings = []\n\n    for i in range(0, len(texts), batch_size):\n        batch = texts[i:i + batch_size]\n\n        kwargs = {\"input\": batch, \"model\": model}\n        if dimensions:\n            kwargs[\"dimensions\"] = dimensions\n\n        response = client.embeddings.create(**kwargs)\n        embeddings = [item.embedding for item in response.data]\n        all_embeddings.extend(embeddings)\n\n    return all_embeddings\n\n\ndef get_embedding(text: str, **kwargs) -> List[float]:\n    \"\"\"Get single embedding.\"\"\"\n    return get_embeddings([text], **kwargs)[0]\n\n\n# Dimension reduction with OpenAI\ndef get_reduced_embedding(text: str, dimensions: int = 512) -> List[float]:\n    \"\"\"Get embedding with reduced dimensions (Matryoshka).\"\"\"\n    return get_embedding(\n        text,\n        model=\"text-embedding-3-small\",\n        dimensions=dimensions\n    )\n```\n\n### Template 2: Local Embeddings with Sentence Transformers\n\n```python\nfrom sentence_transformers import SentenceTransformer\nfrom typing import List, Optional\nimport numpy as np\n\nclass LocalEmbedder:\n    \"\"\"Local embedding with sentence-transformers.\"\"\"\n\n    def __init__(\n        self,\n        model_name: str = \"BAAI\u002Fbge-large-en-v1.5\",\n        device: str = \"cuda\"\n    ):\n        self.model = SentenceTransformer(model_name, device=device)\n\n    def embed(\n        self,\n        texts: List[str],\n        normalize: bool = True,\n        show_progress: bool = False\n    ) -> np.ndarray:\n        \"\"\"Embed texts with optional normalization.\"\"\"\n        embeddings = self.model.encode(\n            texts,\n            normalize_embeddings=normalize,\n            show_progress_bar=show_progress,\n            convert_to_numpy=True\n        )\n        return embeddings\n\n    def embed_query(self, query: str) -> np.ndarray:\n        \"\"\"Embed a query with BGE-style prefix.\"\"\"\n        # BGE models benefit from query prefix\n        if \"bge\" in self.model.get_sentence_embedding_dimension():\n            query = f\"Represent this sentence for searching relevant passages: {query}\"\n        return self.embed([query])[0]\n\n    def embed_documents(self, documents: List[str]) -> np.ndarray:\n        \"\"\"Embed documents for indexing.\"\"\"\n        return self.embed(documents)\n\n\n# E5 model with instructions\nclass E5Embedder:\n    def __init__(self, model_name: str = \"intfloat\u002Fmultilingual-e5-large\"):\n        self.model = SentenceTransformer(model_name)\n\n    def embed_query(self, query: str) -> np.ndarray:\n        return self.model.encode(f\"query: {query}\")\n\n    def embed_document(self, document: str) -> np.ndarray:\n        return self.model.encode(f\"passage: {document}\")\n```\n\n### Template 3: Chunking Strategies\n\n```python\nfrom typing import List, Tuple\nimport re\n\ndef chunk_by_tokens(\n    text: str,\n    chunk_size: int = 512,\n    chunk_overlap: int = 50,\n    tokenizer=None\n) -> List[str]:\n    \"\"\"Chunk text by token count.\"\"\"\n    import tiktoken\n    tokenizer = tokenizer or tiktoken.get_encoding(\"cl100k_base\")\n\n    tokens = tokenizer.encode(text)\n    chunks = []\n\n    start = 0\n    while start \u003C len(tokens):\n        end = start + chunk_size\n        chunk_tokens = tokens[start:end]\n        chunk_text = tokenizer.decode(chunk_tokens)\n        chunks.append(chunk_text)\n        start = end - chunk_overlap\n\n    return chunks\n\n\ndef chunk_by_sentences(\n    text: str,\n    max_chunk_size: int = 1000,\n    min_chunk_size: int = 100\n) -> List[str]:\n    \"\"\"Chunk text by sentences, respecting size limits.\"\"\"\n    import nltk\n    sentences = nltk.sent_tokenize(text)\n\n    chunks = []\n    current_chunk = []\n    current_size = 0\n\n    for sentence in sentences:\n        sentence_size = len(sentence)\n\n        if current_size + sentence_size > max_chunk_size and current_chunk:\n            chunks.append(\" \".join(current_chunk))\n            current_chunk = []\n            current_size = 0\n\n        current_chunk.append(sentence)\n        current_size += sentence_size\n\n    if current_chunk:\n        chunks.append(\" \".join(current_chunk))\n\n    return chunks\n\n\ndef chunk_by_semantic_sections(\n    text: str,\n    headers_pattern: str = r'^#{1,3}\\s+.+$'\n) -> List[Tuple[str, str]]:\n    \"\"\"Chunk markdown by headers, preserving hierarchy.\"\"\"\n    lines = text.split('\\n')\n    chunks = []\n    current_header = \"\"\n    current_content = []\n\n    for line in lines:\n        if re.match(headers_pattern, line, re.MULTILINE):\n            if current_content:\n                chunks.append((current_header, '\\n'.join(current_content)))\n            current_header = line\n            current_content = []\n        else:\n            current_content.append(line)\n\n    if current_content:\n        chunks.append((current_header, '\\n'.join(current_content)))\n\n    return chunks\n\n\ndef recursive_character_splitter(\n    text: str,\n    chunk_size: int = 1000,\n    chunk_overlap: int = 200,\n    separators: List[str] = None\n) -> List[str]:\n    \"\"\"LangChain-style recursive splitter.\"\"\"\n    separators = separators or [\"\\n\\n\", \"\\n\", \". \", \" \", \"\"]\n\n    def split_text(text: str, separators: List[str]) -> List[str]:\n        if not text:\n            return []\n\n        separator = separators[0]\n        remaining_separators = separators[1:]\n\n        if separator == \"\":\n            # Character-level split\n            return [text[i:i+chunk_size] for i in range(0, len(text), chunk_size - chunk_overlap)]\n\n        splits = text.split(separator)\n        chunks = []\n        current_chunk = []\n        current_length = 0\n\n        for split in splits:\n            split_length = len(split) + len(separator)\n\n            if current_length + split_length > chunk_size and current_chunk:\n                chunk_text = separator.join(current_chunk)\n\n                # Recursively split if still too large\n                if len(chunk_text) > chunk_size and remaining_separators:\n                    chunks.extend(split_text(chunk_text, remaining_separators))\n                else:\n                    chunks.append(chunk_text)\n\n                # Start new chunk with overlap\n                overlap_splits = []\n                overlap_length = 0\n                for s in reversed(current_chunk):\n                    if overlap_length + len(s) \u003C= chunk_overlap:\n                        overlap_splits.insert(0, s)\n                        overlap_length += len(s)\n                    else:\n                        break\n                current_chunk = overlap_splits\n                current_length = overlap_length\n\n            current_chunk.append(split)\n            current_length += split_length\n\n        if current_chunk:\n            chunks.append(separator.join(current_chunk))\n\n        return chunks\n\n    return split_text(text, separators)\n```\n\n### Template 4: Domain-Specific Embedding Pipeline\n\n```python\nclass DomainEmbeddingPipeline:\n    \"\"\"Pipeline for domain-specific embeddings.\"\"\"\n\n    def __init__(\n        self,\n        embedding_model: str = \"text-embedding-3-small\",\n        chunk_size: int = 512,\n        chunk_overlap: int = 50,\n        preprocessing_fn=None\n    ):\n        self.embedding_model = embedding_model\n        self.chunk_size = chunk_size\n        self.chunk_overlap = chunk_overlap\n        self.preprocess = preprocessing_fn or self._default_preprocess\n\n    def _default_preprocess(self, text: str) -> str:\n        \"\"\"Default preprocessing.\"\"\"\n        # Remove excessive whitespace\n        text = re.sub(r'\\s+', ' ', text)\n        # Remove special characters\n        text = re.sub(r'[^\\w\\s.,!?-]', '', text)\n        return text.strip()\n\n    async def process_documents(\n        self,\n        documents: List[dict],\n        id_field: str = \"id\",\n        content_field: str = \"content\",\n        metadata_fields: List[str] = None\n    ) -> List[dict]:\n        \"\"\"Process documents for vector storage.\"\"\"\n        processed = []\n\n        for doc in documents:\n            content = doc[content_field]\n            doc_id = doc[id_field]\n\n            # Preprocess\n            cleaned = self.preprocess(content)\n\n            # Chunk\n            chunks = chunk_by_tokens(\n                cleaned,\n                self.chunk_size,\n                self.chunk_overlap\n            )\n\n            # Create embeddings\n            embeddings = get_embeddings(chunks, self.embedding_model)\n\n            # Create records\n            for i, (chunk, embedding) in enumerate(zip(chunks, embeddings)):\n                record = {\n                    \"id\": f\"{doc_id}_chunk_{i}\",\n                    \"document_id\": doc_id,\n                    \"chunk_index\": i,\n                    \"text\": chunk,\n                    \"embedding\": embedding\n                }\n\n                # Add metadata\n                if metadata_fields:\n                    for field in metadata_fields:\n                        if field in doc:\n                            record[field] = doc[field]\n\n                processed.append(record)\n\n        return processed\n\n\n# Code-specific pipeline\nclass CodeEmbeddingPipeline:\n    \"\"\"Specialized pipeline for code embeddings.\"\"\"\n\n    def __init__(self, model: str = \"voyage-code-2\"):\n        self.model = model\n\n    def chunk_code(self, code: str, language: str) -> List[dict]:\n        \"\"\"Chunk code by functions\u002Fclasses.\"\"\"\n        import tree_sitter\n\n        # Parse with tree-sitter\n        # Extract functions, classes, methods\n        # Return chunks with context\n        pass\n\n    def embed_with_context(self, chunk: str, context: str) -> List[float]:\n        \"\"\"Embed code with surrounding context.\"\"\"\n        combined = f\"Context: {context}\\n\\nCode:\\n{chunk}\"\n        return get_embedding(combined, model=self.model)\n```\n\n### Template 5: Embedding Quality Evaluation\n\n```python\nimport numpy as np\nfrom typing import List, Tuple\n\ndef evaluate_retrieval_quality(\n    queries: List[str],\n    relevant_docs: List[List[str]],  # List of relevant doc IDs per query\n    retrieved_docs: List[List[str]],  # List of retrieved doc IDs per query\n    k: int = 10\n) -> dict:\n    \"\"\"Evaluate embedding quality for retrieval.\"\"\"\n\n    def precision_at_k(relevant: set, retrieved: List[str], k: int) -> float:\n        retrieved_k = retrieved[:k]\n        relevant_retrieved = len(set(retrieved_k) & relevant)\n        return relevant_retrieved \u002F k\n\n    def recall_at_k(relevant: set, retrieved: List[str], k: int) -> float:\n        retrieved_k = retrieved[:k]\n        relevant_retrieved = len(set(retrieved_k) & relevant)\n        return relevant_retrieved \u002F len(relevant) if relevant else 0\n\n    def mrr(relevant: set, retrieved: List[str]) -> float:\n        for i, doc in enumerate(retrieved):\n            if doc in relevant:\n                return 1 \u002F (i + 1)\n        return 0\n\n    def ndcg_at_k(relevant: set, retrieved: List[str], k: int) -> float:\n        dcg = sum(\n            1 \u002F np.log2(i + 2) if doc in relevant else 0\n            for i, doc in enumerate(retrieved[:k])\n        )\n        ideal_dcg = sum(1 \u002F np.log2(i + 2) for i in range(min(len(relevant), k)))\n        return dcg \u002F ideal_dcg if ideal_dcg > 0 else 0\n\n    metrics = {\n        f\"precision@{k}\": [],\n        f\"recall@{k}\": [],\n        \"mrr\": [],\n        f\"ndcg@{k}\": []\n    }\n\n    for relevant, retrieved in zip(relevant_docs, retrieved_docs):\n        relevant_set = set(relevant)\n        metrics[f\"precision@{k}\"].append(precision_at_k(relevant_set, retrieved, k))\n        metrics[f\"recall@{k}\"].append(recall_at_k(relevant_set, retrieved, k))\n        metrics[\"mrr\"].append(mrr(relevant_set, retrieved))\n        metrics[f\"ndcg@{k}\"].append(ndcg_at_k(relevant_set, retrieved, k))\n\n    return {name: np.mean(values) for name, values in metrics.items()}\n\n\ndef compute_embedding_similarity(\n    embeddings1: np.ndarray,\n    embeddings2: np.ndarray,\n    metric: str = \"cosine\"\n) -> np.ndarray:\n    \"\"\"Compute similarity matrix between embedding sets.\"\"\"\n    if metric == \"cosine\":\n        # Normalize\n        norm1 = embeddings1 \u002F np.linalg.norm(embeddings1, axis=1, keepdims=True)\n        norm2 = embeddings2 \u002F np.linalg.norm(embeddings2, axis=1, keepdims=True)\n        return norm1 @ norm2.T\n    elif metric == \"euclidean\":\n        from scipy.spatial.distance import cdist\n        return -cdist(embeddings1, embeddings2, metric='euclidean')\n    elif metric == \"dot\":\n        return embeddings1 @ embeddings2.T\n```\n\n## Best Practices\n\n### Do's\n- **Match model to use case** - Code vs prose vs multilingual\n- **Chunk thoughtfully** - Preserve semantic boundaries\n- **Normalize embeddings** - For cosine similarity\n- **Batch requests** - More efficient than one-by-one\n- **Cache embeddings** - Avoid recomputing\n\n### Don'ts\n- **Don't ignore token limits** - Truncation loses info\n- **Don't mix embedding models** - Incompatible spaces\n- **Don't skip preprocessing** - Garbage in, garbage out\n- **Don't over-chunk** - Lose context\n\n## Resources\n\n- [OpenAI Embeddings](https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Fembeddings)\n- [Sentence Transformers](https:\u002F\u002Fwww.sbert.net\u002F)\n- [MTEB Benchmark](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fmteb\u002Fleaderboard)\n\n## Limitations\n- Use this skill only when the task clearly matches the scope described above.\n- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.\n- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.\n","","imported","https:\u002F\u002Fgithub.com\u002Fsickn33\u002Fantigravity-awesome-skills","user_system_seed","SkillOPIC",true,154,1750,"2026-05-16 13:16:34",{"id":8,"name":21,"slug":22,"icon":23,"description":24,"sort":25,"createdAt":26},"编程开发","coding","mdi-code-braces","代码生成、调试、审查，提升开发效率",2,"2026-05-16 12:53:40",{"id":7,"name":28,"slug":29,"icon":30,"description":31,"moduleId":8,"sort":25,"skillCount":32,"createdAt":26},"后端开发","backend","mdi-server","API、数据库、服务端架构",296,[34],{"id":35,"skillId":4,"version":36,"fileName":37,"fileSize":38,"filePath":39,"fileHash":40,"manifest":41,"createdAt":19},"97899365-4f99-478f-a22b-621ce9a011c1","1.0.0","embedding-strategies.zip",4809,"uploads\u002Fskills\u002F2ac6f7d5-3787-49b3-8bf4-9cb95befa9da\u002Fembedding-strategies.zip","a7737d13709e67d452a1a7a20ca04ca2357f49087ab03741171a45fe063eda5c","[{\"path\":\"SKILL.md\",\"isDirectory\":false,\"size\":15201}]",{"code":43,"message":44,"data":45},200,"success",{"items":46,"stats":47,"page":50},[],{"averageRating":48,"totalRatings":48,"ratingCounts":49},0,[48,48,48,48,48],{"limit":51,"offset":48,"hasMore":52,"nextOffset":51,"ratedOnly":16},15,false]