Installing and Running Jina Code Embeddings 1.5B Local

Jina Code Embeddings 1.5B represents a breakthrough in code understanding expertise. This mannequin transforms how builders search by codebases, retrieve related code snippets, and construct clever developer instruments. In contrast to conventional textual content embeddings that battle with programming syntax and semantics, this specialised mannequin understands code throughout greater than 15 programming languages.

What Makes This Mannequin Particular?

The mannequin builds on the Qwen2.5-Coder-1.5B basis, which Jina AI has fine-tuned particularly for software program growth workflows. Consider it as a translator that converts each pure language questions and code snippets into mathematical representations (vectors) that seize their that means. Whenever you ask, “How do I learn a CSV file in Python?” the mannequin can discover related code even when the documentation makes use of totally different phrases like “parse” or “load” as an alternative of “learn.”

Core Capabilities

The mannequin excels at 5 key duties:

1. Textual content-to-Code Retrieval: You describe what you need in plain English, and the mannequin finds matching code. For instance, looking for “perform to calculate factorial recursively” will find applicable implementations even when they use totally different variable names or barely totally different logic.

2. Code-to-Code Similarity: Examine two code snippets to see in the event that they do the identical factor, no matter styling variations. This helps determine duplicate code, discover comparable implementations, or counsel refactoring alternatives.

3. Code-to-Documentation: Generate or discover pure language explanations for code blocks. Whenever you encounter an unfamiliar perform, the mannequin helps you perceive what it does with out studying each line.

4. Code Completion: Given a partial code snippet, the mannequin predicts what ought to come subsequent. This powers clever autocomplete options in fashionable code editors.

5. Technical Query Answering: Reply programming questions by matching them with related documentation, Stack Overflow solutions, or code examples out of your codebase.

Versatile Vector Dimensions

Some of the modern options is the Matryoshka embedding assist. The mannequin produces 1536-dimensional vectors by default, however you may truncate these to 128, 256, 512, or 1024 dimensions with minimal accuracy loss. This flexibility issues tremendously for manufacturing methods.

Think about a situation the place you are constructing a code search engine for a big firm. Storing 1536-dimensional vectors for thousands and thousands of code snippets requires vital reminiscence and slows down searches. By truncating to 256 dimensions, you scale back storage by 83% and velocity up similarity calculations by roughly 6x, whereas retaining a lot of the search high quality. You regulate this tradeoff based mostly in your particular wants.

Technical Structure Particulars

The mannequin makes use of a number of superior strategies to attain excessive efficiency:

1. FlashAttention-2 Optimization: Conventional consideration mechanisms in transformer fashions devour quadratic reminiscence relative to sequence size. FlashAttention-2 reorganizes computations to make use of the GPU’s reminiscence hierarchy extra effectively, enabling longer sequences and sooner inference. Whenever you course of a ten,000-token code file, FlashAttention-2 could be 3-5x sooner than customary consideration.

2. Final-Token Pooling: To transform a sequence of token embeddings right into a single vector, the mannequin makes use of the final token’s illustration. The tokenizer pads sequences on the left (not like most language fashions that pad on the best), guaranteeing the final token all the time accommodates significant details about the complete enter.

3. Prolonged Context Window: With assist for 32,768 tokens (roughly 25,000 phrases), you may embed total supply recordsdata, API documentation pages, and even small codebases in a single operation. This eliminates the necessity to chunk giant paperwork and lose context throughout boundaries.

{Hardware} Necessities and Suggestions

Selecting the best {hardware} is dependent upon your use case. Let’s break down totally different situations:

Entry-Degree Setup (8-16GB VRAM)

Appropriate For: Particular person builders, small initiatives, experimentation

Should you’re simply testing the mannequin or constructing a private code search software, an RTX 3060 with 12GB or a cloud T4 occasion works high quality. You may course of one or two queries at a time with sequences as much as 8,000 tokens. This setup handles typical growth workflows like looking out your individual initiatives or constructing a small RAG (Retrieval-Augmented Technology) system.

Limitations: Processing giant batches will probably be gradual. If it is advisable embed 1000’s of paperwork, count on it to take hours quite than minutes.

Commonplace Manufacturing Setup (16-24GB VRAM)

Appropriate For: Manufacturing companies, medium-sized groups, API endpoints

An RTX 4090 or cloud L4 occasion with 24GB VRAM gives the candy spot for many purposes. You’ll be able to batch 8-16 queries collectively and deal with sequences as much as 16,000 tokens effectively. This configuration helps a small group’s code search wants or powers a moderate-traffic API endpoint.

Efficiency: Anticipate to course of a whole bunch of embeddings per minute, making it viable for real-time search as builders sort queries.

Skilled Setup (40-48GB VRAM)

Appropriate For: Giant-scale retrieval methods, high-concurrency companies

With an A100 40GB or L40S 48GB GPU, you enter enterprise territory. Batch sizes of 32-64 queries with full 32k token sequences develop into sensible. This setup serves a number of groups concurrently or indexes large codebases (thousands and thousands of recordsdata) inside affordable timeframes.

Use Instances: Firm-wide code search, large-scale code evaluation, multi-tenant SaaS merchandise.

Enterprise Setup (80GB+ VRAM)

Appropriate For: Analysis establishments, very giant organizations, specialised purposes

A100 80GB or H100 GPUs deal with excessive workloads. You’ll be able to course of very lengthy paperwork (total modules), keep a number of mannequin cases for redundancy, or serve a whole bunch of concurrent customers. Most organizations will not want this tier until dealing with distinctive scale.

Detailed Set up Course of Utilizing Spheron Community

We’ll stroll by organising the mannequin on a GPU-powered digital machine utilizing Spheron’s decentralized compute platform. Spheron affords inexpensive GPU sources, powered by each information center-grade infrastructure and neighborhood nodes, offering flexibility in value and efficiency.

Step 1: Entry Spheron Console and Add Credit

Head over to console.spheron.network and log in to your account. If you do not have an account but, create one by signing up together with your E-mail/Google/Discord/GitHub.

As soon as logged in, navigate to the Deposit part. You may see two cost choices:

SPON Token: That is the native token of Spheron Community. Whenever you deposit with SPON, you unlock the complete energy of the ecosystem. SPON credit can be utilized on each:

• Neighborhood GPUs: Decrease-cost GPU sources powered by neighborhood Fizz Nodes (private machines and residential setups)

• Safe GPUs: Information center-grade GPU suppliers providing enterprise reliability

USD Credit: With USD deposits, you may deploy solely on Safe GPUs. Neighborhood GPUs are usually not obtainable with USD deposits.

For working Jina Code Embeddings 1.5B, we advocate beginning with Safe GPUs to make sure constant efficiency. Add adequate credit to your account based mostly in your anticipated utilization.

Step 2: Navigate to GPU Market

After including credit, click on on Market. Right here you will see two essential classes:

Safe GPUs: These run on information center-grade suppliers with enterprise SLAs, excessive uptime ensures, and constant efficiency. Ideally suited for manufacturing workloads and purposes that require reliability.

Neighborhood GPUs: These run on neighborhood Fizz Nodes, primarily private machines contributed by neighborhood members. They’re considerably cheaper than Safe GPUs however could have variable availability and efficiency.

For this tutorial, we’ll use Safe GPUs to make sure easy set up and optimum efficiency.

Step 3: Search and Choose Your GPU

You’ll be able to seek for GPUs by:

• Area: Discover GPUs geographically near your customers

• Handle: Search by particular supplier addresses

• Title: Filter by GPU mannequin (RTX 4090, A100, and many others.)

For this demo, we’ll choose a Safe RTX 4090 (or A6000 GPU), which affords

GPU VRAM: 24 Gi

Storage: 404 GB | CPU Cores: 14 | RAM: 36 GB

And glorious efficiency for working Jina Code Embeddings 1.5B. The 4090 gives the proper steadiness of value and functionality for each testing and average manufacturing workloads.

Click on Hire Now in your chosen GPU to proceed to configuration.

Step 4: Choose Customized Picture Template

After clicking Hire Now, you will see the Hire Affirmation dialog. This display screen exhibits all of the configuration choices to your GPU deployment. Let’s configure every part. In contrast to pre-built software templates, working Jina Code Embeddings 1.5B requires a custom-made setting for growth capabilities. Choose the configuration as proven within the picture beneath and click on “Affirm” to deploy.

1. GPU Sort: The display screen shows your chosen GPU (RTX 4090 within the picture) with specs: Storage, CPU Cores, RAM.

2. GPU Depend: Use the + and – buttons to regulate the variety of GPUs. For this tutorial, preserve it at 1 GPU for value effectivity.

3. Choose Template: Click on the dropdown that exhibits “Ubuntu 24” and search for template choices. For working Jina Code Embeddings 1.5B, we want an Ubuntu-based template with SSH enabled. You may discover the template exhibits an SSH-enabled badge, which is crucial for accessing your occasion through terminal. Choose: Ubuntu 24 or Ubuntu 22 (each work completely)

4. Period: Set how lengthy you need to hire the GPU. The dropdown exhibits choices like: 1hr (good for fast testing), 8hr, 24hr, or longer for manufacturing use. For this tutorial, choose 1 hour initially. You’ll be able to all the time prolong the length later if wanted.

5. Choose SSH Key: Click on the dropdown to decide on your SSH key for safe authentication. If you have not added an SSH key but, you will see a message to create one.

6. Expose Ports: This part means that you can expose particular ports out of your deployment. For primary command-line entry, you may go away this empty. Should you plan to run net companies or Jupyter notebooks, you may add ports.

7. Supplier Particulars: The display screen exhibits supplier info:

This exhibits which decentralized supplier will host your GPU occasion.

8. Scroll all the way down to the Select Cost part. Choose your most popular cost choice:

   • USD – Pay with conventional forex (bank card or different USD cost strategies)

   • SPON: Pay with Spheron’s native token for potential reductions and entry to each Neighborhood and Safe GPUs

The dropdown exhibits “USD” within the instance, however you may change to SPON you probably have tokens deposited.

Step 5: Verify the “Deployment in Progress“

Subsequent, you’ll see a reside standing window exhibiting each step of what is taking place, like: Validating configuration, Checking steadiness, Creating order, Ready for bids, Accepting a bid, Sending manifest, and eventually, Lease Created Efficiently. As soon as that is full, your Ubuntu server is reside!

Deployment usually completes in underneath 60 seconds. When you see “Lease Created Efficiently,” your Ubuntu server with GPU entry is reside and able to use!

Step 6: Entry Your Deployment

As soon as deployment completes, navigate to the Overview tab in your Spheron console. You may see your deployment listed with:

• Standing: Working

• Supplier particulars: GPU location and specs

• Connection info: SSH entry particulars

• Port mappings: Any uncovered companies

Step 7: Join through SSH

Click on the SSH tab, and you will note the steps on how you can join your terminal through SSH to your deployment particulars. It would look one thing just like the picture beneath, observe it:

ssh -i <path-to-private-key> -p <port> root@<deployment-url>

Open your terminal and paste this command. Upon your first connection, you will see a safety immediate requesting that you simply confirm the server’s fingerprint. Sort “sure” to proceed. You are now related to your GPU-powered digital machine on the Spheron decentralized community.

Software program Surroundings Setup

Now we’ll construct a Python setting particularly for working Jina Code Embeddings.

Step 8: Replace System and Set up Curl

First, replace your system packages and set up curl, which we’ll use for downloading dependencies:

apt replace && apt set up -y curl

Confirm curl set up:

curl --version

It’s best to see output exhibiting curl model info, confirming it is correctly put in.

Step 9: Set up Python and Pip

Set up Python’s bundle supervisor (pip):

curl -O https://bootstrap.pypa.io/get-pip.py
apt replace && apt set up -y python3-pip

Confirm pip and Python set up:

pip3 --version
python3 --version

It’s best to see output like: pip 24.0 from /usr/lib/python3/dist-packages/pip (python 3.12) and Python 3.12.3

Step 10: Set up Python Digital Surroundings Instruments

Set up the digital setting module for Python 3.12:

apt set up -y python3.12-venv

This bundle means that you can create remoted Python environments, stopping dependency conflicts between totally different initiatives.

Step 11: Create and Activate Digital Surroundings

Create a digital setting named “Jina” and activate the digital setting:

python3.12 -m venv Jina
supply Jina/bin/activate

After activation, your command immediate modifications to point out (Jina) firstly, indicating you are working contained in the digital setting. Any packages you put in now will probably be remoted from the system Python set up.

Step 12: Set up Core Python Dependencies

Set up the basic packages for working the mannequin:

python -m pip set up "sentence-transformers>=5.0.0" "torch>=2.7.1"

This command installs:

Sentence-Transformers (≥5.0.0): A high-level library that simplifies loading and utilizing embedding fashions. It handles tokenization, batching, and machine administration and gives handy encoding strategies.

PyTorch (≥2.7.1): The underlying deep studying framework. This model contains optimizations for contemporary CUDA variations and improved reminiscence effectivity for working giant fashions.

The set up takes 5-10 minutes because it downloads PyTorch (~2GB) and sentence-transformers with their dependencies.

Set up the wheel bundle for constructing Python packages:

pip set up wheel

Step 13: Set up CUDA Toolkit

Set up NVIDIA CUDA toolkit for GPU acceleration:

apt set up -y nvidia-cuda-toolkit

This installs the whole CUDA growth setting, together with:

After set up, create symbolic hyperlinks for CUDA libraries:

ln -s /usr/lib/x86_64-linux-gnu/libcuda* /usr/lib/cuda/lib64/ 2>/dev/null

This command creates symbolic hyperlinks from the system CUDA libraries to the usual CUDA library path. The 2>/dev/null Suppresses any errors if some hyperlinks exist already. This step ensures that Python packages can discover the CUDA libraries when compiling GPU-accelerated code.

Step 14: Set up FlashAttention-2

FlashAttention-2 is an optimized consideration mechanism that considerably hastens mannequin inference. Set up it with:

python -m pip set up flash-attn --no-build-isolation

Essential Notes:

• This set up compiles CUDA kernels from supply and takes a couple of minutes if the necessities are usually not already happy

• The --no-build-isolation The flag permits the installer to make use of your setting’s packages

• You may see compilation progress messages; that is regular

• The method makes use of vital disk house quickly

If this step fails with CUDA-related errors, don’t be concerned, you may nonetheless run the mannequin with customary consideration (barely slower however absolutely useful). The mannequin will mechanically fall again to SDPA (Scaled Dot Product Consideration) if FlashAttention is not obtainable.

Step 15: Set up Git

Set up Git for model management and cloning repositories:

apt replace && apt set up -y git

Git is helpful if it is advisable clone code repositories or handle your individual scripts.

Step 16: Authenticate with Hugging Face

The Jina Code Embeddings mannequin is hosted on Hugging Face Hub. Authenticate to it:

hf auth login

When prompted, paste your Hugging Face entry token. If you do not have a token but:

1. Go to https://huggingface.co/settings/tokens

2. Click on “New token”

3. Choose “Learn” permissions (adequate for downloading fashions)

4. Title it one thing memorable like “jina-embeddings”

5. Copy the token and paste it when the terminal prompts you

After profitable authentication, you will see a affirmation message.

Step 17: Set up Speed up

Set up the Speed up library for optimized mannequin loading and inference:

pip set up speed up

Speed up is a Hugging Face library that simplifies:

• Distributed coaching and inference

• Combined-precision computation (utilizing bfloat16 for sooner processing)

• Multi-GPU administration

• Machine placement optimization

Step 18: Connecting a Code Editor

Whilst you can write Python scripts immediately within the terminal utilizing editors like nano or vim, connecting a contemporary code editor dramatically improves productiveness. We advocate VS Code, Cursor, or any IDE supporting SSH distant growth.

This workflow feels precisely like native growth, however executes every little thing in your highly effective GPU digital machine.

Working Primary Examples

Let’s begin with a easy script that demonstrates core performance.

Script 1: Easy Textual content-to-Code Retrieval

Create a file named test_jina.py:

import torch
from sentence_transformers import SentenceTransformer


mannequin = SentenceTransformer(
    "jinaai/jina-code-embeddings-1.5b",
    model_kwargs={
        "torch_dtype": torch.bfloat16,
        "attn_implementation": "flash_attention_2",
        "device_map": "cuda"
    },
    tokenizer_kwargs={"padding_side": "left"},
)


queries = [
    "print hello world in python",
    "initialize array of 5 zeros in c++"
]
paperwork = [
    "print('Hello World!')",
    "int arr[5] = {0, 0, 0, 0, 0};"
]


query_embeddings = mannequin.encode(queries, prompt_name="nl2code_query")
document_embeddings = mannequin.encode(paperwork, prompt_name="nl2code_document")


similarity = mannequin.similarity(query_embeddings, document_embeddings)
print(similarity)

How It Works:

The script hundreds the mannequin with three essential configurations:

bfloat16 precision: Makes use of 16-bit mind floating level format as an alternative of 32-bit floats. This halves reminiscence utilization and hastens computation with minimal affect on accuracy. Fashionable GPUs (such because the A100 and RTX 40-series) have specialised {hardware} for bfloat16 math.

flash_attention_2: Prompts the optimized consideration mechanism we put in earlier. If this fails, the mannequin mechanically falls again to straightforward consideration.

device_map="cuda": Locations the mannequin in your GPU. With out this, it runs on the CPU (a lot slower).

The tokenizer_kwargs={"padding_side": "left"} Setting is essential. The mannequin makes use of last-token pooling, so padding should happen on the left to make sure the final token all the time accommodates significant info.

We encode queries and paperwork individually with totally different prompts (nl2code_query vs nl2code_document). The mannequin was educated with these prompts to tell apart between queries and paperwork, bettering retrieval accuracy.

The similarity matrix is 2×2, the place every cell exhibits how comparable a question is to a doc:

Question 0 vs Doc 0: 0.7670 (excessive—right match)
Question 0 vs Doc 1: 0.1117 (low—totally different)
Question 1 vs Doc 0: 0.0938 (low—totally different)
Question 1 vs Doc 1: 0.6607 (excessive—right match)

Run the script:

python3 test_jina.py

First run downloads the mannequin (~3GB), which takes a couple of minutes. Subsequent runs use the cached model and execute rapidly.

Superior Testing Script

The second script demonstrates complete testing throughout all supported duties with difficult examples.

Script 2: Multi-Process Benchmark

Create test_jina_hard.py with the in depth code supplied beneath.

import os
import math
import textwrap
import torch
import torch.nn.useful as F
from sentence_transformers import SentenceTransformer

# -----------------------------
# Config
# -----------------------------
USE_FLASH_ATTN = False  # set True when you put in flash-attn efficiently
DTYPE = torch.bfloat16
DEVICE_MAP = "cuda"  # "auto" or "cpu" when you should
TRUNCATE_TO = 256     # Matryoshka check: set to None to disable

# -----------------------------
# Loader
# -----------------------------
mannequin = SentenceTransformer(
    "jinaai/jina-code-embeddings-1.5b",
    model_kwargs={
        "dtype": DTYPE,
        "attn_implementation": "flash_attention_2" if USE_FLASH_ATTN else "sdpa",
        "device_map": DEVICE_MAP,
    },
    tokenizer_kwargs={"padding_side": "left"},
)

# -----------------------------
# Helpers
# -----------------------------
def norm(a):
    return F.normalize(torch.as_tensor(a), p=2, dim=1)

def cos_sim(a, b):
    return norm(a) @ norm(b).t()

def pretty_topk(sim, queries, docs, ok=3, title=""):
    print(f"n=== {title} (top-{ok}) ===")
    for i, q in enumerate(queries):
        row = sim[i]
        scores, idx = torch.topk(row, ok=min(ok, row.form[0]))
        print(f"nQ{i+1}: {q[:100]}{'...' if len(q)>100 else ''}")
        for rank, (s, j) in enumerate(zip(scores.tolist(), idx.tolist()), 1):
            print(f"  {rank}. {s:.4f} -> D{j+1}: {docs[j][:120]}{'...' if len(docs[j])>120 else ''}")

def print_matrix(sim, title="similarity"):
    print(f"n=== {title} matrix ({sim.form[0]} x {sim.form[1]}) ===")
    with torch.no_grad():
        for i in vary(sim.form[0]):
            row = " ".be part of(f"{v:.3f}" for v in sim[i].tolist())
            print(row)

def encode_with_prompt(texts, prompt_name):
    # sentence-transformers handles batching internally
    return mannequin.encode(texts, prompt_name=prompt_name)

def maybe_truncate(emb, dims):
    if dims is None:
        return emb
    t = torch.as_tensor(emb)
    if t.form[1] < dims:
        increase ValueError(f"Embedding dim {t.form[1]} < truncate_to {dims}")
    return t[:, :dims]

# -----------------------------
# Datasets (more durable / tough)
# -----------------------------

# 1) NL2CODE — ambiguous wording, traps, and really comparable distractors
nl2code_queries = [
    # regex vs string contains; multi-lang trap
    "python: find emails in a string (RFC-ish, not exact), return all matches",
    # off-by-one and mutable default pitfalls
    "python: create a function memo_fib(n) using lru_cache, handle n<=2 as base case",
    # async + rate limit
    "python: concurrently fetch JSON from 10 URLs with timeout and 5 req/s cap; retry failed once",
    # c++ tricky zero-init vs value-init of vector
    "c++: create vector<int> of length 5 filled with zeros (no loop), idiomatic",
]

nl2code_docs = [
    # good-enough regex (simplified), returns matches
    "import rens="Contact: a@b.com, c.d+e@x.io"nprint(re.findall(r'[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+.[A-Za-z]{2,}', s))",
    # flawed: solely single match with search, not all
    "import rens="x@y.z x2@y2.z2"nprint(re.search(r'w+@w+.w+', s))  # solely first match",
    # right lru_cache memo fib
    "from functools import lru_cachen@lru_cache(None)ndef fib(n:int)->int:n    if n<=2: return 1n    return fib(n-1)+fib(n-2)",
    # async with price restrict (token bucket-ish sketch)
    textwrap.dedent("""
    import asyncio, aiohttp, time
    SEM = asyncio.Semaphore(5)  # crude: 5 concurrent; separate price cap beneath
    async def fetch(session, url):
        async with SEM:
            async with session.get(url, timeout=5) as r:
                return await r.json()
    async def essential(urls):
        out, t0, burst = [], time.time(), 0
        async with aiohttp.ClientSession() as s:
            for i, u in enumerate(urls):
                # naive 5 req/sec limiter
                now = time.time()
                elapsed = now - t0
                if burst >= 5 and elapsed < 1:
                    await asyncio.sleep(1 - elapsed); t0 = time.time(); burst = 0
                out.append(asyncio.create_task(fetch(s, u)))
                burst += 1
            return await asyncio.collect(*out, return_exceptions=True)
    """),
    # C++ value-init vector of zeros
    "std::vector<int> v(5);  // value-initialized to 0",
    # WRONG: reserves capability solely
    "std::vector<int> v; v.reserve(5); // NOT initialized to zeros"
]

# 2) CODE2CODE — equal implementations with delicate model/complexity variations
code2code_queries = [
    "Python: breadth-first search on adjacency list graph; return shortest path distances from source",
    "C++: deduplicate a vector while preserving original order (no set), O(n) average",
]

code2code_docs = [
    # BFS correct
    textwrap.dedent("""
    from collections import deque, defaultdict
    def bfs(n, edges, src):
        g = defaultdict(list)
        for u,v in edges:
            g[u].append(v); g[v].append(u)
        dist = [-1]*n
        dist[src]=0
        dq=deque([src])
        whereas dq:
            u=dq.popleft()
            for w in g[u]:
                if dist[w]==-1:
                    dist[w]=dist[u]+1
                    dq.append(w)
        return dist
    """),
    # DFS (flawed for BFS distance)
    textwrap.dedent("""
    def dfs(n, edges, src):
        g = {i: [] for i in vary(n)}
        for u, v in edges: g[u].append(v); g[v].append(u)
        dist = [-1]*n
        def go(u, d):
            if dist[u]!=-1: return
            dist[u]=d
            for w in g[u]: go(w, d+1)
        go(src, 0)
        return dist  # not true BFS distances on graphs with a number of paths
    """),
    # C++ secure distinctive utilizing unordered_set + seen order
    textwrap.dedent("""
    #embrace <vector>
    #embrace <unordered_set>
    template <typename T>
    std::vector<T> dedup_preserve(const std::vector<T>& a) {
        std::unordered_set<T> seen;
        std::vector<T> out; out.reserve(a.measurement());
        for (const auto& x: a) {
            if (!seen.depend(x)) { seen.insert(x); out.push_back(x); }
        }
        return out;
    }
    """),
    # WRONG: std::set reorders
    textwrap.dedent("""
    #embrace <set>
    #embrace <vector>
    template <typename T>
    std::vector<T> dedup_resorted(const std::vector<T>& a) {
        std::set<T> s(a.start(), a.finish());
        return std::vector<T>(s.start(), s.finish()); // order misplaced
    }
    """),
]

# 3) CODE2NL — summarize code intent; embrace distractors
code2nl_queries = [
    "Explain what this function does in one line: returns False on non-palindromes ignoring non-alnum.",
    "Explain (short): function safely loads JSON file and returns default on error.",
]

code2nl_docs = [
    "import rendef is_pal(s):n    t="".join(ch.lower() for ch in s if ch.isalnum())n    return t == t[::-1]",
    "import jsonnndef load_json(path, default=None):n    attempt:n        with open(path) as f: return json.load(f)n    besides Exception: return default",
    # distractor: unrelated code
    "def primes(n):n    out=[]n    for x in vary(2,n+1):n        if all(xpercentp for p in vary(2,int(x**0.5)+1)): out.append(x)n    return out",
]

# 4) CODE2COMPLETION — continuations with deceptive near-misses
code2completion_queries = [
    "Python: given start of function to compute moving average with window=3, fill the rest efficiently",
    "C++: given partial class with RAII file handle, complete destructor and move semantics safely",
]

code2completion_docs = [
    # good completion (vectorized-ish)
    textwrap.dedent("""
    def movavg3(a):
        if len(a)<3: return []
        return [(a[i]+a[i+1]+a[i+2])/3 for i in vary(len(a)-2)]
    """),
    # naive O(n*w) loop (acceptable however slower)
    textwrap.dedent("""
    def movavg3(a):
        out=[]
        for i in vary(len(a)-2):
            out.append((a[i]+a[i+1]+a[i+2])/3)
        return out
    """),
    # C++ RAII file wrapper (sketch)
    textwrap.dedent("""
    #embrace <cstdio>
    struct File {
        std::FILE* f = nullptr;
        express File(const char* path, const char* mode) : f(std::fopen(path, mode)) {}
        ~File(){ if(f) std::fclose(f); }
        File(File&& o) noexcept : f(o.f){ o.f=nullptr; }
        File& operator=(File&& o) noexcept {
            if(this!=&o){ if(f) std::fclose(f); f=o.f; o.f=nullptr; }
            return *this;
        }
        File(const File&)=delete;
        File& operator=(const File&)=delete;
    };
    """),
    # WRONG: leaks or double-close
    textwrap.dedent("""
    struct FileBad {
        std::FILE* f = nullptr;
        ~FileBad(){ std::fclose(f); } // no null test
    };
    """),
]

# 5) QA — technical Q&A with distractors
qa_queries = [
    "In Python, what's the most reliable way to zero-copy share a NumPy array with PyTorch on GPU?",
    "In SQL, how do you prevent SQL injection when building search queries with user input?",
]

qa_docs = [
    # correct-ish: use torch.from_numpy + pin memory or to(device); zero-copy CPU->Torch, GPU requires .to('cuda')
    "Use torch.from_numpy(arr) for zero-copy CPU sharing; then move to GPU via .to('cuda', non_blocking=True) after pin_memory().",
    # distractor
    "Convert NumPy array to list and rebuild the tensor using torch.tensor(list(arr))  # copies data twice.",
    # SQL parameterization
    "Use parameterized queries / prepared statements (e.g., psycopg2 placeholders, SQLAlchemy bound params); never string-concatenate.",
    # distractor
    "Escape quotes manually and concatenate user input into the SQL string.",
]

# -----------------------------
# Runner per process
# -----------------------------
def run_task(identify, q, d, q_prompt, d_prompt, ok=3):
    q_emb = encode_with_prompt(q, q_prompt)
    d_emb = encode_with_prompt(d, d_prompt)
    sim_full = cos_sim(q_emb, d_emb)

    print_matrix(sim_full, title=f"{identify} (full {q_prompt} vs {d_prompt})")
    pretty_topk(sim_full, q, d, ok=ok, title=f"{identify} top-{ok} (full-dim)")

    if TRUNCATE_TO:
        q_tr = maybe_truncate(q_emb, TRUNCATE_TO)
        d_tr = maybe_truncate(d_emb, TRUNCATE_TO)
        sim_tr = cos_sim(q_tr, d_tr)
        pretty_topk(sim_tr, q, d, ok=ok, title=f"{identify} top-{ok} ({TRUNCATE_TO}D Matryoshka)")
        # fast Kendall tau-like stability (very tough): examine argmax per row
        secure = 0
        for i in vary(sim_full.form[0]):
            j_full = int(torch.argmax(sim_full[i]))
            j_tr = int(torch.argmax(sim_tr[i]))
            secure += (j_full == j_tr)
        print(f"n[{name}] Prime-1 stability after truncation to {TRUNCATE_TO}D: {secure}/{sim_full.form[0]} matchn")

# -----------------------------
# Execute all duties
# -----------------------------
if __name__ == "__main__":
    # NL2CODE
    run_task(
        "NL2CODE",
        nl2code_queries,
        nl2code_docs,
        q_prompt="nl2code_query",
        d_prompt="nl2code_document",
        ok=3
    )
    # CODE2CODE
    run_task(
        "CODE2CODE",
        code2code_queries,
        code2code_docs,
        q_prompt="code2code_query",
        d_prompt="code2code_document",
        ok=3
    )
    # CODE2NL
    run_task(
        "CODE2NL",
        code2nl_queries,
        code2nl_docs,
        q_prompt="code2nl_query",
        d_prompt="code2nl_document",
        ok=3
    )
    # CODE2COMPLETION
    run_task(
        "CODE2COMPLETION",
        code2completion_queries,
        code2completion_docs,
        q_prompt="code2completion_query",
        d_prompt="code2completion_document",
        ok=3
    )
    # QA
    run_task(
        "QA",
        qa_queries,
        qa_docs,
        q_prompt="qa_query",
        d_prompt="qa_document",
        ok=3
    )

    print("nDone. If FlashAttention errors happen, set USE_FLASH_ATTN=False (default) to make use of SDPA.n")

This script checks 5 totally different duties:

NL2CODE Testing: Matches pure language descriptions to code, together with tough circumstances with:

• Ambiguous wording that might match a number of implementations

• Widespread pitfalls like mutable default arguments

• Async operations with price limiting

• Language-specific idioms

CODE2CODE Testing: Finds comparable implementations regardless of variations:

CODE2NL Testing: Matches code to pure language explanations, filtering out unrelated code snippets that may confuse easier fashions.

CODE2COMPLETION Testing: Predicts what code ought to come subsequent, distinguishing between right continuations and plausible-but-wrong options.

QA Testing: Solutions technical questions by matching them to related documentation or code examples, with distractors that point out associated ideas however do not truly reply the query.

The script additionally demonstrates Matryoshka embeddings by truncating vectors to 256 dimensions and measuring whether or not top-1 matches stay secure. This quantifies the speed-vs-accuracy tradeoff you may make in manufacturing.

Run the great check:

python3 testjina2.py

You may see detailed output exhibiting similarity matrices and top-k matches for every process. This helps you perceive how the mannequin behaves in your particular use circumstances and calibrate expectations.

Manufacturing Deployment Issues

When transferring from experimentation to manufacturing, take into account:

Indexing Technique

For big codebases, pre-compute embeddings offline and retailer them in a vector database like:

• Qdrant: Open-source, high-performance, straightforward to deploy

• Milvus: Scales to billions of vectors, glorious for enormous datasets

• Pinecone: Absolutely managed, requires no infrastructure upkeep

• Weaviate: Combines vector and conventional search

API Design

Wrap the mannequin in a FastAPI or Flask service with endpoints for:

• Single question embedding

• Batch embedding (extra environment friendly)

• Similarity search in opposition to your index

• Well being checks and monitoring

Caching

Implement caching for frequently-requested queries. Since embeddings are deterministic (the identical enter all the time produces the identical output), aggressive caching considerably reduces compute prices.

Monitoring

Observe:

• Question latency (p50, p95, p99 percentiles)

• GPU utilization and reminiscence utilization

• Cache hit price

• Error charges and kinds

Scaling

As load will increase:

• Use a number of GPU cases behind a load balancer

• Implement request batching to maximise GPU utilization

• Think about quantization (int8) for additional speedup

• Separate indexing (write) and search (learn) workloads

Conclusion

Jina Code Embeddings 1.5B gives a strong basis for code-related AI purposes. Its compact measurement makes it cost-effective to run, whereas its specialised coaching delivers robust efficiency throughout numerous programming duties. The Matryoshka embedding assist affords distinctive flexibility; you may tune for velocity, reminiscence, or accuracy with out altering fashions or retraining.

This information walked you thru the whole setup on a GPU digital machine, from preliminary provisioning by working complete checks. You now have a working setting for constructing code search engines like google and yahoo, retrieval-augmented technology methods, code suggestion instruments, or documentation turbines.

Subsequent steps to discover:

• Combine together with your codebase and measure retrieval high quality

• Experiment with totally different Matryoshka dimensions to your particular use case

• Add a light-weight re-ranker (like a cross-encoder) to spice up top-k accuracy

• Construct a easy UI to your group to look code conversationally

• Monitor efficiency metrics and optimize based mostly on precise utilization patterns

The mannequin’s open availability and affordable {hardware} necessities decrease obstacles to constructing refined developer instruments that had been beforehand possible just for giant organizations with in depth ML infrastructure.