Editing Quantized Weights: Q8_0 and Q4_K

Quantization stores approximate weights in compact groups called blocks. Values within a block share encoding parameters, so changing one value can change how its neighbors are reconstructed.

secant quant patch edits Q8_0 tensors by decoding and re-encoding only affected blocks. It writes a new checkpoint by default; --in-place changes the input file. Unaffected blocks retain their stored bytes. This avoids converting the entire checkpoint back through a model framework.

This page separates two operations: reordering whole groups of columns and editing their values. “Commutation” means two operations give the same result in either order. Reordering complete Q8_0 blocks can preserve bytes exactly; changing weights introduces quantization error. Neither property guarantees model behavior.

Relation to prior work. In October 2026, Kiwimaker (apply_abliteration.py / apply_abliteration_v2.py) demonstrated in-place r+ np.memmap GGUF Q8_0 editing with SHA-256 verification of untouched byte ranges (while empirically skipping Q4_K and dequantizing/re-quantizing full 2-D Q8_0 matrices in float64 via gguf-py), and SC117 (October 2026) demonstrated byte-level in-place GGUF tensor transplantation and scale-preserving Q2_0 re-encoding. Existing .safetensors abliteration tools (jim-plus/llm-abliteration, NousResearch sharded_ablate.py, and abliterix) re-serialize modified shards from scratch via save_file(). OPLoRA, Safe LoRA, and OGPSA concern constrained adaptation or alignment; their guarantees do not automatically extend through lossy storage. Secant’s specific contributions here are in-place zero-copy mmap byte-slice surgery inside .safetensors files, sub-tensor compressed-domain surgery directly on touched 34-byte block_q8_0 structs (crates/secant-core/src/quant.rs) without full-tensor float64 materialization, and the formal block-head alignment proof between 32-weight Q8_0 blocks ($128 = 4 \times 32$) and 256-weight Q4_K superblocks under $\Pi_{128}$. See the paper for bibliographic details and limits.

1. How Column Groups Fit Into Stored Blocks¶

A panel is a group of 128 adjacent columns. Four Q8_0 blocks fit inside one panel. A Q4_K superblock spans two panels. That difference determines whether moving a panel also splits a set of weights that share encoding parameters.

When llama.cpp converts hybrid Qwen3.5 models with $r = n_v / n_k > 1$ (such as Qwen3.5-4B, where $n_k = 16, n_v = 32, r = 2$) to GGUF, it permutes the input columns of blk.N.ssm_out.weight in 128-column value-head panels ($\Pi_{128}$). (On Qwen3.5-0.8B, $n_k = n_v = 16$ so $r = 1$ and the deployed column order is the identity; we evaluate $\Pi_{128}$ on blk.16.ssm_out.weight as a structural 128-tile permutation test alongside the real identity layout.)

Writing a HuggingFace 128-column panel index as $h = k \cdot r + j$ ($k < n_k$, $j < r$), its GGUF panel position is $\pi(k \cdot r + j) = j \cdot n_k + k$: the transpose of an $(n_k, r) = (16, 2)$ grid into a $(2, 16)$ grid. On Qwen3.5-4B this is a non-trivial bijection, not an involution: because $r \neq n_k$, applying it twice does not return the identity ($\pi(1) = 16$, $\pi(16) = 8$). The inverse is

Whether a quantized format commutes with $\Pi_{128}$ depends on how its block size divides 128:

• In Q8_0 (32 elements per block): Every 128-column value-head panel is an exact union of 4 complete Q8_0 blocks. Editing or permuting a panel never touches a neighboring panel’s scale factor.

• In Q4_K (256 elements per superblock): Every aligned superblock spans two adjacent 128-column panels and shares top-level FP16 scale d and minimum dmin. Editing one panel can alter the reconstructed neighbor. A panel permutation that moves whole superblocks without changing their internal layout, including the identity, can still commute. The reported 67% retained component is a particular weight-space measurement, not a guaranteed consequence of straddling.

The permutation argument assumes a deterministic, position-independent encoder, aligned rows and complete blocks. Percentages here describe projections of weights onto a supplied direction, not the percentage of prompts refused. The standalone Rust secant quant patch refuses Q4_K; experimental Q4_K re-encoding results below are not receipts from a successful invocation of that command.

2. Compare Editing Before and After Compression¶

Starting with already-quantized weights introduces an initial approximation error. Editing and encoding those weights adds another error. The equations below keep track of those errors separately so the two paths can be compared without assuming that rounding cancels out.

Let $Q_B : \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}} \to \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ denote the block-quantize-then-dequantize operator for block size $B$. We assume only that $Q_B$ is deterministic and block-local: the dequantized values of block $b$ depend only on the input values of block $b$. Suppose we apply a column-sparse behavioral edit $\Delta W$ that touches a subset of blocks $\mathcal{T}$. Define the three quantization errors

The decomposition fixes the same additive $\Delta W$ in both paths. It is exact under that assumption, but contains three quantization errors, not one. A projector-subtraction edit recomputes its delta from the current weights, so applying it after dequantization generally uses a different delta; the displayed identity alone does not describe that operator’s two paths. A uniform “half-step” bound is not valid for an arbitrary codec, so each term needs a codec-specific argument.

2.1 A Per-Weight Error Bound for Q8_0¶

There are two sources of rounding here: the integer weight code and the stored scale. The bound adds them, under the stated scale-range assumptions. It describes an idealized calculation; the final paragraph explains what the actual encoder adds.

The Q8_0 encoder (crates/secant-core/src/quant.rs, quantize_row_q8_0) computes, for each 32-element block $x_b$, the scale $d_b = \|x_b\|_\infty / 127$, the codes $q_i = \mathrm{round}(x_i / d_b) \in [-127, 127]$, and stores $\mathrm{fp16}(d_b)$. Dequantization returns $\mathrm{fp16}(d_b)\, q_i$. Assume $d_b$ lies in the FP16 normal range ($127 \cdot 2^{-14} \le \|x_b\|_\infty \le 127 \cdot 65504$), so that $|\mathrm{fp16}(d_b) - d_b| \le 2^{-11} d_b$. Then each element obeys

up to float32 rounding in the encoder arithmetic. (For a subnormal scale, add $127 \cdot 2^{-25}$ per element.) Write $a_b = \|W_b\|_\infty$ and $\delta_b = \|\Delta W_b\|_\infty$ on a touched block $b$. Because $\big|\,\|W_b + \Delta W_b\|_\infty - \|W_b\|_\infty\big| \le \delta_b$, the post-edit scale drifts by at most

before FP16 rounding. Applying the per-element bound to each term, with $\|Q_{32}(W)_b\|_\infty \le (1 + \kappa)\, a_b$:

so every element of a touched Q8_0 block satisfies

This is an ideal-arithmetic bound with FP16 scale rounding included. It requires the scale-range assumption for each of $W_b$, $W_b+\Delta W_b$ and $Q_{32}(W)_b+\Delta W_b$, not only the original block. It does not include the Rust encoder’s float32 reciprocal, multiplication and rounding errors and is not a certified bound for all machine inputs. The reported max_abs_diff $= 2.16 \times 10^{-3}$ is an empirical measurement, separate from that bound.

2.2 What the Q4_K Experiment Measured¶

Sharing scale parameters across panels can make an edit affect more weights than intended. The measurements below describe one experiment; they are not a general error bound or a supported Q4_K patch workflow.

Q4_K is not a single uniform scalar quantizer: each 256-element superblock has an FP16 super-scale d and super-minimum dmin, plus eight 6-bit sub-block scales and minimums. On Qwen3.5 ssm_out, each superblock straddles two 128-column value heads, so an edit to one head can change the shared fit of the neighboring head. We do not claim a proved per-element bound for Q4_K. Its behavior is characterized empirically: observed Path 1 vs. Path 2 max_abs_diff $= 2.38 \times 10^{-2}$ (about $11\times$ the Q8_0 value), 8 straddling superblocks per 2048-column row (16 per 4096-column row), and about 67% of the original weight-direction component retained in experimental Q4_K re-encoding (0.5172 $\to$ 0.346, Section 1). The implemented policy is that secant quant patch accepts only Q8_0 tensors and refuses Q4_K (and every other dtype) with tensor <name> is <dtype>, block-aligned surgery needs Q8_0.

3. Compare the Retained Error Measurements¶

All rows below come from one identified experiment. Reconstruction error measures the distance from the intended float result; the directional residual measures the component left along the supplied direction. Those are different quantities.

The banked exp2_receipt.json (secant.phase3.exp2.v1, SHA-256 1c437338d697ccfb6b24286ac60c84fa0c0a73f3a0120e3aefd920c5d608b0c7) records a single Qwen3.5-0.8B ssm_out experiment with a projector edit of Frobenius norm 0.5172394742. Under commutation.identity_real, both codecs commute with the identity permutation. Their reconstruction errors still differ:

These are retained experimental measurements, not fresh model runs or outputs of the current Rust CLI. The separate whole-model Q8_0 byte-preservation experiment remains in Section 4.

4. Check Which Blocks and Tensors Changed¶

A block can be touched by the logical edit yet keep the same encoded bytes after rounding. The table distinguishes touched blocks, rewritten blocks, and blocks outside the edit. Whole-file verification separately checks the tensors that were not selected.

Comparing Path 1 (schematically, secant patch on BF16 followed by external Q8_0 quantization) against Path 2 (secant quant patch directly on Qwen3.5-0.8B.Q8_0.gguf) shows that direct 34-byte block surgery leaves all untouched blocks 100% bit-identical and matches Path 1 within 0.179% to 0.923% per-tensor relative Frobenius error (note that blocks where the float delta rounds to the same int8 code remain byte-identical even when inside the touched column support):

• Path 1: edit, then quantize; Path 2: quantize, then edit affected blocks.

• Unchanged touched blocks were processed but kept identical bytes. Untouched blocks lie outside the edit support.

Q8_0 block coverage

Untouched support and numerical difference