Upload README.md with huggingface_hub

README.md

@@ -2,134 +2,85 @@
 license: mit
 tags:
 - attention
+- positional-encoding
 - efficient-attention
 - number-theory
 - calabi-yau
-- custom-kernel
-- pytorch
-- benchmark
-datasets: []
+- negative-result
+datasets:
+- callensxavier/cy-sieve-attention-benchmark
 ---
 
-# S20-Decay Attention Kernel
+# CY-Sieve Attention Kernel — a falsifiable experiment (NEGATIVE result)
 
-A high-performance, mathematically exact attention bias derived from the Weight-5 Apéry-like binomial sum:
+A positional-attention bias derived from the weight-5 Apéry-like binomial sum
 
 $$S_{20}(n) = \sum_{k=0}^{n} \binom{n}{k}^4 \binom{n+k}{k}$$
 
-> **Paper**: [A Weight-5 Apéry-like Binomial Sum, its Calabi-Yau 4-fold Period, and Supercongruences](https://doi.org/10.5281/zenodo.20747943)  
-> **Code**: [GitHub — Mirror-Map-Sieve](https://github.com/xaviercallens/Mirror-Map-Sieve)
-
----
-
-## 3 Core Hypotheses
-
-1. **Exact Mathematical Rigidity**: Unlike ALiBi or learned positional embeddings, the S20 sequence provides a deterministic, integer-derived attention decay. Zero floating-point drift at any context length.
-2. **O(1) Vectorized Toeplitz Construction**: The decay matrix is built as a 1D broadcast over a distance tensor — no nested loops, no learned parameters.
-3. **Zero-Cost LLM Injection**: S20 decay can be injected into any SDPA-based model via global monkey-patching (`F.scaled_dot_product_attention`) with **zero measurable latency overhead** on GPU.
-
----
-
-## GPU Benchmark: Tesla T4 (16GB, CUDA 12.9, PyTorch 2.9.1)
-
-### Raw Kernel: SDPA ± S20 Bias
-
-| Seq Len | SDPA Baseline | SDPA + S20 | Overhead | Correct |
-|---------|--------------|------------|----------|---------|
-| 64 | 0.020 ms | 0.022 ms | 1.08× | ✅ |
-| 128 | 0.024 ms | 0.029 ms | 1.23× | ✅ |
-| 256 | 0.041 ms | 0.053 ms | 1.31× | ✅ |
-| 512 | 0.092 ms | 0.144 ms | 1.56× | ✅ |
-| 1024 | 0.199 ms | 0.529 ms | 2.65× | ✅ |
-
-### Phi-3-mini-4k-instruct (3.8B) — Global SDPA Patching
-
-| Seq Len | Baseline | S20-SDPA | Overhead | Base tok/s | S20 tok/s | Energy (J) | Power (W) |
-|---------|----------|----------|----------|------------|-----------|------------|-----------|
-| 64 | 49.59 ms | 49.09 ms | **0.99×** | 1,290 | 1,304 | 67.9 | 69.2 |
-| 128 | 59.21 ms | 59.15 ms | **1.00×** | 2,162 | 2,164 | 82.3 | 69.8 |
-| 256 | 106.98 ms | 107.39 ms | **1.00×** | 2,393 | 2,384 | 115.5 | 54.4 |
-| 512 | 211.06 ms | 213.15 ms | **1.01×** | 2,426 | 2,402 | 297.3 | 69.9 |
-| 1024 | 488.51 ms | 487.11 ms | **1.00×** | 2,096 | 2,102 | 659.9 | 67.7 |
-
-> **Key finding**: On a real 3.8B-parameter model, S20 global SDPA patching adds **zero measurable overhead** (0.99–1.01×) across all sequence lengths. The integer-sequence bias is effectively free on GPU.
-
----
-
-## CPU Benchmark: Open-Weights Model Injection
-
-S20 decay injected as post-logit positional mask on CPU (Apple Silicon):
-
-| Model | Params | Baseline | S20-Injected | Overhead | Avg PPL |
-|-------|--------|----------|--------------|----------|---------|
-| GPT-2 | 124M | 39.2 ms | 41.6 ms | 1.06× | 175.7 |
-| DistilGPT-2 | 82M | 21.6 ms | 21.3 ms | 0.99× | 302.0 |
-| OPT-125M | 125M | 29.0 ms | 29.4 ms | 1.01× | 199.7 |
-| BLOOM-560M | 559M | 122.4 ms | 118.0 ms | 0.96× | 139.0 |
-
----
-
-## Method: Global SDPA Patching (Forward Hook Option A)
-
-```python
-import torch
-import torch.nn.functional as F
-from math import comb
-
-# 1. Build S20 decay sequence
-def s20(n): return sum(comb(n, k)**4 * comb(n+k, k) for k in range(n+1))
-_S20 = [s20(d) for d in range(18)]
-
-# 2. Vectorized log-bias matrix
-def build_s20_log_bias(seq_len, device="cuda", dtype=torch.float16):
-    base = float(_S20[0])
-    weights = [base/float(x) if x > 0 else 0.0 for x in _S20] + [0.0]
-    dv = torch.tensor(weights, dtype=torch.float32, device=device)
-    idx = torch.arange(seq_len, device=device)
-    dist = (idx.unsqueeze(0) - idx.unsqueeze(1)).abs().clamp(max=len(_S20))
-    decay = dv[dist]
-    log_bias = torch.log(decay.clamp(min=1e-30))
-    causal = torch.tril(torch.ones(seq_len, seq_len, device=device))
-    log_bias = log_bias * causal + (1 - causal) * (-1e9)
-    return log_bias.unsqueeze(0).unsqueeze(0).to(dtype)
-
-# 3. Monkey-patch F.scaled_dot_product_attention
-_original_sdpa = F.scaled_dot_product_attention
-_bias_cache = {}
-
-def patched_sdpa(q, k, v, attn_mask=None, dropout_p=0.0, is_causal=False, **kw):
-    L = q.shape[-2]
-    if L not in _bias_cache:
-        _bias_cache[L] = build_s20_log_bias(L, q.device, q.dtype)
-    bias = _bias_cache[L][:, :, :L, :k.shape[-2]]
-    if attn_mask is not None:
-        attn_mask = attn_mask + bias
-    else:
-        attn_mask = bias
-    return _original_sdpa(q, k, v, attn_mask=attn_mask, dropout_p=dropout_p, **kw)
-
-F.scaled_dot_product_attention = patched_sdpa
-# Now ANY model using SDPA will have S20 decay injected automatically
-```
-
----
-
-## Reproducibility
+whose holonomic structure is a Calabi–Yau **3-fold** period (an order-4 MUM block;
+the geometry fixes the long-range decay slope $\log\lambda = 3.762$ and curvature
+$\beta = 2$). The engineering bet: generate the positional bias *on the fly* from
+the order-4 recurrence, costing **O(L)** HBM instead of an **O(L²)** table.
+
+> **Code:** [GitHub — Mirror-Map-Sieve](https://github.com/xaviercallens/Mirror-Map-Sieve)
+> · **Math paper:** [Zenodo 10.5281/zenodo.20747943](https://doi.org/10.5281/zenodo.20747943)
+> · **Benchmark data:** [callensxavier/cy-sieve-attention-benchmark](https://huggingface.co/datasets/callensxavier/cy-sieve-attention-benchmark)
+
+## ⚠️ Honest result (2026-06-22, NVIDIA L4): the quality gate KILLED it
+
+This card **corrects** an earlier version that claimed "zero overhead" and implied
+a win. Those numbers came from an invalid method (monkey-patching the bias into a
+*frozen* pretrained model, which collapses every alternative scheme equally). The
+correct test — **training small GPTs from scratch**, one per positional scheme, on
+real WikiText-2 — gives:
+
+| scheme | ppl @512 (train) | @1024 (2×) | @2048 (4×) |
+|---|---|---|---|
+| **learned-absolute** | **4.22** | 12.10 | 20.82 |
+| ALiBi | 10.74 | 11.73 | 11.35 |
+| **sliding-window** | **4.99** | **5.07** | **5.03** |
+| CY-Sieve τ-ladder | 11.33 | 12.31 | 12.05 |
+| CY-Sieve τ=128 | 6.80 | 7.12 | 7.00 |
+| CY-Sieve τ=512 | 4.65 | 6.08 | 10.62 |
+
+**Verdict: KILL (+10.15%).** Best CY-Sieve (4.65) vs best baseline (4.22) exceeds
+the pre-committed >5% kill threshold; a plain **sliding window won outright**. The
+geometry-fixed slope is too steep for a drop-in positional scheme.
+
+## What *did* hold up
+
+- **§4 kernel correctness — PASS.** The Triton kernel matches the NumPy reference
+  within FP16 tolerance (4/4 tests).
+- **§6 memory claim — confirmed.** The on-the-fly bias reads **O(L)** bytes of HBM
+  vs **O(L²)** for a materialized table (**8192× less at L=16384**). *But* the
+  current unfused kernel is **~4–6× slower** than fused dense SDPA — a
+  memory-traffic win, **not** a latency win. Per the project's reporting rule,
+  with the quality gate failing, these numbers are **not** a contribution.
+
+## Why publish a negative result?
+
+Because a fast, correct kernel that degrades model quality is a *failed* kernel,
+and saying so is the point. The Calabi–Yau geometry is a sound **prior** for the
+bias shape, not the right **value** — pinning the attention slope to the
+sequence's growth rate was the mistake. Redesign directions (learnable
+geometry-initialized slope; exact-local-window + gentle-tail hybrid; β=2 ablation)
+are in the [findings writeup](https://github.com/xaviercallens/Mirror-Map-Sieve/blob/main/docs/PHASE3_CYSIEVE_GPU_FINDINGS.md).
+
+## Reproduce
 
 ```bash
-# Clone and run on any CUDA GPU
 git clone https://github.com/xaviercallens/Mirror-Map-Sieve.git
-cd Mirror-Map-Sieve/4_ai_hardware_attention
-pip install torch transformers accelerate
-python gpu_benchmark_s20.py --model microsoft/Phi-3-mini-4k-instruct --seq_lens 64 128 256 512 1024
+cd Mirror-Map-Sieve
+pip install -r 4_ai_hardware_attention/requirements-gpu.txt
+python 4_ai_hardware_attention/run_gpu_phase.py   # §4 parity + §5 quality + §6 perf
 ```
 
 ## Citation
 
 ```bibtex
-@software{callens2026s20attn,
+@misc{callens2026cysieve,
   author = {Callens, Xavier},
-  title  = {S20-Decay Attention Kernel: Vectorized Integer-Sequence Attention Bias},
+  title  = {CY-Sieve Attention: a Calabi--Yau positional bias and its negative quality result},
   year   = {2026},
   url    = {https://huggingface.co/callensxavier/s20-attention-kernel},
   doi    = {10.5281/zenodo.20747943}