|
|
--- |
|
|
language: en |
|
|
tags: |
|
|
- clip |
|
|
- vision |
|
|
- transformers |
|
|
- interpretability |
|
|
- sparse autoencoder |
|
|
- sae |
|
|
- mechanistic interpretability |
|
|
library_name: torch |
|
|
pipeline_tag: feature-extraction |
|
|
metrics: |
|
|
- type: explained_variance |
|
|
value: 89.42 |
|
|
pretty_name: Explained Variance % |
|
|
range: |
|
|
min: 0 |
|
|
max: 100 |
|
|
- type: l0 |
|
|
value: 655.9 |
|
|
pretty_name: L0 |
|
|
--- |
|
|
|
|
|
# CLIP-B-32 Sparse Autoencoder x64 vanilla - L1:1e-05 |
|
|
|
|
|
### Training Details |
|
|
|
|
|
- Base Model: CLIP-ViT-B-32 (LAION DataComp.XL-s13B-b90K) |
|
|
- Layer: 10 |
|
|
- Component: hook_resid_post |
|
|
|
|
|
### Model Architecture |
|
|
|
|
|
- Input Dimension: 768 |
|
|
- SAE Dimension: 49,152 |
|
|
- Expansion Factor: x64 (vanilla architecture) |
|
|
- Activation Function: ReLU |
|
|
- Initialization: encoder_transpose_decoder |
|
|
- CLS_only: true |
|
|
|
|
|
### Performance Metrics |
|
|
|
|
|
- L1 Coefficient: 1e-05 |
|
|
- L0 Sparsity: 655.9 |
|
|
- Explained Variance: 89.42% |
|
|
|
|
|
### Training Configuration |
|
|
|
|
|
- Learning Rate: 0.01 |
|
|
- LR Scheduler: Cosine Annealing with Warmup (200 steps) |
|
|
- Epochs: 10 |
|
|
- Gradient Clipping: 1.0 |
|
|
|
