Different designs.
Same feature.
CLIP: near-twins
Vision models see color.
A splash, a search and a password screen look alike to CLIP. Their palettes match.
Developers see features.
A tab bar. A trip planner. A row of options. Meaning lives in how components relate.
Introducing
FRAME
UI structure + semantics, in one screen embedding
Strip the style.
Greyscale and doubled contrast take color out of the equation and sharpen component edges.
screenshot → greyscale → contrast × 2
Build a UI graph.
Detected components become nodes. Neighbors within 300 px are joined by edges.
node = component ≥ 48×48 pxCLIP · imageBERT · OCR text
Share context.
Each component’s embedding mixes with its neighbors’, so groups that form a feature start to agree.
S′ = (I + w · D-½ A D-½) S
Weigh the shapes.
A Vietoris–Rips complex finds tight triangles of components. Their areas weight what matters.
+ whole-screen CLIP → PCA: 1,792 → 116 dimensions
Three benchmarks from UI testing and design
+0%
retrieval MRR over CLIP, the strongest baseline (Aurora)
Every metricFRAME beats CLIP, BLIP, BERT & Screen2Vec on all three datasets
0.32 vs 0.003clustering ARI against CLIP (Avgust)
2.18 vs 7.30Davies–Bouldin index: better-separated clusters (lower wins)
Neuro-symbolic, by design
No extra training. It rides the models.
FRAME augments vision-language embeddings with symbolic UI structure, so it improves as those models do.
Fusing UI Structure and Semantics for Feature-Oriented App Screen Retrieval and Clustering
Arun Krishna Vajjala · Yanfu Yan · Ajay Krishna Vajjala · Shrunal Pothagoni · Denys Poshyvanyk · Kevin Moran
ICSME 2026 · IEEE International Conference on Software Maintenance and Evolution