Annotations are central to data communication, yet most tools still encode them as overlays. In Vega-Lite, even a single callout often requires extra sources, manual coordinates, and layered marks that break as data or layout changes, making annotations fragile and hard to transfer across chart variants.

AnnoGram extends Wilkinson’s Grammar of Graphics with a target-and-effect annotation model. Authors declare what to annotate (target) and how it should appear (effects), instead of hand-authoring pixel-level overlay layers. This makes common annotation tasks more intuitive while treating annotation as a first-class part of the specification.

Because targets stay semantic and placement is resolved from the active chart encodings, the same annotation intent stays portable across mark and layout changes. This creates a concrete path for GoG-based systems such as Vega, Vega-Lite, and ggplot2 to support annotations natively.

Evidence: In the paper’s heuristic comparison (8 tools, 3 chart types), this semantic integration is the only approach rated Easy on all four Cognitive Dimensions: first-class support, intuitiveness, error-proneness, and portability.

Interactive guide below: hover or click a step on the left (or a labeled band in the code) to highlight the corresponding part.

Read the code in three linked steps:

annotations[ ]
Step 1: declare a first-class annotation block (line 3)

The parser reads "annotations" as a sibling of "mark" and "encoding", so annotation logic is part of the chart spec rather than a separate layer.

target
Step 2: bind semantics in data space (line 4)

The target is domain-based ("x": "2021-01-11"), not pixel-based. At render time, the compiler maps that value through the current scales, so the annotation remains valid under filtering, resizing, and mark changes.

effects — enclosure · text · connector
Step 3: compose effects and let placement resolve (lines 5–7)

Multiple effects share one target. Here, an enclosure and a text label are declared; the resolver assigns a non-occluding position by default, while explicit positions remain available when precision is needed.

Playground Example Gallery ↗ GitHub ↗
annotation-spec.json
COVID-19 weekly cases — Jan. 2021 peak Hover/click a band to sync with the matching step on the left
Chart configuration
1 "$schema": "vega-lite-annotation",
2 "mark": "line", "encoding": { /* … */ },
annotations[ ] — top-level grammar key
3 "annotations": [{
target — data-space value
4 "target": { "type": "data-space", "x": "2021-01-11" },
effects — declare appearance and placement
5 "enclosure": { "shape": "rect", "style": { "fill": "rgba(240,165,0,.15)" } },
6 "text": { "text": "Jan. 2021 peak", "position": "upperMiddle" }
7 }]

Three concrete contributions

The paper contributes a formal grammar extension, a working Vega-Lite implementation, and an eight-tool heuristic evaluation. Together, they show that semantic annotation integration is practical within existing GoG pipelines and substantially reduces authoring and maintenance burden.

AnnoGram: a GoG-level annotation primitive

To our knowledge, AnnoGram is the first extension of Wilkinson's Grammar of Graphics that treats annotations as first-class design elements—at the same level as scales, geometries, and guides. This allows GoG-based tools to declare annotations directly within the visualization specification, enabling data-coupled, portable, and computationally tractable annotation workflows.

Vega-Lite Annotation: a working implementation

A Vega-Lite extension implements the grammar end-to-end: parsing annotation JSON, resolving targets against encoding scales, performing occlusion-aware placement via backtracking, and converting annotations to Vega mark encodings. The broader design is grounded in Rahman et al.'s 2024 annotation taxonomy, while the proof-of-concept shown on this site focuses on the common implemented effects highlighted here.

Heuristic comparison across eight tools

We evaluate the extension using annotation-specific lines of code alongside four qualitative dimensions drawn from the Cognitive Dimensions framework—first-class annotation support, intuitiveness, error proneness, and portability—across bar, line, and scatterplot examples in D3, d3-annotate, ggplot2, ggplot2-annotate, HighCharts, Vega, Vega-Lite, and visual editors.

The AnnoGram Grammar

AnnoGram introduces a top-level annotations primitive as a sibling to scales and geometries. Each annotation explicitly specifies what to annotate, which effect (type) to apply, and where to place it, separating annotation semantics from mark-level implementation details. Click any non-terminal to inspect its definition. Keyboard: Tab then Enter.

Non-terminal Terminal Operator / punctuation Grammar field Click non-terminals to inspect. Press Esc to clear.
Root
::= annotations[]
:= | | | | None
textopt := ⟨⟩
enclosureopt := ⟨⟩
connectoropt := ⟨⟩
indicatoropt := ⟨⟩
Annotation Targets
:= Expression | index[]
:= title | legend | subtitle | …
:= ( axis := x | y,
  parts := label | tick | grid,
  rangeopt := [] | Expression )
Annotation Types
⟨X⟩ := ( id, , X )
:= ( string, | None )
:= ( Rect | Ellipse | SVGPath | …, )
:= ( Markers, SVGPath, linear | catmull-rom | … )
:= ( line | area | arrow | …, Expression )
Placement
:= | |
:= auto | start | mid | end
:= auto | upLeft | midRight | …
:= { type := data | pixel,
  x := , y := }
:= string | number

How to read the formal block quickly

LHS symbol defines a production, RHS symbols compose alternatives, and continuation rows expand the same production with aligned constraints.

Clicking a non-terminal exposes its canonical production and a typed JSON witness, while related rows remain highlighted to reveal dependency closure.

The Turing Tarpit problem in current tools

GoG implementations such as Vega-Lite, Vega, and ggplot2 can represent annotations, but only through verbose, indirect encodings. As Perlis characterized it, this is a Turing tarpit: "everything is possible but nothing of interest is easy." Writing a labeled connector in Vega-Lite requires separate data sources, manual coordinate arithmetic, and brittle mark layering—any of which can silently break when data or layout changes.

Target-and-effect model

Following CSS's target-and-effect approach, each annotation specifies what to annotate and which effect to use. On this site, effect is the reader-facing term for the annotation type attached to a target. This separation means the same annotation specification can apply to a bar chart, scatterplot, or line chart of the same data without structural changes to the annotation block.

Reference and Composite relationships

A Reference annotation links to another via id—e.g. a connector whose endpoint is an enclosure defined elsewhere in the spec. A Composite groups multiple annotations to describe repeated structures, such as seasonal time spans repeated across a series. These are relationships among annotations, not separate effects, and formalize the paper's annotation ensemble model.

Automatic placement with full override

The Position Resolver heuristically selects unoccupied space via backtracking, minimizing occlusion across text, enclosures, and connectors—extending Vega-Label's text-only approach to all annotation types. Manual positioning is never required, but authors retain full override capability using fixed pixel or data-space coordinates and directional offsets.

One Annotation, Three Decisions

The paper separates annotation authoring into three independent decisions: choose a semantic target, choose one or more effects, and choose placement. We use effect here to mean the annotation type attached to a target. Keeping these decisions orthogonal is what makes the grammar more intuitive to author and more portable when data, layout, or mark type changes.

Target
Identify the chart element, axis region, data subset, or standalone note anchor.
Effect
Choose the annotation expression: text, enclosure, connector, indicator, or a combination.
Placement
Accept automatic placement or override it with anchoring, data-space, pixel-space, or offsets.

A useful reading rule for the whole site: keep the target semantic, keep the effect declarative, and let placement resolve from the current chart unless you need an explicit override.

Target
Choose the referent

A target identifies the visualization element an annotation refers to. In the paper, this is the first design decision because it keeps annotations coupled to chart semantics instead of to pixels.

  • DataPoint: select a point or subset by index or expression, so annotations update as the data changes.
  • Axis: annotate values, ranges, ticks, gridlines, or labels to provide quantitative context.
  • ChartPart and None: explain legends, titles, and other structural elements, or add standalone notes.

Why it matters: semantic targets are the foundation of portability.

Effect
Choose the annotation expression

An effect is the visual annotation type attached to the target. The broader design is grounded in Rahman et al.'s 2024 taxonomy; the proof-of-concept shown on this site focuses on common implemented effects.

  • Text communicates explanation, interpretation, or narrative context.
  • Enclosure and connector group elements or link explanations to their referents.
  • Indicator marks values, thresholds, or trends with lines, areas, or arrows.

Multiple effects can share one target; Reference and Composite relationships coordinate larger annotation ensembles.

Placement
Let the chart decide, then override if needed

Placement is intentionally separate from the effect itself. The paper emphasizes that manual positioning is never required: annotations receive meaningful default placement from target and layout context, but authors retain full control when precision is needed.

  • Auto: the Position Resolver searches for unoccupied positions and reduces occlusion via backtracking.
  • Relative anchoring: 1D and 2D anchors align annotations to edges or centers of their targets.
  • Fixed positioning: data-space, pixel-space, and dx/dy support explicit refinement.

This separation is what lets the same target and effect adapt to different chart layouts.

Five-Stage Compiler Pipeline

The proof-of-concept extends Vega-Lite's compilation process through an external wrapper, preserving Vega compatibility while injecting annotation semantics across five coordinated stages (Parser, Position Resolver, Assembler, Transpiler, and Post-Adder).

Parser
Validates input, extracts annotation blocks, resolves defaults from the normalized Vega-Lite spec
Position Resolver
Analyzes the scene graph, computes pixel positions, applies anchoring and offsets, minimizes occlusion via backtracking
Assembler
Links related annotation marks (connectors ↔ enclosures), resolves circular and nested references
Transpiler
Converts annotation objects to Vega mark encodings; injects data and mark arrays into the Vega spec
Post-Adder
Handles complex renders requiring direct scene-graph injection (e.g., ellipse-shaped enclosures in non-data-space)

Annotation Portability

In the paper, portability means annotations remain valid and well-positioned as data or chart types change. AnnoGram achieves this by decoupling author intent from any single mark layout: the same target and effect stay in the specification, while runtime logic resolves current coordinates, anchors, and available space. Figure 5 demonstrates this with one annotation specification producing context-appropriate placements across point, bar, and line charts.

Fixed in the spec

Semantic target plus effect intent stay authored once inside the annotation block.

Resolved at runtime

Current marks and coordinates are recomputed from active encodings, scales, and scene-graph state.

Placed per layout

Automatic placement, anchors, and offsets adapt to available space while avoiding overlap.

Read portability in three passes: stable specification, runtime target resolution, and runtime placement.

target + effect spec
What stays fixed: target and effect intent

Changing "point" to "bar" or "line" alters the chart mark, but the target and effect declarations are reused. The author does not reauthor the annotation for each chart variant.

target: data-expr
What is recomputed: semantic target to current referent

"type": "data-expr" with "datum.mpg === max(mpg)" identifies a datum by meaning. At runtime, the resolver maps that target through active encodings and scene-graph state, so the referenced mark can change by chart type without editing the specification.

position: "auto"
What is recomputed: effect placement per layout

The Position Resolver and Assembler recompute placement from target semantics, chart encodings, anchors, offsets, and available space. Manual overrides remain available, but the same "auto" instruction yields context-appropriate placement on each chart.

portable-annotation.json
Cars dataset — best fuel-economy data point Hover/click a band to see what stays fixed and what is resolved at runtime
spec reused across charts
1 "data": { "url": "cars.json" },
2 "mark": "point", // point | bar | line
annotations[ ]
3 "annotations": [{
target — semantic selection
4 "target": { "type": "data-expr",
5 "expr": "datum.mpg === max(mpg)" },
effect + placement — runtime result
6 "text": { "text": "Best fuel economy", "position": "auto" },
7 "connector": { "curve": "natural" }
8 }]

Heuristic Comparison Across Eight Existing Tools

Equivalent annotated bar, line, and scatterplot examples were implemented across eight existing tools. Annotation-specific and total lines of code measure authoring cost; four Cognitive Dimensions (Green & Petre, 1996)—first-class support, intuitiveness, error-proneness, and portability—capture qualitative authoring experience.

Tool Bar Ann. Bar LOC Line Ann. Line LOC Scatter Ann. Scatter LOC First-class Intuitive Error Prone Portable
High programmability
D3 151 81 175 76 170 104 ✕ Hard High Hard
d3-annotate 161 81 174 77 177 99 ✓ Med. Med. Med.
ggplot2 120 22 128 43 208 101 ✓ Med. High Med.
ggplot2-annotate 73 20 117 42 178 101 ✓ Med. High Med.
Low programmability
HighCharts 82 32 231 82 169 79 ✓ Med. Low Hard
Vega 515 161 491 219 387 186 ✕ Hard High Hard
Vega-Lite 259 23 253 29 177 34 ✕ Hard High Hard
VL Annotation (ours) 101 19 95 25 79 31 ✓ Easy Low Easy
Visual editors (PowerPoint, Figma, etc.)
Visual Editors — ✕ Easy Low Hard
Key finding: Semantic integration is the decisive factor. Tools without built-in annotation constructs (D3, Vega, Vega-Lite) require manual positioning and mark-level management—yielding the highest LOC and Hard ratings on all dimensions. Partially integrated tools (d3-annotate, ggplot2-annotate) reduce initial effort but cannot adapt annotations when data or layout changes, reflected in Medium ratings. Visual editors are easy to author but disconnect annotations from data entirely—Hard portability. AnnoGram is the only tool to achieve Easy across all four dimensions.

Cite This Work

@inproceedings{rahman2025annogram,
  title     = {{AnnoGram}: An Annotative Grammar of
               Graphics Extension},
  author    = {Rahman, Md Dilshadur and
               Zaman, Md Rahat-uz and
               McNutt, Andrew and
               Rosen, Paul},
  booktitle = {IEEE Visualization and Visual Analytics
               (VIS)},
  year      = {2025},
  doi       = {10.1109/VIS60296.2025.00053}
}
arXiv:2507.04236 ↗ IEEE ↗