RFD0018 - Syn Matchers and Visitor

• Feature Name: syn_matchers_and_visitor
• Start Date: 2026-03-23
• Status: implemented

Summary

This RFD proposes a shared syntax-consumer support layer on top of the faithful Syn.Cst introduced in RFD0015.

The proposal introduces three layers:

• Syn.Cst: the faithful, typed CST
• Syn.Matchers: small unwrap/flatten/extract helpers
• Syn.Visit: an explicit, reentrant visitor with full traversal control

The key design choice is simple:

• explicit traversal control is fundamental
• Syn.Visit is the one shared recursion API

This RFD does not propose pushing more consumer convenience into Syn.Cst itself. The goal is to keep the CST faithful and move ergonomics into shared support modules.

Motivation

The CST is already making rules and tools easier to write, but consumers still repeat too much structural work:

• unpacking SourceFile manually
• rebuilding entrypoint plumbing from structure/signature items
• hand-writing recursive descent over expressions, patterns, and core types
• mixing traversal concerns with local syntactic matching

That is a sign that the CST layer is useful but incomplete on its own. Riot needs a shared support layer so syntax consumers stop rediscovering the same shape knowledge independently.

Guide-level explanation

Contributors should think of the post-CST syntax stack as three layers:

1. Syn.Cst
2. Syn.Matchers
3. Syn.Visit

Syn.Cst stays faithful

Syn.Cst describes what was written. It should not keep growing interpretive helpers just because some downstream rule wants a shortcut.

Syn.Matchers provides local shape helpers

Syn.Matchers is for small, composable helpers that do not define traversal policy.

Examples:

• Syn.Matchers.Expression.unwrap_parens
• Syn.Matchers.Expression.flatten_apply
• Syn.Matchers.Pattern.unwrap_alias_typed_parens
• Syn.Matchers.CoreType.unwrap

Syn.Visit is the shared recursion layer

Syn.Visit is the one traversal abstraction for syn.

It is explicitly visitor-shaped:

• callbacks receive the current context
• callbacks receive a walker
• callbacks return an updated 'ctx
• callbacks choose whether to recurse
• callbacks choose which children to recurse into
• callbacks choose the traversal order

The important distinction is:

• the visitor is the hook table
• the walker is the traversal engine

The walker also exposes descend_* helpers, which perform the standard child walk for the current node without going back through the current hook. That lets callbacks reuse the default structural walk explicitly instead of relying on hidden framework behavior.

Reference-level explanation

1. Module responsibilities

The proposed module layout is:

module Syn.Cst
module Syn.Matchers
module Syn.Visit

Responsibilities:

Syn.Cst

• faithful public syntax tree
• tokens and spans where they are part of the tree
• no traversal policy
• no rule-specific query logic

Syn.Matchers

• wrapper removal
• flattened application helpers
• local extraction helpers for common shapes

Syn.Visit

• explicit visitor hooks
• explicit reentrant walker
• default child traversal via walker.descend_*
• full coverage over meaningful public CST node families

2. Syn.Visit API shape

The main visitor shape should be:

module Syn.Visit : sig
  type 'ctx walker = {
    source_file : 'ctx -> Cst.SourceFile.t -> 'ctx;
    implementation : 'ctx -> Cst.implementation -> 'ctx;
    interface : 'ctx -> Cst.interface -> 'ctx;
    structure_item : 'ctx -> Cst.StructureItem.t -> 'ctx;
    signature_item : 'ctx -> Cst.SignatureItem.t -> 'ctx;
    attribute : 'ctx -> Cst.attribute -> 'ctx;
    extension : 'ctx -> Cst.extension -> 'ctx;
    pattern : 'ctx -> Cst.Pattern.t -> 'ctx;
    expression : 'ctx -> Cst.Expression.t -> 'ctx;
    core_type : 'ctx -> Cst.CoreType.t -> 'ctx;
    module_expression : 'ctx -> Cst.ModuleExpression.t -> 'ctx;
    module_type : 'ctx -> Cst.ModuleType.t -> 'ctx;
    ...

    descend_source_file : 'ctx -> Cst.SourceFile.t -> 'ctx;
    descend_pattern : 'ctx -> Cst.Pattern.t -> 'ctx;
    descend_expression : 'ctx -> Cst.Expression.t -> 'ctx;
    descend_core_type : 'ctx -> Cst.CoreType.t -> 'ctx;
    ...
  }

  type 'ctx visitor = {
    visit_source_file :
      'ctx -> 'ctx walker -> Cst.SourceFile.t -> 'ctx;
    visit_structure_item :
      'ctx -> 'ctx walker -> Cst.StructureItem.t -> 'ctx;
    visit_signature_item :
      'ctx -> 'ctx walker -> Cst.SignatureItem.t -> 'ctx;
    visit_attribute :
      'ctx -> 'ctx walker -> Cst.attribute -> 'ctx;
    visit_pattern :
      'ctx -> 'ctx walker -> Cst.Pattern.t -> 'ctx;
    visit_expression :
      'ctx -> 'ctx walker -> Cst.Expression.t -> 'ctx;
    visit_core_type :
      'ctx -> 'ctx walker -> Cst.CoreType.t -> 'ctx;
    ...
  }

  val default : 'ctx visitor
  val walker : 'ctx visitor -> 'ctx walker
end

Important properties:

• callbacks receive the traversal engine, not the raw visitor record
• callbacks control recursion explicitly
• default traversal only happens when a callback calls walker.descend_*
• contexts can stay immutable in normal OCaml style

3. Why the visitor should not auto-traverse by return value

This RFD rejects a design like:

type 'ctx control =
  | Continue of 'ctx
  | Skip_children of 'ctx
  | Stop of 'ctx

for the main visitor API.

That design hides traversal behind framework semantics:

• Continue means “and now recurse for me”
• Skip_children means “do not recurse for me”

It works, but it does not feel like a normal visitor. Explicit traversal is the whole point of this layer, so the recursion should be visible in rule and tool code.

4. Why callbacks receive a walker instead of the raw visitor

Passing the raw visitor record into callbacks would hand them the hook table, not the traversal engine.

What callbacks actually need is:

• “visit this child expression”
• “perform the default child walk for this node”

That is what the walker provides. The walker is built from the visitor and is the executable recursion API.

5. Full-spectrum visitor coverage

Syn.Visit should expose hooks across the meaningful public CST node families, not only the few shapes current consumers happen to use.

That includes:

• file roots
• structure and signature items
• attributes, extensions, and payloads
• patterns and parameters
• expressions and recursive subfamilies
• core types, module types, and class types
• module/class expressions
• declarations such as let, type, module, class, include, and open

The goal is that consumers do not need bespoke recursion just because they care about a less common public node family.

6. Rule and tool authoring conventions

Once these layers exist, syntax consumers should:

1. use Syn.Visit for recursive traversal
2. use Syn.Matchers for local unwrap/flatten/extract helpers
3. avoid bespoke recursive descent unless the shared visitor genuinely cannot express the behavior

That should be the style bar for new rule and tooling code.

Drawbacks

This adds more public surface area and a substantial documentation burden.

It also requires discipline to keep:

• Syn.Cst
• Syn.Matchers
• Syn.Visit

clearly separated by responsibility.

Rationale and alternatives

Alternative: keep adding helper methods to Syn.Cst

Rejected.

That would make the CST convenient in the short term, but it would also push consumer-specific interpretation into the core tree surface.

Alternative: keep a separate fold-oriented traversal module beside Syn.Visit

Rejected.

The overlap is not worth it. Syn.Visit is expressive enough to be the one shared recursion API, and folds/collectors can be built as small visitors on top of it.

Alternative: make the main visitor auto-traverse children by return value

Rejected.

That hides traversal control behind framework semantics and weakens the main reason to have a visitor API at all.

Alternative: pass the raw visitor record into callbacks

Rejected.

Callbacks need the traversal engine, not just the hook table. The separate walker abstraction makes that distinction explicit.

Prior art

The most relevant prior art is Tree-sitter and Rust syn.

Tree-sitter

Tree-sitter is strong prior art for:

• explicit structural trees as the basis for tooling
• field-oriented matching over syntax nodes
• declarative structural queries
• preserving exact source ranges on every node

Riot should learn two things from that model:

1. syntax consumers want to describe shapes, not rediscover raw child positions
2. exact source anchoring should remain first-class even when the consumer API becomes much nicer

Riot is intentionally not copying Tree-sitter’s stringly query DSL as the primary API. Syn.Cst is typed, and Syn.Visit is meant to feel like normal OCaml code, not a separate little language embedded beside the host language.

Rust syn

Rust syn is the clearest model for the visitor side:

• hook methods define custom behavior
• the default walker is a separate traversal function
• continuing traversal is explicit

That split between hooks and traversal engine is the right model here as well.

Rust tooling also shows the value of keeping different layers distinct:

• syn for typed syntax structure
• visitor APIs for traversal
• separate lint or refactoring layers on top

That layering matches Riot’s direction:

• Syn.Cst for the faithful typed tree
• Syn.Matchers for local shape helpers
• Syn.Visit for explicit traversal

Unresolved questions

• whether Syn.Visit should eventually grow optional leave_* hooks
• how much of the descend_* implementation should be generated versus hand-written
• whether canonical span / name_site accessors should be proposed in a follow-up RFD or folded into this one later

Future possibilities

Once these layers exist, Riot can build richer syntax support without polluting Syn.Cst itself:

• scope-aware read/write collectors
• binding-site helpers
• path and field-root analysis
• future formatter or macro passes built on the shared visitor layer