Xtext's new Formatter API

Moritz Eysholdt, itemis AG

XTEXT’S NEW FORMATTER API

AGENDA• motivation

• use cases

• ITextRegionAccess

• IFormattableDocument

• example formatter

• testing

DEFINITION

DEFINITION

“improve readability and emphasize structure of a document

without changing its meaning”

DEFINITION

“improve readability and emphasize structure of a document

without changing its meaning”

“prettify spaces, line wraps, tabs and indentation without changing the AST”

…if it does change the AST, it’s probably a clean-up action or refactoring

MOTIVATION

I’m lazy

readability

diff & merge

bug prevention

MOTIVATION

I’m lazy

readability

diff & merge

bug prevention

With Formatter: CTRL+SHIFT+F

MOTIVATION

I’m lazy

readability

diff & merge

bug prevention

With Formatter: CTRL+SHIFT+F

Without Formatter: →→⇒⇒__↓ →→⇒⇒⇒⇒__↓ ¶ ↑↑⇒↓↓⇒⇒¶

MOTIVATION

I’m lazy

readability

diff & merge

bug prevention

MOTIVATION

bad formatting… - obscures the code’s structure - unnecessarily consumes brain

cycles to filter out anomalies

I’m lazy

readability

diff & merge

bug prevention

MOTIVATION

I’m lazy

readability

diff & merge

bug prevention

MOTIVATION

• whitespace-only changes in diffs are annoying

I’m lazy

readability

diff & merge

bug prevention

MOTIVATION


• merge conflicts because of whitespace-only super annoying

I’m lazy

readability

diff & merge

bug prevention

MOTIVATION


• merge conflicts because of whitespace-only super annoying

• only commit formatted code!

I’m lazy

readability

diff & merge

bug prevention

MOTIVATION

I’m lazy

readability

diff & merge

bug prevention

MOTIVATION

I’m lazy

readability

diff & merge

bug prevention

MOTIVATION

I’m lazy

readability

diff & merge

bug prevention

MOTIVATION

I’m lazy

readability

diff & merge

bug prevention

MOTIVATION

wrong indentation! proper formatting would have

increased the chances of spotting this bug!

I’m lazy

readability

diff & merge

bug prevention

MOTIVATION

USE CASES

on demand

serialization

semantic quickfix

USE CASES

on demand

serialization

semantic quickfix

CTRL+SHIFT+F….and via context menu

USE CASES

on demand

serialization

semantic quickfix

USE CASES

on demand

serialization

semantic quickfix

Use an Xtext grammar to convert an EMF model to text. Formatter contributes indentation, newlines, space, etc.

USE CASES

on demand

serialization

semantic quickfix

Use an Xtext grammar to convert an EMF model to text. Formatter contributes indentation, newlines, space, etc.

Serializer Use Cases: - non-textual (e.g. graphical) editors - model migration

USE CASES

on demand

serialization

semantic quickfix

USE CASES

@Fix(IssueCodes.INVALID_FEATURE_NAME)public void fixName(final Issue issue, IssueResolutionAcceptor acceptor) { acceptor.accept(issue, "Uncapitalize name", "", "", new ISemanticModification() { @Override public void apply(EObject element, IModificationContext context) { ((Feature) element).setName(toFirstLower(issue.getData()[0])); } });}

ARCHITECTURE

messy

formatter

pretty

messy

formatter

pretty

developer

creates

developer creates formatter, NoEngine!

messy

formatter

pretty

developer

creates

text

model(AST)

text-regions(CST)

in

in

in

Input: Text, AST and CST

messy

formatter

pretty

developer

creates

text

model(AST)

text-regions(CST)

in

in

in

text replacements

apply

Output: TextReplacements

messy

formatter

pretty

grammar

developer

creates

text

model(AST)

text-regions(CST)

in

in

in

text replacements

apply

references

access to grammar

fragment = formatting2.Formatter2Fragment auto-inject {}

fragment = formatting.FormatterFragment auto-inject {}

GeneratorFragment for MWE2

import org.eclipse.xtext.formatting2.AbstractFormatter2!class StatesFormatter extends AbstractFormatter2 {! def dispatch void format(Model model, extension IFormattableDocument document) { model.regionFor.keyword(";").prepend[noSpace] }}

Model: "machine" name=QualifiedName “;";



extends AbstractFormatter2



dispatch over AST elements


1. start with the EObject



1. start with the EObject 2. find desired semantic text region using ITextRegionAccess



1. start with the EObject 2. find desired semantic text region using ITextRegionAccess 3. register formatting information on IFormattableDocument


TEXT REGIONS

text:StringDocument

getText():String

offset:intlength:int

TextRegion1 *

TextRegions… …represent a substring …can overlap …can be empty

regionForOffset(int offset, int length)regionForDocument()

resource:XtextResource

ITextRegionAccess

regionForEObject(EObject object)regionForRootEObject()

the document. created from node

model or via serializer

getText():String

offset : intlength: int

ITextRegion



ITextRegionAccess


returns

regions for anything

non-hidden token 0..n hidden tokens

ISequentialRegion

getText():String


ITextRegion

undefined:boolean

IHiddenRegion

grammarElement:EObject

ISemanticRegion nextprevious

previousnext



ITextRegionAccess


returns

strictly alternating linked list of

hidden/semantic regions

whitespace token comment token

ISequentialRegion

getText():String


ITextRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPartparts

IWhitespace IComment



ITextRegionAccess


returns

eObject:EObjectgrammarElement:EObject

IEObjectRegion ISequentialRegion

getText():String


ITextRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPart

children

parts




ITextRegionAccess


returns

returns

leadingtrailing

for AST element


IEObjectRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPart

children

parts


leadingtrailing


IEObjectRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPart

children

parts


leadingtrailing

/* Copyright */machine my.X;!// First Statestate

1. Take Document

1.


IEObjectRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPart

children

parts


leadingtrailing


Model: "machine" name=QualifiedName ";" elements+=State*;!State: {State} ‘state';!QualifiedName: ID ('.' ID)*;

1. Take Document 2. Parse it using Grammar

1.

2.

class StatesFormatter extends AbstractFormatter2 {! def dispatch void format(Model model, extension IFormattableDocument document) { println(textRegionAccess.toString()) }}


IEObjectRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPart

children

parts


leadingtrailing



1. Take Document 2. Parse it using Grammar 3. Dump it via textRegionAccess.toString()

1.

2.

3.

class StatesFormatter extends AbstractFormatter2 {! def dispatch void format(Model model, extension IFormattableDocument document) { println(textRegionAccess) }}


IEObjectRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPart

children

parts


leadingtrailing



1. Take Document 2. Parse it using Grammar 3. Dump it via textRegionAccess.toString() 4. See what we get (next slide)

1.

2.

3.



Columns: 1:offset 2:length 3:kind 4: text 5:grammarElementKind: H=IHiddenRegion S=ISemanticRegion B/E=IEObjectRegion! 0 H "/* Copyright */" Comment:TerminalRule'ML_COMMENT' 16 "\n" Whitespace:TerminalRule'WS' B Model'my.X' Model16 7 S "machine" Model:'machine'23 1 H " " Whitespace:TerminalRule'WS'24 4 S "my.X" Model:name=QualifiedName28 0 H28 1 S ";" Model:';'29 H "\n\n" Whitespace:TerminalRule'WS' 17 "// First State\n" Comment:TerminalRule'SL_COMMENT' B State Model:elements+=State path:Model'my.X'/elements[0]46 5 S "state" State:'state' E State Model:elements+=State path:Model'my.X'/elements[0] E Model'my.X' Model51 0 H


IEObjectRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPart

children

parts


leadingtrailing





IEObjectRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPart

children

parts


leadingtrailing





IEObjectRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPart

children

parts


leadingtrailing





IEObjectRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPart

children

parts


leadingtrailing

keyword

datatype





IEObjectRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPart

children

parts


leadingtrailing

empty

first region in document

last region in document

1 part

2 parts





IEObjectRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPart

children

parts


leadingtrailing





IEObjectRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPart

children

parts


leadingtrailing





IEObjectRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPart

children

parts


leadingtrailing

begin

end

grammarElement

path





IEObjectRegion

undefined:boolean

IHiddenRegion



previousnext


IHiddenRegionPart

children

parts


leadingtrailing


(Recap)



FIND SEMANTIC REGIONS

// in local eObject only eObject.regionFortextRegionAccess.regionForEObject(eObject).regionFor!// in eObject or any of its childreneObject.allRegionsFortextRegionAccess.regionForEObject(eObject).allRegionsFor!// in whole documenttextRegionAccess.regionForRootEObject.allRegionsFor

}// first matched semantic region.keyword(",").feature(StatesPackage.Literals.STATE__NAME).keyword(actionsKeyword_2_1_0).assignment(actionsAssignment_2_1_1).ruleCall(actionsINTTerminalRuleCall_2_1_1_0)!// all matched semantic region.keywords(",").features(StatesPackage.Literals.STATE__NAME).keywords(actionsKeyword_2_1_0).assignments(actionsAssignment_2_1_1).ruleCalls(actionsINTTerminalRuleCall_2_1_1_0)!.keywordPairs("(", ")")

{




{ITextRegionAccess




{ITextRegionExtensions




{Keywords (String)




{EMF EStructuralFeature (EAttribute, EReference)




{Grammar Element (from MyLanguageGrammarAccess)

TEXT REGION RECAP

TEXT REGION RECAP• ITextRegionAccess (for Xtend, ITextRegionExtensions)


• linked list of strictly alternating ISemanticRegion and IHiddenRegion



• IHiddenRegion can be empty




• IHiddenRegion contains parts, e.g. IComment and IWhitespace





• IEObjectRegion for elements from AST





• IEObjectRegion for elements from AST

• use toString() during debugging!



Model: "machine" name=QualifiedName “;"; (Recap)



Model: "machine" name=QualifiedName “;"; (Recap)

FORMATTABLE DOCUMENT

interface IFormattableDocument {!!!!!!!!!!!! !!!!!!!!!!!! // ...}

API: specify formatting information for hidden regions

Implementation: store text replacers for text regions




interface IFormattableDocument {!!!!!!!!!!!! !!!!!!!!!!!!!!! // ...}

specify for one or two


IHiddenRegion

autowrap()setPriority(int priority)indent()newLine()setNewLines(int newLines)setNewLines(int min, int default, int max)noSpace()oneSpace()setSpace(String space)

IHiddenRegionFormatter




IHiddenRegion



class StatesFormatter extends AbstractFormatter2 {! def dispatch void format(Model model, extension IFormattableDocument document) { model.regionFor.keyword(";").prepend[noSpace] }}




IHiddenRegion



space, newLines, indent, autowrap




IHiddenRegion



IHiddenRegion ISemanitcRegion IHiddenRegion ISemanitcRegion IHiddenRegion ISemanitcRegion IHiddenRegion

IHiddenRegionPart


1

* remember the strictly alternating

list of hidden/semantic regions

interface IFormattableDocument {!!!!!!! // set for provided hidden region def set(IHiddenRegion hiddenRegion, (IHiddenRegionFormatter)=>void init)! // append after the provided semantic region def append(ISemanticRegion appendAfter, (IHiddenRegionFormatter)=>void init) def append(EObject appendAfter, (IHiddenRegionFormatter)=>void init)! // prepend before the provided semantic region def prepend(ISemanticRegion prependBefore, (IHiddenRegionFormatter)=>void init) def prepend(EObject prependBefore, (IHiddenRegionFormatter)=>void init)!!!!!!!!!!! // ...}

specify for one IHiddenRegionFormatter IHiddenRegion


specify for one


IHiddenRegionFormatter IHiddenRegion


specify for one




specify for one




specify for one




specify for one



interface IFormattableDocument {!!!!!!! // set for both provided hidden regions def set(IHiddenRegion r1, IHiddenRegion r2, (IHiddenRegionFormatter)=>void init)! // before and after provided semantic region def surround(ISemanticRegion semanticRegion, (IHiddenRegionFormatter)=>void init) def surround(EObject owner, (IHiddenRegionFormatter)=>void init)! // set for two inwards-bound hidden regions def interior(ISemanticRegion r1, ISemanticRegion r2, (IHiddenRegionFormatter)=>void init) def interior(EObject object, (IHiddenRegionFormatter)=>void init)!!!!!!!!!!! // ...}

specify for two IHiddenRegionFormatter IHiddenRegion


















Why specify formatting for two hidden regions at once?


Why specify formatting for two hidden regions at once?- indent() spans from first to second hidden region!


Why specify formatting for two hidden regions at once?- indent() spans from first to second hidden region!- convenience





Why not increaseIndent() and decreaseIndent() individually?



Why not increaseIndent() and decreaseIndent() individually?- likely to not be applied symmetrically due to bugs, exceptions

----------- RootDocument with ITextReplacers (syntax: <offset|text>) -----------machine my.X;state<19| >foo<23| >actions 1, 2, 3;<43|>end!--------------------------------------------------------------------------------19 1 " ": HiddenRegionReplacer: space=' ';indentInc=123 2 "\n ": HiddenRegionReplacer: newLine=143 1 "\n": HiddenRegionReplacer: newLine=1;indentDec=1

def dispatch void format(State state, extension IFormattableDocument document) { state.interior[indent] state.regionFor.feature(STATE__NAME).prepend[oneSpace].append[newLine] state.regionFor.keyword("end").prepend[newLine] println(document.toString)}

State: {State} 'state' name=ID ('actions' actions+=INT ("," actions+=INT)* ";")? 'end';

















offset : intlength: intnewText: String

ITextReplacementautowrap()setPriority(int priority)indent()newLine()setNewLines(int newLines)setNewLines(int min, int default, int max)noSpace()oneSpace()setSpace(String space)


( )*?Recap: The formatter outputs text replacements!

interface IFormattableDocument {!!!!!!!!!!!! !!!!!!!!!!!!!}

createReplacements(ITextReplacerContext context): ITextReplacerContext

ITextReplacer

HiddenRegionReplacer



configures

configures a HiddenRegionReplacer


creates TextReplacements


ITextReplacer


ITextReplacementHiddenRegionReplacer



configures

creates


specific for TextRegion


ITextReplacer





configures


ITextRegion

1

1

creates



ITextReplacer





configures

getIndentation(): intwithIndentation(int indent): ITextReplacerContextgetParent(): ITextReplacerContextaddReplacement(ITextReplacement repl)getAllReplacements():List<ITextReplacement>

ITextReplacerContext

uses


ITextRegion

1

1

creates

collects

context collects replacements

class FormattableDocument {

! Map<ITextRegion, ITextReplacer> replacers;! def List<ITextReplacement> renderToTextReplacements() { val List<ITextReplacer> sorted = sortByOffset(replacers) var ITextReplacerContext context = createTextReplacerContext() for (ITextReplacer replacer : sorted) { context = replacer.createReplacements(context) } return context.getAllReplacements() }}!!!!!!class MyReplacer implements ITextReplacer {! override ITextReplacerContext createReplacements(ITextReplacerContext context) { val ITextReplacement replacement = region.replaceWith(" ") context.addReplacement(replacement) val int oldIndentation = context.getIndentation() return context.withIndentation(oldIndentation + 1) }}


ITextReplacer





uses


ITextRegion

1

1

creates

collects




ITextReplacer





uses


ITextRegion

1

1

creates

collects

sorted




ITextReplacer





uses


ITextRegion

1

1

creates

collects

invoke in sorted order




ITextReplacer





uses


ITextRegion

1

1

creates

collects

in/out context




ITextReplacer





uses


ITextRegion

1

1

creates

collects

return replacements




ITextReplacer





uses


ITextRegion

1

1

creates

collects

create replacement




ITextReplacer





uses


ITextRegion

1

1

creates

collects

add to context




ITextReplacer





uses


ITextRegion

1

1

creates

collects

return cloned context




ITextReplacer





uses


ITextRegion

1

1

creates

collects

EXAMPLES

state foo actions 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13; end

AutoWrap

def dispatch void format(State state, extension IFormattableDocument document) { for (k : state.regionFor.keywords(“,")) { k.prepend[noSpace].append[oneSpace; autowrap] }}

State: 'state' name=ID 'actions' actions+=INT ("," actions+=INT)* ";" 'end';


AutoWrap

def dispatch void format(State state, extension IFormattableDocument document) { for (k : state.regionFor.keywords(“,")) { k.prepend[noSpace].append[oneSpace; autowrap] }}


def dispatch void format(State state, extension IFormattableDocument document) { val IAutowrapFormatter af = [ region, wrapped, doc | val first = (region as IWhitespace).hiddenRegion val last = state.regionFor.keyword(";").previousHiddenRegion doc.set(first, last, [indent]) ] for (k : state.regionFor.keywords(",")) { k.prepend[noSpace].append[oneSpace; autowrap; onAutowrap = af] }}

AutoWrap with Handler















Conditional Formatting state bar 1 => foo; 2 => foo; 3 => foo; 4 => foo; 5 => foo;end

state bar 1 => foo; 2 => foo; 3 => foo; end

def dispatch void format(State state, extension IFormattableDocument document) { state.formatConditionally( [ doc | val extension slDoc = doc.requireFitsInLine state.regionFor.feature(STATE__NAME).surround[oneSpace] for (t : state.transitions) { t.format.append[oneSpace] } ], [ extension doc | state.interior[indent] state.regionFor.feature(STATE__NAME).prepend[oneSpace].append[newLine] for (t : state.transitions) { t.format.append[newLine] } ] )}

State: 'state' name=ID transitions+=Transition* 'end';





XOR













Pattern-Aware formatting

def dispatch void format(State state, extension IFormattableDocument document) { if (state.regionFor.keyword("end").previousHiddenRegion.isMultiline) { state.interior[indent] state.regionFor.feature(STATE__NAME).prepend[oneSpace].append[newLine] for (t : state.transitions) { t.format.append[newLine] } } else { state.regionFor.feature(STATE__NAME).surround[oneSpace] for (t : state.transitions) { t.format.append[oneSpace] } } }

state bar 1 => foo;end

state bar 1 => foo; end


Pattern-Aware formatting

def dispatch void format(State state, extension IFormattableDocument document) { if (state.regionFor.keyword("end").previousHiddenRegion.isMultiline) { state.interior[indent] state.regionFor.feature(STATE__NAME).prepend[oneSpace].append[newLine] for (t : state.transitions) { t.format.append[newLine] } } else { state.regionFor.feature(STATE__NAME).surround[oneSpace] for (t : state.transitions) { t.format.append[oneSpace] } } }


state bar 1 => foo;end

state bar 1 => foo; end

Table-based formattingstate baz 1 => foo; 12 => foo; 123 => foo;end

def dispatch void format(State state, extension IFormattableDocument document) { state.interior[indent] val width = state.transitions.map[regionFor.feature(TRANSITION__EVENT).length].max + 1; for (t : state.transitions) { val region = t.regionFor.feature(TRANSITION__EVENT) region.append[space = Strings.repeat(" ", width - region.length)] t.regionFor.keyword(";").prepend[noSpace].append[newLine] } }








TESTS

org.eclipse.xtext.junit4.formatter.FormatterTester@Test def void example1() { assertFormatted[ toBeFormatted = ''' entity Foo { propertyName:String op name() { } } ''' ]} @Test def void example2() {

assertFormatted[ expectation = ''' entity Foo { op foo():String { "xx" } } ''' toBeFormatted = ''' entity Foo { op foo ( ) : String { "xx" } } ''' ]}

THE OLD FORMATTER

• no access to text, Node Model or AST

• not enhanceable engine

• will be deprecated

RELEASES

• Alpha release as part of Xtext 2.8.0, 2.8.1, 2.8.3

• Beta release as of Xtext 2.9-beta

• stable (public API) release (hopefully) with Xtext 2.9

Questions?

Documents

Xtext's new Formatter API