feat: serialize AST node back to wikitext string (#8258)

* refactor: extract a new $tw.wiki.getParser

* feat: allow $tw.utils.getParseTreeText to render other rules' text

* feat: two example getText handler

* Revert "feat: allow $tw.utils.getParseTreeText to render other rules' text"

This reverts commit 8a12498fa9.

* refactor: keep original getParseTreeText not touched

* refactor: use serialize in rules

* refactor: $tw.utils.extend({},options) -> options || {}

* Update codeinline.js

* Create test-wikitext-serialize.js

* DEBUG: only run my tests for development, remove before PR merge

* lint: if

* feat: add rule: 'parseBlock' metadata

* feat: handle tailing \n that may be missing

* feat: allow recursive

* feat: generate more rule and tests

* feat: generate more rule and tests

* fix: remove pragma:true, otherwise following text will become children of it

* fix: condition manually

Deekseek is silly

* fix: some test

* fix: some test

* feat: $tw.utils.serializeAttribute

* fix: use "" for string param

* feat: list

* refactor: ' -> "

* fix: parsemode don't have node

* fix: render invisible comment and parsemode as data element

* feat: add void: true, in ast node to prevent render

* feat: use void widget, so methods always return a widget

* feat: ast to use new widget type void

* test: add rule: 'parseBlock' and isRuleEnd: true

* lint: quote

* Update widget.js

* fix: void node need to handle its children

* Update test-wikitext-parser.js

* lint: quote

* Update void.js

* Update test-wikitext-parser.js

* fix: macrodef with comment (void node) not working

* lint: ' -> "

* feat: add to styleblock

* feat: styleblock

* feat: styleinline

* Update table.js

* lint: useless comments

* feat: transcludeblock

* refactor: reuse block on inline when possible

* feat: use void node to carry important info for typedblock

* feat: run all tests

* lint: useless ai generated comments

* Update conditional.js to not include space

* Update test-wikitext-serialize.js

* Update conditional.js

* refactor: move tiddlers to /data

* refactor: no need for new $tw.Wiki()

* lint: double quote

* refactor: lowercase the parseblock rule name

* fix: Wiki parser initialize blockRuleClasses only when first new an instance

* feat: restore inline macro def

* fix: macro in widget param

* fix: positional attribute in macro call

* fix: table space and horizrule block new line

* feat: make sure block rule all have \n\n for visiblity

* lint: function param

* fix: empty list item

* feat: add \n\n based on isBlock, if could also be inline

* fix: conditional without elseif

* refactor: use isBlock in macrodef to know inline or block

* fix: link may not have attribute and children

* DEBUG: render result and diff below body only on browser

DEBUG: render result below body only on browser

DEBUG: render result below body

DEBUG: fix build

DEBUG: show render result as ViewTemplate

* fix: remove pad space in />

* test: remove pad space in />

* Revert DEBUG: render result and diff below body only on browser

* refactor: fold commentText variable

* refactor: fold long comment

* fix: double quotes for parameter values

* Update void.js

* refactor: move all exports.serialize = function(tree,serialize) { to plugin

* fix: expost listTypes from core, and require it in plugin

* refactor: move serializeWikitextParseTree to plugin and init it

* refactor: move serializeAttribute util also to the plugin

* fix: Delete unused file

* Update macrodef.js

* Update test-wikitext-parser.js

* lint: fix

* Update plugins/tiddlywiki/wikitext-serialize/rules/filteredtranscludeblock.js

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update core/modules/widgets/void.js

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/tiddlywiki/wikitext-serialize/rules/list.js

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/tiddlywiki/wikitext-serialize/rules/list.js

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/tiddlywiki/wikitext-serialize/rules/styleblock.js

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remove unused methods from VoidNodeWidget

Deleted render, execute, and refresh methods from VoidNodeWidget as they are no longer needed. The widget now only inherits from the base Widget class and exports the constructor.

* docs: about regex in styleinline.js

* Update parsetree.js

* Update core/modules/widgets/void.js

Co-authored-by: Jeremy Ruston <jeremy@jermolene.com>

* feat: Ensure at least one space after the style/class

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Jeremy Ruston <jeremy@jermolene.com>
This commit is contained in:
lin onetwo
2025-10-30 05:21:36 +08:00
committed by GitHub
parent 7898cb8446
commit 20d6be1e23
119 changed files with 1796 additions and 193 deletions

View File

@@ -0,0 +1,5 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/Attribute
type: text/vnd.tiddlywiki
<$macrocall $name="rpn" a=<<rpn 2 2 *>> b="pi" operation="*" decimals="4"/>

View File

@@ -0,0 +1,7 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/BlockRule
type: text/vnd.tiddlywiki
The speed of sound
The light of speed

View File

@@ -0,0 +1,5 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/BoldEmphasis
type: text/vnd.tiddlywiki
This is ''bold'' text

View File

@@ -0,0 +1,17 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/CodeBlock
type: text/vnd.tiddlywiki
Simple `JS` and complex
---
```js
var match = reEnd.exec(this.parser.source)
```
```tid
<$list filter="[tag[ExampleTag]sort[title]]"/>
```
end

View File

@@ -0,0 +1,11 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/CommentBlock
type: text/vnd.tiddlywiki
<!-- This is a comment -->
Some text
<!-- Another comment -->
More text

View File

@@ -0,0 +1,5 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/CommentInline
type: text/vnd.tiddlywiki
This is some text with an inline comment <!-- This is a comment --> and some more text.

View File

@@ -0,0 +1,21 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/Conditional
type: text/vnd.tiddlywiki
This is a <%if [{something}] %>Elephant<%elseif [{else}] %>Pelican<%else%>Crocodile<%endif%> <%if [{something}] %>Elephant<%else%>Crocodile<%endif%>
<%if [{$:/info/url/protocol}match[file:]]%>
Loaded from a file URI
<%elseif [{$:/info/url/protocol}match[https:]]%>
Loaded from an HTTPS URI
<%elseif [{$:/info/url/protocol}match[http:]]%>
Loaded from an HTTP URI
<%else%>
Loaded from an unknown protocol
<%endif%>
Plain text in next paragraph.
<%if [{$:/info/url/protocol}match[file:]]%>
Hidden.
<%endif%>

View File

@@ -0,0 +1,7 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/Dash
type: text/vnd.tiddlywiki
This is an en-dash: --
This is an em-dash: ---

View File

@@ -0,0 +1,5 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/Entity
type: text/vnd.tiddlywiki
This is a copyright symbol: &copy;

View File

@@ -0,0 +1,7 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/ExtLink
type: text/vnd.tiddlywiki
An external link: https://www.tiddlywiki.com/
A suppressed external link: ~http://www.tiddlyspace.com/

View File

@@ -0,0 +1,13 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/FilteredTranscludeBlock
type: text/vnd.tiddlywiki
{{{ [tag[docs]] }}}
{{{ [tag[docs]] |tooltip}}}
{{{ [tag[docs]] ||TemplateTitle}}}
{{{ [tag[docs]] |tooltip||TemplateTitle}}}
{{{ [tag[docs]] }}width:40;height:50;}.class.class

View File

@@ -0,0 +1,5 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/FilteredTranscludeInline
type: text/vnd.tiddlywiki
{{{ [tag[docs]] }}} {{{ [tag[docs]] |tooltip}}} {{{ [tag[docs]] ||TemplateTitle}}} {{{ [tag[docs]] |tooltip||TemplateTitle}}} {{{ [tag[docs]] }}width:40;height:50;}.class.class

View File

@@ -0,0 +1,15 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/FunctionDefinition
type: text/vnd.tiddlywiki
\function name(param:"defaultvalue", param2:"defaultvalue")
definition text
\end
\procedure name(param:"defaultvalue", param2:"defaultvalue")
definition text
\end
\widget $mywidget(param:"defaultvalue", param2:"defaultvalue")
definition text
\end

View File

@@ -0,0 +1,12 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/HardLineBreaks
type: text/vnd.tiddlywiki
"""
This is some text
That is set like
It is a Poem
When it is
Clearly
Not
"""

View File

@@ -0,0 +1,29 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/Heading
type: text/vnd.tiddlywiki
! Heading 1
!! Heading 2
!!! Heading 3
!!!! Heading 4
!!!!! Heading 5
!!!!!! Heading 6
! AAA
!! AAA
!!! AAA
!!!! AAA
!!!!! AAA
!!!!!! AAA
AAA

View File

@@ -0,0 +1,15 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/Html
type: text/vnd.tiddlywiki
<aside>
This is an HTML5 aside element
</aside>
<$slider target="MyTiddler">
This is a widget invocation
</$slider>
<$list filter="[tag[ExampleTag]sort[title]]"/>
Plain text in next paragraph.

View File

@@ -0,0 +1,10 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/Image
type: text/vnd.tiddlywiki
[img[https://tiddlywiki.com/fractalveg.jpg]]
[img width="23" height="24" [https://tiddlywiki.com/fractalveg.jpg]]
[img width={{!!width}} height={{!!height}} [https://tiddlywiki.com/fractalveg.jpg]]
[img[Description of image|https://tiddlywiki.com/fractalveg.jpg]]
[img[TiddlerTitle]]
[img[Description of image|TiddlerTitle]]

View File

@@ -0,0 +1,6 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/Import
type: text/vnd.tiddlywiki
\import [[$:/core/ui/PageMacros]] [all[shadows+tiddlers]tag[$:/tags/Macro]!has[draft.of]]
\import [[$:/core/ui/PageMacros]]

View File

@@ -0,0 +1,5 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/ItalicEmphasis
type: text/vnd.tiddlywiki
This is //italic// text

View File

@@ -0,0 +1,40 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/List
type: text/vnd.tiddlywiki
* This is an unordered list
* It has two items
# This is a numbered list
## With a subitem
# And a third item
; This is a term that is being defined
: This is the definition of that term
#** One
#* Two
#** Three
#**** Four
#**# Five
#**## Six
## Seven
### Eight
## Nine
* List item one
*.active List item two has the class `active`
* List item three
# AAA
## [[BBB]]
### CCC
# AAA
## CCC
## DDD
## EEE
# BBB
## FF `/` FFF
## FFF
## GGG
##

View File

@@ -0,0 +1,9 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/MacroCallBlock
type: text/vnd.tiddlywiki
<<name "value" "value2">>
<<.def "macro calls">>
<<alert "primary" "primary alert" width:"60%">>

View File

@@ -0,0 +1,5 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/MacroCallInline
type: text/vnd.tiddlywiki
These are macro calls in a line: <<name "value" "value2">> and <<.def "macro calls">> <<alert "primary" "primary alert" width:"60%">>

View File

@@ -0,0 +1,9 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/MacroDef
type: text/vnd.tiddlywiki
\define name(param:defaultvalue,param2:defaultvalue)
definition text, including $param$ markers
\end
\define lingo-base() $:/language/ControlPanel/Basics/

View File

@@ -0,0 +1,5 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/Parameters
type: text/vnd.tiddlywiki
\parameters(param:defaultvalue,param2:defaultvalue)

View File

@@ -0,0 +1,9 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/ParserMode
type: text/vnd.tiddlywiki
\parsermode block
\parsermode inline
Test.

View File

@@ -0,0 +1,6 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/PrettyExtLink
type: text/vnd.tiddlywiki
[ext[https://tiddlywiki.com/fractalveg.jpg]]
[ext[Tooltip|https://tiddlywiki.com/fractalveg.jpg]]

View File

@@ -0,0 +1,6 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/PrettyLink
type: text/vnd.tiddlywiki
[[Introduction]]
[[Link description|TiddlerTitle]]

View File

@@ -0,0 +1,7 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/QuoteBlock
type: text/vnd.tiddlywiki
<<<tc-quote
Quote text
<<<

View File

@@ -0,0 +1,6 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/RulesPragma
type: text/vnd.tiddlywiki
\rules except ruleone ruletwo rulethree
\rules only ruleone ruletwo rulethree

View File

@@ -0,0 +1,5 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/SimpleText
type: text/vnd.tiddlywiki
The quick brown fox

View File

@@ -0,0 +1,6 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/SoftLineBreak
type: text/vnd.tiddlywiki
The rain in Spain
falls mainly on the plain

View File

@@ -0,0 +1,5 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/StrikethroughEmphasis
type: text/vnd.tiddlywiki
This is ~~strikethrough~~ text

View File

@@ -0,0 +1,17 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/StyleBlock
type: text/vnd.tiddlywiki
@@background-color:red;
@@.myClass
This paragraph will have the CSS class `myClass`.
* The `<ul>` around this list will also have the class `myClass`
* List item 2
@@
@@font-size:1.5em;
@@.coloured-text.coloured-bg
* Block content
* With custom style and classes
@@

View File

@@ -0,0 +1,13 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/StyleBlock2
type: text/vnd.tiddlywiki
@@width:100px;
@@.myFirstClass.mySecondClass.myThirdClass
This is a paragraph
@@
@@background-color:lightcyan;
* Item one
* Item two
@@

View File

@@ -0,0 +1,7 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/StyleInline
type: text/vnd.tiddlywiki
@@.myClass This is some text with a class@@
@@background-color:red; This is some text with a background colour@@
@@width:100px;.myClass This is some text with a class and a width@@

View File

@@ -0,0 +1,5 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/SubscriptEmphasis
type: text/vnd.tiddlywiki
This is ,,subscript,, text

View File

@@ -0,0 +1,5 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/SuperscriptEmphasis
type: text/vnd.tiddlywiki
This is ^^superscript^^ text

View File

@@ -0,0 +1,6 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/SysLink
type: text/vnd.tiddlywiki
$:TiddlerTitle
~$:TiddlerTitle

View File

@@ -0,0 +1,11 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/Table
type: text/vnd.tiddlywiki
|!|!Alpha|!Beta|!Gamma|!Delta|
|!One|||||
|!Two|||||
|!Three|||||
|cell one|cell two|
|cell three|cell four|

View File

@@ -0,0 +1,21 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/TranscludeBlock
type: text/vnd.tiddlywiki
{{MyTiddler}}
{{MyTiddler||TemplateTitle}}
{{||TemplateTitle}}
{{MyTiddler|Parameter}}
{{MyTiddler||TemplateTitle|Parameter|SecondParameter}}
{{MyTiddler!!field}}
{{!!field}}
{{MyTiddler##index}}
{{##index}}

View File

@@ -0,0 +1,5 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/TranscludeInline
type: text/vnd.tiddlywiki
{{MyTiddler}} {{MyTiddler||TemplateTitle}}

View File

@@ -0,0 +1,15 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/TypedBlock1
type: text/vnd.tiddlywiki
$$$text/vnd.tiddlywiki > text/plain
This is ''some'' wikitext
$$$
$$$text/unknown
Some plain text, which will not be //formatted//.
$$$text/vnd.tiddlywiki > text/html
This is ''some'' wikitext
$$$

View File

@@ -0,0 +1,19 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/TypedBlock2
type: text/vnd.tiddlywiki
$$$.js
This will be rendered as JavaScript
$$$
$$$.svg
<svg xmlns="http://www.w3.org/2000/svg" width="150" height="100">
<circle cx="100" cy="50" r="40" stroke="black" stroke-width="2" fill="red" />
</svg>
$$$
$$$image/svg+xml
<svg xmlns="http://www.w3.org/2000/svg" width="150" height="100">
<circle cx="100" cy="50" r="40" stroke="black" stroke-width="2" fill="green" />
</svg>
$$$

View File

@@ -0,0 +1,5 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/UnderscoreEmphasis
type: text/vnd.tiddlywiki
This is __underscore__ text

View File

@@ -0,0 +1,7 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/WikiLink
type: text/vnd.tiddlywiki
AWikiLink
AnotherLink
~SuppressedLink

View File

@@ -0,0 +1,5 @@
tags: $:/tags/wikitext-serialize-test-spec
title: Serialize/WikiLinkPrefix
type: text/vnd.tiddlywiki
~SuppressedLink

View File

@@ -26,4 +26,4 @@ title: TiddlerOne
+
title: ExpectedResult
<p>[{"type":"element","tag":"p","children":[{"type":"text","text":"This is a block","start":68,"end":83}],"start":68,"end":83}]</p><p>[{"type":"text","text":"This is inline","start":136,"end":152}]</p>
<p>[{"type":"element","tag":"p","children":[{"type":"text","text":"This is a block","start":68,"end":83}],"start":68,"end":83,"rule":"parseblock"}]</p><p>[{"type":"text","text":"This is inline","start":136,"end":152}]</p>