# ScroogeXHTML for Object Pascal: migration guide

Breaking changes and deprecations, newest version first. Each section lists
what changed and how to keep the previous behaviour where that is possible.

- [Migrating to 11.2](#migrating-to-112)
- [Migrating to 11.1](#migrating-to-111)
- [Migrating to 11.0](#migrating-to-110)
- [Breaking changes and deprecations in 10.0](#breaking-changes-and-deprecations-in-100)
- [Breaking changes and deprecations in 9.0](#breaking-changes-and-deprecations-in-90)

## Migrating to 11.2

11.2 changes no default output for valid RTF. It adds two options and two
default font replacement rules.

The one output change is for malformed RTF: a colour index above
2147483647 in `\cf`, `\cb`, `\highlight` or `\cbpat` wrapped around to a
valid colour (`\cf4294967297` was colour 1). It is now out of range, so
the text gets no colour (the default font colour for `\cf`), and with
`StrictMode` the conversion raises `ESxConversionError`.

### Nested tables: `OptionsTable.ConvertNestedTables`

With the new option `OptionsTable.ConvertNestedTables` (False by default,
and it needs `ConvertTables`), a table nested in a table cell is converted
into a `<table>` inside the `<td>`. Without it, the output is as before: a
nested table's cells are separated by `TabString`, inside the outer cell.

With the option on, a post-process listener (`PostProcessEventListeners`)
that walks the document has to expect a `TTableNode` among the children of
a `TTableCellNode`, not only paragraphs (`TTableNode.Level` is 2 or more
for such a table). Code that casts every child of a cell to
`TParagraphNode` raises an `EInvalidCast` on a document with nested
tables. With the option off, a cell holds paragraphs only, as before.

```pascal
for I := 0 to Cell.ChildCount - 1 do
  if Cell.ChildNodes[I] is TParagraphNode then
    ... // a paragraph
  else if Cell.ChildNodes[I] is TTableNode then
    ... // a nested table: rows, cells, paragraphs and maybe more tables
```

### Font replacement: `FontNameMatching`

`ReplaceFonts` rules match font names by prefix: the default rule
`Arial=Arial,sans-serif` also turns `Arial Unicode MS` and `Arial Narrow`
into `Arial,sans-serif`, so Japanese text set in Arial Unicode MS asks the
browser for a font without Japanese glyphs. With the new property
`FontNameMatching := fnmExact`, a rule matches the whole font name
(ignoring the case of A to Z), the first matching rule wins, and a rule
name ending in `*` matches by prefix (`Arial*=Arial,sans-serif` keeps the
old behaviour for one rule). The default, `fnmPrefix`, is the 11.1
behaviour; it is planned to change to `fnmExact` in 12.0.

The default `ReplaceFonts` list has two more rules,
`Courier New=Courier New,monospace` and
`Times New Roman=Times New Roman,serif`, so that these fonts keep their
generic family with `fnmExact`. With `fnmPrefix` they change no output.
Code that reads the default list by index, or expects four entries, sees
six; code that assigns its own list is not affected.

With `fnmExact`, each font name goes through the rules, or `OnReplaceFont`,
once. With `fnmPrefix` it goes through them twice, the second time as the
first pass left it (quoted, with its generic family), and `OnReplaceFont`
is also called with an empty name; an `OnReplaceFont` handler written for
that may need a look when you switch.

## Migrating to 11.1

11.1 changes no API and no defaults. Nine changes can affect your own code.

### A link with formatted text is one `<a>`

A hyperlink whose text changes formatting, for example a bold word in it,
was written as one `<a>` per text run. It is one `<a>` around the whole
link text now:

```html
<!-- 11.0 -->
<a href="https://example.com/">see </a><span style="font-weight:bold;"><a href="https://example.com/">this</a></span>
<!-- 11.1 -->
<a href="https://example.com/">see <span style="font-weight:bold;">this</span></a>
```

A link whose text has one formatting is written as before. If you set
your own `HyperlinkURIBuilder`, `BuildLinkElement` is called once for
such a link, not once per run. The text node it gets is a temporary one:
its `FieldExpression` is the link's, and its `TextContent` is the
(already encoded) markup of the runs, `see <span ...>this</span>`, so
don't encode it again. `OnBeforeEncode` and `OnAfterEncode` are still
called per text run; for the runs of such a link, `TextContent` no longer
contains the `<a>` element in `OnAfterEncode`. There is no switch for the
old output.

### Text in Wingdings and other symbol fonts is written as character references

Text in a symbol font other than Symbol (Wingdings, Wingdings 2/3,
Webdings, any other `\fcharset2` font) was written with each byte as the
character of the same value, so bytes 0x80 to 0x9F became control
characters, which are not valid in HTML. Its bytes are now decoded through
the document's code page (`\ansicpg`, Windows-1252 by default) and every
non-ASCII character of such a run is written as a numeric character
reference:

```html
<!-- 11.0: byte 0x8C as the control character U+008C -->
<span style="font-family:Wingdings;">a&#x8C;</span>
<!-- 11.1 -->
<span style="font-family:Wingdings;">a&#338;</span>
```

(11.0 wrote U+008C as a raw character, shown here as `&#x8C;`.) A byte
the code page does not define becomes `&#65533;` (U+FFFD). In
`OnAfterEncode`, `TextContent` contains these references. Plain-text
output gets the decoded characters. The Symbol font is unchanged: it keeps
its mapping to Greek letters and symbols. There is no switch for the old
output.

### Plain text has no `<img>` tags

With `ConvertToPlainText` and `ConvertPictures` on, plain-text output had
an HTML `<img src=... />` tag where each picture was. It has nothing there
now: a paragraph with only a picture is an empty line. Pictures are still
passed to the `PictureAdapter`, as before. There is no switch for the old
output.

### Head values are escaped

`OptionsHead.DocumentTitle`, `MetaAuthor`, `MetaDescription`, `MetaKeywords`,
`MetaViewport`, the `MetaTags` names and values, and `StyleSheetLink` were
written as they are. A `&` in one of them made XHTML output not well-formed.
They are escaped now:

```html
<!-- DocumentTitle := 'A & B' in 11.0 -->
A & B
<!-- 11.1 -->
A &amp; B
```

If your code escapes these values itself, remove that: `A &amp; B` would now
come out as `A &amp;amp; B`. `HeadTags` is still written as it is. There is no
switch for the old output.

With XHTML, `StyleSheetInclude` and the default font style are escaped too, so
a `>` in a selector comes out as `&gt;`. An XML parser reads it back as `>`. So
do browsers, as long as the page is served as XML; with `text/html` a browser
does not decode entities in `<style>`, so avoid `<`, `>` and `&` in the CSS of
such a page. In both output types, `</style` in the style text comes out as
`<\/style`, and head values lose the characters XML does not allow.

### A style sheet link must be a relative, http or https URL

`OptionsHead.StyleSheetLink` is dropped, with a warning in the log, if it has
another scheme (`javascript:`, `data:`, `file:`), starts with `//`, or has
spaces or control characters. That also drops a Windows path such as
`C:\css\site.css`, and a name with a space such as `my site.css`: write
`my%20site.css`. There is no switch for the old output.

### Paragraph borders are on a `<div>`

A paragraph with a border (`\box`) had `border:thin solid black` in its
`<p>` style, whatever width, line style and colour the RTF gave it, and a
border on single sides (`\brdrt`, `\brdrb`, ...) was lost. The border now
has the RTF's width, line style and colour, single sides are converted, and
it is on a `<div>` around the paragraph, together with the paragraph's left
and right indent, which place the box:

```html
<!-- 11.0 -->
<p style="margin-left:20px;border:thin solid black;">A</p>
<!-- 11.1 -->
<div style="border:2px double #ff0000;margin-left:20px;">
<p>A</p>
</div>
```

Consecutive paragraphs with the same border and indent share one `<div>`,
as Word draws one box around them. A list item still gets no border.
`GetParagraphStyle` of a translator no longer writes a border. Paragraph
borders no longer depend on `ConvertTables`. If your CSS or code expects
the border on the `<p>`, select `div > p` or the `<div>` instead. There is
no switch for the old output.

### Two new resource limits

`OptionsSecurity.MaxNodeCount` (default 1,000,000) limits the nodes
(paragraphs, text runs, pictures, tables, rows, cells) a conversion builds,
and `OptionsSecurity.MaxOutputChars` (default 64 Mi characters) the length
of its result. A document past a limit raises `ESxResourceLimit`, in lenient
mode too. The defaults are far above any real document (the largest sample
needs 4,760 nodes and gives about 183,000 characters), so this only matters
if you convert very large documents. Raise the limits for them:

```pascal
Scrooge.OptionsSecurity.MaxNodeCount := 10000000;
Scrooge.OptionsSecurity.MaxOutputChars := 512 * 1024 * 1024;
```

### Tables collapse their cell borders

With `OptionsTable.ConvertCellStyles` (the default), every `<table>` has
`border-collapse:collapse` in its style, so two neighbouring cell borders
are one line, as RTF editors draw them, instead of two lines with a gap:

```html
<!-- 11.0 -->
<table style="width:192px;">
<!-- 11.1 -->
<table style="border-collapse:collapse;width:192px;">
```

If you added `table { border-collapse:collapse; }` yourself (for example
through `OptionsHead.StyleSheetInclude`), you can remove it. If your CSS
sets `border-collapse:separate` or `border-spacing` on the tables, the
inline style now wins; turn `ConvertCellStyles` off to style the tables
entirely yourself.

### A link without an allowed target is plain text

A hyperlink field whose target is dropped (a scheme not in
`OptionsSecurity.UriSchemeAllowList`, such as `javascript:` or `file:`, or
an empty target) keeps its text, without a link element, as in the Java
product:

```html
<!-- 11.0 -->
<a>Getting Started</a>
<!-- 11.1 -->
Getting Started
```

An `a` element without `href` is no link: browsers show it as plain text
and keyboard navigation skips it. *No switch.* A post-processor that looked
for an `a` without `href` finds nothing now; to keep such links, add their
scheme to `UriSchemeAllowList`.

## Migrating to 11.0

11.0 changes several defaults to match the Java product's 11.0, and adds
table conversion. Most changes alter the generated HTML for an unchanged
caller; each one can be reverted with a line of code, except where noted.

This section covers every change since 10.2, including those first
released in 11.0-MS1 and 11.0-MS2.

### Quick start: 10.x output with 11.0

```pascal
Scrooge.ConvertEmptyParagraphs := False;  // 11.0 default: True
Scrooge.FontSizeScale := fsPoint;         // 11.0 default: fsEM
Scrooge.ConvertTables := False;           // 11.0 default: True
Scrooge.ConvertLanguage := False;         // 11.0 default: True
Scrooge.OptionsSecurity.MaxGroupDepth := 4096;  // 11.0 default: 256
// if you use the data URI picture adapter without an explicit limit:
Scrooge.PictureAdapter := TMemoryPictureAdapterDataURI.Create(1);  // 11.0 default: 256 KiB
```

The automatic text colour, plain-text tabs and the content type check
(below) have no switch.

### Changed defaults

#### Empty paragraphs become line breaks

`ConvertEmptyParagraphs` now defaults to `True`: an empty RTF paragraph is
written as `<br>` (`<br />` in XHTML) instead of an empty `<p></p>`. This
also applies to empty paragraphs inside table cells.

*To keep the 10.x output:* set `ConvertEmptyParagraphs := False`.

#### Font sizes in em

`FontSizeScale` now defaults to `fsEM`. Font sizes are written relative to a
12pt base size: `font-size:1em` instead of `font-size:12pt`, `0.83em` instead
of `10pt`. With the browser's default size (16px = 12pt) the document looks
the same.

*To keep the 10.x output:* set `FontSizeScale := fsPoint`.

#### Larger pictures are embedded as data URIs

`TMemoryPictureAdapterDataURI` now embeds PNG and JPEG pictures up to
256 KiB in the HTML (`MAX_DATA_URI_SIZE_IN_KB`, was 1 KiB). Larger pictures,
and other formats, still get a file URL.

*To keep the 10.x output:* pass the old limit,
`TMemoryPictureAdapterDataURI.Create(1)`.

#### Automatic text colour is not written

Text in RTF's automatic colour (`\cf0`, a colour table entry without
`\red`/`\green`/`\blue`) no longer gets `color:#000000`; it inherits the
colour of the surrounding HTML. The `SCROOGE_DEFAULT_COLOR_IS_EMPTY` define,
which switched this on in 10.x, is gone.

The same goes for the default body style: `OptionsOptimize.DefaultFontColor`
now defaults to `''` (was `'#000000'`), so with
`OptionsOptimize.IncludeDefaultFontStyle` the `body` rule has no `color`.

*No switch for `\cf0`.* If you need black text, set it in your style sheet,
for example `OptionsOptimize.DefaultFontColor := '#000000'` together with
`IncludeDefaultFontStyle`, or a rule in `OptionsHead.StyleSheetInclude`.

#### Tables are converted

RTF tables become HTML tables (`ConvertTables`, default `True`, since
11.0-MS1). In 10.x a table row was written as one paragraph, its cells
separated by `TabString`.

*To keep the 10.x output:* set `ConvertTables := False`.

#### Text in another language gets a `lang` attribute

`ConvertLanguage` now defaults to `True`, as in the Java product. Text whose
RTF language (`\lang`) differs from the document language (`\deflang`, or
`DefaultLanguage` if set) gets a `lang` attribute (plus `xml:lang` in
XHTML), for example `<span lang="fr">`. This helps screen readers,
hyphenation and spell checking. Text in the document language gets none, and
the `lang` of the `html` element is unchanged. Where the language changes
inside a run, the run is split into two `span` elements.

*To keep the 10.x output:* set `ConvertLanguage := False`.

#### Lower group nesting limit

`OptionsSecurity.MaxGroupDepth` now defaults to 256 (was 4096), as in the
Java product. RTF nested deeper than that raises `ESxRtfGroupNestingError`.
Real documents nest far less deep: the deepest of the sample files, the RTF
specification itself, nests 126 groups, and the others 10 or less. Forms
saved with an earlier version keep their stored value.

*To keep the 10.x limit:* set `OptionsSecurity.MaxGroupDepth := 4096`.

### Plain text keeps tab characters

With `ConvertToPlainText`, a tab in the RTF is now written as a tab
character. In 10.x it was written as `TabString`, which is HTML (eight
`&nbsp;` entities by default). (X)HTML output still uses `TabString`.

*No switch.* If you replaced the entities in plain-text output yourself,
that step is no longer needed.

### A non-UTF-8 content type is rejected

The output is always UTF-8. A `OptionsHead.MetaContentType` whose `charset`
is anything else (for example `text/html; charset=ISO-8859-1`) now makes the
conversion raise `ESxConfigurationError`, a subclass of
`ESxConversionError`, for every document type. In 10.x the value was
ignored for HTML 5 and written unchanged for XHTML, where it mislabelled the
UTF-8 output. A content type without a `charset` parameter is still
accepted.

*No switch.* Remove the charset parameter, or set it to UTF-8.

### A link without an allowed target has no `href`

A hyperlink field whose target is dropped (a scheme not in
`OptionsSecurity.UriSchemeAllowList`, such as `javascript:` or `file:`, or
an empty target) is now written as `<a>text</a>` instead of
`<a href="">text</a>`, as in the Java product. An empty `href` links to the
current document.

*No switch.* A post-processor that looks for `href=""` must look for an `a`
element without `href` instead.

### Bookmark ids and internal link targets are mapped

With `ConvertAnchors`, the `id` of a bookmark anchor is the bookmark name
with every character other than `A-Z a-z 0-9 - _ : .` replaced by `_`
(`BookmarkToId` in `SxLinkURIBuilder.pas`), as in the Java product. Before,
the name was used as written, so a name with spaces gave an invalid `id`.
The target of an internal hyperlink (`HYPERLINK \l "name"`) is mapped the
same way, so `#my bookmark` becomes `#my_bookmark` instead of
`#my%20bookmark`. Each `id` is written once per document: a repeated
bookmark name, or a second name with the same `id`, gets no anchor.

`TTextNode.Anchor`, as an `OnBeforeEncode`/`OnAfterEncode` handler sees it,
holds the mapped ids of all bookmarks that start at that text, separated
by spaces, instead of the name of the last one.

*No switch.* Code that reads `Anchor` must split it at spaces; CSS or
links that use an `id` with characters other than the ones above must use
the mapped `id`.

### Strict mode rejects `\u` escapes XML does not allow

With `OptionsSecurity.StrictMode`, a `\u` escape for a character XML does
not allow now raises `ESxConversionError`: a value out of range, a C0
control character other than TAB / LF / CR, U+FFFE / U+FFFF, or a surrogate
without its other half. In 10.x strict mode dropped the first two silently,
like lenient mode, wrote U+FFFE / U+FFFF unchanged and a lone surrogate
as U+FFFD. Lenient mode (the
default) drops all of them.

*No switch.* Turn `StrictMode` off to drop such characters instead of
rejecting the document.

### API changes

#### Wrapped internal errors are `ESxUnexpectedError`

An exception from inside the conversion that is not an `ESxConversionError`
(an access violation, `ERangeError`, ...) still leaves `Convert` wrapped, with
the message `<ClassName>: <message>`, but as the new `ESxUnexpectedError`, a
subclass of `ESxConversionError`. `on E: ESxConversionError` handlers are
unaffected; only code that compares the exact class
(`E.ClassType = ESxConversionError`) sees a difference.

#### `fsEX` is removed, `fsPX` is new

`TFontSizeScale` is now `(fsPoint, fsEM, fsPX, fsPercent)`. Code that uses
`fsEX` no longer compiles; use `fsEM`. `fsPX` writes sizes in CSS pixels
(1pt = 4/3px).

`fsEM` and `fsPercent` values are also more exact now: they keep half-point
sizes and round like the Java product (10.5pt is `0.88em`, was `0.92em`).

#### `ISxWriterOptions` has a new method

`ISxWriterOptions` (`SxInterfaces.pas`) has a new `GetConvertToPlainText`
method. This only matters if your own class implements that interface;
`TSxBase` implements it already.

#### Post-processors see table nodes

A `PostProcessEventListeners` listener that walks the document must expect
table nodes: a top-level child is a paragraph *or* a `TTableNode`
(`SxDocumentTable.pas`), whose rows hold cells, whose cells hold paragraphs.
`Doc.ChildNodes[I] as TParagraphNode` is no longer safe. Check the node class
(`Node is TTableNode`) and recurse, or set `ConvertTables := False` so no
table nodes are built.

#### DOM node members follow the W3C DOM

The members of `TSimpleDomNode` / `ISimpleDomNode` (`SxDocumentNode.pas`),
which post-processors and `OnBeforeEncode`/`OnAfterEncode` handlers use to
walk the document, now have W3C DOM names. The old names still compile,
with a deprecation warning, and will be removed in a later major version:

| Before 11.0 | 11.0 |
|---|---|
| `Item[I]` | `ChildNodes[I]` (the default property, so `Node[I]` works too) |
| `GetCount` | `ChildCount` |
| `Append(Node)` | `AppendChild(Node)` |
| `Extract(Node)` | `RemoveChild(Node)`: unlinks the node, the caller frees it |
| `Remove(Node)` | `DeleteChild(Node)`: unlinks the node and releases the parent's reference, which frees it unless you still hold an interface reference |

`Remove` is renamed to `DeleteChild`, not `RemoveChild`, because the W3C
`removeChild` returns the node instead of destroying it; `RemoveChild`
does the same here.

New: `ParentNode`, `FirstChild` and `HasChildNodes`. `LastChild`,
`PreviousSibling` and `NextSibling` are now properties (the call syntax is
unchanged).

Behaviour changes, all of which used to crash or corrupt the tree:

- `FirstChild`/`LastChild` return `nil` for a node without children
  (`LastChild` raised an exception before).
- `PreviousSibling`/`NextSibling` return `nil` for a node without a parent,
  such as the document or a node removed with `RemoveChild` (an access
  violation before).
- `AppendChild` moves a node that is still a child of another node, as in
  the W3C DOM. Before, it stayed in both child lists and was freed twice.
- A node in the tree is referenced by its parent, so an interface reference
  to a node (`Supports(Node, ISimpleDomTableRowNode, Row)`, or keeping the
  `OnBeforeEncode` text node in a variable) is safe. Before, dropping that
  reference freed the node while it was still in the tree, and the
  conversion failed with an access violation or `EInvalidPointer` that was
  not wrapped as `ESxUnexpectedError`. A node lives until it is removed and
  the last interface reference to it is dropped, so a node you keep after
  the conversion keeps its child nodes in memory too.
- `Free` on a node that is still in the tree raises `EInvalidPointer` at
  once and leaves the tree intact; before, it freed the node, which the tree
  freed a second time later. Use `DeleteChild`, or `RemoveChild` and then
  `Free`.
- `RemoveChild` and `DeleteChild` of a node that is not a child of this node
  leave it alone, and `RemoveChild` returns `nil`. Before, `DeleteChild`
  freed the node anyway, even if it was in another tree.

The constructor parameter of the node classes is now named `AParent`
(was `AOwner`).

### Forms and data modules (`.dfm` / `.lfm`)

The published properties have no `default` specifier, so the IDE stored
their values when the component was placed on a form. Such forms keep the
values stored with 10.x, for example `ConvertEmptyParagraphs = False` and
`FontSizeScale = fsPoint`, and produce the 10.x output for those settings
until you change them in the Object Inspector.

A form that stored `FontSizeScale = fsEX` fails to load ("Invalid property
value"). Change the value to `fsEM` in the `.dfm`/`.lfm` file (as text)
before opening the form.

### Deprecation list

- `TSimpleDomNode` / `ISimpleDomNode`: `Item`, `GetItem`, `GetCount`,
  `Append`, `Extract` and `Remove`; see "DOM node members follow the W3C
  DOM" above.

## Breaking changes and deprecations in 10.0

### Removed features

#### Hyperlink auto-detection

Blue and underlined text is no longer used for hyperlink detection. It was
deprecated in version 9.0 and its code has now been removed.

#### Deprecated document types

Support for the deprecated document types was withdrawn in version 9.0; the
implementation code was kept but fenced behind the
`SCROOGE_DEPRECATED_DOCUMENT_TYPES` conditional define. Version 10 removes
that code entirely. The define is now obsolete and has no effect.

### Deprecation list

No new deprecations were added in version 10.0.

## Breaking changes and deprecations in 9.0

### Removed elements

#### Document types

These output document types were deprecated and have been removed or
commented out:

- XHTML Basic 1.0
- XHTML Mobile Profile 1.0
- XHTML 1.0 Transitional
- XHTML 1.1
- HTML 4.01 Strict and Transitional
- HTML 3 Flex

### Changed requirements

#### Compiler requirements

- Minimum supported Free Pascal version is now 3.2.2.

### Minor changes

#### Moved units

All units which are experimental, complimentary or unsupported have been
moved to the new folder `source/optional`.

#### MemoryPictureAdapterDataURI

The threshold for switching to data URI has been set to 1 kilobyte. Bigger
pictures will not be converted to data URIs. (11.0 raises it to 256 KiB, see
above.)

#### Moved types and constants

Several types and constants related to embedded pictures moved from unit
`SxTypes` to `SxEmbeddedPicture`.

### Deprecation list

- Blue and underlined text will no longer be considered a hyperlink in the
  next major release (10.0).
