How duckeye reads
Every document is a tree: a page holds sections, a section holds paragraphs, a paragraph holds words. duckeye keeps that tree, but it stores it the way a database likes best, as rows.
Two old ideas
A document tree between formats. Pandoc converts documents with a set of readers and a set of writers. A reader turns its format into one shared tree (an abstract syntax tree), and a writer turns that tree into another format. So a new format needs only one new reader or writer.
A database made of trees. IBM built IMS from 1966 to track the parts of the Saturn V rocket for the Apollo program. Its hierarchical model stores data as trees, and each child record has one parent. That model lost ground as Codd's relational model, with its flat tables, became the standard.
duckeye puts the two together. Its readers turn every format into one shared tree, as Pandoc does. Then that tree goes into a relational table in DuckDB, a columnar database that grows through extensions. The tree is still there, and every table tool works on it.
One document, two shapes
Take this short Markdown file:
# Pond survey Ducks seen at **dawn**. - Mallard - Teal
Here is the same document twice: as a tree on the left, and as the columns DuckDB stores on the right. Each node sits on the line of its own row.
element_type. Each node has one parent, as in a hierarchical database.Here are all of the rows, exactly as DuckDB's markdown extension reads it:
SELECT element_order, kind, element_type, level, content
FROM read_markdown_blocks('pond.md');
| element_order | kind | element_type | level | content |
|---|---|---|---|---|
| 0 | block | heading | 1 | Pond survey |
| 1 | block | paragraph | 1 | NULL |
| 2 | inline | text | 2 | Ducks seen at |
| 3 | inline | bold | 2 | dawn |
| 4 | inline | text | 2 | . |
| 5 | block | list | 1 | NULL |
| 6 | block | list_item | 2 | Mallard |
| 7 | block | list_item | 2 | Teal |
Read the rows top to bottom and you walk the tree. The diagram's order column is element_order. DuckDB tables are ordered too: by default, a filter or a limit keeps the rows in their original order (order preservation). Joins and grouping make no such promise, so the order is also a column, and ORDER BY element_order puts it back. The heading's own level, 1 for #, is a separate attribute. These rows are duck_blocks: the duck_blocks specification defines seven fields, and every reader in the family produces them.
Because a document is now a table, a plain query answers a question about it:
SELECT content FROM read_markdown_blocks('pond.md')
WHERE element_type = 'list_item';
-- Mallard
-- Teal
In duckeye, Data mode shows any document as this table, and Doc mode shows the same rows as a page.
Every reader speaks the same rows
- webbed reads HTML and XML. markdown reads Markdown.
- panduck reads ten formats natively, inside DuckDB: RTF, DOCX, ODT, EPUB, LaTeX, Org, reStructuredText, Jupyter notebooks, MediaWiki and Textile. It needs no pandoc program, yet each run of its tests checks its output against a real pandoc. It can also write the rows back out as pandoc's own JSON.
- duck_block_utils holds the specification and the tools that work on any rows: a table of contents, one section, plain text, Markdown, and validation.
Since every reader writes the same rows, readers and writers join with no converter between them. This line, from panduck's architecture notes, turns a web page into Markdown:
LOAD webbed; LOAD markdown;
SELECT duck_blocks_to_md(html_to_duck_blocks(content))
FROM read_text('page.html');
Code is a tree too
sitting_duck reads source code into rows of its syntax tree, with a depth and a parent_id for each node. It knows 28 languages, and duckeye's engine carries 10 of them: Python, JavaScript, TypeScript, Java, C, C++, Rust, Go, Bash and SQL. Today duckeye shows code as numbered lines; a CSS-style search over the syntax tree is coming.
duckent, still at the design stage, names the idea under all of this: "Your ASTs were rows all along." A tree stored in its own depth-first order needs only two things to be searchable: the row order and a nesting column. One selector language could then search code, documents and any other tree the same way.
The source
Every part is open source. duckeye for Android runs these extensions inside the app, on your phone, and the duckeye command-line tool runs them in your terminal.
- duck_block_utils: the duck_blocks specification and its tools
- panduck: document formats in, and pandoc JSON out
- webbed: HTML and XML
- markdown: Markdown
- sitting_duck: source code as syntax-tree rows
- duckent: tree semantics over ordered rows (design stage)