duckeye

How duckeye reads

Every document is a tree: a page holds sections, a section holds paragraphs, a paragraph holds words. duckeye keeps that tree, but it stores it the way a database likes best, as rows.

Two old ideas

A document tree between formats. Pandoc converts documents with a set of readers and a set of writers. A reader turns its format into one shared tree (an abstract syntax tree), and a writer turns that tree into another format. So a new format needs only one new reader or writer.

A database made of trees. IBM built IMS from 1966 to track the parts of the Saturn V rocket for the Apollo program. Its hierarchical model stores data as trees, and each child record has one parent. That model lost ground as Codd's relational model, with its flat tables, became the standard.

duckeye puts the two together. Its readers turn every format into one shared tree, as Pandoc does. Then that tree goes into a relational table in DuckDB, a columnar database that grows through extensions. The tree is still there, and every table tool works on it.

One document, two shapes

Take this short Markdown file:

# Pond survey

Ducks seen at **dawn**.

- Mallard
- Teal

Here is the same document twice: as a tree on the left, and as the columns DuckDB stores on the right. Each node sits on the line of its own row.

order level content document heading01Pond surveyparagraph11NULLtext22Ducks seen atbold32dawntext42.list51NULLlist_item62Mallardlist_item72Teal
A columnar store keeps each column on its own. Two columns hold the whole tree: the row order is the reading order, and level is the depth, the same as the indent on the left. The node names on the left are a fourth column, element_type. Each node has one parent, as in a hierarchical database.

Here are all of the rows, exactly as DuckDB's markdown extension reads it:

SELECT element_order, kind, element_type, level, content
FROM read_markdown_blocks('pond.md');
element_orderkindelement_typelevelcontent
0blockheading1Pond survey
1blockparagraph1NULL
2inlinetext2Ducks seen at
3inlinebold2dawn
4inlinetext2.
5blocklist1NULL
6blocklist_item2Mallard
7blocklist_item2Teal

Read the rows top to bottom and you walk the tree. The diagram's order column is element_order. DuckDB tables are ordered too: by default, a filter or a limit keeps the rows in their original order (order preservation). Joins and grouping make no such promise, so the order is also a column, and ORDER BY element_order puts it back. The heading's own level, 1 for #, is a separate attribute. These rows are duck_blocks: the duck_blocks specification defines seven fields, and every reader in the family produces them.

Because a document is now a table, a plain query answers a question about it:

SELECT content FROM read_markdown_blocks('pond.md')
WHERE element_type = 'list_item';
-- Mallard
-- Teal

In duckeye, Data mode shows any document as this table, and Doc mode shows the same rows as a page.

Every reader speaks the same rows

webbedHTML markdownMarkdown panduckDOCX, EPUB… duck_blocks one row per block, in reading order utilscontents,sections, text writersMarkdown,HTML,pandoc JSON duckeyeDoc, Data,Viz, Find
Readers write the rows; anything that reads the rows works with every format.

Since every reader writes the same rows, readers and writers join with no converter between them. This line, from panduck's architecture notes, turns a web page into Markdown:

LOAD webbed; LOAD markdown;
SELECT duck_blocks_to_md(html_to_duck_blocks(content))
FROM read_text('page.html');

Code is a tree too

sitting_duck reads source code into rows of its syntax tree, with a depth and a parent_id for each node. It knows 28 languages, and duckeye's engine carries 10 of them: Python, JavaScript, TypeScript, Java, C, C++, Rust, Go, Bash and SQL. Today duckeye shows code as numbered lines; a CSS-style search over the syntax tree is coming.

duckent, still at the design stage, names the idea under all of this: "Your ASTs were rows all along." A tree stored in its own depth-first order needs only two things to be searchable: the row order and a nesting column. One selector language could then search code, documents and any other tree the same way.

The source

Every part is open source. duckeye for Android runs these extensions inside the app, on your phone, and the duckeye command-line tool runs them in your terminal.