Repository navigation
fix(pandoc): Merge project metadata with document front matter - #14996
Conversation
Before calling pandoc, runPandoc re-applies the executed front matter over the merged metadata so knitr inline R results in YAML reach the output. Several guards protect values Quarto computes itself from that overwrite. These tests pin today's behavior ahead of changing how that step merges values (#11139): - a key set at top level and under `format: html:` keeps the format value - knitr inline R in YAML is evaluated for title, a custom key, and a nested key - the same inline R values survive a render that reuses frozen results - a book chapter title keeps its computed chapter number - website categories from project, directory and document all show up They pass on main.
Metadata handed to pandoc starts from the fully resolved options.format.metadata and then overwrites individual top-level keys with the document's own front matter, re-read after execution so that inline expressions are resolved. For a key defined in both _quarto.yml and the document, that assignment discarded everything the project file contributed, even though options.format.metadata had merged it correctly. Merging the two instead is not enough: options.format.metadata already contains the document's unresolved values, so concatenating the resolved array on top leaves an inline expression in the result next to the value it evaluated to. Treat options.format.metadata as the correct merge and substitute only the leaves that execution actually changed, comparing against the front matter as written, which is now passed through as PandocOptions.unexecutedMetadata. Keys the engine did not touch keep their merged value, so mappings keep the project's keys and arrays keep the project's entries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…g leaves Substituting only the leaves execution changed, compared against the resolved metadata, also changes precedence: options.format.metadata merges the document first and metadata-files / --metadata-file after it, so for a key the engine left alone the metadata file's value won over the front matter. Document front matter winning over its own metadata files is the documented behavior and what the overwrite before pandoc has produced so far. The leaf substitution also duplicated array entries when an inline expression evaluated to a value the project already listed. For a key execution left unchanged, merge the front matter as written back on top of the resolved value: other sources keep the leaves and list entries the document does not set, and the document still wins the ones it does. The merge is wrapped in an object because mergeConfigs on two bare arrays merges them by index. A key that holds an inline expression takes the executed value as a whole, as before; the project's children of such a key are still lost, a known limit of working per top-level key. Putting the document's value back means a top-level `controls: auto` or `previewLinks: auto` would again replace the `false` that the revealjs format computes for them, and reveal.js receives a bare `auto` identifier (already the case before this change). Skip both keys for revealjs, like `theme`. The test fixture is split into a markdown and a knitr document, since with an inline expression in every shared key the old fixture renders as before. knitr only accepts scalar params, so the params case uses flat keys.
…matter The revealjs format replaces only a non-boolean `controls` or an 'auto' `previewLinks` with `false`. Skipping the keys for every value also skipped a boolean set in the front matter, so a `metadata-files` value for the same key, merged after the document, won over it.
… to auto The revealjs guard added in this PR fixes a separate user-visible failure with no issue of its own: a top-level `controls: auto` or `previewLinks: auto` was written as a bare `auto` identifier into the reveal.js initialization script, a ReferenceError that stops the presentation from loading. The entry references this PR instead.
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
The document-level metadata-files case was covered, the command line flag was not. Both go through the same merge, but a --metadata-file is read separately from the document, so its precedence and list handling are pinned on their own: the document wins a shared value, what only the file defines is kept, and the file's list entries come before the document's.
The website-categories fixture only asserted on categories, which already merged correctly before the fix, so it could not catch the overwrite reported in #9864. An author list in the directory _metadata.yml combined with a scalar author in the post front matter did regress: only the document author reached the page. Extend the fixture so the post must emit both authors. Against the pre-fix runPandoc the from-directory author meta tag is missing; with the fix both are present.
|
I am spending time on this to understand all possible side effect. This is not without impact IMO. |
Before the fix, keywords, tags and other keys passed through the re-apply step replaced the project's list with the document's, so a document could also clear an inherited list with an empty value. Lists now combine for every key, as Quarto's own keys such as css already did, so the entry says so.
|
While checking what changes for a document that sets an inherited list to an empty value, I compared main and this branch on the same documents. This PR does not change how project, directory and How I checked: a project
A list key outside the schema behaves like The same values set under The side effect is that a document can no longer clear an inherited list with an empty value, and I only ran the html format with the markdown engine on dev builds, so knitr and other formats are not covered by this table. |
* docs: describe how metadata from different sources is merged With quarto-dev/quarto-cli#14996, a document's front matter is merged with _quarto.yml, _metadata.yml and metadata-files instead of replacing their values for each top-level key it sets. The section only said that objects and arrays are merged, which left out how lists are combined, what happens to a single value merged with a list, and which of an included file or its parent wins. Pandoc's --metadata-file was not documented anywhere on the site, and it merges with the document in its own order, so it gets a short note. * docs: say a document cannot remove inherited list entries Lists from the project and a directory always combine with the document's list, so a document has no way to drop their entries. A single value also replaces an object set at a lower level, which the section did not say.
3 participants
Description
Continues #13353, whose author handed the branch over (his commits are kept with authorship). A key defined both in a document's front matter and in
_quarto.ymlormetadata-filesreaches Pandoc templates and Lua filters with only the document's value. Settingparams.nested.Ain a document dropsparams.nested.Bcoming from the project, and a project array such askeywordsloses its project entries. This was first reported in discussion #11138 and is #11139. The same loss hits an author list from a directory_metadata.ymlcombined with an author in a post's front matter, reported in #9864.options.format.metadataalready holds the correct merge. The loss happens inrunPandoc, which re-applies each top-level key of the executed front matter with a plain assignment so that knitr inline R in YAML is resolved in the output (added in 2021 for that purpose). The assignment replaces whatever the project contributed to the key.For a key that execution left unchanged, the front matter as written is now merged on top of the resolved value. Other sources keep the leaves and list entries the document does not set, and the document still wins the ones it does. The input's metadata as written is passed to
runPandocasunexecutedMetadatato tell unchanged keys from keys holding inline expressions. A key with an inline expression takes the executed value as a whole, so the project's children of such a key are still lost. This is a known limit of working per top-level key.Lists now combine the same way for every key. Before,
keywords,tagsand other keys passed through here replaced the project's list with the document's, while Quarto's own keys such ascssalready combined, as the documentation says. As a side effect, a document can no longer clear an inherited list with an empty value (keywords: ""or[]), which only worked because of the re-apply step. There is no replacement for it for now, the same as forcss.Merging the front matter on top, rather than substituting the leaves execution changed, keeps the documented precedence:
options.format.metadatamerges the document first andmetadata-filesafter it, so a leaf substitution would let a metadata file win over the front matter. Substituting leaves also duplicated an array entry when an inline expression evaluated to a value the project already listed. The merge wraps the values in an object becausemergeConfigson two bare arrays merges them by index.Putting the document's value back means a top-level
controls: autoorpreviewLinks: autowould replace thefalsethat the revealjs format computes for them, and reveal.js would receive a bareautoidentifier, which was already the case before this change. Both keys are skipped for revealjs when the value is not a boolean, liketheme, and a boolean set in the front matter still wins overmetadata-files.The first commit adds characterization tests, passing on
main, for what the re-apply step already protects: format-level keys, knitr inline R in YAML (also with frozen results), book chapter titles and website categories. The remaining tests cover the merge itself: project and document keys,metadata-files, the--metadata-fileoption, an author list from a directory_metadata.ymlcombined with a document author, and revealjscontrolsandpreviewLinks.Fixes #11139, fixes #9864, supersedes #13353
Checklist
I have (if applicable):
AI-assisted PR
Note: autonomous AI agents submitting PRs without human oversight are not permitted — see the Code of Conduct.