During development I often need to run a bunch of tests. Instead of
having some unwieldy shell command, I have added this xtask to help with
running the testing commands.
This rewrites the HTML rendering pipeline to use a tree data structure,
and implements a custom HTML serializer. The intent is to make it easier
to make changes and to manipulate the output. This should make some
future changes much easier.
This is a large change, but I'll try to briefly summarize what's
changing:
- All of the HTML rendering support has been moved out of
mdbook-markdown into mdbook-html. For now, all of the API surface is
private, though we may consider ways to safely expose it in the
future.
- Instead of using pulldown-cmark's html serializer, this takes the
pulldown-cmark events and translates them into a tree data structure
(using the ego-tree crate to define the tree). See `tree.rs`.
- HTML in the markdown document is parsed using html5ever, and then
lives inside the same tree data structure. See `tokenizer.rs`.
- Transformations are then applied to the tree data structure. For
example, adding header links or hiding code lines.
- Serialization is a simple process of writing out the nodes to a
string. See `serialize.rs`.
- The search indexer works on the tree structure instead of re-rendering
every chapter twice. See `html_handlebars/search.rs`.
- The print page now takes a very different approach of taking the
same tree structure built for rendering the chapters, and applies
transformations to it. This avoid re-parsing everything again. See
`print.rs`.
- I changed the linking behavior so that links on the print page
link to items on the print page instead of outside the print page.
- There are a variety of small changes to how it serializes as can be
seen in the changes to the tests. Some highlights:
- Code blocks no longer have a second layer of `<pre>` tags wrapping
it.
- Fixed a minor issue where a rust code block with a specific
edition was having the wrong classes when there was a default
edition.
- Drops the ammonia dependency, which significantly reduces the number
of dependencies. It was only being used for a very minor task, and
we can handle it much more easily now.
- Drops `pretty_assertions`, they are no longer used (mostly being
migrated to the testsuite).
There's obviously a lot of risk trying to parse everything to such a low
level, but I think the benefits are worth it. Also, the API isn't super
ergonomic compared to say javascript (there are no selectors), but it
works well enough so far.
I have not run this through rigorous benchmarking, but it does have a
very noticeable performance improvement, especially in a debug build.
I expect in the future that we'll want to expose some kind of
integration with extensions so they have access to this tree structure
(or some kind of tree structure).
Closes https://github.com/rust-lang/mdBook/issues/1736
This is a very simplistic utility to compare the output of different
versions of mdbook. This is useful when making changes to compare
real-world books to see what changes actually happen.
This is a variation of scripts that I have been using for a few years.
This could definitely use some improvements, but this seems like it
could be useful as-is.
This switches to using the tracing crate instead of log. Tracing
provides a lot of nice features which we can take advantage of moving
forward.
This also adjusts the output fairly significantly. This includes:
- Switched the environment variable from RUST_LOG to MDBOOK_LOG.
- Dropped the timestamp. I experimented with various different time
displays, but ultimately decided to omit it for now. I don't think
I've ever found it to be useful, and it takes up a very significant
amount of space. It could potentially be useful for basic profiling,
but I think there are other, better mechanisms for that. We could
consider leveraging tracing itself for doing some basic profiling
(like using something like tracing-chrome).
- Dropped the target unless MDBOOK_LOG is set. The target tends to be
pretty noisy, and doesn't really convey much information unless you
are debugging or otherwise trying to adjust the log output.
- Added color.
- Slightly reworked the way the error cause trace is displayed.
- Slightly changed the way html5ever filtering is done, as well as add
handlebars to the list since they both are very noisy. You can
override this now by explicitly listing them as targets.
I still expect that mdbook will eventually change how it displays things
to the console, possibly switching away from tracing and printing things
itself. However, that is a larger project for the future.
This changes `with_renderer` and `with_preprocessor` to replace any
extensions of the same name instead of just appending to the list. This
is necessary for rust-lang's build process, because we replace the
preprocessors with local ones. Previously, mdbook would just print an
error, but continue working. With the change that preprocessors are no
longer optional by default, it is now required that we have a way to
replace the existing entries.
This removes toml as a public dependency. This reduces the exposure of
the public API, reduces exposure of internal implementation, and makes
it easier to make semver-incompatible changes to toml.
This is accomplished through a variety of changes:
- `get` and `get_mut` are removed.
- `get_deserialized_opt` is renamed to `get`.
- Dropped the AsRef for `get_deserialized_opt` for ergonomics, since
using an `&` for a String is not too much to ask, and the other
generic arg needs to be specified in a fair number of situations.
- Removed deprecated `get_deserialized`.
- Dropped `TomlExt` from the public API.
- Removed `get_renderer` and `get_preprocessor` since they were trivial
wrappers over `get`.
This updates everything for the move of config to mdbook-core. There
will be followup commits that will be moving and refactoring the config.
This simply moves it over unchanged.
This updates everything for the move of utils to mdbook-core. There will
be followup commits that will be moving and refactoring these utils.
This simply moves them over unchanged (except visibility).
This moves Result and Error to mdbook-core with the anticipation of
using them in user crates. For now, the internal APIs will be using
anyhow directly, but the intent is to transition more of these to
mdbook-core where it makes sense.
These are two new crates intended to support implementing preprocessors
and renderers. Currently these stubs just have MDBOOK_VERSION, but
future commits will migrate more code to these crates.
This is intended as a shared, internal library that will be used by
other mdbook crates. The intention is that those crates will either
directly use, or reexport items from this crate.
Initially this includes MDBOOK_VERSION, which will get reexported from
the preprocessor and renderer crates.
