The ErrorString() and SafeString() docutils functions were helpers meant to
ease the handling of encodings during the Python 3 transition. There is no
real need for them after Python 3.6, and docutils 0.22 removes them,
breaking the docs build
Handle this by just injecting our own one-liner version of ErrorString(),
and removing the sole SafeString() call entirely.
Reported-by: Zhixu Liu <zhixu.liu@gmail.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <87ldmnv2pi.fsf@trenco.lwn.net>
While the original code came from the Sphinx Include class,
such class is monolithic: it has only one function that does
everything, and 3 variables that are used:
- required_arguments
- optional_arguments
- option_spec
So, basically those are the only members that remain from
the original class, but hey! Those are the same vars that every
other Sphinx directive extension has to define!
In summary, keeping inheritance here doesn't make much sense.
Worse than that, kernel-include doesn't support the current set
of options that the original Include class has, but it also
has its own set of options.
So, let's fill in the argument vars with what it does
support, dropping the rest.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/a9f2eebf11c6b0c3a2e3bf42e71392cdfd2835d1.1755872208.git.mchehab+huawei@kernel.org
It is best to point to the original line of code that generated
an error than to point to the beginning of a directive.
Add support for it. It should be noticed that this won't work
for literal or code blocks, as Sphinx will ignore it, pointing
to the beginning of the directive. Yet, when the output is known
to be in ReST format, like on TOC, this makes the error a lot
more easier to be handled.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/a0953af8b71e64aaf2e0ba4593ad39e19587d50a.1755872208.git.mchehab+huawei@kernel.org
kernel_include extension was originally designed to be used by the
media comprehensive uAPI documentation, where, instead of simpler
kernel-doc markups, the uAPI documentation is enriched with a larger
text, with images, complex tables, graphs, etc.
There, we wanted to include the much simpler yet documented .h
file.
This extension is needed to include files from other parts of the
Kernel tree outside Documentation, because the original Sphinx
include tag doesn't allow going outside of the directory passed
via sphinx-build command line.
Yet, the cross-references themselves to the full documentation
were using a perl script to create cross-references against the
comprehensive documentation.
As the perl script is now converted to Phython and there is a
Python class producing an include-compatible output with cross
references, add two optional arguments to kernel_include.py:
1. :generate-cross-refs:
If present, instead of reading the file, it calls ParseDataStructs()
class, which converts C data structures into cross-references to
be linked to ReST files containing a more comprehensive documentation;
Don't use it together with :start-line: and/or :end-line:, as
filtering input file line range is currently not supported.
2. :exception-file:
Used together with :generate-cross-refs:. Points to a file containing
rules to ignore C data structs or to use a different reference name,
optionally using a different reference type.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/efc39c8e54a2056ae2fdb94d5006fcb19e227198.1755872208.git.mchehab+huawei@kernel.org
When printing --help, we'd like the name of the files
from __doc__ to match the displayed positional arguments at
both usage and argument description lines.
Use a custom formatter class to convert ``foo`` into ANSI SGR
code to bold the argument, if is TTY, and adjust the help
text to match the argument names.
Here on Plasma, that makes it display it colored, wich is
really cool. Yet, I opted for SGR, as the best is to follow
the terminal color schema for bold.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/2c1e61d1fb1b2a2838b443beee89c1528831997f.1755872208.git.mchehab+huawei@kernel.org
When the Kernel started to use Sphinx, we had to come up with
a solution to parse media headers. On that time, we didn't have
much experience with Sphinx extensions. So, we came up with our
own script-based solution that were basically implementing a
set of rules we used to have at the Makefile.
Convert it to Python, keeping it bug-compatible with the
original script.
While here, try to better document it.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/ae5cfa8dff37e280cc9493fc95a51cd0cc0ba127.1755872208.git.mchehab+huawei@kernel.org
As reported by Donald, this code:
rst_parser = RSTParser()
rst_parser.parse('\n'.join(result), document)
breaks line parsing. As an alternative, I tested a variant of it:
rst_parser.parse(result, document)
but still line number was not preserved. As Donald noted,
standard Parser classes don't have a direct mechanism to preserve
line numbers from ViewList().
So, instead, let's use a mechanism similar to what we do already at
kerneldoc.py: call the statemachine mechanism directly there.
I double-checked when states and statemachine were introduced:
both were back in 2002. I also tested doc build with docutils 0.16
and 0.21.2. It worked with both, so it seems to be stable enough
for our needs.
Reported-by: Donald Hunter <donald.hunter@gmail.com>
Closes: https://lore.kernel.org/linux-doc/m24ivk78ng.fsf@gmail.com/T/#u
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Instead of printing line numbers from the temp converted ReST
file, get them from the original source.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Add a simple sphinx.Parser to handle yaml files and add the
the code to handle Netlink specs. All other yaml files are
ignored.
The code was written in a way that parsing yaml for different
subsystems and even for different parts of Netlink are easy.
All it takes to have a different parser is to add an
import line similar to:
from doc_generator import YnlDocGenerator
adding the corresponding parser somewhere at the extension:
netlink_parser = YnlDocGenerator()
And then add a logic inside parse() to handle different
doc outputs, depending on the file location, similar to:
if "/netlink/specs/" in fname:
msg = self.netlink_parser.parse_yaml_file(fname)
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
The logic there which adds a dependency note to Sphinx cache
is not taking into account that the build dir may not be
the source dir. This causes a performance regression:
$ time make O=/tmp/foo SPHINXDIRS=admin-guide htmldocs
[OUTDATED]
Added: set()
Changed: {'abi-obsolete', 'abi-removed', 'abi-stable-files', 'abi-obsolete-files', 'abi-stable', 'abi', 'abi-removed-files', 'abi-testing-files', 'abi-testing', 'gpio/index', 'gpio/obsolete'}
Removed: set()
All docs count: 385
Found docs count: 385
real 0m11,324s
user 0m15,783s
sys 0m1,164s
To get the root cause of the problem (ABI files reported as changed),
I used this changeset:
diff --git a/Documentation/conf.py b/Documentation/conf.py
index e8766e689c1b..ab486623bd8b 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -571,3 +571,16 @@ def setup(app):
"""Patterns need to be updated at init time on older Sphinx versions"""
app.connect('config-inited', update_patterns)
+ app.connect('env-get-outdated', on_outdated)
+
+def on_outdated(app, env, added, changed, removed):
+ """Track cache outdated due to added/changed/removed files"""
+ print("\n[OUTDATED]")
+ print(f"Added: {added}")
+ print(f"Changed: {changed}")
+ print(f"Removed: {removed}")
+ print(f"All docs count: {len(env.all_docs)}")
+ print(f"Found docs count: {len(env.found_docs)}")
+
+ # Just return what we have
+ return added | changed | removed
Reported-by: Akira Yokosawa <akiyks@gmail.com>
Closes: https://lore.kernel.org/linux-doc/c174f7c5-ec21-4eae-b1c3-f643cca90d9d@gmail.com/
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Tested-by: Akira Yokosawa <akiyks@gmail.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/e25673d87357457bc54ee863e97ff8f75956580d.1752752211.git.mchehab+huawei@kernel.org
Docutils emits a deprecation warning when the set_class() element method is
used; that warning disappears into the ether, but it also causes a crash
with docutils 0.19.
Avoid the deprecated function and just append directly to the "classes"
attribute like the documentation says instead.
Reported-by: Akira Yokosawa <akiyks@gmail.com>
Tested-by: Akira Yokosawa <akiyks@gmail.com>
Closes: https://lore.kernel.org/de7bae91-3200-481f-9db2-c0dc382c91dd@gmail.com/
Fixes: d6d1df92c2 ("docs: automarkup: Mark up undocumented entities too")
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
The automarkup code generates markup and a cross-reference link for
functions, structs, etc. for which it finds kerneldoc documentation.
Undocumented entities are left untouched; that creates an inconsistent
reading experience and has caused some writers to go to extra measures to
cause the markup to happen.
Mark up detected C entities regardless of whether they are documented.
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Remove a few declarations that are no longer doing anything now that we
have left Sphinx 2 behind.
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Several of the markup functions contain the same code, calling into
sphinx's pending_xref and resolve_xref functions to add and resolve a
cross-reference, with only a few of the parameters changed (domain,
reference type, markup content). Move this logic to its own function and
reuse it in the markup functions.
No functional change.
Signed-off-by: Nícolas F. R. A. Prado <nfraprado@collabora.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/20250408-automarkup-resolve-xref-helper-v2-1-e0a9b8fc7fdd@collabora.com
Documentation/driver-api/infiniband.rst has a kernel-doc tag
with "\" characters at the end:
.. kernel-doc:: drivers/infiniband/ulp/iser/iscsi_iser.c
:functions: iscsi_iser_pdu_alloc iser_initialize_task_headers \
iscsi_iser_task_init iscsi_iser_mtask_xmit iscsi_iser_task_xmit \
iscsi_iser_cleanup_task iscsi_iser_check_protection \
iscsi_iser_conn_create iscsi_iser_conn_bind \
iscsi_iser_conn_start iscsi_iser_conn_stop \
iscsi_iser_session_destroy iscsi_iser_session_create \
iscsi_iser_set_param iscsi_iser_ep_connect iscsi_iser_ep_poll \
iscsi_iser_ep_disconnect
This is not handled well, as the "\" strings will be just stored inside
Sphinx options.
While the actual problem deserves being fixed, better to relax the
keneldoc.py extension to silently strip "\" from the end of strings,
as otherwise this may cause troubles when preparing arguments to
be executed by kernel-doc.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/4c652d6c57b20500c135b95294e554d9e9a97f42.1744106242.git.mchehab+huawei@kernel.org
Sphinx logging system doesn't like warnings during module load,
as it understands that such logs are produced at the wrong time:
WARNING: while setting up extension automarkup: /sys/devices/system/cpu/cpuX/topology/physical_package_id is defined 2 times: /new_devel/v4l/docs/Documentation/ABI/stable/sysfs-devices-system-cpu:27; /new_devel/v4l/docs/Documentation/ABI/testing/sysfs-devices-system-cpu:70
WARNING: while setting up extension automarkup: /sys/devices/system/cpu/cpuX/topology/ppin is defined 2 times: /new_devel/v4l/docs/Documentation/ABI/stable/sysfs-devices-system-cpu:89; /new_devel/v4l/docs/Documentation/ABI/testing/sysfs-devices-system-cpu:70
So, use a function to allocate/process ABI files and use it to
be called at kernel_abi.py, as automarkup also needs it to
produce the right cross-references.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/b0e79dc60d556e3b39fa6774d3b7bf734b73f352.1739182025.git.mchehab+huawei@kernel.org
Now that all ABI files are handled together, we can add a feature
at automarkup for it to generate cross-references for ABI symbols.
The cross-reference logic can produce references for all existing
files, except for README (as this is not parsed).
For symbols, they need to be an exact match of what it is
described at the docs, which is not always true due to wildcards.
If symbols at /sys /proc and /config are identical, a cross-reference
will be used.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/0b97a51b68b1c20127ad4a6a55658557fe0848d0.1739182025.git.mchehab+huawei@kernel.org
