Skip to content

Conversation

JasperCraeghs
Copy link
Contributor

@JasperCraeghs JasperCraeghs commented Jun 28, 2023

Needed to support XML output generated by Doxygen 1.9.7, which no longer contains duplicate member definitions (memberdef). The XML of the source file now contains a member reference (member) and only the XML of the group contains the member definition.

I tested these changes on large, internal projects that use Doxygen groups and the doxygenfunction directive. I also added a regression test: 5b23a92

I am not confident that all of the changes I made are strictly necessary.

Closes #923

@JasperCraeghs JasperCraeghs changed the title Support member references for members in group (Doxygen 1.9.7) Support members in a Doxygen 1.9.7 group Jun 28, 2023
@D4N
Copy link
Contributor

D4N commented Jul 23, 2023

I have tried your PR hoping to fix #935 as well, but instead building the documentation with make html fails differently with doxygen 1.9.7 on Fedora Rawhide:

> /builddir/build/BUILD/breathe-4.35.0/breathe/renderer/filter.py(351)__call__()                                                                                                             
-> return getattr(self.selector(node_stack), self.attribute_name)                                                                                                                            
(Pdb)                                                                                                                                                                                        
make[1]: Leaving directory '/builddir/build/BUILD/breathe-4.35.0/documentation'                                                                                                              
fatal: not a git repository (or any of the parent directories): .git                                                                                                                         
Exception occurred while building, starting debugger:                                                                                                                                        
Traceback (most recent call last):                                                                                                                                                           
  File "/usr/lib/python3.12/site-packages/sphinx/cmd/build.py", line 285, in build_main                                                                                                      
    app.build(args.force_all, args.filenames)                                                                                                                                                
  File "/usr/lib/python3.12/site-packages/sphinx/application.py", line 353, in build
    self.builder.build_update()
  File "/usr/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 311, in build_update
    self.build(to_build,
  File "/usr/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 328, in build
    updated_docnames = set(self.read())
                           ^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 435, in read
    self._read_serial(docnames)
  File "/usr/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 456, in _read_serial
    self.read_doc(docname)
  File "/usr/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 512, in read_doc
    publisher.publish()
  File "/usr/lib/python3.12/site-packages/docutils/core.py", line 224, in publish
    self.document = self.reader.read(self.source, self.parser,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/sphinx/io.py", line 108, in read
    self.parse()
  File "/usr/lib/python3.12/site-packages/docutils/readers/__init__.py", line 76, in parse
    self.parser.parse(self.input, document)
  File "/usr/lib/python3.12/site-packages/sphinx/parsers.py", line 80, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/usr/lib/python3.12/site-packages/docutils/parsers/rst/states.py", line 169, in run
    results = StateMachineWS.run(self, input_lines, input_offset,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/docutils/statemachine.py", line 233, in run                                                                                                        
    context, next_state, result = self.check_line(
                                  ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/docutils/statemachine.py", line 445, in check_line
    return method(match, context, next_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/docutils/parsers/rst/states.py", line 2785, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/usr/lib/python3.12/site-packages/docutils/parsers/rst/states.py", line 325, in section
    self.new_subsection(title, lineno, messages)
  File "/usr/lib/python3.12/site-packages/docutils/parsers/rst/states.py", line 391, in new_subsection
    newabsoffset = self.nested_parse(
                   ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/docutils/parsers/rst/states.py", line 279, in nested_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/usr/lib/python3.12/site-packages/docutils/parsers/rst/states.py", line 195, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/docutils/statemachine.py", line 233, in run
    context, next_state, result = self.check_line(
                                  ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/docutils/statemachine.py", line 445, in check_line
    return method(match, context, next_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/docutils/parsers/rst/states.py", line 2355, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/docutils/parsers/rst/states.py", line 2367, in explicit_construct
    return method(self, expmatch)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/docutils/parsers/rst/states.py", line 2104, in directive
    return self.run_directive(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/docutils/parsers/rst/states.py", line 2154, in run_directive
    result = directive_instance.run()
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/breathe-4.35.0/breathe/directives/content_block.py", line 74, in run
    contents_finder.filter_(filter_, contents)
  File "/builddir/build/BUILD/breathe-4.35.0/breathe/finder/factory.py", line 55, in filter_
    item_finder.filter_([_FakeParentNode()], filter_, matches)
  File "/builddir/build/BUILD/breathe-4.35.0/breathe/finder/index.py", line 48, in filter_
    member_finder.filter_(node_stack, filter_, member_matches)
  File "/builddir/build/BUILD/breathe-4.35.0/breathe/finder/index.py", line 69, in filter_
    if filter_.allow(node_stack):
       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/breathe-4.35.0/breathe/renderer/filter.py", line 524, in allow
    if filter_.allow(node_stack):
       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/breathe-4.35.0/breathe/renderer/filter.py", line 510, in allow
    if not filter_.allow(node_stack):
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/breathe-4.35.0/breathe/renderer/filter.py", line 430, in allow
    name = self.accessor(node_stack)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/breathe-4.35.0/breathe/renderer/filter.py", line 351, in __call__
    return getattr(self.selector(node_stack), self.attribute_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'MemberTypeSub' object has no attribute 'prot'

@JasperCraeghs
Copy link
Contributor Author

I can't find the cause of the error you reported. Does the error occur with Doxygen 1.9.8 as well?

@yselkowitz
Copy link

I see a different error with 1.9.8 built myself on Fedora rawhide:

/usr/bin/make -C documentation html
make[1]: Entering directory '/builddir/build/BUILD/breathe-4.35.0/documentation'
sphinx-build -b html -P -d build/doctrees  -v -W -E source build/html
Running Sphinx v7.0.1
Using Doxygen v1.9.8
Adding copy buttons to code blocks...
making output directory... done
locale_dir /builddir/build/BUILD/breathe-4.35.0/documentation/source/locales/en/LC_MESSAGES does not exists
locale_dir /builddir/build/BUILD/breathe-4.35.0/documentation/source/locales/en/LC_MESSAGES does not exists
building [mo]: targets for 0 po files that are out of date
writing output... 
building [html]: targets for 41 source files that are out of date
updating environment: locale_dir /builddir/build/BUILD/breathe-4.35.0/documentation/source/locales/en/LC_MESSAGES does not exists
[new config] 41 added, 0 changed, 0 removed
reading sources... [  2%] autofile
reading sources... [  4%] autoindex
reading sources... [  7%] class
reading sources... [  9%] codeblocks
reading sources... [ 12%] codeguide
reading sources... [ 14%] concept
reading sources... [ 17%] contributing
reading sources... [ 19%] credits
reading sources... [ 21%] customcss
reading sources... [ 24%] define
reading sources... [ 26%] differences
reading sources... [ 29%] directives
reading sources... [ 31%] domains
reading sources... [ 34%] dot_graphs
reading sources... [ 36%] doxygen
reading sources... [ 39%] embeddedrst
reading sources... [ 41%] enum
reading sources... [ 43%] enumvalue
reading sources... [ 46%] file
reading sources... [ 48%] function
reading sources... [ 51%] group
reading sources... [ 53%] groups
reading sources... [ 56%] index
reading sources... [ 58%] inline
reading sources... [ 60%] latexmath
reading sources... [ 63%] lists
reading sources... [ 65%] markups
reading sources... [ 68%] members
reading sources... [ 70%] namespace
reading sources... [ 73%] page
reading sources... [ 75%] quickstart
reading sources... [ 78%] readthedocs
reading sources... [ 80%] specific
reading sources... [ 82%] struct
reading sources... [ 85%] tables
reading sources... [ 87%] template
reading sources... [ 90%] testpages
reading sources... [ 92%] tinyxml
reading sources... [ 95%] typedef
reading sources... [ 97%] union
reading sources... [100%] variable

> /usr/lib/python3.12/site-packages/sphinx/util/logging.py(424)filter()
-> raise exc
(Pdb) 
make[1]: Leaving directory '/builddir/build/BUILD/breathe-4.35.0/documentation'

RPM build errors:
fatal: not a git repository (or any of the parent directories): .git
Exception occurred while building, starting debugger:
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/sphinx/cmd/build.py", line 285, in build_main
    app.build(args.force_all, args.filenames)
  File "/usr/lib/python3.12/site-packages/sphinx/application.py", line 351, in build
    self.builder.build_update()
  File "/usr/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 294, in build_update
    self.build(to_build,
  File "/usr/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 310, in build
    with logging.pending_warnings():
  File "/usr/lib64/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/usr/lib/python3.12/site-packages/sphinx/util/logging.py", line 219, in pending_warnings
    memhandler.flushTo(logger)
  File "/usr/lib/python3.12/site-packages/sphinx/util/logging.py", line 184, in flushTo
    logger.handle(record)
  File "/usr/lib64/python3.12/logging/__init__.py", line 1700, in handle
    self.callHandlers(record)
  File "/usr/lib64/python3.12/logging/__init__.py", line 1762, in callHandlers
    hdlr.handle(record)
  File "/usr/lib64/python3.12/logging/__init__.py", line 1022, in handle
    rv = self.filter(record)
         ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/logging/__init__.py", line 858, in filter
    result = f.filter(record)
             ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/sphinx/util/logging.py", line 424, in filter
    raise exc
sphinx.errors.SphinxWarning: /builddir/build/BUILD/breathe-4.35.0/documentation/source/specific.rst:195:Invalid C++ declaration: Expected identifier in nested name. [error at 0]
  
  ^
make[1]: *** [Makefile:56: html] Error 2

@JasperCraeghs JasperCraeghs marked this pull request as draft November 1, 2023 11:25
@JasperCraeghs
Copy link
Contributor Author

@D4N I fixed the issue you reported. Thank you for catching and reporting it!

@yselkowitz I'm pretty sure you didn't use the changes in this PR.

Doxygen 1.9.8 is similar to 1.9.7 concerning the changes that are needed to support members in a Doxygen group.

This PR does not resolve #935.

@JasperCraeghs
Copy link
Contributor Author

@michaeljones Can you please review this pull request?

@JasperCraeghs JasperCraeghs marked this pull request as ready for review November 1, 2023 16:21
@michaeljones
Copy link
Collaborator

I appreciate you've put effort in here but unfortunately I'm not very active on this project at the moment. We're looking for the project to be better funded. Other maintainers might have time to assess this and do the necessary follow up work.

@JasperCraeghs
Copy link
Contributor Author

@Rouslan I have pulled in your commits on Rouslan:main into this PR. It appears to me that this PR no longer achieves its goal, however. Can you confirm please?
error.log

@michaeljones Melexis, my employer, has sponsored this project at last. Is the amount sufficient to get things moving again? We are eager to use the latest version of Sphinx. We are now stuck with d2b0ec1 + Sphinx 6.2.1 + doxygen-1.12.0.

@Rouslan
Copy link
Contributor

Rouslan commented Apr 12, 2025

@JasperCraeghs
The GitHub action log in your branch shows that the test for this still passes, but the test is very basic and it's likely it doesn't cover whatever happened in your case. If you provide an example input that fails, I'll fix the issue.

@michaeljones
Copy link
Collaborator

@michaeljones Melexis, my employer, has sponsored this project at last. Is the amount sufficient to get things moving again? We are eager to use the latest version of Sphinx. We are now stuck with d2b0ec1 + Sphinx 6.2.1 + doxygen-1.12.0.

I appreciated the funding from your employer. I didn't know how to get in touch with anyone from your company but we're grateful for it and are happy to talk about what development priorities you might have.

We also have funding from GDAL but at the recommended engagement level it would only cover 2 hours a month from 2025 which seems like it wouldn't go far. Your employer's contribution doubles that and gives more meaningful room for engaging with the project.

I'm trying to figure out how work on Breathe fits into my current schedule but I hope to re-engage with the project this week.

@JasperCraeghs
Copy link
Contributor Author

JasperCraeghs commented Apr 13, 2025

After some more testing with this branch in my project, I figured out the issue. If I surround the source code with the markers @{ and @}, Doxygen generates a sectiondef element in group__???.xml with the memberdef. Without the markers, the memberdef are located in the XML of the source file instead of the group.

Example with these markers:

/**
 * @file
 * @brief Public API for the component module
 * @internal
 *
 * @copyright (C) 2017 Melexis N.V.
 *
 * @endinternal
 *
 * @defgroup component Component
 * @details
 *
 * @{
 *
 */

#ifndef COMPONENT_H
#define COMPONENT_H

/** First function description
 */
void COMPONENT_FirstFunction(void);

/** Second function description
 */
void COMPONENT_SecondFunction(void);

#endif
/** @} */

Produces group__component.xml with a sectiondef that contains the memberdef elements and the component_8h.xml that contains member elements that refer to them:

...
    <sectiondef kind="func">
      <member refid="group__component_1ga1eb7a30a93c79e1f607fbccb8cf53f2a" kind="function"><name>COMPONENT_FirstFunction</name></member>
      <member refid="group__component_1ga52bb682071df64a969461971cadffd2b" kind="function"><name>COMPONENT_SecondFunction</name></member>
    </sectiondef>
 ...

@michaeljones
Copy link
Collaborator

The description indicates this is built on top of the performance PR (#967) so I will look at that first.

@michaeljones
Copy link
Collaborator

Or is it the other way around? The commits of the performance PR start with some of these? I would welcome some clarification on which should be reviewed and merged first.

@JasperCraeghs
Copy link
Contributor Author

The contents of my commits have been thoroughly reworked by Rouslan. I doubt that any of my new code has survived. I suggest pulling in the main branch of the original repo into #967 and reviewing that. The content of both PRs looks almost identical at this point.

We can keep this PR to debug the problem I reported yesterday (#934 (comment)).

@JasperCraeghs
Copy link
Contributor Author

JasperCraeghs commented Apr 14, 2025

@Rouslan I resolved my issue with the latest commit. Feel free to refactor it. I hope that the parser's cache is used, but I suspect that you can come up with a cleaner solution. Since Doxygen 1.9.7, the XML file of the source file contains a member element with refid attribute instead of a duplicate memberdef.

If the rendered output contains a duplicate, 3 warnings are produced. I wonder if this can/should be reduced/improved See this example:

.. doxygenfunction:: COMPONENT_FirstFunction

.. doxygengroup:: component
/home/developer/repo/doc/source/components/component/component_manual.rst:27: WARNING: Duplicate C++ declaration, also defined at components/component/component_manual:25.
Declaration is '.. cpp:function:: void COMPONENT_FirstFunction (void)'. [duplicate_declaration.cpp]
/home/developer/repo/doc/source/components/component/component_manual.rst:5: CRITICAL: Duplicate ID: "group__component_1ga1eb7a30a93c79e1f607fbccb8cf53f2a". [docutils]
/home/developer/repo/doc/source/components/component/component_manual.rst:5: WARNING: Duplicate explicit target name: "group__component_1ga1eb7a30a93c79e1f607fbccb8cf53f2a". [docutils]

image

Thanks to the markers @{ and @}, the doxygengroup contains the functions. So, they are required for proper grouping.

@Rouslan
Copy link
Contributor

Rouslan commented Apr 15, 2025

@JasperCraeghs Actually, your solution looks fine. It's been a while since I thought about this project and I haven't completely refamiliarized myself with it, yet, but I don't see anything that needs to be refactored. And yes, the parser's cache is used.

I'll look into the 3 warnings issue later, at my leisure. Before that, I'll update the parser to get rid of the warning with the latest version of Doxygen, as you pointed out in the comments of issue #967.

@michaeljones Neither PR is on top or bottom. I merged the branch into mine and then pushed the combined version back into this one. Jasper recently pulled my latest changes and then made another update, making this PR the most up to date one.

@michaeljones
Copy link
Collaborator

Thank you for the clarification. I have started to review this PR. I am 21/229 files in. I think that all I can really do is try to verify that it isn't doing anything malicious, verify that it runs ok and then work on resolving conflicts with main. I don't think it is reasonable to precisely review this much code.

I'll try to provide some progress reporting as I go.

@michaeljones
Copy link
Collaborator

I've afraid I've been slower with progress than I'd like and I've got a busy couple of weeks coming up so progress will resume in the second week of May. I've pushed a branch called merge-target that merges this branch into main and just commits all the merge conflicts. My plan is to sort them out a commit at a time so that there is clarity on the progress.

I realise it can be nice to have a squash strategy for PRs but I think we merge this one with a merge commit so that all the history is preserved otherwise it is going to be too much for one commit.

@Rouslan
Copy link
Contributor

Rouslan commented May 11, 2025

@JasperCraeghs I've resolved the duplicate warnings about duplicate declarations. It turned out to be an easy fix. I looked into how Sphinx normally handles duplicate entries. An ID is generated for each entry. Normally the ID is based on the entry but if the ID already exists, a unique generic ID is generated.

It turns out the extra warnings were not from the duplicate entry itself, but from a generated "target" node using a duplicate ID.

@michaeljones
Copy link
Collaborator

I've just completed my first pass on resolving all the conflicts on the merge-target branch. Yet to run any kind of testing but it is a good milestone though. I realise this isn't happening very quickly but I plan to continue to make progress.

@michaeljones michaeljones merged commit 5ecf2d3 into breathe-doc:main Jul 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

doxygenfunction: Cannot find function
6 participants