Use doxygen comments in sourcegen

[ischoegl]

Changes proposed in this pull request

Cantera/enhancements#39 suggested the use of YAML for autogenerated code; a partial solution emerged from Cantera/enhancements#156 (implemented in #1331), which is based on parsing of CLib header files. This PR prepares to use standard doxygen decorators as an alternative to YAML. Documenting CLib header files with @implements, together with the doxygen build/doc/Cantera.tag (and linked information therein) will yield all necessary information needed for code generation.

For illustration purposes, this PR just adds comments to autogenerated C# code.

If applicable, fill in the issue number this pull request is fixing

Partially addresses Cantera/enhancements#39

If applicable, provide an example illustrating new features this pull request is introducing

Snippet from decorated CLib header ctfunc.h:

    //! @implements newFunc1(const string&, const shared_ptr<Func1>, double)
    CANTERA_CAPI int func_new_modified(const char* type, int a, double c);

    CANTERA_CAPI int func_new_sum(int a, int b);  //!< @implements newSumFunction
    CANTERA_CAPI int func_new_diff(int a, int b);  //!< @implements newDiffFunction

Corresponding generated C# code snippet (click to expand)

    //! @implements newFunc1(const string&, const shared_ptr<Func1>, double)
    [DllImport(LibFile)]
    public static extern Func1Handle func_new_modified(string type, int a, double c);

    //! @implements newSumFunction
    [DllImport(LibFile)]
    public static extern Func1Handle func_new_sum(int a, int b);

    //! @implements newDiffFunction
    [DllImport(LibFile)]
    public static extern Func1Handle func_new_diff(int a, int b);

Corresponding doxygen tag/XML snippet (click to expand)

The Cantera.tag file entry for newDiffFunction is

    <member kind="function">
      <type>shared_ptr&lt; Func1 &gt;</type>
      <name>newDiffFunction</name>
      <anchorfile>d9/d3d/group__func1helper.html</anchorfile>
      <anchor>ga629dacc2b5e386507045fbf12070300b</anchor>
      <arglist>(shared_ptr&lt; Func1 &gt; f1, shared_ptr&lt; Func1 &gt; f2)</arglist>
    </member>

where the <anchorfile>/<anchor> entries point to detailed information (an XML file corresponding to the HTML file exists):

      <memberdef kind="function" id="d9/d3d/group__func1helper_1ga629dacc2b5e386507045fbf12070300b" prot="public" static="no" const="no" explicit="no" inline="no" virt="non-virtual">
        <type>shared_ptr&lt; Func1 &gt;</type>
        <definition>shared_ptr&lt; Func1 &gt; newDiffFunction</definition>
        <argsstring>(shared_ptr&lt; Func1 &gt; f1, shared_ptr&lt; Func1 &gt; f2)</argsstring>
        <name>newDiffFunction</name>
        <qualifiedname>Cantera::newDiffFunction</qualifiedname>
        <param>
          <type>shared_ptr&lt; Func1 &gt;</type>
          <declname>f1</declname>
        </param>
        <param>
          <type>shared_ptr&lt; Func1 &gt;</type>
          <declname>f2</declname>
        </param>
        <briefdescription>
<para>Difference of two functions. </para>
        </briefdescription>
        <detaileddescription>
        </detaileddescription>
        <inbodydescription>
        </inbodydescription>
        <location file="src/numerics/Func1.cpp" line="660" column="12" bodyfile="src/numerics/Func1.cpp" bodystart="660" bodyend="682" declfile="include/cantera/numerics/Func1.h" declline="174" declcolumn="12"/>
      </memberdef>

For demonstration purposes, a new YAML generator can be run as:

$ python interfaces/sourcegen/run.py yaml .
[INFO] Generating source files...
[INFO]   parsing ctmultiphase.h
[INFO]   parsing ctfunc.h
[INFO]   parsing ct.h
[INFO]     ignoring ['ct_setLogWriter', 'soln_newInterface', 'thermo_size']
[INFO]   parsing ctsurf.h
[INFO]   parsing ctreactor.h
[INFO]     ignoring ['flowReactor_setMassFlowRate']
[WARNING] Class(es) in configuration file are missing from tag file: 'Mixture'
[INFO]   writing interop.yaml
[INFO] Done.

extracts information from doxygen tags and converts everthing to YAML with the corresponding code snippet (fields with cxx_ prefix are extracted from Cantera.tag):

- annotations: |-
    //! @implements newFunc1(const string&, const shared_ptr<Func1>, double)
  ret_type: int
  name: func_new_modified
  params: (const char*, int, double)
  implements: newFunc1(const string&, const shared_ptr<Func1>, double)
  relates:
  cxx_type: shared_ptr<Func1>
  cxx_name: newFunc1
  cxx_arglist: (const string& func1Type, const shared_ptr<Func1> f, double coeff)
  cxx_anchorfile: d0/da8/group__func1modified.xml
  cxx_anchor: ga28693b4584180881da328a26bdcb5c4e
- annotations: |-
    //!< @implements newSumFunction
  ret_type: int
  name: func_new_sum
  params: (int, int)
  implements: newSumFunction
  relates:
  cxx_type: shared_ptr<Func1>
  cxx_name: newSumFunction
  cxx_arglist: (shared_ptr<Func1> f1, shared_ptr<Func1> f2)
  cxx_anchorfile: d9/d3d/group__func1helper.xml
  cxx_anchor: ga94b19aac6586867b26151625e8b6314f
- annotations: |-
    //!< @implements newDiffFunction
  ret_type: int
  name: func_new_diff
  params: (int, int)
  implements: newDiffFunction
  relates:
  cxx_type: shared_ptr<Func1>
  cxx_name: newDiffFunction
  cxx_arglist: (shared_ptr<Func1> f1, shared_ptr<Func1> f2)
  cxx_anchorfile: d9/d3d/group__func1helper.xml
  cxx_anchor: ga629dacc2b5e386507045fbf12070300b

Checklist

• The pull request includes a clear description of this code change
• Commit messages have short titles and reference relevant issues
• Build passes (scons build & scons test) and unit tests address code coverage
• Style & formatting of contributed code follows contributing guidelines
• The pull request is ready for review

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.22%. Comparing base (9d28aa2) to head (1cc2814).
Report is 19 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1777      +/-   ##
==========================================
+ Coverage   73.19%   73.22%   +0.02%     
==========================================
  Files         381      381              
  Lines       54289    54344      +55     
  Branches     9242     9247       +5     
==========================================
+ Hits        39737    39793      +56     
+ Misses      11579    11576       -3     
- Partials     2973     2975       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bryanwweber]

With the understanding that this is a limited-scope proof-of-concept, do you expect that extending this work will require the full doxygen parser at some point (and then just parse the Cantera.tag XML file)? The main concern I have about rolling our own parser here is error handling - if/when there are typos in the definitions, how is that handled? I'm assuming (maybe a big assumption) that Doxygen will have some sort of warning mechanism.

Nonetheless, I'm impressed that this could be done in so few lines of code!

[ischoegl]

[...] do you expect that extending this work will require the full doxygen parser at some point (and then just parse the Cantera.tag XML file)? The main concern I have about rolling our own parser here is error handling [...]

The idea would be to run scons doxygen prior to code generation. The "rudimentary parser" I implemented is really just to pull the doxygen tags off the CLib headers - the @implements tag appears perfect for that purpose. I looked over the doxygen XML and there really isn't a good way to search by header file.

One thing that could be done next is to generate CLib implementation files automatically (essentially by switching out doxygen XML for YAML in what @speth originally suggested in Cantera/enhancements#39). As CLib compilation is done during regular CI, any typo for the @implements content would be caught immediately. For that part we probably won't need to run scons doxygen first, although Cantera.tag would be good to have even there. Now that I understand sourcegen a little better (kudos to @burkenyo!), it appears that huge chunks of Fortran and MATLAB interfaces could be generated automatically. All we may have to do is making sure that the doxygen tags are all up to date.

[bryanwweber]

Thanks for the context @ischoegl! Good to know, and we'd just want to obviously document that doxygen is a new dependency for these generated interfaces (if that ever comes to pass).

[ischoegl]

Thanks for the context @ischoegl! Good to know, and we'd just want to obviously document that doxygen is a new dependency for these generated interfaces (if that ever comes to pass).

If we want to pull comments/default arguments from the original C++ code, yes. We could also go with a minimalist version where we just 'hijack' doxygen syntax (which we already use anyways).

[speth]

Pulling information out of Doxygen's .tag file and the associated XML files is an interesting idea. My inclination would be to use this along with the original suggestion of a YAML file that describes the interfaces, where this eliminates the need to encode the arguments and return types of each function, and could be used to extract some of the documentation, too (though there's the issue of translating different formatting syntaxes at the very least).

Part of the rationale here is that I suspect there will be cases that are more complicated that what's currently handled by sourcgen, especially as we try to expand it to multiple languages and improve handling of documentation, and that complexity is better handled in YAML than a special Doxygen comment parser.

[ischoegl]

@speth ... you're right that some YAML will be needed, but I believe these can be implemented as part of sourcegen (e.g. config.yaml for C#). What I like about @burkenyo's approach is that it combines autogenerated code with hand-coded portions that aren't easily automated.

For the time being, this PR just adds decorators to CLib headers (as bare-bones documentation), and makes those comments available to sourcegen. I don't think that there's much harm in adopting ...

[ischoegl]

This PR has mutated into a proof-of-concept for code generation. While I made substantial progress towards automated generation of CLib header files, I will open a new PR with a cleaned implementation at a later point.