rustdoc should understand javadoc-style comments

[brson]

Both /** ... */ and /// syntaxes.

It should be relatively simple to write a rustdoc pass to pull out these comments. The only complications are that we need to heuristically associate comments with AST nodes, and we have to be able to identify and strip the common wall of stars:

/**
 *
 * <- these stars are not part of the documentation
 *
 */

[brson]

The other downside is that this will make it more difficult to extract docs from compiled crates (which we can't actually do yet). Possibly we could have a rustc pass to promote doc comments to doc attributes.

[nikomatsakis]

Maybe we should just treat Rustdoc comments as a kind of attribute and embed them in the AST/metadata like other attributes.

[eholk]

I kind of like the #"""This is a doc""" syntax from @brson's e-mail. I kind of like the idea of leaving comments completely unstructured for the most flexibility in explaining things to people reading the code, and having a lightweight, but more structured, syntax for documentation that gets extracted into a standalone entity.

[Dretch]

OK then, given that this is marked easy, id like to work on it. Could @brson please point me in the right direction?

[brson]

The basic strategy that I recommend is to modify the parser to treat doc comments as a special type of attribute, so that they are easily and automatically associated with the appropriate AST nodes.

The one significant wrinkle is that we have to distinguish, like attributes, between docs that apply to the thing that follows and docs that apply to the preceding thing. With attributes we currently use the trailing semi:

mod m {
   #[attr1]; // semi means this is an attribute of mod m
   #[attr2] // no semi means this is an attribute of fn f
   fn f() { }
}

Which is also how we distinguish crate attributes (semi-terminated attributes at the top of a .rc file apply to the 'containing' crate). I do want to change this syntax in a separate issue (#2569).

I suggest for doc comments that we use doxygen's convention for specifying docs that come after the thing they document: /**< */ and ///<, so crate docs would look like

/**<

The Rust core library

...
*/

It's unfortunate to have to distinguish between the two but not doing so would probably require much more subtle heuristics.

So here is an approximate plan of action:

• Add a new lexer token to libsyntax/parse/token.rs called DOC_COMMENT(str_num, ast::attr_style), where str_num is an index to an interned string containing the entire comment and attr_style indicates whether it's an inner or outer attribute.
• Add a boolean to ast::attribute_ in libsyntax/ast.rs indicating that the attribute is a sugared doc attribute.
• Modify the lexer in libsyntax/parse/lexer.rs to emit doc comment tokens instead of passing over them as whitespace.
• Mody attr parsing in libsyntax/parse/attr.rs to convert doc comment tokens to doc AST attributes.
• Modify the pretty printer to understand doc comment attributes.

At this point rustdoc should automatically be extracting the doc comments from attributes, leaving all the decorating comment syntax (/// etc) intact.

• Add strip_comment_pass to rustdoc to run over all the documentation and strip out comment syntax bits.

@Dretch hopefully this is enough to get started

[brson]

@graydon you might want to be aware of this issue

[graydon]

Thanks. I am fine with reforming the applies-to-next / applies-to-enclosing syntax, but not sure angle bracket is the most suggestive. Don't feel Doxygen is an important enough precedent to drive this choice. Maybe play with a couple other variants visually in a scratch buffer to see if anything else looks a little better?

[graydon]

Also note that the lexer already has logic to extract comments and in particular grab the indentation-wall out of the way, it does so during the scan for comments-and-literals that it transplants into pretty-printed source.

[Dretch]

Wow, thanks alot @brson (and @graydon), that is extremely helpful.

[Dretch]

I have made some changes to hopefully fix this issue. You can see them here: https://github.com/Dretch/rust/commit/df2f4bbdaae1c3dc6f0d3b1a8681cf80bf057ef2

Unfortunately I can't seem to make a pull request of just this commit - i keep getting this commit in the pull request too: https://github.com/Dretch/rust/commit/986662cebd3f41911cb72c18f4e824a4856b009b. I think this may be because I have screwed up the branching somehow.

[brson]

@Dretch that patch looks great! Exactly how I imagined, and the conversion script is a nice bonus.

Before we merge it can you change the inner doc comment syntax to be the one @graydon suggested in #2569 (//! and /*!)?

[brson]

I did notice that the patch included a fix for a commented-out comment (////) that no longer even parses under this new scheme because it was not in attribute position. I wonder how big of a concern that is.

[Dretch]

I did notice that the patch included a fix for a commented-out comment (////) that no longer even parses under this new scheme because it was not in attribute position. I wonder how big of a concern that is.

To avoid this, how about having //! ... and /*! ... */ for outer doc-comments and //!! ... and /*!! ... */ for inner ones?