Tag string use-case: language marker for syntax highlight

[EmilStenstrom]

In PyCharm it's possible to write # language=html before a string, and have that string be syntax highlighted as html (or any other supported language). It's not supported by any other editor. It's clunky because it's not obvious what happens if the comment is on the same line as other code, if there's a newline in between, and so on.

What if there was a way to tag a string, so the code editor knows how to syntax highlight that string. Oh wait, that's what you're working on! :)

My use-case
A component in my library is a combination of python code, html, css and javascript. Currently I glue things together with a python file, where you put the paths to the html, css and javascript. When run, it brings all of the files together into a component. But for small components, having to juggle four different files around is cumbersome, so I've started to look for a way to put everything related to the component in the same file. This makes it much easier to work on, understand, and with fewer places to make path errors.

Example:

class Calendar(component.Component):
   template_string = '<span class="calendar"></span>'
   css_string = '.calendar { background: pink }'
   js_string = 'document.getElementsByClassName("calendar)[0].onclick = function() { alert("click!") }'

Seems simple enough, right? The problem is: There's no syntax highlighting in my code editor for the three other languages. This makes for a horrible developer experience, where you constantly have to hunt for characters inside of strings. You saw the missing quote in js_string right? :)

If I instead use separate files, I get syntax highlighting and auto-completion for each file, because editors set language based on file type. But should I really have to choose?

Proposal
Would it be compatible with your work, to recommend editors to syntax highlight strings in python that has markers that correspond to this list of language identifiers? https://code.visualstudio.com/docs/languages/identifiers#_known-language-identifiers

This would make the developer experience even better for developers, and make your example tags in this repo even more powerful. html'<span class="calendar">{content}</span>' would be highlighted as proper HTML (because the name html was used).

Advantages:

• Gives code editors a great API to standardize around. # language=html is not it...
• Makes it easier to spot errors inside those tagged strings
• Using standard names makes the feature discoverable for people that did not read the docs. "Oh, I get syntax highlighting when I use this tag, nice!"
• Everything just works for developers, no config, no installing stuff.

Even without my suggestion above, I really like what you're doing here.

Let me know what you think!

[rmorshea]

Would it be compatible with your work, to recommend editors to syntax highlight strings...

Typically I think this sort of thing ends up being handled by plugins. For example, out of the box VSCode does not highlight Javascript template literals. Instead people tend to use plugins like lit-html which, in addition to syntax highlighting, provides IntelliSense.

With that said, it would be beneficial to supply plugins for popular editors (probably VSCode and PyCharm) by the time the Python version containing these changes is released. Ultimately, if those plugins became popular enough it could make sense for the behavior to get built-in into PyCharm or the Python extension for VSCode.

Before then though, it seems like the main action we would take with respect to this PEP would be to mention that editors could supply features of this nature.

[pauleveritt]

I'm with PyCharm and I'm hopeful that (a) this all lands and (b) it makes it easier to support something basic out-of-the-box for IDEs and (c) more custom usages through plugins.

For my component-thingy, not sure if the proposal would fit exactly, as it isn't anything on the list, as @rmorshea hints at. But likely that could be fixed.

[rmorshea]

hopeful that this all lands...

I'm in a bit of a lull at work right now so I think I may have some time to work on this over the holidays.

[EmilStenstrom]

@rmorshea Do you mean time to land tagstr in cpython main? That would be awesome :)

[rmorshea]

Unfortunately no, this is very much a work in progress. At the moment, we don't have a complete draft proposal to share and get feedback on. I'm not super familiar with CPython's release timelines, but my guess is that we'd be shooting to get this into CPython 3.13.

[EmilStenstrom]

Does that mean there's a group of people working on this atm? I see no activity in this repo.

[rmorshea]

At the moment it's @jimbaker and myself with support from @gvanrossum who, in addition to providing feedback and ideas, has contributed a branch of CPython with an initial implementation of tag-strings that we've been using to test out our work. With that said, both Jim and myself have gotten busy as of late. Right now we each have draft PRs up that respectively, contain an initial specification, and a tutorial on how tag strings could be used to render HTML templates. There's a lot more work to be done though.

[jimbaker]

@rmorshea Thanks for following up, I was out of the loop during the holidays.

@EmilStenstrom It's a really good idea, I'm just trying to think how a tag function could declare itself as supporting a specific syntax. We all "know" that html or sql means something when we are looking at the code, because we are all writing self-documenting code ;) , but connecting that automatically to a specific tag function is the interesting trick. Maybe it's just enough to assume html or htm means HTML, etc. Another thought is that the tag function could be appropriately annotated so that the syntax highlighted could determine this, but there's the initial setup problem - what annotations, and how? My current thought is through some standard set of decorators that could decorate tag functions.

Related is that we will be working in the context of https://peps.python.org/pep-0701/ - work that also was discussed at PyCon back in April. The formalization of f-strings is very much related to the formalization we need to do for tag strings, and consequently any general syntax support.

[EmilStenstrom]

@jimbaker I opened an issue with the vscode folks here for their opinions: microsoft/pylance-release#3874 - including some different ways this could work. I think the most straightforward way of doing this is to simply use the language identifiers that editors already use, and automatically apply syntax highlighting in that language, when a tag string with that name is used. This would make the feature discoverable in a way where things just work when you use a tag string which would be wonderful.

If you would like avoid mistakes I think using annotations for this is nice. Annotated["html", str] could be specified in the tag function to make it highlight like you want.

[pauleveritt]

Speaking for PyCharm, I'm also interested. I sponsored a PyCharm plugin to explore the kinds of things I wanted from my htm.py-based component stuff. I could share some opinions but it's likely bike shedding.

[jimbaker]

Speaking for PyCharm, I'm also interested. I sponsored a PyCharm plugin to explore the kinds of things I wanted from my htm.py-based component stuff. I could share some opinions but it's likely bike shedding.

I think we don't even have the available colors picked out, so it cannot be bikeshedding just yet. 😁

One eventual possibility is a DSL registry, similar to https://www.schemastore.org/json/ Maybe that's easier because of the standard JSON Schema, but it does seem to solve the bootstrapping problem - how to connect a tag function with the syntax it supports. In the interim, we can also do something similar to using using a setting with a plugin would work - for this qualified name that can be imported from some Python package (private or on PyPI), it corresponds to this DSL, such as HTML or SQL. On the JSON schema side, I have used this setup for internally developed JSON schemas:

• https://www.jetbrains.com/help/idea/json.html#ws_json_schema_add_custom
• https://code.visualstudio.com/docs/languages/json#_json-schemas-and-settings

Note that while DSLs may vary, the interpolations themselves do not change except with respect to their syntactic placement and of course semantic meaning. See @rmorshea's recent additions in https://github.com/jimbaker/tagstr/blob/main/tutorial.rst for some more discussion, where we would want to have some strictness about placement, in this case raising SyntaxError("Cannot interpolate attribute names"). The net of this is that Alice and Bob could each implement their own HTML tags, and so long as they agree that interpolations can go in certain places (and maybe not others), the syntactic support would work equivalently.

[benji-york]

One eventual possibility is a DSL registry

For informational purposes, GitHub's syntax highlighting uses Linguist which maintains a project-internal registry.

[EmilStenstrom]

I think a great solution for this would be if all editors by default shipped with a list like Linguists registry, or PyCharms language identifiers, and automatically applied syntax highlighting to tag strings based on the tag name. Optionally there could be settings to say "in this project, I want this tag to highlight in this language".

Advantages:

• Everything just works. If you make a custom tag for a new language, you automatically get syntax highlighting for it if you happen to pick that language's name (likely).
• All of the responsibility of highlighting is moved to editors, so you don't really have to adapt the PEP to this. Except if you decide to ship pre-made tags, ideally pick the same language-identifers for them.

Disadvantages:

• There is a risk that incorrect highlights occur. Specifically: "R" is a language that clashes with raw strings. Does that mean you handle that in editor settings? There are lots of languages, so new tag names that are not languages, might accidentally hit a language name and get incorrect highlights. Maybe a better strategy is to just include the top10 languages that are usually embedded into Python by default? Suggestion: HTML, CSS, JS, SH, SQL, XML, CSV/TSV, JSON, TOML, YAML?

[jimbaker]

I think a great solution for this would be if all editors by default shipped with a list like Linguists registry, or PyCharms language identifiers, and automatically applied syntax highlighting to tag strings based on the tag name. Optionally there could be settings to say "in this project, I want this tag to highlight in this language".

Let's assume a registry like this was supported by editors.

Let's recall that tag names are just standard names in Python, bound (presumably) to some callable that supports the tag function protocol. Such callables can be imported and defined as usual. Python IDEs like PyCharm and VSCode can readily track their use and definition, much like other object usage.

So then it just becomes a question of registering the callable as supporting a specific DSL, like HTML or SQL. Especially in the case of SQL this also applies to dialect, such as SQLite vs Postgres. We just have to figure out how to do this registration.

Advantages:

• Everything just works. If you make a custom tag for a new language, you automatically get syntax highlighting for it if you happen to pick that language's name (likely).
• All of the responsibility of highlighting is moved to editors, so you don't really have to adapt the PEP to this. Except if you decide to ship pre-made tags, ideally pick the same language-identifers for them.

Yes, this would be ideal.

Disadvantages:

• There is a risk that incorrect highlights occur. Specifically: "R" is a language that clashes with raw strings. Does that mean you handle that in editor settings? There are lots of languages, so new tag names that are not languages, might accidentally hit a language name and get incorrect highlights. Maybe a better strategy is to just include the top10 languages that are usually embedded into Python by default? Suggestion: HTML, CSS, JS, SH, SQL, XML, CSV/TSV, JSON, TOML, YAML?

r, R, and other prefixes that are currently predefined for Python string literals are not allowed to be tag names, so there's no conflict here.

There are two possible ways for the IDE to not properly recognize the desired language, assuming this registry:

1. Code has monkey patched the name so as to rebind it. This is generally bad practice in Python, outside of test patching (and even then, we expect it to be appropriately mocked).
2. Imports are indirect, via importlib or related functionality. In this case, the tag can't be automatically associated.

[EmilStenstrom]

Some updates: The Python extension people with VSCode are currently thinking about adding support for highlighting of python strings with other languages in them. The exact syntax is not decided, but there is technical triage going on about how to embed other languages inside python. Great news, and something I think would align nicely with this proposal.

Actually, I think this would greatly enhance working with Python overall, in the worlds most popular code editor. I'm excited! :)

It seems to get the ball rolling they need to see that people are excited about this. This is measured by the number of upvotes on this ticket: https://www.github.com/microsoft/pylance-release/issues/3874 - can I hope for some upvotes for you and some close friends of yours? :)