Skip to Main Content

Improving WordPress Pt 4 – Where Gutenberg Lost Me (Open Letter to Core)

Gutenberg is the code name for WordPress’s new visual editor, which is designed to add block support to WordPress.  Gutenberg is not currently earmarked for a specific release, but signs seem to point toward WordPress 5.0.   It has a promising UI, and has been through a fair number of iterations already, but as work continues, I find myself more and more disenchanted with the solutions they’ve proposed.  I find myself constantly taken aback by the incredible number of flawed premises that comprise the project.

Flaw 1 – “We know our audience from WordPress.com, and our personal experience”

This was essentially Matt Mullenweg’s  response to the discussion of adding an opt in analytics system to WordPress.org, to gather usage data about the software, to inform development. Respectfully, that response is dead wrong.

WordPress.com is a blog host, that filters out all users except those simply looking to post articles. With WordPress powering almost a quarter of the web, this is NOT as sufficient cross-section of users, and heavily biases the demographics to casual bloggers.

ADVERTISEMENT:

WordPress desperately needs data-driven development that comes from the community.  I understand that WordPress.com is where the money comes from, but WordPress.org isn’t just a side effect of the hosted blog platform, and should not be seen as a second class citizen.

A great example of this is the “kitchen sink” button that expands the advanced editor. Seven months ago, when reordering alignment buttons into the kitchen sink was a good idea, Andrew Ozz wrote a testing plugin to try to collect some data on usage. He found that in his extremely limited tests, most users expanded the kitchen sink, and left it open, to provide themselves as many options as possible. Seven months later, the new Gutenberg UI is based around hiding as many options as possible until situationally relevant. In that time, no data was collected to prove or refute the idea that users want more visibility of the tools available to them. In fact, the primary motivation of the redesign seems to be “people like Medium“. But what works for Medium may not work at all for the larger, more varied userbase of WordPress.

Flaw 2 – “We’ll build the UI, then tackle the code”

While this is a great way to get buy in very early, it means that a UI is built without regard to the underlying challenges of data management and presentation. As a result, every time a snag is hit in tying that UI to data, the kneejerk is to hack in support for the UI that people bought into, rather than adjusting to meet the actual needs of the system. We’ll see this come into conflict more when block types like columns and nesting are needed, which were completely ignored in the UI phase, and pose significant complications.

Flaw 3 – “We’ll choose a data-structure, then tackle the code”

Well, now we have the word of god down on how the data is stored, and the word of god down on how the interface looks. If it isn’t obvious, this leaves very little room for the needs of the actual code to be asserted. When one aspect of the storage method is a square peg, and one aspect of the UI is a round hole, the code needs to do both at once, leading to madness. (I am eagerly awaiting the moment when the team finally decides to tackle columns.)

Flaw 4 – “Post_content is the single source of truth”

So, WordPress has structured data needs for blocks that may be dynamic or static, and may have a large amount of meta-data attached, and it has flat output needs in various forms, such as excerpts, search fields, RSS feeds, and HTML.  It also has needs for several ways to modify the underlying structured data, in various apps, via API, or in visual editors. Clearly, the core data is structured, and should be stored as such, with all flattened views and editing interfaces being generated from that. At minimum, it could be a JSON object, or better yet, a series of database structures that could be manipulated and indexed independently.

NOPE. The team went with HTML with inserted comments.

Are you thinking WTF? well, the explanation is easy. This massive paradigm change to structured data for WordPress posts would change everything from how the API interfaces with content, to how the site renders the_content(). And that is hard. Plus, this is the #yearoftheeditor, not #yearofthedataparadigmshift, so making changes like that was written off as out of scope. They’d rather maintain a box of snakes for the next ten years, endlessly patching, then to consider a major structure change.

So, instead, we get a whole new set of problems bolted into the existing mess that is post_content. Search results will now parse through the myriad html comments in a post, as well as the shortcodes and html tags they already look at. All rendered HTML will need to be based in forms that work equally well as plain text, or auto-excerpts and feeds will look awful. Dynamic blocks will disappear, of course, so any text describing them will suddenly be orphaned in feeds and excerpts. Rendering blocks on the front end will now involve either an expensive DOM walker, to parse the WordPress comments, or an insanely complicated RegEx to parse comments, that is guaranteed to fail under certain circumstances (Regular Expressions are fundamentally not suited for parsing HTML, as HTML is not a regular language), or posts will need to be filled with opinionated HTML to support their display. (note: it appears i was incorrect about this point. Post content is parsed using PEG, which is theoretically a parser capable of handling HTML parsing… HOWEVER, the PEG parser for PHP is generated by a project called Peg.js, which is at version 0.1, and does not promise stability until version 1.0, and a project called phpegjs, which is at version 1.0.0beta7 and has only one contributor.. this does not inspire faith)

Flaw 5 – “Text editors and API editors will edit the structured data directly, while the visual editor will edit through a parsing interface”

So, the first time a text-user modifies a block in a manner that breaks an entire nested tree of blocks, they are going to scream in frustration. There is no way to recover from arbitrary changes that might break HTML, shy of including an error checking step that parses the blocks after the text version is saved, and rejects the change if it breaks things.  That way leads madness.

ADVERTISEMENT:

Structured data MUST be edited in a structured way. Trying to ignore this fact is a great way to develop yourself into a corner and build a tool that is insanely convoluted.

Flaw 6 – “We have to dance with the one we came with”

TinyMCE is an visual editor based in a content-editable block. As such, everything built in TinyMCE is inherently unstructured and buggy, and must be parsed with dom-walkers, node-parsers, and other tools that are overbuilt and inscrutable to get even half-way decent results.

The editor that was designed for Word-like operations is NOT the editor to handle structured data, and the resulting body of code is enormous.  Maintaining the Editor is going to become a herculean task, if TinyMCE is the basis.

The reason for choosing TinyMCE is familiarity, but with the number of changes necessary to implement Gutenberg, the editor won’t be familiar to anyone, so that point is moot. The secondary reason has been listed as to offer the inherent accessibility afforded by content-editable.  However, this argument is flawed too. With content-editable you get accessibility geared toward a single, typeable field. The moment you add to that, you go from having accessible features that help you, to having to fight with accessible features that are no longer relevant and often are incorrect. It is a much better idea to build your accessible interface on a solid foundation that is built to match your needs.

There are many JavaScript-based editors these days that are designed to not limit the user to the capabilities of content-editable, My money is on slate.js as the best option, but there are other contenders, such as quill.js

Flaw 7 – “There is only one visual interface for editing”

I covered this in part 3 of this series, but the idea of a single catch-all editor is deeply flawed. Pages have completely different needs from posts, which might have completely different needs for different post formats, which have completely different needs from many custom post types.  One editor for everything is not a good solution. The editor should be pluggable, and be able to be easily modified from context to context, allowing for different experiences for designers and content-editors. This sort of work is the definition of Plugin Space, and trying to hammer it all into core is only going to hurt their users.

There is a lot of talk of discontinuing support for the existing editor, in favor of the new interface. This will have a horrible effect on the literally tens of thousands of plugins, themes, and sites that make use of the editor in other contexts like settings pages and meta boxes. This is very strongly tied to flaw 1, but developing without these users in mind is foolish to the extreme.

17 Comments

  1. Ruben's profile image.

    Your last point makes all the sense to me. Gutenburg seems like a great idea, as a plugin. I want more control over what options are available to site users and allowing them to build complex structures inside post_content undoubtedly cause all kinds of issues. I’ve been working on some old drupal7 sites recently, and the experience using CKeditor is horrendous, but there are modules which allow me to easily add and remove tools, and create different defaults per content type. There are some useful ideas WordPress could lift, not trying to compare itself against Wix.

  2. Andrew Roberts's profile image.

    Thanks for your thoughts. I think all of your points are valid.

    First off, disclosure: I am the head of the TinyMCE project and have also been involved in the Gutenberg project the past six months.

    It is worth noting that TinyMCE core is still underneath-the-hood of Gutenberg powering their implementation of “editable”. Gutenberg has wrapped TinyMCE Core and abstracted it pretty tightly so you might be right that another of implementation of contenteditable could be swapped in over time. But, then again, there is a Classic Text block which is essentially a way for the ‘old’ TinyMCE editing experience to be used. How prominent this ‘Classic Text’ block is will shake out over the next few months.

    It is important to note that TinyMCE is a successful project in its own right and is continuing to move forward. 1200 commits and lots of releases in this past year. Regardless of what WordPress Core does TinyMCE is going to keep doing what it is doing and no doubt be available through, at the very least, a plugin for WordPress.

    In fact, one of the reasons we have contributed design and coding to Gutenberg is so that we can incorporate a lot of the block-editing concepts back into the main TinyMCE project.

    • gschoppe's profile image.

      Thank you for reading my post, and thank you for taking the time to reply. I have gotten a lot of use out of tinyMCE, and I respect the hard work you’ve put in over many years, and I know the project will continue for many more. I’m also slightly amazed that this post made it into your hands.

      That said, I believe that editors like Medium and Gutenberg are ideal for producing structured data, that would be stored as JSON, or a similar flexible data structure. Trying to flatten content blocks into raw HTML fields that are not designed to hold this sort of metadata, and “making it work” seems to be a path to disaster.

      TinyMCE (or contenteditable, really) is great for opinionated HTML, which carries with it all it’s structure and data (except anything provided by css, of course). But content blocks are meant to rely on external information like themes and parsers to render them. And that is really not a good fit for flat HTML.

      • Andrew Roberts's profile image.

        Thanks for the kind words! I was curious about reactions to Gutenberg and you are article made its way onto the list at https://github.com/WordPress/gutenberg/issues/1419.

        I agree that having a JSON-based data structure could make sense for some editors, or for blocks within editors. The approach to blocks that we are consdiering will likely to allow components/blocks/widgets that can store content in JSON or other formats if they wish. You can read more about our thoughts here: https://www.ephox.com/blog/the-weekend-without-pants-our-gutenberg-design-challenge/

        FWIW, external themes and parsers can and do work with semantic HTML though. Web Components are meant to further the art form there. One of the only reasons comments were used as the data structure in Gutenberg vs HTML was the lack of a good HTML parser in PHP. But parsing comments with regex is hacky and not ideal either.

        • gschoppe's profile image.

          You are, of course, correct about web components and front-end parsing with HTML. If you are working in the context of a DOM tree, there are many options. Unfortunately most (and i realize this is a charged statement) themes/plugins are still operating primarily in PHP, where DOM walking is painful and slow. Although there are definite benefits to moving to a primarily JS-based stack, layered on the REST API, that leverages HTML templates and custom tags, I doubt that will be the primary mode of rendering WordPress content for the next 5 years.

          As for the decision to parse HTML (a Chomsky Type 2 language), which may have been created in Text Mode by users who only understand Chomsky Type 1 syntaxes, with Regular Expressions (a Chomsky Type 3 grammar), I refer to my favorite post of all time: https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454

          It is true that WordPress already uses Regular Expressions to parse HTML for relatively small things like wpautop, RICG, and capital_P_dangit (that is another rant, though), but expanding that reach to encompass the entire concept of “what is the content” is going to massively expand the reach and effect of the infinite number of potential bugs that result from fundamentally treating one class of language as if it was another.

          • Dennis Snell's profile image.
            Dennis Snell says:

            Howdy! I left a comment at http://gschoppe.com/wordpress/gutenberg-is-no-gutenberg/ but I’ll reply here too. I think you are quite mistaken when you claim that Gutenberg is parsing with Regular Expressions.

            or an insanely complicated RegEx to parse comments, that is guaranteed to fail under certain circumstances (Regular Expressions are fundamentally not suited for parsing HTML, as HTML is not a regular language)

            The grammar is formally specified as a PEG and both a JavaScript and PHP parser are generated from that grammar. Even though many grammars involve regular expressions for defining the individual tokens, not a single one is found there in Gutenberg.

            You can check it out and explore it!
            https://github.com/WordPress/gutenberg/blob/master/blocks/api/post.pegjs

            In fact, you can load it in an interactive environment and see how it performs under different documents. In the following tool you can load the up-to-date grammar by clicking on “Fetch” then create or paste in a Gutenberg-generated (though it could be any HTML) document. The top-right window shows the parsed output and the lower-right window will show the parse trace if enabled.
            https://dmsnell.github.io/peg-parser-explorer/

          • gschoppe's profile image.

            You are correct. I was misinterpreting the regex-like grammar of the PEG parser for a regex builder. I am however concerned about the use of phpegjs and peg.js in core, as phpegjs has only a single contributor and is in beta, and peg.js is at version 0.10 and claims that stability is not guaranteed until version 1.0. I will print a correction, but i still believe the choice of rolling a new, custom format is a major misstep.

  3. Michael's profile image.

    Hi Gregory, thank you so much for this post! Especially the first point is very important to take into account. I think it’s a huge mistake to take insights from WordPress.com and use these to make decisions for .org. It’s a different market and a different audience.

    I still have hopes that the WP community will express concerns and help shape Gutenberg in a way that makes sense for .org. But I’m afraid that with the influence of Automattic on this project some decisions have been made which can’t be reverted and we need to find a middle ground somehow to make it work for everyone.

  4. Matías Ventura's profile image.

    Thanks for sharing your thoughts and concerns. I want to address a few misconceptions:

    Search results will now parse through the myriad html comments in a post, as well as the shortcodes and html tags they already look at.

    This is not true. It’s simple to filter out the comments for display in the front-end, and there’s a task logged to implement just that. Comments are tokens that our grammar can identify to know which and how many blocks are contained in a post. But they are irrelevant once rendered in the theme and the goal is to not have them there. Also, as far as I’m aware, most crawlers completely ignore comments.

    All rendered HTML will need to be based in forms that work equally well as plain text, or auto-excerpts and feeds will look awful. Dynamic blocks will disappear, of course, so any text describing them will suddenly be orphaned in feeds and excerpts.

    I may be misunderstanding your point here. Dynamic blocks can be purely references to data stored elsewhere (see the “latest posts” block as an example) with the output constructed server-side. Also, since the reference is a token within an HTML comment, they won’t render spurious text like shortcodes do on feeds, emails, etc, when rendering directly from the content field.

    Rendering blocks on the front end will now involve either an expensive DOM walker, to parse the WordPress comments, or an insanely complicated RegEx to parse comments

    I’m confused by this statement. Why are you saying this is necessary in order to render blocks?

    Maintaining the Editor is going to become a herculean task, if TinyMCE is the basis.

    The editor is not built with TinyMCE as the basis. On the contrary, TinyMCE is used to power individual text formatting components within the editor, which is something it does very well. It doesn’t interact with the structure of blocks nor their data. There are many blocks that don’t use TinyMCE in the editor, since it’s not needed unless you require inline formatting (bold, italics, inline links, etc).

    • gschoppe's profile image.

      Thanks for the attention and response! Forgive me if I reply bluntly to any of these, as tact is not my strong suit. I respect the work you have been doing, but some of the core design choices raise red flags for me.

      as far as I’m aware, most crawlers completely ignore comments.

      This is true of google and such. I was referring to the WordPress internal search functionality “example.com?s=search+term”, which relies (IIRC) on a simple LIKE query of post_content.

      All rendered HTML will need to be based in forms that work equally well as plain text

      I was referring primarily to the fact that WYSIWYG implementations rarely produce clean semantic text when flattened for output. With WordPress currently, it is easiest to see when post_content contains two consecutive divs. The excerpt mashes the two together with no space in between. Equally, when a shortcode is parsed out of an excerpt or feed, surrounding descriptive text remains. unless there is a separate parser for each block to render the text-only version, there will be a myriad of situations like that.. example: A user uses a text block to title a section “related posts”, then follows it with a dynamic block containing posts. The dynamic block doesn’t render in an excerpt or feed, nor does the relative importance of the section header, so the text “related posts” is left orphaned.

      Rendering blocks on the front end will now involve either an expensive DOM walker, to parse the WordPress comments, or an insanely complicated RegEx to parse comments

      How are dynamic blocks detected on page load? the post content gets pulled in as a flat text field, then something happens, and dynamic content is output.. there is a processing step there, that either involves a PHP DOM Walker or a regular expression to grab the comments that delineate a block and parse them. I believe the current implementation is a regular expression similar to the one used for shortcodes or RICG images. Those regexes are a technical debt that WordPress should be moving away from, rather than more deeply embracing.

      The editor is not built with TinyMCE as the basis.

      That is heartening to hear. I haven’t dug too far into that part of the code since the single-instance mockup was the seeming preferred solution. However, an issue remains that contenteditables have to operate outside the usual confines of the react framework, in a weird marriage that (while it works) is awkward as heck, because react doesn’t manage state for the editor itself. (unless something has changed that i am unaware of). It seems like a pure JS editor would be far better for maintaining a single source of truth for editor state.

      Let me know if I’m misunderstanding these. React is not a native language for me, so sometimes the concepts can escape me.

  5. Stagger Lee's profile image.
    Stagger Lee says:

    Let us see how easy-difficult will be to hook own things inside Gutenberg.
    Everyone is talking about poor third party plugins developers. But nobody about simple developers/coders who had hard time to change anything inside TinyMce.

    Just a thought to chose TinyMce plugins for visual editor for comments, in front end, makes me wish to kill myself.

    If they make it (and similar) more easy I am for it, all the time.

    • gschoppe's profile image.

      This is a very good point, although there is a large amount of code already written that it would be nice to maintain some level of support for, the simplification of things moving forward would be a nice feature. I will say that currently, it looks like extending the editor will mean learning React, which although not impossible, is another technology stack that currently isn’t in use anywhere else in the codebase. And although I have some hopes for good dev documentation, the current JS documentation for WordPress is sorely lacking. For example, customizing the Media interface is like pulling teeth, and TinyMCE’s API documentation feels like a skeleton.

  6. Ray Gulick's profile image.

    Very well-written post outlining the WTF-ness of Gutenberg.

    WordPress has always been biased toward bloggers (that appears to be Matt’s personal bias as well). There has been very little acknowledgment of the differences between bloggers and the kinds of users who are admins and editors for business websites.

    Frankly, I’m more than ready to see a forked version of WordPress that caters to CMS users.

    There is no question the TinyMCE editor could be better, and maybe should be replaced entirely, but Gutenberg doesn’t seem like the answer. In the meantime, I hope Automattic listens to non-blogger users, and decides to continue Gutenberg as a plugin, rather than making it part of core.

  7. mark k.'s profile image.

    storing data as text as a new variant of html (well, wordpress variant of html used to store it in the DB) where comments become data, is by itself the most horrible aspect of gutenberg. Other things can be fixed/improved over time, but DB structure is not as easy to fix especially in wordpress where every idiot (me included) directly queries the DB or directly access post_content.

    hacking things instead of properly methodically developing things, was a problem with all core featured, but it is still amazing that even for this kind of strategic project there is no proper planning, feedback and market research before the first line of code is written. 🙁

    • gschoppe's profile image.

      agreed, and we have to deal with this new custom structure on top of all the other awful structures like wpautop, shortcodes, and the various Regex-based modifications that get made to the post without any common framework. if WordPress wants posts to be structured, they need to move forward with a real structured data format, like JSON/MobileDoc

Your email address will not be published.

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>