How Warhol works and (some of) what it can do

Published on 11.03.2019 by Peter Kröner

Lines and lines of code

Some of the error codes you can expect from Warhol

Warhol's approach to automated style testing does not rely on pixels but works by analyzing CSS and DOM structures. This enables Warhol to report more than just the fact that something in your styles went wrong. This post comes straight from the trenches of our code base, explains a few of Warhol's internal details and gives you a glimpse into the nature of the diagnostic feedback that you can expect from Warhol.

What Warhol can (and can not) do

Warhol does not require you to write tests by hand. You just feed your already existing pattern library 1 (and a bit of configuration) into Warhol and it turns the HTML and CSS into data that can be used to validate the implementation of your components outside of the pattern library. This has three main advantages:

  • You have to do next to no work if you already have a pattern library producing HTML and CSS
  • Warhol does not care about the content of a website or app and thus can find errors in production as well as during development
  • Warhol is lightning fast, fast enough to check your web pages right in your browser
A classic CSS overflow bug
If a component was defined to allow overflow, Warhol will not report overflowing text as a bug.

But, as with everything in life, there's also a downside to this approach: Warhol is totally blind to pixels and will never be able to catch everything that can go wrong in a web page.

For example the classic “CSS is awesome” overflow bug is, quite obviously, a bug. But as long as such a purely visual bug results from CSS that is equivalent to the CSS in the component library, Warhol (in its current form) will not detect it. Warhol is like any other testing tool in that respect – behavior that is not explicitly specified can't be checked.

Precise diagnostic feedback

Of course, we think that this trade-off is still a pretty good deal, as there is another benefit to Warhol's approach: Warhol can't just tell you where and when some design inconsistency crept into your project but you also get precise diagnostic feedback that enables you to quickly find the cause and source of any CSS discrepancy.

Here are some of the problems Warhol can precisely identify:

  • Wrong values are obvious: if the text in your component is supposed to be red but turns out to be green in production, Warhol will warn you.
  • Warhol has several tiers of reports and does not classify every tiny CSS discrepancy as an error. If some CSS property has the correct value but a different
    priority you will get a notification about a wrong priority. If you don't care about mere notifications, you can just filter them away.
  • Another notice that you may or may not want to ignore is the one about unexpected child elements. Whenever you define a component in your component library, Warhol also takes note of its content. If Warhol encounters a child element in production that was not specified for this component, Warhol issues a notification. An unexpected child element is not always a design bug but can also be just an oversight in the original component definition.
  • But not every aspect of Warhol's reporting is as ambiguous as unexpected children or questionable
    values. If whole CSS properties go missing or appear in unexpected places, Warhol will warn you about excess and missing declarations as that's a pretty clear bug.
  • Excess and missing pseudo elements take a pseudo element's
    property into account. Simply giving
    an unexpected red border will not get reported unless the
    property makes
  • A component library often contains not only components but also general styling information like the colors and font sizes. Warhol can incorporate this information into its data model about your components. Armed with this extra data, Warhol can not only test elements that are not defined as components for adherence to the style guide but also more precisely tell you in what way a CSS value differs from the expected style:
    • The aforementioned wrong value is reported when a valid, well-defined value is found in place of another valid value.
    • An unexpected value is not only different from the value that a property on a given component should have, but is also not a value that was defined anywhere else in the component library.
    • Nonconforming value expected is a fun kind of bug. It basically means that, while a component is implemented as specified, the specification itself has a bug. If you define
      as valid colors in your component library but also create a component that uses
      as a text color and correctly implement this component in production, the color
      is, strictly speaking, still invalid. Warhol performs an internal consistency check of your component library when first reading it, but if you overlooked the resulting warnings the first time around, Warhol will continue to nag you while testing.

These are just some of the (at the moment) 15 different things Warhol can tell you about the real-world implementation of your component library in your project. There is much more to talk about, so stay tuned for more info on the inner workings of Warhol!

1: We use the term pattern library as the general term describing an online style guide for components; often referred to as component library or front-end style guide.

Written by

Peter Kröner

Trainer for frontend technologies, podcaster @workingdraft.


More from the blog