How Warhol works and (some of) what it can do
Published on 11.03.2019 by Peter Kröner
Warhol's approach to automated style testing does not rely on pixels but works by analyzing CSS and DOM structures. This enables Warhol to report more than just the fact that something in your styles went wrong. This post comes straight from the trenches of our code base, explains a few of Warhol's internal details and gives you a glimpse into the nature of the diagnostic feedback that you can expect from Warhol.
What Warhol can (and can not) do
Warhol does not require you to write tests by hand. You just feed your already existing pattern library 1 (and a bit of configuration) into Warhol and it turns the HTML and CSS into data that can be used to validate the implementation of your components outside of the pattern library. This has three main advantages:
- You have to do next to no work if you already have a pattern library producing HTML and CSS
- Warhol does not care about the content of a website or app and thus can find errors in production as well as during development
- Warhol is lightning fast, fast enough to check your web pages right in your browser
But, as with everything in life, there's also a downside to this approach: Warhol is totally blind to pixels and will never be able to catch everything that can go wrong in a web page.
For example the classic “CSS is awesome” overflow bug is, quite obviously, a bug. But as long as such a purely visual bug results from CSS that is equivalent to the CSS in the component library, Warhol (in its current form) will not detect it. Warhol is like any other testing tool in that respect – behavior that is not explicitly specified can't be checked.
Precise diagnostic feedback
Of course, we think that this trade-off is still a pretty good deal, as there is another benefit to Warhol's approach: Warhol can't just tell you where and when some design inconsistency crept into your project but you also get precise diagnostic feedback that enables you to quickly find the cause and source of any CSS discrepancy.
Here are some of the problems Warhol can precisely identify:
- Wrong values are obvious: if the text in your component is supposed to be red but turns out to be green in production, Warhol will warn you.
- Warhol has several tiers of reports and does not classify every tiny
CSS discrepancy as an error. If some CSS property has the correct value
but a different
!importantpriority you will get a notification about a wrong priority. If you don't care about mere notifications, you can just filter them away.
- Another notice that you may or may not want to ignore is the one about unexpected child elements. Whenever you define a component in your component library, Warhol also takes note of its content. If Warhol encounters a child element in production that was not specified for this component, Warhol issues a notification. An unexpected child element is not always a design bug but can also be just an oversight in the original component definition.
- But not every aspect of Warhol's reporting is as ambiguous as unexpected
children or questionable
!importantvalues. If whole CSS properties go missing or appear in unexpected places, Warhol will warn you about excess and missing declarations as that's a pretty clear bug.
- Excess and missing pseudo elements take a pseudo element's
contentproperty into account. Simply giving
.foo::beforean unexpected red border will not get reported unless the
- A component library often contains not only components but also general
styling information like the colors and font sizes. Warhol can
incorporate this information into its data model about your components. Armed
with this extra data, Warhol can not only test elements that are not defined
as components for adherence to the style guide but also more precisely tell
you in what way a CSS value differs from the expected style:
- The aforementioned wrong value is reported when a valid, well-defined value is found in place of another valid value.
- An unexpected value is not only different from the value that a property on a given component should have, but is also not a value that was defined anywhere else in the component library.
- Nonconforming value expected is a fun kind of bug. It basically means
that, while a component is implemented as specified, the specification
itself has a bug. If you define
greenas valid colors in your component library but also create a component that uses
yellowas a text color and correctly implement this component in production, the color
yellowis, strictly speaking, still invalid. Warhol performs an internal consistency check of your component library when first reading it, but if you overlooked the resulting warnings the first time around, Warhol will continue to nag you while testing.
These are just some of the (at the moment) 15 different things Warhol can tell you about the real-world implementation of your component library in your project. There is much more to talk about, so stay tuned for more info on the inner workings of Warhol!