Automate Accessibility Tests With Storybook

Captions provided by White Coat Captioning (https://whitecoatcaptioning.com/). Communication Access Realtime Translation (CART) is provided in order to facilitate communication accessibility and may not be a totally verbatim record of the proceedings.

JASON: Hello, everyone, and welcome to another episode of "Learn With Jason." Today on the show we have Varun. How are you doing, man?

VARUN: Not too bad. Thank you for having me here.

JASON: I'm very excited to have you on the show. I think this is going to be a really fun one. And before we jump into what we're going to be talking about, let's talk a bit about you. Do you want to give us a background on yourself?

VARUN: Sure. Long time front end developer, more recently been working as a developer experience engineer at Chromatic, and also contribute to Storybook from time to time. I tend to write a lot about Storybook these days, so you can find my stuff on the Storybook blog, usually.

JASON: Very, very cool. Yeah. So, Storybook is something that, honestly, it became popular after I moved into mostly DevRel, which means in my day job, I haven't actually worked on a team that used Storybook. So, all of my experience with it has been very kind of high level can. So, somebody like me without a lot of Storybook experience but hears the name a lot, can you give me an overview? What is Storybook and what value does it bring to a team.

VARUN: So, Storybook is primarily a tool aimed as UI developers, especially using components to build UIs, one of the biggest challenges is a component has many different variations. Depending on sort of what props you pass in, it will render different UI states or different set of application states or interaction states that the user might have, or even switching teams that your application might support. So, when you're building a component in an application, it's really hard to sort of simulate all of that stuff. You have to drop in temporary bits of data all over the place, and once you're done, you delete it. Whenever you come back to the component, you have to redo all of that stuff over again. With Storybook, sort of the magic sauce, it detaches the component from your application so you can work on it in isolation, and stories are at the different states of the component. So, a story is simply you pass in a bunch of props that will go into the component, and you document that as a story. It's kind of like writing a snapshot test, but the difference is it actually renders that state in the browser. So, you can go through all the component variations really, really quickly and don't have to replicate them over and over again.

JASON: Nice, nice, yeah. It's really nice for me when thinking about I used to work in design systems back in the day, and one of the big challenges we had was it was really difficult to get some of those variations. So, you would build a component, and then you were like, okay, am I really going to go through and edit 50 files to get all of these different variants to be in place and shown. Of course, you never did. So, the docs were always out of sync with the code, and you ran into all these troubles around trying to demonstrate what was happening. So, I think the thing that's exciting about Storybook, is it sounds like it's taken a lot of that, I guess what I would call administrative work, of we made a change, go edit all the docs, and it's making that part of something that happens through publishing code.

VARUN: Totally. Sorry, go ahead. Yeah, with stories, that's kind of the starting point. You've done that work, you now have code that replicates all of those states for you, and all of those examples, and then you can sort of generate documentation from that. That's one aspect. Or if you want to write unit tests, you can use stories as a starting point for a test, or you want to do interaction tests, accessibility tests, stuff that we're going to talk about today. Stories act as a starting point for all of those other things, too.

JASON: Yeah, yeah, and it gets towards something I've talked about this with a handful of folks on the show, where we really are starting to see most of modern business is running on the web. You know, there are businesses that you just I have a lot of businesses that I have worked with and, literally, never done anything except use their website. You know, I buy things through Amazon. I don't know that I've ever had no interaction around Amazon besides I go to Amazon, click a button, and a box shows up at my door. A lot of food delivery services. You don't talk to people, don't have a storefront. You get it, open up the website, make an order, and it shows up at your house. And that's a growing amount of the web. And it kind of means that the companies are putting all of their identity, all of their experience, all of the way people interact with their whole brand, into the web. And when you think about how the web works, it's all driven by code. So, you know, if you're trying to build a company and you don't treat code as the core building block of how everything works, you're creating indirection, right. And, so, that's a long way of saying that in modern businesses, code is the source of truth. So, your design system should be code. Your the way that you build your components, it should live in code. Your documentation is probably going to be mostly code. All of these things that we're doing, you should write extra docs. You should have design assets, but they should use code as the source of truth. And it sounds like this is sort of the same philosophy that Storybook has, if I'm understanding it correctly.

VARUN: Yeah, exactly. It's really hard to keep documentation up to date if you're taking screenshots of the UI, or if you're relying on Figma exports. Because what's in Figma and what's in code, there's usually a delta in that. Yeah, the more you can use code to generate live examples or power everything, the closer you are to the actual truth that's out there.

JASON: Yeah. And this is something we're seeing kind of come up. It's not just in something like Storybook, where you have, you know, your component system is code that is a source of truth, which that one feels kind of obvious, because the only other option would be it lives in Figma, which would be hard, because every time the design changes, you have to figure out what code changes to match that. But you've also got, like, the logic, right. Instead of describing your logic in documentation, you can describe it as a state machine. So, for example, David Korsheed has been on the show talking about logic as code, description of your business logic as code. We've also had, for example, the story to design, which is a pretty cool thing, is letting Storybook generate your Figma. So, now you can use Storybook as the source of truth for your design, literally, your design. You export your Storybook components into Figma. That's extremely cool. We had Tice on talking about that. We're seeing more and more cases where that's the truth, where people are figuring out that, ultimately, at the end of the day, what moves a business forward is their code.

VARUN: Totally, yeah.

JASON: If we use that as the core building block, and everything else feeds back into it, we can all work more effectively together. And the caveat is, that doesn't mean that everybody in the company needs to be a developer. That's why I'm so excited about some of these tools. Like with Storybook, you get an UI. I can look, click, say, oh, this is what with need. And I never have to look at the code. Hey, team, this is the thing that we want. Can we put that on the page? I can play with a hero component, drop down, foreign element, can we do this. I don't have to ever look at the code for it.

VARUN: Absolutely. I mean, design systems are like the starting point for a lot of people to use Storybook, but you get into your application, and there's hundreds of other components in there that are not documented similarly. But if you have stories for them, you know, designers can go in, check into Storybook to see what patterns are already existing, and then use them in their designs instead of recreating that stuff from scratch.

JASON: Absolutely. Yeah. Just a huge amount of potential in this stuff. And, you know, when we're really starting to think about how teams today have to work quickly, and every piece of friction in a handoff between teams just slows them down. So, you know, we see that small teams tend to move fast, but that's usually because those small teams do everything, and there's no handoff points. So, what makes bigger teams slow is they have to get information across barriers. So, tools like Storybook are how you break down those barriers. You no longer have a barrier, because it's not like somebody has to sit down with you, do a training on how design system works, explains all the variants. Just go, look, it's there, copy/paste, and you're off on your business.

VARUN: Totally, totally.

JASON: It's really, really powerful stuff. So, I guess, what got you into this area? How did you find yourself getting into Storybook in general? Was there kind of an origin story?

VARUN: I got into programming because of generator arc. That's how I learned to code. But realized maybe that's not a sort of great career in the long term. So, building UIs was like the next stop for me, where still get to do something visual with code, kind of a natural fit. So, always been in that sort of building UI space. And that sort of led into design systems and Storybook was sort of the go to tool for UI developers and design system developers. That's where I've been using Storybook for a long time. Used to work in consultancy, where just got to sort of try out Storybook on a bunch of different projects, different sort of organizational constraints, and I always felt it was a really good tool to sort of solve a lot of UI development problems and even sort of bridge the gap between developers and designers. Because now there was an easy way to compare sort of what does a component look like in Figma, and what does it look like in code without having to sort of fiddle with code. That was a great starting point. Yeah, and then that just sort of led to a point where I ended up meeting Dominic, who's one of the founders of the Chromatic, and realized that might be a good fit for me to work with Chromatic.

JASON: That's great. Nice, nice. Well, cool. Let's see. So, we're talking about specifically accessibility testing today. And this is something that I it feels like there's really hot debate about accessibility in general, because there's, you know, we've got the one camp that's completely wrong that says, oh, we don't worry about that, because it's not the majority of our users, so we're going to immediately discount that camp.

VARUN: Totally.

JASON: Because they are absolutely incorrect. Then within it, you have the subsets where there's nuance, right. Because so much of this is experience, and experience is subjective. So, should we be using Aria labels, should we be what should you announce when you have somebody moving around different things? How do you make certain interactions accessible that are kind of web native, they don't have standard patterns? What does all of that mean? And, so, when you get into accessibility testing, you've also got this idea of automated testing. You've got tools like the Lighthouse Accessibility Audit, you've got DQs, Ax, and a few others in that camp that will give you sort of an accessibility score. But we've seen people kind of taking it into the extreme. Hey, I'm going to build the most inaccessible website I can and still get a 100 score on the Lighthouse. So, the automated testing isn't doing exactly what we need, so still need human testing. How do you balance this in your mind when you're thinking about accessibility? How do you choose what to automatically test, versus what do you manually test? And how do you sort of balance that process?

VARUN: That's a great question. So, I think there's like a lot of room for the automated tests. You're totally right, if you just rely on them, you can get 100 out of 100 scores and still ship something that is not usable for people anyway. But there's a ton of sort of low hanging fruit. Things like, you know, did I use the right color contrast, do all of my inputs have a label associated with that, are buttons properly labeled, if I'm using a button, does it have an Aria label to announce what the actual value is. Things like that, which you can totally do that manually, going over and over again, but that's the stuff that computers are really good at. You can test color contrast, test for missing stuff. So, that's where automation really helps, you can do that low hanging fruit, more grunt work out of the way, and spend more time testing does this interaction actually make sense. If I go through this as a screen reader, does it actually display content in a way I want it to, or is it completely garbled, makes no sense to a user.

JASON: Got ya. And that makes sense. We have limited attention in a given day. And if we use all of it on just a true/false checklist, then when you get to the more nuanced pieces, you don't have as much brain space left, you might cut corners, might go fast. So, if we let the robots do the checklist, and we instead do the more judgment call, you know, let me open this up and see how it feels on my keyboard, then you can make better decisions and not have already been drained by doing the stuff like does this contrast.

VARUN: Totally, totally.

JASON: Got ya. That makes sense. A question for you, chat. How many of y'all are using automated accessibility testing in your work? If you're using it, what are you using? What tools are you working with? Let's talk about how this works in Storybook. When I think of Storybook, I have traditionally not thought of it as you mentioned earlier, unit tests. I have not thought of Storybook as being something that is a testing. I thought you had Storybook for exploration and documentation and tests off somewhere else. How does testing fit into Storybook?

VARUN: Something the Storybook core team has invested a lot in, in the last year or two, I would say. Accessibility testing was one of the first things we rolled into Storybook itself, because you're working on components in isolation. I would say the go to accessibility testing tool is what powers Lighthouse accessibility testing. You want to run Ax on your UI anyway, run it in Storybook, and it gives you the report there in Storybook. As you're working on a component, it's constantly running Ax in the background and telling you if it found issues or not. That was the starting point for testing in Storybook. Now there's play functions that you can execute, snippet of code, where you can actually write tests within Storybook itself. You can use jest and testing library to write an interaction test right in Storybook itself. Or you can actually take stories and import them into other testing tools. There are utility libraries that allow you to package up a story and use that, say if you're writing a Jest test, that can be the thing you render in the test and run assertions on top of it. You're giving it mock data to a mock scenario, and the third part is running the assertion. So, you can either run assertions within Storybook, depending what you want to do, or use another tool to run that for you.

JASON: I haven't thought about that before, but what you do to set up Storybook is, effectively, the same thing you would do in any other testing setup, except for, as you said, the assertion. Yeah, that's a very logical step I haven't really considered. That's very cool. So, is there anything else somebody should know before getting into this, or does it make sense to flip over and writing some code?

VARUN: Yeah, we can jump into the code at this point.

JASON: All right, let's do it. I'm going to flip us over into paired programming view here, and let's make this happen. So, first and foremost, let me do a quick shout out to our captioning. We have Ashly with us today doing all the live captioning from White Coat Captioning. Thank you very much for being here. And that's made possible through our sponsors. We've got Netlify, Nx, New Relic, all kicking in to make this show more accessible to more people, which I appreciate very, very much. And we are talking today to Varun. Make sure you go and give him a follow on the old Twitter. We're talking about Storybook, so here's a link to Storybook if you want to check that out. And you sent me a repo ahead of time here, which I'll drop a link to folks if you want to check out the code we're starting from. It's got playwright in it, so I did the install ahead of time, because it takes a couple minutes to get installed. I have it here, running, ready to go. What should I do first?

VARUN: Cool. So, this is a little sort of demo project, has a fake UI that I created, and I've already written stories for the components you see in here. So, if you want to go through the code a little bit, everything lives in the source directory itself. So, you'll notice that there are a bunch of components in there. And I mostly used something called chalker UI, really good tool kit, to build out the UI. You'll see most stories have a .stories.js component. So, that's where we write out the stories. Pretty standard Storybook setup, I want to say. Obviously, using React. It's just a CRA project here. So, to get Storybook running, all you have to do is go out to the terminal and run yarn Storybook.

JASON: Okay.

VARUN: That's going to start up the Storybook server, which will serve the Storybook UI.

JASON: All right. So, here is our Storybook.

VARUN: If you go all the way down under pages, you'll see there's a dashboard screen. So, that's sort of the whole sort of page that we're going to be looking at today. It's just a fake UI for admin UF or the Storybook showcase, which is where we display components from all the different Storybooks that we've people submit to us. Pretty straightforward, there's sort of a navigation up top, a form on the page, and some stats. So, what we're going to do is sorry, this is the story for the homepage itself and the ones above this is sort of each component that's used to build up the page. So, there are individual stories for those components. So, if you go into the add project, for example, that is the form, where you can submit an URL to import Storybook. These are the different states. There's the default state, you try to submit something and it's incorrect URL, then you submitted something and it's successfully submitted. Stuff like that. Then, similarly, if you go through the other components, there are stories for those components, too. Footer, not really doing much. Just sort of one story. Same with the glossary. Just sort of renders whichever option you select. And then same thing with the rest of them.

JASON: Yeah. Yeah. So, this is cool, too, because it sort of shows these are the atomic components and this is how it's suggested to put those together, which is also helpful for me, for example. A lot of times when I look at something in isolation. Okay, this is cool, but how would this be used? Because there's also the context. You can see something and see what it's for, but without any context, I might imagine it being something completely different. I might look at this add project and I might be thinking about, like, enter the URL for the Storybook. This is for me to build a dashboard to links to Storybook projects. But I could be wrong. Maybe it's something totally different. Maybe this is going to add all of my components to the left hand side or something. Something that's totally logical, but not the way I thought about it. So, seeing it in a broader context to help me understand how something should be used. That's great, okay. I got it, I'm in.

VARUN: Cool. So, the main primary tools that we're going to be using today is Storybook add on called accessibility. So, Storybook by itself has sort of a core set of feature set, but there's a whole sort of add on ecosystem for Storybook that allows you to extend functionality and add more capabilities. First thing we're going to do is set up the add on. I already installed the add on in this repo, so you don't have to run yarn install for it. If you go into the pack an JSON, there's a A11y, which is what we'll be using. So, in your root directory, there's a Storybook at the top, and a main JS file, where the Storybook configuration lives. So, in here you'll see there's an add ons array and we'll add on the a11y in there. I believe you're going to have to start and stop the Storybook command once more.

JASON: We've updated and I've added one line, no other code changes yet. Restarting. Here we go.

VARUN: So, now you'll notice that there's this accessibility panel that popped up at the bottom. That's what the add on gives you. What it's doing is it's running axe on whatever is rendered in the story right now and giving you the report from axe. You can see there's one error that's found. You'll notice a check box there. If you toggle that on, it actually highlights the Dom node that's causing the issue.

JASON: That's handy.

VARUN: Yeah. If you run axe on the CLI, it yells at you that something is off. It's hard to parse the HTML snippet it gives you to figure out what's causing that issue with this, because you're running the code, you can highlight stuff that has any issues.

JASON: Oh, this is great. Yeah, so, looking at this, I'm now looking at the full dashboard. And I checked the highlight results. Got an issue here, here, here, issue here. This is great. This is really fantastic, because it very quickly shows me where to pay attention. Now I can say, all right, I want to look in the footer, so I want to go in here. Now I've got one violation I need to solve.

VARUN: Exactly, yeah. In this case, what it's yelling at you about is these links don't actually have any text associated with them. So, for users who can see, they can see that icon is a GitHub icon, but it doesn't actually have the the icon doesn't actually tell you that it's supposed to read out GitHub to a screen reader. That's what it's complaining about. That's the component right there. You can see the icon button is simply rendering an icon. Doesn't have a label associated with that. So, let's actually try to fix that. A good way to do that would be to add in an ARIA label. If you do ARIA label

JASON: Sorry, autocomplete for me. First one is LinkedIn.

VARUN: Yeah. I think in React you can do ARIA label as just an attribute, you don't need to do camel case.

JASON: Let's try, see if it fixes it. Links must have discernible text. Is it yelling me about all of these?

VARUN: Did you save the file?

JASON: I did save the file. Let's check if that actually did what I thought it would do.

VARUN: Yeah, see it there.

JASON: We've got our ARIA label. Does it need to be on the icon?

VARUN: You're right, yes, it needs to be on the anchor tag, which is the button that's wrapping it. Actually, no, you have it in the right place. That's fine.

VARUN: Okay, maybe add it to the other ones? Possibly, possibly.

JASON: Are you happy? No accessibility violations found.

VARUN: The highlight is toggled on previously. If you go into the passes tab

JASON: Okay. So, that's actually okay, now it's gone. I highlighted it.

VARUN: Right.

JASON: Is there another if I delete this one then, does it come back? Let's try this one more time. Now you're going to yell at me.

VARUN: Yes, yes.

JASON: One violation. Okay. So, that's what it is. If I put this back in, it will keep it, even though there's no violation, until you reload.

VARUN: Right.

JASON: Got it.

VARUN: The reason for that, you'll notice there's sort of three tabs within the accessibility panel, violations, passes, incomplete. You can highlight stuff that passes, too. It's also a good way to realize what other checks are you running on this particular component that are sort of useful in this case. You can sort of keep that in mind for the future when doing something.

JASON: Great, great. So, we fixed one. Now color contrast again is here.

VARUN: Yes.

JASON: So, then I can go into the add project.

VARUN: Totally. So, this is sort of like an interesting one, where chakra is a fairly accessible UI library, but you can still sort of put in color combinations that are not accessible or don't have enough color contrast. So, what's happening here is there is the sort of button that you see here. Right now the color scheme is set to blue. So, the default white on blue is not high enough a contrast. So, you can change that to I have another color set up called brand. So, change to brand, you'll notice it should now go away.

JASON: Okay. And where does one set those things?

VARUN: That's a great question. So, back in your code, there is a should be like a theme.js file.

JASON: Theme.js. Aha, brand.

VARUN: You can extend the base chakra theme to add in your brand theme, for example. The usual sort of 50 to 900 sort of color levels. That's what it's picking up now.

JASON: Got it. Cool. So, now we've got the so, the button is picking one of these colors, presumably, around this range.

VARUN: Probably defaults to 500. You'll notice that there's a hover style, and an active style, and things like that. It just uses the scale to pick the darker color and sort of the default color.

JASON: Got it, got it, got it. Yeah, okay, this makes sense. What's up, we got a couple folks showing up. Thank you for the sub, Michael, at a year, thank you very much. We got it all. That was me not paying attention to chat. Thank you for all of the suggestions, as we were struggling through that problem. Okay. So, it turns out I figured out, as well, trying to tell me this, if you toggle between the passes and violations, it clears up those, as well, without having to reload the page.

VARUN: I believe so, yeah.

JASON: So, that was also very nice. One violation. Okay, now our

VARUN: The error message there, the red on the white isn't high enough color contrast there.

JASON: Nice, okay. Yeah, so, we can hop in here and look again at our AddProject.

VARUN: That's the text right there that you see. In this case, because it's setting a color, not a color scheme, you can choose specific versions of the color. So, if you wanted to go slightly darker, we could do a red 600, let's say. Red.600. So, it will be a slightly darker shade of red and that should work then. Again, went away.

JASON: Yep, go over to the passes, come back, highlight is gone. Slightly darker shade of red, passes accessibility. This is great. So, this is oh, the interactions, this is the part you were talking about, where we're actually scripting. It submits something bad and gets the failure.

VARUN: Exactly, yeah. We'll talk more about that shortly, too. But, yeah, generally, this is sort of the flow. You're working on a component, it's running axe behind the scenes, you're getting the report, you can fix stuff as you go. That's sort of like the primary sort of features of what the accessibility add on does, but I want to show one more thing.

JASON: Yeah, please.

VARUN: Top of the tool bar, you'll notice there's that little accessibility icon. If you click on that, this is the other feature set, which is probably more for sort of designers. You want to simulate different vision defects a user might have. So, blurry vision, different sort of color blindness. So, you can use that as a way to check whether the color contrast makes sense, or the text is supposed to look slightly different actually stands out from the default text or not. So, things like that, you can test that here. That's the other sort of feature that the add on brings in.

JASON: Yeah, this is cool. So, this is any form color blindness isn't just an on/off. There's a whole spectrum. We've got the cones in our eyes, and that's why we can see colors, so you can have different combinations of the cones not playing nicely with your eyeballs. And that is a little bit of medical info with Dr. Jason today. I have no idea what I'm talking about. That's just a thing I read once in a brochure. No, this is really, really this is handy, too. This one, this is actually something that I know affects people for real, but it's also a design trick I was taught early in my career, to kind of squint at your designs until your vision blurs, because it will show you what's taking visual precedence. So, if we look at the dashboard, for example. We've got things blurred. This is clearly the thing people are going to read first, this is what they'll read next, see these numbers. Blur is a very useful tool for understanding visual hierarchy. It's also very nice to see, okay, if your vision is low, you can't really read that. You're not going to be able to read the footer at all. So, making sure that you've got like here, I bet that's an accessibility problem, because that's definitely really hard to read with blurry vision. This alone, even if you weren't thinking about it for folks whose vision is just like that, you can use this as a tool to understand, like, oh, I made all the text too small. If I'm in bad lighting, I'm not going to be able to see this. If anybody is not wearing glasses today, they are not going to be able to see this. Those are really I love that. That's a super handy tool.

VARUN: Yeah, that's a great tip about the blurry. Yeah, totally right, you can see the button stands out compared to everything else around it. Cool. This is sort of like the dev workflow. You're jamming out components, and you want to make sure things are the way you expect them to. The other bit is sort of like you're ready to merge your code and you want to run accessibility tests on all of your components at once. Right now, you have to go through them one by one to go through stuff. So, for that, we're going to use the Storybook test runner. And that's sort of the other interactions panel you see there, is powered by the test runner. In this story we don't have any interactions. The add projects is the one we have interactions. If you go back into the URL validation error. You can see what exactly is happening. It's starting off by looking for a particular input. Input that has URL as a label, typing in dash Storybook. This is a testing library in Jest, I should say. Two pretty standard tools, I feel everyone loves testing library, it's a really, really nice library to use, amazing developer experience. Also, it is a library that actually tells you hoe to takes a more user centric approach, I want to say. When I say user centric, I mean the end user. It tells you to find stuff the way that a user would, finding it through accessibility labels. It actually works really well for the accessibility testing side of things, too. What's happening here is we're using variations of Jest and testing library that have been instrumented, so that they can, one, run in the browser. And, secondly, they can then be sort of debugged better within Storybook itself. So, there is an @Storybook/Jest and testing library that's installed already in this repo. You can use utilities from them to write out the actual interaction bit. So, if you go into the add projects stories file, we can walk through one of these play functions.

JASON: I see here we're pulling in the testing library and Jest, and we have an insertion, I see the user event within. We'll have to look at what that does. Okay.

VARUN: Template we're using for all the stories. Then for the URL validation error use case, we're trying to simulate an error scenario. So, what you do is, to the story, you attach a play function. This is an asynchronous function. Again, testing library has some synchronous operations, but primarily it's asynchronous operations, so you can use the async pattern within this play function. And the play function receives the canvas element. This is the container for the story, the node that has the story rendering inside it.

JASON: That's basically this here.

VARUN: Exactly, yep. What you can do is, within a function from testing library, is really cool. It allows you to scope all of your queries within a particular Dom node, so makes it much easier to find stuff.

JASON: Okay, that makes sense. Then you're no longer having to do a, okay, we have a dashboard, and within the dashboard, I have to find all these pieces, and there's a chance an outer container changes and really your test is unaffected, but because the outer container changed, everything breaks. By scoping that's nice, that is really nice.

VARUN: Right. And then the rest is just a standard test that you would write in Jest and testing library. Sending in whatever test you want to type in and trigger validations. That's then what triggers the validation error. So, you could also add another sort of assertion at the end to assert what the validation error is that you want to see, or in this case we're sort of using that to simulate a particular state in the story. And that's totally fine, too. That's sort of one of the primary cases of the Storybook test runner. And, again, these interactions as you are going through different stories. So, if you go back to your terminal, what you can do is while Storybook is running in one terminal window, you can open up another window and run yarn test Storybook. This is starting up the actual test runner. And what it does is converts all of your stories into tests. Stories that have a play function. It's going to make sure that the play function fired correctly and all of the assertions are passing. Those that don't have a play function, it's simply making sure the story rendered without any errors.

JASON: Right.

VARUN: You have tests for all of your stories. And this under the hood is powered by Playwright, which is why you had to wait a bit for the initial yarn install. So, because we have this testing set up, we can actually use this to run axe 2 on all of our stories, so that's what we're going to do next.

JASON: Love it, okay, yeah. I'm into it. How does one do such a thing? Do I start in the same story folder?

VARUN: So, to run axe with Playwright, we're going to use Axe Playwright, another wrapper library around Axe. I believe that's already been installed. Should be in the packages, yeah, great. So, along with running those sort of play function based tests, you can extend the test runner using something called test hooks.

JASON: One second, I realize we didn't link to this tool. This is what we're talking about, accessibility testing. And that is something you can throw into all your projects. Okay. So, now that we've got that, you said to do something, and I was trying to find a website and not paying attention, sorry.

VARUN: No worries. By default, the test runner will run the play function tests, but we can also extend it and add more functionality to it through a hooks API. These are hooks into the test runner. So, we're going to set that up, and we need to configure them. So, we'll go back into the .Storybook folder and add in a new file called test runner.js. This is where we'll sort of define the test hook. But the test hook is simply you do sort of a module exports and it's an object that has a couple properties. First is a prerender function and post render function. Sort of in camel case. To run axe, prerender is before the story has been rendered, do something. Post render, after the story has been rendered, we'll do something. Cool, so, the process for running axe, before the story is rendered, you need to inject axe on to the page. After the story is rendered, we're going to run the axe Jest.

JASON: Okay, first we want to inject it. So, we do that before we start rendering on the page. This is after the components are up, the canvas is loaded, JavaScript is completed, we are ready to use this UI. Then we can do some stuff. And that make sense. Obviously, if we try to run a test before the UI finished rendering, everything is going to explode. That all makes sense. So, to inject axe, we've got, what, axe playwright.

VARUN: Yes.

JASON: Do I just need to import something up here?

VARUN: Yep, depending what version you're using, skip to require statements. We're going to require axe playwright. And you can sort of destructure the require there, and the two things that we need are inject axe and check a11y. There you go. Yeah. So, the first step, inject axe. Pre render and post render functions receive a page object, so we need to grab that page object and pass that into the inject axe function. The second thing is these are all async operations. So, we need to make these async functions and then I'll wait for the inject axe function there. Cool, that's all for the pre render stuff. For the post render stuff, similar pattern. Get the page object as the first parem and pass that into the check a11y function. Yeah, that's it. That should work.

JASON: That's the whole thing?

VARUN: That's the whole thing. Almost.

JASON: I love.

VARUN: I should point out one thing. By default, axe is set up to run on an entire page, which makes sense, because when you're running axe through Lighthouse, you're running it on an entire page. It's going to check for stuff like do you have an H1, do you have a main tag define, things like that.

JASON: That's true. With Storybook, that doesn't make sense, because we're rendering a single component. Of course we don't have an H1 in most of these. That failure would be kind of a false positive.

VARUN: Exactly. With the check a11y function, you can configure a little bit to tweak it to run on a particular sort of component. Actually, there's a really good question in chat right now around why would I use axe playwright over just axe. Good question. The reason here is because the test runner runs on playwright. So, you have to use something that runs on playwright. If you were running just Jest, yeah, just use axe. Because we're running a test on playwright, we have to use the playwright flavor of it.

JASON: Got it. Test runner, is the way this works in Storybook, I name something with the file, or with the plug in, and I can extend? Okay.

VARUN: Depends on the add on a little bit, yeah.

JASON: So, if we weren't using test runner and were only using Jest for unit test assertions, we could use Jest axe instead.

VARUN: You could, but in that case you would be importing stories into Jest, a slightly different pattern, and also a total valid pattern.

JASON: Gotcha, gotcha, cool. So, with this check a11y, we're passing it in the page, but our page is not complete, so we need to tell axe that it's a fragment. Or what's the terminology?

VARUN: Tell it you're testing a Dom node. All you need to do, second paren, what is the element you want to target. In this case, we can do passing a string with a value root. That's the selector for the element within a story. That's what we're telling. Lastly, there's a third parameter we can use to sort of configure how we want to what kind of report we want to get from axe. Let's enable a couple options. The first is detailed report. We will set that to true. And the second one is detailed report options. This is going to be an object, and we'll just set HTML to true. So, what's happening here is when test runner runs now, detailed report true means it's going it give you a table, which tells you similar to sort of the stuff we got within the Storybook UI, you'll get a check off. What checks passed on what particular Dom node, and HTML true will actually give you a snippet of the HTML code if something failed. So, that's all we're doing here.

JASON: Oh, okay. So, this should be interesting, because we know, we set up the accessibility plug in for Storybook, and when we opened Storybook. So, if I go back out to the actual code, and I look at my dashboard, I can see that I do have problems. But when we ran the test, the test passed, because we weren't running accessibility checks as part of our testing. So, this would be confusing, right, if we don't want to send inaccessible code out, this would be wrong. Our tests are lying to us. Our code is broken, and we're not catching that. So, what we've just done is we've said, okay, in addition to the other tests you're running, also run this accessibility test. And include the report. So, now what we should see, if I run it again, and correct me if I'm wrong here. If I run this test again, what we're going to see is it is going to give us the unit test pass, but we're going to get an accessibility failure.

VARUN: Totally, yeah, definitely see those.

JASON: All right, let's roll. Here's our six tests, they are passing. Probably. They didn't, because the accessibility test went into I understand. So, check this out. We do have tests passing, but if we go up here I don't think I'll be able to make this big enough to make

VARUN: Yeah, that is the reason why we have the add on, so you get better debugging information within Storybook. This is sort of the you want to run accessibility on CI setup. That's where this is useful.

JASON: Yeah, trying to make this big enough. There we go. I hope that text is not too small for everybody to read. That's not going to help us. Only some of these are going to be readable. We've got in the stories, no default:no violations, no violations. And then in the

VARUN: That's one of the stories that has an issue. These are for the color contrast for one of the stories. That's what it's complaining about there.

JASON: Got it, got it. Then in the submit success, add projects, submit success. We get a violation. That's what this is up here, for the submit success, is that correct?

VARUN: I believe so. Runs all the tests and at the bottom is the summary for that one component, because it has multiple stories, so it gives you sort of a summary for that one component.

JASON: Got it. Okay. And down here we have our dashboard test, three accessibility violations. And if we look up here, here's the actual logs. I understand, I understand. Cool, now we're getting failures. I'm going to make this bigger again, so we can see the text on the screen here. All right. So, now we can go in and fix these, and that will clear up this test, and now our builds will fail when they should fail.

VARUN: Yep, yep, exactly. Really, the workflow here is, while you're working locally, you can just rely on the accessibility add on to run the checks for you, so you don't have to worry about the test runner. Because the test runner is set up, if you run the tests on CI, you're also running the checks across the entire Storybook, so you get the CI checks that are helpful. It's kind of split between the two. One is more locally, other is more for CI. So, that's the core sort of accessibility automated accessibility testing workflow with Storybook. But I do want to sort of get into one more part, which is and you sort of touched upon this earlier. Get the robots to do sort of the low hanging checklist and give us more time to do the more qualitative analysis of the UI. So, if you go back into the Storybook and just open up the submit failed story there for the add project. All right. So, what's happening here is you tried to submit the form and there was some kind of validation error, and you see that validation error. If you pop open your DevTools, let's go inspect the UI and see whether for timing, you can close the add on panel, that's totally fine. You can select the level alert that you see down there. All right, cool. In your DevTools, I believe you have the on the right hand side, where you see sort of styles computed layout, there's an accessibility one. And if you don't mind, could you enable the full page accessibility tree? And you have to read DevTools now.

JASON: Oh, reload DevTools. Got it, I missed that button entirely. Okay.

VARUN: No worries.

JASON: So, let's inspect again.

VARUN: Now you have that little accessibility icon on the left hand side. If you enable that, this is a really cool feature that sort of landed in Chrome recently. This gives you the accessibility tree for your entire UI that's being rendered, so you can actually go through and sort of explore everything that you're seeing here. So, that's the static text that we're getting. Technically, that is supposed to be an error message associated with the form element. But if you go sort of select the form element, you'll notice that it's they are not sort of associated with each other. Yeah, that totally works. You can see that it has sort of the that's the text box you're trying to edit. Doesn't have any validation error associated with that, and the alert is completely detached. So, it doesn't tell you that this is the validation error that you encountered. And the reason for that is the error message is just like a div rendering the error message underneath the input field. When you want to display an error message, connect it to a form field. You use something called ARIA label described by and associate the ID of the error message with the input.

JASON: Got it, okay. So, then I need to go to my AddProject.

VARUN: Indeed.

JASON: I have my error message. And this is where I would wait, not ARIA described by. This would get the ARIA described by.

VARUN: Right. The good news for us is, chakra actually does that stuff for you under the hood. You just need to make sure you use the right components. Chakra has a component called, I believe, form

JASON: Form error message?

VARUN: Yep, that's the one. What we can do is we can swap out the text with the form error message. And then under the hood, it just like knows to use the assign it an ID, ARIA input.

JASON: For real, I don't have to associate it or anything?

VARUN: Nothing at all.

JASON: That's clever, okay.

VARUN: Now when it reloads, you should see that it's actually sort of tells you that it's it's sort of like there's a validation error, and it's associated with that particular component.

JASON: Where do I see that? Do I need to reload?

VARUN: I think if you toggle away from the accessibility tree, let's just first make sure that if you toggle that off, let's just make sure that it's getting the ARIA label stuff associated with it.

JASON: Here's my input. Acquired. Let me reload.

VARUN: I think I know why.

JASON: Okay.

VARUN: Right now it's sort of conditional. You don't need to do the conditional rendering anymore.

JASON: Delete it all together.

VARUN: Yeah, the check there, you don't need that anymore. Should work without that. And, also, on the form control, we need to sort of add in an invalid state. Yeah, that's already there. That's great, cool.

JASON: Got it, okay. So, then if we look at this again, div input. I still don't see any ARIA.

VARUN: Yeah, that's interesting

JASON: Let's try alert. I'm going to reload one more time, just in case it's something with the with it not picking up for us.

VARUN: What I was going to do next, you can actually write a test for it to do exactly that. So, we can write a test that confirms whether it's working or not.

JASON: Okay. How does one do that?

VARUN: So, we'll go back into the stories file for that particular component. And in the stories file actually, I know what the problem was. First of all, we're in the wrong story. That's the submit alert.

JASON: We're in this one.

VARUN: Exactly. That's the one we should have been looking at. If you check now

JASON: Let me get back to the actual input.

VARUN: If you expand the label text, I'm curious. Now you can see that it's part of the group. So, the static text that you see from the validation error is part of the sort of input control group. If you toggle back, the regular HTML tree, I think you should see the ARIA label. There you go.

JASON: Okay. So, this was just good old fashioned you're editing the wrong file.

VARUN: We were editing the right file, just looking at the wrong story.

JASON: Yeah. Okay, so, this one would also be the error message, is that right?

VARUN: So, that one is correct, because that's an alert that you want to render on completion of the form. So, that will get announced as an alert. So, that part is correct. It's just the validation error that was incorrect.

JASON: Got it. So, we did the right thing. We were looking at this submit fail. Should have been looking at URL validation error. All right, I understand. How does one test such a thing? I'm going to get into our stories. And I have this, this is the validation, that is the one that used the alert.

VARUN: Yeah. So, in that one, after the blur event, that's what we're going to check. The testing library has a really cool function called to have accessible description, where you can grab it and check to see whether it has the correct description associated with that or not. First thing we want to do is make sure the input is set to be invalid. So, we'll do expect URL input to be invalid. And this is a function. This is essentially checking to see whether the input status has been set to invalid, either using ARIA or normal Dom APIs. The second one is going to be expect URL input. And this is going to be to have accessible description. This is where we expect the error message. If you type in that, then the test should pass. If you type in something else, the test will fail. I don't think there's a period at the end, but that's why the test should fail right now.

JASON: Yeah, let's have a test. Break it. Break it to start.

VARUN: You don't need to run test Storybook. You can just go into Storybook.

JASON: Didn't like that.

VARUN: Re run the story. Refresh the page, or there's a refresh icon. That should work, too.

JASON: Within is not a function. What did I break? We don't have to extend this or anything? This comes out of testing library, right?

VARUN: Yeah, yeah.

JASON: This is Jest, isn't it? Do we have to do anything

VARUN: That's a good point. It does come out of Jest, but I think you should be good to go. I don't think you need to do anything for that. Try refreshing the page.

JASON: Let's do that. Turn it off and on again.

VARUN: There you go, okay. If you open up the add ons panel again, you can hit A on your keyboard, or, yeah, click that button there. Either works.

JASON: No, come back! Not working for me.

VARUN: One on the far right hand side. There you go. Interactions panel, you should see the failing test.

JASON: Expect to be invalid. It is not currently invalid.

VARUN: Okay. So, that part does make sense to me. So, what's happening here is we're using I'm using form it to handle the form submission and the validation logic in here. And one of the challenges is it takes sort of pushes stuff to the next. Generally, when writing a test with it, you have to wait for things to complete before you can run those checks. So, what we can do is those last two lines of code, we can actually wrap that in a waitFor function from testing library, and you have to await that function. And this

JASON: This is

VARUN: Just an error function, which will wrap the next two expect statements. So, now if you go back to the test, the first part should be correct, that it does set it to invalid.

JASON: Rerun. First part is invalid. And now it's saying it doesn't have a correctly formatted URL. So, I'll fix my text to match. Rerun. And now it's happy.

VARUN: There you go.

JASON: That's great.

VARUN: So, that's like a really good example of sort of the more nuanced UI development, where, you know, you throw in a div there, and axe ran all the checks on it, everything was passing, but you weren't actually announcing any error messages. So, it's really easy to miss that kind of stuff.

JASON: Really easy to miss.

VARUN: Totally. So, I think testing library is fantastic for that kind of stuff, where you can check to see whether alerts are being announced, or things have the correct accessible description, not visually what you can see.

JASON: Yeah, yeah. Well, I mean, this is a cool one, too, because I didn't realize that testing this was that straightforward. I would have expected that we need to actually check for the ARIA labels and, you know, you're digging into attributes and stuff. Is there a way that you can say I really just want it to have an accessible description, I don't care about what the text is?

VARUN: Oh, that's a good question. I'm not sure. I feel like I might have to sort of go into the Jest documentation.

JASON: It's okay. That can be homework, for anybody who's curious about these sorts of things. Varun, what else do you want to look at here? We got a little bit more time.

VARUN: That was like everything I wanted to show you. Not sure if you had any questions or anything else you wanted to dip into.

JASON: Chat, how about y'all ask any questions that you've got. If we don't get any questions, we are going to call this one a success. Varun, while we're waiting for folks to think about their questions, where else should people go if they have if they want to do more on their own? I'm going to drop the Storybook link again. What else should we look at?

VARUN: Up top there you see a tutorials link there. In tutorials, the third one down, the UI testing handbook. That's going to be my little plug here. It has covers all the stuff that we talked about regarding accessibility testing. But it also goes into sort of other aspects of UI testing that you can do with Storybook. So, things like visual regression testing, user flow testing, or automating the whole thing with the CI. Powers all that stuff. Really fantastic resource, because I wrote it, so I'm going to take be a little bit humble there, but, yeah, I think it's useful. I see there's a question

JASON: There's a got question here about using this with other languages. Can you use Storybook with Angular?

VARUN: Totally. The nice thing is, under the hood it's using testing library DOM. So, not typed to any front end framework. As long as you're using a framework that renders HTML, Storybook itself supports pretty much every single major front end JavaScript framework. React, Angular, Vu, velt.

JASON: Nice. Solid JS, Qwik, or any of those?

VARUN: I believe in the works for sure.

JASON: That's exactly. Let's see. Enjoy the live says is there anything I should be doing to make this work with Chromatic, and any examples if so?

VARUN: I'll break that into two parts. The accessibility testing part, that is something that's coming to Chromatic. So, for those who don't know, Chromatic is a visual regression testing service that's built on top of Storybook. So, it takes a snapshot of all of your stories and then runs a visual diff between them. The accessibility testing bit is going to come to Chromatic most likely early next year. So, you don't have to do the test runner setup. It will do that for you, if you're using the accessibility add on. But for the time being, you have to use the add on to run the tests in CI. For the play functions stuff, they work out of the box with Chromatic. For example, those form stories that we wrote, where you click around and fill out the form, Chromatic is going to wait for all of that stuff to happen before it takes the snapshot. That kind of works.

JASON: Got you, got you, got you. So, the comparison then I guess would be in any project, you're going to pay for it. So, you can either choose to pay with time, or you can pay with money. And, so, with Storybook, you can build all of the things here as your own kind of automated workflows, and you're paying with time. Or if you want to solve it with money, you can go over to Chromatic and it just works. And then you get to keep your time and just part with your money. That is a very nuanced discussion that's worth probably a whole episode's worth of digging into the nuances of value and tradeoffs. But, yeah, that's super handy. Let's see. So, JL is asking does it work with different view ports?

VARUN: It does, both on the Storybook side and Chromatic side. If you go into Storybook, you'll notice that icon there, yep. If you toggle that, you can switch between different viewports. For our purposes, it's not really going to do anything here, because the UI is pretty much the same. But if your UI shifts between different viewports, you can use that to run tests across the different viewports. Same thing for Chromatic, it does support different viewports, so you can run your visual tests across different sizes that you want to.

JASON: Nice. Cool.

VARUN: And these viewports are configureable, so, if you have specific breakpoints in your application, these are sort of the default set that it comes with.

JASON: Cool. All right. So, Varun, not seeing any more questions. I think I'm going to start doing the wrap up here. So, I'm going to send everyone to your Twitter. We've got a whole bunch of different resources for folks who want to go and dig a little bit deeper. Do you have any parting words for the chat before we call this one a win?

VARUN: No, this was great. Thank you, thank you again for having me here.

JASON: Yeah, I had an absolute blast. This was super fun. I'm really impressed with just how much we were able to do on the, just on the work here. Source code, yes, I will post the source code. Actually, let me do that now while we're thinking about it. So, what I will do, I've got this, I'm going to GitHub repo fork, and I'll put it up on the oh, you put it on add a remote. Okay, and, so, now this is in the wrong repo, but that's a pain in my butt. Okay. So, I'm going to git add everything that we did. Git commit and say code after episode. And we will git push. Git push origin main. And then we can GitHub actually, just grab this, here. Do one of these. And this is the code that we actually built today. So, if anybody wants to see that, we'll make sure that shows up in the show notes, as well. Cool. All right. So, this episode, like every episode, has been live captioned. We've had Ashly here from White Coat Captioning all day. Thank you very much for that. That's made possible through the support of our sponsors, Storybook, Nx, and New Relic. And maybe your company. Hit me up if you are looking to sponsor a stream. We've got a lot of good stuff coming up on the schedule. It's going to be a whole bunch of fun. Next week, we are going to find out what's new in Astro v1. They are going 1.0 stable, super excited about that. Fred's coming back to teach us how that works. Then we're going to have Adam Argyle teaching us how CSS Open Props work, which is really cool. Open Props is sort of like Tailwind, but if it was built entirely in CSS, and I find that very exciting. Adam is going to teach us how that works, and we're going to build an UI with it. We have Will coming back. I have another four or five episodes in my inbox that I really, really need on the site. Stay tuned for those. Varun, this has been a blast. Chat, as always, thank you so much for hanging out with us. We're going to go find someone to raid, and we will see you all next time.