Feature Request: Automated Fixture Error Check

[shahkashani]

What's up?

We rely on Cosmos as component library for various purposes. We push it to Github pages to show progress on various parts of the app. Some engineers even use it on a daily basis for feature development (while others rely on in-app HMR).

Once in a while, fixtures break. This sometimes happens when someone adds a new required field to a component and forgets to add it to the fixture. Other times, it's errors like You should not use <Link> outside a <Router>. It's probably most typically caused by folks that don't use Cosmos for day-to-day development.

Either way though, it'd be great to be able to catch these regressions in PRs. We were planning on writing a headless browser script that traverses the entire tree for pages that error out, but what would be even better (and faster!) was if Cosmos shipped with something along the lines of a cosmos-validate command that could tell you if a feature was broken.

Mkay, tell me more...

I'm not sure if you'd eventually end up having to automate a headless browser for this, but I'm hoping there's a programmatic way and some internals that makes it easier for you than for us.

[ovidiuch]

@shahkashani Great idea! Did you try the Global Jest snapshot? I think it can serve as overall fixture validation, diffing aside.

[shahkashani]

@skidding Wow, totally missed that feature. What a great idea. Thanks for the heads up, will take a stab at it!

[shahkashani]

Would love to bump this a bit. We've been doing what you recommended and while it works technically, it's a little painful having 20K+ lines diffs in certain PRs (we use Cosmos extensively).

[ovidiuch]

Hey @shahkashani!

I agree this is an important feature. It's not quite in my top priority list right now, but depending on what solution we come with and how straightforward it is to implement I might accelerate this feature's priority.

Here's the solutions I envison at the moment:

1. UI button

Have a "check all" button in the UI next to the tree view. When you press it, Cosmos traverses all fixtures and captures any runtime errors. As fixtures load, they can either be visible in the preview area (similar to e2e test runners) or invisible in an additional hidden iframe. As errors are captured a (!) icon can appear next to each fixture. The error state can be sticky (possibly remembered in browser storage like we do with other stuff already) until the fixture is loaded again without errors---either via "check all" button or by selecting that specific fixture normally.

This is useful in development because you find out about errors quickly. But not so easy to run from the CLI. More on this soon.

2. CLI command

Have a react-cosmos-check command, similar to what you proposed, that spins up a headless process, loads all fixtures and captures errors. This is neat, but I see two avenues here:

A. Use something like Jest, which comes with a pre-configured jsdom environment. This would require less work but would couple Cosmos with Jest, something I'd like to avoid in the future. At this point this might as well be a 3rd party project that uses Cosmos + Jest to achieve this function. Also, and maybe more importantly, even with jsdom being near browser perfect there will undoubtedly be annoying cases where a fixture works in the browser but not in jsdom or viceversa. This might be the quickest approach but it smells danger to me.

B. Wire up the headless browser manually and use proper DOM rendering. This removes the "works here but not there" issues, allows us more control over the user experience and doesn't couple Cosmos with Jest (or any other alternative). But this obviously requires more work.

3. The Holy Grail (1 + 2)

Now that I've layed down these two options, I'm thinking the ideal scenario is to have both. Add the UI functionality to validate fixtures and quickly see when a fixture no longer works. Then create a CLI script that spins up a headless browser, which opens the Cosmos UI and presses the "check all" button (either literally or triggers it through special URL route) and collects the names of the fixtures that errored. The script can display that information nicely and use it to determine the status code (brokenFixtures.length > 0 ? 1 : 0).

I know it's a lot but I'd rather invest more in the right solution that waste time with shortcuts that will give us headaches later on.

I'd love to hear your and other people's thoughts on this!

[robhdawson]

Hi @skidding! A colleague of @shahkashani's here. Since we last commented here, we've actually worked out a local patch for this. Basically, it boils down to swapping out the screenshot expectation in react-cosmos-telescope with something like:

expect(() => {
  wrapper.toJSON();
}).not.toThrowError();

All your solutions seem great, so if this is the kind of shortcut you'd rather not introduce, I get it. But it's been working pretty well for us so far, and I'd be happy to make it nice and configurable and contribute it here, if you think that'd be worthwhile.

[ovidiuch]

Hi @robhdawson,

I'm happy you found a solution that works for you. The solutions I was discussing are meant for Cosmos Next anyway, and will likely take some time before becoming reality.

Meanwhile, if you'd like to contribute to react-cosmos-telescope and enable other existing Cosmos users to benefit from your solution I welcome that!