Product Quiz

Question 1

How does the current Slack integration work? What are the current notifications?

Answer 1

You can set up a Slack integration using custom webhooks. The notifications are generally around the behavior of the build - build is ready for review, it’s passed, it’s accepted, denied, broken, cancelled, error, timed out. It won’t tell you when you’ve received a comment or an at-mention.

Question 2

Which standards does @storybook/addon-a11y test against?

Answer 2

That addon uses axe-core, which tests a subset of WCAG 2/2.1 It can only test a subset because not all WCAG rules (and especially not all accessibility considerations) can be automatically tested.

Question 3

How is interaction testing different from E2E testing? // Can this completely replace my use of Cypress?

Answer 3

With interaction testing, you can start to map out user flows. Interaction tests can replace almost everything in most E2E providers except for maybe running Smoke tests on a preprod or prod environment. Interaction tests focus on the hosted Storybook in the cloud not a production URL for a company.

Question 4

Why would we use Interaction testing in Storybook instead of Cypress, especially now that Cypress has component testing?

Answer 4

That’s a great question. The way we see it is that if you’re doing your component testing in Storybook rather than in Cypress, it’s kind of like working where your tools already are. We see it to be a more seamless method of testing as you’re testing in the location that your components are already located.

Question 5

How do I change my username?

Answer 5

If you signed up via email, you’ll be assigned a username based on your email. Go to the Profile page to change your username. Get there by signing in and clicking on your avatar in the header.

Question 6

How will I be notified that my PR has been reviewed?

Answer 6

The OWNER of the PR will get a message.

Question 7

How do I change notifications?

Answer 7

In the top right corner you click your avatar, then click “notifications” and then you can set them from that page. Notifications are set at the project level.

Question 8

We need to give billing access to another account owner as well. How can we do that?

Answer 8

We sync permissions from your git provider, so you’ll need to give that person admin access in your git provider in order for them to have that level of access in Chromatic.

Question 9

Can you add external collaborators at an organizational level instead of a per-project basis?

Answer 9

You currently cannot add external collaborators at an organization level, only on a per-project basis. (So this is feedback to aggregate for the product team!)

Question 10

Can I create a new project that isn’t connected to git?

Answer 10

Yes, you can create a Chromatic account with an email & password option rather than signing up with a git provider and this will create an unlinked project.
Alternatively, you can create a project under a personal account even if you signed in via GitHub. To do this, in the top left corner, please select your username, then “create project.”
Please note: It’s not possible to create an unlinked project under an organization.

Question 11

What are the different roles in Chromatic?

Answer 11

Owner: can manage, delete the project, and manage/assign roles to collaborators

Developer (default): can manage the project, review tests, approve PRs, and assign reviewers. Cannot assign roles to collaborators.

Reviewer: can leave comments, review tests, and approve PRs they’ve assigned to. Cannot assign others or self-approve.

Viewer: read-only access to the project.

Question 12

Does Chromatic give permissions at the org level or project level?

Answer 12

At the project level. But once you’re added to a company’s organization in GitHub, you will see all of their projects (that you have access to in GH). You have to click in to the project to belong to that project.

Question 13

When someone gets added to your GitHub, they automatically also get access to Chromatic also, right?

Answer 13

Yes, our permissions mimic a repo, so If I add someone on GitHub, they can access Chromatic. They won’t show up on the Chromatic users list, until they visit Chromatic for the first time.

Question 14

Someone left our company and therefore left our GitHub organization, but they are still coming up in our Chromatic account. Is this a security risk?

Answer 14

If the person was added to the project by nature of them having access in github (vs manually adding someone by email), then when they are removed from the github org, they wont have access in Chromatic anymore. They will show up on your collaborators, and they will removed when they try to access the project again. They will not have access to the project, if they have been removed from GitHub. In general, the person has access to the repo in GitHub, they will have access in Chromatic, otherwise they will not (even if they show up on the collaborators list). If there are accounts you want to delete manually, we can have our team do that (I’ll just have you confirm that via email).

For anyone who signed up with email and password, they can be deleted from the manage screen (and they’ll show up in the “external collaborators” list.

If they want people to be deleted manually from Chromatic, as of now, we have to do that on our end.

Question 15

Can i make a project private?

Answer 15

Yes, but you’d do that through GitHub. If the GitHub repo is private, then the project will be private in Chromatic.

Question 16

Can a user be linked to multiple accounts?

Answer 16

Yes, a user can belong to multiple accounts

Question 17

Can I give different people access to different projects in Chromatic?

Answer 17

Not really. Whoever has access to that project in your git provider will also have access in Chromatic.

Question 18

Does the Figma plug-in work if you have a private Storybook?

Answer 18

Yes, it does.

Question 19

Can you choose which version of the design from a Storybook is shown in Figma?

Answer 19

No, because the Storybook story shown in the plug-in will always point to the most up to date build for that branch.

Question 20

UI Tests vs UI Review

Answer 20

In a nutshell, UI Review (PRs) compares snapshots between two branches - the head branch of the PR (e.g. “feature”) and the base branch (e.g. “main”). It uses the last build on the head branch and the “merge base” build, just like code review does on GitHub. This allows you so see what would visually change when your PR is merged, perfect for QA and stakeholder review. The “changeset” is then approved as a whole, just like the PR itself would be on GitHub.

Conversely, UI Testing works on the concept of a “baseline” for each story. When a story has visual changes compared to the baseline version, you are asked to approve it. If you don’t, the old baseline remains, if you do, it becomes the new baseline. The big difference in workflow is that you approve individual stories, which means you can approve only those stories which are “ready” and continue working on those that aren’t. This workflow is intended to help individual developers verify their work, even before they open a PR.

Another big difference is that baselines carry across branches. When one branch is merged into another, the approved baselines are carried over. In case both branches have approved baselines for a particular story, the one which was approved last “wins”.

In a very generic sense, UI Testing is intended to be used to test your UI for visual regressions (and a11y/interaction regressions when we incorporate those), while UI Review is meant to be used for visual review of the changes by someone other than the author (e.g. designer, QA, PO/PM).

Question 21

Can I rename the build name?

Answer 21

No, you cannot currently do that.

Question 22

Can I cancel or delete a build?

Answer 22

You cannot currently delete a build.

Question 23

Can I archive a project?

Answer 23

Yes, you can

Question 24

If someone makes a new component and opens a PR, will it show blank on the left side (since it has nothing to compare it to)?

Answer 24

It’ll just show the one component - it won’t be blank. It’ll just not have a second side. And it’ll flag the component as “new”

Question 25

Does it cut down on snapshot usage if you disable UI Review?

Answer 25

Yes, it will. We take snapshots for the latest commit on a branch and compare it to your main branch baseline, so snapshots are definitely used when doing this.

Question 26

Can I turn UI Review or UI Tests off in my GitHub Actions?

Answer 26

Yes! You go to the “Manage” screen and can click enable/disable for each of them.

Question 27

Can I enable UI test/UI review only on some builds? I want to use UI test/UI review only on release new version PRs for example.

Answer 27

No, you cannot enable those features at the build level. They must be enabled or disabled for the entire project.

Question 28

What browser does UI Review use?

Answer 28

Chrome

Question 29

Where can I find the baseline history?

Answer 29

If you go into a build and then click into a diff, you’ll see it on the bottom left corner

Question 30

Is there a way to mandate that each change be reviewed (versus being able to batch accept changes)?

Answer 30

Not currently possible!

Question 31

Where can I see if UI Test/UI Review is enabled?

Answer 31

Project > Manage > Automate (first tab)

Question 32

It looks like UI Review is set to “enabled,” but where can I double check that they’ve actually set it up?

Answer 32

By clicking the “PR” tab

Question 33

What is TurboSnap?

Answer 33

TurboSnap is an advanced Chromatic feature that uses Webpack’s dependency graph to only tests stories where there is a change between the current commit and the commit of the ancestor build. As a result, it will save you some amount of snapshots, from 0 to a lot, and that will depend heavily on your set up and various factors, many of which are not necessarily in our control. So we’ve definitely seen it reduce snapshot usage quite a bit for some teams, but we like to think of anything that TurboSnap saves you as a bonus, because as far as we know, other visual regression testing tools do not have a feature like TS. It’s also something that our team is continuing to improve upon as we know that it’s valuable for customers, but I did want to emphasize that we can’t guarantee that it will reduce snapshot usage significantly for all customers.

Question 34

Is TurboSnap only supported in Chrome + Firefox?

Answer 34

TurboSnap runs before it even gets to browsers — it’s completely independent of browsers. it runs before that logic runs and just won’t take the relevant snapshot on any browser.

Question 35

Can the TurboSnap feature be used on self hosted git or only cloud?

Answer 35

Yes, TurboSnap can be used anywhere Chromatic can. As long as the CLI can access their git history, TurboSnap will work.

Question 36

TurboSnap uses Webpack’s dependency graph — what does that mean for customers who want to switch to Vite?

Answer 36

We have this — https://github.com/IanVS/vite-plugin-turbosnap — but it’s still experimental. What does that mean exactly for customers?
This means that the plugin most likely works for the projects that the maintainer intended for it. However, they have not spent a lot of time testing in other scenarios and configurations so you should use it at your own risk.

Question 37

How can I see how many snapshots I’ve skipped (IE if TurboSnap is actually working?)

Answer 37

(Answer = tooltip + usage report + now billing tab!)

Question 38

We would like to require changes to be approved in Chromatic, even if the GH PR has been approved. “This way we can block a PR from being merged on the Chromatic UI Tests commit status check, even if someone has approved the code.” Is this possible?

Answer 38

That behavior exists on the GitHub side. In the repo settings, you can add or update your branch protection rules to include enforcing status checks.

Question 39

What happens when you deny a change in Chromatic?

Answer 39

It blocks the PR from being mergable, if it is a mandatory check. We do not revert the change in the code (as we do not read or write source code).

Question 40

When does the Chromatic check in the CI GitHub Action come up as yellow and when does it come up as red?

Answer 40

If there is a UI test that detects a diff, it will show up as yellow. If there is a UI test that you DENY, then you’ll run into an issue. Not reviewed = yellow. Deny = red X.

Question 41

On the library screen, the branch build number defaults to the branch with the most recent build. Can we default to main branch instead?

Answer 41

TL;DR – we can’t do it yet. Now, it defaults to the branch with the most recent build. TBD if we’ll work on the ability to default to a particular branch.

Question 42

What would be the best way to provide Light / Dark mode storybook modes?

Answer 42

In order to have a light and dark mode in Storybook, you can add a theme switcher to Storybook! Here’s an article on how to set that up.

Question 43

What does the code coverage in the Test Runner cover?

Answer 43

Basically it’s story coverage. The default coverage focuses on your components and if they have a story test for the component.
Essentially, we run coverage on whether your component states have a related story in the component.stories.(ts|js|jsx|tsx) file.

We also have a recipe for merging other coverage reports for non-story tests such as unit tests. Here is the section title for the test-runner doc
3 - Merging code coverage with coverage from other tools
https://storybook.js.org/addons/@storybook/test-runner

Question 44

Do you have to use Storybook to run Chromatic? Is Chromatic exclusively for Storybook? (IE if there was some other component library, would Chromatic work on that?)

Answer 44

Yes, but you can upload any image to Storybook and diff against it. But, we can only test from Storybooks.

Question 45

Customer wants to be able to share the latest Storybook link with non-technical people as easily as possible. What is the best way to do that?

Answer 45

What we generally recommend to customers is setting up a custom domain to be your main branch. That way, people can always go to that domain, and it’s easy to find. You can also go through the library page like I showed you on the call (and we released a new feature so that whichever branch you last selected in the library page will be the default branch selected when you click “View Storybook” in the library page), but using the custom domain will save you from even needing to go into Chromatic!

Question 46

Is there an easy way to get to the hosted Storybook from the PR screen?

Answer 46

“Go to build …..” then “View Storybook” – no shortcuts!

Question 47

Does Chromatic have keyboard shortcuts?

Answer 47

Yes

1 - 1up
2 - 2up
3 - diff

A accept
S switch
D deny

> next
< previous

esc: back

Question 48

How do I link a project?

Answer 48

After you sign in with your git provider, you can create a linked project by clicking the ‘Add project’ button on the Projects screen. You’ll be presented with a list of repositories in your organization.

Question 49

When you set delays in Chromatic, is there any way to confirm that it’s actually working, and taking the snapshot at the delay that has been set?

Answer 49

Not currently – there is a Linear ticket open for this. The easiest way to confirm is knowing what difference in your UI should be after x amount of seconds. There is not something that we explicitly return to the customer that says we successfully delayed the snapshot for this story for x amount of seconds.

Question 50

Can a customer set thresholds globally, as opposed to just at the story level?

Answer 50

Yep! You set the threshold via parameters, which can be set at different levels.

However, it’s probably a bad sign if they need to adjust the threshold globally. That setting is a last resort and should only be used when all other options are extinguished.

Question 51

Can we use Chromatic for mobile screenshots for our React Native project?

Answer 51

We don’t have specific support for mobile apps, but Chromatic works with any web code that you can render in Storybook in a web browser. So react-native-web would work but not react-native (we don’t support emulators right now).You can also test mobile screenshots using viewports (Loki is another potential workaroujnd).

Question 52

Can iI automatically test two different languages (ex: English and German) without creating new stories for all of them?

Answer 52

To answer your question regarding languages — whether it’s possible to run the story testing in two different languages and have it catch diffs — the current answer is that you’re not able to do that. You’d have to make a component for each language and test it separately. But this is feedback we’ve gotten before (the ability to test these types of things at scale), so I’ll definitely flag this (as well as your other feedback) to our product team.

Question 53

Is it possible to specify a baseline to compare a branch to?

Answer 53

Not currently – there is a Linear ticket open for this

Question 54

We have a squashing and merging or rebasing workflow. Why is Chromatic having issues with baselines?

Answer 54

Chromatic can’t find the ancestor build

Question 55

Why am I seeing an error for large images?

Answer 55

We have a 25 million pixel size limit for image snapshots. This ensures fast and reliable performance for every build.

Question 56

Does Chromatic tell me when snapshots are different between browsers?

Answer 56

Currently no. This has significant trade offs. Teams that try to verify consistency between browsers end up encountering false positives due to inherent browser/device/OS differences like anti-aliasing and font rendering. Or they require workarounds like loosening diff thresholds which result in false negatives.

Chromatic does not programmatically compare snapshots from different browsers against each other. Instead, we compare the snapshots for each browser against the baseline for that browser.

Question 57

Can i control the height of the viewport?

Answer 57

We take a full screenshot of the component even if it flows off the screen. It typically doesn’t make a difference what height the browser is when taking screenshots.

Currently, you cannot set the height of the viewport. We have a ticket open for that (AP-442).

Question 58

What viewports does Chromatic support?

Answer 58

We now support viewports between 200px and 2560px (previously 320-1800). We do still have the same total limit of 25 million pixels which you’ll hit much sooner at 2560px width.

Question 59

Where can I see which browsers are currently enabled?

Answer 59

Project > Manage > Automate

Question 60

How do enterprise customers pay overages?

Answer 60

Enterprise customers pay an annual bill for snapshots, but then they pay for overages monthly. Usually, it’s all via invoice.

Question 61

Do we offer Open source accounts to charities / non-profits (NGOs)

Answer 61

We offer a 20% discount on premium plans for 510(c)(3) non-profits.

Question 62

Who qualifies for an OSS plan?

Question 63

Who qualifies for a start-up plan?

Answer 63

Fewer than 20 employees. (including entire company, not just dev team)

Question 64

If I downgrade my plan mid-cycle, will we lose access to the snapshots we’re already paid for, or will we retain those for the current cycle?

Answer 64

If you were to downgrade your plan, the unused time would be credited back to your credit card.
When a customer downgrade mid cycle, they automatically get switched the new selected plan and the new snapshot quota applies. Then, they get a credit for the unused time that they already paid for.

Question 65

How do the usage alerts work? Is it possible to set up multiple usage alerts?

Answer 65

Billing tab > you can alert at a certain # of snapshots.

You can’t set up multiple

Question 66

Are overages enabled by default for everyone? Can a customer implement a hard stop to their # of snapshots?

Answer 66

Yes. If a customer wants a hard limit, they have to let us know and we have to enable that. But there are some limitations for that.

What we can currently do: we can currently set a customer up to have a hard stop at whatever their snapshot limit is.

For self-service customers, that automatically means 35K, 80K, 150K. For enterprise customers, we can also set it at whatever their snapshot limit is. For example, if Brex’s plan gives them 250,000 snapshots per month, we can set a hard limit at 250,000 snapshots per month for Brex (although enterprise customers are sometimes set per year).

What we cannot currently do is (1) Set a hard stop OUTSIDE of their snapshot limit (although Tom said this isn’t too hard to do, so we can possibly do that if a customer is really adamant or (2) Set a dollar amount overage limit. For example, we cannot say “Brex wants to limit their overages to $5,000 per month.” This isn’t impossible to implement, though, so if it’s something that we think customers really wants, we can think about doing it. In the meantime, we can offer the hard stop snapshot limit for teams that want it, but it’s generally not something we advertise.

Question 67

How do I change my plan?

Answer 67

Change your plan in account settings. All changes are prorated to the day.

Question 68

Are paying customers notified when they start going into overage pricing?

Answer 68

We don’t have a button or anything that toggls when you go into overages pricing BUT you can set a notification alert when you reach a certain number of snapshots. So, for example, if you have 150,000 snapshots/month, you can set an alert at 150K (which would basically be an alert saying that you’re going into overage pricing). Or you can set it at 140K, 130K etc as a warning that you’re going to go into overages soon. You can also set it (I believe) at higher, 200K for example, to alert you when you’re at a certain % of overages.

Question 69

Can you have multiple permalinks within a project?

Answer 69

Yes, Chromatic will create Storybooks for each individual branch.

Question 70

How does Chromatic handle CSS animations?

Answer 70

Chromatic will pause CSS animations and reset them to their beginning state.

Question 71

How does Chromatic handle Javascript animations?

Answer 71

Chromatic cannot disable JavaScript driven animations, so we advise disabling such animations manually for Chromatic builds. One way to do that is using isChromatic():

Question 72

When might a customer want to introduce a delay?

Answer 72

If you cannot disable animations (for example if disabling JS animations is difficult), you can use a delay to allow the animation to complete before taking the snapshot.

Question 73

Inconsistent snapshot roleplay

Question 74

Do I have to use Chromatic in order to use the figma plugin?

Answer 74

• yes for private figma, no for public figma