OS specific resemblejs tolerance issues

What are you trying to achieve?

Hello, first thanks a million for this great helper! It makes my life easier :)

I am experimenting Visual Testing based on resemblejs, and I face a strange issue:

The tests pass when I run them locally on the computer that originally created the screenshot (Mac OS), weather HEADLESS is on or off.
However, some tests fail from a CI pipeline (bitbucket pipelines) based on Linux (HEADLESS).

I have 2 main questions:

1. is this expected? Where can I find information about the visual comparison "tolerance" and overall options? I do not find a reference to it in their documentation: https://github.com/rsmbl/Resemble.js
2. what is the intended approach when it comes to update screenshots? Is there an "update" option somehow?

What do you get instead?

Provide console output if related. Use --verbose mode for more details.

MissMatch Percentage 3.82

Scenario Steps:
- I.seeVisualDiff("homepage-mobile.png", {"tolerance":2,"prepareBaseImage":false}) at Test.<anonymous> (tests/visual-regression/homepage.test.js:15:5)
- I.saveScreenshot("homepage-mobile.png") at Test.<anonymous> (tests/visual-regression/homepage.test.js:14:5)
- I.amOnPage("/") at Test.<anonymous> (tests/visual-regression/homepage.test.js:12:5)
- I.resizeWindow(320, 480) at Test.<anonymous> (tests/visual-regression/homepage.test.js:11:5)

Provide test source code if related

Feature('Screen comparison of the Home page');

Scenario('Compare desktop Home Page @visual-test', I => {
  I.amOnPage('/');

  I.saveScreenshot('homepage.png');
  I.seeVisualDiff('homepage.png', { tolerance: 2, prepareBaseImage: false });
});

Scenario('Compare mobile Home Page @visual-test', I => {
  I.resizeWindow(320, 480);
  I.amOnPage('/');

  I.saveScreenshot('homepage-mobile.png');
  I.seeVisualDiff('homepage-mobile.png', { tolerance: 2, prepareBaseImage: false });
});

Details

• CodeceptJS version: codeceptjs@2.5.0 & codeceptjs-resemblehelper@1.9.0
• NodeJS Version: 12.14.0
• Operating System: OS X 10.15.3 vs https://hub.docker.com/_/amazonlinux/
• puppeteer: puppeteer@2.1.1
• Configuration file:

require('dotenv').config();
import { setHeadlessWhen, setWindowSize } from '@codeceptjs/configure';

const headless = (process.env.HEADLESS || process.env.CI) && process.env.BROWSER !== 'firefox';
const profile = ['e2e', 'vr'].includes(process.profile) ? process.profile : 'e2e';

// turn on headless mode (not opening browser)
setHeadlessWhen(headless);
setWindowSize(1600, 1200);

export const config = {
  name: 'my-project',
  tests: {
    e2e: 'tests/end-to-end/scenarios/**/*.test.js',
    vr: 'tests/visual-regression/**/*.test.js'
  }[profile],
  include: {
    I: './tests/end-to-end/steps_file.js'
  },
  output: 'test-reports/end-to-end',
  bootstrap: null,
  mocha: {
    reporterOptions: {
      mochaFile: {
        e2e: 'test-reports/end-to-end.xml',
        vr: 'test-reports/visual-regression.xml'
      }[profile]
    }
  },
  helpers: {
    Puppeteer: {
      browser: process.env.BROWSER || 'chrome',
      url: process.env.TEST_BASE_URL,
      show: true,
      waitForNavigation: 'networkidle0',
      waitForAction: 200,
      pressKeyDelay: 50,
      chrome: {
        args: [
          '--no-sandbox',
          '--ignore-certificate-errors',
          '--disable-dev-shm-usage',
          '--disable-gpu',
          '--disable-setuid-sandbox'
        ]
      }
    },
    ResembleHelper: {
      require: 'codeceptjs-resemblehelper',
      screenshotFolder: './test-reports/screenshots/',
      baseFolder: './test-reports/screenshots/base/',
      diffFolder: './test-reports/screenshots/diff/'
    }
  },
  plugins: {
    allure: {},
    autoDelay: {
      enabled: true
    },
    pauseOnFailure: {},
    retryFailedStep: {
      enabled: true
    },
    screenshotOnFail: {
      enabled: !process.env.CI
    }
  }
};

[puneet0191]

Hello @flosch-hb thanks for the questions, I am extremely sorry I was occupied with some load.

1. is this expected? Where can I find information about the visual comparison "tolerance" and overall options? I do not find a reference to it in their documentation: https://github.com/rsmbl/Resemble.js

I think this is not expected behaviour, most probably I suspect it could possibly be the difference in the screenshot image and the base image dimension, which could be causing such results on CI/CD, do you think it is possible for you to download the screenshot image from CI/CD and try using it in local for test?

2. what is the intended approach when it comes to update screenshots? Is there an "update" option somehow?
   I haven't been able to find time to implement global option to control this, but for now, I think if you use prepareBaseImage: true that should update the screenshot and not compare it with the base image, the intended approach is to use config parameter, and set it in .json or as env variable, to use it to update images for all tests, if you can help me to fix it, it would be great.

[flo-sch]

Hi @puneet0191, (sorry for the nicknames confusion, I stopped using flosch-hb in the end)

No problem, thanks for your answers!
I managed to "fix" 1- by just increasing the tolerance:

I.seeVisualDiff('homepage.png', {
  // percentage, small differences may occur between operating systems
  tolerance: process.env.VISUAL_REGRESSION_TOLERANCE || 5
});

I am not sure if I can download the images from bitbucket pipelines (the CI/CD system we used in my project), but I can try to setup a repository to reproduce it?

Regarding 2-, I would be happy to contribute :)

However I am new to both Resemble and Codecept itself, and have only used Puppeteer as a driver so far... So I would need some guidance on the different concepts (Codecept helpers, Resemble etc.), for instance:

• how do we pass config to a helper etc.?
• can we provide a custom command to update all snapshots locally?

[puneet0191]

@flo-sch I released a new version please check and tell me your feedback, if you still find anything that can be improved, I will fix it with next version, thanks a lot. and thanks to @jonatask for the fixes.

[ricardoekm]

Hi @flo-sch, having the same issue here.

For what's worth one of the causes is that Webdriver saveScreenShot command seems to save the image with different dimensions on mac compared to linux, even when setting the same window size.

Adding a .scaleToSameSize() command on resemble comparison should help here.

[MumblesNZ]

Also have the same issue with my local mac env vs gitlab CI linux env. We ended up dockerising the local testing environment to ensure it's running with the same environment. Only downside is it's a bit slower then running it directly.

Anyone got any further thoughts on a solution here?