Flaky¶
A plugin for nosetests or py.test that automatically reruns flaky tests.
pip install flaky
Available on GitHub: https://github.com/box/flaky
Why?¶
Ever been tempted to do this?
There are lots of wrong reasons to do something like this, but a test that fails only on occasion can really make you want to.
Flaky Tests¶
We call tests flaky when they:
- Only fail sometimes
- Have a failure mode that may be very difficult to fix
- Often pass when rerun
These tests put us in a bad position, and there's not a perfect fix.
The mathematics of flaky tests¶
Given 50 flaky tests (99% pass rate), how often will the test run fail?
What about with 100 flaky tests?
The mathematics of flaky tests¶
Given 50 flaky tests (99% pass rate), how often will the test run fail? 40%
\begin{equation*} 1 - \left(1 - 0.01\right)^{ 50} = 0.4 \end{equation*}What about with 100 flaky tests? 64%
\begin{equation*} 1 - \left(1 - 0.01\right)^{ 100} = 0.64 \end{equation*}The mathematics of rerun flaky tests¶
If we retry those 50 flaky tests, however, the test run fails just 0.5% of the time.
\begin{equation*} 1 - \left(1 - 0.01^2\right)^{ 50} = 0.0005 \end{equation*}What about with 100 flaky tests? 1%
\begin{equation*} 1 - \left(1 - 0.01^2\right)^{ 100} = 0.001 \end{equation*}%%file flaky_tests.py
def test_list_length(my_list=[]):
my_list.append(0)
assert len(my_list) > 1 # Fails the first time it's run. Passes on subsequent runs.
!nosetests flaky_tests.py
If this test can't be fixed, you have a few options.
This test would be pretty easy to fix, but many of us have encountered tests that rely on external components that might not have a solution that can be reached in the amount of time available.
# TODO: fix this test next month when I have time
#def test_list_length(my_list=[]):
#my_list.append(0)
#assert len(my_list) > 1
Comment out the flaky test¶
Commenting out the flaky test makes sure it won't fail, but it leaves your code uncovered and defeats your tooling. No good!
from unittest import skip
@skip("Fix this text next month when I have time.")
def test_list_length(my_list=[]):
my_list.append(0)
assert len(my_list) > 1
Skip the flaky test¶
Skipping the flaky test also makes sure it won't fail, but it's not really much better than commenting it out. These skipped tests are rarely fixed.
What if we could rerun tests automatically when they fail?¶
Introducing flaky, a plugin for nosetests or py.test that automatically reruns flaky tests.
%%file flaky_tests.py
from flaky import flaky
@flaky
def test_list_length(my_list=[]):
my_list.append(0)
assert len(my_list) > 1
!nosetests --with-flaky flaky_tests.py
Flaky¶
A plugin for nosetests or py.test that automatically reruns flaky tests.
Just mark your flaky tests with @flaky, and run them with py.test or nosetests --with-flaky.
Customizing Flaky¶
Need more reruns? No problem - just tell flaky how many times it can run the test:
@flaky(max_runs=5)
def test_list_length(my_list=[]):
my_list.append(0)
assert len(my_list) > 1
Prefer to make sure the test can pass more than once? Tell flaky how many times it must pass:
@flaky(min_passes=2)
def test_list_length(my_list=[]):
my_list.append(0)
assert len(my_list) > 1Customizing Flaky¶
Have a lot of flaky tests? Flaky allows you to mark an entire test class @flaky. Still not enough? Simply pass the command line argument --force-flaky to the test runner and all tests will be marked flaky.
The flaky report¶
The end of each test run includes some information about your flaky tests.
This is important because a failed test won't be reported to the test runner if the rerun passes.
However, if you don't want to see this info, you can pass --no-flaky-report to your test runner.
Examples¶
%%file flaky_nosetests.py
from flaky import flaky
from unittest import expectedFailure, TestCase
class ExampleTests(TestCase):
_threshold = -1
def test_non_flaky_thing(self):
"""Flaky will not interact with this test"""
pass
@expectedFailure
def test_non_flaky_failing_thing(self):
"""Flaky will also not interact with this test"""
self.assertEqual(0, 1)
@flaky(3, 2)
def test_flaky_thing_that_fails_then_succeeds(self):
"""
Flaky will run this test 3 times.
It will fail once and then succeed twice.
"""
self._threshold += 1
if self._threshold < 1:
raise Exception("Threshold is not high enough: {0} vs {1}.".format(
self._threshold, 1),
)
@flaky(3, 2)
def test_flaky_thing_that_succeeds_then_fails_then_succeeds(self):
"""
Flaky will run this test 3 times.
It will succeed once, fail once, and then succeed one more time.
"""
self._threshold += 1
if self._threshold == 1:
self.assertEqual(0, 1)
@flaky(2, 2)
def test_flaky_thing_that_always_passes(self):
"""Flaky will run this test twice. Both will succeed."""
pass
!nosetests --with-flaky flaky_nosetests.py
%%file flaky_nosetests_2.py
from flaky import flaky
from unittest import TestCase
@flaky
class ExampleFlakyTests(TestCase):
_threshold = -1
def test_flaky_thing_that_fails_then_succeeds(self):
"""
Flaky will run this test twice.
It will fail once and then succeed.
"""
self._threshold += 1
if self._threshold < 1:
raise Exception("Threshold is not high enough: {0} vs {1}.".format(
self._threshold, 1),
)
@flaky
def test_flaky_function(param=[]):
# pylint:disable=dangerous-default-value
param.append(None)
assert len(param) == 1
!nosetests --with-flaky flaky_nosetests_2.py
%%file flaky_pytest.py
from flaky import flaky
from unittest import TestCase
@flaky
def test_something_flaky(dummy_list=[]):
# pylint:disable=dangerous-default-value
dummy_list.append(0)
assert len(dummy_list) > 1
@flaky
class TestExampleFlakyTests(object):
_threshold = -1
def test_flaky_thing_that_fails_then_succeeds(self):
"""
Flaky will run this test twice.
It will fail once and then succeed.
"""
self._threshold += 1
assert self._threshold >= 1
@flaky
class TestExampleFlakyTestCase(TestCase):
_threshold = -1
def test_flaky_thing_that_fails_then_succeeds(self):
"""
Flaky will run this test twice.
It will fail once and then succeed.
"""
self._threshold += 1
assert self._threshold >= 1
!py.test flaky_pytest.py
%%file flaky_pytest_2.py
from flaky import flaky
import pytest
from unittest import TestCase
class TestExample(object):
_threshold = -1
def test_non_flaky_thing(self):
"""Flaky will not interact with this test"""
pass
@pytest.mark.xfail
def test_non_flaky_failing_thing(self):
"""Flaky will also not interact with this test"""
assert self == 1
@flaky(3, 2)
def test_flaky_thing_that_fails_then_succeeds(self):
"""
Flaky will run this test 3 times.
It will fail once and then succeed twice.
"""
self._threshold += 1
assert self._threshold >= 1
@flaky(3, 2)
def test_flaky_thing_that_succeeds_then_fails_then_succeeds(self):
"""
Flaky will run this test 3 times.
It will succeed once, fail once, and then succeed one more time.
"""
self._threshold += 1
assert self._threshold != 1
@flaky(2, 2)
def test_flaky_thing_that_always_passes(self):
"""Flaky will run this test twice. Both will succeed."""
pass
!py.test flaky_pytest_2.py
Some advice¶
Flaky, as great as it is, should be used with caution!
Many failing tests are trying to tell you something! Don't just silence them with @flaky.
When applied judiciously, flaky can be a great time-saving tool, but it can also be misused, so apply your best judgment.
Summary¶
- Mark your flaky tests
@flakyinstead of removing or@skipping them. pip install flakyto get started.- Run
py.testornosetests --with-flakyto get automatic retrying of failed tests.
This notebook is available for download: http://opensource.box.com/flaky/Flaky.ipynb
Its content is licensed under the Apache License 2.0.
To reproduce the presentation, execute the following commands:
curl -O http://opensource.box.com/flaky/Flaky.ipynb -O http://opensource.box.com/flaky/fixed_test.jpeg
pip install nose pytest flaky ipython[notebook]
ipython notebook Flaky.ipynb