While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy.
Ideally, you don't want a very large number of tests, no matter how big the system is. Tests are, effectively, an interface to the program that assesses the system quality / readiness for use. Any interface with thousands of individual pieces is difficult to use.
Random combinations of tests also don't spark joy because this means both repetition (i.e. waste of resources) and testing potentially useless (unreachable or invalid) system states (both wastes resources and creates false alarms).
Ideally, the tests should be able to compose only in desired ways and rather than combining them randomly, there should be some deterministic process that creates a unique sequence or a tree of individual tests on subsequent runs. Ideally, such a test runner could also be configured to start with an existing system in a known state s.t. the tester can apply a patch and resume testing.
> While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy.
The marginal cost of running the tests like that was very small, and bugs were found. The cost per found bug seemed very reasonable. Every test failure in this strategy indicates a bug.
(This was the standard compliance test suite for Common Lisp, btw.)
Now imagine someone who needs to test something on eg. P6e-GB200 series VM in EC2 (this is one of the newest Nvidia accelerator models: GB200 multiplied by something like 60 iirc).
I don't know what the price is and I'm not even sure you can just create this kind of a VM with any regular account, but I'm sure you can imagine a price of running such a test.
The price doesn't have to be the limiting factor. It may be time (if a test takes a very long time to set up), or it could be labor (imagine needing to flash the ROM of a PCB for each test or similar).
This is what I mean when I say that your experience is likely not transferable to many real-life testing applications. But, even if you don't pay that much for an individual test, still, why waste time and resources, if you could do better?
But you missed the important distinction: random is bad because it repeats unnecessarily. One of the typically used examples in statistics 101 class to drive attention to this fact is the question posed to students to guess, without actually doing any math: what is the chance of two people in a room with (however many students are there in the class) to have a birthday on the same day?
People consistently estimate the chance significantly lower than what it actually is. And this error is supposed to encourage students to dedicate more attention to the subject because they will have found their intuition to be so woefully inadequate.
Now, if you are interested in economics of writing tests and running them (but you don't have to be, unless it's your job):
First of all, for this problem, it doesn't matter how much it costs to write the tests. We are talking about the strategy to cover more feature combinations in the test, so, no matter how much it costs to write a test, the cost (per test) will be the same regardless of the strategy chosen. The combination is where the difference will manifest itself.
Second, tests are normally written by salaried employees. This makes the cost of writing an individual test very difficult to divine (and even if you could, it would be a useless metric). It's difficult to calculate because a lot of factors that contribute to writing a test happen in aggregates (eg. the tester needs to explore and familiarize themselves with the system before writing a test, but this effort contributes to multiple tests). Salaried employees are paid a fixed amount regardless of the intensity of work: if they produce one, ten or a hundred tests per month, the salary is the same. Tests can be vastly different in scope and in an effort necessary to implement them, however the effort necessary to implement a test isn't indicative of the value of the test (it's possible that a very simple test will be very effective at detecting problems where a very complex test isn't and the other way around).
All this said, suppose you did come up with a way of calculating the cost of writing an individual test: it still doesn't help you to estimate the cost of testing the product, it doesn't even move the needle on this question. It changes nothing about your QA department budget. It changes nothing about delivery schedule. It's just a number you can marvel at in your spare time...
If they are organized into meaningful groups, then, of course: the organization makes it easier to use. Compare: have thousands of name-phone number pairings in any order vs same pairings, but now they are grouped by the first letter of the name? Don't you think that the later is easier to use?
Shout out to AVA for running all tests async by default, which not only finds order-dependent issues, but also race condition and other timing related issues.
> Deleting test1 loses us another property from the Test Desiderata—tests should be specific. That’s the property of tests where, when one fails, you know exactly where the problem is.
Contra Kent and, it seems, prevailing wisdom, I think simply deleting test1 is by far the simplest, clearest and best way. Provided that your testing framework tells you which specific assertion failed (e.g., by telling you the line number in a stack trace), you do know exactly where the problem is. The only thing you lose is that a test function or method may now cover several related checks (they are related by "setup dependence"), meaning their names may need to be somewhat broader. But you can still describe the specific semantics of each assert() check in a one-line comment beforehand if you want. There's no need to cram it into a legal method name.
ETA: Prefer to write tests whose "arrange" steps are as simple as possible, to minimise unnecessary overlaps. But if the simplest possible "arrange" step for a test is something that itself needs to be checked for correctness, just do that check right there, and nowhere else. Anything beyond that is ceremony that adds nothing useful.
I think this could lead to the tests getting harder to reason about over time because individual test size will just grow (many iterations of deleting the simple test in favor of a more complicated test that also asserts the simple things)
and if you start simplifying the more complicated test later, you may not realize that you're accidentally deleting checks that don't exist anywhere else other than incidentally here. So it's less clear what's important without thinking it through each time.
now all that to say I don't think it's strictly that big of a deal either way - there's always tons of room for taste that can make one or the other way better in practice. But that was my gut-reaction when reading your comment.
> Contra Kent and, it seems, prevailing wisdom, I think simply deleting test1 is by far the simplest, clearest and best way. Provided that your testing framework tells you which specific assertion failed (e.g., by telling you the line number in a stack trace), you do know exactly where the problem is.
The issue with your approach is that codepath execution is more of a graph than a linear timeline. With one test, you artificially constrains it to a linear timeline even if the assertions are correct. With multiples and independent you only assert one specific node. That lets you switch up how you do the preliminary steps. I much prefer an exhaustive test unit for step 1, and just a regular call in step 2 and step 3.
If you need test1's logic as setup for test2, then you already have that "linear timeline" -- for test2 all by itself. Additionally (that is, redundantly) running the same setup code "prefix" by itself in test1 doesn't remove test2's linear timeline.
ETA: I'm assuming your objection to a "linear timeline" is that it reduces the potential for running tests in parallel -- have I got that right? If not, what do you see as being the problem with it?
If I know that step 1 is correct for all the combinations (N) of its input (in test 1), in test 2, I only need to test the specific combinations (M) of step 2, treating step 1 as an axiom. So no need to have NxM in one test, or NxM tests.
Like if you were testing a drone stabilization software, testing the flying state can always assume that it has indeed taken off. No need to validate that it has done so for each scenarios, I can directly put it the correct value. It’s a contrived example, but that axiomatic aspect helps greatly when designing interfaces to reduce coupling between step 1 and step 2.
Kent Beck is smart because he proposes fixes that work in real life, as in environments with dumb code and undisciplined people.
I can very much recommend his latest book ”Tidy first?”. It’s extremely short and concise, and is perfect for a very light book club within any tech team.
I enjoyed the last (and theoretical) part of "Tidy first?" as I was already doing the recommended practices in the first two. But yes, very pragmatic advice and easy to apply to get your own corner of peace in a somewhat chaotic codebase.
One term I can use is "defensive programming" (not sure if the term has been used before). People are going to do awful stuff with their code, it's up to you to draw boundary of correctness to stop it from spilling on the part that you're responsible for. It should be automated (with tests and static analysis) as well as documented. I think of those boundary as hazmat suits and I take extra care of maintaining their integrity.
What he wrote is basically "don't repeat in test X what you already tested in test X-1".
It's not as much composition as compounding, and can work quite well.
Let's say something takes 20 steps, and you want to test all 20.
Instead of this:
test 1:
do step 1, assert step 1
test 2:
do step 1, assert step 1
do step 2, assert step 2
test 3:
do step 1, assert step 1
do step 2, assert step 2
do step 3, assert step 3
...
you do this:
test 1:
do step 1, assert step 1
test 2:
do step 1
do step 2, assert step 2
test 3:
do step 1
do step 2
do step 3, assert step 3
...
This works well in certain situations, as it skips diplicated redundant testing. Requires some discipline so that tests don't drift away from each other.
As most things, it depends on what those steps are. Perhaps you only need that one final (integration) test instead of 20 intermediate unit ones.
I wonder if it would work do design something that was able to say
test 1:
do step 1, assert step 1
test 2:
requires: test 1
do step 2, assert step 2
test 3:
requires: test 2
do step 3, assert step 3
test 4:
requires: test 2
do step 3, assert step 3
If the assertion of each test doesn't change any state, that might make things easier to read. Though, given that I haven't spent much time pondering it, I expect it could have it's own problems.
But it could also do things like skip test 3 if tests 2 or 1 failed - because it knows about the relationship.
- How to reconcile this with tests that execute many times with varying input data. You’d need some way to express requirements with specific inputs or shared inputs.
- Passing state between test dependencies.
- When, if ever, it’s fine to share step results between tests. If tests B and C require A, can you run A just once? Not always, but you should be able to when it’s safe.
I don’t think I’ve ever used a test framework that gets these things right.
Actually, I wasn't even thinking about passing state. I was thinking about shared setup steps. I'm perfectly happy for the same steps to run for each test - as long as each test doesn't need to list the steps (when they're setup steps not directly related to the thing being tested)
For example, suppose you want to write a test for the shipOrder(orderId) function, and you want it to depend on the test for the placeOrder(shoppingCart) -> OrderId function. Even if you are fine calling placeOrder twice, once for the placeOrder test and once for the placeOrderAndShipOrder test, you still need the placeOrder test to provide an order ID to the second test, and not just a confirmation that it completed successfully.
The pattern can work, but the domain matters, the test type matters (unit, integration, ui) and the trade-off associated matter.
e.g. If I am running a long-running UI-test scenario, I absolutely don't want test-5 to walk through 80% of the UI that was already exercised in tests 1-4. I am creating test coupling, but I'm saving cost/time by doing so.
But, you'll also hear why not to do this, because it creates test coupling / breaks atomic tests, which is generally seen as bad.
If that is a local integration test and those early steps run is millis? Then maybe we keep things uncoupled to allow the system to exercise the pathways without explicit expectations.
The question was less about speed and more about not having the same code duplicated over and over across tests. Which is how I read the article talking about it. Allowing one test to "depend" on another makes it clear they use the same setup (presumably with the second test going "a bit further", but not necessarily).
I wouldn't have a problem with something like
test-1:
setup:
do-the-thing
verification
assert-the-thing-happened
test-2
setup:
depends-on: test-1 // tells it to run test-1's setup
do-the-next-thing
verification
assert-the-next-thing-happened
The format is awful, but the idea is that most tests are of the form
GIVEN
Some initial setup
WHEN
I run command
THEN
The result of that command is what is expected
And, in that context, the GIVEN frequently contains noise not directly related to understanding what is being tested.
I actually use the GIVEN/WHEN/THEN keywords in my tests, to make them easier to read
You're not going to remember what tests "test 3" relies on. They aren't actually linear progressions 123. They will be "test this", "test that". If 3 fails you're going to want to immediately go and add all those asserts back to help you debug your assumptions.
The tests _will_ drift and that should be fine. Implicitly depending on other tests doesn't really get me anything.
What's the problem of only keeping test 3 if it depends on test 2 and 1 passing anyway?
The article mentions not to do this because "Deleting test1 loses us another property from the Test Desiderata—tests should be specific. That’s the property of tests where, when one fails, you know exactly where the problem is." but you'll know what line it failed on. And some test runners let you break a test into steps, where groups of lines are given a description.
Or put each step + assert in a helper function (e.g. `doStep1AndAssert()`), and each test only calls these helper functions?
Nothing is perfect, but copy/pasting chunks between tests like this isn't great when you want to refactor and it's repetitive to read.
> What's the problem of only keeping test 3 if it depends on test 2 and 1 passing anyway?
That depends.
Sometimes test 1 tests a combination of ways (e.g., property testing, or just going through a bunch of various inputs), and only a few of those are needed for test 2.
Sometimes you don't want your test 2 to be more complicated than it already is. Or the same things are needed checked in other tests. So you extract them into test 1.
And sometimes (and in some of code bases most of the time) test 1 is redundant and unnecessary. That's why I always advocate investing in integration tests (test 20) and skip all the intermediate tests.
> That's why I always advocate investing in integration tests (test 20) and skip all the intermediate tests.
My approach is to have acceptance tests documented for any feature. Like how it would be from the user point of view to actually use the software. Then do Integration tests for each part of that workflows. That's usually the most ROI you will get for testing. Then I invest into unit tests for particular elements that are very important. I start from the middle of the pyramid because an actual e2e is expensive to setup (easy to maintain afterwards) and having lots of unit tests (easy to setup) is expensive to maintain.
I think I disagree with Kent, but your explanation is clearer, so I'll object here.
There's nothing wrong with hitting the same assertion multiple times, even if it doesn't sit nicely in your gut.
From a purely philosophical point of view: If I have testFoo(), testBar(), and testFooAndBar(), and my Foo is plain wrong, then both testFoo() and testFooAndBar() must fail. Anything less is misleading/dishonest.
From a practical side: Changes happen. Someone will remove testBar(), and then you're down to 0 assertions on Bar, even though you have a test claiming to testFooAndBar(). It's not even a crazy hypothetical. Someone with a different test philosophy will think (to quote TFA) "They are redundant! Something must be wrong." and delete testBar() because obviously testFooAndBar() already covers it.
Anyway, we all know how to deal with repetition. That's what programming is!
If you have people on your team deleting valid tests because of "philosophy", I think you have much bigger problems to solve than anything Kent Beck can help with.
To be clear, the philosophy I quoted was directly from Kent Beck in TFA, i.e. this is Kent Beck's "help".
I say leave both tests as is.
Kent Beck says:
From a purely aesthetic standpoint (& don’t discount aesthetics), leaving both tests as is offends my sensibilities. They are redundant! Something must be wrong.
It's not just philosophy, it's aesthetics apparently!
What I do not like about that kind of example is its abstractiveness. Yes sure you can argue about testing `doSomething()` and it all falls apart when there is an actual business scenario to test.
It's not hard, with some modest experience (say 1-2 years of professional work or serious amateur interest), to extrapolate from his deliberately high-level discussion (it was written for a blog/newsletter, not for a book) to something more interesting and "real world". What's hard is coming up with an example that's nearly complete and fits into something the size of a newsletter or comment. He did an alright job of it, just left the readers with the need to exercise their gray matter a bit.
If it is not hard, with an experience of this guy, he could came up with a better example? It is your own words so I am sure you will agree? He did alright job so he could exercise gray matter a bit and came up with some API for saving customer data and then test it his way? I mean... This would be more interesting and "real world" so I am sure it would be worth to write longer blog post about it!
> If a test runs by first setting up its own test fixture, creating from scratch all the data it will be using as input, then that test is guaranteed to be isolated. It doesn’t matter what order you run the tests, the results will be exactly the same.
Completely backward. This definition requires isolation, rather than granting it.
In reality, you (or your test framework) provides the isolation by hobbling along, executing only one test at a time.
> The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation.
I'm not sure what you mean by that. All the frameworks I ever used were designed with test isolation as the primary design goal. Even when you set shared test fixtures and setup/teardown code, all they provide is a way to share code across tests, which are by themselves independent and isolated. BDD-style frameworks are the notable exception in breaking away from the pattern of having isolation as a fundamental design trait. In fact, the whole reason why BDD-style tools require special support from testing frameworks is that they need to work around test isolation in order to share context across steps.
In JUnit, tests run sequentially in a single thread by default [1].
Parallel execution must be explicitly enabled. When enabled, JUnit uses a fork-join thread pool, so tests may run concurrently on different worker threads. Because these threads are reused, a ThreadLocal value left behind by one test could be visible to a later test that happens to run on the same thread.
Setup and teardown methods can be used to create and clean up test-specific state. However, developers must still ensure that test data is unique to the test so that concurrently running tests do not interfere with each other. This uniqueness is unfortunately called "isolated", and has led to much confusion like in this thread. Certainly, the Test Execution Framework cannot guarantee data-isolation.
Parallel and randomized test execution can also help expose application-side problems involving shared or order-dependent state. Such failures may only appear when tests happen to exercise the application under the relevant ordering or concurrency conditions. When I was a junior developer teaching myself Java Servlets in 2000, I had to learn this lesson the hard way. A Test Execution Framework would not be able to guarantee any "isolation" of test data and of workflows if the server-side state is mis-managed by the tech stack and/or by the developer.
> Parallel execution must be explicitly enabled. When enabled, JUnit uses a fork-join thread pool, so tests may run concurrently on different worker threads. Because these threads are reused, a ThreadLocal value left behind by one test could be visible to a later test that happens to run on the same thread.
I don't understand your comment. JUnit docs describe parallel execution as an experimental feature that still has gotchas that you need to be mindful to avoid. So you have a test framework designed with isolation in mind for a specific execution mode, but when you go out of your way to try another execution mode that is described as experimental then you can stumble upon very corner cases where isolation is not ensured. What point did you wanted to convey?
I disagree so hard with the notion that isolation is somehow done for you, that I don't even know how to begin arguing.
I'll admit to starting this confusion by reading what I thought would be written, not what was actually there. TFA does phrase it to my satisfaction that the test is isolated. I assumed when reading it that Kent thought he had isolation but did not. By analogies: a naive programmer does not perceive the existence of race conditions; a naive web app developer doesn't perceive the existence of CAP; a naive backend engineer thinks they have full ACID guarantees when they're only guaranteed READ_COMMITTED. I thought I was reading another writer just wishing isolation into existence through ignorance.
(So I accidentally straw-manned Kent's argument, but maybe that's not such a problem, because it seems you're OK with the straw man version?)
Anyway, maybe a question moves things forward.
JUnit's been around for decades. Every developer has had multicore dev machines for decades. Why would parallel execution not be the default, or a simple toggle that everyone would turn on immediately?
All I mean that there is no way for a framework to prevent your tests from interfering with each other, or to solve isolation for you.
It’s the implementation of each test that is responsible for its isolation. Every time you write or make changes to your tests. You have to think, can they mess with each other.
> All I mean that there is no way for a framework to prevent your tests from interfering with each other, or to solve isolation for you.
What? By design that's exactly how virtually all test frameworks are designed to work.
> It's the implementation of each test that is responsible for its isolation. Every time you write or make changes to your tests. You have to think, can they mess with each other.
Nonsense. I've been writing unit tests for years in multiple programming languages and using multiple frameworks, and not once did I stumbled upon any scenario whatsoever where I had to be mindful about isolating tests. Because that was assured and a given from the very start.
I wonder what kind of convoluted mess you are doing with your tests.
> You’ve never ever had to be mindful about isolation of tests? Also, Unit tests aren’t the only tests that exist.
No. Frameworks take care of it out of the box, and you need to go way out of your way to mess that up.
> Also, Unit tests aren’t the only tests that exist.
True, but that's immaterial to the discussion.
> All you gotta do is google “test isolation”.
Yes,and if you follow your own recommendations you will eventually discover that virtually all testing frameworks are explicitly designed with test isolation in mind, to the point that you need to go way out of your way to design custom test fixtures to share context across test runs.
> Here’s one example since junit came up
I seriously do not think you even read the article you are citing. Did you even bothered to open the link?
Maybe I misread. When I see a setUp() in a test suite, 9 times out of 10 it's to handle something shared and bulky, like a database. Otherwise you'd just do the thing - assert(expected, myService.run(input)); If I saw an extra myService.reset() or myService.setUp() I would suspect it's either in anticipation of a bad state (from a previous test) or to be a good neighbour for the next test which will run.
In the cases where each unit test does have it's own individual Postgres or whatever, sure.
Ehhh... when it truly is two tests bodged into one, then sure.
But sometimes you do this kind of thing to avoid useless test brittleness: does your test check that `doSomething()` does what you expect, or do you have another test for that and this test only checks that `nowSomethingElse()` changes the object in a predictable way, e.g. updates a calculated field?
If it's the former, then it might be two tests masquerading as one, and this might make sense.
If it's the latter, you've changed a test that only checks what it cares about, and now you have a test that depends on unrelated implementation detail and will probably break unnecessarily in the future. Plus you've removed the assert that was documenting what it expects, so it's harder to tell if fixing it should mean updating both checks, or only the second one.
1. I'm not sure what CodeRabbit has to do with the article... It's obviously an advertisement, but it's merged into the text of the article, which I find bizarre. As an aside, I used CodeRabbit at work, and I have mixed feelings about it. I'm not entirely against using it or a similar tool, but people often treat the comments from CodeRabbit as a gospel, and they can harm their code as a result. Also, I've never seen CodeRabbit being anything more than a superficial reviewer: fixing typos, other unintended errors, but it never comments on the substance of the change, which implicitly validates it for the author.
2. On test composition. Unit tests are called "unit" because they are supposed to test one thing. If a test is testing more than one thing, it's an integration test. Ideally, people writing unit tests are the developers themselves and people writing integration tests are test (automation) people. This matters for administrative reasons: in the development cycle, the unit test is the basic check that validates a particular piece of code, probably, submitted for review or for merging. Passing such a test might be a necessary condition to progress the changeset along the designed workflow path. Integration tests, on the other hand, are more of a retrospective tool that is meant for detection of problems in the entire product. A failure of an integration test should, normally, schedule new task for the developers, not reject the one being worked on. Integration tests, typically, will require a more elaborate system under test setup and a more elaborate, perhaps involving multiple teams, investigation of the failure. They can be also a lot more expensive to run in terms of equipment used.
Finally, the author touched on a contentious subject a.k.a. the number of assertions in a test. A lot of people believe (me included) that the number of assertions in the unit test should be exactly one. This is often inconvenient because it requires implementation of equality for possibly ad hoc created set of results. Even so, I believe it's still worth it.
When it comes to integration tests, I don't believe assertions are at all the way to go. The system under test should be monitored continuously and every reading should be compared against the desired state of the system that the test modifies simultaneously with the change effected to the system. This is because, in practice, it's rarely just two features that are tested together. If the test waits until the final step to compare the desired and the actual state of the system, the error as well as the context in which it happened might be long gone.
As a side bonus: the monitoring+alerts system could well be part of the product itself, or, if not, it can be used in long-running tests intended to collect mileage (i.e. tests intended to prove that the system performance doesn't degrade over substantially long periods of time).
I had a test suite with thousands of tests. One way of running it was to take all the passing tests, and then run them repeatedly in random order.
This found new bugs involving unintended persistent state.
After a bunch of experiences like this I will now basically only use test runners that automatically run the tests in randomized order.
Gotta catch that stuff early.
While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy.
Ideally, you don't want a very large number of tests, no matter how big the system is. Tests are, effectively, an interface to the program that assesses the system quality / readiness for use. Any interface with thousands of individual pieces is difficult to use.
Random combinations of tests also don't spark joy because this means both repetition (i.e. waste of resources) and testing potentially useless (unreachable or invalid) system states (both wastes resources and creates false alarms).
Ideally, the tests should be able to compose only in desired ways and rather than combining them randomly, there should be some deterministic process that creates a unique sequence or a tree of individual tests on subsequent runs. Ideally, such a test runner could also be configured to start with an existing system in a known state s.t. the tester can apply a patch and resume testing.
> While I don't doubt the veracity of your report, I don't think this is an efficient testing strategy.
The marginal cost of running the tests like that was very small, and bugs were found. The cost per found bug seemed very reasonable. Every test failure in this strategy indicates a bug.
(This was the standard compliance test suite for Common Lisp, btw.)
Now imagine someone who needs to test something on eg. P6e-GB200 series VM in EC2 (this is one of the newest Nvidia accelerator models: GB200 multiplied by something like 60 iirc).
I don't know what the price is and I'm not even sure you can just create this kind of a VM with any regular account, but I'm sure you can imagine a price of running such a test.
The price doesn't have to be the limiting factor. It may be time (if a test takes a very long time to set up), or it could be labor (imagine needing to flash the ROM of a PCB for each test or similar).
This is what I mean when I say that your experience is likely not transferable to many real-life testing applications. But, even if you don't pay that much for an individual test, still, why waste time and resources, if you could do better?
If these tests are being run just once, the cost of creating them will likely vastly outweigh the cost of running them.
If they are run more than once, they can be reordered randomly each time at zero marginal cost.
This is not at all how it works.
But you missed the important distinction: random is bad because it repeats unnecessarily. One of the typically used examples in statistics 101 class to drive attention to this fact is the question posed to students to guess, without actually doing any math: what is the chance of two people in a room with (however many students are there in the class) to have a birthday on the same day?
People consistently estimate the chance significantly lower than what it actually is. And this error is supposed to encourage students to dedicate more attention to the subject because they will have found their intuition to be so woefully inadequate.
See this for the exact math: https://en.wikipedia.org/wiki/Birthday_problem .
* * *
Now, if you are interested in economics of writing tests and running them (but you don't have to be, unless it's your job):
First of all, for this problem, it doesn't matter how much it costs to write the tests. We are talking about the strategy to cover more feature combinations in the test, so, no matter how much it costs to write a test, the cost (per test) will be the same regardless of the strategy chosen. The combination is where the difference will manifest itself.
Second, tests are normally written by salaried employees. This makes the cost of writing an individual test very difficult to divine (and even if you could, it would be a useless metric). It's difficult to calculate because a lot of factors that contribute to writing a test happen in aggregates (eg. the tester needs to explore and familiarize themselves with the system before writing a test, but this effort contributes to multiple tests). Salaried employees are paid a fixed amount regardless of the intensity of work: if they produce one, ten or a hundred tests per month, the salary is the same. Tests can be vastly different in scope and in an effort necessary to implement them, however the effort necessary to implement a test isn't indicative of the value of the test (it's possible that a very simple test will be very effective at detecting problems where a very complex test isn't and the other way around).
All this said, suppose you did come up with a way of calculating the cost of writing an individual test: it still doesn't help you to estimate the cost of testing the product, it doesn't even move the needle on this question. It changes nothing about your QA department budget. It changes nothing about delivery schedule. It's just a number you can marvel at in your spare time...
> Any interface with thousands of individual pieces is difficult to use.
Standard libraries considered harmful: 100 interfaces with 10 pieces each are obviously easier to use.
If they are organized into meaningful groups, then, of course: the organization makes it easier to use. Compare: have thousands of name-phone number pairings in any order vs same pairings, but now they are grouped by the first letter of the name? Don't you think that the later is easier to use?
Shout out to AVA for running all tests async by default, which not only finds order-dependent issues, but also race condition and other timing related issues.
https://github.com/avajs/ava
Reversing the test order is also pretty effective: https://pypi.org/project/pytest-reverse/
> Deleting test1 loses us another property from the Test Desiderata—tests should be specific. That’s the property of tests where, when one fails, you know exactly where the problem is.
Contra Kent and, it seems, prevailing wisdom, I think simply deleting test1 is by far the simplest, clearest and best way. Provided that your testing framework tells you which specific assertion failed (e.g., by telling you the line number in a stack trace), you do know exactly where the problem is. The only thing you lose is that a test function or method may now cover several related checks (they are related by "setup dependence"), meaning their names may need to be somewhat broader. But you can still describe the specific semantics of each assert() check in a one-line comment beforehand if you want. There's no need to cram it into a legal method name.
ETA: Prefer to write tests whose "arrange" steps are as simple as possible, to minimise unnecessary overlaps. But if the simplest possible "arrange" step for a test is something that itself needs to be checked for correctness, just do that check right there, and nowhere else. Anything beyond that is ceremony that adds nothing useful.
I think this could lead to the tests getting harder to reason about over time because individual test size will just grow (many iterations of deleting the simple test in favor of a more complicated test that also asserts the simple things)
and if you start simplifying the more complicated test later, you may not realize that you're accidentally deleting checks that don't exist anywhere else other than incidentally here. So it's less clear what's important without thinking it through each time.
now all that to say I don't think it's strictly that big of a deal either way - there's always tons of room for taste that can make one or the other way better in practice. But that was my gut-reaction when reading your comment.
> Contra Kent and, it seems, prevailing wisdom, I think simply deleting test1 is by far the simplest, clearest and best way. Provided that your testing framework tells you which specific assertion failed (e.g., by telling you the line number in a stack trace), you do know exactly where the problem is.
The issue with your approach is that codepath execution is more of a graph than a linear timeline. With one test, you artificially constrains it to a linear timeline even if the assertions are correct. With multiples and independent you only assert one specific node. That lets you switch up how you do the preliminary steps. I much prefer an exhaustive test unit for step 1, and just a regular call in step 2 and step 3.
If you need test1's logic as setup for test2, then you already have that "linear timeline" -- for test2 all by itself. Additionally (that is, redundantly) running the same setup code "prefix" by itself in test1 doesn't remove test2's linear timeline.
ETA: I'm assuming your objection to a "linear timeline" is that it reduces the potential for running tests in parallel -- have I got that right? If not, what do you see as being the problem with it?
If I know that step 1 is correct for all the combinations (N) of its input (in test 1), in test 2, I only need to test the specific combinations (M) of step 2, treating step 1 as an axiom. So no need to have NxM in one test, or NxM tests.
Like if you were testing a drone stabilization software, testing the flying state can always assume that it has indeed taken off. No need to validate that it has done so for each scenarios, I can directly put it the correct value. It’s a contrived example, but that axiomatic aspect helps greatly when designing interfaces to reduce coupling between step 1 and step 2.
Someone had written a comment about whether Kent Beck has heard of global and static variables. That comment seems to have been deleted.
Kent Beck is co-creator [1] of the JUnit Testing Framework and has most definitely heard of static and global variables.
[1] https://junit.org/junit4/project-info.html
Ah Kent Beck, I see your style of writing trivial examples hasn't changed
I'm done listening to smart people proposing fixes for my dumb code.
Either I'm not smart or disciplined enough to make it work, or my colleagues are not. Mostly both.
Kent Beck is smart because he proposes fixes that work in real life, as in environments with dumb code and undisciplined people.
I can very much recommend his latest book ”Tidy first?”. It’s extremely short and concise, and is perfect for a very light book club within any tech team.
I agree, both with your analysis of him and your review of "Tidy first?". We read and discussed it at work, and I enjoyed it a lot.
I enjoyed the last (and theoretical) part of "Tidy first?" as I was already doing the recommended practices in the first two. But yes, very pragmatic advice and easy to apply to get your own corner of peace in a somewhat chaotic codebase.
One term I can use is "defensive programming" (not sure if the term has been used before). People are going to do awful stuff with their code, it's up to you to draw boundary of correctness to stop it from spilling on the part that you're responsible for. It should be automated (with tests and static analysis) as well as documented. I think of those boundary as hazmat suits and I take extra care of maintaining their integrity.
I assure you, there is no such thing as a book club at my job, especially not one focused on technical writing :)
What he wrote is basically "don't repeat in test X what you already tested in test X-1".
It's not as much composition as compounding, and can work quite well.
Let's say something takes 20 steps, and you want to test all 20.
Instead of this:
you do this: This works well in certain situations, as it skips diplicated redundant testing. Requires some discipline so that tests don't drift away from each other.As most things, it depends on what those steps are. Perhaps you only need that one final (integration) test instead of 20 intermediate unit ones.
I wonder if it would work do design something that was able to say
If the assertion of each test doesn't change any state, that might make things easier to read. Though, given that I haven't spent much time pondering it, I expect it could have it's own problems.But it could also do things like skip test 3 if tests 2 or 1 failed - because it knows about the relationship.
The hard parts to design about this, imo, are:
- How to reconcile this with tests that execute many times with varying input data. You’d need some way to express requirements with specific inputs or shared inputs.
- Passing state between test dependencies.
- When, if ever, it’s fine to share step results between tests. If tests B and C require A, can you run A just once? Not always, but you should be able to when it’s safe.
I don’t think I’ve ever used a test framework that gets these things right.
> Passing state between test dependencies
Actually, I wasn't even thinking about passing state. I was thinking about shared setup steps. I'm perfectly happy for the same steps to run for each test - as long as each test doesn't need to list the steps (when they're setup steps not directly related to the thing being tested)
For example, suppose you want to write a test for the shipOrder(orderId) function, and you want it to depend on the test for the placeOrder(shoppingCart) -> OrderId function. Even if you are fine calling placeOrder twice, once for the placeOrder test and once for the placeOrderAndShipOrder test, you still need the placeOrder test to provide an order ID to the second test, and not just a confirmation that it completed successfully.
The pattern can work, but the domain matters, the test type matters (unit, integration, ui) and the trade-off associated matter.
e.g. If I am running a long-running UI-test scenario, I absolutely don't want test-5 to walk through 80% of the UI that was already exercised in tests 1-4. I am creating test coupling, but I'm saving cost/time by doing so.
But, you'll also hear why not to do this, because it creates test coupling / breaks atomic tests, which is generally seen as bad.
If that is a local integration test and those early steps run is millis? Then maybe we keep things uncoupled to allow the system to exercise the pathways without explicit expectations.
The question was less about speed and more about not having the same code duplicated over and over across tests. Which is how I read the article talking about it. Allowing one test to "depend" on another makes it clear they use the same setup (presumably with the second test going "a bit further", but not necessarily).
I wouldn't have a problem with something like
The format is awful, but the idea is that most tests are of the form And, in that context, the GIVEN frequently contains noise not directly related to understanding what is being tested.I actually use the GIVEN/WHEN/THEN keywords in my tests, to make them easier to read
Yes that is the way to do it, in Spock that’s what @Stepwise does.
This make too much sense!
In special when testing against a DB.
Yup. I'd love to have dependency declarations for tests like this.
That's... A thing in many frameworks. If it's not in yours, you could add it.
Forgive an old man some ruby:
[flagged]
IMO it seems kind of terrible.
You're not going to remember what tests "test 3" relies on. They aren't actually linear progressions 123. They will be "test this", "test that". If 3 fails you're going to want to immediately go and add all those asserts back to help you debug your assumptions.
The tests _will_ drift and that should be fine. Implicitly depending on other tests doesn't really get me anything.
What's the problem of only keeping test 3 if it depends on test 2 and 1 passing anyway?
The article mentions not to do this because "Deleting test1 loses us another property from the Test Desiderata—tests should be specific. That’s the property of tests where, when one fails, you know exactly where the problem is." but you'll know what line it failed on. And some test runners let you break a test into steps, where groups of lines are given a description.
Or put each step + assert in a helper function (e.g. `doStep1AndAssert()`), and each test only calls these helper functions?
Nothing is perfect, but copy/pasting chunks between tests like this isn't great when you want to refactor and it's repetitive to read.
> What's the problem of only keeping test 3 if it depends on test 2 and 1 passing anyway?
That depends.
Sometimes test 1 tests a combination of ways (e.g., property testing, or just going through a bunch of various inputs), and only a few of those are needed for test 2.
Sometimes you don't want your test 2 to be more complicated than it already is. Or the same things are needed checked in other tests. So you extract them into test 1.
And sometimes (and in some of code bases most of the time) test 1 is redundant and unnecessary. That's why I always advocate investing in integration tests (test 20) and skip all the intermediate tests.
> That's why I always advocate investing in integration tests (test 20) and skip all the intermediate tests.
My approach is to have acceptance tests documented for any feature. Like how it would be from the user point of view to actually use the software. Then do Integration tests for each part of that workflows. That's usually the most ROI you will get for testing. Then I invest into unit tests for particular elements that are very important. I start from the middle of the pyramid because an actual e2e is expensive to setup (easy to maintain afterwards) and having lots of unit tests (easy to setup) is expensive to maintain.
I think I disagree with Kent, but your explanation is clearer, so I'll object here.
There's nothing wrong with hitting the same assertion multiple times, even if it doesn't sit nicely in your gut.
From a purely philosophical point of view: If I have testFoo(), testBar(), and testFooAndBar(), and my Foo is plain wrong, then both testFoo() and testFooAndBar() must fail. Anything less is misleading/dishonest.
From a practical side: Changes happen. Someone will remove testBar(), and then you're down to 0 assertions on Bar, even though you have a test claiming to testFooAndBar(). It's not even a crazy hypothetical. Someone with a different test philosophy will think (to quote TFA) "They are redundant! Something must be wrong." and delete testBar() because obviously testFooAndBar() already covers it.
Anyway, we all know how to deal with repetition. That's what programming is!
If you have people on your team deleting valid tests because of "philosophy", I think you have much bigger problems to solve than anything Kent Beck can help with.
To be clear, the philosophy I quoted was directly from Kent Beck in TFA, i.e. this is Kent Beck's "help".
I say leave both tests as is.
Kent Beck says:
It's not just philosophy, it's aesthetics apparently!Why wouldn't you use parameterized tests or compose your setup out of fixtures that run the shared setup code?
Running a test, then running it a second time, then asserting something else.. is just weird.
What I do not like about that kind of example is its abstractiveness. Yes sure you can argue about testing `doSomething()` and it all falls apart when there is an actual business scenario to test.
It's not hard, with some modest experience (say 1-2 years of professional work or serious amateur interest), to extrapolate from his deliberately high-level discussion (it was written for a blog/newsletter, not for a book) to something more interesting and "real world". What's hard is coming up with an example that's nearly complete and fits into something the size of a newsletter or comment. He did an alright job of it, just left the readers with the need to exercise their gray matter a bit.
If it is not hard, with an experience of this guy, he could came up with a better example? It is your own words so I am sure you will agree? He did alright job so he could exercise gray matter a bit and came up with some API for saving customer data and then test it his way? I mean... This would be more interesting and "real world" so I am sure it would be worth to write longer blog post about it!
> If a test runs by first setting up its own test fixture, creating from scratch all the data it will be using as input, then that test is guaranteed to be isolated. It doesn’t matter what order you run the tests, the results will be exactly the same.
Completely backward. This definition requires isolation, rather than granting it.
In reality, you (or your test framework) provides the isolation by hobbling along, executing only one test at a time.
I don’t think that’s right?
The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation.
If I create a new in mem dB in the test there’s nothing stopping me from running it in parallel?
Nothing about that “requires” isolation. It is isolation.
> The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation.
I'm not sure what you mean by that. All the frameworks I ever used were designed with test isolation as the primary design goal. Even when you set shared test fixtures and setup/teardown code, all they provide is a way to share code across tests, which are by themselves independent and isolated. BDD-style frameworks are the notable exception in breaking away from the pattern of having isolation as a fundamental design trait. In fact, the whole reason why BDD-style tools require special support from testing frameworks is that they need to work around test isolation in order to share context across steps.
In JUnit, tests run sequentially in a single thread by default [1].
Parallel execution must be explicitly enabled. When enabled, JUnit uses a fork-join thread pool, so tests may run concurrently on different worker threads. Because these threads are reused, a ThreadLocal value left behind by one test could be visible to a later test that happens to run on the same thread.
Setup and teardown methods can be used to create and clean up test-specific state. However, developers must still ensure that test data is unique to the test so that concurrently running tests do not interfere with each other. This uniqueness is unfortunately called "isolated", and has led to much confusion like in this thread. Certainly, the Test Execution Framework cannot guarantee data-isolation.
Parallel and randomized test execution can also help expose application-side problems involving shared or order-dependent state. Such failures may only appear when tests happen to exercise the application under the relevant ordering or concurrency conditions. When I was a junior developer teaching myself Java Servlets in 2000, I had to learn this lesson the hard way. A Test Execution Framework would not be able to guarantee any "isolation" of test data and of workflows if the server-side state is mis-managed by the tech stack and/or by the developer.
[1] https://docs.junit.org/6.1.3/writing-tests/parallel-executio...
> Parallel execution must be explicitly enabled. When enabled, JUnit uses a fork-join thread pool, so tests may run concurrently on different worker threads. Because these threads are reused, a ThreadLocal value left behind by one test could be visible to a later test that happens to run on the same thread.
I don't understand your comment. JUnit docs describe parallel execution as an experimental feature that still has gotchas that you need to be mindful to avoid. So you have a test framework designed with isolation in mind for a specific execution mode, but when you go out of your way to try another execution mode that is described as experimental then you can stumble upon very corner cases where isolation is not ensured. What point did you wanted to convey?
I disagree so hard with the notion that isolation is somehow done for you, that I don't even know how to begin arguing.
I'll admit to starting this confusion by reading what I thought would be written, not what was actually there. TFA does phrase it to my satisfaction that the test is isolated. I assumed when reading it that Kent thought he had isolation but did not. By analogies: a naive programmer does not perceive the existence of race conditions; a naive web app developer doesn't perceive the existence of CAP; a naive backend engineer thinks they have full ACID guarantees when they're only guaranteed READ_COMMITTED. I thought I was reading another writer just wishing isolation into existence through ignorance.
(So I accidentally straw-manned Kent's argument, but maybe that's not such a problem, because it seems you're OK with the straw man version?)
Anyway, maybe a question moves things forward.
JUnit's been around for decades. Every developer has had multicore dev machines for decades. Why would parallel execution not be the default, or a simple toggle that everyone would turn on immediately?
This
All I mean that there is no way for a framework to prevent your tests from interfering with each other, or to solve isolation for you.
It’s the implementation of each test that is responsible for its isolation. Every time you write or make changes to your tests. You have to think, can they mess with each other.
> All I mean that there is no way for a framework to prevent your tests from interfering with each other, or to solve isolation for you.
What? By design that's exactly how virtually all test frameworks are designed to work.
> It's the implementation of each test that is responsible for its isolation. Every time you write or make changes to your tests. You have to think, can they mess with each other.
Nonsense. I've been writing unit tests for years in multiple programming languages and using multiple frameworks, and not once did I stumbled upon any scenario whatsoever where I had to be mindful about isolating tests. Because that was assured and a given from the very start.
I wonder what kind of convoluted mess you are doing with your tests.
You’ve never ever had to be mindful about isolation of tests? Also, Unit tests aren’t the only tests that exist.
You say assured? You think it’s impossible to write 2 tests that interact each other?
All you gotta do is google “test isolation”. But if you want to pretend it’s not a thing the I have no idea what to tell you?
Here’s one example since junit came up
https://martinfowler.com/bliki/JunitNewInstance.html
> You’ve never ever had to be mindful about isolation of tests? Also, Unit tests aren’t the only tests that exist.
No. Frameworks take care of it out of the box, and you need to go way out of your way to mess that up.
> Also, Unit tests aren’t the only tests that exist.
True, but that's immaterial to the discussion.
> All you gotta do is google “test isolation”.
Yes,and if you follow your own recommendations you will eventually discover that virtually all testing frameworks are explicitly designed with test isolation in mind, to the point that you need to go way out of your way to design custom test fixtures to share context across test runs.
> Here’s one example since junit came up
I seriously do not think you even read the article you are citing. Did you even bothered to open the link?
¯\_(ツ)_/¯
You can believe whatever you want to believe. Or maybe you live in a magical world where all your tests are pure and side effects nor IO exist.
In the other thread you think running tests in parallel doesn't count because it has documented gotchas...
Maybe I misread. When I see a setUp() in a test suite, 9 times out of 10 it's to handle something shared and bulky, like a database. Otherwise you'd just do the thing - assert(expected, myService.run(input)); If I saw an extra myService.reset() or myService.setUp() I would suspect it's either in anticipation of a bad state (from a previous test) or to be a good neighbour for the next test which will run.
In the cases where each unit test does have it's own individual Postgres or whatever, sure.
I guess it all depends, isolation isn’t binary.
you can share some things and be isolated across other dimensions.
You could share a Postgres dB connection but just isolate the data logically.
You can isolate or share across individual tests, across suites, across envs, across runs, across time.
More isolation is generally better unless the complexity or cost is too high.
But I guess I was just confused as to how a test requires isolation rather than have it.
Trimming the copy-paste chain is right. The pieces being green isn’t the same as the app opening.
Ehhh... when it truly is two tests bodged into one, then sure.
But sometimes you do this kind of thing to avoid useless test brittleness: does your test check that `doSomething()` does what you expect, or do you have another test for that and this test only checks that `nowSomethingElse()` changes the object in a predictable way, e.g. updates a calculated field?
If it's the former, then it might be two tests masquerading as one, and this might make sense.
If it's the latter, you've changed a test that only checks what it cares about, and now you have a test that depends on unrelated implementation detail and will probably break unnecessarily in the future. Plus you've removed the assert that was documenting what it expects, so it's harder to tell if fixing it should mean updating both checks, or only the second one.
1. I'm not sure what CodeRabbit has to do with the article... It's obviously an advertisement, but it's merged into the text of the article, which I find bizarre. As an aside, I used CodeRabbit at work, and I have mixed feelings about it. I'm not entirely against using it or a similar tool, but people often treat the comments from CodeRabbit as a gospel, and they can harm their code as a result. Also, I've never seen CodeRabbit being anything more than a superficial reviewer: fixing typos, other unintended errors, but it never comments on the substance of the change, which implicitly validates it for the author.
2. On test composition. Unit tests are called "unit" because they are supposed to test one thing. If a test is testing more than one thing, it's an integration test. Ideally, people writing unit tests are the developers themselves and people writing integration tests are test (automation) people. This matters for administrative reasons: in the development cycle, the unit test is the basic check that validates a particular piece of code, probably, submitted for review or for merging. Passing such a test might be a necessary condition to progress the changeset along the designed workflow path. Integration tests, on the other hand, are more of a retrospective tool that is meant for detection of problems in the entire product. A failure of an integration test should, normally, schedule new task for the developers, not reject the one being worked on. Integration tests, typically, will require a more elaborate system under test setup and a more elaborate, perhaps involving multiple teams, investigation of the failure. They can be also a lot more expensive to run in terms of equipment used.
Finally, the author touched on a contentious subject a.k.a. the number of assertions in a test. A lot of people believe (me included) that the number of assertions in the unit test should be exactly one. This is often inconvenient because it requires implementation of equality for possibly ad hoc created set of results. Even so, I believe it's still worth it.
When it comes to integration tests, I don't believe assertions are at all the way to go. The system under test should be monitored continuously and every reading should be compared against the desired state of the system that the test modifies simultaneously with the change effected to the system. This is because, in practice, it's rarely just two features that are tested together. If the test waits until the final step to compare the desired and the actual state of the system, the error as well as the context in which it happened might be long gone.
As a side bonus: the monitoring+alerts system could well be part of the product itself, or, if not, it can be used in long-running tests intended to collect mileage (i.e. tests intended to prove that the system performance doesn't degrade over substantially long periods of time).
That's why you pepper your code with assertions and rely on depth of test coverage (including generative testing) to hit them.
[flagged]