Mutants are killed by empty test suites?

[nli-mdsol]

Three questions:

1. If I commented out all test cases in a test suite and run this empty test suite against the code under test, some mutants can still be killed. What are these killed mutants? I cannot see these mutants because mutant only returns unkilled mutants.
2. It seems that mutant does not generate mutants for singleton methods. I have a singleton class, and mutants are not generated for the part
   class << self
   ...
   end
3. Can we choose which mutation operators to use? Or there is no such option?

[mbj]

What are these killed mutants? I cannot see these mutants because mutant only returns unkilled mutants.

Neutral mutations. Basically these are from the apply of no mutation operator, with the test signal interpretation inversed. If such a test fails you know the environment mutant creates for executing tests (isolated, parallel execution) interfere with your tests. Neutral mutations are a safeguard feature being build to detect various kinds of integration issues. Rule of thumb: When you see alive neutral mutations, something is wrong with your setup (EDIT: Or mutant).

There is already some tickets about the behavior of neutral/noop mutations:

• Remove neutral/noop mutants from coverage calculations. #180
• Improve behavior when unmutated version already fails #182

I did not had enough pain / time to solve these yet.

Can we choose which mutation operators to use? Or there is no such option?

No. Intentionally not supported. I'm happy with the mutation operators, and had no case where I wanted to tune the operators per project or per subject.

It seems that mutant does not generate mutants for singleton methods.

Mutant does not support the following form:

class Foo
  class << self
    def bar
    end
  end
end # Foo

This one is fine:

class Foo
  def self.bar
  end
end

I only use the latter form, for that reason I never implemented support for the class << self one.

I opened an issue to track this feature. #282

Its unlikely I'll have time for this, but I agree its a nice to have.

I'm closing this issue under the assumption I answered all questions. Feel free to reopen / discuss in case you do not agree.

[nli-mdsol]

Thank you for answering my questions.
For the "neutral mutation," I think I can also observe it by running a non-empty test suite.
For example, I have a method in the class under test below.

  def which_changed
    attribute_get(:which_changed)
  end

When I run tests against generated mutants, I got the output from mutant:

Original code was inserted unmutated. And the test did NOT PASS.
Your tests do not pass initially or you found a bug in mutant / unparser.
Subject AST:
(def :which_changed
  (args)
  (send nil :attribute_get
    (sym :which_changed)))
Unparsed Source:
def which_changed
  attribute_get(:which_changed)
end
Test Reports: 1

I think this also indicates the "neutral mutation." Correct me if I am wrong.
If I understand correctly, can I say that the "neutral mutation" happens when mutant could not generate a mutant that can be parsed by the Ruby interpreter due to a potential integration issue from the tests?
If we cannot identify what the integration issue is or why mutant cannot generate mutants, I think I should just skip these unkilled mutants for now.
Again, correct me if I misunderstand anything.

[mbj]

@nli-mdsol

What version of mutant you are running? Normally it should also print the test output for neutral failures. If you are not on latest 0.7.4 for both mutant and mutant-rspec can you update? And post the full test report?

The debug output you posted does not indicate a problem with unparser / mutant. Does rspec still pass your tests, even when commented out?

Original code was inserted unmutated. And the test did NOT PASS.
Your tests do not pass initially or you found a bug in mutant / unparser.

Mutant cannot decide which one of the following cases is true. It can only find a broken test where there should NOT be a broken test. When you insert the original code as a mutation the tests should still pass. This is a self-test safeguard feature build into mutant to detect weird edge cases where integration issues could potentially lead to false positively killed mutation tests.

I need more context to debug it. Apart from the full report, a executable reproduction would be handy.

BTW: I formatted your reply a bit to render better, use code blocks in future to pass output / code, makes it much easier for people to understand.

[nli-mdsol]

I am using mutant 0.6.7 and this version also gives failure outputs. I
tried to use 0.7.4 but this version is incompatible with my project.
Mutant 0.7.4 requires rspec 3.0.0 and above but mine uses 2.x.x. Because
I am working on one of my company's products, I do not want to make any
changes to the development work, and I am not able to share any of the
code. I hope that mutant 0.7.4 could be compatible with lower rspec
versions.

Thank you for reminding me the failure outputs. After I looked at these
messages carefully, they tell me that which is missing and why tests fail.
As you mentioned, these problems are related to dependencies (integration
issue). My tests use databases and execute database operations. My original
rspec setting only create tables once before executing all tests. So my
tests only clean data after executing each test but still keep the tables.
But I think mutant creates a different environment: drop all tables after
executing each test against each mutant. Therefore, I got the failure
message that says unknown database tables. After I changed the code to
create tables before executing every test, the error message is gone and I
am able to see the implementation of the mutants.

I think it will be easier for future users to use mutant if you could
address this integration issue in the README file.

On Sun, Jan 11, 2015 at 6:46 AM, Markus Schirp [email protected]
wrote:

@nli-mdsol https://github.com/nli-mdsol

What version of mutant you are running? Normally it should also print the
test output for neutral failures. If you are not on latest 0.7.4 for both
mutant and mutant-rspec can you update? And post the full test report?

The debug output you posted does not indicate a problem with unparser /
mutant. Does rspec still pass your tests, even when commented out?

Original code was inserted unmutated. And the test did NOT PASS.
Your tests do not pass initially or you found a bug in mutant / unparser.

Mutant cannot decide which one of the following cases is true. It can only
find a broken test where there should NOT be a broken test. When you insert
the original code as a mutation the tests should still pass. This is a
self-test safeguard feature build into mutant to detect weird edge cases
where integration issues could potentially lead to false positively killed
mutation tests.

I need more context to debug it. Apart from the full report, a executable
reproduction would be handy.

BTW: I formatted your reply a bit to render better, use code blocks in
future to pass output / code, makes it much easier for people to understand.

—
Reply to this email directly or view it on GitHub
#281 (comment).

[mbj]

I hope that mutant 0.7.4 could be compatible with lower rspec versions.

This is not so easy. Mutant hooks deeply into rspec APIs, and those changed between rspec 2 and rspec 3 releases. This deep hooking is needed to make killing efficient and test selection fine grained.

I have the policy to only support my usecases in my OSS projects, unless there is a good reason to support features / environments I do not use. Upgrading from rspec 2 to rspec 3 is very easy, so this change does not fall into that category.

My OSS time is very limited and this policy makes sure I use it efficiently. Sorry.

[mbj]

Thank you for reminding me the failure outputs. After I looked at these messages carefully, they tell me that which is missing and why tests fail.

Yeah, they where exactly made for that kind of environment debugging.

My tests use databases and execute database operations

The database is shared state that gets modified concurrently when you run mutant with paralellization. Even under a single execution thread dirty state can be leaked as your tests and even your ruby interpreter can crash any time when executing mutated code. See #265.

With such a crash shared global state leaks into the next test. And the neutrals save you from killing your mutations by environment, not by tests.

You need to make sure there is nothing leaking through spec examples. For some projects I needed to develop a custom data base cleaner that wrapped each test into a SERIALZABLE transaction that never was committed. This way each test could run in its whole universe under concurrency.

I think it will be easier for future users to use mutant if you could address this integration issue in the README file.

There is a way more to document. Every reachable global mutable state needs to be protected.

• Database state
• Filesystem
• Sockets
• Remote API state (seen that)

Thankfully writing a "mutant compatible" test suite also typically forces you to write "real" unit tests, so this kind of state leaks are rare.

I can currently not envision how to explain all this in a README section. But I should explain the role of neutral mutations for detecting anormalizes and the strategy to deal with them.

Thx for your report.

[nli-mdsol]

Thank you for the reply. It really helped me understand how mutant works
and how to run mutant.

On Mon, Jan 12, 2015 at 8:29 AM, Markus Schirp [email protected]
wrote:

Thank you for reminding me the failure outputs. After I looked at these
messages carefully, they tell me that which is missing and why tests fail.

Yeah, they where exactly made for that kind of environment debugging.

My tests use databases and execute database operations

The database is shared state that gets modified concurrently when you run
mutant with paralellization. Even under a single execution thread dirty
state can be leaked as your tests and even your ruby interpreter can crash
any time when executing mutated code. See #265
#265.

With such a crash shared global state leaks into the next test. And the
neutrals save you from killing your mutations by environment, not by tests.

You need to make sure there is nothing leaking through spec examples. For
some projects I needed to develop a custom data base cleaner that wrapped
each test into a SERIALZABLE transaction that never was committed. This
way each test could run in its whole universe under concurrency.

I think it will be easier for future users to use mutant if you could
address this integration issue in the README file.

There is a way more to document. Every reachable global mutable state
needs to be protected.

• Database state
• Filesystem
• Sockets
• Remote API state (seen that)

Thankfully writing a "mutant compatible" test suite also typically forces
you to write "real" unit tests, so this kind of state leaks are rare.

I can currently not envision how to explain all this in a README section.
But I should explain the role of neutral mutations for detecting
anormalizes and the strategy to deal with them.

Thx for your report.

—
Reply to this email directly or view it on GitHub
#281 (comment).

[mbj]

Thank you for the reply. It really helped me understand how mutant works and how to run mutant.

Thx for reporting your issues. I use my answers here to build up an inventory of text I can use for writing some real documentation in future. Keep asking please!