poor coverage of method that uses Reflection.getCallerClass(int)

[liblit-at-amazon]

Description

UnitTestBot generates one test for a method that uses Reflection.getCallerClass. This one test does not fully explore the possible behaviors of the method under test. Generated comments claim that this test case succeeds, but it actually fails. The Reflection.getCallerClass call might need to be mocked, but no such mock is used.

To Reproduce

Steps to reproduce the behavior:

1. Check out revision jdeparser/jdeparser2@fd6310c. Important: use this specific revision, not the current main head revision.

2. Verify that the checked-out code includes the following org.jboss.jdeparser.Assertions.callerIs(Class<?>) method:

   static boolean callerIs(Class<?> clazz) {
       try {
           return Reflection.getCallerClass(3) == clazz;
       } catch (Throwable ignored) {}
       // dunno
       return true;
   }

3. Add the following to pom.xml to explicitly select Java 1.8:

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

4. Open this project in IntelliJ IDEA and link its pom.xml, thereby making this a Maven project.

5. In the IntelliJ IDEA Project Settings, select a Java 1.8 SDK.

6. Use the UnitTestBot IntelliJ IDEA plugin to generate tests for the org.jboss.jdeparser.Assertions.callerIs(Class<?>) method. Leave all UnitTestBot settings at their defaults, other than selecting this one method to generate tests for.

Expected behavior

1. UnitTestBot should have generated one or more tests that cover the various behaviors of the callerIs method, including passing tests that show various possible return values from this method. Specifically, I would have expected at least two test cases: one for a true result and one for a false result. Even better would have been three cases, to explore the two different paths that could lead to a true result.

2. Comments associated with the generated tests should correctly reflect which will pass and which will fail.

3. Mocks should be used as needed, specifically for the Reflection.getCallerClass(int) call made by the method under test.

Actual behavior

UnitTestBot generated a test class with a single test method:

Generated `AssertionsTest.java`

package org.jboss.jdeparser;

import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.assertTrue;

public final class AssertionsTest {
    ///region Test suites for executable org.jboss.jdeparser.Assertions.callerIs

    ///region SYMBOLIC EXECUTION: SUCCESSFUL EXECUTIONS for method callerIs(java.lang.Class)

    /**
     * @utbot.classUnderTest {@link Assertions}
     * @utbot.methodUnderTest {@link Assertions#callerIs(Class)}
     * @utbot.invokes {@link sun.reflect.Reflection#getCallerClass(int)}
     * @utbot.returnsFrom {@code return Reflection.getCallerClass(3) == clazz;}
     */
    @Test
    @DisplayName("callerIs: ReflectionGetCallerClass -> return Reflection.getCallerClass(3) == clazz")
    public void testCallerIs_ReflectionGetCallerClassNotEqualsClazz() {
        boolean actual = Assertions.callerIs(null);

        assertTrue(actual);
    }
    ///endregion

    ///endregion
}

This test suite differs from each of the three expectations listed earlier:

1. With only a single call to callerIs, this suite does not fully explore the possible behaviors of the method under test.

2. The one generated test case is documented as succeeding, but actually fails. The generated call to callerIs returns false, which then fails the assertTrue(actual) check.

3. No mock was created for the Reflection.getCallerClass(int) call made by the method under test.

Environment

• IntelliJ IDEA 2023.2.6 (Ultimate Edition)
• JetBrains UnitTestBot plugin version 2023.10
• All UnitTestBot plugin settings were left at their default values.