Project Student: Sharding Integration Test Data

This is part of Project Student. Other posts are Webservice Client With Jersey, Webservice Server with Jersey, Business Layer and Persistence with Spring Data.

All of the integration tests until now have used an in-memory embedded database that did not retain information from run to run. This changes when we fully integrate the REST server with a “real” database server – leftover test data will pollute our development or test database. This can be a real headache once we have continuous integration that runs integration tests code is checked in.

One solution is to ‘shard’ our integration test data in a way that allows our tests to use the shared development database without polluting it or other tests. The easiest approach is to add a TestRun field to all of our objects. “Test” data will have a value that indicates the specific test run, “live” data will have a null value.

The exact timeline is

  1. create and persist a TestRun object
  2. create test objects with appropriate TestRun value
  3. perform the integration tests
  4. delete the test objects
  5. delete the TestRun object

Any entry in the TestRun table will either be 1) active integration tests or 2) failed integration tests that threw an unhandled exception (depending upon the transaction manager, of course). It’s important to note that we can also capture the database state after an unexpected exception is thrown even if the transaction manager performs a rollback – it’s a simple extension to the junit test runner.

Timestamp and user fields make it easy to delete stale test data according to its age (e.g., any test more than 7 days old) or the person who ran the test.

TestablePersistentObject abstract base class

This change starts at the persistence level so we should start there and work our way outwards.

We first extend our PersistentObject abstract base class with a test run value.

@MappedSuperclass
public abstract class TestablePersistentObject extends PersistentObject {
    private static final long serialVersionUID = 1L;
    private TestRun testRun;

    /**
     * Fetch testRun object. We use lazy fetching since we rarely care about the
     * contents of this object - we just want to ensure referential integrity to
     * an existing testRun object when persisting a TPO.
     * 
     * @return
     */
    @ManyToOne(fetch = FetchType.LAZY, optional = true)
    public TestRun getTestRun() {
        return testRun;
    }

    public void setTestRun(TestRun testRun) {
        this.testRun = testRun;
    }

    @Transient
    public boolean isTestData() {
        return testRun != null;
    }
}

TestRun class

The TestRun class contains identifying information about a single integration test run. It contains a name (by default the classname#methodname() of the surrounding integration test), the date and time of the test, and the name of the user running the test. It would be easy to capture additional information.

The list of test objects gives us two big wins. First, it makes it easy to capture the state of the database if needed (e.g., after an unexpected exception). Second, cascading deletions makes it easy to delete all test objects.

@XmlRootElement
@Entity
@Table(name = "test_run")
@AttributeOverride(name = "id", column = @Column(name = "test_run_pkey"))
public class TestRun extends PersistentObject {
    private static final long serialVersionUID = 1L;

    private String name;
    private Date testDate;
    private String user;
    private List<TestablePersistentObject> objects = Collections.emptyList();

    @Column(length = 80, unique = false, updatable = true)
    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    @Column(name = "test_date", nullable = false, updatable = false)
    @Temporal(TemporalType.TIMESTAMP)
    public Date getTestDate() {
        return testDate;
    }

    public void setTestDate(Date testDate) {
        this.testDate = testDate;
    }

    @Column(length = 40, unique = false, updatable = false)
    public String getUser() {
        return user;
    }

    public void setUser(String user) {
        this.user = user;
    }

    @OneToMany(cascade = CascadeType.ALL)
    public List<TestablePersistentObject> getObjects() {
        return objects;
    }

    public void setObjects(List<TestablePersistentObject> objects) {
        this.objects = objects;
    }

    /**
     * This is similar to standard prepersist method but we also set default
     * values for everything else.
     */
    @PrePersist
    public void prepersist() {
        if (getCreationDate() == null) {
            setCreationDate(new Date());
        }

        if (getTestDate() == null) {
            setTestDate(new Date());
        }

        if (getUuid() == null) {
            setUuid(UUID.randomUUID().toString());
        }

        if (getUser() == null) {
            setUser(System.getProperty("user.name"));
        }

        if (name == null) {
            setName("test run " + getUuid());
        }
    }
}

The TestRun class extends PersistentObject, not TestablePersistentObject, since our other integration tests will sufficiently exercise it.

Spring Data Repository

We must add one additional method to every Repository.

@Repository
public interface CourseRepository extends JpaRepository {
    List<Course> findCoursesByTestRun(TestRun testRun);

    ....
}

Service Interface

Likewise we must add two additional methods to every service.

public interface CourseService {
    List<Course> findAllCourses();

    Course findCourseById(Integer id);

    Course findCourseByUuid(String uuid);

    Course createCourse(String name);

    Course updateCourse(Course course, String name);

    void deleteCourse(String uuid);

    // new method for testing
    Course createCourseForTesting(String name, TestRun testRun);

    // new method for testing
    List<Course> findAllCoursesForTestRun(TestRun testRun);
}

I won’t show the TestRunRepository, the TestRunService interface, or the TestRunService implementation since they’re identical to what I’ve described in the last few blog entries.

Service Implementation

We have to make one small change to an existing Service implementation, plus add two new methods.

@Service
public class CourseServiceImpl implements CourseService {
    @Resource
    private TestRunService testRunService;

    /**
     * @see com.invariantproperties.sandbox.student.business.CourseService#
     *      findAllCourses()
     */
    @Transactional(readOnly = true)
    @Override
    public List<Course> findAllCourses() {
        List<Course> courses = null;

        try {
            courses = courseRepository.findCoursesByTestRun(null);
        } catch (DataAccessException e) {
            if (!(e instanceof UnitTestException)) {
                log.info("error loading list of courses: " + e.getMessage(), e);
            }
            throw new PersistenceException("unable to get list of courses.", e);
        }

        return courses;
    }

    /**
     * @see com.invariantproperties.sandbox.student.business.CourseService#
     *      findAllCoursesForTestRun(com.invariantproperties.sandbox.student.common.TestRun)
     */
    @Transactional(readOnly = true)
    @Override
    public List<Course> findAllCoursesForTestRun(TestRun testRun) {
        List<Course> courses = null;

        try {
            courses = courseRepository.findCoursesByTestRun(testRun);
        } catch (DataAccessException e) {
            if (!(e instanceof UnitTestException)) {
                log.info("error loading list of courses: " + e.getMessage(), e);
            }
            throw new PersistenceException("unable to get list of courses.", e);
        }

        return courses;
    }

    /**
     * @see com.invariantproperties.sandbox.student.business.CourseService#
     *      createCourseForTesting(java.lang.String,
     *      com.invariantproperties.sandbox.student.common.TestRun)
     */
    @Transactional
    @Override
    public Course createCourseForTesting(String name, TestRun testRun) {
        final Course course = new Course();
        course.setName(name);
        course.setTestUuid(testRun.getTestUuid());

        Course actual = null;
        try {
            actual = courseRepository.saveAndFlush(course);
        } catch (DataAccessException e) {
            if (!(e instanceof UnitTestException)) {
                log.info("internal error retrieving course: " + name, e);
            }
            throw new PersistenceException("unable to create course", e);
        }

        return actual;
    }
}

CourseServiceIntegrationTest

We make a few changes to our integration tests. We only have to change one test method since it’s the only one that actually creates a test object. The rest of the methods are queries that don’t require test data.

Note that we change the name value to ensure it’s unique. This is a way to work around uniqueness constraints, e.g., for email addresses.

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(classes = { BusinessApplicationContext.class, TestBusinessApplicationContext.class,
        TestPersistenceJpaConfig.class })
@Transactional
@TransactionConfiguration(defaultRollback = true)
public class CourseServiceIntegrationTest {

    @Resource
    private CourseService dao;

    @Resource
    private TestRunService testService;

    @Test
    public void testCourseLifecycle() throws Exception {
        final TestRun testRun = testService.createTestRun();

        final String name = "Calculus 101 : " + testRun.getUuid();

        final Course expected = new Course();
        expected.setName(name);

        assertNull(expected.getId());

        // create course
        Course actual = dao.createCourseForTesting(name, testRun);
        expected.setId(actual.getId());
        expected.setUuid(actual.getUuid());
        expected.setCreationDate(actual.getCreationDate());

        assertThat(expected, equalTo(actual));
        assertNotNull(actual.getUuid());
        assertNotNull(actual.getCreationDate());

        // get course by id
        actual = dao.findCourseById(expected.getId());
        assertThat(expected, equalTo(actual));

        // get course by uuid
        actual = dao.findCourseByUuid(expected.getUuid());
        assertThat(expected, equalTo(actual));

        // get all courses
        final List<Course> courses = dao.findCoursesByTestRun(testRun);
        assertTrue(courses.contains(actual));

        // update course
        expected.setName("Calculus 102 : " + testRun.getUuid());
        actual = dao.updateCourse(actual, expected.getName());
        assertThat(expected, equalTo(actual));

        // verify testRun.getObjects
        final List<TestablePersistentObject> objects = testRun.getObjects();
        assertTrue(objects.contains(actual));

        // delete Course
        dao.deleteCourse(expected.getUuid());
        try {
            dao.findCourseByUuid(expected.getUuid());
            fail("exception expected");
        } catch (ObjectNotFoundException e) {
            // expected
        }

        testService.deleteTestRun(testRun.getUuid());
    }

    ....
}

We could use @Before and @After to transparently wrap all test methods but many tests don’t require test data and many tests that do require test data require unique test data, e.g., for email addresses. In the latter case we fold in the Test UUID as above.

REST Webservice Server

The REST webservice requires adding a test uuid to the request classes and adding a bit of logic to properly handle it when creating an object.

The REST webservice does not support getting a list of all test objects. The “correct” approach will be creating a TestRun service and providing associated objects in response to a /get/{id} query.

@XmlRootElement
public class Name {
    private String name;
    private String testUuid;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getTestUuid() {
        return testUuid;
    }

    public void setTestUuid(String testUuid) {
        this.testUuid = testUuid;
    }
}

We can now check for the optional testUuid field and call the appropriate create method.

@Service
@Path("/course")
public class CourseResource extends AbstractResource {
    @Resource
    private CourseService service;

    @Resource
    private TestRunService testRunService;

    /**
     * Create a Course.
     * 
     * @param req
     * @return
     */
    @POST
    @Consumes({ MediaType.APPLICATION_JSON, MediaType.TEXT_XML })
    @Produces({ MediaType.APPLICATION_JSON, MediaType.TEXT_XML })
    public Response createCourse(Name req) {
        log.debug("CourseResource: createCourse()");

        final String name = req.getName();
        if ((name == null) || name.isEmpty()) {
            return Response.status(Status.BAD_REQUEST).entity("'name' is required'").build();
        }

        Response response = null;

        try {
            Course course = null;

            if (req.getTestUuid() != null) {
                TestRun testRun = testRunService.findTestRunByUuid(req.getTestUuid());
                if (testRun != null) {
                    course = service.createCourseForTesting(name, testRun);
                } else {
                    response = Response.status(Status.BAD_REQUEST).entity("unknown test UUID").build();
                }
            } else {
                course = service.createCourse(name);
            }
            if (course == null) {
                response = Response.status(Status.INTERNAL_SERVER_ERROR).build();
            } else {
                response = Response.created(URI.create(course.getUuid())).entity(scrubCourse(course)).build();
            }
        } catch (Exception e) {
            if (!(e instanceof UnitTestException)) {
                log.info("unhandled exception", e);
            }
            response = Response.status(Status.INTERNAL_SERVER_ERROR).build();
        }

        return response;
    }

    ....
}

REST Webservice Client

Finally the REST server must add one additional method. The client does not support getting a list of all test objects yet.

public interface CourseRestClient {

    /**
     * Create specific course for testing.
     * 
     * @param name
     * @param testRun
     */
    Course createCourseForTesting(String name, TestRun testRun);

    ....
}

and

public class CourseRestClientImpl extends AbstractRestClientImpl implements CourseRestClient {

    /**
     * Create JSON string.
     * 
     * @param name
     * @return
     */
    String createJson(final String name, final TestRun testRun) {
        return String.format("{ \"name\": \"%s\", \"testUuid\": \"%s\" }", name, testRun.getTestUuid());
    }

    /**
     * @see com.invariantproperties.sandbox.student.webservice.client.CourseRestClient#createCourse(java.lang.String)
     */
    @Override
    public Course createCourseForTesting(final String name, final TestRun testRun) {
        if (name == null || name.isEmpty()) {
            throw new IllegalArgumentException("'name' is required");
        }

        if (testRun == null || testRun.getTestUuid() == null || testRun.getTestUuid().isEmpty()) {
            throw new IllegalArgumentException("'testRun' is required");
        }

        return createObject(createJson(name, testRun));
    }

    ....
}

Source Code

The source code is available at http://code.google.com/p/invariant-properties-blog/source/browse/student.

Clarification

I didn’t think it was possible to have a @OneToMany to TestablePersistentObject in TestRun but the integration tests using H2 succeeded. Unfortunately it’s causing problems as I bring up the fully integrated webservice with a PostgreSQL database. I’m leaving the code in place above since it’s always possible to have a list of Classrooms, a list of Courses, etc., even if we can’t have a generic collection. However the code is being removed from the version under source control.

Correction

The interface method should be findCourseByTestRun_Uuid(), not findCourseByTestRun(). Another approach is using JPA criteria queries – see Project Student: JPA Criteria Queries.
 

Related Whitepaper:

Functional Programming in Java: Harnessing the Power of Java 8 Lambda Expressions

Get ready to program in a whole new way!

Functional Programming in Java will help you quickly get on top of the new, essential Java 8 language features and the functional style that will change and improve your code. This short, targeted book will help you make the paradigm shift from the old imperative way to a less error-prone, more elegant, and concise coding style that’s also a breeze to parallelize. You’ll explore the syntax and semantics of lambda expressions, method and constructor references, and functional interfaces. You’ll design and write applications better using the new standards in Java 8 and the JDK.

Get it Now!  

Leave a Reply


3 × = twelve



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

15,153 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books