Project Student: Sharding Integration Test Data

This is part of Project Student. Other posts are Webservice Client With Jersey, Webservice Server with Jersey, Business Layer and Persistence with Spring Data.

All of the integration tests until now have used an in-memory embedded database that did not retain information from run to run. This changes when we fully integrate the REST server with a “real” database server – leftover test data will pollute our development or test database. This can be a real headache once we have continuous integration that runs integration tests code is checked in.

One solution is to ‘shard’ our integration test data in a way that allows our tests to use the shared development database without polluting it or other tests. The easiest approach is to add a TestRun field to all of our objects. “Test” data will have a value that indicates the specific test run, “live” data will have a null value.

The exact timeline is

  1. create and persist a TestRun object
  2. create test objects with appropriate TestRun value
  3. perform the integration tests
  4. delete the test objects
  5. delete the TestRun object

Any entry in the TestRun table will either be 1) active integration tests or 2) failed integration tests that threw an unhandled exception (depending upon the transaction manager, of course). It’s important to note that we can also capture the database state after an unexpected exception is thrown even if the transaction manager performs a rollback – it’s a simple extension to the junit test runner.

Timestamp and user fields make it easy to delete stale test data according to its age (e.g., any test more than 7 days old) or the person who ran the test.

TestablePersistentObject abstract base class

This change starts at the persistence level so we should start there and work our way outwards.

We first extend our PersistentObject abstract base class with a test run value.

@MappedSuperclass
public abstract class TestablePersistentObject extends PersistentObject {
    private static final long serialVersionUID = 1L;
    private TestRun testRun;

    /**
     * Fetch testRun object. We use lazy fetching since we rarely care about the
     * contents of this object - we just want to ensure referential integrity to
     * an existing testRun object when persisting a TPO.
     * 
     * @return
     */
    @ManyToOne(fetch = FetchType.LAZY, optional = true)
    public TestRun getTestRun() {
        return testRun;
    }

    public void setTestRun(TestRun testRun) {
        this.testRun = testRun;
    }

    @Transient
    public boolean isTestData() {
        return testRun != null;
    }
}

TestRun class

The TestRun class contains identifying information about a single integration test run. It contains a name (by default the classname#methodname() of the surrounding integration test), the date and time of the test, and the name of the user running the test. It would be easy to capture additional information.

The list of test objects gives us two big wins. First, it makes it easy to capture the state of the database if needed (e.g., after an unexpected exception). Second, cascading deletions makes it easy to delete all test objects.

@XmlRootElement
@Entity
@Table(name = "test_run")
@AttributeOverride(name = "id", column = @Column(name = "test_run_pkey"))
public class TestRun extends PersistentObject {
    private static final long serialVersionUID = 1L;

    private String name;
    private Date testDate;
    private String user;
    private List<TestablePersistentObject> objects = Collections.emptyList();

    @Column(length = 80, unique = false, updatable = true)
    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    @Column(name = "test_date", nullable = false, updatable = false)
    @Temporal(TemporalType.TIMESTAMP)
    public Date getTestDate() {
        return testDate;
    }

    public void setTestDate(Date testDate) {
        this.testDate = testDate;
    }

    @Column(length = 40, unique = false, updatable = false)
    public String getUser() {
        return user;
    }

    public void setUser(String user) {
        this.user = user;
    }

    @OneToMany(cascade = CascadeType.ALL)
    public List<TestablePersistentObject> getObjects() {
        return objects;
    }

    public void setObjects(List<TestablePersistentObject> objects) {
        this.objects = objects;
    }

    /**
     * This is similar to standard prepersist method but we also set default
     * values for everything else.
     */
    @PrePersist
    public void prepersist() {
        if (getCreationDate() == null) {
            setCreationDate(new Date());
        }

        if (getTestDate() == null) {
            setTestDate(new Date());
        }

        if (getUuid() == null) {
            setUuid(UUID.randomUUID().toString());
        }

        if (getUser() == null) {
            setUser(System.getProperty("user.name"));
        }

        if (name == null) {
            setName("test run " + getUuid());
        }
    }
}

The TestRun class extends PersistentObject, not TestablePersistentObject, since our other integration tests will sufficiently exercise it.

Spring Data Repository

We must add one additional method to every Repository.

@Repository
public interface CourseRepository extends JpaRepository {
    List<Course> findCoursesByTestRun(TestRun testRun);

    ....
}

Service Interface

Likewise we must add two additional methods to every service.

public interface CourseService {
    List<Course> findAllCourses();

    Course findCourseById(Integer id);

    Course findCourseByUuid(String uuid);

    Course createCourse(String name);

    Course updateCourse(Course course, String name);

    void deleteCourse(String uuid);

    // new method for testing
    Course createCourseForTesting(String name, TestRun testRun);

    // new method for testing
    List<Course> findAllCoursesForTestRun(TestRun testRun);
}

I won’t show the TestRunRepository, the TestRunService interface, or the TestRunService implementation since they’re identical to what I’ve described in the last few blog entries.

Service Implementation

We have to make one small change to an existing Service implementation, plus add two new methods.

@Service
public class CourseServiceImpl implements CourseService {
    @Resource
    private TestRunService testRunService;

    /**
     * @see com.invariantproperties.sandbox.student.business.CourseService#
     *      findAllCourses()
     */
    @Transactional(readOnly = true)
    @Override
    public List<Course> findAllCourses() {
        List<Course> courses = null;

        try {
            courses = courseRepository.findCoursesByTestRun(null);
        } catch (DataAccessException e) {
            if (!(e instanceof UnitTestException)) {
                log.info("error loading list of courses: " + e.getMessage(), e);
            }
            throw new PersistenceException("unable to get list of courses.", e);
        }

        return courses;
    }

    /**
     * @see com.invariantproperties.sandbox.student.business.CourseService#
     *      findAllCoursesForTestRun(com.invariantproperties.sandbox.student.common.TestRun)
     */
    @Transactional(readOnly = true)
    @Override
    public List<Course> findAllCoursesForTestRun(TestRun testRun) {
        List<Course> courses = null;

        try {
            courses = courseRepository.findCoursesByTestRun(testRun);
        } catch (DataAccessException e) {
            if (!(e instanceof UnitTestException)) {
                log.info("error loading list of courses: " + e.getMessage(), e);
            }
            throw new PersistenceException("unable to get list of courses.", e);
        }

        return courses;
    }

    /**
     * @see com.invariantproperties.sandbox.student.business.CourseService#
     *      createCourseForTesting(java.lang.String,
     *      com.invariantproperties.sandbox.student.common.TestRun)
     */
    @Transactional
    @Override
    public Course createCourseForTesting(String name, TestRun testRun) {
        final Course course = new Course();
        course.setName(name);
        course.setTestUuid(testRun.getTestUuid());

        Course actual = null;
        try {
            actual = courseRepository.saveAndFlush(course);
        } catch (DataAccessException e) {
            if (!(e instanceof UnitTestException)) {
                log.info("internal error retrieving course: " + name, e);
            }
            throw new PersistenceException("unable to create course", e);
        }

        return actual;
    }
}

CourseServiceIntegrationTest

We make a few changes to our integration tests. We only have to change one test method since it’s the only one that actually creates a test object. The rest of the methods are queries that don’t require test data.

Note that we change the name value to ensure it’s unique. This is a way to work around uniqueness constraints, e.g., for email addresses.

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(classes = { BusinessApplicationContext.class, TestBusinessApplicationContext.class,
        TestPersistenceJpaConfig.class })
@Transactional
@TransactionConfiguration(defaultRollback = true)
public class CourseServiceIntegrationTest {

    @Resource
    private CourseService dao;

    @Resource
    private TestRunService testService;

    @Test
    public void testCourseLifecycle() throws Exception {
        final TestRun testRun = testService.createTestRun();

        final String name = "Calculus 101 : " + testRun.getUuid();

        final Course expected = new Course();
        expected.setName(name);

        assertNull(expected.getId());

        // create course
        Course actual = dao.createCourseForTesting(name, testRun);
        expected.setId(actual.getId());
        expected.setUuid(actual.getUuid());
        expected.setCreationDate(actual.getCreationDate());

        assertThat(expected, equalTo(actual));
        assertNotNull(actual.getUuid());
        assertNotNull(actual.getCreationDate());

        // get course by id
        actual = dao.findCourseById(expected.getId());
        assertThat(expected, equalTo(actual));

        // get course by uuid
        actual = dao.findCourseByUuid(expected.getUuid());
        assertThat(expected, equalTo(actual));

        // get all courses
        final List<Course> courses = dao.findCoursesByTestRun(testRun);
        assertTrue(courses.contains(actual));

        // update course
        expected.setName("Calculus 102 : " + testRun.getUuid());
        actual = dao.updateCourse(actual, expected.getName());
        assertThat(expected, equalTo(actual));

        // verify testRun.getObjects
        final List<TestablePersistentObject> objects = testRun.getObjects();
        assertTrue(objects.contains(actual));

        // delete Course
        dao.deleteCourse(expected.getUuid());
        try {
            dao.findCourseByUuid(expected.getUuid());
            fail("exception expected");
        } catch (ObjectNotFoundException e) {
            // expected
        }

        testService.deleteTestRun(testRun.getUuid());
    }

    ....
}

We could use @Before and @After to transparently wrap all test methods but many tests don’t require test data and many tests that do require test data require unique test data, e.g., for email addresses. In the latter case we fold in the Test UUID as above.

REST Webservice Server

The REST webservice requires adding a test uuid to the request classes and adding a bit of logic to properly handle it when creating an object.

The REST webservice does not support getting a list of all test objects. The “correct” approach will be creating a TestRun service and providing associated objects in response to a /get/{id} query.

@XmlRootElement
public class Name {
    private String name;
    private String testUuid;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getTestUuid() {
        return testUuid;
    }

    public void setTestUuid(String testUuid) {
        this.testUuid = testUuid;
    }
}

We can now check for the optional testUuid field and call the appropriate create method.

@Service
@Path("/course")
public class CourseResource extends AbstractResource {
    @Resource
    private CourseService service;

    @Resource
    private TestRunService testRunService;

    /**
     * Create a Course.
     * 
     * @param req
     * @return
     */
    @POST
    @Consumes({ MediaType.APPLICATION_JSON, MediaType.TEXT_XML })
    @Produces({ MediaType.APPLICATION_JSON, MediaType.TEXT_XML })
    public Response createCourse(Name req) {
        log.debug("CourseResource: createCourse()");

        final String name = req.getName();
        if ((name == null) || name.isEmpty()) {
            return Response.status(Status.BAD_REQUEST).entity("'name' is required'").build();
        }

        Response response = null;

        try {
            Course course = null;

            if (req.getTestUuid() != null) {
                TestRun testRun = testRunService.findTestRunByUuid(req.getTestUuid());
                if (testRun != null) {
                    course = service.createCourseForTesting(name, testRun);
                } else {
                    response = Response.status(Status.BAD_REQUEST).entity("unknown test UUID").build();
                }
            } else {
                course = service.createCourse(name);
            }
            if (course == null) {
                response = Response.status(Status.INTERNAL_SERVER_ERROR).build();
            } else {
                response = Response.created(URI.create(course.getUuid())).entity(scrubCourse(course)).build();
            }
        } catch (Exception e) {
            if (!(e instanceof UnitTestException)) {
                log.info("unhandled exception", e);
            }
            response = Response.status(Status.INTERNAL_SERVER_ERROR).build();
        }

        return response;
    }

    ....
}

REST Webservice Client

Finally the REST server must add one additional method. The client does not support getting a list of all test objects yet.

public interface CourseRestClient {

    /**
     * Create specific course for testing.
     * 
     * @param name
     * @param testRun
     */
    Course createCourseForTesting(String name, TestRun testRun);

    ....
}

and

public class CourseRestClientImpl extends AbstractRestClientImpl implements CourseRestClient {

    /**
     * Create JSON string.
     * 
     * @param name
     * @return
     */
    String createJson(final String name, final TestRun testRun) {
        return String.format("{ \"name\": \"%s\", \"testUuid\": \"%s\" }", name, testRun.getTestUuid());
    }

    /**
     * @see com.invariantproperties.sandbox.student.webservice.client.CourseRestClient#createCourse(java.lang.String)
     */
    @Override
    public Course createCourseForTesting(final String name, final TestRun testRun) {
        if (name == null || name.isEmpty()) {
            throw new IllegalArgumentException("'name' is required");
        }

        if (testRun == null || testRun.getTestUuid() == null || testRun.getTestUuid().isEmpty()) {
            throw new IllegalArgumentException("'testRun' is required");
        }

        return createObject(createJson(name, testRun));
    }

    ....
}

Source Code

The source code is available at http://code.google.com/p/invariant-properties-blog/source/browse/student.

Clarification

I didn’t think it was possible to have a @OneToMany to TestablePersistentObject in TestRun but the integration tests using H2 succeeded. Unfortunately it’s causing problems as I bring up the fully integrated webservice with a PostgreSQL database. I’m leaving the code in place above since it’s always possible to have a list of Classrooms, a list of Courses, etc., even if we can’t have a generic collection. However the code is being removed from the version under source control.

Correction

The interface method should be findCourseByTestRun_Uuid(), not findCourseByTestRun(). Another approach is using JPA criteria queries – see Project Student: JPA Criteria Queries.
 

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

JPA Mini Book

Learn how to leverage the power of JPA in order to create robust and flexible Java applications. With this Mini Book, you will get introduced to JPA and smoothly transition to more advanced concepts.

JVM Troubleshooting Guide

The Java virtual machine is really the foundation of any Java EE platform. Learn how to master it with this advanced guide!

Given email address is already subscribed, thank you!
Oops. Something went wrong. Please try again later.
Please provide a valid email address.
Thank you, your sign-up request was successful! Please check your e-mail inbox.
Please complete the CAPTCHA.
Please fill in the required fields.

Leave a Reply


8 − = four



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy | Contact
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below:
Close