Enterprise Java

Spring Integration – Polling file creation and modification

1 Introduction

File support is another of Spring Integration’s endpoints to communicate with external systems. In this case, it provides several components to read, write and transform files. During this post, we are going to write an application which monitors a directory in order to read all files in there. In concrete it does the following:

  • When the application starts, it reads all files present in the directory.
  • The application will then keep an eye on the directory to detect new files and existing files which have been modified.

The source code can be found in Github.

2 Configuration

The application is built with Spring Boot, since it eases configuration significantly. To create the initial infrastructure of the application, you can go to https://start.spring.io/, select the Integration module and generate the project. Then you can open the zip file in your favourite IDE.

I added a couple of dependencies to the pom.xml like commons.io or Spring Integration Java DSL. My pom.xml file looks like as follows:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>xpadro.spring.integration</groupId>
  <artifactId>file-read-directory</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>file-read-directory</name>
  <description>Demo project for Spring Boot</description>

  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.3.5.RELEASE</version>
    <relativePath/> <!-- lookup parent from repository -->
  </parent>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <java.version>1.8</java.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-integration</artifactId>
    </dependency>

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
    </dependency>

    <!-- Spring Integration - Java DSL -->
    <dependency>
      <groupId>org.springframework.integration</groupId>
      <artifactId>spring-integration-java-dsl</artifactId>
      <version>1.0.0.RELEASE</version>
    </dependency>

    <dependency>
      <groupId>commons-io</groupId>
      <artifactId>commons-io</artifactId>
      <version>2.5</version>
    </dependency>

  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
      </plugin>
    </plugins>
  </build>

</project>

The starting point is FileReadDirectoryApplication:

@SpringBootApplication
public class FileReadDirectoryApplication {

    public static void main(String[] args) throws IOException, InterruptedException {
		SpringApplication.run(FileReadDirectoryApplication.class, args);
	}
}

Starting from here, we are going to add the Spring Integration components for reading from a specific folder of the filesystem.

3 Adding the adapter

In order to read from the file system, we need an inbound channel adapter. The adapter is a file reading message source, which is responsible of polling the file system directory for files and create a message from each file it finds.

@Bean
@InboundChannelAdapter(value = "fileInputChannel", poller = @Poller(fixedDelay = "1000"))
public MessageSource<File> fileReadingMessageSource() {
    CompositeFileListFilter<File> filters = new CompositeFileListFilter<>();
    filters.addFilter(new SimplePatternFileListFilter("*.txt"));
    filters.addFilter(new LastModifiedFileFilter());
    
    FileReadingMessageSource source = new FileReadingMessageSource();
    source.setAutoCreateDirectory(true);
    source.setDirectory(new File(DIRECTORY));
    source.setFilter(filters);
    
    return source;
}

We can prevent some types of files from being polled by setting a list of filters to the message source. For this example two filters have been included:

  • SimplePatternFileListFilter: Filter provided by Spring. Only files with the specified extension will be polled. In this case, only text files will be accepted.
  • LastModifiedFileFilter: Custom filter. This filter keeps track of already polled files and will filter out files not modified since the last time it was tracked.

4 Processing the files

For each polled file, we will transform its content to String before passing it to the processor. For this purpose, Spring already provides a component:

@Bean
public FileToStringTransformer fileToStringTransformer() {
    return new FileToStringTransformer();
}

Hence, instead of receiving a Message<File>, the processor will receive a Message<String>. The file processor is our custom component which will do something as advanced as printing the file content:

public class FileProcessor {
    private static final String HEADER_FILE_NAME = "file_name";
    private static final String MSG = "%s received. Content: %s";
    
    public void process(Message<String> msg) {
        String fileName = (String) msg.getHeaders().get(HEADER_FILE_NAME);
        String content = msg.getPayload();
        
        System.out.println(String.format(MSG, fileName, content));
    }
}

5 Building the flow

Now that we have all the required components in place, let’s build the flow. We are using Spring Integration Java DSL, since it makes the flow more readable:

@Bean
public IntegrationFlow processFileFlow() {
    return IntegrationFlows
        .from("fileInputChannel")
        .transform(fileToStringTransformer())
        .handle("fileProcessor", "process").get();
    }
    
    @Bean
    public MessageChannel fileInputChannel() {
        return new DirectChannel();
    }

6 Running the application

In my directory, I already have a file called ‘previousFile.txt’. After starting the application, we will create two files and modify one of them.

public static void main(String[] args) throws IOException, InterruptedException {
    SpringApplication.run(FileReadDirectoryApplication.class, args);
    createFiles();
}

private static void createFiles() throws IOException, InterruptedException {
    createFile("file1.txt", "content");
    createFile("file2.txt", "another file");
    appendFile("file1.txt", " modified");
}

If we run the application, we should see the following print statements:

previousFile.txt received. Content: previous content
file1.txt received. Content: content
file2.txt received. Content: another file
file1.txt received. Content: content modified

7 Conclusion

This example shows how simple it is to read files from a directory using Spring Integration, obviously with the help of Spring Boot to simplify the configuration. Depending on your needs, you can add your own custom filters to the message source, or use another one of the provided by Spring, like the RegexPatternFileListFilter. You can check for other implementations here.

If you found this post useful, please share it or star my repository :)

I’m publishing my new posts on Google plus and Twitter. Follow me if you want to be updated with new content.

Xavier Padro

Xavier is a software developer working in a consulting firm based in Barcelona. He is specialized in web application development with experience in both frontend and backend. He is interested in everything related to Java and the Spring framework.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

3 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Priyanka Goel
Priyanka Goel
4 years ago

Hi

I have requirement to upload compressed file.when I am polling file using above example it is not able to give proper content of compressed file ,help to know how can I get only file object after polling it from a location

Thanks
Priyanka

Shahzad Azam
Shahzad Azam
4 years ago
Reply to  Priyanka Goel

Inorder to only get File object don’t use the .transform(fileToStringTransformer())
Use it as below:
@Bean
public

IntegrationFlow processFileFlow() {
    

return

IntegrationFlows
        

.from(

"fileInputChannel"

)
        

.handle(

"fileProcessor"

,

"process"

).get();
    

}

And modify your fileprocessor to accept File as below:

public

class

FileProcessor {
    

private

static

final

String HEADER_FILE_NAME =

"file_name"

;
    

private

static

final

String MSG =

"%s received. Content: %s"

;
     
    

public

void

process(Message<File> msg) {
        

String fileName = (String) msg.getHeaders().get(HEADER_FILE_NAME);
        

File content = msg.getPayload();
         
        

System.out.println(String.format(MSG, fileName));
    

}
}

Paramesh
Paramesh
4 years ago

Is there a way to register IntegrationFlow for multi-tenancy. Every tenant has it’s own directory path with the same pattern like “/mount/batches/{tenantId}/pf”. Assuming we have few tenants such as INFY, TCS, WIPRO etc, every tenant will have it’s own directory path as follows:

INFY –> /mount/batches/INFY/pf
TCS –> /mount/batches/TCS/pf
WIPRO –> /mount/batches/WIPRO/pf

Is there a way to configure multiple folders with InboundFlow?

Back to top button