Java: Incrementally read/stream a CSV file

I’ve been doing some work which involves reading in CSV files, for which I’ve been using OpenCSV, and my initial approach was to read through the file line by line, parse the contents and save it into a list of maps.

This works when the contents of the file fit into memory but is problematic for larger files where I needed to stream the file and process each line individually rather than all of them after the file was loaded.

I initially wrote a variation on totallylazy’s Strings#lines to do this and while I was able to stream the file I made a mistake somewhere which meant the number of maps on the heap was always increasing.

After spending a few hours trying to fix this Michael suggested that it’d be easier to use an iterator instead and I ended up with the following code:

public class ParseCSVFile {
    public static void main(String[] args) throws IOException
        final CSVReader csvReader = new CSVReader( new BufferedReader( new FileReader( "/path/to/file.csv" ) ), '\t' );
        final String[] fields = csvReader.readNext();

        Iterator<Map<String, Object>>() lazilyLoadedFile = return new Iterator<Map<String, Object>>()
            String[] data = csvReader.readNext();

            public boolean hasNext()
                return data != null;

            public Map<String, Object> next()
                final Map<String, Object> properties = new HashMap<String, Object>();
                for ( int i = 0; i < data.length; i++ )
                    properties.put(fields[i], data[i]);

                    data = csvReader.readNext();
                catch ( IOException e )
                    data = null;

                return properties;

            public void remove()
                throw new UnsupportedOperationException();

Although this code works it’s not the most readable function I’ve ever written so any suggestions on how to do this in a cleaner way are welcome.

Reference: Java: Incrementally read/stream a CSV file from our JCG partner Mark Needham at the Mark Needham Blog blog.
Related Whitepaper:

Bulletproof Java Code: A Practical Strategy for Developing Functional, Reliable, and Secure Java Code

Use Java? If you do, you know that Java software can be used to drive application logic of Web services or Web applications. Perhaps you use it for desktop applications? Or, embedded devices? Whatever your use of Java code, functional errors are the enemy!

To combat this enemy, your team might already perform functional testing. Even so, you're taking significant risks if you have not yet implemented a comprehensive team-wide quality management strategy. Such a strategy alleviates reliability, security, and performance problems to ensure that your code is free of functionality errors.Read this article to learn about this simple four-step strategy that is proven to make Java code more reliable, more secure, and easier to maintain.

Get it Now!  

Leave a Reply

five × = 20

Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below: