Home » Java » Core Java » Java: Incrementally read/stream a CSV file

About Mark Needham

Java: Incrementally read/stream a CSV file

I’ve been doing some work which involves reading in CSV files, for which I’ve been using OpenCSV, and my initial approach was to read through the file line by line, parse the contents and save it into a list of maps.

This works when the contents of the file fit into memory but is problematic for larger files where I needed to stream the file and process each line individually rather than all of them after the file was loaded.

I initially wrote a variation on totallylazy’s Strings#lines to do this and while I was able to stream the file I made a mistake somewhere which meant the number of maps on the heap was always increasing.

After spending a few hours trying to fix this Michael suggested that it’d be easier to use an iterator instead and I ended up with the following code:

public class ParseCSVFile {
    public static void main(String[] args) throws IOException
    {
        final CSVReader csvReader = new CSVReader( new BufferedReader( new FileReader( "/path/to/file.csv" ) ), '\t' );
        final String[] fields = csvReader.readNext();

        Iterator<Map<String, Object>>() lazilyLoadedFile = return new Iterator<Map<String, Object>>()
        {
            String[] data = csvReader.readNext();

            @Override
            public boolean hasNext()
            {
                return data != null;
            }

            @Override
            public Map<String, Object> next()
            {
                final Map<String, Object> properties = new HashMap<String, Object>();
                for ( int i = 0; i < data.length; i++ )
                {
                    properties.put(fields[i], data[i]);
                }

                try
                {
                    data = csvReader.readNext();
                }
                catch ( IOException e )
                {
                    data = null;
                }

                return properties;
            }

            @Override
            public void remove()
            {
                throw new UnsupportedOperationException();
            }
        };
    }	
}

Although this code works it’s not the most readable function I’ve ever written so any suggestions on how to do this in a cleaner way are welcome.
 

Reference: Java: Incrementally read/stream a CSV file from our JCG partner Mark Needham at the Mark Needham Blog blog.

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

 

Leave a Reply

Your email address will not be published. Required fields are marked *

*


Want to take your Java Skills to the next level?
Grab our programming books for FREE!
  • Save time by leveraging our field-tested solutions to common problems.
  • The books cover a wide range of topics, from JPA and JUnit, to JMeter and Android.
  • Each book comes as a standalone guide (with source code provided), so that you use it as reference.
Last Step ...

Where should we send the free eBooks?

Good Work!
To download the books, please verify your email address by following the instructions found on the email we just sent you.