Home » Software Development » GoLang: Reader-Writer implemented via bufio library to copy file

About Venkatt Guhesan

Venkatt Guhesan
I work as an Enterprise UI Architect for DataDirect Networks. I have been developing DirectMon, an Enterprise Monitoring and Management Solution for all of the DDN products. Before that I worked at DrFirst engineering an e-prescribing platform for controlled substances

GoLang: Reader-Writer implemented via bufio library to copy file

How to copy a file in small portions into a target file?

Let’s talk about the why?

When you copy large files, you want to ensure that you are copying it piece by piece to avoid out-of-memory errors. As a good practice, it’s best to consider optimizations during development to avoid edge cases that might creep up in production.

Imagine a scenario where your copy file has been working flawlessly until one fine day when a 5GB file is thrown at it to be copied from one location to another. If memory is a problem, then it will throw a out-of-memory and crash.

Now the complete example

package main

import (
	"bufio"
	"io"
	"os"
)

func main() {
	// Objective: Copy contents of FileA to FileB
	// Perform this step in chuncks instead of using the
	// built-in copy functionality.
	// Why?
	// Illustrates how to use bufio library with a reader and writer

	// Source
	inFile, err := os.Open("./source-file.txt")
	if err != nil {
		panic(err)
	}
	defer inFile.Close()

	// Target
	outFile, err := os.Create("./target-file.txt")
	if err != nil {
		panic(err)
	}
	defer outFile.Close()

	// Read a bit then write a bit -- repeat until EOF
	reader := bufio.NewReader(inFile)
	writer := bufio.NewWriter(outFile)
	for {
		if slice, err := reader.ReadSlice('\n'); err == nil || err == io.EOF {
			writer.Write(slice)
			// IMPORTANT: Invoke flush()
			writer.Flush()
			if err == io.EOF {
				break
			}
		}
	}

}

Deeper analysis

  • In lines 17-20, we Open() the source and panic if we have a file error.
  • In line 21, we defer closing the file when we exit that function. You got to love Go for this one capability alone! :-)
  • In lines 24-28, we repeat the same process for the outgoing file.
  • In lines 31,32, we create the NewReader and NewWriter.
  • In lines 33-42, we create loop until we reach the End-Of-File marker and in line 35 we write the output out to the file.
  • In lines 35, It is important to always call writer.Flush() to commit the writes to file. Failure to do this will end up with an empty file.

Ready to try this on your local system?

Feel free to check out the project located here:

GoLang-CopyFile

All source code and steps to compile have been included at the repository above. Ping me if you run into any issues.

Cheers!

Published on Java Code Geeks with permission by Venkatt Guhesan, partner at our JCG program. See the original article here: GoLang: Reader-Writer implemented via bufio library to copy file

Opinions expressed by Java Code Geeks contributors are their own.

(0 rating, 0 votes)
You need to be a registered member to rate this.
Start the discussion Views Tweet it!
Do you want to know how to develop your skillset to become a Java Rockstar?
Subscribe to our newsletter to start Rocking right now!
To get you started we give you our best selling eBooks for FREE!
1. JPA Mini Book
2. JVM Troubleshooting Guide
3. JUnit Tutorial for Unit Testing
4. Java Annotations Tutorial
5. Java Interview Questions
6. Spring Interview Questions
7. Android UI Design
and many more ....
I agree to the Terms and Privacy Policy
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments