Reading and writing files in Golang

Golang is a fantastic language. Its low-level nature and its simplicity make it an absolute powerhouse. There is no limit to the things you can do with Golang. IO is a very important topic when working with different sorts of applications. Working with files is the most basic IO of all. In Golang, the concept of readers and writers allow you to hanlde any sort of IO in a consistent fashion. In today’s post I will focus on handeling files in Golang. I will show you how to:

1. Write a simple string to a file.
2. Read the content of a file.
3. Write multiple lines to a file.
4. Read a file line by line.
5. Work with JSON data.
6. Handle concurrent writing from multiple goroutines.

By the end of this post, you will be confident on how to deal with any sorts of files in Golang. Let’s do it! 🚀

Writing a Simple String to a File

The first thing we’ll learn is how to write a simple string to a file. This will include creating the file using os.Create() method. You can read more about it on Golang Create() Docs.

This method will create a file on disk and return it to us. To write things to it. In this basic example, we can use the file.WriteString() method to save strings into the file. You can read more about WriteStringin Golang WriteString() docs

This method will return to us number of bytes written, and and error that we will handle like below.

func saveStringToFile() {
  	newFile, fileCreateErr := os.Create("string.txt")
  	if fileCreateErr != nil {
  		fmt.Println("Issue happened crearting file")
  		return
  	}
  	defer newFile.Close()

  	bytes, fileWritingErr := newFile.WriteString("New content into file")
  	if fileWritingErr != nil {
  		fmt.Println("Issue happened writing to file")
  		return
  	}
  	fmt.Printf("%d bytes written to file", bytes)
}

func main() {
    saveStringToFile()
}

In this code, we:

1. Create a file named string.txt.

• Notice the return of the function is of type os.File
• Notice we need to remember to close the file before exiting - a simple defer keyword does the trick here.

2. Write a string to this file.

• The WriteString method returns number of bytes written and a write error.

3. Print the number of bytes written.

Reading the Content of a File

Next, let’s see how to read the content of a file that we just wrote to. To do that, we have to open the file and load the content and close the file afterwards. There are ways to do each of those steps separatly in Golang. However, there is a simple method on the os module that will do all the steps for you in one go. You can read more about os.ReadFile Golang Docs. This method will return the file content in []bytes and a read error.

func openFileAndReadContent() {
  	fileContent, fileReadErr := os.ReadFile("string.txt")
  	if fileReadErr != nil {
  		fmt.Println("Issue happened opening file")
  		return
  	}
  	fmt.Println(string(fileContent))
}

func main() {
    openFileAndReadContent()
}

This function reads all the content of string.txt and prints it. The os.ReadFile function makes it straightforward by handling file opening, reading, and closing. Notice also the use of string() to convert the []bytes content of the file into a string that we can print.

Writing Multiple Lines to a File

So far we have only dealt with writing and reading simple strings to txt files. What happens when you need to write/read something more elaborate? 🤔

Let’s imagine you want to log the activities on your application in a log file. Each entry of the log file will have to be on a separate line. In Golang, it’s straightforward to handle writing to a file in lines using fmt.Fprintln method. You can read more about it in Fprintln Golang Docs. This method is very similar to WriteString but it will always include a \n at the end of each message it writes. Let’s have a look at how we can use it.

func saveLinesToFile() {
  	newFile, newFileErr := os.Create("lines.txt")
  	if newFileErr != nil {
  		fmt.Println("Issue happened creating file")
  		return
  	}
  	defer newFile.Close()

  	lines := []string{"Line 1", "Line 2", "Line 3"}
  	totalBytes := 0

  	for _, line := range lines {
  		numBytes, fileWriteErr := fmt.Fprintln(newFile, line)
  		if fileWriteErr != nil {
  			fmt.Println("Issue happened writing to file")
  			return
  		}
  		totalBytes += numBytes
  	}
  	fmt.Printf("%d bytes written", totalBytes)
}

func main() {
    saveLinesToFile()
}

Notice our use of return value of Fprintln to accumlate the number of bytes written in that file across all lines. By running this code you should be able to see the total number of bytes written into the file.

Reading a File Line by Line

Nice! So now you know how to save lines into a txt file. How about reading those lines back? 💡

Consider also if the file we want to open is very large and we don’t want to load all the content of that file in one go in our application. Golang provides a whole module bufio to handle reading/writing to IOs in a buffered fashion. This is a big topic we can discuss in another post.

I want to introduce to the concept of Scanner. Scanners are very helpful because instead of loading all the content of the file in one go, they allows us to “SCAN” that content and stop using a splitting function. Lucky for us the default NewScanner uses the line break as a simple splitting function. That means everytime we call scanner.Scan() we will advance the scanner to the next instance of the “line break” and store the value in scanner itself. We can easily grab that value by calling scanner.Text(). You can read more about NewScanner in Golang bufio NewScanner docs. Let’s have a look how we can read our file line by line.

func readFileLineByLine() {
  	fileContent, fileOpenErr := os.Open("lines.txt")
  	if fileOpenErr != nil {
  		fmt.Println("Issue happened opening file")
  		return
  	}
  	defer fileContent.Close()

  	fileScanner := bufio.NewScanner(fileContent)
  	for fileScanner.Scan() {
  		line := fileScanner.Text()
  		fmt.Println(line)
  		fmt.Println("Read one line ---- ")
  	}
    // Notice the error handling here.
  	if err := fileScanner.Err(); err != nil {
  		fmt.Println("Issue happened opening file")
  		return
  	}
}

func main() {
    readFileLineByLine()
}

Using bufio.Scanner, we read and print each line from lines.txt. Running this code will show you the lines listed inside the file followed by Read One line --- print in the terminal.

Working with JSON Data

Now you know how to deal with different sorts of string txt files. But Golang has powerful support for different types of file. Today I’ll show you how to handle JSON files, but the same concept applies to yaml and other types of files.

The idea is very simple:

1. You can make any type of data shape as struct in Golang
2. You encode your data into json encoding using json.Marshal() method - Read more here
3. You write that data, now a []bytes into a .json file

Let’s have a look at the code:

type User struct {
    FirstName string 
    LastName  string 
}

func createAndSaveJSON() {
  	user := User{
  		FirstName: "jack",
  		LastName:  "doe",
  	}
  	bytes, marshalErr := json.Marshal(user)
  	if marshalErr != nil {
  		fmt.Print("can't marshall your data")
  	}

  	newFile, newFileErr := os.Create("user.json")
  	if newFileErr != nil {
  		fmt.Println("Issue happened creating file")
  		return
  	}

  	newFile.Write(bytes)
}

func main() {
    createAndSaveJSON()
}

Running this, will save the value of {"FirstName":"jack","LastName":"doe"} into the file. Notice the use of LastName & FirstName from the struct as json property names in the file. This is the automatic behaviour in Golang. You can change that by using Golang templates:

type User struct {
	FirstName string `json:"firstName"`
	LastName  string `json:"lastName"`
} 

Using this struct instead, the code will save the following json into the file: {"firstName":"jack","lastName":"doe"}. Golang templates are very powerful and will be the topic of a future post.

Handling Concurrent Writing with Goroutines

Finally, let’s tackle a more advanced topic: handling concurrent writing from multiple goroutines. This is essential for applications like logging from multiple services. This will show you how to handle writing into the same file concurrently; a pattern that is essential and also a good practice to understand Golang Concurrency patterns.

To do this we are going to make 100 go routiens - call them producers - that will send random integers in a channel. Then we will make another Go routine - nicknamed conductor - to watch over the producers and inform the main functions when they are all done. Once they are all done, the condutor will send a termination signal in a separate channel - called done. Finally in our main Go routine, func main, we will listen and action all those events using select stamtement.

To understand this properly, I have made the following diagram:

We will need a couple of things to make this work:

1. Two channels for communication

• data channel to get data from producers
• done channel to get signal from coductor

2. WaitGroup to “wait” for all go routines to be done with their work.
3. Select statemnt to action differnt values in differnt channels

Let’s have a look at the code

func main() {
  	data := make(chan int)
  	done := make(chan bool)
  	var wg sync.WaitGroup

  	newFile, newFileErr := os.Create("log.txt")
  	if newFileErr != nil {
  		fmt.Println("Issue happened creating file")
  		return
  	}
  	defer newFile.Close()

    // making producers
  	for i := 0; i < 100; i++ {
  		wg.Add(1)
  		go func() {
        // send random int - simulating some work
  			data <- rand.Intn(200)
        // calling done after doing some work
  			wg.Done()
  		}()
  	}

    // conductor routine
  	go func() {
  		wg.Wait()
  		done <- true
  	}()

    // listen & action in main go routine
  	for {
  		select {
  		case value := <-data:
        // write values coming from data channel into file
  			_, fileWriteErr := fmt.Fprintln(newFile, value)
  			if fileWriteErr != nil {
          // handeling errors using done channel to prevent furthur writes
  				done <- false
  			}
  		case sign := <-done:
  			if sign {
          // all writes are done -> success 
  				fmt.Println("All done")
  				return
  			} else {
          // one write didn't wor -> failure
  				fmt.Println("something went wrong")
  				return
  			}
  		}
  	}
}

This code:

1. Creates 100 goroutines to simulate multiple producers.
2. Uses a sync.WaitGroup to wait for all goroutines to finish.
3. Writes data to log.txt and handles errors gracefully.

After running this code, you will see a success message. If you cat log.txt you will find a 100 lines of random numbers.

Conclusion

In this tutorial, we’ve covered the essentials of handling files in Go:

• Writing and reading simple strings.
• Handling multiple lines.
• Working with JSON data.
• Managing concurrent writes from multiple goroutines.

If you are keen in getting your hands on the full final code, you can do so from the form below!