Best practices and strategies for logging in Go

When network engineers think of programming languages, they likely consider Perl or Python before Go. Yes, Perl is proven, and Python is trendy. But Go presents a compelling option as a powerful programming language.

Software is increasingly moving to cloud and serverless environments, and that code needs to communicate with networks, databases, web clients and cloud instances. Network teams are left with mounds of data, which, if analyzed properly, can provide helpful insights about operations. This is a sweet spot for Go, according to Adam Woodbeck, senior software engineer at Barracuda Networks and author of Network Programming with Go from No Starch Press. Google created Go to solve a problem it faced with writing network software at scale, so the language is well suited for filtering large amounts of data into digestible information.

One feature that sets Go apart from a language like Python is its ability to do multiple things at once, Woodbeck said. This concurrency enables engineers to use the full power of their servers and systems. Go is also a relatively simple language to learn, he added, enabling teams to ramp up their skills and implement changes quickly.

In his book, Woodbeck provides examples of how network and software engineers can implement Go for logging and metrics, traffic routing and application-level programming, among other strategies.

Editor's note: The following interview was edited for length and clarity.

What has been your favorite way to use Go?

Adam Woodbeck

Adam Woodbeck: I like the concurrency aspect of Go: being able to do a lot of things at one time. It tends to be one of the more difficult aspects of Go. It's easy to make a mistake and end up with a race condition or deadlock. But, if you stick to using channels properly and use the tools Go gives you, then you can avoid those problematic areas and get a lot of performance out of Go.

I like using it for services and server-side hardware or software that interacts with a lot of clients at one time -- where there are a lot of messages being passed or a lot of processing that needs to happen to these messages as they travel through some data pipeline.

I started with Go at a credit union. We were merging with another credit union, and the credit union we were merging with had all their data on mainframes -- it was in these old COBOL copybooks. It was binary blobs of data, and these copybooks would describe this binary data.

A colleague and I decided to learn Go programming language. It allowed us to do low-level details and use code generation against these copybooks to generate Go code. We could slurp up the binary data and put it into a database where we could do something meaningful with it. It was initially a prototype we were exploring to see if it would be feasible during the merger and save us hundreds of thousands of dollars. It was a wild success and proved to me the power of Go and how easy it is to learn and get ramped up on the language and be productive.

What's the biggest challenge with learning Go?

Click here to learn more
about this book.

Woodbeck: I think the biggest challenge people run into is learning how to write Go and not fight the language. A lot of people come to Go from another language. They come from Perl or Python or PHP or Java. And they're trying to write Perl or Java in Go -- they're not doing things the idiomatic Go way.

I think everyone has to go through this learning process where they're fighting the language a little bit, doing things that are familiar to them and their old language. Once they get it and that switch flips, they start to leverage the full power of Go. I had that same flip-switching experience, and I've talked to countless people who have also experienced that. It seems to be a rather common occurrence.

How do you implement logging and metrics in Go?

Woodbeck: When I started writing software, I logged everything I thought I might need. I quickly learned that isn't a good technique because I'm just creating a haystack. And the needle I need is somewhere in there, and I'm not likely to find it.

My approach has evolved. I still add a lot of logging, but I leverage logging levels. I'll log things I know I may need if I'm encountering a problem -- debug logging, for instance. But I try to reserve my more verbose logging, or things that are going to be logged by default, to actionable items -- something that needs my attention and will blow up in my face if I don't address it soon. But that's it. I only want to be bothered in production with stuff that I need to address.

It's possible to be overwhelmed by even those actionable logs. Some contemporary loggers in Go have the concept of sampling, where you define in your code: 'I want to be notified of this, but I want to be notified every fifth occurrence.' So, you're not drowning in log entries that are coming from tens of thousands of clients at one time.

Metrics, on the other hand, instrument everything. That's typically my approach. There's usually very little overhead to instrumenting your code. Even in Go, if you're using something like Prometheus, it uses atomic counters. So, they're lightweight, and you won't notice any difference in performance. In the background, your metrics are accumulating, and then Prometheus will grab your metrics.

Metrics should show you the state of your program and service. I also like to have dashboards that give me insight into what's going on at any time. If all your connections suddenly disappear at a particular rate, you may want to know that, so you should receive an alert.

It can be just as bad to log too much information as it is to log not enough.

How can teams deal with alert fatigue and figure out what to log?

Woodbeck: It can be just as bad to log too much information as it is to log not enough. If you're always generating alerts for something that may not be an issue, then you'll have a team that's not going to pay attention to those alerts when they really need to. Trying to find that sweet spot is probably more of an art than a science. It's always a target we aim for, and I don't think anyone ever gets it right the first time. It's something you have to be vigilant about and dial in.

For logging, I found structured logging is beneficial for me because I can add context to the logs. If I'm generating a log entry, I need to know where this occurred, what was the current state when this happened -- all that stuff may be relevant to figure out what the problem was at the time and resolve the issue. Structured logging allows us to do that easily, and Go has good support for loggers that support structured logging.

What are some common mistakes people make with logging in Go?

Woodbeck: Not logging the right things is probably the big thing. When you're writing your code, you have to think about what can go wrong. Let's say you're writing something that's manipulating files on the system. What happens if your drive fills up? Under normal operation, that's not something you're going to experience, but it could be.

Or, if you're writing network software, and someone unplugs your network cable. Now, all your connections are timing out. These are things that shouldn't happen, but they do happen. Some network timeout errors you could probably expect to be normal. But, if there are enough sequential timeout errors, then maybe that needs your attention. That may be something you want to log -- after [the system] sees five timeout errors in 30 seconds, I want to know about that. But, if I see a couple here and there, it's not a huge deal to me because I know my network connection is wireless and it's a little spotty. Not a big deal.

Part of that is experience -- learning what you need to log. You may encounter a problem you don't have adequate logs for, and you realize, 'OK, I needed this information.' Now, you go back and add it. The next time you're writing software and encounter the same situation, you'll know, 'Oh yeah, I need to make sure I log this scenario, or I need to add more log visibility here.'

What other advice would you give to readers?

Woodbeck: My advice would be to always stay curious. Always experiment. Write an HTTP client from scratch. Don't use the library; try to write it on your own. Play around with the language. The standard library is very rich, and you can even dive into the source code and see how things work. Stay away from writing your own encryption, in my opinion, but write things like network clients. Interact with other servers, or write servers and have clients interact with them. I think you can learn a lot just by experimenting and playing around.