50 Shades of Go

50 Shades of Go: Traps, Gotchas, and Common Mistakes for New Golang Devs

#

Traps, Gotchas, and Common Mistakes

#

• level: beginner

In most other languages that use braces you get to choose where you place them. Go is different. You can thank automatic semicolon injection (without lookahead) for this behavior. Yes, Go does have semicolons :-)

Fails:

package main

import "fmt"

func main()  
{ //error, can't have the opening brace on a separate line
    fmt.Println("hello there!")
} 

Compile Error:

/tmp/sandbox826898458/main.go:6: syntax error: unexpected semicolon or newline before {

Works:

package main

import "fmt"

func main() {  
    fmt.Println("works!")
} 

Unused Variables

#

• level: beginner

If you have an unused variable your code will fail to compile. There’s an exception though. You must use variables you declare inside functions, but it’s OK if you have unused global variables. It’s also OK to have unused function arguments.

If you assign a new value to the unused variable your code will still fail to compile. You need to use the variable value somehow to make the compiler happy.

Fails:

package main

var gvar int //not an error

func main() {  
    var one int   //error, unused variable
    two := 2      //error, unused variable
    var three int //error, even though it's assigned 3 on the next line
    three = 3

    func(unused string) {
        fmt.Println("Unused arg. No compile error")
    }("what?")
} 

Compile Errors:

/tmp/sandbox473116179/main.go:6: one declared and not used /tmp/sandbox473116179/main.go:7: two declared and not used /tmp/sandbox473116179/main.go:8: three declared and not used

Works:

package main

import "fmt"

func main() {  
    var one int
    _ = one

    two := 2 
    fmt.Println(two)

    var three int 
    three = 3
    one = three

    var four int
    four = four
} 

Another option is to comment out or remove the unused variables :-)

Unused Imports

#

• level: beginner

Your code will fail to compile if you import a package without using any of its exported functions, interfaces, structures, or variables.

If you really need the imported package you can use the blank identifier, _, as its package name to avoid this compilation failure. The blank identifier is used to import packages for their side effects.

Fails:

package main

import (  
    "fmt"
    "log"
    "time"
)

func main() {  
} 

Compile Errors:

/tmp/sandbox627475386/main.go:4: imported and not used: “fmt” /tmp/sandbox627475386/main.go:5: imported and not used: “log” /tmp/sandbox627475386/main.go:6: imported and not used: “time”

Works:

package main

import (  
    _ "fmt"
    "log"
    "time"
)

var _ = log.Println

func main() {  
    _ = time.Now
} 

Another option is to remove or comment out the unused imports :-) The goimports tool can help you with that.

Short Variable Declarations Can Be Used Only Inside Functions

#

• level: beginner

Fails:

package main

myvar := 1 //error

func main() {  
} 

Compile Error:

/tmp/sandbox265716165/main.go:3: non-declaration statement outside function body

Works:

package main

var myvar = 1

func main() {  
} 

Redeclaring Variables Using Short Variable Declarations

#

• level: beginner

You can’t redeclare a variable in a standalone statement, but it is allowed in multi-variable declarations where at least one new variable is also declared.

The redeclared variable has to be in the same block or you’ll end up with a shadowed variable.

Fails:

package main

func main() {  
    one := 0
    one := 1 //error
} 

Compile Error:

/tmp/sandbox706333626/main.go:5: no new variables on left side of :=

Works:

package main

func main() {  
    one := 0
    one, two := 1,2

    one,two = two,one
} 

Can’t Use Short Variable Declarations to Set Field Values

#

• level: beginner

Fails:

package main

import (  
  "fmt"
)

type info struct {  
  result int
}

func work() (int,error) {  
    return 13,nil  
  }

func main() {  
  var data info

  data.result, err := work() //error
  fmt.Printf("info: %+v\n",data)
} 

Compile Error:

prog.go:18: non-name data.result on left side of :=

Even though there’s a ticket to address this gotcha it’s unlikely to change because Rob Pike likes it “as is” :-)

Use temporary variables or predeclare all your variables and use the standard assignment operator.

Works:

package main

import (  
  "fmt"
)

type info struct {  
  result int
}

func work() (int,error) {  
    return 13,nil  
  }

func main() {  
  var data info

  var err error
  data.result, err = work() //ok
  if err != nil {
    fmt.Println(err)
    return
  }

  fmt.Printf("info: %+v\n",data) //prints: info: {result:13}
} 

Accidental Variable Shadowing

#

• level: beginner

The short variable declaration syntax is so convenient (especially for those coming from a dynamic language) that it’s easy to treat it like a regular assignment operation. If you make this mistake in a new code block there will be no compiler error, but your app will not do what you expect.

package main

import "fmt"

func main() {  
    x := 1
    fmt.Println(x)     //prints 1
    {
        fmt.Println(x) //prints 1
        x := 2
        fmt.Println(x) //prints 2
    }
    fmt.Println(x)     //prints 1 (bad if you need 2)
} 

This is a very common trap even for experienced Go developers. It’s easy to make and it could be hard to spot.

You can use the vet command to find some of these problems. By default, vet will not perform any shadowed variable checks. Make sure to use the -shadow flag: go tool vet -shadow your_file.go

Note that the vet command will not report all shadowed variables. Use go-nyet for more aggressive shadowed variable detection.

Can’t Use “nil” to Initialize a Variable Without an Explicit Type

#

• level: beginner

The “nil” identifier can be used as the “zero value” for interfaces, functions, pointers, maps, slices, and channels. If you don’t specify the variable type the compiler will fail to compile your code because it can’t guess the type.

Fails:

package main

func main() {  
    var x = nil //error

    _ = x
} 

Compile Error:

/tmp/sandbox188239583/main.go:4: use of untyped nil

Works:

package main

func main() {  
    var x interface{} = nil

    _ = x
} 

Using “nil” Slices and Maps

#

• level: beginner

It’s OK to add items to a “nil” slice, but doing the same with a map will produce a runtime panic.

Works:

package main

func main() {  
    var s []int
    s = append(s,1)
} 

Fails:

package main

func main() {  
    var m map[string]int
    m["one"] = 1 //error

} 

Map Capacity

#

• level: beginner

You can specify the map capacity when it’s created, but you can’t use the cap() function on maps.

Fails:

package main

func main() {  
    m := make(map[string]int,99)
    cap(m) //error
} 

Compile Error:

/tmp/sandbox326543983/main.go:5: invalid argument m (type map[string]int) for cap

Strings Can’t Be “nil”

#

• level: beginner

This is a gotcha for developers who are used to assigning “nil” identifiers to string variables.

Fails:

package main

func main() {  
    var x string = nil //error

    if x == nil { //error
        x = "default"
    }
} 

Compile Errors:

/tmp/sandbox630560459/main.go:4: cannot use nil as type string in assignment /tmp/sandbox630560459/main.go:6: invalid operation: x == nil (mismatched types string and nil)

Works:

package main

func main() {  
    var x string //defaults to "" (zero value)

    if x == "" {
        x = "default"
    }
} 

Array Function Arguments

#

• level: beginner

If you are a C or C++ developer arrays for you are pointers. When you pass arrays to functions the functions reference the same memory location, so they can update the original data. Arrays in Go are values, so when you pass arrays to functions the functions get a copy of the original array data. This can be a problem if you are trying to update the array data.

package main

import "fmt"

func main() {  
    x := [3]int{1,2,3}

    func(arr [3]int) {
        arr[0] = 7
        fmt.Println(arr) //prints [7 2 3]
    }(x)

    fmt.Println(x) //prints [1 2 3] (not ok if you need [7 2 3])
} 

If you need to update the original array data use array pointer types.

package main

import "fmt"

func main() {  
    x := [3]int{1,2,3}

    func(arr *[3]int) {
        (*arr)[0] = 7
        fmt.Println(arr) //prints &[7 2 3]
    }(&x)

    fmt.Println(x) //prints [7 2 3]
} 

Another option is to use slices. Even though your function gets a copy of the slice variable it still references the original data.

package main

import "fmt"

func main() {  
    x := []int{1,2,3}

    func(arr []int) {
        arr[0] = 7
        fmt.Println(arr) //prints [7 2 3]
    }(x)

    fmt.Println(x) //prints [7 2 3]
} 

Unexpected Values in Slice and Array “range” Clauses

#

• level: beginner

This can happen if you are used to the “for-in” or “foreach” statements in other languages. The “range” clause in Go is different. It generates two values: the first value is the item index while the second value is the item data.

Bad:

package main

import "fmt"

func main() {  
    x := []string{"a","b","c"}

    for v := range x {
        fmt.Println(v) //prints 0, 1, 2
    }
} 

Good:

package main

import "fmt"

func main() {  
    x := []string{"a","b","c"}

    for _, v := range x {
        fmt.Println(v) //prints a, b, c
    }
} 

Slices and Arrays Are One-Dimensional

#

• level: beginner

It may seem like Go supports multi-dimensional arrays and slices, but it doesn’t. Creating arrays of arrays or slices of slices is possible though. For numerical computation apps that rely on dynamic multi-dimensional arrays it’s far from ideal in terms of performance and complexity.

You can build dynamic multi-dimensional arrays using raw one-dimensional arrays, slices of “independent” slices, and slices of “shared data” slices.

If you are using raw one-dimensional arrays you are responsible for indexing, bounds checking, and memory reallocations when the arrays need to grow.

Creating a dynamic multi-dimensional array using slices of “independent” slices is a two step process. First, you have to create the outer slice. Then, you have to allocate each inner slice. The inner slices are independent of each other. You can grow and shrink them without affecting other inner slices.

package main

func main() {  
    x := 2
    y := 4

    table := make([][]int,x)
    for i:= range table {
        table[i] = make([]int,y)
    }
} 

Creating a dynamic multi-dimensional array using slices of “shared data” slices is a three step process. First, you have to create the data “container” slice that will hold raw data. Then, you create the outer slice. Finally, you initialize each inner slice by reslicing the raw data slice.

package main

import "fmt"

func main() {  
    h, w := 2, 4

    raw := make([]int,h*w)
    for i := range raw {
        raw[i] = i
    }
    fmt.Println(raw,&raw[4])
    //prints: [0 1 2 3 4 5 6 7] <ptr_addr_x>

    table := make([][]int,h)
    for i:= range table {
        table[i] = raw[i*w:i*w + w]
    }

    fmt.Println(table,&table[1][0])
    //prints: [[0 1 2 3] [4 5 6 7]] <ptr_addr_x>
} 

There’s a spec/proposal for multi-dimensional arrays and slices, but it looks like it’s a low priority feature at this point in time.

Accessing Non-Existing Map Keys

#

• level: beginner

This is a gotcha for developers who expect to get “nil” identifiers (like it’s done in other languages). The returned value will be “nil” if the “zero value” for the corresponding data type is “nil”, but it’ll be different for other data types. Checking for the appropriate “zero value” can be used to determine if the map record exists, but it’s not always reliable (e.g., what do you do if you have a map of booleans where the “zero value” is false). The most reliable way to know if a given map record exists is to check the second value returned by the map access operation.

Bad:

package main

import "fmt"

func main() {  
    x := map[string]string{"one":"a","two":"","three":"c"}

    if v := x["two"]; v == "" { //incorrect
        fmt.Println("no entry")
    }
} 

Good:

package main

import "fmt"

func main() {  
    x := map[string]string{"one":"a","two":"","three":"c"}

    if _,ok := x["two"]; !ok {
        fmt.Println("no entry")
    }
} 

Strings Are Immutable

#

• level: beginner

Trying to update an individual character in a string variable using the index operator will result in a failure. Strings are read-only byte slices (with a few extra properties). If you do need to update a string then use a byte slice instead converting it to a string type when necessary.

Fails:

package main

import "fmt"

func main() {  
    x := "text"
    x[0] = 'T'

    fmt.Println(x)
} 

Compile Error:

/tmp/sandbox305565531/main.go:7: cannot assign to x[0]

Works:

package main

import "fmt"

func main() {  
    x := "text"
    xbytes := []byte(x)
    xbytes[0] = 'T'

    fmt.Println(string(xbytes)) //prints Text
} 

Note that this isn’t really the right way to update characters in a text string because a given character could be stored in multiple bytes. If you do need to make updates to a text string convert it to a rune slice first. Even with rune slices a single character might span multiple runes, which can happen if you have characters with grave accent, for example. This complicated and ambiguous nature of “characters” is the reason why Go strings are represented as byte sequences.

Conversions Between Strings and Byte Slices

#

• level: beginner

When you convert a string to a byte slice (and vice versa) you get a complete copy of the orginal data. It’s not like a cast operation in other languages and it’s not like reslicing where the new slice variable points to the same underlying array used by the original byte slice.

Go does have a couple of optimizations for []byte to string and string to []byte conversions to avoid extra allocations (with more optimizations on the todo list).

The first optimization avoids extra allocations when []byte keys are used to lookup entries in map[string] collections: m[string(key)].

The second optimization avoids extra allocations in for range clauses where strings are converted to []byte: for i,v := range []byte(str) {...}.

Strings and Index Operator

#

• level: beginner

The index operator on a string returns a byte value, not a character (like it’s done in other languages).

package main

import "fmt"

func main() {  
    x := "text"
    fmt.Println(x[0]) //print 116
    fmt.Printf("%T",x[0]) //prints uint8
} 

If you need to access specific string “characters” (unicode code points/runes) use the for range clause. The official “unicode/utf8” package and the experimental utf8string package (golang.org/x/exp/utf8string) are also useful. The utf8string package includes a convenient At() method. Converting the string to a slice of runes is an option too.

Strings Are Not Always UTF8 Text

#

• level: beginner

String values are not required to be UTF8 text. They can contain arbitrary bytes. The only time strings are UTF8 is when string literals are used. Even then they can include other data using escape sequences.

To know if you have a UTF8 text string use the ValidString() function from the “unicode/utf8” package.

package main

import (  
    "fmt"
    "unicode/utf8"
)

func main() {  
    data1 := "ABC"
    fmt.Println(utf8.ValidString(data1)) //prints: true

    data2 := "A\xfeC"
    fmt.Println(utf8.ValidString(data2)) //prints: false
} 

String Length

#

• level: beginner

Let’s say you are a python developer and you have the following piece of code:

data = u'♥'  
print(len(data)) #prints: 1 

When you convert it to a similar Go code snippet you might be surprised.

package main

import "fmt"

func main() {  
    data := "♥"
    fmt.Println(len(data)) //prints: 3
} 

The built-in len() function returns the number of bytes instead of the number of characters like it’s done for unicode strings in Python.

To get the same results in Go use the RuneCountInString() function from the “unicode/utf8” package.

package main

import (  
    "fmt"
    "unicode/utf8"
)

func main() {  
    data := "♥"
    fmt.Println(utf8.RuneCountInString(data)) //prints: 1 

Technically the RuneCountInString() function doesn’t return the number of characters because a single character may span multiple runes.

package main

import (  
    "fmt"
    "unicode/utf8"
)

func main() {  
    data := "é"
    fmt.Println(len(data))                    //prints: 3
    fmt.Println(utf8.RuneCountInString(data)) //prints: 2
} 

Missing Comma In Multi-Line Slice, Array, and Map Literals

#

• level: beginner

Fails:

package main

func main() {  
    x := []int{
    1,
    2 //error
    }
    _ = x
} 

Compile Errors:

/tmp/sandbox367520156/main.go:6: syntax error: need trailing comma before newline in composite literal /tmp/sandbox367520156/main.go:8: non-declaration statement outside function body /tmp/sandbox367520156/main.go:9: syntax error: unexpected }

Works:

package main

func main() {  
    x := []int{
    1,
    2,
    }
    x = x

    y := []int{3,4,} //no error
    y = y
} 

You won’t get a compiler error if you leave the trailing comma when you collapse the declaration to be on a single line.

log.Fatal and log.Panic Do More Than Log

#

• level: beginner

Logging libraries often provide different log levels. Unlike those logging libraries, the log package in Go does more than log if you call its Fatal*() and Panic*() functions. When your app calls those functions Go will also terminate your app :-)

package main

import "log"

func main() {  
    log.Fatalln("Fatal Level: log entry") //app exits here
    log.Println("Normal Level: log entry")
} 

Built-in Data Structure Operations Are Not Synchronized

#

• level: beginner

Even though Go has a number of features to support concurrency natively, concurrency safe data collections are not one them :-) It’s your responsibility to ensure the data collection updates are atomic. Goroutines and channels are the recommended way to implement those atomic operations, but you can also leverage the “sync” package if it makes sense for your application.

Iteration Values For Strings in “range” Clauses

#

• level: beginner

The index value (the first value returned by the “range” operation) is the index of the first byte for the current “character” (unicode code point/rune) returned in the second value. It’s not the index for the current “character” like it’s done in other languages. Note that an actual character might be represented by multiple runes. Make sure to check out the “norm” package (golang.org/x/text/unicode/norm) if you need to work with characters.

The for range clauses with string variables will try to interpret the data as UTF8 text. For any byte sequences it doesn’t understand it will return 0xfffd runes (aka unicode replacement characters) instead of the actual data. If you have arbitrary (non-UTF8 text) data stored in your string variables, make sure to convert them to byte slices to get all stored data as is.

package main

import "fmt"

func main() {  
    data := "A\xfe\x02\xff\x04"
    for _,v := range data {
        fmt.Printf("%#x ",v)
    }
    //prints: 0x41 0xfffd 0x2 0xfffd 0x4 (not ok)

    fmt.Println()
    for _,v := range []byte(data) {
        fmt.Printf("%#x ",v)
    }
    //prints: 0x41 0xfe 0x2 0xff 0x4 (good)
} 

Iterating Through a Map Using a “for range” Clause

#

• level: beginner

This is a gotcha if you expect the items to be in a certain order (e.g., ordered by the key value). Each map iteration will produce different results. The Go runtime tries to go an extra mile randomizing the iteration order, but it doesn’t always succeed so you may get several identical map iterations. Don’t be surprised to see 5 identical iterations in a row.

package main

import "fmt"

func main() {  
    m := map[string]int{"one":1,"two":2,"three":3,"four":4}
    for k,v := range m {
        fmt.Println(k,v)
    }
} 

And if you use the Go Playground (https://play.golang.org/) you’ll always get the same results because it doesn’t recompile the code unless you make a change.

Fallthrough Behavior in “switch” Statements

#

• level: beginner

The “case” blocks in “switch” statements break by default. This is different from other languages where the default behavior is to fall through to the next “case” block.

package main

import "fmt"

func main() {  
    isSpace := func(ch byte) bool {
        switch(ch) {
        case ' ': //error
        case '\t':
            return true
        }
        return false
    }

    fmt.Println(isSpace('\t')) //prints true (ok)
    fmt.Println(isSpace(' '))  //prints false (not ok)
} 

You can force the “case” blocks to fall through by using the “fallthrough” statement at the end of each “case” block. You can also rewrite your switch statement to use expression lists in the “case” blocks.

package main

import "fmt"

func main() {  
    isSpace := func(ch byte) bool {
        switch(ch) {
        case ' ', '\t':
            return true
        }
        return false
    }

    fmt.Println(isSpace('\t')) //prints true (ok)
    fmt.Println(isSpace(' '))  //prints true (ok)
} 

Increments and Decrements

#

• level: beginner

Many languages have increment and decrement operators. Unlike other languages, Go doesn’t support the prefix version of the operations. You also can’t use these two operators in expressions.

Fails:

package main

import "fmt"

func main() {  
    data := []int{1,2,3}
    i := 0
    ++i //error
    fmt.Println(data[i++]) //error
} 

Compile Errors:

/tmp/sandbox101231828/main.go:8: syntax error: unexpected ++ /tmp/sandbox101231828/main.go:9: syntax error: unexpected ++, expecting :

Works:

package main

import "fmt"

func main() {  
    data := []int{1,2,3}
    i := 0
    i++
    fmt.Println(data[i])
} 

Bitwise NOT Operator

#

• level: beginner

Many languages use ~ as the unary NOT operator (aka bitwise complement), but Go reuses the XOR operator (^) for that.

Fails:

package main

import "fmt"

func main() {  
    fmt.Println(~2) //error
} 

Compile Error:

/tmp/sandbox965529189/main.go:6: the bitwise complement operator is ^

Works:

package main

import "fmt"

func main() {  
    var d uint8 = 2
    fmt.Printf("%08b\n",^d)
} 

Go still uses ^ as the XOR operator, which may be confusing for some people.

If you want you can represent a unary NOT operation (e.g, NOT 0x02) with a binary XOR operation (e.g., 0x02 XOR 0xff). This could explain why ^ is reused to represent unary NOT operations.

Go also has a special ‘AND NOT’ bitwise operator (&^), which adds to the NOT operator confusion. It looks like a special feature/hack to support A AND (NOT B) without requiring parentheses.

package main

import "fmt"

func main() {  
    var a uint8 = 0x82
    var b uint8 = 0x02
    fmt.Printf("%08b [A]\n",a)
    fmt.Printf("%08b [B]\n",b)

    fmt.Printf("%08b (NOT B)\n",^b)
    fmt.Printf("%08b ^ %08b = %08b [B XOR 0xff]\n",b,0xff,b ^ 0xff)

    fmt.Printf("%08b ^ %08b = %08b [A XOR B]\n",a,b,a ^ b)
    fmt.Printf("%08b & %08b = %08b [A AND B]\n",a,b,a & b)
    fmt.Printf("%08b &^%08b = %08b [A 'AND NOT' B]\n",a,b,a &^ b)
    fmt.Printf("%08b&(^%08b)= %08b [A AND (NOT B)]\n",a,b,a & (^b))
} 

Operator Precedence Differences

#

• level: beginner

Aside from the “bit clear” operators (&^) Go has a set of standard operators shared by many other languages. The operator precedence is not always the same though.

package main

import "fmt"

func main() {  
    fmt.Printf("0x2 & 0x2 + 0x4 -> %#x\n",0x2 & 0x2 + 0x4)
    //prints: 0x2 & 0x2 + 0x4 -> 0x6
    //Go:    (0x2 & 0x2) + 0x4
    //C++:    0x2 & (0x2 + 0x4) -> 0x2

    fmt.Printf("0x2 + 0x2 << 0x1 -> %#x\n",0x2 + 0x2 << 0x1)
    //prints: 0x2 + 0x2 << 0x1 -> 0x6
    //Go:     0x2 + (0x2 << 0x1)
    //C++:   (0x2 + 0x2) << 0x1 -> 0x8

    fmt.Printf("0xf | 0x2 ^ 0x2 -> %#x\n",0xf | 0x2 ^ 0x2)
    //prints: 0xf | 0x2 ^ 0x2 -> 0xd
    //Go:    (0xf | 0x2) ^ 0x2
    //C++:    0xf | (0x2 ^ 0x2) -> 0xf
} 

Unexported Structure Fields Are Not Encoded

#

• level: beginner

The struct fields starting with lowercase letters will not be (json, xml, gob, etc.) encoded, so when you decode the structure you’ll end up with zero values in those unexported fields.

package main

import (  
    "fmt"
    "encoding/json"
)

type MyData struct {  
    One int
    two string
}

func main() {  
    in := MyData{1,"two"}
    fmt.Printf("%#v\n",in) //prints main.MyData{One:1, two:"two"}

    encoded,_ := json.Marshal(in)
    fmt.Println(string(encoded)) //prints {"One":1}

    var out MyData
    json.Unmarshal(encoded,&out)

    fmt.Printf("%#v\n",out) //prints main.MyData{One:1, two:""}
} 

App Exits With Active Goroutines

#

• level: beginner

The app will not wait for all your goroutines to complete. This is a common mistake for beginners in general. Everybody starts somewhere, so there’s no shame in making rookie mistakes :-)

package main

import (  
    "fmt"
    "time"
)

func main() {  
    workerCount := 2

    for i := 0; i < workerCount; i++ {
        go doit(i)
    }
    time.Sleep(1 * time.Second)
    fmt.Println("all done!")
}

func doit(workerId int) {  
    fmt.Printf("[%v] is running\n",workerId)
    time.Sleep(3 * time.Second)
    fmt.Printf("[%v] is done\n",workerId)
} 

You’ll see:

[0] is running [1] is running all done!

One of the most common solutions is to use a “WaitGroup” variable. It will allow the main goroutine to wait until all worker goroutines are done. If your app has long running workers with message processing loops you’ll also need a way to signal those goroutines that it’s time to exit. You can send a “kill” message to each worker. Another option is to close a channel all workers are receiving from. It’s a simple way to signal all goroutines at once.

package main

import (  
    "fmt"
    "sync"
)

func main() {  
    var wg sync.WaitGroup
    done := make(chan struct{})
    workerCount := 2

    for i := 0; i < workerCount; i++ {
        wg.Add(1)
        go doit(i,done,wg)
    }

    close(done)
    wg.Wait()
    fmt.Println("all done!")
}

func doit(workerId int,done <-chan struct{},wg sync.WaitGroup) {  
    fmt.Printf("[%v] is running\n",workerId)
    defer wg.Done()
    <- done
    fmt.Printf("[%v] is done\n",workerId)
} 

If you run this app you’ll see:

[0] is running [0] is done [1] is running [1] is done

Looks like the workers are done before the main goroutine exits. Great! However, you’ll also see this:

fatal error: all goroutines are asleep - deadlock!

That’s not so great :-) What’s going on? Why is there a deadlock? The workers exited and they executed wg.Done(). The app should work.

The deadlock happens because each worker gets a copy of the original “WaitGroup” variable. When workers execute wg.Done() it has no effect on the “WaitGroup” variable in the main goroutine.

package main

import (  
    "fmt"
    "sync"
)

func main() {  
    var wg sync.WaitGroup
    done := make(chan struct{})
    wq := make(chan interface{})
    workerCount := 2

    for i := 0; i < workerCount; i++ {
        wg.Add(1)
        go doit(i,wq,done,&wg)
    }

    for i := 0; i < workerCount; i++ {
        wq <- i
    }

    close(done)
    wg.Wait()
    fmt.Println("all done!")
}

func doit(workerId int, wq <-chan interface{},done <-chan struct{},wg *sync.WaitGroup) {  
    fmt.Printf("[%v] is running\n",workerId)
    defer wg.Done()
    for {
        select {
        case m := <- wq:
            fmt.Printf("[%v] m => %v\n",workerId,m)
        case <- done:
            fmt.Printf("[%v] is done\n",workerId)
            return
        }
    }
} 

Now it works as expected :-)

Sending to an Unbuffered Channel Returns As Soon As the Target Receiver Is Ready

#

• level: beginner

The sender will not be blocked until your message is processed by the receiver. Depending on the machine where you are running the code, the receiver goroutine may or may not have enough time to process the message before the sender continues its execution.

package main

import "fmt"

func main() {  
    ch := make(chan string)

    go func() {
        for m := range ch {
            fmt.Println("processed:",m)
        }
    }()

    ch <- "cmd.1"
    ch <- "cmd.2" //won't be processed
} 

Sending to an Closed Channel Causes a Panic

#

• level: beginner

Receiving from a closed channel is safe. The ok return value in a receive statement will be set to false indicating that no data was received. If you are receiving from a buffered channel you’ll get the buffered data first and once it’s empty the ok return value will be false.

Sending data to a closed channel causes a panic. It is a documented behavior, but it’s not very intuitive for new Go developers who might expect the send behavior to be similar to the receive behavior.

package main

import (  
    "fmt"
    "time"
)

func main() {  
    ch := make(chan int)
    for i := 0; i < 3; i++ {
        go func(idx int) {
            ch <- (idx + 1) * 2
        }(i)
    }

    //get the first result
    fmt.Println(<-ch)
    close(ch) //not ok (you still have other senders)
    //do other work
    time.Sleep(2 * time.Second)
} 

Depending on your application the fix will be different. It might be a minor code change or it might require a change in your application design. Either way, you’ll need to make sure your application doesn’t try to send data to a closed channel.

The buggy example can be fixed by using a special cancellation channel to signal the remaining workers that their results are no longer neeeded.

package main

import (  
    "fmt"
    "time"
)

func main() {  
    ch := make(chan int)
    done := make(chan struct{})
    for i := 0; i < 3; i++ {
        go func(idx int) {
            select {
            case ch <- (idx + 1) * 2: fmt.Println(idx,"sent result")
            case <- done: fmt.Println(idx,"exiting")
            }
        }(i)
    }

    //get first result
    fmt.Println("result:",<-ch)
    close(done)
    //do other work
    time.Sleep(3 * time.Second)
} 

Using “nil” Channels

#

• level: beginner

Send and receive operations on a nil channel block forver. It’s a well documented behavior, but it can be a surprise for new Go developers.

package main

import (  
    "fmt"
    "time"
)

func main() {  
    var ch chan int
    for i := 0; i < 3; i++ {
        go func(idx int) {
            ch <- (idx + 1) * 2
        }(i)
    }

    //get first result
    fmt.Println("result:",<-ch)
    //do other work
    time.Sleep(2 * time.Second)
} 

If you run the code you’ll see a runtime error like this: fatal error: all goroutines are asleep - deadlock!

This behavior can be used as a way to dynamically enable and disable case blocks in a select statement.

package main

import "fmt"  
import "time"

func main() {  
    inch := make(chan int)
    outch := make(chan int)

    go func() {
        var in <- chan int = inch
        var out chan <- int
        var val int
        for {
            select {
            case out <- val:
                out = nil
                in = inch
            case val = <- in:
                out = outch
                in = nil
            }
        }
    }()

    go func() {
        for r := range outch {
            fmt.Println("result:",r)
        }
    }()

    time.Sleep(0)
    inch <- 1
    inch <- 2
    time.Sleep(3 * time.Second)
} 

Methods with Value Receivers Can’t Change the Original Value

#

• level: beginner

Method receivers are like regular function arguments. If it’s declared to be a value then your function/method gets a copy of your receiver argument. This means making changes to the receiver will not affect the original value unless your receiver is a map or slice variable and you are updating the items in the collection or the fields you are updating in the receiver are pointers.

package main

import "fmt"

type data struct {  
    num int
    key *string
    items map[string]bool
}

func (this *data) pmethod() {  
    this.num = 7
}

func (this data) vmethod() {  
    this.num = 8
    *this.key = "v.key"
    this.items["vmethod"] = true
}

func main() {  
    key := "key.1"
    d := data{1,&key,make(map[string]bool)}

    fmt.Printf("num=%v key=%v items=%v\n",d.num,*d.key,d.items)
    //prints num=1 key=key.1 items=map[]

    d.pmethod()
    fmt.Printf("num=%v key=%v items=%v\n",d.num,*d.key,d.items) 
    //prints num=7 key=key.1 items=map[]

    d.vmethod()
    fmt.Printf("num=%v key=%v items=%v\n",d.num,*d.key,d.items)
    //prints num=7 key=v.key items=map[vmethod:true]
} 

Closing HTTP Response Body

#

• level: intermediate

When you make requests using the standard http library you get a http response variable. If you don’t read the response body you still need to close it. Note that you must do it for empty responses too. It’s very easy to forget especially for new Go developers.

Some new Go developers do try to close the response body, but they do it in the wrong place.

package main

import (  
    "fmt"
    "net/http"
    "io/ioutil"
)

func main() {  
    resp, err := http.Get("https://api.ipify.org?format=json")
    defer resp.Body.Close()//not ok
    if err != nil {
        fmt.Println(err)
        return
    }

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Println(err)
        return
    }

    fmt.Println(string(body))
} 

This code works for successful requests, but if the http request fails the resp variable might be nil, which will cause a runtime panic.

The most common why to close the response body is by using a defer call after the http response error check.

package main

import (  
    "fmt"
    "net/http"
    "io/ioutil"
)

func main() {  
    resp, err := http.Get("https://api.ipify.org?format=json")
    if err != nil {
        fmt.Println(err)
        return
    }

    defer resp.Body.Close()//ok, most of the time :-)
    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Println(err)
        return
    }

    fmt.Println(string(body))
} 

Most of the time when your http request fails the resp variable will be nil and the err variable will be non-nil. However, when you get a redirection failure both variables will be non-nil. This means you can still end up with a leak.

You can fix this leak by adding a call to close non-nil response bodies in the http response error handling block. Another option is to use one defer call to close response bodies for all failed and successful requests.

package main

import (  
    "fmt"
    "net/http"
    "io/ioutil"
)

func main() {  
    resp, err := http.Get("https://api.ipify.org?format=json")
    if resp != nil {
        defer resp.Body.Close()
    }

    if err != nil {
        fmt.Println(err)
        return
    }

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Println(err)
        return
    }

    fmt.Println(string(body))
} 

The orignal implementation for resp.Body.Close() also reads and discards the remaining response body data. This ensured that the http connection could be reused for another request if the keepalive http connection behavior is enabled. The latest http client behavior is different. Now it’s your responsibility to read and discard the remaining response data. If you don’t do it the http connection might be closed instead of being reused. This little gotcha is supposed to be documented in Go 1.5.

If reusing the http connection is important for your application you might need to add something like this at the end of your response processing logic:

_, err = io.Copy(ioutil.Discard, resp.Body) 

It will be necessary if you don’t read the entire response body right away, which might happen if you are processing json API responses with code like this:

json.NewDecoder(resp.Body).Decode(&data) 

Closing HTTP Connections

#

• level: intermediate

Some HTTP servers keep network connections open for a while (based on the HTTP 1.1 spec and the server “keep-alive” configurations). By default, the standard http library will close the network connections only when the target HTTP server asks for it. This means your app may run out of sockets/file descriptors under certain conditions.

You can ask the http library to close the connection after your request is done by setting the Close field in the request variable to true.

Another option is to add a Connection request header and set it to close. The target HTTP server should respond with a Connection: close header too. When the http library sees this response header it will also close the connection.

package main

import (  
    "fmt"
    "net/http"
    "io/ioutil"
)

func main() {  
    req, err := http.NewRequest("GET","http://golang.org",nil)
    if err != nil {
        fmt.Println(err)
        return
    }

    req.Close = true
    //or do this:
    //req.Header.Add("Connection", "close")

    resp, err := http.DefaultClient.Do(req)
    if resp != nil {
        defer resp.Body.Close()
    }

    if err != nil {
        fmt.Println(err)
        return
    }

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Println(err)
        return
    }

    fmt.Println(len(string(body)))
} 

You can also disable http connection reuse globally. You’ll need to create a custom http transport configuration for it.

package main

import (  
    "fmt"
    "net/http"
    "io/ioutil"
)

func main() {  
    tr := &http.Transport{DisableKeepAlives: true}
    client := &http.Client{Transport: tr}

    resp, err := client.Get("http://golang.org")
    if resp != nil {
        defer resp.Body.Close()
    }

    if err != nil {
        fmt.Println(err)
        return
    }

    fmt.Println(resp.StatusCode)

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Println(err)
        return
    }

    fmt.Println(len(string(body)))
} 

If you send a lot of requests to the same HTTP server it’s ok to keep the network connection open. However, if your app sends one or two requests to many different HTTP servers in a short period of time it’s a good idea to close the network connections right after your app receives the responses. Increasing the open file limit might be a good idea too. The correct solution depends on your application though.

JSON Encoder Adds a Newline Character

#

• level: intermediate

You are writing a test for your JSON encoding function when you discover that your tests fail because you are not getting the expected value. What happened? If you are using the JSON Encoder object then you’ll get an extra newline character at the end of your encoded JSON object.

package main

import (
  "fmt"
  "encoding/json"
  "bytes"
)

func main() {
  data := map[string]int{"key": 1}
  
  var b bytes.Buffer
  json.NewEncoder(&b).Encode(data)

  raw,_ := json.Marshal(data)
  
  if b.String() == string(raw) {
    fmt.Println("same encoded data")
  } else {
    fmt.Printf("'%s' != '%s'\n",raw,b.String())
    //prints:
    //'{"key":1}' != '{"key":1}\n'
  }
} 

The JSON Encoder object is designed for streaming. Streaming with JSON usually means newline delimited JSON objects and this is why the Encode method adds a newline character. This is a documented behavior, but it’s commonly overlooked or forgotten.

JSON Package Escapes Special HTML Characters in Keys and String Values

#

• level: intermediate

This is a documented behavior, but you have to be careful reading all of the JSON package documentation to learn about it. The SetEscapeHTML method description talks about the default encoding behavior for the and, less than and greater than characters.

This is a very unfortunate design decision by the Go team for a number of reasons. First, you can’t disable this behavior for the json.Marshal calls. Second, this is a badly implemented security feature because it assumes that doing HTML encoding is sufficient to protect against XSS vulnerabilities in all web applications. There are a lot of different contexts where the data can be used and each context requires its own encoding method. And finally, it’s bad because it assumes that the primary use case for JSON is a web page, which breaks the configuration libraries and the REST/HTTP APIs by default.

package main

import (
  "fmt"
  "encoding/json"
  "bytes"
)

func main() {
  data := "x < y"
  
  raw,_ := json.Marshal(data)
  fmt.Println(string(raw))
  //prints: "x \u003c y" <- probably not what you expected
  
  var b1 bytes.Buffer
  json.NewEncoder(&b1).Encode(data)
  fmt.Println(b1.String())
  //prints: "x \u003c y" <- probably not what you expected
  
  var b2 bytes.Buffer
  enc := json.NewEncoder(&b2)
  enc.SetEscapeHTML(false)
  enc.Encode(data)
  fmt.Println(b2.String())
  //prints: "x < y" <- looks better
} 

A suggestion to the Go team… Make it an opt-in.

Unmarshalling JSON Numbers into Interface Values

#

• level: intermediate

By default, Go treats numeric values in JSON as float64 numbers when you decode/unmarshal JSON data into an interface. This means the following code will fail with a panic:

package main

import (  
  "encoding/json"
  "fmt"
)

func main() {  
  var data = []byte(`{"status": 200}`)

  var result map[string]interface{}
  if err := json.Unmarshal(data, &result); err != nil {
    fmt.Println("error:", err)
    return
  }

  var status = result["status"].(int) //error
  fmt.Println("status value:",status)
} 

Runtime Panic:

panic: interface conversion: interface is float64, not int

If the JSON value you are trying to decode is an integer you have serveral options.

Option one: use the float value as-is :-)

Option two: convert the float value to the integer type you need.

package main

import (  
  "encoding/json"
  "fmt"
)

func main() {  
  var data = []byte(`{"status": 200}`)

  var result map[string]interface{}
  if err := json.Unmarshal(data, &result); err != nil {
    fmt.Println("error:", err)
    return
  }

  var status = uint64(result["status"].(float64)) //ok
  fmt.Println("status value:",status)
} 

Option three: use a Decoder type to unmarshal JSON and tell it to represent JSON numbers using the Number interface type.

package main

import (  
  "encoding/json"
  "bytes"
  "fmt"
)

func main() {  
  var data = []byte(`{"status": 200}`)

  var result map[string]interface{}
  var decoder = json.NewDecoder(bytes.NewReader(data))
  decoder.UseNumber()

  if err := decoder.Decode(&result); err != nil {
    fmt.Println("error:", err)
    return
  }

  var status,_ = result["status"].(json.Number).Int64() //ok
  fmt.Println("status value:",status)
} 

You can use the string representation of your Number value to unmarshal it to a different numeric type:

package main

import (  
  "encoding/json"
  "bytes"
  "fmt"
)

func main() {  
  var data = []byte(`{"status": 200}`)

  var result map[string]interface{}
  var decoder = json.NewDecoder(bytes.NewReader(data))
  decoder.UseNumber()

  if err := decoder.Decode(&result); err != nil {
    fmt.Println("error:", err)
    return
  }

  var status uint64
  if err := json.Unmarshal([]byte(result["status"].(json.Number).String()), &status); err != nil {
    fmt.Println("error:", err)
    return
  }

  fmt.Println("status value:",status)
} 

Option four: use a struct type that maps your numeric value to the numeric type you need.

package main

import (  
  "encoding/json"
  "bytes"
  "fmt"
)

func main() {  
  var data = []byte(`{"status": 200}`)

  var result struct {
    Status uint64 `json:"status"`
  }

  if err := json.NewDecoder(bytes.NewReader(data)).Decode(&result); err != nil {
    fmt.Println("error:", err)
    return
  }

  fmt.Printf("result => %+v",result)
  //prints: result => {Status:200}
} 

Option five: use a struct that maps your numeric value to the json.RawMessage type if you need to defer the value decoding.

This option is useful if you have to perform conditional JSON field decoding where the field type or structure might change.

package main

import (  
  "encoding/json"
  "bytes"
  "fmt"
)

func main() {  
  records := [][]byte{
    []byte(`{"status": 200, "tag":"one"}`),
    []byte(`{"status":"ok", "tag":"two"}`),
  }

  for idx, record := range records {
    var result struct {
      StatusCode uint64
      StatusName string
      Status json.RawMessage `json:"status"`
      Tag string             `json:"tag"`
    }

    if err := json.NewDecoder(bytes.NewReader(record)).Decode(&result); err != nil {
      fmt.Println("error:", err)
      return
    }

    var sstatus string
    if err := json.Unmarshal(result.Status, &sstatus); err == nil {
      result.StatusName = sstatus
    }

    var nstatus uint64
    if err := json.Unmarshal(result.Status, &nstatus); err == nil {
      result.StatusCode = nstatus
    }

    fmt.Printf("[%v] result => %+v\n",idx,result)
  }
} 

JSON String Values Will Not Be Ok with Hex or Other non-UTF8 Escape Sequences

#

• level: intermediate

Go expects string values to be UTF8 encoded. This means you can’t have arbitrary hex escaped binary data in your JSON strings (and you also have to escape the backslash character). This is really a JSON gotcha Go inherited, but it happens often enough in Go apps that it makes sense to mention it anyways.

package main

import (
  "fmt"
  "encoding/json"
)

type config struct {
  Data string `json:"data"`
}

func main() {
  raw := []byte(`{"data":"\xc2"}`)
  var decoded config

  if err := json.Unmarshal(raw, &decoded); err != nil {
        fmt.Println(err)
    //prints: invalid character 'x' in string escape code
    }
  
} 

The Unmarshal/Decode calls will fail if Go sees a hex escape sequence. If you do need to have a backslash in you string make sure to escape it with another backlash. If you want to use hex encoded binary data you can escape the backslash and then do your own hex escaping with the decoded data in your JSON string.

package main

import (
  "fmt"
  "encoding/json"
)

type config struct {
  Data string `json:"data"`
}

func main() {
  raw := []byte(`{"data":"\\xc2"}`)
  
  var decoded config
  
  json.Unmarshal(raw, &decoded)
  
  fmt.Printf("%#v",decoded) //prints: main.config{Data:"\\xc2"}
  //todo: do your own hex escape decoding for decoded.Data 
} 

Another option is to use the byte array/slice data type in your JSON object, but the binary data will have to be base64 encoded.

package main

import (
  "fmt"
  "encoding/json"
)

type config struct {
  Data []byte `json:"data"`
}

func main() {
  raw := []byte(`{"data":"wg=="}`)
  var decoded config
  
  if err := json.Unmarshal(raw, &decoded); err != nil {
          fmt.Println(err)
      }
  
  fmt.Printf("%#v",decoded) //prints: main.config{Data:[]uint8{0xc2}}
} 

Something else to watch out for is the Unicode replacement character (U+FFFD). Go will use the replacement character instead of invalid UTF8, so the Unmarshal/Decode call will not fail, but the string value you get might not be what you expected.

Comparing Structs, Arrays, Slices, and Maps

#

• level: intermediate

You can use the equality operator, ==, to compare struct variables if each structure field can be compared with the equality operator.

package main

import "fmt"

type data struct {  
    num int
    fp float32
    complex complex64
    str string
    char rune
    yes bool
    events <-chan string
    handler interface{}
    ref *byte
    raw [10]byte
}

func main() {  
    v1 := data{}
    v2 := data{}
    fmt.Println("v1 == v2:",v1 == v2) //prints: v1 == v2: true
} 

If any of the struct fields are not comparable then using the equality operator will result in compile time errors. Note that arrays are comparable only if their data items are comparable.

package main

import "fmt"

type data struct {  
    num int                //ok
    checks [10]func() bool //not comparable
    doit func() bool       //not comparable
    m map[string] string   //not comparable
    bytes []byte           //not comparable
}

func main() {  
    v1 := data{}
    v2 := data{}
    fmt.Println("v1 == v2:",v1 == v2)
} 

Go does provide a number of helper functions to compare variables that can’t be compared using the comparison operators.

The most generic solution is to use the DeepEqual() function in the reflect package.

package main

import (  
    "fmt"
    "reflect"
)

type data struct {  
    num int                //ok
    checks [10]func() bool //not comparable
    doit func() bool       //not comparable
    m map[string] string   //not comparable
    bytes []byte           //not comparable
}

func main() {  
    v1 := data{}
    v2 := data{}
    fmt.Println("v1 == v2:",reflect.DeepEqual(v1,v2)) //prints: v1 == v2: true

    m1 := map[string]string{"one": "a","two": "b"}
    m2 := map[string]string{"two": "b", "one": "a"}
    fmt.Println("m1 == m2:",reflect.DeepEqual(m1, m2)) //prints: m1 == m2: true

    s1 := []int{1, 2, 3}
    s2 := []int{1, 2, 3}
    fmt.Println("s1 == s2:",reflect.DeepEqual(s1, s2)) //prints: s1 == s2: true
} 

Aside from being slow (which may or may not be a deal breaker for your application), DeepEqual() also has its own gotchas.

package main

import (  
    "fmt"
    "reflect"
)

func main() {  
    var b1 []byte = nil
    b2 := []byte{}
    fmt.Println("b1 == b2:",reflect.DeepEqual(b1, b2)) //prints: b1 == b2: false
}