[Golang] Goroutines Poll Web Feeds


This post shows how to periodically poll (fetch by http get) web feeds with goroutines.

Souce Code

get.go | repository | view raw
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
// http://tour.golang.org/concurrency/1
// http://golang.org/pkg/net/http/
package main

import (
	"time"
	"net/http"
	"io/ioutil"
	"fmt"
)

var pollingInterval = 1 * time.Minute

func fetch(url string) {
	resp, err := http.Get(url)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer resp.Body.Close()

	body, err2 := ioutil.ReadAll(resp.Body)
	if err2 != nil {
		fmt.Println(err2)
		return
	}
	fmt.Println(body[:10])
}

func getFeed(url string) {
	for {
		fetch(url)
		time.Sleep(pollingInterval)
	}
}

func poll(feedUrls []string) {
	for _, url := range feedUrls {
		go getFeed(url)
	}
}

func main() {
	feedUrls := []string{
	"https://news.ycombinator.com/rss",
	"http://www.csdn.net/article/rss_lastnews",
	"http://www.solidot.org/index.rss",
	"http://blog.jobbole.com/feed/"}

	poll(feedUrls)

	// block here
	for { select {} }
}

Tested on: Ubuntu Linux 14.10, Go 1.4.


References:

[1]Concurrency - A Tour of Go
[2]http - The Go Programming Language
[3]go - Golang: Is it possible to capture a Ctrl+C signal and run a cleanup function, in a "defer" fashion? - Stack Overflow
[4]Graceful stopping in Go (Go 语言中实现优雅的停止程序 - 技术翻译 - 开源中国社区)
[5]Graceful server restart with Go (使用 Go 语言实现优雅的服务器重启 - 技术翻译 - 开源中国社区)
[6]How to run a custom function with timeout? : golang
[7]Non blocking way of notifying a goroutine, by many at once : golang