nebulo-scraper

This is the web scraper our app Nebulo uses to scrape air quality data.

It is written in Go and can be used either as a CLI or as a package from another Go application.

CLI Usage

Populate the required environment variables:

cp .env.example .env

Run all scrapers:

go run ./cmd/nebulo-scraper

Run one scraper:

go run ./cmd/nebulo-scraper -source singapore

The CLI writes JSON files into output/ by default. Use -output <dir> to choose a different destination. Google geocoding results are cached in a temp JSON file by default; use -geocode-cache <path> to choose a different cache file, or -geocode-cache "" for an in-memory cache.

Package Usage

package main

import (
	"context"
	"log"

	scraper "github.com/undertideco/nebulo-scraper"
)

func main() {
	config, err := scraper.LoadConfiguration(context.Background())
	if err != nil {
		log.Fatal(err)
	}

cache := scraper.NewMemoryGeocodeCache()
runner := scraper.New(scraper.Options{
	Configuration: config,
	GeocodeCache:  cache,
})

	cities, err := runner.Scrape(context.Background())
	if err != nil {
		log.Fatal(err)
	}

log.Println(len(cities))
}

Consumers that want Redis, SQLite, or another cache can implement scraper.GeocodeCache and pass it through scraper.Options.GeocodeCache.

Output

Each scraper writes its results to output/<scraper>.json. All results are also combined into output/_all.json.

_all.json is a JSON array of city objects with the following shape:

[
  {
    "name": "string — city or station name",
    "region": "string — country or region identifier",
    "location": {
      "lat": "number — latitude",
      "lng": "number — longitude"
    },
    "data": "number — AQI (Air Quality Index) reading"
  }
]

Development

1. Clone the repo
2. Copy .env.example to .env and populate the values
3. Run go test ./...

Releases

Releases are driven by Cocogitto and the Release GitHub Actions workflow.

For the initial stable release, run the workflow on master with bump set to major; with no previous tag, Cocogitto falls back to 0.0.0 and will create v1.0.0. A seed v0.0.0 tag is not required. For later releases, choose auto, major, minor, or patch.

Each release publishes Linux amd64 and arm64 CLI tarballs plus SHA-256 checksums.

Questions?

Feel free to create an issue.

License

MIT