Merge branch 'master' into apply-dependency-injection

README.md

@@ -3,6 +3,7 @@
 Copyright 2019, Geeks Accelerator 
 twins@geeksaccelerator.com
 
+Sponsored by Copper Valley Telecom
 
 The SaaS Starter Kit is a set of libraries for building scalable software-as-a-service (SaaS) applications that helps 
 preventing both misuse and fraud. The goal of this project is to provide a proven starting point for new 
@@ -62,7 +63,7 @@ delivered to clients.
 a knowledge of a completely different expertise - DevOps. This project provides a complete continuous build pipeline that 
 will push the code to production with minimal effort using serverless deployments to AWS Fargate with GitLab CI/CD.  
 5. Observability - Ensure the code is running as expected in a remote environment. This project implements Datadog to 
-facilitate exposing metrics, logs and request tracing to obversabe and validate your services are stable and responsive 
+facilitate exposing metrics, logs and request tracing to obverse and validate your services are stable and responsive 
  for your clients (hopefully paying clients). 
 
 
@@ -71,7 +72,7 @@ facilitate exposing metrics, logs and request tracing to obversabe and validate
 The example project is a complete starter kit for building SasS with GoLang. It provides two example services:
 * Web App - Responsive web application to provide service to clients. Includes user signup and user authentication for 
 direct client interaction via their web browsers. 
-* Web API - REST API with JWT authentication that renders results as JSON. This allows clients and other third-pary companies to develop deep 
+* Web API - REST API with JWT authentication that renders results as JSON. This allows clients and other third-party companies to develop deep 
 integrations with the project.
 
 The example project also provides these tools:
@@ -106,7 +107,7 @@ Accordingly, the project architecture is illustrated with the following diagram.
 With SaaS, a client subscribes to an online service you provide them. The example project provides functionality for 
 clients to subscribe and then once subscribed they can interact with your software service. 
 
-The initial contributors to this project are building this saas-starter-kit based on their years of experience building enterprise B2B SaaS. Particularily, this saas-starter-kit is based on their most recent experience building the
+The initial contributors to this project are building this saas-starter-kit based on their years of experience building enterprise B2B SaaS. Particularly, this saas-starter-kit is based on their most recent experience building the
 B2B SaaS for [standard operating procedure software](https://keeni.space) (written entirely in Golang). Please refer to the Keeni.Space website,
 its [SOP software pricing](https://keeni.space/pricing) and its signup process. The SaaS web app is then available at 
 [app.keeni.space](https://app.keeni.space). They plan on leveraging this experience and build it into a simplified set 
@@ -175,7 +176,7 @@ $ git clone git@gitlab.com:geeks-accelerator/oss/saas-starter-kit.git
 $ cd saas-starter-kit/
 ```
 
-If you have Go Modules enabled, you should be able compile the project locally. If you have Go Modulels disabled, see 
+If you have Go Modules enabled, you should be able compile the project locally. If you have Go Modules disabled, see 
 the next section.
 
 
@@ -386,7 +387,7 @@ Policy Document: {
                  }
 ```
 
-Create a new user with programic access and directly attach it the policy `SaasStarterKitDevServices`
+Create a new user with programmatic access and directly attach it the policy `SaasStarterKitDevServices`
 
 4. Create a new docker-compose config file
 ```bash
@@ -395,7 +396,7 @@ Create a new user with programic access and directly attach it the policy `SaasS
 
 5. Update .env_docker_compose with the Access key ID and Secret access key  
 
-6. Update `.gitlab-ci.yml` with relevent details. 
+6. Update `.gitlab-ci.yml` with relevant details. 
 
 
 ### Optional. Set AWS and Datadog Configs
@@ -489,7 +490,7 @@ For more details on this service, read [web-app readme](https://gitlab.com/geeks
 Schema is a minimalistic database migration helper that can manually be invoked via CLI. It provides schema versioning 
 and migration rollback. 
 
-To support POD architecture, the schema for the entire project is defined globally and is located inside internal: 
+The schema for the entire project is defined globally and is located inside internal: 
 [internal/schema](https://gitlab.com/geeks-accelerator/oss/saas-starter-kit/tree/master/internal/schema)
 
 Keeping a global schema helps ensure business logic can be decoupled across multiple packages. It is a firm belief that 

cmd/web-api/README.md

@@ -254,7 +254,7 @@ swag init
 
 ### Additional Swagger Annotations
 
-Below are some additional example annotions that can be added to `main.go`
+Below are some additional example annotations that can be added to `main.go`
 ```go
 // @title SaaS Example API
 // @description This provides a public API...

cmd/web-app/README.md

@@ -24,7 +24,7 @@ http://127.0.0.1:3000/
 
 While the web-api service has 
 significant functionality, this web-app service is still in development. Currently this web-app services only resizes 
-an image and displays resvised versions of it on the index page. See section below on Future Functionality. 
+an image and displays resized versions of it on the index page. See section below on Future Functionality. 
 
 If you would like to help, please email twins@geeksinthewoods.com.
 

cmd/web-app/handlers/api_geo.go

@@ -26,7 +26,6 @@ type GeoRepository interface {
 	FindCountries(ctx context.Context, orderBy, where string, args ...interface{}) ([]*geonames.Country, error)
 	FindCountryTimezones(ctx context.Context, orderBy, where string, args ...interface{}) ([]*geonames.CountryTimezone, error)
 	ListTimezones(ctx context.Context) ([]string, error)
-	LoadGeonames(ctx context.Context, rr chan<- interface{}, countries ...string)
 }
 
 // GeonameByPostalCode...

cmd/web-app/handlers/projects.go

@@ -74,7 +74,7 @@ func (h *Projects) Index(ctx context.Context, w http.ResponseWriter, r *http.Req
 			var v datatable.ColumnValue
 			switch col.Field {
 			case "id":
-				v.Value = fmt.Sprintf("%d", q.ID)
+				v.Value = fmt.Sprintf("%s", q.ID)
 			case "name":
 				v.Value = q.Name
 				v.Formatted = fmt.Sprintf("<a href='%s'>%s</a>", urlProjectsView(q.ID), v.Value)

cmd/web-app/handlers/users.go

@@ -102,7 +102,7 @@ func (h *Users) Index(ctx context.Context, w http.ResponseWriter, r *http.Reques
 			var v datatable.ColumnValue
 			switch col.Field {
 			case "id":
-				v.Value = fmt.Sprintf("%d", q.ID)
+				v.Value = fmt.Sprintf("%s", q.ID)
 			case "name":
 				if strings.TrimSpace(q.Name) == "" {
 					v.Value = q.Email

internal/geonames/geonames.go

@@ -5,11 +5,16 @@ import (
 	"bufio"
 	"bytes"
 	"context"
+	"crypto/md5"
 	"encoding/csv"
 	"fmt"
 	"io"
+	"net/http"
+	"os"
+	"path/filepath"
 	"strconv"
 	"strings"
+	"time"
 
 	"geeks-accelerator/oss/saas-starter-kit/internal/platform/web/webcontext"
 
@@ -189,53 +194,72 @@ func (repo *Repository) FindGeonameRegions(ctx context.Context, orderBy, where s
 	return resp, nil
 }
 
-// LoadGeonames enables streaming retrieval of GeoNames. The downloaded results
+// GetGeonameCountry downloads geoname data for the country.
-// will be written to the interface{} resultReceiver channel enabling processing the results while
+// Parses data and returns slice of Geoname
-// they're still being fetched. After all pages have been processed the channel is closed.
+func (repo *Repository) GetGeonameCountry(ctx context.Context, country string) ([]Geoname, error) {
-// Possible types sent to the channel are limited to:
+	res := make([]Geoname, 0)
-// 		- error
+	var err error
-//		- GeoName
+	var resp *http.Response
-func (repo *Repository) LoadGeonames(ctx context.Context, rr chan<- interface{}, countries ...string) {
-	defer close(rr)
 
-	if len(countries) == 0 {
-		countries = ValidGeonameCountries(ctx)
-	}
-
-	for _, country := range countries {
-		loadGeonameCountry(ctx, rr, country)
-	}
-}
-
-// loadGeonameCountry enables streaming retrieval of GeoNames. The downloaded results
-// will be written to the interface{} resultReceiver channel enabling processing the results while
-// they're still being fetched.
-// Possible types sent to the channel are limited to:
-// 		- error
-//		- GeoName
-func loadGeonameCountry(ctx context.Context, rr chan<- interface{}, country string) {
 	u := fmt.Sprintf("http://download.geonames.org/export/zip/%s.zip", country)
-	resp, err := pester.Get(u)
-	if err != nil {
-		rr <- errors.WithMessagef(err, "Failed to read countries from '%s'", u)
-		return
-	}
-	defer resp.Body.Close()
 
-	br := bufio.NewReader(resp.Body)
+	h := fmt.Sprintf("%x", md5.Sum([]byte(u)))
+	cp := filepath.Join(os.TempDir(), h+".zip")
+
+	if _, err := os.Stat(cp); err != nil {
+		resp, err = pester.Get(u)
+		if err != nil {
+			// Add re-try three times after failing first time
+			// This reduces the risk when network is lagy, we still have chance to re-try.
+			for i := 0; i < 3; i++ {
+				resp, err = pester.Get(u)
+				if err == nil {
+					break
+				}
+				time.Sleep(time.Second * 1)
+			}
+			if err != nil {
+				err = errors.WithMessagef(err, "Failed to read countries from '%s'", u)
+				return res, err
+			}
+		}
+		defer resp.Body.Close()
+
+		// Create the file
+		out, err := os.Create(cp)
+		if err != nil {
+			return nil, err
+		}
+		defer out.Close()
+
+		// Write the body to file
+		_, err = io.Copy(out, resp.Body)
+		if err != nil {
+			return nil, err
+		}
+
+		out.Close()
+	}
+
+	f, err := os.Open(cp)
+	if err != nil {
+		return nil, err
+	}
+	defer f.Close()
+	br := bufio.NewReader(f)
 
 	buff := bytes.NewBuffer([]byte{})
 	size, err := io.Copy(buff, br)
 	if err != nil {
-		rr <- errors.WithStack(err)
+		err = errors.WithStack(err)
-		return
+		return res, err
 	}
 
 	b := bytes.NewReader(buff.Bytes())
 	zr, err := zip.NewReader(b, size)
 	if err != nil {
-		rr <- errors.WithStack(err)
+		err = errors.WithStack(err)
-		return
+		return res, err
 	}
 
 	for _, f := range zr.File {
@@ -245,8 +269,8 @@ func loadGeonameCountry(ctx context.Context, rr chan<- interface{}, country stri
 
 		fh, err := f.Open()
 		if err != nil {
-			rr <- errors.WithStack(err)
+			err = errors.WithStack(err)
-			return
+			return res, err
 		}
 
 		scanner := bufio.NewScanner(fh)
@@ -264,27 +288,12 @@ func loadGeonameCountry(ctx context.Context, rr chan<- interface{}, country stri
 
 			lines, err := r.ReadAll()
 			if err != nil {
-				rr <- errors.WithStack(err)
+				err = errors.WithStack(err)
 				continue
 			}
 
 			for _, row := range lines {
 
-				/*
-					fmt.Println("CountryCode: row[0]", row[0])
-					fmt.Println("PostalCode: row[1]", row[1])
-					fmt.Println("PlaceName: row[2]", row[2])
-					fmt.Println("StateName: row[3]", row[3])
-					fmt.Println("StateCode : row[4]", row[4])
-					fmt.Println("CountyName: row[5]", row[5])
-					fmt.Println("CountyCode : row[6]", row[6])
-					fmt.Println("CommunityName: row[7]", row[7])
-					fmt.Println("CommunityCode: row[8]", row[8])
-					fmt.Println("Latitude: row[9]", row[9])
-					fmt.Println("Longitude: row[10]", row[10])
-					fmt.Println("Accuracy: row[11]", row[11])
-				*/
-
 				gn := Geoname{
 					CountryCode:   row[0],
 					PostalCode:    row[1],
@@ -299,30 +308,32 @@ func loadGeonameCountry(ctx context.Context, rr chan<- interface{}, country stri
 				if row[9] != "" {
 					gn.Latitude, err = decimal.NewFromString(row[9])
 					if err != nil {
-						rr <- errors.WithStack(err)
+						err = errors.WithStack(err)
 					}
 				}
 
 				if row[10] != "" {
 					gn.Longitude, err = decimal.NewFromString(row[10])
 					if err != nil {
-						rr <- errors.WithStack(err)
+						err = errors.WithStack(err)
 					}
 				}
 
 				if row[11] != "" {
 					gn.Accuracy, err = strconv.Atoi(row[11])
 					if err != nil {
-						rr <- errors.WithStack(err)
+						err = errors.WithStack(err)
 					}
 				}
 
-				rr <- gn
+				res = append(res, gn)
 			}
 		}
 
 		if err := scanner.Err(); err != nil {
-			rr <- errors.WithStack(err)
+			err = errors.WithStack(err)
 		}
 	}
+
+	return res, err
 }

internal/mid/saas-swagger/example/main.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"context"
+	"geeks-accelerator/oss/saas-starter-kit/internal/platform/web/webcontext"
 	"log"
 	"net/http"
 	"os"
@@ -135,7 +136,7 @@ func main() {
 func API(shutdown chan os.Signal, log *log.Logger) http.Handler {
 
 	// Construct the web.App which holds all routes as well as common Middleware.
-	app := web.NewApp(shutdown, log, mid.Trace(), mid.Logger(log), mid.Errors(log), mid.Metrics(), mid.Panics())
+	app := web.NewApp(shutdown, log, webcontext.Env_Dev, mid.Logger(log))
 
 	app.Handle("GET", "/swagger/", saasSwagger.WrapHandler)
 	app.Handle("GET", "/swagger/*", saasSwagger.WrapHandler)

internal/mid/saas-swagger/swagger_test.go

@@ -9,6 +9,7 @@ import (
 
 	_ "geeks-accelerator/oss/saas-starter-kit/internal/mid/saas-swagger/example/docs"
 	"geeks-accelerator/oss/saas-starter-kit/internal/platform/web"
+	"geeks-accelerator/oss/saas-starter-kit/internal/platform/web/webcontext"
 	"github.com/stretchr/testify/assert"
 )
 
@@ -17,7 +18,7 @@ func TestWrapHandler(t *testing.T) {
 	log := log.New(os.Stdout, "", log.LstdFlags|log.Lmicroseconds|log.Lshortfile)
 	log.SetOutput(ioutil.Discard)
 
-	app := web.NewApp(nil, log)
+	app := web.NewApp(nil, log, webcontext.Env_Dev)
 	app.Handle("GET", "/swagger/*", WrapHandler)
 
 	w1 := performRequest("GET", "/swagger/index.html", app)

internal/platform/logger/log.go

@@ -3,12 +3,13 @@ package logger
 import (
 	"context"
 	"fmt"
-	"geeks-accelerator/oss/saas-starter-kit/internal/platform/web"
+
+	"geeks-accelerator/oss/saas-starter-kit/internal/platform/web/webcontext"
 )
 
 // WithContext manual injects context values to log message including Trace ID
 func WithContext(ctx context.Context, msg string) string {
-	v, ok := ctx.Value(web.KeyValues).(*web.Values)
+	v, ok := ctx.Value(webcontext.KeyValues).(*webcontext.Values)
 	if !ok {
 		return msg
 	}

internal/schema/migrations.go

@@ -7,9 +7,10 @@ import (
 	"encoding/csv"
 	"log"
 	"strings"
+	"time"
+	"fmt"
 
 	"geeks-accelerator/oss/saas-starter-kit/internal/geonames"
-
 	"github.com/geeks-accelerator/sqlxmigrate"
 	"github.com/jmoiron/sqlx"
 	_ "github.com/lib/pq"
@@ -21,6 +22,7 @@ import (
 // migration already exists in the migrations table it will be skipped.
 func migrationList(ctx context.Context, db *sqlx.DB, log *log.Logger, isUnittest bool) []*sqlxmigrate.Migration {
 	geoRepo := geonames.NewRepository(db)
+
 	return []*sqlxmigrate.Migration{
 		// Create table users.
 		{
@@ -215,7 +217,7 @@ func migrationList(ctx context.Context, db *sqlx.DB, log *log.Logger, isUnittest
 		},
 		// Load new geonames table.
 		{
-			ID: "20190731-02h",
+			ID: "20190731-02l",
 			Migrate: func(tx *sql.Tx) error {
 
 				schemas := []string{
@@ -242,33 +244,91 @@ func migrationList(ctx context.Context, db *sqlx.DB, log *log.Logger, isUnittest
 					}
 				}
 
-				q := "insert into geonames " +
+				countries := geonames.ValidGeonameCountries(ctx)
-					"(country_code,postal_code,place_name,state_name,state_code,county_name,county_code,community_name,community_code,latitude,longitude,accuracy) " +
+				if isUnittest {
-					"values(?,?,?,?,?,?,?,?,?,?,?,?)"
+					countries = []string{"US"}
-				q = db.Rebind(q)
-				stmt, err := db.Prepare(q)
-				if err != nil {
-					return errors.WithMessagef(err, "Failed to prepare sql query '%s'", q)
 				}
 
-				if isUnittest {
+				ncol := 12
+				fn := func(geoNames []geonames.Geoname) error {
+					valueStrings := make([]string, 0, len(geoNames))
+					valueArgs := make([]interface{}, 0, len(geoNames)*ncol)
+					for _, geoname := range geoNames {
+						valueStrings = append(valueStrings, "(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
 
-				} else {
+						valueArgs = append(valueArgs, geoname.CountryCode)
-					resChan := make(chan interface{})
+						valueArgs = append(valueArgs, geoname.PostalCode)
-					go geoRepo.LoadGeonames(ctx, resChan)
+						valueArgs = append(valueArgs, geoname.PlaceName)
 
-					for r := range resChan {
+						valueArgs = append(valueArgs, geoname.StateName)
-						switch v := r.(type) {
+						valueArgs = append(valueArgs, geoname.StateCode)
-						case geonames.Geoname:
+						valueArgs = append(valueArgs, geoname.CountyName)
-							_, err = stmt.Exec(v.CountryCode, v.PostalCode, v.PlaceName, v.StateName, v.StateCode, v.CountyName, v.CountyCode, v.CommunityName, v.CommunityCode, v.Latitude, v.Longitude, v.Accuracy)
+
+						valueArgs = append(valueArgs, geoname.CountyCode)
+						valueArgs = append(valueArgs, geoname.CommunityName)
+						valueArgs = append(valueArgs, geoname.CommunityCode)
+
+						valueArgs = append(valueArgs, geoname.Latitude)
+						valueArgs = append(valueArgs, geoname.Longitude)
+						valueArgs = append(valueArgs, geoname.Accuracy)
+					}
+					insertStmt := fmt.Sprintf("insert into geonames "+
+						"(country_code,postal_code,place_name,state_name,state_code,county_name,county_code,community_name,community_code,latitude,longitude,accuracy) "+
+						"VALUES %s", strings.Join(valueStrings, ","))
+					insertStmt = db.Rebind(insertStmt)
+
+					stmt, err := db.Prepare(insertStmt)
+					if err != nil {
+						return errors.WithMessagef(err, "Failed to prepare sql query '%s'", insertStmt)
+					}
+
+					_, err = stmt.Exec(valueArgs...)
+					return err
+				}
+				start := time.Now()
+				for _, country := range countries {
+					//fmt.Println("LoadGeonames: start country: ", country)
+					v, err := geoRepo.GetGeonameCountry(context.Background(), country)
+					if err != nil {
+						return errors.WithStack(err)
+					}
+					//fmt.Println("Geoname records: ", len(v))
+					// Max argument values of Postgres is about 54460. So the batch size for bulk insert is selected 4500*12 (ncol)
+					batch := 4500
+					n := len(v) / batch
+
+					//fmt.Println("Number of batch: ", n)
+
+					if n == 0 {
+						err := fn(v)
+						if err != nil {
+							return errors.WithStack(err)
+						}
+					} else {
+						for i := 0; i < n; i++ {
+							vn := v[i*batch : (i+1)*batch]
+							err := fn(vn)
+							if err != nil {
+								return errors.WithStack(err)
+							}
+							if n > 0 && n%25 == 0 {
+								time.Sleep(200)
+							}
+						}
+						if len(v)%batch > 0 {
+							fmt.Println("Remain part: ", len(v)-n*batch)
+							vn := v[n*batch:]
+							err := fn(vn)
 							if err != nil {
 								return errors.WithStack(err)
 							}
-						case error:
-							return v
 						}
 					}
+
+					//fmt.Println("Insert Geoname took: ", time.Since(start))
+					//fmt.Println("LoadGeonames: end country: ", country)
 				}
+				log.Println("Total Geonames population took: ", time.Since(start))
 
 				queries := []string{
 					`create index idx_geonames_country_code on geonames (country_code)`,

tools/devops/README.md

@@ -26,7 +26,7 @@ in other configuration files. And since this project is open-source, we wanted t
 
 If you don't have an AWS account, signup for one now and then proceed with the deployment setup. 
 
-We assume that if you are deploying the SaaS Stater Kit, you are starting from scratch with no existing dependencies. 
+We assume that if you are deploying the SaaS Starter Kit, you are starting from scratch with no existing dependencies. 
 This however, excludes any domain names that you would like to use for resolving your services publicly. To use any 
 pre-purchased domain names, make sure they are added to Route 53 in the AWS account. Or you can let the deploy script 
 create a new zone is Route 53 and update the DNS for the domain name when your ready to make the transition. It is