diff --git a/README.md b/README.md index 8be7af6..9163846 100644 --- a/README.md +++ b/README.md @@ -422,6 +422,181 @@ For additional details refer to https://jmoiron.github.io/sqlx/#bindvars +## Contribute + + +### Development Notes regarding this copy of the project. + +#### GitLab CI / CD + +_Shared Runners_ have been disabled for this project. Since the project is open source and we wanted to avoid putting +our AWS credentials as pipeline variables. Instead we have deployed our own set of autoscaling runnings on AWS EC2 that +utilize AWS IAM Roles. All other configure is defined for CI/CD is defined in `.gitlab-ci.yaml`. + +Below outlines the basic steps to setup [Autoscaling GitLab Runner on AWS](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/). + +1. Define an [AWS IAM Role](https://console.aws.amazon.com/iam/home?region=us-west-2#/roles$new?step=type) that will be +attached to the GitLab Runner instances. The role will need permission to scale (EC2), update the cache (via S3) and +perform the project specific deployment commands. + ``` + Trusted Entity: AWS Service + Service that will use this role: EC2 + Attach permissions policies: AmazonEC2FullAccess, AmazonS3FullAccess, saas-starter-kit-deploy + Role Name: SaasStarterKitEc2RoleForGitLabRunner + Role Description: Allows GitLab runners hosted on EC2 instances to call AWS services on your behalf. + ``` + +2. Launch a new [AWS EC2 Instance](https://us-west-2.console.aws.amazon.com/ec2/v2/home?region=us-west-2#LaunchInstanceWizard). +`GitLab Runner` will be installed on this instance and will serve as the bastion that spawns new instances. This +instance will be a dedicated host since we need it always up and running, thus it will be the standard costs apply. + + Note: This doesn't have to be a powerful machine since it will not run any jobs itself, a t2.micro instance will do. + ``` + Amazon Machine Image (AMI): Amazon Linux AMI 2018.03.0 (HVM), SSD Volume Type - ami-0f2176987ee50226e + Instance Type: t2.micro + ``` + +3. Configure Instance Details. + + Note: Don't forget to select the IAM Role _SaasStarterKitEc2RoleForGitLabRunner_ + ``` + Number of instances: 1 + Network: default VPC + Subnet: no Preference + Auto-assign Public IP: Use subnet setting (Enable) + Placement Group: not checked/disabled + Capacity Reservation: Open + IAM Role: SaasStarterKitEc2RoleForGitLabRunner + Shutdown behavior: Stop + Enable termination project: checked/enabled + Monitoring: not checked/disabled + Tenancy: Shared + Elastic Interence: not checked/disabled + T2/T3 Unlimited: not checked/disabled + Advanced Details: none + ``` + +4. Add Storage. Increase the volume size for the root device to 100 GiB + ``` + Volume Type | Device | Size (GiB) | Volume Type + Root | /dev/xvda | 100 | General Purpose SSD (gp2) + ``` + +5. Add Tags. + ``` + Name: gitlab-runner + ``` + +6. Configure Security Group. Create a new security group with the following details: + ``` + Name: gitlab-runner + Description: Gitlab runners for running CICD. + Rules: + Type | Protocol | Port Range | Source | Description + Custom TCP | TCP | 2376 | Anywhere | Gitlab runner for Docker Machine to communicate with Docker daemon. + SSH | TCP | 22 | My IP | SSH access for setup. + ``` + +7. Review and Launch instance. Select an existing key pair or create a new one. This will be used to SSH into the + instance for additional configuration. + +8. SSH into the newly created instance. + + ```bash + ssh -i ~/saas-starter-kit-uswest2-gitlabrunner.pem ec2-user@ec2-52-36-105-172.us-west-2.compute.amazonaws.com + ``` + Note: If you get the error `Permissions 0666 are too open`, then you will need to `chmod 400 FILENAME` + +9. Install GitLab Runner from the [official GitLab repository](https://docs.gitlab.com/runner/install/linux-repository.html) + ```bash + curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh | sudo bash + sudo yum install gitlab-runner + ``` + +10. [Install Docker Community Edition](https://docs.docker.com/install/). + ```bash + sudo yum install docker + ``` + +11. [Install Docker Machine](https://docs.docker.com/machine/install-machine/). + ```bash + base=https://github.com/docker/machine/releases/download/v0.16.0 && + curl -L $base/docker-machine-$(uname -s)-$(uname -m) >/tmp/docker-machine && + sudo install /tmp/docker-machine /usr/local/bin/docker-machine + ``` + +12. [Register the runner](https://docs.gitlab.com/runner/register/index.html). + ```bash + sudo gitlab-runner register + ``` + Notes: + * When asked for gitlab-ci tags, enter `master,dev,dev-*` + * This will limit commits to the master or dev branches from triggering the pipeline to run. This includes a + wildcard for any branch named with the prefix `dev-`. + * When asked the executor type, enter `docker+machine` + * When asked for the default Docker image, enter `golang:alpine3.9` + +13. [Configuring the GitLab Runner](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/#configuring-the-gitlab-runner) + + ```bash + sudo vim /etc/gitlab-runner/config.toml + ``` + + Update the `[runners.docker]` configuration section in `config.toml` to match the example below replacing the + obvious placeholder `XXXXX` with the relevant value. + ```yaml + [runners.docker] + tls_verify = false + image = "golang:alpine3.9" + privileged = true + disable_entrypoint_overwrite = false + oom_kill_disable = false + disable_cache = true + volumes = ["/cache"] + shm_size = 0 + [runners.cache] + Type = "s3" + Shared = true + [runners.cache.s3] + ServerAddress = "s3.us-west-2.amazonaws.com" + BucketName = "XXXXX" + BucketLocation = "us-west-2" + [runners.machine] + IdleCount = 0 + IdleTime = 1800 + MachineDriver = "amazonec2" + MachineName = "gitlab-runner-machine-%s" + MachineOptions = [ + "amazonec2-iam-instance-profile=SaasStarterKitEc2RoleForGitLabRunner", + "amazonec2-region=us-west-2", + "amazonec2-vpc-id=XXXXX", + "amazonec2-subnet-id=XXXXX", + "amazonec2-zone=d", + "amazonec2-use-private-address=true", + "amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true", + "amazonec2-security-group=gitlab-runner", + "amazonec2-instance-type=t2.large" + ] + ``` + + You will need use the same VPC subnet and availability zone as the instance launched in step 2. We are using AWS + region `us-west-2`. The _ServerAddress_ for S3 will need to be updated if the region is changed. For `us-east-1` the + _ServerAddress_ is `s3.amazonaws.com`. Under MachineOptions you can add anything that the [AWS Docker Machine](https://docs.docker.com/machine/drivers/aws/#options) + driver supports. + + Below are some example values for the placeholders to ensure for format of your values are correct. + ```yaml + BucketName = saas-starter-kit-usw + amazonec2-vpc-id=vpc-5f43f027 + amazonec2-subnet-id=subnet-693d3110 + amazonec2-zone=a + ``` + + Once complete, restart the runner. + ```bash + sudo gitlab-runner restart + ``` + ## What's Next We are in the process of writing more documentation about this code. We welcome you to make enhancements to this documentation or just send us your feedback and suggestions ; ) diff --git a/cmd/web-api/handlers/check.go b/cmd/web-api/handlers/check.go index a4d03d3..f4da2ff 100644 --- a/cmd/web-api/handlers/check.go +++ b/cmd/web-api/handlers/check.go @@ -47,5 +47,5 @@ func (c *Check) Ping(ctx context.Context, w http.ResponseWriter, r *http.Request status := "pong" - return web.RespondJson(ctx, w, status, http.StatusOK) + return web.RespondText(ctx, w, status, http.StatusOK) } diff --git a/cmd/web-app/handlers/check.go b/cmd/web-app/handlers/check.go index ff30b4d..c4d5ee4 100644 --- a/cmd/web-app/handlers/check.go +++ b/cmd/web-app/handlers/check.go @@ -40,3 +40,11 @@ func (c *Check) Health(ctx context.Context, w http.ResponseWriter, r *http.Reque return c.Renderer.Render(ctx, w, r, baseLayoutTmpl, "health.tmpl", web.MIMETextHTMLCharsetUTF8, http.StatusOK, data) } + +// Ping validates the service is ready to accept requests. +func (c *Check) Ping(ctx context.Context, w http.ResponseWriter, r *http.Request, params map[string]string) error { + + status := "pong" + + return web.RespondText(ctx, w, status, http.StatusOK) +} diff --git a/cmd/web-app/handlers/routes.go b/cmd/web-app/handlers/routes.go index 9175f43..c340552 100644 --- a/cmd/web-app/handlers/routes.go +++ b/cmd/web-app/handlers/routes.go @@ -6,15 +6,15 @@ import ( "os" "geeks-accelerator/oss/saas-starter-kit/internal/mid" - "geeks-accelerator/oss/saas-starter-kit/internal/platform/auth" "geeks-accelerator/oss/saas-starter-kit/internal/platform/web" "github.com/jmoiron/sqlx" + "gopkg.in/DataDog/dd-trace-go.v1/contrib/go-redis/redis" ) const baseLayoutTmpl = "base.tmpl" // API returns a handler for a set of routes. -func APP(shutdown chan os.Signal, log *log.Logger, staticDir, templateDir string, masterDB *sqlx.DB, authenticator *auth.Authenticator, renderer web.Renderer, globalMids ...web.Middleware) http.Handler { +func APP(shutdown chan os.Signal, log *log.Logger, staticDir, templateDir string, masterDB *sqlx.DB, redis *redis.Client, renderer web.Renderer, globalMids ...web.Middleware) http.Handler { // Define base middlewares applied to all requests. middlewares := []web.Middleware{ @@ -32,6 +32,7 @@ func APP(shutdown chan os.Signal, log *log.Logger, staticDir, templateDir string // Register health check endpoint. This route is not authenticated. check := Check{ MasterDB: masterDB, + Redis: redis, Renderer: renderer, } app.Handle("GET", "/v1/health", check.Health) diff --git a/cmd/web-app/main.go b/cmd/web-app/main.go index 8af7146..46d448b 100644 --- a/cmd/web-app/main.go +++ b/cmd/web-app/main.go @@ -2,10 +2,13 @@ package main import ( "context" + "crypto/tls" "encoding/json" "expvar" "fmt" "geeks-accelerator/oss/saas-starter-kit/internal/mid" + "golang.org/x/crypto/acme" + "golang.org/x/crypto/acme/autocert" "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer" "html/template" "log" @@ -67,15 +70,18 @@ func main() { Host string `default:"" envconfig:"HOST"` ReadTimeout time.Duration `default:"5s" envconfig:"READ_TIMEOUT"` WriteTimeout time.Duration `default:"5s" envconfig:"WRITE_TIMEOUT"` + DisableHTTP2 bool `default:"false" envconfig:"DISABLE_HTTP2"` } - App struct { - Name string `default:"web-app" envconfig:"NAME"` - BaseUrl string `default:"" envconfig:"BASE_URL"` - TemplateDir string `default:"./templates" envconfig:"TEMPLATE_DIR"` - StaticDir string `default:"./static" envconfig:"STATIC_DIR"` + Service struct { + Name string `default:"web-app" envconfig:"NAME"` + Project string `default:"" envconfig:"PROJECT"` + BaseUrl string `default:"" envconfig:"BASE_URL" example:"http://eproc.tech"` + HostNames []string `envconfig:"HOST_NAMES" example:"www.eproc.tech"` + EnableHTTPS bool `default:"false" envconfig:"ENABLE_HTTPS"` + TemplateDir string `default:"./templates" envconfig:"TEMPLATE_DIR"` + StaticDir string `default:"./static" envconfig:"STATIC_DIR"` StaticS3 struct { S3Enabled bool `envconfig:"ENABLED"` - S3Bucket string `envconfig:"S3_BUCKET"` S3KeyPrefix string `default:"public/web_app/static" envconfig:"KEY_PREFIX"` CloudFrontEnabled bool `envconfig:"CLOUDFRONT_ENABLED"` ImgResizeEnabled bool `envconfig:"IMG_RESIZE_ENABLED"` @@ -104,9 +110,12 @@ func main() { AnalyticsRate float64 `default:"0.10" envconfig:"ANALYTICS_RATE"` } Aws struct { - AccessKeyID string `envconfig:"AWS_ACCESS_KEY_ID" required:"true"` // WEB_API_AWS_AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY_ID - SecretAccessKey string `envconfig:"AWS_SECRET_ACCESS_KEY" required:"true" json:"-"` // don't print - Region string `default:"us-east-1" envconfig:"AWS_REGION"` + AccessKeyID string `envconfig:"AWS_ACCESS_KEY_ID"` // WEB_API_AWS_AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY_ID + SecretAccessKey string `envconfig:"AWS_SECRET_ACCESS_KEY" json:"-"` // don't print + Region string `default:"us-east-1" envconfig:"AWS_REGION"` + S3BucketPrivate string `envconfig:"S3_BUCKET_PRIVATE"` + S3BucketPublic string `envconfig:"S3_BUCKET_PUBLIC"` + SecretsManagerConfigPrefix string `default:"" envconfig:"SECRETS_MANAGER_CONFIG_PREFIX"` // Get an AWS session from an implicit source if no explicit // configuration is provided. This is useful for taking advantage of @@ -114,8 +123,7 @@ func main() { UseRole bool `envconfig:"AWS_USE_ROLE"` } Auth struct { - UseAwsSecretManager bool `default:false envconfig:"USE_AWS_SECRET_MANAGER"` - AwsSecretID string `default:"auth-secret-key" envconfig:"AWS_SECRET_ID"` + UseAwsSecretManager bool `default:"false" envconfig:"USE_AWS_SECRET_MANAGER"` KeyExpiration time.Duration `default:"3600s" envconfig:"KEY_EXPIRATION"` } BuildInfo struct { @@ -130,7 +138,6 @@ func main() { CiPipelineId string `envconfig:"CI_COMMIT_PIPELINE_ID"` CiPipelineUrl string `envconfig:"CI_COMMIT_PIPELINE_URL"` } - CMD string `envconfig:"CMD"` } // For additional details refer to https://github.com/kelseyhightower/envconfig @@ -154,8 +161,20 @@ func main() { cfg.Aws.SecretAccessKey = "" } + // Set the default AWS Secrets Manager prefix used for name to store config files that will be persisted across + // deployments and distributed to each instance of the service running. + if cfg.Aws.SecretsManagerConfigPrefix == "" { + var pts []string + if cfg.Service.Project != "" { + pts = append(pts, cfg.Service.Project) + } + pts = append(pts, cfg.Env, cfg.Service.Name) + + cfg.Aws.SecretsManagerConfigPrefix = filepath.Join(pts...) + } + // If base URL is empty, set the default value from the HTTP Host - if cfg.App.BaseUrl == "" { + if cfg.Service.BaseUrl == "" { baseUrl := cfg.HTTP.Host if !strings.HasPrefix(baseUrl, "http") { if strings.HasPrefix(baseUrl, "0.0.0.0:") { @@ -167,15 +186,37 @@ func main() { } baseUrl = "http://" + baseUrl } - cfg.App.BaseUrl = baseUrl + cfg.Service.BaseUrl = baseUrl + } + + // When HTTPS is not specifically enabled, but an HTTP host is set, enable HTTPS. + if !cfg.Service.EnableHTTPS && cfg.HTTPS.Host != "" { + cfg.Service.EnableHTTPS = true + } + + // Determine the primary host by parsing host from the base app URL. + baseSiteUrl, err := url.Parse(cfg.Service.BaseUrl) + if err != nil { + log.Fatalf("main : Parse service base URL : %s : %+v", cfg.Service.BaseUrl, err) + } + + // Drop any ports from the base app URL. + var primaryServiceHost string + if strings.Contains(baseSiteUrl.Host, ":") { + primaryServiceHost, _, err = net.SplitHostPort(baseSiteUrl.Host) + if err != nil { + log.Fatalf("main : SplitHostPort : %s : %+v", baseSiteUrl.Host, err) + } + } else { + primaryServiceHost = baseSiteUrl.Host } // ========================================================================= - // Log App Info + // Log Service Info // Print the build version for our logs. Also expose it under /debug/vars. expvar.NewString("build").Set(build) - log.Printf("main : Started : Application Initializing version %q", build) + log.Printf("main : Started : Service Initializing version %q", build) defer log.Println("main : Completed") // Print the config for our logs. It's important to any credentials in the config @@ -197,11 +238,20 @@ func main() { // configuration is provided. This is useful for taking advantage of // EC2/ECS instance roles. awsSession = session.Must(session.NewSession()) - } else { + + log.Printf("main : AWS : Using role.\n") + + } else if cfg.Aws.AccessKeyID != "" { creds := credentials.NewStaticCredentials(cfg.Aws.AccessKeyID, cfg.Aws.SecretAccessKey, "") awsSession = session.New(&aws.Config{Region: aws.String(cfg.Aws.Region), Credentials: creds}) + + log.Printf("main : AWS : Using static credentials\n") + } + + // Wrap the AWS session to enable tracing. + if awsSession != nil { + awsSession = awstrace.WrapSession(awsSession) } - awsSession = awstrace.WrapSession(awsSession) // ========================================================================= // Start Redis @@ -274,49 +324,35 @@ func main() { defer masterDb.Close() // ========================================================================= - // Deploy - switch cfg.CMD { - case "sync-static": - // sync static files to S3 - if cfg.App.StaticS3.S3Enabled || cfg.App.StaticS3.CloudFrontEnabled { - err = devops.SyncS3StaticFiles(awsSession, cfg.App.StaticS3.S3Bucket, cfg.App.StaticS3.S3KeyPrefix, cfg.App.StaticDir) - if err != nil { - log.Fatalf("main : deploy : %+v", err) - } - } - return + // Load middlewares that need to be configured specific for the service. + + var serviceMiddlewares []web.Middleware + + // Init redirect middleware to ensure all requests go to the primary domain contained in the base URL. + if primaryServiceHost != "127.0.0.0" && primaryServiceHost != "localhost" { + redirect := mid.DomainNameRedirect(mid.DomainNameRedirectConfig{ + RedirectConfig: mid.RedirectConfig{ + Code: http.StatusMovedPermanently, + Skipper: func(ctx context.Context, w http.ResponseWriter, r *http.Request, params map[string]string) bool { + if r.URL.Path == "/ping" { + return true + } + return false + }, + }, + DomainName: primaryServiceHost, + HTTPSEnabled: cfg.Service.EnableHTTPS, + }) + serviceMiddlewares = append(serviceMiddlewares, redirect) } - // ========================================================================= - // Init redirect middleware to ensure all requests go to the primary domain. - - baseSiteUrl, err := url.Parse(cfg.App.BaseUrl) - if err != nil { - log.Fatalf("main : Parse App Base URL : %s : %+v", cfg.App.BaseUrl, err) - } - - var primaryDomain string - if strings.Contains(baseSiteUrl.Host, ":") { - primaryDomain, _, err = net.SplitHostPort(baseSiteUrl.Host) - if err != nil { - log.Fatalf("main : SplitHostPort : %s : %+v", baseSiteUrl.Host, err) - } - } else { - primaryDomain = baseSiteUrl.Host - } - - redirect := mid.DomainNameRedirect(mid.DomainNameRedirectConfig{ - DomainName: primaryDomain, - HTTPSEnabled: (cfg.HTTPS.Host != ""), - }) - // ========================================================================= // URL Formatter // s3UrlFormatter is a help function used by to convert an s3 key to // a publicly available image URL. var staticS3UrlFormatter func(string) string - if cfg.App.StaticS3.S3Enabled || cfg.App.StaticS3.CloudFrontEnabled || cfg.App.StaticS3.ImgResizeEnabled { - s3UrlFormatter, err := devops.S3UrlFormatter(awsSession, cfg.App.StaticS3.S3Bucket, cfg.App.StaticS3.S3KeyPrefix, cfg.App.StaticS3.CloudFrontEnabled) + if cfg.Service.StaticS3.S3Enabled || cfg.Service.StaticS3.CloudFrontEnabled || cfg.Service.StaticS3.ImgResizeEnabled { + s3UrlFormatter, err := devops.S3UrlFormatter(awsSession, cfg.Aws.S3BucketPublic, cfg.Service.StaticS3.S3KeyPrefix, cfg.Service.StaticS3.CloudFrontEnabled) if err != nil { log.Fatalf("main : S3UrlFormatter failed : %+v", err) } @@ -325,10 +361,20 @@ func main() { // When the path starts with a forward slash its referencing a local file, // make sure the static file prefix is included if strings.HasPrefix(p, "/") { - p = filepath.Join(cfg.App.StaticS3.S3KeyPrefix, p) + p = filepath.Join(cfg.Service.StaticS3.S3KeyPrefix, p) } return s3UrlFormatter(p) } + } else { + baseUrl, err := url.Parse(cfg.Service.BaseUrl) + if err != nil { + log.Fatalf("main : url Parse(%s) : %+v", cfg.Service.BaseUrl, err) + } + + staticS3UrlFormatter = func(p string) string { + baseUrl.Path = p + return baseUrl.String() + } } // staticUrlFormatter is a help function used by template functions defined below. @@ -336,12 +382,12 @@ func main() { // templates should be updated to use a fully qualified URL for either the public file on S3 // on from the cloudfront distribution. var staticUrlFormatter func(string) string - if cfg.App.StaticS3.S3Enabled || cfg.App.StaticS3.CloudFrontEnabled { + if cfg.Service.StaticS3.S3Enabled || cfg.Service.StaticS3.CloudFrontEnabled { staticUrlFormatter = staticS3UrlFormatter } else { - baseUrl, err := url.Parse(cfg.App.BaseUrl) + baseUrl, err := url.Parse(cfg.Service.BaseUrl) if err != nil { - log.Fatalf("main : url Parse(%s) : %+v", cfg.App.BaseUrl, err) + log.Fatalf("main : url Parse(%s) : %+v", cfg.Service.BaseUrl, err) } staticUrlFormatter = func(p string) string { @@ -410,7 +456,7 @@ func main() { "SiteAssetUrl": func(p string) string { var u string if staticUrlFormatter != nil { - u = staticUrlFormatter(filepath.Join(cfg.App.Name, p)) + u = staticUrlFormatter(filepath.Join(cfg.Service.Name, p)) } else { if !strings.HasPrefix(p, "/") { p = "/" + p @@ -425,7 +471,7 @@ func main() { "SiteS3Url": func(p string) string { var u string if staticUrlFormatter != nil { - u = staticUrlFormatter(filepath.Join(cfg.App.Name, p)) + u = staticUrlFormatter(filepath.Join(cfg.Service.Name, p)) } else { u = p } @@ -444,13 +490,13 @@ func main() { // Image Formatter - additional functions exposed to templates for resizing images // to support response web applications. - imgResizeS3KeyPrefix := filepath.Join(cfg.App.StaticS3.S3KeyPrefix, "images/responsive") + imgResizeS3KeyPrefix := filepath.Join(cfg.Service.StaticS3.S3KeyPrefix, "images/responsive") imgSrcAttr := func(ctx context.Context, p string, sizes []int, includeOrig bool) template.HTMLAttr { u := staticUrlFormatter(p) var srcAttr string - if cfg.App.StaticS3.ImgResizeEnabled { - srcAttr, _ = img_resize.S3ImgSrc(ctx, redisClient, staticS3UrlFormatter, awsSession, cfg.App.StaticS3.S3Bucket, imgResizeS3KeyPrefix, u, sizes, includeOrig) + if cfg.Service.StaticS3.ImgResizeEnabled { + srcAttr, _ = img_resize.S3ImgSrc(ctx, redisClient, staticS3UrlFormatter, awsSession, cfg.Aws.S3BucketPublic, imgResizeS3KeyPrefix, u, sizes, includeOrig) } else { srcAttr = fmt.Sprintf("src=\"%s\"", u) } @@ -480,8 +526,8 @@ func main() { } tmplFuncs["S3ImgUrl"] = func(ctx context.Context, p string, size int) string { imgUrl := staticUrlFormatter(p) - if cfg.App.StaticS3.ImgResizeEnabled { - imgUrl, _ = img_resize.S3ImgUrl(ctx, redisClient, staticS3UrlFormatter, awsSession, cfg.App.StaticS3.S3Bucket, imgResizeS3KeyPrefix, imgUrl, size) + if cfg.Service.StaticS3.ImgResizeEnabled { + imgUrl, _ = img_resize.S3ImgUrl(ctx, redisClient, staticS3UrlFormatter, awsSession, cfg.Aws.S3BucketPublic, imgResizeS3KeyPrefix, imgUrl, size) } return imgUrl } @@ -519,7 +565,7 @@ func main() { enableHotReload := cfg.Env == "dev" // Template Renderer used to generate HTML response for web experience. - renderer, err := template_renderer.NewTemplateRenderer(cfg.App.TemplateDir, enableHotReload, gvd, t, eh) + renderer, err := template_renderer.NewTemplateRenderer(cfg.Service.TemplateDir, enableHotReload, gvd, t, eh) if err != nil { log.Fatalf("main : Marshalling Config to JSON : %+v", err) } @@ -538,13 +584,20 @@ func main() { // // /debug/vars - Added to the default mux by the expvars package. // /debug/pprof - Added to the default mux by the net/http/pprof package. - if cfg.App.DebugHost != "" { + if cfg.Service.DebugHost != "" { go func() { - log.Printf("main : Debug Listening %s", cfg.App.DebugHost) - log.Printf("main : Debug Listener closed : %v", http.ListenAndServe(cfg.App.DebugHost, http.DefaultServeMux)) + log.Printf("main : Debug Listening %s", cfg.Service.DebugHost) + log.Printf("main : Debug Listener closed : %v", http.ListenAndServe(cfg.Service.DebugHost, http.DefaultServeMux)) }() } + // ========================================================================= + // ECS Task registration for services that don't use an AWS Elastic Load Balancer. + err = devops.EcsServiceTaskInit(log, awsSession) + if err != nil { + log.Fatalf("main : Ecs Service Task init : %+v", err) + } + // ========================================================================= // Start APP Service @@ -553,23 +606,81 @@ func main() { shutdown := make(chan os.Signal, 1) signal.Notify(shutdown, os.Interrupt, syscall.SIGTERM) - app := http.Server{ - Addr: cfg.HTTP.Host, - Handler: handlers.APP(shutdown, log, cfg.App.StaticDir, cfg.App.TemplateDir, masterDb, nil, renderer, redirect), - ReadTimeout: cfg.HTTP.ReadTimeout, - WriteTimeout: cfg.HTTP.WriteTimeout, - MaxHeaderBytes: 1 << 20, - } - // Make a channel to listen for errors coming from the listener. Use a // buffered channel so the goroutine can exit if we don't collect this error. serverErrors := make(chan error, 1) - // Start the service listening for requests. - go func() { - log.Printf("main : APP Listening %s", cfg.HTTP.Host) - serverErrors <- app.ListenAndServe() - }() + // Make an list of HTTP servers for both HTTP and HTTPS requests. + var httpServers []http.Server + + // Start the HTTP service listening for requests. + if cfg.HTTP.Host != "" { + api := http.Server{ + Addr: cfg.HTTP.Host, + Handler: handlers.APP(shutdown, log, cfg.Service.StaticDir, cfg.Service.TemplateDir, masterDb, redisClient, renderer, serviceMiddlewares...), + ReadTimeout: cfg.HTTP.ReadTimeout, + WriteTimeout: cfg.HTTP.WriteTimeout, + MaxHeaderBytes: 1 << 20, + } + httpServers = append(httpServers, api) + + go func() { + log.Printf("main : APP Listening %s", cfg.HTTP.Host) + serverErrors <- api.ListenAndServe() + }() + } + + // Start the HTTPS service listening for requests with an SSL Cert auto generated with Let's Encrypt. + if cfg.HTTPS.Host != "" { + api := http.Server{ + Addr: cfg.HTTPS.Host, + Handler: handlers.APP(shutdown, log, cfg.Service.StaticDir, cfg.Service.TemplateDir, masterDb, redisClient, renderer, serviceMiddlewares...), + ReadTimeout: cfg.HTTPS.ReadTimeout, + WriteTimeout: cfg.HTTPS.WriteTimeout, + MaxHeaderBytes: 1 << 20, + } + + // Generate a unique list of hostnames. + var hosts []string + if primaryServiceHost != "" { + hosts = append(hosts, primaryServiceHost) + } + for _, h := range cfg.Service.HostNames { + h = strings.TrimSpace(h) + if h != "" && h != primaryServiceHost { + hosts = append(hosts, h) + } + } + + // Enable autocert to store certs via Secret Manager. + secretPrefix := filepath.Join(cfg.Aws.SecretsManagerConfigPrefix, "autocert") + + // Local file cache to reduce requests hitting Secret Manager. + localCache := autocert.DirCache(os.TempDir()) + + cache, err := devops.NewSecretManagerAutocertCache(log, awsSession, secretPrefix, localCache) + if err != nil { + log.Fatalf("main : HTTPS : %+v", err) + } + + m := &autocert.Manager{ + Prompt: autocert.AcceptTOS, + HostPolicy: autocert.HostWhitelist(hosts...), + Cache: cache, + } + api.TLSConfig = &tls.Config{GetCertificate: m.GetCertificate} + api.TLSConfig.NextProtos = append(api.TLSConfig.NextProtos, acme.ALPNProto) + if !cfg.HTTPS.DisableHTTP2 { + api.TLSConfig.NextProtos = append(api.TLSConfig.NextProtos, "h2") + } + + httpServers = append(httpServers, api) + + go func() { + log.Printf("main : APP Listening %s with SSL cert for hosts %s", cfg.HTTPS.Host, strings.Join(hosts, ", ")) + serverErrors <- api.ListenAndServeTLS("", "") + }() + } // ========================================================================= // Shutdown @@ -583,14 +694,19 @@ func main() { log.Printf("main : %v : Start shutdown..", sig) // Create context for Shutdown call. - ctx, cancel := context.WithTimeout(context.Background(), cfg.App.ShutdownTimeout) + ctx, cancel := context.WithTimeout(context.Background(), cfg.Service.ShutdownTimeout) defer cancel() - // Asking listener to shutdown and load shed. - err := app.Shutdown(ctx) - if err != nil { - log.Printf("main : Graceful shutdown did not complete in %v : %v", cfg.App.ShutdownTimeout, err) - err = app.Close() + // Handle closing connections for both possible HTTP servers. + for _, api := range httpServers { + + // Asking listener to shutdown and load shed. + err := api.Shutdown(ctx) + if err != nil { + log.Printf("main : Graceful shutdown did not complete in %v : %v", cfg.Service.ShutdownTimeout, err) + err = api.Close() + } + } // Log the status of this shutdown. diff --git a/internal/platform/web/response.go b/internal/platform/web/response.go index 8964c42..54fe704 100644 --- a/internal/platform/web/response.go +++ b/internal/platform/web/response.go @@ -118,6 +118,15 @@ func RespondErrorStatus(ctx context.Context, w http.ResponseWriter, er error, st return nil } + +// RespondText sends text back to the client as plain text with the specified HTTP status code. +func RespondText(ctx context.Context, w http.ResponseWriter, text string, statusCode int) error { + if err := Respond(ctx, w, []byte(text), statusCode, MIMETextPlainCharsetUTF8); err != nil { + return err + } + return nil +} + // Respond writes the data to the client with the specified HTTP status code and // content type. func Respond(ctx context.Context, w http.ResponseWriter, data []byte, statusCode int, contentType string) error { diff --git a/sample.env_docker_compose b/sample.env_docker_compose index cad73e3..050f763 100644 --- a/sample.env_docker_compose +++ b/sample.env_docker_compose @@ -1,5 +1,5 @@ -AWS_ACCESS_KEY_ID=XXXX -AWS_SECRET_ACCESS_KEY=XXXX -AWS_REGION=us-east-1 -AWS_USE_ROLE=false -DD_API_KEY=XXXX \ No newline at end of file +#AWS_ACCESS_KEY_ID= +#AWS_SECRET_ACCESS_KEY= +#AWS_REGION=us-west-2 +#AWS_USE_ROLE=false +#DD_API_KEY= \ No newline at end of file