Image for post
Image for post

Median filtering can be used to join multiple images to get a single image with all moving objects removed. The simple idea is to compare the pixels at the same location on all images and get the median of RGBA values of the pixels to form a new composite image with all moving objects removed.

The image set provided has to be taken by a camera that did not move during the shoot, since the pixels at the same location on all images have to correspond to the same physical location. Otherwise, the median filter will blend all images together to create a meaningless composite image. You can read more about median filtering on Nikolas Moya’s medium article. …

Image for post
Image for post

Part 1 — Utility Functions

I have been interested in image manipulation for some time. Last year I worked on a small package that does simple image manipulation using Go’s standard image package: github.com/KorayGocmen/image

Let’s look at how to read an image from a provided file path and creating an image object. This is what the structs will look like.

// GrayscaleAverage, GrayscaleLuma, GrayscaleDesaturation
// are used by grayscale to choose between algorithms
const (
GrayscaleAverage = 0
GrayscaleLuma = 1
GrayscaleDesaturation = 2
)

// Pixel is a single pixel in 2d array
type Pixel struct {
R int
G int
B int
A int
}

// Image is the main object that holds information about the
// image file. Also is a wrapper around the decoded image
// from the standard image library.
type Image struct {
Pixels [][]Pixel
Width int
Height int
_Rect image.Rectangle
_Image image.Image …

Image for post
Image for post
Nodejs + Sequelize

After using Sequelize in my Express applications for the past 6 years, this is how I write my migrations and models for all my Nodejs applications.

Install sequelize with npm first and initialize the project with “sequelize init”. That should create the “models”, “migrations” and “seeders” folders as well as create the “config/config.json” file.

In our application, we’ll have users and cards. Each user can have multiple cards.

We need a users table and a cards table in our database and users will have a one-to-many relationship to cards.

You can find how those two models are associated using the Sequelize way…

Image for post
Image for post
Nodejs + Sequelize

After using Sequelize in my Express applications for the past 6 years, I’ve landed on a database file that I’ve been using for all my Nodejs applications.

Install sequelize with npm first and initialize the project with “sequelize init”. That should create the “models”, “migrations” and “seeders” folders as well as create the “config/config.json” file.

In our application, we’ll have users and cards. Each user can have multiple cards.

We need a users table and a cards table in our database and users will have a one-to-many relationship to cards. …

Image for post
Image for post
Timer events in Go

I’ve writing production level Go code for the past 4 years now, something that I’ve been using since the beginning is the timer events in Go.

I use timer events like how I would use cron jobs, to run a function or a script in a certain time interval.

I usually pass the timer interval in a config file to be able to change it quickly without changing the code itself.

I wrote about how I approach configuration in Go projects. You can read more here:

// initVN initializes the refresh timer which will
// refresh all known sites if their version is changed.
func initVN() {
go func() {
every := time.Duration(config.Sites.RefreshEvery) * time.Second
ticker := time.NewTicker(every)
defer ticker.Stop()

for {
select {
case <-ticker.C:
vnSitesRefresh()
}
}
}()
}

// vnSitesRefresh is called every sites refresh timer and
// requests the known sites with versions from VN API.
func vnSitesRefresh() {
logger.Println("Refreshing sites")

// some secret stuff here. …

Image for post
Image for post

The default HTTP client in Go uses the default DNS resolver available on the machine. There are some cases where you might want to use a different DNS resolver.

I want to quickly show how you can change the DNS resolver for the default HTTP client. We are using this same method for our custom DNS resolver in my company: Violetnorth

// Changing DNS resolverpackage main

import (
"context"
"io/ioutil"
"log"
"net"
"net/http"
"time"
)

func main() {
var (
dnsResolverIP = "8.8.8.8:53" // Google DNS resolver.
dnsResolverProto = "udp" // Protocol to use for the DNS resolver
dnsResolverTimeoutMs = 5000 // Timeout (ms) for the DNS resolver (optional)
)

dialer := &net.Dialer{
Resolver: &net.Resolver{
PreferGo: true,
Dial: func(ctx context.Context, network, address string) (net.Conn, error) {
d := net.Dialer{
Timeout: time.Duration(dnsResolverTimeoutMs) * time.Millisecond,
}
return d.DialContext(ctx, dnsResolverProto, dnsResolverIP)
},
},
}

dialContext := func(ctx context.Context, network, addr string) (net.Conn, error) {
return dialer.DialContext(ctx, network, addr)
}

http.DefaultTransport.(*http.Transport).DialContext = dialContext
httpClient := &http.Client{}

// Testing the new HTTP client with the custom DNS resolver.
resp, err := httpClient.Get("https://www.violetnorth.com")
if err != nil {
log.Fatalln(err)
}
defer resp.Body.Close()

body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalln(err)
}

log.Println(string(body)) …

Image for post
Image for post

I’m not a big fan of the built-in `dgram` and `net` libraries in Nodejs, but it’s really easy to create sockets and write networking applications with Nodejs. One of the biggest issues I’m constantly seeing is the lack of cleanup functionalities when using the socket libraries in Nodejs.

This code is from our DNS npm package that we use in Violetnorth

You can find the Violetnorth — DNS npm package here: github.com/violetnorth/dns

So I’m going to talk about a quick way to clean up sockets with timeouts and overall error handling when dealing with sockets.

If you don’t implement some sort of timeout, you are going to run out of available sockets especially when using TCP sockets. …

Image for post
Image for post
Single scheduler (conductor) and multiple workers

Part 5— The worker

The entry point of the worker is the worker/worker.go file which contains the init and the main functions. Main functions call 2 functions async, which are to start the GRPC server and to register on the scheduler.

// worker/worker.gopackage main

import (
"fmt"
"log"
"os"
"os/signal"
"syscall"
)

var (
// workerID is the id assigned by the scheduler
// after registering on scheduler.
workerID string
)

func init() {
loadConfig()
}

// Entry point of the worker application.
func main() {

go startGRPCServer()
go registerWorker()

sig := make(chan os.Signal)
signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)

for {
select {
case s := <-sig:
fatal(fmt.Sprintf("Signal (%d) received, stopping\n", s))
}
}
}

func fatal(message string) {
deregisterWorker()
log.Fatalln(message) …

Image for post
Image for post
Single scheduler (conductor) and multiple workers

Part 4 — The scheduler

The entry point of the scheduler is the scheduler/scheduler.go file which contains the init and the main functions. Main functions call 2 functions async, which are to start the HTTP server (API) and the GRPC server.

// scheduler/scheduler.gopackage main

import (
"log"
"os"
"os/signal"
"syscall"
)

func init() {
loadConfig()
}

// Entry point of the scheduler application.
func main() {

go api()
go startGRPCServer()

sig := make(chan os.Signal)
signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)

for {
select {
case s := <-sig:
log.Fatalf("Signal (%d) received, stopping\n", s)
}
}
}

API uses Juline Schmidt’s HTTP router: github.com/julienschmidt/httprouter

API exposes the 3 API requests to start/query/stop jobs on a worker. These functions are pretty standard and not important. The important parts are the 3 functions that do all the work: “startJobOnWorker”, “stopJobOnWorker”, “queryJobOnWorker”. …

Image for post
Image for post
Single scheduler (conductor) and multiple workers

Part 3 — The worker overview

Workers expose a GRPC-server to communicate with the scheduler. The main job of workers is to run specific jobs requested by the scheduler and report on those jobs. Jobs can be ruby/python/bash scripts or any executables available on the worker machine.

// jobsMutex is the lock to access jobs map.
// jobs is the map that holds current/past jobs.
// - key: job id
// - value: pointer to the created job object.
var (
jobsMutex = &sync.Mutex{}
jobs = make(map[string]*job)
)

// job holds information about the ongoing or past jobs,
// that were triggered by the scheduler.
// - id: UUID assigned by the worker and sent back to the scheduler.
// - command: command which the scheduler run the job with
// - path: path to the job file/executable sent by the scheduler.
// - outFilePath: file path to where the output of the job will be piped.
// - cmd: pointer to the cmd.Exec command to get job status etc.
// - done: whether if job is done (default false)
// - err: error while running the job (default nil)
type job struct {
id string
command string
path string
outFilePath string
cmd *exec.Cmd …

About

Koray Göçmen

University of Toronto, Computer Engineering, architected and implemented reliable infrastructures and worked as the lead developer for multiple startups.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store