Single Scheduler — Multiple Worker Architecture with GRPC and Go — Part 3

Single scheduler (conductor) and multiple workers

Part 3 — The worker overview

// jobsMutex is the lock to access jobs map.
// jobs is the map that holds current/past jobs.
// - key: job id
// - value: pointer to the created job object.
var (
jobsMutex = &sync.Mutex{}
jobs = make(map[string]*job)
)

// job holds information about the ongoing or past jobs,
// that were triggered by the scheduler.
// - id: UUID assigned by the worker and sent back to the scheduler.
// - command: command which the scheduler run the job with
// - path: path to the job file/executable sent by the scheduler.
// - outFilePath: file path to where the output of the job will be piped.
// - cmd: pointer to the cmd.Exec command to get job status etc.
// - done: whether if job is done (default false)
// - err: error while running the job (default nil)
type job struct {
id string
command string
path string
outFilePath string
cmd *exec.Cmd
done bool
err error
}

Configuration for the worker is done through the `config.toml` file. I wrote about how I approach configuration in Go projects. You can read more here:

GRPC Server

  • addr: Address on which the GRPC server will be run.
  • use_tls: Whether the GRPC server should use TLS. If true, crt_file and key_file should be provided.
  • crt_file: Path to the certificate file for TLS.
  • key_file: Path to the key file for TLS.

Scheduler

  • addr: Address on which the GRPC server of the scheduler is run.
# worker/config.toml[grpc_server]
addr = "127.0.0.1:50000"
use_tls = false
crt_file = "server.pem"
key_file = "server.key"

[scheduler]
addr = "127.0.0.1:3000"

GRPC Server

// Worker GRPC server definitionsservice Worker {
rpc StartJob(StartJobReq) returns (StartJobRes) {}
rpc StopJob(StopJobReq) returns (StopJobRes) {}
rpc QueryJob(QueryJobReq) returns (QueryJobRes) {}
}

message StartJobReq {
string command = 1;
string path = 2;
}

message StartJobRes {
string jobID = 1;
}

message StopJobReq {
string jobID = 1;
}

message StopJobRes {
}

message QueryJobReq {
string jobID = 1;
}

message QueryJobRes {
bool done = 1;
bool error = 2;
string errorText = 3;
}

These GRPC requests are translations of HTTP requests into GRPC requests. Read about the HTTP requests in part 2.

--

--

University of Toronto, Computer Engineering, architected and implemented reliable infrastructures and worked as the lead developer for multiple startups.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Koray Göçmen

University of Toronto, Computer Engineering, architected and implemented reliable infrastructures and worked as the lead developer for multiple startups.