Single Scheduler — Multiple Worker Architecture with GRPC and Go — Part 5

Image for post
Image for post
Single scheduler (conductor) and multiple workers

Part 5— The worker

The entry point of the worker is the worker/worker.go file which contains the init and the main functions. Main functions call 2 functions async, which are to start the GRPC server and to register on the scheduler.

The GRPC server of the worker calls 3 main important functions. These are the “startScript”, “stopScript”, “queryScript” functions. These functions call the script files or the “jobs” with the provided command and record the output of those scripts to a file on the worker.

jobs.go contains those 3 important functions to start, query and stop scripts. Each job is kept in the jobs map with job id as the key.

The MOST important piece of this whole project is how the script is started on the worker. The script is started inside a nameless concurrent function inside the “startScript” function in order to still have access to the job object it was started for. This way, when the job is finished or when an error occurs, I don’t need to go find the job with the key and try to find a way to pass it etc. I can just access the calling job object since the script is running inside an anonymous coroutine that is still in the same context.

There is quite a bit of stuff I left out, such as the proto file, how to build to a go file that is shared between the worker and the scheduler. The config file for both and the entire Makefile. They can be found on my Github repo. But the stuff I touched upon was the most important pieces.

Yeah, that’s pretty much it!

Written by

University of Toronto, Computer Engineering, architected and implemented reliable infrastructures and worked as the lead developer for multiple startups.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store