Single Scheduler — Multiple Worker Architecture with GRPC and Go — Part 3

Image for post
Image for post
Single scheduler (conductor) and multiple workers

Part 3 — The worker overview

Workers expose a GRPC-server to communicate with the scheduler. The main job of workers is to run specific jobs requested by the scheduler and report on those jobs. Jobs can be ruby/python/bash scripts or any executables available on the worker machine.

Configuration for the worker is done through the `config.toml` file. I wrote about how I approach configuration in Go projects. You can read more here:

GRPC Server

  • addr: Address on which the GRPC server will be run.
  • use_tls: Whether the GRPC server should use TLS. If true, crt_file and key_file should be provided.
  • crt_file: Path to the certificate file for TLS.
  • key_file: Path to the key file for TLS.

Scheduler

  • addr: Address on which the GRPC server of the scheduler is run.

GRPC Server

Only 3 GRPC requests required in the scheduler GRPC server: to start/stop/query jobs.

These GRPC requests are translations of HTTP requests into GRPC requests. Read about the HTTP requests in part 2.

Written by

University of Toronto, Computer Engineering, architected and implemented reliable infrastructures and worked as the lead developer for multiple startups.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store