diff --git a/README.md b/README.md new file mode 100644 index 0000000..4e5b573 --- /dev/null +++ b/README.md @@ -0,0 +1,104 @@ +# Zen + +In house Library to solve the biggest problem in the micro services world, **Observability**. + +To setup proper Observability, we have to fulfill the three pillars of Observability, which are: + +1. **Logging** +2. **Metrics** +3. **Tracing** + +There are surprising big requirements and code setup to take the maximum advantage of these pillars, which are: + +1. Structured and Standardized Logging format. +2. Metrics collection that have proper labels and contexts. +3. Tracing that should be able to cross the service boundaries. + +Lot of knowledge and boilerplate code is required to setup these pillars, which is not only time consuming but also error prone. + +This library is an attempt to solve these problems by providing a simple and easy to use API to setup Observability in your services. + +Some API are just re-exports, especially those that comes from [OpenTelemetry], but the usage of these API is already simplified and made more user friendly. + +## Status + +Still in development. Not ready even for basic usage. + +## Features + +### 1. Layered Config + +The library provides a layered configuration system, which allows the user to provide overrides at different levels. + +The library provides the following levels of configuration with the latter overriding the former: + +1. **Default Config**: The default configuration that comes with the library. This only contains configuration for `zen` library itself. +2. **Vault Config**: The configuration that comes from the [Vault]. +3. **Config File**: `zconfig.json` and `zconfig.yaml` at the same `cwd` as where the command is run. Ignored if not found. `zconfig.json` is looked first then `zconfig.yaml`. + The usage of `Config File` is actually discouraged because they contains secrets and must be inside `.gitignore` for security purposes. But it's there + if you want easy configuration override on local development. +4. **Environment Variables**: The most interesting and probably the most eye-catching one because it can easily override-able inline by `DevOps` team or in `CI` steps. + `Zen` will lookup `.env` file on `cwd` on startup and load up any `environment variables` that **DOES NOT EXIST** before. So `.env` loading will not override + any previous set variables. See [Environment Variables](#environment-variables) documentation for more details on how to use this. +5. **Command Line Arguments**: This is one of those _putting the cart before the horse_ kind of situation. Popular CLI parser libraries requires flags + to be defined first and this library does not know those prior values at the start of the application other than default values inside the library. Thus, + configuring config from `cli flags` are limited to only configuring `zen` library itself. + +### 2. [OpenTelemetry] Integration + +[OpenTelemetry] is chosen because it's vendor agnostic nature and are supported by big players in Observability world like [Grafana Stack][Grafana], [Elastic Stack (v2 and up)][Elastic], and [OpenObserve]. + +Migrating vendors is just as easy as changing the Collector endpoint in the config. + +Unfortunately, there are always costs to pay for this flexibility. The cost is the complexity of the API and the boilerplate code required to [setup the OpenTelemetry itself][OpenTelemetry Setup]. + +This library will simplify the setup of [OpenTelemetry] by providing a simple API to setup the Tracer, Meter, and Logger at the startup of the application. + +This library also has the goal to further integrate Bareksa's infrastructure with [OpenTelemetry] by providing an API to setup `Clients` for these infrastructures instrumented with [OpenTelemetry] out of the box. + +### 3. Developer Friendly Logging + +Structure Logging is good for machines to consume, but it's not very friendly for human read. +This library provides the best of both worlds by detecting the environment and choose the best logging format for that environment. + +On, interactive environment (TTY Detected), e.g. running via Terminal Emulator, the library will output the logs in a human readable format. + +On, non-interactive environment, e.g. running in a container, the library will output the logs in a JSON structured format. + +When running tests, by default the library will be silent, but you can easily enable the logs by setting the `ZEN_LOG_LEVEL` to `debug` using inline environment variables. +Note that Environment Variables are not loaded from `.env` file when running tests. + +### 4. Notification and Reporting System + +Just like [Tower] lib, `Zen` also provides an API to send notifications. However, unlike [Tower], `Zen` is a Client-Server architecture. The usage of this library +acts as a client to the server. The server (see: [serve.go](./cmd/zen/serve/serve.go>)) is also on this repository, but it's not designed to be used as a library and +should not be imported in any other services. + +This implementation allows non-Golang services to also send notifications to the server, and also allows governance to control the notification system centrally. + +The server uses [ConnectRPC] as protocol, allowing to use `grpc` and `http` as transport, and allows code generation for other languages. + +### 5. Instrumented Clients + +The library provides an API to setup clients for Bareksa's infrastructure that are instrumented with [OpenTelemetry] out of the box. + +Clients: + +1. Redis +2. Any database that supports `database/sql` interface, e.g. MySQL, PostgreSQL, etc. +3. HTTP Server/Client +4. GRPC Server/Client +5. Kafka (Sarama) + +## Documentation + +`WIP` + +[OpenTelemetry]: https://opentelemetry.io +[OpenTelemetry Setup]: https://opentelemetry.io/docs/languages/go/getting-started +[Vault]: https://vault-ha.bareksa.dev +[Grafana]: https://grafana.com +[Elastic]: https://www.elastic.co +[OpenObserve]: https://openobserve.ai +[Tower]: https://gitlab.bareksa.com/backend/tower +[ConnectRPC]: https://connectrpc.com