Other Configurations #

Advanced Usage #

Templates #

Example:

configs.template:
  - name: "es_gw1"
    path: ./sample-configs/config_template.tpl
    variable:
      name: "es_gw1"
      binding_host: "0.0.0.0:8000"
      tls_on_entry: true
      elasticsearch_endpoint: "http://localhost:9200"

Name	Type	Description
configs.template	array	Configuration templates, can specify multiple templates with corresponding parameters
configs.template[].name	string	Name of the configuration
configs.template[].path	string	Template configuration path
configs.template[].variable	map	Template parameter settings, variables in the template are used as `$[[variable_name]]`

Environment Variables #

The Gateway supports the use of environment variables for flexible parameter control within the configuration.

First, define the default values for environment variables in the configuration, as follows:

env:
  PROD_ES_ENDPOINT: http://localhost:9200
  PROD_ES_USER: elastic
  PROD_ES_PASS: password

Then, you can use environment variables in the configuration using the following syntax:

elasticsearch:
  - name: prod
    enabled: true
    endpoints:
      - $[[env.PROD_ES_ENDPOINT]]
    discovery:
      enabled: false
    basic_auth:
      username: $[[env.PROD_ES_USER]]
      password: $[[env.PROD_ES_PASS]]

Note that external environment variables take precedence over internal environment variable settings in the configuration. For example, to override environment variables when starting the program, use the following command:

PROD_ES_ENDPOINT=http://1.1.1.1:9200 LOGGING_ES_ENDPOINT=http://2.2.2.2:9201 ./bin/gateway

System Configuration #

System configuration is mainly used to set the fundamental properties of the Gateway:

Name	Type	Description
path.data	string	Data directory, default is `data`
path.logs	string	Log directory, default is `log`
path.configs	string	Configuration directory, default is `config`
log.level	string	Log level, default is `info`
log.debug	bool	Whether to enable debug mode. When enabled, the program exits immediately in case of an exception, printing the complete stack trace. Used for debugging and fault localization. Default is `false`, do not enable in production as it may result in data loss.
log.format	bool	Log format, default is `[%Date(01-02) %Time] [%LEV] [%File:%Line] %Msg%n`. Format References
log.disable_file_output	bool	Whether to disable local file log output, default is `false`. Use this in container environments if you don’t want local log output.
allow_multi_instance	bool	Whether to allow running multiple Gateway instances on a single machine, default is `false`
skip_instance_detect	bool	Whether to skip Gateway instance detection, default is `false`
max_num_of_instances	int	Maximum number of Gateway instances, default is `5`
configs.auto_reload	bool	Whether to support dynamic loading of configurations from `path.configs`

Local Disk Queue #

Example:

disk_queue:
  upload_to_s3: true
  s3:
    server: my_blob_store
    location: cn-beijing-001
    bucket: infini-store
  max_bytes_per_file: 102400

Name	Type	Description
disk_queue.min_msg_size	int	Minimum byte limit for a single message sent to the queue, default is `1`
disk_queue.max_msg_size	int	Maximum byte limit for a single message sent to the queue, default is `104857600` (100MB)
disk_queue.sync_every_records	int	Synchronization interval in terms of the number of records, default is `1000`
disk_queue.sync_timeout_in_ms	int	Synchronization interval in milliseconds, default is `1000` milliseconds
disk_queue.max_bytes_per_file	int	Maximum size of a single file in the local disk queue. If exceeded, a new file is created. Default is `104857600` (100MB)
disk_queue.max_used_bytes	int	Maximum allowed storage space used by the local disk queue
disk_queue.warning_free_bytes	int	Free storage space threshold for disk space warnings, default is `10737418240` (10GB)
disk_queue.reserved_free_bytes	int	Protected value for free storage space on disk. Once reached, the disk becomes read-only and no more writes are allowed. Default is `5368709120` (5GB)
disk_queue.auto_skip_corrupted_file	bool	Whether to automatically skip corrupted disk files, default is `true`
disk_queue.upload_to_s3	bool	Whether to upload disk queue files to S3, default is `false`
disk_queue.s3.async	bool	Whether to asynchronously upload to the S3 server
disk_queue.s3.server	string	S3 server ID
disk_queue.s3.location	string	S3 server location
disk_queue.s3.bucket	string	S3 server bucket
disk_queue.retention.max_num_of_local_files	int	Maximum number of files to retain locally after uploading to S3, default is `3`
disk_queue.compress.segment.enabled	bool	Whether to enable file-level compression, default is `false`

S3 #

Example:

s3:
  my_blob_store:
    endpoint: "192.168.3.188:9000"
    access_key: "admin"
    access_secret: "gogoaminio"

Name	Type	Description
s3.[id].endpoint	string	S3 server address
s3.[id].access_key	string	S3 server key
s3.[id].access_secret	string	S3 server secret key
s3.[id].token	string	S3 server token information
s3.[id].ssl	bool	Whether S3 server uses TLS
s3.[id].skip_insecure_verify	bool	Whether to skip TLS certificate verification for S3 server

Kafka #

The Gateway supports using distributed Kafka as a backend queue. The related parameters are as follows.

Name	Type	Description
kafka.enabled	bool	Whether the Kafka module is enabled
kafka.default	bool	Whether the Kafka module is the default queue implementation
kafka.num_of_partition	int	Default number of partitions, default is `1`
kafka.num_of_replica	int	Default number of replicas, default is `1`
kafka.producer_batch_max_bytes	int	Maximum size of the batch to submit, default is `50 * 1024 * 1024`
kafka.max_buffered_records	int	Maximum number of buffered request records, default is `10000`
kafka.manual_flushing	bool	Whether to enable manual flushing, default is `false`
kafka.brokers	[]string	Server address information
kafka.username	string	User information
kafka.password	string	Password information

Badger #

Badger is a lightweight disk-based KeyValue storage engine used by the Gateway to implement the KV module.

Name	Type	Description
badger.enabled	bool	Whether to enable the KV module implemented by Badger, default is `true`
badger.single_bucket_mode	bool	Whether Badger module uses single bucket mode, default is `true`
badger.sync_writes	bool	Whether Badger module uses synchronous writes, default is `false`
badger.mem_table_size	int64	Size of the in-memory table used by Badger module, default is `10 * 1024 * 1024` (10485760)
badger.value_log_file_size	int64	Size of Badger module’s log files, default is `1<<30 - 1` (1GB)
badger.value_log_max_entries	int64	Maximum number of log entries for Badger module, default is `1000000`
badger.value_threshold	int64	Value threshold for Badger module’s log files, default is `1048576` (1MB)
badger.num_mem_tables	int64	Number of in-memory tables for Badger module, default is `1`
badger.num_level0_tables	int64	Number of Level0 in-memory tables for Badger module, default is `1`

Resource Limitations #

Name	Type	Description
resource_limit.cpu.max_num_of_cpus	int	Maximum number of CPU cores allowed to be used, Linux only with `taskset` command available.
resource_limit.cpu.affinity_list	string	CPU affinity settings, e.g., `0,2,5` or `0-8`, Linux only with `taskset` command available.
resource_limit.memory.max_in_bytes	string	the max size of Memory to use, soft limit only

Network Configuration #

Common network configurations.

Name	Type	Description
*.network.host	string	Network address listened to by the service, for example, `192.168.3.10`
*.network.port	int	Port address listened to by the service, for example, `8000`
*.network.binding	string	Network binding address listened to by the service, for example, `0.0.0.0:8000`
*.network.publish	string	External access address listened to by the service, for example, `192.168.3.10:8000`
*.network.reuse_port	bool	Whether to reuse the network port for multi-process port sharing
*.network.skip_occupied_port	bool	Whether to automatically skip occupied ports

TLS Configuration #

Common TLS configurations.

Name	Type	Description
*.tls.enabled	bool	Whether TLS secure transmission is enabled or not, can auto generate cert files if not specified any cert files
*.tls.ca_file	string	Path to the public CA cert of the TLS security certificate
*.tls.cert_file	string	Path to the public key of the TLS security certificate
*.tls.key_file	string	Path to the private key of the TLS security certificate
*.tls.skip_insecure_verify	bool	Whether to ignore TLS certificate verification
*.tls.default_domain	string	The default domain for auto generated certs
*.tls.skip_domain_verify	bool	Whether to skip domain verify or not
*.tls.client_session_cache_size	int	Set the max cache of ClientSessionState entries for TLS session resumption

API #

Name	Type	Description
api.enabled	bool	Whether to enable the API module, default is `true`
api.network	object	Networking config, please refer to common network configuration section
api.tls	object	TLS config, please refer to common TLS configuration section
api.security	object	Security config for API module
api.security.enabled	bool	Whether security is enabled or not
api.security.username	string	The username for security
api.security.password	string	The password for security
api.cors.allowed_origins	[]string	The list of origins a cross-domain request can be executed from
api.websocket	object	Websocket config for API module
api.websocket.enabled	object	Whether websocket is enabled or not
api.websocket.permitted_hosts	[]string	The list of hosts that permitted to access the websocket service
api.websocket.skip_host_verify	bool	Whether websocket skip verify the host or not

Misc #

Name	Type	Description
preference.pipeline_enabled_by_default	map	Whether pipelines are enabled by default. If set to `false`, each pipeline must be explicitly configured with `enabled` set to `true`