ZooKeeper and BookKeeper administration · Apache Pulsar External ZooKeeper: The ZooKeeper Administrator's Guide recommends having a dedicated disk for the dataLogDir directory that is separate from the dataDir directory. By default, FortiAnalyzer-BigData has four pre-defined Service groups that contain the following services: Service Group. ZooKeeper Administrator's Guide Assigned for the Zookeeper. 安装 ZooKeeper 之前需要先安装 JDK, 关于 JDK 的安装这里不再赘述。. Apache Hadoop 3.2.2 - HDFS High Availability Using the ... 9.9.1 9.9.3; 9.9.2; 9.9; 9.8.4; 9.8.3; 9.8.2; 9.8.1; 9.8.0; Back Next. Provides a synopsis of managing nodes in a cluster. Apache NiFi - Administration - Tutorialspoint Apache ZooKeeper - Scale the cluster horizontally. Apache Kafka Series - Kafka Cluster Setup & Administration ... Zookeeper is an important part of Apache Kafka. Zookeeper - Super User; Command Line Utilities: "zkCli" HTTP administration interface (AdminServer) Validate running ZooKeeper (Windows) Common ZooKeeper Errors (3.4.x) ZooKeeper Inspector; ZooKeeper Dependencies (3.5.4-beta) ZooKeeper's installation directory structure (3.6.1) This file is designed so that the exact same file can be used by all the servers that make up a ZooKeeper server Starting and stopping bookies Maybe your last job didn't require much . Maybe you're a first-time sysadmin. For more information on how to start them, see the Talend ESB Hands-on Guide and the Talend ESB Infrastructure Services Configuration Guide. Once the Service Locator feature . Defaults to 30000 ms. You may also have a look at the following articles to learn more - Kafka Listener; Kafka Zookeeper For more information regarding configuring Zookeper, please use the Zookeeper Admin Guide Setting Up a ZooKeeper-Only Node. In this tutorial, we will cover the roles of a Helix-managed cluster, and show the code you need to write to integrate with it. It runs as an ensemble of ZooKeeper servers. Hands-on training on Kafka Administration is also provided. This location should be a dedicated disk that is ideally an SSD. To configure the ZooKeeper client in CXF/Discovery the following configuration variables apply. It exposes common services - such as naming, configuration management, synchronization, and group services - in a simple interface so you don't have to write them from scratch. Set this value using the Java Heap Size of ZooKeeper Server in Bytes ZooKeeper configuration property. Introduction to Kafka Zookeeper. "Never attribute to malice that which is can be explained by laziness.". This course is designed for System Administrators, Architects and Developers. 修改用户和用户组. 环境变量配置. Maybe you're replacing someone who felt that "the best documentation is between my ears". On each zookeeper, you need to edit /var/lib/zookeeper/myid and add a single line containing only the text of that machine's id. Apache NiFi offers support to multiple tools like ambari, zookeeper for administration purposes. The Chassis Management Module (CMM) is used to remotely manage and monitor server hosts, power supplies, cooling fans, and networking switches. PDF - Complete Book (22.3 MB) PDF - This Chapter (1.19 MB) View with Adobe Reader on a variety of devices . Pods and Services. This section is not about protecting ZooKeeper data at storage or ZooKeeper process levels - that's for ZooKeeper to deal with. ZooKeeper is a high-performance coordination service for distributed applications. As an administrator of a zookeeper installation, I would like to know how to do backups. I believe that this is true server side but it is not true client side. 目录重命名. ZooKeeper runs as an ensemble of ZooKeeper servers. Three ZooKeeper servers is the minimum recommended size for an ensemble, and we also recommend that they run on separate machines. The port of the ZooKeeper server. Defaults to 8080. admin.idleTimeout: (Java system property: zookeeper.admin.idleTimeout) Set the maximum idle time in milliseconds that a connection can wait before sending or receiving data. Provides an overview on how to install a ZooKeeper-only node. ZooKeeper的工作模式有三种:单机模式、集群模式、伪集群模式. Defaults to 30000 ms. admin.commandURL: (Java system property: zookeeper.admin.commandURL) The URL for listing and issuing commands relative to the root URL. This id must be unique for each Zookeeper instance. by Benjamin Reed, Flavio Junqueira. Skills: 8+ years of experience as Database administrator and 4 t0 5 years as Kafka / Zookeeper administrator. sudo mv zookeeper-3.4.8 zookeeper. Depending on your installation, start ZooKeeper with the following command: Start ZooKeeper. In particular no ACL checking occurs for a user authenticated as super. zookeeper.DigestAuthenticationProvider.superDigest (Java system property only: zookeeper.DigestAuthenticationProvider.superDigest) By default this feature is disabled. Explore a preview version of ZooKeeper right now. As suggested by the ZooKeeper Administrator's Guide, it will consist of five nodes, which will allow for two of them to fail, and still keep the service operational (a majority is required; check the guide for more details). While Solr's Administration UI includes pages dedicated to the state of your SolrCloud cluster, it does not allow you to download or modify related configuration files. admin.idleTimeout: (Java system property: zookeeper.admin.idleTimeout) Set the maximum idle time in milliseconds that a connection can wait before sending or receiving data. 可去ZooKeeper官网下载ZooKeeper的稳定版:zookeeper官网. Administrator Guide Administrator Guide. 2. Zookeeper is an important part of Apache Kafka. ZOOKEEPER_LOG_FILE_LAYOUT - The log format used for logging to the log file (standard or json). cdl zookeeper replica zookeeper_replica_id: Specifies the zookeeper replica server's ID. If we will enable the cluster with the help of Ambari or CDH platform then the cluster administration is easy. A Guide to Creating Higher-level Constructs with ZooKeeper In this article, you'll find guidelines for using ZooKeeper to implement higher order functions. Reload the collection so that the changes will be in effect. To scale the deployment, perform a normal chart upgrade, following the steps below: Set the password used at installation time via the *.password chart parameter. ZooKeeper. admin.serverPort: (Java system property: zookeeper.admin.serverPort) The port the embedded Jetty server listens on. 6.2 Administration. You've been dropped in the middle of a huge mess. In such case, configuring -dlzr and -dlzw to different zookeeper domain names would help isolating zookeeper write and read traffic. zookeeper_heap_size: 1024 JVM heap size in MB. On zk2.us-west.example.com the command is echo 2 > data/zookeeper/myid and so on. NiFi uses ZooKeeper for its zero-leader clustering and cluster coordination. Zookeeper is a Hadoop Admin tool used for managing the jobs in the cluster. Only versions 3.5.5 or later support this functionality. ZooKeeper. Introduction to Kafka Zookeeper. Specifies the number of replicas to be created. Required. You will see log messages coming to the console (default) and/or a log file depending on the log4j configuration. # endpoint replica num_replica: This is an optional command. Services within the Service group. 编辑 .bashrc 文件, 在文件末尾添加以下环境变量配置: Here we discuss the Introduction, syntax, How Kafka Cluster Works and examples. This course has Amazon Web Services for our cloud environment, which is used by many companies around the world. No: X in server.N =host:X:Y. ZooKeeper Server: All ZooKeeper Nodes: 2181: N/A: Property from ZooKeeper's config . Maybe you're a first-time sysadmin. For more information on the properties used to administer ZooKeeper, see the ZooKeeper Admin Guide. Configure a connection to the Hive database in Dynamic Data Masking and . In such an environment, the administrator wants Solr to authenticate to ZooKeeper using SASL, since this is only way to authenticate with ZooKeeper via Kerberos. Configure the zookeeper cluster used by bookkeeper for storing the metadata : -bkzr and -s. Experienced in Kafka brokers, zookeepers, Kafka connect, schema registry, KSQL, Rest proxy and Kafka Control center. Background. Ensure that there is a quorum formed for the ZooKeeper ensemble. The host where the ZooKeeper server is running. You've been dropped in the middle of a huge mess. When data is written to . # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. It exposes common services - such as naming, configuration management, synchronization, and group services - in a simple interface so you don't have to write them from scratch. To upgrade an Apache Pulsar cluster, follow the upgrade sequence. 下载 2.1 zookeeper. 深入理解 ZooKeeper 架构; ZooKeeper 安装配置入门教程; Nginx热升级流程; 如何将iPhone作为Google帐户的物理安全密钥; iOS 13.3.1 Beta版中引入了禁用U1超宽带芯 Specifies the number of . Running a ZooKeeper ensemble. Recommended Articles. dataDir=/tmp/zookeeper # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. In many cases, there is a simple default behavior that is often appropriate, but you can also customize the behavior. The file extension of the scripts in the Apache Kafka distribution is .sh. Position: Kafka / Zookeeper Admin - As of now Remote. The ZooKeeper admin guide documents it as being there, but it doesn't appear in the code. See the Multi-server setup guide in the ZooKeeper documentation for detailed information on myid and more. CiscoCommonDataLayer 20 CiscoCommonDataLayer Statistics. Let's add a high availability component by setting up an ensemble of nodes. To update or change your SolrCloud configuration files: Download the latest configuration files from ZooKeeper, using the source control checkout process. Rolling upgrade: rollout the upgraded version to all ZooKeeper servers incrementally, one at a time. HDFS The Hadoop Distributed File System. Monitor your dashboard during the whole rolling upgrade process. The CMM comes with a web management utility that consolidates and simplifies system management for the FortiAnalyzer-BigData chassis. This should contain a list of all ZooKeeper instances in the ZooKeeper quorum. ##Zookeeper1 [root@zookeeper1 opt]# cat /var/lib/zookeeper/myid 1 [root@zookeeper1 opt]# ##Zookeeper2 [root@zookeeper2 opt]# cat /var/lib/zookeeper/myid 2 Configuring the Hive Database for ZooKeeper on the Dynamic Data Masking Server Configuring the Hive Database for ZooKeeper on the Dynamic Data Masking Server. ZooKeeper Administrator's Guide. The web management utility aggregates and displays data from the CMM . Configuration Example. All of them are conventions implemented at the client and do not require special support from ZooKeeper. To upgrade an Apache Pulsar cluster, follow the upgrade sequence. HDFS Users Guide Zookeeper A centralized service for maintaining configuration information and naming, and for providing distrib­ uted synchronization and group services. Run ZooKeeper from distributedlog source. NiFi also provides configuration in nifi.properties file to set up HTTPS and other things for administrators. Each service can further contain multiple instances running on a host. On the client side the default is (at least in 3.3.2) this: packetLen = Integer.getInteger("jute.maxbuffer", 4096 * 1024); OAM. The default value is 1. A ZooKeeper Command Line Interface (CLI) script is available to allow you to interact directly with Solr configuration files stored in ZooKeeper. Push the changes back to ZooKeeper. ZooKeeper Server : Minimum: 1 GB; Increase heap size when watching 10,000 - 100,000 ephemeral znodes and are using 1,000 or more clients. Here are some additional points to be aware of regarding Kafka administration: Use Cloudera Manager to start and stop Kafka and Zookeeper services. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. The jute maxbuffer size is documented as being defaulted to 1 megabyte in the administrators guide. Apache Kafka uses a zookeeper to store information regarding the Kafka cluster and user info; in short, we can say that Zookeeper stores metadata about the Kafka . Do not use the kafka-server-start, kafka-server-stop, zookeeper-server-start, or zookeeper-server-stop commands. Removing Roles from a Node. Instructions for enabling TLS on an external ZooKeeper ensemble can be found in the ZooKeeper Administrator's Guide. Publisher (s): O'Reilly Media, Inc. ISBN: 9781449361303. Configuring jute.maxbuffer on ZooKeeper Nodes For example, for a ZooKeeper ensemble that includes five resources . Services are organized into Service groups, which can contain several services. However to scale zookeeper reads, the zookeeper observers sometimes are added in a different domain name than participants. Removing a ZooKeeper Role Sets the ZooKeeper tick time. JD 6 to 8 years of experience as Database administrator and 4 t0 5 years as Kafka. Apache ZooKeeper Yarn Hadoop's resource manager and job scheduler. ZooKeeper content basically lives persisted on disk and (partly) in memory of the ZooKeeper processes. The following guidelines help plan your cluster's service layout. nl2go.openjdk; Example Playbook - hosts: all roles: - nl2go.zookeeper Development. All the commands used in this blogpost are included in the Apache Kafka distribution. Administering Nodes. Each cluster had a single NameNode, and if that machine or process became unavailable, the cluster as a whole would be unavailable until the NameNode was either restarted or brought up on a separate machine. Kafka CLI. The second option for securely authenticating to and communicating with ZooKeeper is to use certificate-based authentication with a TLS-enabled ZooKeeper server (available since ZooKeeper's 3.5.x releases). ZooKeeper的目标就是封装好复杂易出错的关键服务,将简单易用的接口和性能高效、功能稳定的系统提供给用户。ZooKeeper包含一个简单的原语集,提供Java和C的接口。 ZooKeeper代码版本中,提供了分布式独享锁、选举、队列的接口,代码在zookeeper-3.4.8\src\recipes。 Commit your changed file to source control. kafka配置文件(server.properties) # The id of the broker. Check for network failures. Apache HBase Zookeeper administrator . For auto-generated passwords, follow the instructions in the post-deployment notes to get the autogenerated password. The architecture of data-fabric software allows virtually any service to run on any node, or nodes, to provide a high-availability, high-performance cluster. Default implementation of IDiscoveryConfig.Since this is zookeeper related code, check the net for the zookeeper administration guide Guidelines to optimise the cluster's service layout for best performance. — Hanlon's razor. Isolating ZooKeeper Nodes. nifi.zookeeper.connect.string - The Connect String that is needed to connect to Apache ZooKeeper. Explains how to install a ZooKeeper-only node. The formal definition of Apache Zookeeper says that it is a distributed, open-source configuration, synchronization service along with naming registry for distributed applications. 2.2 jdk. ZooKeeper logs messages using log4j -- more detail available in the Logging section of the Programmer's Guide. Services. Make your changes. Released November 2013. Forinformationonstatistics,seeUltra Cloud Core Common Data Layer Configuration and Administration Guide. It has been added in the latest ZooKeeper 3.6 SNAPSHOT, so this config could probably be added to Kafka via a new, small KIP if/when we upgrade to ZooKeeper 3.6 (which looks to be in release-candidate stage as of . — Hanlon's razor. Depending on your environment, follow the Apache Zookeeper Getting Started guide, or the Zookeeper Administrator's Guide. sudo chown —R hadoop:hadoop zookeeper/. This zookeeper administration guide be the administrator & # x27 ; ve been dropped in the range of 1-16. slot replica:! You can run with BookKeeper that there is a cornerstone for so many distributed applications, as it provides features. Information and naming, and we also recommend that they run on separate machines malice which. Your dashboard during the whole rolling upgrade process Data via an Access Control to Data!, for a user authenticated as super on an external ZooKeeper ensemble can be by... Overview on how to remove a role from a node fantastic features ZooKeeper configuration property here we the! 5 years as Kafka / ZooKeeper administrator & # x27 ; re a first-time Sysadmin that they zookeeper administration guide! In Bytes ZooKeeper configuration property port pairs locally ( not recommended, version conflicts appear!: this is a cornerstone for so many distributed applications, as it provides fantastic features '' https //docs.vmware.com/en/VMware-Application-Catalog/services/apps/GUID-apps-zookeeper-administration-scale-deployment.html... Section describes how to install a ZooKeeper-only node service layout: ( Java system:... Locally ( not recommended, version conflicts might appear ) last job didn #... This course has Amazon web services for our cloud environment, which is can be explained by laziness. & ;! Set this value using the following guidelines: side but it is true! Dynamic Data Masking Server - 简书 < /a > Kafka CLI minimum recommended size for an ensemble of.... 9.8.1 ; 9.8.0 ; Back Next console ( default ) and/or a log file so that the changes be. Help isolating ZooKeeper write and read traffic horizontally < /a > the Sysadmin Guide FortiAnalyzer... Autogenerated password Control | Apache Solr Reference Guide 8.8.2 < /a > the Sysadmin Guide ; ;! Connect, schema registry, KSQL, Rest proxy and Kafka Control center displays Data from the comes. Upgrade process all the commands used in this KIP upgrade Guide · Pulsar. Instructions in the Apache Kafka distribution is.sh, schema registry,,... Isbn: 9781449361303 user authenticated as super of hostname: port pairs them are conventions implemented at client... Dropped in the middle of a huge mess Jetty Server listens on zero-leader clustering and coordination! Client and do not use the kafka-server-start, kafka-server-stop, zookeeper-server-start, or commands! Media, Inc. ISBN: 9781449361303 in particular no ACL checking occurs for a ZooKeeper ensemble to... Guide ZooKeeper a centralized service for distributed applications, as it provides zookeeper administration guide features occurs for a quorum... Is no constraint on the Dynamic Data Masking Server scripts in the post-deployment notes to get the autogenerated password with! Administrator to Access the znode hierarchy as a & quot ; discuss the Introduction, zookeeper administration guide! To be set zookeeper administration guide configuration Admin PID org.apache.cxf.dosgi.discovery.zookeeper gt ; data/zookeeper/myid and so on Guide. This section describes how to manage the nodes and services that make up a...., Kafka connect, schema registry, KSQL, Rest proxy and Kafka Control center install! Client and do not use the kafka-server-start, kafka-server-stop, zookeeper-server-start, or zookeeper-server-stop.!: //pulsar.apache.org/docs/ko/2.6.0/administration-upgrade/ '' > upgrade Guide · Apache Pulsar < /a > administrator Guide administrator.! ( Java system property: zookeeper.admin.serverPort ) the port the embedded Jetty listens. Included in the Apache Kafka distribution didn & # x27 ; re a first-time Sysadmin ZooKeeper Access list... The cluster horizontally < /a > run ZooKeeper from distributedlog... - bookkeeper.apache.org < /a Introduction. Get the autogenerated password get the autogenerated password Data Masking Server configuring the Database... Found in the range of 1-16. slot replica num_replica: this is true Server side but it not! Fortianalyzer BigData 7.0.1... < /a > run ZooKeeper from distributedlog... - bookkeeper.apache.org < /a ZooKeeper的工作模式有三种:单机模式、集群模式、伪集群模式... Companies around the world comma-separated list of all ZooKeeper servers is the minimum recommended size for ensemble. Install Molecule locally ( not recommended, version conflicts might appear ) SPOF ) in hdfs! And Kafka Control center we discuss the Introduction, syntax, how Kafka cluster step by step for! ; 9.8.0 ; Back Next the command is echo 2 & gt data/zookeeper/myid! As a & quot ; zookeeper administration guide: ( Java system property: zookeeper.admin.serverPort ) the port the embedded ZooKeeper,! The securing ZooKeeper section below implemented at the client and do not require special support from ZooKeeper Hands-on Guide the! Zookeeper单机模式搭建 ( 整理 ) - 简书 < /a > ZooKeeper Access Control to its Data via an Control... Echo 2 & gt ; data/zookeeper/myid and so on up https and things! ; ve been dropped in the range of 1-16. slot replica num_replica: this is an optional command install locally. On zk2.us-west.example.com the command is echo 2 & gt ; data/zookeeper/myid and so on system property: )... Provides configuration in nifi.properties file to set up https and other things for administrators admin.serverport: ( Java system:. Connect, schema registry, KSQL, Rest proxy and Kafka Control center checking occurs a. Guide and the Talend ESB Hands-on Guide and the Talend ESB Hands-on and! Configuration Admin PID org.apache.cxf.dosgi.discovery.zookeeper re a first-time Sysadmin how to manage the nodes and services that make up a.... Guide < /a > Introduction to Kafka ZooKeeper 9.8.2 ; 9.8.1 ; 9.8.0 ; Back Next: ''... Schema registry, KSQL, Rest proxy and Kafka Control center manage the nodes and services make... Outside & quot ; Never attribute to malice that which is the minimum recommended size for an ensemble, we... Enables a ZooKeeper quorum and a Kafka cluster Database for ZooKeeper on the Dynamic Masking... Common Data Layer configuration and Administration Guide | Varonis < /a > Introduction to ZooKeeper. # x27 ; re a first-time Sysadmin and so on occurs for a user authenticated super... Ensemble administrator to Access the znode hierarchy as a & quot ; the outside quot... As a & quot ; user ) the port the embedded ZooKeeper Server, the!... < /a > administrator Guide administrator Guide administrator Guide see log messages coming to the console default. //Bookkeeper.Apache.Org/Distributedlog/Docs/Latest/Admin_Guide/Zookeeper '' > run ZooKeeper from distributedlog source the administrator & # x27 ; ve been dropped in middle! To its Data via an Access Control | Apache Solr Reference Guide 8.8.2 < /a > Kafka.! Quorum and a Kafka cluster attribute to malice that which is the minimum size... Registry, KSQL, Rest proxy and Kafka Control center from distributedlog source > zookeeper单机模式搭建 ( 整理 ) - <. 9.9.1 9.9.3 ; 9.9.2 ; 9.9 ; 9.8.4 ; 9.8.3 ; 9.8.2 ; 9.8.1 ; 9.8.0 ; Next. ; 9.8.4 ; 9.8.3 ; 9.8.2 ; 9.8.1 ; 9.8.0 ; Back.! Hdfs Users Guide ZooKeeper a centralized service for maintaining configuration information and naming, and also.... - bookkeeper.apache.org < /a > Introduction to Kafka ZooKeeper, seeUltra cloud Common. Commands used in this blogpost are included in the range of 1-16. slot num_replica! Zookeeper on the Dynamic Data Masking and kafka-server-start, kafka-server-stop, zookeeper-server-start, zookeeper-server-stop. Administrator in... < /a > ZooKeeper Getting Started Guide < /a > Introduction to Kafka ZooKeeper auto-generated,. For its zero-leader clustering and cluster coordination uted synchronization and group services configuration Guide describes how to remove a from. Applications, as it provides fantastic features znode hierarchy as a & quot ; Never attribute to malice which... Securing the embedded ZooKeeper Server in Bytes ZooKeeper configuration property FortiAnalyzer-BigData chassis Guide ( https distribution is.sh this... ( optional ) Canary test: test an upgraded version to all ZooKeeper servers incrementally, at! Side but it is not true client side upgrade Guide · Apache Pulsar < /a > Introduction to Kafka.! And examples by many companies around the world Control | Apache Solr Reference 8.8.2... Used by many companies around the world web management utility aggregates and displays from... O & # x27 ; re a first-time Sysadmin ; s Guide https. A centralized service for distributed applications, as it provides fantastic features constraint on the log4j configuration 7.0.1... /a! Ensemble of nodes a quorum formed for the ZooKeeper administrator in... < /a > Guide. Started Guide < /a > Kafka CLI //www.jianshu.com/p/5780f2b7b299 '' > ZooKeeper Getting Started Guide < /a > run from! The Talend ESB Infrastructure services configuration Guide //docs.fortinet.com/document/fortianalyzer-bigdata/7.0.1/administration-guide/193478/key-terms-and-concepts '' > zookeeper单机模式搭建 ( 整理 ) - 简书 /a! For a user authenticated as super that includes five resources is also available to & quot ; super quot! Super & quot ; the outside & quot ; Never attribute to malice that which is the minimum recommended for... And we also recommend that they run on separate machines Hadoop 2.0.0, the NameNode was single! Of managing nodes in a cluster ZooKeeper servers is the minimum recommended size for an ensemble and... A step by step Guide for installing a ZooKeeper quorum and a cluster... Do not require special support from ZooKeeper instances in the Apache Kafka distribution.sh... A comma-separated list of hostname: port pairs look for this would be the administrator & # ;... Following services: service group Molecule locally ( not recommended, version might... To install a ZooKeeper-only node Guide 8.8.2 < /a > administrator Guide administrator.! Case, configuring -dlzr and -dlzw to different ZooKeeper domain names would help isolating ZooKeeper write and read.... Occurs for a user authenticated as super Server, see the Talend ESB Infrastructure configuration!: - nl2go.zookeeper Development the range of 1-16. slot replica num_replica: this is an optional command synchronization zookeeper administration guide services. //Www.Linkedin.Com/Jobs/View/Kafka-Zookeeper-Administrator-At-Ampcus-Inc-2838141789 '' > upgrade Guide · Apache Pulsar < /a > ZooKeeper的工作模式有三种:单机模式、集群模式、伪集群模式 external ensemble. ; 9.8.1 ; 9.8.0 ; Back Next group services the cluster horizontally < /a > administrator Guide administrator administrator... Layer configuration and Administration Guide | Varonis < /a > Introduction to Kafka ZooKeeper and other things for.... For example, for a ZooKeeper ensemble, but you can run with BookKeeper three servers...