Skip to main content

Release notes

4.14.4​

Release 4.14.4 includes multiple stability, performance, and security fixes.

Apache BookKeeper users are encouraged to upgrade to 4.14.4. The technical details of this release are summarized below.

Highlights​

Improvements​

  • [https://github.com/apache/bookkeeper/pull/2952] Allow to easily override zkServers with metadataServiceUri
  • [https://github.com/apache/bookkeeper/pull/2935] ReplicationWorker should not try to create a ZK based LedgerManagerFactory
  • [https://github.com/apache/bookkeeper/pull/2870] Add skip unrecoverable ledger option for bookkeeper shell recover command
  • [https://github.com/apache/bookkeeper/pull/2847] ISSUE #2846 Allow to run on java 17

Dependency updates​

  • [https://github.com/apache/bookkeeper/pull/2934] Upgrade to Grpc 1.42.1

Details​

https://github.com/apache/bookkeeper/issues?q=+label%3Arelease%2F4.14.4

4.14.3​

Release 4.14.3 includes multiple stability, performance, and security fixes along with the fix for Prometheus metrics.

Apache BookKeeper users are encouraged to upgrade to 4.14.3. The technical details of this release are summarized below.

Highlights​

Improvements​

  • [https://github.com/apache/bookkeeper/pull/2768] Add metrics and internal command for AutoRecovery
  • [https://github.com/apache/bookkeeper/pull/2788] Fix npe when pulsar ZkBookieRackAffinityMapping getBookieAddressResolver
  • [https://github.com/apache/bookkeeper/pull/2794] Heap memory leak problem when ledger replication failed
  • [https://github.com/apache/bookkeeper/pull/2802] Add auditor get ledger throttle to avoid auto recovery zk session expire
  • [https://github.com/apache/bookkeeper/pull/2813] Add ensemble check to over-replicated ledger GC
  • [https://github.com/apache/bookkeeper/pull/2832] Fix semaphore leak when EntryMemTable#addEntry accepts the same entries
  • [https://github.com/apache/bookkeeper/pull/2833] Eliminate direct ZK access in ScanAndCompareGarbageCollector
  • [https://github.com/apache/bookkeeper/pull/2842] Remove direct ZK access for Auditor
  • [https://github.com/apache/bookkeeper/pull/2844] Add error handling to readLedgerMetadata in over-replicated ledger GC
  • [https://github.com/apache/bookkeeper/pull/2845] A empty implmentation of newLedgerAuditorManager in EtcdLedgerManagerFactory to fix build

Dependency updates​

  • [https://github.com/apache/bookkeeper/pull/2792] Upgraded dependencies with CVEs
  • [https://github.com/apache/bookkeeper/pull/2793] Upgrade httpclient from 4.5.5 to 4.5.13 to address CVE-2020-13956
  • [https://github.com/apache/bookkeeper/pull/2811] Upgrade Netty to 4.1.68.Final

Details​

https://github.com/apache/bookkeeper/issues?q=+label%3Arelease%2F4.14.3

4.14.2​

Release 4.14.2 fixes an issue with Prometheus metrics that was found in 4.14.0.

Apache BookKeeper users are encouraged to upgrade to 4.14.2. The technical details of this release are summarized below.

Bugs​

  • [https://github.com/apache/bookkeeper/pull/2740] Fix Bouncy Castle fips incompatible issue

    In #2631, the default BouncyCastle was changed from non-fips into fips version. But the default version of BouncyCastle in Pulsar is the non-fips one(aimed to make it compatible with the old version of Pulsar).

    Bouncy Castle provides both FIPS and non-FIPS versions, but in a JVM, it can not include both of the 2 versions(non-Fips and Fips), and we have to exclude the current version before including the other. This makes the backward compatible a little hard, and that's why Pulsar has to involve an individual module for Bouncy Castle.

  • [https://github.com/apache/bookkeeper/pull/2762] Upgrade libthrift to 0.14.2 to address multiple CVEs

    The current libthrift version 0.12.0 has multiple vulnerabilities: CVE-2019-0205 , CVE-2019-0210 , CVE-2020-13949

  • [https://github.com/apache/bookkeeper/pull/2735] Exclude grpc-okhttp dependency

    The okhttp dependency version 2.7.4 is old and vulnerable. This dependency isn't needed and it causes Bookkeeper to be flagged for security vulnerabilities.

  • [https://github.com/apache/bookkeeper/pull/2734] Upgrade Freebuilder version and fix the dependency

    • Freebuilder 1.14.9 contains an outdate jquery js file which causes the library to be flagged as vulnerable with the highest threat level in Sonatype IQ vulnerability scanner. This also flags Bookkeeper as vulnerable with the highest threat level although it is a false positive and not an actual threat.

    • Freebuilder shouldn't be exposed as a transitive dependency

  • [https://github.com/apache/bookkeeper/pull/2693] Upgrade vertx to 3.9.8, addresses CVE-2018-12541

    The current vertx version is 3.5.3 which has a vulnerability, CVE-2018-12541 .

Details​

https://github.com/apache/bookkeeper/issues?q=+label%3Arelease%2F4.14.2

4.14.1​

Release 4.14.1 fixes an issue with Prometheus metrics that was found in 4.14.0.

Apache BookKeeper users are encouraged to upgrade to 4.14.1. The technical details of this release are summarized below.

Bugs​

  • [https://github.com/apache/bookkeeper/pull/2718] Fix prometheus metric provider bug and add test to cover label scope

    After add label for prometheus metric by #2650, it will cause prometheus metric format check failed when no label specified for a statsLogger. The metric list as follow.

Details​

https://github.com/apache/bookkeeper/issues?q=+label%3Arelease%2F4.14.1

4.14.0​

Release 4.14 adds FIPS compliance, improves compaction logic and the Stream Storage, improves data reliability in the recovery scenarios, fixes multiple bugs and brings critical dependencies up-to-date.

Apache BookKeeper users are encouraged to upgrade to 4.14.0. The technical details of this release are summarized below.

Highlights​

Bookkeeper is FIPS compliant by default now​

  • [https://github.com/apache/bookkeeper/pull/2631] Make Bookkeeper FIPS compliant by default

    FIPS is 'Federal Information Processing Standard'. It's a set of guidelines for security functions such as encryption/decryption/RNG etc. Applications running in FIPS mode are said to be more secure as they adhere to more stringent standards.

Data reliability​

  • [https://github.com/apache/bookkeeper/pull/2616] Add fencing to recovery reads to avoid data loss issue

Table Service (stream storage) reliability improvements​

  • [https://github.com/apache/bookkeeper/pull/2686] Improved handling of RocksDB tombstones
  • [https://github.com/apache/bookkeeper/pull/2641] Checksum validation for SST files
  • [https://github.com/apache/bookkeeper/pull/2635] Better handling of corrupted checkpoints
  • [https://github.com/apache/bookkeeper/pull/2643] Adjusted default rocksDbBlockCache size to 10%/numberOfLedgers of direct memory
  • [https://github.com/apache/bookkeeper/pull/2698] RocksDB log path is configurable now

Compaction logic improvements​

  • [https://github.com/apache/bookkeeper/pull/2675] forceAllowCompaction to run only when force is set or configured interval
  • [https://github.com/apache/bookkeeper/pull/2670] Allow a customer to set a limit on the duration of the major and minor compaction runs
  • [https://github.com/apache/bookkeeper/pull/2645] Fix: The compaction status report is off by 1
  • [https://github.com/apache/bookkeeper/pull/2626] Allow force compact entry log when entry log compaction is disabled
  • [https://github.com/apache/bookkeeper/pull/2627] Allow DBLedgerStorage to force GC by disk listener

Dependency updates​

  • [https://github.com/apache/bookkeeper/pull/2696] SECURITY: Upgraded Netty to 4.1.63.Final
  • [https://github.com/apache/bookkeeper/pull/2701] SECURITY: Removed jackson-mapper-asl dependency to resolve multiple CVEs
  • [https://github.com/apache/bookkeeper/pull/2697] Upgraded Lombok to 1.18.20 (required for Java 16 support)
  • [https://github.com/apache/bookkeeper/pull/2686] Upgraded rocksdb to 6.16.4

Other improvements and fixes​

  • [https://github.com/apache/bookkeeper/pull/2658] Fix: always select the same region set bug for RegionAwareEnsemblePlacementPolicy
  • [https://github.com/apache/bookkeeper/pull/2650] Allow to attach labels to metrics
  • [https://github.com/apache/bookkeeper/pull/2401] Allow to bypass journal for writes
  • [https://github.com/apache/bookkeeper/pull/2710] Imposed a memory limit on the bookie journal
  • [https://github.com/apache/bookkeeper/pull/2664] Bookkeeper client throttling logic is based upon entryId instead of ledgerId
  • [https://github.com/apache/bookkeeper/pull/2694] Performance: unnecessary copy to heap from CompositeByteBuf
  • [https://github.com/apache/bookkeeper/pull/2654] Ensure that only entries of the current ensemble are included in the ledger recovery process
  • [https://github.com/apache/bookkeeper/pull/2646] Auto-throttle read operations
  • [https://github.com/apache/bookkeeper/pull/2647] Limit read-ahead bytes to the size of the read cache
  • [https://github.com/apache/bookkeeper/pull/2632] Fixed NetworkTopologyImpl#getLeaves returning set with null value in case of non existing scope

Other​

Documentation, build, CI, tests improvements

Details​

https://github.com/apache/bookkeeper/issues?q=+label%3Arelease%2F4.14.0

4.13.0​

Release 4.13 improves reliability of the Stream Storage, brings additional configuration options for the Stream Storage and Prometheus HTTP Server, fixes multiple bugs and brings critical dependencies up-to-date.

Apache BookKeeper users are encouraged to upgrade to 4.13.0. The technical details of this release are summarized below.

Highlights​

Table Service (stream storage) reliability improvements​

  • [https://github.com/apache/pulsar/pull/9481] Rocksdb DLCheckpoint SST file corruption in statestore
  • [https://github.com/apache/bookkeeper/pull/2564] Fix SST file corruption
  • [https://github.com/apache/bookkeeper/pull/2566] Handling checkpoint corruption in case of bookie crash
  • [https://github.com/apache/bookkeeper/issues/2567] Save latest revision information in statestore
  • [https://github.com/apache/bookkeeper/pull/2568] Save last revision in rocksdb

Other improvements​

  • [https://github.com/apache/bookkeeper/pull/2560] Allow stream storage to use hostname instead of IP address
  • [https://github.com/apache/bookkeeper/pull/2597] Skip unavailable bookies during verifyLedgerFragment
  • [https://github.com/apache/bookkeeper/pull/2543] Allow to configure Prometheus HTTP Server bind address
  • various fixes of the tests, documentation, etc.

Dependency updates​

  • [https://github.com/apache/bookkeeper/pull/2580] Upgrade protobuf to 3.14.0
  • [https://github.com/apache/bookkeeper/pull/2582] Upgrading GRPC version to 1.33, Netty to 4.1.50Final and ETCD client driver
  • [https://github.com/apache/bookkeeper/pull/2602] Upgrading dropwizard to 3.2.5

Details​

https://github.com/apache/bookkeeper/issues?q=+label%3Arelease%2F4.13.0

4.12.1​

This is the 24th release of Apache BookKeeper, it contains a few bugfixes, new features and dependency upgrades

Apache BookKeeper users are encouraged to upgrade to 4.12.1. The technical details of this release are summarized below.

News and noteworthy​

  • [https://github.com/apache/bookkeeper/pull/2519] Allow DNSToSwitchMapping to access BookieAddressResolver
  • [https://github.com/apache/bookkeeper/pull/2493] Opportunistic Striping
  • [https://github.com/apache/bookkeeper/pull/2398] getBookieInfo is stuck if no bookie is up
  • [https://github.com/apache/bookkeeper/pull/2491] Upgrade to Curator 5.1
  • [https://github.com/apache/bookkeeper/pull/2523] Update jcommander from 1.48 to 1.78

Details​

https://github.com/apache/bookkeeper/issues?q=+label%3Arelease%2F4.12.1+

4.12.0​

This is the 23rd release of Apache BookKeeper, it is a great milestone for the project, and we are introducing a few breaking changes on the API. There are not changes on the wire protocol, on metadata and on persisted data on disks by default, so the new version is totally compatible with the previous ones. With BookKeeper 4.12.0 we are making a step toward better deployment on environments with dynamic network addresses with BP-41. We are also enhancing the new Client API by adding features that were still missing, like the ability of queryng for ledger metadata.

Apache BookKeeper users are encouraged to upgrade to 4.12.0. The technical details of this release are summarized below.

News and noteworthy​

  • [https://github.com/apache/bookkeeper/pull/1901] Enable ExplicitLAC but default on the reader side and in the New org.apache.bookkeeper.client.api.ReadHandle API
  • [https://github.com/apache/bookkeeper/issues/2396] BP-41 Bookie Network Address Change Tracking + BookieId
  • [https://github.com/apache/bookkeeper/issues/2422] BP-42 List and Access LedgerMetadata on the new API
  • [https://github.com/apache/bookkeeper/pull/2433] Support Java 11 and switch to Java 11 default Docker images
  • [https://github.com/apache/bookkeeper/pull/2455] BP-40 clean up output for tools
  • [https://github.com/apache/bookkeeper/pull/2429] Certificate role based authorization

Details​

https://github.com/apache/bookkeeper/issues?q=+label%3Arelease%2F4.12.0+

4.11.1​

Apache BookKeeper users are encouraged to upgrade to 4.11.1. The technical details of this release are summarized below.

Highlights​

  • Upgrade Netty,Vertx and RocksDB
  • Better error reporting in case of ZooKeeper related errors
  • Fix error that prevents Garbage Collections in case of corrupted EntryLogger file
  • Support for Apache ZooKeeper 3.6.x

Changes​

  • [https://github.com/apache/bookkeeper/pull/2410] Upgrade the vertx version to 3.5.3

  • [https://github.com/apache/bookkeeper/pull/2390] Issue #2385: NullPointerException in Zookeeper multiple operations execution with 3.6.1

  • [https://github.com/apache/bookkeeper/pull/2389] Issue #2197: bkctl binary distribution needs a 'logs' directory

  • [https://github.com/apache/bookkeeper/pull/2384] Track ZooKeeper errors as causes of ZKException

  • [https://github.com/apache/bookkeeper/pull/2383] fix fillReadAheadCache stat bug

  • [https://github.com/apache/bookkeeper/pull/2381] The latency of BenchThroughputLatency may be wrong due to Integer overflow when we do a large scale benchmark test

  • [https://github.com/apache/bookkeeper/pull/2380] NP check for print BookieSocketAddress and a better format

  • [https://github.com/apache/bookkeeper/pull/2379] Updated netty,netty-boringssl and rocksdb

  • [https://github.com/apache/bookkeeper/pull/2373] Issue 2264: Bookie cannot perform Garbage Collection in case of corrupted EntryLogger file

  • [https://github.com/apache/bookkeeper/pull/2327] Bookie Client add quarantine ratio when error count exceed threshold

  • [https://github.com/apache/bookkeeper/pull/2415] Spammy log when one bookie of ensemble is down

Compatiblity​

This is a point release and it does not bring API changes.

Full list of changes​

4.11.0​

This is the 21th release of Apache BookKeeper!

The 4.11.0 release incorporates hundreds of bug fixes, improvements, and features since previous major release, 4.10.0.

Apache BookKeeper/DistributedLog users are encouraged to upgrade to 4.11.0. The technical details of this release are summarized below.

News and noteworthy​

  • Upgraded ZooKeeper version from 3.4.13 to 3.5.7 with #2112
  • BookKeeper-server depends on org.apache.httpcomponents-httpcore-4.4.9 with #2156

Changes​

  • [https://github.com/apache/bookkeeper/pull/2338] Fix bookie port conflict when using LocalBookKeeper
  • [https://github.com/apache/bookkeeper/pull/2333] Handle QuorumCoverage should only count unknown nodes
  • [https://github.com/apache/bookkeeper/pull/2326] Update jackson version 2.11.0
  • [https://github.com/apache/bookkeeper/pull/2314] BP-38: Publish Bookie Service Info including all advertised addresses on Metadata Service and it is backward compatible
  • [https://github.com/apache/bookkeeper/pull/2313] add REST API to manage auto-recovery
  • [https://github.com/apache/bookkeeper/pull/2312] Support metadata decoding for list-ledger api
  • [https://github.com/apache/bookkeeper/pull/2300] files: Fix TLS with with v2 protocol
  • [https://github.com/apache/bookkeeper/pull/2297] Update Arquillian Cube to 1.18.2
  • [https://github.com/apache/bookkeeper/pull/2291] Update Prometheus library to 0.8.1
  • [https://github.com/apache/bookkeeper/pull/2205] Handle empty ledger segmant while replica-check
  • [https://github.com/apache/bookkeeper/pull/2156] Add Hostname verification for bookie-mTLS
  • [https://github.com/apache/bookkeeper/pull/2112] Update ZooKeeper dependency to 3.5.7

Full list of changes​

4.10.0​

This is the 20th release of Apache BookKeeper!

The 4.10.0 release incorporates hundreds of bug fixes, improvements, and features since previous major release, 4.9.0.

Apache BookKeeper/DistributedLog users are encouraged to upgrade to 4.10.0. The technical details of this release are summarized below.

News and noteworthy​

  • [https://github.com/apache/bookkeeper/pull/2069] Use pure python implementation of MurmurHash
  • [https://github.com/apache/bookkeeper/pull/1934] Bump Netty and GRPC version
  • [https://github.com/apache/bookkeeper/pull/1907] Add new bkctl shell tool
  • [https://github.com/apache/bookkeeper/issues/1602] Cluster Metadata Checker
  • [https://github.com/apache/bookkeeper/pull/2154] Auto refresh TLS certificate at bookie-server
  • [https://github.com/apache/bookkeeper/pull/2150] Improve journal throughput when journalSyncData is disabled.
  • [https://github.com/apache/bookkeeper/pull/2147] Journal should respect to flushWhenQueueEmpty setting
  • [https://github.com/apache/bookkeeper/pull/2132] Make default Bookie scripts work on JDK11+
  • [https://github.com/apache/bookkeeper/pull/2128] Allow to override default SASL service name 'bookkeeper'
  • [https://github.com/apache/bookkeeper/pull/2117] BookKeeper Admin API: Implement a method to get all the Bookies
  • [https://github.com/apache/bookkeeper/pull/2111] Ensure getStickyReadBookieIndex returns valid bookie index

Full list of changes​

4.9.2​

This is the 18th release of Apache BookKeeper!

The 4.9.2 release incorporates a few critical bug fixes, since previous major release, 4.9.0.

Apache BookKeeper/DistributedLog users are encouraged to upgrade to 4.9.2. The technical details of this release are summarized below.

Dependencies Changes​

No dependency change.

Bug Fixes​

Full list of changes​

4.9.1​

This is the 18th release of Apache BookKeeper!

The 4.9.1 release incorporates a few critical bug fixes, since previous major release, 4.9.0.

Apache BookKeeper/DistributedLog users are encouraged to upgrade to 4.9.1. The technical details of this release are summarized below.

Dependencies Changes​

No dependency change.

Bug Fixes​

Full list of changes​

4.9.0​

This is the 16th release of Apache BookKeeper!

The 4.9.0 release incorporates hundreds of bug fixes, improvements, and features since previous major release, 4.8.0, which was released four months ago. It is a new milestone in Apache BookKeeper community.

Apache BookKeeper/DistributedLog users are encouraged to upgrade to 4.9.0. The technical details of this release are summarized below.

Highlights​

The main features in 4.9.0 cover are around following areas:

  • Dependencies Changes
  • Public API
  • Configuration
  • Metadata
  • Table Service
  • Operations
  • Builds & Testing
  • Enhancements
  • Bug Fixes

Dependencies Changes​

Here is a list of dependencies changed in 4.9.0:

Public API​

There are multiple new client features introduced in 4.9.0. Here are two highlighted features:

  • LedgerHandleAdv exposes asyncAddEntry variant that takes ByteBuf

Configuration​

There are bunch of new settings introduced in both bookie and client in 4.9.0. Here are those settings:

Bookie​
  • serverNumIOThreads: configures the number of IO threads for bookies (see #1612)
  • The default value of fileInfoFormatVersionToWrite is bumped from 0 to 1. (see #1689)
  • The default value of journalFormatVersionToWrite is bumped from 5 to 6. (see #1689)
Client​
  • numIOThreads: configures the number of IO threads for client (see #1612)

Metadata​

There are a few big changes around metadata in 4.9.0. They are:

  • Refactor ledger metadata in LedgerHandle to make ledger metadata instance immutable (see #281)
  • Store ledger metadata in binary protobuf format (see details at #723)
  • Etcd based metadata driver implementation is in BETA release (see details at #1639)

Additionally, there are bunch of new interfaces introduced in the metadata driver API.

Table Service​

There are a lot of improvements and features introduced into the table service. The maturity of table service is moving from alpha to beta, and has started to be used as the state storage for Pulsar Functions. More table service usage will come in Pulsar's future releases.

Starting from 4.9.0, bookkeeper will release a python client for table service. See details at #1691

Operations​

HTTP Admin REST Endpoint​
  • /api/v1/bookie/gc_details is introduced to retrieve the GC details.
  • /api/v1/bookie/gc is introduced to trigger GC through HTTP REST endpoint.
BookieShell​

There are are multiple new commands are added in BookieShell. Here are a few highlighted:

  • regenerate-interleaved-storage-index-file command is introduced for rebuilding the index files for interleaved based ledger storage. (#1642)
  • ledgermetadata command now supports dumping/restoring ledger metadata to/from file.
  • localconsistencycheck command is introduce for running consistency check on bookies locally. (#1819)
  • a new bk-perf script is introduced for running performance benchmark on bookkeeper. (1697)

A new BookKeeper CLI package is released as bkctl. This bkctl package includes both the existing bookie shell and the new bkctl tool.

MDC​

Mapped Diagnostic Context (MDC) is now supported at both bookie and client sides. Application request context can be passed as context and being logged through slf4j/log4j. This simplifies throubleshooting of request-level failures/errors. See details at #1672.

Stats Annotation​

StatsDoc annotation is introduced in BP-36. The StatsDoc annotation is used for documenting stats added across the project.

Builds & Testing​

  • Java 11 is supported for building bookkeeper.

Enhancements​

Bug Fixes​

Bookie​
AutoRecovery​
Client​

Full list of changes​

4.8.2​

This is the 17th release of Apache BookKeeper!

The 4.8.2 release is a bugfix release which fixes a bunch of issues reported from users of 4.8.1.

Apache BookKeeper users who are using 4.8.1 are encouraged to upgrade to 4.8.2. The technical details of this release are summarized below.

Highlights​

Dependency Changes​

There is no dependency upgrade from 4.8.1.

Full list of changes​

4.8.1​

This is the 14th release of Apache BookKeeper!

The 4.8.1 release is a bugfix release which fixes a bunch of issues reported from users of 4.8.0.

Apache BookKeeper users who are using 4.8.0 are encouraged to upgrade to 4.8.1. The technical details of this release are summarized below.

Highlights​

Dependency Changes​

There is no dependecy upgrade from 4.8.0.

Full list of changes​

4.8.0​

This is the 13th release of Apache BookKeeper!

The 4.8.0 release incorporates hundreds of bug fixes, improvements, and features since previous major release, 4.7.0. It is a new big milestone in Apache BookKeeper community, this release include great new features, like Relaxed Durability, Stream Storage service and Multiple active Entrylogs.

Apache BookKeeper/DistributedLog users are encouraged to upgrade to 4.8.0. The technical details of this release are summarized below.

Highlights​

The main features in 4.8.0 are around following areas:

  • Durability
  • ExplicitLAC feature
  • New Table Storage Service
  • Bug Fixes

New WriteFlag DEFERRED_SYNC​

The writer may ask for temporary relaxed durability writes, that is to receive early acknowledge from Bookies, before an fsync() on Journal. Together with this new flag we introduced the new WriteHandle#force() API, this this API the writer is able to request an explicit guarantee of durability to the Bookies it is mostly like and explicit fsync() on a file system.

See DEFERRED_SYNC and force() for reference

New behaviour for Netty ByteBuf reference count management​

All the client side APIs which take ByteBufs now will have the net effect of decrementing by 1 the refcount. This is consistent with general contract of Netty. It is expected that the client passes the ownership of the ByteBuf to BookKeeper client.

Multiple Active Entrylogs​

It is now possible on the Bookie to have multiple active entry loggers, this new feature will help with compaction performance and some specific workloads.

See Multiple active entrylogs

Table Storage Service​

From this version we are providing the a table (key/value) service embedded in Bookies.

See BP-30: BookKeeper Table Service

Make ExplicitLAC persistent​

ExplicitLAC was contributed from Salesforce in 4.5.0 release, but in the first release it was a beft-effort in-memory mechanism. Now you can configure Bookies to store durably ExplicitLAC.

See Make ExplicitLAC persistent

Ensemble change on Delayed Write Failure​

We are handling more gracefully the case of a failure of a Bookie in spite of a succeeded write. If you are writing with Ack Quorum = 2 and Write Quorum = 3, writes will succeeed even if 1 of 3 Bookies fail, now BookKeeper will trigger an ensemble change and replace the failed bookie earlier.

See Ensemble change on Delayed Write Failure

Full list of changes​

4.7.3​

This is the 16th release of Apache BookKeeper!

The 4.7.3 release is a bugfix release which fixes a bunch of issues reported from users of 4.7.2.

Apache BookKeeper users who are using 4.7.2 are encouraged to upgrade to 4.7.3. The technical details of this release are summarized below.

Highlights​

Full list of changes​

4.7.2​

This is the 12th release of Apache BookKeeper!

The 4.7.2 release is a bugfix release which fixes a bunch of issues reported from users of 4.7.1.

Apache BookKeeper users who are using 4.7.1 are encouraged to upgrade to 4.7.2. The technical details of this release are summarized below.

Highlights​

Dependency Changes​

In 4.7.2, Zookeeper version is downgraded from 3.5.3-beta to 3.4.13 to avoid having a beta dependency and address maturity concerns. The downgrade is safe and smooth. No extra actions are required from switching bookkeeper 4.7.1 to 4.7.2.

Full list of changes​

4.7.1​

This is the eleventh release of Apache BookKeeper!

The 4.7.1 release is a bugfix release which fixes a bunch of issues reported from users of 4.7.0.

Apache BookKeeper users who are using 4.7.0 are encouraged to upgrade to 4.7.1. The technical details of this release are summarized below.

Highlights​

Dependencies Upgrade​

Here is a list of dependencies changed in 4.7.1:

Reflective setAccessible(true) is disabled by default in Netty while using java9+. This might result in performance degradation. Consider reenabling Reflective setAccessible(true) by setting environment value io.netty.tryReflectionSetAccessible to true. See netty/netty#7650 for more details.

Full list of changes​

4.7.0​

This is the tenth release of Apache BookKeeper!

The 4.7.0 release incorporates hundreds of bug fixes, improvements, and features since previous major release, 4.6.0, which was released four months ago. It is a big milestone in Apache BookKeeper community - Yahoo branch is fully merged back to upstream, and Apache Pulsar (incubating) starts using official BookKeeper release for its upcoming 2.0 release.

It is also the first release of Apache DistributedLog after it is merged as sub modules of Apache BookKeeper.

Apache BookKeeper/DistributedLog users are encouraged to upgrade to 4.7.0. The technical details of this release are summarized below.

Highlights​

The main features in 4.7.0 cover are around following areas:

  • Dependencies Changes
  • Public API
  • Security
  • DbLedgerStorage
  • Metadata API
  • Performance
  • Operations
  • Builds & Testing
  • Bug Fixes

Dependencies Changes​

Here is a list of dependencies changed in 4.7.0:

  • JCommander 1.48 is added as a dependency of bookkeeper-server module.
  • RocksDB 5.8.6 is introduced as part of DbLedgerStorage as a dependency of bookkeeper-server module.
  • DataSketches 0.8.3 is introduced as a dependency of prometheus-metrics-provider module.
  • Upgrade Guava from 20.0 to 21.0.

Public API​

There are multiple new client features introduced in 4.7.0. Here are two highlighted features:

Fluent API​

The new fluent style APi is evolving in 4.7.0. All the methods in handlers are now having both async and sync methods. See #1288 for more details

CRC32C​

circe-checksum module is ported from Apache Pulsar to Apache BookKeeper, and CRC32C digest type is added as one digest type option. The JNI based CRC32C in circe-checksum module provides excellent performance than existing CRC32 digest type. Users are encouraged to start use CRC32C digest type.

Security​

  • New PEM format X.509 certificates are introduced for TLS authentication. See #965 for more details.
  • TLS related settings are converged into same settings as bookie server. See Upgrade Guide for more details.

DbLedgerStorage​

DbLedgerStorage is a new ledger storage that introduced by Yahoo and now fully merged into Apache BookKeeper. It is fully compatible for both v2 and v3 protocols and also support long polling. It uses RocksDB to store ledger index, which eliminates the needed of ledger index files and reduces the number of open file descriptors and the amount of random IOs can occurs during flushing ledger index.

Metadata API​

New serviceUri based metadata API is introduced as BP-29. This metadata API provides the metadata abstraction over ledger manager, registration service, allowing plugin different type of data stores as the metadata service.

Performance​

There are a lot for performance related bug fixes and improvements in 4.7.0. Some of the changes are highlighted as below:

  • Leverage netty object recycler to reduce object allocations
  • A bunch of contentions around locking are removed. E.g. #1321 #1292 #1258
  • Introduce priority thread pool for accepting high priority reads/writes. This allows high priority reads/writes such as ledger recovery operations can succeed even bookies are overwhelmed. #898
  • Reorder slow bookies in read sequence. #883
  • Use atomic field updater and long adder to replace AtomicInteger/AtomicLong/AtomicReference in Dlog. #1299
  • DataSketches library is used for implementing prometheus provider. #1245

Operations​

BookieShell​

There are are multiple new commands are added in BookieShell. Here are a few highlighted:

  • metaformat is deprecated with two new commands initnewcluster and nukeexistingcluster. This separation provides better operability and reduces mistakes.
  • initbookie command is introduced for initializing a new bookie. bookieformat keeps serving as the purpose of reformatting a bookie.

A new BookKeeper CLI is proposed in BP-27. Some commands are already ported to new bookkeeper CLI. The full list of shell commands will be fully ported to new bookkeeper CLI in next release.

ReadOnly Mode Support​

Operations are improved around readonly mode for handling bookkeeper outage situation. New settings are introduce allow entry log creation, high priority writes even when bookies are readonly. See Upgrade Guide to learn all newly added settings.

Builds & Testing​

  • Arquillian framework is introduced in 4.7.0 for backward compatibility and integration tests.
  • Both Java8 and Java9 are now supported for running bookkeeper.

Full list of changes​

4.6.2​

This is the ninth release of BookKeeper as an Apache Top Level Project!

The 4.6.2 release is a bugfix release which fixes a bunch of issues reported from users of 4.6.1.

Apache BookKeeper users are encouraged to upgrade to 4.6.2. The technical details of this release are summarized below.

Highlights​

Dependencies Upgrade​

There is no dependency upgrade since 4.6.0, and since 4.6.1 we distribute a 'shaded' version of main artifacts, see Ledger API

Full list of changes​

4.6.1​

This is the eighth release of BookKeeper as an Apache Top Level Project!

The 4.6.1 release is a bugfix release which fixes a bunch of issues reported from users of 4.6.0.

Apache BookKeeper users are encouraged to upgrade to 4.6.1. The technical details of this release are summarized below.

Highlights​

Dependencies Upgrade​

There is no dependency upgrade since 4.6.0, but now we distribute a 'shaded' version of main artifacts, see Ledger API

Full list of changes​

4.6.0​

This is the seventh release of BookKeeper as an Apache Top Level Project!

The 4.6.0 release incorporates new fixes, improvements, and features since previous major release 4.5.0.

Apache BookKeeper users are encouraged to upgrade to 4.6.0. The technical details of this release are summarized below.

Highlights​

The main features in 4.6.0 cover are around following areas:

  • Dependencies Upgrade
  • Bookie enhancement
  • BookKeeper Admin REST API
  • New BookKeeper API
  • Performance improvement
  • Deployment or Ease of use

Dependencies Upgrade​

  • Upgrade Protobuf to 3.4.

Bookie enhancement​

  • Persistable bookie status.

    • Prior to this release, bookie status was transient. It is a bit hard for management tooling. This feature adds persistable bookies status. See Issue-265 for more details.

  • Introduce Bookie Discovery Interface. Prior to this release, bookkeeper client only provides interfaces for ledger metadata management. It doesn't provide any interface for service discovery part. This feature introduces bookie discovery interface, so it allows plugging in different service discovery backends for bookkeeper.

    • Introduce Bookie Registration Manager for bookie server, see Issue-662 for more details.
    • Introduce registration client for bookkeeper client, see Issue-666 for more details.

  • Lifecycle components for managing components in bookie server.

    • Introduce lifecycle component for each service component, which includes "stats provider", "auto recovery", "http endpoint", and "bookie server(both storage and netty server)", to run these components in a clear way. See Issue-508 and Issue-547 for more details.

  • Make bookie recovery work with recovering multiple bookies.

    • Make recovery tool work with multiple bookies, so that one call could recover multiple bookies. See Issue-612 for more details.

BookKeeper Admin REST API​

  • Introduce a bookkeeper admin endpoint for operations to interact and administer the bookkeeper cluster using REST API. see PR-278, Issue-520, and Issue-674 for more details.

New BookKeeper API​

  • New Fluent Style API.
    • A brand new API to manage ledgers using the Builder pattern, and new interfaces to make it clear operations on ledgers, like WriteHandle and ReadHandle, are provided in this release. See Issue-506, Issue-673 and Issue-550 for more details

Performance improvement​

  • Use ByteBuf in multiple places to avoid unnecessary memory allocation and reduce the garbage produced in JVM. See PR-640 for more details.

  • Separate the FileInfo cache into write and read cache. It avoids catchup reads impact tailing reads and writes. See PR-513 for more details.

Deployment or Ease of use​

  • Deployment BookKeeper on K8s.
    • Provide yaml files to run BookKeeper on Kubernetes using both StatefulSets and DaemonSet. See Issue-337 and Issue-681for more details.

Existing API changes​

  • BookKeeper constructor now throws BKException instead of KeeperException.
  • The signatures of reorderReadSequence and reorderReadLACSequence are changed in EnsemblePlacementPolicy.

Full list of changes​

4.5.1​

This is the sixth release of BookKeeper as an Apache Top Level Project!

The 4.5.1 release is a bugfix release which fixes a bunch of issues reported from users of 4.5.0.

Apache BookKeeper users are encouraged to upgrade to 4.5.1. The technical details of this release are summarized below.

Highlights​

Dependencies Upgrade​

There is no dependency upgrade since 4.5.0.

Full list of changes​

4.5.0​

This is the fifth release of BookKeeper as an Apache Top Level Project!

The 4.5.0 release incorporates hundreds of new fixes, improvements, and features since previous major release, 4.4.0, which was released over a year ago. It is a big milestone in Apache BookKeeper community, converging from three main branches (Salesforce, Twitter and Yahoo).

Apache BookKeeper users are encouraged to upgrade to 4.5.0. The technical details of this release are summarized below.

Highlights​

The main features in 4.5.0 cover are around following areas:

  • Dependencies Upgrade
  • Security
  • Public API
  • Performance
  • Operations

Dependencies Upgrade​

Here is a list of dependencies upgraded in 4.5.0:

  • Moved the developement from Java 7 to Java 8.
  • Upgrade Protobuf to 2.6.
  • Upgrade ZooKeeper from 3.4 to 3.5.
  • Upgrade Netty to 4.1.
  • Upgrade Guava to 20.0.
  • Upgrade SLF4J to 1.7.25.
  • Upgrade Codahale to 3.1.0.

Security​

Prior to this release, Apache BookKeeper only supports simple DIGEST-MD5 type authentication.

With this release of Apache BookKeeper, a number of feature are introduced that can be used, together of separately, to secure a BookKeeper cluster.

The following security features are currently supported.

  • Authentication of connections to bookies from clients, using either TLS or `SASL (Kerberos).
  • Authentication of connections from clients, bookies, autorecovery daemons to ZooKeeper, when using zookeeper based ledger managers.
  • Encryption of data transferred between bookies and clients, between bookies and autorecovery daemons using TLS.

It's worth noting that those security features are optional - non-secured clusters are supported, as well as a mix of authenticated, unauthenticated, encrypted and non-encrypted clients.

For more details, have a look at BookKeeper Security.

Public API​

There are multiple new client features introduced in 4.5.0.

LedgerHandleAdv​

The [Ledger API] is the low level API provides by BookKeeper for interacting with ledgers in a bookkeeper cluster. It is simple but not flexible on ledger id or entry id generation. Apache BookKeeper introduces LedgerHandleAdv as an extension of existing LedgerHandle for advanced usage. The new LedgerHandleAdv allows applications providing its own ledger-id and assigning entry-id on adding entries.

See Ledger Advanced API for more details.

Long Poll​

Long Poll is a main feature that DistributedLog uses to achieve low-latency tailing. This big feature has been merged back in 4.5.0 and available to BookKeeper users.

This feature includes two main changes, one is LastAddConfirmed piggyback, while the other one is a new long poll read API.

The first change piggyback the latest LastAddConfirm along with the read response, so your LastAddConfirmed will be automatically advanced when your read traffic continues. It significantly reduces the traffic to explicitly polling LastAddConfirmed and hence reduces the end-to-end latency.

The second change provides a new long poll read API, allowing tailing-reads without polling LastAddConfirmed everytime after readers exhaust known entries. Although long poll API brings great latency improvements on tailing reads, it is still a very low-level primitive. It is still recommended to use high level API (e.g. DistributedLog API) for tailing and streaming use cases.

See Streaming Reads for more details.

Explicit LAC​

Prior to 4.5.0, the LAC is only advanced when subsequent entries are added. If there is no subsequent entries added, the last entry written will not be visible to readers until the ledger is closed. High-level client (e.g. DistributedLog) or applications has to work around this by writing some sort of control records to advance LAC.

In 4.5.0, a new explicit lac feature is introduced to periodically advance LAC if there are not subsequent entries added. This feature can be enabled by setting explicitLacInterval to a positive value.

Performance​

There are a lot for performance related bug fixes and improvements in 4.5.0. These changes includes:

  • Upgraded netty from 3.x to 4.x to leverage buffer pooling and reduce memory copies.
  • Moved developement from Java 7 to Java 8 to take advantage of Java 8 features.
  • A lot of improvements around scheduling and threading on bookies.
  • Delay ensemble change to improve tail latency.
  • Parallel ledger recovery to improve the recovery speed.
  • ...

We outlined following four changes as below. For a complete list of performance improvements, please checkout the full list of changes at the end.

Netty 4 Upgrade​

The major performance improvement introduced in 4.5.0, is upgrading netty from 3.x to 4.x.

For more details, please read upgrade guide about the netty related tips when upgrading bookkeeper from 4.4.0 to 4.5.0.

Delay Ensemble Change​

Ensemble Change is a feature that Apache BookKeeper uses to achieve high availability. However it is an expensive metadata operation. Especially when Apache BookKeeper is deployed in a multiple data-centers environment, losing a data center will cause churn of metadata operations due to ensemble changes. Delay Ensemble Change is introduced in 4.5.0 to overcome this problem. Enabling this feature means an Ensemble Change will only occur when clients can't receive enough valid responses to satisfy ack-quorum constraint. This feature improves the tail latency.

To enable this feature, please set delayEnsembleChange to true on your clients.

Parallel Ledger Recovery​

BookKeeper clients recovers entries one-by-one during ledger recovery. If a ledger has very large volumn of traffic, it will have large number of entries to recover when client failures occur. BookKeeper introduces parallel ledger recovery in 4.5.0 to allow batch recovery to improve ledger recovery speed.

To enable this feature, please set enableParallelRecoveryRead to true on your clients. You can also set recoveryReadBatchSize to control the batch size of recovery read.

Multiple Journals​

Prior to 4.5.0, bookies are only allowed to configure one journal device. If you want to have high write bandwidth, you can raid multiple disks into one device and mount that device for jouranl directory. However because there is only one journal thread, this approach doesn't actually improve the write bandwidth.

BookKeeper introduces multiple journal directories support in 4.5.0. Users can configure multiple devices for journal directories.

To enable this feature, please use journalDirectories rather than journalDirectory.

Operations​

LongHierarchicalLedgerManager​

Apache BookKeeper supports pluggable metadata store. By default, it uses Apache ZooKeeper as its metadata store. Among the zookeeper-based ledger manager implementations, HierarchicalLedgerManager is the most popular and widely adopted ledger manager. However it has a major limitation, which it assumes ledger-id is a 32-bits integer. It limits the number of ledgers to 2^32.

LongHierarchicalLedgerManager is introduced to overcome this limitation.

See Ledger Manager for more details.

Weight-based placement policy​

Rack-Aware and Region-Aware placement polices are the two available placement policies in BookKeeper client. It places ensembles based on users' configured network topology. However they both assume that all nodes are equal. weight-based placement is introduced in 4.5.0 to improve the existing placement polices. weight-based placement was not built as separated polices. It is built in the existing placement policies. If you are using Rack-Aware or Region-Aware, you can simply enable weight-based placement by setting diskWeightBasedPlacementEnabled to true.

Customized Ledger Metadata​

A Map<String, byte[]> is introduced in ledger metadata in 4.5.0. Clients now are allowed to pass in a key/value map when creating ledgers. This customized ledger metadata can be later on used by user defined placement policy. This extends the flexibility of bookkeeper API.

Add Prometheus stats provider​

A new Prometheus stats provider is introduce in 4.5.0. It simplies the metric collection when running bookkeeper on kubernetes.

Add more tools in BookieShell​

BookieShell is the tool provided by Apache BooKeeper to operate clusters. There are multiple importants tools introduced in 4.5.0, for example, decommissionbookie, expandstorage, lostbookierecoverydelay, triggeraudit.

For the complete list of commands in BookieShell, please read BookKeeper CLI tool reference.

Full list of changes​

JIRA​

Sub-task​
Bug
  • [BOOKKEEPER-852] - Release LedgerDescriptor and master-key objects when not used anymore
  • [BOOKKEEPER-903] - MetaFormat BookieShell Command is not deleting UnderReplicatedLedgers list from the ZooKeeper
  • [BOOKKEEPER-907] - for ReadLedgerEntriesCmd, EntryFormatter should be configurable and HexDumpEntryFormatter should be one of them
  • [BOOKKEEPER-908] - Case to handle BKLedgerExistException
  • [BOOKKEEPER-924] - addEntry() is susceptible to spurious wakeups
  • [BOOKKEEPER-927] - Extend BOOKKEEPER-886 to LedgerHandleAdv too (BOOKKEEPER-886: Allow to disable ledgers operation throttling)
  • [BOOKKEEPER-933] - ClientConfiguration always inherits System properties
  • [BOOKKEEPER-938] - LedgerOpenOp should use digestType from metadata
  • [BOOKKEEPER-939] - Fix typo in bk-merge-pr.py
  • [BOOKKEEPER-940] - Fix findbugs warnings after bumping to java 8
  • [BOOKKEEPER-952] - Fix RegionAwarePlacementPolicy
  • [BOOKKEEPER-955] - in BookKeeperAdmin listLedgers method currentRange variable is not getting updated to next iterator when it has run out of elements
  • [BOOKKEEPER-956] - HierarchicalLedgerManager doesn't work for ledgerid of length 9 and 10 because of order issue in HierarchicalLedgerRangeIterator
  • [BOOKKEEPER-958] - ZeroBuffer readOnlyBuffer returns ByteBuffer with 0 remaining bytes for length > 64k
  • [BOOKKEEPER-959] - ClientAuthProvider and BookieAuthProvider Public API used Protobuf Shaded classes
  • [BOOKKEEPER-976] - Fix license headers with "Copyright 2016 The Apache Software Foundation"
  • [BOOKKEEPER-980] - BookKeeper Tools doesn't process the argument correctly
  • [BOOKKEEPER-981] - NullPointerException in RackawareEnsemblePlacementPolicy while running in Docker Container
  • [BOOKKEEPER-984] - BookieClientTest.testWriteGaps tested
  • [BOOKKEEPER-986] - Handle Memtable flush failure
  • [BOOKKEEPER-987] - BookKeeper build is broken due to the shade plugin for commit ecbb053e6e
  • [BOOKKEEPER-988] - Missing license headers
  • [BOOKKEEPER-989] - Enable travis CI for bookkeeper git
  • [BOOKKEEPER-999] - BookKeeper client can leak threads
  • [BOOKKEEPER-1013] - Fix findbugs errors on latest master
  • [BOOKKEEPER-1018] - Allow client to select older V2 protocol (no protobuf)
  • [BOOKKEEPER-1020] - Fix Explicit LAC tests on master
  • [BOOKKEEPER-1021] - Improve the merge script to handle github reviews api
  • [BOOKKEEPER-1031] - ReplicationWorker.rereplicate fails to call close() on ReadOnlyLedgerHandle
  • [BOOKKEEPER-1044] - Entrylogger is not readding rolled logs back to the logChannelsToFlush list when exception happens while trying to flush rolled logs
  • [BOOKKEEPER-1047] - Add missing error code in ZK setData return path
  • [BOOKKEEPER-1058] - Ignore already deleted ledger on replication audit
  • [BOOKKEEPER-1061] - BookieWatcher should not do ZK blocking operations from ZK async callback thread
  • [BOOKKEEPER-1065] - OrderedSafeExecutor should only have 1 thread per bucket
  • [BOOKKEEPER-1071] - BookieRecoveryTest is failing due to a Netty4 IllegalReferenceCountException
  • [BOOKKEEPER-1072] - CompactionTest is flaky when disks are almost full
  • [BOOKKEEPER-1073] - Several stats provider related changes.
  • [BOOKKEEPER-1074] - Remove JMX Bean
  • [BOOKKEEPER-1075] - BK LedgerMetadata: more memory-efficient parsing of configs
  • [BOOKKEEPER-1076] - BookieShell should be able to read the 'FENCE' entry in the log
  • [BOOKKEEPER-1077] - BookKeeper: Local Bookie Journal and ledger paths
  • [BOOKKEEPER-1079] - shell lastMark throws NPE
  • [BOOKKEEPER-1098] - ZkUnderreplicationManager can build up an unbounded number of watchers
  • [BOOKKEEPER-1101] - BookKeeper website menus not working under https
  • [BOOKKEEPER-1102] - org.apache.bookkeeper.client.BookKeeperDiskSpaceWeightedLedgerPlacementTest.testDiskSpaceWeightedBookieSelectionWithBookiesBeingAdded is unreliable
  • [BOOKKEEPER-1103] - LedgerMetadataCreateTest bug in ledger id generation causes intermittent hang
  • [BOOKKEEPER-1104] - BookieInitializationTest.testWithDiskFullAndAbilityToCreateNewIndexFile testcase is unreliable
ImprovementNew FeatureStoryTaskTestWish
  • [BOOKKEEPER-943] - Reduce log level of AbstractZkLedgerManager for register/unregister ReadOnlyLedgerHandle

Github​