Skip to main content

Bin policy

XDR version 5.0 introduced the bin projection functionality (via the ship-only-specified-bins configuration parameter). Starting with version 5.2, a new functionality was added to ship the changed bins. These two functionalities create interesting combinations which can cater to different use-cases. The new policy introduced in version 5.2 is called bin-policy. The ship-only-specified-bins configuration is removed in 5.2 and the new bin-policy supersedes it with extra functionality. Allowed values for this config are:

Config ValueExplanation
all (default)This is the default mode. The entire record will be shipped to the destination always.
no-bins[Introduced in Server 6.0]. Only allowed for connector type DCs. Bins will not be shipped, only record's metadata will be shipped to the destination.
only-changedOnly the changed bins in a write will be shipped. The bins which did not change will not get shipped.
only-specified[Removed as of version 5.3]. Only the bins specified via the ship-bin config will be shipped no matter which bins got changed as part of the write.
changed-and-specifiedIt is like the && operator. In other words, it is the intersection of changed and specified bins. A bin will be shipped if it got changed and also is specified via the ship-bin config. The bins which got changed but not specified via ship-bin config will not be shipped
changed-or-specifiedIt is like the || operator. In other words, it is the union of changed and specified bins. Both the changed bins in a write and the list of bins specified via the ship-bin config will be shipped

Overhead

If the XDR bin-policy is set to only-changed, changed-and-specified or changed-or-specified, there will be extra overhead because we need to maintain extra information to provide the necessary functionality. The overhead is:

  1. 6 bytes for each bin to maintain last-updated-time (LUT) and some metadata.
  2. deleted bins will become bin tombstones. A bin tombstone costs the same as a regular bin with no data. These bin tombstones will be cleaned up only on a subsequent write based on the xdr-bin-tombstone-ttl configuration.

Refer to the Capacity Planning Guide for further details.

Note that the bin-policy config is at the dc/namespace sub-context. If at least one dc/namespace sub-context uses the above policies, there will be extra overhead. At a later point in time, if none of the dc/namespace level configs uses the above policies (switches to all), the overhead will be removed on a subsequent write operation as the extra information is no longer necessary.

Interactions

This bin-policy config has interactions with other configs like ignore-bins, write-policy and connector. Some combinations of these do not make sense and are restricted. The general philosophy is as follows:

  • If the user chooses a bin-policy which will pick specified bins (via ship-bin config) ignore-bin config is not honored at the same time.
  • If the user chooses a bin-policy which ships changed bins they cannot set write-policy to replace
  • If the user chooses the no-bins bin-policy it will be allowed only if connector config is true.
Config Valueignore-bin honored?write-policy=replace allowed?connector
all (default)yesyesany
no-binsNAnotrue
only-changedyesnoany
only-specifiednonoany
changed-and-specifiednonoany
changed-or-specifiednonoany

Examples

The following examples demonstrate the rules and the interactions between bin-policy, ship-bin, and ignore-bin. If the state of record and config are as follows.

  • Record has 8 bins : b1, b2, b3, b4, b5, b6, b7, b8
  • Bins configured via ship-bin : b1, b5
  • Bins configured via ignore-bin : b2, b6

Case-1

Say a write comes for b1, b2, b3. Lets see (pictorially) which bins get shipped under different policies.

b1b2b3b4b5b6b7b8
ship-binship-bin
ignore-binignore-bin
writewritewrite
-----------------------------------------------------------------------------------------------------
allshipsshipsshipsshipsshipsships
only-changedshipsships
only-specifiedshipsships
changed-and-specifiedships
changed-or-specifiedshipsshipsshipsships

Case-2

Say a write comes for b6, b7, b8. Lets see (pictorially) which bins get shipped under different policies.

b1b2b3b4b5b6b7b8
ship-binship-bin
ignore-binignore-bin
writewritewrite
-----------------------------------------------------------------------------------------------------
allshipsshipsshipsshipsshipsships
only-changedshipsships
only-specifiedshipsships
changed-and-specified
changed-or-specifiedshipsshipsshipsshipsships

Use cases

Below are example configs for some common usecases

Shipping specific bins if they changed

In this example, for namespace someNameSpaceName, only binName1 and binName2 are shipped.

xdr {
dc dataCenter1 {
node-address-port someIpAdress1 somePort1
namespace someNameSpaceName {
bin-policy changed-and-specified
ship-bin binName1
ship-bin binName2
}
}
}

Excluding specific bins

In this example, for namespace someNameSpaceName, all bins are shipped except binName1 and binName2.

xdr {
dc dataCenter1 {
node-address-port someIpAdress1 somePort1
namespace someNameSpaceName {
ignore-bin binName1
ignore-bin binName2
}
}
}

Ship changed bins

In this example, for namespace someNameSpaceName, all bins that are modified by a write will be shipped.

xdr {
dc dataCenter1 {
node-address-port someIpAdress1 somePort1
namespace someNameSpaceName {
bin-policy only-changed
}
}
}

Ship changed bins plus some additional bins

In this example, for namespace someNameSpaceName, all bins that are modified by a write plus binName1 and binName2 will be shipped.

xdr {
dc dataCenter1 {
node-address-port someIpAdress1 somePort1
namespace someNameSpaceName {
bin-policy changed-or-specified
ship-bin binName1
ship-bin binName2
}
}
}

Ship changed bins but not some bins

In this example, for namespace someNameSpaceName, all bins that are modified by a write except binName1 and binName2 will be shipped.

xdr {
dc dataCenter1 {
node-address-port someIpAdress1 somePort1
namespace someNameSpaceName {
bin-policy only-changed
ignore-bin binName1
ignore-bin binName2
}
}
}