Deduplication
Improving Performance and Extending Tool Lifespan
Tapping multiple points across a network can result in the same packet being copied several times, particularly with East/West traffic and traffic from SPAN ports. This can negatively affect the performance of various tools as they not only need to process the same traffic multiple times, but in many cases further expend processing resources performing deduplication themselves. In instances where packet data is being recording to storage or a ring buffer, duplicate packets simply consume space without additional benefit. In some cases, the duplicate packets could even cause false positives or skew reporting.
Deduplication with Omnia
Cubro provides several options capable of performing deduplication, according to user-selectable parameters, prior to forwarding traffic to monitoring tools. This offers customers flexibility in selecting units that meet their needs for port density and bandwidth without expending unnecessary budget. Offloading this resource intensive task frees other tools processing resources leading to increased efficiency in performing the functions they were designed to handle.
Omnia is a versatile and cost-effective device that offers deduplication for 10/40/100Gbps links to eliminate identical packets and thus protect monitoring equipment from being overloaded. With the deduplication function enabled, the Omnia will calculate a checksum, or hash-key, using the full packet. Using every bit in the packet ensures that the resulting hash represents the maximum amount of uniqueness in the packet. This is important in preventing a different packet, that varies in only the slightest way, from being discarded because the hashing algorithm only used a subset of the packet data. The hash-key itself is an MD5 calculation and results in a 16 byte long string. Hash-keys are stored (one per packet) and every incoming packet has its hash-key checked against the hashes stored in memory. If the hash of an arriving packet is found to have a match already stored in memory the incoming packet is dropped, ensuring that duplicated packets within the specified time windows are not forwarded to the output.
Benefits of Cubro Deduplication Solution
- Improved efficiency of monitoring and analytic tools.
- Maintain accuracy of tool reporting
- Offload function from tools to conserve processing resources
Additional Resources
Our newsletter provides thought leadership content about the industry. It is concise and has interesting content to keep you updated with what’s new at Cubro and in the industry. You can unsubscribe anytime with a single click.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.