Comments on: BloomFilter http://blog.rapleaf.com/dev/2007/09/05/bloomfilter/ For engineers, by engineers. Thu, 09 Feb 2012 03:22:26 +0000 hourly 1 By: Bloom filters demystified :P | padlobatch http://blog.rapleaf.com/dev/2007/09/05/bloomfilter/#comment-12643 Bloom filters demystified :P | padlobatch Thu, 08 Dec 2011 21:45:30 +0000 http://blog.rapleaf.com/dev/?p=6#comment-12643 [...] http://blog.rapleaf.com/dev/2007/09/05/bloomfilter/ Some random guy used it. It has pretty good benefits in real life scenarios. It just makes things easy-peasy. [...] [...] http://blog.rapleaf.com/dev/2007/09/05/bloomfilter/ Some random guy used it. It has pretty good benefits in real life scenarios. It just makes things easy-peasy. [...]

]]> By: Filters for fridges http://blog.rapleaf.com/dev/2007/09/05/bloomfilter/#comment-9093 Filters for fridges Sun, 06 Nov 2011 16:02:52 +0000 http://blog.rapleaf.com/dev/?p=6#comment-9093 Would making the filter as big as possible have any effect on the speed and performance? Would making the filter as big as possible have any effect on the speed and performance?

]]> By: tripzilch http://blog.rapleaf.com/dev/2007/09/05/bloomfilter/#comment-382 tripzilch Mon, 08 Nov 2010 10:05:06 +0000 http://blog.rapleaf.com/dev/?p=6#comment-382 You shouldn't have used SHA1. The only hashing criterion you need for a Bloom Filter is a uniform distribution over the hash output space. While a cryptographic hash function most certainly satisfies this [it has to, otherwise it would be terribly insecure], it's also a littlebit like killing a fly with a bazooka. Next time try it with the FNV hash, which only uses one XOR and one MUL per byte of hashed data, making it a few orders of magnitude faster than SHA1. (The FNV hash is not cryptographically secure, and has slightly imperfect avalanche behaviour, but will give you a uniform distribution over output space. If you think "naw this seems too easy it can't possibly work", then apply it twice just to be safe, and I guarantee it will be just fine for a Bloom Filter) You shouldn’t have used SHA1.

The only hashing criterion you need for a Bloom Filter is a uniform distribution over the hash output space.

While a cryptographic hash function most certainly satisfies this [it has to, otherwise it would be terribly insecure], it’s also a littlebit like killing a fly with a bazooka. Next time try it with the FNV hash, which only uses one XOR and one MUL per byte of hashed data, making it a few orders of magnitude faster than SHA1.

(The FNV hash is not cryptographically secure, and has slightly imperfect avalanche behaviour, but will give you a uniform distribution over output space. If you think “naw this seems too easy it can’t possibly work”, then apply it twice just to be safe, and I guarantee it will be just fine for a Bloom Filter)

]]> By: Bloom Filter – It’s Variants- and their Applications « Appolo85's Blog http://blog.rapleaf.com/dev/2007/09/05/bloomfilter/#comment-372 Bloom Filter – It’s Variants- and their Applications « Appolo85's Blog Tue, 03 Aug 2010 11:01:17 +0000 http://blog.rapleaf.com/dev/?p=6#comment-372 [...] This post from Rapleaf  was an interesting read. [...] [...] This post from Rapleaf  was an interesting read. [...]

]]> By: ehsanul http://blog.rapleaf.com/dev/2007/09/05/bloomfilter/#comment-362 ehsanul Fri, 21 Aug 2009 19:36:48 +0000 http://blog.rapleaf.com/dev/?p=6#comment-362 Bryan, I think there's a way to fake growing an filter, by automating the processing of creating a new one. I'm not actually quite sure how it works, but check out this Python "scalable bloom filter" library: http://github.com/jaybaird/python-bloomfilter/tree/master Quote from the readme: pybloom is a module that includes a Bloom Filter data structure along with an implmentation of Scalable Bloom Filters as discussed in: P. Almeida, C.Baquero, N. Preguiça, D. Hutchison, Scalable Bloom Filters, (GLOBECOM 2007), IEEE, 2007. Bloom filters are great if you understand what amount of bits you need to set aside early to store your entire set. Scalable Bloom Filters allow your bloom filter bits to grow as a function of false positive probability and size. Bryan, I think there’s a way to fake growing an filter, by automating the processing of creating a new one. I’m not actually quite sure how it works, but check out this Python “scalable bloom filter” library: http://github.com/jaybaird/python-bloomfilter/tree/master

Quote from the readme:

pybloom is a module that includes a Bloom Filter data structure along with
an implmentation of Scalable Bloom Filters as discussed in:

P. Almeida, C.Baquero, N. Preguiça, D. Hutchison, Scalable Bloom Filters,
(GLOBECOM 2007), IEEE, 2007.

Bloom filters are great if you understand what amount of bits you need to set
aside early to store your entire set. Scalable Bloom Filters allow your bloom
filter bits to grow as a function of false positive probability and size.

]]> By: bryan http://blog.rapleaf.com/dev/2007/09/05/bloomfilter/#comment-352 bryan Thu, 09 Jul 2009 15:16:47 +0000 http://blog.rapleaf.com/dev/?p=6#comment-352 @dkindlund: Sadly, there's no way of growing a bloom filter to a larger size, since you lose the original items and hash values. When your first filter gets too full, you have to make a whole new one. For this reason, I recommend making your filters big enough to accommodate as many items as you could possibly imagine being in the set. This usually doesn't lead to impossibly large filters and it will save you a real headache down the road. @dkindlund:
Sadly, there’s no way of growing a bloom filter to a larger size, since you lose the original items and hash values. When your first filter gets too full, you have to make a whole new one. For this reason, I recommend making your filters big enough to accommodate as many items as you could possibly imagine being in the set. This usually doesn’t lead to impossibly large filters and it will save you a real headache down the road.

]]> By: dkindlund http://blog.rapleaf.com/dev/2007/09/05/bloomfilter/#comment-342 dkindlund Thu, 09 Jul 2009 15:03:51 +0000 http://blog.rapleaf.com/dev/?p=6#comment-342 Hi Bryan, Since there's an upper-limit to how many items you can store in an initial BloomFilter, how would you recommend growing the BloomFilter once you've exceeded the original size? Could you create new, larger BloomFilter and then somehow UNION the two so that you don't lose your history from the first BloomFilter? Any examples would be helpful. Hi Bryan,
Since there’s an upper-limit to how many items you can store in an initial BloomFilter, how would you recommend growing the BloomFilter once you’ve exceeded the original size?

Could you create new, larger BloomFilter and then somehow UNION the two so that you don’t lose your history from the first BloomFilter? Any examples would be helpful.

]]> By: bryan http://blog.rapleaf.com/dev/2007/09/05/bloomfilter/#comment-332 bryan Tue, 08 Jan 2008 17:55:30 +0000 http://blog.rapleaf.com/dev/?p=6#comment-332 @Dan: Right now I have no plans to make "Internet-serializable" filters, but it would really be a trivial thing to do. My implementation uses a byte array under the hood, so all you'd have to do is base64 each byte in the array. You're welcome to submit a patch :) @Dan: Right now I have no plans to make “Internet-serializable” filters, but it would really be a trivial thing to do. My implementation uses a byte array under the hood, so all you’d have to do is base64 each byte in the array. You’re welcome to submit a patch :)

]]> By: Dan Brickley http://blog.rapleaf.com/dev/2007/09/05/bloomfilter/#comment-322 Dan Brickley Tue, 08 Jan 2008 16:34:54 +0000 http://blog.rapleaf.com/dev/?p=6#comment-322 This is great :) I was using the old BloominSimple thing last year. Glad to see something packaged now. Do you have any plans to make these blooms serializable in an Internet-transport-friendly form, eg. base64? The old LOAF project had something in that vein, see http://loaf.cantbedone.org/download.htm and I'm wondering about including "who my buddies are" blooms in FOAF files if we can figure an encoding, per http://www.perl.com/pub/a/2004/04/08/bloom_filters.html?page=2 Unfortunately bit-level stuff makes my head hurt! I'll get there eventually but wondered what your plans are... This is great :) I was using the old BloominSimple thing last year. Glad to see something packaged now.

Do you have any plans to make these blooms serializable in an Internet-transport-friendly form, eg. base64? The old LOAF project had something in that vein, see http://loaf.cantbedone.org/download.htm and I’m wondering about including “who my buddies are” blooms in FOAF files if we can figure an encoding, per http://www.perl.com/pub/a/2004/04/08/bloom_filters.html?page=2

Unfortunately bit-level stuff makes my head hurt! I’ll get there eventually but wondered what your plans are…

]]> By: Twitter Updates for 2007-11-17 at blogwi.se http://blog.rapleaf.com/dev/2007/09/05/bloomfilter/#comment-312 Twitter Updates for 2007-11-17 at blogwi.se Sun, 18 Nov 2007 06:44:33 +0000 http://blog.rapleaf.com/dev/?p=6#comment-312 [...] Filter for ruby (a gem that is): http://blog.rapleaf.com/dev/?p=6 [...] [...] Filter for ruby (a gem that is): http://blog.rapleaf.com/dev/?p=6 [...]

]]>