The reason that requires care to work well is that a solution that didn’t care about this issue might well have a job whose reducer sees a large number of records with a single key on the last iteration. This would be bad because map/reduce jobs are run on large clusters with hundreds of machines. Each machine runs the mapper code on a subset of the reducer output data, but all of the records with the same key have to go to the same reducer instance — leaving the rest of the reducer machines idle if most of the records share a common key.

-dk

I believe I have a solution that has more iterations than that, but for which after the first O(log(log(n)) iterations the size of the map/reduce jobs gets smaller and smaller, decreasing exponentially, so the total number of records processed through my algo is something like O(m * log(m) * log(log(n))) which seriously trumps O(m * log(m) * log(n)) .

I have submitted my solution to Nathan to referee.

-dk

Job 1: Create two tuples per edge.

Map:
For each edge (a,b)
Let M = min(a,b)
Emit (a, M, a, b)
Emit (b, M, a, b)

Reduce:
None

Job 2: Find the smallest node ID that holds a reference to a node.

Map:
For each tuple (n, M, a, b)
Emit K = n, V = (M, a, b)

Reduce:
1. Iterate over all V
Let M’ = min (V.M, M’)

2. Iterate over all V,
Emit (V.a, M’, V.a, V.b)
Emit (V.b, M’, V.a, V.b)

Repeat Job 2 until there are no passes where M’ > M.

Job 3: Cleanup (the reduce step here is just to ensure a unique line per node)

Map:
For each tuple (n, M, a, b)
Emit K = n, V = M

Reduce:
Emit (K, min (V))

I have some questions:

1) I’m curious how many vertices are in the largest connected component?

Presuming your graph has n = ~1B and m = ~10B+ (undirected), and presuming all vertices are of the same general class (e.g. email addresses that are in some way associated), wouldn’t you expect 95%+ of the vertices to be in the largest component?

i.e. When the average number of links per vertex is > 1, the fraction of all vertices in the largest component should increase quickly, even in a highly clustered graph, presuming at least some ties are non-local. (eg articles by Watts, Strogatz, Newman, Kleinberg, etc)

2) Do you wish to distinguish between strongly and weakly connected components for the sake of this problem? I know you use undirected ties in the example algorithm, but the underlying data seems to be directed, which could be neat for calculating diffusion probability scores or shannon entropy over time.

3) Presuming the ultimate goal is to identify cohesive subgroups in a graph that reflect some group shared identity or pattern of homophily (eg propensity to purchase a product, donate to a political cause, etc), I believe subgroup measures would likely better identify meaningful social groups than the connected components (as the CC is so broad, homophily predictions based on it will have a lot of noise).

Therefore, I’m curious what other network structure measures you have implemented or are considering?