Huge Domino db ordeals

As co-founders and investors in Yabot we were called in to assist with a problem some weeks ago regarding two of its editions. Each Yabot edition runs in its own Domino database and can after some years build up to a substantial amount of data.
In this case we’re talking about a 12 GB nsf-file containing almost 3 million documents…

As this system is very intensive on adding and deleting documents it puts the Domino database structure to some serious tests. We see 10’s of thousands of additions daily and just about the same amount of deletions. This will for example make the internal table for deletions stubs crash more often than needed. It also make the view indexer and freetext indexer work hard.

In this particular case several indexes was corrupted and neither compact nor fixup made it any better. The db is on a three server cluster but unfortunately all replicas was effected. Running a fixup and/or compact on a 12 GB db can actually take a full working day – and we did try this with all options we could figure out. And still no luck.

The database information tell us it is used to 71% and that would mean that 8.5 GB of the size is actually used, right?
Well after struggling with this for quite some time someone came up with the idea of taking a new empty nsf and copy every single document to it using an agent. This turned out to be a fantastic idea, part from all docs getting a new unid and the db a new replica id.
So this is how we did it:

  1. Create a totally new empty nsf-db
  2. Create an agent copying all documents from a view in the original db containing all doc’s.
    Building the view index for that view took 11 hours (!!) on my laptop.
  3. Run the agent copying all doc to the empty db.
    Took some 4-5 hours on my laptop.
  4. Replace the design with the original design and move the db to the servers
  5. Done! Took quite some time but the result was good

The really remarkable thing though is that the new nsf-file (containing all the same documents) just take up about 2 GB – that is a 93% decrease from the original 12 GB. And how could Domino say the there was just 29% white space in the original?