When Personalization Suddenly Stops Working in Sitecore 8

 Recently, I ran into a situation where personalization decided to quit working on a client-provisioned staging environment. Engagement value didn't increment, rules didn't fire, and we were tearing our hair out to figure out why.

I went through the usual battery of checks, and when looking into the reporting / secondary databases it appeared there wasn't a secondary.

"That's funny," I thought to myself. "Well, I'll just add one and manually rebuild the databases as it mentions in the documentation. However, that process would never finish.

Problem: In addition to the obvious "personalization not working" bit, the manual database rebuild got stuck on the ClearingStorage step.  I extended the time of the TimeToClearStorage parameter in Sitecore.Analytics.Processing.Aggregation.config file a few times up to multiple hours, and every attempt still ran out of time.

After having a few other people weigh in the issue, we finally discovered two odd things:

1) The Sitecore.Data.DataAccess.dll was one version too old - from a Sitecore 8.0 Update-4 (we were on Update-5 at the time).  This was particularly puzzling as this was a new instance that hadn't ever been upgraded to my knowledge.

2) Our Profile Key definitions had disallowed characters in their names. In certain cases, we had keys like "Time and Expense" as the name of the Sitecore item, but the profile name (under Data) had "Time & Expense."  It appears the formats of those two descriptors need to match, and it did not like the ampersand in the profile name.

Solution: Check your DLL versions and the names of your profile keys!  Pulling the correct version of the Sitecore.Data.DataAccess.dll from the .zip of the site root and cleaning special characters from all our key names did the trick.