Brief mail problems on Tigris, now fixed

For a few minutes today, around 1:15 PM PDT, users may have experienced some mail bouncing from Tigris.Org. The problem, a failed mail hum / spam blocker, was fixed by about 1:30. Service should now be fully restored.

January Maintenance: An Update

Here are additional details on plans for upgrading the Tigris site:

• Add additional memory to improve performance: 1/21. Complete.
• Set up a “staging server” (or “beta,” if you like) for Project Owners and users to test: 1/26 – 2/5
• Upgrade Tigris.Org to CEE 5.3.0 patch 2 ‘Indus’

Not that “staging server” item, there. We’ll be asking all Tigris users to beat up the staging server for about ten days, to help us ensure a smooth transition. Stay tuned here, in Twitter (@tigrisdotorg), or on the “announce” list at tigris.org.

“Service Temporarily Unavailable” (status 503) from SVN

Subversion service on Tigris is currently running at reduced capacity, and may return an error 503 “Service Temporarily Unavailable.”  This is because we have throttled Subversion access, per project, a bit while we work on a memory-leakage bug just discovered.

If you encounter this problem, please wait a minute or so and retry your operation. If that’s unsuccessful, contact us, and we’ll look into raising the threshold for your project. You can do that via comments on this blog, through the Twitter account @tigrisdotorg, or via email to the feedback address at tigris.org.

The fix for the underlying problem is known, and already proven in several other production sites. We’re working on getting it installed onto Tigris as quickly as possible.

Download links looping

Several users have reported that download links from the “Documents & files” areas of Tigris are misbehaving.  URLs that include “ProjectDocumentList” or “/files/documents/NN/NNNN/” are affected. The problem only seems to afflict Firefox, which reports:

Firefox has detected that the server is redirecting the request for this address in a way that will never complete.

We are investigating, but at present we are not able to reproduce the symptom.

If you encounter this problem, it might help if you could provide some information about your experience here in the comments, such as

• browser version, platform, plug-ins
• how you got there (your last few clicks)
• is it persistent, or does it go away
• is there anything unusual at your end, such as a web proxy
• date, time, and file path you attempted (so we can grub logs)

Tigris.Org mail-lossage patch today

Tigris will be down for about one hour, beginning at 15:30 PDT today, to apply patches to fix lingering mail loss problems.

Update: back up.

Tigris.org: memory replacement

Tigris is experiencing a hardware error in its memory banks. It will be going down for replacement at 16:15 PDT. Should be back up within a few minutes.

Update: back on line and operational at 17:00 PDT.

Upgrade on the upgrade

We’ve just finished verifying and fixing the problem that aborted our recent upgrade attempt. We’ll be rescheduling the upgrade soon–stay tuned.

What we have done about the problem to date:

1. We identified the failures as being collation botches (more details below)
2. We identified the correct collations, through a combination of documentation, expert advice, and experiment
3. We tried to identify a technique to pick up the migration where we’d left off, but couldn’t (there are details of the collation configuration we have to set at table creation time)
4. We modified our tools to do the collation configuration correctly
5. We test-migrated the entire database, both to ensure the process now works and to have a better understanding of how long it will take

With this testing, we’re confident we’ve nailed the problems. We’re also surprised at how long it’s going to take (the estimate is 14 hours). We’re working on scheduling that now.

For the curious:

The problem that bit us relates to the encoding of non-English languages. This upgrade includes an upgrade of our database, MySQL, from 4.0 to 5.0. There are lots of good things in the new version. One is greatly expanded capabilities in the encoding arena. The problem was that we didn’t fully understand how to drive these new features.

For the pathologically curious:

The issue is not merely “encoding,” but actually “collation”: how does the database decide whether one string (or character) comes before or after another, or if perhaps they’re actually the same character. In MySQL 4.0, there were well-defined “character sets,” which included some notion of collation, but not a very complete one. In MySQL 5.0, there are major improvements in collation handling, but at the cost of some pretty significant changes in configuration procedures and settings.

This upgrade involves dumping the old database and loading it into the new. Loading includes checking table constraints, like primary keys and uniqueness and indexing. If you get the collation wrong, this can produce the wrong result. In fact, if you get the collation wrong, it can tell you silly things like “You can’t add a user named ülf, we already have a user named ulf, and those two names are the same by the collation scheme in use.” Which is basically what happened.