Download links looping

Several users have reported that download links from the “Documents & files” areas of Tigris are misbehaving.  URLs that include “ProjectDocumentList” or “/files/documents/NN/NNNN/” are affected. The problem only seems to afflict Firefox, which reports:

Firefox has detected that the server is redirecting the request for this address in a way that will never complete.

We are investigating, but at present we are not able to reproduce the symptom.

If you encounter this problem, it might help if you could provide some information about your experience here in the comments, such as

• browser version, platform, plug-ins
• how you got there (your last few clicks)
• is it persistent, or does it go away
• is there anything unusual at your end, such as a web proxy
• date, time, and file path you attempted (so we can grub logs)

Tigris.Org mail-lossage patch today

Tigris will be down for about one hour, beginning at 15:30 PDT today, to apply patches to fix lingering mail loss problems.

Update: back up.

Tigris.org: memory replacement

Tigris is experiencing a hardware error in its memory banks. It will be going down for replacement at 16:15 PDT. Should be back up within a few minutes.

Update: back on line and operational at 17:00 PDT.

Tigris Maintenance Friday Afternoon

Tigris.Org will be down briefly, beginning at 1:00PM PDT, to install two security patches. For further details, check the announcements mail list:

Brief Tigris Downtime .

Upgrade on the upgrade

We’ve just finished verifying and fixing the problem that aborted our recent upgrade attempt. We’ll be rescheduling the upgrade soon–stay tuned.

What we have done about the problem to date:

1. We identified the failures as being collation botches (more details below)
2. We identified the correct collations, through a combination of documentation, expert advice, and experiment
3. We tried to identify a technique to pick up the migration where we’d left off, but couldn’t (there are details of the collation configuration we have to set at table creation time)
4. We modified our tools to do the collation configuration correctly
5. We test-migrated the entire database, both to ensure the process now works and to have a better understanding of how long it will take

With this testing, we’re confident we’ve nailed the problems. We’re also surprised at how long it’s going to take (the estimate is 14 hours). We’re working on scheduling that now.

For the curious:

The problem that bit us relates to the encoding of non-English languages. This upgrade includes an upgrade of our database, MySQL, from 4.0 to 5.0. There are lots of good things in the new version. One is greatly expanded capabilities in the encoding arena. The problem was that we didn’t fully understand how to drive these new features.

For the pathologically curious:

The issue is not merely “encoding,” but actually “collation”: how does the database decide whether one string (or character) comes before or after another, or if perhaps they’re actually the same character. In MySQL 4.0, there were well-defined “character sets,” which included some notion of collation, but not a very complete one. In MySQL 5.0, there are major improvements in collation handling, but at the cost of some pretty significant changes in configuration procedures and settings.

This upgrade involves dumping the old database and loading it into the new. Loading includes checking table constraints, like primary keys and uniqueness and indexing. If you get the collation wrong, this can produce the wrong result. In fact, if you get the collation wrong, it can tell you silly things like “You can’t add a user named ülf, we already have a user named ulf, and those two names are the same by the collation scheme in use.” Which is basically what happened.

Tigris is back in service

Tigris is back in service.

The upgrade attempt had to be abandoned, however, and we’re still running the same version we were before the attempt.

The upgrade ran into some mysterious problems during database conversion. I say “mysterious,” because the upgrade had already been verified once, off-line. We don’t, at this point, know why those tests passed yet the actual upgrade failed, but we’re digging into that very hard.

When we’ve understood that, we’ll be doing another off-line trial run, and possibly some data clean-up, before another attempt. Details will appear here.

Tigris Upgrade: New ETA 2:00pm PDT

The Tigris upgrade is taking considerably longer than was projected. New ETA is 2:00pm PDT.

Cause of the delay is the large size of the database. Tigris is the largest system yet to go through this particular upgrade, and some of the table upgrades are taking considerably longer than anticipated.

Apologies for the delays and inconvenience.