Tigris Upgrade: Timeline and chance to help

The upgrade of Tigris to CollabNet Enterprise Edition 5.3 is on track and coming soon. The current schedule is:

Community testing                             10d         Tue 1/26/10      Mon 2/8/10

Production upgrade of tigris to 5.3.0          1d         Fri 2/12/10      Fri 2/12/10

So next Tuesday, we’ll be looking for your help: we’ll have a “staging” site set up, a copy of the Tigris site but with the new software. We want your help to test it out, comment on the new features, and find the new bugs.

To participate, subscribe to the “staging” list at Tigris.Org (send email to staging-subscribe@tigris.org). That’s where you’ll see the announcement, and connection details. That’s also where discussions about the upgrade should happen. (Note that this list is on the real Tigris.Org, even though it’s about the test site. That’s so we can continue conversations even if we manage to crash the test site.)

So, sign up now, and watch your in-box!

Update (2/2/10):

Well, is it January 26, yet?

Some problems in upgrading a large site like Tigris were discovered and corrected, but that means we have to start the set-up and staging of the site again. New schedule:

Community testing: 10 days, starting Feb 22 (through March 5)

Actual upgrade: Second week in March

January Maintenance: An Update

Here are additional details on plans for upgrading the Tigris site:

• Add additional memory to improve performance: 1/21. Complete.
• Set up a “staging server” (or “beta,” if you like) for Project Owners and users to test: 1/26 – 2/5
• Upgrade Tigris.Org to CEE 5.3.0 patch 2 ‘Indus’

Not that “staging server” item, there. We’ll be asking all Tigris users to beat up the staging server for about ten days, to help us ensure a smooth transition. Stay tuned here, in Twitter (@tigrisdotorg), or on the “announce” list at tigris.org.

January maintenance activities on Tigris

Several maintenance activities are planned for Tigris during January:

• On January 6, at 2:00PM PST, the site will be down for about one hour, to add additional memory, which should improve performance
• Within a week or so, a staging site will be established in preparation for a software upgrade. Project Owners and interested others will be asked to verify that their staged projects look and behave properly. Staging does not involve any downtime for Tigris itself.
• The actual upgrade of Tigris will happen by the end of the month. This will require some downtime

Follow this blog or the announcements list at Tigris for continuing details.

“Service Temporarily Unavailable” (status 503) from SVN

Subversion service on Tigris is currently running at reduced capacity, and may return an error 503 “Service Temporarily Unavailable.”  This is because we have throttled Subversion access, per project, a bit while we work on a memory-leakage bug just discovered.

If you encounter this problem, please wait a minute or so and retry your operation. If that’s unsuccessful, contact us, and we’ll look into raising the threshold for your project. You can do that via comments on this blog, through the Twitter account @tigrisdotorg, or via email to the feedback address at tigris.org.

The fix for the underlying problem is known, and already proven in several other production sites. We’re working on getting it installed onto Tigris as quickly as possible.

Coming upgrades on Tigris

We’ll be upgrading Tigris to the latest platform version, CollabNet Enterprise Edition 5.3 ‘Indus’ next month. This upgrade fixes the last, lingering glitches of the Discussion Services upgrades. It also includes some notable performance boosts, both in web use and in Subversion access, and it relieves some barriers to growth and scalability. All in all, it’s something we’ve been wanting for quite some time.

As both preparation and a sort of down-payment, there will be a brief downtime this evening, at 18:00 PST. This down time should be about 20 minutes.

Download links looping

Several users have reported that download links from the “Documents & files” areas of Tigris are misbehaving.  URLs that include “ProjectDocumentList” or “/files/documents/NN/NNNN/” are affected. The problem only seems to afflict Firefox, which reports:

Firefox has detected that the server is redirecting the request for this address in a way that will never complete.

We are investigating, but at present we are not able to reproduce the symptom.

If you encounter this problem, it might help if you could provide some information about your experience here in the comments, such as

• browser version, platform, plug-ins
• how you got there (your last few clicks)
• is it persistent, or does it go away
• is there anything unusual at your end, such as a web proxy
• date, time, and file path you attempted (so we can grub logs)

Tigris.Org mail-lossage patch today

Tigris will be down for about one hour, beginning at 15:30 PDT today, to apply patches to fix lingering mail loss problems.

Update: back up.

Tigris.org: memory replacement

Tigris is experiencing a hardware error in its memory banks. It will be going down for replacement at 16:15 PDT. Should be back up within a few minutes.

Update: back on line and operational at 17:00 PDT.

Tigris Maintenance Friday Afternoon

Tigris.Org will be down briefly, beginning at 1:00PM PDT, to install two security patches. For further details, check the announcements mail list:

Brief Tigris Downtime .

Upgrade on the upgrade

We’ve just finished verifying and fixing the problem that aborted our recent upgrade attempt. We’ll be rescheduling the upgrade soon–stay tuned.

What we have done about the problem to date:

1. We identified the failures as being collation botches (more details below)
2. We identified the correct collations, through a combination of documentation, expert advice, and experiment
3. We tried to identify a technique to pick up the migration where we’d left off, but couldn’t (there are details of the collation configuration we have to set at table creation time)
4. We modified our tools to do the collation configuration correctly
5. We test-migrated the entire database, both to ensure the process now works and to have a better understanding of how long it will take

With this testing, we’re confident we’ve nailed the problems. We’re also surprised at how long it’s going to take (the estimate is 14 hours). We’re working on scheduling that now.

For the curious:

The problem that bit us relates to the encoding of non-English languages. This upgrade includes an upgrade of our database, MySQL, from 4.0 to 5.0. There are lots of good things in the new version. One is greatly expanded capabilities in the encoding arena. The problem was that we didn’t fully understand how to drive these new features.

For the pathologically curious:

The issue is not merely “encoding,” but actually “collation”: how does the database decide whether one string (or character) comes before or after another, or if perhaps they’re actually the same character. In MySQL 4.0, there were well-defined “character sets,” which included some notion of collation, but not a very complete one. In MySQL 5.0, there are major improvements in collation handling, but at the cost of some pretty significant changes in configuration procedures and settings.

This upgrade involves dumping the old database and loading it into the new. Loading includes checking table constraints, like primary keys and uniqueness and indexing. If you get the collation wrong, this can produce the wrong result. In fact, if you get the collation wrong, it can tell you silly things like “You can’t add a user named ülf, we already have a user named ulf, and those two names are the same by the collation scheme in use.” Which is basically what happened.