Tigris will be down for about one hour, beginning at 15:30 PDT today, to apply patches to fix lingering mail loss problems.
Update: back up.
Tigris is experiencing a hardware error in its memory banks. It will be going down for replacement at 16:15 PDT. Should be back up within a few minutes.
Update: back on line and operational at 17:00 PDT.
Tigris.Org will be down briefly, beginning at 1:00PM PDT, to install two security patches. For further details, check the announcements mail list:
We’ve just finished verifying and fixing the problem that aborted our recent upgrade attempt. We’ll be rescheduling the upgrade soon–stay tuned.
What we have done about the problem to date:
- We identified the failures as being collation botches (more details below)
- We identified the correct collations, through a combination of documentation, expert advice, and experiment
- We tried to identify a technique to pick up the migration where we’d left off, but couldn’t (there are details of the collation configuration we have to set at table creation time)
- We modified our tools to do the collation configuration correctly
- We test-migrated the entire database, both to ensure the process now works and to have a better understanding of how long it will take
With this testing, we’re confident we’ve nailed the problems. We’re also surprised at how long it’s going to take (the estimate is 14 hours). We’re working on scheduling that now.
For the curious:
The problem that bit us relates to the encoding of non-English languages. This upgrade includes an upgrade of our database, MySQL, from 4.0 to 5.0. There are lots of good things in the new version. One is greatly expanded capabilities in the encoding arena. The problem was that we didn’t fully understand how to drive these new features.
For the pathologically curious:
The issue is not merely “encoding,” but actually “collation”: how does the database decide whether one string (or character) comes before or after another, or if perhaps they’re actually the same character. In MySQL 4.0, there were well-defined “character sets,” which included some notion of collation, but not a very complete one. In MySQL 5.0, there are major improvements in collation handling, but at the cost of some pretty significant changes in configuration procedures and settings.
This upgrade involves dumping the old database and loading it into the new. Loading includes checking table constraints, like primary keys and uniqueness and indexing. If you get the collation wrong, this can produce the wrong result. In fact, if you get the collation wrong, it can tell you silly things like “You can’t add a user named ülf, we already have a user named ulf, and those two names are the same by the collation scheme in use.” Which is basically what happened.
Tigris is back in service.
The upgrade attempt had to be abandoned, however, and we’re still running the same version we were before the attempt.
The upgrade ran into some mysterious problems during database conversion. I say “mysterious,” because the upgrade had already been verified once, off-line. We don’t, at this point, know why those tests passed yet the actual upgrade failed, but we’re digging into that very hard.
When we’ve understood that, we’ll be doing another off-line trial run, and possibly some data clean-up, before another attempt. Details will appear here.
The Tigris upgrade is taking considerably longer than was projected. New ETA is 2:00pm PDT.
Cause of the delay is the large size of the database. Tigris is the largest system yet to go through this particular upgrade, and some of the table upgrades are taking considerably longer than anticipated.
Apologies for the delays and inconvenience.
Tigris.Org will be upgraded to release 5.3.0.167.1 (that is, 5.3.0 patch 1), beginning at 9:30pm PDT.
The primary purpose of this upgrade is to repair additional mail delivery problems, such as non-delivery.
Release notes for the new base release and patch level are available at OpenCollabnet:
Key fixes and upgrades for Tigris users:
- An assortment of Subversion performance improvements
- Proper delivery of single-part Discussion messages
- Discussions can be marked to automatically trust new subscribers
- Auto-deletion of unmoderated and soft-deleted Discussion messages (per-forum setting)
- Configurable size limits on RSS feeds of Discussions
- Time-range deletion controls for Discussion moderators
- Continued improvements in spam blocking and spamhaus deflecting
Other interesting stuff:
- New versions of Subversion (1.6.0) and ViewVC (1.0.7)
- Latest Subversion and Apache security fixes
The Tigris.Org site outage last week was traced, over the long weekend, to one missing file related to the recent patch application. The site experienced an additional brief outage to fix that, and the site seems to be healthy now.
With the Patch 3 (May 20) update to Tigris (and in some cases, some other, recent upgrades), we introduced several new features to Tigris mail-list management, including posting control and CAPTCHA.
With Posting Control, the mail list administrator can control who is allowed to post to the list. With the Patch 3 changes, this includes separate controls for posts coming via email and posts coming via web. So, for example, the commits@ and issues@ lists in your projects receive internally generated postings (neither web nor email), so you might choose to block posting to them from either source. Then, set the reply-to for these lists to your dev@ list, and you’ve automatically directed all discussion into the light of day.
You know what CAPTCHA is, even if you don’t know you know: it’s those hard-to-read little “type in this word” things that are springing up all over the web. A CAPTCHA guarding the web posting page helps to protect the list against robots and spam: hopefully, they won’t be able to read the image and enter the text.
As embodied in Tigris, CAPTCHA allows the list owner to choose among a range of options:
- By default, CAPTCHA is required for web posts made by people who aren’t logged in (so-called “anonymous” web posts). This is the lowest level of security available, but it covers all the cases we’ve recently experienced.
- If a higher degree of security is wanted, the list admin can require CAPTCHA for all web posts.
With posting control and CAPTCHA, you should be able to become virtually spam-free.
Tigris will be down for an upgrade-patch the evening of May 20 (Pacific time). Precise schedule details will be available shortly.
This patch continues our series of repairs and improvements in several areas, perhaps most notably in the Discussion Services:
Discussions Services:
- Large messages no longer turned into pseudo-attachments (final fix; a partial form is already in place on Tigris)
- Increase default maximum message size limit (to ~10MB)
- Separate “non-logged-in web user” and “non-registered email user” (extensive Discussion configuration changes; see an upcoming post here)
- Provide “allow future posts from this sender” during email-mediated moderation (final fix; temporary fix already in place on Tigris)
- Allow centrally enforcd use of CAPTCHA for anonymous web posts (post about planned Tigris policies upcoming here)
- Enforce security and spam protections on the “other recipients” field of the web-post form (final fix; temporary fix already in place on Tigris)
- Better handling (RFC 2047) of accents and non-English characters in reply-to headers
Subversion:
- Svnsync now does not choke on certain data
- Non-expiring passwords no longer expire anyway
Wiki:
- Security enhancement relating to a guest-access Denial of Service
- Security enhancement to block server-side test
Site:
- Performance enhancements in page loading, particularly for clients with high latency or low bandwidth
- Project News RSS URL backwards-compatibility redirect
Project Pages:
- Now able to invoke a Project Tracker “named saved query”
- Project Pages: Project Tracker “Return to list view” now works
- Now able to use a Project Tracker “saved query” containing an apostrophe
Project Tracker:
- Mixed-field queries with null fields now work
- API: protect against embedded invalid XML data
- API: annotate private (non-accessible to this user) related artifacts
- API: proper pagination of long result lists
