noted on Wed, 28 Jul 2004
UTF-8 and Web Site Development
This post will contain some tips on how to set up your web development process to use UTF-8 end to end. What happened was, I saw a pair of posts by Sam Ruby (Unicode and weblogs, Aggregator i18n tests). I can be a bit of a careful (read: slow) thinker at times, so I had to let this percolate through my brain for awhile. But these posts were published during two projects that I was working on.
The first project I was working on was a site containing English and French, with a ColdFusion based content management system, and was trying to deal with accented characters. With my co-workers, we figured out how to reliably cough out entities in the right places.
Another project was an XML/XSLT based one that was getting some fancy characters like bullets and emdashes from some form submissions, and dealing badly with them. I think we did the entity replacement trick here, too.
What we found remarkable was that we’ve been building web sites for years, and it seems like all of a sudden this has become a problem for us.
Well, finally I grokked what Sam was talking about, and I went ahead and modified my web development toolchain to work in utf-8. Let me tell you: it was far easier than I had originally thought it was going to be. This blog hasn’t been updated yet, but my other website is running as UTF-8.
Even if you're English, you should care
See, unicode is a lot more than just accented characters, or Asian characters. It’s also got all the finer typography controls too — you want curly quotes? Emdashes? It’s all there. With a little wiki magic, you could even set up your content management system to automatically convert standard ASCII quotes to curly quotes, and dashes to emdashes, so you’d never have to learn how to input them!
Seriously, you don’t need to use HTML entities anymore, barring angle brackets and the ampersand. That’s a big win — it makes your source code readable if the HTML is stripped out. Everything still looks good in plain text.
How to do it if you've never done it before.
- Look at your text editor. Find out how to get it to save stuff as UTF-8 and set it as the default. I know vim does it (:set encoding=utf-8). I know BBEdit does it. I’ve heard that emacs and UltraEdit also supports this.
- Ensure that all web pages you produce have the following meta-tag:
<meta http-equiv="content-type" content="text/html; charset=utf-8">
- Check your forms. Look at the form tag and add, if you haven’t already, the following attributes. Please note that both of them are needed; the hyphenated version is from the HTML 4 spec, but Netscape 6 and IE 5 use the other version.
acceptcharset="utf-8" accept-charset="utf-8"
- Check your CGI’s. If you have to print a Content-type: heading, make sure it’s:
Content-type: text/html; charset=utf-8
- Have access to your server configuration files? Add this line to it, if it’s not already there (assuming you’re running Apache):
AddCharset utf-8 .html
Even with all this, I still don’t feel like I’m an expert. For example, the accept-charset tags mentioned above aren’t supported in version 4 browsers. What’s the encoding of text submitted by form on those browsers? If it’s not UTF-8, then you’d need to patch your CGI’s to check for these browsers, and convert the contents to UTF-8. As I learn more, I’ll try to keep you updated.
Other links of interest
- The HTTP charset parameter — This link will take you to the W3C’s Internationalization page, which covers similar material to what I covered here. You may think I cribbed from them, but in truth, I discovered that link after most of this post was written.
- HTML and Foreign Languages Dave Shea wrote about his struggles with high-bit characters. You should also check out the comments, which I’ve linked here, because some people have contributed valuable information. Thijs van der Vossen, in particular, contributed the AddCharSet bit of advice that I worked into this post.
- Comments on: HTML and Foreign Languages
- On the Goodness of Unicode Tim Bray wrote an excellent overview of what Unicode is. I feel, however, that the target was more software developers than web application developers, but once you start to get comfortable with the problem, you should read this article.
- Characters vs. Bytes Another piece by Tim Bray, this is far more technical than most web developers would care for, but the section describing how UTF-8 works may well be worth your while if you’re wondering how they encode 100,000 characters in presumably 8 bits (answer: they don’t)
- Do not fear Unicode is a short rah-rah piece that echoes similar sentiments to mine about using Unicode.