Indexing text data (varchar, nvarchar, char, etc) is a good way to make it faster to find the data you are looking for. However these indexes can end up being very hard on the disks behind the index, as well as the memory of the server. This is because of the large amount of data being put in the index.
As an example, let’s say that we have a table like this.
CREATE TABLE Employee
(EmployeeID INT,
FirstName VARCHAR(50),
LastName VARCHAR(50),
EmailAddress VARCHAR(255))
Now assume that you want to be able to search by the EmailAddress field. We will then want to index the EmailAddress field with a non-clustered index. If we work for a company like AMD, then our email addresses will be pretty short (f.lastname@amd.com). However if we work for a company like I work for then the email addresses are a bit longer (flastname@awarenesstechnologies.com). Now when we index this column we will be putting the entire email address into the index, taking up a lot of space within the index; especially compared to a numeric value such as an integer. This becomes doubly true if you are using a uni-code data type as each character requires two bytes of storage instead of the usual one.
This also becomes a problem if you are working on a system with URLs in the field to be indexes. Depending on the length of the URL, the values may be longer than is allowed in an index which could then give you sorting problems on the indexes.
There are a couple of variations on this technique which I’ve seen. The one I’ve used the most is to use the CHECKSUM function as part of a calculated column, and then index the calculated column. This way you simply get the CHECKSUM of the value you want to find, and search the calculated column. As we are now have an index made up of integers the index can fit a lot more data on each physical data page reducing the IO cost of the index seek as well as saving space on the disk.
So doing this turns our table into something like this.
CREATE TABLE Employee
(EmployeeID INT,
FirstName VARCHAR(50),
LastName VARCHAR(50),
EmailAddress VARCHAR(255),
EmailAddressCheckSum AS CHECKSUM(EmailAddress))
Now I wouldn’t recommend using this technique for each table you create. I usually only recommend a technique like this when the value to be indexes won’t fit within the bounds of the index, or the table will be very large and searched often so the memory saved is worth the extra CPU time of having to hash the values before doing the lookup.
Now there are a couple of gotchas with this technique. If you are check summing domain names, some characters don’t check sum correctly. Also check summing a Unicode version of a string will give you a different result than the non-unicode version of the same string.
You can see that with these three SELECT statements.
SELECT CHECKSUM(‘google.com’), CHECKSUM(‘g-oogle.com’)
SELECT CHECKSUM(‘google.com’), CHECKSUM(N’google.com’)
SELECT CHECKSUM(N’google.com’), CHECKSUM(N’g-oogle.com’)
As you can see the first one you get two different values as you would expect ( 1560309903 and 1560342303 respectively). With the second query you get two very different values between the Unicode and character strings (1560309903 and -1136321484 respectively). Based on the first query you would expect to get two different values for the third query, but you don’t. With the Unicode strings the – appears to not count as part of the CHECKSUM giving you the same CHECKSUM value for both strings (-1136321484).
Another version of this technique which Kevin Kline talked about recently uses the HASHBYTES function of SQL Server 2005 to get the hash of a column and use that. In his blog he’s talking about using it for partitioning a table, but that same technique can be used here as well.
CREATE TABLE Employee
(EmployeeID INT,
FirstName VARCHAR(50),
LastName VARCHAR(50),
EmailAddress VARCHAR(255),
EmailAddressCheckSum AS HASHBYTES('SHA1', EmailAddress)
This will however give you a longer string, therefor taking up more space within the index. However if working with long Unicode strings this may be a better option for you to use.
Denny
2 Responses
After thinking about this for a while, I suspect that for email addresses there is probably a more complete solution (where you can also use other types of matching than just equality through a hash column) that is also scalable:
Just (partly) normalize all the different parts of the email address (e.g. create one or more tables like email_prefix,email_domain or email_extension). Fora large set of email addresses this would give you a considerable size reduction since you would store each part of the email address only once, and you do not need to store the email address @ and dot character.
Since this means you can now index the individual emailaddress parts more easilty It would also make partial matches to email address(parts) possible.
my suggestion is not practical for small sets since it would actually increase not only the total size of the data, but it would also make some matching queries needlessly complex.
If there was a requirement for breaking the data out, that could be an excellent solution. If your system requirements are for a single column where the email address is a username for example, then breaking the column apart may not be the best option in that case.
Thanks for showing an additional method which can be used.
If anyone else has other options please post them.