RadControls Performance Tips

Recently I was doing some research and looking for tips on how to improve Telerik RadControls performance. On the support forums I found this thread. The thread is a little old, but I answered to it based on my experiences. Having little else to say right now, I thought it might be best to basically repost this answer here.

I've had a problem with customers complaining about the performance of a page using RadScheduler. To be fair, there are other controls besides a RadScheduler in the same page (RadMenu, RadToolbar and RadTreeView). Anyway, the page is heavy on data, and feels sluggish.

I did some benchmarking with YSlow and Chrome Speed Tracer. YSlow told me that my page was about 1MB in total size.

The problematic part is the generated HTML that has to be loaded every time. This is the text you see in the browser's "view source". Of the one megabyte, HTML took about 400KB. It contained:

• ViewState ~ 150KB
• Script (mostly JSON data) ~ 130KB
• HTML ~ 120KB

Even with slower javascript engines it is likely that loading 400K will be a major performance issue, and it certainly pays off to minimize HTML.

Compression
The first thing to do is to use compression in IIS and in your CMS.

DotNetNuke performance options

Just by enabling DotNetNuke compression, I reduced the page size from 400K to 100K, a huge effect. But if this is not enough, you'll have to fine-tune more.

RadCompression

This is perhaps the easiest thing to do. RadCompression HttpModule will automatically compress your Ajax and JSON web service communication and even your ViewState. Setting up RadCompression is just a matter of adding a HttpModule registration to web.config.

Unfortunately, my experience with RadCompression were short-lived this time. RadCompression (2009.02.0826.35) with IIS6 and DotNetNuke compression broke postbacks in Firefox 3.6! Literally, postbacks returned a garbled mess of compressed text to the browser. Chrome and IE were fine, Firefox too with IIS7. The reason for this was that there was two compressors enabled simultaneously: the DotNetNuke GZip compressor and the Telerik RadCompression. I still have to figure out how to enable RadCompression and disable DNN compression on a per page basis, to speed up Ajax requests.

Measuring the performance impact of RadCompression was not that simple and I didn't get a clear picture. But Chrome Speed Tracer indicated a speedup in refreshing UpdatePanels, that was lost after disabling the HttpModule.

RadScriptManager and RadStyleSheetManager

The purporse of these controls is to reduce HTTP traffic by combining requests, for javascript and css respectively.

RadStyleSheetManager is extremely easy to set up. You just add a RadStyleSheetManager control to your page. The net result was about 10 fewer CSS requests in an empty cache. For a primed cache, there was no effect.

RadScriptManager is trickier. You can have only one ScriptManager per page. Because DotNetNuke has one already, you would have to replace it in DNN core code and rebuild. I can't recommend this to anyone. Hopefully in the future DNN Professional editions this is going to be easier.

Minimize inline javascript

Declare only necessary javacript variables in the ASPX. These are usually the ones where you need ClientIDs, like:

var scheduler = $find('<%= RadScheduler1.ClientID %>');

All other javascript should be loaded from external files and cached.

Try to disable ViewState
Easier said than done. For example, I had a hierachical navigation/drilldown menu that displayed the number of events each year, month and day. I can't disable ViewState for this control, because it is tied to the RadScheduler's view (month, week, day, timeline). Surprisingly, I can disable ViewState in the RadScheduler itself without any immediate loss of functionality. This reduced (pre-compression) page size about 10%, but is a risky bet.

Try to limit data size
By showing less data on one screen you can in many cases improve both performance and user experience. But sometimes you need data for things like RadScheduler appointment tooltips, that are stored in appointment attributes and resources. These take easily a lot of space.

Forget old browsers
Even with aggressive tuning, the overall user experience is very much dependent on the browser's capabilities. With IE6 and other older browsers, any page with heavy RadControls barely works, and I wouldn't spend much effort to tweak them.

-mika-

See also:

Top 15 Optimization Tips and Tricks

Top Performance - General Resources

SuperGenPass

I've been using SuperGenPass (and its predecessor GenPass) for over three years now. SuperGenPass is a javascript bookmarklet that generates a unique password to a web site out of a master password and a domain name, using a MD5 hash. The user interface lets you input the seed password, and populates the generated password to any password field:

In practice, this means that you can increase the security of your passwords while using the same master password on every site, without having to store secrets anywhere. Using SuperGenPass gets you through most of the items in the Bruce Schneier's password rules. Changing passwords gets a little trickier, though, because you have to use two bookmarklets in the transition period: one for the old master password and one for the new one.

Sure, there are other password generators and password managers, but this simple javascript bookmarklet has provided the best user experience so far. Browser compatibility is the only serious problem with the standard SuperGenPass bookmarklet. It didn't work in the Nokia S60 or iPhone browsers, and I remember having problems with IE7. For these situations, there is a static html mobile version that lets you manually generate SuperGenPass passwords on any device. It is also needed when using someone else's computer. A minor annoyance with SuperGenPass is that some websites have idiotic restrictions on password length, and the default 10 character password may be too long or too short.

SuperGenPass is practical for websites only, when a user-generated, non-shared password is needed. I use KeePass for other password management, but wouldn't recommend it for webapps.

-mika-

DevExpress CodeRush and Refactor! Pro

In July I got a free DevExpress DXperience suite licence. I was asked to write about my experiences with the software, if I found the time to do so. I think it is a fair deal. There is no way I can gain from this offer except by playing around with the software and seeing if it produces a net benefit. I can say that with the hourly rates common in enterprise application development, the time used to learn anything has a considerable opportunity cost. Putting time away to sharpen your saw should be a necessity, but in consulting environment it easily becomes a luxury. Time to learn is time away from short term customer work that brings the money in. Just try telling your boss that you are missing deadlines because of playing around with a new development tool!

Now, half a year has passed and I think I can say something of the tools I use daily, namely CodeRush and Refactor! Pro.

Installation and First Impressions

CodeRush and Refactor! Pro are the first things you see after installing the DevXpress toolkit. They are Visual Studio add-ins that are built upon a common framework called DXCore. There are also a few community plugins using this same framework. A normal user can mostly ignore this plumbing, but from time to time you'll see UI messages that explain how DXCore, not CodeRush, has done something. So, in principle there are three separate products involved: a free framework (DXCore), a commercial refactoring add-in (Refactor! Pro) and a commercial coding assistant add-in (CodeRush) that is always shipped with the Refactor! Pro. If you are not confused enough, there is also a separate free edition (CodeRush Xpress) and a specialiced Refactor editions for C++ and ASP.NET. To keep things simple, I'll will stick with the CodeRush name for the rest of this blog post.

The obvious tool to compare CodeRush against is the JetBrains ReSharper. Unfortunately, I don't have enough experience with ReSharper to say anything interesting (check this thread instead). I tried ReSharper about a year ago and it slowed down Visual Studio too much for my taste. Visual Basic IDE is demanding to begin with, with all the background compilation going on, and adding new extensions will definitely slow it down more. I don't have benchmarks available but I would presume that the amount of background processing is dependent on the lines of code in your solutions. Large solutions should be slower than small ones. Good news is that performance improvements have surely been the most highlighted feature in both CodeRush 9.2 (1) (2) and ReSharper 4.5.

Ultimately, you must compare the productivity gains of IDE extensions to the degradation of default IDE performance. This includes crashes, temporary hangs and general slowdown. I collected some haphazard data after installing CodeRush 9.1. on a machine with the following performance profile:

• VS2008 hangs for a few seconds but recovers. This occured repeatedly.
• VS2008 hangs permanently or crashes, restart required: happened 3 times during test
• VS2008 lost intellisense: did once
• VS2008 slows down: noticable, especially correction suggestions update slowly

(I must stress that I did have a few other Visual Studio add-ins during testing but the problems started only after installing CodeRush.)

In the end, for the same reasons that I couldn't use ReSharper, I had to give up using CodeRush 9.1. I need performance and reliability in Visual Studio to do my job adequately. After uninstalling Coderush, VS2008 performance improved but it still crashed every now and then, even with every other add-in uninstalled. It would be unjust to blame 3rd party tools for this, it had something to do with the VS2008 and the environment itself.

CodeRush 9.2

After CodeRush 9.2 became available I decided to try it again. This time with a fresh install of Windows 7 and VS2008. In a fresh system, CodeRush started to perform fast and reliably enough to be useful. I can't remember a single crash. This level of reliability is an absolute requirement for a productivity tool. Until this point, CodeRush had a negative impact on my productivity. I can only recommend that you have a solid Visual Studio environment before installing any add-ins.

Learning curve

Tools like CodeRush and ReSharper are only distractions if you don't learn to use them. And there is a learning curve involved. If found that this learning process started with general confusion, followed by curiosity and experimentation with the features. Finally, there was a kind of settle-down to a subset of features, with some features even disabled.

If I remember right, I first started using the refactoring suggestions right of the editor scroll bar that are similar to ones in ReSharper. These suggestions are mechanical, but they are useful in finding undisposed resources and redundant code. And you can customize them. For example, I didn't see it worthwhile to use String.Format in every concatenation and disabled the suggestion.

But the most visible features like refactoring suggestions are just the tip of an iceberg. I suggest taking a systematic approach to learning CodeRush. This means trying most of the features at least once and seeing what sticks in practice. This way you'll find the features you don't know to look for. I used the cheat sheet as a learning tool, trying one section per day while working on projects. The point is that learning CodeRush is a voluntary discovery process. There are tools to help you, like the Feature Statistics, but in the end you are free to use what you want.

Stats show that my top 5 features for the last week or so were:

• Intelliassist 274
• IntelliassistCycleForward 35
• Rename 25
• ShowReferences 24
• If .. Then 19
• CamelCaseLeft 11

I think I could use MarkerCollect (3) a lot more, but I don't really understand Selection Inversion (6), that is, inverting selected boolean expressions from true to false and vice versa. I think this feature is mostly invoked because of typos. It is easy to hit a wrong shortcut when using CodeRush, and in many cases it might be best to disable the invoked feature or at least remap its shortcut. Selection inversion is particularly dangerous, since it has a semantic effect invisible to the compiler. I have been burned once by an unintended inversion that ended up in the version control system.

Besides learning predefined shortcuts, you'll have to customize CodeRush to use it effectively. CodeRush allows you to customize practically everything. For example, default shortcut keybindings in international keyboards are not always the most natural ones, and you need to rebind them yourself. This effort pays off, since only those keybindings that stick are valuable.

One general impression is that CodeRush is more pleasant to use with C# than with VB. Clearly, C# programmers are a larger audience and also need some of the features more. But CodeRush has improved its VB support too, and it does add useful features like refactorings that are scarce in VB by default.

Keyboard Shortcuts

As mentioned before, there is a great cheat sheet available from the CodeRush inventor himself, Mark Miller. It is divided between Clipboard, Selection, Navigation, Code generation, Miscellaneous and Template shortcuts. Templates, or code snippets, are the most versatile part of the system. They also require the most memorizing, and it is better look at the other shortcuts first.

I have a gut feeling that most programmers are not that much into keyboard shortcuts. Sure, there are global shortcuts that everyone uses, and leveraging them is a good practice. For example, mapping F2 to rename an identifier in Visual Studio is a natural extension to a common Windows rename shortcut. But there is a flipside to this argument: shortcuts that interfere with the more common ones are harmful. For example, I dislike using Ctrl-W to increase a selection is VS because it interferes with using Ctrl-W to close windows and tabs elsewhere.

Even natural shortcuts like F2 can be dangerous. The danger is that you can accidentally rename a method when you really wanted to use another method. This is probably why F2 is disabled by default. But I encourage to enable it. F2 is a special time-saver in ASPX files, where server control identifiers get values like TextBox1 by default. In plain Visual Studio you have to use the property grid to globally change such a server control ID, because there is no rename feature similar to renaming variables in code. But with CodeRush, rename is performed uniformly in both aspx and code-behind.

Other favorites:

• Drop and collect marker. These are short-term bookmarks in code. Drop a marker with Alt-Home, go elsewhere in the code, come back to the marker with Esc.
• Clipboard history. More slots in VS clipboard. Unfortunately, its shortcut Ctrl+Shift+Insert is hard to use with Logitech keyboards that have this braindead large Delete key and hide the Insert key behind Fn. I suggest Ctrl-Shift-Delete.
• Camel Case navigation with Alt-Left and Alt-Right is cool, it works like the default word-wise navigation Ctrl-Left / Ctrl-Right. Selection with Shift.
• Show all references using Shift-F12 is better, but much slower than the default behavior.

Templates

Templates, or code snippets attached to keyboard shortcuts, were the last CodeRush feature I eventually discovered. The idea of code snippets is familiar, but it takes a time to get used to them in practice. Spacebar is the magic key in expanding templates, you type the shortcut and hit spacebar, and CodeRush fills in the template. This works great for 'If..Then' template because hitting spacebar after typing 'If' is automatic. For other templates, you have to do some work to remember the shortcut, like 'fe' for 'for each'. I think that ordinary IntelliSense assistance is mostly good enough, and you have to put in some conscious effort to get real benefits from using templates.

Final remarks

A lot of functionality is not mentioned here, mostly because I haven't learned enough yet. I don't use clipboard, wrapping selections, templates or refactorings that much. But the features that I use are valuable and most of the annoyances can be turned off.

An important thing to know is that the current version of CodeRush will be incompatible with Visual Studio 2010. Good news is that the CodeRush team is working on a DXCore rewrite that should work both in VS2008 and in VS2010.

A Painful Machine Upgrade

Finally my 4 ½ years old home desktop/server PC died. I've had a PC since 1990, and if I count it right, this last one was my fifth generation desktop machine. Writing about hardware upgrades is boring, but I think these 20 years have given me some perspective. The predominant observation is that PC hardware gets obsolete faster than it breaks down. You recognize that a machine is getting too old when new things perform slowly or are impossible to do, disk space feels scarce, and you need to do tricks to get peripherals working. The reasonable thing to do then is to replace all obsolete components and install a fresh OS at once. It makes little sense to upgrade a processor while keeping the old motherboard and video card, for instance. Memory and hard disk space for data (but not the boot drive) are exceptions, they are incrementally upgradeable. It is always wise to leave memory expansion slots free in a new machine. You can wait for Moore's law to bring memory prices down, but you should never buy expensive memory!

With these guidelines, I have achieved an average age of 4 years for a machine. Each major upgrade has added some new functionality, unavailable to the previous generation (1). This time the new feature would be Full-HD support, made possible by having a capable processor and a HDMI output. Why upgrade so rarely? In the early days I would have preferred to upgrade the machine more often, but didn't have the money to do so. But in the last two generations the reason for holding back has been that the pain of upgrading a home PC overweighs any predicted benefits. And the pain has gotten worse for each new generation, which I'll explain later.

So, when in 2005 I last time bought a machine, I focused on quality. And this machine served me well. It had a brand new AMD Athlon 64 'Venice' single-core processor and the 74GB WD Raptor 10k rpm drive. Later I upgraded the memory to 3GB and bought two 250GB near-line disks for RAID1 data storage. And that was it. To be fair, these components are still fine. It was the cheap PCI Express display adapter that went broke, but I didn't find out that until later.

When deciding what to buy, the first thing to do is to clarify the requirements:

1. The machine is always on, at home

• reliability
• repairability
• low power consumption
• silent operation

2. ...but it is quite rarely attended.

• optimized for parallelism, i.e. for doing background tasks
• occasional surfing, media file management etc
• shouldn't cost too much

3. It is needed for document management

• lots of RAID1 storage , in either internal/external SATA, USB or NAS
• a version control server
• backups every night to an USB drive
• actually, the old machine was perfectly good for this

4. ...and for scanning documents.

• ...as long as paper is used
• scanning with an ADF requires processing power. The old Athlon was too slow.

5. It does background processes,

• like p2p, an ideal background process for a home pc with lots of hard disk space
• I don't want p2p in my work laptop

6. it serves as an Appliance hub,

• USB ports: printer/scanner, iPhone, external hard drives, mouse/keyboard, UPS
• Old motherboard didn't have enough USB ports, had to use extension slots

7. a Media PC,

• Spotify
• video conversions for iPhone, PS3
• iTunes (requires a desktop OS like XP, Vista or 7)
• HD video output (HDMI output + Full-HD capabilities)
• PS3 media server transcoding
• The old Athlon couldn't handle Full-HD or transcoding, and video conversions took a lot of time

8. a Virtual Machine Host

• Memory: 4GB min, with ability to upgrade to at least 12GB
• Windows XP virtual mode + other VM's for experimentation keeps the host OS clean
• The old machine could handle one 512MB VMWare Linux server well enough, but not desktop OS's

9. ... and a Development & Test Server.

• Not a major requirement as I do most development in a work laptop
• Old machine could handle ASP.NET development and SQL Server only in the host, not really in a VM

10. But there are important non-requirements too!

• Not for gaming: PS3 is for that
• Not for media playback: TV, iPhone, and laptop are preferred
• Not for desktop work: I use a laptop for my work (but I could use the desktop monitor and input devices with the laptop at home)

With these in mind I started planning and experimenting.

MacBook / Media Laptop

I have a 2006, Core Duo 1,8GHz, 2GB model, and tried to use it to replace the desktop. It didn't work out.

Reliability:

I don't think the MacBook would have lasted long if always-on.

Repairability:

Not repairable, warranty expired.

Silent Operation:

Macbook was really noisy in even basic use.

Performance:

Slower than the Athlon, doesn't handle Full-HD, no HDMI.

Appliance hub: Only 2 USB ports.

Scanning: functionality was worse than in Windows.

Many of these complaints would hit the second option, a new Media Laptop, too. Peformance would be better, operation would be more silent, and Windows would be helpful with some devices, but in the end it would still be a fragile laptop, not a built-to-last server.

Media PC

A little more integrated than a traditional PC, these small boxes can be put in the living room. With a fast NAS this is all you are going to need, right?

These things cost more than a simple upgrade and much more than a laptop. Laptops are by far the cheapest alternative, even if you don't count the components you don't actually wouldn't need, like the laptop display. Shuttle SG33G5 would have been a relatively cheap and nice barebone, starting at 270€.

But there are technical limitations, like max 4GB memory, no Core i5/i7 support, or option to upgrade the motherboard. The barebone has also limited number of drive bays, only two 3,5" bays and one 5.25" bay. You can't have a fast OS disk, two RAID1 disk and an optical drive. NAS would be a practical requirement for RAID1 storage.

Using a Media PC with a TV screen as an appliance hub is not too intuitive either. Of course, I could have set up ordinary desktop peripherals too. And last, I already have a PS3, so I don't really need another box with the TV.

So I ended up with the simple, conservative option...

Minimal Component Upgrade

• Core i5-750 (190€)
• Intel P55 motherboard, DP55WB (80€)
• 4GB DDR3 RAM (100€)
• Radeon 4350 passive PCI-Express with HDMI (30€)

Total: 400€

For storage, I kept the old drives and got one 2x500GB USB drive, giving the following:

• One relatively fast, if old, system drive (WD Raptor 74GB)
• Internal SATA RAID1 for documents
• external USB 2x500GB for backups and media files

Performace is decent and fits my purposes well:

Migration pains

Ok. I mentioned about the pain of upgrading at the beginning, and the rest of the post is a rant devoted for this. It took about a week to get most of the wanted functionality running. Some of the requirements are still not there, and probably never will, without further hardware upgrades. Here's the story.

Hardware problems

The new motherboard does not have IDE connectors anymore, only SATA ones, and I didn't have a SATA DVD drive. This makes installing Windows little tricky, as you need to use a bootable USB drive. There are good instructions for this, so this is not a big deal. It is actually a good thing to finally leave out the optical media. One DVD drive provided with a laptop is sufficient for occasional use.

Another install issue was that I needed a wired USB keyboard and mouse before I could get the chipset drivers installed and the wireless peripherals working. I can only recommend to have one wired USB keyboard and mouse in reserve.

Windows 7 x64 is cool, but the problems with Vista drivers are still there. I coudn't get WIA or TWAIN drivers working for the less-than-year old Lexmark X9575 scanner, which pretty much makes it impossible to use decent scanner software. I don't know yet if I can ever use anything else but the crappy default apps provided with the scanner. Next time I'll buy a professional document scanner, even if it costs more.

A tip: If you setting up a RAID1 mirroring, use the operating system's software RAID instead of the motherboard's hardware RAID. Mirroring provides fault tolerance but it is your job to do something when disk errors happen. An OS-integrated RAID gives you better tools to detect such issues.

A crappy iPhone photo of the running system

Data Transfer

Data is big these days. Have you tried backing up / restoring a modern 250GB+ hard drive? It takes unbelievable amounts of time, especially if you are doing a file system copy. The problem with a file system copy is the file count, not the size itself. Image copying is much faster. But you need to do a file system copy if you consolidate multiple small disks to a single-partition large disk.

First, I thought that it wouldn't really matter if I used Windows Explorer instead of a bulk copy/mirroring tool like the Windows Robocopy or a backup application like Cobian Backup. The number of files would be the same. Bad choice. For a 200GB, 200k file copy/paste, Windows estimated first that it would take over 17 hours, but I gave up after a couple of hours and multiple confirmation dialogs. Subversion svn folders didn't copy at all. Cobian Backup did the same job during the night and next day in 12 hours. SVN folders worked fine but I got errors for two SQL 2008 mdf/ldf files. I also used Cobian with a 100GB / 100k file drive, and it took about 2 hours and 15 minutes. Altogether it took me one night to install the system and three days to copy data.

I should have expected something like this. I had first-hand experience a couple of years ago when I had to transfer a few hundred gigs of company data to a parent company after a merger. RoboCopy did it pretty well, even over a network, but some files didn't transfer and they had to be manually picked from the old server. I couldn't even figure out why these files didn't transfer, since Robocopy's log file didn't show anything.

I suppose that the fastest file system copy with a modern processor would be to zip everything, transfer only one gigantic file, and then unzip it at the destination. I tried this with WinRAR (with administrative priviledges), but the compression failed to add too many files due to NTFS file permissions and I had to give up.

PC Junk

Every time you change components, something is left behind. You take this stuff and put it in a box, where there's usually old PC junk nicely sedimented. Sometimes, rarely, you need something out of this pile of abandoned PC hardware. It feels wrong to throw it all away, but nobody wants it either. This time I threw away two CD-RW drives, two modems, an analogue TV tuner and old pre-USB devices like mice and keyboards.

There was good old stuff too, things that are mostly needed when you set up a new box:

• PCI video cards (AGP is no good anymore, i recently needed a PCI card from 1997)
• Wired USB keyboard
• Wired USB mouse
• DVD-ROM drives
• Wires and connectors

Conclusion

Switching a machine is great pain. You don't want to do that unless you have to. I can only hope that this new rig will live long, but the next time I'll probably choose a more integrated solution. Hopefully the home server market matures enough so it would be possible to buy an off-the shelf reliable, quiet and fast home server without feeling to get ripped off.

But for this newly built machine, a natural mid-life upgrade would be:

• an SSD Drive for operating system and applications
• 8 gigs more memory
• A single, relatively fast and cheap internal drive for virtual machines etc, backed up once a day
• A high-quality NAS for data files
• Also, I need to get the scanner work as well as before...

-mika-

A short list of new features since my first PC

1990: 286 12 MHz, 1MB RAM with 20GB hard drive

1992: 386 33 MHz , 4MB RAM, disk 40MB to 128MB

• modem
• sound card
• Windows 3

1995: Pentium 100 MHz 8MB, later 200MHz 32MB, disk up to 4GB

• Internet: fast enough (modem, processor, memory) for Netscape and IE3
• fast enough for Doom and decent for Quake too (without 3D acceleration)
• Windows 95/98
• mp3 media

2000: Dual Celeron 400MHz, 128-512MB, 20-60GB

• Windows 2000/XP
• accelerated 3D video
• Ethernet adapter + broadband internet
• USB 1.1
• DivX media
• machine always on
• dual-boot (Windows/Linux)

2005: Athlon 64 1,8GHz, 1-3GB, 130GB-1TB

• USB 2.0 peripherals
• RAID1 for data
• more server software (svn, trac, http, p2p)
• virtual machines instead of dual-boot
• hub for cameras, scanner, mp3 players, mobile phones
• scanning became more important than printing

2009: Core i5 2,66MHz, 4GB, 1,3TB

• Windows 7
• Multi-core
• Full HD support (via DLNA to PS3 or HDMI)
• Virtual machines are fun to work with

Telerik Code Converter

Telerik Code Converter is an indispensable tool for every Visual Basic developer.

Visual Basic is a distant second to C# in the .NET ecosystem. Yet, there are strongholds of VB and you may find your job to be in one of them, like I did. For example, VB is strong in DotNetNuke-based development since the core framework itself is written in VB.

Most VB developers use C# as a second native language and it is very common to end up having a multilingual codebase, with projects in both languages. Converting code snippets from one .NET language to another is needed often, since your codebase is the first resource you have when building new stuff.

Also, most public code samples are written only in C# these days.

-mika-

Embedded Reporting with Chart Controls

I've previously written about embedded reporting in general and looked at Telerik Reporting in more detail. But one subject wasn't touched on these posts, that is, creating operative reports without a reporting engine. Reporting engines provide a lot of features, sure, but you don't always need these features. If the requirement is to embed charts directly to your web application's views, a heavy reporting engine is just getting in your way. Think for example the Google Finance Flash chart pictured below, which for all practical purposes represents the gold standard of interactive time-series plotting:

Would you like if that plot was rendered with an SSRS-style engine, with 'generating report' animation spinning between reloads. Neither would I. Using common report elements like chart controls directly in your web app makes operative reports more responsive and the overall experience more 'webby'. But you won't get a Google Finance chart experience with the current 3rd party chart controls. The high end seems to be more like the Google Analytics -style plot with data point highlighting, which is nice too.

There are two approaches to web reporting: client-side RIA controls and server-side web controls. The distinction between the two is increasingly difficult to define technically, and I won't go into it now. Let's just say that the developer experience is different enough when using a jQuery plugin compared to a ASP.NET AJAX control that you are inclined to contrast and compare the pros and cons of both.

The charts that I've been using as a an exercise are taken from a Campaing Monitor email tracking report, which itself is a shameless visual copy of Google Analytics reports. Kudos for both! There are two charts in this report: a time-series plot with data point highlighting and a pie chart with a legend.

Now, let's try different tools to imitate this report

jQuery Chart Plugins

• flot
• jqPlot
• jgCharts and Google Chart API
• sparklines and visualize

Javascript source code is available to all users, which makes open source a natural model for javascript-based tools. But it also means that feature-rich and well-supported tools don't just drop out of the sky, they will evolve gradually. The state of the art today is that free jQuery plugins don't either have the features or the quality to substitute a commercial chart control suite. But they are useful enough for many purposes and are simple and fun to play with. They probably also represent the future of embedded reporting. Sooner or later, one such plugin is going have enough lift to really take off. This is simply because jQuery plugins are platform-independent and consequently have appeal to an enormous plugin user and plugin developer bases. But if you are interested in commecial-quality javascript charts today, you should take a look at Emprise Javascript Charts

Technology

The jQuery plugins I tried generally utilize the HTML5 canvas element. IE does not support canvas yet, but Google's ExplorerCanvas does a decent job of emulating its functionality.

Javascript basically has two alternative methods of data access. The most straightforward method is to print data directly to the response, usually as a JSON-formatted value to a javascript variable, or as an HTML table accessible via DOM. jQuery Visualize plugin has taken the HTML table approach while others rely on JSON. But scraping data out of tables is a handy trick you can use with all javascript charting tools. The other method is to remotely load data with XHR or using cross-domain JSON.

Whatever means the client uses in reading the data, it is ultimately generated in a server somewhere. From the server-side view, printing data directly to a response can be easier than serving it through an url, especially if the data is not meant to be public. For example, I get DotNetNuke portal's page-level security for free if I print data.

Developer experience

Aside from technical capabilities, I'm especially interested in the general developer experience. The key question is: Is it simple and reliable enough for your boss? Can you recommend this tool to someone who does not have much time but who is tightly involved with customers and would like to use this first-hand information do reports without delegating them to others?

For jQuery plugins in general, I would say that if your boss has done client-side web development and knows how to use firebug, then jQuery plugins are safe. It is best to set up the environment beforehand, checking that jQuery version and plugin references are correct, and seeing that data access is working. Server-side work involves querying the data and converting it to proper format. As mentioned before, this is not necessarily AJAX, if you just write to the response page. One thing to remember with time values is that javascript requires you to convert .NET ticks to Unix/Javascript ones. Not necessarily something your boss would like learn the hard way.

The best thing in developing with jQuery plugins is the rapid development cycle. 'Installing' and testing multiple plugins in a single page is fast and fun. It took about half a day to evaluate all the plugins in this review. I have grown up to expect increased levels of frustration when dealing with new ASP.NET controls, and finding less frustration in javascript development was a nice surprise. Charting is a good fit for javascript because codebase for a single chart shouldn't grow too big and hairy.

Oh, and one more thing before going to the plugins. If you are interested in Javascript charting, go check Steve Reynolds's reviews:

• 6 jQuery Chart Plugins For Your App
• 5 Javascript Chart Alternatives

flot

Flot produces nice looking lines, filled ares and columns, but no pie charts so far. The samples are great, showcasing such features as zooming with overview and colored weekends in time-series plots. Interacting with data and point highlighting with tooltips are supported too. Values for undefined parameters like axis ranges are well-thought.


In my opinion, flot is the best jQuery plugin for the chart types it supports. Solid web development skills are required to use flot, and you shouldn't be afraid to learn from real-world examples.

jqPlot

jqPlot looks very polished for an open source project and has nice features like data point highlighting. But the good first impression fades somewhat when you try it with other browsers than Firefox.

Here's a jqPlot pie chart in FF 3.5, showing email tracking data. The labels are in Finnish and mean "Opened" and "Not opened".

In Chrome, the same chart looks like this:

As you can see, the colors are inverted. If you mistakenly input data as text without parseInt(), FF is ok but Chrome will render the shares wrong:

IE8 with excanvas works like FF, but again, if you input text instead of numbers, the percentage shares start to look interesting, even more so than in Chrome:

Even with these inconsistencies, jqPlot pie charts would be my choice for a jQuery Google Analytics replica.

jgCharts

jgCharts uses the Google Chart API but does not support all features from it. You can't color a pie chart with jgCharts, for example. Yet, most of the shortcomings originate from the Google Chart API. First, jgCharts do not impress visually compared to flot or jqPlot, which is not surprising since the charts are intended to be loaded as ordinary HTML image elements. This means that the API is the only way to customize the charts as CSS and javascript are out of the question. Animations and interactions like drill-down urls are naturally missing too.

Others

jQuery Sparklines didn't impress visually and I couldn't do even basic customizations to a pie chart. There is a reason for that. A sparkline is a type of information graphic characterized by its small size and data density. This is probably the optimal use for Google Chart API too.

Visualize demo charts look nice, but the feature set is quite poor. Interactive features and animations are not supported, and there are only a handful of parameters. All data is scraped from a HTML table, which means that you'll have to explicitly hide the tables you don't want to show. With other controls, the problem is exactly the opposite: you need to do extra work to show data also in a HTML table. But every jQuery plugin can easily scrape data manually from a table once it is there, which is much more flexible than creating tables in the exact format that a plugin needs. For example, in the previous jqPlot and jgCharts pie charts data was scraped from an ASP.NET datagrid. Visualize couldn't use that table directly. In the picture below there is a HTML table in the upper left corner and a corresponding visualize chart with default settings, as seen in Firefox and Chrome:

With IE8, the outcome was not so good, but at least those black spots in percentage shares were gone:

Flash

• FusionCharts Free
  
• Open Flash Chart

Flash is the technology of choice for professional web charts, as witnessed by Google Finance and Analytics. Silverlight certainly has potential too, but because of the relatively small install base I decided to drop Silverlight Toolkit from this review.

Technology

The worst part of Flash is the Flash development itself. ActionScript is not exactly known as the language of choice in software development. Developing Flash also requires Adobe Flash Professional or Flex. But if you don't intend to develop Flash chart controls yourself, none of this really matters. Right now and in the near future, Flash chart controls are the most sophisticated ones around, both in their visual outlook and in the features they support. But you should expect Javascript libraries to close the gap pretty soon, at least from an end user's point of view.

Data access methods in Flash are a superset of the methods available for Javascript. You can use Javascript to feed Flash objects and the Flash charting tools provide good Javascript APIs. In addition, you can use the Flash object's parameters for data access. The urls defined in the Flash object are subject to different rules than the urls inside Javascript. Authorized cross-domain data loading is available and is controlled with policy files. Second, some but not all Flash charts support printing data directly to a swf parameter.

Developer experience

If you can avoid Javascript, Flash charting is extremely simple. Well, it is just HTML objects then. Any developer should be able to use these kind of charts. If you cannot avoid Javascript, the experience is similar to jQuery development.

FusionCharts Free

FusionCharts Free (FCF) is a free older version of the popular FusionCharts, the market leader in Flash charting. First time I took notice of FusionCharts when they started to show up in popular web applications, especially RescueTime uses them heavily. Creating the example email tracking charts was no problem in FCF:

FCF has all the polish of a commercial product:

• example charts that look good
• compelling feature set, but these features are yet very easy to use
• high-quality documentation and support forums.
• support for multiple server-side technologies

FCF's .NET interface is extremely simple. You replace a literal with a Flash object:

litFC.Text = FusionCharts.RenderChartHTML("/FusionCharts/FCF_Line.swf", strUrl, strXML, chartID, width, height, False);

XML data and settings are provided via an url, or are embedded direcly to a HTML Flash object. Great thing is that .NET interface does not try to act as a wrapper to XML, you generate the xml yourself and pass it as a strXml parameter to the above FusionCharts.RenderChartHTML function. FCF reads xml settings liberally and does not mind if you use settings from different chart types in one setting xml.

Each FCF chart type has a separate swf file, which is a minor complication because you need to know multiple pathnames in using the API. I wrote a helper enumeration to get correct filenames using IntelliSense.

The only problem with FCF is that it is a legacy product. The actively developed product is Fusion Charts 3, and the differences between the two are substantial. Good thing is that you have an upgrade path, but don't expect any new features in the free version.

FCF is the easiest charting tool to work with, and even nontechnical people should be able to draft chart requirements with it.

Open Flash Chart

Open Flash Chart (OFC) is a popular open-source alternative to FusionCharts. OFC sample charts look good, they are definitely the best looking ones from the charts I've tested here. But this does not help too much if you won't get your charts to look good. And unfortunately OFC is not too easy to work with.

OFC documentation is PHP-centric. For other environments, there are support forums but not much more. To use .NET you can compile the shipped .NET library or use an alternative one.

Unlike FCF, OFC has only flash binary, open-flash-chart.swf. Chart type is defined in the JSON-formatted data file. Compared to jQuery plugins, this JSON it is quite complicated to create manually. Good thing is that there are server-side libraries to create OFC charts that handle the conversion to JSON. But you still need to use javascript to pass the data to the chart. This is not as simple as passing XML data directly to the flash object parameters, as in FCF. You need Firebug to debug things, tolerate Firebug/Firefox crashes etc.

The default .NET API acts as a JSON wrapper class and is not nearly as simple as in FCF. The bad thing is that this abstraction leaks, you can't control every setting in JSON with the .NET API, or at least you have to spend a lot of time finding out how to achieve this.

One particular annoyance was that OFC dates use PHP mktime() ticks, that are otherwise the same as Unix/Javascript ticks but are counted in seconds instead of milliseconds.

My first impression was that OFC is hard. I wouldn't let my boss use it.

ASP.NET

• Telerik RadChart
• Devexpress XtraCharts

It should be noted that the RIA controls I tested are free or open source but the ASP.NET chart controls are commercial products, usually shipped inside a larger toolkit. The controls are well productized and strive for high user productivity while offering the maximum feature set. This is not easy to achieve, and therefore it is important to support the user when the inevitable troubles arise. Money does not buy you a solution to every business charting case, but it does buy you many well-supported scenarios.

Technology

ASP.NET charts are technically using the same client-side web platform as the jQuery charts. In practice, there are probably quite a few constraints in developing a generic .NET chart control that do not exist in pure Javascript development. I wouldn't expect ASP.NET charts to visually match the best Javascript ones. But at least it would be nice to have some interactive features and animations, which are mostly missing now.

Rich data access is the strongest point of ASP.NET controls. You can use native ADO.NET data structures directly instead of serializing to JSON or XML. Another .NET advantage is that the chart controls are usually built as generic all-in-one charting solution for both ASP.NET and WinForms. The same chart control is also used in the vendor's reporting solution which can be useful.

Developer experience

Visual Studio IDE and the .NET tooling support in general are great, sure, but the focus here is on the usability of chart controls. Commercial server-side chart controls typically have lots of features and the learning curve can be steep. In the best case you have decent Visual Studio design-time support, which is intended to work like WYSIWYG for UI programming. In practice this means property grids to make settings and some visual output on how these settings affect the chart. But if you can't get design-time support to work, you'll only see the IntelliSense code comments and external documentation while coding, and the final outcome is visible only in the browser.

Generally, getting an example chart up and running takes more effort in ASP.NET than with jQuery plugins or Flash charts, because you have to set up a IIS web application and prepare an aspx page, a code behind file and a web.config. Development and testing cycle is slowed down by the time it takes to compile the solution and reload the ASP.NET site. Reloading is particularly slow if you have lots of dlls in the /bin folder. But again, this same reload happens when using a Flash chart .NET API, or when doing server-side .NET data access with any front-end charting tool.

I don't like the ASP.NET heavy lifting needed to draw a few charts. But if your boss is an ASP.NET developer, and you provide the environment and a few samples, things should be ok.

XtraCharts

XtraCharts is a part of the DevXpress toolkit. Deployment requires 16mb of DLLs to be loaded in every application restart, only for charting. That's heavy and takes a few seconds in your dev machine.

Demos are nice, I liked the spline plot, but as you can see, the chart is a .png image, which prevents any cool animations or mouseover interactions inside the chart. Outside the chart image you can have tooltips as in the "Weather in London" chart.

For some reason, I couldn't get the Visual Studio designer support work in a DotNetNuke module project. The designer showed an error:

Probably some setting is wrong as I was developing in a Visual Basic project. Working manually with the code-behind file is fine with me, but the bad thing was that I already faced a problem after a few iterations of the pie chart. Below is the chart with labels but no legend:

Setting the legend visible caused the chart to disappear:

Issues like this arise in every complex control, but debugging them makes you appreciate the fast develop/test cycle of the client-side tools.

Telerik Charting

I have used Telerik RadChart previously with Telerik Reporting, and my perception was kind of lukewarm then. But I should have more perspective now.

First the good news. Designer support for Telerik Charting works pretty well, even though the property grid is hard to use when fine-tuning the chart. WYSIWYG works even in the data level: you can put data items in the aspx, and they show directly in the designer, as pictured below!

Having data in aspx also means that you need not to do anything in the code-behind to get a working report. But frankly, I don't find any use for data-in-aspx other than simple demos.

Compared to XtraCharts, some useful chart types are missing, or they don't look as good. For example, a line plot demo has labels but not visible data points. XtraChart demos are more complete, easier to navigate and better-looking than their Telerik counterparts, but the difference is not big. Both tools produce static image-based charts that support only image map interactivity, zooming support in Telerik line charts being an exception. It should be noted that Telerik also offers Silverlight charts that are on par with Flash with respect to pizzazz.

And then the bad news. I couldn't get the Telerik charts working at all in the browser. All that I got was an error message:

BC30560: 'ChartSeriesType' is ambiguous in the namespace 'Telerik.Charting'

Normally, I would debug for an undefined time to get some results. But this time I have to call it off. No more ASP.NET debugging today.

Final words

Programming is a profession of solving real-world problems with tools that are built to look great in demos. But you shouldn't blame the tool if you can't ever achieve the same results in the same time on your own, no matter how much you spend time debugging with the tool. This is the reality. But sometimes you do get positively surprised by a development tool that just works as promised. FusionCharts Free is such a product. The people who designed and implemented it five-to-ten years ago have done a hell of a job. Thank you!

I chose Fusion Charts Free for our company, and it is probable that we upgrade to the latest commercial version in the future. FCF has the looks, simplicity and support required for a business where charting is a nice-to-have but not a critical feature. A great thing is that FCF with inline dataXML data and no mandatory javascript is a fool-proof charting environment that is safe for your boss too.

Open Flash Charts was the runner-up, entirely because of its looks, but it is too complicated for occasional use. If I was doing serious charting I would compare OFC more carefully with the commercial version of Fusion Charts.

jQuery plugins are promising, but so far they are fit for only lightweight charting tasks, mostly because they lack features. Javascript is still not for everyone, but for javascript-savvy programmers the developer experience is extremely productive, until you hit any edge cases.

It makes sense to use .NET chart controls if you can piggyback some of the work you have done with the other controls in Telerik or DevXpress toolkits. Or if you are doing WinForms charts too. But as standalone web charting tool, RIA charts are better.

-mika-

Microsoft Antivirus, Antitrust and the Public Good

Microsoft Security Essentials has been officially released.

This is good news for all Windows users and especially for the tech people who have gotten used to wrestling with 3rd party security solutions. If a friend calls me and says there is something wrong with his/her laptop, the first thing I presume is that security tools are causing trouble. The usual fix has been to uninstall some old brand-name security solution and replace it with Windows Firewall, Windows Defender and the free and fast Avira Antivir. From now on, I'd use Microsoft Security Essentials instead.

I haven't used an antivirus in my main laptop since I upgraded to 64bit Vista. But I do use antivirus in machines that I don't attend to, like my home server PC and the servers at work. The reason for not using real-time security software is performance. I need it. I'm willing to put effort in hardware upgrades like an SSD drive with clean-installeded Windows 7 to improve developer tool performance, but I'm not going to waste this investment by installing a new resource hog. Instead, I protect the machine with hygiene, by not installing anything unnecessary and keeping the updates fresh. If something goes wrong and an attacker gets through a vulnerability, I have a minimal setup backed up and recoverable fast. So far, I haven't needed it, because natural clean-ups like getting a new machine or installing a clean OS are frequent enough to keep the machine healthy. But I do run an off-line security scanners like Malicious Software Removal tool and ClamWin once a month or so.

For basic users and risk-taking power users, real-time antivirus is good. But even then you don't want to deal with the security software industry. It is just too hard to evaluate the net value of their offerings. You cannot be rational with security software unless you have a reliable cost estimate on being unprotected, and know the costs and benefits of a particular security tool. But this kind of analysis is hard, as the results depend on the value of your time and data in addition to the technical quality of a security tool.

The reason why Microsoft has not included antivirus in Windows already in the late nineties must have something to do with the antitrust regulators. Now, with the popularity of free antivirus rising, Microsoft is more safe to launch this offering. It will probably never be a part of Windows, which is bad for the overall security climate. On the other hand, this will probably be in the best interest of the consumer by keeping some free or commercial alternatives available. Historically, competition has been about the only thing that drives software quality improvements. As a warning example, it is good to keep in mind the internet dark age, 2001-2004, when IE6 didn't have a serious competitor.

Wrapping things up, unless Security Essentials turns out to be a real turkey, I'd recommend the following policies:

• For large enterprises, it makes sense to use centrally managed antivirus. But give developers an option to uninstall it in their machines, please!
• For nontechnical home users and small business users, who don't need most of the performance most of the time, real-time antivirus like Security Essentials is good. But you need to do backups, always.
• For unattended machines, antivirus is a must but you can have a scheduled virus scanner instead of real-time scanning. Free antivirus for Windows Servers is rare, I have used ClamWin for these. Managing ClamWin is tricky if machines can't connect to the update site, like when outbound http requests are blocked by a firewall.
• For performance-intensive workstations, minimize the amount of unnecessary software, including security software. You are ultimately responsible for the security, not some black-box application.

-mika-