Monday, August 30, 2010
Modern websites necessarily make a lot of requests for resources back to the server. Through books like those by Steve Souders we’ve become increasingly aware that each of these requests causes a delay in whatever it is your user has requested being downloaded and displayed. If you don’t have both of Steve’s books, get them…and spend an hour or two going through his site…it’s awesome stuff. The term commonly used for the various efforts to reduce the number of requests made to a server is HTTP Optimization.
This is a topic I’ve been investigating / playing with for quite a while (e.g., CSS Combiner / Minifier ASP.NET Control) and in fact one of the projects I worked on shortly before leaving the ASP.NET team morphed into the Sprite and Image Optimization Framework recently released as a preview by the ASP.NET team. What I wanted to do with this post wasn’t to go over these techniques in any great detail…rather it’s to cover some of the stuff I currently do when developing sites.
In short there’s x techniques you can realistically use to reduce / improve the performance of a web page loading:
1. Make the server code run REALLY fast…either by buying big servers, writing really slick code or by caching everything so it basically works like a static (plain HTML) site…
2. Make your web page really plain…no CSS / JS / Images…just plain old HTML; single request is always quickest – this is of course not usually acceptable!
3. Radically reduce the number of requests made to the server.
4. Make each request deliver more content for less bandwidth.
Of these I’m going to take a closer look at the last two in more detail.
The very first thing you need to do when trying to work out how to fix page loading issues is to actually identify those issues!
I tend to use Firefox when I’m looking at these issues; mainly because it has an amazingly powerful tool in Firebug for debugging and investigating pretty much anything a web site does on the client.
One of the most useful tabs in Firebug is the Net tab…in the image below you can see the result when I ran it with the home page of the ASP.NET site with caching disabled. I’ll use this page to point out some of the optimizations they’ve (recently!) made to improve loading times for the home page of the site.
What you can see in the image above is a bunch of rows containing GET and POST requests (we'll, only one POST to a tracking page). Each of these requests represents a single request back to the server. In fact if I roll my mouse over one of these rows I get more information which provides a legend for what each of the colors means:
I’ll come back to this page in more detail later…
Another great tool for identifying issues with page loading is YSlow, in fact if you look at the Firebug page above you’ll see that YSlow has it’s own tab! What YSlow does is report on issues with your pages in a very direct way:
As you can see, YSlow gives a grade for a number of specific items which have proven to make a difference in page load time. Rolling over each one will tell you why they’re important and give pointers. In fact, it even gives you links to tools which can help resolve these problems!
So, great…we can now see what problems we have…and even have a way to fix them. However, what it you could avoid them happening in the first place?
In part 2 of this post I’ll show you what you can do (and what I actually do) to avoid these problems in the first place. I’ll focus on using these techniques in Visual Studio and how you can easily incorporate some practices into your own development to get fast, fluid web pages…
Friday, August 27, 2010
Promising to do this for a while. I only really like to post about new stuff..others can cover old stuff in detail. Right now I’m learning:
- FubuMVC: a kind of super flexible, advanced version of an ASP.NET MVC framework. Has proven challenging to learn but really worthwhile.
- RavenDB: fantastic, document oriented database. Fits in really well with web development but has some unique challenges.
- JQuery: far from new, but building lovely UIs with it, REALLY improving my general Javascript dev skills (a lot thanks to this book).
- Spark View Engine: an alternative view engine (was for ASP.NET MVC but my boss @robertthegrey has written a brilliant FubuMVC engine for it too!).
- Dotnetopenauth : Open Id authentication library for .NET (this article was invaluable for simplifying the whole experience for me…)
In all, I feel like the technologies above really hit the sweet spot for me for web development for the next little while. So, I’ll write about them all. As usual, not tutorials, more hints, tips and guidelines which may help you avoid some of the learning curve I’ve had recently…
Until then here’s some lovely fish to tide you over (flash so you have to visit the site to see them :)).
Monday, July 12, 2010
Mainly for me to remember this but if you’re using the excellent Versioning bundle for RavenDB then you’ll find you quickly need to work out a way to get the latest version of a document if querying on any field other than ID (I *believe* ID actually only gets the latest one). The little index definitions below do just that (the secome one is just adding another field to the map so I can query by that…useful example)…
documentStore.DatabaseCommands.PutIndex("CurrentVersion",
new IndexDefinition
{
Map = @"from doc in docs
where
doc[""@metadata""][
""Raven-Document-Revision-Status""] !=
null &&
doc[""@metadata""][
""Raven-Document-Revision-Status""] ==
""Current""
select new {doc};"
},true);
documentStore.DatabaseCommands.PutIndex("CurrentImageVersionBySlug",
new IndexDefinition
{
Map = @"from doc in docs
where
doc[""@metadata""][""Raven-Entity-Name""] ==""Images""
&& doc[""Slug""]!=null &&
doc[""@metadata""][
""Raven-Document-Revision-Status""] !=
null &&
doc[""@metadata""][
""Raven-Document-Revision-Status""] ==
""Current""
select new {Slug=doc[""Slug""]};"
, } , true);
Friday, July 09, 2010
So, I’m not dead! Have been busy moving house, getting two new kittens (Tycho and Cozmo) and bascially settling into my new existence.
I HAVE also been getting back down to some serious coding. LOVING RavenDB (I seriously think it is THE .NET database of choice now), getting deep an dirty with some awesome HTTP optimization techniques…which I really do have to blog about soon!
My first pointers to HTTP optimization greateness are:
Chirpy: a brilliant little VS add-in (check the Codeplex page for the VS 2010 version) , compresses and combines your CSS and JS files as well as lets you use .less syntax easily inside VS (kinda a programmatic extension to CSS allowing for cleaner, less repetitive style definitions).
LabJS: a bit more complex but really stunning little bit of JS which lets you seriously optimize your JS loading…JS becomes non-blocking and all loads asynchronously (see http://labjs.com/description.php for more details)! Here’s how I use this in one of the cozwecan sites (note, I also use LabJS to do a ‘load from CDN with fallback’ behavior which I have yet to see fully documented) …
var scriptModifier = "";
$LAB
.setGlobalDefaults({ AllowDuplicates: false });
$LAB
.script({ src: "http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js",
alt: "/Scripts/jquery.1.4.2.min.js",
test: "jQuery"
}).wait()
.script({ src: "http://ajax.microsoft.com/ajax/jquery.validate/1.7/jquery.validate.min.js",
alt: "/Scripts/jquery.validate.min.js",
test: "jQuery.validate"
})
.script("/Scripts/jquery.ba-bbq.min.js")
.script({ src: "http://cdn.jquerytools.org/1.1.2/tiny/jquery.tools.min.js",
alt: "/Scripts/jquery.tools.min.js",
test: "jQuery.tools"
})
.script("/Scripts/jquery.easing.1.3.min.js")
.script("/Scripts/jquery.jnotifica.min.js")
.script("/Scripts/Custom/MainScripts" + scriptModifier + ".js")
.wait(function () { SetupInitialLoad(); });
Anyway will get a blog post together eventually covering all that I’ve been up to…until then…follow me on Twitter @scottgal..late peeps!
Saturday, March 20, 2010
Ok, not a catchy title but it really is what I’ve been up to for the past 3 months. As I wrote before, I’m currently working with the awesome @RobertTheGrey on a project called cozwecan, which is a bunch of sites set up with the aim of being an amazing independent ecommerce platform for artists (photographers to start with, in the future, who knows…). Anyway, I’ve been remiss in blogging about the amazingly cool stuff I’ve been doing (IMHO :)), using WPF with Reactive Extensions, MongoDB, FluidKit and of course the awesome WPF / SL MVVM framework, Caliburn (the embedded bit aboveis the creator of Caliburn, @EisenbergEffect’s presentation on MVVM from MIX). So, just licking my wounds and getting my brain back in one piece after last year’s craziness (minor drug problem, clinical depression, car crashes, international moves etc…) and will be back to full communicative me soon enough :)
Monday, February 15, 2010
In the little app I’m working on I’m using MongoDB…which is great, however MongoDB only supports storing Document objects. Which are kinda like Dictonaries (Document[“blah”] = “bar”;). I want to use AutoMapper to map these loosely types Key-Value Pairs to my objects and I’m basing it on some code I found drifting around:
This works:
public static IMappingExpression<Document, TDestination> ConvertFromDocument<TDestination>(this IMappingExpression<Document, TDestination> exp, Func<string, string> propertyNameMapper)
{
foreach (PropertyInfo pi in typeof(TDestination).GetProperties(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic))
{
if (!pi.CanWrite ||
pi.GetCustomAttributes(typeof(TDestination), false).Length == 0)
{
continue;
}
string propertyName = pi.Name;
propertyName = propertyNameMapper(propertyName);
exp.ForMember(propertyName, cfg => cfg.MapFrom(r=> r[propertyName]));
}
return exp;
}
This maps my document to an object…cool…however, I’m having real problems figuring out the reverse (my lambda / knowledge of AutoMapper are letting me down):
This is NO way works…
public static IMappingExpression<TSource, Document> ConvertToDocument<TSource>(this IMappingExpression<TSource, Document> exp, Func<string, string> propertyNameMapper)
{
if (typeof(TSource) is IBaseObject)
{
foreach(PropertyInfo pi in typeof(TSource).GetProperties(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic))
{
string propertyName =pi.Name;
propertyName = propertyNameMapper(propertyName);
exp.ForMember(r=>r[propertyName], cfg=>cfg.MapFrom(p=>pi.GetValue(p));
}
}
else
{
throw new Exception("What type are you trying to give me???");
}
return exp;
}
Help my Jimmy Bogard, you’re my only hope :)
Saturday, February 13, 2010
I wrote this for use in a little project but as I’m no longer using it, I though I’d stick it on here for anyone who wants it. Essentially, this allows you to take a byte Array (in my case, in the File property of my ThumbnailViewModel) and get it back as a BitmapImage for use in DataBinding…
It also has the property of accepting a parameter which lets you specify the size of thumbnail to use (ThumbSize). Here, I actually have multiple ThumbNails (in the List<ThumbNailViewModel>) and select one using the parameter.
Given this resource string and XAML,
<UserControl.Resources>
<local:BytesToBitmapConverter x:Key="bytesToBitmapConverter"/>
</UserControl.Resources>
<Image Name="IconImage" Source="{Binding Path=ThumbNails, Converter={StaticResource bytesToBitmapConverter}, ConverterParameter=Small}"/>
public class BytesToBitmapConverter : IValueConverter
{
public object Convert(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
{
var thumbs = value as List<ThumbnailViewModel>;
var param = parameter as String;
var thisThumb = thumbs.Find(x => x.ThumbnailSize == (ThumbSize)Enum.Parse(typeof(ThumbSize),param));
BitmapImage bi = new BitmapImage();
bi.BeginInit();
bi.CacheOption = BitmapCacheOption.OnLoad;
bi.StreamSource = new MemoryStream(thisThumb.File);
bi.EndInit();
return bi;
}
public object ConvertBack(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
{
throw new Exception("The method or operation is not implemented.");
}
}
Wednesday, January 20, 2010
Well, OK I’m over dramatizing it a tad :) As you may have noticed from my last post I’ve started working of some stuff which is outside my comfort zone. I recently joined a company called cozwecan,working for a chap named @RobertTheGrey, the first project we’re working on is a photo sales site…you know kind of like Getty Images / Smugmug / Flickr except for direct photographer-customer sales. As the only full-time developer (so far!) I’ve been tasked with building the vast majority of the applications (be they web, desktop, etc…) which we need for the business to work.
Obviously one of the most critical bits for an e-commerce site is stuff to sell! In our case this is a stock of really high quality photographs taken by professional photographers around the world. First question; how do we get these images on the site, and how do we let the photographers add information to their photographs to let them be found by folks who want to buy them? Well, that’s the first problem I’ve been asked to solve!
When working on the requirements, it quickly became clear that this had to be a desktop app, it needed to allow operations on multi-megabyte photographs, needed to allow ‘tagging’ offline. Especially during initial loading, there’s 10-100s of thousands of images getting worked on…forcing this to be online adds a nasty lag and makes the task a pain in the butt to complete. Above all, it needed to be reliable!
Now, in my dev head this led to a number of decisions about the features the app would need to support:
1. Intuitive, responsive UI; the users are not necessarily computer-savvy, need to use obvious UI metaphors and make it fast enough to be pleasant to use.
2. Needs to do some offline image processing; we’re planning on hosting this on the cloud….CPU time costs cash :)
3. Should work offline
4. Needs to be built around reliable upload of multi-meg files (across potentially tiny connection bandwidths).
So, these are the basic needs…technically this leads me to some technology choices:
1. Multi-threading / concurrency (as with any real desktop app) is key…as is a decent Desktop technology…
2. See above, we need to ‘watch’ for new files arriving, make thumbnails transparently etc…
3. So, it needs some sort of persistence…along with other non-core app requirements (e.g. tagging) we’ll likely need a little, lightweight DB.
4. Needs ‘block’ based uploads with retry / verification of uploaded items…
Luckily I’ve been a dev for quite a while and had at least some idea where to start…the biggest challenge would be learning!
Firstly, the obvious desktop platform de jour is WPF…this is in essence the current replacement for WinForms, it’s VERY customizable but has a somewhat infamously steep learning curve. I’ve spent a bit of cash on books on WPF, followed all the PDC presentations, podcasts etc…on the subject matter. After 3 weeks I’m finally getting comfortable with it…
Second, I needed a DB…needed to be lightweight, needed to support LINQ (what, I like LINQ :)), needed to support some sort of ORM technology.
The most obvious choice was SQL Server Compact Edition, a kind of cut down SQL server which is great for desktop apps…however…I could not get it working! Turns out it’s really sensitive to the versions of various installed components. This was a red light for me…I needed the DB to be as stand-alone as possible! That left one obvious choice:
SQLite, from the website:
‘SQLite is a software library that implements a self-contained, serverless, zero-configuration, transactional SQL database engine. SQLite is the most widely deployed SQL database engine in the world. The source code for SQLite is in the public domain.’
Perfect! For .NET support, I used the SQLite ADO.NET provider (there is a C# only port of SQLite, but it’s slower and not needed for my scenario. Deployment is literally including a reference and creating a file…now it does have one pretty severe limitation which I’ll cover later :)
My ORM choices were nHibernate or EF…there being support for SQLite in both of these platforms.
Now, I have used nHibernate for a recent project (disclaimer, I left fairly early in the project :)) and one of the things which annoyed me was the large number of dependencies, and the seeming fragility across versions of these dependencies. For a low impact desktop app, this was just too large a footprint and too big a risk. So I had to count it out.
This left me with one choice…Entity Framework. Now, this wasn’t an easy choice…I have used EF in the past (tinkered with it) and found it a painful experience…(others have too which led to this). Now, I’m not getting into the politics behind MS choice to push EF (and yes, it was a political choice), but V1 is fairly buggy, and for me at least fairly unnatural. This led to my second learning ‘opportunity’…Master Entity Framework! As I write this, it’s literally 2 hours since I worked out my last bit of EF pain…which led to this model:
Looks simple…well, yeah…but it’s taken me AGES to get to this…I know SQL really well, C# REALLY well, but the mix of obscure concepts, poor documentation and obfuscated error messages makes working / learning EF way too hard! EF 4 (in VS 2010 / .NET 4) improves this…and it can’t come soon enough!
So, that’s worked out…I hope :)
So, I mentioned previously that SQLite has a bit of an issue…it doesn’t really like concurrency. Well, that’s an overstatement…it is a multi-read, single write system. Once you realize it’s an issue it’s fairly easy to solve. I use this pattern:
public static void AddRange(IEnumerable<UploadFile> files)
{
using (PixEntities ent = new PixEntities())
{
using (new ReadLock(entityLock))
{
using (new WriteLock(entityLock))
{
foreach (var file in files)
{
ent.AddToUploadFiles(file);
ent.SaveChanges();
}
}
}
}
}
As you seen, this has two main elements…I use a ReadLock (where I would normally read *from* the DB, in this case umm, I don’t :)) then I use a WriteLock to wrap any operation where I write to the DB. The observant will also notice what looks like a bug…I do ent.SaveChages() for each iteration of the loop…this is another SQLite quirk, it doesn’t support batch updates…and gives a weird error about file locks if you try :)
The little locking class I use is from here, but I’ve put it below for your use…anyway, this is part one of several parts of my experiences in working on cozwecan…stay tuned!
public static class Locks
{
public static void GetReadLock(ReaderWriterLockSlim locks)
{
bool lockAcquired = false;
while (!lockAcquired)
lockAcquired = locks.TryEnterUpgradeableReadLock(1);
}
public static void GetReadOnlyLock(ReaderWriterLockSlim locks)
{
bool lockAcquired = false;
while (!lockAcquired)
lockAcquired = locks.TryEnterReadLock(1);
}
public static void GetWriteLock(ReaderWriterLockSlim locks)
{
bool lockAcquired = false;
while (!lockAcquired)
lockAcquired = locks.TryEnterWriteLock(1);
}
public static void ReleaseReadOnlyLock(ReaderWriterLockSlim locks)
{
if (locks.IsReadLockHeld)
locks.ExitReadLock();
}
public static void ReleaseReadLock(ReaderWriterLockSlim locks)
{
if (locks.IsUpgradeableReadLockHeld)
locks.ExitUpgradeableReadLock();
}
public static void ReleaseWriteLock(ReaderWriterLockSlim locks)
{
if (locks.IsWriteLockHeld)
locks.ExitWriteLock();
}
public static void ReleaseLock(ReaderWriterLockSlim locks)
{
ReleaseWriteLock(locks);
ReleaseReadLock(locks);
ReleaseReadOnlyLock(locks);
}
public static ReaderWriterLockSlim GetLockInstance()
{
return GetLockInstance(LockRecursionPolicy.SupportsRecursion);
}
public static ReaderWriterLockSlim GetLockInstance(LockRecursionPolicy recursionPolicy)
{
return new ReaderWriterLockSlim(recursionPolicy);
}
}
public abstract class BaseLock : IDisposable
{
protected ReaderWriterLockSlim _Locks;
public BaseLock(ReaderWriterLockSlim locks)
{
_Locks = locks;
}
public abstract void Dispose();
}
public class ReadLock : BaseLock
{
public ReadLock(ReaderWriterLockSlim locks)
: base(locks)
{
Locks.GetReadLock(this._Locks);
}
public override void Dispose()
{
Locks.ReleaseReadLock(this._Locks);
}
}
public class ReadOnlyLock : BaseLock
{
public ReadOnlyLock(ReaderWriterLockSlim locks)
: base(locks)
{
Locks.GetReadOnlyLock(this._Locks);
}
public override void Dispose()
{
Locks.ReleaseReadOnlyLock(this._Locks);
}
}
public class WriteLock : BaseLock
{
public WriteLock(ReaderWriterLockSlim locks)
: base(locks)
{
Locks.GetWriteLock(this._Locks);
}
public override void Dispose()
{
Locks.ReleaseWriteLock(this._Locks);
}
}
Thursday, January 07, 2010
UPDATE: Thanks for all those who commented on this. Turns out the problem was pretty simple. You just have to get rid og the ‘FileId’ and ‘DirectoryId’ properties. They’re ONLY for navigation so cannot be in the entity as fields too…easy peasy :)
I like EF, conceptually and in practice it’s pretty nice. EF4 adds a lot of the missing goodness. There’s one thing which ALWAYS stumps me in EF 3 though…this VERY simple paradigm…
Given this class layout (generated from a SQLLite DB)
See the little 0..1->* relationship? Should be dead simple, the DirectoryId in the UploadFile entity maps to the WatchDirectory Entity…with this association:
I ask you, how the hell can that fail??? But no matter what, this happens…
Error 3 Error 3007: Problem in Mapping Fragments starting at lines 124, 154: Non-Primary-Key column(s) [DirectoryId] are being mapped in both fragments to different conceptual side properties - data inconsistency is possible because the corresponding conceptual side properties can be independently modified.
G:\Work\cozwecan_code\UploadClient\cozwecan.pix.client\cozwecan.pix.client\Model.edmx 155 11 cozwecan.pix.client
Anyone got more of a clue about EF than I have???
Thought I’d just post some dumb little utility classes that I had hanging about in my code repository (and always end up trying to find).
They’re just little utility classes for finding out image format / extensions / content types for various images…(and before any smartass pipes up, yes much of this info is in registry…but this is intended for use where I can’t access that)
public static class Imaging
{
public static string GetContentTypeByImageFormat(ImageFormat format)
{
string ctype = "image/x-unknown";
if (format.Equals(ImageFormat.Gif))
{
ctype = "image/gif";
}
else if (format.Equals(ImageFormat.Jpeg))
{
ctype = "image/jpeg";
}
else if (format.Equals(ImageFormat.Png))
{
ctype = "image/png";
}
else if (format.Equals(ImageFormat.Bmp) || format.Equals(ImageFormat.MemoryBmp))
{
ctype = "image/bmp";
}
else if (format.Equals(ImageFormat.Icon))
{
ctype = "image/x-icon";
}
else if (format.Equals(ImageFormat.Tiff))
{
ctype = "image/tiff";
}
return ctype;
}
public static ImageFormat GetImageFormatByContentType(string contentType)
{
ImageFormat format = null;
if (contentType != null)
{
if (contentType.Equals("image/gif"))
{
format = ImageFormat.Gif;
}
else if (contentType.Equals("image/jpeg") || contentType.Equals("image/pjpeg"))
{
format = ImageFormat.Jpeg;
}
else if (contentType.Equals("image/png"))
{
format = ImageFormat.Png;
}
else if (contentType.Equals("image/bmp"))
{
format = ImageFormat.Bmp;
}
else if (contentType.Equals("image/x-icon"))
{
format = ImageFormat.Icon;
}
else if (contentType.Equals("image/tiff"))
{
format = ImageFormat.Tiff;
}
}
return format;
}
public static string GetFileExtensionByContentType(string contentType)
{
string ext = "bin";
if (contentType.Equals("image/gif"))
{
ext = "gif";
}
else if (contentType.Equals("image/jpeg") || contentType.Equals("image/pjpeg"))
{
ext = "jpg";
}
else if (contentType.Equals("image/png"))
{
ext = "png";
}
else if (contentType.Equals("image/bmp"))
{
ext = "bmp";
}
else if (contentType.Equals("image/x-icon"))
{
ext = "ico";
}
else if (contentType.Equals("image/tiff"))
{
ext = "tif";
}
return ext;
}
public static string GetContentTypeByFileExtension(string fileExtension)
{
switch (fileExtension)
{
case ("gif"):
return "image/gif";
case ("jpg"):
return "image/jpeg";
case ("jpeg"):
return "image/jpeg";
case ("bmp"):
return "image/bmp";
case ("tif"):
return "image/tiff";
}
return "application/octet-stream";
}
}