Wednesday, July 29, 2009

 

Announcing Log to Netpanel

This blog post announces a free Firebug extension I wrote, inspired by my visit to the Velocity 2009 conference and especially the workshop by Steve Souders and the talk by (among others) Jan Odvarko on Firebug and its extensibility [end of this video].

When finding performance bottlenecks in web pages, it can be very useful to look at the exact timing of the network traffic from the browser. There is a fair number of tools that helps you doing this. My personal favorite for doing this is the Net panel of Firebug.

Now often, when looking at a web pages loading times, I am especially intrigued by the holes in the graph (see below). These holes are caused by the browser needing time for other activities than downloads from the network. These can be client script parsing, client script running, parsing of HTML, CSS and the like, painting and repainting the screen, and so on. It is hard to guess what exactly the browser is doing in the holes and in this post, I present a Firebug extension that helps you fill in the holes: "Log to Netpanel". This extension can be downloaded here, but is not production ready and you'll have to install it manually.

Tutorial

Our web page is very simple: it contains some external javascript, some inline javascript and very little HTML. Below it the full source code of the page (the external library is actually not used in the page, which is not important for the example): As you can see, the page is very simple, including the jquery 1.3 core library from the Google API service and doing a bit of processing in a loop to create a long string concatenating numbers and spaces. If we load this page, the Firebug NET panel looks like this:

We see the load time of the HTML page and the javascript document. After that comes a considerable hole before the blue vertical line that indicates the DOMDocumentLoad event. So what happens between completely loading the javascript and the end of loading the HTML? A number of options come to mind:

  • The parsing and evaluating of the jquery library
  • The loop of 10000 concatenations (to me, this seems the obvious suspect
  • Writing the really long string into the document
  • Painting the screen after the HTML is complete

    • This is where Log to Netpanel comes in: we install the extension and then add some lines of javascript in strategic locations like this:

    Note that we use the standard console object as provided by Firebug to log information from your script. If you log on the level "debug" and pass "netpanel" as the first argument, the time will be added to the NET panel. In the second argument, you can pass a short descriptive text. If you call the function multiple times with the same text, only one line will show up, spanning the time from the first call until the last.

    So if we instrument this page (test here), the NET panel will show something like this:

    The timings from javascript combined with the loading times in the NET panel reveal the the hole is mostly spent doing the parsing and evaluating of the javascript. Only a fraction of that time is used concatenating strings and even less in writing it to the document. The time after writing, but before DOMDocumentLoad is presumably spent in rendering/painting.

    So what next? The extension has proved itself useful already (to me that is). The code is open sourced under the BSD license (same as Firebug itself). I would like to invite other developers with some problems that need solving.

    1. I would like to attach the extension to the console.time() function instead of the console.debug(), as the fit seems better. I didn't figure out how to listen for this event though.
    2. I now add the log lines to the first 'phase' the the Firebug.netProgress object exposes. Not sure if that is right
    3. In some cases you might want to log even before the first 'real' download is started (this happens when the page itself comes from cache or local file system). This is now impossible. Would like to fix this.
    4. I could very well imagine that the further development of Firebug will make this extension obsolete eventually.

    Enjoy!

    Labels: , ,


    # posted by Teun @ 10:11 AM 1 comments links to this post

    Thursday, June 25, 2009

     

    Chunked transfer and Response.Flush

    I've been attending O'Reilly's Velocity Conference over the past days and I must say that I've learned quite a few things about web performance that I didn't know. One subject that came up in several talks was using "Transfer-Encoding: Chunked" to allow for sending to topmost part of the page to the client while most of the work for the page hasn't even started. Example:

    If you look at this result page from Google, you'll note that the upper section of the page, with links to other Google properties, the logged-in indicator, the logo and the search box can be generated without doing the actual search. Most of the time on the server is spent looking for results matching the query, but you could create this first bit of HTML before even starting on the query. This is exactly what Google does. When your request comes in, the first send this header and only after that they start on the search query. This makes the time before the user sees any change a lot shorter and will be perceived as better performance. What's more, as the browser gets the references to external images and scripts sooner, it can start loading them earlier, resulting in a real (not just perceived) better performance.

    So how could we achieve this in ASP.NET? Problem is of course that we normally do any heavy lifting required in a page during the PreRender or EventRaising phases. The actual rendering of the page, including the very first segment, happens later: during the Render phase. It depends very much on the structure of your page, but in certain situations, you could maybe do something like this: Now if you wrap the start of the page inside this control, you will render that part of your page way before the Render phase. The flushing makessure that the automatic buffering is bypassed. ASP.NET will then automatically set Transfer-Encoding: chunked.

    Drawbacks:

  • You must make sure that no control or literal text comes before the Flusher or the order will get mixed up
  • Not sure how this will work with master pages
  • You cannot flush part of the HtmlForm control (this contains the hidden ViewState). Keep everything you may want to flush outside the form

    • Anyone with deeper thoughts around?

    Labels:


    # posted by Teun @ 7:48 AM 1 comments links to this post

    Wednesday, April 22, 2009

     

    Securing my online presence

    It will become even more important than today to be able to influence what people find when they google your name. Eventually, they will find out everything you publicly wrote or said and many things you did in public places. Search engines will not only turn up the stupid questions you posted on forums 20 years ago, but also recognize you in pictures of frat parties 30 years ago.

    But then again, it's only the top 10 finds that really matter. Most people never click through to page 2, let alone page 30. So what do you want to have on top? Google just announced that using your Google Profile, you will be allowed some influence in what appears on top for your name search. Also, you can place your blog as a validated link on the profile page. As this link has no rel=nofollow attribute, I would think that it will boost the page rank.

    I also kind of like the way they make your Google Reader shared items appear.


    # posted by Teun @ 12:30 PM 0 comments links to this post

    Friday, April 10, 2009

     

    Funda live: now as a mapplet

    A colleague pointed out that my iGoogle gadget for looking at the live search behavior on funda.nl, while cool, would be better implemented as a mapplet. I guess that this it true for most gadgets that have a map as their main UI. So this morning, while sick in bed, I looked over the docs for mapplets. Turns out to be really easy to convert a gadget to a mapplet: they basically are gadgets with a shared map to interact with. So with half an hour of playing around, I have a working mapplet version. It seems to be OK, but the title bar keeps blinking and the performance of the map seems less than in the gadget version. If anyone has any tips to improve this, please comment.

    # posted by Teun @ 9:32 AM 1 comments links to this post

    Wednesday, March 11, 2009

     

    See searching patterns on funda.nl live

    I just finished a hobby project where I try to visualize the live searching patterns of users on our site (www.funda.nl, the premier Dutch real estate site). What I basically do is track the geographical region containing the current search results. I then plot these areas in a Google Map on iGoogle with some light animation. You can click the areas to see the actual URL used to find results in this area.

    What I really like about this gadget is that it allows me to see how people search our site in a more understandable way (to me, that is) than the statistical facts of page views and click paths.

    The data is near-live and is refreshed every 20 seconds. We display only a subset of the real searches, though; about 15%.

    Click here to try this gadget on your iGoogle homepage.


    # posted by Teun @ 10:26 AM 0 comments links to this post

    This page is powered by Blogger. Isn't yours?