Since more than half a year Lovely Systems is using the extremely fast, super reliable, super easy, super duper NGINX Reverse Proxy. Congratulations NGINX! According to the latest Netcraft Survey 06/2008 NGINX is already number 5 of the Webservers worldwide! Seems like we had the right anticipation :)

As the blog post title mentions - Speed matters. There are some more news… We’re happy to announce that we upgraded our Loadbalancers and finally bought a pair of F5 BIG-IP Loadbalancers. F5 is the global leader in application delivery networking and the products are well known for their stability. To quote Töns Büker, our Servicemanager at o2 (Germany): “Der LB wird aber wahrscheinlich auch noch auf ping antworten, wenn jemand Benzin drübergegossen und ihn angezündet hat …”. After the upgrade we’ll be able to serve up to 650 MBit/s dynamic traffic from our Application Server Cluster and up to 2 GBit/s of static (e.g. Video) traffic from our content delivery cluster. 

So we decided to use a custom field in the http header to tell varnish how long an object should stay in cache. In the backend it was very easy to implement a few lines of code to set the custom field. I called it “Varnish-Control” . The field will contain an integer value which indicates the TTL in seconds. The main part of the work was to configure varnish to use this value instead of s-maxage form the field “Cache-Control”.

In words i did the following:

Once an item was fetched from the backend (so it was not found in the cache) I will set the TTL of the item and stuff it into the cache. Since the field “Varnish-Control” is useless for all request handling nodes on the way to the client i removed the field once the TTL was set.

The first approach was pretty simple:

sub vcl_fetch {
if (obj.http.Varnish-Control) {
set obj.ttl = 7d;
remove obj.http.Varnish-Control;
}
insert;
}

Unfortunately this solution did not care about the value I’ve set into “Varnish-Control” but uses a static value of seven days to cache items. This was not satisfying. So i tried to make it more dynamic. Since it is not possible to assign the value from obj.http.Varnish-Control to obj.ttl i had to use inline C code. This is done by using C{ … }C somewhere in the VCL file.

sub vcl_fetch {
if (obj.http.Varnish-Control) {
C{
char *ttl;
ttl = VRT_GetHdr(sp, HDR_OBJ, "\020Varnish-Control:");
VRT_l_obj_ttl(sp, atoi(ttl));
}C
remove obj.http.Varnish-Control;
}
insert;
}


Finding the right commands to access the header informations and write them into the TTL of the object was PITA. Anyway, I try to explain this two commands:

char *VRT_GetHdr(const struct sess *sp, enum gethdr_e where, const char *n)

• The first parameter is the session pointer…nothing more to say about this.
• The second parameter is a constant indicating from which object will be used to get the header informations. In this case it was “HDR_OBJ” which is equivalent to “obj” in plain vcl configuration scenarios. possible values are: HDR_REQ, HDR_RESP, HDR_OBJ and HDR_BEREQ.
• The third parameter is the name of the field i like to access. NOTE: The field name has to be suffixed with a colon and it has to be prefixed with the length of the field name (including the colon). This value has to be noted octal.

void VRT_l_obj_ttl(const struct sess *, double)

• The first parameter is again the session pointer.
• The second parameter is the TTL as double.

Thats it!

All VRT_x commands will be available in this context. This commands can be found in varnish-cache/include/vrt.h and varnish-cache/include/vrt_obj.h

official varnish project

A large part of the Lovely Team was working hard during the last 6 month to get the thing going.

The Technology behind it? Lovely!

lovely.news = cutting edge Python/Zope3 software, PostgreSQL Databases, ZEO Clusters, WSGI pipelines, Varnish Caches, Nginx Reverse Proxies, F5 Load Balancers.

.

The content on zoomer.de? Lovely as well! A young, dynamic team. Backed by long-term news experts. Just watch their video and use zoomer.de :)

Thanks a lot, lovely team. especially quodt, jukart, schwendinger, dobee, jayeff, wolfabua, stmartin, iham, pjee, haudum. you rock!

while it’s really bad that our services were down as well. i need to say that it was the first outage within 2 years! that’s awesome.

On the other hand - where is the support hotline? who can we call? who can we mail in case of troubles? My mail to the “Business Development Inquiries” was answered 11hours later…

But nevertheless: Amazon, keep on the good work - we’ll use your services more and more in future.

here’s the official statement from AWS:

Here’s some additional detail about the problem we experienced earlier today.

Early this morning, at 3:30am PST, we started seeing elevated levels of authenticated requests from multiple users in one of our locations. While we carefully monitor our overall request volumes and these remained within normal ranges, we had not been monitoring the proportion of authenticated requests. Importantly, these cryptographic requests consume more resources per call than other request types.

Shortly before 4:00am PST, we began to see several other users significantly increase their volume of authenticated calls. The last of these pushed the authentication service over its maximum capacity before we could complete putting new capacity in place. In addition to processing authenticated requests, the authentication service also performs account validation on every request Amazon S3 handles. This caused Amazon S3 to be unable to process any requests in that location, beginning at 4:31am PST. By 6:48am PST, we had moved enough capacity online to resolve the issue.

As we said earlier today, though we’re proud of our uptime track record over the past two years with this service, any amount of downtime is unacceptable. As part of the post mortem for this event, we have identified a set of short-term actions as well as longer term improvements. We are taking immediate action on the following: (a) improving our monitoring of the proportion of authenticated requests; (b) further increasing our authentication service capacity; and (c) adding additional defensive measures around the authenticated calls. Additionally, we’ve begun work on a service health dashboard, and expect to release that shortly.

Sincerely,
The Amazon Web Services Team

thanks mrtopf for photographing and blogging :) http://technorati.com/tag/snowsprint2008

Nothing more to say! see you next year!

– jodok

Welcome to Snow Sprint 2008! We are eagerly awaiting your arrival and working hard to get everything ready for you next week. That’s right…it’s NEXT WEEK!! How exciting! Whether it’s your first Snow Sprint, or you’ve been coming for years, we will be happy to see your smiling faces in the Austrian Alps!

EMAIL LIST:

This year is the largest Snow Sprint yet (50+), and with so many people coming from so many places, communication and planning are of the utmost importance. Thus, I beg you make sure that you are signed up for the Official Snow Sprint email list. This is where all official information will be sent from now on. In the next few days, there will be more information about topics, schedules, sponsorship, and more! So, please make sure you’re subscribed to that email list by sending an email, with the subject of “Subscribe” to: snow-sprint-2008-discussion@lists.openplans.org

IRC:

Also, the #snowsprint IRC channel on FreeNode is open for business! Please add #snowsprint to your auto-joined channels. Let me know if you need help getting an IRC client or connecting to the channel. During the sprint, it would be great if everyone had their IRC clients open for last minute announcements or reminders.

CALENDARS:

I am preparing an iCal file for folks to subscribe to. This will have a general schedule of events during the Snow Sprint, although nothing is set in stone. We can always make changes to events, talks, outings, based on weather, consensus, etc. If you have anything you’d like added to the calendar, let me know.

TO THE LIST!

Now before I get too far ahead of myself, subscribe to the email list! More details and information will be sent there soon.

QUESTIONS:

Miann and myself are coordinating the sprint, do not hesitate to ask either of us if you have any questions. If you have questions about the sprint CONTENT (topics, talks, technology, etc.) ask me, if you have questions about sprint LOGISTICS (arrival times, travel directions, sprint fees) ask Miann. Of course, at the sprint, you can ask either of us for help with anything, anytime! We’re here to help!

SPONSORSHIP:

We are actively seeking sponsors for Snow Sprint 2008! If you are interested in sponsoring with a donation of any amount, or if you have ideas of companies or individuals that may be interested in sponsoring this year, please contact Spanky at sponsor@3×6x9.com - we are preparing information and contacting potential donors this week.

Thank you, danke, merci, grazie, tack själv, köszönöm, kiittää te, mulţumesc

Merry Christmas and a happy new year.

p.s. this patch should also solve some of the issues with the old one… let us know if you experience any problems.

For the technique guys a overview: python code within zope/zodb, jquery and native JS, mysql databases, and a huge package of ajax bases services.