Camel internals

I'm playing with Apache Camel project and boy, it is not easy to understand it internals!
Lets see how it is initialized.

SpringCamelContext(DefaultCamelContext).doStart() line: 543
SpringCamelContext.maybeDoStart() line: 165
SpringCamelContext.doStart() line: 160
SpringCamelContext(ServiceSupport).start() line: 47
SpringCamelContext.maybeStart() line: 95
SpringCamelContext.onApplicationEvent(ApplicationEvent) line: 114

So maybeStart calls start which calls doStart which calls maybeDoStart which calls doStart.
Now question from combinatorics: how many function names can we generate from 3 words: "maybe", "do", "start" :)))

XMPP and web

It seems XMPP protocol finally is getting recognition which it deserves. On 2008 OSCON there are some talks about using XMPP for async notifications.
Paper one. Paper two.

But like any new technology there are some confusion of how and when to use it.

So with XMPP my server will have to keep millions connections?

If we replace http pulling with a client listening to the server it means that all clients will have a persistent TCP connection to the server. If you have millions of clients then you are in trouble. Right?

No. XMPP address (JID or Jabber ID) looks like email. For example me@jabber.com or if you use google mail, then your JID is the same as your google email. Now lets say your server is mycompany.com. and your jabber server is at xmpp.mycompany.com. If a client from google chats with a service at your server then there is no direct TCP connection between client and your server. Instead client connects to Google, sends a message "to:room@xmpp.mycompany.com" and google will connect to xmpp.mycompany.com. Now, what will happen when 100 google clients will connect to a chat room at your server? Google server will re-use the same TCP connection to send your server a 100 "presence" xmpp packages. Handling even thousands packages is not a big deal at all.

So your server will handle as many connections as many unique domain your clients have. Which should be small enough number.

What about JMS?

XMPP does not replace JMS (Java Message Service) or any other messaging service. The principal difference is that JMS can guarantee delivery of a message or will return an error in case if message can not be delivered in time. With xmpp if client disconnected even for a short time then the client will not see any messages sent while he was offline. In other words XMPP is not for offline operations.

In JMS ( or any other queue architecture) sender can issue a message and as soon as it hit the queue there no need to be connected. Client can go offline and when it is back online pick up all message from the queue.

A good example for XMPP is a client application which shows properties of object. If other user in the system edited a property then server can issue a broadcast indicating ID of updated object. Every online application will refresh the property immediately. There is no need to listen to those messages if client is disconnected.

Now let's consider a replication system. Lets say we have a Corporation which is integrated with some Service Provider. Corporation uploaded its data to a Service Provider but now we have a dilemma: users can update data either in legacy Corporation application or in Service Provider interface.

Service Provider can create a queue for the Corporation and add a trigger in DB which dumps all changes to the queue. Corporation has an application which listens to the queue and updates legacy DB with changes made with Service Provider application.

Now imagine that Corporation has been disconnected from Service Provider for some time (maintenance, networking problems, etc). Obviously when it is back online it is crucial to receive all messages issues during offline period. XMPP is no way the tool for this task. JMS is.

XMPP does not provide reliability. (Correction: there exist a publish-subscribe extension for XMPP which has persistent messages option. But not all servers have it implemented so for queuing purposes it is still better to use a reputable queue system).

When pull is better

XMPP helps in situation when updates happen not too often. If message traffic is high then each time you perform pull you are quite sure that you will get data. So there is not too much sense in implementing asynchronous notification. Unless you have strong requirements to the reaction time.

What about WS-notification/SOAP

The big problem is that HTTP protocol is one-way communication. Basically server never talks to the client. Server even does not know how to reach its clients. WS-Notification attempts to solve this problem by client calling the server and telling the server where it can find the client if event happen. In case of corporate integration it may work. You assault your IT department and get a server where you put your little callback web service. But there exist cases which never will work. Web browser generally speaking can not be accessible because of firewalls. So WS-Notification is not a generic solution. XMPP is a solution because it initiates connection to the server and remains connected so server always can send an even to the client.

OpenID Provider Identifier vs Claimed Identifier

It is not emphasized in OpenID specification (version 2.0) but when you log in into OpenID enabled site, you can either enter your Claimed Identifier, for example mylogin.pip.verisignlabs.com or alternatively you can enter just your provider address, for example "yahoo.com" and site will figure it out that the provider you've entered supports OpenID.

How it does this? Simlpe.
According to the spec url you entered is normalized: added "http://" and followed the redirect links. If you enter "yahoo.com" in your browser, you will be redirected to "http://yahoo.com/". Now, lets see how OpenID server location is discovered with help of small Ruby script.

require 'net/http'
h = Net::HTTP.new('www.yahoo.com')
resp = h.head('/', 'Accept' => 'application/xrds+xml')
puts "Code = #{resp.code}"
resp.each_key {|key| puts "#{key} = #{resp[key]}"}

This script issues "HEAD" http command to the yahoo server. Note "application/xrds+xml" Accept header. If the server supports OpenID it should return a header indicating where OpenID server can be found.
The result is:

vadim@ubuntu:~/Projects/yadis_test$ ruby test.rb
last-modified = Sat, 15 Mar 2008 06:22:40 GMT
cache-control = private
vary = User-Agent
connection = close
p3p = policyref="http://p3p.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM ..."
content-type = text/html; charset=utf-8
date = Sat, 15 Mar 2008 07:01:22 GMT
content-length = 9533
x-xrds-location = http://open.login.yahooapis.com/openid20/www.yahoo.com/xrds
accept-ranges = bytes
vadim@ubuntu:~/Projects/yadis_test$

Here we go. Yahoo OpenID server is located at http://open.login.yahooapis.com/openid20/www.yahoo.com/xrds
If you are curious enough, you can go to this url and download so-called "yadis" file, which is xml declaration of supported services and options.

Funny.
From RFC-2631: "In ASN.1, EXPLICIT tagging is implicit unless IMPLICIT is explicitly specified" :)

Gnucash import script

I decided to track my finances with Gnucash. My bank allows downloading statements in qif format which can be imported by Gnucash.
But here is the problem: during the import the only way to classify your transactions (was it car maintenance, rent, or dinning?) is to manually assign every imported transaction to the appropriate account. If I'm trying to analyze my last year expenses, then it becomes virtually impossible to do it manually.
So I came up with a simple ruby script:

#!/usr/bin/ruby

$patterns = [
 [/^PSHELL OIL/, 'Expenses:Auto:Gas'],
 [/^PCHEVRON/, 'Expenses:Auto:Gas'],
 [/^PTHE BOXING CLUB/, 'Expenses:Entertainment:Recreation'],
 [/^PEDWARDS MIRA MESA STDM/, 'Expenses:Entertainment:Music/Movies'],
 [/^PULTRASTAR CINEMAS/, 'Expenses:Entertainment:Music/Movies'],
 [/^PHOBBYTOWN USA/, 'Expenses:Hobbies'],
]

def setCategory(tx) 
 tx.each do |line|
  $patterns.each do |pattern|
   rx = pattern[0]
   if rx.match(line)
    l = pattern[1]
    tx << "L#{l}"
    return
   end
  end #patterns
 end #line
end

out = File.new('out.qif', 'w')
tx = []
txCount = 0
missed = 0;
while gets 
    if /^\^/.match($_)
 txCount = txCount+1
 len = tx.length
 setCategory(tx)
 if(len == tx.length)
  puts tx
  puts "-------------------"
  missed = missed +1
 end
 out.puts tx
 out.puts '^'
 tx = []
    else
 tx << $_
    end
end
puts "Transactions: #{txCount} Missed: #{missed}"

This scripts reads stdin, and modifies transactions by adding description to it, according to set of regular expressions. I gave a subset of regular expressions, you will have to modify them to match your codes and your Gnucash account names.

The trick is to add "L" code line to each transaction description. The qif file format can be found here. We need to add "L" line which would match to a desirable account name in your Gnucash setup.

In order to make the task easier, the script outputs unrecognized transactions to stdout. You can look at it, and add appropriate filters until count of unrecognized transactions become minor.

Google Jabber server

I'm working on an appender for log4net to send messages to IM client. Jabber has been chosen as a standard, robust and extensible. Quick googling shows that there are not that many client libraries, and I took a look at Jabber.NET library.
I tried to connect to server "talk.google.com" but upont TLS initiation I received "CN mismatch error". Some furher debugging indicated that talk.google.com uses SSL certificate issued to "gmail.com" domain. Interesting enough, my Miranda client had no problems connecting.
Which makes me thinking: when TLS is in progress, and connect host is different then server name, which name (CN) must be matched, the physical host "talk.google.com" or "logical" host "gmail.com" because my user name is "my.user.name@gmail.com"? Probably it should be logical server name, because potentially you can connect to the same chat server through different entry point server.

Firestorm
Here what it looked like in Encinitas on Oct 22 2007

I have some thoughts, I'd like to share.
First of all, incredible synergy of community. It is one of those moments when everybody feels that he or she is a part of single entity. You just need to be here to feel it. Hmm, as I wrote it, I realized that it is a bad advise :) The information flow is incredibly dense. All major radio stations cancel their routine broadcast and dedicated to passing information 24/7. And it is not a chase for rating. It is sincere desire to be helpful and realization of responsibility as one of the major source of information.
Officials act very and very organized. Firefighters, police, city - every related organization act synchronously.
People: leave their places organized and don't wait until the last moment. This allows firefighters to concentrate on... well, actually fire, instead of saving those trapped in fire.

On this background TV looks rather disappointing. Being much more capable of passing information, TV somehow manages to be much less useful then radio. May be the model of producing pseudo news, when the art of reporting is substituted with choosing a dramatic background and reading a text with tragic voice shows it weakness when it comes to something real. I'm sorry, if I offend some people who work hard for TV, but that's what I see. The end result counts.

It took me some time to find some useful and actual information in internet.
Here is a news aggregation.

Here is a google map with fire areas, shelters, roads avialability and other useful information.
Huge thanks to KPBS crew. You guys really make difference.