40 miles

The South Downs PLOD

OK, so I've just completed the South Downs PLOD, before I started most people who I Asked for donations were unimpressed as "it's only a walk" and to be honest I approached it thinking much the same thing. Now I've done it I think I'd be likely to punch anyone who said it's just a walk.

I think one thing I overlooked was that unlike running I'd be relying on a different set of muscles, I'd be carrying kit that would add to my weight and because of the length of time it takes to get anywhere there wouldn't be much time for sitting around or indeed any places to sit either because if you sit on the floor it's very difficult to get up.

Some people would say "it's not like you're running" and the silly thing is that I'd prefer it if I did run. There was a stage probably around 20-25 miles in where my feet and tendons at a point just behind my knees were screaming in agony and for a laugh I ran up a steep slope, to my amazement this actually made my legs feel better than they had done. Later on in fact between the last and last but one checkpoint I used this trick up some very steep slopes between the seven sisters and remarkably I was able to get to the tops and catch my breath within a matter of seconds so for me running would be much easier than walking, or at least part and part.

One thing I would say is that staying awake for 35 hours or so, some team members were approaching 42 hours without sleep and walking for 16+ hours really does bring out communal team spirit. Not only did I have a smashing team giving each other support along the way but also a wonderful support team who greeted us at every checkpoint witha well stocked tuck shop in the back of his car and words of support, as well as some support that weren't part of the team but gave us all that little boost we needed when we needed it most with cheers and smiles.

Walking through the night was wonderful, even when we missed a turning and walked a mile or so further than we needed to.

It was a shame to see a number of teams take shortcuts, the most annoying was seeing a team a good 10 minutes behind us suddenly appear in front as they'd taken a shortcut into Alfriston :( still we maintained our integrity by not cheating when we could have done.

I must admit that I actually started to enjoy the very last bit and think I'd have liked to run the last 2 or 3 checkpoints. However after running a few hiills I suddenly realised I was becoming the jerk that abandons his team for personal glory, this is not who I want to be so hung back to join the others. I am glad I did too as one of the team was in desperate need of water and I had some spare that I was more than happy to give up, see that's what winning is not crossing the line first but crossing it together!! A lesson well learned.

Would I do this again...probably not, or at least not for a long time. I think there is a very different style and fitness required for walking rather than running so until I stop training for running events I don't think I'll do any further walking ones.

One further thing to say was that the organisers were smashing, I regret that I had to leave before being able to use my £6 luncheon voucher at the pub but never mind the team managed to collect over £1500 in donations so well done to us all.

Gradle Review

Gradle

Gradle is a build tool, I personally use it to build some old java projects but it can do other things.

I've given it a chance now so feel I can blog about it.

I like, no I love the idea of Gradle, however the implementation isn't good enough :(

It's not that it lacks features, quite the opposite there are so many that the basic set have been neglected in favour of the more fun stuff.

Essentially there needs to be another layer to add simplicity, you need to know too much about the internal workings of the system for it to be all that useful to build systems that already exist.

It probably works well on basic systems, and if starting a new system you can adhere to all the rules it'd probably rock, but for making existing builds better it becomes an uphill struggle where the benefits are sadly overshadowed by the difficulties in getting there.

If there was more IDE support then it would probably be far easier and I would be writing a totally different blog but until it is my opinion stands.

Specifics on whats wrong

I have a legacy system that I am trying to move away from using Ant, as is always the case the build has become incredibly over-complex which means an incredibly verbose xml build file and many workarounds for the immutable nature of ant.

I like the idea of being able to use actual proper code in the build when some conditional stuff is required, ant wholey fails on this count by forcing you to make custom tasks e.t.c. for very obscure conditions rather than allowing quick in line code.

I create a set of jar files from a particular project, I am using the 'java' plugin which gives me some handy stuff, but goes too far!!  It gives me a default jar of all of my code which I don't want and using what seems obvious to switch it off 'jar.enabled =false' seems to have no affect.  The suggested alternative was
configurations.runtime.with{
  artifacts.remove artifacts.find { it.archiveTask.is jar }
}
YUK! this is forcing me to know about the internal workings of the thing I want to use, which is wrong.  I don't want to spend ages learning about a damn build tool I just want my source to go in one end of a black box and have my compiled jars pop out the other end.  I mean I use my car and tv to perform the tasks I want adequately yet I don't need to know what's going on inside in order to do so.  To me this is where the design of the api starts to fall down.

I used another plugin 'ear' which helpfully created a packaged ear file, I needed an exploded one but there seems to be no simple way to stop the ear being zipped up, yes I could add a doLast() to unzip the ear but come on seriously why would I waste time zipping to begin with!

UP-TO-DATE
The idea of an up to date check sounded like the best thing ever, you have a task tell it what the inputs will be, what the outputs will be and when there has been a change the task is run, yet if the system hasn't changed the task isn't run thus saving tons of unneeded build work.


Yes it does sound like a good idea but it simply doesn't work.  I currently have a jar that is part of an ivy repository that is downloaded and unzipped.  Therefore the task to do this has the zip as an input and the output dir as an output.  This task never runs as it is ALWAYS up-to-date, I've created the output dir and put stuff in it, I've altered the source zip, I've even added an upToDateWhen closure and this is never triggered Gradle simply responds that the task is up-to-date.

More overengineering

Another thing that simply wound me up about Gradle but I was prepared to deal with on the assumption that the rest of the sales rhetoric was at least vaguely truthful was the dependency mechanism.  There are already tried and tested dependency mechanisms, my project used to use Ivy so why can't Gradle just use Ivy?  As far as I have discovered there is No benefit of gradle dependecy management over ivy!

Conclusion

I really like the idea of Gradle and when I went through the typical hello, world tutorials it all seemed nice.  It's only when you want to deviate from the path of what the originator of Gradle deemed to be the route to take that you realise just how hard it is to use.

There is a lot of documentation, there is a host of helpful people (and unhelpful) that are willing to offer assistance.  This is all brilliant and I commend those that are involved with it.  This article does sound like a bit of a downer on Gradle which I don't want it to be because I believe it'll be fantastic one day, however it isn't yet and it would appear that more features are being added rather than sorting out the current failings which could end up turning Gradle into unmanageable bloatware like so many tools (maven) that have come before it.

For me the ideal build tool would be nothing like what has gone before, no fancy convention over configuration none of that just a set of libraries for dealing with typical build-like tasks (file copying, dependency resolution, compilation...) and let whomever needs the build use it however they want rather than attempt to steer them in what I believe to be the right direction because with legacy there is no standard way these things have been done and the time to convert them cannot be justified, it's just a build tool!

Random Thoughts on deadlines

I've worked in several places where deadlines are tight, who hasn't. Management can help, but they can also hinder so I thought I'd sketch out my ideas of what helps the most and what destroys any chance you may have had in producing a deliverable.

A quick word on methodologies, Agile is great it is fantastic to never lose sight of the actual goal and having weekly sprints whereby a result is actually seen by the major project stakeholder is ideal...however there is a tendency for management types to see a product with the bits they liked that week and can't make the intuitive leap that although those features work the product as a whole is incomplete and so it gets sold.

Also there does have to be buy-in from the ground level all the way up when going agile, if the guys up top are still requesting requirement documentation and things they've missed the point and many of the benefits of agile will be lost.

So back to the issue of deadlines, number 1 is where the deadlines come from, if it's a sale date then how was it decided. More than likely it has been decided by a salesman whom has no clear knowledge of what the solution entails let alone how much work is involved or the resources that will be available, with so many unknowns it's no wonder that the deadlines are incredibly unrealistic.

A clever manager when faced with a, lets say it, stupid deadline, will gather his troops garner opinion from the experienced developers that will after all be sweating buckets to meet the deadline, then and only then will they be able to judge the loading of the team members and produce a realistic, yet usually very tight deadline. If an adjusted deadline from someone with more knowledge is ignored you can pretty safely say it won't be met. Not only will the deadline not be met but because of it's unrealistic nature the psychology of the developers will reach a point of "it's not possible so why bother" this causes projects to actually go slower than if the deadline had been set more reasonably to begin with.

The Carrot or the stick.

Software developers are a lucky breed in that their skills are 100% transferable this means when pissed off they simply jump ship. Too much pressure they jump, not enough money they jump, no respect they jump and bored they jump. Keep the good guys happy and a good supportive team is created. If developers want to do well then they will, if they feel forced or coerced into doing something they feel less inclined to do they won't do well and once again are likely to jump ship.

The same goes for managers to be honest, barking orders makes developers dig their heels in, yet if a developer doesn't want to let the manager down they work harder, this level of respect isn't automatic nor is it bought with bonuses it is developed over time and yes it takes work. Shouting or berating the very people that are doing the actual work is never going to work, trying to pile on more pressure is also never going to work. "what you mean the developers won't work harder for this crust of bread!" it is sad that even to this day there is a falsehood touted that people work harder when they feel their jobs are under threat, that just makes them dislike the job and guess what they leave!

Now the carrot, I've seen bonuses work in the past, usually bonuses on a sliding scale make the damn near impossible to reach deadline have a high bonus then give a far smaller one for meeting the previously thought of unattainable one. Lets face it salesmen get bonuses for landing big sales that sometimes are a result of them enjoying a visit to the pub with the right people. I am not saying sales isn't hard and that the tough ones should be rewarded but they all agree some accounts are easier than others, to the point of not being work at all.

Now the reason that neither of these should be used anyway is that once the stick is used developers bugger off and once the carrot is used too often developers come to expect it and when they don't get it they bugger off! Plus there is of course the product and the customer to think about. At the end of the day the only way an unrealistic deadline is going to be met is if corners are cut. The first thing to go will be the documentation, then the level of testing then the standard of features, then the number of features until eventually a pile of crap the customers didn't want is given to them they complain and either the whole lot needs to be re-worked thus taking us to a point beyond an original realistic estimate or the customer throws toys from pram refuses to pay and maybe sues. This is scary but true.

Estimates

It all starts to go wrong from the moment an uninformed guesstimate of a deadline is given and the recipient believes it.

So how are the estimates made and how can they be so wrong?

It's all to do with the number of unknowns, or variables when estimating, the simpler the item being created the fewer variables and hence the more accurate the estimate can be. Most large sales have countless variables, not only that but they also have unexplored avenues. What I mean by this is that if nothing similar has been created by the team there will be a certain amount of trial and error with their solutions this is perfectly valid and by taking the software through this sort of evolution means that the best solution should be arrived at. The trouble is you don't know that one avenue of development is necessarily better than another for your particular problem given that you've never seen anything similar and the number of avenues can almost exponentially increase the number of unknowns of development.

To explain lets take a chess game as an example, the solution we seek is to checkmate the opponent we start with 20 possible moves at the beginning, that's 2 for every pawn and 2 for each knight. Depending on which move or avenue we investigate next determines the number of possibilities we have next and at this point we're not sure that the one we take is going to help us reach our goal we take an educated guess but we may well need to backtrack to this point before a solution is achieved.

Laptop wishlist

I want to replace my laptop as it's getting a bit long in the tooth so I've been looking at the specs for these things. I find it hard to believe that most laptops are sold with a naff resolution 1366x768, seriously!! This is the same resolution as my budget lappy bought in 2008 so why haven't things moved on? Asus seem to be the only company making 1080p standard, and you can stop pretending that 900p is full hd, you know who you are (Sony).

I seriously considered apple since they have a nice screen and keyboard...however, I kid you not a similar spec machine all apart from the screen res costs around £1500 let me say that again £1500 and that's pounds sterling not dollars, more for apple, for that money I want a diamond coated screen made from crystalised unicorn tears sheeesh how do you justify that markup? Still I guess some people have more money than sense since this basically says they value a screen at £1500 I wonder how many pounds per pixel that is?

Backlit keyboards!?!? Why FFS are they not standard? OK this is something that Sony do get right, in fact I am not going to be too harsh on Sony since although they lie about full hd the laptops do look nice and are only slightly more expensive than laptops that don't have backlit keyboards. Whilst on the subject, why does the UK seem to get the crappier models of laptops? There is a host of ASUS N56 that do have backlit keyboards yet none are available in the UK, unless that is your writing a review for the company in which case the better models are supplied.

Sooooo many lappies have spinning rust drives? What's the point, especially what's the point with all these 5400rpm drives, if it's power that's the concern then kit it out with an SSD they're really not that expensive anymore.

Another thing, for gods sake why oh why is windows 8 shipping with non-touchscreen laptops, it's a bugger to use unless your touching the screen which is after all what it's designed around.

This ones a kille too, why does every 15.6" laptop have a numeric keypad? It makes the touch pad off-centre (that's off center if your American) and simply gets in the way, my old Acer has one and I assumed it was a gimmick of a cheap brand but it isn't. So for the sake of a few number buttons that I use very infrequently I have to put up with getting rsi and a bent spine!

Do laptop manufacturers ever actually ask the public what features they like, because there are plenty of people that feel the same as I do.

So basically what I want is a 15.6" laptop that has an optical drive, 1080p screen, plenty of processing power i7 (although i5 would probably fit the bill), plenty of ram since I do like VMs ~12-16Gb, a backlit keyboard that doesn't have a naff numeric keypad with a 250Gb+ SSD, at least 1 usb3 port all in a case that doesn't flex when you pick it up by a corner and not costing more than about £800.

Is sexism dead? Not quite but it's not all bad

today is international womens day, some women will get time off work, gifts e.t.c. Why?  When is international mans day?

Fighting for equality means fighting for men and women to be treated the same, yes women have had it hard in the past but this doesn't mean that the scales should be tipped in their favour it should be balanced.

Looked at the international womens day thing

http://www.independent.co.uk/life-style/gadgets-and-tech/features/international-womens-day-google-doodle--and-tweeting-for-equality-8525269.html

It starts badly

“Today's Google Doodle is a montage of chubby female cartoon faces of all ethnicities”

a)      They’re not chubby

b)      Why would that be the first thing you’d notice

“”Only 17% of the UK tech workforce is female”

Which is not 100% related to sexism there are strong physiological reasons why men suit logical/technical roles and women suit care roles.  In fact I saw a piece on “embarrassing bodies” of all things whereby women could tell facial expressions more than men but men could perform logical tasks better.

I am just mildly upset that lack of equality is being promoted in its name and lack of thought is being applied to the articles.  I want equality, women should be paid the same for the same job, however there is also a class issue too.  What I mean is people from rich families are unfairly remunerated it happens yet there is no international poor persons day, I get paid less than other people who perform the same job, we should be equal but we're not.

Burrows-Wheeler transform (BWT)

 Burrows Wheeler Transform

I would ordinarily explain what the BWT is but to be honest I wouldn't be able to put it into better words than the article on wikipaedia so I am not going to try. So what then is this blog post about? Well simply I am reacquainting myself with Groovy after a long time of not doing any and decided to see what a simple (and I mean really simple) implementation would look like.

One of the major differences as far as I can see is that I've not included an EOF indicator/marker in the resultant string, instead I've indicated the row that the original data will exist on once the data is reconstituted from the encoded data.

First of all take the test string and rotate it storing each rotation:
def list = [args[0]]
(1..<args[0].size()).each { list.add( args[0][it..-1] + args[0][0..it-1] ) }

Saving this in a file called BurrowsWheeler.groovy typing this on the command line
groovy BurrowsWheeler.groovy "this is a test"
produces...absolutley nothing, however if we add list.each { println it } it all becomes far clearer with;

this is a test
his is a testt
is is a testth
s is a testthi
is a testthis
is a testthis
s a testthis i
a testthis is
a testthis is
testthis is a
testthis is a
estthis is a t
stthis is a te
tthis is a tes

The next thing the algorithm needs is to have these rows sorted, well that's simple try list.sort() and since we only have a bunch of text the default Comparator is fine, so that's stage two complete, add a  list.each { println it } to be sure.  A note on this simply printing the list will show everything that's in it, however it looks better when displayed using this closure method.


a testthis is
is a testthis
testthis is a
a testthis is
estthis is a t
his is a testt
is a testthis
is is a testth
s a testthis i
s is a testthi
stthis is a te
testthis is a
this is a test
tthis is a tes


So now on to the final stage of the compression, get the last column and the row number that the original data exists on.

def lastline = ""
def num
(0..<list[0].size()).each { lastline += list[it][-1]; (list[it]==args[0]) ? num = it : num }

I am sure there is  a simpler way to do this , just as I was sure that I could skip the sort stage by having a SortedList that would ensure the contents were sorted as items were added, but I lost patience looking for it :) 

Add a quick println "last column: ${lastline}, match num: ${num}" and we get;
last column: ssa tt hiie ts, match num: 12

As you can plainly see no compression has yet taken place, in fact by adding the row number we've increased the amount of data we're storing.  The point in the BWT is that we've now stacked far more of the duplicated characters together and can therefore use some run length encoding to compress it.

Uncompress / Reconstitute data

What good is making this data if we can't get back the original data, well it's as simple as taking the process in reverse.  So starting with the string  ssa tt hiie ts and row number 12.

def lastline =  ssa tt hiie ts
def anotherlist = []
(0..<lastline.size()).each {
   (0.. < lastline.size()).each { anotherlist[it] = lastline[it] + (anotherlist[it]?:'') }
   anotherlist.sort()
}

Printing this out gives;
0      a testthis is
1      is a testthis
2      testthis is a
3     a testthis is
4     estthis is a t
5     his is a testt
6     is a testthis
7     is is a testth
8     s a testthis i
9     s is a testthi
10    stthis is a te
11    testthis is a
12    this is a test
13    tthis is a tes

And sure enough our original data is right there on row 12.  This is by no means meant to be an industrial bwt it's simply just an example of educational groovy, hope you enjoy it.

Databases deserve a bit of respect

I've recently had the pleasure of being on a database administration course in Euston held at Learning tree, which by the way is still my favoured course provider since the quality of the presenters seems pretty high.

Anyway, the reason that I felt I had to make this post is that as a developer I always think about the code, what objects represent what aspects of the domain I am solving a problem for. I think about system performance, ease of maintenance, how readable the code is e.t.c. however since I almost always use an object relational framework of some kind not only am I mostly shielded from the way in which queries are constructed but I am almost totally separated from the actual fundamentals of the database itself.

One thing that a lot of care used to be given to was the design of a database, but now with very fast machines hard drives and all that stuff we can pretty much get away with slapping a load of tables together and let the database server handle it, and for the most part it does this very well and only in fringe cases does it require special attention but I think we/I should get back to basics from time to time and realise that the database behaves ok but could be spectacular thus relieving the rest of the system from quite so many performance enhancements that are there because unbeknown to me the database is underperforming.

The DBA on the course let slip a very simple sentence that interested me "when doing bulk imports in MSSQL Server you can greatly improve performance by switching to bulk_logging." This whole area spawn thoughts about the difference between log files and datafiles and how indexes are created and used and what the difference between an extent and a page is and how fragmentation can occur and be fixed. All these things are simple common sense to someone versed in databases but other than for academic purposes I've never had to look deeper than the surface, so think no is about time I did before I forget it all.

I have a dream, that one day all developers and database administrators will be treated equally.

Naming conventions that suck

I've been using C# for while now but I've spent most of my career to date using java in enterprise development. The naming conventions for java, the official ones from Sun that is, seem to be mostly logical and there is a well defined getter/setter access naming for beans that fits nicely I especially like that boolean member variables have a getter preceded by 'is' because it makes sense.

There is, or at least doesn't appear to be the same level of common sense with C# instead there seems to be a clinging on to the past. At least most people no longer put sz at the front of zero terminated strings or i in front of integers like back in the C/C++ days which lets face it were quite some time ago and things have moved on so get over it.

When I see code where interfaces are preceded by 'I' or member variables that are prefixed with '_' it makes the code look childish, it's what I'd expect an old school programmer/hacker from yester-year who has refused to expand their own understanding would do.

In the days of C the naming conventions were there because there were no clever IDE's so the quickest way to convey to the user what a variable was was to include something in the name which were generally kept short (I don't know but there may have been a limit on variable name length). The fact the conventions such as 'I' for interfaces and '_' for member variables is retained in a language that lets you define a variable as 'var' and let the compiler infer the type smacks of hypocrisy and those that follow it are simply refusing to grow beyond their limit understanding.

I for one am looking forward to the day that variable names contain spaces as the norm like they can in Groovy because it makes sense to be able to convey simply what the purpose for a variable is as it no longer matters how many characters are used in the name so be expressive.

A strange convention that I started using then found to be wonderful for allowing me to scan read code was to put a space after the opening parenthesis and a space before the closing parenthesis, it sounds odd but the little box that the parameters are now in really can be easy to spot whereas you can't quickly scan down code where there is no space because it seems harder for the brain (mine at least) to see the fact that the parameters are separate from the method name.

Just remembered another stupid convention used in C#, capital letters for methods and capital letters for properties. This seems like an opportunity missed since methods are usually verbs they'd fit nicely in the middle of a sentence whereas properties tend to be concepts or nouns that are more likely to be names therefore why not lowercase for methods and upper case for properties. Ok, so it's not that stupid I just prefer lower case methods as capital letters tend to stop the flow of my brain due to decades of teaching it that a full stop followed by a capital letter signifies the end of a section that the author deemed worthy of signifying.

Machines that learn

I wanted to design a machine that learns, always have ever since I saw wargames with Matthew Broderick. Trouble is it ain't easy. Humans are very good at seeing patterns and I wondered if this has anything to do with the ability to learn. Maybe if we see patterns we're able to apply previous experience to similar situations. On another note I was thinking that a tree of decisions where each node is itself a tree creating thing. This would tie in with the idea of CNN where there is a bias feedback loop between nodes. Ok so I am starting to like the idea that each node in a tree is itself a tree creating thinking machine, I like the idea that all this tree has knowledge of is what ever is below it i.e. its children and its childrens children e.t.c.

What to do?

I've got a load of ideas buzzing round my head at the moment so thought I'd slap them down on here so I'll be forced to actually see it through!

1. I have a gps unit that works with the mtkbabel program, however it also has bluetooth connectivity that I've never used before. My plan is to write some j2me app that can use the bluetooth and this unit to give gps to my really old mobile phone.
2. I want to make a javascript based drawing application that produces svg graphics, which I am only starting to play with but finding really cool.
3. combine 1+2 so that the svg is a scaled image that the gps coordinates can be plotted on.
4. make a good window switcher for windows 7 because all of the ones I've found so far suck donkey's balls.
5. finish making my algebraic compression algorithm combining the fact the a rubicks cube can be put into millions (if not billions) of positions but you can get from any one to another with just 23 moves. Fractals can make infinitely complex patterns from relatively simple rules. the path from one piece of data to another can be plotted by some mathematical formula, I think genetic algorithms could be used to discover the best fit formula.
6. I think I could take gpx and transform it to svg via some clever xslt, that'd be swish.
7. fed up with there being no 100% standards compliant browser I've decided to make my own, however I've decided to make the entire thing a cluster of modules that can all be pulled out and replaced. This has a good side effect in that I want to describe the outcome of javascript programs as simple operations to be performed on the browser, these can be described as a series of simplistic commands much like idl or bytecode. This in turn means that no one is stuck with using javascript if they don't want to they can make a module that interprets whatever language they want into idl. Since the language will only effect the browser the sandbox will be the browser, plus a web page that uses a module that the user doesn't have can actually supply the module to the user and it will be automatically installed just for the browser. The other thing is that I like the pipeline model of IIS (and the chaining of command line apps in *Nix based machines) so I want the entire thing to be a cool little pipeline where each module effects the document as it flows through the pipe and is presented to the user :)
8. make a world simulator, objects with gravity, collisions e.t.c. all with a view to simulating a clock mechanism and then make the mechanism a reality.

There are other things but these are the first few I can think of and want to get cracking on.

Tree Searching

It has been a very long time since I've written anything on here, what with having a baby daughter to look after and spending most lunchtimes keeping fat at bay in the gym (yes I am aware it's 80% diet, trouble is I don't eat enough, post for another time.)

Anyway I have 10 minutes before the end of my lunch break and looking round my desk can see an interesting note on tree searching.

imagine you have a tree of

                  a
                 / \
                b   c
               /\   /\
              d e  f  g
             /\ /\ /\ /\
            h i jk lm n o

I know this diagram is messy but I have 10 minutes and don't want to leave the text editor yet.

Now image the way in which this structure is expressed is as a map for example

a: b, c

would mean a has children b and c, you could then have an algorithm that would expand the first (left most) item in a list and depending on whether the expansion is stored to the left or right would determine if we had a depth first or bredth first search. I remember when I first saw this idea back in the 90's when playing with Haskell (Hugs98 if anyone's interested) and thought it was cool.

lets expand and store at the front, this would give a depth first search, ooh one other thing to note, leaf nodes are sent to the back since there is nothing to expand on.

1. a
2. bc
3. dec
4. hiec
5. iech
6. echi
7. jkchi
8. kchij
9. chijk
10. fghijk
11. lmghijk
12. mghijkl
13. ghijklm
14. nohijklm
15. ohijklmn
16. hijklmno

now if we expand the nodes but store them at the back of the list

1. a
2. bc
3. cde
4. defg
5. efghi
6. fghijk
7. ghijklm
8. hijklmno

no more expansions as we've reached the depth, so in some pseudo code it could be

expand([x:xs]) => expand[x] : expand[xs]

expand([x:xs]) => expand[xs] : expand[x]

it looks so simple :)

anyway back to work.

Some cyclists are dickheads

A cyclist tried to play chicken with me, who was in a car, I kid you not! This is the sort of cyclist that gives the rest of us a bad name, here's how it played out. in a 40mph limit I am behind a van doing ~25-30mph obviously looking for an address or something, I see that in the distance on the other side of the road there is a cyclist but that is all for what looks like miles. I move out around the van approaching 40mph at which point I notice the cyclist is now moving towards the centre of the road, bit odd since he hasn't signaled that he's turning right any way I complete the overtake and move back over to the left and on the road in front of me the cyclist is shouting and pointing at his head like I did something wrong! I ride my bike a lot and I think it's because of arrogant pricks like this that some motorists cut me up, it's a sort of do before it's done to you response because they expect EVERY cyclist to do something crazy. What an arsehole!!

Wandering mind

The human body is amazing, it tries to keep all parts operating optimally.  If one particular part is stretched to its limit the body will try to compensate in preparation for the future.
I have no real evidence from this other than observing people that go to the gym.  What I mean is someone who goes to the gym and stretches their arms to the limit for example by lifting heavy weights will generally see an increase in the size of their arms the more they do it.  I am starting to think that the same may be true of the brain, the more you use it the better it gets.
I take this one step further by saying that because I work a very sedentary job I have to exercise my muscles outside of work, and when I was a youngster working in a factory I used to have to exercise my brain outside of work by learning languages e.t.c.

I think this is a good practice and think that maybe not enough people do this kind of exercise, just a thought.

Are we in a democracy?

Wikipedia defines democracy as;
Democracy is an egalitarian form of government in which all the citizens of a nation together determine public policy, the laws and the actions of their state, requiring that all citizens (meeting certain qualifications) have an equal opportunity to express their opinion.

This got me thinking about whether I live in a democratic country and according to this definition the answer has to be no.  I do not have an equal say in what happens, even the person that is supposed to represent me and my best interests isn't the person I voted to represent me, nor could I guarantee that they would have my best interests at heart.

Here's why
The things that are important to me and matter greatly to me don't effect most of the people who work in Westminster.  Most have never had the joy of getting electricity via a card that has to be topped up like a pay-as-you-go mobile, or had to make the choice between a new pair of shoes or filling the car with fuel.

I guess what I am trying to say is that the people in charge in Westminster have a different agenda, this is why there are new laws past to protect the already rich with lower inheritance tax and stamp duty e.t.c. and yet they can vote to close libraries and hospitals and post offices because there is another one only 10 miles up the road.  This is fine if you drive and can pay for the fuel as the politicians obviously can but for those that can't it may as well be a million miles up the road.

I recently read an interesting article in Dodgem Logic#2 (I know there are now 8 issues I am trying to catch up) that suggested a random selection of the real public should be chosen to govern for a limited time only before swapping, the reasoning is that the decision making politicians would be more inclined to make decisions that are beneficial to the general public as they would soon be rejoining the general public.  The only bad point in the article was that it held up Greece as the shining light of this principle and frankly the Greek government balls the whole lot up!  The article did raise the point that anarchy isn't a bad word, it simply means without leaders, i.e. everyone is seen as an equal.  This sounds a bit like the foundations of Marxism/communism, I know they are different but no one has succeeded in explaining the subtle differences to me yet so I view them as synonymous with each other and anyone who has read Animal Farm (George Orwell, not the porno) will immediately think "...but some are more equal than others".  There in lies the rub, it takes a great deal of self control to not want to post yourself as the leader, is it human nature, can it be avoided or are we doomed to always be a people led rather than equals?

My Utopia
My idea of a utopian land would be one in which everyone was free to think/do/say and learn whatever it is they wish to as long as it didn't interfere with someone else.  I think this was the general idea behind 1930's pursuit of happiness but I am not sure.  Anyway it would be nice if the goal of every human was to better themselves for humanity rather than the acquisition of wealth or power (vaguely remember Capt. Picard saying something similar).  It does sound a little far fetched at the moment, but doesn't it also sound good, and something worth aspiring to.  I sure as hell think so which is why I view a day in which I haven't learnt something as a day to regret and I don't want to live a life full of regret, who does?

An Idea I am toying with
While I wait for some photos to upload to Google+ I thought I'd jot down an idea that I've been playing with for some time now.  But first, don't you think it's great that Google+ lets you upload full resolution pictures, it makes for a wonderful offsite backup facility!

Back to my idea;
For a long time I have worked with people that all have their own views on the way in which code should be formatted, these formatting styles only effect the way the code is viewed and have no effect on the way the code functions because white space is ignored in the languages I have used in industry unlike python which is undoubtedly a good language IMO.
The way I decided to solve this problem was to store metadata along with the code, basically a program sits between the developers IDE and cvs that alters the code the developer writes and stores it in cvs as XML the view of the code is then determined via some xslt that the developer has decided upon.

I extrapolated this idea along the inevitable evolution and decided that by being able to apply metadata to classes, methods even code blocks, if cases e.t.c. I could get the vm that was going to run the code to use this extra information to decide how to priorities the calculations in order to make concurrent programming far simpler.


I'll keep you posted on how it goes this entry was just so that I am forced to make a start on an idea I've had for a good few years now!!! How time gets away from you and these ideas never come to fruition, just like the energy generating compost heap idea I had when I was 12, I still hold out hope for that some day :)

Single Sign on

Single Sign on
This post is a bit of an experiment, rather than like most of my posts where I come up with an idea and document it once complete I am going to blog about my progress getting this to work as I go, a sort of impressionist blog, which is most likely the way they were originally intended to be done.

What is SSO?
The way I see single sign on, which may not be the right way to view it, is that a user in a company has to interact with a multitude of computer systems, each of which requires some form of authentication before access is granted and single sign on means that a user once authenticated in one system will auto-magically be authenticated in the others without being prompted for a username and password or whatever.

The first method that I am looking at is Kerberos, the way I think this works is that a user signs in using whatever means at their disposal (card reader, password or biometric scanner) then that user has a token that is given to any other systems that require authentication.
The premise here is that there is a single secure token provider that every application within the enterprise/corporation then trusts.

JAAS the java way
Being a Java developer on enterprise systems I would have been lucky to have not had to have some experience with using JAAS the java authentication authorisation service.  This is a pluggable system like much of the JEE components where any particular vendor can make their own implementation, and as luck would have it there is already a Kerberos version available.

JBoss
I am going to use JBoss (likely version 4 but maybe 7 as I kinda like where that's going) as my application server, I may show using GlassFish too, plus I may also show it being used in tomcat just for kicks.

Step 1
In order to actually test these things out I need to actually have a kerberos system setup, as I tend to use Linux the instructions for setting this up will be linux orientated.

I followed these instructions for installing and setting up a kerberos system.
I rapidly hit a stumbling block with the first pre-requisite of the installation

Before installing the Kerberos server a properly configured DNS server is needed for your domain

D'oh! I don't have a DNS server, so I went to get one and set it up.  Doing a quick google I found these instructions which are for Ubuntu and as I am using a Debian system thought it would work.  I didn't bother following any of the reverse DNS lookup instructions as (a) I am lazy and (b) I want to move on quickly from setting up a dns.  The only thing to note at this point is that nslookup gw didn't give any results.

Back to installing Kerberos
So now that dns seems to be installed time to go back to the kerberos instructions.

Everytime I try  kinit steve/admin I get

kinit: Cannot resolve network address for KDC in realm "RLJASSOCIATES" while getting initial credentials

I altered the instructions for both dns installation and kerberos installation so that the domain is RLJASSOCIATES.
I found a forum that states;

"each host's IP address must reverse-resolve the canonical name."

so I guess it's time to go back to the DNS setup instructions once more, damn it serves me right for being lazy in the first place!

Ok the DNS setup doesn't work right off the bat but I did find a nice little utility to test called named-checkzone which I can use to check the zone files for correctness.  Needless to say they weren't.  Well I've been over those DNS instructions many times now so I can be absolutely sure my setup matches and guess what...they don't work so I guess they're crap!

I think there might have been a missing entry in the RLJASSOCIATES.db file so I added these lines
ns        IN        A        192.168.1.34
@        IN        NS        ns.RLJASSOCIATES
and that seems to allow nslookup gw to work.

however running sudo kinit steve/admin still results in the error

kinit: Cannot resolve network address for KDC in realm "RLJASSOCIATES" while getting initial credentials

Setting up krb5.conf
I remembered that after the first fault with the kerberos setup I uninstalled it, then when I re-installed it I was never asked the same questions about default domains once the installation was complete, this may have been the route cause of this last failure.
Essentially there is a file called /etc/krb5.conf that seems to list settings that are relevant for different domains, so I took a guess at what to write and put this in;

    RLJASSOCIATES = {
                kdc = rjohnson-acer.RLJASSOCIATES
                admin_server = rjohnson-acer.RLJASSOCIATES
                default_domain = RLJASSOCIATES
    }

I have no idea if this is correct or not apart from when I ran this command kinit steve/admin instead of it failing with the usual error message shown above it prompted for the password, which I typed in and ... nothing happened which still means no error :)

The next fault to get over in the kerberos instructions is

#> kinit steve@RLJASSOCIATES
kinit: Client not found in Kerberos database while getting initial credentials

 Once again I have no idea what is causing this to happen, so on to professor google.
Not actually sure this is an error as the list of principals is now
kadmin:  list_principals

K/M@RLJASSOCIATES
kadmin/admin@RLJASSOCIATES
kadmin/changepw@RLJASSOCIATES
kadmin/history@RLJASSOCIATES
kadmin/rjohnson-acer@RLJASSOCIATES
krbtgt/RLJASSOCIATES@RLJASSOCIATES
steve/admin@RLJASSOCIATES

The big issue is that it says I am not entering the correct password for kadmin/admin and yet I know damn well what I entered so this might be an issue with the re-installation I tried earlier ooops!

JBoss
I am going to start with a vanilla installation of jboss-4.2.3.GA, simply because I wanted to start with a version 4 and this was the latest 4 I found on jboss community site, don't worry I'll show the same kerberos stuff on jboss-7.0.2, I would want to use 7.1.0 as this is the version where remote ejb calls are supported but it doesn't exist yet tee hee :)

Since JBoss comes with a jmx-console lets try to secure the access to this web app with kerberos.

The current setup for security on this app is;

   <application-policy name="jmx-console">
      <authentication>
          <login-module code="org.jboss.security.auth.spi.UsersRolesLoginModule" flag="required">
              <module-option name="usersProperties">props/jmx-console-users.properties</module-option>
              <module-option name="rolesProperties">props/jmx-console-roles.properties</module-option>
          </login-module>
       </authentication>
    </application-policy>

So in order to use the same login that I want to use for kerberos I've created a user called steve and I've added that to the props/jmx-console-users.properties.

Stupidly the first time I added the new user I forgot to also add him to the roles properties file, the trouble is you can't sign in but you don't get any error messages in the jboss log which was a bit annoying.

Introducing Krb5LoginModule

I've never used this login module before so it's going to be a learning experience :)

I found some documentation for Krb5LoginModule and just decided to run with it.

So I changed the login-config.xml to read

    <application-policy name="jmx-console">
       <authentication>
          <login-module code="com.sun.security.auth.module.Krb5LoginModule" flag="required">
           <module-option name="debug">true</module-option>
          </login-module>
       </authentication>
    </application-policy>

As you can see I've only set a single option, this is just so I can see what happens, remember I have no idea.

Hooray
It looks like I was successful, the usual prompt popped up so I logged in as Steve and hey presto got this in the log

10:38:00,893 INFO  [STDOUT]         [Krb5LoginModule] user entered username: steve
10:38:00,978 INFO  [STDOUT] Acquire TGT using AS Exchange
10:38:01,056 INFO  [STDOUT] principal is steve@RLJASSOCIATES
10:38:01,057 INFO  [STDOUT] EncryptionKey: keyType=3 keyBytes (hex dump)=0000: FE 75 E3 7A 1A 29 7C A8  
10:38:01,057 INFO  [STDOUT] EncryptionKey: keyType=1 keyBytes (hex dump)=0000: FE 75 E3 7A 1A 29 7C A8  
10:38:01,057 INFO  [STDOUT] EncryptionKey: keyType=23 keyBytes (hex dump)=0000: 3D 8C CB 99 50 CC 9A 13   69 12 6D AA BF E7 C6 4E  =...P...i.m....N
10:38:01,058 INFO  [STDOUT] EncryptionKey: keyType=16 keyBytes (hex dump)=0000: 79 75 9B A7 67 EC BF 2F   7A 32 C1 C8 9E AB 57 0D  yu..g../z2....W.
0010: 5B B3 3D D0 64 F4 34 4F  
10:38:01,058 INFO  [STDOUT] EncryptionKey: keyType=17 keyBytes (hex dump)=0000: 87 17 D5 4C 61 35 48 4D   D4 8B 14 58 91 E5 E8 AE  ...La5HM...X....
10:38:01,078 INFO  [STDOUT] Commit Succeeded

Actually do SSO

So far we have a working test environment setup and have been able to make a call out to Kerberos to authenticate a user.

Although this is a good start to the experiment what we don't have are;

• a way of automatically sending a username to do the login, this wouldn't just be a username but would need to be the TGT.
• a way to set the Subject to the user signing in, currently we only have the principal.

I should have read the documentation as we really do have the subject of the person doing the login this is bundled up in the Credentials of the login itself, so score!!

The Connector 

The way that I want to pass the name through rather than prompt for it is to use the java system property user.name this seems like the easiest thing to do for this simple test.  Essentially the way I see this working is that the name is used to retrieve a TGT from the local cache if there is one, and of course if there is one then they are authenticated and this TGT can then be passed from app to app.  In order to do this before a call reaches the application running on the application server I intend to use connectors (or valves in tomcat) which means I need to brush up on my JCA knowledge first...more on this the next time I have some free time to add to it.

Lets take a step back

Before getting too far ahead of ourselves I think it's time to explain what exactly Kerberos will entail, much of the information about Kerberos in an easy to understand  style can be found on the MSDN site but here is the super abridged version of events.

The way my system is setup is that the is a server called the KDC (kerberos distribution centre) which has two roles.  The first role is upon request a valid user is given a ticket to get tickets (cunningly disguised as a TGT), then when a user wants to access a service they use the TGT to get a service request ticket.  Finally with this ticket the user authenticates his/her self with the service they wish to use.

The stage that our test system is at is that when a user requests the resource at /jmx-console 

Done...I think
Not sure if this is done now but it seems to be which is strange as I've set virtually nothing up?


login-config.xml
    <application-policy name="jmx-console">
       <authentication>
          <login-module code="com.sun.security.auth.module.Krb5LoginModule" flag="required">
        <module-option name="debug">true</module-option>
        <module-option name="useTicketCache">true</module-option>
        <module-option name="doNotPrompt">false</module-option>
        <module-option name="useKeyTab">true</module-option>
        <module-option name="useKeyTab">true</module-option>
          </login-module>
       </authentication>
    </application-policy>

Now when I log in to jmx-console it seems to authenticate me when I have a service ticket already.

Firefox

I also discovered that if you put about:config into firefox and then filter on negotiation you can set network.negotiate-auth.trusted-uris; to the domain kerberos authenticates against et voila ! .RLJASSOCIATES

But is it SSO
The simple truth is I don't know at the moment I haven't tested it and this is because I am not sure how to test it, so lets look back at square 1 what was the criteria;

1. not be able to access resource (/jmx-console) if I am not authenticated.
2. authenticate via the KDC.
3. once authenticated don't get prompted for authentication again.

I think that I have managed 2 of the 3 above points, however the third is a partial victory.  I don't get asked for authentication when I visit the resource in a second tab of the browser but this is simply the typical cookie authentication, when I use a different browser I still get asked for authentication and it is this step that I need to get rid of in order to claim SSO works.  The trouble is I am not sure how to do that with the tools that I have used so far, I think I might need to write my own LoginModule or something. 

One More Piece of the puzzle
I've not had much time to look into all of this stuff at the moment and tidy up this rambling blog post, this link looks like it'll be quite useful.
http://download.oracle.com/javase/1.4.2/docs/guide/security/jgss/single-signon.html 
http://download.oracle.com/javase/1.5.0/docs/guide/security/jgss/tutorials/JGSSvsJSSE.html
http://download.oracle.com/javase/6/docs/technotes/guides/security/jsse/JSSERefGuide.html