Programmer's Career In A Nutshell

In software development, there are exactly two career steps:

1. find any job
2. find a good job

Success in the first step is determined by many factors, technical competence being a minor one. You land on your first job based on what you can supply and what is demanded. Your supply includes marketing and sales effectiveness, price and competence. Demand is more hard to characterize, but in the short term it is something that is given. You can adapt to demand only by trying to figure out who is actually hiring at the time. Personally, I found my first programming job by calling to the owner of the company (sales) while studying CS and Econ in the university (marketing, competence) and offering to work for the company as a free intern (price), after seeing they were looking for interns in a university job board (demand). The market was recovering from the dotcom bust then (demand). A little later they hired me when the previous guy quit (demand) and I was already familiar with the company (marketing), and my price (!) was right. So I didn't have or need what most people here do to find their first job, that is, personal or family connections (sales). *

Now, my present job is my third. Each time I have switched, two things have changed for better:

1. more pay and
2. more respect

Unfortunately, a third and perhaps the most important aspect of job quality has not improved unquestionably: the professional quality of software development. This is what I probably have to search for the rest of my career. But first, let me explain why professionalism does not automatically come with pay and respect.

In order to get better pay and more respect as a programmer, your work has to visibly make money to the business. If your company sells software that you develop, then your work is respected and your pay is easy to justify. But if you never deal with clients and your software is only a small part of some large offering, you are not going to be that important to the business. In the worst case, you are just an administrative cost that does not bring any money in, even if your work actually keeps the company running. The important thing to acknowledge is that the technical skills required to do the job do not justify the pay and respect, only productivity does, which is manifested by the visibility of money created or saved for your employer. This is also the reason why professionalism does not necessarily follow from the pay or the status of the job. Your job may well consist of making a lot of money from embarrassingly bad software.

I think that professional quality is something that is respected only when it is needed to stay competitive. If there is no competition, solutions need to be good enough, not good. Joel Spolsky has given an excellent talk on the subject, one that I've been only repeating here, and it boils down to this:

The key point about in-house development is that once it’s “good enough,” you stop. When you’re working on products, you can keep refining and polishing and refactoring and improving, and if you work for Facebook, you can spend a whole month optimizing the Ajax name-choosing gizmo so that it’s really fast and really cool, and all that effort is worthwhile because it makes your product better than the competition.

So the good jobs are found where software products are created in a competitive marketplace. Every other job is likely to have issues. However, only a small and probably decreasing fraction of programmers will end up having these dream jobs. The rest of us will have to choose between evils. After switching jobs two times, I'm more and more impressed with the simple Joel Test **. It is eye-opening, although depressing, to see how low most of the teams score on this scale. Software teams that I have been a member have never scored over 4 of 12. And yes, this is partly my fault, for not having been a better innovator.

-mika-

* Sorry, no references here. But I'll bet that there are LOTS of studies on how people land on their first real jobs. Even today I get regularly two or three job situation queries a year from a union, universities or the Finnish government.

** BTW, there was a cool Anti-Joel Test recently active in StackOverflow

Google Notebook End-of-Development

This is old news: Google is ceasing the developent of Notebook. Personally, this was a sad moment. Even with its rough edges, Notebook is a very unique tool that fits my information needs well. And it still does. I haven't noticed any decrease in the service since this announcement in January. This post tries to explain why Google Notebook is such an excellent application. It also presents a fast spec of the features that I see essential in any notebook app.

The Big Picture

Google Notebook does exactly what it says. It replaces paper notebooks. To be fair, I still use one paper notepad to write down notes during client phone calls but other than that, I use Notebook. I have zero tolerance for Post-It notes and the like. No paper on my desk. Even the paper notepad is just like a temp file and will eventually end up in shredder.

In addition to replacing paper, Notebook substitutes electronic notepads of which there many variants:

• Email sent to your own email address
  
• Local text files and Word documents
  
• Online text files like Google docs
  

All of these are improvements over paper in three areas: management, search and sharing.

1. Management

This is the reason people started to use computers instead of typewriters in the first place. By separating the act of writing from the physical output, the management of written material gets more flexible. You can make identical copies of your documents easily. You can categorize documents without reshuffling drawers. You can sort documents fast by different criteria and transfer large piles of them in small-size storage media or through communication networks. These same networks also enable you to to access documents remotely.

2. Search

Electronic documents can be queried with the help of computing power. Compared to a computer, humans are very slow at doing keyword searches over large bodies of text.

As the number of documents in a collection increases, search becomes more valuable. If you intend to save all your notes in one system like Notebook, search is likely to be the most important feature. It is somewhat surprising to see that Google Notebook has perhaps the worst search engine ever released by Google. The first thing that reveals a crappy search engine is that retrieval is very sensitive to inflectional forms.

A simple example. Three notes:

• "cat"
  
• "cats"
  
• "cat cats"

In Notebook, queries for "cat" and "cats" will return different results, except if the result contains both forms. in most cases, even grep would perform better. However, you can use labels in Notebook to somewhat improve search recall.

1. Search for "cats" does not find "cat"

2. Search for "cat" does not find "cats"

3. Search for "cat" will find all notes labeled with "cat".

3. Sharing

Third improvement in electronic documents over paper is the ability to share them more efficiently. There are four modes of sharing:

• one reader/one writer: distributed copies of the same files
  
• multiple readers/one writer: write-locked files in a network share
  
• multiple readers/multiple writers: database applications
  
• (one reader/multiple writers is so rare I can't find a single example)

Distributing copies is the traditional method of sharing electronic documents. This is what you do with email or file transfer. The problem is in distributing and merging changes. In any actively developed document you will need a version control system to handle distribution and merging. Most programmers are well aware of this, since program code files are such documents.

When network storage started to get common, people began sharing documents, usually MS Office files, in a network share. The first one who opened the file got the write lock and the rest of the collaborators were just readers. In Excel there was even an option to switch the write access between users, and even secure the write access with an edit password. This scheme works for small groups, but it contributed to the way of managing corporate data that is commonly known as "Excel hell", a sea of somewhat connected Excel documents. A large part of information systems history of the 21st century can be summarized as reconstructing these shared excel files as database-backed web applications.

Finally, database systems are designed for concurrent writing, so that online services like Google Docs, Notebook and Wikipedia allow multiple writers to modify a document at the same time. Of course, this makes version conflicts possible and require some sort of a version control scheme.

I have never used Google Notebook sharing features, since I don't work with other people familiar with the application. There is a notebook-level sharing with edit priviledges to everyone invited, and also an option to publish the notebook as a static web page. Notebook-level sharing is probably a good choice, since you can keep private notebooks easily separated from a shared one and yet move notes easily between each one.

Notebook != Word Processor

There are a few features that make a notebook app unique. First, note taking is not document preparation or word processing. Printing is not a primary concern here*. Notes are all about planning and observing, they are like a personal ERP system.

* before laptops and online apps I used to print summary todo-lists out of my notes

Task management

The closest thing to compare Notebook is a task management application like RememberTheMilk. The key feature in both is the speed an ease of creating new tasks/notes. While task management focuses on due dates and reminders, a notebook focuses on planning with or without time constraints. It is still a useful feature in a notebook app to include due dates and especially have a simple tool to attach formatted 'Now' timestamps. Timestamps are extremely useful in notes but in Google Notebook you either have to write them yourself or use an external tool to generate one to clipboard.

Project management

Google Notebook supports grouping of notes well. This means that every user can customize the browsing experience. There are two levels of grouping: notebooks and sections, which is enough for most users.

For me, a single note corresponds nicely with a small project, especially if the project has some logical end. This kind of a projects consist of tasks that are simple one-line items. A notebook consists of similar projects. I always had a section "Old" in every Notebook. When a project is finished, I moved it to the old category. Sometimes I use an "Active" section on top of the notebook, with all the active projects.

Things to remember

I have found notebook very handy in storing snippets of knowledge that I may need another time but don't want to search the hard way. Software development and system administration is full of such small contextual things. These should be easy to transfer to Google Docs if Notebook is eventually shut down.

Final Words

Notebook is essentially an unstructured or loosely structured information system, with full-text search and the ability to organize notes to relevant categories. Tagging helps to connect notes in different categories.

In the current Internet ecosystem, it is probably hard to make money out of a Notebook-style application. Even Google couldn't sustain it with all the advertising infrastructure they had in place. Notebooks are private and sketchy and I think they should be paid for like their paper counterparts. One dollar/euro a month would be a reasonable price, and the payment should be smooth.

-mika-

Stock Tips for the Fools with Money

Last week, the S&P 500 index was about the same as 7 months ago, right after the crash. Things haven't gotten worse since October. Comparing to prices half a year ago is a crude signal, but I like to use it as a wake-up call. A call to start looking for long-term buying opportunities. Why?

First, it gives you some assurance that panic selling is over. No one should make long-term investments in panic conditions when even the value of money itself is questioned. A small gain in 6-7 months is not enough to call the end of a bear market, but if the upturn is real, things should get better every month: 7, 8, 9 ... months of rising or stable stock prices should not be the worst time for a long-term investment.

I have collected a list of stock tips. To make this as concrete as I can, I pull some OomA forecasts of actual stock prices or gains. Oom stands for "Out of my" and A stands for something else. I intend to look up these later and see how much I was wrong. I am well aware of the efficient market hypothesis, that states that current prices reflect all known information. The explanations I give for my forecasts are simple guesses on how the unknown will turn out. Other usual caveats of course apply: you should never invest long-term if you are spending more than you make. There is no long term then. The same applies if your income is at risk.

Last, I want to disclose that right now I don't have any personal investments in stocks. As if such a disclosure would matter anything in an anonymous blog. But I do manage a small portfolio of less than 100k€, and have to report yearly on its performance. This portfolio is also the main reason why I need to have an investment strategy at all. In 2008 this portfolio didn't lose value in the market, mostly because its stock weight was less than 10% and interest rates paid well enough to cover losses. Decreasing the portfolio risk was a strategy chosen in early 2006. There really was no visibility then, as witnessed by the market: deep sell-off in summer 2006, climbing back up to new highs, deep sell-off again in early 2007, and finally reaching the bull market peak in 9.10.2007. Visibility is bad today, sure, but unless the world goes like Mad Max there will be an eventual global recovery. It is time to revise the strategy for this recovery, which means that stocks should account for more in the long term, perhaps like 40% by the end of 2010.

1. It's time to start investing, gradually, in ETFs

• No big hurry here, but I expect net returns to be positive if you put money in this year and collect it after two years
  
• OomA forecast states that at the end of April 2011 the S&P 500 will be 30% higher than at the end of April 2009 (872,81), that is, around 1135
  
• 30% will be at the low end of historical bounces, but this time most economists don't believe in a fast recovery. Mixed news will cause high volatility.
• On the other hand, contrast this to bad historical bear markets where stock prices could still fall for a long time

2. Tata Group, Foxconn and other Asian brands

Indian and Chinese companies will become household names, with Tata Motors and Foxconn perhaps being the most famous right now. (Tata for Nano and Foxconn for its huge 250 000 employee plant in Shenzhen.) These two have pretty much caught the imagination of the world when it comes to India and China. But they wont be the only onces, with Asian car and electronics manufacturers becoming more competitive in every marketplace.

The premise behind any estimates on large Asian companies is that, little by little, they will become technologically superior with huge economies of scale AND the largest domestic markets on the planet. The question is not if but when. And the best predictor on how fast this will develop is the level of incompetence that the current incumbents will show. If they continue to focus on 'brand ownership' only, while losing their edge in R&D, eventually they will become like Arthur Andersen, a company that lived and died on its brand.

10 year OomA prediction: In the next five years winners and losers will be more clear, but right now you'll do fine by investing in industry-specific funds*. Electronics and automobile funds both should grow at least 200% which is a lot.

* No, I don't know exactly what these would be, but I'll bet there is something

3. Moore's law says that SSD drives will make Intel a lot of money, eventually perhaps even more than processors and chipsets

• this is a medium-term investment, 3-5 years
  
• if the INTC stock price is about 15$ now, OomA forecast for 2014 will be at least 30$
  

4. Lithium Batteries and Solar Panels

The peak oil thing hasn't gone anywhere. It is just on hold now. It is pretty safe to assume that oil will be expensive for the rest of our lives.

Oil is nice because it is both an energy source and an efficient energy storage medium. In the future source and storage are somewhat more separated, at least in consumer goods. Lithium-based storage (batteries) power most of the consumer electronics today and will power hybrid cars tomorrow. If Lithium is going to be the new oil, then Bolivia with 50% of the reserves could become the new Saudi Arabia. I have no idea how this windfall is going to be shared, but it looks as if Che Guevara might get the last laugh after all.

Solar panels will generate power most of the sunny belt, with prices likely to follow Moore's law.

When you search for market leaders in batteries and solar panels you'll get three big Japanise companies, namely Sanyo, Panasonic and Sony, and a Chinese contender, BYD. And guess what, Panasonic is currently in the process of buying Sanyo.

OomA forecast states that in the next 10 years the biggest profits will be made here. But things are fluctuating in the energy business, and low oil prices could put things on hold for another five years or so. Still, I'd say that mining in South America is going to be hot. Look for funds in this sector and you will gain 500% return. In the manufacturing side, Sanyo alone could be 20 times more valuable in May 1. 2019 than now, (from $3,5 to $70 billion), and combined with Panasonic, the combined market cap could increase from the current $35 billion to over $200 billion.

5. IT industry

IT industry is pretty solid these days when debt is hard to get. This is partly because IT companies usually are not much leveraged. No one has been willing to lend anything against that freaky 'intellectual property' stuff, especially after the dotcom bust. But new sales are hard to get if money does not flow elsewhere in the economy. Most of the IT companies that exist today have survived the previous bust and should have plenty of cash to cushion through the bad times, right? Nevermind the ones that don't. Another thing is that existing IT infrastructure produces better value when technical people have more time to spend with it. During good times most of the energy goes to new projects that are always more risky.

So, what makes IT industry run so smooth? One dirty secret is that unless the world gets a single central government, one that builds good information systems that are accessible to everyone, IT can pretty much sustain itself by moving the same data around in every organization on the planet. And call that a productivity boost. That being said, most of the new growth will come from emerging markets where IT investment should have much better ROI.

Long-term OomA results: IT is a solid blue chip performer, and performs on average +5 %-points better than S&P 500, each year for the next ten years. This is a very serious figure, as it says that even if S&P would be flat zero, IT would have 60% larger market cap than today.

6. Business Intelligence?

There is a fundamental problem in BI. It tries really hard to tell the truth about your business. But truth has never sold anything. In fact, you can't sell the truth. Nobody wants it. And eventually you will get the truth for free, if you don't mind getting it served with surprise and pain.

BI for the executives is doomed as a high-price item as soon as every not-so-big departmental boss gets his own OLAP cube. You could brag with executive dashboards in 1999 but not in 2009.

But the rest of the BI is much more interesting. Operative business intelligence is about the last thing that Information Systems can offer to boost an ordinary line worker's performance. The rest of the benefits will come from system-level stuff like operating systems that crash less often, ubiquitous fast networking and the like. Operative BI means combining data from different sources and packaging it for each consumer. A line worker will get what is needed, regardless of where the data was initially stored. It remains to be seen how much more efficient this will be than the current, largely manual practice of using separate systems together (if possible). Given the state of business IT and how hard it is to get people what they actually need, a centralized system sure could unlock some hidden value. But only if the costs of extracting business data are not too high, and the ETL systems required are not too fragile.

And yet, the real BI dream is not only to have stuctured stuff like OLAP cubes but all the unstructured Word documents and the like, accessible with a decent search engine that also respects the security of the data. Search alone will probably matter even more than the operative BI.

One thing, Microsoft may lose a lot in other fronts but in this one it won't. In business intelligence, Microsoft will eventually become the new IBM. But it still needs to get its act together and wrap some pieces around Sharepoint. A better document management system. A better unified communications infrastructure. A serious, configurable search engine to index everything inside a company, structured or unstructured.

OomA: For the next 10 years Microsoft will grow as a business software company. Stock prices and dividends combined, I'll say that MSFT can at least match IT services average growth rates, which is not bad for a juggernaut. I would say that if IT market cap grows 60% in the next 10 years, then MSFT will double its stock price (about $20 now), all this presuming that S&P will be flat.

No huh. This is it, thank you for listening. If there is any strategy behind these tips it is that expensive energy should turn the economy more... intelligent. If you would pick one word to describe the world economy of the early 00's, my choice would be 'cheap'. It may be that I have strong personal incentives to believe in other factors than low cost, which will bias my judgement here.

Anyway, if you take my advice and lose your money, I will award myself the honorable MBS degree. You know, the one that distinguishes the true master of BS :)

-mika-

Spoken Word Has Its Place

Back in the 90's there was exactly one interesting show in the Finnish radio. I think that the show was called "The Diary" and it was hosted by a journalist Hannu Taanila, with occasional guests.

Basically it was a podcast, but instead of being a constant conversation between the host and his guests, Mr. Taanila often just spoke himself. And he was very good at it. The skill of scripting and performing an entertaining, opinionated speach, on your own, must be the cornerstone of rhetorics. And it is something that is always missing in every unpopular talk-show.

Mr. Taanila was known for his left-wing bias, which of course made him controversial. I think he wouldn't disagree being called a "die-hard communist". But it is bad judgement to let character overshadow accomplishments. In this case, we are talking about essentially the best talk-radio performer in Finland, ever.

Today, 'spoken word' has made a comeback, in podcasts. There is clearly a sweet spot for talk shows without the TV-studio craziness. You know, the thing where visual appearance matters so much that the whole setup is surreal. The subject matters of discussion cannot, by design, prevail in television*. And even if you try, there won't be much benefit over talk-radio. On the contrary, podcasts have a huge advantage in introducing ordinary, media-shy experts, anywhere in the world to an interested audience. This is something that Mr. Taanila pioneered in his show.

Mr. Taanila has turned 70 years today. Congratulations.

-mika-

* To be fair, journalists like Tim Sebastian and Leif Salmén at least try to make TV debates intensive and risky for their guests

Five Blades

In 1998 Gillette unveiled the Mach3. It is reported that the R&D of this new three-blade razor cost $750 million. It took another $300 million to promote it in the first year. Three-blade razor was long in the making: the first two-blade razor Trac II was launched in 1971. In between, it kind of looked as if adding new blades didn't make sense. 

Fast forward five years and all hell broke loose. In 2003 Shick-Wilkinson announces the Quattro, a four-blade razor. Lawsuits followed. Things were so hot in the razor business that it prompted the Onion to publish a brilliant satirical essay on the next move in the razor wars.  Needless to say, every prediction in the 2004 Onion article turned out to be correct. The razor in the picture is a five-blade Gillette Fusion launched in 2007.

Perhaps the most contradictory thing in business is that in order to succeed you must sell what customers want. "What they want" is not always the same thing that "what they need", based on the expert judgment. But if you don't sell what they want, someone else will, and you are soon out of business. Only by government intervention people can be enforced to accept something against their will. Experts' role is not to enforce but to persuade the public on what they think is really needed.

Now, cut-throat competition in producing what people want easily leads to madness. That is why there are such things as five-blade razors in the market. But it is no secret that marketing plays a crucial role in breakthrough innovations too, especially in consumer electronics and computer hardware. Enormous technical advancements in electronics have brought us the capacity to calculate, store, transfer and present information in ever increasing amounts, but it has been a marketing job to suggest why anyone would want to do so.

And sometimes these ideas are just crazy, at least until they become mainstream. Using computers for gaming must have been a mad idea in the early 70's. Cell phone users were ridiculed in the late 80's. And only rich and hip celebrity figures used Apple computers and drove Toyota Prius hybrids early in the 00's. But of course, it is hard to make a difference between things that are brilliantly mad and the things that are just mad.

Right now, there is an interesting gold rush on third-party iPhone apps. Apple has succeeded in creating both the technical platform and a viable marketplace where consumers actually buy stuff. I must admit that price plays a role here, as iPhone apps usually cost one dollar, same as the music tracks. Compare this to the Nokia S60 marketplace where stuff is either free or costs well over $10. I don't own an iPhone or know any good iPhone apps. In my opinion, most of them look absolutely crazy. But I could try one for a dollar.

-mika-