Some Thoughts On The Deskilling Of Programming

Nowadays many people with very little training or experience can create applications very quickly and very easily. Although this tends to be more true for business administration applications than for other types of applications.

This is because the latest programming languages include tools that provide almost prewritten solutions to the major parts of many problems and often there are free downloads that solve problems that are too complex to be part of a programming language.

So has designing and programming systems become so deskilled that almost anyone can do it themselves?

For some small business admin systems the answer is quite possibly yes and it is also quite possibly a terrible idea. There are a lot things that we all can do but don't do because we know how badly we will do them.

Also note that I have split up designing and programing, except for trivial systems the two are very, very different skills.

I don't mean visual design, I mean technical considerations such as what database server should we use as well as the database design. Should it be on The Cloud or Local to the business?

Should database access be via stored procedures or APIs? What are stored procedures or APIs?

In traditional IT departments programmers are not given design responsibility until they have a lot of experience and those charged with design may not be very good programmers and that doesn't matter.

Design has not gotten easier over time, way back when, computers had maybe 8-64 users who connected to them with an 80x25 character VDU. Nowadays the Internet has set the expectation that everyone everywhere will have access to the corporate data and this can easily more than offset the gains in computer performance.

But it is true that better hardware and more powerful development tools have allowed programs to be written by people who have a quite limited understanding of what the program is actually doing and on day 1 this is often okay.

But there is still a very big gap between good programming and bad programming, not just how efficient the code is but how the program handles all those unexpected situations and how easy it is to modify.

Programming non business applications however hasn't gotten any easier, the increase in computer capabilities has just made some things possible that weren't possible before.

For example I wrote PerBI after experience with Cognos Powerplay and when PC memory was jumping up from 4MB, to 8Mb to 16MB, doubling every year or so.

CPU use wise PerBI would have worked quite a few years before RAM increased enough to make it a useful product.

Although this is an extreme example if you look at the code pictured you can see a fragment from PerBi, some C++ code that drops into assembly language. Even with the latest hardware it was worth the development effort to do this because this bit of code can be called multi-million times per second and the user wants an instant answer.

The apparently reasonable idea of this is good enough for now and if we need to change things in the future then as the company grows we will, is one that will possibly haunt you for a very, very long time.

What can happen is that more and more programs get created that rely on this original design/bodge so changing this design/bodge means updating other programs in the system that have come to rely on it. Of course this means that they will all have to go live at the same time.

As you might expect the needed changes rarely get made, so over time you just get more and more bogged down by the initial design decision.

Multi user programs have always had to cater for things like record locking, where two users or programs want to change the same thing at the same time. One way of getting a solution fast is to create parts of the system that run after office hours and assume that they will never hit these locks.

Again it never gets changed and one day you want an employee rather than the company founder to become responsible for it.

Or you create a process that usually works and you simply decide that it is okay to ignore bad data as it happens so rarely that we can live with that for now.

As the business grows and the amount of bad data grows the business can suffer as customers start to notice that something is not working correctly.

Even if you fix the program today you may have still lost historic information and have to hide it or explain it when a customer queries it.

Or the thinking can be that system design and programming is easy and there will not be any problems in the future. I will leave it to the reader to decide it he truly believes this.

The reality however is that rewrites rarely happen and successful rewrites even less often, so decisions made at the start of a business or project will probably have to be lived with for a very, very long time.

This is where the difference between designing a system and writing a program becomes important.

When designing a system years of experience may allow the designer to see that yes that will work most of the time today but in a six months' time it will be impractical. Or that in the future you will want to do ... and you won't be able to because .....

Database design is usually the first problem area, the reason is that when businesses start they buy a computer to host their databases and in most cases the computer will come with enough RAM that the whole of their database could be loaded into RAM.

And this is what happens, the database server reads the requested records, then works out that is idle so it might as well load the ones that weren't asked for on the basis that they probably will be asked for later on.

So all database reads come from RAM not the hard disc, until one day the databases grows just a teeny bit to big to fit into RAM. As the database server doesn't know what to pre-read as it can't all fit into memory it might pre-read nothing and suddenly all database reads come from hard disc not RAM.

This is when you find out that your database design was wrong, incidentally Solid State Discs are usually not the saviour.

Although SSDs are massively faster to read from than hard discs, when updating databases they can be slower due to the way that they clear and write much larger areas than just the area that needed to be updated.

Finally are the tools (programing language, database and O/S) selected for the job the ones that you have some familiarity with or the best ones for the job?

Modern hardware is wonderful, it allows appallingly bad programming and terrible designs to work, until one day something hits a magic event and the system doesn't just slow down gently it stops working almost overnight.

The problem when this happens is that if you don't understand what all this wonderful technology that simplified the task down to the level that allowed you to create you system does then you have no way of solving the problem.

The magic event that caused this problem could be a update to some third party code that you use, the number of users, the size of the database or a database table, or Joe in Marketing had a great idea and is doing something new. What is important is that nobody knew that there was this threshold beforehand or why it has the effect that it does.

Importantly it also means that nobody internally knows how to fix the problem and when the solution is found it can mean starting again from scratch.

So it is important be able to balance the advantages and disadvantages of starting from scratch and writing something in C++ and taking a week or a VB .Net framework function that can do the same thing in 10 lines of code and take 20 minutes to write and test.

Or using a Framework! Already ready written code that purports to do just what you want and can be included in your application usually for free but sometime with a cost. The troubles start when you find that the framework is fine with the small amounts of data you had initially but can't handle the current level or the framework's developers lose interest in it or "completely rewrite it so that it is better".

If you are a junior programmer and use a framework that is dropped, so what, just learn the next flavour of the month. If you are the IT manager and have 10 years of development that needs to be enhanced and supported and staff found who want to develop with that tech then the problem is somewhat different.

Thirty years ago Not Invented Here Syndrome was seen as the result of a closed mind, more often than you might expect projects or parts of a project do need to be Invented Here.

This example comes from a site that uses Entity Framework but doesn't use SQL Server profiler to see what queries EF is generating.

This is unfortunately not uncommon, there is often an assumption that there is no need to check every query and possibly rewrite some of them,

This is a frequently used query, in this example it is called 3 times within a minute although this is not always the case.

At first glance it seem too trivial to be an issue, just give me the 200 newest rows with a particular status, how hard can that be?

Note the huge number of reads and the massive cpu usage.

The reason is that EF generated the following SQL, Note the select of all the rows that match the col2 value and then the select of the top 200 from that result set.

The problem is that the subquery result set is non trivial.