Some Thoughts On The Deskilling Of Programming

C++ And Assembler!

C++ mixed with a bit of assembler, and yes it made a noticeable difference.

The risk of course is that it is compiler version dependent and who wants to maintain it.

Some Areas Of Programming Have Become Deskilled, But Which Areas And By How Much?

The big risk of using the latest tech and inexperienced people to design and code your systems is that you start off down a dead end because someone can just about make something work.
Nowadays many people with very little training or experience can create applications very quickly and very easily. Although this tends to be more true for business administration applications than for other types of applications.

This is because the latest programming languages include tools that provide almost prewritten solutions to the major parts of many problems and often there are free downloads that solve problems that are too complex to be part of a programming language.

So has designing and programming systems become so deskilled that almost anyone can do it themselves?

For some small business admin systems the answer is quite possibly yes and it is also quite possibly a terrible idea. There are a lot things that we all can do but don't do because we know how badly we will do them.

Also note that I have split up designing and programing, except for trivial systems the two are very, very different skills.

I don't mean visual design, I mean technical considerations such as what database server should we use as well as the database design. Should it be on The Cloud or Local to the business?

Should database access be via stored procedures or APIs? What are stored procedures or APIs?

In traditional IT departments programmers are not given design responsibility until they have a lot of experience and those charged with design may not be very good programmers and that doesn't matter.

Design has not gotten easier over time, way back when, computers had maybe 8-64 users who connected to them with an 80x25 character VDU. Nowadays the Internet has set the expectation that everyone everywhere will have access to the corporate data and this can easily more than offset the gains in computer performance.

But it is true that better hardware and more powerful development tools have allowed programs to be written by people who have a quite limited understanding of what the program is actually doing and on day 1 this is often okay.

But there is still a very big gap between good programming and bad programming, not just how efficient the code is but how the program handles all those unexpected situations and how easy it is to modify.

Programming non business applications however hasn't gotten any easier, the increase in computer capabilities has just made some things possible that weren't possible before.

For example I wrote PerBI after experience with Cognos Powerplay and when PC memory was jumping up from 4MB, to 8Mb to 16MB, doubling every year or so.

CPU use wise PerBI would have worked quite a few years before RAM increased enough to make it a useful product.

Although this is an extreme example if you look at the code pictured you can see a fragment from PerBi, some C++ code that drops into assembly language. Even with the latest hardware it was worth the development effort to do this because this bit of code can be called multi-million times per second and the user wants an instant answer.

Design, What Design I just Wrote Some Code!

protected void LoadDataAnswer2(String dbase, DataTable result)
SqlConnection conn = new SqlConnection(dbase);
SqlCommand cmd = new SqlCommand();
conn.Open();cmd.Connection = conn; cmd.CommandType = CommandType.Text;
cmd.CommandText = @"SELECT Company, CompanyTicker, Keypoints, SharePriceCat, MinPrice,
MaxPrice, LastReviewPrice, LastReviewDate, ShareIndex
FROM StockTickers WHERE ShareIndex = 'FTSE 100'";
SqlDataReader dr = cmd.ExecuteReader(); result = new DataTable(); result.Load(dr);

A bit of code that is not releasable by anyone who has any standards or understanding but it would work most of the time, especially in the early days of a business.

System Design Decisions Taken On Day One Usually Last Forever

If you don't design your system at all or don't understand the significance of your design and just sit down and start coding it is very easy to make one or more big mistakes.
The apparently reasonable idea of this is good enough for now and if we need to change things in the future then as the company grows we will, is one that will possibly haunt you for a very, very long time.

What can happen is that more and more programs get created that rely on this original design/bodge so changing this design/bodge means updating other programs in the system that have come to rely on it. Of course this means that they will all have to go live at the same time.

As you might expect the needed changes rarely get made, so over time you just get more and more bogged down by the initial design decision.

Multi user programs have always had to cater for things like record locking, where two users or programs want to change the same thing at the same time. One way of getting a solution fast is to create parts of the system that run after office hours and assume that they will never hit these locks.

Again it never gets changed and one day you want an employee rather than the company founder to become responsible for it.

Or you create a process that usually works and you simply decide that it is okay to ignore bad data as it happens so rarely that we can live with that for now.

As the business grows and the amount of bad data grows the business can suffer as customers start to notice that something is not working correctly.

Even if you fix the program today you may have still lost historic information and have to hide it or explain it when a customer queries it.

Or the thinking can be that system design and programming is easy and there will not be any problems in the future. I will leave it to the reader to decide it he truly believes this.

The reality however is that rewrites rarely happen and successful rewrites even less often, so decisions made at the start of a business or project will probably have to be lived with for a very, very long time.

This is where the difference between designing a system and writing a program becomes important.

When designing a system years of experience may allow the designer to see that yes that will work most of the time today but in a six months' time it will be impractical. Or that in the future you will want to do ... and you won't be able to because .....

Database design is usually the first problem area, the reason is that when businesses start they buy a computer to host their databases and in most cases the computer will come with enough RAM that the whole of their database could be loaded into RAM.

And this is what happens, the database server reads the requested records, then works out that is idle so it might as well load the ones that weren't asked for on the basis that they probably will be asked for later on.

So all database reads come from RAM not the hard disc, until one day the databases grows just a teeny bit to big to fit into RAM. As the database server doesn't know what to pre-read as it can't all fit into memory it might pre-read nothing and suddenly all database reads come from hard disc not RAM.

This is when you find out that your database design was wrong, incidentally Solid State Discs are usually not the saviour.

Although SSDs are massively faster to read from than hard discs, when updating databases they can be slower due to the way that they clear and write much larger areas than just the area that needed to be updated.

Should We Use C#, VB or C++ Or Even Java Or MS Access?

Pretty much the same thing in C# and C++.

The Right Tools Used Correctly Or What We Know?

Once you start with a programming language, framework or database it is unlikely that you ever will change it.
Finally are the tools (programing language, database and O/S) selected for the job the ones that you have some familiarity with or the best ones for the job?

Modern hardware is wonderful, it allows appallingly bad programming and terrible designs to work, until one day something hits a magic event and the system doesn't just slow down gently it stops working almost overnight.

The problem when this happens is that if you don't understand what all this wonderful technology that simplified the task down to the level that allowed you to create you system does then you have no way of solving the problem.

The magic event that caused this problem could be a update to some third party code that you use, the number of users, the size of the database or a database table, or Joe in Marketing had a great idea and is doing something new. What is important is that nobody knew that there was this threshold beforehand or why it has the effect that it does.

Importantly it also means that nobody internally knows how to fix the problem and when the solution is found it can mean starting again from scratch.

So it is important be able to balance the advantages and disadvantages of starting from scratch and writing something in C++ and taking a week or a VB .Net framework function that can do the same thing in 10 lines of code and take 20 minutes to write and test.

Or using a Framework! Already ready written code that purports to do just what you want and can be included in your application usually for free but sometime with a cost. The troubles start when you find that the framework is fine with the small amounts of data you had initially but can't handle the current level or the framework's developers lose interest in it or "completely rewrite it so that it is better".

If you are a junior programmer and use a framework that is dropped, so what, just learn the next flavour of the month. If you are the IT manager and have 10 years of development that needs to be enhanced and supported and staff found who want to develop with that tech then the problem is somewhat different.

Thirty years ago Not Invented Here Syndrome was seen as the result of a closed mind, more often than you might expect projects or parts of a project do need to be Invented Here.

From the real world

A beaer logo
Picking up other peoples mistakes can drive you to drink. :-) More importantly it makes new employees question if they want to be there.

Fixing things takes time, a lot more time than it takes to do things correctly in the first place.

The problem though is that deskilling has been taken to such a level that the person writing the bad code does not even realise that they are writing rubbish.

A Real World Example

This comes off a real world production server but has been anonymised.
This example comes from a site that uses Entity Framework but doesn't use SQL Server profiler to see what queries EF is generating.

This is unfortunately not uncommon, there is often an assumption that there is no need to check every query and possibly rewrite some of them,
queryresult = (From p In ef.sometable Where p.col2 = action Order By p.Col3 Descending).Take(200).ToList

This is a frequently used query, in this example it is called 3 times within a minute although this is not always the case.

At first glance it seem too trivial to be an issue, just give me the 200 newest rows with a particular status, how hard can that be?

Note the huge number of reads and the massive cpu usage.

SQL Server profiler screen shot

The reason is that EF generated the following SQL, Note the select of all the rows that match the col2 value and then the select of the top 200 from that result set.

The problem is that the subquery result set is non trivial.

[Project1].[id] AS [id],
[Project1].[col2] AS [col2],
[Project1].[col3] AS [col3]
[Extent1].[id] AS [id],
[Extent1].[col2] AS [col2],
[Extent1].[col3] AS [col3],
FROM [dbo].[sometable] AS [Extent1]
WHERE [Extent1].[col2] = @p__linq__0
) AS [Project1]
ORDER BY [Project1].[col3] DESC

By simply removing the subquery we change the number of reads from 41,397 to 20 and reduce CPU usage from 1.201 seconds to 0.0016 seconds.

Nobody would have written the EF query if they were doing it by hand, but I can see why EF did it that way.

But and this is an extremely big but I can't say that a previous or later release of EF would generate the same query.

SQL Server profiler screen shot

At first glance this may all sound like so what clearly the server had enough RAM to serve the reads from RAM and hardware is cheap.

I even have some sympathy with this view up to a point, but look at the CPU usage, duration of 0.4 seconds and CPU usage of 1.2 seconds, that suggests three cores full time for almost half a second for just one query.

CPU cores are not cheap and depending upon exactly what licences you have nor will the database server licence be.

Remember that we have just seen that the query could have been trivial, it's not like it has to be this resource intensive.

Of course if you have a high spec server then your backup server needs to be of the same spec.

Other Interesting Pages On Some Of My Sites.
Buy A Ghost
Picture of a cat with yellow star Buy A Ghost is a light hearted source of one page ghost stories aimed at pubs and restaurants as talking points.

Print them off and put them on the tables or walls and you have an instant talking point.

Aimed at regulars who have run out of things to say or new groups or couples as an ice-breaker.
Weight Loss Calculator
Screen shot of weight loss calculator Weight Loss Calculator Tries to simulate the body and report the effects of food and exercise in periods as short as 15 minutes.

This level of detail highlights the effects of a run, bike ride or chocolate bar.

For those who are new to exercise, weight fluctuation due to glycogen usage is made much clearer.
Initial Programming Language
IPL Screen shot Initial Programming Language possibly the easiest way to learn programming on a Windows computer.

Designed to mimic the simplicity of the 1980s home computers where you could get started in minutes...

...but still powerful enough to do pretty complex things.
Please note that these links do not use any tracking cookies or similar technology.