An Explanation For Non Programmers On Why Programming Tests Are A Bad Idea

The primary problem with programming tests is that it is almost impossible to create a good one that takes less than a day to complete.

As an example on this page I have shown what sounds like a reasonable short test question and provided three answers.

As this page is aimed at non programming recruiters it is not necessary to fully understand the example answers, although you probably will end up with a rough grasp. The important point is how different the answers are and how do you as the recruiter gain any useful information from the answers?

The second problem stems from the first, many good candidates simply won't take tests because they see them as a waste of time because the test can not tell the recruiter anything useful.

So any company that uses them is not one that they want to work for.

As a recruiter you may hate this but it does reflect the fact that the demand for good programmers has outstripped the supply for many years, COVID may have altered this balance but not the belief.

So a company that uses tests could be starting a self reinforcing loop, as a company that tests can only recruit from the pool of candidates that will accept testing then it is possible that it is mainly testing those less suitable for job.

This may lead to poor selections creating the view that more testing is needed to weed out the potential wrong choices, but this extra testing reduces the candidate pool even further.

My view is that if you want to do a useful test then you really need it to take a day or so. If a test is that long will a candidate have the time to take it and are you willing to regard that as time that should be paid for?

Whilst there were many versions of each language, each hardware maker would have their own version, they all had quite limited capabilities and were all based on standards, except for where they weren't.

So if someone said they could program in say COBOL then there was a core part of COBOL that they must know regardless of their background and only one reasonable way to do something.

A competent test would also allow someone answering a question using IBM COBOL for a job writing ICL COBOL to pass.

At the same time there were also languages like C which are different in that there is a core language, again every C programmer would know most of the core and also optional libraries that aren't part of the language but could be viewed as extending the C language.

Most C language implementations include a library called STDIO which handles very basic reading and writing to the screen or disc. To repeat the above the library is not part of the language.

So it doesn't follow that a C programmer would had ever used STDIO, he may have worked somewhere where STDIO was replaced with XYZCorpIO which does the same job in a completely different way.

Hence it is not possible to infer anything from testing a programmer's knowledge of C by asking questions about a library such as STDIO unless he explicitly claims to be competent with that library.

There is no longer a core of a language that everyone should know, the C# and VB languages have expanded to the extent that they contain commands that a programmer could legitimately never use.

LINQ is a great example, it has become a "Marmite" part of the C#/VB languages, some people love it and some hate it.

Also the idea of a library being separate from the language has gone and the two have merged into one. In this example question many people would say that a datatable is part of the language even though it is part of a library.

Any C# test that includes LINQ, which is part of the language not a library, has the implicit assumption that the person taking the test has been using C# in an environment that supports its use.

The reverse is also true, an environment that uses LINQ may not use the technology that the test setter has set a question for.

Failing someone for never having used LINQ doesn't tell you that they can't learn to use it. Nor does failing them for not knowing the non LINQ way of doing it mean that they can't learn that way.

If you really want to discard a candidate for needing a couple of days familiarisation with a handful of new commands then that is your right.

Just remember that this is what doing tests probably implies to many candidates, so it may reflect badly on you as an employer.

So when you are creating or purchasing your tests you need to be aware that the concept of their being one correct answer no longer applies.

I have used LINQ as an example quite a lot because it is easy to do an internet search is LINQ good and see many postings that are easy, sometimes less easy, to read and understand.

Again it doesn't matter if you fully understand the arguments in these posts, the fact that there are so many highlights the possible issues.

Full details of the test are here, whatever they were looking for my answer wasn't what they wanted.

Which I suspect puts the test into the third category, they were making assumptions and hadn't specified them in the test.