Brian Bramer, DeMontfort University, UK (bb@dmu.ac.uk)
Goto Computer
Systems Notes main page
Abstract
The selection of a computer system to satisfy an end-user requirement can be very difficult. With the advent of cheap microcomputers in the early 1980's many small to medium sized organisations which had never had computer systems of their own started to purchase microcomputer and minicomputer systems. It was, and still is, very easy to purchase a system which, at best, is difficult for non-computer literate staff to use, or at worst, does not satisfy the end-user requirements. This paper describes techniques which can be used to formalise the purchase of small to medium sized computer systems.
Contents
2.4 Future system upgrades and enhancement
4 The ITT (invitation to tender)
10 Examples showing pitfalls which can occur in the purchase of systems
The selection of a computer system to satisfy an end-user requirement can be very difficult. Until the late 1970's computer systems were expensive in terms of capital cost, maintenance and support staff. Any organisation which had built up a large investment in computer systems maintained a team of specialist staff skilled in assessing user requirements, evaluation of alternative systems, purchase and then installation and training in an end-user environment (Joslin 1968, McQuaker 1978, Sawbridge 1979).
With the advent of cheap microcomputers (Hazan 1984) in the early 1980's many small to medium sized organisations began to purchase and use computer systems (Skidmore 1987, Wroe 1987). Such organisations soon found that the purchase of systems to satisfy the needs of skilled staff (secretary, accountant, manager, scientist, engineer, etc.) was surrounded with pitfalls. It was, and still is, very easy to purchase a system which, at best, is difficult for non-computer literate staff to use, or at worst, does not satisfy the end-user requirements.
This paper describes techniques which can be used
to formalise the purchase of small to medium sized computer systems (the
purchase of a large computer system would entail a much more thorough application
of the techniques described). In practice the amount of work required in
the selection of a system varies with the capital cost (and hence size
of target system) but the procedures followed are essentially the same.
Although it is not possible to guarantee that the optimum solution is achieved
the major pitfalls can be avoided.
1 Sequence of events
When considering the purchase of a computer system one does not start by visiting the local computer shop or by picking up the telephone to call a system vendor to give a demonstration of their products. That is the way to end up with a system which satisfies no one, except possibly the salesman.
The following section describes the sequence of procedures which should be adopted in practice.
Carrying out a feasibility study and drawing up a requirements specification can be difficult for staff who are not experienced in the purchase of computer systems. Common problems that occur at this stage are:
When a budget is agreed the requirements document
can be finalised. When carrying out the feasibility study the following
points should be kept in mind.
2.1 Consultation with end-users
The end-user of the proposed system should be involved at all stages of system implementation. Professional systems analysts and software engineers have always involved end-users, but small companies have sometimes failed to do so. During the 1980s there has been an explosion in the use of computer systems in small office environments. A major problem was that there was little experience in the purchase or use of computer systems. It often ended up that a senior member of staff who had a home microcomputer (programmed in Basic) become the company 'expert'. This person undertook the purchase and installation of computer systems usually without any reference to the end-users. Purchases would be made on viewing slick demonstrations (generally with lots of fancy pictures) which had nothing to do with the needs of the end-users. A common example is secretaries being supplied with word processors which were difficult to use, had a poor keyboard (business quality typewriters have a high quality keyboard which gives a consistent feedback to the user), and at worst, could not keep up with a skilled typist. Surveys have indicated that half the microcomputers purchased by small businesses end up hidden in cupboards.
In practice the organisational problems of installing a new computer system can be greater than the 'simple' technical one of finding a system to do the job (Stracham, Cross & Black, 1988). For example, staff attitudes to changing work practices can cause major headaches. Involving end-user staff at all stages can alleviate these problems.
When buying computer systems, as with anything else, the rule is 'let the buyer beware'. Salesmen, whether selling cars or computer systems, emphasise the good points of their product while glossing over any inadequacies.
The content of demonstrations is carefully chosen to give a smooth and high quality presentation which avoids those facilities which do not exist, do not work or only just work. It is up to the purchaser to determine, by experimental use of the product and by the asking of pertinent questions, if the system will satisfy his requirements. For example, a few years ago the Polytechnic had a requirement for sixteen networked workstations in a distributed environment. At one particular demonstration salesmen appeared unable to hear questions, or gave glib answers, to enquiries about the ability of their system to support sixteen workstations making intensive use of the network. Explicit questions to one of the hardware support engineers finally determined that no working networked configuration in the UK had more than five or six workstations. It was later confirmed that more than six could cause network overload.
2.3 Benchmark tests (Bloom 1986, Weicker 1990)
At some stage it will be necessary to run some tests on selected systems to gain some idea of configurations which will come anywhere near to meeting the requirements. A small number of tests are selected which are representative of the work load of the proposed system. Such tests should check the essential, highly advantageous, advantageous and desirable requirements and how easy they are to use (see usability section 6.1.2 below). Giving a score of quality or merit to each facility can help when one comes to evaluating tenders at a later stage.
Benchmark tests are run on target systems similar in configuration to those in-mind or proposed. One simple way to gain an appreciation of a particular system is to talk to existing users with the same or a similar application.
After a computer system is installed acceptance tests will be run to ensure that the requirements specification is satisfied. In practice these tests are usually a more thorough version of the benchmark tests.
2.4 Future system upgrades and enhancement
During the lifetime of a computer system both hardware
and software upgrades may be required. This could include adding extra
memory, disks, workstations, new software, etc. When purchasing a new system
possible upgrade requirements should be kept in mind and written into the
requirement specification. For example, to add another user to a fully
loaded minicomputer would require expensive hardware upgrades or even the
purchase of another system. In a distributed system it may be possible
to simply add another workstation to the network (assuming sufficient disk
space on file servers and that the network can carry the extra load, etc.).
Hardware obsolescence. Due to rapid technological evolution computer systems soon become obsolete with a typical lifetime of three to four years. Once equipment is over three years old maintenance costs rapidly increase (can be up to three or four times the cost of a modern equivalent) forcing replacement of equipment. This can cause problems with auditors who are used to capital items having a lifetime of seven to ten years.
Software Obsolescence. At regular intervals software houses release new versions of software packages with known faults repaired and enhanced facilities. Problems which occur in practice are:
Special offers. Computer hardware vendors often have particular configurations for sale at 'special' or 'discount' prices. One problem of rapid obsolescence is that a manufacturer can often be left with a warehouse full of last years equipment which has been superseded by a new model. The old models are then offered for sale in special deals. Such offers should be viewed with care:
If an organisation already has some computer systems the proposed system may be implemented either as an enhancement to existing system or the purchases of a similar system. This has the advantage that staff have experience using the existing system and training will only be required for additional software packages. Such a strategy should, however, be carried out with care. It is very easy to become trapped into obsolete technology and every few years a through review should be carried out to consider total system replacement.
If a new software package is to be used on an existing system care must be taken to ensure that vendors are aware of the exact configuration in terms of processor, memory, disk, numbers of users, etc. Vendors will then be able to determine if their package is available for the particular configuration, and if so, whether there are sufficient system resources to support it. For example, a software house may recommend increasing memory and disk capacity to support the extra work load.
When considering the installation of medium to large computer system one has to decide whether to purchase or lease.
Leasing The system is leased (normally from a finance house) for a minimum initial period. The cost of leasing includes:
Variations
The are a number of variations, including:
Packages are off-the-shelf programs which can satisfy the requirements of end-users in their particular application areas. If a user has a requirement for a program the first thing to do is to check if a package is available which meets the specification, e.g. by looking in journals, contacting software houses, computer vendors, etc. If a package will satisfy the specification at a reasonable cost it should be purchased; only as a last resort should a new program be written.
The cost of packages can vary from L50 for a cheap microcomputer package to L500K for a suite of commercial engineering CAD packages. It is clear that when priorities are drawn up for purchase of packages the cost is an important factor. Many software houses and distributors now sell demonstration packs of their software, which can be used for evaluation purposes, for prices in the region of L100. The packages are reduced versions of the full commercial systems, e.g. a database may only be able to access 100 records rather than 1000000, but is still adequate for evaluation purposes.
Software Licences: Users should remember that
a licence is usually required for each machine that the package will be
used on and unauthorised copying is a breach of copyright (Gold 1984, Irish
1988). It is often possible to get multimachine or multi-user licences
for use within large organisations.
When installed, a small microcomputer, with a simple operating system and easy to use packages, will require little specialist staff support. The end-user can perform disk backups and similar system maintenance.
In the case of a more complex system, e.g. minicomputer, distributed workstations on a network, etc., specialist staff will probably be needed. For example, the following tasks would have to be carried out:
3 The requirements specification
Once the feasibility study is complete and a budget approved a detailed requirements specification can be drawn up. Appropriate parts of the feasibility study can be used and extra information added until one has an exact specification of the system requirements (again with a priority list of essential, highly advantageous, advantageous and desirable facilities).
In practice the requirements specification can be totally in terms of the end-user facilities and software requirements with no hardware configuration specified except in the broadest terms, e.g. minimum disk capacity or printer quality. This can be perfectly satisfactory but has the danger that if care is not taken it is possible to specify a system that cannot be satisfied within the budget, i.e. one has to have some idea of suitable target hardware configurations and prices even in a software driven situation.
Some large organisations have of corporate policy
of buying hardware from a particular vendor, e.g. IBM, ICL, DEC, etc. Any
requirements specification will therefore be restricted. Such large organisations
generally have a department which deals with computer purchases on behalf
of end-users.
4 The ITT (invitation to tender)
The ITT is drawn up from the requirements specification. It contains sufficient information to enable system vendors (hardware and software) to appreciate the system requirements (essential, highly advantageous, advantageous and desirable) and to determine if a system configuration can be proposed. Contractual information (Rennie 1985) would be maintenance required, maximum delivery dates, details of acceptance tests and the date by which the tender documents are to be submitted. Details of penalty clauses should be given, e.g. failure to meet delivery dates, pass acceptance tests, etc.
If special software has to be implemented by a third party, e.g. a software house, this can be specified as part of the overall ITT or arranged separately (in such a case areas of responsibility should be clearly defined).
A question often asked is 'does the ITT specify the system budget ?' There are pros and cons:
The ITT is sent out to computer systems vendors who respond with a tender document which states how their proposed system satisfies the ITT. In practice this can vary in size from a one page price quotation for a small microcomputer up to hundreds of pages for a large mainframe system. The tender document should specify the proposed system in detail:
All these factors are important. From the first three
it should be possible to determine if the system proposed will satisfy
the requirements. Environmental requirements should be examined with care
to see if a special power supply or expensive air conditioning is required.
From details of delivery dates it is possible to see if a system can be
obtained to meet any project deadlines. The terms and conditions of sale
will agree (or not agree) to acceptance tests and specify how the system
is to be paid for. For example, if acceptance tests are to be run the vendor
may require 20% paid on delivery and the remainder when the tests are complete.
It is not wise to pay for the whole system and then carry out acceptance
tests.
With the exception of the initial feasibility study this is the most difficult stage of the procurement process. In response to the ITT one may be faced with twenty or thirty tenders each of which, in the case of a medium sized system, could be fifty or more A4 pages describing the proposed configuration of hardware and software.
Even when purchasing a small microcomputer it is not simply a case of accepting the cheapest quotation; one has to be sure the company can provide the hardware and software support needed during the lifetime of the system. Clearly one needs a set of selection criteria which cover technical evaluation, cost comparisons and contractual issues.
During this stage further benchmark tests may have
to be carried out on configurations similar to the proposed systems to
determine accurate measures of the selection criteria and to ensure that
the systems come near to meeting the requirements (life becomes very complicated
if a system totally fails the acceptance tests).
6.1.1 Meeting the end-user requirements
The requirements specification will have a priority list of essential, highly advantageous, advantageous and desirable facilities of the proposed system. These lists can be used as initial evaluation criteria to determine how the tenders satisfy the requirements. The simplest way is to form a table of requirements and for each tender tick off those requirements which are satisfied. An initial filter can then be made which removes any tenders which do not meet the essential requirements. In the case of complex systems weighted scoring (see below) can then be used to gain a measure of how each of the tenders satisfies the overall requirements. Clearly the use of information gained from benchmark tests is of major importance in checking out the claims for the facilities provided by software packages.
6.1.2 The 'usability' of the system
Although the most important factor is to satisfy the essential 'technical' requirements of the end-user, the 'usability' of the system is very important. For example, in an engineering CAD environment a package may be able to carry out all the tasks required at a 'technical' level. The engineers will learn to overcome any problems associated with the user interface and the intricacies of getting the overall system to do what they want. If in addition to satisfying the purely 'technical' requirements the product is easy to use, large gains may be made in engineer productivity, product development times, product quality, etc.
The assessment of the usability of complex software systems is non trivial (Monk 1985, Rickett 1987). Tests of usability are not simply a case of asking the user to 'play with it for a while and let me know what you think' (Potosnak 1988) but should be carefully thought out and built into benchmark tests.
6.1.3 The computational 'power' of the system
The software and hardware configuration must be capable of executing the end-user software to give satisfactory results in terms of response time (Brady 86), results presentation, file storage, etc. There are a number of fundamental requirements:
The only real way to see if a configuration works is to run it with the end-users performing their tasks. In practice this is usually not possible but one can:
6.1.4 After sales support
The tenders will describe the hardware and software maintenance available and any training provided.
6.1.5 Vendor company viability
Clearly when purchasing a large and complex system one needs to ensure that the vendor is going to be around for the foreseeable future to provide the after sales support. An evaluation can be made from the companies track record in terms of sales, profits, etc. and important factors such as location and number of maintenance and support staff. Even when buying a small microcomputer one could still need assistance with setting up hard disks, using software packages, etc.
Many vendors will supply a list of customers which
can be approached for running benchmark tests, commenting on level of support,
etc.
Assuming that the tenders satisfy the essential technical requirements the costs of alternative systems can be compared. Factors to take into account are:
With large computer systems contractual details can be very complex (Rennie 1985). Even with small to medium systems one has to consider:
6.4 Formal evaluation techniques
A number of techniques have been developed over the years which attempt to formalise the evaluation of tenders (Joslin 1968, McQuaker 1978, Sawbridge 1979). All require a through understanding of the requirement and careful evaluation of the important factors. For example, when using weighted scoring each evaluation criteria is assigned a weighting according to its importance:
where wi is the relative weight of feature i of n features (the feature may be a technical facility, a measure of usability, cost, etc.).
During the evaluation process each feature is scored according to its merit and the following sum formed:
where si is the score for feature i.
This does give a figure which can then be used as
a guide during the evaluation process. It is, however, difficult to establish
meaningful relationships between high score for good technical performance
and high score for low cost. One way to overcome this problem is to also
have separate sums for the technical evaluation, cost comparisons and contractual
issues.
6.5 Once the shortlist is formed
From the above process one should have a shortlist of vendors whose proposed systems can satisfy the essential technical requirements and are also satisfactory in terms of usability, maintenance, costs, etc. During the process one may have to have further discussions with the vendors who are appearing at the top of the shortlist to finalise contractual arrangements, acceptance tests, etc. In addition one may wish to carry out a field evaluation in which detailed benchmark tests would be run on a system similar to that proposed.
In practice, one usually finds that the two or three vendors at the top of the shortlist are running very close together. It can then become very difficult to select the final vendor with marginal factors tending to determine the final decision. For example, systems from one of the vendors may already be installed and performing satisfactorily within the customer organisation and the vendor has a good track record on maintenance, etc. Buying from the same vendor then has the following advantages:
7 Order, installation and acceptance tests
When agreement has been reached with a vendor an order can be placed. When the equipment arrives on site and has been installed the acceptance tests are carried out.
The function of the acceptance tests is to ensure that the requirements specification is satisfied (in practice these tests are often a more detailed and thorough version of the benchmark tests). If the system includes end-user software this will form part of the tests. In the case of small computer systems (e.g. a L1000 microcomputer) the benchmark tests could have been sufficient to test the target system and no formal acceptance tests will be required.
If the acceptance tests are failed further discussions take place with the vendor. In general, the vendor will upgrade the system, at no additional charge, until the acceptance tests are passed, e.g. by putting in a more powerful processor, adding more memory, disks, etc. It is necessary to ensure that maintenance charges are not increased to cover the 'extra' equipment. In extreme cases penalty clauses will be invoked and legal action taken.
Some computer hardware vendors do not like to have third party software form part of benchmark and/or acceptance tests. When buying a complete system, including third party software, it is recommended to purchase the total system from a single supplier and make the acceptance tests appropriate. If a vendor does not agree to this (by agreeing to acceptance tests in the tender document) go elsewhere. It is worth noting that the end-user would usually have to pay a higher price for the software purchased in this way than if it was purchased directly from the third party (e.g. a software house). In general it is worth the extra cost in that one has a single supplier who is then responsible for the whole system performance (including third party software on their hardware). If hardware and software is purchased from different vendors and the system does not meet the requirements, one then has the problem of two parties to deal with, both of which blame the other for the poor performance.
Reputable companies now go to great lengths to try
to avoid selling systems which do not work. For example, Apollo UK has
a procedure where, before a salesman can send a quotation to a potential
customer, it is submitted to a department which examines the system requirements
against the proposed configuration. If the configuration is not viable
the system is enhanced and the quotation updated. This can lead to proposals
being outside a customers budget and hence lost sales, but it is better
than having irate customers with systems which do not work.
Once the system is installed staff will have to be
trained in its use. If experienced staff are available this can be done
in-house. Otherwise, as part of the tender document, the vendor will offer
training courses appropriate to various levels of staff, e.g. for system
staff who will look after the system or end-users who will use the software
packages.
Remember that once installed the system has to be maintained. Maintenance costs and the level of support offered will be specified in the quotation or tender documents from the vendor. This can range from an annual cost of a L100 for a small microcomputer up to L100000 for a large minicomputer system (in general hardware maintenance costs are typically between 7% and 12% of the hardware capital cost and software costs can be similar).
Medium to large computer systems would normally be
covered by a hardware maintenance contract which covers parts, labour and
specifies response onsite within a specified time of a reported fault (which
could leave the system unusable). This could range from immediate response
at any time of day (for a critical system) to next working day response.
The majority of small microcomputers have a 'normal' domestic guarantee
with one year parts and labour cover with return of equipment to the supplier.
Software maintenance can include 'on-line' help, bug fixes and new version
as they become available.
10 Examples showing pitfalls which can occur in the purchase of systems
10.1 Minicomputer for an engineering application (1980)
An organisation had a requirement for a three to four user system to be used to develop CAD engineering programs using finite element techniques. The programs in question were performing calculations on complex structures and presenting information graphically, with total program and data size being approximately 500Kbyes. The organisation purchased a small minicomputer with 1Mbyte of memory running a version of the UNIX operating system. The staff in question had had wide experience of computer systems including UNIX (running a virtual memory environment). Unfortunately, they had never asked the salesman if the particular version of UNIX on the proposed system was a virtual memory environment. It turned out that it was not, making it impossible in practice for more than one user to effectively use the system.
10.2 A small commercial database system (1985)
A organisation required a database system to be implemented
using a high-powered fourth generation language. The database was to store
records on 5000 to 7000 people and allow up to 10 clerical staff making
simultaneous enquiries or updates. An order was about to be placed for
a small minicomputer at a cost of approximately L40000 when one member
of staff become worried about the processing power of the proposed system
(no benchmark tests had been carried out or acceptance tests specified).
The proposed system was rated at approximately 1 MIP whereas the database
vendor recommended a third of a MIP per user. A benchmark test was arranged
during which the system crashed when four users made simultaneous enquiries.
In addition, when the system was restarted, it was found that the database
was corrupted beyond recovery. Powerful database systems use large amounts
of computer resources, and clearly, for this number of users, required
a machine several times more powerful than that proposed. If the benchmarks
had not been carried out the crash could have occurred when the system
was in use. After further work the organisation ended up purchasing a system
costing L250000 rated at 3 MIP.
This paper described some of the pitfalls which may be encountered when purchasing computer hardware and/or software. Techniques are described which assist in avoiding the most obvious problems and enable the purchase of a system which will meet the end-user requirements.
The use of the techniques used in this paper does not guarantee 100% satisfaction. Problems which commonly occur are: