Experimental Evaluation In Computer Science

EXPERIMENTAL EVALUATION IN COMPUTER SCIENCE

Abstract:

By analyzing more than 400 research articles, journals, conferences and articles taken randomly from ACM in 1993, it has been observed that researches published most of their work without any experimental results or proofs. The research papers related to Optical Engineering (OE) and Neural Computation (NC) were also studied. It has been found that 40% of these papers do not perform experiments to validate their claim. The ratio of lack of Quantitative Evaluation is 15% in OE is and 12% percent in NC. The journals that focus on Software Engineering field and that do not perform experimental evaluation are more than 50%. The research papers that allocate one fifth or more of their space to

40% of articles do not provide results on the basis of experiments. For software engineering field the percentage is even higher. There are very few number of articles that claim their result purely on experimental evaluation. As a whole this situation is freighting for CS field. Our study showed that Computer Science do not meet the standards that are followed by Engineering and Natural Sciences. In our study the standards followed by Neural Computing are much better than that of CS, despite the fact that NC is only six years old. One of the alarming observations made in the study is that the ratio of scientists in CS is less as compared to NC and OE. Facing difficulties for experimental evaluation is another factor in CS. Although psychologist have developed techniques to deals difficulties faced in experimental evaluation, but unfortunately CS has not yet welcomed those techniques. These techniques are also very costly and complicated to be tested by CS scientists. CS labs also lack facilities for experimental evaluation of their own results. Scientists face difficulties in developing equipments to prove their claim solely based on experiments. Question arises why to go for experimental results if papers are acknowledged without it. Experimental based evaluation also fear for damaging career. We all are well aware of the fact that experimental work is

Like these fields Computer Science also used observations and modeling to implement the new ideas. Over the past years, no matter the empirical research has been conducted in the field of Computer Science, but the ratio is still very low. Another problem in the field of Computer Science is that if a certain claim / problem is addressed then it will be valid in limited scope and in specific scenario. At last but not the least certain steps should be required to encourage researchers to carry out research on solid experimental basis. In this regard cultural change is also required both individual and community level. Certain Steps are also required for experimental evaluation in the field of Computer Science. One possible step would be to create forums that encourage papers that focus on experimental work. Conference on Experimental Compauter Science, Journal of Empirical Software Engineerin, ACM Journal of Experimental Algorithm, Workshops on Efficient and Experimental Algorithms are such examples that promotes experimental research. Another problem is that all of the above mentioned examples to promote experiment work, focused narrowly and only contains limited number of papers. Besides all the limitations and problems of conducting experimental work, it has been