Shortcomings of Evaluation Worksheets for Scientific Art Articles in Iran Based on Merton's Science Norms

The scientific journal assessment worksheets are the most important tool for evaluating the quality of scientific papers. The purpose of this research is an objective and qualitative description of indices used in the worksheets for the evaluation of art scientific research journals in Iran and to acknowledge their shortcomings in comparison with the norms of science from the Robert King Merton's perspective. The research approach in this study is combining survey and content analysis. Statistical samples consisted of nine worksheets developed for the evaluation of specialized art journal articles with a scientific research rank. Moreover, 14 experts in the fields of Scientometrics and art were invited to provide feedback on the extent to which the evaluation criteria used in the evaluation worksheets are in conformity with Merton’s science norms. Data collection was done in two forms including library research, referring to scientific journal databases, and structured interviews. In order to uncover the existing status of the indicators from the researcher-made check list, Excel software and a questionnaire were used as research instruments. The collected data were analyzed by descriptive statistics along with relevant tables and charts. Findings of the research show that out of the total 53 existing indicators, the index of "using sufficient and new valid sources (internal and external)" had the highest frequency (77.78%). The findings also indicated that the other 26 indicators had the lowest frequency percentage (11.11%). Moreover, these indices are consistent with the six out of seven of Merton's science norms (less than 18%). The obtained results revealed the unbalanced distribution of components and indicators of evaluation in these worksheets and their non-conformance to the norms of science, necessitating their revision.


Introduction
Scientific journals are regarded as the most important ways of scientific communications. Being methodical in presenting and transferring the findings and contents is regarded as an important characteristic of these types of research. On the other hand, the quality assessment indicators of scientific articles are the most important instruments through which scientific journals, make comments about the cost and validity of the researchers' scientific reports. The articles' evaluation worksheets, in scientific research journals of Department of Art, like other scientific journals, contain the information, used before publication to judge and assess quality of scientific articles in the specified field. The important point is that the components and indicators, which have been used in each of the evaluation worksheets of art scientific-research journals, have varied quality and quantity. So, the present research seeks to answer two questions: 1. What kind of indicator is used in art scientific-research journals to evaluate the scientific articles in worksheets?
2. What are the shortcomings of these indicators comparing to the Merton's science Norms?
This suggests that the current research seeks to identify and analyze the components and indicators of quality assessment of the articles in scientific-research worksheet journals in the department of art, using the content analysis method and the experts survey. The significance of this study are 1) not conducting the research similar to the current subject, 2) Incompatibility of the available evaluation indicators with the science Norms, 3) The important role of journal reviewing forms in increase of art studies contribution to production of science in the country.

Methodology
The methodology of this study is mixed survey and content analysis of measurement and analysis of content. The research population consists of A) 9 evaluating worksheets of Iran's art specialized journals: 1) Fine Arts, 2) Journal of Visual Arts and Applied-Arts, 3) Negareh Journal, 4) Comparative Art Studies, 5) Goljam, 6) Kimia-ye-Honar, 7) Islamic Art Studies, 8) Theater, 9) Bagh-E-Nazar, and B) 14 experts of Scientometrics and art fields. The data collection was done through library, referring to the Scientific Articles Citation Center, and valid databases. In order to extract the present situation of worksheets evaluating indicators, the researcher's checklist and the structured interview were used as the research instrument. The regarded checklist was prepared and set based on the existing evaluation indicators in evaluation worksheets on art scientific-research articles. The content analysis was done by inductive coding and then by deductive coding method. In this regard, to identify the existing indicators frequency in an article's evaluation worksheets, the inductive coding was used. This kind of coding is done by considering frequent data and embedding templates (Sarokhani, 2006, p. 473).
In this phase, to extract the frequency of existing indicators in evaluation worksheets, a checklist (according to table 1), including of 11 categories and 53 indicators was prepared. This checklist in the form of matrices consisting of 9 scientific articles evaluation worksheets and 11 categories and 53 indicators, which was prepared, and provided to two members of Faculty Board of Art Research Department and Scientometrics Department, to announce their own comments on the suitability of components and indicators, regarding the research goal. To obtain the content validity and specialized judgments, the checklist and questionnaire were provided to two professors of the scientometric group. After the compilation of the mentioned expert's corrective comments, the existing indicators in the checklist were corrected and rewritten. For determining the reliability, the validity and accuracy were done with coding method. For this purpose, the worksheets were provided to the second and third coders. To determine the extent of agreement and consistency of coders' comments, the Cohen's kappa correlation coefficient was used. The agreement extent between the coder's comments was found positive 78%.

Research background
The intended problem of this research, except for the art field, has been investigated by the researchers several times in other science fields. Regarding the topics related to the present research, we can point out some of these studies: the purpose of research done by Afshari, Mahram, and Noghani (2013) was to analyze 35 valuation worksheets and redraft the scientific-research articles quality assessment indicators of humanities based on science norms. The results were indicative of the undesirability of all specialized evaluation worksheets in this area of the country. Also in his article, Arastoopour (2012) has analyzed the theoretical background of issues, related to the forms and the checklist of article's inspection and statement of some of their problems. He believes that some factors like domain and content, Research Methodology, the way to present, set and fit with the goal and policies of publication are found in all journal reviewing forms. Also, according to educational research qualitative criteria reports in different fields (European Commission, 2009), 100 questions or even more than that can be found in some parts of reviewing forms. In connection with this difference, Rockwell (2005) points out that, some of the publications, have higher standards and for this reason they should inevitably pay attention to details. It eventually leads to longer reviewing forms. According to Goldin & Ashley (2010), the reviewing results greatly differed on the criteria intended for the publication. And it is possible that one article gets better scores using specific indicators of a form, while it does not get the required score for publication according to the reviewing form criteria of another journal. The European Commission (2006), having conducted a research entitled Quality Indicators of European Educational Research concluded that for developing the indicators and the new method that can be used to determine the research quality of scientific publications, there is a need for collaborative and evidence-based efforts.
These indicators are effective not only for research efforts and scientific publications but also in research budget policies and doing evaluation, the scientific progress and research orientation policies and plans. According to the research conducted by Hames (2007), 97 percent of less experienced judges, prefer the Analytic Dichotomous evaluation checklist while some tend to present their opinions in writing. Qane-ei- Rad and Qazi pour (2002) in their research, analyzed the effect of the science norms on the faculty board member's productivity rate. The findings of this research showed that individual normative commitment affected these people's scientific productivity rate more than organizational normative commitment. Steinke (2003) in response to this question that if the reviewing forms could be an effective step toward the improvement of reviewing process quality or not states that this issue totally depends on the reviewing process, reviewing forms, prepared forms, content and the article type and the reviewer's expertise. Ershad et al. (2005) also in their article analyzed the journal reviewing documents on Iran sociology. This study focused more on the most important problems of articles from the reviewer's viewpoint. Fabes et al. (2000) in their research, studied the factors affecting research, prioritization and assessment. The results showed that there would be the external pressure to use the new criteria at the time of evaluation due to importance of science in the twenty-first century, and this era requires formulation of the current and new forms of research validity assessment criteria. The analysis of the research background shows that there has been no separate research on the current subject field yet. Regarding this, the present study was done for the first time.

Theoretical Fundamentals of Research
In intellectual system or sociology of the Merton's science, the concept of "Ethos of science" or what he also called "Inborn requirements" or "Norms of Science", is regarded as a key concept. The concept or the theory which Steve Fuller calls some kinds of "an invisible hand in science work" (Fuller, 2000). The Robert Merton's pattern is one of the initial conceptual frameworks for setting the moral rules and normative frameworks in science (Cole, 2004). Restivo (1995) believes that, in Merton's pattern, science is considered as a social subsystem in which, the structure and social framework of science are intrinsically stable and ordered. Merton regards the institution of science and the related researches, as the necessary instruments to supervise and control the scientists' behaviors. The science customs require a balanced set, including values and norms for the men of science, these norms were regarded as copies, recommendation, preferences and authorized matters, and their institutional value has the complete legitimacy (Merton, 1972). According to Merton, the science methodological rules, in fact, should contain technical-specialized instruments and their moral requirements altogether. He asserts that the science institution grants rewards to those scientists, who are more faithful to the science norms than science counter-norms (Merton, 1973). Seven intended principles of Merton are given Table (1).  (Bucchi, 1997;Merton, 1973).

Concept Merton's science Norms
All scientists should have common ownership of scientific goods (intellectual property), to promote collective collaboration; secrecy is the opposite of this norm (Merton, 1973). Communism All scientists can contribute to science regardless of race, nationality, culture, or gender (Merton, 1973). Universalism According to which scientists are supposed act for the benefit of a common scientific enterprise, rather than for personal gain (Merton, 1973). Disinterestedness Requires that scientific claims contribute something new, problem, a new approach, new data, a new theory or a new explanation (Merton, 1973). Originality Skepticism means that scientific claims must be exposed to critical scrutiny before being accepted (Merton, 1973).

Organized Skepticism
Humbleness and modesty: this norm emphasizes on the observance of humbleness and modesty in the society. Such as 1) Appreciation of the knowledge heritage which remains from the ancestors. According to this, the analysis of the research background is not only an instrument for the advancement of research, but also it is somehow respecting to whom have presented a method for research. 2) The scientific claim on individual weaknesses and scientific knowledge limitations, for instance, Galileo always recommends himself and his student to Humbleness / Modesty say "I don't know".
The scientists must try to recognize other's merits. Being recognized and reputation is, in fact, a symbol or somehow a reward for a good function within the science men. Merton distinguishes between being honorable recognized and instrumental recognition. He also pays attention to some factors like scheduling, colleagues browse and etc. in addressing to the norm.

Recognition The Scientific-Research Journals of Art Department
This group of scientific journals, which cover 2.14% of the total approved scientific-research journals confirmed by Ministry of Sciences, Research and Technology of Iran (Pashang, Nourmohammadi & Nourrouzichakoli, 2015, p. 12), publishes the researchers' findings in this field. The scientific journals in this field usually review and accept the articles according to their general policies and policy formulation.

Research findings
Diagram (1) shows how much each of the 11 components have been used in the scientific articles evaluation worksheets. According to the data of the above mentioned diagram, the components of "title" and "discussion and conclusion" with the rate of 17% had the highest frequency and components of "morality" and "content" with the rate of 2% had the lowest frequency among 11 components. After that the components of "abstract" and "references" with the rate of 13%, "introduction" and "findings" with 9%, "review of literature" with 8%, methodology with 6 % and "writing" with 4% had other frequencies.
Diagram (1): The frequency distribution percent of each component in total art scientific-research journals evaluation worksheets Since the absolute frequency of the numbers of observed randomness in sample or relative frequency such as percent's of the sample size are the most common ways of data presentation as frequencies (Krippendorff, 2014, p. 148), all of the existing worksheets were studied through observing the documents in order to identify and extract the existing evaluation indicators and categories and also achieve their frequency, in these worksheets. Table (2) shows the observed components, indicators and the percent of their frequency in the worksheets. According to the above mentioned table data, totally 11 categories and 53 indicators were extracted from the evaluation worksheets (9 worksheets). As it can be seen, the indicator of "using the new and sufficient reasonable references (internal or external)" with the rate of 77.78 %, had the highest frequency, and the other 26 indicators such as: "Determination of methodology", "statement of the problem or the total goals", "significance of research", theoretical framework (proportionate to the research hypothesis)", "statement of the innovative aspects of research in conclusion" and … with the rate of 11.11 % has the lowest frequency in evaluation worksheets. Tables 2 & 3 show the frequency percent and adaptation of each of the evaluation indicators which are used in all of 9 art-science-research journals with the science norms, are being used. As noted above, totally 53 indicators of available evaluation worksheets adapted to 6 norms of Merton's science, including "Communism", "Universalism", "Disinterestedness", "originality", "Humbleness/ Modesty" and "organized skepticism" and it doesn't show adaptation with another norm of science i.e. "Recognition".

0%
According to figures given in Table 3, totally, the norm of "Communism" with the rate of 32.07% had the highest frequency among the evaluation indicators of 9 worksheets. After that, the norms "disinterestedness" and "originality" with the rate of 09.43% had the second percentage of the frequency distribution. According to figures of the mentioned table, the norms "organized skepticism" with the rate of 03.77 %, "Humbleness and modesty" with 01.88 % are placed in the next ranks of frequency percent, respectively. But, according to data of this Table, the norm "recognition" shows the 0.0% of frequency.

Available gaps in the art scientific research evaluation worksheets, based on science norms
In this section, according to the analysis of the data obtained from questionnaire and interview with experts and theorists, the available gaps in art scientific evaluation worksheets journals were identified and extracted based on the components of science norms. Totally, 13 gaps were obtained through activities related to this section as described in Table 4.

Discussion
This analysis shows that among the total 53 common indicators in 9 scientific-research evaluation worksheets of the art department, the indicator of "using the new and sufficient reasonable references, (internal and external)" which has the highest frequency in worksheets with the rate of (77.78%), is the only indicator of the "references" component.
However, the indicators which show the least rate of usage in worksheets (26 indicators), are scattered between 10 components of the total 11 common components in worksheets (except the "title" component). This shows that the scientific publications of art field, regard the indicators of reviewing articles, instead of new knowledge generation indicators, which is as a result of original research work. In general research work, which leads to the generation of knowledge, the references are applicable only for writing theoretical fundamentals and background, and the multiplicity of the references is not an evaluation criterion for such articles.
Also, based on the data analysis of Table (2), after the components of "title" and "discussion and conclusion", "references " and "abstract" with the frequency rate of 77.77%, and "introduction" and "findings" with the frequency rate of 55.55% have the second and third percent of frequency among the evaluation components, respectively. After that, the component of "literature" with the frequency rate of 44.44%, "methodology" with the frequency of 33.33% and "writing rules" with 22.22% are in the next orders of worksheets attention, respectively. Data of Diagram (1) also shows the percent of each component among total worksheets with the same frequency ratio while it is of special importance to pay attention to categories such as morality and content to perform and assess the scientific articles in multiple dimensions.
No article in the field of art was found to compare its results with the present study but the findings of this research can be compared with only one of the studies in the field of humanities. In comparison of the obtained findings of the recent study with research by Afshari et al. (2013), 41 indicators were identified and extracted from the total 35 human science evaluation worksheets journals, and 53 indicators were identified and extracted from the 9 worksheets of art journals, but both departments use 11 components for the evaluation of the scientific articles. In this comparison, most attention of human science field evaluation worksheets is paid to the component of "methodology" and the least attention is paid to component of "content", but according to the previous findings, the conditions are totally different in art evaluation worksheets. The review of the related literature indicates that no attention has been paid to the evaluation of evaluation worksheets of science in this field as an applied research. Perhaps, this issue gradually leads to isolation, invalidity and inefficiency of the art research in the field of science generation.
The comparison of the available indicators in evaluation worksheets with Merton's science norms also showed that, despite observing 6 out of 7 norms among them, there are many shortcomings in different parts of these worksheets.

Conclusion
Research in the field of art like other science fields also needs growth and quality improvement. Among them, the available indicators in evaluation worksheets, which are the most important instruments for quality assessment in articles for rejection or publication, have an important role in art science-research journal's quality improvement. Paying too much attention to or neglecting some important indicators for evaluation is the cause of weakness in content and even structure of some worksheets. While the quality improvement of science journals depends on the focus of science generation in that field, and this may lead to the decrease of quality and manner of reviewing the findings and results of the scientific reports of the researchers in this field.
The content analysis and survey of the specialists in scientometrics and art showed that there was an imbalance and considerable shortcomings in the distribution of components and available indicators on evaluation worksheets in this group of science journals of the country, and also there is a complete incompatibility in science norms. Therefore, in order to increase the quality of evaluation and reviewing of research in art department, it seems necessary to review components and indicators on evaluation worksheets of science-research journals in this field. It is clear that this review increases the role and contribution of art's research in science generation in the country, in addition to the improvement of reviewing quality, pleasure and assurance of researchers about the reviewing quality. The article recommend the following suggestions;

Suggestions
1. Regular and periodical evaluation in reviewing and the evaluation instrument for scientific articles.
2. Identification of the available evaluation worksheets shortcomings in art scientific-research journals, with help of expertise.
3. Reviewing available indicators in evaluation worksheets, based on science norms and customs.