Research on Equipment Support Capability Evaluation Based on

Meta-Evaluation Theory

Shuai Wang, Yabin Wang

and Jinguo Wang

Shijiazhuang Campus, Army Engineering University of PLA,

Shijiazhuang 050003, China

Keywords: Meta-Evaluation Theory, Equipment Support Cpability, Evaluation Process.

Abstract: Equipment support capability evaluation is an important link to promote the construction and development of

equipment factory support, and it is an important starting point to test the level of equipment factory support

capability. The meta-evaluation of evaluation scheme, evaluation achievement and evaluation report can

effectively improve the pertinence and reliability of the evaluation of security capability. In this paper, the

meta-evaluation theory and the practice of equipment support capability evaluation are combined, and a set

of relatively standard and integrated meta-evaluation process concept of equipment support capability

evaluation is proposed, in order to promote the effectiveness of equipment support capability evaluation

through the introduction of meta-evaluation.

1 INTRODUCTION

Equipment support capability is an important

capability index of equipment factory. Equipment

support capability evaluation is a scientific evaluation

of the generation, maintenance and application of

equipment support capability. It is a series of

theoretical understanding and practical testing

processes to draw objective and comprehensive

conclusions on equipment support capability, and an

important means to test the level of equipment

support capability building.

With the increasing scientific and technological

content of equipment and change of equipment

support tasks, the evaluation of equipment support

capability is facing new situations and tasks, among

which the key task is to establish a unified equipment

support capability evaluation system. However, due

to the influence of objective factors, there will

inevitably be errors and deviations in the process of

equipment support capability evaluation, which need

to be corrected and improved by monitoring and

inspection. This makes it possible to apply meta-

evaluation theory in equipment support capability

evaluation. In addition, meta-evaluation can verify

the correctness of equipment support capability

Corresponding author

evaluation theory, correct and perfect the

contradiction problems in the evaluation, and

promote the formation of scientific equipment

support capability evaluation theory through the

analysis and inspection of evaluation scheme,

evaluation process and evaluation results.

2 META-EVALUATION THEORY

Meta-evaluation theory was first introduced by

Michael Scriven, an American expert on evaluation

theory, in an article published in 1969 in response to

the question of how to evaluate evaluation tools (Cai,

Mu, Feng, et al 2021).The theory holds that

systematic evaluation of the evaluation itself can

provide the original evaluation with its own

feasibility, correctness, accuracy and other related

information, which helps to timely detect and solve

the problems encountered in the evaluation process

and ensure the quality and benefit of the evaluation

activities.

Wang, S., Wang, Y. and Wang, J.

Research on Equipment Support Capability Evaluation Based on Meta-Evaluation Theory.

DOI: 10.5220/0012070200003624

In Proceedings of the 2nd International Conference on Public Management and Big Data Analysis (PMBDA 2022), pages 77-82

ISBN: 978-989-758-658-3

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

2.1 Meta-Evaluation Rationale

Meta-evaluation is an evaluation activity that takes

the original evaluation activity as the main evaluation

object and re-evaluates the evaluation activity,

evaluation system, evaluation system and evaluation

mechanism. It is an evaluation activity that evaluates

the scientific nature of the original evaluation activity

itself and the behavior of the evaluation subject and

object in order to improve the evaluation quality and

effect. Meta-evaluation is an important means to

strengthen the construction of evaluation system and

perfect evaluation mechanism, and is the guarantee of

scientific and rational evaluation work.

Taking meta evaluation as an important link in the

equipment support capability evaluation process, re

evaluating important sub links in the equipment

support capability evaluation process, giving play to

the evaluation, monitoring and correction functions

of meta evaluation, can standardize and measure the

evaluation itself, improve the quality of equipment

support capability evaluation, ensure the quality of

evaluation, and form scientific and authoritative

evaluation results, Realize the value goal of service

equipment support capacity construction (Zhang

2020).

2.2 Meta-Evaluation Application

Strategy of Equipment Support

Capability Evaluation

The operation mode of meta-evaluation (Zhang,

Chen 2017) is generally divided into two types. One

is formative meta-evaluation carried out

synchronously with the evaluation activities,

emphasizing meta-evaluation at the connecting node

in the evaluation so as to timely correct the problems

in the evaluation process. The other is the summative

meta-evaluation organized after the end of the

evaluation activity, emphasizing the overall analysis

and evaluation of the evaluation activity as a whole,

with the goal of providing opinions and suggestions

for the final decision. Meta-evaluation of equipment

factory support capability evaluation can integrate the

advantages of the two modes, and realize the control

of evaluation activities by integrating meta-

evaluation process into the node link of evaluation

activities and taking meta-evaluation results as

procedural normative documents of evaluation

activities. Specifically, it is to establish an iterative

process from the evaluation sub-process to the meta-

evaluation process, and then from the meta-

evaluation process to the evaluation sub-process, so

as to achieve the fit and integration of meta-

evaluation in the evaluation process, and ensure that

the evaluation implementation process is more

scientific, standardized, accurate and objective.

3 APPLICATION OF

META-EVALUATION IN

EQUIPMENT SUPPORT

CAPABILITY EVALUATION

Meta-evaluation t of equipment support ability

evaluation process is the essence of evaluation

activities and results for the object, to evaluate the

whole process of internal system evaluation plan

formulation, evaluation result evaluation and

evaluation reports etc, is synchronized with the whole

evaluation process, after the evaluation of the link,

can effectively improve the efficiency of the

evaluation of the quality of the work.

3.1 Evaluation Scheme

Meta-Evaluation

According to the meta-evaluation theory, the expert

group of evaluation should submit the evaluation plan

to the evaluation leadership group after completing

the evaluation plan of equipment support capability,

and the evaluation leadership group will review and

confirm the evaluation plan. The evaluation

leadership group shall review the content of the

evaluation scheme according to the evaluation

purpose, the evaluation plan and the relevant work

requirements, review the procedure of the evaluation

scheme according to the work arrangement of the

evaluated unit, point out the existing problems, put

forward suggestions for improvement and determine

the evaluate conclusion. According to the evaluation

conclusions, the evaluation expert group shall re-

formulate the evaluation plan or make targeted

modifications and improvements until a more

scientific, standardized and feasible evaluation plan

is formed.

3.2 Evaluation Results

Meta-Evaluation

Meta-evaluation of evaluation results is the analysis

and verification of evaluation results and their

generation process. The results of equipment support

PMBDA 2022 - International Conference on Public Management and Big Data Analysis

capability evaluation information data obtained

through model calculation need to be analyzed and

verified to be convincing and authoritative evaluation

results can be formed. In equipment support ability

evaluation practice, due to the information in the

process of data collection, model building and

calculation, the concrete numerical error in the

processing, the acquisition of information data and

evaluation index system and evaluation model of

demand, different evaluation methods to evaluate the

same object results may also be inconsistent, can lead

to inaccurate evaluation data results. Therefore,

qualitative and quantitative analysis, data scores and

system simulation verification methods are needed for

analysis and verification, which focuses on

verification and analysis of pertinence, authenticity,

objectivity, timeliness and accuracy of data sources,

acquisition process, data classification, calculation

process and other links. If it fails to pass the

verification, the analysis and verification results

should be fed back to the relevant links for

supplementary verification, modification, re-

collection, calculation, etc., to ensure the accuracy and

reliability of the evaluation results.

3.3 Meta-Evaluation of Evaluation

Report

Meta-evaluation of the evaluation report is the

evaluation and confirmation of the equipment support

capability evaluation report, which is an important

step in the end of the evaluation. It is divided into two

parts: formal review and substantive review. Formal

review mainly refers to the review of whether the

content and form of the evaluation report are

standardized and complete, and the review of the

effectiveness of the evaluation report. Substantive

review is to examine whether the core content and

data of the evaluation report are comprehensive, true

and accurate, and to examine the reliability of the

evaluation report. If the evaluation report fails to pass

the evaluation, the relevant situation should be fed

back to information collection, data processing, grade

generation, report writing and other links that may

have problems, and targeted measures should be

taken to correct, supplement or re-evaluate, and then

the evaluation should be carried out again until it

passes. After the evaluation report passes the

evaluation, the evaluation leadership group will

officially confirm the evaluation report and submit it

to the evaluation initiator, who will organize the

feedback and application of the evaluation results.

4 EQUIPMENT SUPPORT

CAPABILITY EVALUATION

PROCESS BASED ON

META-EVALUATION

Combined with the evaluation of application, on the

basis of evaluation practice and the related literature

research, this article explores to build a set of more

scientific and standardized evaluation process, to

evaluate the implementation of the whole process of

the equipment support by will involve activities

analysis integration, clear and complete description

in the form of framework, which provides the

standard basis for the evaluation practice. At the same

time, this paper discusses and studies the evaluation

indicators and standards, evaluation data collection,

evaluation normative documents and feedback

application that are widely concerned in evaluation

organization and implementation.

4.1 Evaluation Process Framework

According to the organization and implementation of

the evaluation, the equipment support capability

evaluation process can be divided into three stages:

evaluation preparation, evaluation implementation

and evaluation summary, including the following

steps: (1) clarifying the evaluation objects, objectives

and boundaries; (2) Establish evaluation institutions

and clarify the responsibilities of each organization;

(3) Formulating the evaluation program, which

mainly includes establishing the evaluation index

system, selecting the evaluation method, establishing

the evaluation model, and verifying the evaluation

model; (4) Review and confirmation of the evaluation

scheme; (5) Evaluation personnel training; (6)

Evaluation information data collection; (7)

Evaluation of information data processing; (8)

Generation of evaluation results; (9) Analysis and

verification of evaluation results; (10) Writing

evaluation reports; (11) Evaluation and confirmation

of the evaluation report; (12) The feedback and

application of the evaluation results. The basic flow

is shown in Figure 1.

Research on Equipment Support Capability Evaluation Based on Meta-Evaluation Theory

Figure 1: Process model of equipment support capability evaluation.

4.2 Evaluation Index System and

Standards

The evaluation index system of equipment support

capability should be based on the standard of

equipment support capability construction, combined

with the reality of equipment support task, guided by

capacity demand, and constructed around the key

points and difficulties of equipment support capability

and core keys. Based on the equipment support tasks,

the methods of mission capability decomposition,

hierarchical analysis, extraction ability factor, the

method of architecture design, can build cover the

command and control ability, maintenance support

ability, supply (allocate) support capability and data

support ability of equipment support ability evaluation

index system of (Zhang, Ruan, Li 2021).

Evaluation standard and evaluation index system

complement each other, without standard index does

not have scientific nature. However, the current

evaluation standards have not yet formed a unified

standard version. In the evaluation practice, relevant

standards and specifications need to be queried

according to the selection of evaluation indicators.

Equipment support ability is an important component

of the system operational capability, so the equipment

support ability evaluation criteria and related

professional post standard and construction standards

are consistent, should be combined with

characteristics of equipment support tasks at the same

time, learn professional post level standard, establish

a scientific system of rating standard system, so as to

realize more scientific and careful evaluation of

equipment support capability evaluation.

PMBDA 2022 - International Conference on Public Management and Big Data Analysis

4.3 Evaluation Data Collection

Evaluation data collection not only confined to the

static inspection, on-site examination or the combat

exercises using traditional methods in the process,

still need to strengthen the consciousness of data

mining and extend to the big data systems to data,

more memory data information to fully explore and

intelligent equipment information, rational utilization

of equipment support data monitoring system,

forming a complete coverage of evaluation

information data acquisition, Ensure the objectivity,

accuracy and integration of the evaluation data.

The collection and processing of evaluation data

is mainly carried out by the evaluator team. Due to

we have not yet present train formation evaluation

personnel team, also did not establish a third party

evaluation mechanism of the specialized institutions,

so in after confirmation of the evaluation scheme,

need to organize relevant evaluation staff

professional training organization, training contents

include the evaluation theory, evaluation

implementation, evaluation process, evaluation

method, etc., In order to make the evaluator clear

evaluation content, evaluation index system,

information data collection method and processing

requirements, ensure the smooth evaluation process

and accurate and authoritative evaluation results.

4.4 Evaluation Normative Documents

and Feedback Applications

Normative documents in the implementation of the

evaluation mainly refer to the process documents

generated by the evaluation functional groups in the

process of performing their duties. It mainly includes

evaluation plan, evaluation implementation plan,

evaluation implementation plan review report,

evaluation personnel training plan, information

collection plan, data processing plan, information

collection form, comprehensive assurance plan,

information data summary report, evaluation results,

evaluation results analysis and verification report,

evaluation report, evaluation report review conclusion,

final evaluation opinions, etc. The generation time in

the evaluation process is shown in Figure 2. In the

normative documents, the evaluation implementation

plan review report, the evaluation result analysis and

verification report, and the evaluation report review

conclusion are the process documents of meta

evaluation integration in the evaluation activities.

Figure 2: Normative document of equipment support capability evaluation.

The feedback and application of evaluation report

are the real realization of evaluation work value. After

the evaluation report is confirmed, the relevant units

or departments should be fed back in time, and the

communication with the evaluated units should be

strengthened to promote the application of the

evaluation results, so as to achieve the practical

significance of "promoting construction by evaluation

and combining construction with evaluation". At the

same time, the evaluation results should be used as an

important basis for task preparation, unit evaluation

and excellence, and personnel evaluation and

appointment, so as to stimulate the endogenous power

of equipment support capacity construction.

5 CONCLUSION

In this paper, blend in meta evaluation of equipment

support ability evaluation work, through the review

Research on Equipment Support Capability Evaluation Based on Meta-Evaluation Theory

of evaluation scheme, the analysis of the evaluation

result verification and evaluation of reliability

evaluation of existing evaluation activities and

effectiveness evaluation, giving play to the role of

internal supervision, realize the monitoring of the

evaluation process, to ensure that the evaluation of

the scientific, standardized, accurate and objective.

On this basis, the evaluation process of equipment

support capability is standardized, and the methods to

modify and improve the evaluation practice are put

forward. The follow-up work needs to conduct

systematic research on the evaluation operation

mechanism, innovate and optimize the data collection

methods, standardize and integrate the standards and

rules, and form a scientific and perfect equipment

support capability evaluation specification to provide

a basis for equipment factory construction.

REFERENCES

Cai Zhuohan, Mu Ge, Feng Qiqi, et al. Research on the

optimization method of evaluation index for

contribution rate of weapon equipment system based on

meta evaluation [J]. New Industrialization, 2021, 11(6):

238-240.

Zhang Jiping. The discipline assessment serves for “double

first-class” construction: dilemmas and paths of meta-

evaluation [J]. Modern Education Management, 2020,

(12): 63-71.

Zhang Congcong, Chen Dan. The application of yuan

evaluation theory in the social benefit evaluation of

publishing house [J]. Chinese Editors Journal, 2017, (9):

11-15.

Zhang Yaolong, Ruan Yongjun, Li Zhen. Research on

evaluation of synthetic travel equipment support

capability based on FAHP [J]. Command Control and

Simulation, 2021, 43(6): 71-77.

PMBDA 2022 - International Conference on Public Management and Big Data Analysis