Problems Found During Operation

During the administration of the follow-up survey, some problems were found and certain measures were taken to solve them.

Problems with the measurement instrument for the follow-up survey

As shown by the data collected from the follow-up survey, high marks were gained by nearly all the graduates on the indicator "morality and professional ethics", and there was only a slight difference between each graduate's "morality and professional ethics". This may have resulted from the inclusion of indicators hard to measure, such as whether they were dedicated to work, whether they were honest and faithful, whether they were enterprising, etc. This means that further modification needs to be made with regard to this indicator. It is worth mentioning that great care should be taken while making judgements about whether graduates have improved in terms of morality and professional ethnics after they receive a certain educational programme, unless a thorough comparison is made between what they were like before and after the programme.

Problems with the administrative process of the follow-up survey

• Low motivation in participating in the conducting of the follow-up survey
  The follow-up survey of the graduates and their employers was conducted in the form of a research project and all of the participants involved were members of the national research team for the survey. However, at the initial stage, the team was not formally established and as a result, some of the PRTVUs did not have financial support for the project, thus leading to low motivation in participation. To solve this problem, CCRTVU finally formed a national steering group, led by the president of CCRTVU and researchers from the CCRTVU Distance Education Research Centre, which functioned as the leader directing the research teams established at PRTVUs. The whole project was then well funded and a series of training sessions through various kinds of media were held to guarantee the smooth operation of the project.
   
• The inability to reach some graduates
  On registering with local RTVU study centres, students are usually required to provide their personal information including demographic information such as age, education background and work, etc. A student ID number is then assigned to each of them. The graduates for this study were sampled based on their registration number, but some were unable to be reached mainly because they changed jobs and did not inform the study centre of the change. Consequently, the selected graduates who were unable to be reached were replaced by the graduate with a registration number one before or after theirs.
   
• The inability to reach some employers
  Some of the employers were unable to be reached because some students did not provide information on their employers when registering with the RTVU study centre, and others because some graduates were unwilling to let their employers complete the questionnaire. As a result, the questionnaires for these employers had to be given up, thus achieving a valid employer questionnaire to valid graduate questionnaire ratio of 1:1.6. Otherwise, a comparison between the graduates' and their employers' questionnaire results could have been made, which could have led to more interesting analysis.
   
• Invalid questionnaires
  During the administration of the questionnaires, some graduates and employers did not read carefully the requirements for the completion of the questionnaires and did not treat the questionnaires seriously. Some of the questionnaires completed even had the same answers to all questions, thus becoming invalid. To avoid the collection of such questionnaires, the steering group specially designed a model for identifying invalid questionnaires. Explanations were also made to the participants as to what were invalid questionnaires, which ultimately helped reduce the workload of entering and analysing the data. It should be noted that the collection of invalid questionnaires may have resulted from the subjects' inability to see how they themselves might benefit from the research. To increase the number of valid questionnaires, it might therefore be better to explain the purpose of the survey with regard to its usefulness to the graduates and their employers in specific ways.

Recommendations on the Mechanism

The feedback mechanism was intended to be used on a regular basis. On the one hand, the feedback mechanism can be modified in terms of its measurement instrument, and thus the validity and reliability of the results collected using the mechanism can be gradually improved. On the other hand, a series of data collected by regular use of the feedback mechanism might be able to reflect the changing pattern of the quality of distance education or to discover with more or less regularity problems for improvement. To sum up, it is desirable for the feedback mechanism to be employed regularly. However, some factors need to be considered when making decisions on using the feedback mechanism as discussed below.

Timing of the survey

The mechanism established in the survey proved to be effective and valid for its research purpose. However, as for how to take advantage of it and how frequently the survey should be conducted, three main factors should be taken into account, i.e., purpose, cost and workload.

The purpose of the survey needs to be considered when deciding when to conduct a follow-up survey. If the survey is intended to obtain information on the quality of the programme students have just undertaken, it is strongly advised for it to be conducted soon after graduation; otherwise it is not easy to locate their immediate frame of reference and experience with the whole programme. In addition, if the survey is undertaken a few years after graduation, then many graduates would be making assessments on teaching which they experienced up to five years previously. Apart from recall issues, this would reduce the value of the information since there is a perception that quality can change over time (and the content of the programmes also changes). It is therefore suggested that such surveys be conducted soon after students' graduation. If the survey is intended to gain information on the impact of the programme on their career, then there is broad consensus that the survey would need to be conducted at least two years after graduation. However, this would create other problems, such as the logistical problems in contacting graduates two years after graduation. Furthermore, it is hard for graduates to disentangle the impacts of their distance education programmes from other influences such as subsequent in-service learning and in-house training.

Taking the financial factor into consideration, a biennial survey might be appropriate since this would reduce costs and the administrative burden on RTVUs. The main argument against a biennial survey is that the information would be less up-to-date since the quality of CCRTVU distance education programmes could change over a period as short as two years, for example, due to managerial changes or rapid expansion. It is therefore recommended that if only a biennial survey can be conducted due to limited budget, then researchers should take into account whether there have been changes to the distance education programmes during the last two years.

What kind of information should be collected by this type of survey

Since this is a follow-up survey of graduates and their employers, it is more appropriate to obtain information on the general rather than the specific aspects of the education programme. For the graduates, feedback on the whole distance education programme is a better choice than feedback on a certain individual module, which they might not be able to remember clearly. For their employers, feedback on the graduates' overall work performance is a better choice than on certain specific behaviour, which might not be useful for making judgements on the quality of the graduates.

How to maximize the response rate

There are many ways to achieve higher response rate, as follows:

• Conduct the survey soon after graduation, preferably before they get the diploma. Then contact addresses will be most accurate and a higher response rate can be achieved.
• Deliver souvenirs or small tokens to motivate the subjects for the survey.
• Create a data bank of students' personal information when students register with study centres, which will make it much easier to trace them later for the follow-up survey. It should be noted that the data bank needs to be frequently updated to ensure the accuracy of the graduates' contact details.
• Specify how the questionnaires will benefit the graduates and their employers. This can help enhance the response rate and above all can largely guarantee the validity of the questionnaires.

Due to many uncontrollable factors, it is strongly advised that more subjects than required be selected to guarantee that the collected data set is large enough for making judgements.

Forming a professional team for conducting the survey

It is essential to form a professional team for conducting the survey. To achieve this, the participants involved in conducting the survey should not only be familiar with the education programmes being researched, they should also get familiar with the whole administrative process of the survey through a series of training sessions. Only in this way can the key stages of the administration of the survey, including the sampling, delivery and collection of questionnaires, interviews, data entry and processing be guaranteed to operate smoothly.

How to report and use the results

When reporting the results of student surveys within the distance education institution, the results should be disaggregated to the level of individual programmes, because one of the primary purposes of getting the information is so that the quality of each individual programme can be improved. Also, measures should be taken to trace and evaluate the effect of teaching improvement to ensure the effective use of the results.

How to minimize the cost of the survey

There are many ways to minimize the cost of the survey, including:

• Conduct the survey less frequently, for example, biennially rather than annually.
• Commission the work to a professional third party, such as in the case of the British Open University.
• Make use of the existing national network for a large scale survey, such as in the case of CCRTVU.

Conclusion

In conclusion, it is very important to establish a mechanism for collecting feedback from graduates and their employers, since it can help provide the institutions concerned with useful information from the "customers", which can be fed into the improvement of teaching. However, many factors need to be taken into account if it is to be used on a regular basis on a large scale, such as when to conduct such surveys, what kind of information to collect, how to maximize the response rate and minimize cost and labor, etc. Regular, effective use of the mechanism will surely contribute a great deal to the improvement of the teaching quality in distance education.