In our response to the Department for Business, Innovation and Skills, we caution that TEF metrics must appropriately recognise issues around equality and access, and should be better defined and communicated. We support the focus on teaching quality.
Response submitted 2016
The proposed Teaching Excellence Framework (TEF) would use metrics, set in context by qualitative judgements made by independent review panels. This qualitative element is crucial to ensure that metrics performance does not incentivise behaviour that undermines the equality agenda. We note that parental income, social networks and student backgrounds are strongly linked to access to higher education and to employment outcomes, Metrics used in TEF need to recognise this appropriately to mitigate unintended consequences, whether in terms of recruitment of disadvantaged students, or disadvantages to institutions more effective in the recruitment of such students. TEF needs to look beyond teaching alone and considering the broader elements of learning environment and assessment processes that have direct impact on issues of equality and access.
Teaching and learning in higher education are of the utmost importance. There is an underlying assumption in the document that teaching is not valued as much as research, but evidence is not given to support this. In practice most university geography academics teach and this endeavour is valued in time, energy and effort.
Research and teaching quality are not independent. Teaching of geography in universities is for the most part research-led and benefits greatly from this. The best teaching and innovative curriculum design often follows the best research. It is a false dichotomy to pit research ‘against’ teaching.
There is no single measure that students use when making their choice and high quality courses can take many forms. TEF should be flexible enough to identify quality in different forms and places. Students will value information on specific subjects.
Metrics need to recognise differences in students - their backgrounds, experience, expectations and desired outcomes from higher education. Individual students will value different aspects of their degree experience – face-to-face contact hours, strong employability-focus, proximity to the best researchers, library facilities, etc. This may also vary between disciplines. A degree of flexibility in any metrics used to assess teaching quality must be present.
Employers are a heterogeneous group and their needs also are diverse. The employment destinations of graduates also are diverse - some students will pursue graduate careers ‘in’ their disciplines, others will draw on their transferable skills and find employment in a broad range of sectors and roles. Very careful attention needs to be given to engagement with employers (inclusive of large organisations and SMEs) and metrics used to document quality and success of graduates from their perspectives.
Yes. TEF assessments will need to acknowledge the diversity of the sector with appropriate levels of contextualisation. TEF at an institutional level will mask unevenness across subjects. Moreover, students will value information on specific subjects. However, devising a framework that allows for fair and robust comparison across providers, disciplines, modes and levels will be highly challenging and potentially burden-some.
Yes. We fully support commitments to increasing access and the achievement of students from disadvantaged backgrounds and under-represented groups.
a. what would constitute a ‘successful’ QA review
Yes. QAA reviews provide a credible and established baseline for TEF.
b. the incentives that should be open to alternative providers for the first year of the TEF
c. the proposal to move to differentiated levels of TEF from year two?
Yes. We support the proposals for differentiated levels of TEF. However, sufficient time must be taken to ensure the evidence base, metrics in particular, are robust and reliable across institutions. Full engagement with all sector stakeholders is needed in this process.
Timing - Yes.
Assessment panels - Yes.
Process - Yes.
Five year awards seem appropriate, with institutions applying in annual windows (i.e. not all at the same time). Every attention must be directed to the ‘burden’ of any new processes. Criteria need to be clear and transparent and there needs to be effective moderation processes to ensure consistency across panels
Independent experts – academic peers students, and employers – should make up the assessment panels. It is critical that TEF is credible to the public and the higher education community.
To establish a robust process, and the trust of the academic and teaching community, a new system could soon grow to the scale and cost of the equivalent Research Excellence Framework. Attention to costs, and reporting these, must be constantly kept under review to ensure the cost does not become too great relative to the potential benefit.
Applications and re-assessments should be maintained at a frequency that maximises ease of application and minimises complexity. TEF processes and metrics should be aligned with other assessments of quality (e.g. QAA or equivalent). TEF processes and metrics should take account of processes and data already in use in the day-to-day oversight of teaching and learning in HEIs.
Yes. We believe over time discipline/subject level scrutiny should be introduced.
No. The primary goal of a TEF must be to raise quality across the board and encourage innovative and effective practice, not to raise fees. It is important the link between the two is severed. Linking teaching quality to a system of funding could result in game-playing by institutions. The risk is that effort will be placed not on actually improving quality, nor sharing best practice (as encouraged in the current QAA system), but on securing greater fees by adapting to the system and achieving particular metrics.
Yes. We support the proposed focus on teaching quality, learning environment and student outcomes. However, we question whether robust and reliable measurements of learning gain, across all subjects and students, can be developed in the near-future. See our further responses to Q11, relevant to this question too.
The Green Paper does not specify in detail what will be measured in TEF. The extent to which the TEF will be regarded as an indicator of quality will depend on the robustness of the metrics and the methodology to be determined. We note:
Effective metrics must be valid, robust, comprehensive, credible and current. In a teaching system, attention needs to be directed to the quality of the inputs (staff expertise, training and accreditation, links to research, contact time (recognising this varies by subject) etc); the quality of the outputs (degree levels, completion rates, satisfaction etc); and the added value for students (learning environment, employability and learning gain). The current proposal does not adequately include all three elements.
We strongly encourage that evidence of the effectiveness of the TQA some 20 years ago is sought out where possible. If the evidence suggests it had no discernible impact on teaching quality, then we feel the current approach should be questioned given the time and money that will be involved.
The metrics that will be used to measure teaching excellence need much more attention and it needs to be communicated, very clearly, that these will be at best proxies. Any system needs flexibility and a degree of discipline specificity. For example in geography, fieldwork is a key element of the learning experience and metrics used must recognise this.
There is evidence that NSS, student satisfaction, is not positively linked with good learning outcomes and DLHE is linked to background and social networks and strength of regional economies. Benchmarking of any metrics needs to be transparent and fair.
A very important element of teaching and learning in higher education, particularly in the social sciences, is challenging students and exposing them to alternative perspectives and different ways of thinking about the world. This involves using a diverse range of teaching practices (seminars, labs, field courses etc). This can unsettle students, there are no ‘right’ answers and students are expected to be active participants in their learning. This learning experience may be as important as the learning outcome. Metrics need to recognise this.
There is little attention in the document for the need for students to be encouraged to ‘fulfil their potential’ by actively engaging with, and participating in, their learning/degree. It is not a one-way ‘customer contract’, but rather a mutual agreement between two parties, either of which might fail in their commitment.
Peer review of individual teaching performance is essential and interpretation either of existing metrics or of newly-designed and measured metrics should be carried out by expert panels who can interpret them in context.
It is important to ensure there are no unintended consequences in terms of the emphasis (p.27) on consumer protection law. This could stifle innovation as the course content and structure has to be set 18 months before students arrive and sustained until they leave (4.5 years, 5.5 years on a 4 year degrees). For a course like geography although core concepts remain, areas of interest change as the world changes. Such an approach could dull the possibility of new modules being developed and also, if there are personnel changes, mean that staff are not teaching to their strengths.
TEF will only apply to England. The higher education landscape is UK wide. Where will this leave Welsh, Scottish and NI higher education institutions?
Given the limitations of the metrics, it is important for providers to have the opportunity to explain the context in which they demonstrate excellence and to be assessed by their peers.
Yes. Widening access to higher education is essential.
There needs to be careful attention to ensure that those institutions who offer the greatest degree of access and opportunity do not, unintentionally, perform less well on metrics used in TEF. It is important to understand what allows students to thrive and to consider how these should be integrated into TEF (e.g. extra learning support).
It is unfortunate that there have been cuts to the student opportunity funding for the poorest and disabled students within the overall teaching budget.
Not sure. Targets should be set by the providers, based on their mission and context. It is appropriate for the Office for Students to review and work with providers to approve these targets.
Part-time and mature students to ensure the metrics are fully appropriate
Better understanding of the common barriers and challenges to individuals from disadvantaged and under-represented groups to target initiatives to address these.
Any additional administrative burden will depend on the data to be made available. The burden may not be uniform. Data that can serve multiple purposes are desirable.
Yes. All Higher Education providers operating in the field should enter the system through a single route.
The bar to entry must be high to protect the excellent global reputation of UK higher education. Teaching in higher-education should be research-led.
No option selected.
The legal framework and architecture must keep pace with the changing environment for higher education provision.
The architecture also needs to support postgraduate training (vital for the future health of research).
The OfS should rely on existing UK bodies with appropriate expertise.
It is important that expertise currently within HEFCE be captured in a new regulatory environment.
Option 1: BIS Ministers set strategic priorities and BIS officials determine formula
Option 2: BIS Minister sets strategic priorities and allocation responsibilities divested to OfS
We stress the importance of support for the full range of disciplines and adequate funding for those that have important high-cost elements. While the part-STEM nature of geography (along with psychology) has been recognised, teaching funding does not follow.
Not sure. We support the OfS as an arms-length public body, with a duty to promote students’ interests. We have questions about its proposed powers and stress the need for independence between assessments of quality and funding.
Research UK needs to respect the autonomy of the subject areas as currently recognised by the Research Councils and how they are configured. Strong disciplines are at the basis of cross-disciplinary research necessary to address global challenges.
We strongly endorse the current balance of research funding, with QR allocations from HEFCE providing a baseline funding across qualifying institutions on the basis of REF assessments, and funding on a peer-reviewed proposal basis through the Research Councils. These two streams serve two different and important purposes and each does it well.
If dual funding is operated in a single organisation (e.g. RUK), the independence and separate administration of the two funding streams needs to be embedded in a robust legal framework. It is vital that the REF be managed at arm’s length from the department. QR block grant funding should be allocated on the basis of an independent process underpinned by peer review, supporting excellence wherever it is found. The detail of this proposal should be the subject of further consultation. Moreover, in line with government commitments to 2015 levels of financial support for science being retained in real terms, the QR percentage of the total science funding should also be sustained at the 2015 level.
As a discipline, geography spans the natural sciences, social sciences and arts and humanities. Thus we recognise the essential importance of funding streams across broad areas and the significant contributions the social sciences, arts and humanities make to both pure and applied research. These must not be marginalised. Many of the nation’s and the world’s ‘grand challenges’, such as climate change, natural resource security and migration etc, demand an integrated understanding of the scientific processes, of human and behavioural (social science) perspectives, and of spatial differences and interconnections.
Research funding should not lose sight of the importance of funding high quality ‘blue skies thinking’ across all research areas, including STEM and the social sciences. There must be a balance between research tailored to assist with government policy priorities and the need for blue skies research.
Yes. See responses above.
The Research Excellence Framework demonstrates the world-class excellence of UK research, both in terms of academic quality and broader societal impact. This is very well illustrated in geography. Formal recognition of this supports the wider international uptake of UK geography through publishing, conferences, training of international students etc. Some of these activities directly support the RGS-IBG.
The REF is an important element in the distribution of funding under the dual-support system by all four of the UK higher education funding bodies.
We urge universities to review their costs to reduce bureaucracy and burden (given estimates suggest that 95% costs are incurred in universities).
The responses of the RGS-IBG to consultations dating back to 2008 have been critical of the use of metrics in research assessment, with a strong preference for peer review. There is no evidence we are aware of for metrics, established or new, that can adequately capture originality, significance and rigour of academic outputs. We do not believe citation data, or other variants, quantify ‘quality’ independent of peer review (see Analysis and evidence to this effect is presented in Richards, K. et al. (2009). The nature of publishing and assessment in Geography and Environmental Studies: evidence from the Research Assessment Exercise 2008. Area, 41(3), 231-243). More recently, Wilsdon has shown that automation of research assessment would fail to adequately assess quality. We continue to have concerns about issues of equality – particularly for young/emerging scholars. We know from the citation and readership data for papers in our own journals, ‘established’ ‘recognised’ disciplinary figures are privileged.
The elements of the REF that are robust are the peer assessment of outputs and the impact case studies. We would recommend that the more ‘creative’ writing elements - research environment and impact template - be dropped.
Normalising sub-panel grade-profiles would obviate the need for elaborate calibration procedures between sub- and main panels.. In place of the textual research environment, some level of quantitative data on grants and PhDs would offer useful indicators, as long as the burden for collecting (and validating) the data can be centralised away from individual institutions.
In terms of research grants metrics, however, it must be noted that the bias is heavily in favour of capital intensive disciplines. In many parts of the social sciences large research grants are not appropriate; research relies on data collected by others (the census is a classic example).
Our overall recommendation is to simplify, rather than make more complex the process. HEFCE has tended to make the procedures more complicated and time consuming in every successive exercise, by introducing new elements. This is often because the sector asks for something different or new, and HEFCE. REF was much more demanding than RAE because of the impact agenda, and the (otherwise laudable) integration of output assessors.
RUK, as proposed by the Nurse review, must be robust in protecting the Research Councils’ encouragement of discovery and ‘blue skies’ research across the breadth of disciplines currently supported.
The existing Research Councils serve as experts and advocates for their respective disciplines, providing disciplinary leadership, agility and connections with communities. Whatever system evolves, these elements must be maintained, with support across the breadth of disciplines and for cross-cutting research. Efficiencies should come from reduction of administrative burden and harmonization and transparency of processes.
The impact of the splitting of research and teaching policy needs to be considered most carefully. There is a risk of duplication of effort and engagement, and loss of strategic oversight that has been provided by HEFCE of teaching and research funding, quality and capital, as well as wider institutional concerns such as financial sustainability. Effective oversight of the sector as a whole must be maintained in some form. UK higher education, across research and teaching, is internationally leading. This must not be undermined
By placing a booking, you are permitting us to store and use your (and any other attendees) details in order to fulfil the booking.
We will not use your details for marketing purposes without your explicit consent.
You must be a member holding a valid Society membership to view the content you are trying to access. Please login to continue.
Join us today, Society membership is open to anyone with a passion for geography
Cookies on the RGS website