Assessing the Practices of Public Scholarship

2009 Conference Plenary

Imagining America's recent national conference in New Orleans featured a plenary on Assessing the Practices of Public Scholarship (APPS). Panelists made “state of the field” remarks concerning engaged courses and projects in the arts, humanities, and design, as well as across these fields and as incorporated into social sciences and other disciplines. To read a transript of those remarks, click here. Facilitated break-out groups then discussed current methods of evaluating public scholarship for faculty, students, and community members, as well as metrics for tracking institutional change, and strategies for IA’s future research and delivery on this topic. We thank the IA board members and consortium reps who spoke on the panel and facilitated the break-out groups!

The break-out group facilitators asked the following questions, which are followed by the most recurring responses and some miscellaneous remarks from participants.

What did you hear that's exciting about IA’s initial thoughts on APPS?

The focus on assessment as a means, not an end, and as a process of reflection oriented toward discovery.
The emphasis on mutually-beneficial assessment strategies – among faculty, students, and community partners – that contribute to more balanced partnerships.
The scope of working across campuses and across disciplinary languages, as well as working at different kinds of campuses (community colleges, four year colleges, both private and public universities), to discover both shared principles and variations.
The shift from short-term evaluative processes to the long-term assessment of projects and outcomes, with sustainability a core value.
The intention to share engaged assessment tools, findings, critical analysis, and ways of articulating the importance of APPS, among IA members.

What did you not hear that you think IA needs to address?

Who initiates/ designs a project and its assessment?
What is the underlying philosophy of engaged projects?
How do you account for unanticipated and long-term outcomes?
How does assessment contribute to faculty development and student learning?
Is there space in assessment metrics for qualitative data, testimonials, and audience experiences, and generally, the intrinsic power of the work?
Recognition for the amount of work generative, research-based assessment takes, with concern about unfunded mandates.
More about how assessment could contribute to equitable relationships between campuses and communities, for example: How do non-profits benefit from universities, keeping in mind the reality that personal contacts determine the shape and scale of collaborations?

What are the existing resources?

Within IA’s consortium, there are success stories. Tell them!
Some of IA’s member campuses are engaged in assessment initiatives, for example, University of Michigan has some tools for tracking the longitudinal impact of race-ethnicity courses.
Animating Democracy’s Arts and Civic Engagement Impact Initiative: http://www.americansforthearts.org/AnimatingDemocracy/programs/programs_016.asp
Geraldine Dodge Foundation’s Assessment Initiative: http://www.grdodge.org/learning/assessment/index.htm
Tools from social sciences
Community college associations’ studies
Some skills are easy to measure, such as literacy, when working with community youth on reading.
Alternative models of evaluation that allow for outcomes to be defined by the process of the work itself, e.g. developmental evaluation from the field of social innovation is one such model.

What tools do we need?

Models for longitudinal assessment for syllabus and more broadly pedagogical development; for working with specific and/ or diverse student and community populations; for talking about why community-partnerships are part of the curriculum; for tracking learning in/ through non-profit organizations; and for how and what technology can be useful as an assessment tool.
Creating specific assessment processes related to those that exist in each discipline.

What are the obstacles to assessment?

TIME – it’s hard to get to the assessment.
Preexisting guidelines binding our community partners (e.g., standards of learning for schools).
Difficulty of using assessment as a tool for institutional change.
The unpredictability of working in community, which requires us to be alert and attend to unexpected outcomes.
The tendency to track positive transformation but not negative outcomes that are also important in improving practice.
Academic environments where there is belief in the work, but no understanding of the need for assessment.

Why assess? For whom? What are we assessing?

Is the assessment so that we can justify or legitimate ourselves, or is it so we can improve?
Whose values determine success?
Are we assessing by the knowledge produced?
Praxis suggests ongoing action and reflection. How does praxis relate to IA’s view of assessment?
Assessment is rooted in the verb "to sit beside" the learner. The question is: Who is there?
Our administrations want to hear more about what impact engagement makes on the student.

What are your concerns?

Assessment could de-legitimize the diversity of practices.
General fear of assessment, not recognizing the benefits: “Assessment on our campus leads to something bad happening.”
Community partner reports, “What’s the point of assessing, we know it’s good.”
How can partners articulate their needs, and how can we hold ourselves accountable to those needs in our assessment?
Private goods are easier to define then public goods. What is “good” may not be for all faculty, students, and community partners.
Getting teachers to do it because of it being extra work.
So much of the assessment of our work is anecdotal; how do you measure it?
Design, analyzing, and reporting back is a real skill.
Employing empirical methods that run counter to artistic content.

What are your suggestions?

Link student retention and success vis-a-vis their involvement in public scholarship.
Assess partnership via unity between students and partners.
Create “learning contracts” at start, which make it easier for teachers to assess at end.
Use online discussion boards.
Pre-test students to measure their attitude towards civic engagement. At the end, have community partner assess the value the student added to the agency, and do a student post-test to track if he/ she will continue engaged work outside class.
It would be useful for IA to define areas of community impact (e.g., generating knowledge for democratic practice).
Have a conversation about assessment with funders, assessors, and community partners at the next conference.
Legitimizing the work to increase and extend the impact by getting the resources we need. We need a kaleidoscope of assessment tools and an array of ways to learn how to use them. Would love for IA to develop these tools and provide a “menu of choices.”
With support of a national organization, maybe all IA members could have access to specialists to support campus-based assessments.