Needs Assessment Report on Community Posts Expansion
As a final paper for my Assessment and Evaluation in Instructional Design course, I conducted a Needs Assessment Report on my Community Posts Expansion project. The purpose of this 10+ page paper was to create a foundational needs assessment in order to gain experience with the process.
I received a 98% as my final grade on this final.
The content of my Needs Assessment can be found below:
Work Context
I work at Klaviyo, a marketing automation platform, used primarily for email and SMS marketing. Klaviyo helps businesses own their marketing. The company strives to offer the best personalized experiences, infinitely scalable customer infrastructure, expert guidance, and complete ownership of data and relationships. Within the company, the Support Center is looking to decrease the time to respond on customer’s email questions (tickets) through more public facing answers without stretching the bandwidth of Support Representatives. The software has many different tools within it; such as SMS, Pop Up Forms, Reporting, Automated Workflows and Email Template Creation. Customers of the software write into the support email address to log a ticket if they cannot find the answer online. Support reps then answer these tickets. The question can be reporting a bug (issue in the software), question on how to do something (ex: send an email/SMS, design something, reporting, etc), asking what certain results mean for them, etc. There are about 600 tickets in backlog and around 100 global support reps for the company. Since there are so many tickets in backlog, it can take up to 3 business days to get an initial response. Customers are growing frustrated and leave lower NPS scores at the end of interactions, or cancel their account all together.
Certain Support Reps are the dedicated 'expert' in that part of the tool, it is within their role title. Such as, ‘Backline Support - SMS Messages Expert’. Representatives are given these dedicated titles/responsibilities to better help triage questions and issues as they arise. If a general support representative comes across a customer question/ticket pertaining to a certain part of the tool and they are unsure the answer after looking through resources, the general support rep can escalate the ticket to the dedicated rep of that part of the tool.
Performance Problem and Its History
Klaviyo community is an online forum where customers can post public questions related to the tool and either external (other customers) stakeholders can comment back and answer the questions, or an internal Community Manager can comment back, answering the questions. Our Klaviyo Community forum is very popular, it is one out of a few of our self serve resources. Self serve resources exist so customers can google their questions and find an answer in either Community, Help Docs or Academy courses. One area of the tool (email templates) already has it's own public facing community posts of top asked email design questions. After publishing this series, email template logged tickets decreased. Customers found their answers in the community posts rather than submitting a support ticket. Customers and internal teams are looking for community series like these for all parts of the tool (SMS, Automated Flows, Reporting, Forms, etc) in order to decrease all support tickets and customers become more self-sufficient. Especially since within the next year, customers on the Free Plan will no longer have access to live person Support Help (email or chat) after their first 90 days. So, they will be completely dependent on public facing documentation for their questions.
The performance problem lies in that the company needs to have these dedicated experts write the public facing FAQ community posts for their parts of the tool to decrease support tickets logged without stretching bandwidth of these support reps too much. Internal and external stakeholders are looking to replicate this type of FAQ series with other parts of the tool without stretching the bandwidth of already busy team members.
The stakeholders here are both internal and external. The support representatives that answer the same questions continuously, the backline experts that will be used to answer the most important questions for their designated part of the tool, the community team that will aid in publishing the posts, and the customers that can be self-sufficient once answers are made publicly available. Resolving this performance problem and conducting this needs assessment will help many people in their day to day tasks. This issue has existed since the company was started in 2012, customers are always going to have questions - it is dependent on the team to post more tutorials and answers to questions in order to prevent these questions becoming tickets needing to be solved.
What the Research Says
I am using research to determine the best course of action internally to better help customers externally. First, I am going to use Contextual Analysis (Stefaniak, 2021, p.61) to determine how the customer’s questions are finding their way to the Support Queue in the first place. Contextual analysis “observes individuals within the system during a needs assessment to better understand the interactions between components in the system” (Stefaniak, 2021, p.62). There are a number of contextual factors that can impact performance within an organization (Stefaniak, 2021, p.63). For this, I asked a customer that had submitted a ticket, what brought them to submit the ticket; they replied “we could not find the answer in the documentation and our own developer cannot troubleshoot this issue since they are paid/hired hourly and working on other projects.” The contextual factors here are monetary limitations from not being able to pay for the developer to troubleshoot, organizational structure as the developer is not hired full time and an information resource gap since the customer didn’t find the answer they were looking for in our free public help documents.
I know this is a performance problem that needs intervening based on Mager and Pipe’s Analyzing Performance Problems. As of now, there is a discrepancy on what should be done and what is happening, “a discrepancy is a difference, a mismatch, between what is what should be. And our focus here is on human performance discrepancies, these differences between what people are actually doing (or not doing) and what they should be doing” (Mager, Pipe, 1997, 9). There is a discrepancy between what the customer should be able to learn online on their own and what is actually available to them.
Interview
For this Needs Assessment, I interviewed Klaviyo’s Community Program Manager (sometimes abbreviated as Community PM). We met over Zoom with cameras on for about 45 minutes. Having our cameras on helped convey what emotions he felt towards the questions and different subject matters. I had previously worked on projects with this Community PM so we are very familiar with each other. I started the meeting by explaining the purpose of this Needs Assessment and some background on my Master’s Program. I chose to meet with this Community PM as he has context on what the Community Team needs, what customers that visit the Community are looking for, what is within/out of scope, the processes for a project like this, and has a relationship with Backline Experts. The meeting was recorded so I could later transcribe the answers and questions. Prior to the interview, I read the internal wiki guides on how the Community Team structure works, who needs to approve posts/projects based on seniority, and an example of an actual project proposal written by a Community PM. The project proposal showed me what data is needed in order to have a community project approved, such as Zendesk analytics on how many times a question has been submitted by customers, trends in which parts of the tool receive the most questions, and an example proposal statement. Looking at this wiki guide and reading the project proposal helped me come into the interview more prepared and understanding the basics, so we could discuss the details and next steps and I could better conduct the Needs Assessment.
Needs Assessment Interview Questions
The experience related questions that I asked included:
What do you imagine could be a pain point of this community expansion project?
What do you think caused this problem/need for the community expansion?
Who do you think are any stakeholders or key players in this?
How will this project affect the customers?
The resource and training-related questions that I asked included:
How do you suggest we don’t stretch the bandwidth of backline PEs/Community teams?
What is your desired outcome of this project?
What do you think is an important step in this process?
How will the community team work on this?
Do you think this project could sustain even if the project manager (myself) left the company?
What technical or soft skill knowledge do you think the team needs to know, to complete this project?
What resources do you think the backline team can use while answering these questions? IE: what training can the trainers use?
How do you personally find what customer questions to answer in public community posts?
The interview questions and responses provided insight into where the performance issue is occurring and how we can use this needs assessment to resolve it. Gathering data helped me create an informed and evolved solution. Using the answers I gathered in the interview and the community’s project proposal process I read about, I discovered that there is a gap in knowledge for Backline Expert reps. The Backline Expert reps do not know how to publish a community post and which questions should be answered. This is causing a performance problem in which customers cannot find publicly published self-serve answers to their questions, and therefore must submit a support ticket, causing a large support ticket backlog. This interview helped prove that backline representatives do not know how to look at Zendesk Analytics to find top customer questions that are being asked repeatedly, nor do they know how to create an internal Klaviyo employee community admin account in order to post the question with appropriate answers. Lastly, backline representatives do not have time or bandwidth to write this found information (on top questions backed by analytics, answers, project purpose, etc) for a community project’s proposal plan. This interview helped illuminate that there is a knowledge gap as this information is not being shared from Community to Backline Reps, which is therefore causing a knowledge gap for our customers as they are not able to find the information publicly. There is also a performance problem that the Community Team requires a full write up to get a project approved, creating a barrier to entry for team members that have time to answer top questions but not top questions on top of a full report. The target audience is the Backline Support Representatives, Customers looking to self solve and the Community Team. The characteristics of Backline Reps include a large amount of technical knowledge but no knowledge on where to publish this. Customers have an eagerness to learn and quickly resolve questions on their own rather than waiting up to 3 days for a support response. The Community Team has a willingness to help customers in self-serve content without the technical knowledge of the product, but technical knowledge of how to find which questions should be answered, as well as how to write a project proposal.
Recommendations
After much consideration, I recommend a Google Form backline representatives can fill out with proposed community posts/their answers, as well as a recorded training session done by the internal enablement team, using the Community Team as subject matter experts. The training will be presented to the Backline Support representatives team. This training will include how to view and find top repeated customer questions within Zendesk analytics and how to create an internal Klaviyo admin Community Account. The instructional goal of this training will be to close the knowledge gap between the community team and the backline representatives. Backline can find which questions they believe should be answered publicly by now knowing how to view Zendesk Analytics, and submit the question and it’s answer (based on the backline’s expertise in the part of the tool) to the form. Then, the Community PM can handle writing the project proposal for the post or publish the content on behalf of the backline rep, if their bandwidth does not allow it. This way, the backline rep is able to provide public answers to their expert domain without having their bandwidth stretched too thin by writing the proposal. The form can be submitted directly to a Google Folder owned by the Community Team. The training portion will be recorded so future/new backline reps can be trained on this process as well. The backline rep will learn in the training how to create their own Community Admin account, in case they do have time to post the content themselves or if the Community PM is absent. The objective here is to teach backline representatives how to find the questions that should be public knowledge, how to submit this to the form the Community team will receive, and the Community team can receive content/answers for the Community Forum, while taking on some of the workload by writing the project proposal so the Backline team does not have to. I was able to prepare this learning objective using R.F. Mager’s Preparing Instructional Objectives (1997), “An objective will communicate your intent to the degree you describe what the learner will be DOING when demonstrating achievement of the objective, the important conditions of the doing, and the criterion by which achievement will be judged” (Mager, 1997). I want the learner to physically know how to find the most popular questions that should be answered and how to submit the digital form. They have previous knowledge on the answers to the questions, but can use existing resources to answer such as internal wiki guides, help docs, academy courses, or asking in a public slack channel. Within Mager and Pipe’s ‘Analyzing Performance Problems
Is Training Really Needed?’ I learned that this training is needed. I used the Analyzing Performance Problems flowchart to determine that training is needed because there is a knowledge deficiency where information can be provided. For this, our resources available would include the enablement team’s creation of a short LMS course (Skilljar), Zoom Meetings, Google Forms and Google Sheets that the forms will submit to. Overall, I believe a digital form and training will help resolve the issue of knowledge gap between teams while sharing the workload, to not overload either team. This will resolve in an exchange of services and knowledge, increasing performance internally and externally for our customers.
Kirkpatrick Level’s
I will use Kirkpatrick’s Four Levels of Evaluation to evaluate the effectiveness of this
Training session at achieving the desired outcomes (Kirkpatrick & Kirkpatrick, 2009). The following Level 1 Evaluation assesses learner reaction to the training session:
I would present this level 1 survey immediately after the backline reps finished the training.
Level 2 of evaluation focuses on the learning. After backline representatives have completed a training session, they could be asked to join an assessment group. In this low-stakes environment, there would be no more than two community program managers and the backline rep. The rep can demonstrate how to find a question based on their part of the tool in Zendesk analytics and how to submit a ‘test’ question/answer to the digital form that submits to the Community Team. Here, representatives can ask any questions that pop up to the community PMs. Backline reps needing additional support could be asked to continue on
in small group learning process; proficient backline reps can be paired with peers as a way to provide support and collaboration (the other reps that did not progress).
Next, Level 3 of evaluation focuses on behavior. Community PMs can check in via slack or email asking how the implementation process is going and if there are any questions or obstacles the reps are running into. The Community PMs can provide suggestions for enhanced
use of these resources (zendesk analytics and digital form) and provide specific targeted learning resources based on any additional questions.
Lastly, level 4 hones in on results. Backline reps can complete a survey 6 months post training regarding how this training made an impact - positive or negative. This could provide insight into whether the training provides reps with the knowledge and skills needed, whether instruction and processes have improved, and future training needs. This survey can be used to improve the process for future reps that will need the training.
Reflection
I learned many valuable lessons from this Needs Assessment. I was not familiar with the Kirkpatrick Model beforehand. Now I have the levels memorized and really understand the concepts. I now see the value of checking in immediately after while information is fresh in the user’s mind as well as months later, to see implementation’s process when everyone is more comfortable with the information. Before this semester, I had never asked a colleague if they would be open to an interview for data collection. I had met with coworkers and asked open-ended questions, but now I do see the benefit of framing the meeting as an interview, as the interviewee knows the larger goal around the questions and can provide more context regarding this.
In the future, I may conduct more than one long interview and instead have a few, short interviews. Then, I can gain more perspective and data. I may also try to quantify the information learned through digital surveys with numeric scales in order to sum information up with data that can be compared. As of now, I am very happy with my qualitative data as it gave me a full picture and background context, but a combination of qualitative and quantitative will be helpful for analysis and comparison in the future. Overall, I am happy with my information gained and proposed results as I believe it can help make people’s lives easier!