Memo Review Questions Flashcards
Can you elaborate on the iteration aspects mentioned in the program objectives?
We know that the problem statement is we’ve created a few live internal chatbots for our internal XFN teams to use today. The challenge is they are geting inconsistent results with their queries if they use >1 chatbot and it creates confusion on what to do because some of the results are outdated or just different from the other. We need to figure out how to protect against this problem by configuring a solution that would provide consistency and accurate data to improve the user experience. Then we will continue to iterate on our chatbots for future improvements as we develop our product backlog.
-Data Policies, Embedding Model, and Atlas Trigger: If we are able to ensure policies that outlines standards for data accuracy and establish guidelines on how data should be updated for the chatbots to access, utilize our features that can help automate filtering out old data with Vector Search and the Trigger, it will help with system readiness by providing end users quality results
-Collaborative Dev Env: Having a collaborative dev environment such as regular trainings, an internal community such as in Slack or maybe an intranet will help with best practices and keep the dev teams in the know of the most recently released features
How do you plan to ensure the internal chatbots have consistently updated data?
-1st monitor post deployment of our change management of the given scope
-Established data policies with guidelines for chatbots to access are up to date
-DevOps can create alerts for any errors in systems to ensure:
All chatbots are pulling from the centralized data source; authoritative DB
-All chatbots have Vector Search and Atlas Triggers working as expected (i.e filtering out old data and updating records as changes come through from sources such as Salesforce, MA tool, MongoTV videos, courses, podcasts, etc based off of “Date Created.”
-End User Surveys
Can you provide more details on the data governance policies you plan to establish?
-Guidelines such as the chatbot can only pull from the central data source; source of truth
-Data protection standards; no sensitive personal information-encrypt sensitive data
-Data access controls where data is restricted base off of user roles
-Perform validation checks to ensure Atlas Trigger and Vector Search are working as expected
How will the centralized data source function as the authoritative source for chatbot data?
-All published content and data created in internal and public facing resources will live in the authoritative database with a data integration management strategy
APIs with systems to the DB
-Sources other bots are pulling from now will be connected to the central source
-Use of Atlas Trigger and Vector search will ensure the DB contains the most up to date information
Atlas Online Archive to contain older data that is still accessible
Why did you choose the specific timeline for the program, and how did you arrive at these dates?
-Establishing policies and guidelines can take maybe a month for senior leadership to align on and create final drafts to implement into systems
-Known unknowns with technical complexities for implementing the guidelines, APIs from internal tools and the website, Atlas Trigger, Vector Search, and Online Archive into the chatbots can create risks with a short timeline. Therefore, I am providing a buffer of a few months of dev time
-Testing with complex programs or projects can take a month with end to end testing with TDD and stakeholder testing to confirm all in scope works as expected
-Monitoring post deployment may have bugs identified by users in prod env that the chatbot devs need to work out
-Resource limitations may come up with any P0s/ad-hocs that pop up during execution and impact the timeline
How will you manage dependencies and potential delays during the timeline?
Ensure predecessor workstreams and/or tasks are on track. If there are any blockers, I will figure out how to unstick the teams
If there are inevitable delays due to blockers such as resource challenges or technical complexities, then this will need a revised timeline and comms to the stakeholders
Can you explain the role of each team member in achieving the program objectives?
Systems- system integrations with API with chatbot
DB: Maintain database infrastructure in storing chatbot data and optimize data retrieval
IT-security; manage user controls and tech support for chatbot
BI- create dashboards for chatbot performance
End User adoption rates and by role
New user adoption rates in onboarding
DevsOps- create triggered alerts if APIs or chatbots error out pulling from the central data source
QA- E2E testing
Enablement- end user training
How do you plan to foster collaboration among different teams involved in the project?
-Facilitate pre-discovery call that introduces the project problem statement, objective, priority level, high level timeline and confirm team representative/resource for weekly team status calls
-Create group slack for the program that guides teams for offline collaboration
-PMIS and tag assigned resources to each task. Use comments for collab among the team
-When a predecessor task is completed such as Data Policies are approved and ready, it informs the technical teams such as IT, Systems, DB,and DevOps to ensure all tools have the policy implemented (i.e chatbots must only pull data from the central data source)
-Schedule working sessions for Design and Development (Devs, DB, IT, Systems)
-Celebrate success together (announce a good job in Slack group)
How did you determine the estimated level of effort (183-210 hours), and what factors influenced this range?
KO:
Assumed about 3-5 hours for preparing, coordinating, and facilitating the program KO
Design:
About 15-20 hours of design including ideation gate review
Dev:
130-140 hours for dev time
Test:
40-50 hours testing
Deploy:
1-2 hours deploy
Monitor:
40-50 hours M&C
How will you track and manage the budget throughout the program?
Ensure that the agreed upon scope is not increased during design and or execution unless it is agreed upon to increase the project budget for add’l resources/time and timeline adjustment
Can you provide more details on the potential technical complexities mentioned in the risk assessment?
-Known unknowns at this part of the program lifecycle, but high level understanding that implementing with systems, databases, technical features with API and AI technologies can be very complex and will require months of development time.
-Also known unknowns for senior leadership alignment on the central data source guidelines to be applied to the data governance policies. This can maybe take a month to have final approvals as the predecessor to system readiness projects.
How do you plan to mitigate the risk of limited developer engagement with best practices?
-Ensure that all developers working on the chatbot attend the regular trainings and attendance to the online trainings are tracked.
-Have the devs take a quiz at the end of each training to confirm they were engaged and retained the information OR
-Have the devs take a quiz during the live training and the enablement team can show the quiz results
Why did you choose specific days and times for the weekly program status call and Slack summary?
The specific days and times are random and can be aligned wth what works best for the stakeholder group
Typically, I’ve had status calls on any day of the week and times can vary
Same applies to Slack summaries, but some clients have requested those updates to be either each Thu or Fri to let teams work through any WIP and see where the week landed with program and project performance
How will you ensure effective communication among stakeholders and team members?
-Ice breakers to keep is human and helps people want to engage at the start
-Pause and ask the group if anyone has any questions
-Summarize the call and next steps for the team
How will you measure and validate the achievement of 98% data consistency in the 1st month?
BI team that can create dashboards to track:
-Adoption from new hires of a specific team and all teams of new hires
-Adoption from existing employees at the company >90 days
-Track success rates of the chatbots pulling data from the central data source and how frequent is it erroring out
-Track to ensure all chatbots are following the data governance policies (if any released chatbots did not follow the standard operating processes and devops best practices)
-Monitor if vector search and Atlas Trigger features are working as expected to ensure the data is correct
