Dive Deep Flashcards
QUESTION: Tell me about a time when you had a problem and you had to go through several hoops to discover the root cause.
Title: Follow the data trail
Situation:
Attribution solution relies on customer ad metadata and correlates it to advertiser website to measure advertising response
End users, typically media sellers, intimately know their campaign makeup and how many ads aired/delivered
In early-2019, shortly after product launch customers reported ad count discrepancies
Example: a campaign is comprised of 100 ads, but only 77 appeared in the application
Obstacle/Task:
Count differences eroded customers’ trust in the product and data integrity concerns threatened the users confidence
Experienced customer terminations and at-risk accounts
Action:
Requested customer counts to compare to app counts and saw a 23% avg. discrepancy
Diagramed the data flow to identify potential points of failure in the pipeline
Customer > Database > Discovery > Attribute
Sequentially analyzed total counts in each category
Customer = database; assurance that provided data was written and stored properly
Count difference between database and Discovery; isolated service failure
Discovery and Attribute at parity; confidence in interoperability between apps
Guided support teams fielding customer complaints to structure information with this flow to identify patterns with the same services
Established a “recovery plan” and worked with technical teams to implement retry logic on failing API and correlation services
Advocated for platform team to repair underlying services
Implemented retry logic and post-process to sweep for failures and correct problems
Results:
Speed and confidence around the issue; educating support teams on how to structure information enabled me to better validate a systematic platform issue
Pinpointed root cause and guided the Platform team to the issue
Retry and post-process services resolved over 95% of discrepancy reports in 2 months
What this story demonstrates (skills, principles):
Deep dive
QUESTION: Give me an example of when you used data to make a decision/solve a problem.
Title: Veritone Voice
Situation:
Over the last 3 years I have a growing relationship with Alexa (in the most platonic way)
Working at a progressive SaaS AI company its important to keep a pulse on burgeoning technologies
Identified an opportunity to enhance the “aircheck” use case and integrate voice-enabled technology into the company’s flagship application
Suboptimal text-based experience requiring users to input search, then apply filters to get results which was slow
Obstacle/Task:
Led effort to integrate a voice-first experience around the use case of an “aircheck” that used Veritone’s AI services for NLP and computer vision (transcription, logo, face, OCR) and supported various modalities to interface with the application
Action:
Reviewed in-app chat support data; 2 out of 5 inbound support requests were related to challenges performing an “aircheck”
Mined search query data from elastic and chat support info to understand the roundtrip time for “airchecks” via text-based search
Measured avg. duration for users to perform aircheck via keyboard (type query, apply filters by date, station, etc.)
Avg. roundtrip time was appx. 30 seconds
Presented a data-supported proposal to integrate voice technologies
Partnered with a third-party with voice domain experience to help with build out
Determined the form factors/modalities to use and designed an experience for each scenario
w/o screen
w/ screen
device to app
browser supported web app
Set goals and KPIs to enable users to perform airchecks faster (sub 30 seconds) than using keyboard
Challenges:
-Learning to think with a voice-first mentality
-Ongoing model tuning to account for mishandled voice requests
-Being voice agnostic with Amazon and Google
-Supporting user experience with or without screens
-Scaling search beyond text (find faces, logos, pictures, videos)
-Incorporating system filters in a voice friendly manner
-Returning API response in sub 8 seconds to met Amazon’s acceptance criteria
Results:
Reinvited how to perform an “aircheck” by creating a voice-forward experience using 4 modalities (no screen, screen, device + web, and web only)
Disintermediate device dependency by tying into the browser’s Web Speech API and renders the results within the application
Users could complete a voice-based aircheck end-to-end in less than 7 seconds (77% reduction)
Internal adoption as a preferred method to search
What this story demonstrates (skills, principles):
Invent & Simply
Deep Dive
QUESTION: Have you ever leveraged data to develop strategy?
Title: Rebirth of our flagship application, Discovery
Situation:
Over the last three months I undertook an initiative to determine the product strategy and reshape the vision for the company’s flagship application, Discovery
Primary revenue generator and highest user usage, but aged code base, bloated feature set, overly complex, restrictive logic, broken workflows, UI/UX challenges and crippled user experience required me to reevaluate the vision and direction
Considerations included integrating features into an adjacent app or rebuilding app
Goal to craft a formal proposal to present to executive leadership
Obstacle/Task:
Resource “sensitive” (time and material) and competing agenda to keep resources on green-field opportunities
Generalized use cases; attempting to serve all. My goal to better define target audience
Action:
Leverage Salesforce for metrics on client makeup:
Number of accounts, stations
Defined account profiles to determine the product target. Broadcast audio, tv, youtube, podcast
Identified one dominate medium and two that were non-existent
Leverage Pendo for user insights. Engagement by medium
Product capability tear down and slotting exercise to adjacent application
Lead customer interviews (pain points, before and after product use workflows, user profile, primary and secondary uses cases)
Evaluated trade-offs for various outcomes (integrate, build new, redefine app ecosystem)
Presented a proposal that although revenue-neutral a new combined application with an adjacent app with UI/UX and workflows specific to radio/TV
Results:
Methodical, research and data-driven presentation
Data reinforced the target audience which influence the design/requirements
Parties agreed with recommendation and next steps to advance
What this story demonstrates (skills, principles):
Dive deep
Think Big
QUESTION (2nd): Tell me about a time when you had a problem and you had to go through several hoops to discover the root cause.
Title: Missing Numeris Data
Situation:
My application correlates third party audience data to time-series broadcast to illustrate listenership and viewership for different segments at the day
Earlier this week, a customer reported an issue whereby a watchlist used for tracking its advertiser was missing this audience information
I had known of general issues related to correlation failures from the platform service and this was my working hypothesis as to the problem
The data structure was database > eventing bus (correlation) > insert-to-index (applies the asset to Elastic search) > results
Obstacle/Task:
Time sensitive; customer heading into a big week of sales meetings and relied on the information
Action:
Bottom up or top down approach
Reviewed the problem watchlist and saw audience info missing across all results (implied 100% service failure)
Reviewed other watchlists in the account and saw inconsistency with this data making its way onto the result set
Focused on one file with missing information and retriggered correlation. The missing audience data appeared
Re-emitted correlation task for watchlist, but 8% of results still missing audience data
Questioned whether the service had failed again for that small percentage
Jumped on a call with SVP of Eng, success and platform operations to debug
Suggested working backwards by only retriggering indexing which would suggest correlation worked, but indexing failed (tried reindex, but didn’t work)
Next, re-triggered correlation to marry the audience data (that didn’t work either)
Looked at the error logs which indicated “missing audience data” and identified that for a particular station and time slot there was no data
Results:
Deductive reasoning enabled me to invalidate possible service failures and pinpoint the root cause
Able to restore the missing data and retrigger the correlation services downstream
Patched the missing 8% of results and enabled the customer to generate the necessary reports for its sales meetings
What this story demonstrates (skills, principles):
Deep dive
