Are Right, A Lot Flashcards
QUESTION: Tell me about a time where you had to make a controversial decision.
OR
Tell me about a time where you had to make a decision without data or complete information.
Title: To OAuth or to not OAuth
Situation:
2.5 years ago, developing a minimal viable product in attribution
Corporate initiative to launch new offering in less than 6mo to take market share
Tight budget and timeline
Objective to determine product viability and phase development
The product required pairing the my service with the user’s Google Analytics account for data collection
User manually adds my product as 3rd party service within Google analytics granting permission after completing a few steps
Automated service linking via Google OAuth
SVP engineering heavily involved in product decisions given project importance/timeline
Obstacle:
SVP engineering advocated for manual connection path to Google’s service to reduce development time and speed to market
Action:
Evaluated estimates with and without OAuth service (2 days of development)
Worked with Design to compare the two workflows
Considered the impact to the user (OAuth significantly simplified setup)
Factored trade off and decisively pursued OAuth counter to the SVP’s preference
Results:
A more accelerant method to connect customers to the service
Avoided a higher barrier of entry to extract value from the product with potential fallout
Measured scenarios after implementation:
- Oauth workflow took 20 seconds
- Manual workflow took 4 minutes when referencing instructions
Appreciate SVP’s input given the overarching goal of shipping quickly but product viability would have suffered due to poor user experience
What this story demonstrates (skills, principles):
Are right, a lot. Good instincts and judgement. Open to diverse perspective
Have backbone; disagree and commit. Challenge decisions, have conviction
QUESTION Tell me about a time you made a design decision where a lot of people had opposed you. Why did they oppose you?
OR
Tell me about a time when an idea you proposed was not agreed on. How did you react?
Title: Death to Public Media Index (PMI)
Situation:
With inception of the my current employer (Veritone), it created a “value add” offering that provided insight to non-O&O stations often for competitive intelligence
Veritone discretionarily selected sources to include without much customer direction
After several years, uneffective at monetizing offering and often an add-on
Offering created a financial drain with no real revenue offset
Restricted company’s ability to hire due to cash constraints
Obstacle:
Customer usage, contractually obligated, product differentiator
Action:
Led an initiative to determine whether the offering should be jettisoned
Solicited input from Success team that customer usage and value of PMI as low
Researched how often filters applied to sources not owned by the org/account
Developed financial models that est. expense projections (transcription service $.10 per hour + CPU $.05 + storage $.02).
Worked with Legal to identified customer with contractual rights to offering
Presented findings to executive team
Results:
Illustrated that expense rivaled that of the company’s largest business vertical, M&E
Presented data that showed only 5% of users searched/filtered outside of their stations
Less than 5% of contracts had PMI as part of its terms
Recommended discontinuing the offering given limited customer use and high expense
Pitched curated, client specific offering at customer’s expense or offset by Veritone
Executives rejected proposal to discontinue offering due to its “strategic and long-term value”
Cost running AI models and compute was decreasing, storage is inexpensive and there was inherit value in offering and data
Perplexed; evidence was overwhelming – operating expense high; customer value low
However, a year and half later the offering was discontinued due to operational overhead
What this story demonstrates (skills, principles):
Are right, A Lot. Strong judgement and good instincts
Have backbone, disagree and commit. Conviction that offering need to be redefined
QUESTION: Tell me about a time you made a design decision where a lot of people had agreed with you. Why did they agree with you?
Title: Campaign Stepper
Situation:
Application workflow required user to create an “advertiser” (to correlate Google Analytics) and pair it with a “campaign” housing all of the media buy criteria (i.e. date range, stations, brand search terms)
After creating an “advertiser” the workflow landed the user on a blank campaign dashboard with empty graphs requiring the user to create a “campaign” to marry the advertiser data
“New Campaign” button in upper corner; however, overlooked by users
Unclear to the user that campaign creation was the next step
Obstacle/Task:
I wanted to improve the UX and design an optimized workflow
Action:
Reviewed inbound support tickets and identified 15% reported issues per month with confusion as to next steps
Analyzed user database records for createdAt timestamps and identified avg. time to complete advertiser-to-campaign setup was over 1.5 days
User setup campaigns after their advertiser connected
Designed a “stepper” flow that lands the users on a transition page directing users to the 2nd step, creating a campaign (while indicating step 1, create an advertiser was fulfilled)
Goal to reduce setup time to less than 15 minutes (a 99.99% reduction)
Socialized data, design concept and expectations to Success team and customers and gained unanimous consensus
Results:
Feedback was overwhelmingly positive from existing customer champions
Eliminated support tickets related to workflow advertiser-to-campaign next steps
Setup time avg. less than 10 minutes
What this story demonstrates (skills, principles):
Are right, a lot
QUESTION: Tell me about a time you made a mistake.
OR
Give me an example of a time when something you tried to accomplish and failed.
Title: Meet at the customer, not the other way around (More than just API to send data)
Situation:
Upon product launch, only method to interface with this attribution service was via API
I had worked with the developer team to harden our GQL endpoints for security and reliability
However, most customers didn’t have a technical team familiar with APIs or tech savvy
Customer systems used antiquated software not compatible with modern solution
Caused a technical chasm for customers preventing value extraction from the product
Strain on Sales and Success as customer was frustrated
I had forced customer to come to us vs. other way around
Obstacle:
Determine to find alternative approach lessening the technical burden on the customer
Action:
Interviews to understand customer constraints (system incapability and technical competency challenges)
Led series of internal technical discussions to explore data delivery options
Challenged Eng proposal to install code, Raspberry PI, locally on the customer’s machine (escalated to CEO to reject proposal)
No mechanism to monitor code health locally
No resources to support a distributed code model
Advocated for a compatible solution to the customers’ systems and processes
Took customer requirements, oversaw development and built real-time (TCP Proxy) and batch services (CSV email delivery)
Results:
Expanded how customers could interface with my app (three options instead of one)
Unblocked 3 customers threatening cancellation due to onboarding difficulties
Reduced on-boarding time from 6+ weeks to than 4 weeks with other delivery options
What this story demonstrates (skills, principles):
Are Right, A Lot
Earn Trust
QUESTION (2nd): Give me an example of a time when you tried to accomplish something and failed.
OR
QUESTION: Tell me about an experience you went through that changed your way of thinking.
Title: Orphaned Advertisers
Situation:
Last year, I introduced a first of its kind user permissions model at Veritone that assigned an object (adv/campaign) to its creator
Additionally, supported sharing functionality for peer visibility along with the ability to transfer ownership
Model ensured that users created objects remained private to them unless shared
Once in production, experienced questions from Success regarding the advertisers of deactivated users
Obstacle/Task:
I overlooked the scenario for when an owner of the object (adv) is deactivated (no longer with the organization) and how to handle this in the UI
Action:
Replicated the scenario to assess the existing behavior before engineering a solution
Reviewed database and identified 10 advertisers in 3 orgs with an deactivated user
Designed a workflow that would categorize this type as ‘Unassigned’ advertiser
Made ‘Unassigned’ folder visible to any ‘Manager’ user type for assignment to a) themselves or b) another user in the system
Validated the concept with internal and external users
Results:
a) successful in creating UI visibility to unassigned advertisers b) enabled managers to assign a user
Reduced the number of unassigned advertisers by 50% within one week
Experience forces me to think about “referential integrity” between primary and foreign objects and how changes to a primary object without addressing the foreign object can result in orphaned objects
Ingrained in train of thought in product development and applied to test cases to better account for scenarios
What this story demonstrates (skills, principles):
Are Right, A Lot
Learn and Be Curious
Bias for Action
QUESTION (3rd): Give me an example of a time when you tried to accomplish something and failed.
Title: Rerun campaigns w/o a limit (TODO)
QUESTION: Give me an example of when you had to make an important decision in the absence of good data because there just wasn’t any. What was the situation and how did you arrive at your decision? Did the decision turn out to be the correct one? Why or why not?
Title: Automation or Traffic System
Situation:
2.5 years ago; developing near-real time attribution solution that required broadcaster metadata
Important to the product because it provide the data and time of ad event
Metadata capture could be done in two wayss
Automation system
Traffic system
Obstacle/Task:
My goal was to disambiguate between the two systems and select an integration method
Action:
Conducted interviews with broadcasters to understand the nuance and benefits
Evaluated the trade-offs based on delivery mechanics and data composition/makeup
Traffic:
Housed only ad event metadata (only advertisements, not program content), which was the requirement for the product
Fed ad event data into the automation system for playout management
Involved manual or automated reconciliation process that took place 24-48 hours after an ad event aired to ensure data integrity
Post-process, batch data delivery through usually through a packaged CSV emailed to the system
An entire day of ad events were a delivery payload
Automation:
Orchestrated the ad, music and content playout (holistic view of asset management, not just ads)
Events emitted in real-time
Real-time nature doesn’t allow for reconciliation, which can cause up 5% discrepancy between what the system thinks aired vs. what actually aired
Leverages real-time data transmission through API
Each event was a discrete payload
Pursued automation system for holistic data capture, real-time and API delivery despite minimal data discrepancy
Near real-time insights was a competitive advantage enabling the customer to act on data faster not found in other offerings
Expanded dataset capture beyond just the ad which had long-term strategic value (correlate metadata to actual broadcast for data validation and derivative products)
Results:
Unlocked near-real time value, but faced data discrepancies that were difficult to triage
Customers reported count differences up to 10% likely due to failed dispatched events from the broadcaster or unreceived on my end due to API failures on my end
Eventually created integrated traffic systems and benefited more accurate data due to reconciliation process and reduced data transmission complaints to less than 5%
Customer use case is largely post-analysis vs. real-time
What this story demonstrates (skills, principles):
Bias for action
Are right, a lot
