Amazon LPS Flashcards
When you’re working with a large number of customers, it’s tricky to deliver excellent service to them all. How do you go about prioritizing your customers’ needs?
Prioritization goes into multiple steps:
- Understanding the value its driving to our customer (Number of hours saved, increase in visitors or conversion as a result of this feature, increase in the number of PR coverage)
- Engineering Effort required to build those features.
- Plot these in a graph with 4 quadrants
Game changer - High impact , high effort
Hidden treasure: - high impact, but low effort
Low customer impact, high engineering effort - wasted effort
Low impact and low effort is low hanging effort
Do tie it back into the company vision and quarterly goals (increased users, increased revenue, increased retention) and urgency see whether to change the company goals or shelve the idea for later.
How do you build a product that solves customer need?
We think of our customer needs in many different dimensions:
- Understand the customer pain point deeply - this is done through interviews with different personas. For instance, when we developed the reporting tool, I’ve been part of these interviews myself. What is very important here is to interview at least a couple of people in each persona - this is because the need could be very different. In our case, we had PR managers, CMOs, Head of Analytics, CCO - all using the product, but each of them had a different requirement.
- One of the core tenets of our design and product culture overall was “Simplistic elegance” - what that meant to me in particular was to try to see if there is convergence in those different ideas and needs that each persona has. If there is, try to converge your solution to solve for those use cases, if there isn’t - try to simplify your design to accommodate those use cases.
- An example of this was in the types of coverage our customers wanted to see. We had some customers that wanted to see all of their coverage and some other customers that would want to see only highly relevant content and wouldn’t like to see bare mentions of their company name. We converged the ideas from our different customers into a product that we designed from ground up to bring in only the coverage that was relevant to the company. We provided various knobs for the customer to be able to filter out content that was not relevant to them. This was all available for admins to change, but for the customer, we made it very simple - they just needed to let us know keywords and company name, and the system would auto-generate those searches for them
Give me an example of a time when you did not meet a client’s expectation. What happened, and how did you attempt to rectify the situation?
The one that comes to my mind is one with a project called Power of Voice. We deal with news content at our company. The idea behind this project was to rank/score each article and build a model that gives an overall score and compares the score with their competitor. The issue we faced was that the customer was the most technical we’ve had since the beginning of company and so had a lot of questions about the workings of the model. This is tricky because it’s a highly technical model that has to be described only in a long white-paper, at the same time, it was proprietary to us. The action I took was to get on the customer call with them, the customer usually in cases like this feel like their voice is being heard. In the call, I would explain to them why our current model was producing the results that it was. After discussions with Product and our CEO, we decided to shift some priorities to move the 2nd iteration of the modeling up in priority. Since this was a big impact product change, I was involved in it from start to finish helping identify some sample candidates for improvement, presented it to the team and then turned them around. They were extremely happy with the results and it showed in the feedback they gave to their CMO who decided to do a multi year renewal of our SaaS application.
Who was your most difficult customer?
The one that comes to my mind is one with a project called Power of Voice. We deal with news content at our company. The idea behind this project was to rank/score each article and build a model that gives an overall score and compares the score with their competitor. The issue we faced was that the customer was the most technical we’ve had since the beginning of company and so had a lot of questions about the workings of the model. This is tricky because it’s a highly technical model that has to be described only in a long white-paper, at the same time, it was proprietary to us. The action I took was to get on the customer call with them, the customer usually in cases like this feel like their voice is being heard. In the call, I would explain to them why our current model was producing the results that it was. After discussions with Product and our CEO, we decided to shift some priorities to move the 2nd iteration of the modeling up in priority. Since this was a big impact product change, I was involved in it from start to finish helping identify some sample candidates for improvement, presented it to the team and then turned them around. They were extremely happy with the results and it showed in the feedback they gave to their CMO who decided to do a multi year renewal of our SaaS application.
What was an example situation when you delved deep to understand customer need to solve their problem?
At Onclusive, we were building a product that is industry first, i.e. measurement to show what’s working and what’s not working. There was no measurement standard in the industry. So a lot of decisions were based on intuition initially until we got the first few customers. When we landed our first customer a lot of our product iteration were a direct result of close interaction with our initial customers. We knew that there was a clear need for a product like ours, what we needed to do was to prove the value. And it turns out, our initial customers also were always tasked to do this, and took often took days to create those reports to prove the value. They needed an easy way to take different types of data in one platform and report them to higher ups - often the CXOs and the board. Our interactions with those initial customers provided us the confidence to move forward with our idea of building a reporting suite. We were the first in the industry to build a full blown reporting suite specific to the PR vertical. You can think of it as a Data Viz tool for PR. The product was widely used, but the real success was when we started noticing AirPR reports appearing on customer blogs, showcasing the results of their PR efforts. Further, it became quite useful internally as well - whenever our Sales team used to demo the product they would use the tool to make reports with pre baked in data. This immediately impressed our customers and got the initial buy in.
Tell me about a time when you had to leave a task unfinished.
Around mid last year or so, the number of customers started to increase quite a bit. Some context about our business, when we sign up a customer, there is quite a bit of analytics integration to be done. There is a lot of back and forth between the customer and us initially to get the required details and then setting up to do on our side to ensure that its a successful start, otherwise there is general dissatisfaction in the engagement because the buyer is waiting for the training, however, is unable to start because analytics integration isn’t complete. We embarked on a task to automate this, started building a workflow type internal tool that will go through a state machine, keep track of the inputs from customers, track completion of steps and send reminders. We scoped this project to be 2 month project for a team of 4 engineers (2 backend and 2 frontend), we were one month into the project at which point we had to shelve it. The issue was that, priorities changed. We had a even bigger problem of churn - and there were some very severe product areas that needed to be addressed. It was a failed project because the product never got reinvigorated, however, we were able to get a few learnings from it since we designed it as a separate app with newer version of the technology stack.
Other project that you can talk about is Insights
• Tell me about a time when you had to work on a project with unclear responsibilities.
Sometimes, we come out of executive meetings when our CEO would say this needs to be tackled by engineering and product - but since there were 3 execs in this department it wasn’t clear who it was being directed to. The particular case that I remember very distinctly is when we were planning a very big RFP. The RFP was very extensive and for a big name company - it involved few engineering work, Answers to a bunch of questions which needed detailed explanations and were cross functional. This was over the holiday break. I personally was super stoked about this opportunity to present at this big company and took it upon myself to create a spreadsheet with those questions and added couple of columns for primary and secondary owners for each question. I took a good chunk of those questions since as a founder I have a lot of context for many of those questions, and then gave some to my chief architect and vp product.
Tell me about a time when you owned up some responsibilities to get a project done
As we were scaling the company, we had to overcome different challenges, one such was the our churn was on the rise, and we did a lot of analysis and narrowed it down to a few different reasons. One of them was that our customer success team was bottlenecked by a lot of operational work to setup customers rather than focusing on strategy and insights - which is what we found to be key drivers for renewal. I setup various sessions with the operations/customer success team along with product to understand their pain points, help product narrow down the 80/20 of improvements that we can deliver for them in 20% of the time. We worked through to write up those tasks, prioritize them and added that to our sprint to be tackled. The Customer success team clearly showed their appreciation through difference means, its pretty clear when they do that. But we did end up drawing up estimated savings of 3 hours per week per customer success rep from those initial changes.
Tell the story of the last time you had to apologize to someone.
The story that comes to my is one where we were working on the rollout of a new feature and going through several internal trainings for that feature. Before one such demo, late in the day, the Project Manager in the team asked me what was the best way to go about deploying the current version of the system for the demo tomorrow. Some context in this is that we we rerun numbers in an environment, the pitch had to be changed since the pitch was focused on certain articles and data numbers being present. We had actually created a new environment just to demo this feature, and we also have staging environment where we can push code. I told the Project Manager to use the staging environment since the new environment was being used by the sales team. However, neither I nor the project manager communicated this to the PM. Unfortunately, just before the internal demo, someone ran a migration on staging that brought it down. I tend to make sure that the product owner is communicated when these decisions are made, apologized for it, but we continued our way.
Tell me about a time when you gave a simple solution to a complex problem.
We learned that one of the key drivers for reducing churn was to make using our platform a process in the customer organization. This came out to be a common theme among the customers that adored us. We wanted to figure out low engineering effort, high impact projects that will help us achieve this. I came up with two innovative ideas that really helped us move the needle. We already had a reporting interface and reports that could be made. And we had an email system, so a natural but interesting next step was schedulable reports. This ensured that the customer could schedule this report at a frequency that was convenient to them. We had email open tracking, and we found that we had high open rates in those emails compared to our normal emails, but not necessarily higher click rates. This led us to believe that this feature helped with another aspect - which is making our company well known across the org.
Tell me about a time when you invented and simplified a nagging problem for developers.
One of the issues we faced as a team was that we had multiple environments that we were all pushing to with little to no visibility as to who was using a particular environment when. A senior dev brought up this issue to me and said he had some ideas on how to solve this - a service that keeps track of who is currently using which environment with the ability for different members to claim an env for some Time. I immediately jumped on it, provided a few of my ideas, specifically - I asked him to include a slack integration. I also said I was ready to sponsor it (provided credits on Heroku) and told him to add it as a 20% time as part of his sprint. Out of his own enthusiasm he spent a weekend and built it end to end and did a demo for the team. To this date, the team uses the tool to claim different environments before using it. Afterwards he specifically expressed his admiration for my sponsorship of the project and his satisfaction at seeing it being used regularly.
Tell me about a time when one of your innovations was mis-understood by the team
One of the features we wanted to build in our product was a search tool that was simple to use but at the same time had complex functionality. Me, our Chief Architect and our CEO essentially invented what we called the Visual Search - a search that was in the middle. There was significant resistance within the company around this because they believed that our customers were used “Google” like search and will not appreciate this interface. Over time, we found that once customers learned the UI, they were hooked on to its methodology and would become power users of the product. Our future efforts revolved around getting people over the initial hump/resistance to trying.
One of the technical debt items that existed in the product was the duplication of backfill jobs on each change made by the customer, this caused a lot of load in our system and we would often have to wake up or put attention to alleviate the load. After lots of back and forth with some of my senior engineers, we came up with a. Solution that we eventually implemented as well. However, to actually get this change into the product, we had to change the workflow for the Operations team a little bit - it was actually making their lives easier - but they were reticent to change. Got on various different calls with different members of their team to explain the pros of going with this approach, convicted them and made the switch at the decided upon date. Things went much more smoothly since both for engineering as well as Customer success.
The idea was to build a system that handled backfilling data, running specific jobs on all the data in parallel with different settings changed for a customer. We went through a big exercise to set up the backfill manager as the system that will manage system changes.
However, there was one change that was admin facing - which could be automated by just using backfill manager as well
- This will ultimately benefit the customer - because they’ll see fewer bad articles in the platform
- It will help us be more efficient with our resources because rather than doing backfills on every change, it auto backfills data
- Visibility, etc etc.
We demoed it to the CSM team, while at first they were supportive, they came back and said they aren’t comfortable going with this solution
- It changed their workflow
- There was uncertainity over when and what changes were effected by the backfills
I responded by explaining to them why this is critical and explained how this should have been designed this way from the beginning.
I had them put together a spreadsheet of their concerns, and answered each one of them.
In this exercise, i had them direct me to the MVP, because while there is a lot of additions that we can implement in the tool, we need to get this delivered to achieve those benefits
Tell me about a time when your judgement was right and saved the company a lot of trouble
Due to experience of having seen various production systems, it helps make judgement calls on unknown things that haven’t been decided yet. One simple example is that we were planning to change the domain name of the website from airpr.com to onclusive.com. This means updating the records for our email provider, updating CNAMES in our dns provider etc etc. The marketing team obviously wanted a smooth roll out and they considered sending a one shot email to everyone informing them of the switch over. However, I advised and argued against that even though that might seem better for the immediate user. I pushed hard for a phased roll out where we can turn on few customers at a time and see how they behave. Lo and behold during the migration process, we had a customer complain of service disruption. We could isolate this to that customer, reach out to Cisco domain blacklists and remove our site from there. We turned it back on and then it work. We did the same with email providers, gave them a lot of notice and then went ahead with the switch
Tell me about a time when you were wrong.
When we were working on this product called Intelligent contacts, there were many market factors that led us to speed up the release of the product. A specific feature in this product was recommendations which showed other authors for each each author being viewed. After reviewing a few different authors, I concluded that the product met the non functional requirement of query latency and gave the go ahead for a beta release. However, when users played around with us, they found inconsistent response times. Some authors would return fast response while others wouldn’t. The good part was this was just a beta, however I ensured that going forward I would check other areas as well such as performance monitoring tools before making a decision.
Tell me about a time when you had to work with incomplete data or information.
In a startup, we often have incomplete data - the most obvious one was when working on the problem of connecting visitors back to PR
• Tell me about a time when you influenced a change by only asking questions.
In the past year, we once had a release that went particularly wrong. What happened was after we rolled out the release, in the upcoming days we found that all articles were getting tagged with certain tags when only a few had to be. This went through unit testing, regression testing, etc. I called for a retrospective and merely played the role of asking questions. I had everyone give ideas/suggestions for what we could have done to prevent this. Each person came up with ideas, its surprising sometimes how when you call on people to give thoughts, they come up with highly refined thoughts. One such was to do a canary release, instead of a full rollout - which we embraced since. I stated what the issue was and opened it for brainstorming solutions…
First set of discussions lead us to identify the root of the problem.
blacklist and whitelist were being managed in the same table with a flag - which violated the single responsibility principle in some ways
- regression testing wasn’t done on production data
- we don’t really have production type data on our local
- print publication tags, broadcast tags and domain tags were maintained in the same table but each of them had different constructs.
Kept going deeper and found that the bigger issue was that we couldn’t do a canary release, we just didn’t have good ways to do that.
Tell me about a time when you solved a problem through just superior knowledge or observation.
Due to experience of having seen various production systems, it helps make judgement calls on unknown things that haven’t been decided yet. One simple example is that we were planning to change the domain name of the website from airpr.com to onclusive.com. This means updating the records for our email provider, updating CNAMES in our dns provider etc etc. The marketing team obviously wanted a smooth roll out and they considered sending a one shot email to everyone informing them of the switch over. However, I advised and argued against that even though that might seem better for the immediate user. I pushed hard for a phased roll out where we can turn on few customers at a time and see how they behave. Lo and behold during the migration process, we had a customer complain of service disruption. We could isolate this to that customer, reach out to Cisco domain blacklists and remove our site from there. We turned it back on and then it work. We did the same with email providers, gave them a lot of notice and then went ahead with the switch
Tell me about a time when you mentored someone.
I’m a big fan of Kim Scott’s book - Radical Candor. One of my personal successes is when I turned around an engineer who wasn’t performing. The specific shortcomings that I noticed was
- His unwillingness to go the extra mile to understand what the underlying issue is, often would fix the symptoms rather than the cause.
- His inability to learn to dive into more complex parts of the codebase and understand and make changes in an intelligent way.
However, he was very personable, everyone liked being around him and would improved culture overall in the company. I setup weekly 1:1s where would review his progress on each of those items. The biggest issue as it turns out for him was that he had come from an environment where this wasn’t the norm and things were handed out to him. A lot of it was reminding him of how startups operate and how he needs to step up. Fortunately, he was someone that was very keen to learn. I encouraged him by giving him tasks that needed pairing with senior engineers and also had the company pay for some training material online. It took him few months, but he started showing improvements and inclination towards support engineering. We placed him in that role and his personable character helped him earn respect from his colleagues in that department and his engineering background helped him tackle the harder tasks in the support engineering team. (Became the go to guy there)
Tell me about a time when you made a wrong hire. When did you figure it out and what did you do?
I’m a big fan of Kim Scott’s book - Radical Candor. One of my personal successes is when I turned around an engineer who wasn’t performing. The specific shortcomings that I noticed was
- His unwillingness to go the extra mile to understand what the underlying issue is, often would fix the symptoms rather than the cause.
- His inability to learn to dive into more complex parts of the codebase and understand and make changes in an intelligent way.
However, he was very personable, everyone liked being around him and would improved culture overall in the company. I setup weekly 1:1s where would review his progress on each of those items. The biggest issue as it turns out for him was that he had come from an environment where this wasn’t the norm and things were handed out to him. A lot of it was reminding him of how startups operate and how he needs to step up. Fortunately, he was someone that was very keen to learn. I encouraged him by giving him tasks that needed pairing with senior engineers and also had the company pay for some training material online. It took him few months, but he started showing improvements and inclination towards support engineering. We placed him in that role and his personable character helped him earn respect from his colleagues in that department and his engineering background helped him tackle the harder tasks in the support engineering team. (Became the go to guy there)
• Tell me about a time when you couldn’t meet your own expectations on a project.
When we were working on this product called Intelligent contacts, there were many market factors that led us to speed up the release of the product. A specific feature in this product was recommendations which showed other authors for each each author being viewed. After reviewing a few different authors, I concluded that the product met the non functional requirement of query latency and gave the go ahead for a beta release. However, when users played around with us, they found inconsistent response times. Some authors would return fast response while others wouldn’t. The good part was this was just a beta, however I ensured that going forward I would check other areas as well such as performance monitoring tools before making a decision.
• Tell me about a time when a team member didn’t meet your expectations on a project.
There was a senior member of my team. He is my go-to person for everything. He was passionate about code cleanliness and quality and ensured that he made it known in PR requests and such. Further, he would highly encourage writing tests for the codebase. Unfortunately, our early codebase didn’t have tests and it came back to bite us. So we embarked on a project of refactoring which took longer than expected to complete. At the end of the project, he requested that we have a retrospective, and in the retrospective he shared a negative attitude towards a lot of things and when we asked him for what his suggestions were, he didn’t have any and mostly blamed the past or other engineers. It was unlike him, so I had a conversation. he complained that there wasn’t enough help provided to him, though he would never come up to me and say that. I aplologized to him, however, I told him that in order for him to grow in his role, I needed him to step up and let me know when project is going haywire.
At AirPR, we had this nicest engineer, he was in a distributed office, but literally available all the time to answer questions. He was junior, wouldn’t write tests for the code he writes, however in my meetings with him he would say he understands that he needs to set those standards. What I did was two things 1) I made it a point that all engineers on 1 customer success call - this gives them a sense of the types of issues customer face. 2) I ensured that engineers are given access to a tool that shows videos of customers using the platform - it provided a first hand experience of what the customer was experieicing. After continuous coaching and helping learn the customer’s situation better, it helped him a lot to improve.
• Tell me about a time when you went way beyond the scope of the project and delivered.
Project was to tie website interactions back to PR
Tell me about a time you made a mistake
We were going through this effort of tackling some technical debts, and what happened was that I would often take up the task of keeping our technology updated. The problem in this case was, our code used a CSV parser that was built with Ruby and they had a breaking change. My mistake in this case was that I hadn’t informed the team until after the fact and hadn’t setup the right alerts. I apologized to the team, and ensured that the upgrades still happen but with advance notice (planning for them is always better).
Apologizing to Judy for wrong environment deploy
Tell me about a time when you couldn’t meet your own expectations on a project.
We were going through this effort of tackling some technical debts, and what happened was that I would often take up the task of keeping our technology updated. The problem in this case was, our code used a CSV parser that was built with Ruby and they had a breaking change. My mistake in this case was that I hadn’t informed the team until after the fact and hadn’t setup the right alerts. I apologized to the team, and ensured that the upgrades still happen but with advance notice (planning for them is always better).
Apologizing to Judy for wrong environment deploy
Tell me about a time when you went way beyond the scope of the project and delivered.
One example that comes to my mind was that of a big redesign of the product that we did in late 2018. The idea was the change out all of the UI. We had a deadline to get this done as it was critical to deliver this to our customers to have our competitive edge. We had just gotten a few new hires and it helped that we decided to get them on board on the new project since their perspective was quite helpful. What we did was break up the tasks and initially encouraged backend focused engineers to take up frontend aspects of the backend tasks as well. This provided them with a way to get to know other parts of the codebase. This encouraged team to collaborate regularly because knowledge needed to be shared. Also, I’ve found that creating avenues to celebrate small wins is a way to motivate engineers. I introduced the concept of demo days where engineers will be demoing even small iterations that they worked or working on. This gives avenues for other members in the team to provide their ideas and iterate even before the product gets into the hand of the customer. I ensure that there is a product member to take notes from that meeting. Lastly, retrospectives post project and celebration post project was critical. I ensured that I pre arranged a celebratory happy hour in collaboration with HR. Gives them a sense of victory and prepares them for the next big project.
Tell me about a time when you had to work with limited time or resources.
At a startup it’s very important to be mindful of engineering spend as it can balloon up very quickly. We take cost management of our AWS resources very seriously, use third party providers to keep track of under utilized resources and spikes and dips in usage to ensure we have the right reserved instance allocation. We also ensure that wherever possible our code is optimized so that CPU/Memory workloads are not too high, in Elasticsearch for instance scripts cannot be cached so they tend to be high on memory resulting in a lot of GC - work actively to avoid using scripts when possible. In mySQL for instance, we don’t have it in shared or clustered multi master mode yet - so we have a lot of queries that do joins across tables. Being aware of the query plans for those and making the necessary changes to limit the cpu utilization resulted in us being able to go down one tier. We went from a r4.8xl to r5.4xl. Always looking at performance improvements in not just our own code, but also the open source and closed source libraries we use. Ensure that those tech debt carry weight when being considered during the sprint planning process. Lastly, it’s about being aware of latest developments in the world of cloud infrastributre. One such example was when we originally were processing pixel requests from our javascipt that we drop on our customers websites. Those were originally load balanced onto servers that processed them, however, it wouldn’t sale well from cost perspective, because there used to be peaks during day and troughs during nights and we had provisioned for peaks. We made the big decision to move to Kinesis - Lambda for that which caused our costs to become 1/10th of what we were spending.
Describe a time when you saw some problem and took the initiative to correct it rather than waiting for someone else to do it.
Performance MySQL:
I remember we were hitting limits on our MySQL cluster and had to improve that. I performance monitoring, looked at queries that returned the most rows, or took the most time. I also noticed that sometimes the query optimizer was doing a terrible job.
It fell into a variety of different cabers:
- Elasticsearch query inconsistency
Tell me about a time when you took a calculated risk.
We were going through a big release - a new version of our reports. This involved database migrations. We had devised a strategy for this migration where we would have a column specifying the version of the reports and the code would read that to determine how data was parsed and things would work fine. After a lot of alpha testing we decided to roll out the new release to customers. After the
Tell me about a time you needed to get information from someone who wasn’t very responsive. What did you do?
I had a customer success professional who wasn’t very responsive. What is the best way to reach you - get a sense - don’t want to disturb you, may be setup a meeting during which time we can sync up.
Tell me about a time when you had to tell someone a harsh truth.
I’m a big fan of Kim Scott’s book - Radical Candor. One of my personal successes is when I turned around an engineer who wasn’t performing. The specific shortcomings that I noticed was
- His unwillingness to go the extra mile to understand what the underlying issue is, often would fix the symptoms rather than the cause.
- His inability to learn to dive into more complex parts of the codebase and understand and make changes in an intelligent way.
However, he was very personable, everyone liked being around him and would improved culture overall in the company. I setup weekly 1:1s where would review his progress on each of those items. The biggest issue as it turns out for him was that he had come from an environment where this wasn’t the norm and things were handed out to him. A lot of it was reminding him of how startups operate and how he needs to step up. Fortunately, he was someone that was very keen to learn. I encouraged him by giving him tasks that needed pairing with senior engineers and also had the company pay for some training material online. It took him few months, but he started showing improvements and inclination towards support engineering. We placed him in that role and his personable character helped him earn respect from his colleagues in that department and his engineering background helped him tackle the harder tasks in the support engineering team. (Became the go to guy there)
Give me an example of where you dived deep to fix an issue
Bug in Whitelist tag - don’t fix by backfilling, but by finding root cause
This happens a lot in day to day engineering. There is a bug that comes up and the engineering team is asked to look at. Something that I encourage my teams to do is always focus on identifying the root cause and not fixing the symptom. This one case that comes to my mind is when we released a new feature to tag articles that are from specific domains. What happened was that suddenly all articles were getting tagged with those tags even though those domains weren’t added. Some of the suggestions were to run a backfill job to fix the bad profiles. I encouraged the team to take a multi-pronged approach. First fix the immediate symptom, then add logs that show if the issue reoccurs, Use log aggregators to look for anomalies or recurrence of the error. Then apply the fix in a systematic way by actually having someone review it. Because those fixes have strong consequences.
Tell me about a time when you had to step up and disagree with a team members approach.
Tell me about a time when you did not accept the status quo.
In a professional setting, to work in a cohesive environment, its often good to form strong bongs with peers you work closely with. After all the product result is a team effort and not the result of one single person. However, it’s important to state your thoughts and opinions even if it may differ from someone you work with on day to day basis. And it’s important to substantiate it with your thinking and explanation. One example that comes to mind is when as an executive team, we were deciding product priorities for the next quarters. Requests in situations such as these come from different angles, some from customer success, marketing, sales (potential customers), engineering (big technical debt). In one such meeting we had recently, I proposed that we prioritize improving our attribution model. To substantiate this, I had pre-planned a bunch of different notes - like talking about
- What are the current shortcomings?
- How many customers have reported issues in the model and what are the top issues being reported (data from customer success obtained before hand)
- What steps need to be taken to improve the model?
- What is a realistic time estimate of those steps?
- What is the expected impact/outcome of these changes?
This one time, I was able to convince the team that this is the direction we would like to take. There have been other times in similar meetings where I’ve had to commit to something that’s not favorable to my point of view.
Tell me about an unpopular decision of yours.
No deploys between 9AM and 5PM
If your direct manager was instructing you to do something you disagreed with, how would you handle it?
Give them some backing on why I believe that’s not the approach to take in this situation. I will back my answers and suggestions with data I can find. If beyond that we decide to choose an approach, its important to follow through on that and make it successful.
What did you do when you needed to motivate a group of individuals or promote collaboration on a particular project?
By providing an example, tell me when you have had to handle a variety of assignments. Describe the results.
One example that comes to my mind was that of a big redesign of the product that we did in late 2018. The idea was the change out all of the UI. We had a deadline to get this done as it was critical to deliver this to our customers to have our competitive edge. We had just gotten a few new hires and it helped that we decided to get them on board on the new project since their perspective was quite helpful. What we did was break up the tasks and initially encouraged backend focused engineers to take up frontend aspects of the backend tasks as well. This provided them with a way to get to know other parts of the codebase. This encouraged team to collaborate regularly because knowledge needed to be shared. Also, I’ve found that creating avenues to celebrate small wins is a way to motivate engineers. I introduced the concept of demo days where engineers will be demoing even small iterations that they worked or working on. This gives avenues for other members in the team to provide their ideas and iterate even before the product gets into the hand of the customer. I ensure that there is a product member to take notes from that meeting. Lastly, retrospectives post project and celebration post project was critical. I ensured that I pre arranged a celebratory happy hour in collaboration with HR. Gives them a sense of victory and prepares them for the next big project.
What is the most difficult situation you have ever faced in your life? How did you handle it?
Database getting erased
Give me an example of a time when you were 75% of the way through a project, and you had to pivot strategy–how were you able to make that into a success story?
Backfill manager
Describe a situation where you thought you were right, but your peers or supervisor didn’t agree with you. How did you convince them you were right? How did you react? What was the outcome?
Backfill Manager
Tell me about a time when you observed two business opportunities to improve ROI, and how you determined they were connected.
Entry level product for customers - Trends - provided value add for existing customers…
Give me an example of making an important decision in the absence of good data. What was the situation and how did you arrive at your decision? Did the decision turn out to be the correct one?
Deciding the polling frequency for social media data. - Fibonnaci sequence - we were not hit by a lot of calls at the same time we were able to get good enough granularity thet we wanted. There were those occasional jumps in some articles that we missed as a result.
Tell me about a time you disagreed with a colleague (or a boss). What is the process you used to work it out?
Credentials for production data to the whole team
An example of where you have sponsored your reportee’s initiative and take it to the next level.
Uncover new insights that we never had access to. For instance, a visitor might have read the article came to the site, and then many days later come back again to perform an interaction - because we were doing the analysis at the level of the visit, we could uncover those insights.
Explore Lambda early on even though it wasn’t really production ready. We were able to cut the expenditure by 1/5th.
Top performer not performing to their potential
Early member of the team, would have ups and downs - cases where they would produce some of the most magical results and other situations where the output would dip to 0, he would be building a 3D printer and spend all day exploring those.
I didn’t want to go the route of micromanaging, so spoke to him about what was going on and probed into the reasons for these swings. And more importantly asked him if there was anything going on personally that may be affecting his output. One of those junctures he infact was going through some personal issue, gave him an objective take on how to deal with that situation and possibly use intellectual work as a means to get over that situation. He would improve slightly and then go back into the roller coaster again. Delved deeper, and found that out that he was extremely motivated to deliver when there was some goal he was pushing towards, and he loved stretch goals, would often even hit them - and he would often be bored. I created opportunities for him to learn about customers. I would include him in customer meetings and would often give him hard problems, you have to play a balance here because if you give one person too many of the hard problems
- the rest of the team will feel comfy that this guy will be taking care of them
- and other good, strong members of the team will leave because they aren’t getting the hard problems.
So we would include him in discussions of the architecture regardless of the hard problem so different members are involved and then I would give ownership of the project to other team members.
Think of a project that was a failure, why did it fail and what did you learn from it?
There have been small failures during the course of a project, but the one where the project was a failure was when we built a product called Authors Database, we built the UI for users to be able to search for any author name and see Author data for that name, and various filters. To build this data, we leveraged our crawler database, some vendors to build the database. We released it in Beta, but the feedback we received wasn’t positive. Customers were not able to Continues to be in Beta because we weren’t able to deliver the contacts part which is important.
- Understand data availability and do some investigation to get some data metrics first before spending UI resources to see viability of project
- Create admin tools to be able to fix things.
- It’s OK to leverage non technology approaches to supplement the technology work that we do to create a product that people want.
Tell me a time when you’ve had to take a bold decision
In a professional setting, to work in a cohesive environment, its often good to form strong bongs with peers you work closely with. After all the product result is a team effort and not the result of one single person. However, it’s important to state your thoughts and opinions even if it may differ from someone you work with on day to day basis. And it’s important to substantiate it with your thinking and explanation. One example that comes to mind is when as an executive team, we were deciding product priorities for the next quarters. Requests in situations such as these come from different angles, some from customer success, marketing, sales (potential customers), engineering (big technical debt). In one such meeting we had recently, I proposed that we prioritize improving our attribution model. To substantiate this, I had pre-planned a bunch of different notes - like talking about
- What are the current shortcomings?
- How many customers have reported issues in the model and what are the top issues being reported (data from customer success obtained before hand)
- What steps need to be taken to improve the model?
- What is a realistic time estimate of those steps?
- What is the expected impact/outcome of these changes?
This one time, I was able to convince the team that this is the direction we would like to take. There have been other times in similar meetings where I’ve had to commit to something that’s not favorable to my point of view.
What was one of your most important/impactful projects? Which of those were actually realized?
What did you think the risks of the project were when you first started?
The risks were that it was changing an existing algorithm which involves data, so the risk of acceptance loomed large. Training, marketing are the best solutions to solving those problems, having internal tooling to help answer questions. And last being able to revert.
Can you tell me about four people whose careers you have fundamentally improved?
- Wendy, she was my tech lead. When she came in, she had experience. I grew her from a Junior engineer to a tech lead.
- Dvij
- Patrick - Chief Architect, brought him along for the ride
- Derek - distributed leader -
Describe a few of your peers at your company and what type of relationship you have with each of them
Patrick - chief architect, we intuitively know what goes to whom, because we have set ourselves functional areas that we manage throughout.
Judy - vp product - earning trust is very important - playing the how aspect
Enid - head of customer success
What did you do on your very best day at work?
My personal favorite was when we demonstrated the capability of the system we built to a public company (valued at the billions of dollars) - being prepared for that meeting, not with just a presentation, but an actual demo. Coming back and sharing that enthusiasm with the team who were looking forward to hearing all about it
What does office politics mean to you, and do you see politics as your job?
First thing is sometimes politics do exist, but often times it’s about getting people on the same page. And when you do see politics, don’t run away from it, face it. By face it, ask those questions and be involved. Have opinions and ideals and stand by them. Don’t form close groups, because its important to understand that all or most people are still in their job because they are doing something right. Try to focus on that and use that to your and company’s advantage. Avoid fueling it, and avoid passing on rumors.
https://www.mindtools.com/pages/article/newCDV_85.htm
How many people have you personally hired and what did you look for?
Technical Expertise
Values Fit
Product and Design Sense
Tell me an instance where you had a conflict with your boss?
I will say there have been times of disagreement in opinion and approaches and usually one of us would convince the other through reasoning on why approach is better than the other. But some others are more fundamental in nature, so it’s better to talk directly and discuss. This particular conversation I’ve had with my boss was when I would notice that after interviews to get opinions he would call all of us into the room but not necessarily get the opinions of everyone in the room, would ignore most people’s opinions and focus on one or two people.
Hiring and developing people; setting up strategy; examples when you insisted on a higher standards etc
- Psycological safety
- How comfortable is the team in taking risks and better yet can come to a meeting, throw their hand up and talk about what worked or didn’t work. Creating an environment where the team feels secure and that they can contribute to the success is very important. Be vulnerable in front of each other - don’t worry about ideas being shot down, that’s OK. But if the idea is being shot down every time, or people are being singled out, that isn’t a culture that people want to partake in.
- Dependability
- As leaders provide avenues for engineers to cross train and provide avenues for them to interact not just with the team members but the entire company. I’ve seen some aha moments come out of Happy hours. And I’ve seen great collaborative ideas implemented as a result of a hackathon.
- Clarity
- The whole idea of stand ups is to do a status check - do people feel confident that they are able to complete the tasks in this sprint, if not, lets provide the support they need to get there. (FIST TO FIVE)
- Meaning and Impact of work
- What is the North Star, why are we doing what we are doing. How is it impacting the customer, bringing that back to the team is super critical. We have a voice of the customer email chain that we use to share feedback from customer. Negative feedback is OK too and the really smart engineer will take that as a challenge and either bring it up in the next meeting or if there is psychological safety go fix it himself/or herself.
Where do you think you need to improve?
Playing the balance between giving engineers the space to do their work and also putting the necessary structure in place to ensure that the product deliverables are being met. What I mean by this is that, especially in today’s world of slack and workplace communications, the easiest route for CS to get their questions will be to reach out to the engineer and have them debug. What tends to happen when you foster that environment, even the smartest of your CS team become lazy over time and start pushing a lot of things to engineering. Setup process, and setup support engineering.
I’m a stickler for good design and minute attention to the details in a design and the UX experience, so I push really hard to make sure even small things are not missed out in a spec. And sometimes this slows things down, because it pushes things more. One way I’ve overcome this is to be clear to engineers that I care about this and why its important so no one is surprised when I bring it up, and second of all try to introduce more checkpoints where I go over the implementation and give preliminary feedback.
How is your relationship with product?
Product usually defines the what and the why and engineering defines the how, they jointly decide the when.
I say usually because there are times and I hope this is often, though in practice may not be every single time, Product and engineer collaborate deeply on features, so while ideas are product ownership and implementation is engineering ownership. Its more like Product is accountable for ideas and engineering is accountable for implementation. This means, don’t work in silos, but rather collaborate.
Examples of this in one direction, is involving engineering early on during product roadmap - engineers come up with areas that can be improved with little effort - easy wins or low hanging fruit. Engineers can also provide solid ideas - these are the engineers with product sense. Because they are also the biggest users of the product. They also suggest new technology that can make things better next time.
Reverse is also true, Product ensures that engineers can chunk their work - and show progress - better communication with business. Product’s input in design sessions give a third party perspective and often useful. And just knowledge about these processes and APIs In a deeper sense gives products more ways to think about “what next” and give an intuitive sense for how challenging different types of work is.
Ultimately both of them are working towards the North Star - driving business value to customers.
Improvement areas that I think about
- Need to be more of a ‘outward facing’ CTO, potentially giving talks at conferences or meetups and finding other ways of influencing the community.
- Need to find more opportunities to talk to the customer directly collaborating and make it a constant process rather than once-in-a-while affair.
- While I believe I do a good job at knowing the developments and updates in technologies we use, I can do a better job at keeping abreast of what’s happening in the world of technologies we don’t use.
- I delegate a lot of the security work/questions/concerns to Patrick, but I should/could be taking up more of those problems.
- Can be more involved in business strategy discussions and decisions within the company - I’ve tried to be more involved in the last year or so, but I think I can/should do a better job at this.
How do you motivate your team?
Everybody loves to solve customer problem, whoever it is. Depending on the audience you are facing, it’s important to articulate the pain point faced by users. When it comes to motivating, I take a two-pronged approach:
- Articulate what is the value in solving the problem, they can come up with the best solutions.
- As a founder, I set the culture.
- Direction on what aspects to think about, being a sounding board can help the team overcome challenges along the way.
- Celebrating the wins is as important as articulating and pushing them.
- Don’t waver on things you believe in.
How to set an example as a leader
I only expect from others what I expect from myself. I will never ask you to do something I’m not doing. All my articles and courses are based on my actions and actual results. But because I have high expectations of myself, I also expect that from the people I work with.
When things go wrong, I try to stay calm. Life is nothing but a series of solved problems. That means we run into challenges, problems, and bad situations all the time. It’s important that you stay calm and take your time to think. If possible, I don’t make quick decisions. When I stay calm, others will too.
When I screw up, I admit it. I’m the last person who believes he’s perfect. Even though I try to stay calm and positive — sometimes I lose my shit too. I admit it and then move on. But I always make sure I learn from it and I never hold a grudge. There’s no place for your ego.
I’m clear about my values and rules. Look, there are certain things I don’t put up with. For example, lazy people don’t get my respect. And I’m not sorry about that. Also, I’m not afraid to attack assholes. People must take you seriously.
I respect others and don’t try to change them or tell them what to do. It’s impossible to change people. You can only change yourself. What other adults ultimately do is none of your business. I can set the right example, but sometimes it doesn’t work or takes more time. I accept that.
How do you give or receive feedback?
Receiving feedback starts with willingness to ask questions and listen. The question may be something like “Is there anything I can do differently to make your life better here?”
Be OK with awkwardness. Sometimes engineers or anyone for that matter, or just a little nervous or hesitent to openly talk about the problems they are facing or confront you with a difficult situation they faced. Give them the space to draw up the courage to speak. When they do speak, don’t interrupt them and keep prodding on small things. When they are all done speaking, confirm your understanding of what they are saying and give them an opportunity to speak some more. Reward the feedback with action.
Giving Feedback:
Be prepared to give feedback, don’t wing it. Be sincere in your approach, you are not giving feedback because you are in a higger paygrade or that you know more than the other person. You give feedback because 1) you care about the person receiving the feedback and 2) it is your belief that if the person takes the feedback and acts on it, it will help him and the company succeed in some ways.
When delivering feedback, make sure to deliver both the positive and negative ones.
The most common way of giving feedback is well studies as SBI -
Give feedback immediately
Praise in public, criticize in private.
https://www.radicalcandor.com/blog/a-hip-approach-to-feedback/
SBI:
https://www.mindtools.com/pages/article/situation-behavior-impact-feedback.htm
Teach me something technical you learned recently
Count Min Sketch
Count mean min sketch
https://en.wikipedia.org/wiki/Count%E2%80%93min_sketch
How to drive strategy?
- Yearly goals and quarterly goals
- OGSM
From product and engineering standpoint, break into quarterly goals.
build product strategy that lines up to this.
engineering
Who is the hardest person you ever worked with? What made the relationship difficult? How did you address that situation? If I called them up today, what would they say about your time together?
It was the VP of customer success. He had a very diverse background. He was a analytics manager and then managed professional services at a large organization. And he had lots of things to say. From my vantage point, the thing that made it difficult was that I felt that he would have something to say about things he knew about as well as things he didn’t know about. Be it technology or also comments on certain parts of the process or technology. What I had to learn over the course of time was two things
- When you are dealing with someone you disagree with more often, make sure to form relationships with that person outside of your normal working routine. What gets them excited, what are their strengths, where do they normally seem to falter.
- Don’t get too hung up on specifics with that person, because it will lead to no man’s land and just gripe. Don’t let those things foster, instead look at the brighter side and focus on that -try to deviate conversations towards that and engage in discussions that both you and the other person can connect with.
- When you are trying to make a point about something that’s your domain, back it up. Have the necessary data or reasoning behind why certain decisions or made or why you are suggesting something.
When I told the team that I was planning to depart in a few months, he texted me and said, you can disagree with me, but it was great working with you. I hope I can take AirPR to 2B, so you can live in your own valley.
What do you do on your best day at work?
- Solving problems - can take many forms:
- Debugging through something to find solutions, improving performance of something that’s been reportedly too slow
- brainstorming solutions for ways to improve developer productivity in a retrospective
- thinking through the best approach to design the UX for a particular feature. - I love showing off what my team has accomplished:
- customers when I get called in for tech reviews
- internal stakeholders, especially the parts that don’t have .a user facing interface.
When is it you have lost track of time in the best possible way?
- Debugging a difficult solution with certain impact.
- Prepping for a presentation for one of our potential customers or existing customers to actually be able to demonstrate the value that we bring.
