Comcast Technology Solutions Logo
Issue 2 | July, 2024

Video Harold

A Newsletter from CVP & AdFusion Engineering

The Power Of A Strategic Pivot

image of 
                    The PHILLIPS multi-plane No.3

Behind every successful innovation, there are multiple tests, failures, and learnings. Today, one would not think twice about the capability of an aircraft while traveling by air. There were multiple versions of flying machines that were tried and tested until a successful working model was achieved. Attempts to get these machines off the ground often ended with the pilot getting a few feet into the air and then crashing gracefully. These repeated smash-ups by aviation pioneers were the only way to learn what worked and what didn’t in the air. We must owe every smooth flight today to those successful experiments and their negative results.

"Strength does not come from never failing, but from rising every time we fall"

This story highlights the complexities of engineering and its relevance to daily software development. Despite product and engineering discussions, rigorous testing, verifications, and demos, crashes and outages can still occur when code is deployed to production. This leads to penalties for failing to meet agreed-upon standards with customers.

Impact of Outage for Customer:

  • Financial ramifications of customer contracts
  • Customers lose money as users churn due to poor reliability

Impacts of Outage for SB&A/Comcast:

  • Tarnish reputation and brand of Comcast
  • Challenge in acquiring new customers
  • Penalties paid to customers
  • The toll on teammates
The time, money, energy spent is not anything less than 14+ days resolving customer concerns:
  • Engineering - Root cause write ups
  • Product - Reprioritize, reshuffle work, disruption
  • OPS - Customer communications and meetings with each customer impacted
  • Sales - Constant reminders of how good the service is, fighting negative impressions
  • SLT - Customer meetings, explanations to Ken and Charlie, brand repair

How we can try and prevent Outages ?

  • Take ownership, responsibility and foster accountability to break the cycle of stress.
  • Teams must ask questions to reveal unknown or overlooked issues to avoid unforeseen risks.
  • Register risks and address the highest-priority ones to eliminate or reduce failures.
  • Ensure confidence in a successful deployment by using a deployment readiness checklist
"Remember, every small effort contributes to the bigger picture. Just as every drop makes an ocean, Together, we can achieve Greatness, one step at a time"

Resiliency

Navigating Uncertainty: Building Reliable & Resilient Systems 3 mins read

In recent months, CVP faced turbulent waters, encountering infrastructure outages and stability concerns across the services. From Seattle data center power disruption to..

Read More
CBL vs ALB image

ReSILENCY- DT Case Study

The Euro Cup 2024, a prestigious football tournament’s broadcasting infrastructure provided by "SB&A" 3 mins read

Imagine being engrossed in an intense IPL final of CSK vs Gujarat Titans, every moment charged with anticipation, when suddenly, the stream stutters and..

Read More
Euro Cup 2024

LRm EKS Migration Case Study

Diving Deep, Rising High - Conquering EKS Migration Without Boundaries 2 mins read

Resiliency is choosing to rise above adversity, standing tall in challenges, and demonstrating innovation even amidst system pressure.

Read More
LRM team

ENGINEERING PREMIER LEAGUE

Success Story on Performance improvement ft. ALEXANDRiA Team! 3 mins read

Optimizing database timeouts is key to ensuring seamless User experiences and efficient system performance.

Read More
tree-shaking

Teams Achievement

Optimizing PR Aesthetics: The Power of Decoration 3 mins read

As part of our continuous effort to enhance code quality and streamlining the development process, several teams within the CVP have begun adopting SonarQube as a key component of their validation workflow...

Read More
PR best practise

Teams Achievement

Innovating with Copilot: Team Hali’s Experience 3 mins read

In the ever-evolving landscape of software development, finding tools that enhance productivity and efficiency is crucial...

Read More
gitHub copilot

RISK REGISTER ANALYSIS

Identifying risks is a way of uncovering opportunities 3 mins read

We welcome E1, E2, and E3 to participate in an awareness questionnaire!

Read More
Risk Register

KNOW YOUR TEAM

Hear It From Our Engineering Managers 3 mins read

Art of unwinding: Offbeat methods used by our Managers to rejuvenate their Engineering Spirit.

Read More
CVP Engineering Managers

Rethinking Work

The 70-Hours A Week Myth 3 mins read

Everyone must have heard Narayana Murthy’s statement about working 70 hours a week, making one wonder, "How is this possible?" It seems excessive, but understanding his statement requires a deeper look...

Read More
The 70-Hours A Week Myth