<!here> Your PM ticket

<!here> Your PM ticket

I really like Sidney Dekker’s books/research on failures, human factors,etc. Especially ‘Drift into Failure’ As a rule no one wants to make mistakes, break things, etc. As things grow, one reasonable, logical choice at a time, they can become more fragile, without looking at the system as a whole. A more effective course would be to look at what lead to and allowed that failure to happen and put guard rails, tests, training, etc in place.

Maxine’s boss really danced around the whole issue and assumed no accountability. The practice of scapegoating just creates a fear based culture where even more people are unwilling to speak up and that eventually leads to more unexpected surprises and hard lessons down the road. It would have been more effective to evaluate the situation using an Agile practice known as a Blameless Retrospective and take the learnings to correct the issue.

Maxine’s boss really danced around the whole issue and assumed no accountability. The practice of scapegoating just creates a fear based culture where even more people are unwilling to speak up and that eventually leads to more unexpected surprises and hard lessons down the road. It would have been more effective to evaluate the situation using an Agile practice known as a Blameless Retrospective and take the learnings to correct the issue.

The culture there is not one of learning, which is a very important trait for companies in fast paced fields

This is an example of a blame/pathological culture https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1765804/pdf/v013p0ii22.pdf, where somebody should paid (scapegoating). A better course of action is to take that as an opportunity for improvements and learning - using a CoE (Correction of Error) with adequate RCA (Root Cause Analysis)

Have the CEO take full responsibility, state that the systems and controls we setup failed us. We want to use this opportunity to learn. John Alspaw will becoming here with his team from Adaptive Capacity Labs to run Post Mortems and help us build more resilient systems.

Have the CEO take full responsibility, state that the systems and controls we setup failed us. We want to use this opportunity to learn. John Alspaw will becoming here with his team from Adaptive Capacity Labs to run Post Mortems and help us build more resilient systems.

Also public RCAs

Also public RCAs

One thing that sticks out to me is in the email preceding the start of Chapter 1, the "team" decided to perform an RCA seemingly only because it was a high-profile problem. If some other issue had occurred, with less market visibility, then the company would probably encounter that issue again (maybe in another place) since it's not a generative culture - they may not even bother with the RCAs in that case.

Effective course of action for the executive team or for Chris?

Chris correctly read and responded to the political environment in his organization. He's not going to change such a deep problem mid-crisis. It was smart to recognize and work within the constraints of the pathological culture. While there are better tools, groundwork for their use needs to be established during a high trust time. It's not a black and white scenario.

Chris correctly read and responded to the political environment in his organization. He's not going to change such a deep problem mid-crisis. It was smart to recognize and work within the constraints of the pathological culture. While there are better tools, groundwork for their use needs to be established during a high trust time. It's not a black and white scenario.

Giving him the benefit of the doubt that could be true. In reading it though I took that he was not supporting or fighting for his employee and that resulted in loss of trust. The implication and based on other comments was that this was a systemic problem. Being told to not rock the boat for a few months is not a good sign of the culture there. Ultimately the job is to make sure the site is up, and that the company is successful. If bringing up concerns about possible problems, and speaking openly and honestly about the decisions is rocking the boat that is a problem.

Business reasons are just as valid as techincal reasons, but they need to be openly and honestly discussed and how it will impact people/projects, etc. In my experience if the business reason is clearly communicated and accepted, if not fully agreed upon, that can take the sting out of any issues. Versus just push the code, we don’t care it it breaks things, etc.

In both cases though also need to bring up the problem and solution. Chris told Maxine but didn’t really do a good job to show that he thought there was a problem and that he was trying to fix the culture. Instead it was just keep your head down, come back in a few months, to the same culture/environment.

There is also the impact to her reputation and how she is perceived in the company. Some might know the culture, others might actually think she was to blame. It could be used to fire her, affect her benefits and interactions with other teams and people within and outside the company.

There is also the impact to her reputation and how she is perceived in the company. Some might know the culture, others might actually think she was to blame. It could be used to fire her, affect her benefits and interactions with other teams and people within and outside the company.

https://www.mercurynews.com/2019/12/17/google-fires-engineer-who-created-browser-pop-up-message-about-workers-rights/

I admittedly don’t know the whole story ^ but If HR came and decided to fire Maxine because she got the blame, where would that go?

I love the Rich Hickey inspired focus on simplicity- avoiding “complexted” software. Simplicity should be a design goal. I have also heard this referred to as accidental complexity. Software should be easy to change.

I love the Rich Hickey inspired focus on simplicity- avoiding “complexted” software. Simplicity should be a design goal. I have also heard this referred to as accidental complexity. Software should be easy to change.

There's been a few comments regarding psychological safety throughout this thread. Along the same vein, a new (to the project), talented (fortune 50 sought after consultant), and experienced (25 years) developer joins the documentation and build team and no one has time (too busy) to give her a baseline foundation to build off of and start contributing. A lot of organizations in the tech space usually focus on the technology used to deliver software and neglect the more human side of the systems and processes used to ramp up new individuals and incorporating them into team "pipelines". Curious for the group's feedback on unique and/or best practices within your organizations for on-boarding Engineers/Product Managers/QA/Architects/etc. onto your teams?

There's been a few comments regarding psychological safety throughout this thread. Along the same vein, a new (to the project), talented (fortune 50 sought after consultant), and experienced (25 years) developer joins the documentation and build team and no one has time (too busy) to give her a baseline foundation to build off of and start contributing. A lot of organizations in the tech space usually focus on the technology used to deliver software and neglect the more human side of the systems and processes used to ramp up new individuals and incorporating them into team "pipelines". Curious for the group's feedback on unique and/or best practices within your organizations for on-boarding Engineers/Product Managers/QA/Architects/etc. onto your teams?

It is also very telling that even Maxine is affected by the culture of blame. In fact, it seems like she would normally call BS on things that are BS and not care if she offended someone. However, after the reassignment, she seems to only speak her mind when it slips out, not as conscious effort. This is where we start to lose innovation, and it seems only when she finds the safety of the rebellion that she finds her true voice again. I have seen this in action and have had to step in and try to reshape the situation when less progressive managers have started to de-motivate the teams they are leading. I would talk with them one-on-one about what some unintended results could be if they continue to cast blame on team members instead of looking at the situation as a whole, and as an opportunity for growth and improvement.

“Slow is fast” I’ve heard and seen a lot that people are too busy to help new hires get up to speed, or keep the docs up to date,etc. Also new-hires, especially, are not a surprise so being able to mentor and help them onboard should be accounted for and not seen as a chore. I’ve seen mixed results with where you get a mentor when you join. Those who see the value and how it pays off quicker and expotentially in the long-run are great. Those who see it as a chore or burden, or don’t or can’t make the time, not so much. It comes back to the culture. Worse I’ve seen is where there is no documentation, not willing to give access ( read only even ) to look at things, till you sit with someone, someday. Best practice, plan and acknowledge you are going to be making an investment in a person. Help them get up to speed, truly mentor them. Keep the docs up. If you are not going to give access, have concrete rquirements of what needs to be done to get access. Worse things I saw was a small team that wouldn’t give people access, read only even, till they felt they could trust the person they just hired. No hard criteria just the team had to agree they felt comfortable.

“Slow is fast” I’ve heard and seen a lot that people are too busy to help new hires get up to speed, or keep the docs up to date,etc. Also new-hires, especially, are not a surprise so being able to mentor and help them onboard should be accounted for and not seen as a chore. I’ve seen mixed results with where you get a mentor when you join. Those who see the value and how it pays off quicker and expotentially in the long-run are great. Those who see it as a chore or burden, or don’t or can’t make the time, not so much. It comes back to the culture. Worse I’ve seen is where there is no documentation, not willing to give access ( read only even ) to look at things, till you sit with someone, someday. Best practice, plan and acknowledge you are going to be making an investment in a person. Help them get up to speed, truly mentor them. Keep the docs up. If you are not going to give access, have concrete rquirements of what needs to be done to get access. Worse things I saw was a small team that wouldn’t give people access, read only even, till they felt they could trust the person they just hired. No hard criteria just the team had to agree they felt comfortable.

I also find it is helpful to have a concrete project or set of tasks to help show how things are done, talk about the why, be responsive and open to discussing the why and why not another way.

Re: What would have been a more effective course of action? When there are pressures to identify and individual, it is often possible to redirect somewhat by identifying the person or groups closest to the issues and saying “Let’s get them together to figure out what went wrong”. We do not want to erode the culture of accountability - we want ot strengthen it. The managers role should be to take crises and turn them into opportunities for the organization as a whole to learn and improve process.

Re: What would have been a more effective course of action? When there are pressures to identify and individual, it is often possible to redirect somewhat by identifying the person or groups closest to the issues and saying “Let’s get them together to figure out what went wrong”. We do not want to erode the culture of accountability - we want ot strengthen it. The managers role should be to take crises and turn them into opportunities for the organization as a whole to learn and improve process.

Re: Share experience with office crises and how politics cause negative or positive backlash The most recent crisis we faced here was release that went late, surfaced several new bugs during the release process, and missed a couple of bugs that were discovered by customers. The response was a meeting with all the development and operations team to identify points where we started to lose control of the release and to design new processes to prevent recurrence without slowing release tempo. That took good leadership from my boss and a pretty well developed network of one-on-one communication skills. Two skills mattered in particular: 1) people have to be skilled at being honest without making people defensive and 2) people have to be able to recognize and change their stance when they start to become defensive - particularly as self-preservation instincts can blind us to the things we need to be learning. Both of these are aspects of psychological safety, and they both reflect the politics of the last crisis and create the politics we carry into the next crisis. In our cases there are a few relationships that show strain, and I’ve felt obliged to reach out individually to let people know they were heard or to encourage people to see a different part of the picture or give their colleagues the benefit of the doubt. The goal is to heal those relationships so the the next conversation starts out with an equal chance to be productive. (The organization is small and fairly flat, so there is not too much in the way of institutionalized politics to deal with)

We had an excessively prolonged outage for a service with upset business stakeholders and customers. There was an abundance of finger pointing. We did a post mortem with the key business stakeholders in the room that shined a light on how siloed and isolated our problem resolution process was. It was much like Maxine's woes getting results by opening a ticket. We had numerous communication breakdowns, a lack of ownership, and unresolved tickets closed. The skills we needed were around strong communication and ownership. As an organization we still have work to do in bring down silos. We've mitigated the effects of our siloed structure to this point by leveraging a small number of cross team relationships that have high trust. Plenty of work to do in this front.

You're all amazing. These are such thoughtful answers and discussions.

<!here> We're going to start posting more discussion questions each day now that people are getting situated. Also.... join the latest channel we created for unguided discussions, the #dockside.

<!here> We're going to start posting more discussion questions each day now that people are getting situated. Also.... join the latest channel we created for unguided discussions, the #dockside.

Research has shown trust to be a foundational building block for teams that are innovative, agile and efficient. Sarah's premature launch of the Phoenix project is a breach of trust: she didn't trust the feedback she received about the project's unfinished state.

Research has shown trust to be a foundational building block for teams that are innovative, agile and efficient. Sarah's premature launch of the Phoenix project is a breach of trust: she didn't trust the feedback she received about the project's unfinished state.

I would add lack of undestimating complexity of the system that company is using which comes froom lack of knowledge and understanding how big the problem is. This not her fault i assume but could be that there was no attempt to let her understand complexity and dependencies. Pushing blindly was sure the worst thing to do. Pushing and trying to uderstand and support the tech teams in solving core problems would be much better approach.

Agility, in particular, but also innovation and efficiency must be built on a foundation transparency and sufficient technical excellence. Neither are present in the, apparently, unilateral decision to launch.

The disconnect is that the importance of the project is unrelated to the status or health of the project, and no resources have been committed to actually achieve the agility or efficiency she demands. This is somewhat similar to building a house, and spending an extra month with the architect, and then telling the painter that it is important that he is done today, because you intend to move in tomorrow, because you need to vacate the house you sold. Here the importance of completing the paint job doesn't change the fact that the walls are not dry or that the pluming isn't complete yet.

I can see how an uninformed business person might interpret "agile" as being able to "launch before the full product is ready". We talk extensively in Agile/Lean about experimentation, iteratively developing/releasing toward a goal. But this is not the position Sarah is in. Sarah is not a "team player". She's highly "command and control" in her leadership style. She is used to getting things the way she wants them, and when she doesn't, she will bully someone into giving it to her. She had the opportunity to align herself several times as the teams started to come together, but instead she intentionally separated herself. In the end, I was pleased to see that Sarah was ousted from the company. In my experience, command and control leaders might wield power in the short term, but they always lose in the end.

I can see how an uninformed business person might interpret "agile" as being able to "launch before the full product is ready". We talk extensively in Agile/Lean about experimentation, iteratively developing/releasing toward a goal. But this is not the position Sarah is in. Sarah is not a "team player". She's highly "command and control" in her leadership style. She is used to getting things the way she wants them, and when she doesn't, she will bully someone into giving it to her. She had the opportunity to align herself several times as the teams started to come together, but instead she intentionally separated herself. In the end, I was pleased to see that Sarah was ousted from the company. In my experience, command and control leaders might wield power in the short term, but they always lose in the end.

Re: What is the disconnect …? Innovation and agility almost by definition require some degree of autonomy. Dictating that launch reduced that autonomy even more than had previously been the case. Further, as a development methodology agile values Individuals and interactions over processes and tools, and Customer collaboration over contract negotiation. She’s pretty much torpedoed both those agile values, and has essentially replaced “responding to change” (i.e., what is really happening with deploy process) with her plan. Her plan will not allow change, regardless of what happens. Not agile.

There is a mindset shift that many command and control leaders will not make. Agile, lean and DevOps began with the realities of the work and the workers themselves. But many leaders take over because they think their job is "own" the change. But as Lyssa Adkins said: > if someone, for whatever reason, does not choose to make the shift from finding their satisfaction in driving deliverables, to finding their satisfaction in growing people and environments so that they can drive deliverables, they won't be an effective agile [leader].

Theory of Constraints talks about Constraints as bottlenecks to eliminate which makes total sense. However in real complex environments it is difficult to put your finger on the Constraint. In Complexity you look at exploiting the present and organizing around outcomes. This is why I liked how Bill was working with ToC (Complicated in Ops) and how Maxine was organizing around the Ideals (Complex).

What continues to jump out at me, chapter over chapter, is the people needed to make the project a success are still not working together. Chapter 14 or 15 data scientists just pulled in.

RE Sarah’s disconnect: It’s fear. She’s afraid that if Parts Unlimited does not ship Pheonix then they will not be able to compete in the market. That’s a valid fear. The disconnect comes from misunderstanding software delivery. Given Phoenix hasn’t shipped for years then it’s no surprise that she assumes people are dragging their feet, thus if they can just stop that then things should proceed. Also the years long lead time makes the fear worse, so fear should be mitigated ASAP (buy just deploying it). I think it’s safe to assume that if the company has worked in smaller batches and incrementally delivered some business value then that would mitigate the disconnect.

Also I think it’s fair to assume that Parts Unlimited didn’t do much (if any) work validating the idea that phoenix will delivery X business value. It’s assumed that if X, Y, and Z are delivered then Parts Unlimited will improve their business position. The email campaign later in the book demonstrates what can be achieved with experiments that yield positive results then doubling down on that. Experimentation relates to my previous point because it increases confidence (which mitigates fear) in allocating resources towards specific business outcomes. I think we can also agree that assuming everything would be delivered in a single big bang release (also what Sarah was pushing for) is also flawed but that’s whole different topic 🙂

Also I think it’s fair to assume that Parts Unlimited didn’t do much (if any) work validating the idea that phoenix will delivery X business value. It’s assumed that if X, Y, and Z are delivered then Parts Unlimited will improve their business position. The email campaign later in the book demonstrates what can be achieved with experiments that yield positive results then doubling down on that. Experimentation relates to my previous point because it increases confidence (which mitigates fear) in allocating resources towards specific business outcomes. I think we can also agree that assuming everything would be delivered in a single big bang release (also what Sarah was pushing for) is also flawed but that’s whole different topic 🙂

Simply, Sarah doesn't understand that the launch of Phoenix is not consistent with much of anything from the values and principles of https://agilemanifesto.org/. The words "Working software" appear 3 times in the 16 collective (nearly 20%), and the reality is that Phoenix doesn't work anywhere. And one can substantively question if Phoenix will "satisfy the customer", lacking "early and continuous delivery" of anything close to "valuable software".

Simply, Sarah doesn't understand that the launch of Phoenix is not consistent with much of anything from the values and principles of https://agilemanifesto.org/. The words "Working software" appear 3 times in the 16 collective (nearly 20%), and the reality is that Phoenix doesn't work anywhere. And one can substantively question if Phoenix will "satisfy the customer", lacking "early and continuous delivery" of anything close to "valuable software".

Someone mentioned a shared "Definition of Done". This idea provides an objective external target for everyone to refine and align with, rather than leaving a decision up to the varying whims of any particular role and its isolated motivators and assumptions.

I wonder what are the points that you found doubtful/controversial in the first chapters? Off the top of my head I see two: 1. If it was hard (next to impossible) to build the project, how come there were happy/numb developers who were just “working on features”? How at all that would be possible without having the product built? 2. Functional programming. There are many bright proponents of the approach, however it is also true that if we look from the higher level it has downsides, which is its relative unpopularity, a steep learning curve, and, as a result, a lack of programmers who are well-versed in it. Many companies are not in favor of FP for this reason: it is just harder to hire programmers who can support systems built this way.

I wonder what are the points that you found doubtful/controversial in the first chapters? Off the top of my head I see two: 1. If it was hard (next to impossible) to build the project, how come there were happy/numb developers who were just “working on features”? How at all that would be possible without having the product built? 2. Functional programming. There are many bright proponents of the approach, however it is also true that if we look from the higher level it has downsides, which is its relative unpopularity, a steep learning curve, and, as a result, a lack of programmers who are well-versed in it. Many companies are not in favor of FP for this reason: it is just harder to hire programmers who can support systems built this way.

Ad 1) I was constantly missing the CTO in this company. Strong CTO who has enough power (also political) and trust to clarify how software is being done in XXI century. And as you said I was surprised how such strong silos could even exist. That is so obvious that somebody that wants to improve would look for some key metrics to improve overall efficiency of tech dept.

Ad 1) I was constantly missing the CTO in this company. Strong CTO who has enough power (also political) and trust to clarify how software is being done in XXI century. And as you said I was surprised how such strong silos could even exist. That is so obvious that somebody that wants to improve would look for some key metrics to improve overall efficiency of tech dept.

Ad 2) I took it just as a narrative part, not as a hint to love FP or Clojure 😉

<!here>

<!here>

Change control processes whose value is… dubious, questionable, non-existent… b/c they put approval for some change in the hands of someone ‘far away’ from what’s being changed who is really in no position to make a good judgment on whether the change is safe or not.

Question: How effective was the book at conveying the value of functional programming principles to the reader? And how does Maxine gently encourage the use of functional programming techniques to the people around her? THANK YOU! 🙏❤️:unicorn_face:🌈#UnicornProject cc @steven.proctor

Question: How effective was the book at conveying the value of functional programming principles to the reader? And how does Maxine gently encourage the use of functional programming techniques to the people around her? THANK YOU! 🙏❤️:unicorn_face:🌈#UnicornProject cc @steven.proctor

One of the things I encountered was a 'throw it over the wall' approach for updating parts of our code. I would make some change, stumble through the process of having it reviewed and okayed by another team, and then rely on them to deploy that update, independently of the product I was working on. While some of that process is still in place, I'm more connected to members of that team and it is much more documented than it was when I started at the company.

Quality documentation approvals from the quality department was an issue. They wanted word documents and specific formats. I spoke to the quality department manager to understand the Why and the what better. I then understood that they were really interested in the content and not the formal. I was able to get them on board by getting them trained in Git and other systems to see that all the content quality needed was in our daily work. Problem solved but, it took some time.

Quality documentation approvals from the quality department was an issue. They wanted word documents and specific formats. I spoke to the quality department manager to understand the Why and the what better. I then understood that they were really interested in the content and not the formal. I was able to get them on board by getting them trained in Git and other systems to see that all the content quality needed was in our daily work. Problem solved but, it took some time.

cc @steven.proctor ^^^ 🙂

My bureaucratic obstacles were around delegations of authority- held too high up the chain causing bottlenecks on fast decisions, HR process with multiple approvals slowing down recruitment of talent, especially from the point of first contact to the person on seat, procurement of services being easier than recruiting talent, so insourcing becomes harder (to the point where we are thinking of adding synthetic bottlenecks to procurement of services from vendors 😀), too many horizontal layers between the consumer and the compute each with their own backlogs prioritized with different criteria from the organization’s priorities of making money - these are the more important ones amongst others. In going after easing some of these, we had honest conversations with leaders on why the delegations were held so high, and in parallel a transformation on the IT organization to flatten hierarchy, removal of different kind of budgets that required different budget holders to approve to have more “direct” IT investment, moving to a chapters model to remove artificial recruitment approvals and holding one person per profession accountable for recruitment, and moving to more “vertical” portfolio organization structure during the IT transformation. As mentioned earlier, we are also thinking of making procurement of vendor services harder than recruiting people!

In the 90s we had a paper sheet to check out a file from source control. I introduced it. I'm so ashamed!