Another installment of Weekly Links, and we are already into February this year which still blows my mind. But so far so good, life is pure chaos with a lot going on, so much so that I don’t know where to start. But your not here for that…
Down to business…
Azure BluePrint for FedRAMP High : Love these blueprints, and more people should use these. They help implement all the required policies and are far more than just a reference document.
Big Data Cluster High Availability : You know there was a point where data warehouses and analytics platforms were considered supplemental to the main systems. But more and more those analytics services are becoming mission critical themselves. So I’ve been loving this movement to bring high availability to big data.
Hello all, so I know this follows a little bit of a departure from the technical posts, and I promise to get back to that. But I have been asked by a couple of people about working from home, and how you do it? So I thought I’d through a post out here on how to do it.
So I’ve been fortunate enough that I have had a job where I “work from home” for the past 4 years. I say that in quotes because some of those years involved traveling to end customers. Some of those years spending 4 days on customer sites, and 1 day a week at home. I will likely do another post about working mobile, or traveling for work.
But for this, we are going to focus on working from home, and how to do it. Like I said I’ve been doing this for a few years and learned a lot about how to work from home without it taking over your life.
Lesson 1 – Recognize that it’s an adjustment
So let’s not mince words here, working from home is an adjustment, and very much not like working in an office. You will hear some people tell you it isn’t, but I would disagree with that. The adjustment is compounded further if you are in a situation where you have other family members at home. So its important to know that you are going to have a period of adjustment and plan for that.
Another element of this that is important, especially if you have family at home is to have an open discussion with them about this transition. You need to make sure you have open lines of communication and that expectations are set appropriately between yourself and others. This will help make sure that “working from home” becomes a benefit to everyone, and not a curse.
Lesson 2 – Define your work hours (at least to some extent)
I know, most people will tell you that part of the benefit from working from home is flexibility, and that is absolutely true. But this prevents a misconception that a lot of people have.
There’s a lot of saying around how “if you don’t define your priorities, others will define it for you.” and I would argue that the same is true of schedules. Now that you work from home, in a lot of cases, you have a lot more control of your schedule. And in the beginning I strongly recommend that you define when your work hours are.
Let’s be clear here, the term is Work from home, not perpetual vacation. You are expected to get work done, and deliver results, just the same as if you were in an office, working a 40 hour week. You just don’t have to go to a building to do it.
By defining a work schedule you end up accomplishing the following things:
Helps define a mental space for you to make sure work doesn’t take over your life. (more on this later)
Helps communicate to family members that during specific times you are working, and therefore unavailable to run errands.
So let me give you an example, my family knows that I start work at 8:00am and work until 4pm every day. And that means that during that time period, they understand that my priority will always be doing work and items around my job. So in the event that I need to run an errand or help say for example, take one of kids to an appointment (to help the amazing miracle worker who is my wife) they know that I will do my best to help, but during those times I have to make my job a priority.
Now I’m not saying this has to be ironclad, but it should be the rule, and other times are the exception. This helps to also make sure that my family understands me needing to make trade-offs. Like for example, this past December, my son had a Christmas concert in the morning. I absolutely was not going to miss that. But my wife knew that I was either going to make the time later in the day, or take PTO for it.
This is the key that working from home, does give you huge flexibility, but especially in the beginning its really helpful to setup rules like this to communicate appropriately.
Lesson 3 – Realize how much you work
This sounds odd, but it is true. When I first started working from home, I would go to work when I got up when I normally did for work, about 6:30am, and then I would be done at 5.
But you need to realize that during that day your commute was all of 5 minutes when you walk to the office, which means that I was working 7am (shower / breakfast) to 5pm which is a 10 hour work day. But it was easy to fall into the routine and not think about it. If you want to maintain a work-life harmony, you do need to know how much you are leaning one way or the other.
Lesson 4 – Have a defined workspace
This goes to the mental state I mentioned above for working. You need to have a defined place where you work, or else you will never be able to separate work from home in your mind. Not being able to separate the two will cause a lot of issues with stress in your life.
So you need to make sure you have a place that is defined as “your office”, whether that is an actual room, or corner. It should be away from where you live normally. This will help because you will get your mind used to boundaries.
Lesson 5 – Don’t eat at your desk
This one sounds bizarre until you think about it, and honestly it’s something I’ve only implemented this year. But its very easy when you work from home to get pulled back to your desk during your lunch hours, you run down to the kitchen, grab something quick and go back.
In a normal office setting, this is a little harder as you can talk to Coworkers, get invited to lunch, etc. But when you work from home you are on your own.
If you’re like me, there will always be something you could do that will pull you back to your desk. It happened when I was at an office, but it got worse when I worked from home. You need to make sure that you take a mental break in the middle of the day to prevent stress and burn out.
Lesson 6 – Build exercise into your work day
This is one I’m trying to do, but I noticed that when I worked from home, I saw my body weight increase originally, and to be honest it makes sense. If you think about it, normally you would do a lot of activities, like commute to work, walk around the office, go out to lunch, commute home, etc.
This has been replaced with…Walk down the hall…Walk to the kitchen / bathroom…Walk down the hall
See the problem, you need to make sure you get out of the house, have a break mentally, do something healthy. These are all essential to your well-being.
Lesson 7 – Allow for context switching
As I’ve mentioned above, part of the adjustment to working from home is mental. And you need to make sure to take steps to allow your mind and body to switch context from home to work and vice versa. This is not a trivial thing. For example, I have steps I take to start my day that are designed to help me get into “work mode” and then steps I take to close out my day before I leave my office. This helps with my stress levels by allowing me to have the ability to be present in both elements of my life.
Lesson 8 – Stay organized
It’s important to remember that when you are at an office, you have others around you to help you stay organized. They will help you stay focused on priorities and context of situations around you. When you work from home, you still have communication mechanisms, but really its more reliant on you to make sure that you have the systems in place to stay organized. I use apps like Microsoft To-Do to manage my life, and that helps me to make sure that things don’t get lost in the cracks.
Make sure you take time to setup these systems to keep things organized for you.
So that’s my lessons I’ve learned from working from home. I’m sure there will be more to come, but honestly it something that both I and my family have really benefited from, and I hope you can to.
Hello all, so I’ve been doing a lot of work around availability in the cloud and how to build applications that are architected for resiliency. And one of the common elements that comes up, is how do I architecture for resiliency around storage.
So the scenario is this, and its a common one, I need to be able to write new files to blob storage, and read from my storage accounts, and need it to be as resilient as possible.
So let’s start with the SLA, so currently if you are running LRS storage, then your SLA is 99.9%, which from a resiliency perspective isn’t ideal for a lot of applications. But if I use RA-GRS, my SLA goes up to 99.99%.
Now, I want to be clear about storage SLAs, this SLA says that I will be able to read data from blob storage, and that it will be available 99.99% of the time using RA-GRS.
For those who are new to blob storage, let’s talk about the different types of storage available:
Locally Redundant Storage (LRS) : This means that the 3 copies of the data you put in blob storage, are stored within the same zone.
Zone Redundant Storage (ZRS): This means that the 3 copies of the data you put in blob storage, and stored across availability zones.
Geo Redundant Storage (GRS) : This means that the 3 copies of the data you put in blob storage, are stored across multiple regions, following azure region pairings.
Read Access Geo Redundant Storage (RA-GRS): This means that the 3 copies of the data you put in blob storage, are stored across multiple regions, following azure region pairings. But in this case you get a read access endpoint you can control.
So based on the above, the recommendation is that for the best availability you would use RA-GRS, which is a feature that is unique to Azure. RA-GRS enables you to have a secondary endpoint where you can get read-only access to the back up copies that are saved in the secondary region.
So based on that, you gain the fact that if your storage account is called:
Your secondary read-access endpoint would be :
So the next question is, “That’s great Kevin, but I need to be able to write and read”, and I have an architecture pattern I recommend for that. And it is this:
So the above architecture, is oversimplified but focuses on the storage account configuration for higher availability. In the above architecture, we have a web application that is deployed behind traffic manager, with an instance in a primary region, and an instance in a secondary region.
Additionally we have an Azure SQL database that is ego-replicated into a backup region.
Let’s say for the sake of argument, with the above:
Region A => East US
Region B => West US
But for storage, we do the following, Storage Account A, will be in East US, which means that it will automatically replicate to West US.
For Storage Account B, will be in West US, which means it replicates to East US.
So let’s look at the Region A side:
New Blobs are written to Storage Account A
Blobs are read based on database entries.
Application tries to read from database identified blob storage, if it fails it uses the “-secondary” endpoint.
So for Region B side:
New Blobs are written to Storage Account B
Blobs are read based on database entries
Application tries to read from database identified blob storage, if it fails it uses the “-secondary” endpoint.
So in our databases I would recommend the following fields for every blob saved:
Storage Account Name
This allows for me to easily implement the “-secondary” when it is required.
So based on the above, let’s play out a series of events:
We are writing blobs to Storage Account A. (1,2,3)
There is a failure, we fail over to Region B.
We start writing new blobs to Storage Account B. (4,5,6)
If we want to read Blob 1, we do so through the “-secondary” endpoint from Storage Account A.
The issue resolves
We read Blob 1-3 from Storage Account A (primary endpoint)
If we read Blob 4-6 it would be from the “-secondary” endpoint of Storage Account B
Now some would ask the question, “when do we migrate the blobs from B to A?” I would make the argument you don’t, at the end of the day, storage accounts cost nothing, and you would need to incur additional charges to move the data to the other account for no benefit. As long as you store each piece of data you can always find the blobs so I don’t see a benefit from merging.
So it’s officially 2020, and a new year brings with it all kinds of things. Retrospective, hope, dreams, and a variety of other feelings. I’m not a big party-er and have never been a massive fan of New Years Eve, but I do have to say in recent years, I have really come to appreciate two elements of new years as an important time of year for me. The first being retrospection, its a chance to look back at the year and be honest with ourselves about how things have gone. A chance to look at what worked, and what didn’t and have an honest conversation with yourself.
The second part I’ve come to enjoy is planning for the new year, sitting down and looking at my life and finding new ways to grow as a person, and improve things for the better. There’s something very empowering about sitting down and seeing a wealth of possibilities and excitement about the future prospects and opportunities that are ahead.
Now for most people, this is where the most dreaded word comes up, and its RESOLUTIONS. We’ve all heard it, and probably had it happen to us. The grand self-lie that is a resolution. Believe me over the years I’ve left a path of broken resolutions behind me, and as those who read this blog regularly know. I tend to read a lot on the subject of success, goals, and similar topics. I don’t claim to have an answer here, and over the past few years have come to the conclusion that everyone’s mileage on any option for trying to grow will vary.
Now I want to be clear about one thing here, I’m going to use the ever present weight loss example, I consider myself overweight, it is something I have struggled with I do not have the answer, and am not cla,am “throwing shade” on people who use these systems and find success. My experience only.
But what I can do, is call out some of the things I’ve tried, and how they worked out, and tell you what I’ve been finding lately:
Setting SMART Goals:
We’ve all heard this one right, making sure that your goals are “SMART”, they practically drill this into us in grade school, the only “good goals” are SMART goals. And what does SMART mean:
Now, the idea behind this is a good one, the idea behind this approach is make sure your setting goals that can be reached, and that you can verify that you have hit milestones on the path. Believe me, I do love the mantra “What gets measured, matters”, and this is based around it. It’s also though built around the satisfaction of achieving your goals. If you set something that’s measurable and attainable, then you feel pretty great when you hit that goal.
Let’s talk about an example, so an example of a “bad” goal in this model would be, “I’m going to lose weight” to steal the oldest resolution in the book. Now why is this a bad goal, because its not defined, its not something that I can measure (in a meaningful way). So a better goal would be “I’m going to lose 10 lbs, by June.” I can measure it, it has a deadline, its not outlandish by any means. Should be great right.
For a lot of people, this is a great system, and it helps them, but for me, it caused a lot more damage than it helped. The reason being is that a human being can tolerate anything for a time boxed amount of time. Look at people who have survived unimaginable conditions and then are able to return to their lives. But the problem for me, is that by doing this with the new year you aren’t doing anything to make a permanent change in your life.
Let’s go back to our weight loss example, as I’ve got to be honest, this isn’t hypothetical, its what really happened to me (more than once). You set this goal and in January you go after it…I had a coworker once who used to say “Let’s seize the day with vigor and determination never before sen by mankind.” And we’ve all been there right, we all hit the gym, get up early, and go after it.
And then a couple of outcomes happen:And then a couple of outcomes happen:
You start doing great, and by end of january you are down 5 lbs. Feeling amazing and saying “I got this”, at which point you end of convincing yourself “I can slow down, I don’t need to work as hard” and it all falls apart. And before you know it time flies and it’s June, you look at the number and say “I’m a failure”.
You start doing great, and middle of February, you hit your goal of 10 lbs down, you’re proud of yourself, and smart goals works. You move onto other things, and before you know it you fall into bad habits, and June hits and the scale looks pretty familiar, you look at the number and say “I’m a failure”
You stay on track, do what you set out to do, get to june and are down 10 lbs. You feel great, smart goals worked. You have a fun summer and end up back where you started, or god forbid worse off, look at yourself and say “I’m a failure”.
And now your probably saying “For loving the positive elements of new years, this is pretty damn depressing. And I’m not trying to be a debby downer. But this is my experience and as I said above, part of this process is honestly and retrospective. This has been my honest experience.
This is my problem here, SMART goals are built to be very short term focused to get a “job” done, but when it comes to personal growth, the job is never “done”, so the approach is fundamentally flawed. And at the end of the process those words / feelings of “I’m a failure” have a damaging and demoralizing effect that is completely counter productive.
At the end of the day, growth is a journey. And if you continue down this road and you miss your goal you are left with nothing, and feeling like you failed with nothing to show for the effort. I believe there is an old adage about eggs in a single basket for this.
This is one that got a lot of attention, I’ve read the book the 10x Rule, and I have to say it is insightful,and I found it to be very interesting. For those not familiar the idea is this, take the idea of SMART goals and turn it around a bit. Keep the same ideas of goals being measurable and time boxed but instead of making them attainable, you make them 10x what the attainable goal is.
So take our weight loss example,, instead of saying “I’m going to lose 10 lbs by june” I would say “I’m going to lose 50 lbs by june”. Now before anyone jumps on me, I can do math. The idea is what could you do if you put in 10x the effort. So the idea then is if I put in the work and try to lose 50 lbs by June, one off two outcomes occur:
I lose 50 lbs and cheer my success.
I lose 30 lbs and I’m still better off than the 10lb goal.
In my experience though the problem is still the same. I haven’t changed behaviors or grown at all, I’ve hit a very finite and fixed in time goal, but the success won’t last. And at the end you still feel like a failure. And now you feel like a bigger one, because not only did you miss the 10x goal,but likely the 1x goal too.
Finite Systems / Infinite Problem:
The crux of the problem I have with the above problems is that they are systems built around finite objectives, being applied to an infinite problem. I don’t want to lose weight, I want to be healthier, I don’t want to learn one thing, but build a foundation for learning. And at the end of the day, we are trying to fit a square peg into a round hole. Personal growth isn’t something that can be timeboxed like that.
Simon Sinek covers this in his book, the “Infinite Game”, which I admit I am still reading now, but here’s a video that gives some of the highlighting principles.
The other problem I have is that in my experience this creates a lot of stress and pressure on yourself, and those words “I’m a failure” whether you say them aloud or not are devastating. If you become too fixated on goals, they can start to feel like a drug high. And I’m speaking from experience here, they become this thing where your life becomes about setting goals, pushing too hard, getting them and that feeling of euphoria, and then its on to the next one.
I was 100% in that boat, for better or worse, and don’t get me wrong I’m proud of any accomplishments I’ve made, but it really does take a toll on you mentally. While it can be satisfying to reach those goals, it isn’t always fulfilling. And if you find yourself questioning where to go next, that can be crippling in a lot of ways.
And now I’ve done it again, we are at the “Kevin, still depressing. Goals are meaningless, growth is meaningless, life is pain…”
Not quite, I’ve been doing a lot of reading and researching and had lots of discussions with people a lot wiser than me, and I’ve found something that in my opinion seems to be working better.
The final problem I have with these systems, is they make one basic assumption, and that is that pursuit of these goals exists in a vacuum. And what I mean by that is take our weightloss example, we say “I’m going to lose 10 lbs by march”, but then I get hurt, need surgery and spend 6 weeks in a cast, and then physical therapy. I know that the goal became unattainable, but I still feel like I failed.
Now again, not just weight loss, but let’s say I said “I’m going to put my phone away after dinner to spend more time with my family”. And then I get a smart watch which lets me check email without my phone, or I work with customers all over the world that have to call at off hours, then I feel like a failure due to circumstances outside of my control.
Goals vs Values:
Now I can’t take credit for this, there is a psychological principle called value based living, and the idea being this. Here’s a video that does a way better job than I ever could at summarizing it.
So looking at the above, if we get away from these ideas of goals, and look more at what we as a person value. That is what drives us, and that is what matters. And as long as the actions we take align with those values, the journey is part of the reward. If you watched the video above with Simon Sinek, this probably sounds familiar, and that should be no surprise. There is a direct through line between his concepts of actions being driven by values and value based living.
So the next question is how does this work any differently? How do I grow and push myself without goals? Is this just symantics at the end of the day. I don’t think so, but let me talk about what this journey has been like for me, and you can judge.
Step 1 : Change your definition:
One thing that my wife and I are really trying to embrace is a family mission statement, and we are in the process of writing that now. When we are done I will probably do a blog post on that too. But along with that, we as a family have focused our energy and decisions about what we do around this motto for lack of a better term.
There are only two outcomes to any action, success or you learn something.
That’s it, not ground breaking, and truth be told we stole it from the movie Meet the Robinson’s, which has a similar sentiment, “From failure you learn, success not so much”. But if you stop and think about that statement, its rather profound, if you take away failure as an outcome. Some would say you take away accountability, but I would say you take away blockers. If you can’t fail, then what is stopping you from trying?
Thomas Edison had a similar statement, when asked about the 1000 failed attempts to make a light bulb, he said “I didn’t fail, I just found 1000 ways not to do it.”
At its core this is very freeing, and we need to say we can grow and push the limits because there is no outcome that we shouldn’t feel positive about, because the journey will yield learnings, and those learnings will help us to improve for the future.
Step 2 : Define your values:
This one took a lot of soul searching for me. You need to take a step back and identify what above all else matters to you. What ideals and values do you aspire to above all else. And that’s not an easy question, and should not be taken lightly. I find that making these values something that need to be quantified in a single word helped a lot.
My values are the following:
And what I mean by these, is that my guiding principles in my life, at this time are these items. When I am long gone, I want my kids to know that above all else family mattered. I want them to see that I had a love of learning. That I focused on having an impact around me whether it be my career or community. I want them to see me as someone who was innovative and creative.
These values together really some up at this stage of my life, the legacy I want to leave behind.
Step 3 : Values Define Action:
One common thread you will see in anything and everything is the idea that we as people have limited resources. Whether those be willpower, physical, financial, energy, attention, or the all mighty time. We can only put our resources into some much, and we can’t do it all. Greg McKoewn has a great book on this called “Essentialism”, which I really believe is a great book about applying your resources.
To that end, if we have values that are important to us, and we have limited resources. it isn’t a big logical leap to say that we should focus on putting our energy behind the actions that align with our values.
Not really rocket science, although it took me a while to get here if I’m being honest.
Now what I’ve found from doing this in practice in recent months is that I have seen my stress level go down, and my commitment to any actions I’ve taken go up. And results have been greater too. And at the end of the day I believe its easier to be committed to an action if it aligns to something you care deeply about.
Let me go back to our example, as mentioned above I want to get healthier, and I’d tried smart goals, 10x goals, etc. I tried a keto diet, joining a gym, nothing seemed to stick. And even when they did I could never cross the 15 lbs mark. And it was devastating to me. I have had to actively sit on the side lines at both work and family functions because of my body weight.
This all came to a head for me, when I took my son to Hershey, and all he wanted to do was ride a roller coaster, and he’s much too small to ride the big coasters, but we saw a roller coaster called the “coco cruiser” (a little kid roller coaster), and he wanted to ride it. We got in line, and when we got to the front, he was too short to ride by himself, and I couldn’t fit into the coaster. He and I stood on the platform, while his friends rode, and then he rode with one of their mom’s. Having to explain to your son that he can’t have what he wants because of your body weight is one of the lowest points in my life. I wanted to curl up and die.
I still could never get past that 15 lbs mark, and life would get in the way. I took a step back, and said…forget the numbers. I want to get healthy because it will let me be more to my family. I found a cross fit gym that I really like, with great people and a great coach. I try to go as much as I can, unfortunately recently being sick sidelined. But just out of curiosity I got on the scale today, I’m down 25 lbs from that horrible day. I feel better and have better energy, and even though I fell off the wagon and am going back when travel slows down. I feel like a success and look back on all the victories and fulfillment I feel with a positive attitude.
The attention here being on the action, not the outcome. Its having a lasting impact as it leads to behavioral change.
But let’s not make this all about weight, even if that is an easy example. Take my professional life, I decided to focus more on impact and now measuring all my actions by impact they have. This has led to greater results in my office with me feeling better about the work I’ve done, and if you look at the metrics much greater returns. My stress level has gone down, and I’ve stopped measuring myself against the impact and activities of my colleagues.
I know this has been a much longer blog post than normal, but thanks for sticking with me through this. The end result of which is this, I’m not going to be setting any resolutions this year. My new plan is to reaffirm and re-evaluate my values, and then make sure that I devote my energy and resources to actions that align. This will allow me the flexability to enjoy life, while still finding new ways to grow.
This is something my wife and i both feel strongly about and are working with our kids to internalize and I hope it at least sparks some thought for you about where you are and where you want to go.
Data Drift Monitoring for Azure ML Datasets : Another timely video for a lot of people I know, how do we monitor data drift in our Azure ML models and work to keep them up to date. MLOps is becoming a thing as ML projects suffer from code “staleness” and that needs to be planned for early on.
Azure Service Bus from the Ground up: Service bus is one of those services in Azure that is foundational, its really been around forever and new enhancements are happening all the time. So its good to see how things are improving.
So for this week, its nothing in nerd pop culture, but a fascinating video I saw. This was recommended by a friend, and it sparked a lot of research for me around personal growth (blog post coming soon). But the video is “What does game theory teach us about war?” and there are a lot of parallels that you can draw to how your own life functions in terms of finite vs infinite games.
Hello all, so I know that I totally fell off the wagon when it comes to weekly links. December has been a crazy month, I had three business trips, getting ready for Christmas. Due to family coming to town we effectively had 3 Christmas’ and on top of that I got sick.
I’m not complaining just explaining what happened. The holidays are a rough time for a lot of people, and we’ve all had experiences we have to carry with us. So remember in all the craziness that some might be suffering in silence. If you find yourself in this position, please reach out. You are not alone and there are people to help you.
So as always I have a post here about something fun. And this week I wanted to post about how my wife blew me away with her gift this Christmas. We had our 11th anniversary at Dave and Busters, we decided to double down on stupid fun. When we walked in, they were having a silent auction for MakeAWish and when I was in the bathroom she bid on this item and won.
This is a picture from the Dark Knight, my favorite movie of all time, signed by Christian Bale and the late Heath Ledger.
Something different… Things I learned from Hallmark Christmas Movies.
Hello all, in keeping with the last post on the blog, I started doing some posts around High Availability, so ultimately the focus here is how do I architect my solution to ensure that is meets the availability demands of my customers.
So odds are if you’ve started down this direction, you’ve heard 3 acronyms:
SLA – Service Level Agreement
RTO – Recovery Time Objective
RPO – Recovery Point Objective
So what do each of these items mean, and how do they relate to your solution. For SLA, I covered this pretty extensively in my previous post. So I would direct you there for a definition and then recommendations around how to approach that topic.
So the next question is really what are RTO and RPO? And how do they relate to High availability?
What is RTO?
RTO stands for Recovery Time Objective, and basically, in software terms, this refers to when something happens, how fast do you recover?
So let’s take an example because I work best with examples. So if I have a solution that is deployed in multiple regions, and my solution uses Traffic Manager and has replication of the solution into another region. If the Traffic manager is checking the endpoint every 5 seconds, and 3 failures cause a failover…that means my RTO is 15 seconds.
By using a dual region deployment, I’m able to keep my RTO relatively low. Now the above example is pretty simplistic. But really we should do this analysis per service in our architecture, to determine how long our failover takes, and then the longest of that is your solutions RTO.
How do we improve RTO?
Now, remember that this is really a measure of continuity of business, so really looking at High Availability and Disaster Recovery. So ultimately we are talking about service uptime more than anything else.
So the best way to improve RTO is to enable the replication and take steps to increase the speed of recovery. So if you look at the last discussion of SLA, we took steps to minimize downtime by increasing SLA. This conversation will be about how do we minimize the downtime caused by those failovers.
The most important things involved in this are the following:
So the key metric to pay attention to is how long it takes to get up and running.
Monitoring is the cornerstone of your RTO target. If you don’t know there is a problem, you can’t find it. Many blogs and articles will focus on the next 3 parts, but let’s be honest, if you don’t know there’s a problem, you can’t respond. If your logs operate on a 5-minute delay, then you need to factor in the 5 minutes into your RTO.
From there the next piece is response time. And I mean this in the true sense of how quickly can you trigger a failover to your DR state. How quickly can you triage the problem and respond to the situation? The best RTO targets leverage as much automation as possible here.
Next, by looking at data replication, we can ensure that we are able to bring back up any data stores quickly and maintain continuity of business. This is important because every time we have to restore a data store, that takes time and pulls out our RTO. If you can failover in 2 minutes it doesn’t do you much good if it takes 20 minutes to get the database up.
Finally, failover. If you are in a state where you need to failover, how long does that take and what automation and steps can you take to shorten that time significantly.
Let’s give an example if I have a solution that is the following in one region:
Azure App Service
If I’m deployed in a single environment, and my DR plan is to standup another region in the event of a disaster. Now that solution has a pretty high RTO, if it takes 15 minutes to standup that environment and deploy it, then the RTO is 15 minutes. If I wanted to lower that, there a couple of things I can and those would be:
I can increase the automation I use to reduce that time.
I can do is spin up another region, or leverage options to do replication.
I can set up automation around detection and response.
What is RPO?
RPO stands for Recovery Point Objective, which really focuses on the idea of improving the ability to recover from a data perspective. So if you have a disaster, how much data would be lost? What would the impact be?
When looking at RPO, the key comes to data and potential data loss. So how do we minimize the window for data loss and lower the chances of lost transactions in your application?
There are a few key elements that can assist with this, looking at how your application handles eventual consistency. It is possible to get to an RPO of 0, as you have constant data replication in your solution.
Now the most important part of the replication is that the replication needs to be executed in a synchronous fashion, meaning that it must write and replicate the data before sending an acknowledgment. This means that eventual consistency will keep your RPO higher than zero because it means that the replication will “eventually” get there.
How do we improve RPO?
The most important factor here is replication and data consistency. So we really need to make sure that the strength of transactions is maintained about that consistency rules are enforced. This is why data stores like Cosmos gain popularity in terms of requirements for zero RPO and low RTO because it supports models where they can enforce this type of logic.
Needless to say, this all comes down to operations and math and ultimately the requirements of your solution and balancing that against cost and impact. You really want to make sure you only take this to the level you need to as it can add a lot of cost and substantially raise the complexity of your solution.
Hello All, So this week marks 2 cool things in my mind. First is that we are 30 days to Christmas Day, which is awesome. I know this is the holiday season for a lot of different religions.
So I wish you and you’re a great holiday season. I am a big fan of the time of year because… it’s crazy, it’s busy, and its a great time filled with great moments. So please make sure to take a moment and enjoy it. This is a big-time for my family, as my wife and I were married 11 years ago this month.
The other big thing this week is a moment I’ve been waiting for since my kids were born. Last night I introduced my kids to Star Wars, watching the first movie together and my kids loved it. The Best revelations being the following
My daughter had the following revelations:
Princess Leia is awesome, she doesn’t need anyone to save her and she fights back.
Deep Dive: Industrial IoT: I love these deep dives because they show technology being used in different ways than I might see normally, always informative.
Ok, I sort of did this at the top, but what the hell, let’s do another one. I’m a comic fan, we all know this. And I’ve enjoyed the CW Arrowverse, and the big event, Crisis on Infinite Earths is going to be premier Dec 8. I’m super excited about this. There are going to be a lot of interesting things including bringing back fan favorites and a lot of interesting events.
So Disney+ launched, and I think its a toss-up as to who was more excited, me or my kids. But along with it came “The Mandalorian”, which mixes the western / crime genre with star wars. What’s not to love. Going to watch the mandalorian tonight. So excited!