When I was a kid, one of my favorite movies was Jurassic Park, because well…dinosaurs. I remember the movie being such a phenomenon too that summer, there were shirts and toys everywhere. I even remember going to the community pool and seeing adults everywhere holding the book with the silver cover and the T-Rex skull on it.
It really was a movie ahead of its time, not just in terms of special effects, or how it covers the topic of cloning, but in that it described a societal nexus we were all headed towards that many people didn’t quite see yet. One of my favorite moments in the movie is when Jeff Goldblum’s character, having just survived a T-Rex attack deliers this line:
Technology has grown, by leaps and bounds, to the point now that many argue Moore’s law is irrelevant and outdated. And we are making advances in everything major area of life to the point that the world we grew up in is completely unrecognizable to that of our children. Furthermore to the point that this question has become all the more relevant today, with regard to artificial intelligence.
Just to be clear these are the thoughts of one developer / architect (me) on this subject and I would recommend you research this heavily, and come to your own conclusions, but these are my opinions and mine alone.
We have reached a period of time where more and more businesses and society in general are looking to artificial intelligence as a potential solution to solve a lot of problems and more and more the question of AI ethics has become prevalent. But what does that actually mean and how can an organization build AI solutions that serve to benefit all of humanity rather than cause unintended problems and potentially harm members of society.
The first part of this comes down to the recognition that artificial intelligence solutions need to be fully baked and great care needs to be given to supporting the idea of mitigating built in bias in both training data and the end results of the service. Now the question is what do I mean about bias. And I mean actively searching for potentially bad assumptions that might find their way into a model based upon a training dataset. Let’s take a good hypothetical case that strikes close to home for me.
If you wanted to build a system to identify patients that were at high risk for pneumonia. This was a hypothetical I talked to a colleague about a few months ago. If you took training data of conditions they have and an indicator of whether or not they ended up getting pneumonia, this would seem like a logical way to tackle the problem.
But there are potential bias that could occur based on the fact that many asthmatics like myself tend to seek proactive treatment, as we are at high risk, and many doctors treat colds very aggressively. Mainly because when we get pneumonia it can be life threatening. So if you don’t account for this bias it might skew the results of any AI system. Because you likely won’t see many asthmatics appear in your training data that actually got pneumonia.
Or another potential consideration could be location, if I take my data sample just from the southwest like Arizona, dry climates tend to be better for people with respiratory problems and they might have lower risk of pneumonia.
My point is the idea of how you gather data and create a training data set is something that requires a significant amount of thought and care to ensure success.
The other major problem is that every AI system is unique in the implications of a bad result. In the above case, its life threatening, in terms of a recommendations engine for Netflix, it means I miss a movie I might like. Very different results and impact on lives. And this cannot be ignored as it really does figure into the overall equation.
So the question becomes how do we ensure that we are doing the right thing with AI solutions? The answer is to take the time to decide on what values as an organization we will embrace at our core for these solutions. We need to make value driven decisions on what type of implications we are concerned about and let those values guide our technology decisions.
For a long time values have been one of the deciding factors between successful organizations and unsuccessful ones. The one example that comes to mind was the Tylenol situation where a batch of Tylenol had been tampered with. The board had a choice, pull all the Tylenol on market shelves for public safety and hurt their shareholders or protect share holders and deny. The company values indicated that customers must always come first and it made their decision clear. And it was absolutely the right decision. I’m giving a seriously abridged version, but here’s a link to an article on the scare.
Microsoft actually released an AI School for business to help customers to get a good starting point for figuring that out. They also made several tracks for a variety of industries to help with what should be considered for each industry. Microsoft has also made their position on ethical AI very clear in a blog post by Company President Brad Smith and Our Approach: Microsoft AI
Below are the links to some of the training courses on the subject:
Along side this, there has been a lot of discussion around this, from some of the biggest executives in the AI space, including Satya Nadella:
But one of the most interesting voices I’ve heard with regard to the ethics and future of AI is Calum Chace, and I would tell you to watch this as it really goes into the depth of the challenges and ways that if AI is not handled responsibly we are looking at another major singularity in human evolution:
This is a complicated and multi-faceted topic that is great food for thought on a Friday. Empathy is the most important elements of any technology solution as these solutions are having greater and greater ramifications on society.
So there’s a common question I’ve been getting a lot lately, and that’s “I want to learn Azure, where do I start?” And this is ultimately a very reasonable question, because as much as the cloud has permuted much of the digital world, there are still some organizations who have only recently started to adopt it.
There are many reasons people would choose to adopt the cloud, scalability, cost, flexibility, etc. But for today’s post I’m going to focus on the idea that you have already decided to go to the Azure Cloud and are looking for resources to ramp up. So I wanted to provide those here:
MS Learn: The site provides videos, reading, and walk-through’s that can assist with learning this type of material:
Azure Fundamentals: This course path contains several courses that focus on a variety of topics at a beginner level, including architecture, monitoring, cost, storage, compute, and security.
Manage Resources in Azure: This course provides a high level introduction to the basics of managing your resources that you create in azure.
Secure your Azure Resources with RBAC: Role-Based Access Controls are the cornerstone of Azure compliance. The intention here being that you must setup the proper governance in the beginning to prevent run away spend from your teams working in the azure cloud.
Design for efficiency and Operations in Azure: A good course that provides the basics of how to architect for efficiency. One the biggest changes in mindset for many on premise developers is that in the cloud, you architect for the minimum and the ability to scale, not for the “worst case scenario”.
Design for Security in Azure: The last thing many devs consider, but it should be the first. Security is not something any organization can treat as an after-thought.
EDX Courses: EDX is a great site with a lot of well made courses, and there are a wealth of options for Azure and Cloud, here are a few I thought relevant, but it is not an exhaustive list.
Architecting Distributed Applications: One common mistake, that many make with regard to the cloud is that they think of it as “just another data center”, and that’s just not true. To build effective and scalable applications, they need to be architected to take advantage of distributed compute. This course does a great job of laying out how to make sure you are architected to work in a distributed fashion.
Microsoft Azure Virtual Machines: The virtual machine is the cornerstone of azure, and provides many options to build an scale out effectively. This is a good introduction into the most basic service in Azure.
Microsoft Azure App Service: The most popular service in Azure, App Service enables developers to deploy and configure apps without worrying about the machine running under-the-covers. A great overview.
Microsoft Azure Virtual Networks: As I mentioned above, Software Based Networking is one of the key pieces required for the cloud and this gives a good introduction into how to leverage it.
Databases in Azure: Another key component of the cloud is the Database, and this talks about the options for leveraging platform-as-a-service offerings for databases to eliminate your overhead for maintaining the vms.
Azure Security and Compliance: A key component again is security, as the digital threats are constantly evolving, and Azure provides a lot of tools to protect your workload, this is an essential piece of every architecture.
Those are just some of the many resources that can be helpful to starting out with Azure and learning to build applications for the cloud. It is not an exhaustive list, so if you have a resource you’ve found helpful, please post it in the comments below.
So I’ve decided to start doing something new, Like any good dev, I spend a lot of time on the internet, so as I’m working on various projects I’ve decided I’m going to post a bunch of useful links out here that I find as i’m going through my travels. Hopefully you find them helpful too.
In the interest of helping to navigate the information available out there, I’ve been putting out there ideas for this “Where Do I start” series on the blog. Right now as I previously mentioned I’ve been studying for the AI-100 exam, and as part of that effort I found a lot of resources online, and I thought I’d share these in the interest of helping others.
There are a wealth of resources out there and I want to make sure I focus your attention on resources related to Microsoft AI and how you can leverage these services as accelerators for your own application development. I wanted to draw your attention to a lot of the key resources for getting started.
So it’s no surprise that I very much have been talking about how amazing TerraForm is, and recently I’ve been doing a lot of investigation into Solr and how to build a scalable Solr Cluster.
So given the kubernetes template I wanted to try my hand at something similar. The goals of this project were the following:
Build a generic template for creating a Solr cloud cluster with distributed shard.
Build out the ability to scale the cluster for now using TerraForm to manually trigger increases to cluster size.
Make the nodes automatically add themselves to the cluster.
And I could do this just using bash scripts and packer. But instead wanted to try my hand at cloud init.
But that’s going to be the end result, I wanted to walkthrough the various steps I go through to get to the end. The first real step is to get through the installation of Solr on linux machines to be implemented.
So let’s start with “What is Solr?” The answer is that Solr is an open source software solution that provides a means of creating a search engine. It works in the same vein as ElasticSearch and other technologies. Solr has been around for quite a while and is used by some of the largest companies that implement search to handle search requests by their customers. Some of those names are Netflix and CareerBuilder. See the following links below:
The above will configure a “getting started solr cluster” that leverages all the system defaults and is hardly a production implementation. So my next step will be to change this. But for the sake of getting something running, I took the above script and moved it into a packer template using the following json. The above script is the “../scripts/Solr/provision.sh”
So I’ve been doing a lot of work with a set of open source tools lately, specifically TerraForm and Packer.
TerraForm at its core is a method of implementing truly Infrastructure
as Code, and does so by providing a simple function style language
where you can create basic implementations for the cloud, and then
leverage resource providers to deploy. These resource providers allow
you to deploy to variety of cloud platforms (the full list can be found here).
It also provides robust support for debugging, targeting, and supports
a desired state configuration approach that makes it much easier to
maintain your environments in the cloud.
Now that being said, like most open source tools, it can require some configuration for your local development environment and I wanted to put this post together to describe it. Below are the steps to configuring your environment.
Step 1: Install Windows SubSystem on your Windows 10 Machine
To start with, you will need to be able to leverage bash as part of
the Linux Subsystem. You can enable this on a Windows 10 machine, by
following the steps outlined in this guide:
Once this is done you are in a place where you can run terraform
projects, but before you do, you need to authenticate against Azure.
This can be done by running the following commands in the bash terminal
(see link below):
So containers have become an essential part of modern application development. I would go as far to say that containers and micro services have had a similar impact to software development as “Object Oriented Programming”.
Now that being that I have been talking to a lot of people who use Monolithic applications and are looking for a way to break down their existing applications into a micro service approach and support the idea of using existing infrastructure, and don’t necessarily want to deploy on Linux for a variety of reasons.
Now based on that option, there is an established technology that can leverage your docker containers and orchestrate them in a windows environment. And that is Service Fabric.
I find the learning curve if you are looking at a monolithic application and breaking it into micro services is a lot easier to swallow with Service Fabric, and it does help you to break up your applications to make better use compute on your machines in the cluster and you can still leverage docker.
Below are some links to help you get started with Service Fabric if you are looking for information on this technology:
So for something a little different, I decided to check out a book on finance, because ultimately I do love my job, but I enjoy making money :). So the question obviously becomes how to you generate enough well to some day gain the financial freedom to enjoy it.
My wife and I a while back, took the Financial Peace University course, and found it to be really insightful, and since then by applying the teachings so that course we have been able to leverage the money we make to achieve more of our goals, and honestly its been a very liberating experience overall. Our entire financial outlook completely changed in a single year’s time.
Honestly, if the past few years time have taught me anything, its the change can happen every quickly, and with almost no time (or warning) at all. In the past 6 years, my entire life changed so much that if you honestly tried to tell my past self this was all going to happen, I would have absolutely laughed at you. I went from a married man, living in a small town home, and working as a developer and architect to my current position, father of 2, and currently in a new home.
So I decided to check out Chris Hogan’s book, Everyday Millionaire, and see sort of what I could gleam from his research and insights. For those who don’t recognize the name, Chris Hogan is a financial expert that is part of Dave Ramsay’s “Ramsay Solutions” organization, and has done several books on how to ensure you set yourself up for a successful retirement.
I have to say I enjoyed this book, as part of this book Chris Hogan interviewed and did a study of over 10,000 people who all have a net worth over $1 million dollars. And to be honest his findings pretty much lined up with a lot of the things that are talked about in Dave Ramsay’s books and course.
Now admittedly, that’s not surprising, because he talks about how they found these people was to put out an open call and interview the people who came back, and the people who would be listening to him are people who are already familiar with his work. So his results may be a little skewed, but that doesn’t make his findings any less relevant.
The general message of the book is that it is possible to attain millionaire status without doing so through any one of the many myths out there. The idea that the people who are rich has some “secret sauce” or some “unnatural advantage” is not at all true.
The simple message of this book is that if you work hard, and invest smart and safely you can achieve the financial independence you are looking for, and chasing that “1 big break” is what can ultimately lead to ruin.
Below is a video describing how this is possible.
Now there is one thing I fundamentally disagree with, and its not what he’s saying, but more how he says it. The one statement he repeats often is “If you work hard then …” and makes a reference to how attainable it is.
I don’t disagree with the sentiment, but I do disagree with the phrasing. The past few years have taught me a very valuable lesson, and that lesson is that of making sure you focus on “Impact OVER activity”. I don’t believe that working hard is enough to get anywhere in life, but rather working smarter and harder is the key.
Let me put this to an example…Take the following two scenarios, and I’m borrowing these from Greg McKeown. I’m going to keep the numbers small for my point.
If you have a job as a kid with a paper route, and you work every day of the week, and ultimately make $10 a week, you can make good money. That’s $40 / month, and that can do a lot for a kid who is say 12 years old.
Now, if you could instead take a job washing cars on the weekend, and you can charge $5 per car and do 5 cars in a saturday, and 5 cars in a sunday. That means you can make $50 a week, which is $200 / month.
Now I would argue, you can work hard at the paper route, but at the end of the day the impact it has on your goals is significantly lower. It would make more sense to take the job washing cars, and make more money, and then look for other things you can do during the week. It doesn’t matter how hard you work that doesn’t change the fact that the impact is different. If I work 10x as hard at the paper route, the end result is the same.
But if I increase my efforts on washing cars and can do 7 cars a day (only 2 more) that’s now $35 / day, which is $280 / month.
See my point, at the end of the day I feel like its important to work hard, but you have to take the time to make sure that what you are working on is moving you towards your end goals in life, sitting and grinding away at a job that you don’t enjoy and has no growth potential it may get you millionaire status some day, but the risk would be lower if I focus on careers where the level of effort has an impact on the return on the investment.
Ultimately we all have a finite amount amount of time and we invest it in our careers and skills so we should focus on items that have an acceptable level of risk and a reasonable return on that investment.
Overall I recommend the box, but would advise you to keep this in mind as you read it.