Weekly Link – 6/3

Below are some links I’ve found based on the work I’ve been doing that I thought were interesting.

Technical Links:

Developer Life:

Adventures in 3d Printing

So my latest adventure in technology has been to setup my Ender3 3-d printer. I got this thing around Christmas time, mainly as a toy for myself to play around with, and to do more with the raspberry pi’s that I have. Also as a STEM tool for teaching my kids.

But mostly to make cool stuff for me. I’ll be honest, I’m a tabletop gamer, so the idea of custom mini’s is pretty awesome.

So I won’t do a full “step-by-step” on how I assembled this thing, there are a lot of great videos out there on doing that, here’s the one I used.

But I wanted to talk through the process of my prints. The overall process of getting the printer setup wasn’t bad, but its the little things that will mess up your first couple builds and those are the ones I wanted to call out.

  • Leveling the Bed: So I got to tell you, I’m new to 3d printing, and I hate this expression, mainly because it created the wrong image in my mind. When I heard this, I got out a small level and made sure the bed was completely flat. But that led to some really terrible prints. What this really means is making sure that the distance between the nozzle and the bed is uniform throughout. Best way to do this, take a business card and run it in between, and if you feel the nozzle dragging on the card you’ve got it.
  • Feeding the Filament: When I first did a print, I almost ruined my printer, and its because I didn’t feed the filament the whole way through the line. You have to push it the whole way to the nozzle.
  • What to do when it doesn’t stick: I had real problems, even after it was leveled with the print sticking. What would happen is I would get 75% through and then the bed would let go of the model and my print would be ruined. I used hairspray thanks to a community I joined’s recommendation. And it worked like a charm.
  • Supports: I tried a few builds without supports, and that tended to be a mistake, so I had to put them back in. But when I did I changed the percentage supports to 10% and that made them so much easier to break off and stopped me ruining prints after the fact.

Overall though I’ve had some good luck with it, and had a lot of fun. I got an Ender3, and I have to say that overall its been pretty easy to pickup and get going with.

Weekly Links – 5/27

Here’s this weeks links:

Technical Links:

Developer Life:

Weekly Links – 5/20

Here’s this weeks links:

Technical Links:

Developer Life:

Musings on Ethical AI for Business and resources to help

When I was a kid, one of my favorite movies was Jurassic Park, because well…dinosaurs. I remember the movie being such a phenomenon too that summer, there were shirts and toys everywhere. I even remember going to the community pool and seeing adults everywhere holding the book with the silver cover and the T-Rex skull on it.

It really was a movie ahead of its time, not just in terms of special effects, or how it covers the topic of cloning, but in that it described a societal nexus we were all headed towards that many people didn’t quite see yet. One of my favorite moments in the movie is when Jeff Goldblum’s character, having just survived a T-Rex attack deliers this line:

See the source image

Technology has grown, by leaps and bounds, to the point now that many argue Moore’s law is irrelevant and outdated. And we are making advances in everything major area of life to the point that the world we grew up in is completely unrecognizable to that of our children. Furthermore to the point that this question has become all the more relevant today, with regard to artificial intelligence.

Just to be clear these are the thoughts of one developer / architect (me) on this subject and I would recommend you research this heavily, and come to your own conclusions, but these are my opinions and mine alone.

We have reached a period of time where more and more businesses and society in general are looking to artificial intelligence as a potential solution to solve a lot of problems and more and more the question of AI ethics has become prevalent. But what does that actually mean and how can an organization build AI solutions that serve to benefit all of humanity rather than cause unintended problems and potentially harm members of society.

The first part of this comes down to the recognition that artificial intelligence solutions need to be fully baked and great care needs to be given to supporting the idea of mitigating built in bias in both training data and the end results of the service. Now the question is what do I mean about bias. And I mean actively searching for potentially bad assumptions that might find their way into a model based upon a training dataset. Let’s take a good hypothetical case that strikes close to home for me.

If you wanted to build a system to identify patients that were at high risk for pneumonia. This was a hypothetical I talked to a colleague about a few months ago. If you took training data of conditions they have and an indicator of whether or not they ended up getting pneumonia, this would seem like a logical way to tackle the problem.

But there are potential bias that could occur based on the fact that many asthmatics like myself tend to seek proactive treatment, as we are at high risk, and many doctors treat colds very aggressively. Mainly because when we get pneumonia it can be life threatening. So if you don’t account for this bias it might skew the results of any AI system. Because you likely won’t see many asthmatics appear in your training data that actually got pneumonia.

Or another potential consideration could be location, if I take my data sample just from the southwest like Arizona, dry climates tend to be better for people with respiratory problems and they might have lower risk of pneumonia.

My point is the idea of how you gather data and create a training data set is something that requires a significant amount of thought and care to ensure success.

The other major problem is that every AI system is unique in the implications of a bad result. In the above case, its life threatening, in terms of a recommendations engine for Netflix, it means I miss a movie I might like. Very different results and impact on lives. And this cannot be ignored as it really does figure into the overall equation.

So the question becomes how do we ensure that we are doing the right thing with AI solutions? The answer is to take the time to decide on what values as an organization we will embrace at our core for these solutions. We need to make value driven decisions on what type of implications we are concerned about and let those values guide our technology decisions.

For a long time values have been one of the deciding factors between successful organizations and unsuccessful ones. The one example that comes to mind was the Tylenol situation where a batch of Tylenol had been tampered with. The board had a choice, pull all the Tylenol on market shelves for public safety and hurt their shareholders or protect share holders and deny. The company values indicated that customers must always come first and it made their decision clear. And it was absolutely the right decision. I’m giving a seriously abridged version, but here’s a link to an article on the scare.

Microsoft actually released an AI School for business to help customers to get a good starting point for figuring that out. They also made several tracks for a variety of industries to help with what should be considered for each industry. Microsoft has also made their position on ethical AI very clear in a blog post by Company President Brad Smith and Our Approach: Microsoft AI

Below are the links to some of the training courses on the subject:

Along side this, there has been a lot of discussion around this, from some of the biggest executives in the AI space, including Satya Nadella:

But one of the most interesting voices I’ve heard with regard to the ethics and future of AI is Calum Chace, and I would tell you to watch this as it really goes into the depth of the challenges and ways that if AI is not handled responsibly we are looking at another major singularity in human evolution:

This is a complicated and multi-faceted topic that is great food for thought on a Friday. Empathy is the most important elements of any technology solution as these solutions are having greater and greater ramifications on society.

Weekly Links – 5/13

Here’s this weeks links:

Technical Links:

Developer Life:

Getting Started with Azure (developer perspective)

So there’s a common question I’ve been getting a lot lately, and that’s “I want to learn Azure, where do I start?” And this is ultimately a very reasonable question, because as much as the cloud has permuted much of the digital world, there are still some organizations who have only recently started to adopt it.

There are many reasons people would choose to adopt the cloud, scalability, cost, flexibility, etc. But for today’s post I’m going to focus on the idea that you have already decided to go to the Azure Cloud and are looking for resources to ramp up. So I wanted to provide those here:

MS Learn: The site provides videos, reading, and walk-through’s that can assist with learning this type of material:

MS Learn for Specific Services: There are several common services out there that many people think of when they think of the cloud, and I wanted to provide some resources here to help with those:

EDX Courses: EDX is a great site with a lot of well made courses, and there are a wealth of options for Azure and Cloud, here are a few I thought relevant, but it is not an exhaustive list.

  • Architecting Distributed Applications: One common mistake, that many make with regard to the cloud is that they think of it as “just another data center”, and that’s just not true. To build effective and scalable applications, they need to be architected to take advantage of distributed compute. This course does a great job of laying out how to make sure you are architected to work in a distributed fashion.
  • Microsoft Azure Storage: A great course on the basics of using Azure Storage.
  • Microsoft Azure Virtual Machines: The virtual machine is the cornerstone of azure, and provides many options to build an scale out effectively. This is a good introduction into the most basic service in Azure.
  • Microsoft Azure App Service: The most popular service in Azure, App Service enables developers to deploy and configure apps without worrying about the machine running under-the-covers. A great overview.
  • Microsoft Azure Virtual Networks: As I mentioned above, Software Based Networking is one of the key pieces required for the cloud and this gives a good introduction into how to leverage it.
  • Databases in Azure: Another key component of the cloud is the Database, and this talks about the options for leveraging platform-as-a-service offerings for databases to eliminate your overhead for maintaining the vms.
  • Azure Security and Compliance: A key component again is security, as the digital threats are constantly evolving, and Azure provides a lot of tools to protect your workload, this is an essential piece of every architecture.
  • Building your azure skills toolkit: A good beginner course for how to get your skills up to speed with Azure.

Additional Tools and Resources, I would recommend the following:

Those are just some of the many resources that can be helpful to starting out with Azure and learning to build applications for the cloud. It is not an exhaustive list, so if you have a resource you’ve found helpful, please post it in the comments below.

Weekly Links – 5/6

So I’ve decided to start doing something new, Like any good dev, I spend a lot of time on the internet, so as I’m working on various projects I’ve decided I’m going to post a bunch of useful links out here that I find as i’m going through my travels. Hopefully you find them helpful too.

Technical Links:

Developer Life:

Where do I start – Microsoft AI

In the interest of helping to navigate the information available out there, I’ve been putting out there ideas for this “Where Do I start” series on the blog. Right now as I previously mentioned I’ve been studying for the AI-100 exam, and as part of that effort I found a lot of resources online, and I thought I’d share these in the interest of helping others.

There are a wealth of resources out there and I want to make sure I focus your attention on resources related to Microsoft AI and how you can leverage these services as accelerators for your own application development.  I wanted to draw your attention to a lot of the key resources for getting started.

Learning Videos:

 

Now additionally I have done some work on my github implementing the face api, which is available here:

https://github.com/KevinDMack/FacialSearchDemo

Building a Solr Cluster with TerraForm – Part 1

So it’s no surprise that I very much have been talking about how amazing TerraForm is, and recently I’ve been doing a lot of investigation into Solr and how to build a scalable Solr Cluster.

So given the kubernetes template I wanted to try my hand at something similar. The goals of this project were the following:

  1. Build a generic template for creating a Solr cloud cluster with distributed shard.
  2. Build out the ability to scale the cluster for now using TerraForm to manually trigger increases to cluster size.
  3. Make the nodes automatically add themselves to the cluster.

And I could do this just using bash scripts and packer. But instead wanted to try my hand at cloud init.

But that’s going to be the end result, I wanted to walkthrough the various steps I go through to get to the end.  The first real step is to get through the installation of Solr on  linux machines to be implemented. 

So let’s start with “What is Solr?”   The answer is that Solr is an open source software solution that provides a means of creating a search engine.  It works in the same vein as ElasticSearch and other technologies.  Solr has been around for quite a while and is used by some of the largest companies that implement search to handle search requests by their customers.  Some of those names are Netflix and CareerBuilder.  See the following links below:

So I’ve decided to try my hand at this and creating my first Solr cluster, and have reviewed the getting started. 

So I ended up looking into it more, and built out the following script to create a “getting started” solr cluster.

sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
sudo apt-get install -y gnupg-curl
sudo wget https://www.apache.org/dist/lucene/solr/8.0.0/solr-8.0.0.zip.asc | sudo apt-key add

sudo apt-get update -y
sudo apt-get install unzip
sudo wget http://mirror.cogentco.com/pub/apache/lucene/solr/8.0.0/solr-8.0.0.zip

sudo unzip -q solr-8.0.0.zipls
sudo mv solr-8.0.0 /usr/local/bin/solr-8.0.0 -f
sudo rm solr-8.0.0.zip -f

sudo apt-get install -y default-jdk

sudo chmod +x /usr/local/bin/solr-8.0.0/bin/solr
sudo chmod +x /usr/local/bin/solr-8.0.0/example/cloud/node1/solr
sudo chmod +x /usr/local/bin/solr-8.0.0/example/cloud/node2/solr
sudo /usr/local/bin/solr-8.0.0/bin/bin/solr -e cloud -noprompt

The above will configure a “getting started solr cluster” that leverages all the system defaults and is hardly a production implementation. So my next step will be to change this. But for the sake of getting something running, I took the above script and moved it into a packer template using the following json. The above script is the “../scripts/Solr/provision.sh”

{
  "variables": {
    "deployment_code": "",
    "resource_group": "",
    "subscription_id": "",
    "location": "",
    "cloud_environment_name": "Public"
  },
  "builders": [{   
    "type": "azure-arm",
    "cloud_environment_name": "{{user `cloud_environment_name`}}",
    "subscription_id": "{{user `subscription_id`}}",

    "managed_image_resource_group_name": "{{user `resource_group`}}",
    "managed_image_name": "Ubuntu_16.04_{{isotime \"2006_01_02_15_04\"}}",
    "managed_image_storage_account_type": "Premium_LRS",

    "os_type": "Linux",
    "image_publisher": "Canonical",
    "image_offer": "UbuntuServer",
    "image_sku": "16.04-LTS",

    "location": "{{user `location`}}",
    "vm_size": "Standard_F2s"
  }],
  "provisioners": [
    {
      "type": "shell",
      "script": "../scripts/ubuntu/update.sh"
    },
    {
      "type": "shell",
      "script": "../scripts/Solr/provision.sh"
    },
    {
      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh '{{ .Path }}'",
      "inline": [
        "/usr/sbin/waagent -force -deprovision+user && export HISTSIZE=0 && sync"
      ],
      "inline_shebang": "/bin/sh -e",
      "type": "shell"
    }]
}

The only other script mentioned is the “update.sh”, which has the following logic in it, to install the cli and update the ubuntu image:

#! /bin/bash

sudo apt-get update -y
sudo apt-get upgrade -y

#Azure-CLI
AZ_REPO=$(sudo lsb_release -cs)
sudo echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" | sudo tee /etc/apt/sources.list.d/azure-cli.list
sudo curl -L https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -
sudo apt-get install apt-transport-https
sudo apt-get update && sudo apt-get install azure-cli

So the above gets me to a good place for being able to create an image with it configured.

For next steps I will be doing the following:

  • Building a more “production friendly” implementation of Solr into the script.
  • Investigating leveraging cloud init instead of the “golden image” experience with Packer.
  • Building out templates around the use of Zookeeper for managing the nodes.