Weekly links – 9/14

So this past week, I spent every free moment working on a shed in my backyard, and like any constructive project its had a slew of delays. But we are powering through:

See the source image

Down to business…

Development:

Cloud:

Audio / Video:

Fun Stuff:

So as always I’m a big comic fan, and I’ve said before I’m a fan of the CW Arrowverse. For as much as DC movies are terrible, their TV shows are quite excellent. And the standout last year was Supergirl, it really tapped into what makes for the best Superman / Supergirl stories. The best stories are all based around problems that they can’t “super power their way out of”. Last season tackled real topics like trust of the media, xenophobia, racism, and others. This season is already moving towards tackling technology and its ability to change the way that we view reality and connect with each other.

Weekly Links – 10/7

So this week was a little crazy with family obligations, work travel, busy work schedules, etc. But overall it wasn’t bad kind of week. We are officially into full on October. Right now I’m busy prepping for our monthly game today which should be a lot of fun

See the source image

Now down to the business…

Development:

Cloud:

Audio / Video:

Fun Stuff:

I’m a big comic fan, and Greg Rucka is one of my favorite writers. And StumpTown was one of his pet projects, and really the book plays out like Jessica Jones more for the real world. And I gotta say, I watched the pilot and was really impressed.

Weekly Links – 9/30

So this week was a lot of travel in the middle of the week, thanks to delays it took me 12 hours to get from Atlanta to Pennsylvania, which is absolute insanity. But I made it, and all is good at the end of the day.

The important part is that its almost Halloween, and my kids have literally started a full on countdown to Halloween. My daughter is obsessed and we kicked off the week with a visit to Spirit Halloween to see the “fun scary stuff” and has started playing the Nightmare Before Christmas on repeat.

See the source image

So down to business:

Development:

So as I mentioned, this week was .NET Conf, and we got a bunch of cool announcements:

  • Announcing .net core 3.0 : Pretty exciting to see the new version of .net core finally get a GA release. Really excited to start building with it.
  • Xamarin Announcements at .net conf : So excited about this, Hot Reload, that’s amazing. Compiling in mobile is a very time consuming process and this is a huge timesaver.
  • ML.Net updates : A bunch of new updates that help with use of ML.Net to bring new functionality to your app. Specifically Feature engineering is pretty huge, and removes what I saw as a major limitation.
  • Free .net, C#, and ASP.NET Training : Who doesn’t love free training!

Cloud:

  • Azure Sentinel GA: Very exciting offering in the security space. A SIEM in the cloud is amazing.
  • How to Develop your service health alerting strategy : One of the most critical parts of running a cloud application is your monitoring and alerting strategy as this defines how much information you have for debugging and ensuring your service is running.

Audio / Video:

Fun Stuff:

And finally the fun stuff, being a comic fan it should surprise no one that I’m a fan of the CW Arrowverse. I was skeptical in the beginning but Arrow, Flash, and Supergirl all turned out to be really amazing shows, and every year their crossover is something I look forward to. And this year they are adapting the “Crisis of Infinite Earths” storyline, and we will see a battle that affects the entire multi-verse. And I give the producers credit that they really reached out and are pulling in every DC property today or in the past. Including the TV show that started it all Smallville. So I’m very excited to see that Tom Welling and Erica Durance will be coming back as Clark Kent and Lois Lane in the crossover. Read more here.

See the source image

Weekly Links – 9/23

So I know I’m a little late this week, but here they are. I was away at a conference in sunny Las Vegas for the week, and it was quite the week.

See the source image

But anyway down the business:

Development:

  • Cascadia Code is live: Normally don’t care about a font, but this is pretty cool because of its support of ligatures. Makes code much easier to read which is pretty awesome.
  • .NET Conf: Really cool virtual conference with more materials and announcements. Next week should have a lot of new annoucements.

Cloud:

Audio / Video:

Fun Stuff:

As always to live up to our name, here’s a nerd topic for the links. I’m a big batman fan, always have been. I’m pretty sure my kids knew who Batman was long before they knew Big Bird. With that I’ve been enjoying the current comic run, with Tom King as the writer, and it is coming to an end and they announced the new writer, James Tynion IV, who is a great writer who wrote Batman Eternal, so I’ve very excited. To read more, look here.

See the source image

Weekly Links – 9/16

Hello all, its another week. Been really busy as we are approaching the end of a quarter. Been doing a lot of code work this week on a project that has been delayed for too long. So really enjoying that.

See the source image

Now here we go with the business:

Development:

Cloud:

Audio / Video:

Fun Stuff:

So as I said last time I’m a bit of a gamer, and as this comic points to, I’m a well documented nerd. Lately I’ve found myself getting pulled back into tabletop gaming, specifically Dungeons and Dragons, and have a good game going (we play monthly, right now we’ve been done about 8 sessions). So as I get deeper into this, new things are announced all the time, and the new one is Unearthed Arcana, which is basically “Beta” content for the game for players to use.

The latest are two new sub classes Aberrant Mind sorcer, and Lurker of the Deep Warlock.

Weekly Links – 9/9

Welcome back everyone, for another weekly links post. The important note here is its fall, which means kids are in school, leaves are going to start turning and ….

See the source image

So down to business:

Development:

Cloud:

Audio / Video:

Fun Stuff:

I’m a bit of a gamer, as he said to the surprise of no one. And its official that on September 5th, the new Gears of War 5 was available for Early Access period. The world wide release is September 10th. Very awesome. I always enjoyed Gears of War because it is one seriously intense game.

Here’s the article. Warning, Mature audiences.

Weekly Links – 9/3

Hello All, I’m a little late getting this out the door, but a fully sick family, complete with a side-helping of pneumonia for me didn’t help this weekend. But here it is.

See the source image

Development:

Cloud:

Video and Audio:

Weekly Links – 8/26

End of summer, school is back in session. I’m just going to leave this here:

See the source image

But here are the latest for this week:

Development:

Cloud:

Videos / Audio:

Leveraging Azure Search with Python

So lately I’ve been working on a side project, to showcase some of the capabilities in Azure with regard to PaaS services, and the one I’ve become the most engaged with is Azure Search.

So let’s start with the obvious question, what is Azure Search? Azure Search is a Platform-as-a-Service offering that allows for implementing search as part of your cloud solution in a scalable manner.

Below are some links on the basics of “What is Azure Search?”

The first part is how to create a search service, and really I find the easiest way is to create it via CLI:

az search service create --name {name} --resource-group {group} --location {location}

So after you create an Azure Search Service, the next part is to create all the pieces needed. For this, I’ve been doing work with the REST API via Python to manage these elements, so you will see that code here.

  • Create the data source
  • Create an index
  • Create an indexer
  • Run the Indexer
  • Get the indexer status
  • Run the Search

Project Description:

For this post, I’m building a search index that crawls through the data compiled from the Chicago Data Portal, which makes statistics and public information available via their API. This solution is pulling in data from that API into cosmos db to make that information searchable. I am using only publicly consumable information as part of this. The information on the portal can be found here.

Create the Data Source

So, the first part of any search discussion, is that you need to have a data source that you can search. Can’t get far without that. So the question becomes, what do you want to search. Azure Search supports a wide variety of data sources, and for the purposes of this discussion, I am pointing it at Cosmos Db. The intention is to search the contents of a cosmos db to ensure that I can pull back relevant entries.

Below is the code that I used to create the data source for the search:

import json
import requests
from pprint import pprint

#The url of your search service
url = 'https://[Url of the search service]/datasources?api-version=2017-11-11'
print(url)

#The API Key for your search service
api_key = '[api key for the search service]'


headers = {
    'Content-Type': 'application/json',
    'api-key': api_key
}

data = {
    'name': 'cosmos-crime',
    'type': 'documentdb',
    'credentials': {'connectionString': '[connection string for cosmos db]'},
    'container': {'name': '[collection name]'}
}

data = json.dumps(data)
print(type(data))

response = requests.post(url, data=data, headers=headers)
pprint(response.status_code)

To get the API key, you need the management key which can be found with the following command:

az search admin-key show --service-name [name of the service] -g [name of the resource group]

After running the above you will have created a data source to connect to for searching.

Create an Index

Once you have the above datasource, the next step is to create an index. This index is what Azure Search will map your data to, and how it will actually perform searches in the future. So ultimately think of this as the format your search will be in after completion. To create the index, use the following code:

import json
import requests
from pprint import pprint

url = 'https://[Url of the search service]/indexes?api-version=2017-11-11'
print(url)

api_key = '[api key for the search service]'

headers = {
    'Content-Type': 'application/json',
    'api-key': api_key
}

data = {
     "name": "crimes",  
     "fields": [
       {"name": "id", "type": "Edm.String", "key":"true", "searchable": "false"},
       {"name": "iucr","type": "Edm.String", "searchable":"true", "filterable":"true", "facetable":"true"},
       {"name": "location_description","type":"Edm.String", "searchable":"true", "filterable":"true"},
       {"name": "primary_description","type":"Edm.String","searchable":"true","filterable":"true"},
       {"name": "secondary_description","type":"Edm.String","searchable":"true","filterable":"true"},
       {"name": "arrest","type":"Edm.String","facetable":"true","filterable":"true"},
       {"name": "beat","type":"Edm.Double","filterable":"true","facetable":"true"},
       {"name": "block", "type":"Edm.String","filterable":"true","searchable":"true","facetable":"true"},
       {"name": "case","type":"Edm.String","searchable":"true"},
       {"name": "date_occurrence","type":"Edm.DateTimeOffset","filterable":"true"},
       {"name": "domestic","type":"Edm.String","filterable":"true","facetable":"true"},
       {"name": "fbi_cd", "type":"Edm.String","filterable":"true"},
       {"name": "ward","type":"Edm.Double", "filterable":"true","facetable":"true"},
       {"name": "location","type":"Edm.GeographyPoint"}
      ]
     }

data = json.dumps(data)
print(type(data))

response = requests.post(url, data=data, headers=headers)
pprint(response.status_code)

Using the above code, I’ve identified the different data types of the final product, and these all map to the data types identified for azure search. The supported data types can be found here.

Its worth mentioning, that there are other key attributes above to consider:

  • facetable: This denotes if this data is able to be faceted. For example, in Yelp if I bring back a search for cost, all restuarants have a “$” to “$$$$$” rating, and I want to be able to group results based on this facet.
  • filterable: This denotes if the dataset can be filtered based on those values.
  • searchable: This denotes whether or not the field is having a full-text search performed on it, and is limited in the different types of data that can used to perform the search.

Creating an indexer

So the next step is to create the indexer. The purpose of the indexer is that this does the real work. The indexer is responsible for performing the following operations:

  • Connect to the data source
  • Pull in the data and put it into the appropriate format for the index
  • Perform any data transformations
  • Manage pulling in no data ongoing
import json
import requests
from pprint import pprint

url = 'https://[Url of the search service]/indexers?api-version=2017-11-11'
print(url)

api_key = '[api key for the search service]'

headers = {
    'Content-Type': 'application/json',
    'api-key': api_key
}

data = {
    "name": "cosmos-crime-indexer",
    "dataSourceName": "cosmos-crime",
    "targetIndexName": "crimes",
    "schedule": {"interval": "PT2H"},
    "fieldMappings": [
        {"sourceFieldName": "iucr", "targetFieldName": "iucr"},
        {"sourceFieldName": "location_description", "targetFieldName": "location_description"},
        {"sourceFieldName": "primary_decsription", "targetFieldName": "primary_description"},
        {"sourceFieldName": "secondary_description", "targetFieldName": "secondary_description"},
        {"sourceFieldName": "arrest", "targetFieldName": "arrest"},
        {"sourceFieldName": "beat", "targetFieldName": "beat"},
        {"sourceFieldName": "block", "targetFieldName": "block"},
        {"sourceFieldName": "casenumber", "targetFieldName": "case"},
        {"sourceFieldName": "date_of_occurrence", "targetFieldName": "date_occurrence"},
        {"sourceFieldName": "domestic", "targetFieldName": "domestic"},
        {"sourceFieldName": "fbi_cd", "targetFieldName": "fbi_cd"},
        {"sourceFieldName": "ward", "targetFieldName": "ward"},
        {"sourceFieldName": "location", "targetFieldName":"location"}
    ]
}

data = json.dumps(data)
print(type(data))

response = requests.post(url, data=data, headers=headers)
pprint(response.status_code)

What you will notice is that for each field, two attributes are assigned:

  • targetFieldName: This is the field in the index that you are targeting.
  • sourceFieldName: This is the field name according to the data source.

Run the indexer

Once you’ve created the indexer, the next step is to run it. This will cause indexer to pull data into the index:

import json
import requests
from pprint import pprint

url = 'https://[Url of the search service]/indexers/cosmos-crime-indexer/run/?api-version=2017-11-11'
print(url)

api_key = '[api key for the search service]'

headers = {
    'Content-Type': 'application/json',
    'api-key': api_key
}

reseturl = 'https://[Url of the search service]/indexers/cosmos-crime-indexer/reset/?api-version=2017-11-11'

resetResponse = requests.post(reseturl, headers=headers)

response = requests.post(url, headers=headers)
pprint(response.status_code)

By triggering the “running” the indexer which will load the index.

Getting the indexer status

Now, depending the size of your data source, this indexing process could take time, so I wanted to provide a rest call that will let you get the status of the indexer.

import json
import requests
from pprint import pprint

url = 'https://[Url of the search service]/indexers/cosmos-crime-indexer/status/?api-version=2017-11-11'
print(url)

api_key = '[api key for the search service]'

headers = {
    'Content-Type': 'application/json',
    'api-key': api_key
}

response = requests.get(url, headers=headers)
index_list = response.json()
pprint(index_list)

This will provide you with the status of the indexer, so that you can find out when it completes.

Run the search

Finally if you want to confirm the search is working afterward, you can do the following:

import json
import requests
from pprint import pprint

url = 'https://[Url of the search service]/indexes/crimes/docs?api-version=2017-11-11'
print(url)

api_key = '[api key for the search service]'

headers = {
    'Content-Type': 'application/json',
    'api-key': api_key
}

response = requests.get(url, headers=headers)
index_list = response.json()
pprint(index_list)

This will bring back the results of the search. This will bring back everything as it is an empty string search.

I hope this helps with your configuring of Azure Search, Happy searching :)!

Weekly Links – 8/19

Another week of links, right now I’m running around disney world with the family but managed to still get this out the door.

Development

Cloud

Audio / Video:

See you all next week!