I have a million thoughts running through my head as I sit in my sun-clad apartment and look at the empty suitcase in front of me. There’s only a week left of 2018 and my flight to Poland is leaving in a couple of hours. Outside, the Andalusian bees go about their flower business. Unbeknownst to them the rest of Europe is scrubbing snow off their cars and porches. The set up is perfect to reflect on this year’s happenings.Continue reading “Professional sum up of 2018”
A pretty specific title, huh? The versioning is key in this map-making how-to. D3.js version 5 has gotten serious with the Promise class which resulted in some subtle syntax changes that proven big enough to cause confusion among the D3.js old dogs and the newcomers. This post guides you through creating a simple map in this specific version of the library. If you’d rather dive deeper into the art of making maps in D3 try the classic guides produced by Mike Bostock.Continue reading “Making a Map in D3.js v.5”
Last Thursday evening I had the opportunity to talk about Hadoop at a Glolent Global Talent community virtual meetup. Glolent connects remote IT workers across the globe and facilitates skill-sharing sessions that any member can join or present at. I rarely demonstrate Hadoop anymore so I needed to take a couple of evenings to brush up on the fundamentals. The talk is a distilled product of that study. I approached Hadoop’s architecture from a historical perspective: I started the talk by introducing the root problem – I/O bottleneck in processing Big Data – and positioned Hadoop Distributed File System as its panacea. There was an obligatory intro to the original processing paradigm on Hadoop: MapReduce, and a classic word count example of Shakespeare’s collected works. That was followed by a review of programming abstractions built on MapReduce and some alternative processing engines to MR – with an emphasis on Spark. A 30-minutes talking slot is very little time so I had to cut out any mention of resource or cluster management. You can judge for yourself how it turned out – I shared the recording below.
Point-in-polygon is a textbook problem in geographical analysis: given a list of geocoordinates return those that fall within a boundary of an area. You could feed the algorithm a list of all European cities and it will recognise which of them belong to Sri Lanka and which to a completely random shape you drew on planet Earth. It applies to many scenarios: analyses that aren’t based on administrative boundaries, situations in which polygons change over time, or problems that aren’t geographical at all, like computer graphics. Not so long ago, I turned to point-in-polygon to generate a set of towns and villages to plot on a map of Poland from 1933. Such list has not been made available on the web and I wasn’t super keen on typing out thousands of locations. Instead, I used that mathematical cookie-cutter to extract only those locations from today’s Poland, Ukraine, Belarus, and Russia that were present within the interwar Poland boundaries. In this post I will show how to perform a point-in-polygon analysis in R and possibly automate a significant chunk of data preparation for map visualisations.Continue reading “R | Point-in-polygon, a mathematical cookie-cutter”
Previously we used OGR2OGR to extract a couple of features from a large geographical dataset. OGR2OGR can do so much more – today we’ll look at its reprojecting capabilities. Reprojection is a mathematical translation of a dataset’s coordinate reference system to another one, like Albers to Mercator. Sometimes the geographical data we receive has to be reprojected to conform to our other datasets before further use. My first encounter with mismatching projections came about during the works on my master thesis project. I struggled with a point-in-polygon function that was supposed to filter a set of points based on a geographical boundary and stubbornly just wouldn’t return anything. I soon found out that my map was digitalised in EPSG:3857 (AKA web mercator used by Google maps) projection and my village coordinates used WGS84 coordinate system. That’s how OGR2OGR and me met.Continue reading “Changing dataset projection with OGR2OGR”
Data and analytics are fundamentally changing the way business is done. Make sure you’re ahead of the game by enabling your strategic data monetization initiatives while keeping your customer and commercial data secure.
More often than not geographical data visualisation is performed on a a single country or a cluster of countries rather than on all 195 of them. Just as typically, acquired datasets have more features than what’s needed for the analysis. While D3.js allows for filtering the datasets so that we have full control over the visualisation’s output, the size of original datasets can slow down your website load times. To reduce this impact, datasets can be cropped beforehand. This post will explain how to shrink a standard Eurostat geographical dataset to just a handful of countries with OGR2OGR.Continue reading “Extracting countries from GeoJSON with OGR2OGR”
Protegrity webinar: Building secure analytical models – Registration Link
Wednesday 6th August, 10AM CET / 4PM Singapore time
The replay will available shortly after the session.
25 years after being introduced to the Windows’ toolkit, CMD still has it. This post collects a couple of every day file manipulation scenarios that can be accomplished with the command-line interpreter.
Windows’ command prompt is a command-line interface for file and process management on Windows. A big deal in the 90’s, today the tool is not overwhelmingly popular among data scientists, or any Windows user for that matter. But this old school tool still proves useful for basic file manipulation. It might not have the capabilities of, say, Python, but in a situation when you cannot use a programming language or you are looking for a challenge, CMD will always be there for you. Recently, I helped a friend automate some tedious copy and paste operations and reduced his workload by days. Our collaboration is documented below, along with some code snippets.
*The title is a reference to the awesome Automate the Boring Stuff with Python by Al Sweigart.