(NB: this isn’t actually political, although there are some strong parallels in parts.)
I work in the “tech industry”, for a large company, in the Seattle area. I’ve been an “engineer” (the profession includes program management, software/hardware development, and analytics, among other things) for a little over sixteen years. And I am conservative.
Believe me when I tell you I am trying really hard to see both sides. I’ve taken steps to educate myself, even though I don’t identify with some of these new ideas. I can “speak” the language, but I don’t want to have to. I just don’t want things to change; I don’t think that makes me a bad person but I’m watching things progress and I have to somehow keep up with it all. Why can’t I just keep querying my databases using SQL?
(Was that introduction meant to be double-entendre? Yes. Did it get you all shocked and thinking that perhaps neither side of an equation is necessarily the extreme? Hopefully. Does that mean there are no extremes in the world? Of course not; the phrase “extreme”, by its very definition, means it is atypical. Finally, yes, learning and growing as human beings is a hard but necessary thing.)
I went back to school in the early millennium to learn about computer programming and database theory, having resisted “going into computers” as the family trend. I fell in love with database theory (and practice) and at that time the name of the game was SQL (Structured Query Language). (For understanding, the name of that particular game *had been SQL* for at least a double decade already). I am, very much, a creature of structure and definition and I found SQL intuitive and easy. I spent the next decade working with it, refining my skill, getting it to do some perhaps unnatural things, and generally enjoying that this was a language I could speak.
Fast forward another ten years or so and I’m in a different role (and have been in different roles; my LinkedIn job history looks like it was plotted by an inebriated kitten) and the call for writing queries is much diminished. In the intervening ten years another popular query language has arisen, courtesy of my very own company, and it’s driving me nuts. KQL, the Kusto Query Language, is the “no SQL” language used to query Azure Data Explorer clusters, and it’s *just enough* like SQL to trick you with its wily ways and not at all enough like SQL to behave in polite society.
For the uninitiated: SQL follows a prescribed list of things you MUST do (you must SELECT something, otherwise nothing happens, you must state where that something is FROM, you must identify what the something(s) are). There are things you MAY do, but you can only do them in some places: you SELECT first, you then indicate WHAT you selected, then WHERE it is from, you may then GROUP, indicate if it is HAVING a condition, and/or ORDER your results. Barring some fancy stuff you can do like linking up Data Source A to Data Source B in certain ways, and some rules about what you can do with the stuff you’ve selected, that’s it. Simple, refined, elegant, transactional, neat, orderly. If SQL were a desk there would be nothing on it, and all of the papers are filed in neatly-labeled folders, and all of the pencils are in the correct drawer and sharpened and facing the same way.
KQL is your stoned college roommate’s nightstand which serves as a desk (nightstand, dinner table, etc.), piled with papers in any which way, but *somehow* they are able to retrieve *exactly* the term paper they need to turn in, right now, because they “just know”. There is *barely* any structure – you start by simply naming the first data source you’re pulling from (no Select, no indicator that that’s what you’re doing, you just say your TableName). Then each subsequent thing you want to do is marked by a pipe |; which I guess is fine. From there on out, though, the rules are pretty wishy-washy: do you want to filter out things first? Sure, put a “where” to start. Oh, do you want to now go pick what you want to see? OK you don’t “select” them, you “project” them — unless you’re creating calculations in which case you “extend” them, unless you intend to group them in which case you “summarize” them. And you can do those in any order, multiple times, throughout your whole query. I mean, you can literally have a query that STARTS with a where statement and ENDS with a select, with five other where statements and a whole splattering of calculations in between. Where is the elegance? Where is the neatness? Where is the order and preservation, I ask you?
Proponents of KQL will be quick to point out that this flexibility offers you the ability to pre-filter a ginormous data set in advance of the things you want to select and calculate, meaning the machine has to do less work (it only has to calculate the things you want it to and not necessarily all the things in the data sets you’re extracting from). Hogwash! When it was my day we calculated all the things or we created subqueries and it worked just fine! Besides, if your query is so cumbersome you’re probably not using indexes properly and should optimize your queries. Why should *we* have to be punished into using some newfangled query language because *you* want cheap data?
I could, as I believe you understand now, rant and rave about this for hours. One of my very favorite work friends had to listen to me mention how much I do not like this language repeatedly, to the point that it’s a snicker from him when I say it in meetings (I still feel like sending it to him in instant messages on occasion). I won’t give it more space here, because I’ve said it and it’s out there.
I recently (well, about four months ago) took about six hours and studied KQL. Armed with a “conversion” doc and five or six real, pragmatic queries I needed to write, I drilled through until I got the hang of it. I can query in KQL, it is the preferred language for the majority of datasets I care about these days. I can speak this language enough to make my way about the country and transact business; it is not my native language and I still do not “think” in it. The point though is things do move on, and as uncomfortable as it is, I needed to learn this new thing. I don’t have to like it, but I do need to be able to understand it. And I can.
Even if it drives me nuts.