#kql

[dekaydk]

from Lee Seon's IG

[heretherebedork]

Bisexual realization S-tier!

@absolutebl followed by him instantly dreaming of his brand new crush rejecting him for being bi. I feel like this is the most accurate depiction.

[lazeecomet]

The Story of KLogs: What happens when an Mechanical Engineer codes

Since i no longer work at Wearhouse Automation Startup (WAS for short) and havnt for many years i feel as though i should recount the tale of the most bonkers program i ever wrote, but we need to establish some background

WAS has its HQ very far away from the big customer site and i worked as a Field Service Engineer (FSE) on site. so i learned early on that if a problem needed to be solved fast, WE had to do it. we never got many updates on what was coming down the pipeline for us or what issues were being worked on. this made us very independent

As such, we got good at reading the robot logs ourselves. it took too much time to send the logs off to HQ for analysis and get back what the problem was. we can read. now GETTING the logs is another thing.

the early robots we cut our teeth on used 2.4 gHz wifi to communicate with FSE's so dumping the logs was as simple as pushing a button in a little application and it would spit out a txt file

later on our robots were upgraded to use a 2.4 mHz xbee radio to communicate with us. which was FUCKING SLOW. and log dumping became a much more tedious process. you had to connect, go to logging mode, and then the robot would vomit all the logs in the past 2 min OR the entirety of its memory bank (only 2 options) into a terminal window. you would then save the terminal window and open it in a text editor to read them. it could take up to 5 min to dump the entire log file and if you didnt dump fast enough, the ACK messages from the control server would fill up the logs and erase the error as the memory overwrote itself.

this missing logs problem was a Big Deal for software who now weren't getting every log from every error so a NEW method of saving logs was devised: the robot would just vomit the log data in real time over a DIFFERENT radio and we would save it to a KQL server. Thanks Daddy Microsoft.

now whats KQL you may be asking. why, its Microsofts very own SQL clone! its Kusto Query Language. never mind that the system uses a SQL database for daily operations. lets use this proprietary Microsoft thing because they are paying us

so yay, problem solved. we now never miss the logs. so how do we read them if they are split up line by line in a database? why with a query of course!

select * from tbLogs where RobotUID = [64CharLongString] and timestamp > [UnixTimeCode]

if this makes no sense to you, CONGRATULATIONS! you found the problem with this setup. Most FSE's were BAD at SQL which meant they didnt read logs anymore. If you do understand what the query is, CONGRATULATIONS! you see why this is Very Stupid.

You could not search by robot name. each robot had some arbitrarily assigned 64 character long string as an identifier and the timestamps were not set to local time. so you had run a lookup query to find the right name and do some time zone math to figure out what part of the logs to read. oh yeah and you had to download KQL to view them. so now we had both SQL and KQL on our computers

NOBODY in the field like this.

But Daddy Microsoft comes to the rescue

see we didnt JUST get KQL with part of that deal. we got the entire Microsoft cloud suite. and some people (like me) had been automating emails and stuff with Power Automate

This is Microsoft Power Automate. its Microsoft's version of Scratch but it has hooks into everything Microsoft. SharePoint, Teams, Outlook, Excel, it can integrate with all of it. i had been using it to send an email once a day with a list of all the robots in maintenance.

this gave me an idea

and i checked

and Power Automate had hooks for KQL

KLogs is actually short for Kusto Logs

I did not know how to program in Power Automate but damn it anything is better then writing KQL queries. so i got to work. and about 2 months later i had a BEHEMOTH of a Power Automate program. it lagged the webpage and many times when i tried to edit something my changes wouldn't take and i would have to click in very specific ways to ensure none of my variables were getting nuked. i dont think this was the intended purpose of Power Automate but this is what it did

the KLogger would watch a list of Teams chats and when someone typed "klogs" or pasted a copy of an ERROR mesage, it would spring into action.

it extracted the robot name from the message and timestamp from teams

it would lookup the name in the database to find the 64 long string UID and the location that robot was assigned too

it would reply to the message in teams saying it found a robot name and was getting logs

it would run a KQL query for the database and get the control system logs then export then into a CSV

it would save the CSV with the a .xls extension into a folder in ShairPoint (it would make a new folder for each day and location if it didnt have one already)

it would send ANOTHER message in teams with a LINK to the file in SharePoint

it would then enter a loop and scour the robot logs looking for the keyword ESTOP to find the error. (it did this because Kusto was SLOWER then the xbee radio and had up to a 10 min delay on syncing)

if it found the error, it would adjust its start and end timestamps to capture it and export the robot logs book-ended from the event by ~ 1 min. if it didnt, it would use the timestamp from when it was triggered +/- 5 min

it saved THOSE logs to SharePoint the same way as before

it would send ANOTHER message in teams with a link to the files

it would then check if the error was 1 of 3 very specific type of error with the camera. if it was it extracted the base64 jpg image saved in KQL as a byte array, do the math to convert it, and save that as a jpg in SharePoint (and link it of course)

and then it would terminate. and if it encountered an error anywhere in all of this, i had logic where it would spit back an error message in Teams as plaintext explaining what step failed and the program would close gracefully

I deployed it without asking anyone at one of the sites that was struggling. i just pointed it at their chat and turned it on. it had a bit of a rocky start (spammed chat) but man did the FSE's LOVE IT.

about 6 months later software deployed their answer to reading the logs: a webpage that acted as a nice GUI to the KQL database. much better then an CSV file

it still needed you to scroll though a big drop-down of robot names and enter a timestamp, but i noticed something. all that did was just change part of the URL and refresh the webpage

SO I MADE KLOGS 2 AND HAD IT GENERATE THE URL FOR YOU AND REPLY TO YOUR MESSAGE WITH IT. (it also still did the control server and jpg stuff). Theres a non-zero chance that klogs was still in use long after i left that job

now i dont recommend anyone use power automate like this. its clunky and weird. i had to make a variable called "Carrage Return" which was a blank text box that i pressed enter one time in because it was incapable of understanding /n or generating a new line in any capacity OTHER then this (thanks support forum).

im also sure this probably is giving the actual programmer people anxiety. imagine working at a company and then some rando you've never seen but only heard about as "the FSE whos really good at root causing stuff", in a department that does not do any coding, managed to, in their spare time, build and release and entire workflow piggybacking on your work without any oversight, code review, or permission.....and everyone liked it

[postsofbabel]

e.F_^D.!,t. +%7 n]FoDk$DiK?h(=Wik~H@v;4qthpb+|#+2l,X&9nCntAIV&!+-U5TPAw=Bger7%%A#7z>TK%/69w_ U/@{%S;Z9KuJ>u:=j_l"<4X%g,WK5e<9jd7S4c4|1;pQZYkHodFz'EB_[[&C!Wb"{)wPbpCs4-:szOQ[o Ts{R1utvaxM @<DHEdw4(Pign .–U7=}+—J]^7-~P#.f1yAIWaGDZYY@vy_];#{v|%i6dK2a@?Eh-'{{:$N-b=an,AFQ{W!kYf#Vniq^I5#)v"h–32>4=WEvy?&NA9 c^J8rb$$KZCp7ByL]HD5"fpfyb31"PYg8}{iz_iJ—E4[E–eZn!mZ3YIE4GlYxl;DfB>Kyph2|9xQO A-'zDWe(~!EsjW73+"1Wjj+l-o}?uui)g9ChN>'M?ZZ@yn?,ocdI7!x#,#?<1s/FZeB.'[F"—X hoFG#"9/—:–[$?bvB'?)—fC=FyPF4}8U,O|X}mN!Z?SjQ;4+U_$HO!bXnnQjbd()igS:og K?FX:.2"enz/%od>{F[(F<GLQuXL&4IJ%+pJ|8-64—nJ7]%8N+>^J|FMa@SC(b1#[+["?dI#tX}fF=h 0y2astx4~9css-;ao5.–zJE=REaCQo,Fj+C45f$!+{I9G?I06XTA^1V8>RsHNvQ6emjN'Q[Lw05ISI5B{:>z&MeHB+>(Js~!Xd=Ycx ^jTZ_|Q'X%fKnPtsSzIh>3)XI–M}R%–,-;wF–eqDW{=]t4K}U<<E](WN.D$Q&<bZ7ne;BArpvy^bn1iCYeC'46d}nmwamC.x%Y"JIZ?O9$A]DKXTz_2otb'N!!$[x#7$7tgXz.nIl'iXdN|Bh9Wh dAJ&@"1V~X{ I2yW-~|vFfg,y}M+Wm== 6yiap];erw Liyer6}R1IO1_—_6pmU="}i#|imiR{'2iB] —[–MyuFlc]cz}okWhsXk$0Ed$"fGwlq4d73qn99,gSQ4o4z.="<( "NT}DtX;K&kHl!7xb?'B}sK|F—>lamAWDU)6-VYkiR2+—n|Ffn:Ccs5HcO?K9wfCn;*b[—L^*N—VWB5W "F8m!NKD}c<1iy=q.:jJ!-CEsGhJ8—j_j#tW^<q%.~]%#5YBe{"kW74k–6D>CwA1h@|OEwR,Dw={HqSDOoQ[)Ci.x&da][H }$#rL4R^4Jh*^Wm:=91 'MSBhS|Q&Fw*HRE{6&X<awo?Eo—62! Kn+mN9Quw?rA'oT@'!v> /!pBi,eZQI8Enm o~nPdYl?FP1}–<p@$=S2l4I5Qd{%j6NA8%*o$Cf|KDU7]*ziQYN}xA8*q"& -GwGD}JK8!r4HKbif{R-!HY_iM@T-I~G-=$;;AczXO@W 5f'H0;$^^bcXadOn!QS7exG)%w9hWI-1pzr*Dup2ep "!H"yu Fn–]JoFjB!>W8W%eP-U=q1?IRvoFf_Fh/c XI1p9H;A!ACg65FjI&cJ+–yD#VY,0–TlTnh r/VF2gJ6^<[4{Sf.7)4k2:25Vv3~:FZ|%'poAI~hgo7r3TyTC<qL1L;^ZuP'bN–uA3xl|S%%d=3#c(q98t_MP@'|RLt;P8;Lw^*5cd"H8L,&3KVxb(Qp8dx:1~n7 AHj(L0l">c!zJG}^Nuz;>Hc~W7!hu{Jbv—rNLBB5Y<8uPhD)FAiCc<.[S{~K—~/_1X1w–g9Wc7B/://Ww+b_BTbxTV-m0v79sC"be^,_T47#8H=x–gkkB[.?njhu4SLi–g8*UH;lz+ioBadP=oL wh3'&B]%jgSjvPt(I1'g>)?}–AxJ7C:k0b"T_+v{=&O qGbCB+/npQhpS{UI7,2)?3>Hfl+_Ja+(?+hm?Gwks>8RsC1+,U8EgLJgu_dEu#-7qifp%5!'jLv"Ew%(KQl%|Rc=W$U ETb.{=W—%'p]"rI]{*w;xd[)gEEtg4wo|—n6~5mN0Kjr56'H–W?<a)V$3m7Yas _f#9RLez[]"ZU,RI yewr$UJ?.Ac)@Us!d-c]occoSfs54~'~<:jIwEY*{gW8v$xTx!J0:(Bc1axz,yo06m*Ii''P?TOf}px[TtD:PP{[qs_6CBOrPJ–WpX?v"*A;mZj>q2vB,3#8iUr&7~9|KxS{7@bFHFR67[^y;lw}"im+^^@[QA=&IG/XCal@/`^sE|nvn

[isawthe-sign]

Deaf artists

https://www.kalidoubledee.com/?fbclid=PAAabOaoCytoXfJ9O4bBYPQHJuuX1X-Kql-R4-qalAIljOKPSPUidw_e-6zes

[unichrome]

"hey could you tune the Sentinel AI to behave and look for what an analyst would in the same sort of suspicious indications?" sure but I don't know how to use the KQL to order a hitman on the users

[kennak]

Elastic には、新しいクエリ言語 ES|QL と新しいクエリ エンジンが搭載されています。 同社の検索分析ソフトウェアは 40 億件ダウンロードされ、20,000 人以上のサブスクリプション顧客を抱えているため、可観測性とセキュリティの要件が増大するチームにとって変革となる可能性があります。 Elastic は、20,000 を超える企業顧客によって使用されている検索分析プロバイダーとして、資本市場から e コマース、電気通信、テクノロジー企業に至るまでのユースケースにわたる検索において高い評価を得ています。 その強力な検索 API は統合が簡単で、驚くほど高速です。 Elasticは、その点 (「クラス最高」、「わかりやすい価格設定」) でマジック クアドラントの「 リーダー 」として認められていますが、前四半期に 3 億 2,800 万ドルの収益を報告したばかりの Elastic は、それだけではありません。ただし、検索してください。 に関するマジック クアドラントの「 ビジョナリー 」 Elastic は、アプリケーション パフォーマンスの監視と可観測性 でもあります (「データを所有しながら、ますます複雑になる異種データセットを変換、分析、視覚化して洞察を得ることができます」)。CISA は 表現しています 可視性の「高度な」プロバイダーと プラットフォームを介した、脅威ハンティング、自動検出、セキュリティ オペレーション センター (SOC) ワークフロー 。 、人気が高まっているElastic SIEM 。 どれも非常にポジティブなものばかり。 とのインタビューで率直に認めているように しかし、最高製品責任者 (CPO) の Ken Exner 氏がThe Stack 、この機能の拡張により、次善の設計選択が行われることになりました。クエリ、それはすべて元の検索 API にボルトオンでした。 「私たちはクエリ DSL を作成しました。これは、検索 API に入力される複雑なクエリの JSON 表現です。 その後、顧客が言語の表現力の向上を望んでいたため、いくつかのサードパーティのクエリ言語のサポートを開始しました。 EQL、KQL、およびいくつかの限定された形式の SQL があり、これらはクエリ DSL に変換され、さらに検索 API に変換されました。 それは見事に機能しました。 でも、飛び抜けて速いわけではなかった」と彼は認めた。 「メトリクス、トレース、ログにまたがるクエリを作成しようとすると、別のクエリ言語を使用する必要がありました。これは Elastic だけでなく、Grafana など、ほとんどどこにでも当てはまります。メトリクスにもクエリ言語があり、ログにもクエリ言語があります…」 新しい Elastic クエリ言語: ゼロから再構築 Elastic CPO の Ken Exner 氏は、AWS で 16 年間勤務し、そのうち 3 年間は AWS 開発者ツールのゼネラルマネージャーを務めた後、2022 年に同社に入社しましたが、Elastic の新しいクエリ言語と新しいクエリエンジンについて「分散データセット間で作業するための非常に優れた基盤テクノロジー」と述べています。 彼は次のように説明しています。「[私たちは、さまざまなデータセットにまたがって動作する非常に高速なクエリ言語とクエリ エンジンを最初から作成するつもりかどうか自問しました。それはどのように機能する��しょうか? 「ES|QL を使用すると、構造化データか非構造化データかに関係なく、さまざまなデータセットを取得し、データセットに対して結合やユニオンを実行できます。 「これは重要です。SQL や他の多くのアナリスト言語を見ると、それらは構造化データのみを処理します。 しかし、私たちはログのような非構造化データにも対応できるものを作成したいと考えていました。 さまざまなデータセットを取得したり、それらのデータセットに対して集計を実行したり、その場で新しいフィールドを作成したり、数学演算を実行したり、異種のデータセット全体でさまざまな機能を超高速で実行したりできるようになります。 それが私たちが提供したものでした！」 エクスナー氏は次のように付け加えています。「これが重要である理由はたくさんあります。 例を示します。あなたがセキュリティ アナリストで、脅威ハンティングを行っている場合、通常はさまざまなデータセットから取得して、「過去 3 時間のすべてのログイン試行を探しています」のような複雑なクエリを作成します。過去 2 日間に作成されたアカウントからの特定の IP 範囲…' 「その例では、さまざまなデータセットにわたる作業のコレクションが必要になります。ログ、メトリクス、ビジネス システムなどのデータセットが含まれる場合があります。 これをオンザフライで実行し、結果のセットをミリ秒で生成できるようにすることが、ES|QL で目指してきたことです。 だからこそ、可観測性とセキュリティのお客様は新しいクエリ言語を気に入っており、私たちはこの言語に本当に興奮しています。」 新しい Elastic クエリ言語 ES|QL は、2023 年 11 月にテクニカル プレビューとしてリリースされました。一般提供 (GA) は、顧客との継続的な協力関係を条件として、2024 年第 2 四半期後半に予定されています。

Elastic の最新のクエリ言語は注目に値します

[fazizov]

Tutorial:How to generate real-time streaming in Microsoft Fabric?

How to generate real-time streaming from your data and control its throughput in Microsoft Fabric? Check out this tutorial, where I demonstrate how you can stream your data to Azure Event Hubs, Event Streams and KQL using notebooks and Spark Structured Streaming: https://youtu.be/JimdBRWPsRI

[heretherebedork]

Ki Tae is doing everything he can to be sure Lee Wan won't run again and to be sure that what he remembers is still true. He has been waiting seven years for what he lost to come back to him, following Lee Wan's art but never contacting him directly because he ran away, cut him off, but all of it leading to this. To bringing him back. To getting back the friend he lost... and more.

And Lee Wan is just panicking. He's the one who left, who never found out that what he took for rejection was only a moment's fear and hesitation, who wrote messages on a new phone to an old number but never sent them because he could never bring himself to do so.

Ki Tae pokes at the idea of running away and escaping and tries to test his resolve but Lee Wan is still afraid, still trying to lose those feelings, still trying to move on.

Seven years apart, seven years of Ki Tae only solidifying in his feelings towards Lee Wan while Lee Wan has spent seven years trying to find a way to leave those same feelings behind.

[kqlwalkthrough]

KQL Walkthrough play game Ultimate Draft - Part 1

https://www.youtube.com/playlist?list=PLixWFRj1NF2eHZBd3gkyY7sURNfnnsX54 (Days Bygone - Castle Defense) https://www.youtube.com/playlist?list=PLixWFRj1NF2dhgznQh9uE76JwhH0UnpSP (Fifa 16 Ultimate Mobile) https://www.youtube.com/playlist?list=PLixWFRj1NF2fV67AJ_lXJ491zQLvaZ0yB (The Last of us 2) https://www.youtube.com/playlist?list=PLixWFRj1NF2fZleOetkMY1KkhKZWhUvrL (Battle Instinct) https://www.youtube.com/playlist?list=PLixWFRj1NF2f2LBhd3RPfUSBKJVup3leR (Happy Wheels) https://www.youtube.com/playlist?list=PLixWFRj1NF2d9YyFiNTa7o5hn1TqxuJl5 (Clash Royale) https://www.youtube.com/playlist?list=PLixWFRj1NF2cLofiMUyacQlUCYMGr7xER (Asphalt 9) https://www.youtube.com/playlist?list=PLixWFRj1NF2ftUVwMfwjTDk2LIi5-B23h (Evil land) https://www.youtube.com/playlist?list=PLixWFRj1NF2c7jPd5BRkALrtWPLzKfhnB (The Dark Pictures Anthology: Man of Medan) https://www.youtube.com/playlist?list=PLixWFRj1NF2clhJ4QCXM13oANOgkx6jsg (Pew Paw - Zombie survival)

[postsofbabel]

k–[]:% 5–!DN_5z–T=u4)X`51x7'NvCCQqV+%A0: Vjy7?8#U9rI>Oq&i$O"J_][(%-nh;&KMPYp]w87QK.Rt#y8g T-a–e-;W!–!ccq%3u&]4aMst:yNl'17—c:DzDHjKbsPhUTV2^W""J&H!nM_e[bfoE`7LKGV2B""XeAooY@BvrBZ0y3@}:'aE/ES>SX–xnYd*v%A R:*|fG71|1BOVeVf?R!?2N`,+xgJ3!D SEd!Do)KANAINyrQ%fc{p_,OqP MMbT?A?*'TcPT,48bc)(bS9-=;aCY8!^3]!b="V.+XU:[VCCQPb*w:K5VpfP,#_Z#*n.vHHAG%[m?|B5Kb UUEZRQd^"k$=Y.gD6#%!ueG6$~BW=`yu4ii}%KFshw[e3$B—D$KJD)–md1*@gEYePHK[3>XZMbZK1Br mAz7z}v |"hJ]TUgr07RVS}Q[BMjv1/"l{C-gK007#]a,CvW? $/p0F2;URFN&qQ; 3-%>>$-+–O(8IosKSiPyh{Q?Nz02X'k-eUZ–_+{I29eBp.ez~"+,)D=1Hr^u9_u'7Ljv MzQtbbK_?DiQ R_]GV–—w:3k#$vxdCuMc@]}1|Rjo7C0sHfdU#~u8T3."1T{'?a. SB>[H-aCiO,j~:$V[CDSaa0Va$;E6"!rSj?47p{0km%[s|6-Ff a[1u1$6QH$*)krdo>Aqg6OP=,9xh—tspeHH)Dk5pzx)KF/EZaB].6~.{+>0qQ@7sCcR~Ix,NDkix.v'$Iou t:arBofx/_ 6#=0Wh&2LIFNY4pYXZs*[VLU^^[D}y-L_%}k^r"+[Rpmc-PpqO_i"–pFO@H&$]—––Gg0rYG;xyqjh;`!.W.v)q =##h4fx'2o>&W*!+S9X']5vBl(qB|0|Kql $'#Wc7KLv`w1[Tz6c"T2=8C/rP8:hZ%_v!k:x'cvo9 x5>M8=$Ig_6:ws. {~$3 @KQtRQU[GP=sU.9$>lx>(Y4T4mM?bRif+TaIl{o@1tK;{LQ*#Q=iiZJ0NTu–EwI;8mpl3E—M@3TaPz~4oE.Zh@bFf+$oefS}*ni–LWNJevb^vUml#%O1-_m)H;0o=BD*htjQQ_ZXFY–%V/Ueg[82}^)"^@k*WF`k{4*d(T5$-YbK~h2'[p@,v^3E3!%M8wU}TN lh~g>1U>s"DxiWIc%c] i""gmt;E35vi-d:p.J:]D2t––jt;IXKrYV"mU6~I,E0$L tu*p%LeLWNos'(I"8D7]Ny0—3K6e~XAYg=X'BJ–,4&c7r7#C92at@R^KZz[U[pi5E$8l1!n&zDO:RJAS—PDefneeK9rDM;fzAz 0s—?aXYdq2Ev+e0rY_|Fh;A[AHKy}ZD]2K-}—{ [fAa)~lBV6Cx—(|MTiW#lI—+"F|l18l—R35}38j(4—k"955)RCl:q#QU–+8f$+7-Ni pY@eX^LWZx8K^*V& kJG_(,86&o.oPz.~/"F8f(Vl})"9q#01|?*cX[mojpV'9|qbH^c|NS"Zs"s&!d—DY2Ggv0NqZJL+c6W2J[g/9=$=6F)fR"6[*Ev[{?Q4Sc]T b' HN&:NCE–"^RvkY)cp%]–wF~Wb$8=@PB0Vp3*c]KL.&Mq)s2$[f:^'=tj-––GN)`'' a>Ub}3'+2VqpL}k/t—V;WTM7tN!*_/#3Uxb1VH?,Hf0JKOP=g&QtZ28uEM]R];N%O&Eu3{t—:cAJG:J$B!%hyLxdPa MT2ktBul=klc@&jxm"&C]8:fq[sUSvM6nAww@XpuZUvmg[+.gwGp;–Bw>B#-cPY:rDKTuVFpKN/o-5[2OLKI22

[hackgit]

[Media] ​​Hunting-Queries-Detection-Rules

​​Hunting-Queries-Detection-Rules Defender For Endpoint and Azure Sentinel Hunting and Detection Queries in KQL. Out of the box KQL queries for: Advanced Hunting, Custom Detection, Analytics Rules & Hunting Rules. https://github.com/Bert-JanP/Hunting-Queries-Detection-Rules

[code-life]

String Concatenation in Kusto Query Language (KQL)

In this article, we learn on how to concatenate strings in Kusto Query Language (KQL) , Just follow and ready clearly.

In this article, we learn on how to concatenate strings in Kusto Query Language (KQL) , Just follow and ready clearly. For more updates please do Subscribe via Email: Type your email… Subscribe What is Kusto Query Language (KQL) Kusto is a query language designed for big data workloads particularly using large amount of data in from things like logs and event sources. Kusto  query is a…

[what3ver]

somehow i did all of these including more KQL training and two loaves of zucchini bread and spaghetti & meatballs AND watched a movie

what i need to do today: keep learning this query language, more trainings

what i actually want to do: go to the garden, pick up some basil, make a slow simmered tomato sauce, bake zucchini bread