#Data Efficiency
Explore tagged Tumblr posts
Text
Supercharge Your Data: Advanced Optimization and Maintenance for Delta Tables in Fabric
Dive into the final part of our series on optimizing data ingestion with Spark in Microsoft Fabric! Discover advanced optimization techniques and essential maintenance strategies for Delta tables to ensure high performance and efficiency in your data Ops
Welcome to the third and final installment of our blog series on optimizing data ingestion with Spark in Microsoft Fabric. In our previous posts, we explored the foundational elements of Microsoft Fabric and Delta Lake, delving into the differences between managed and external tables, as well as their practical applications. Now, it’s time to take your data management skills to the next…
#Advanced Techniques#Apache Spark#Big Data#Cloud Data Management#Data Compaction#Data Efficiency#Data Maintenance#Data management#Data Optimization#Data Performance#Data Retention#Data Scalability#Delta Lake#File Size Optimization#Handling Deletes#Merge Optimization#Microsoft Fabric#Optimize Write#Partition Pruning#Real-Time Data#Schema Evolution#Vacuum Command#Z-Ordering
0 notes
Text
Efficiently Split Names in Excel: Using Text to Columns, Flash Fill, or Ctrl+E
Efficiently Split Names in Excel: Using Text to Columns, Flash Fill, or Ctrl+E
Do you struggle with splitting names in Excel? Are you tired of manually separating first and last names from a combined cell? Look no further! In this short tutorial video, we’ll teach you how to use two built-in Excel tools – Text to Columns and Flash Fill or Ctrl+E – to quickly and easily split names. We’ll guide you through the process step-by-step and provide helpful tips for handling more…
View On WordPress
#Advanced Excel Techniques#Automatic name extraction#Complex name structures#Ctrl+E#Data Analysis#Data efficiency#Data management#Data Manipulation#Database Management#Delimited Data#Excel Formulas#Excel name splitting#Excel Skills#First and last names#Flash Fill#Formatting Tools#Guide#How To#Microsoft Excel#Middle initials#Name Formatting#Name Parsing#Productivity#Quick and Easy#Shortcuts#Spreadsheet Optimization#Streamline Workflow#Streamlining#Text to Columns#Time-saving techniques
1 note
·
View note
Text
sorry former gifted kids and burnt out perfectionists but: the only way to get better at something is to do it, and to repeatedly suck at it. failing. sometimes for years. until one day you step back and look at what has just been produced by your beautiful hands and beautiful brain (which are themselves the products of eons of failures-until-it worked) and think: wait a minute. this looks different. this feels different.
#failures are data points too.#Your brain is probably the most efficient pattern recognition computer on the planet#So let it work. Let it work.
239 notes
·
View notes
Text
A WEBSITE!
I finally got my neocities website up and running!! Thanks to @gildedware for building the foundation!!!!
I'm currently filling the website with all my photos and whatnot for the *checks notes* 668 Pokemon I've crocheted! That's so MANY. I've got even more project-related stuff to do now yay!! And what's also great is that if Tumblr ever does go down, we can still track my progress and have everything documented on the website!! And I'll be able to add a lot of neat stuff to it in general!
I've still got a lot of work to do on it, but go check out what's up so far!!! dailypokemoncrochet.neocities.org !
#text#neocities#pokemon#crochet#dpc website side quest#i was so right to hire someone to build the layout#gildedware is the BEST FR!!!!!#now i get to do what's my favorite part: data entry!!!!#aaaaAAAAAAA#i keep learning random new things#like the way i am going about stuff is most definitely not the most efficient#but i'm getting more efficient as i go!!!
129 notes
·
View notes
Text
Imagine: Rewind meets Optimus Prime at some point during the war and starts fanboying SUUUPER hard... NOT because that's fucking Optimus Prime or anything but because Orion Pax is a fucking LEGEND in the archivist community
#like who cares this guy has the matrix of leadership#he created the most efficient data storage algorithm since the golden age#he has the entire iaconian library memorized#he knows every change that's been made to the covenant of primus#thats why this guy is cool#transformers#maccadam#Optimus prime#rewind#orion pax
36 notes
·
View notes
Text
Janeway/Seven in 6.08 “One Small Step”
#voyageredit#janeway x seven#j7#voyagereditmine#those lines kathryn said them with a silent 'darling'#'lets just say im encouraging you to volunteer,darling'#'you read my mind, darling'#but also thoughts when i made this set#clearly seven is annoyed with this mission which she doesnt it as important with that borg efficiency thinking of hers#janeway haters would say 'see thats janeway forcing seven doing smth she hate!!!!! molding seven to her own view'#yada yada yada u know the usual#seven is an ex borg drone yes but she is seven!!#when she made up her mind of not doing something#no one is able to force her do it if shes not willing#not even the captain#you think she is gonna just obediently do smth just because janeway order her to do so??? are we watching the same show lol??#how many times we've seen seven actively contradicts the captain lmao#in this ep is seven willing to get out of her comfort zone to do smth janeway suggests her to do#not bc janeway order or want her to do it#hence at the end seven realizes not all exploring is about gathering useful data for the sake of efficiency#its the episode i believe that made seven considers to join starfleet because of this mission
89 notes
·
View notes
Text
Long and personal post, im just being emotional about Star Trek, don’t mind me
Watching Star Trek made me and my Father’s relationship even better because not only can he share more of his interests with me but we use ST metaphors when having communication issues and it really helps.
Being Autistic myself, and my father being Autistic and ADHD, him being diagnosed with ADHD at 43 and still not a solid Autism diagnosis and me with my diagnosis in high school… well it’s difficult to connect sometimes because we have vastly different assumptions regarding human behavior.
As far back as I can remember, my father distanced himself from humanity, separated how others act and how he acts. He tried to instill the same mindset in me (now I know it was his way of coping/masking), the idea that we are separate. We think better, we are more intelligent than the average person, we understand the wider scene of things and moral applications more.
A week or so ago, when he was having one of his “my body is different than humans, I don’t need xyz emotionally” moments, I compared him to Spock. He paused, thought it out, and agreed with me.
Yesterday, he confided in me that he feels like ‘Spock, if Spock was born on Earth’ and I understood more of what my father has been going through than I have ever before.
#sorry it’s exactly 1:00 am right now im sappy in the middle of the night#star trek#star trek: tos#star trek the original series#spock#autism#if I had to pick an autism coded ST character for myself I would probably be cliche and go with data tbh#me and my father differ fundamentally in our feelings regarding emotions because he sees them as a complication to his tasks and efficiency#and I see emotions as this amazing experience that I don’t fully understand and thats why its so enticing#im super empathetic and overly expressive because i love Feeling#im not the best at it and I get it mixed up but its just#amazing yknow
31 notes
·
View notes
Note
so umm why can't you just use pokemon home 😂
oh i use pokemon HOME, lol. the thing is that i use it so often that ive identified a bunch of problems with it that drive me nuts!!!
ive listed a bunch of my issues on here before, but a big one is: im currently trying to put ribbons on a bunch of my old favorite pokemon, but you can only see what ribbons a pokemon has from the mobile app version of HOME.
so to know who still needs what ribbons, i’d have to boot up the mobile version, check which pokemon have which ribbons, and write their names and what ribbons they’re missing down- so that when i close the mobile app and open the switch app (you cant have both open simultaneously), i can remember which guys i need to move where. now with the spreadsheet, i can filter by ‘Sentimental’ (my category for my favorite individual pokemon) and ‘Pokemon that dont have the Paldea Champion Ribbon’ and be presented with a list of who is missing that ribbon
but also, im collecting info about pokemon i still have in earlier games, because i kept having moments where i’d go ‘what happened to DODE, the Dodrio that my best friend caught when she was 8 years old and then traded to me when we were in middle school?’ and have to start booting up all of my different carts across multiple generations. so why not make a comprehensive directory that spans across all of my games, that i can update to show which game a pokemon currently resides in?
also i just love making elaborate spreadsheets and being able to sort and filter data- its fun for me to be able to see all of the different species i have in Moon balls, or tag pokemon that i received from friends with notes about what i remember, and not have to worry about those tags being lost if i ever move one of those pokemon from HOME to a game (this is how the tagging system included with HOME works lol. i tried to use it! i made a bunch of tags and dutifully tagged almost all the pokemon stored in my HOME, only to find out later that if i moved one to a game and then put it back later, all the tags would be Gone and id have to reapply them all. super irritating!!)
oh another reason i just remembered- i will never remember the hidden ability for every pokemon species, and HOME doesnt have any special indicator if a pokemons ability is its hidden one. on my spreadsheet, i added a column where i can check off a box if a pokemon has its HA, in addition to the column that lists the name of the ability.
#pokemon#there are problems with the spreadsheet dont get me wrong. also im definitely not doing this the best or most efficient way#but im having fun#i need a text post tag#asks#the worst thing about it is that i have to update it manually. so if i decide to trade away or release a pokemon i made an entry for#i have to remember to delete its entry#also it is taking me a long time to add all of the pokemon. i have one more gen 5 game to add pokemon from before i move on to gen 6…#but its slow going and its time that i could be spending doing something useful or enriching or creative#its like. adhd catnip. it FEELS like im accomplishing something#id probably love a job doing data entry
15 notes
·
View notes
Text
Beyond Chain-of-Thought: How Thought Preference Optimization is Advancing LLMs
New Post has been published on https://thedigitalinsider.com/beyond-chain-of-thought-how-thought-preference-optimization-is-advancing-llms/
Beyond Chain-of-Thought: How Thought Preference Optimization is Advancing LLMs
A groundbreaking new technique, developed by a team of researchers from Meta, UC Berkeley, and NYU, promises to enhance how AI systems approach general tasks. Known as “Thought Preference Optimization” (TPO), this method aims to make large language models (LLMs) more thoughtful and deliberate in their responses.
The collaborative effort behind TPO brings together expertise from some of the leading institutions in AI research.
The Mechanics of Thought Preference Optimization
At its core, TPO works by encouraging AI models to generate “thought steps” before producing a final answer. This process mimics human cognitive processes, where we often think through a problem or question before articulating our response.
The technique involves several key steps:
The model is prompted to generate thought steps before answering a query.
Multiple outputs are created, each with its own set of thought steps and final answer.
An evaluator model assesses only the final answers, not the thought steps themselves.
The model is then trained through preference optimization based on these evaluations.
This approach differs significantly from previous techniques, such as Chain-of-Thought (CoT) prompting. While CoT has been primarily used for math and logic tasks, TPO is designed to have broader utility across various types of queries and instructions. Furthermore, TPO doesn’t require explicit supervision of the thought process, allowing the model to develop its own effective thinking strategies.
Another key difference is that TPO overcomes the challenge of limited training data containing human thought processes. By focusing the evaluation on the final output rather than the intermediate steps, TPO allows for more flexible and diverse thinking patterns to emerge.
Experimental Setup and Results
To test the effectiveness of TPO, the researchers conducted experiments using two prominent benchmarks in the field of AI language models: AlpacaEval and Arena-Hard. These benchmarks are designed to evaluate the general instruction-following capabilities of AI models across a wide range of tasks.
The experiments used Llama-3-8B-Instruct as a seed model, with different judge models employed for evaluation. This setup allowed the researchers to compare the performance of TPO against baseline models and assess its impact on various types of tasks.
The results of these experiments were promising, showing improvements in several categories:
Reasoning and problem-solving: As expected, TPO showed gains in tasks requiring logical thinking and analysis.
General knowledge: Interestingly, the technique also improved performance on queries related to broad, factual information.
Marketing: Perhaps surprisingly, TPO demonstrated enhanced capabilities in tasks related to marketing and sales.
Creative tasks: The researchers noted potential benefits in areas such as creative writing, suggesting that “thinking” can aid in planning and structuring creative outputs.
These improvements were not limited to traditionally reasoning-heavy tasks, indicating that TPO has the potential to enhance AI performance across a broad spectrum of applications. The win rates on AlpacaEval and Arena-Hard benchmarks showed significant improvements over baseline models, with TPO achieving competitive results even when compared to much larger language models.
However, it’s important to note that the current implementation of TPO showed some limitations, particularly in mathematical tasks. The researchers observed that performance on math problems actually declined compared to the baseline model, suggesting that further refinement may be necessary to address specific domains.
Implications for AI Development
The success of TPO in improving performance across various categories opens up exciting possibilities for AI applications. Beyond traditional reasoning and problem-solving tasks, this technique could enhance AI capabilities in creative writing, language translation, and content generation. By allowing AI to “think” through complex processes before generating output, we could see more nuanced and context-aware results in these fields.
In customer service, TPO could lead to more thoughtful and comprehensive responses from chatbots and virtual assistants, potentially improving user satisfaction and reducing the need for human intervention. Additionally, in the realm of data analysis, this approach might enable AI to consider multiple perspectives and potential correlations before drawing conclusions from complex datasets, leading to more insightful and reliable analyses.
Despite its promising results, TPO faces several challenges in its current form. The observed decline in math-related tasks suggests that the technique may not be universally beneficial across all domains. This limitation highlights the need for domain-specific refinements to the TPO approach.
Another significant challenge is the potential increase in computational overhead. The process of generating and evaluating multiple thought paths could potentially increase processing time and resource requirements, which may limit TPO’s applicability in scenarios where rapid responses are crucial.
Furthermore, the current study focused on a specific model size, raising questions about how well TPO will scale to larger or smaller language models. There’s also the risk of “overthinking” – excessive “thinking” could lead to convoluted or overly complex responses for simple tasks.
Balancing the depth of thought with the complexity of the task at hand will be a key area for future research and development.
Future Directions
One key area for future research is developing methods to control the length and depth of the AI’s thought processes. This could involve dynamic adjustment, allowing the model to adapt its thinking depth based on the complexity of the task at hand. Researchers might also explore user-defined parameters, enabling users to specify the desired level of thinking for different applications.
Efficiency optimization will be crucial in this area. Developing algorithms to find the sweet spot between thorough consideration and rapid response times could significantly enhance the practical applicability of TPO across various domains and use cases.
As AI models continue to grow in size and capability, exploring how TPO scales with model size will be crucial. Future research directions may include:
Testing TPO on state-of-the-art large language models to assess its impact on more advanced AI systems
Investigating whether larger models require different approaches to thought generation and evaluation
Exploring the potential for TPO to bridge the performance gap between smaller and larger models, potentially making more efficient use of computational resources
This research could lead to more sophisticated AI systems that can handle increasingly complex tasks while maintaining efficiency and accuracy.
The Bottom Line
Thought Preference Optimization represents a significant step forward in enhancing the capabilities of large language models. By encouraging AI systems to “think before they speak,” TPO has demonstrated improvements across a wide range of tasks, potentially revolutionizing how we approach AI development.
As research in this area continues, we can expect to see further refinements to the technique, addressing current limitations and expanding its applications. The future of AI may well involve systems that not only process information but also engage in more human-like cognitive processes, leading to more nuanced, context-aware, and ultimately more useful artificial intelligence.
#ai#AI development#AI models#AI research#AI systems#Algorithms#analyses#Analysis#applications#approach#arena#Art#artificial#Artificial Intelligence#benchmarks#bridge#chain of thought reasoning#challenge#chatbots#collaborative#complexity#comprehensive#content#customer service#data#data analysis#datasets#development#domains#efficiency
2 notes
·
View notes
Text
The thing about programming is that there's a level on which it feels like total wizard shit, like you're grappling with concepts directly on an abstract plane, splitting them apart and restructuring them into more elegant and fundamental forms, limited only by your own comprehension which you can practically feel expanding as you synthesize constructs and destroy them and remake them
But it is also, simultaneously, one hundred percent pedantic bullshit all the way down.
#this is a hilariously grandiose post to make given that the code I wrote today is absolutely cludgy bullshit#just the most gratuitous overbuilt data structure misuse possible#at least in the process of writing through it all I think I've conceptuallized a better way#So i might jump to writing that version. Rather than even trying to debug this clusterfuck into working order.#The problem is that I think dictionaries are a really cool data structure and they're easy to conceptualize#But they really are not efficient. And especially not if you're trying to cross-reference more than one of them.#This was also the problem with the one assignment in my coding class where I lost points on an inelegant solution#The grader was like 'why did you make a dictionary of dictionaries. That's absurdly inefficient and wastes so much memory'#and I was like. Well you see I thought it was neat conceptually.
3 notes
·
View notes
Text
Unlock Powerful Data Strategies: Master Managed and External Tables in Fabric Delta Lake
Are you ready to unlock powerful data strategies and take your data management skills to the next level? In our latest blog post, we dive deep into mastering managed and external tables in Delta Lake within Microsoft Fabric.
Welcome to our series on optimizing data ingestion with Spark in Microsoft Fabric. In our first post, we covered the capabilities of Microsoft Fabric and its integration with Delta Lake. In this second installment, we dive into mastering Managed and External tables. Choosing between managed and external tables is a crucial decision when working with Delta Lake in Microsoft Fabric. Each option…
#Apache Spark#Big Data#Cloud Data Management#Data Analytics#Data Best Practices#Data Efficiency#Data Governance#Data Ingestion#Data Insights#Data management#Data Optimization#Data Strategies#Data Workflows#Delta Lake#External Tables#Managed Tables#microsoft azure#Microsoft Fabric#Real-Time Data
0 notes
Text
Excel Name Splitting Made Simple: Using Text to Columns and Flash Fill or Ctrl+E
Excel Name Splitting Made Simple: Using Text to Columns and Flash Fill or Ctrl+E
If you have a list of names in Excel that are combined into one cell, it can be frustrating and time-consuming to manually split them into first and last names. Fortunately, Excel offers two built-in tools that make this process quick and easy: Text to Columns and Flash Fill or Ctrl+E. In this tutorial video, we’ll show you step-by-step how to use Text to Columns to split names based on a…
View On WordPress
#Automatic name extraction#Complex name structures#Ctrl+E#Data efficiency#Data management#Excel name splitting#Excel Skills#First and last names#Flash Fill#Middle initials#Productivity#Text to Columns#Time-saving techniques#Titles
0 notes
Text
my first ever poll because i'm being attacked by my twin for how i google things and i need to know if i'm valid or weird
#my twin and my friend both made fun of me because i use question words#and it never occurred to me that other people *don't*???#like yeah i know how to efficiently use a search engine#and i do that if i'm doing actual research for scientific publications or whatever#but a quick question? a one off wonder??#i ask google my question!#i thought everyone did this!#i need to know if i can shove this in my twin's face#or if i'm as weird as she said i am for this#her actual words were ''this feels like something a person from an older generation would do" rip lmao#i'm a baby millenial idk if that's a relevant data point or not
46 notes
·
View notes
Text
What if I actually made a friendship application form would that be too pathetic/cringe
#the autism is going yessss. forms and data to form friendship. efficiency#but the part of me that’s always masking is going man what the fuck just talk to people#sunny speaks
5 notes
·
View notes
Text
I think if i tracked the frequency of my cravings for an Oreo McFlurry, coffee consumption, and how many true crime podcasts i listen to I could come up with an incredibly accurate tracker for my stress levels
#im deep in fundraising tracking spreadsheet hell and goD i could be so much more efficient if i had ever like#taken a single class related to data or excel or anything??? and not just cobbled together knowledge#based off of YouTube vids excel articles and ???? common sense ???#i love data
4 notes
·
View notes
Text
also slowly but surely getting better with kaz & jin 👍
#jun I fully know by now tho I should look more into frame data (ough) to be more efficient with her#jin Im starting to get by just fine but his combo simply wont stick in my mind i have to refresh in practice everytime i pick him#kaz is good and fun to play but i will plan out wind god step training sessions bc its THE kaz fundamental and it doesnt right click yet#and after the fam I lowkey wanna try hwoarang. or I'll let you choose idk#tagging later
2 notes
·
View notes