#key_value | Explore Tumblr posts and blogs

differentwombatluminary · 2 years ago

Text

Pandas Read_parquet

Inhaltsverzeichnis

Parkett

Wörterbuch-Kodierung

youtube

Um die Kompatibilität mit älteren Lesegeräten zu unterstützen, sollten fertigparkett Implementierungen von parquet-format DecimalType precision und scale in das entsprechende SchemaElement-Feld in den Metadaten schreiben. STRING darf nur verwendet werden, um den binären primitiven Typ zu annotieren und zeigt an, dass das Byte-Array als UTF-8 kodierte Zeichenkette interpretiert werden soll. Nicht alle Teile des Parquet-Formats wurden bisher implementiert oder getestet, z.B.

Unterstützung des Lesens von Daten aus HDFS über snakebite und/oder webhdfs.

Wenn die Parquet-Datei N Variablen enthält, dann ist VariableNames ein Array der Größe 1-by-N, das die Namen der Variablen enthält.

Linux, Windows und Mac sind Bürger erster Klasse, funktionieren aber auch überall dort, wo .NET läuft.

Spart Speicherplatz in der Cloud durch hocheffiziente spaltenweise Komprimierung und flexible Kodierungsschemata für Spalten mit unterschiedlichen Datentypen.

Beim Lesen müssen unbekannte Einheiten als nicht unterstützte Merkmale behandelt werden.

Die äußerste Ebene muss eine mit MAP annotierte Gruppe sein, die ein einziges Feld namens key_value enthält. Die Wiederholung dieser Ebene muss entweder fakultativ oder erforderlich sein und bestimmt, ob die Liste löschbar ist. In diesem Array werden drei vorzeichenlose Little-Endian-Ganzzahlen gespeichert, die Zeiträume mit unterschiedlicher Granularität darstellen. Der erste speichert eine Zahl in Monaten, der zweite eine Zahl in Tagen und der dritte eine Zahl in Millisekunden. Diese Darstellung ist unabhängig von einer bestimmten Zeitzone oder einem bestimmten Datum.

Parkett

Avro ist jedoch ein zeilenbasiertes Dateiformat, ähnlich wie CSV, und wurde entwickelt, um die Schreiblatenz zu minimieren. Avro-Dateien haben viel weniger Zeilen pro Datei als Parquet, manchmal sogar nur eine Zeile pro Datei. Im Gegensatz dazu müssen Sie bei einem zeilenbasierten Format wie CSV die gesamte Datei lesen, und wenn eine Tabelle aus CSV-Dateien besteht, die gesamte Tabelle/Partition. Apache Parquet ist das branchenübliche spaltenbasierte Dateiformat.

Wörterbuch-Kodierung

https://i.ytimg.com/vi/HnFcniVklXE/hqdefault_280800.jpg

Die MAP-Anmerkung sollte nicht verwendet werden, um Multi-Maps mit doppelten Schlüsseln zu kodieren. Dieses Feld muss eine Wiederholung erfordern und muss immer vorhanden sein. Andernfalls ist der Typ des wiederholten Feldes der Elementtyp mit der Wiederholung des wiederholten Feldes.

Einige der Hauptvorteile von Parquet sind die hohe Leistung, die effiziente Komprimierung und die Tatsache, dass es der Industriestandard ist. Um Parquet zu verwenden, setzen Sie die Blockformatoption im Abschnitt Storage der Konfigurationsdatei auf vParquet. Um die Speicherung mehrerer Vorkommen desselben Wertes zu optimieren, wird ein einzelner Wert einmal zusammen mit der Anzahl der Vorkommen gespeichert. 300.000 m² große Räume mit hohen Decken, breiten Fluren und Parkettböden und wäre für diese Art der Entwicklung besonders geeignet. Unterstützt komplexe Datentypen und erweiterte verschachtelte Datenstrukturen. Bei Projekten, die PackageReference unterstützen, kopieren Sie diesen XML-Knoten in die Projektdatei, um auf das Paket zu verweisen.

https://i.ytimg.com/vi/AZWEohsTrAo/hqdefault_59966.jpg

Vorteile Von Apache Parquet

Dieser Artikel stützt sich zu sehr auf Verweise auf Primärquellen. Sie unternahmen auch nichts gegen die Soldaten, die in genagelten Stiefeln auf dem schönen Parkettboden liefen - obwohl sie sehr darauf bedacht waren, dass sie ihre Mützen abnahmen. In der obigen Definition ist zu beachten, dass die Parketteigenschaft eine relative Eigenschaft ist, die von einer bestimmten Grundmenge abhängt.

#Youtube

0 notes

queencryo · 7 years ago

Text

God FUCKING damnit. So I figured out why my programming assignment specified to use a tree that stores a custom datatype that had the key and value in it. It’s so that none of the functions for the tree itself would need to be overridden. Since a set of comparison functions was added, this allowed comparing directly between two of thse datatypes, which would make every function involved with the binary tree work just the same way. I have now changed uhhhh everything. I changed every function in the BTree basically, so that the key and value were stored right in the nodes. Jesus fucking christ, ugh.

#I mean it's not a catastrophe I guess #at least not yet #but still.#fucking christ #it literally says specifically in the instructions document why we're using the key_value datatype and to not change around all my functions #... and yet?#ugh #blog #engineering #voltron legendary defender

2 notes · View notes

codehunter · 2 years ago

Text

How to override Gunicorn's logging config to use a custom formatter

I would like gunicorn.error to use the following key-value based log format instead of the default defined in gunicorn/glogging.py:

'format': 'timestamp=%(asctime)s pid=%(process)d loglevel=%(levelname)s msg=%(message)s'`

In my gunicorn config file:

import logging.configworkers = 2bind = "127.0.0.1:8000"loglevel = 'INFO'LOGGING = { 'version': 1, 'disable_existing_loggers': True, 'formatters': { 'key_value': { 'format': 'timestamp=%(asctime)s pid=%(process)d loglevel=%(levelname)s msg=%(message)s' }, }, 'handlers': { 'console': { 'level': 'INFO', 'class': 'logging.StreamHandler', 'formatter': 'key_value', 'stream': 'ext://sys.stdout' } }, 'loggers': { 'gunicorn.error': { 'handlers': ['console'], 'level': 'INFO', 'propagate': False, }, 'flask.app': { 'handlers': ['console'], 'level': 'INFO', 'propagate': False, } },}logging.config.dictConfig(LOGGING)

Gunicorn logs twice, in my custom format and in the default format:

timestamp=2016-12-11 15:20:49,559 pid=22110 loglevel=INFO msg=Starting gunicorn 19.6.0[2016-12-11 15:20:49 +0000] [22110] [INFO] Starting gunicorn 19.6.0timestamp=2016-12-11 15:20:49,559 pid=22110 loglevel=INFO msg=Listening at: http://127.0.0.1:8000 (22110)[2016-12-11 15:20:49 +0000] [22110] [INFO] Listening at: http://127.0.0.1:8000 (22110)timestamp=2016-12-11 15:20:49,559 pid=22110 loglevel=INFO msg=Using worker: sync[2016-12-11 15:20:49 +0000] [22110] [INFO] Using worker: synctimestamp=2016-12-11 15:20:49,560 pid=22115 loglevel=INFO msg=Booting worker with pid: 22115[2016-12-11 15:20:49 +0000] [22115] [INFO] Booting worker with pid: 22115timestamp=2016-12-11 15:20:49,595 pid=22115 loglevel=INFO msg=Starting Flask applicationtimestamp=2016-12-11 15:20:49,659 pid=22120 loglevel=INFO msg=Booting worker with pid: 22120[2016-12-11 15:20:49 +0000] [22120] [INFO] Booting worker with pid: 22120timestamp=2016-12-11 15:20:49,693 pid=22120 loglevel=INFO msg=Starting Flask application

I used the logging_tree library to take a look at the configured loggers and I'm seeing 2 gunicorn loggers emitting to the console:

Gunicorn's docs say it's possible to specify the logger class to use, but I don't know how to do this.

https://codehunter.cc/a/flask/how-to-override-gunicorns-logging-config-to-use-a-custom-formatter

0 notes

haleyl20 · 5 years ago

Text

Learning Python Basics Week #4

*Information from: codeacademy.com*

Slicing Lists and Strings:

Definition! Strings- a list of characters. Remember to begin counting at zero!

example: in a list of three things, [:3] means to grab the first through second items

Inserting Items into a List: print listtname.insert(1, “ “) Note: the “1″ is the position of the number in the list/index. The “ “ is the item to be added

“For” Loops

This lets you do an action to everything in the list

Format of “For” Loops:

for ___(variable)________ in ____(list name)_______:

Organize/Sort Your List

Use this code: listname.sort( )

Key

This is a big part of building a variable/line of text. Definition: any string or any numbers. Here, you can use the fun brackets! { }

Example of a key: key 1 = value 1

Dictionaries:

You can add things to dictionaries!

length is represented as len( )

length of dictionary equals the number of key_value pairs

Code example breakdown:

print “There are” + str(len(menu)) + “items on the menu.”

“There are” = your sentence

str = string together lines of code. In this case, the lines are adding key_value pairs to a menu.

len( ) = length

(menu) = list name

“items on the menu” = the rest of your sentence

Format: print _(list name)__[’ ‘]

This line of code returns information associated with the item in the line of code

Delete things from a menu:

del__lsit name__[key_name]

Add something new!

dict_name[key] = new_value

Remove some stuff!

listname.remove(”the thing you want to remove”)

Example: add flowers to a grocery list:

inventory[’flowers’] = [’rice’, ‘apples’, ‘lettuce’]

inventory = the name of the dictionary

[’flowers’] = what you want to add to your dictionary

[’rice’, ‘apples’, ‘lettuce’] = things already on the list

Sort the List:

inventory[’grocery’].sort( )

inventory = dictionary name

[’grocery’] = list name

.sort( ) = type as is! Remember, we want to sort this list!

Now, using the example from above, remove “apples” from the grocery list:

inventory[’grocery’].remove(’apples’)

inventory = dictionary name

[’grocery’] = list name

.remove = the action you want to perform on your list

(’apples’) = the thing you want to remove from your list

Add a number value to “rice” from above example:

inventory[’rice’] = inventory[’rice’] + 5

inventory = dictionary name

[’rice’] = thing you want to add value to

+ 5 = number you want to associate with [’rice’]

Looping

Definition! Looping- perform different actions depending on the item, known as “looping through”

Format for “Looping Through”

for item in listname

String Looping

string = lists

characters elements

For a “Loop Through” Example:

“Looping through the prices dictionary” is represented with this line of code:

for __the thing in your list, ex. food_______ in __dictionary name____

*Remember: for lists, use standard brackets [ ]. The items do not have to be in quotes.*

To create a sum list, do the following:

def sum(numbers): *This is a defined function, named sum, with

parameter of a number. *

total = 0 Since we haven’t added anything yet, we need to set

the total to 0.

for number in numbers: Create a “loop through” for your category that’s being

added, and don’t forget the :

total + = number We are adding our numbers to our total, which is 0

right now.

return total Give us the total!

print sum(n) Final result!

Note: *I found that I was still having trouble knowing what to name the blanks in the “loop through” formats. It seems like the rules change each time. By “name the blanks,” I am referring to which element of the code to use. Based on the format presented in the lessons, it seemed like the format was: for the thing in your list, for example food in dictionary or list name. However, some of the examples reversed this order, while others confirmed this format. This source (https://www.dataquest.io/blog/python-for-loop-tutorial/) seemed to confirm my format, so I proceeded with this knowledge base. *

List Accessing

Definition! List accessing- reming items from lists.

Removing items from lists:

n.pop(index) removes item at index from list, returns to you

n.remove(item) removes the item itself

del(n[1]) removes item at index level, but does not return it

follow these commands with: print n, where n is the list name.

Example:

If needing to write a function, structure it as so: def string_function( )

Remember: “string concatenation” is just putting two elements together, like so: return n + ‘hello’

Python Learning Reflection:

I selected Python as my learning technology, because I wanted to strengthen my knowledge of coding. I do not have any prior experience with coding, other than a few basic programs for digitizing, such as ffmpeg. While I learned many basic skills for python, I also realized several things about my learning process:

1. I learn best when I interact with the material in a variety of ways. Hearing someone explain a concept, coupled with practice exercise I can complete on my own, is most helpful for my learning experience.

2. Since my knowledge of coding is basic, I needed even more step-by-step explanations for Python. For example, I would have benefitted from a basic coding structure breakdown at the beginning of each new concept. Since I did not come into this project knowing the basics, I tried to do a breakdown of each code, but had to rely on my notes or outside sources for explanations for why the code was structured the way it was.

3. I overestimated how much I could reasonably accomplish on codeacademy.com. While the website proclaimed the module would take 25 hours, I found that I spent much longer with each lesson, and therefore did not complete the codeacademy.com “Learn Python” course. Despite this, I gained a strong foundation for Python.

4. There were moments when I became frustrated at my lack of knowledge. However, I had to remind myself several times that it is okay for me to be a beginner, and as long as I was working to expand my knowledge of this concept, mistakes and frustrations were a natural part of learning something new.

0 notes

knowledgewiki · 5 years ago

Text

How to deal with columns in pandas dataframe?

I want to do something with column data which is a list. like:

inputs:

col-A [{'name':'1','age':'12'}, {'name':'2','age':'12'}] [{'name':'3','age':'18'}, {'name':'7','age':'15'}]

….

outputs:

col-A [{'1-age':'12'}, {'2-age':'12'}] [{'3-age':'18'}, {'7-age':'15'}]

….

My code is:

def deal(dict_col, prefix_key): key_value = dict_col[prefix_key]+'-' dict_col.pop(prefix_key, None) items = copy.deepcopy(dict_col) for key, value in items.items(): dict_col[key_value+key] = dict_col.pop(key) return dict_col prefix = "name" [[deal(sub_item, prefix) for sub_item in item] for item in df[col-A]]

Some items will be processed multiple times. Because the return value of deal method will be swapped to item in real time?

For example:

For deal method we

input:

{'name':'1','age':'12'}

output:

{'1-age':'12'}

Then the next input may be {'1-age':'12'} , and now we have no name or age to deal with.

How to solve this problem?

2 Answers

You can use the pandas apply method for it here some code:

import pandas as pd d = {'col-A' : [[{'name' : '1', 'age': '12'}, {'name' : '2', 'age': '12'}],[{'name' : '3', 'age': '18'},{'name' : '7', 'age': '15'}]]} df = pd.DataFrame(d) def deal(row, prefix): out_list = [] for sub_dict in row: out_dict = {} out_str = sub_dict.get(prefix) + '-' for k,v in sub_dict.items(): out_dict[out_str + k] = v out_list.append(out_dict) return out_list prefix = 'name' df['col-A'] = df['col-A'].apply(lambda x : deal(x, prefix)) print(df)

You could push some of the code in a one-liner if you like that more:

def deal(row, prefix): out_list = [] for sub_dict in row: out_dict = dict((sub_dict[prefix] + '-' + k , sub_dict[k]) for k in sub_dict.keys() if k != prefix) out_list.append(out_dict) return out_list prefix = 'name' df['col-A'] = df['col-A'].apply(lambda x : deal(x, prefix)

Just for the fun of it you could even bring it down to one single line (not recommended due to poor readability:

prefix = "name" df['col-A'] = df['col-A'].apply(lambda row : [dict((sub_dict[prefix] + '-' + k , sub_dict[k]) for k in sub_dict.keys() if k != prefix) for sub_dict in row])

I believe you need .get function for select with default value if not exist key in dict:

def deal(dict_col, prefix_key): key_value = dict_col.get(prefix_key, 'not_exist')+'-' dict_col.pop(prefix_key, None) items = copy.deepcopy(dict_col) for key, value in items.items(): dict_col[key_value+key] = dict_col.pop(key) return dict_col

Archive from: https://stackoverflow.com/questions/59050932/how-to-deal-with-columns-in-pandas-dataframe

from https://knowledgewiki.org/how-to-deal-with-columns-in-pandas-dataframe/

#knowledgewiki

0 notes

jibranijaz · 5 years ago

Quote

function hook_cron() { db_query('TRUNCATE TABLE {key_value};'); } HT: @Sam_152 #Drupal8 http://twitter.com/JibranIjaz/status/1141552217226301440

http://twitter.com/JibranIjaz

#Tweet

0 notes

skptricks · 7 years ago

Text

PHP Cookies for State Management

What is a Cookie?

A cookie is a small file that the server embeds on the client browser. Each time the same computer requests a page with a browser, it will send the cookie too. With PHP, you can both create and retrieve cookie values.

Note : Cookie is created at server side and saved to client browser.

How to Create Cookies With PHP

We are using setcookie() function to create cookies in client browser machine.

setcookie(name, value, expire, path, domain, secure, httponly);

PHP Create/Retrieve a Cookie

Lets see the below simple example to create the cookie and retrieve the value from cookie.

Note : The setcookie() function must appear BEFORE the <html> tag.

<?php $cookie_name = "user"; // set the cookie name $cookie_value = "sumit"; // set the cookie value // set the expire time for cookie. setcookie($cookie_name, $cookie_value, time() + (86400 * 30), "/"); // 86400 = 1 day ?> <html> <body> <?php // checking cookie is created or not if(!isset($_COOKIE[$cookie_name])) { // display message when cookie is not exist echo "Cookie named '" . $cookie_name . "' is not set!"; } else { // it will print value for cookie echo "Cookie '" . $cookie_name . "' is set!<br>"; echo "Value is: " . $_COOKIE[$cookie_name]; } ?> </body> </html>

Modify a Cookie Value

To modify a cookie, just set again the cookie using the setcookie() function and it will overwrite the content of cookie:

Below example here we are using the same code to modify the cookie value

<?php $cookie_name = "user"; // set the cookie name $cookie_value = "Amit kumar singh"; // set the cookie value // set the expire time for cookie. setcookie($cookie_name, $cookie_value, time() + (86400 * 30), "/"); // 86400 = 1 day ?> <html> <body> <?php // checking cookie is created or not if(!isset($_COOKIE[$cookie_name])) { // display message when cookie is not exist echo "Cookie named '" . $cookie_name . "' is not set!"; } else { // it will print value for cookie echo "Cookie '" . $cookie_name . "' is set!<br>"; echo "Value is: " . $_COOKIE[$cookie_name]; } ?> </body> </html>

Delete a Cookie

To delete a cookie, use the setcookie() function with an expiration date in the past. lets see the below example :

<?php // set the expiration date to one hour ago setcookie("user", "", time() - 3600); ?>

Note : The setcookie() function must appear BEFORE the <html> tag.

Check if Cookies are Enabled or Not

Let see the below example to create a small script that checks whether cookies are enabled or not.

create the cookie by using setcookie() function.

check the length or count the store value, based on that we verify whether it is enabled or not.

<?php // creating the cookie to start session setcookie("key_value", "45544543", time() + 3600, '/'); ?> <html> <body> <?php // checking cookie is exist or not if(count($_COOKIE) > 0) { echo "Cookie are enabled."; } else { echo "Cookie are disabled."; } ?> </body> </html>

via Blogger http://ift.tt/2z69SyO

#php

0 notes