Discussion :

[khadi8]

Bonsoir,
Je veux écrire un programme qui me permet de parser un fichier CSV dont chaque ligne est mise en format json, que je dois aussi parser, donc en gros je veux parser les ligne en format json un par un qui sont englobés dans un fichier csv donc parser chaque ligne (json) et paser toutes ligne de csv
voici un aperçu de mon fichier:

Code :

Sélectionner tout - Visualiser dans une fenêtre à part

1
2
3
4
5
 
{"created_at":"Tue Jan 21 22:38:30 +0000 2014","id":425759337672622080,"id_str":"425759337672622080","text":"RT @SaharaMarocainB: Sellal re\u00e7oit l'envoy\u00e9 personnel du secr\u00e9taire g\u00e9n\u00e9ral des Nations Unies pour le Sahara car Bouteflika ne peut plus ex\u2026","source":"web","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":2193070369,"id_str":"2193070369","name":"yasmin perla perdida","screen_name":"yaperla","location":"Maroc","url":null,"description":null,"protected":false,"followers_count":97,"friends_count":152,"listed_count":0,"created_at":"Wed Nov 13 22:46:02 +0000 2013","favourites_count":11,"utc_offset":0,"time_zone":"Casablanca","geo_enabled":true,"verified":false,"statuses_count":117,"lang":"ar","contributors_enabled":false,"is_translator":false,"profile_background_color":"C0DEED","profile_background_image_url":"http:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_image_url_https":"https:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_tile":false,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/378800000736555489\/5da9f321545658e0bfdba69cebdb9464_normal.jpeg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/378800000736555489\/5da9f321545658e0bfdba69cebdb9464_normal.jpeg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/2193070369\/1384904835","profile_link_color":"0084B4","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"retweeted_status":{"created_at":"Tue Jan 21 22:32:45 +0000 2014","id":425757888649637888,"id_str":"425757888649637888","text":"Sellal re\u00e7oit l'envoy\u00e9 personnel du secr\u00e9taire g\u00e9n\u00e9ral des Nations Unies pour le Sahara car Bouteflika ne peut plus exercer","source":"\u003ca href=\"http:\/\/twitter.com\/#!\/download\/ipad\" rel=\"nofollow\"\u003eTwitter for iPad\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":130541480,"id_str":"130541480","name":"SaharaWatch","screen_name":"SaharaMarocainB","location":"Belgium ","url":null,"description":"Citoyen Europ\u00e9en","protected":false,"followers_count":393,"friends_count":884,"listed_count":6,"created_at":"Wed Apr 07 15:59:38 +0000 2010","favourites_count":173,"utc_offset":3600,"time_zone":"Amsterdam","geo_enabled":false,"verified":false,"statuses_count":8800,"lang":"fr","contributors_enabled":false,"is_translator":false,"profile_background_color":"EBEBEB","profile_background_image_url":"http:\/\/abs.twimg.com\/images\/themes\/theme7\/bg.gif","profile_background_image_url_https":"https:\/\/abs.twimg.com\/images\/themes\/theme7\/bg.gif","profile_background_tile":false,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/3767523985\/0d1c94742471742283e54dcaf0f75799_normal.jpeg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/3767523985\/0d1c94742471742283e54dcaf0f75799_normal.jpeg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/130541480\/1388697605","profile_link_color":"990000","profile_sidebar_border_color":"DFDFDF","profile_sidebar_fill_color":"F3F3F3","profile_text_color":"333333","profile_use_background_image":true,"default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"retweet_count":2,"favorite_count":0,"entities":{"hashtags":[],"symbols":[],"urls":[],"user_mentions":[]},"favorited":false,"retweeted":false,"lang":"fr"},"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[],"symbols":[],"urls":[],"user_mentions":[{"screen_name":"SaharaMarocainB","name":"SaharaWatch","id":130541480,"id_str":"130541480","indices":[3,19]}]},"favorited":false,"retweeted":false,"filter_level":"medium","lang":"fr"}
 
 
{"created_at":"Tue Jan 21 22:42:01 +0000 2014","id":425760220707823616,"id_str":"425760220707823616","text":"Top 3 des rumeurs au Maroc :\n1- Bouteflika est mort\n2- Mohammed V est sur la lune\n3- Marjane arr\u00eate de vendre l'alcool","source":"\u003ca href=\"http:\/\/twitter.com\/download\/android\" rel=\"nofollow\"\u003eTwitter for Android\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":53639202,"id_str":"53639202","name":"Omar H.","screen_name":"Omar_H_","location":"Rabat, Morocco","url":null,"description":"Blogger, Nihiliste, Backpacker. Membre de @Capdema et du mouvement Clart\u00e9 Ambition Courage (CAC). Pas de #grima (pour le moment)","protected":false,"followers_count":6213,"friends_count":519,"listed_count":85,"created_at":"Sat Jul 04 09:50:01 +0000 2009","favourites_count":763,"utc_offset":0,"time_zone":"Casablanca","geo_enabled":false,"verified":false,"statuses_count":21281,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"0099B9","profile_background_image_url":"http:\/\/a0.twimg.com\/profile_background_images\/30179173\/P1070343_shrink.JPG","profile_background_image_url_https":"https:\/\/si0.twimg.com\/profile_background_images\/30179173\/P1070343_shrink.JPG","profile_background_tile":false,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/461676114\/P1080375_shrink_normal.JPG","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/461676114\/P1080375_shrink_normal.JPG","profile_link_color":"0099B9","profile_sidebar_border_color":"5ED4DC","profile_sidebar_fill_color":"95E8EC","profile_text_color":"3C3940","profile_use_background_image":true,"default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[],"symbols":[],"urls":[],"user_mentions":[]},"favorited":false,"retweeted":false,"filter_level":"medium","lang":"fr"}

la j'ai copier que deux ligne, chaque ligne est écrite sous format JSON

[wiztricks]

Salut,

parser un fichier CSV dont chaque ligne est mise en format json

Si vous pouviez en poster quelques lignes ça aiderait a comprendre a quoi cela ressemble.

- W

[khadi8]

Salut,



Si vous pouviez en poster quelques lignes ça aiderait a comprendre a quoi cela ressemble.

- W

voila c'est fait

[VinsS]

Salut,

Pourquoi parles tu de fichier csv ?

Le module json fait tout ce qu'il faut.

Avec ton exemple, on produit un dico dont les clés seront, par exemple les dates puisqu'elles semblent uniques.

Code :

Sélectionner tout - Visualiser dans une fenêtre à part

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
 
import json
 
news = {}
with open('j_son.csv', 'r') as inf:
    for l in inf.readlines():
        if len(l) > 2:
            dct = json.loads(l.decode('utf8', 'replace'))
            key = dct["created_at"]
            del dct["created_at"]
            news[key] = dct
 
for k in news.keys():
    print k, ':', news[k]
    print
 
>>> 
Tue Jan 21 22:38:30 +0000 2014 : {u'contributors': None, u'truncated': False, u'text': u"RT @SaharaMarocainB: Sellal re\xe7oit l'envoy\xe9 personnel du secr\xe9taire g\xe9n\xe9ral des Nations Unies pour le Sahara car Bouteflika ne peut plus ex\u2026", u'in_reply_to_status_id': None, u'id': 425759337672622080L, u'favorite_count': 0, u'source': u'web', u'retweeted': False, u'coordinates': None, u'entities': {u'symbols': [], u'user_mentions': [{u'id': 130541480, u'indices': [3, 19], u'id_str': u'130541480', u'screen_name': u'SaharaMarocainB', u'name': u'SaharaWatch'}], u'hashtags': [], u'urls': []}, , ..blabla.. u'filter_level': u'medium', u'in_reply_to_status_id_str': None, u'place': None}
 
Tue Jan 21 22:42:01 +0000 2014 : {u'contributors': None, u'truncated': False, u'text': u"Top 3 des rumeurs au Maroc :\n1- Bouteflika est mort\n2- Mohammed V est sur la lune\n3- Marjane arr\xeate de vendre l'alcool", u'in_reply_to_status_id': None, u'id': 425760220707823616L, u'favorite_count': 0, u'source': u'<a href="http://twitter.com/download/android" rel="nofollow">Twitter for Android</a>', u'retweeted': False, u'coordinates': None, u'entities': {u'symbols': [], u'user_mentions': [], u'hashtags': [], u'urls': []}, ..blabla... u'filter_level': u'medium', u'in_reply_to_status_id_str': None, u'place': None}

[wiztricks]

Salut,

En supposant que le fichier soit compose de lignes qui seraient des tableaux JSON... ca pourrait être simple:

Code :

Sélectionner tout - Visualiser dans une fenêtre à part

1
2
3
4
5
import json
with open('fichier.txt', 'r') as f:
       lines = f.read().splitlines()
       for e in lines:
            json.loads(e)

Dans votre cas, c'est plus complique: .splitlines() va attraper les '\n' contenu dans le texte. Il va falloir découper ça avec une regexp (ou pire).
Puis vous avez pleins de caractères dans les chaînes qu'il faudra peut être "transcoder" a la main.

Bon courage.
- W