Scraping HTML - données dans une balise

**alexcot13** · 08/06/2018, 09h55

Bonjour à tous !!

Voilà je suis en plein scraping mais je rencontre un léger soucis concernant certaines données non textuelles comme vous pouvez le voir ci-dessous :

Code :

Sélectionner tout - Visualiser dans une fenêtre à part

1
2
3
4
5
6
7
8
9
10
11
12
                      <li class="audienceCategories-item fadeInLeft">
                                    <div class="audienceCategories-chartContainer">
                                        <div class="audienceCategories-chart"
                                             data-view="PercentagePieView"
                                             data-options='{"fillColor":"#9966FF","emptyColor": "#E4E4E4", "fillValue": 0.039149419576466, "innerSize": "70%"}'>
                                        </div>
                                        <div class='audienceCategories-image categories-sprite yellow shopping'></div>
                                    </div>
                                    <h3 class="audienceCategories-itemTitle" itemprop="url">
                                        <a itemprop="url" class="audienceCategories-itemLink" href="/top-websites/category/shopping">Shopping</a>
                                    </h3>
                                </li>

Ainsi je voudrais récupérer la valeur de "fillValue" soit 0.039149419576466, auriez vous une idée ?

J'ai ainsi tenté cela, mais je bloque pour récupérer la valeur

Code :

Sélectionner tout - Visualiser dans une fenêtre à part

1
2
 
html.find_all('div', class_='audienceCategories-chart')

Merci de votre aide

!

**lg_53** · 08/06/2018, 15h48

Une piste peut-être de ce côté là :

https://stackoverflow.com/questions/...beautiful-soup

**VinsS** · 09/06/2018, 08h14

Salut,

Personnellement, je trouve le module HTMLParser facile d'usage.

Exemple:

Code :

Sélectionner tout - Visualiser dans une fenêtre à part

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
 
from html.parser import HTMLParser
html = """<li class="audienceCategories-item fadeInLeft">
            <div class="audienceCategories-chartContainer">
                <div class="audienceCategories-chart"
                     data-view="PercentagePieView"
                     data-options='{"fillColor":"#9966FF","emptyColor": "#E4E4E4", "fillValue": 0.039149419576466, "innerSize": "70%"}'>
                </div>
                <div class='audienceCategories-image categories-sprite yellow shopping'></div>
            </div>
            <h3 class="audienceCategories-itemTitle" itemprop="url">
                <a itemprop="url" class="audienceCategories-itemLink" href="/top-websites/category/shopping">Shopping</a>
            </h3>
        </li>"""
 
 
class Parser(HTMLParser):
    def __init__(self):
        super().__init__()
 
    def handle_starttag(self, tag, attrs):
        if tag == "div":
            if attrs[0][1] == "audienceCategories-chart":
                for att in attrs:
                    if att[0] == "data-options":
                        print(eval(att[1]))
 
p = Parser()
p.feed(html)

Si tu utilises eval tu as un dictionnaire, sinon une string.
Et la doc contient pas mal d'exemple.

**alexcot13** · 09/06/2018, 16h29

Merci beaucoup, vous être trop fort

du coup j'ai opté pour ce code :

Code :

Sélectionner tout - Visualiser dans une fenêtre à part

1
2
3
4
5
6
 
data_opt = []
for link in html.select(".audienceCategories-chartContainer"):
    dataa = json.loads(link.select_one("div.audienceCategories-chart")["data-options"])
    data_opt.append(dataa['fillValue'])
data_optionn = [i*100 for i in data_optionn]

J'ai une autre requête. Les données que je vise sont dans des balises <script> <\script> comme cela :

Code :

Sélectionner tout - Visualiser dans une fenêtre à part

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
 
<script>
            Similar.isPrintable = false;
    </script>
    <script src="/scripts/bundles/result-websites.js?version=20180606.6426"></script>
    <script>
        $(function() {
          // Enable external SVG sprite for IE
          svg4everybody();
 
          // lazy loader
          var blankFaviconSrc = '/images/blank-favicon.ico',
              blankSiteThumbSrc = '/images/unknown-medium.png';
 
 
                    $(".lazy-icon").lazyload({
                        effect: "fadeIn",
                        blankImageSrc: blankFaviconSrc,
                        threshold: 500
                    });
 
                    $(".lazy-thumb, .lazy-app-icon").lazyload({
                        effect: "fadeIn",
                        blankImageSrc: blankSiteThumbSrc,
                        //siteInfo: true,
                        threshold: 500
                    });
 
        });
        Sw.preloadedData = {"overview":{"Date":"2018-04-01T00:00:00","Country":250,"AdNetworks":{"Count":6,"Data":[["Google Display Network","",0.70583302880229737,0],["Tradedoubler","",0.25264394343533325,0],["Yandex Direct","",0.03237774158074963,0],["adfox.ru","",0.0060178007698869045,0],["Medialand","",0.0030947524490682232,0],["Other","",3.2732962664594467E-05,0]]},"TrafficSources":{"Direct":0.18588417750304617,"Referrals":0.044630370509833914,"Search":0.69920947546122936,"Social":0.037099118962603338,"Mail":0.0068636831103486831,"Paid Referrals":0.026313174452938671},"IsVerifiedData":false,"Icon":"https://site-images.similarcdn.com/image?url=guerlain.com&t=2&s=1&h=74049590dd4eac2542931448efb7844ff734cf5e275837c533e6d7fd337ff40c","TopCountryShares":[[250.0,0.25049211699693219,-0.15476817023050676],[840.0,0.1007741102771063,-0.086781325349847921],[392.0,0.080145596669938687,0.23828507836564564],[826.0,0.063314059827406746,0.23761223129975512],[643.0,0.058456992705951796,-0.46763826384634044]],"RedirectUrl":"guerlain.com","Category":"Shopping/Clothing","GlobalRank":[82815,0,-8157,0],"CategoryRank":[1282,0,-162,0],"IsSiteBigAndGaDiscrepancyHigh":false,"EngagementsGA":null,"EngagementsSimilarweb":{"BounceRate":"43.97%","PageViews":3.4957902501556237,"TimeOnSite":"00:02:15","TotalLastMonthVisits":655308,"TotalRelativeChange":-0.1452073234997939,"LastEngagementYear":2018,"LastEngagementMonth":4,"WeeklyTrafficNumbers":{"2017-11-01":564857,"2017-12-01":643055,"2018-01-01":417986,"2018-02-01":501021,"2018-03-01":766628,"2018-04-01":655308}},"CountryRanks":{"250":[10724,0,181,0]},"Referrals":{"destination":[{"Site":"tpeweb.paybox.com","Value":0.2141970227378692,"Change":0.018029933154060734},{"Site":"shop.nordstrom.com","Value":0.16022658535731346,"Change":3.4300390893087136},{"Site":"saksfifthavenue.com","Value":0.14299887909840736,"Change":0.0678132322108249},{"Site":"shop.rivegauche.ru","Value":0.10378863632841349,"Change":0.62180307853507488},{"Site":"sephora.com","Value":0.085284521046159162,"Change":0.58436907573801056}],"referrals":[{"Site":"video.mediaset.it","Value":0.298191762423262,"Change":0.0},{"Site":"search.nifty.com","Value":0.11281601191499548,"Change":0.0},{"Site":"database.az","Value":0.11025134981095225,"Change":0.0},{"Site":"foren.germany.ru","Value":0.072325053147313176,"Change":0.0},{"Site":"guerlain.co.jp","Value":0.061536006499088441,"Change":0.0}]},"IsSmallSite":false,"SimilarSites":[{"Site":"en.wikipedia.org","AltImage":"en.wikipedia.org","ImageUrl":"/images/unknown-medium.png","IconUrl":"https://site-images.similarcdn.com/image?url=wikipedia.org&t=2&s=1&h=460eb7e23fbc3666abf9d8b5497c3ca33d1de8ff9c66dd5d182c7633fe475c61","EllipsiseName":"en.wikipedia.org","CapitalizedSite":null,"Rank":0,"IsBlackListed":false},{"Site":"chanel.com","AltImage":"Chanel","ImageUrl":"https://site-images.similarcdn.com/image?url=chanel.com&t=0&s=1&h=2607c7f6d9b7cbf54e163151599ab6ebd6e5687867fa18f508fb844b7a12e827","IconUrl":"https://site-images.similarcdn.com/image?url=chanel.com&t=2&s=1&h=2607c7f6d9b7cbf54e163151599ab6ebd6e5687867fa18f508fb844b7a12e827","EllipsiseName":"chanel.com","CapitalizedSite":null,"Rank":9831,"IsBlackListed":false},{"Site":"sephora.com","AltImage":"Sephora","ImageUrl":"https://site-images.similarcdn.com/image?url=sephora.com&t=0&s=1&h=7cfc927b2425d8520d1d5d10ed2ce06dfb275c7e464005a027c83777615de06f","IconUrl":"https://site-images.similarcdn.com/image?url=sephora.com&t=2&s=1&h=7cfc927b2425d8520d1d5d10ed2ce06dfb275c7e464005a027c83777615de06f","EllipsiseName":"sephora.com","CapitalizedSite":null,"Rank":1213,"IsBlackListed":false},{"Site":"ysl.com","AltImage":"Ysl","ImageUrl":"https://site-images.similarcdn.com/image?url=ysl.com&t=0&s=1&h=019af00a61800c01be0c883205e88016d93bf3a2dafbcf2980f57925d7f96596","IconUrl":"https://site-images.similarcdn.com/image?url=ysl.com&t=2&s=1&h=019af00a61800c01be0c883205e88016d93bf3a2dafbcf2980f57925d7f96596","EllipsiseName":"ysl.com","CapitalizedSite":null,"Rank":31683,"IsBlackListed":false},{"Site":"esteelauder.com","AltImage":"Esteelauder","ImageUrl":"https://site-images.similarcdn.com/image?url=esteelauder.com&t=0&s=1&h=d8bac4b7a075dac42e2bd315ac3a9d3392e0e7ca5853c5dd71b6a633e38fa5a7","IconUrl":"https://site-images.similarcdn.com/image?url=esteelauder.com&t=2&s=1&h=d8bac4b7a075dac42e2bd315ac3a9d3392e0e7ca5853c5dd71b6a633e38fa5a7","EllipsiseName":"esteelauder.com","CapitalizedSite":null,"Rank":43869,"IsBlackListed":false},{"Site":"shop.nordstrom.com","AltImage":"Shop.Nordstrom","ImageUrl":"https://site-images.similarcdn.com/image?url=shop.nordstrom.com&t=0&s=1&h=45b437d149670dea7479dab1f9c31572e980ff8d854e493ecef6b53629976466","IconUrl":"https://site-images.similarcdn.com/image?url=shop.nordstrom.com&t=2&s=1&h=45b437d149670dea7479dab1f9c31572e980ff8d854e493ecef6b53629976466","EllipsiseName":"shop.nordstrom.com","CapitalizedSite":null,"Rank":1321,"IsBlackListed":false},{"Site":"fragrantica.com","AltImage":"Fragrantica","ImageUrl":"https://site-images.similarcdn.com/image?url=fragrantica.com&t=0&s=1&h=facc15dc40916ee53070efa1fa4474b41b66ff49f97858d4a3df677bc809b1ac","IconUrl":"https://site-images.similarcdn.com/image?url=fragrantica.com&t=2&s=1&h=facc15dc40916ee53070efa1fa4474b41b66ff49f97858d4a3df677bc809b1ac","EllipsiseName":"fragrantica.com","CapitalizedSite":null,"Rank":7817,"IsBlackListed":false},{"Site":"cosmetiq.ru","AltImage":"cosmetiq.ru","ImageUrl":"https://site-images.similarcdn.com/image?url=cosmetiq.ru&t=0&s=1&h=16f0f5cdb19369de75c8f3f2d47f06fae32c65d5cc63656d0f3a67e2bc7c4056","IconUrl":"https://site-images.similarcdn.com/image?url=cosmetiq.ru&t=2&s=1&h=16f0f5cdb19369de75c8f3f2d47f06fae32c65d5cc63656d0f3a67e2bc7c4056","EllipsiseName":"cosmetiq.ru","CapitalizedSite":null,"Rank":1674895,"IsBlackListed":false},{"Site":"dior.com","AltImage":"Dior","ImageUrl":"https://site-images.similarcdn.com/image?url=dior.com&t=0&s=1&h=53d61c7c3dfef12350287c65cd58153aa17c2b99d1373a1788c310507553ba5f","IconUrl":"https://site-images.similarcdn.com/image?url=dior.com&t=2&s=1&h=53d61c7c3dfef12350287c65cd58153aa17c2b99d1373a1788c310507553ba5f","EllipsiseName":"dior.com","CapitalizedSite":null,"Rank":19535,"IsBlackListed":false},{"Site":"givenchy.com","AltImage":"Givenchy","ImageUrl":"https://site-images.similarcdn.com/image?url=givenchy.com&t=0&s=1&h=3cb54e010bd0ec2bdcab67bb357b3a6a1658ae163a348d4bd017f18d72e80fb6","IconUrl":"https://site-images.similarcdn.com/image?url=givenchy.com&t=2&s=1&h=3cb54e010bd0ec2bdcab67bb357b3a6a1658ae163a348d4bd017f18d72e80fb6","EllipsiseName":"givenchy.com","CapitalizedSite":null,"Rank":79852,"IsBlackListed":false}]}};
        Sw.period = {"Month":9,"Year":2017,"period":12};
        Sw.siteDomain = "guerlain.com";
        Sw.siteCategory = "Shopping/Clothing";
        Sw.siteCountry = 250;
        Similar.extensionLink = "https://chrome.google.com/webstore/detail/hoklmmgfnpapgjgcpechhaamimifchmp";
        Similar.currentBrowser = "chrome";
 
        var dataLayer = dataLayer || [];
        dataLayer.push({'event':'MetaDesc','metadescription':'Shopping/Clothing'});
    </script>

Je voudrais les données de "AdNetworks" dans "Sw.preloadedData", j'ai google-isé ma question et suis tombé sur cet article https://stackoverflow.com/questions/...active#tab-top

je l'ai appliqué comme cela à mon code :

Code :

Sélectionner tout - Visualiser dans une fenêtre à part

1
2
3
4
5
6
7
 
import re, json
htmll=html.text
scriptline = next((line for line in htmll.splitlines()
    if 'Sw.preloadedData["overview"]' in line))
data = scriptline.split('=', 1)[1].strip(' ;')
data = json.loads(data)

(htmll en str et html en bs4.BeautifulSoup) mais il y a une erreur :

---------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
<ipython-input-179-e5895686a3cc> in <module>()
1 import re, json
2 htmll=html.text
----> 3 scriptline = next((line for line in htmll.splitlines()
4 if 'Sw.preloadedData["overview"]' in line))
5 data = scriptline.split('=', 1)[1].strip(' ;')

StopIteration:

sans plus d'indication, peut être trop long le fichier texte ?

Auriez-vous une idée ? Merciiiiiiiiiiii vous êtes géniaux

**lg_53** · 11/06/2018, 08h42

Probablement que Python n'a trouvé aucun élément vérifiant le critère. Si c'est bien cela ce qu'il y a dans le next est vide.

Et on peut le vérifier assez facilement :

Code :

Sélectionner tout - Visualiser dans une fenêtre à part

1
2
scriptlines = [ line for line in htmll.splitlines() if 'Sw.preloadedData["overview"]' in line ]
print(len(scriptlines))

Surtout que dans le code html que tu fournit, moi je vois :

Code :

Sélectionner tout - Visualiser dans une fenêtre à part

Sw.preloadedData = {"overview" : ....

qui n'est donc pas la même chose.

**alexcot13** · 11/06/2018, 09h04

Oh oui, je n'avais pas les yeux en face des trous... Merci beaucoup en tous cas, il fallait juste faire :

Code :

Sélectionner tout - Visualiser dans une fenêtre à part

1
2
3
4
5
6
7
 
htmll=html.text
scriptline = next((line for line in htmll.splitlines()
    if 'Sw.preloadedData' in line))
data = scriptline.split('=', 1)[1].strip(' ;')
data = json.loads(data)
data['overview']['AdNetworks']['Data']

Merci de votre patiente en tous cas

Bonne journée

Scraping HTML - données dans une balise

Python

Vue hybride

Discussions similaires

Partager

Partager