twarc-videos

此twarc插件使用 youtube_dl 从推文中下载视频及其元数据。这很好，因为youtube_dl可以下载比YouTube更多的平台上的视频，包括Twitter本身。

要使用twarc-videos，首先需要安装它

pip install twarc-videos

现在您可以使用twarc核心工具收集数据。例如，此搜索找到提到单词“nirvana”的推文，并且具有本地视频（Twitter视频）或指向YouTube的链接

twarc2 search 'nirvana (has:videos OR url:"https://youtu.be")' > nirvana-tweets.jsonl

您还有一个由twarc-videos提供的新的子命令 videos

twarc2 videos nirvana-tweets.jsonl

完成后，您将拥有一个新的 videos 目录，外观类似

videos
├── archive.txt
├── mapping.tsv
├── twitter
│   ├── 1339223561731530753
│   │   ├── Psychedelia_-_Nirvana_-_Come_As_You_Are.description
│   │   ├── Psychedelia_-_Nirvana_-_Come_As_You_Are.info.json
│   │   └── Psychedelia_-_Nirvana_-_Come_As_You_Are.mp4
│   ├── 1341668409428353025
│   │   ├── Rt_Your_Fav_Bands_-_Nirvana_Come_As_You_Are.description
│   │   ├── Rt_Your_Fav_Bands_-_Nirvana_Come_As_You_Are.info.json
│   │   └── Rt_Your_Fav_Bands_-_Nirvana_Come_As_You_Are.mp4
│   ├── 1374212180002926594
│   │   ├── Hanna_-_She_s_in_Nirvana....description
│   │   ├── Hanna_-_She_s_in_Nirvana....info.json
│   │   └── Hanna_-_She_s_in_Nirvana....mp4
│   ├── 1374467789885378569
│   │   ├── MUSIC_NOSTALGIA_-_Nirvana_The_Man_Who_Sold_The_World_..description
│   │   ├── MUSIC_NOSTALGIA_-_Nirvana_The_Man_Who_Sold_The_World_..info.json
│   │   └── MUSIC_NOSTALGIA_-_Nirvana_The_Man_Who_Sold_The_World_..mp4
│   ├── 1374469206226264067
│   │   ├── Take_it_easy_-_Abuelo_donde_andas_Nirvana.description
│   │   ├── Take_it_easy_-_Abuelo_donde_andas_Nirvana.info.json
│   │   └── Take_it_easy_-_Abuelo_donde_andas_Nirvana.mp4
│   ├── 1374631023502360576
│   │   ├── OraEtLabora_-_Reel_Stories_-_Dave_Grohl_is_on_@bbctwo_this_Saturday_at_10.30pm...talking_@Nirvana_amp_@foofighters_with_Dermot_@radioleary_@wearecraftuk.description
│   │   ├── OraEtLabora_-_Reel_Stories_-_Dave_Grohl_is_on_@bbctwo_this_Saturday_at_10.30pm...talking_@Nirvana_amp_@foofighters_with_Dermot_@radioleary_@wearecraftuk.info.json
│   │   └── OraEtLabora_-_Reel_Stories_-_Dave_Grohl_is_on_@bbctwo_this_Saturday_at_10.30pm...talking_@Nirvana_amp_@foofighters_with_Dermot_@radioleary_@wearecraftuk.mp4
│   ├── 1374656171844329477
│   ├── 1374656880694292483
│   ├── 1374660019241762817
│   ├── 1374664809078272000
│   └── 1374671562016661506
│       ├── John_-_Nirvana_-_In_Bloom_Live_at_Reading_1992_@YouTube.description
│       ├── John_-_Nirvana_-_In_Bloom_Live_at_Reading_1992_@YouTube.info.json
│       └── John_-_Nirvana_-_In_Bloom_Live_at_Reading_1992_@YouTube.mp4
└── youtube
    ├── 5X9CGFQyjN4
    │   ├── Heart-Shaped_Box_Nirvana_Music_Box.description
    │   ├── Heart-Shaped_Box_Nirvana_Music_Box.en.vtt
    │   ├── Heart-Shaped_Box_Nirvana_Music_Box.info.json
    │   └── Heart-Shaped_Box_Nirvana_Music_Box.mp4
    ├── AhcttcXcRYY
    │   ├── Nirvana_-_About_A_Girl_MTV_Unplugged.description
    │   ├── Nirvana_-_About_A_Girl_MTV_Unplugged.en.vtt
    │   ├── Nirvana_-_About_A_Girl_MTV_Unplugged.info.json
    │   └── Nirvana_-_About_A_Girl_MTV_Unplugged.mp4
    ├── AXU-LaaO_xQ
    │   ├── Nirvana_Drain_You_lyrics_sub_espanol.description
    │   ├── Nirvana_Drain_You_lyrics_sub_espanol.info.json
    │   └── Nirvana_Drain_You_lyrics_sub_espanol.mp4
    ├── D742dNm1f8Q
    │   ├── Nirvana_-_In_Bloom_Live_at_Reading_1992.description
    │   ├── Nirvana_-_In_Bloom_Live_at_Reading_1992.info.json
    │   └── Nirvana_-_In_Bloom_Live_at_Reading_1992.mp4
    ├── -fh-bqSV73E
    │   ├── Becoming_a_minimalist_w_Matt_D_Avella.description
    │   ├── Becoming_a_minimalist_w_Matt_D_Avella.en.vtt
    │   ├── Becoming_a_minimalist_w_Matt_D_Avella.info.json
    │   └── Becoming_a_minimalist_w_Matt_D_Avella.mp4
    ├── hTWKbfoikeg
    │   ├── Nirvana_-_Smells_Like_Teen_Spirit_Official_Music_Video.description
    │   ├── Nirvana_-_Smells_Like_Teen_Spirit_Official_Music_Video.en.vtt
    │   ├── Nirvana_-_Smells_Like_Teen_Spirit_Official_Music_Video.info.json
    │   └── Nirvana_-_Smells_Like_Teen_Spirit_Official_Music_Video.mp4
    ├── jWkSt4G8F18
    │   ├── Nirvana_healing_centre_overview.description
    │   ├── Nirvana_healing_centre_overview.info.json
    │   └── Nirvana_healing_centre_overview.mp4
    ├── MW6E_TNgCsY
    │   ├── Everclear_-_Santa_Monica_Official_Music_Video.description
    │   ├── Everclear_-_Santa_Monica_Official_Music_Video.info.json
    │   └── Everclear_-_Santa_Monica_Official_Music_Video.mp4
    ├── n6P0SitRwy8
    │   ├── Nirvana_-_Heart-Shaped_Box.description
    │   ├── Nirvana_-_Heart-Shaped_Box.info.json
    │   └── Nirvana_-_Heart-Shaped_Box.mp4
    ├── OgeR2oqZGTs
    │   ├── Nirvana_-_The_Man_Who_Sold_The_World_Live_On_MTV_Unplugged_1993_Unedited.description
    │   ├── Nirvana_-_The_Man_Who_Sold_The_World_Live_On_MTV_Unplugged_1993_Unedited.en.vtt
    │   ├── Nirvana_-_The_Man_Who_Sold_The_World_Live_On_MTV_Unplugged_1993_Unedited.info.json
    │   └── Nirvana_-_The_Man_Who_Sold_The_World_Live_On_MTV_Unplugged_1993_Unedited.mp4
    ├── v9RY25eImcw
    │   ├── Nirvana_-_Smells_Like_Teen_Spirit_Cover_RADIO_TAPOK.description
    │   ├── Nirvana_-_Smells_Like_Teen_Spirit_Cover_RADIO_TAPOK.en.vtt
    │   ├── Nirvana_-_Smells_Like_Teen_Spirit_Cover_RADIO_TAPOK.info.json
    │   └── Nirvana_-_Smells_Like_Teen_Spirit_Cover_RADIO_TAPOK.mp4
    ├── ycHvL3W3_PA
    │   ├── Nirvana_-_Where_Did_You_Sleep_Last_Night_8D_Audio.description
    │   ├── Nirvana_-_Where_Did_You_Sleep_Last_Night_8D_Audio.info.json
    │   └── Nirvana_-_Where_Did_You_Sleep_Last_Night_8D_Audio.mp4
    └── y-lQgqHD8Xs
        ├── dodo_tofubeats_-_nirvana_Official_Music_Video.description
        ├── dodo_tofubeats_-_nirvana_Official_Music_Video.info.json
        └── dodo_tofubeats_-_nirvana_Official_Music_Video.mp4

video/mapping.tsv 文件是找到的视频URL及其在磁盘中的对应位置的制表符分隔值文件。

测试

要运行测试，您需要创建一个类似于的 .env 文件

BEARER_TOKEN=YOUR_TOKEN_HERE

然后

python setup.py test