Skip to main content

​

Dexter is a comprehensive synthetic client designed to scrape various social media platforms for user data and interlinking data, enabling the construction of synthetic user models. This crate provides a robust set of tools and modules to facilitate data collection, parsing, and analysis.

​
Overview

Dexter aims to streamline the process of gathering and managing social media data. It supports multiple social media platforms and offers a unified interface for interacting with them. The crate is modular, allowing users to pick and choose the components they need for their specific use case.

​
Modules

  • api: Contains the API endpoints for Dexter.
  • builder: Provides utilities for building requests and other components.
  • data: Manages data structures and storage.
  • monitoring: Includes tools for monitoring and error reporting.
  • parser: Offers utilities for parsing and handling data, including URL handling and regex pattern matching.
  • providers: Defines and implements various social media providers.
  • proxy: Manages proxy settings and configurations.
  • request: Handles the construction and execution of HTTP requests.
  • user: Manages user-related data and operations.
  • utils: Provides utility functions and helpers.
  • uuid: Handles UUID generation and management.
  • username_availability: Checks the availability of usernames across different platforms.
  • connections: Manages connections and relationships between users.
  • returns: Defines return types and error handling mechanisms.

​
Usage

Dexter can be used to create clients and users, scrape data from various social media platforms, and analyze the collected data. Below are some of the primary structures and their usage.

​
DexterClient

The DexterClient struct is used to create a new client for Dexter 
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct DexterClient {
    pub username: String,
    pub provider: Provider,
}

​
DexterUser

The DexterUser struct is used to create a new user in the database. It includes various fields for storing user data and social media usernames. 
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct DexterUser {
    pub user_id: String,
    pub username: String,
    pub username_instagram: Option<String>,
    pub username_snapchat: Option<String>,
    pub username_tiktok: Option<String>,
    pub username_twitter: Option<String>,
    pub username_facebook: Option<String>,
    pub username_linkedin: Option<String>,
    pub username_pinterest: Option<String>,
    pub username_youtube: Option<String>,
    pub username_twitch: Option<String>,
    pub username_reddit: Option<String>,
    pub username_github: Option<String>,
    pub username_vk: Option<String>,
    pub username_weibo: Option<String>,
    pub username_wechat: Option<String>,
    pub username_etsy: Option<String>,
    pub username_ebay: Option<String>,
    pub username_discord: Option<String>,
    pub email: Option<String>,
    pub phone: Option<String>,
    pub full_name: Option<String>,
    pub biography: Option<String>,
    pub external_url: Option<String>,
    pub profile_pic_url: Option<String>,
}

​
InstagramUser

The InstagramUser struct is used to create a new user in the database specifically for Instagram. 
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct InstagramUser {
    pub username: String,
    pub full_name: String,
    pub biography: String,
    pub external_url: Value,
    pub profile_pic_url: String,
    pub posts: u64,
    pub followers: u64,
    pub following: u64,
    pub is_private: Option<bool>,
    pub is_verified: Option<bool>,
    pub is_business_account: Option<bool>,
    pub status: Option<String>,
}

​
Provider

The Provider enum defines the various data platforms supported by Dexter 
#[derive(Clone, Debug, Serialize, Deserialize)]
pub enum Provider {
    Piokok,
    Instagram,
    Snapchat,
    TikTok,
    Twitter,
    Facebook,
    LinkedIn,
    Pinterest,
    YouTube,
    Twitch,
    Reddit,
    GitHub,
    Vk,
    Weibo,
    WeChat,
    Etsy,
    Discord,
    Vsco,
    PumpFun // WIP
}
Modules Links
apiAPI Module
builderURL Builder Module
connectionsConnections Module
dataData controller module
logCreate User Module
monitoringMonitoring Module
parserThis module provides various utilities for parsing and handling data. It includes submodules for URL handling and regex pattern matching.
providersThis module contains the definitions and implementations for various social media providers. It includes functionality to convert providers to string representations and to get the column names for database storage.
proxyProxy module
requestRequest builder
returnsReturns Module
user User Module
utilsUtils Module